extension-prometheus

Steadybit extension-prometheus

A Steadybit check implementation to gather Prometheus metrics within chaos engineering experiment executions. These can be used as checks within experiments, e.g., to implement pre- and post-conditions.

Learn about the capabilities of this extension in our Reliability Hub.

Configuration

The extension supports all environment variables provided by steadybit/extension-kit.

Installation

Kubernetes

Detailed information about agent and extension installation in kubernetes can also be found in our documentation.

Recommended (via agent helm chart)

All extensions provide a helm chart that is also integrated in the helm-chart of the agent.

You must provide additional values to activate this extension.

--set extension-prometheus.enabled=true \
--set extension-prometheus.prometheus.name="dev" \
--set extension-prometheus.prometheus.origin="http://prometheus-server.default.svc.cluster.local" \

Additional configuration options can be found in the helm-chart of the extension.

Alternative (via own helm chart)

If you need more control, you can install the extension via its dedicated helm-chart.

helm repo add steadybit-extension-prometheus https://steadybit.github.io/extension-prometheus
helm repo update
helm upgrade steadybit-extension-prometheus \
  --install \
  --wait \
  --timeout 5m0s \
  --create-namespace \
  --namespace steadybit-agent \
  --set prometheus.name="dev" \
  --set prometheus.origin="http://prometheus-server.default.svc.cluster.local" \
  steadybit-extension-prometheus/steadybit-extension-prometheus

Openshift Prometheus behind Thanos

To be able to reach prometheus through thanos querier on prometheus, please allow the service account from the extension to fetch metrics:

oc adm policy add-cluster-role-to-user cluster-monitoring-view -z steadybit-extension-prometheus -n steadybit-agent

Then you need to get a dedicate token for the extension, the rotation of the token is your responsability:

oc create token steadybit-extension-prometheus --namespace steadybit-agent

You can now pass the authorization token via helm values:

prometheus:
  headerKey: Authorization
  headerValue: Bearer ${YOUR_GENERATED_TOKEN}

This will create a kubernetes secret with header key and value, then use it for environment variable of the extension deployment.

Linux Package

Please use our agent-linux.sh script to install the extension on your Linux machine. The script will download the latest version of the extension and install it using the package manager.

After installing, configure the extension by editing /etc/steadybit/extension-prometheus and then restart the service.

Extension registration

Make sure that the extension is registered with the agent. In most cases this is done automatically. Please refer to the documentation for more information about extension registration and how to verify.

Version and Revision

The version and revision of the extension:

• are printed during the startup of the extension
• are added as a Docker label to the image
• are available via the version.txt/revision.txt files in the root of the image

Importing your own certificates

You may want to import your own certificates. Mount a volume with the certificates and reference it in extraVolumeMounts and extraVolumes in the helm chart.

This example uses a config map to store the *.crt-files in a configmap:

kubectl create configmap -n steadybit-agent prometheus-self-signed-ca --from-file=./self-signed-ca.crt

extraVolumeMounts:
	- name: extra-certs
		mountPath: /etc/ssl/extra-certs
		readOnly: true
extraVolumes:
	- name: extra-certs
		configMap:
			name: prometheus-self-signed-ca
extraEnv:
	- name: SSL_CERT_DIR
		value: /etc/ssl/extra-certs:/etc/ssl/certs

FAQ

I don't see metrics from my VictoriaMetrics instance

By default, Victoria Metrics does not immediately return the recently written samples. You can set the latency_offset parameter to 1 to disable this behavior. This can be done by setting the environment variable STEADYBIT_EXTENSION_ADDITIONAL_REQUEST_PARAMS to latency_offset,1.