metrics

delivery-metrics

Delivery metrics is a service designed to handle delivery team's metrics. It implements a specialized Prometheus Pushgateway designed to be used from our CI jobs, as well as an API scraper for publishing metrics based on our pipelines.

It can easily keep track of histograms implementing an HTTP API around Prometheus metrics.

Architecture

delivery-metrics is a Go software exposing an HTTP endpoint, on the /metrics path we have a standard prometheus metrics, on the /api path we have custom handlers to add data to our metrics. API scraping is implemented in dedicated goroutines.

Deployment

delivery-metrics implements continous delivery using argocd-image-updater which automatically updates the deployment in ArgoCD when a new image is published to registry.ops.gitlab.net/gitlab-org/release/tools/delivery-metrics.

Useful links:

• Delivery-metrics chart: https://gitlab.com/gitlab-com/gl-infra/charts/-/tree/main/gitlab/delivery-metrics
• Delivery-metrics service in ArgoCD: https://gitlab.com/gitlab-com/gl-infra/argocd/apps/-/tree/main/services/delivery-metrics

Logs

Logs for Delivery-metrics can be found at https://console.cloud.google.com/kubernetes/deployment/us-east1/ops-gitlab-gke/delivery/delivery-metrics/logs?project=gitlab-ops&pli=1.

Access control

Push Gateway

The /api path requires a token to allow write operations. The token must be provided as an HTTP header named X-Private-Token.

The software compares the user provided token with the content of the AUTH_TOKEN environment variable.

API scraper

In order to perform API requests on ops.gitlab.net, delivery-metrics requires a read only API token stored into the DELIVERY_METRICS_OPS_TOKEN environment variable.

Working with histograms

It is possible to add values to an histogram making a POST request to the metric observe method. The request is form-encoded and requires 2 parameters:

• value: the observed value
• labels: a comma-separated list of label values. Label values are positional.

example

curl -X POST \
    -H "X-Private-Token: MYTEST" \
    -F value=18000 \
    -F "labels=coordinator_pipeline,success" \
    "http://127.0.0.1:2112/api/deployment_duration_seconds/observe"

Metric reset

It is possible to reset a metric by sending a DELETE request, without parameters, to the metric's endpoint.

example

curl -X DELETE \
    -H "X-Private-Token: MYTEST" \
    "http://127.0.0.1:2112/api/release_pressure"

Experiments

It is possible to add temporary experimental metrics under the experiments subsystem.

To add a new metrics, create a new go file in the package with an init function that appends the new handler to the pluggable slice, then add an acceptance test for the new metric.

When the experiment is concluded, simply remove the go file and the acceptance test.

The delivery_experiments_total counter is provided to track the number of experiments.

Example code to add a new metric (metrics/internal/experiments/foo.go):

package experiments

import (
	"gitlab.com/gitlab-org/release-tools/metrics/internal/handlers"
	"gitlab.com/gitlab-org/release-tools/metrics/internal/metrics"
	"gitlab.com/gitlab-org/release-tools/metrics/internal/metrics/labels"
)

func init() {
    fooCounter, err := metrics.NewCounterVec(
        metrics.WithName("foo_total"),
        metrics.WithSubsystem(subsystem),
        metrics.WithHelp("Number of foo"),
        metrics.WithLabel(labels.SuccessOrFailed("result")),
        metrics.WithCartesianProductLabelReset(),
    )
    if err != nil {
        panic(err)
    }

    pluggables = append(pluggables, handlers.NewCounter(fooCounter))
}

Acceptance tests

Delivery metrics has acceptance tests running in CI against the docker image compiled from the branch codebase.

It is possible to run those tests locally with ./scripts/delivery-metrics.sh build, ./scripts/delivery-metrics.sh run (from another shell), and ./scripts/delivery-metrics.sh acceptance-tests.

The above solution is designed for CI environments and will run on docker (or nerdctl), it is possible to run it locally without docker using the following commands from the metrics folder.

# start delivery-metrics and let it run
JOB_WEBHOOK_RELEASE_TOOLS_OPS_TOKEN="release-tools-token" JOB_WEBHOOK_DEPLOYER_OPS_TOKEN="deployer-token" AUTH_TOKEN="acceptance tests" go run .

# open another shell and run the acceptance tests
go test -tags=acceptance -v

Webhook Sources

Delivery-metrics receives job event webhooks from multiple projects for deployment pipeline metrics. These are all projects in which the deployment pipeline triggers downstream pipelines. Webhook events are received from the following projects:

• https://dev.gitlab.org/gitlab/charts/components/images
• https://dev.gitlab.org/gitlab/omnibus-gitlab
• https://ops.gitlab.net/gitlab-com/gl-infra/deployer
• https://ops.gitlab.net/gitlab-com/gl-infra/k8s-workloads/gitlab-com
• https://ops.gitlab.net/gitlab-org/quality/staging-canary
• https://ops.gitlab.net/gitlab-org/quality/staging
• https://ops.gitlab.net/gitlab-org/release/tools/

Adding new secrets/tokens as environment variables to delivery-metrics

Each project's webhook events will contain a token. Delivery-metrics will match the received token against the expected token. Expected tokens are stored in Vault, which are pulled into K8s.

The tokens need to be added to:

• Vault under https://vault.gitlab.net/ui/vault/secrets/k8s/list/ops-gitlab-gke/delivery/delivery-metrics-webhooks/
• ArgoCD: https://gitlab.com/gitlab-com/gl-infra/argocd/apps/-/blob/main/services/delivery-metrics/env/ops/clusters/ops-gitlab-gke/values-vault-secrets.yaml

Generating metrics using job event webhooks

job_webhook.go defines a handler for job event webhook requests. Requests are routed to different functions based on the project the request came from. It generates metrics by using the metrics package (for example metrics.NewCounterVec) and modifies them based on the data received from incoming requests. (see examples in the file). Example MR.

Storing metrics data in Consul

Metrics can be stored in the Ops Consul service to enable the metric value to be persisted through restarts. Storing metrics data in Consul allows Delivery-metrics to recover the data after a restart. Without this, every metric starts with a nil value after Delivery-metrics restarts, which can have negative consequences for some metrics.

Call the metrics.WithConsulMemory(consulHost, consulPort, logger) method when initializing a metric. This will store the metric's data in Consul and recover it on restart.

Consul Web UI

To have a look at current metric data in Consul:

1. glsh kube use-cluster ops
2. kubectl port-forward service/consul-gl-consul-expose-servers 8500:8500 -n consul
3. Open http://localhost:8500 in a browser

These steps are taken from the runbook for connecting to the production Consul service.

Running Consul locally

1. Install Consul - https://developer.hashicorp.com/consul/install

2. Run Consul locally with consul agent -dev

3. Modify the delivery-metrics.sh script.

   Specifically the line that runs the delivery-metrics image to add -consul-host 'localhost' and --network="host", so that the full line looks like:

   $docker run --rm --network="host" --name delivery_metrics -e JOB_WEBHOOK_RELEASE_TOOLS_OPS_TOKEN="release-tools-token" \
    -e JOB_WEBHOOK_DEPLOYER_OPS_TOKEN="deployer-token" -e AUTH_TOKEN="acceptance tests" -p "$PORT:2112" \
    "$image" -log-format color -consul-host 'localhost'
   

4. Run the following in a terminal:

   ./scripts/delivery-metrics.sh build
   ./scripts/delivery-metrics.sh run
   

5. Start a pry console with

   • DELIVERY_METRICS_TOKEN='acceptance tests'
   • DELIVERY_METRICS_URL='http://localhost:2112' and run the following:

   ReleaseTools::Metrics::Client.new.set('auto_deploy_environment_state', 100, labels: 'ready,gstg,cny')
   ReleaseTools::Metrics::Client.new.inc('auto_deploy_environment_state', labels: 'ready,gstg,cny')
   
   ReleaseTools::Metrics::Client.new.set('auto_deploy_environment_state', 10, labels: 'locked,gstg,cny')
   ReleaseTools::Metrics::Client.new.inc('auto_deploy_environment_state', labels: 'locked,gstg,cny')
   

6. Open http://localhost:2112/metrics in a browser to see if the metric value is reflected.

7. Run consul kv export 'group-delivery/' in a terminal to check that Consul contains the metric and its values.

8. Terminate ./scripts/delivery-metrics run and run it again. You should see a log line like the following when you run it again: INFO[0000] Successfully loaded metric data from Consul data="map[ready,gstg,cny:100]" metric=delivery_auto_deploy_environment_state

9. Open http://localhost:2112/metrics in a browser to see if the metric value is initialized correctly to its previous value. You can also check the logs when you ran ./scripts/delivery-metrics run.