metrics

metrics

Package metrics provides an isolated Prometheus registry that integrates with the LabKit v2 component lifecycle. There is no shared global state — each Metrics instance owns its own prometheus.Registry, so components and tests never interfere with each other.

Quick start

m, err := metrics.New()
if err != nil {
    log.Fatal(err)
}

a.Register(m) // Start registers go_ and process_ collectors; Shutdown is a no-op

// Mount alongside the existing health endpoints.
srv.Router().Get("/-/metrics", m.Handler())

Defining metrics

Use BuildName to prefix metric names with the configured namespace (default "gitlab"), then register with MustRegister:

requestsTotal := prometheus.NewCounterVec(prometheus.CounterOpts{
    Name: m.BuildName("http", "requests_total"),
    Help: "Total number of HTTP requests partitioned by feature category and status.",
}, []string{metrics.LabelFeatureCategory, metrics.LabelStatus})

m.MustRegister(requestsTotal)

// Increment on every request:
requestsTotal.WithLabelValues("code_review", "2xx").Inc()

BuildName(subsystem, name) follows the Prometheus convention of joining parts with underscores, omitting empty segments:

Standard labels

Use the label name constants to keep dashboards and alert rules consistent across services:

Standard bucket sets

DurationBuckets provides SLO-aligned histogram boundaries for HTTP and RPC latency (100ms to 60s). The five boundaries include 1 (satisfied threshold) and 10 (tolerated threshold) to match the Workhorse and Rails SLI thresholds used across GitLab's metrics catalog:

prometheus.NewHistogramVec(prometheus.HistogramOpts{
    Name:    m.BuildName("http", "request_duration_seconds"),
    Buckets: metrics.DurationBuckets,
}, []string{metrics.LabelEndpointID, metrics.LabelStatus})

Memory metrics

Start registers a cgroup memory collector, so every service reports the memory limit the kernel enforces on it alongside its current usage.

gitlab_memory_usage_bytes / gitlab_memory_limit_bytes{source="cgroup_max"}

The source label distinguishes three different quantities:

Always select on source. cgroup_high is routinely set below cgroup_max, so min(gitlab_memory_limit_bytes) silently switches to the soft limit, and sum(gitlab_memory_limit_bytes) double-counts.

Absent means unknown, never zero

When a memory value could not be determined, it is omitted rather than published as zero, so absent() alerting works and the ratio never divides by zero.

What the numbers refer to

The limit is the minimum over the process's cgroup and all its ancestors: a container's own cgroup routinely reads max while the pod cgroup carries the limit the kernel enforces. Usage, anonymous memory and events are read from the process's own cgroup, so in a multi-container pod the limit may be the pod's while usage is one container's, and utilisation will read low.

If the process's own cgroup carries no memory interface files, because its parent never enabled +memory in cgroup.subtree_control, everything is read from the nearest ancestor that does. Usage there may include sibling cgroups, but the alternative is falling back to MemTotal and reporting the machine's memory as the ceiling while an ancestor enforces something far smaller.

Note that applications that place child processes in their own cgroups — Gitaly's spawned git processes, for example — are not fully described by these metrics. LabKit reports the application's own cgroup; per-child accounting stays application-specific.

Every value is read on each scrape. Only the resolved cgroup path is cached, for 30 seconds, since resolution parses mountinfo but its answer changes only if the process is migrated between cgroups. So usage tracks the current scrape, and a limit rewritten by in-place pod vertical scaling (KEP-1287) or VPA is picked up immediately rather than reported stale.

cgroup v1 and non-Linux

The absence of the v1 series is stable rather than flapping, so rate() sees no counter resets — but a mixed v1/v2 fleet has a ragged series set.

Nothing is exported on macOS or Windows.

Set DisableMemoryCollector to opt out.

HTTP service example

The pattern below wires Metrics into an httpserver.Server. Metrics are defined once at construction time and incremented inside the handler closure:

func main() {
    ctx := context.Background()

    a, _ := app.New(ctx)

    m, _ := metrics.New()
    a.Register(m)

    // Define metrics before Start is called.
    requestsTotal := prometheus.NewCounterVec(prometheus.CounterOpts{
        Name: m.BuildName("http", "requests_total"),
        Help: "Total HTTP requests partitioned by endpoint and status.",
    }, []string{metrics.LabelEndpointID, metrics.LabelStatus})

    requestDuration := prometheus.NewHistogramVec(prometheus.HistogramOpts{
        Name:    m.BuildName("http", "request_duration_seconds"),
        Help:    "HTTP request latency.",
        Buckets: metrics.DurationBuckets,
    }, []string{metrics.LabelEndpointID})

    m.MustRegister(requestsTotal, requestDuration)

    srv := httpserver.NewWithConfig(&httpserver.Config{
        Addr:   ":8080",
        Tracer: a.Tracer(),
    })

    srv.Router().Get("/api/projects/{id}", func(w http.ResponseWriter, r *http.Request) {
        start := time.Now()

        // ... handler logic ...

        requestsTotal.WithLabelValues("GET /api/projects/{id}", "2xx").Inc()
        requestDuration.WithLabelValues("GET /api/projects/{id}").Observe(time.Since(start).Seconds())

        w.WriteHeader(http.StatusOK)
    })

    // Expose metrics at /-/metrics alongside /-/liveness and /-/readiness.
    srv.Router().Get("/-/metrics", m.Handler())

    a.Register(srv)
    a.Run(ctx)
}

Note: for high-traffic services consider using prometheus.NewHistogramVec with NativeHistogramBucketFactor instead of fixed Buckets to reduce cardinality and scrape payload size.

Configuration

m, err := metrics.NewWithConfig(&metrics.Config{
    Name:              "primary-metrics", // component name in logs   (default: "metrics")
    Namespace:         "myservice",       // metric name prefix       (default: "gitlab")
    EnableOpenMetrics: true,              // serve OpenMetrics/exemplars (default: false)

    DisableMemoryCollector: true,         // skip gitlab_memory_*      (default: false)
})

When writing tests, pass a fresh prometheus.Registry to isolate the metric state completely (see Testing below).

Sharing a registry

Pass an existing registry via Registry to serve LabKit metrics from an endpoint another library already exposes. If that library also registers Go runtime and process collectors, set DisableRuntimeCollectors so Start leaves them alone.

OpenMetrics and exemplars

By default Handler serves only the classic Prometheus text format. Set EnableOpenMetrics to also serve the OpenMetrics format to scrapers that negotiate it (Accept: application/openmetrics-text). This is required for exemplars: a histogram sample tagged with the active trace_id lets an operator jump from a latency spike straight to the distributed trace behind it. Scrapers that do not request OpenMetrics still receive the classic text format.

m, err := metrics.NewWithConfig(&metrics.Config{EnableOpenMetrics: true})
if err != nil {
    log.Fatal(err)
}

latency := prometheus.NewHistogram(prometheus.HistogramOpts{
    Name:    m.BuildName("http", "request_duration_seconds"),
    Buckets: metrics.DurationBuckets,
})
m.MustRegister(latency)

// Attach the current trace_id to the observation as an exemplar.
latency.(prometheus.ExemplarObserver).ObserveWithExemplar(
    time.Since(start).Seconds(),
    prometheus.Labels{"trace_id": traceID},
)

Injecting into sub-components

Pass only the prometheus.Registerer interface to components that should register their own collectors but not control the full Metrics instance. This limits the API surface and makes dependencies explicit:

type Worker struct {
    jobsTotal prometheus.Counter
}

func NewWorker(reg prometheus.Registerer, m *metrics.Metrics) (*Worker, error) {
    c := prometheus.NewCounter(prometheus.CounterOpts{
        Name: m.BuildName("worker", "jobs_total"),
        Help: "Total jobs processed.",
    })
    if err := reg.Register(c); err != nil {
        return nil, err
    }
    return &Worker{jobsTotal: c}, nil
}

// Wire up:
worker, err := NewWorker(m.Registerer(), m)
a.Register(worker)

Testing

Use metricstest.New for an isolated Metrics backed by a fresh registry. The component is not started — Go runtime and process collectors are absent, keeping gathered output minimal and assertions deterministic:

import "gitlab.com/gitlab-org/labkit/v2/testing/metricstest"

func TestWorker_countsJobs(t *testing.T) {
    m := metricstest.New(t)

    worker, err := NewWorker(m.Registerer(), m)
    require.NoError(t, err)
    require.NoError(t, worker.ProcessJob(ctx))

    families := metricstest.Gather(t, m)

    require.Contains(t, families, "gitlab_worker_jobs_total")
    assert.Equal(t, 1.0, families["gitlab_worker_jobs_total"].Metric[0].Counter.GetValue())
}

metricstest.Gather returns a map[string]*dto.MetricFamily keyed by fully-qualified metric name, so you can assert on counters, gauges, and histogram observations without a running Prometheus server.