trace

Trace Package

The trace package provides a thin, ergonomic wrapper around OpenTelemetry (OTEL) distributed tracing for GitLab Go services.

Engineers in GitLab must use this package to instrument their services so that trace data is emitted consistently across all systems.

Quick Start

tracer, shutdown, err := trace.New(ctx)
if err != nil {
    log.Fatal(err)
}
defer shutdown(ctx)

ctx, span := tracer.Start(ctx, "my-operation")
defer span.End()

span.SetAttribute("user.id", userID)

New exports spans via OTLP/HTTP to localhost:4318 by default. The collector endpoint and other settings can be configured without code changes via the GITLAB_TRACING environment variable (see below) or via Config.

Environment Variables

The recommended way to configure tracing in cloud-native deployments is via the GITLAB_TRACING environment variable. It accepts a connection string that controls the collector endpoint, TLS, service name, and sampling in a single value.

Connection string format

<scheme>://<host>:<port>[?param=value&...]

Examples

# Insecure local collector, 10% sampling
GITLAB_TRACING="otlp://localhost:4318?sampler=probabilistic&sampler_param=0.1"

# TLS-enabled remote collector, always sample, custom service name
GITLAB_TRACING="otlps://collector.example.com:4317?service_name=gitaly&sampler=const&sampler_param=1"

In Kubernetes:

env:
  - name: GITLAB_TRACING
    value: "otlps://otel-collector.observability:4317?service_name=my-service&sampler=probabilistic&sampler_param=0.05"

Precedence

Values set in Config take precedence over GITLAB_TRACING. The env var fills in only fields that are left at their zero value in the Config struct, so you can mix both: use GITLAB_TRACING for the endpoint and set SampleRate programmatically, for example.

Programmatic Configuration

tracer, shutdown, err := trace.NewWithConfig(ctx, &trace.Config{
    // ServiceName is included as the service.name resource attribute on every span.
    // Defaults to the binary name when empty.
    ServiceName: "gitaly",

    // ServiceVersion is included as the service.version resource attribute.
    ServiceVersion: "16.0.0",

    // Endpoint is the base URL of the OTLP HTTP collector (no path suffix).
    Endpoint: "https://collector.example.com:4318",

    // Insecure disables TLS. Useful for local development.
    Insecure: true,

    // Headers are sent with every export request, e.g. for authentication.
    Headers: map[string]string{"x-honeycomb-team": token},

    // SampleRate controls the fraction of traces that are sampled [0.0, 1.0].
    // Zero value defaults to 1.0 (always sample).
    SampleRate: 0.1,

    // DisableGlobalProvider prevents registering the provider as the global
    // OTEL provider. Leave false (default) for services that use third-party
    // OTEL-instrumented libraries.
    DisableGlobalProvider: false,
})

To explicitly drop all spans (useful in tests that use the real provider):

tracer, shutdown, err := trace.NewWithConfig(ctx, &trace.Config{
    SampleRate: trace.SampleRateDropAll,
})

Creating Spans

Always pair Start with a deferred End. The returned context carries the active span — pass it to all downstream calls so child spans are nested correctly.

ctx, span := tracer.Start(ctx, "process-merge-request")
defer span.End()

// Annotate the span with typed attributes.
span.SetAttribute("mr.iid", mergeRequestIID)
span.SetAttribute("project.id", projectID)
span.SetAttribute("dry_run", false)

SetAttribute accepts Go primitives and their slice variants directly:

Any other type falls back to fmt.Sprintf("%v", value).

Recording Errors

result, err := doWork(ctx)
if err != nil {
    span.RecordError(err) // records the exception event AND sets status=Error
    return err
}

RecordError records the exception event and sets the span status to codes.Error. The raw OTEL Span.RecordError only records the event; this wrapper handles both in one call.

Retrieving the Active Span from Context

span := trace.SpanFromContext(ctx)
span.SetAttribute("queue.depth", depth)

SpanFromContext always returns a non-nil span. If no span is active in the context a no-op span is returned — all methods are safe to call on it.

Context Propagation

Child spans are automatically nested under the active span when the enriched context is passed downstream:

ctx, parent := tracer.Start(ctx, "handle-request")
defer parent.End()

// child is nested under parent in the trace.
ctx, child := tracer.Start(ctx, "query-database")
defer child.End()

Using the Full OpenTelemetry API

Span embeds oteltrace.Span, so any OTEL method not wrapped by labkit is accessible directly:

// Add a structured event.
span.AddEvent("cache-miss", oteltrace.WithAttributes(
    attribute.String("cache.key", key),
))

// Add a span link to a remote trace.
span.AddLink(oteltrace.Link{SpanContext: remoteSpanCtx})

// Retrieve the trace-id and span-id.
sc := span.SpanContext()
traceID := sc.TraceID().String()

Where labkit's helper method shadows an OTEL method with a different signature, reach the original via the embedded field:

// End with an explicit timestamp.
span.Span.End(oteltrace.WithTimestamp(endTime))

// RecordError with event options.
span.Span.RecordError(err, oteltrace.WithAttributes(
    attribute.String("error.source", "upstream"),
))

To pass the underlying tracer to a third-party OTEL-instrumented library:

oteltracer := tracer.OTELTracer() // returns oteltrace.Tracer
thirdPartyLib.Init(oteltracer)

Testing

Use tracetest.NewRecorder to capture completed spans in memory without a live collector. The recorder always samples and does not register a global provider, keeping tests isolated.

import "gitlab.com/gitlab-org/labkit/v2/testing/tracetest"

func TestMyHandler(t *testing.T) {
    tracer, rec := tracetest.NewRecorder()

    ctx, span := tracer.Start(context.Background(), "handle-request")
    span.SetAttribute("user.id", 42)
    span.RecordError(errors.New("something went wrong"))
    span.End()

    spans := rec.Ended()
    require.Len(t, spans, 1)

    s := spans[0]
    assert.Equal(t, "handle-request", s.Name)
    assert.Equal(t, int64(42), s.Attributes["user.id"])
    assert.True(t, s.HasError)
    assert.Equal(t, codes.Error, s.StatusCode)
    assert.Equal(t, "something went wrong", s.StatusMsg)
}

rec.Reset() clears recorded spans between sub-tests when a single recorder is shared across a table-driven test.