trace

Trace Package

The trace package provides a thin, ergonomic wrapper around OpenTelemetry (OTEL) distributed tracing for GitLab Go services.

Engineers in GitLab must use this package to instrument their services so that trace data is emitted consistently across all systems.

Quick Start

tracer, shutdown, err := trace.New(ctx)
if err != nil {
    log.Fatal(err)
}
defer shutdown(ctx)

ctx, span := tracer.Start(ctx, "my-operation")
defer span.End()

span.SetAttributes(attribute.String("user.id", userID))

New exports spans via OTLP/HTTP to localhost:4318 by default. The collector endpoint and other settings can be configured without code changes via the GITLAB_TRACING environment variable (see below) or via Config.

Environment Variables

The recommended way to configure tracing in cloud-native deployments is via the GITLAB_TRACING environment variable. It accepts a connection string that controls the collector endpoint, TLS, service name, and sampling in a single value.

Connection string format

<scheme>://<host>:<port>[?param=value&...]

Examples

# Insecure local collector, 10% sampling
GITLAB_TRACING="otlp://localhost:4318?sampler=probabilistic&sampler_param=0.1"

# TLS-enabled remote collector, always sample, custom service name
GITLAB_TRACING="otlps://collector.example.com:4317?service_name=gitaly&sampler=probabilistic&sampler_param=1"

In Kubernetes:

env:
  - name: GITLAB_TRACING
    value: "otlps://otel-collector.observability:4317?service_name=my-service&sampler=probabilistic&sampler_param=0.05"

Precedence

Values set in Config take precedence over GITLAB_TRACING. The env var fills in only fields that are left at their zero value in the Config struct, so you can mix both: use GITLAB_TRACING for the endpoint and set SampleRate programmatically, for example.

Programmatic Configuration

tracer, shutdown, err := trace.NewWithConfig(ctx, &trace.Config{
    // ServiceName is included as the service.name resource attribute on every span.
    // Defaults to the binary name when empty.
    ServiceName: "gitaly",

    // ServiceVersion is included as the service.version resource attribute.
    ServiceVersion: "16.0.0",

    // Endpoint is the base URL of the OTLP HTTP collector (no path suffix).
    Endpoint: "https://collector.example.com:4318",

    // Insecure disables TLS. Useful for local development.
    Insecure: true,

    // Headers are sent with every export request, e.g. for authentication.
    Headers: map[string]string{"x-honeycomb-team": token},

    // SampleRate controls the fraction of traces that are sampled [0.0, 1.0].
    // An explicit 0 drops all spans. Defaults to 1% only when Config is nil.
    SampleRate: 0.1,
})

To explicitly drop all spans (useful in tests that use the real provider):

tracer, shutdown, err := trace.NewWithConfig(ctx, &trace.Config{
    SampleRate: trace.SampleRateDropAll,
})

Parsing a Tracing URL from Any Source

Some services receive a tracing connection string from an external source (e.g. a Rails JSON response or a CLI flag) rather than from GITLAB_TRACING. Use ParseTracingConfig to parse it, then use the fields directly with the OTEL SDK:

cfg := trace.ParseTracingConfig(urlFromRails)
if cfg == nil {
    // URL was empty or unparseable — proceed without tracing.
    return
}

// Build exporter options directly from the parsed fields.
var exporterOpts []otlptracehttp.Option
if cfg.Endpoint != "" {
    exporterOpts = append(exporterOpts, otlptracehttp.WithEndpointURL(cfg.Endpoint+cfg.URLPath))
}
if cfg.Insecure {
    exporterOpts = append(exporterOpts, otlptracehttp.WithInsecure())
}

exporter, err := otlptracehttp.New(ctx, exporterOpts...)
if err != nil { ... }

provider := sdktrace.NewTracerProvider(
    sdktrace.WithBatcher(exporter),
    sdktrace.WithSampler(cfg.Sampler()),
)
defer provider.Shutdown(ctx)

// Wrap the provider for the labkit Tracer API, or use it directly.
tracer := trace.NewWithProvider(provider)

To read from the environment without going through New:

cfg := trace.ParseTracingConfigFromEnv()
if cfg == nil {
    // GITLAB_TRACING is unset or unparseable.
    return
}
// cfg.Endpoint, cfg.URLPath, cfg.Insecure, cfg.ServiceName, cfg.SampleRate are available.

TracingConfig fields

Sampler() uses SampleRate only when SamplerSet is true. When SamplerSet is false it returns sdktrace.ParentBased(sdktrace.AlwaysSample()), which is the OpenTelemetry SDK default (sample all root spans, follow parent decision for child spans).

Creating Spans and Using the OTel API

Tracer.Start returns the raw oteltrace.Span interface — the full OTel span API (attributes, events, links, context propagation, etc.) is available without any labkit wrapper. See the OpenTelemetry Go instrumentation guide and the go.opentelemetry.io/otel/trace package reference.

Recording errors

trace.RecordError records the exception event and sets the span status to codes.Error in one call. The raw OTel Span.RecordError only records the event; setting the status requires a separate span.SetStatus call.

result, err := doWork(ctx)
if err != nil {
    trace.RecordError(span, err)
    return err
}

Accessing underlying OTel types

To pass the tracer to a library that requires oteltrace.Tracer directly:

oteltracer := tracer.OTELTracer()
thirdPartyLib.Init(oteltracer)

To retrieve the underlying provider:

provider := tracer.Provider()
thirdPartyLib.SetProvider(provider)

Testing

Use tracetest.NewRecorder to capture completed spans in memory without a live collector. The recorder always samples and does not register a global provider, keeping tests isolated.

import "gitlab.com/gitlab-org/labkit/v2/testing/tracetest"

func TestMyHandler(t *testing.T) {
    tracer, rec := tracetest.NewRecorder()

    ctx, span := tracer.Start(context.Background(), "handle-request")
    span.SetAttributes(attribute.Int("user.id", 42))
    trace.RecordError(span, errors.New("something went wrong"))
    span.End()

    spans := rec.Ended()
    require.Len(t, spans, 1)

    s := spans[0]
    assert.Equal(t, "handle-request", s.Name)
    assert.Equal(t, int64(42), s.Attributes["user.id"])
    assert.True(t, s.HasError)
    assert.Equal(t, codes.Error, s.StatusCode)
    assert.Equal(t, "something went wrong", s.StatusMsg)
}

rec.Reset() clears recorded spans between sub-tests when a single recorder is shared across a table-driven test.