tracing

package
v1.0.0-rc.4 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 3, 2026 License: Apache-2.0 Imports: 27 Imported by: 0

Documentation

Overview

Package tracing provides distributed tracing for the gridctl MCP gateway. It initializes the OpenTelemetry SDK, propagates W3C trace context, stores completed traces in an in-memory ring buffer, and optionally exports traces and metrics to OTLP.

Index

Constants

View Source 
const (
	AttrMethodName         = attribute.Key("mcp.method.name")
	AttrProtocolVersion    = attribute.Key("mcp.protocol.version")
	AttrGenAIToolName      = attribute.Key("gen_ai.tool.name")
	AttrGenAIOperationName = attribute.Key("gen_ai.operation.name")
	AttrErrorType          = attribute.Key("error.type")
	AttrRPCResponseStatus  = attribute.Key("rpc.response.status_code")
	AttrNetworkTransport   = attribute.Key("network.transport")
	AttrServerAddress      = attribute.Key("server.address")
	AttrServerPort         = attribute.Key("server.port")
	AttrServiceName        = attribute.Key("service.name")
	AttrServiceVersion     = attribute.Key("service.version")
	AttrServiceInstanceID  = attribute.Key("service.instance.id")
)

Semantic-convention keys. Development status; kept in this file so a registry rename is one edit.

View Source 
const (
	AttrServerName          = attribute.Key("gridctl.server.name")
	AttrReplicaID           = attribute.Key("gridctl.replica.id")
	AttrClientName          = attribute.Key("gridctl.client.name")
	AttrTransport           = attribute.Key("gridctl.transport")
	AttrProtocolGeneration  = attribute.Key("gridctl.protocol.generation")
	AttrOutputFormat        = attribute.Key("gridctl.output.format")
	AttrAutoscaleDirection  = attribute.Key("gridctl.autoscale.direction")
	AttrAutoscaleReason     = attribute.Key("gridctl.autoscale.reason")
	AttrUsageInputTokens    = attribute.Key("gridctl.usage.input_tokens")
	AttrUsageOutputTokens   = attribute.Key("gridctl.usage.output_tokens")
	AttrUsageCacheRead      = attribute.Key("gridctl.usage.cache_read_tokens")
	AttrUsageCacheCreation  = attribute.Key("gridctl.usage.cache_creation_tokens")
	AttrUsageDirection      = attribute.Key("gridctl.usage.direction")
	AttrDispatchDisposition = attribute.Key("gridctl.dispatch.disposition")
	AttrDispatchStage       = attribute.Key("gridctl.dispatch.stage")
	AttrDispatchReason      = attribute.Key("gridctl.dispatch.reason")
	AttrReplicaState        = attribute.Key("gridctl.replica.state")
	AttrRunsDropReason      = attribute.Key("gridctl.runs.drop_reason")
	AttrOTLPSignal          = attribute.Key("gridctl.otlp.signal")
)

Gridctl keys. Inventions that used to sit in reserved namespaces.

View Source 
const (
	AttrLegacyServerName         = attribute.Key("server.name")
	AttrLegacyMCPServerName      = attribute.Key("mcp.server.name")
	AttrLegacyMCPServer          = attribute.Key("mcp.server")
	AttrLegacyReplicaID          = attribute.Key("mcp.replica.id")
	AttrLegacyClientName         = attribute.Key("mcp.client.name")
	AttrLegacyProtocolGeneration = attribute.Key("mcp.protocol.generation")
	AttrLegacyToolName           = attribute.Key("mcp.tool.name")
	AttrLegacyBareToolName       = attribute.Key("tool.name")
	AttrLegacyOutputFormat       = attribute.Key("output.format")
	AttrLegacyAutoscaleDirection = attribute.Key("mcp.autoscale.direction")
	AttrLegacyAutoscaleReason    = attribute.Key("mcp.autoscale.reason")
	AttrLegacyUsageInput         = attribute.Key("gen_ai.usage.input_tokens")
	AttrLegacyUsageOutput        = attribute.Key("gen_ai.usage.output_tokens")
	AttrLegacyUsageCacheRead     = attribute.Key("gen_ai.usage.cache_read.input_tokens")
	AttrLegacyUsageCacheCreation = attribute.Key("gen_ai.usage.cache_creation.input_tokens")
)

Legacy keys as emitted before the rename. Readers fall back to these permanently because seeded traces keep them.

View Source 
const (
	MethodToolsCall      = "tools/call"
	OperationExecuteTool = "execute_tool"
	SpanNameToolsCall    = "tools/call"
)

Operation and method values.

View Source 
const (
	NetworkPipe = "pipe"
	NetworkTCP  = "tcp"

	TransportStdio   = "stdio"
	TransportProcess = "process"
	TransportSSH     = "ssh"
	TransportHTTP    = "http"
	TransportSSE     = "sse"
	TransportOpenAPI = "openapi"
	TransportA2A     = "a2a"
)

Network and gridctl transport values.

View Source 
const (
	SignalTraces      = "traces"
	SignalMetrics     = "metrics"
	SignalUnknown     = "unknown"
	SignalPathTraces  = "/v1/traces"
	SignalPathMetrics = "/v1/metrics"
)

OTLP signal names for the export-failure counter.

View Source 
const (
	InstrumentServerDuration = "mcp.server.operation.duration"
	InstrumentClientDuration = "mcp.client.operation.duration"
	InstrumentToolCalls      = "gridctl.gateway.tool.calls"
	InstrumentTokens         = "gridctl.gateway.tokens" //nolint:gosec // instrument name, not a credential
	InstrumentReplicas       = "gridctl.gateway.replicas"
	InstrumentRunsDropped    = "gridctl.runs.dropped"
	InstrumentExportFailures = "gridctl.otlp.export.failures"
)

Instrument names.

View Source 
const AliasRemovalRelease = "1.1.0"

AliasRemovalRelease is the release that drops legacy attribute keys. Those keys are emitted until that release. The old network.transport values are gone now; their mapping is documented until the same release. Diagnostic output, not a versioned contract.

View Source 
const BaggageMaxBytes = 8192

BaggageMaxBytes is the cap on a forwarded baggage value. Over the cap the whole value is dropped. Baggage is never copied into span attributes, logs, run records, or the trace buffer.

View Source 
const DefaultMetricsInterval = 60 * time.Second

DefaultMetricsInterval is the periodic reader interval when unset.

View Source 
const DefaultMetricsTimeout = 10 * time.Second

DefaultMetricsTimeout is the exporter timeout when unset.

Variables

View Source 
var DurationHistogramBuckets = []float64{0.01, 0.02, 0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10, 30, 60, 120, 300}

DurationHistogramBuckets are the MCP operation-duration boundaries, in seconds.

View Source 
var LegacyAliases = map[attribute.Key][]attribute.Key{
	AttrServerName:         {AttrLegacyServerName, AttrLegacyMCPServerName, AttrLegacyMCPServer},
	AttrReplicaID:          {AttrLegacyReplicaID},
	AttrClientName:         {AttrLegacyClientName},
	AttrProtocolGeneration: {AttrLegacyProtocolGeneration},
	AttrGenAIToolName:      {AttrLegacyToolName, AttrLegacyBareToolName},
	AttrOutputFormat:       {AttrLegacyOutputFormat},
	AttrAutoscaleDirection: {AttrLegacyAutoscaleDirection},
	AttrAutoscaleReason:    {AttrLegacyAutoscaleReason},
	AttrUsageInputTokens:   {AttrLegacyUsageInput},
	AttrUsageOutputTokens:  {AttrLegacyUsageOutput},
	AttrUsageCacheRead:     {AttrLegacyUsageCacheRead},
	AttrUsageCacheCreation: {AttrLegacyUsageCacheCreation},
}

LegacyAliases maps each new key to the old keys emitted for the same value. A new key may alias more than one old key. network.transport is absent: the key stays and only its values change.

Functions

func AliasesEnabled 

func AliasesEnabled() bool

AliasesEnabled reports whether the deprecation window is still open.

func BaggageDrops 

func BaggageDrops() uint64

BaggageDrops reports how many oversized baggage values have been dropped in this process. Tests and the debug log share the count.

func BuildResource 

func BuildResource(id ResourceIdentity) *sdkresource.Resource

BuildResource returns the shared resource. The schema URL is left empty: these are the stable service.* keys, not a pinned semconv schema.

func CapBaggage 

func CapBaggage(value string) (string, bool)

CapBaggage returns value unchanged when it is within the cap. An oversized value is dropped (empty string, dropped=true) and counted.

func ClientNameKeys 

func ClientNameKeys() []string

ClientNameKeys is the fallback order for a client name attribute.

func EndpointHost 

func EndpointHost(raw string) string

EndpointHost returns the host (and port) of an OTLP endpoint for logs. It never returns userinfo, path, or query. An unparseable value yields "".

func FirstString 

func FirstString(attrs map[string]string, keys ...string) string

FirstString returns the first non-empty value among keys. Readers use this so a new key wins and a seeded old key still resolves.

func NoteBaggageDrop 

func NoteBaggageDrop(n int)

NoteBaggageDrop records one oversized drop and logs the byte count, never the value.

func ResolveOTLPEndpoint 

func ResolveOTLPEndpoint(raw, signalPath string) (string, error)

ResolveOTLPEndpoint parses raw and, when the path is empty or "/", appends signalPath ("/v1/traces" or "/v1/metrics"). Any other path is used verbatim. Both exporters call this: WithEndpointURL uses the path as-is and never appends a signal path itself.

func ServerNameKeys 

func ServerNameKeys() []string

ServerNameKeys is the fallback order for a server name attribute.

func SetAttributesWithAliases 

func SetAttributesWithAliases(span trace.Span, attrs ...attribute.KeyValue)

SetAttributesWithAliases stamps attrs and, while the window is open, each legacy alias of those keys with the same value.

func ToolNameKeys 

func ToolNameKeys() []string

ToolNameKeys is the fallback order for a tool name attribute.

func TransportKeys 

func TransportKeys() []string

TransportKeys is the fallback order for the transport label. The key did not change; readers still look at network.transport.

Types

type Buffer 

type Buffer struct {
	// contains filtered or unexported fields
}

Buffer stores completed traces in a thread-safe ring buffer and implements sdktrace.SpanExporter so it can be registered with the OTel TracerProvider.

func NewBuffer 

func NewBuffer(maxSize int, retention time.Duration) *Buffer

NewBuffer creates a trace ring buffer with the given capacity and retention.

func (*Buffer) Count 

func (b *Buffer) Count() int

Count returns the number of traces currently stored.

func (*Buffer) ExportSpans 

func (b *Buffer) ExportSpans(_ context.Context, spans []sdktrace.ReadOnlySpan) error

ExportSpans implements sdktrace.SpanExporter. Spans are accumulated by trace ID. When the local-root span is exported, all accumulated spans are finalised into a TraceRecord.

func (*Buffer) Filter 

func (b *Buffer) Filter(opts FilterOpts) []TraceRecord

Filter returns traces matching the given options, newest first.

func (*Buffer) GetByID 

func (b *Buffer) GetByID(traceID string) *TraceRecord

GetByID returns the trace with the given ID, or nil if not found.

func (*Buffer) GetRecent 

func (b *Buffer) GetRecent(n int) []TraceRecord

GetRecent returns up to n most-recent traces, newest first.

func (*Buffer) MaxSize 

func (b *Buffer) MaxSize() int

MaxSize returns the ring buffer capacity.

func (*Buffer) SeedFromFile 

func (b *Buffer) SeedFromFile(path string, n int) error

SeedFromFile reads up to the last n OTLP-JSON envelope lines from path and rehydrates them into TraceRecords in the buffer. Used on daemon startup to preload the in-memory ring with pre-restart trace history from a persisted per-server traces.jsonl file. Missing or empty files return nil — that's expected on the very first run with persistence enabled.

The on-disk format is one tracepb.TracesData per line, written by pkg/telemetry.TracesFileClient. Lines that fail to parse are skipped without aborting the seed.

func (*Buffer) SetIncludeInfra 

func (b *Buffer) SetIncludeInfra(v bool)

SetIncludeInfra controls whether spans from non-gridctl instrumentation scopes are stored. Call before the buffer is registered as an exporter.

func (*Buffer) Shutdown 

func (b *Buffer) Shutdown(_ context.Context) error

Shutdown implements sdktrace.SpanExporter. Flushes any pending in-progress traces.

type CappedCarrier 

type CappedCarrier struct {
	// contains filtered or unexported fields
}

CappedCarrier wraps a TextMapCarrier and drops an oversized baggage value on Set. Other keys pass through.

func NewCappedCarrier 

func NewCappedCarrier(inner propagation.TextMapCarrier) CappedCarrier

NewCappedCarrier wraps inner. A nil inner is not useful; callers pass a header or meta carrier.

func (CappedCarrier) Get 

func (c CappedCarrier) Get(key string) string

Get returns the inner value.

func (CappedCarrier) Keys 

func (c CappedCarrier) Keys() []string

Keys returns the inner keys.

func (CappedCarrier) Set 

func (c CappedCarrier) Set(key, value string)

Set stores the pair, dropping baggage that exceeds BaggageMaxBytes.

type Config 

type Config struct {
	// Enabled controls whether tracing is active. Default: true.
	Enabled bool `yaml:"enabled"`

	// Sampling is the head-based sampling rate [0.0, 1.0]. Default: 1.0 (sample all).
	Sampling float64 `yaml:"sampling"`

	// Retention is how long completed traces are kept in the in-memory buffer.
	// Parsed as a Go duration string (e.g. "24h", "1h"). Default: "24h".
	Retention string `yaml:"retention"`

	// Export selects an exporter: "otlp" or "" (none). Default: "".
	Export string `yaml:"export,omitempty"`

	// Endpoint is the OTLP endpoint URL (e.g. "http://localhost:4318").
	// Required when Export is "otlp". A pathless URL is posted to /v1/traces.
	Endpoint string `yaml:"endpoint,omitempty"`

	// Headers are sent on the OTLP trace export request. Values are redacted
	// in logs. Names only are logged at startup.
	Headers map[string]string `yaml:"headers,omitempty"`

	// MaxTraces is the ring buffer capacity. Default: 1000.
	MaxTraces int `yaml:"max_traces,omitempty"`

	// IncludeInfra admits spans from non-gridctl instrumentation scopes
	// (e.g. Docker SDK HTTP self-instrumentation) into the in-memory buffer.
	// The OTLP export path is unaffected. Default: false.
	IncludeInfra bool `yaml:"include_infra,omitempty"`

	// ServiceName is the resource service.name. Empty falls back to gridctl-gateway.
	ServiceName string `yaml:"-"`

	// ServiceVersion is the resource service.version.
	ServiceVersion string `yaml:"-"`

	// Metrics configures the optional OTLP metrics exporter. Nil disables it.
	Metrics *MetricsConfig `yaml:"-"`
}

Config holds tracing configuration for the gateway.

func DefaultConfig 

func DefaultConfig() *Config

DefaultConfig returns a Config with sensible defaults.

func (*Config) RetentionDuration 

func (c *Config) RetentionDuration() time.Duration

RetentionDuration parses Retention as a time.Duration, returning the default (24h) on error.

type ExportErrorHandler 

type ExportErrorHandler struct {
	// contains filtered or unexported fields
}

ExportErrorHandler logs exporter failures at warn at most once per minute per distinct message and counts them per signal. Installed via otel.SetErrorHandler in Provider.Init.

func NewExportErrorHandler 

func NewExportErrorHandler(logger *slog.Logger) *ExportErrorHandler

NewExportErrorHandler returns a handler that rate-limits identical messages to once per minute.

func (*ExportErrorHandler) Failures 

func (h *ExportErrorHandler) Failures(signal string) uint64

Failures returns the in-process count for signal.

func (*ExportErrorHandler) Handle 

func (h *ExportErrorHandler) Handle(err error)

Handle implements otel.ErrorHandler. Errors wrapped as "metric export:" or "trace export:" are counted on that signal. Anything else is unknown.

func (*ExportErrorHandler) Note 

func (h *ExportErrorHandler) Note(signal string, err error)

Note records one export failure for signal, logs at warn when the message has not been logged in the last minute, and increments the in-process counter.

func (*ExportErrorHandler) SetHost 

func (h *ExportErrorHandler) SetHost(signal, host string)

SetHost records the endpoint host logged with failures for a signal. Header values are never stored here.

func (*ExportErrorHandler) SetRecorder 

func (h *ExportErrorHandler) SetRecorder(fn func(signal string))

SetRecorder installs the hook that increments gridctl.otlp.export.failures. Nil disables the metric side; the in-process counter still increments.

type FilterOpts 

type FilterOpts struct {
	ServerName  string
	ErrorsOnly  bool
	MinDuration time.Duration
	Limit       int
}

FilterOpts configures trace list filtering.

type MetaCarrier 

type MetaCarrier struct {
	// contains filtered or unexported fields
}

MetaCarrier implements propagation.TextMapCarrier for MCP _meta maps. It enables standard OTel Extract()/Inject() calls on the JSON-RPC params._meta field, following MCP spec PR #414.

func NewMetaCarrier 

func NewMetaCarrier(meta map[string]any) *MetaCarrier

NewMetaCarrier wraps an existing _meta map (or creates one if nil).

func (*MetaCarrier) Get 

func (c *MetaCarrier) Get(key string) string

Get returns the value for a key, normalising to lowercase.

func (*MetaCarrier) Keys 

func (c *MetaCarrier) Keys() []string

Keys returns all keys present in the _meta map.

func (*MetaCarrier) Map 

func (c *MetaCarrier) Map() map[string]any

Map returns the underlying _meta map (e.g. to re-marshal into JSON).

func (*MetaCarrier) Set 

func (c *MetaCarrier) Set(key, value string)

Set stores a key-value pair in the _meta map.

type MetricsConfig 

type MetricsConfig struct {
	Endpoint string
	Headers  map[string]string
	Interval time.Duration
	Timeout  time.Duration
}

MetricsConfig is the resolved OTLP metrics export block. HTTP/protobuf only.

type Provider 

type Provider struct {
	Buffer *Buffer
	// contains filtered or unexported fields
}

Provider wraps the OTel TracerProvider, the optional MeterProvider, and the in-memory trace buffer. Call Init to configure and register the global providers. Call Shutdown to flush and shut down cleanly.

func NewProvider 

func NewProvider(cfg *Config) *Provider

NewProvider creates a Provider from config. Call Init to activate it.

func (*Provider) ErrorHandler 

func (p *Provider) ErrorHandler() *ExportErrorHandler

ErrorHandler returns the export error handler, or nil before Init.

func (*Provider) Identity 

func (p *Provider) Identity() ResourceIdentity

Identity returns the resource identity. Valid after NewProvider.

func (*Provider) Init 

func (p *Provider) Init(ctx context.Context) error

Init initialises the OTel SDK, registers the global TracerProvider, MeterProvider, propagator, and error handler, and sets up the in-memory buffer plus optional OTLP exporters.

func (*Provider) RegisterExporter 

func (p *Provider) RegisterExporter(exp sdktrace.SpanExporter)

RegisterExporter attaches an additional SpanExporter to the live TracerProvider via a BatchSpanProcessor. Used by the telemetry persistence layer to plug a per-server file exporter alongside the in-memory buffer and the optional OTLP exporter without rebuilding the provider. No-op when tracing is disabled or Init has not yet run.

func (*Provider) SetLogger 

func (p *Provider) SetLogger(logger *slog.Logger)

SetLogger sets the logger for tracing startup messages.

func (*Provider) SetResourceIdentity 

func (p *Provider) SetResourceIdentity(serviceName, version string)

SetResourceIdentity overrides the service name and version used for the shared resource. Empty name keeps the gridctl-gateway fallback. The instance id generated at construction is preserved.

func (*Provider) Shutdown 

func (p *Provider) Shutdown(ctx context.Context) error

Shutdown flushes pending spans and metrics, then shuts the providers down. Cancellation of ctx is honored. ForceFlush runs before Shutdown for each signal that was started.

type ResourceIdentity 

type ResourceIdentity struct {
	ServiceName string
	Version     string
	InstanceID  string
}

ResourceIdentity is the three resource attributes shared by the trace and meter providers. InstanceID is generated once per provider instance.

func NewResourceIdentity 

func NewResourceIdentity(serviceName, version, instanceID string) ResourceIdentity

NewResourceIdentity fills an empty service name and instance id.

type SpanEvent 

type SpanEvent struct {
	Name      string            `json:"name"`
	Timestamp time.Time         `json:"timestamp"`
	Attrs     map[string]string `json:"attrs,omitempty"`
}

SpanEvent is a timestamped event recorded on a span.

type SpanRecord 

type SpanRecord struct {
	TraceID    string            `json:"trace_id"`
	SpanID     string            `json:"span_id"`
	ParentID   string            `json:"parent_id,omitempty"`
	Name       string            `json:"name"`
	StartTime  time.Time         `json:"start_time"`
	EndTime    time.Time         `json:"end_time"`
	DurationMs int64             `json:"duration_ms"`
	Status     string            `json:"status"`
	IsError    bool              `json:"is_error"`
	Attrs      map[string]string `json:"attrs,omitempty"`
	Events     []SpanEvent       `json:"events,omitempty"`
}

SpanRecord is a completed OTel span stored in the trace buffer.

type TraceRecord 

type TraceRecord struct {
	TraceID    string       `json:"trace_id"`
	Operation  string       `json:"operation"` // root span name
	ServerName string       `json:"server_name,omitempty"`
	MethodName string       `json:"method_name,omitempty"`
	Tool       string       `json:"tool,omitempty"`
	ClientName string       `json:"client_name,omitempty"`
	StartTime  time.Time    `json:"start_time"`
	EndTime    time.Time    `json:"end_time"`
	DurationMs int64        `json:"duration_ms"`
	SpanCount  int          `json:"span_count"`
	IsError    bool         `json:"is_error"`
	Spans      []SpanRecord `json:"spans"`
}

TraceRecord groups all spans belonging to a single trace.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.