Documentation
¶
Overview ¶
Package telemetry provides OpenTelemetry-based observability for Trace.
This package initializes the OTel SDK with opinionated defaults for tracing and metrics, while allowing backend flexibility through exporter configuration.
Philosophy ¶
Be opinionated about the API, flexible about the backend. OpenTelemetry IS the abstraction layer. We use OTel APIs directly (no custom interfaces), and users swap backends by changing exporter configuration, not code.
Trace Backend (default: Jaeger via OTLP) ¶
Jaeger is the default trace backend. Since Jaeger 1.35+, it supports OTLP natively, which is the recommended protocol. Users can swap to Datadog, New Relic, or other OTLP-compatible backends via environment variables.
Metrics Backend (default: Prometheus) ¶
Prometheus is the default metrics backend. Metrics are exposed at /metrics endpoint for scraping. Users can swap to OTLP push-based metrics if needed.
Logging ¶
Uses slog (Go 1.21+ stdlib) for structured logging. LoggerWithTrace injects trace_id and span_id into log entries for correlation in Grafana/Loki.
Usage ¶
cfg := telemetry.DefaultConfig()
shutdown, err := telemetry.Init(ctx, cfg)
if err != nil {
return fmt.Errorf("init telemetry: %w", err)
}
defer shutdown(ctx)
// Now otel.Tracer() and otel.Meter() are configured
tracer := otel.Tracer("mypackage")
meter := otel.Meter("mypackage")
Environment Variables ¶
Standard OTel environment variables are supported:
- OTEL_EXPORTER_OTLP_ENDPOINT: OTLP endpoint (default: localhost:4317)
- OTEL_TRACES_EXPORTER: otlp, stdout, or none (default: otlp)
- OTEL_METRICS_EXPORTER: prometheus, otlp, stdout, or none (default: prometheus)
- ALEUTIAN_ENV: environment name (default: development)
Thread Safety ¶
All exported functions are safe for concurrent use after Init() returns.
Index ¶
- Variables
- func AddSpanEvent(span trace.Span, name string, attrs ...attribute.KeyValue)
- func CombinedMiddleware(tracerName string, metrics *Metrics) func(http.Handler) http.Handler
- func ExtractContext(ctx context.Context, headers http.Header) context.Context
- func ExtractFromMap(ctx context.Context, carrier map[string]string) context.Context
- func ExtractFromRequest(req *http.Request) context.Context
- func HasActiveSpan(ctx context.Context) bool
- func Init(ctx context.Context, cfg Config) (shutdown func(context.Context) error, err error)
- func InjectContext(ctx context.Context, headers http.Header)
- func InjectToMap(ctx context.Context, carrier map[string]string) map[string]string
- func LoggerWithNode(ctx context.Context, logger *slog.Logger, nodeName string) *slog.Logger
- func LoggerWithSession(ctx context.Context, logger *slog.Logger, sessionID string) *slog.Logger
- func LoggerWithTrace(ctx context.Context, logger *slog.Logger) *slog.Logger
- func MetricsHandler() http.Handler
- func MetricsMiddleware(metrics *Metrics) func(http.Handler) http.Handler
- func PropagateToRequest(ctx context.Context, req *http.Request) *http.Request
- func RecordError(span trace.Span, err error, attrs ...attribute.KeyValue)
- func RecordErrorf(span trace.Span, format string, args ...interface{})
- func SetSpanAttributes(span trace.Span, attrs ...attribute.KeyValue)
- func SetSpanOK(span trace.Span)
- func SpanFromContext(ctx context.Context) trace.Span
- func SpanID(ctx context.Context) string
- func StartSpan(ctx context.Context, tracerName, spanName string, ...) (context.Context, trace.Span)
- func TraceID(ctx context.Context) string
- func TracingMiddleware(tracerName string) func(http.Handler) http.Handler
- type Config
- type MapCarrier
- type Metrics
Constants ¶
This section is empty.
Variables ¶
var ( // ErrNilContext is returned when a nil context is passed. ErrNilContext = errors.New("context must not be nil") // ErrUnknownExporter is returned when an unknown exporter type is specified. ErrUnknownExporter = errors.New("unknown exporter type") // ErrAlreadyInitialized is returned if Init is called more than once. ErrAlreadyInitialized = errors.New("telemetry already initialized") )
Sentinel errors for the telemetry package.
Functions ¶
func AddSpanEvent ¶
AddSpanEvent adds an event to the span with optional attributes.
Description:
Records a timestamped event on the span. Events are useful for marking significant points in a span's lifecycle.
Inputs:
span - The span to add the event to. May be nil. name - Event name describing what happened. attrs - Optional attributes to include with the event.
Example:
telemetry.AddSpanEvent(span, "cache_miss", attribute.String("key", cacheKey))
Thread Safety: Safe for concurrent use.
func CombinedMiddleware ¶
CombinedMiddleware creates middleware that adds both tracing and metrics.
Description:
Convenience function that combines TracingMiddleware and MetricsMiddleware. Tracing is applied first (outer), then metrics (inner).
Inputs:
tracerName - Name for the tracer. metrics - Pre-configured Metrics instance.
Outputs:
Middleware function that wraps http.Handler.
Example:
metrics, _ := telemetry.NewMetrics(otel.Meter("trace"))
mux := http.NewServeMux()
handler := telemetry.CombinedMiddleware("trace.http", metrics)(mux)
http.ListenAndServe(":8080", handler)
Thread Safety: Safe for concurrent use.
func ExtractContext ¶
ExtractContext extracts trace context from incoming HTTP headers.
Description:
Uses the globally configured propagator (set in Init) to extract W3C TraceContext and Baggage from HTTP headers. The returned context contains the extracted trace information and can be used to create child spans.
Inputs:
ctx - Base context to extend with trace information. headers - HTTP headers containing trace context (e.g., traceparent, tracestate).
Outputs:
context.Context - Context with trace information attached.
Returns the original context if no trace headers are present.
Example:
func handleRequest(w http.ResponseWriter, r *http.Request) {
ctx := telemetry.ExtractContext(r.Context(), r.Header)
ctx, span := tracer.Start(ctx, "handleRequest")
defer span.End()
// ... handle request
}
Thread Safety: Safe for concurrent use.
func ExtractFromMap ¶
ExtractFromMap extracts trace context from a string map.
Description:
Useful for extracting trace context from non-HTTP transports like message queues, gRPC metadata, or custom headers.
Inputs:
ctx - Base context to extend. carrier - Map containing trace context keys.
Outputs:
context.Context - Context with trace information attached.
Example:
func handleMessage(ctx context.Context, msg *Message) {
ctx = telemetry.ExtractFromMap(ctx, msg.Headers)
ctx, span := tracer.Start(ctx, "handleMessage")
defer span.End()
}
Thread Safety: Safe for concurrent use.
func ExtractFromRequest ¶
ExtractFromRequest extracts trace context from an incoming HTTP request.
Description:
Convenience wrapper that extracts trace context from the request headers and returns an updated context.
Inputs:
req - HTTP request containing trace headers.
Outputs:
context.Context - Context with trace information attached.
Example:
func handleIncoming(w http.ResponseWriter, r *http.Request) {
ctx := telemetry.ExtractFromRequest(r)
// Use ctx for creating child spans
}
Thread Safety: Safe for concurrent use.
func HasActiveSpan ¶
HasActiveSpan returns true if the context contains a valid, recording span.
Description:
Checks whether the context has an active span that is recording. Useful for conditional instrumentation.
Inputs:
ctx - Context to check.
Outputs:
bool - True if context has a valid, recording span.
Example:
if telemetry.HasActiveSpan(ctx) {
span := telemetry.SpanFromContext(ctx)
span.AddEvent("expensive_operation_start")
}
Thread Safety: Safe for concurrent use.
func Init ¶
Init initializes the telemetry stack with the given configuration.
Description:
Sets up OpenTelemetry TracerProvider and MeterProvider based on the configuration. After Init returns successfully, otel.Tracer() and otel.Meter() can be used throughout the application.
Inputs:
ctx - Context for initialization (used for exporter connections). cfg - Telemetry configuration. Use DefaultConfig() for sensible defaults.
Outputs:
shutdown - Function to call on application exit for cleanup. Must be called. error - Non-nil if initialization fails.
Example:
shutdown, err := telemetry.Init(ctx, telemetry.DefaultConfig())
if err != nil {
return fmt.Errorf("init telemetry: %w", err)
}
defer shutdown(context.Background())
Thread Safety: Call once at application startup.
func InjectContext ¶
InjectContext injects trace context into outgoing HTTP headers.
Description:
Uses the globally configured propagator (set in Init) to inject W3C TraceContext and Baggage into HTTP headers. Use this when making outgoing HTTP requests to propagate trace context.
Inputs:
ctx - Context containing active span information. headers - HTTP headers to inject trace context into.
Example:
func callDownstream(ctx context.Context, url string) error {
req, _ := http.NewRequestWithContext(ctx, "GET", url, nil)
telemetry.InjectContext(ctx, req.Header)
resp, err := http.DefaultClient.Do(req)
// ...
}
Thread Safety: Safe for concurrent use.
func InjectToMap ¶
InjectToMap injects trace context into a string map.
Description:
Useful for propagating trace context through non-HTTP transports. If carrier is nil, a new map is created and returned.
Inputs:
ctx - Context containing active span information. carrier - Map to inject trace context into. May be nil.
Outputs:
map[string]string - Map with trace context injected.
Example:
func publishMessage(ctx context.Context, payload []byte) error {
headers := telemetry.InjectToMap(ctx, nil)
msg := &Message{Payload: payload, Headers: headers}
return queue.Publish(msg)
}
Thread Safety: Safe for concurrent use.
func LoggerWithNode ¶
LoggerWithNode returns a logger with trace context and node name.
Description:
Combines LoggerWithTrace with a node identifier for DAG execution logging. Useful for distinguishing log entries from different pipeline nodes.
Inputs:
ctx - Context containing span context. logger - Base logger to enhance. nodeName - Name of the current DAG node (e.g., "PARSE_FILES").
Outputs:
*slog.Logger - Logger with trace_id, span_id, and node fields.
Example:
func (n *ParseNode) Execute(ctx context.Context, inputs map[string]any) (any, error) {
logger := telemetry.LoggerWithNode(ctx, n.logger, n.Name())
logger.Info("parsing files", slog.Int("count", len(files)))
}
Thread Safety: Safe for concurrent use.
func LoggerWithSession ¶
LoggerWithSession returns a logger with trace context and session ID.
Description:
Adds a session identifier for tracking related operations across multiple requests or pipeline executions.
Inputs:
ctx - Context containing span context. logger - Base logger to enhance. sessionID - Unique session identifier.
Outputs:
*slog.Logger - Logger with trace_id, span_id, and session_id fields.
Thread Safety: Safe for concurrent use.
func LoggerWithTrace ¶
LoggerWithTrace returns a logger with trace context injected.
Description:
Extracts trace_id and span_id from the context and adds them as structured log fields. This enables log correlation in Grafana/Loki with traces in Jaeger.
Inputs:
ctx - Context containing span context. May be nil or have no active span. logger - Base logger to enhance. Must not be nil.
Outputs:
*slog.Logger - Logger with trace_id and span_id fields added if available.
Returns the original logger if no valid span context.
Example:
func (s *Service) Process(ctx context.Context) error {
logger := telemetry.LoggerWithTrace(ctx, s.logger)
logger.Info("processing started")
// Log output: {"level":"INFO","msg":"processing started","trace_id":"abc123","span_id":"def456"}
}
Thread Safety: Safe for concurrent use.
func MetricsHandler ¶
MetricsHandler returns the HTTP handler for the /metrics endpoint.
Description:
Returns the Prometheus metrics handler if Prometheus exporter is enabled. Returns nil if metrics are disabled or a different exporter is used.
Outputs:
http.Handler - The metrics handler, or nil if unavailable.
Example:
handler := telemetry.MetricsHandler()
if handler != nil {
http.Handle("/metrics", handler)
}
Thread Safety: Safe for concurrent use.
func MetricsMiddleware ¶
MetricsMiddleware creates HTTP middleware that records request metrics.
Description:
Records HTTP request count, duration, and active request count. Metrics include labels for method, path, and status code.
Inputs:
metrics - Pre-configured Metrics instance.
Outputs:
Middleware function that wraps http.Handler.
Example:
metrics, _ := telemetry.NewMetrics(otel.Meter("trace"))
mux := http.NewServeMux()
handler := telemetry.MetricsMiddleware(metrics)(mux)
http.ListenAndServe(":8080", handler)
Thread Safety: Safe for concurrent use.
func PropagateToRequest ¶
PropagateToRequest injects trace context into an outgoing HTTP request.
Description:
Convenience wrapper that extracts the context from the request, injects trace context into the request headers, and returns the request with updated context.
Inputs:
ctx - Context containing active span information. req - HTTP request to inject trace context into.
Outputs:
*http.Request - Request with context and trace headers updated.
Example:
func callAPI(ctx context.Context, url string) error {
req, _ := http.NewRequest("GET", url, nil)
req = telemetry.PropagateToRequest(ctx, req)
resp, err := client.Do(req)
// ...
}
Thread Safety: Safe for concurrent use.
func RecordError ¶
RecordError records an error on the current span with proper status.
Description:
Records the error as a span event and sets the span status to Error. If the span or error is nil, this is a no-op.
Inputs:
span - The span to record the error on. May be nil. err - The error to record. May be nil. attrs - Optional additional attributes to record with the error.
Example:
result, err := performOperation()
if err != nil {
telemetry.RecordError(span, err, attribute.String("operation", "parse"))
return err
}
Thread Safety: Safe for concurrent use.
func RecordErrorf ¶
RecordErrorf records a formatted error message on the current span.
Description:
Creates an error from the format string and records it on the span. Useful when you want to add context to an error before recording.
Inputs:
span - The span to record the error on. May be nil. format - Printf-style format string. args - Format arguments.
Example:
if err := validate(input); err != nil {
telemetry.RecordErrorf(span, "validation failed for %s: %v", input.Name, err)
return err
}
Thread Safety: Safe for concurrent use.
func SetSpanAttributes ¶
SetSpanAttributes sets attributes on the span.
Description:
Adds or updates attributes on the span. Safe to call with nil span.
Inputs:
span - The span to set attributes on. May be nil. attrs - Attributes to set.
Example:
telemetry.SetSpanAttributes(span,
attribute.Int("result_count", len(results)),
attribute.String("query_type", "find_callers"),
)
Thread Safety: Safe for concurrent use.
func SetSpanOK ¶
SetSpanOK marks the span as successful.
Description:
Sets the span status to OK. Use this when an operation completes successfully and you want to explicitly mark the span.
Inputs:
span - The span to mark as OK. May be nil.
Example:
result, err := performOperation()
if err != nil {
telemetry.RecordError(span, err)
return nil, err
}
telemetry.SetSpanOK(span)
return result, nil
Thread Safety: Safe for concurrent use.
func SpanFromContext ¶
SpanFromContext returns the current span from the context.
Description:
Convenience wrapper around trace.SpanFromContext. Returns a no-op span if no span is present in the context.
Inputs:
ctx - Context potentially containing a span.
Outputs:
trace.Span - The current span, or a no-op span if none exists.
Example:
func logWithSpan(ctx context.Context, msg string) {
span := telemetry.SpanFromContext(ctx)
span.AddEvent(msg)
}
Thread Safety: Safe for concurrent use.
func SpanID ¶
SpanID returns the span ID from the context as a string.
Description:
Extracts the span ID from the span context. Returns empty string if no valid span context is present.
Inputs:
ctx - Context potentially containing a span.
Outputs:
string - Hex-encoded span ID, or empty string if unavailable.
Thread Safety: Safe for concurrent use.
func StartSpan ¶
func StartSpan(ctx context.Context, tracerName, spanName string, opts ...trace.SpanStartOption) (context.Context, trace.Span)
StartSpan creates a new span from the context using the global tracer.
Description:
Convenience wrapper that uses otel.Tracer() to create spans without explicitly managing tracer instances. Uses consistent naming conventions.
Inputs:
ctx - Parent context. May contain existing span context. tracerName - Tracer name (typically package path, e.g., "trace.graph"). spanName - Span name (typically "package.Type.Method" or operation name). opts - Optional span start options (attributes, links, etc.).
Outputs:
context.Context - Context with the new span attached. trace.Span - The created span. Caller must call span.End().
Example:
func (g *Graph) Query(ctx context.Context, query string) ([]Result, error) {
ctx, span := telemetry.StartSpan(ctx, "trace.graph", "Graph.Query",
trace.WithAttributes(attribute.String("query", query)),
)
defer span.End()
// ... perform query
}
Thread Safety: Safe for concurrent use.
func TraceID ¶
TraceID returns the trace ID from the context as a string.
Description:
Extracts the trace ID from the span context. Returns empty string if no valid span context is present.
Inputs:
ctx - Context potentially containing a span.
Outputs:
string - Hex-encoded trace ID, or empty string if unavailable.
Example:
traceID := telemetry.TraceID(ctx)
logger.Info("operation complete", slog.String("trace_id", traceID))
Thread Safety: Safe for concurrent use.
func TracingMiddleware ¶
TracingMiddleware creates HTTP middleware that adds distributed tracing.
Description:
Wraps each HTTP request in a span with standard HTTP semantic attributes. Extracts trace context from incoming headers for distributed tracing. Sets span status to Error for 5xx responses.
Inputs:
tracerName - Name for the tracer (e.g., "trace.http").
Outputs:
Middleware function that wraps http.Handler.
Example:
mux := http.NewServeMux()
mux.HandleFunc("/api/query", handleQuery)
handler := telemetry.TracingMiddleware("trace.http")(mux)
http.ListenAndServe(":8080", handler)
Thread Safety: Safe for concurrent use.
Types ¶
type Config ¶
type Config struct {
// ServiceName identifies this service in traces and metrics.
ServiceName string `json:"service_name"`
// ServiceVersion is the version string for this service.
ServiceVersion string `json:"service_version"`
// Environment identifies the deployment environment (development, production).
Environment string `json:"environment"`
// TraceExporter selects the trace exporter: "otlp", "stdout", or "none".
TraceExporter string `json:"trace_exporter"`
// MetricExporter selects the metric exporter: "prometheus", "otlp", "stdout", or "none".
MetricExporter string `json:"metric_exporter"`
// OTLPEndpoint is the OTLP receiver endpoint for traces.
OTLPEndpoint string `json:"otlp_endpoint"`
// OTLPInsecure disables TLS verification for OTLP connections.
OTLPInsecure bool `json:"otlp_insecure"`
// PrometheusPort is the port for the /metrics endpoint (default: 9090).
PrometheusPort int `json:"prometheus_port"`
// SampleRate is the percentage of traces to sample (0.0-1.0).
// Default: 1.0 (100%) for development, recommend 0.1 (10%) for production.
// When set to 1.0, AlwaysSample() is used.
// When set to 0.0, NeverSample() is used.
// Otherwise, TraceIDRatioBased() is used.
SampleRate float64 `json:"sample_rate"`
// AllowDegraded allows the service to start even if telemetry backends are unavailable.
// When true, Init() logs a warning but returns success with noop providers.
// Default: false (strict mode for catching configuration errors in development).
AllowDegraded bool `json:"allow_degraded"`
}
Config controls telemetry behavior.
All fields have sensible defaults via DefaultConfig().
func DefaultConfig ¶
func DefaultConfig() Config
DefaultConfig returns opinionated defaults for development.
Environment variables override defaults where applicable:
- ALEUTIAN_ENV: environment name
- OTEL_TRACES_EXPORTER: trace exporter type
- OTEL_METRICS_EXPORTER: metric exporter type
- OTEL_EXPORTER_OTLP_ENDPOINT: OTLP endpoint
type MapCarrier ¶
MapCarrier implements propagation.TextMapCarrier for map[string]string.
Description:
Allows trace context propagation with simple string maps, useful for non-HTTP transports like message queues or gRPC metadata.
type Metrics ¶
type Metrics struct {
// HTTPRequestsTotal counts total HTTP requests by method, path, and status.
HTTPRequestsTotal metric.Int64Counter
// HTTPRequestDuration records HTTP request duration in seconds.
HTTPRequestDuration metric.Float64Histogram
// HTTPActiveRequests tracks currently active HTTP requests.
HTTPActiveRequests metric.Int64UpDownCounter
// GraphBuildsTotal counts total graph build operations by status.
GraphBuildsTotal metric.Int64Counter
// GraphBuildDuration records graph build duration in seconds.
GraphBuildDuration metric.Float64Histogram
// GraphQueriesTotal counts total graph queries by type and status.
GraphQueriesTotal metric.Int64Counter
// GraphQueryDuration records graph query duration in seconds.
GraphQueryDuration metric.Float64Histogram
// GraphSymbolsTotal counts total symbols indexed by language.
GraphSymbolsTotal metric.Int64Counter
// WeaviateRequestsTotal counts total Weaviate operations by type and status.
WeaviateRequestsTotal metric.Int64Counter
// WeaviateRequestDuration records Weaviate operation duration in seconds.
WeaviateRequestDuration metric.Float64Histogram
// WeaviateCircuitState tracks circuit breaker state (0=closed, 1=open, 2=half-open).
WeaviateCircuitState metric.Int64ObservableGauge
// MemoryRetrievalsTotal counts total memory retrieval operations by status.
MemoryRetrievalsTotal metric.Int64Counter
// MemoryStoresTotal counts total memory store operations by type.
MemoryStoresTotal metric.Int64Counter
// MemoryRetrievalDuration records memory retrieval duration in seconds.
MemoryRetrievalDuration metric.Float64Histogram
// MCTSIterationsTotal counts total MCTS iterations.
MCTSIterationsTotal metric.Int64Counter
// MCTSIterationDuration records MCTS iteration duration in seconds.
MCTSIterationDuration metric.Float64Histogram
// MCTSNodesExplored counts total MCTS nodes explored.
MCTSNodesExplored metric.Int64Counter
// ErrorsTotal counts total errors by type and component.
ErrorsTotal metric.Int64Counter
}
Metrics contains pre-defined metrics for the Aleutian Trace service.
Description:
Provides standard counters, histograms, and gauges for HTTP requests, graph operations, Weaviate interactions, and memory operations. All metrics use the "trace_" prefix for consistent naming.
Thread Safety: Safe for concurrent use after creation.
func NewMetrics ¶
NewMetrics creates a new Metrics instance with all metrics registered.
Description:
Registers all pre-defined metrics with the provided meter. Returns an error if any metric registration fails.
Inputs:
meter - The OTel meter to use for metric registration.
Outputs:
*Metrics - The metrics instance with all counters and histograms initialized. error - Non-nil if metric registration fails.
Example:
meter := otel.Meter("trace")
metrics, err := telemetry.NewMetrics(meter)
if err != nil {
return fmt.Errorf("create metrics: %w", err)
}
metrics.HTTPRequestsTotal.Add(ctx, 1, ...)
Thread Safety: Safe for concurrent use after creation.
func (*Metrics) RegisterWeaviateCircuitState ¶
func (m *Metrics) RegisterWeaviateCircuitState(meter metric.Meter, stateFunc func() int64) (metric.Registration, error)
RegisterWeaviateCircuitState registers a callback for the Weaviate circuit state gauge.
Description:
Sets up an observable gauge that reports the current circuit breaker state. The callback is invoked each time metrics are scraped.
Inputs:
meter - The OTel meter to use for registration. stateFunc - A function that returns the current circuit state (0=closed, 1=open, 2=half-open).
Outputs:
metric.Registration - Registration handle for cleanup. error - Non-nil if registration fails.
Source Files