Documentation
¶
Overview ¶
Package observability constructs the real OTLP tracer provider Forge hands to the runtime seam in forge-core/runtime/tracing.go.
This package is pure library — it does NOT read forge.yaml, env, or CLI flags. Configuration arrives as a fully-resolved TracingConfig from the cli-level wiring (Phase 2 of the OTel v1 initiative, #103). Anti-coupling: the runtime seam in forge-core/runtime depends only on the OTel TracerProvider interface; this package supplies a real SDK implementation that the cli installs at startup.
See issue #102 (Phase 1) and the master tracking issue #108.
Index ¶
- Constants
- Variables
- func NewTracerProvider(ctx context.Context, cfg TracingConfig, transport http.RoundTripper) (*sdktrace.TracerProvider, error)
- func WrapHTTPTransport(base http.RoundTripper) http.RoundTripper
- type SpanRecorder
- func (r *SpanRecorder) ExportSpans(_ context.Context, spans []sdktrace.ReadOnlySpan) error
- func (r *SpanRecorder) FindSpan(name string) (sdktrace.ReadOnlySpan, bool)
- func (r *SpanRecorder) FindSpans(name string) []sdktrace.ReadOnlySpan
- func (r *SpanRecorder) Reset()
- func (r *SpanRecorder) Shutdown(_ context.Context) error
- func (r *SpanRecorder) Spans() []sdktrace.ReadOnlySpan
- type TracingConfig
Constants ¶
const ( // AttrGenAISystem identifies the LLM vendor: "anthropic", // "openai", "ollama", "openai-compatible". AttrGenAISystem = "gen_ai.system" // AttrGenAIRequestModel is the model the agent ASKED for. AttrGenAIRequestModel = "gen_ai.request.model" // AttrGenAIResponseModel is the model the vendor REPORTED back. // Often identical to the request, but enterprise routers can // substitute (e.g. Anthropic returns a versioned suffix). AttrGenAIResponseModel = "gen_ai.response.model" // AttrGenAIUsageInputTokens / AttrGenAIUsageOutputTokens — counted // directly from the provider's usage block. Drives cost // dashboards. AttrGenAIUsageInputTokens = "gen_ai.usage.input_tokens" AttrGenAIUsageOutputTokens = "gen_ai.usage.output_tokens" // AttrGenAIResponseFinishReasons mirrors the Anthropic/OpenAI // "stop_reason" / "finish_reason" — "stop", "tool_use", // "max_tokens", "end_turn", etc. AttrGenAIResponseFinishReasons = "gen_ai.response.finish_reasons" // AttrForgeAgentID is the agent_id from forge.yaml — the operator's // primary identifier. Dashboards typically group by this first. AttrForgeAgentID = "forge.agent.id" // AttrForgeTaskID is the A2A task id (the same id surfaced in audit // events and the X-Forge-Task-Id response header). Lets operators // jump from an audit event to the corresponding trace. AttrForgeTaskID = "forge.task.id" // AttrForgeCorrelationID is the cross-process correlation id // (X-Forge-Correlation-Id header). Survives across the dispatcher // boundary so a span can be tied back to the inbound request. AttrForgeCorrelationID = "forge.correlation_id" // AttrForgeWorkflowID / Stage / Step are the FWS-2 orchestrator // correlation ids extracted from inbound X-Workflow-* headers. // Spans get these when the inbound request was issued by an // orchestrator (otherwise the keys are absent — backends should // treat missing keys as "ad-hoc invocation"). AttrForgeWorkflowID = "forge.workflow.id" AttrForgeWorkflowStageID = "forge.workflow.stage.id" AttrForgeWorkflowStepID = "forge.workflow.step.id" // AttrForgeA2AMethod is the JSON-RPC method name on the inbound // span — "tasks/send", "tasks/get", "tasks/cancel". Span name is // also derived from this, but the explicit attribute simplifies // querying. AttrForgeA2AMethod = "forge.a2a.method" // AttrForgeLoopIteration is the executor iteration counter on the // agent.execute parent span — set after the loop finishes so // dashboards can chart "iterations per task." AttrForgeLoopIteration = "forge.loop.iteration" // AttrForgeToolName / AttrForgeToolError name the tool call // instrumentation. AttrForgeToolName = "forge.tool.name" AttrForgeToolError = "forge.tool.error" // AttrForgeTaskFinalState is the terminal A2A TaskState the loop // resolved to — "completed", "failed", "canceled". Set on the // agent.execute span just before End. AttrForgeTaskFinalState = "forge.task.final_state" // AttrGenAIInputMessages is the structured inbound message array // the agent sent to the LLM — a JSON array of role+content pairs. // Per OTel GenAI semantic conventions (current). Supersedes the // deprecated `gen_ai.prompt` flat-string attribute. AttrGenAIInputMessages = "gen_ai.input.messages" // AttrGenAIOutputMessages is the structured response array from // the model — a JSON array of role+content pairs (single element // for a non-streaming, single-choice completion). Per OTel GenAI // semantic conventions (current). Supersedes the deprecated // `gen_ai.completion` flat-string attribute. AttrGenAIOutputMessages = "gen_ai.output.messages" // AttrForgeToolArgs is the raw arguments JSON the agent passed to // a tool. Set on tool.<name> spans. AttrForgeToolArgs = "forge.tool.args" // AttrForgeToolResult is the raw output the tool returned. Set on // tool.<name> spans. AttrForgeToolResult = "forge.tool.result" // AttrForgeGuardrailGate is the library gate that fired — one of // "input", "context", "tool_call", "output", "stream". Sourced // directly from `Result.Gate` (single source of truth, matches // fields.gate on the guardrail_check audit event). AttrForgeGuardrailGate = "forge.guardrail.gate" // AttrForgeGuardrailDecision is the library decision — one of // "allow", "mask", "block", "warn". Sourced from `Result.Decision`. AttrForgeGuardrailDecision = "forge.guardrail.decision" // AttrForgeGuardrailType is the first violation's Type field — e.g. // "pii", "moderation", "security". Omitted when violations list is // empty (the "allow" path). AttrForgeGuardrailType = "forge.guardrail.type" // AttrForgeGuardrailCategory is the first violation's Category — // e.g. "ssn", "email", "hate_speech". Omitted when the violation // has no category. AttrForgeGuardrailCategory = "forge.guardrail.category" // AttrForgeGuardrailViolationCount is the length of // `Result.Violations`. Useful for SIEM "show me high-violation // invocations" queries without joining against the full evidence // stream. AttrForgeGuardrailViolationCount = "forge.guardrail.violation_count" // AttrForgeGuardrailEvidence is the triggering content. Set only // when TracingConfig.CaptureContent is true. Passes through // PrepareSpanContent (redact-then-truncate) just like the other // #130 content attributes. For mask decisions evidence carries the // post-mask content (the same payload the LLM actually saw); for // block / warn decisions it carries the original triggering text // (the library never produces a masked variant in those paths). // Matches the audit-event evidence rule from PR #156. AttrForgeGuardrailEvidence = "forge.guardrail.evidence" )
const ( ProtocolHTTPProtobuf = "http/protobuf" ProtocolGRPC = "grpc" )
Protocol values recognized by TracingConfig.Protocol. The OTLP spec allows additional encodings; Forge ships HTTP+protobuf (the default, port 4318 convention) and gRPC (port 4317). HTTP+JSON is deliberately not exposed — backends that need it can configure their collector to translate, and Forge's own egress-allowlist wiring (Phase 6) tracks the host, not the encoding.
const ( SamplerAlwaysOn = "always_on" SamplerAlwaysOff = "always_off" SamplerTraceIDRatio = "traceidratio" SamplerParentBasedAlwaysOn = "parentbased_always_on" SamplerParentBasedAlwaysOff = "parentbased_always_off" SamplerParentBasedTraceIDRatio = "parentbased_traceidratio" )
Sampler values recognized by TracingConfig.Sampler. The string set is the same one the OTEL_TRACES_SAMPLER env var documents, so operators who already know the standard names don't have to learn Forge- specific ones.
const ( DefaultProtocol = ProtocolHTTPProtobuf DefaultSampler = SamplerParentBasedAlwaysOn DefaultSamplerRatio = 1.0 DefaultTimeout = 10 * time.Second )
Defaults applied when TracingConfig leaves a field at the zero value. Centralized here so the cli flag layer (Phase 2) and the tests reference the same constants.
Variables ¶
var ErrDisabled = errors.New("tracing disabled")
ErrDisabled is returned by NewTracerProvider when cfg.Enabled is false. Distinct sentinel so the cli wiring can branch on it (`errors.Is(err, ErrDisabled) → install the no-op tracer`) without turning a legitimate "tracing off" state into a startup error.
Functions ¶
func NewTracerProvider ¶
func NewTracerProvider(ctx context.Context, cfg TracingConfig, transport http.RoundTripper) (*sdktrace.TracerProvider, error)
NewTracerProvider builds a real OTLP-backed sdktrace.TracerProvider from a resolved TracingConfig. Returns (nil, ErrDisabled) when the config is disabled or has no endpoint — the cli treats both as "install the noop tracer" without crashing the agent.
The transport parameter is the egress-enforced http.RoundTripper the cli layer supplies (typically the SafeTransport wrapping http.DefaultTransport). For ProtocolHTTPProtobuf the transport is injected into the OTLP HTTP client so the exporter's traffic flows through the same allowlist + post-DNS IP guard every other in- process Forge HTTP client uses. For ProtocolGRPC the OTel grpc exporter dials directly via grpc.Dial — the egress enforcer cannot wrap that path, so for gRPC the endpoint host must be in the build- time egress allowlist and Forge relies on the NetworkPolicy + (Linux) the agent's outbound firewall to bound the traffic. HTTP is the recommended default for that reason.
Caller owns Shutdown. Phase 2's runner wiring calls `defer tp.Shutdown(ctx)` at the same lifecycle hook the audit-sink Close call lives at, so the batch processor flushes on a clean exit.
func WrapHTTPTransport ¶
func WrapHTTPTransport(base http.RoundTripper) http.RoundTripper
WrapHTTPTransport decorates an http.RoundTripper with OpenTelemetry HTTP instrumentation. Every request the wrapped transport handles produces an "http.client" span automatically — method, host, status code, and trace-context injection come for free, with no per-call-site changes.
Wiring is one line at the runner setup point: after the egress enforcer builds its transport, the runner passes it through this wrapper before stashing the http.Client downstream code uses. LLM providers / MCP clients / channel adapters that retrieve the egress-enforced client therefore get HTTP-level spans for free.
Nil-tolerant: passing nil returns nil so a caller that builds a transport conditionally (and decided not to) doesn't have to guard.
Span propagation: otelhttp.NewTransport reads the OTel global TextMapPropagator (Phase 0 installs traceparent + baggage). The outbound request therefore carries the parent span's traceparent header automatically — Phase 5 (#106) end-to-end propagation rides on this without further work here.
The instrumentation honors the global TracerProvider, so when the tracer is the noop (tracing disabled) the wrapper is effectively a pass-through — no per-request overhead beyond a single interface dispatch.
Types ¶
type SpanRecorder ¶
type SpanRecorder struct {
// contains filtered or unexported fields
}
SpanRecorder is an in-memory sdktrace.SpanExporter for tests. It captures every finished span and returns the captured slice on demand so test assertions can pin span hierarchy, attributes, and status without spinning up a real collector.
Production code must NEVER use this — the recorder retains every span forever. Phase 3 instrumentation tests (#104) and any future phase that asserts on spans use it via NewTestTracerProvider.
func NewTestTracerProvider ¶
func NewTestTracerProvider() (*sdktrace.TracerProvider, *SpanRecorder)
NewTestTracerProvider constructs an sdktrace.TracerProvider that records every span synchronously into the returned SpanRecorder. The processor is *SimpleSpanProcessor* (not BatchSpanProcessor) so tests do not need to call ForceFlush before reading recorded spans — every span is exported on End.
Sampler is AlwaysSample so a test can install the provider and immediately get spans regardless of the production sampler default.
Typical test setup:
tp, rec := observability.NewTestTracerProvider()
coreruntime.SetTracerProvider(tp)
defer coreruntime.ResetTracerProviderForTest()
defer tp.Shutdown(context.Background())
// ... exercise code under test ...
got := rec.FindSpans("agent.execute")
func (*SpanRecorder) ExportSpans ¶
func (r *SpanRecorder) ExportSpans(_ context.Context, spans []sdktrace.ReadOnlySpan) error
ExportSpans implements sdktrace.SpanExporter.
func (*SpanRecorder) FindSpan ¶
func (r *SpanRecorder) FindSpan(name string) (sdktrace.ReadOnlySpan, bool)
FindSpan returns the first recorded span whose name matches. Useful when a test produces one span per name (the common case for non-loop instrumentation). For loop sites (per-iteration spans of the same name), iterate Spans() directly.
func (*SpanRecorder) FindSpans ¶
func (r *SpanRecorder) FindSpans(name string) []sdktrace.ReadOnlySpan
FindSpans returns every recorded span whose name matches, in the order spans were exported.
func (*SpanRecorder) Reset ¶
func (r *SpanRecorder) Reset()
Reset clears every recorded span. Call between test sub-stages when reusing the same recorder.
func (*SpanRecorder) Shutdown ¶
func (r *SpanRecorder) Shutdown(_ context.Context) error
Shutdown implements sdktrace.SpanExporter. The recorder has no background goroutines so Shutdown is a no-op; the test owns the recorder's lifecycle.
func (*SpanRecorder) Spans ¶
func (r *SpanRecorder) Spans() []sdktrace.ReadOnlySpan
Spans returns a snapshot of every span recorded so far. Safe to call from any goroutine; the returned slice does not share backing memory with the recorder so a test may sort / mutate it freely.
type TracingConfig ¶
type TracingConfig struct {
// Enabled gates the whole subsystem. When false, NewTracerProvider
// returns (nil, ErrDisabled) and the cli installs the noop tracer.
// Default false (off-by-default per the initiative ruling).
Enabled bool
// Endpoint is the OTLP target URL. For http/protobuf:
// "https://otel-collector.svc.cluster.local:4318/v1/traces". For
// gRPC: "otel-collector.svc.cluster.local:4317". Required when
// Enabled is true; an empty endpoint with Enabled=true is treated
// the same as Enabled=false (the cli logs a warning and installs
// the noop tracer — never crash the agent over telemetry config).
Endpoint string
// Protocol selects the OTLP encoding + transport. One of
// ProtocolHTTPProtobuf (default) or ProtocolGRPC. The egress
// allowlist + NetworkPolicy generated by `forge package` (Phase 6)
// allowlists the endpoint host, not the encoding.
Protocol string
// Sampler is one of the SamplerXxx constants. Default
// SamplerParentBasedAlwaysOn — when an upstream caller's
// traceparent header says "sample," we honor it; when no parent
// context exists, we sample everything. Standard server-side
// posture.
Sampler string
// SamplerRatio applies to the *TraceIDRatio* samplers (0.0 — 1.0).
// Ignored for the always_on / always_off variants. Default 1.0.
SamplerRatio float64
// Headers are OTLP request headers (typically auth tokens).
// Prefer env-driven values in the cli layer so secrets don't
// end up committed to forge.yaml.
Headers map[string]string
// Timeout bounds each exporter request. Default 10s.
Timeout time.Duration
// ServiceName is OTel's `service.name` resource attribute, the
// primary identifier backends group spans by. The cli layer
// resolves $OTEL_SERVICE_NAME first, then falls back to the
// agent_id from forge.yaml. Required when Enabled is true.
ServiceName string
// ServiceVersion is OTel's `service.version`. Sourced from the
// agent's `version` in forge.yaml.
ServiceVersion string
// RuntimeVersion is the Forge build version, surfaced as
// `forge.runtime.version`. Lets backends compare agent runs across
// Forge upgrades.
RuntimeVersion string
// ResourceAttrs are additional resource attributes merged with the
// service.* / forge.* attributes above. The cli layer parses
// $OTEL_RESOURCE_ATTRIBUTES into this map.
ResourceAttrs map[string]string
// Redact and CaptureContent are consumed by the Phase 3 span
// instrumentation, not by this constructor — but they live in the
// config struct so the cli layer wires every knob in one place.
// Redact defaults true; CaptureContent defaults false (the
// enterprise opt-in for prompt/completion content per the
// initiative ruling).
Redact bool
CaptureContent bool
}
TracingConfig is the fully-resolved tracing configuration passed from the cli layer. Pure data — no env, no file lookups. Phase 2 (#103) resolves env / forge.yaml / CLI-flag precedence and hands a finished TracingConfig to NewTracerProvider.