Documentation
¶
Overview ¶
Package telemetry records and exports the CLI's OpenTelemetry metrics and traces.
Metrics: tool outcomes, token usage, and sessions, recorded into OTel SDK instruments named per the GenAI semantic conventions and infer-action's exporter, so they line up with the gateway's OTLP ingest and existing dashboards.
Traces: one root span per session, child spans for each LLM turn and each tool call. No prompt/response content is recorded.
Both signals share the same resource and OTLP endpoint/headers config. Local file export is always attempted; OTLP/HTTP export is opt-in via an endpoint (config or OTEL_EXPORTER_OTLP_ENDPOINT). Metrics use delta temporality (required by the gateway ingest, and what makes the local files trivially summable by `infer stats`).
Index ¶
- Constants
- Variables
- func Archive(dir string, cutoff time.Time)
- func NewToolService(inner agentdomain.ToolService, rec *Recorder) agentdomain.ToolService
- func ParseSince(s string) (time.Time, error)
- func PropagationTransport(base http.RoundTripper) http.RoundTripper
- func SetSpanError(ctx context.Context, err error)
- func SetSpanUsage(ctx context.Context, inputTokens, outputTokens int)
- type CostFunc
- type ModelStat
- type Options
- type Recorder
- func (r *Recorder) ChildEnv(ctx context.Context) []string
- func (r *Recorder) Flush(ctx context.Context)
- func (r *Recorder) Meter() metric.Meter
- func (r *Recorder) RecordSession(mode, outcome string, dur time.Duration)
- func (r *Recorder) RecordTool(tool, outcome, errType string, dur time.Duration)
- func (r *Recorder) RecordUsage(model string, prompt, completion, cached, cacheWrite int)
- func (r *Recorder) SetConversationID(id string)
- func (r *Recorder) Shutdown(ctx context.Context)
- func (r *Recorder) SpanContext(ctx context.Context) context.Context
- func (r *Recorder) StartLLMTurnSpan(ctx context.Context, model string) (context.Context, trace.Span)
- func (r *Recorder) StartSession(mode string) func(outcome string)
- func (r *Recorder) Tracer() trace.Tracer
- type SessionStat
- type Stats
- type ToolStat
- type TraceSession
- type TraceSpan
Constants ¶
const ( ExecInteractive = "interactive" ExecHeadless = "headless" ExecDaemon = "daemon" )
Execution modes (resource attribute infer.execution.mode).
const ( ToolSuccess = "success" ToolError = "error" ToolRejected = "rejected" ErrTypeTool = "tool_error" )
Tool outcomes (attribute infer.tool.outcome; error.type on non-success).
const ( RunSuccess = "success" RunFailed = "failed" RunStoppedEarly = "stopped_early" )
Session/run outcomes (attribute infer.run.outcome) - infer-action's enum.
Variables ¶
var ( Version = "dev" ExecutionMode = ExecHeadless )
Process-wide facts stamped onto every metric via the resource. Version is the build version; ExecutionMode distinguishes interactive chat from headless `infer headless`. cmd sets these before building the service container.
Functions ¶
func Archive ¶
Archive moves session files older than cutoff into an archive/ subdir instead of deleting them; Aggregate's non-recursive glob then skips them. Best-effort.
func NewToolService ¶
func NewToolService(inner agentdomain.ToolService, rec *Recorder) agentdomain.ToolService
NewToolService wraps inner so tool executions are recorded. The container only applies this when rec is non-nil, so the disabled tool path carries no decorator at all.
func ParseSince ¶
ParseSince converts a window like "7d"/"24h"/"30m" to an absolute cutoff. Empty means all time (zero cutoff). time.ParseDuration has no day unit, so "Nd" is handled here.
func PropagationTransport ¶
func PropagationTransport(base http.RoundTripper) http.RoundTripper
PropagationTransport returns an http.RoundTripper that injects W3C trace-context and baggage headers into every outgoing request. When base is nil, http.DefaultTransport is used.
func SetSpanError ¶
SetSpanError marks the span in ctx failed: error.type attribute, recorded error event, and Error status, per the semconv recording-errors rules.
func SetSpanUsage ¶
SetSpanUsage stamps token usage (gen_ai.usage.*) onto the span in ctx.
Types ¶
type CostFunc ¶
type CostFunc func(model string, prompt, completion, cached, cacheWrite int) (input, output, total float64)
CostFunc returns the input, output, and total cost for a model's token counts (wraps convdomain.PricingService.CalculateCost). cached and cacheWrite are the cache-read and cache-creation subsets of prompt tokens. Pass nil to skip cost.
type ModelStat ¶
type ModelStat struct {
Model string `json:"model"`
Prompt int `json:"prompt"`
Cached int `json:"cached"`
Completion int `json:"completion"`
Total int `json:"total"`
Cost float64 `json:"cost"`
}
ModelStat aggregates token usage and cost for one model. Cached is the cached-prompt subset of Prompt (not added to Total); it stays 0 for telemetry files written before cache_read datapoints existed.
type Options ¶
type Options struct {
Enabled bool
Dir string
SessionID string
OTLPEndpoint string
OTLPHeaders map[string]string
OTLPInterval time.Duration
ReceiverAddress string
Cost CostFunc
// AttrSessionIDKey / AttrToolCallIDKey override the baggage member names;
// empty falls back to the OTel semconv defaults.
AttrSessionIDKey string
AttrToolCallIDKey string
}
Options configures a Recorder. Dir + SessionID locate the per-process local file; OTLP* enable the optional remote export.
type Recorder ¶
type Recorder struct {
// contains filtered or unexported fields
}
Recorder maps recorded events onto OTel instruments and spans. A nil *Recorder is a valid no-op, so callers guard hot paths with `if rec != nil` and the container skips wrapping when disabled.
func New ¶
New builds a Recorder, or nil when disabled or no sink could be created. The local file sink is always attempted; the OTLP sink is added when an endpoint is configured (or OTEL_EXPORTER_OTLP_ENDPOINT is set). Sink failures are logged and dropped so telemetry never breaks a run.
func (*Recorder) ChildEnv ¶
ChildEnv returns the W3C trace-context environment for a subprocess launched under the current tool span, plus its OTLP span sink: a configured endpoint passes through, an env-configured one is inherited, otherwise the ephemeral loopback receiver keeps child spans in the local session store. Nil when the recorder is nil or no span is active.
func (*Recorder) Flush ¶
Flush forces an immediate export of everything recorded so far, without tearing down (used by tests and callers that want data on disk now). Safe on nil.
func (*Recorder) Meter ¶
Meter returns the meter from the provider, or nil when the recorder is nil or the provider is nil. Used by subsystems (e.g. the daemon) to register their own instruments on the shared meter provider.
func (*Recorder) RecordSession ¶
RecordSession records one completed agent session (infer.agent.runs + infer.agent.run.duration). outcome is one of RunSuccess/RunFailed/RunStoppedEarly.
func (*Recorder) RecordTool ¶
RecordTool records one tool execution (infer.agent.tool.calls + gen_ai.execute_tool.duration).
func (*Recorder) RecordUsage ¶
RecordUsage records one request's token usage (gen_ai.client.token.usage, one datapoint per token type) and the derived infer.client.cost split. cached and cacheWrite are the cache-read and cache-creation subsets of prompt tokens; their datapoints are only emitted when non-zero.
func (*Recorder) SetConversationID ¶
SetConversationID tags subsequent metric datapoints with the given conversation id (gen_ai.conversation.id) so aggregation can scope to it. No-op on a nil recorder or empty id.
func (*Recorder) Shutdown ¶
Shutdown flushes the final export and releases resources for both OTel signals. Safe on nil.
func (*Recorder) SpanContext ¶
SpanContext grafts the session root span onto ctx so spans created from the returned context parent to it. Safe on nil.
func (*Recorder) StartLLMTurnSpan ¶
func (r *Recorder) StartLLMTurnSpan(ctx context.Context, model string) (context.Context, trace.Span)
StartLLMTurnSpan creates a span for one LLM request with GenAI semconv attributes and CLIENT kind (a remote call to the gateway). Safe on nil (returns ctx unchanged and a no-op span).
func (*Recorder) StartSession ¶
StartSession begins the session root span. The returned end function stamps infer.run.outcome and ends the span. Safe on nil.
type SessionStat ¶
type SessionStat struct {
Execution string `json:"execution"`
Mode string `json:"mode"`
Count int `json:"count"`
}
SessionStat counts sessions by execution mode (interactive/headless) and agent mode.
type Stats ¶
type Stats struct {
Tools []ToolStat `json:"tools"`
Models []ModelStat `json:"models"`
Sessions []SessionStat `json:"sessions"`
Empty bool `json:"-"`
}
Stats is the `infer stats` aggregate over the local telemetry files. Empty is true when no datapoints matched (clean empty-store render).
type ToolStat ¶
type ToolStat struct {
Name string `json:"name"`
Calls int `json:"calls"`
Failures int `json:"failures"`
AvgMs int64 `json:"avg_ms"`
}
ToolStat aggregates one tool. Failures counts the error outcome (a rejection is not a failure). AvgMs is the mean execution duration.
type TraceSession ¶
TraceSession identifies one session with a non-empty local trace file (<dir>/<id>-traces.jsonl).
func TraceSessions ¶
func TraceSessions(dir string) ([]TraceSession, error)
TraceSessions lists sessions with non-empty trace files under dir, newest first. Empty files (sessions that recorded no spans) are skipped.
type TraceSpan ¶
type TraceSpan struct {
Name string `json:"name"`
Start time.Time `json:"start"`
DurationMs float64 `json:"duration_ms"`
Error string `json:"error,omitempty"`
Attributes map[string]string `json:"attributes,omitempty"`
Children []*TraceSpan `json:"children,omitempty"`
}
TraceSpan is one node of a session's span tree.
func LoadTraceTree ¶
LoadTraceTree reads a session's trace file and assembles the span tree. Spans whose parent is not in the file (the root, or orphans from a partial file) become roots; siblings are ordered by start time. Malformed lines are skipped best-effort.
Source Files