Documentation
¶
Overview ¶
Package observability holds mast's telemetry surface: the FIXED Prometheus metric registry, and env-gated OTel trace-export setup.
Design contract (docs/observability-design.md):
- Metric names live here and only here. Callers increment pre-declared families through typed methods; they cannot mint new metric names or labels. That is the cardinality-control point (open question #5: specialists/workloads emit events, not metrics).
- The session eventlog is the source of truth; these metrics are a real-time *view* derived from the same event stream the budget meter folds (pkg/budget.Meter.Observe) — Observe here is shaped the same way and is fed from the same loop.
- Session ID is never a metric label (cardinality). Correlation at session grain goes through logs and traces.
- Traces are ADK v2's own span tree; mast decorates, it does not re-invent (see otel.go).
Index ¶
- Constants
- func SetupOTel(ctx context.Context) (shutdown func(context.Context) error, enabled bool, err error)
- type Registry
- func (r *Registry) A2ATask(workload, outcome string)
- func (r *Registry) AGUIRun(workload, outcome string)
- func (r *Registry) AGUIRunDuration(workload string, seconds float64)
- func (r *Registry) Abort(workload string)
- func (r *Registry) AddCost(workload string, usd float64)
- func (r *Registry) AutoResume(workload, outcome string)
- func (r *Registry) BudgetTrip(workload string)
- func (r *Registry) GatePause(workload, source string)
- func (r *Registry) HITLPause(workload string)
- func (r *Registry) HITLResume(workload string)
- func (r *Registry) Handler() http.Handler
- func (r *Registry) MarkerWriteFailure(workload, operation string)
- func (r *Registry) Observe(ev *session.Event, workload string)
- func (r *Registry) Prime(workload string)
- func (r *Registry) ScheduledFire(workload, outcome string)
- func (r *Registry) TimedPauseFire(workload, outcome string)
- func (r *Registry) TurnComplete(workload, outcome string)
Constants ¶
const ( OutcomeOK = "ok" OutcomeError = "error" OutcomeBudgetExceeded = "budget_exceeded" // OutcomeWatchdogHalt is a turn the behavioral watchdog stopped // under --watchdog=enforce. Distinct from error because it is the // backstop working, and distinct from budget_exceeded because the // session has runway left — an alert on this one should page // differently from either. OutcomeWatchdogHalt = "watchdog_halt" )
Turn outcomes for TurnComplete. A fixed vocabulary — free-form outcome strings would be a label-cardinality leak.
const ( TokenKindPrompt = "prompt" TokenKindCandidates = "candidates" )
Token kinds for the mast_tokens_total{kind} label.
const ( MarkerOpMark = "mark" // MarkInterrupted (beginDrain / late-start mark) MarkerOpClear = "clear" // ClearInterrupted (clean finish during drain) MarkerOpPause = "pause" // PauseGate on a planned stop with --pause-sessions )
Marker-write operations for MarkerWriteFailure (mast_marker_write_failures_total{operation}). One per durable interruption-marker store write the shutdown drain performs; a non-zero count means an interrupted turn a restart may never surface (the Error log at the failure site names the session).
const ( GatePauseOperator = "operator" // POST /pause (incl. hard pause) GatePausePlannedStop = "planned_stop" // drain mark with --pause-sessions )
Gate-pause sources for GatePause (mast_gate_pauses_total{source}).
const ( // TimedPauseResumed: the timer drove a resume/consume to completion. TimedPauseResumed = "resumed" // TimedPauseSkipped: the fire was a benign no-op — an operator // resumed first, or the daemon was draining when the timer fired. TimedPauseSkipped = "skipped" // TimedPauseError: the resume/consume the timer attempted failed; // the timer is rescheduled and will retry. TimedPauseError = "error" )
Timed-pause fire outcomes (mast_timed_pause_fires_total{outcome}). A fixed vocabulary mirroring the scheduler fire callback's branches.
const ( // ScheduledFireRan: the tick drove a turn to completion. ScheduledFireRan = "ran" // ScheduledFireSkipped: the tick came due while the daemon was // draining, so no turn was started. Not an error and not retried — // the next tick is the retry. ScheduledFireSkipped = "skipped" // ScheduledFireError: the turn the tick started failed. The cadence // is unaffected; the next tick fires on schedule. ScheduledFireError = "error" // ScheduledFireMissed: the tick passed with nobody to run it — the // daemon was down, or a previous run overran the interval — and was // coalesced away rather than caught up. Incremented once per // skipped tick, so a crash-looping daemon is visible as a rising // missed count rather than as silence. ScheduledFireMissed = "missed" )
Scheduled-trigger outcomes (mast_scheduled_fires_total{outcome}) for the v0.4 W4.1 cadence. The family counts what the cadence DID with each tick, which is why three of the four outcomes are not runs: a scheduled workload that stopped doing its work is indistinguishable from a healthy one unless the ticks it declined to run are counted too.
const ( // AutoResumeResumed: a continuation turn ran to completion and the // interruption marker was cleared. AutoResumeResumed = "resumed" // AutoResumeCleared: the trailing event was already a clean model // turn (stale marker / clear race); the marker was cleared without // running a turn. AutoResumeCleared = "cleared" // AutoResumeSkippedStale: the interruption is older than the // freshness window; the marker was left for an operator. AutoResumeSkippedStale = "skipped_stale" // AutoResumeSkippedAmbiguous: the session carries a dangling mutating // intent (ambiguous effect); left for an operator ack. AutoResumeSkippedAmbiguous = "skipped_ambiguous" // AutoResumeSkippedLoopbreak: the per-session attempt breaker or the // per-boot cap tripped. AutoResumeSkippedLoopbreak = "skipped_loopbreak" // AutoResumeSkippedSuperseded: a concurrent turn advanced the session // after the scan (M1 TOCTOU recheck). AutoResumeSkippedSuperseded = "skipped_superseded" // AutoResumeSkippedUnsupported: a shape slice-1 does not drive // (non-coordinator dispatch, dangling sub-agent delegation, or a // multi-event repair). AutoResumeSkippedUnsupported = "skipped_unsupported" // AutoResumeError: the continuation turn was attempted and failed; // the marker was left in place. AutoResumeError = "error" )
Auto-resume outcomes for AutoResume (mast_autoresume_total{outcome}). A fixed vocabulary, mirroring cmd/mast's boot-time auto-resume decision tree (#41): every interrupted candidate the boot pass inspects lands in exactly one of these.
const ( A2ATaskSubmitted = "submitted" A2ATaskWorking = "working" A2ATaskInputRequired = "input-required" A2ATaskCompleted = "completed" A2ATaskFailed = "failed" A2ATaskCanceled = "canceled" A2ATaskRejected = "rejected" )
A2A server task outcomes for A2ATask (mast_a2a_server_tasks_total {workload,outcome}). The vocabulary mirrors the A2A task-lifecycle states mast reports (docs/a2a-design.md "Task lifecycle mapping"); the string VALUES match pkg/a2a's TaskState constants — the server passes string(state), so these two lists must not drift.
const ( AGUIRunSuccess = "success" AGUIRunInterrupted = "interrupted" AGUIRunError = "error" AGUIRunAborted = "aborted" AGUIRunRejected = "rejected" )
AG-UI server run outcomes for AGUIRun (mast_agui_runs_total {workload,outcome}). A fixed vocabulary mirroring the AG-UI run dispositions the server reports (docs/ag-ui-design.md): a completed run, a run that paused for human input (a clean interrupt, resumable via RunAgentInput.Resume), an errored run, an operator/client abort, and a pre-stream refusal (auth/scope/rate-limit/drain/not-resumable). The string VALUES match pkg/agui's internal outcome constants — the server passes them through, so these two lists must not drift. Both sides are pinned to the same literals: pkg/agui's own test locks its unexported constants, and a cmd/mast test locks these exported ones, so a move on either side fails a build.
Variables ¶
This section is empty.
Functions ¶
func SetupOTel ¶
SetupOTel installs the global OTLP trace exporter + W3C propagator when standard OTel env config asks for it (OTEL_EXPORTER_OTLP_ENDPOINT or OTEL_EXPORTER_OTLP_TRACES_ENDPOINT). Endpoint, headers, protocol details, etc. are all read from the environment by the exporter — mast adds nothing beyond a service.name resource default.
mast does not open custom spans in v0.1: ADK v2's runner emits the unified span tree (session/turn/node/tool), and mast only decorates. This function only makes that tree leave the process.
Returns a shutdown func (flushes the batch exporter) and whether export was enabled. When the env vars are absent it is a no-op: shutdown is non-nil and trivially succeeds.
Types ¶
type Registry ¶
type Registry struct {
// contains filtered or unexported fields
}
Registry is the fixed set of mast metric families. Construct one per process with New and expose it via Handler on the inject listener.
func New ¶
func New() *Registry
New constructs the registry with every base family pre-registered. The underlying prometheus.Registry is private: nothing outside this package can register additional collectors through it.
func (*Registry) A2ATask ¶ added in v0.2.0
A2ATask records one A2A server task lifecycle transition with the given outcome (one of the A2ATask* constants; a pkg/a2a TaskState value). Safe on a nil *Registry.
func (*Registry) AGUIRun ¶ added in v0.2.0
AGUIRun records one AG-UI server run outcome (one of the AGUIRun* constants; a pkg/agui outcome value). Safe on a nil *Registry.
func (*Registry) AGUIRunDuration ¶ added in v0.2.0
AGUIRunDuration records one AG-UI run's wallclock duration in seconds. Safe on a nil *Registry; negative durations are ignored.
func (*Registry) Abort ¶ added in v0.2.0
Abort records a terminal abort durably recorded through the operator surface (the marker write succeeded; the in-flight turn is swept separately).
func (*Registry) AddCost ¶
AddCost accumulates spend (in USD) attributed to a workload. The amount comes from the budget meter — pricing stays in one place; this is only the export surface. Non-positive deltas are ignored.
func (*Registry) AutoResume ¶ added in v0.2.0
AutoResume records one boot-time auto-resume decision with the given outcome (one of the AutoResume* constants).
func (*Registry) BudgetTrip ¶
BudgetTrip records a turn aborted on a budget ceiling.
func (*Registry) GatePause ¶ added in v0.2.0
GatePause records a gate pause durably recorded, by source (one of the GatePause* constants: an operator request or a planned stop).
func (*Registry) HITLPause ¶
HITLPause records a HITL interrupt explicitly, for callers that detect the pause outside the event stream. Callers already feeding events through Observe must not also call this for the same interrupt (Observe counts RequestedInput events itself).
func (*Registry) HITLResume ¶
HITLResume records an operator resume being fed into a session.
func (*Registry) Handler ¶
Handler returns the Prometheus scrape handler for this registry, suitable for mounting at /metrics on an existing mux.
func (*Registry) MarkerWriteFailure ¶ added in v0.2.0
MarkerWriteFailure records a failed interruption-marker store write during the shutdown drain, by operation (one of the MarkerOp* constants). A non-zero count is an alert condition: a marker that did not land means a restart may never surface the interrupted turn.
func (*Registry) Observe ¶
Observe folds one runner event into the counters. Shaped like pkg/budget's Meter.Observe so both hooks sit side by side on the event-stream loop. Events without UsageMetadata contribute no model-call or token counts; nil events are ignored. Safe on a nil *Registry so callers can leave telemetry unwired.
func (*Registry) Prime ¶
Prime materializes every family's time series for the given workload at zero, so a scrape sees all base families from process start (before the first turn) and PromQL rate()/increase() have a defined origin. Call once at startup per served workload.
func (*Registry) ScheduledFire ¶ added in v0.4.0
ScheduledFire records one scheduled-trigger tick with the given outcome (one of the ScheduledFire* constants). Called once per tick, including the ticks that produced no run at all.
func (*Registry) TimedPauseFire ¶ added in v0.2.0
TimedPauseFire records one timed-pause scheduler fire with the given outcome (one of the TimedPause* constants).
func (*Registry) TurnComplete ¶
TurnComplete records one finished turn with the given outcome (one of the Outcome* constants).