Documentation
¶
Overview ¶
Package observability holds mast's telemetry surface: the FIXED Prometheus metric registry, and env-gated OTel trace-export setup.
Design contract (docs/observability-design.md):
- Metric names live here and only here. Callers increment pre-declared families through typed methods; they cannot mint new metric names or labels. That is the cardinality-control point (open question #5: specialists/workloads emit events, not metrics).
- The session eventlog is the source of truth; these metrics are a real-time *view* derived from the same event stream the budget meter folds (pkg/budget.Meter.Observe) — Observe here is shaped the same way and is fed from the same loop.
- Session ID is never a metric label (cardinality). Correlation at session grain goes through logs and traces.
- Traces are ADK v2's own span tree; mast decorates, it does not re-invent (see otel.go).
Index ¶
- Constants
- func SetupOTel(ctx context.Context) (shutdown func(context.Context) error, enabled bool, err error)
- type Registry
- func (r *Registry) A2ATask(workload, outcome string)
- func (r *Registry) AGUIRun(workload, outcome string)
- func (r *Registry) AGUIRunDuration(workload string, seconds float64)
- func (r *Registry) Abort(workload string)
- func (r *Registry) AddCost(workload string, usd float64)
- func (r *Registry) AutoResume(workload, outcome string)
- func (r *Registry) BudgetTrip(workload string)
- func (r *Registry) Dispatch(workload, outcome string)
- func (r *Registry) GatePause(workload, source string)
- func (r *Registry) HITLPause(workload string)
- func (r *Registry) HITLResume(workload string)
- func (r *Registry) Handler() http.Handler
- func (r *Registry) MarkerWriteFailure(workload, operation string)
- func (r *Registry) MonitorAck(workload, outcome string)
- func (r *Registry) MonitorDigestWake(workload string)
- func (r *Registry) MonitorNotify(workload, outcome string)
- func (r *Registry) Observe(ev *session.Event, workload string)
- func (r *Registry) ParkNotify(workload, outcome string)
- func (r *Registry) Prime(workload string)
- func (r *Registry) ProviderRetry(workload, outcome string)
- func (r *Registry) ScheduledFire(workload, outcome string)
- func (r *Registry) TimedPauseFire(workload, outcome string)
- func (r *Registry) TurnComplete(workload, outcome string)
Constants ¶
const ( OutcomeOK = "ok" OutcomeError = "error" OutcomeBudgetExceeded = "budget_exceeded" // OutcomeWatchdogHalt is a turn the behavioral watchdog stopped // under --watchdog=enforce. Distinct from error because it is the // backstop working, and distinct from budget_exceeded because the // session has runway left — an alert on this one should page // differently from either. OutcomeWatchdogHalt = "watchdog_halt" // OutcomeRefusalLoop is a turn the write gate stopped because the // model kept re-proposing a call an operator had already refused // (#449). Separate from watchdog_halt on the same reasoning that // made watchdog_halt separate from error, applied one level down: // this one latches nothing and needs no reset, so a rate on it is // a count of models arguing with their operators rather than a // queue of sessions waiting on a human. Separate from error // because nothing failed. OutcomeRefusalLoop = "refusal_loop" // OutcomeLoopStop is a turn the in-turn loop guard ended because // the model kept making one call, with the same arguments and the // same result, after a note on that result told it so (#514), or // because it reached budget.max_model_calls_per_turn (#519). It // latches nothing, like refusal_loop, and is separate from it for // the same reason refusal_loop is separate from watchdog_halt: the // remedy differs. This one points at the model or the workload's // tools, not at an operator's decision. OutcomeLoopStop = "loop_stop" )
Turn outcomes for TurnComplete. A fixed vocabulary — free-form outcome strings would be a label-cardinality leak.
const ( TokenKindPrompt = "prompt" TokenKindCandidates = "candidates" )
Token kinds for the mast_tokens_total{kind} label.
const ( MarkerOpMark = "mark" // MarkInterrupted (beginDrain / late-start mark) MarkerOpClear = "clear" // ClearInterrupted (clean finish during drain) MarkerOpPause = "pause" // PauseGate on a planned stop with --pause-sessions )
Marker-write operations for MarkerWriteFailure (mast_marker_write_failures_total{operation}). One per durable interruption-marker store write the shutdown drain performs; a non-zero count means an interrupted turn a restart may never surface (the Error log at the failure site names the session).
const ( GatePauseOperator = "operator" // POST /pause (incl. hard pause) GatePausePlannedStop = "planned_stop" // drain mark with --pause-sessions )
Gate-pause sources for GatePause (mast_gate_pauses_total{source}).
const ( // TimedPauseResumed: the timer drove a resume/consume to completion. TimedPauseResumed = "resumed" // TimedPauseSkipped: the fire was a benign no-op — an operator // resumed first, or the daemon was draining when the timer fired. TimedPauseSkipped = "skipped" // TimedPauseError: the resume/consume the timer attempted failed; // the timer is rescheduled and will retry. TimedPauseError = "error" )
Timed-pause fire outcomes (mast_timed_pause_fires_total{outcome}). A fixed vocabulary mirroring the scheduler fire callback's branches.
const ( // ScheduledFireRan: the tick drove a turn to completion. ScheduledFireRan = "ran" // ScheduledFireSkipped: the tick came due while the daemon was // draining, so no turn was started. Not an error and not retried — // the next tick is the retry. ScheduledFireSkipped = "skipped" // ScheduledFireError: the turn the tick started failed. The cadence // is unaffected; the next tick fires on schedule. ScheduledFireError = "error" // ScheduledFireMissed: the tick passed with nobody to run it — the // daemon was down, or a previous run overran the interval — and was // coalesced away rather than caught up. Incremented once per // skipped tick, so a crash-looping daemon is visible as a rising // missed count rather than as silence. ScheduledFireMissed = "missed" )
Scheduled-trigger outcomes (mast_scheduled_fires_total{outcome}) for the v0.4 W4.1 cadence. The family counts what the cadence DID with each tick, which is why three of the four outcomes are not runs: a scheduled workload that stopped doing its work is indistinguishable from a healthy one unless the ticks it declined to run are counted too.
const ( // MonitorNotifyPosted: a new message opened a timeline. MonitorNotifyPosted = "posted" // MonitorNotifyAppended: the assessment was added to the message the // previous speaking cycle posted. MonitorNotifyAppended = "appended" // MonitorNotifyReplaced: the append could not be applied (the // ingress had forgotten the message body) and the whole timeline was // re-sent as an edit. MonitorNotifyReplaced = "replaced" // MonitorNotifyRolled: the append overflowed the platform's message // limit and switchboard rolled the timeline into a threaded // continuation, which mast now addresses instead. MonitorNotifyRolled = "rolled" // MonitorNotifyQuiet: the classification was empty, so no model was // woken and nothing was sent. MonitorNotifyQuiet = "quiet" // MonitorNotifyHealth: mast spoke about itself — the monitoring // cycle failed, or recovered after failing. Counted apart from the // assessments because it is the one message a cycle sends that no // model wrote. MonitorNotifyHealth = "health" // MonitorNotifyError: the ingress refused, or could not be reached. // The cycle's work is not replayed; the next cycle is a fresher // sample. MonitorNotifyError = "error" )
Monitoring-notification outcomes (mast_monitor_notifications_total {outcome}) for the v0.5 W4.5 egress leg. The family counts what a monitoring cycle DID about telling somebody, which is why "quiet" is an outcome: a cycle that decided there was nothing to say is the intended common case, and a monitor whose chat has gone silent is only distinguishable from a healthy quiet one by whether the quiet count is still rising.
const ( // ParkNotifySent: the announcement reached the chat ingress. ParkNotifySent = "sent" // ParkNotifyThrottled: the announcement was dropped because the // park budget was spent. The park is unaffected and still // answerable; what was lost is the push. A rising count means a // workload is parking faster than an operator can be told, which is // itself worth an alert. ParkNotifyThrottled = "throttled" // ParkNotifyError: the ingress refused, or could not be reached. // There is no retry — see internal/notify — so this is a park the // operator was not told about. ParkNotifyError = "error" )
Park-announcement outcomes (mast_park_notifications_total{outcome}) for the #451 out-of-band half of a park. The family answers one question: did the operator get told this daemon stopped to ask something. A park the gate raised and nobody heard about is the failure the feature exists to remove, so both ways of not hearing about it are counted rather than logged only.
const ( // DispatchOK: the specialist ran to the end of its stream. DispatchOK = "ok" // DispatchHalted: a sink stopped the sub-run — a per-specialist // budget ceiling or a watchdog trip. mast's own decision, already // logged and already visible to the planner as a labelled partial; // counted here so the denominator is every dispatch. DispatchHalted = "halted" // DispatchRateLimited: the sub-runner failed and the failure reads // as the provider saying "not now" (429/RESOURCE_EXHAUSTED and // friends, per attach.ClassifyTurnError). // // Split out of DispatchFailed rather than folded into it because it // is the one dispatch failure that is both transient and the // operator's business: it means capacity, not a broken workload, // and burying it in a general failure count is what made it // invisible. // // Since #452 step 2 an outbound model call IS retried once, so a // dispatch landing here is one the retry did not save: either the // second attempt was rejected too, or the process-wide cooldown // declined it. Which of the two is in // mast_provider_retries_total{outcome}, and the pair is worth // reading together — a rate-limited dispatch alongside a `declined` // retry says the fan-out is wider than the quota, which is a // workload-shape problem rather than a provider problem. DispatchRateLimited = "rate_limited" // DispatchFailed: the sub-runner failed some other way. Excludes // DispatchRateLimited, so the two sum to every failed dispatch. DispatchFailed = "failed" )
Planner-dispatch outcomes (mast_dispatches_total{workload,outcome}) for the #452 visibility half. One count per invoke_specialist dispatch, closed however it closed.
The family exists because a dispatch that dies under the specialist is the one event on this seam that produces nothing else: a completed dispatch produces a result, a halt produces a labelled partial and a log line, and a dispatch killed by a provider rejection produced — until this — a tool error the planner was free to absorb by doing the specialist's work itself. Upstream measured what that costs: four runs in a 38-run archive silently stopped being delegated runs, at full parent-context cost, and still scored.
const ( // ProviderRetryRecovered: a retry was served and the call then // produced content. The provider's "not now" cost a wait instead of // a result, which is the whole point of the family — without it a // provider under worsening pressure produces green, increasingly // slow turns with nothing to point at. ProviderRetryRecovered = "recovered" // ProviderRetryExhausted: a retry was served and the call failed // anyway. The caller got the provider's error; whatever was // downstream of that call ended short. ProviderRetryExhausted = "exhausted" // ProviderRetryDeclined: the error qualified for a retry and the // process-wide cooldown refused one, because this process already // spent a retry inside the window. // // Its own outcome rather than folded into exhausted because it is a // fact about mast and not about the provider: it means this process // is meeting rejections faster than one a minute, which is what a // fan-out wider than its quota looks like from the inside. An // operator acts on it by narrowing the roster or raising the quota, // neither of which is the response to an exhausted retry. ProviderRetryDeclined = "declined" )
Provider-retry outcomes (mast_provider_retries_total{workload,outcome}) for the #452 resilience half.
One count per model call that met a transient provider rejection — not per call. The denominator is mast_model_calls_total, and an event per successful call would bury the handful that matter: mast's own measurement of this is three rows in thirty-one on one night (internal/modelretry).
Distinct from mast_dispatches_total, which counts delegations. A retry happens on any outbound model call — the planner's own turn, a specialist's, a monitoring cycle's — and most of them are not dispatches at all.
const ( // MonitorAckForwarded: the producer's ack tool accepted it. MonitorAckForwarded = "forwarded" // MonitorAckError: it did not. The attributed record is still // durable — mast knows who asked and that the suppression did not // take, which is the state an operator has to be able to see. MonitorAckError = "error" )
Monitoring-ack outcomes (mast_monitor_acks_total{outcome}) for the v0.5 W4.6 ingress leg. Two outcomes and no more: mast either got the operator's acknowledgement to the producer that owns the suppression, or it did not. Whether a suppression was redundant, already in force, or applied to a subject nothing is currently reporting are all the producer's questions, and counting mast's guesses at them here would be counting an opinion mast is careful not to have.
const ( // AutoResumeResumed: a continuation turn ran to completion and the // interruption marker was cleared. AutoResumeResumed = "resumed" // AutoResumeCleared: the trailing event was already a clean model // turn (stale marker / clear race); the marker was cleared without // running a turn. AutoResumeCleared = "cleared" // AutoResumeSkippedStale: the interruption is older than the // freshness window; the marker was left for an operator. AutoResumeSkippedStale = "skipped_stale" // AutoResumeSkippedAmbiguous: the session carries a dangling mutating // intent (ambiguous effect); left for an operator ack. AutoResumeSkippedAmbiguous = "skipped_ambiguous" // AutoResumeSkippedLoopbreak: the per-session attempt breaker or the // per-boot cap tripped. AutoResumeSkippedLoopbreak = "skipped_loopbreak" // AutoResumeSkippedSuperseded: a concurrent turn advanced the session // after the scan (M1 TOCTOU recheck). AutoResumeSkippedSuperseded = "skipped_superseded" // AutoResumeSkippedUnsupported: a shape slice-1 does not drive // (non-coordinator dispatch, dangling sub-agent delegation, or a // multi-event repair). AutoResumeSkippedUnsupported = "skipped_unsupported" // AutoResumeError: the continuation turn was attempted and failed; // the marker was left in place. AutoResumeError = "error" )
Auto-resume outcomes for AutoResume (mast_autoresume_total{outcome}). A fixed vocabulary, mirroring cmd/mast's boot-time auto-resume decision tree (#41): every interrupted candidate the boot pass inspects lands in exactly one of these.
const ( A2ATaskSubmitted = "submitted" A2ATaskWorking = "working" A2ATaskInputRequired = "input-required" A2ATaskCompleted = "completed" A2ATaskFailed = "failed" A2ATaskCanceled = "canceled" A2ATaskRejected = "rejected" )
A2A server task outcomes for A2ATask (mast_a2a_server_tasks_total {workload,outcome}). The vocabulary mirrors the A2A task-lifecycle states mast reports (docs/a2a-design.md "Task lifecycle mapping"); the string VALUES match internal/a2a's TaskState constants — the server passes string(state), so these two lists must not drift.
const ( AGUIRunSuccess = "success" AGUIRunInterrupted = "interrupted" AGUIRunError = "error" AGUIRunAborted = "aborted" AGUIRunRejected = "rejected" // AGUIRunQueueFull counts runs refused because the addressed thread // already had agui.run_queue.depth+1 runs in flight. Separate from // Rejected because it is the refusal an operator answers with capacity or // a bundle key rather than with a caller's credentials. AGUIRunQueueFull = "queue_full" )
AG-UI server run outcomes for AGUIRun (mast_agui_runs_total {workload,outcome}). A fixed vocabulary mirroring the AG-UI run dispositions the server reports (docs/ag-ui-design.md): a completed run, a run that paused for human input (a clean interrupt, resumable via RunAgentInput.Resume), an errored run, an operator/client abort, and a pre-stream refusal (auth/scope/rate-limit/drain/not-resumable), and a run shed because its thread's queue was full. The string VALUES match internal/agui's internal outcome constants — the server passes them through, so these two lists must not drift. Both sides are pinned to the same literals: internal/agui's own test locks its unexported constants, and a cmd/mast test locks these exported ones, so a move on either side fails a build.
Variables ¶
This section is empty.
Functions ¶
func SetupOTel ¶
SetupOTel installs the global OTLP trace exporter + W3C propagator when standard OTel env config asks for it (OTEL_EXPORTER_OTLP_ENDPOINT or OTEL_EXPORTER_OTLP_TRACES_ENDPOINT). Endpoint, headers, protocol details, etc. are all read from the environment by the exporter — mast adds nothing beyond a service.name resource default.
mast does not open custom spans in v0.1: ADK v2's runner emits the unified span tree (session/turn/node/tool), and mast only decorates. This function only makes that tree leave the process.
Returns a shutdown func (flushes the batch exporter) and whether export was enabled. When the env vars are absent it is a no-op: shutdown is non-nil and trivially succeeds.
Types ¶
type Registry ¶
type Registry struct {
// contains filtered or unexported fields
}
Registry is the fixed set of mast metric families. Construct one per process with New and expose it via Handler on the inject listener.
func New ¶
func New() *Registry
New constructs the registry with every base family pre-registered. The underlying prometheus.Registry is private: nothing outside this package can register additional collectors through it.
func (*Registry) A2ATask ¶
A2ATask records one A2A server task lifecycle transition with the given outcome (one of the A2ATask* constants; a internal/a2a TaskState value). Safe on a nil *Registry.
func (*Registry) AGUIRun ¶
AGUIRun records one AG-UI server run outcome (one of the AGUIRun* constants; a internal/agui outcome value). Safe on a nil *Registry.
func (*Registry) AGUIRunDuration ¶
AGUIRunDuration records one AG-UI run's wallclock duration in seconds. Safe on a nil *Registry; negative durations are ignored.
func (*Registry) Abort ¶
Abort records a terminal abort durably recorded through the operator surface (the marker write succeeded; the in-flight turn is swept separately).
func (*Registry) AddCost ¶
AddCost accumulates spend (in USD) attributed to a workload. The amount comes from the budget meter — pricing stays in one place; this is only the export surface. Non-positive deltas are ignored.
func (*Registry) AutoResume ¶
AutoResume records one boot-time auto-resume decision with the given outcome (one of the AutoResume* constants).
func (*Registry) BudgetTrip ¶
BudgetTrip records a turn aborted on a budget ceiling.
func (*Registry) Dispatch ¶
Dispatch records how one planner invoke_specialist dispatch ended (one of the Dispatch* constants). Called once per dispatch, from the host's SubRunSink.Close.
Counted per dispatch and not per specialist: the specialist name is workload-authored and unbounded, and the question this family answers — is this workload losing delegations, and to what — does not need it. The log line the same close writes names the specialist, which is where an operator goes once the count says to look.
func (*Registry) GatePause ¶
GatePause records a gate pause durably recorded, by source (one of the GatePause* constants: an operator request or a planned stop).
func (*Registry) HITLPause ¶
HITLPause records a HITL interrupt explicitly, for callers that detect the pause outside the event stream. Callers already feeding events through Observe must not also call this for the same interrupt (Observe counts RequestedInput events itself).
func (*Registry) HITLResume ¶
HITLResume records an operator resume being fed into a session.
func (*Registry) Handler ¶
Handler returns the Prometheus scrape handler for this registry, suitable for mounting at /metrics on an existing mux.
func (*Registry) MarkerWriteFailure ¶
MarkerWriteFailure records a failed interruption-marker store write during the shutdown drain, by operation (one of the MarkerOp* constants). A non-zero count is an alert condition: a marker that did not land means a restart may never surface the interrupted turn.
func (*Registry) MonitorAck ¶
MonitorAck records one operator acknowledgement taken on the ingress (one of the MonitorAck* constants). Counted per ack rather than per subject: two operators acking the same finding is two people asking for quiet, and an audit that collapsed them would lose the second.
func (*Registry) MonitorDigestWake ¶
MonitorDigestWake records a cycle that spoke because the notify deadman expired rather than because the classification changed. Its own family rather than a notification outcome: the operator question it answers is "has this workload been silent long enough that I had to prove I was alive", which is not the same question as how the message was delivered.
func (*Registry) MonitorNotify ¶
MonitorNotify records what one monitoring cycle did about telling somebody (one of the MonitorNotify* constants). Called once per cycle that reached the egress leg, including the quiet ones.
func (*Registry) Observe ¶
Observe folds one runner event into the counters. Shaped like pkg/budget's Meter.Observe so both hooks sit side by side on the event-stream loop. Events without UsageMetadata contribute no model-call or token counts; nil events are ignored. Safe on a nil *Registry so callers can leave telemetry unwired.
func (*Registry) ParkNotify ¶
ParkNotify records what became of one park's out-of-band announcement (one of the ParkNotify* constants). Called once per park on a daemon with a park egress configured, and not at all on one without — a workload that never asked to be told is not counted as failing to be told.
func (*Registry) Prime ¶
Prime materializes every family's time series for the given workload at zero, so a scrape sees all base families from process start (before the first turn) and PromQL rate()/increase() have a defined origin. Call once at startup per served workload.
func (*Registry) ProviderRetry ¶
ProviderRetry records one model call's brush with a transient provider rejection (one of the ProviderRetry* constants). Called from the retry policy's observer, which the host installs at startup.
Not labelled by model, though the retry policy knows the name. A `model:` override is workload-authored, so the label set would be unbounded in the same way a specialist name is — and the question this family answers is whether this workload is being shed, which the log line names the model for.
func (*Registry) ScheduledFire ¶
ScheduledFire records one scheduled-trigger tick with the given outcome (one of the ScheduledFire* constants). Called once per tick, including the ticks that produced no run at all.
func (*Registry) TimedPauseFire ¶
TimedPauseFire records one timed-pause scheduler fire with the given outcome (one of the TimedPause* constants).
func (*Registry) TurnComplete ¶
TurnComplete records one finished turn with the given outcome (one of the Outcome* constants).