observability

package
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 10, 2026 License: Apache-2.0 Imports: 13 Imported by: 0

Documentation

Overview

Package observability holds mast's telemetry surface: the FIXED Prometheus metric registry, and env-gated OTel trace-export setup.

Design contract (docs/observability-design.md):

  • Metric names live here and only here. Callers increment pre-declared families through typed methods; they cannot mint new metric names or labels. That is the cardinality-control point (open question #5: specialists/workloads emit events, not metrics).
  • The session eventlog is the source of truth; these metrics are a real-time *view* derived from the same event stream the budget meter folds (pkg/budget.Meter.Observe) — Observe here is shaped the same way and is fed from the same loop.
  • Session ID is never a metric label (cardinality). Correlation at session grain goes through logs and traces.
  • Traces are ADK v2's own span tree; mast decorates, it does not re-invent (see otel.go).

Index

Constants

View Source
const (
	OutcomeOK             = "ok"
	OutcomeError          = "error"
	OutcomeBudgetExceeded = "budget_exceeded"
)

Turn outcomes for TurnComplete. A fixed vocabulary — free-form outcome strings would be a label-cardinality leak.

View Source
const (
	TokenKindPrompt     = "prompt"
	TokenKindCandidates = "candidates"
)

Token kinds for the mast_tokens_total{kind} label.

View Source
const (
	MarkerOpMark  = "mark"  // MarkInterrupted (beginDrain / late-start mark)
	MarkerOpClear = "clear" // ClearInterrupted (clean finish during drain)
	MarkerOpPause = "pause" // PauseGate on a planned stop with --pause-sessions
)

Marker-write operations for MarkerWriteFailure (mast_marker_write_failures_total{operation}). One per durable interruption-marker store write the shutdown drain performs; a non-zero count means an interrupted turn a restart may never surface (the Error log at the failure site names the session).

View Source
const (
	GatePauseOperator    = "operator"     // POST /pause (incl. hard pause)
	GatePausePlannedStop = "planned_stop" // drain mark with --pause-sessions
)

Gate-pause sources for GatePause (mast_gate_pauses_total{source}).

View Source
const (
	// TimedPauseResumed: the timer drove a resume/consume to completion.
	TimedPauseResumed = "resumed"
	// TimedPauseSkipped: the fire was a benign no-op — an operator
	// resumed first, or the daemon was draining when the timer fired.
	TimedPauseSkipped = "skipped"
	// TimedPauseError: the resume/consume the timer attempted failed;
	// the timer is rescheduled and will retry.
	TimedPauseError = "error"
)

Timed-pause fire outcomes (mast_timed_pause_fires_total{outcome}). A fixed vocabulary mirroring the scheduler fire callback's branches.

View Source
const (
	// AutoResumeResumed: a continuation turn ran to completion and the
	// interruption marker was cleared.
	AutoResumeResumed = "resumed"
	// AutoResumeCleared: the trailing event was already a clean model
	// turn (stale marker / clear race); the marker was cleared without
	// running a turn.
	AutoResumeCleared = "cleared"
	// AutoResumeSkippedStale: the interruption is older than the
	// freshness window; the marker was left for an operator.
	AutoResumeSkippedStale = "skipped_stale"
	// AutoResumeSkippedAmbiguous: the session carries a dangling mutating
	// intent (ambiguous effect); left for an operator ack.
	AutoResumeSkippedAmbiguous = "skipped_ambiguous"
	// AutoResumeSkippedLoopbreak: the per-session attempt breaker or the
	// per-boot cap tripped.
	AutoResumeSkippedLoopbreak = "skipped_loopbreak"
	// AutoResumeSkippedSuperseded: a concurrent turn advanced the session
	// after the scan (M1 TOCTOU recheck).
	AutoResumeSkippedSuperseded = "skipped_superseded"
	// AutoResumeSkippedUnsupported: a shape slice-1 does not drive
	// (non-coordinator dispatch, dangling sub-agent delegation, or a
	// multi-event repair).
	AutoResumeSkippedUnsupported = "skipped_unsupported"
	// AutoResumeError: the continuation turn was attempted and failed;
	// the marker was left in place.
	AutoResumeError = "error"
)

Auto-resume outcomes for AutoResume (mast_autoresume_total{outcome}). A fixed vocabulary, mirroring cmd/mast's boot-time auto-resume decision tree (#41): every interrupted candidate the boot pass inspects lands in exactly one of these.

View Source
const (
	A2ATaskSubmitted     = "submitted"
	A2ATaskWorking       = "working"
	A2ATaskInputRequired = "input-required"
	A2ATaskCompleted     = "completed"
	A2ATaskFailed        = "failed"
	A2ATaskCanceled      = "canceled"
	A2ATaskRejected      = "rejected"
)

A2A server task outcomes for A2ATask (mast_a2a_server_tasks_total {workload,outcome}). The vocabulary mirrors the A2A task-lifecycle states mast reports (docs/a2a-design.md "Task lifecycle mapping"); the string VALUES match pkg/a2a's TaskState constants — the server passes string(state), so these two lists must not drift.

View Source
const (
	AGUIRunSuccess     = "success"
	AGUIRunInterrupted = "interrupted"
	AGUIRunError       = "error"
	AGUIRunAborted     = "aborted"
	AGUIRunRejected    = "rejected"
)

AG-UI server run outcomes for AGUIRun (mast_agui_runs_total {workload,outcome}). A fixed vocabulary mirroring the AG-UI run dispositions the server reports (docs/ag-ui-design.md): a completed run, a run that paused for human input (a clean interrupt, resumable via RunAgentInput.Resume), an errored run, an operator/client abort, and a pre-stream refusal (auth/scope/rate-limit/drain/not-resumable). The string VALUES match pkg/agui's internal outcome constants — the server passes them through, so these two lists must not drift. Both sides are pinned to the same literals: pkg/agui's own test locks its unexported constants, and a cmd/mast test locks these exported ones, so a move on either side fails a build.

Variables

This section is empty.

Functions

func SetupOTel

func SetupOTel(ctx context.Context) (shutdown func(context.Context) error, enabled bool, err error)

SetupOTel installs the global OTLP trace exporter + W3C propagator when standard OTel env config asks for it (OTEL_EXPORTER_OTLP_ENDPOINT or OTEL_EXPORTER_OTLP_TRACES_ENDPOINT). Endpoint, headers, protocol details, etc. are all read from the environment by the exporter — mast adds nothing beyond a service.name resource default.

mast does not open custom spans in v0.1: ADK v2's runner emits the unified span tree (session/turn/node/tool), and mast only decorates. This function only makes that tree leave the process.

Returns a shutdown func (flushes the batch exporter) and whether export was enabled. When the env vars are absent it is a no-op: shutdown is non-nil and trivially succeeds.

Types

type Registry

type Registry struct {
	// contains filtered or unexported fields
}

Registry is the fixed set of mast metric families. Construct one per process with New and expose it via Handler on the inject listener.

func New

func New() *Registry

New constructs the registry with every base family pre-registered. The underlying prometheus.Registry is private: nothing outside this package can register additional collectors through it.

func (*Registry) A2ATask added in v0.2.0

func (r *Registry) A2ATask(workload, outcome string)

A2ATask records one A2A server task lifecycle transition with the given outcome (one of the A2ATask* constants; a pkg/a2a TaskState value). Safe on a nil *Registry.

func (*Registry) AGUIRun added in v0.2.0

func (r *Registry) AGUIRun(workload, outcome string)

AGUIRun records one AG-UI server run outcome (one of the AGUIRun* constants; a pkg/agui outcome value). Safe on a nil *Registry.

func (*Registry) AGUIRunDuration added in v0.2.0

func (r *Registry) AGUIRunDuration(workload string, seconds float64)

AGUIRunDuration records one AG-UI run's wallclock duration in seconds. Safe on a nil *Registry; negative durations are ignored.

func (*Registry) Abort added in v0.2.0

func (r *Registry) Abort(workload string)

Abort records a terminal abort durably recorded through the operator surface (the marker write succeeded; the in-flight turn is swept separately).

func (*Registry) AddCost

func (r *Registry) AddCost(workload string, usd float64)

AddCost accumulates spend (in USD) attributed to a workload. The amount comes from the budget meter — pricing stays in one place; this is only the export surface. Non-positive deltas are ignored.

func (*Registry) AutoResume added in v0.2.0

func (r *Registry) AutoResume(workload, outcome string)

AutoResume records one boot-time auto-resume decision with the given outcome (one of the AutoResume* constants).

func (*Registry) BudgetTrip

func (r *Registry) BudgetTrip(workload string)

BudgetTrip records a turn aborted on a budget ceiling.

func (*Registry) GatePause added in v0.2.0

func (r *Registry) GatePause(workload, source string)

GatePause records a gate pause durably recorded, by source (one of the GatePause* constants: an operator request or a planned stop).

func (*Registry) HITLPause

func (r *Registry) HITLPause(workload string)

HITLPause records a HITL interrupt explicitly, for callers that detect the pause outside the event stream. Callers already feeding events through Observe must not also call this for the same interrupt (Observe counts RequestedInput events itself).

func (*Registry) HITLResume

func (r *Registry) HITLResume(workload string)

HITLResume records an operator resume being fed into a session.

func (*Registry) Handler

func (r *Registry) Handler() http.Handler

Handler returns the Prometheus scrape handler for this registry, suitable for mounting at /metrics on an existing mux.

func (*Registry) MarkerWriteFailure added in v0.2.0

func (r *Registry) MarkerWriteFailure(workload, operation string)

MarkerWriteFailure records a failed interruption-marker store write during the shutdown drain, by operation (one of the MarkerOp* constants). A non-zero count is an alert condition: a marker that did not land means a restart may never surface the interrupted turn.

func (*Registry) Observe

func (r *Registry) Observe(ev *session.Event, workload string)

Observe folds one runner event into the counters. Shaped like pkg/budget's Meter.Observe so both hooks sit side by side on the event-stream loop. Events without UsageMetadata contribute no model-call or token counts; nil events are ignored. Safe on a nil *Registry so callers can leave telemetry unwired.

func (*Registry) Prime

func (r *Registry) Prime(workload string)

Prime materializes every family's time series for the given workload at zero, so a scrape sees all base families from process start (before the first turn) and PromQL rate()/increase() have a defined origin. Call once at startup per served workload.

func (*Registry) TimedPauseFire added in v0.2.0

func (r *Registry) TimedPauseFire(workload, outcome string)

TimedPauseFire records one timed-pause scheduler fire with the given outcome (one of the TimedPause* constants).

func (*Registry) TurnComplete

func (r *Registry) TurnComplete(workload, outcome string)

TurnComplete records one finished turn with the given outcome (one of the Outcome* constants).

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL