Documentation
¶
Overview ¶
Package telemetry is an outbound adapter that derives OpenTelemetry traces and Prometheus metrics from the harness's domain event stream.
It implements both port.EventSink (so it observes every session.Event the loop emits) and port.ToolCallRecorder (so it observes per-tool execution timing). The adapter is self-contained: the leader wires it by teeing the telemetry sink into the Engine's EventSink and ToolCallRecorder, mounting MetricsHandler at /metrics, and passing a TracerProvider to NewTracing.
Context ¶
port.EventSink.Emit(ctx, ev) carries the run's context.Context, but session.Event has no session-id field. When the ctx carries a trace span (e.g. the run goroutine was started under an inbound request span), Tracing parents its run span to it, so concurrent runs correlate to their originating request. When the ctx carries no span, Tracing falls back to a single provider-level root per sink instance (see Tracing). Tool child spans are produced with accurate latency, and the run lifecycle span captures stop reason and token usage.
Index ¶
- Constants
- Variables
- func ExpvarHandler() http.Handler
- func LatencyViews() []sdkmetric.View
- func MarshalSnapshotJSON(s RuntimeSnapshot) ([]byte, error)
- func MetricsHandler(reg *prometheus.Registry) http.Handler
- func NewAdminMux(reg *prometheus.Registry, recorder *FlightRecorder) *http.ServeMux
- func NewSink(sinks ...port.EventSink) port.EventSink
- func RegisterPprof(mux *http.ServeMux)
- func RegisterProcessGauges(mp metric.MeterProvider, diag port.Diagnostics) error
- func StartGoroutineWatchdog(ctx context.Context, threshold int, interval time.Duration, count func() int, ...)
- type FlightRecorder
- type Metrics
- func (m *Metrics) Emit(ctx context.Context, ev session.Event)
- func (m *Metrics) EmitLearning(activity learning.Activity)
- func (m *Metrics) EmitSchedule(payload session.SchedulePayload, duration time.Duration)
- func (m *Metrics) EmitSessionLoadFailure(class port.SessionLoadFailureClass)
- func (m *Metrics) ToolCall(id session.SessionID, call session.ToolCall, result session.ToolResult, ...)
- func (m *Metrics) WithRole(role string) *RoleMetrics
- type OTLPConfig
- type Providers
- type RoleMetrics
- type RuntimeSnapshot
- type SlowTurn
- type SlowTurnBuffer
- type Tracing
Constants ¶
const ( DefaultFlightRecorderMaxBytes uint64 = 8 << 20 // 8 MiB ring buffer DefaultFlightRecorderMinAge = 5 * time.Second )
Bounded defaults for the flight-recorder window. The recorder keeps a ring buffer of recent execution-trace events in memory; these bounds cap that overhead. MaxBytes takes precedence over MinAge (stdlib semantics): the window holds at most ~DefaultFlightRecorderMaxBytes of trace data, covering at least DefaultFlightRecorderMinAge of wall time when activity is light.
Sized for "the last few seconds of activity, a few MB resident": low enough to be safe to arm by default (decision: --flight-recorder defaults ON), large enough that a snapshot taken right after a slow turn still contains it.
const ( // RoleMain is the main (operator-facing) engine. RoleMain = "main" // RoleSubagent is the Subagent delegation family (explorer, per-def, model-override). RoleSubagent = "subagent" // RoleMember is the agent-team member/lead family. RoleMember = "member" // RoleParallel is the Parallel fan-out family (branches and the judge). RoleParallel = "parallel" // RoleUserModel is the user-model review child. RoleUserModel = "usermodel" // RoleChild is the safe fallback for an unrecognised child role. RoleChild = "child" )
Role family values for the attrRole label. This is a CLOSED, bounded set — the cardinality contract of the role dimension. The composition layer (internal/app's roleFamily) maps every engine role onto exactly one of these six values before it ever reaches a metric; def names, member names, model ids, and session ids must NEVER appear as a role value. RoleChild is the fail-safe bucket for any role the mapping does not recognise. Every member must correspond to a role an engine actually carries — an unmatchable value in the model-facing filter enums is a trap (the reason there is no "fork" family: fork branches/judges have session-id prefixes, never a Deps.Role).
const ( // ProtocolGRPC exports spans over OTLP/gRPC (the default). ProtocolGRPC = "grpc" // ProtocolHTTP exports spans over OTLP/HTTP (protobuf). ProtocolHTTP = "http" )
Protocol selects the OTLP transport used by the exporter.
const DefaultSlowTurnCapacity = 256
DefaultSlowTurnCapacity is the fixed size of the slow-turn ring buffer. The buffer is intrinsically bounded: it holds at most this many of the most-recent turns, evicting the oldest when full. 256 is enough recent history for an agent to reason over a slow patch without growing the model context (the MCP tool paginates) and keeps the per-process footprint trivially small (a few KiB of scalars).
Variables ¶
var ErrFlightRecorderAlreadyActive = errFlightRecorder("flight recorder already active process-wide")
ErrFlightRecorderAlreadyActive is returned by ProcessFlightRecorder when an arming attempt is COALESCED onto an already-started shared recorder, or REJECTED because a distinct recorder already owns the single process-wide slot. It is exported so a second consumer (e.g. the embedded server in cmd/mecatui, decision 7) can recognise the loss with errors.Is and fall back to the existing instance rather than treating it as a fatal start failure. The returned *FlightRecorder (when non-nil) is still the live shared instance, so the caller can use it; the error signals only that the caller did not arm it.
Functions ¶
func ExpvarHandler ¶
ExpvarHandler returns the expvar HTTP handler backing /debug/vars, after publishing a curated "mecatl_runtime" expvar.Func that returns the current Snapshot(). The curated var is preferred over relying on expvar's default "memstats" entry, which calls runtime.ReadMemStats and triggers a stop-the-world pause on every scrape; Snapshot() reads lock-free runtime/metrics instead.
expvar's default handler still also exposes "memstats" and "cmdline" (the package registers them in init). That is acceptable on the loopback admin surface; the curated mecatl_runtime var is the one Phase-2 reads.
func LatencyViews ¶
LatencyViews returns the explicit-bucket-histogram views for EVERY latency instrument (tool/turn duration, TTFT, inter-token, tool-queue; ADR 0045, superseding ADR 0018 §5 decision 2). It is the single source of truth for the latency aggregation: any MeterProvider feeding NewMetrics MUST install these (sdkmetric.WithView(LatencyViews()...)), or the latency series fall back to the SDK default explicit buckets (a coarser ladder that misses the ms low end and the minute-scale high end the latencyBucketBoundaries cover).
The explicit ladder renders as classic Prometheus le= buckets, so the text /metrics exposition + promtool + the perf-MCP reducer's classicLadder path all yield p50/p90/p99 with zero scrape config — the issue #158 fix. (OTLP push, were it ever wired, would carry the same explicit buckets; the exponential-tail precision OTLP could have aggregated is unconsumed today since no OTLP metrics reader exists.)
Aggregation choice is a view-on-the-provider/reader concern, NOT a per-instrument hint, which is why it belongs to whoever assembles the MeterProvider.
func MarshalSnapshotJSON ¶
func MarshalSnapshotJSON(s RuntimeSnapshot) ([]byte, error)
MarshalSnapshotJSON is a small convenience used by tests and any caller that wants the snapshot bytes directly (Phase-2 reduces server-side instead of shipping the raw blob, but the JSON form is the canonical interchange).
func MetricsHandler ¶
func MetricsHandler(reg *prometheus.Registry) http.Handler
MetricsHandler returns an http.Handler that serves the given registry in the Prometheus text exposition format, suitable for mounting at /metrics. It is a thin wrapper over promhttp that keeps the handler construction (and the promhttp.HandlerOpts choice) in one place; the composition root still names *prometheus.Registry to wire the handler, which architecture.md §2 permits.
func NewAdminMux ¶
func NewAdminMux(reg *prometheus.Registry, recorder *FlightRecorder) *http.ServeMux
NewAdminMux builds the loopback admin mux: /metrics (read-only, secret-free) plus the runtime-introspection surface — pprof, expvar (/debug/vars), and, when recorder is non-nil, the FlightRecorder snapshot (/debug/flightrecorder).
It lives in the telemetry adapter (not a cmd main) so BOTH composition roots — the standalone cmd/mecated daemon and the cmd/mecatui embedded server (decision 7 in docs/adr/0018-perf-observability.md) — serve the IDENTICAL admin surface from one helper, rather than each hand-rolling a mux that could drift.
recorder may be nil (FlightRecorder disabled), in which case /debug/flightrecorder is NOT mounted and a GET returns 404.
SECURITY: pprof/FlightRecorder/expvar output can embed prompt text, file paths, and goroutine stacks. The returned mux MUST be served only on a loopback-bound listener — never on the public gRPC/HTTP service surface (decision 6 in docs/adr/0018-perf-observability.md).
func NewSink ¶
NewSink returns a port.EventSink that fans out every Event to each of the given sinks, in the order provided. It lets a single Engine EventSink drive both the Metrics and Tracing adapters.
func RegisterPprof ¶
RegisterPprof registers the standard net/http/pprof handlers EXPLICITLY on the given mux. It deliberately does NOT rely on the package's init-time blank import (which mutates http.DefaultServeMux); registering on a caller-supplied mux keeps the profiling surface confined to the loopback admin listener and off the public service mux.
It mounts the four dynamic endpoints (/debug/pprof/ Index, cmdline, profile, symbol, trace) plus one /debug/pprof/<name> handler per predefined profile (heap, goroutine, allocs, mutex, block, threadcreate) via pprof.Handler.
SECURITY: pprof output (goroutine dumps, heap, the CPU/trace profiles) can embed prompt text, file paths, and other request data. Mount this ONLY on a loopback-bound listener — never on the public gRPC/HTTP service surface.
func RegisterProcessGauges ¶
func RegisterProcessGauges(mp metric.MeterProvider, diag port.Diagnostics) error
RegisterProcessGauges registers the process-level observable gauges on the given MeterProvider's meter. Currently it registers mecatl.process.rss (the resident set size in bytes), read lock-free via readRSS on each collection.
The gauge is registered ONLY where RSS is actually readable (Linux): off Linux rssSupported() is false and the series is simply absent (decision 9 — the RSS gauge ships for general long-session memory visibility, no leak-specific alarm). It is a separate registration from NewMetrics because the domain instruments derive from the event/log stream, whereas this is an async observation of the OS process; keeping it apart lets a caller opt out.
It returns an error if the instrument fails to construct.
func StartGoroutineWatchdog ¶
func StartGoroutineWatchdog(ctx context.Context, threshold int, interval time.Duration, count func() int, logger *slog.Logger)
StartGoroutineWatchdog launches a background ticker that samples the live goroutine count (via the injected count func, normally runtime.NumGoroutine) every interval and logs a slog.Warn when it exceeds threshold — the live leak ALARM of decision 10 in docs/adr/0018-perf-observability.md, complementing the test-time goleak gate and the runtime collector's goroutine-count /metrics series.
The goroutine exits when ctx is cancelled (shutdown), so the watchdog itself never leaks — verified by the package's goleak-free shutdown and by TestStartGoroutineWatchdogStopsOnCancel. count and logger are injected so the behaviour is unit-testable without spawning real goroutines or racing the global logger.
It lives in the telemetry adapter (not a cmd main) so BOTH composition roots — the standalone cmd/mecated daemon and the cmd/mecatui embedded server — arm the SAME helper. The injected count func keeps it import-clean (no runtime import here; the caller passes runtime.NumGoroutine).
A threshold <= 0 is a no-op (the alarm is disabled and no goroutine is spawned). A non-positive interval falls back to 30s.
Types ¶
type FlightRecorder ¶
type FlightRecorder struct {
// contains filtered or unexported fields
}
FlightRecorder wraps the stdlib runtime/trace.FlightRecorder (Go 1.26) with a bounded in-memory window. It continuously records execution-trace events into a ring buffer; Snapshot writes the current window out as a parseable trace (the trigger Phase-2's perf-over-MCP server calls on a tail-latency turn).
SECURITY: an execution trace can embed goroutine stacks and timing that correlate to request data. Snapshots must only be served on the loopback admin surface — never the public service mux.
The wrapper is concurrency-safe: Start/Stop are guarded so a double Start or a Stop-before-Start is a no-op rather than an error, which keeps the daemon startup/shutdown wiring simple and idempotent.
func NewFlightRecorder ¶
func NewFlightRecorder(cfg trace.FlightRecorderConfig) *FlightRecorder
NewFlightRecorder constructs a recorder with the given config. A zero-value config field falls back to the bounded defaults above, so NewFlightRecorder (FlightRecorderConfig{}) is the recommended "sane bounded window" call.
func ProcessFlightRecorder ¶
func ProcessFlightRecorder(cfg trace.FlightRecorderConfig) (*FlightRecorder, error)
ProcessFlightRecorder returns the single shared, started FlightRecorder for this process, constructing and arming it on the first call with the given config and returning the same instance on every later call. cfg is honoured ONLY on the first call (it constructs the singleton); a later call's cfg is ignored and the call is COALESCED — it returns the existing instance wrapped with ErrFlightRecorderAlreadyActive so the caller knows it did not own the arming. If the first arming itself failed (e.g. the slot was taken by a recorder created outside this accessor), every call returns that error.
This is the wiring every composition root should use (cmd/mecated today, the cmd/mecatui embedded server next), so a second consumer cannot silently lose a Start against the one-recorder-per-process stdlib constraint.
LIMITATION — the slot is process-lifetime, not re-armable. The singleton is constructed and started exactly once; once its OWNER Stops it (the caller that received a nil error armed it and is the only one that should Stop it), it CANNOT be re-armed in the same process — a subsequent ProcessFlightRecorder returns the now-stopped instance, and Start() on it will fail because the stdlib slot has already been used and torn down. This is fine for the production wiring (one mecated, or one embedded server, per process, armed at startup and stopped at shutdown), but it is a hard constraint for a test binary or any future multi-embed: only ONE owner per process may arm-then-stop the recorder; later owners must coalesce (and must NOT Stop what they did not arm — see cmd/mecatui/embed's ownsRecorder gate).
func (*FlightRecorder) Enabled ¶
func (f *FlightRecorder) Enabled() bool
Enabled reports whether the recorder is currently running.
func (*FlightRecorder) Snapshot ¶
func (f *FlightRecorder) Snapshot(w io.Writer) (int64, error)
Snapshot writes the current trace window to w and returns the byte count. It errors if the recorder is not running (nothing to snapshot) or if a concurrent Snapshot is already in progress (a stdlib WriteTo constraint).
func (*FlightRecorder) SnapshotBytes ¶
func (f *FlightRecorder) SnapshotBytes() ([]byte, error)
SnapshotBytes returns the current trace window as a byte slice. It is the convenience the Phase-2 MCP tool and the Phase-1 /debug/flightrecorder endpoint use; the returned buffer begins with the Go execution-trace magic header and is parseable by golang.org/x/exp/trace.
func (*FlightRecorder) Start ¶
func (f *FlightRecorder) Start() error
Start begins recording. It is idempotent: a second Start on THIS instance while already running is a no-op returning nil. Only one flight recorder may be active process-wide (a stdlib constraint) — prefer ProcessFlightRecorder, which hands every caller the single shared instance and enforces that invariant, over arming a second distinct recorder directly.
func (*FlightRecorder) Stop ¶
func (f *FlightRecorder) Stop()
Stop ends recording and releases the ring buffer. It is idempotent: a Stop when not running is a no-op. Call it from the daemon's shutdown path so the recorder's background subscription is torn down (important for the upcoming goleak gate — a leaked recorder would otherwise hold runtime trace state).
type Metrics ¶
type Metrics struct {
// contains filtered or unexported fields
}
Metrics is an OpenTelemetry-backed telemetry adapter. It implements both port.EventSink (deriving counters/gauges from the event stream) and port.ToolCallRecorder (deriving tool-call counters and a latency histogram).
All series use bounded attribute sets: tool names and stop reasons are bounded domain values, and no series is ever labelled by session id or free text. The instruments are created from a metric.Meter obtained from the injected MeterProvider; the provider's prometheus exporter (wired in Setup) renders them on the /metrics registry.
func NewMetrics ¶
func NewMetrics(mp metric.MeterProvider) (*Metrics, error)
NewMetrics constructs a Metrics adapter from an OTel MeterProvider. It implements both port.EventSink and port.ToolCallRecorder. The provider is expected to have a prometheus exporter reader and the latency explicit-bucket-histogram views installed (see Setup); NewMetrics itself only creates the instruments.
It returns an error if any instrument fails to construct — the OTel meter API is fallible, unlike client_golang's panic-on-misuse registration.
func (*Metrics) Emit ¶
Emit records OTel metrics derived from a single domain Event, attributing every series to the MAIN engine (role="main"). A child engine's events must flow through WithRole instead, so each series carries the role label uniformly. The ctx is the run's context (threaded from port.EventSink.Emit) and is passed to every instrument operation so the SDK can attach exemplars from an active span.
func (*Metrics) EmitLearning ¶
EmitLearning records only closed activity/reason/sensitivity labels.
func (*Metrics) EmitSchedule ¶
func (m *Metrics) EmitSchedule(payload session.SchedulePayload, duration time.Duration)
EmitSchedule records scheduled-task fire metrics. It is the composition-injected callback target the scheduler invokes (via Config.ScheduleMetrics) for every fired/skipped/failed fire. Schedule metrics are NOT a role-family (issue #233): a fire mints a fresh session whose OWN run already carries role="main" via its EventSink, so these instruments carry NO role label — they are a separate schedule-lifecycle dimension.
duration is the fire's wall-clock cost (due→terminal — measured from the tick/due time captured before Store.Claim, not from the Claim op itself). The scheduler passes it only for a fired/failed fire (the value of time.Since(now) captured in fireClaimed); a SKIPPED fire (no run) passes duration 0 and the fire-duration histogram is skipped. The fires counter is ALWAYS bumped (labelled by outcome). Nil-safe: a nil Metrics is a no-op (the byte-identical no-metrics path).
func (*Metrics) EmitSessionLoadFailure ¶ added in v0.0.26
func (m *Metrics) EmitSessionLoadFailure(class port.SessionLoadFailureClass)
EmitSessionLoadFailure increments the ownership-safe load-failure counter. Invalid values fail closed to the unknown label.
func (*Metrics) ToolCall ¶
func (m *Metrics) ToolCall(id session.SessionID, call session.ToolCall, result session.ToolResult, queued, took time.Duration)
ToolCall records the per-tool call counter and the duration/queue-time latency histograms, attributed to the MAIN engine (role="main"); a child engine's calls flow through WithRole instead. It satisfies port.ToolCallRecorder. ToolCallRecorder carries no ctx, so the recordings use a background context — exemplar correlation is best-effort here. queued is the dispatch wait (enqueue→execution start); took is the execution wall time. Both are recorded with the same tool attribute.
func (*Metrics) WithRole ¶
func (m *Metrics) WithRole(role string) *RoleMetrics
WithRole returns a role-scoped dual-interface view (port.EventSink + port.ToolCallRecorder) over the SAME underlying instruments, tagging every series with role. Pass one of the bounded Role* constants (the composition layer's roleFamily mapping guarantees this for child engines).
type OTLPConfig ¶
type OTLPConfig struct {
// Endpoint is the collector address, e.g. "localhost:4317" for gRPC or a
// URL/host for HTTP. An empty Endpoint disables tracing: Setup installs
// nothing and returns a no-op shutdown.
Endpoint string
// Protocol selects the transport: "grpc" (default) or "http". Any other
// value is rejected by Setup.
Protocol string
// Insecure skips TLS when dialing the collector (development only).
Insecure bool
// ServiceName sets the resource service.name attribute. Defaults to
// "mecatl" when empty.
ServiceName string
// Headers are sent with every export request (e.g. auth headers).
Headers map[string]string
// Timeout bounds a single export request. Zero uses the exporter default.
Timeout time.Duration
// SampleRatio is the parent-based head-sampling ratio in [0,1]. Values <= 0
// select always-on sampling (the default); values >= 1 also sample every
// trace. Non-root spans follow their parent's sampling decision.
SampleRatio float64
// Version sets the resource service.version attribute when non-empty.
Version string
// InstallationID sets the optional mecatl.installation.id resource attribute.
InstallationID string
// --- OTLP metrics push (optional; the prometheus reader stays always on) ---
// MetricsEndpoint is the collector address for an OTLP METRICS push reader
// (a PeriodicReader over an otlpmetricgrpc/otlpmetrichttp exporter). An empty
// MetricsEndpoint installs NO periodic reader — the scrape-only path is
// byte-identical. When set, the periodic reader joins the prometheus reader
// on the SAME MeterProvider so /metrics AND the push both export the domain
// instruments.
MetricsEndpoint string
// MetricsProtocol selects the metrics transport: "grpc" (default) or "http".
// Any other value is rejected by Setup.
MetricsProtocol string
// MetricsInsecure skips TLS when dialing the metrics collector (dev only).
MetricsInsecure bool
// MetricsHeaders are sent with every metrics export request.
MetricsHeaders map[string]string
// MetricsTimeout bounds a single metrics export request. Zero uses the
// exporter default.
MetricsTimeout time.Duration
// MetricsPushInterval is the PeriodicReader export cadence. Zero uses the
// SDK default (60s). A short-lived caller (mecatequi) should set a small
// interval AND call Shutdown to force a final flush before exit.
MetricsPushInterval time.Duration
}
OTLPConfig configures the OTLP trace exporter and the SDK TracerProvider that Setup installs. A zero Endpoint disables tracing entirely.
type Providers ¶
type Providers struct {
// Tracer is the TracerProvider (a no-op provider when tracing is disabled).
// It is also installed globally via otel.SetTracerProvider, so
// NewTracing(otel.GetTracerProvider()) picks it up.
Tracer trace.TracerProvider
// Meter is the MeterProvider feeding the domain instruments. Pass it to
// NewMetrics. It is always a real SDK provider (metrics are always on, even
// when OTLP tracing is disabled) so /metrics has data to serve.
Meter metric.MeterProvider
// Registry is the prometheus registry the metrics exporter registers on.
// Serve it via MetricsHandler at /metrics.
Registry *prometheus.Registry
// Shutdown flushes and stops BOTH providers. Always non-nil.
Shutdown func(context.Context) error
}
Providers bundles the OTel providers Setup installs, so callers wire metrics and traces from one place instead of a positional return list.
func Setup ¶
func Setup(ctx context.Context, cfg OTLPConfig) (Providers, error)
Setup builds the OTel metrics and (optionally) tracing pipelines.
Metrics are ALWAYS installed: a prometheus-exporter reader registers the domain instruments on a fresh prometheus.Registry (returned for /metrics), the latency histograms are configured as explicit-bucket histograms via metric.Views (ADR 0045 — so the classic text exposition carries real le= buckets / quantiles), and the runtime collector (go.goroutine.count, GC, heap, …) is started against the MeterProvider. The MeterProvider is NOT installed globally — it is returned in Providers.Meter for explicit injection into NewMetrics.
Tracing is installed only when cfg.Endpoint is non-empty: Setup builds an OTLP span exporter and an SDK TracerProvider with a batch span processor, a parent-based sampler, and a resource carrying service.name and (optionally) service.version. It installs the provider via otel.SetTracerProvider and a W3C TraceContext propagator via otel.SetTextMapPropagator. When cfg.Endpoint is empty, tracing is disabled and Providers.Tracer is a no-op provider.
The gRPC trace exporter is constructed lazily and does not dial the collector until the first export, so Setup returns promptly even against an unreachable endpoint.
An OTLP metric exporter (push to a collector) is an optional seam: when cfg.MetricsEndpoint is set, Setup attaches a PeriodicReader over an otlpmetricgrpc/otlpmetrichttp exporter to the MeterProvider's reader slice alongside the always-on prometheus reader. The reader slice assembly (below in newMeterProvider) keeps both exporters exporting the SAME instruments.
type RoleMetrics ¶
type RoleMetrics struct {
// contains filtered or unexported fields
}
RoleMetrics is a role-scoped view over a *Metrics: it implements BOTH port.EventSink and port.ToolCallRecorder, delegating every Add/Record to the shared instruments with the role attribute appended. The role MUST be one of the bounded Role* family constants — the composition layer maps engine roles onto that closed set BEFORE constructing a RoleMetrics, so no def/member name or session id can ever become a label value. It holds no state of its own and spawns no goroutine.
type RuntimeSnapshot ¶
type RuntimeSnapshot struct {
// Goroutines is the live goroutine count from /sched/goroutines, falling back
// to runtime.NumGoroutine() when the sample is unavailable.
Goroutines int `json:"goroutines"`
// NumCPU is the number of logical CPUs usable by the process.
NumCPU int `json:"num_cpu"`
// GOMAXPROCS is the current GOMAXPROCS setting.
GOMAXPROCS int `json:"gomaxprocs"`
// HeapAllocsTotalBytes is the CUMULATIVE number of bytes ever allocated to
// the heap since process start (/gc/heap/allocs) — a monotonic counter that
// reads 100+ GB on a long-lived process. It is NOT the live heap; live
// heap-object memory is HeapObjectBytes (heap_object_bytes). Renamed from
// heap_alloc_bytes, which collided with the LIVE-gauge meaning of Go
// MemStats.HeapAlloc.
HeapAllocsTotalBytes uint64 `json:"heap_allocs_total_bytes"`
// HeapObjects is the count of live-or-unswept heap objects
// (/gc/heap/objects), not a cumulative allocation count.
HeapObjects uint64 `json:"heap_objects"`
// TotalMemoryBytes is all memory mapped by the runtime (/memory/classes/total).
TotalMemoryBytes uint64 `json:"total_memory_bytes"`
// HeapObjectBytes is live heap-object memory (/memory/classes/heap/objects).
HeapObjectBytes uint64 `json:"heap_object_bytes"`
// GCPauseCount is the number of GC pauses observed (sum of the pause
// histogram counts). Legitimately 0 before the first GC — see Available.
GCPauseCount uint64 `json:"gc_pause_count"`
// GCPauseP99UpperBoundNs is the histogram BUCKET UPPER BOUND containing the
// ~p99 rank of the GC pause distribution, in nanoseconds, derived from the
// /gc/pauses histogram. It is a representative bucket bound, NOT an exact
// quantile — the field name says so deliberately so a Phase-2 consumer does
// not over-trust the precision. Legitimately 0 before the first GC.
GCPauseP99UpperBoundNs uint64 `json:"gc_pause_p99_upper_bound_ns"`
// RSSBytes is the process resident set size in bytes, or 0 when it could not
// be read (non-Linux, or /proc unavailable). See process_rss_linux.go.
RSSBytes uint64 `json:"rss_bytes"`
// UptimeSeconds is the wall-clock age of the process in seconds.
UptimeSeconds float64 `json:"uptime_seconds"`
// Available is the set of curated runtime/metrics sample names that were
// actually present (and thus read) for this snapshot. It makes metric
// presence EXPLICIT: a name absent here was not published by this toolchain,
// whereas a name present here was read even if its value happens to be zero.
// Phase-2 consumers should test membership here rather than inferring absence
// from a zero field. The runtime counters (Goroutines/NumCPU/GOMAXPROCS/
// UptimeSeconds) and RSSBytes are always populated and are NOT listed here —
// Available tracks only the runtime/metrics-derived fields.
Available []string `json:"available"`
}
RuntimeSnapshot is a plain, JSON-serialisable view of the process runtime state at one instant. It carries NO OTel/SDK types deliberately: it is the reusable read contract that Phase-2's perf-over-MCP server projects directly into tool output (decision 4 in docs/adr/0018-perf-observability.md). Treat the field set + JSON tags as a stable wire shape — additive changes only. One deliberate exception on record: heap_alloc_bytes was RENAMED to heap_allocs_total_bytes — the old key read as a live-heap gauge (the Go MemStats.HeapAlloc meaning) when the value is actually the cumulative allocation counter, a misread worth a one-time break.
Byte counts are bytes; durations are nanoseconds (GCPauseP99UpperBoundNs) or seconds (UptimeSeconds) as named.
PRESENCE vs ZERO: a zero on a runtime/metrics-derived field carries NO semantic load — several fields are legitimately zero (GCPauseCount and GCPauseP99UpperBoundNs before the first GC; any counter with GOGC=off). Do NOT read "zero" as "absent". To learn which metrics were actually present this snapshot, consult Available: it lists the curated runtime/metrics names that were read (i.e. exist on this toolchain) this snapshot, so absence is explicit and zero is just a value.
func Snapshot ¶
func Snapshot() RuntimeSnapshot
Snapshot reads a curated set of runtime/metrics samples plus runtime counters into a plain RuntimeSnapshot DTO. It is allocation-light and does NOT trigger a stop-the-world ReadMemStats (runtime/metrics is sampled lock-free), so it is safe to call from a request handler or an MCP tool on a hot path.
Missing sample names are tolerated: the corresponding field is left zero. The goroutine count falls back to runtime.NumGoroutine() if the sample is absent.
type SlowTurn ¶
type SlowTurn struct {
// TurnIndex is the 0-based turn index within its run (Event.Turn).
TurnIndex int
// DurationMs is the turn's model-call wall-clock duration in milliseconds.
DurationMs int64
// TTFTMs is the time-to-first-token in milliseconds (0 if not measured).
TTFTMs int64
// InterTokenMaxMs is the worst inter-token gap in milliseconds (0 if not measured).
InterTokenMaxMs int64
// EndedAt is the wall-clock time the turn ended.
EndedAt time.Time
// Role is the BOUNDED engine role family that produced the turn (one of the
// Role* constants — "main", "subagent", "member", …). It is a closed enum,
// never a def/member name or session id, so redaction-by-shape still holds.
Role string
}
SlowTurn is one entry of the slow-turn ring buffer. It carries SCALARS ONLY — turn index, the per-turn latency numerics, and the end timestamp. There is deliberately NO prompt text, tool arguments, or session id field: redaction by storage shape (decision 6 / §2.4 of docs/adr/0018-perf-observability.md) means the buffer physically cannot leak conversation content, no matter how the data is later surfaced. The shape mirrors mcpperf.SlowTurn so the cmd/embed wiring can bridge the two with a trivial field copy (telemetry must NOT import mcpperf — wrong direction; see SlowTurnBuffer.Recent).
type SlowTurnBuffer ¶
type SlowTurnBuffer struct {
// contains filtered or unexported fields
}
SlowTurnBuffer is a fixed-size, in-memory ring buffer of recent turns, recording ONE scalar SlowTurn per EvTurnEnd it observes. It satisfies port.EventSink so telemetry.NewSink can fan EvTurnEnd into it alongside the metrics/tracing sinks — it sees the exact same TurnEndPayload the latency histograms do, with no second event path.
It is the concrete backing for mcpperf's SlowTurnSource read seam: the cmd / embed composition root bridges SlowTurnBuffer.Recent (returning the telemetry SlowTurn type) to the adapter interface (which wants mcpperf.SlowTurn), so the dependency points inward (cmd → telemetry, cmd → mcpperf) and telemetry never imports the adapter.
It spawns no goroutine — all work happens synchronously inside Emit under a mutex — so it is goleak-clean by construction. Reads (Recent) and writes (Emit) are concurrency-safe; the engine may emit from a run goroutine while an MCP tool reads.
func NewSlowTurnBuffer ¶
func NewSlowTurnBuffer(capacity int, clock func() time.Time) *SlowTurnBuffer
NewSlowTurnBuffer builds a slow-turn ring buffer of the given capacity (<= 0 falls back to DefaultSlowTurnCapacity). clock supplies the EndedAt timestamp for turns whose payload carries no end time of its own; nil falls back to time.Now.
func (*SlowTurnBuffer) Emit ¶
func (b *SlowTurnBuffer) Emit(_ context.Context, ev session.Event)
Emit records a SlowTurn for every EvTurnEnd, ignoring all other event types, attributing the turn to the MAIN engine (role="main"); a child engine's turns flow through WithRole instead (mirroring Metrics.WithRole). It stores ONLY scalars from the TurnEndPayload (and the Event.Turn index) plus the bounded role family — no text ever enters the buffer. The oldest entry is overwritten once the ring is full (bounded memory). ctx is unused: the buffer derives nothing from it.
func (*SlowTurnBuffer) Recent ¶
func (b *SlowTurnBuffer) Recent(thresholdMs int64) []SlowTurn
Recent returns the whole bounded set of recorded turns, NEWEST FIRST, filtered to turns whose DurationMs is at least thresholdMs (thresholdMs <= 0 returns every recorded turn). The order is stable across calls on an unchanged buffer, which is the contract mcpperf.SlowTurnSource relies on for stable in-memory cursor pagination and an accurate totalCount.
It returns telemetry.SlowTurn (its own type); the cmd/embed wiring maps each to mcpperf.SlowTurn so telemetry need not import the adapter.
func (*SlowTurnBuffer) WithRole ¶
func (b *SlowTurnBuffer) WithRole(role string) port.EventSink
WithRole returns a port.EventSink view over the SAME ring buffer whose recorded turns carry the given bounded role family value (mirroring Metrics.WithRole). It holds no state of its own and spawns no goroutine.
type Tracing ¶
type Tracing struct {
// contains filtered or unexported fields
}
Tracing is an OpenTelemetry-backed telemetry adapter. It implements port.EventSink and turns the event stream into spans:
- a run span opened on the first event of a run (session.init, or the first event seen if init is missing) and ended on the terminal result event, with the stop reason mapped to span status and token usage recorded as attributes;
- a child span per tool, opened on tool.call and ended on the matching tool.result, keyed by ToolCallID.
Context ¶
EventSink.Emit carries the run's context.Context. When that ctx carries a trace span (the run goroutine was started under an inbound request span), Tracing parents the run span to it, so concurrent runs each link to their originating request. When the ctx carries no span, Tracing falls back to a single "current run" span per Tracing instance: a result event closes the open run span, and the next session.init opens a fresh one. Tool spans are correlated by ToolCallID, which is unique within a run. The span map is mutex-guarded so concurrent Emit calls are safe.
Note: because session.Event has no session id, the single-instance fallback still cannot distinguish two concurrent runs whose ctx carries no span; that case relies on each run having its own ctx span (or its own Tracing/EventSink) for correct correlation.
func NewTracing ¶
func NewTracing(tp trace.TracerProvider) *Tracing
NewTracing constructs a Tracing adapter that creates spans from tp. It implements port.EventSink.