Documentation
¶
Overview ¶
Package agent computes agent-observability reports over the same generic event log everything else in the engine runs on — no new storage, no schema change. Two reserved event names carry the signal: agent_tool_call (one tool/MCP dispatch, with latency + error) and agent_turn (one conversation turn, with role + time-to-first-token). Every number here is COMPUTED and deterministic — latency percentiles reuse the exact nearest-rank code the p90/p95/p99 measures use (trends.Percentile), so the provably-correct brand holds. The one number we refuse to compute unless the app sends it is resolution rate (the honesty covenant): ResolutionRate stays nil unless a turn actually carries a `resolved` bool — never fabricated.
Index ¶
Constants ¶
const ( EventToolCall = "agent_tool_call" EventTurn = "agent_turn" )
Reserved event names, deliberately NOT $-prefixed so they sidestep the autocapture bot filter at ingest (that filter only drops $-prefixed events from crawler UAs). Documented for the paste-to-install skill so an agent instruments its own observability with these exact names.
const ( PropConversationID = "conversation_id" PropLabeledBy = "labeled_by" // the model that inferred the labels — required, never blank PropLabeledAt = "labeled_at" // RFC3339 stamp of when the label was written )
Reserved properties on an agent_label event. Everything else on it is a label.
const ( DefaultSampleLimit = 20 MaxSampleLimit = 100 MaxSampleTurns = 40 MaxTextRunes = 500 )
Sampling caps: a model has to be able to read the result in one go, so the sample is bounded on every axis (conversations, turns per conversation, characters per turn) and the caller is told exactly what was left out.
const EventLabel = "agent_label"
EventLabel is the third reserved agent event name: one model-written label set for one conversation. Like the other two it is deliberately NOT $-prefixed.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type ClientSplit ¶
ClientSplit is one client's share of a tool's calls (cursor / claude-code / copilot / …).
type ConversationSample ¶ added in v0.11.0
type ConversationSample struct {
Event string `json:"event"`
Conversations []SampledConversation `json:"conversations"`
Returned int `json:"returned"`
Total int `json:"total"` // conversations in the window, before the cap
Limit int `json:"limit"`
TurnsWithText int `json:"turns_with_text"`
TurnsWithoutText int `json:"turns_without_text"`
Note string `json:"note"`
}
ConversationSample is what sample_conversations returns: whole conversations plus the counts that keep it honest (how many exist, how many came back, how many turns actually had text).
func SampleConversations ¶ added in v0.11.0
SampleConversations returns whole conversations for the user's own model to read: agent_turn events in [from, to) grouped by conversation_id, turns ordered oldest-first inside each conversation, conversations ordered most-recent-first (last turn descending, conversation_id ascending as the tie-break) so the same input always produces the same sample — no randomness anywhere. limit defaults to 20 and is capped at 100; turns per conversation are capped and long text is truncated so a model can actually consume the result. A turn without a `text` property comes back without text; we never fabricate what was said.
type Conversations ¶
type Conversations struct {
Event string `json:"event"`
Conversations int `json:"conversations"`
Turns int `json:"turns"`
MedianTurns float64 `json:"median_turns"`
P90Turns float64 `json:"p90_turns"`
ReAskRate float64 `json:"reask_rate"` // fraction with >=2 consecutive user turns (no assistant between)
AbandonRate float64 `json:"abandon_rate"` // fraction whose last turn is a user turn
TTFTP50 float64 `json:"ttft_p50"` // time-to-first-token percentiles over ttft_ms
TTFTP90 float64 `json:"ttft_p90"`
TTFTP99 float64 `json:"ttft_p99"`
ResolutionRate *float64 `json:"resolution_rate"` // nil unless a turn carries a `resolved` bool
}
Conversations is the conversation-health report over agent_turn events. Every field is computed except ResolutionRate, which stays nil unless the app actually sends a `resolved` bool on a turn — we never invent whether a conversation was resolved.
func ComputeConversations ¶
func ComputeConversations(events []event.Event, from, to time.Time) Conversations
ComputeConversations groups agent_turn events (already filtered + dev-env-scoped) by conversation_id within [from, to). Turns per conversation drive the median/p90 (via trends.Percentile). Ordering each conversation by timestamp yields re-ask (two consecutive user turns with no assistant between) and abandon (ends on a user turn). TTFT percentiles run over every turn's ttft_ms. ResolutionRate is computed ONLY when at least one turn carried a `resolved` bool anywhere in the set — otherwise it is nil (the honesty covenant).
type ErrorTaxonomy ¶
type ErrorTaxonomy struct {
Event string `json:"event"`
Tool string `json:"tool,omitempty"`
Total int `json:"total"`
Types []ErrorType `json:"types"`
}
ErrorTaxonomy is the breakdown of failing tool calls by their `error_type`, optionally scoped to a single tool. Total is the number of error calls considered (the taxonomy sums to it).
func ComputeErrorTaxonomy ¶
ComputeErrorTaxonomy runs query.Breakdown over `error_type` for the failing agent_tool_call events (error truthy) in [from, to), optionally restricted to one tool. A typo'd or unseen tool matches nothing and returns an empty taxonomy — honest empty, never an error.
type LabelBreakdown ¶ added in v0.11.0
type LabelBreakdown struct {
Event string `json:"event"`
Label string `json:"label"`
Values []LabelValue `json:"values"`
Conversations int `json:"conversations"` // conversations in the window
Labeled int `json:"labeled"` // ...carrying this label
Unlabeled int `json:"unlabeled"`
LabeledBy []string `json:"labeled_by"` // distinct models behind the counted labels
Inferred bool `json:"inferred"`
Note string `json:"note"`
}
LabelBreakdown is the conversation count per label value, wrapped in the honesty context that makes it safe to read: which model(s) wrote the labels, and how many conversations in the window are still unlabeled. Inferred is always true — the VALUES are a model's judgement; the counts are computed from the event log.
func ComputeLabelBreakdown ¶ added in v0.11.0
ComputeLabelBreakdown counts conversations per value of one label. Conversations come from agent_turn events in [from, to); labels come from agent_label events joined on conversation_id, regardless of when the label was written (a label is a statement about a conversation, not something that happened at that moment — windowing it would silently drop last week's conversations the moment you label them today). When a conversation was labeled more than once for the same label, the most recent write wins (event id breaks a timestamp tie, so the result is deterministic). Buckets and ordering come from query.Breakdown, the same grouping every other report uses.
Nothing here is invented: an unlabeled set returns an honest empty result that says how to label, and every result carries labeled/unlabeled counts plus the labelling model(s).
type LabelValue ¶ added in v0.11.0
LabelValue is one value of a label and how many conversations carry it.
type SampledConversation ¶ added in v0.11.0
type SampledConversation struct {
ConversationID string `json:"conversation_id"`
TurnCount int `json:"turn_count"` // real number of turns, even if some were dropped
Start time.Time `json:"start"`
End time.Time `json:"end"`
Labeled bool `json:"labeled"` // already carries at least one agent_label event
Turns []SampledTurn `json:"turns"`
TurnsOmitted int `json:"turns_omitted,omitempty"`
}
SampledConversation is one whole conversation, turns oldest-first, ready to be read and labeled.
type SampledTurn ¶ added in v0.11.0
type SampledTurn struct {
Role string `json:"role"`
Timestamp time.Time `json:"timestamp"`
HasText bool `json:"has_text"`
Text string `json:"text,omitempty"`
TextTruncated bool `json:"text_truncated,omitempty"`
}
SampledTurn is one turn as the user's model sees it. Text is only ever what the app actually sent on the turn: a turn with no `text` property comes back with role and timing only, and HasText false. We never invent conversation content.
type ToolHealth ¶
type ToolHealth struct {
Event string `json:"event"`
Tools []ToolStat `json:"tools"`
Calls int `json:"calls"`
Errors int `json:"errors"`
}
ToolHealth is the tool-call report: one row per tool, plus the overall totals.
func ComputeToolHealth ¶
func ComputeToolHealth(events []event.Event, from, to time.Time) ToolHealth
ComputeToolHealth groups agent_tool_call events (already filtered + dev-env-scoped by the caller) by their `tool` property within [from, to), and for each tool computes call/error counts, error rate, latency p50/p90/p99 over `latency_ms` (via trends.Percentile — the same nearest-rank the measures use), and the client split via query.Breakdown over `client`. A tool that doesn't exist simply doesn't appear — an honest empty report, never an error.
type ToolStat ¶
type ToolStat struct {
Tool string `json:"tool"`
Calls int `json:"calls"`
Errors int `json:"errors"`
ErrorRate float64 `json:"error_rate"` // Errors/Calls, 0 when no calls
LatencyP50 float64 `json:"latency_p50"` // over latency_ms, nearest-rank
LatencyP90 float64 `json:"latency_p90"`
LatencyP99 float64 `json:"latency_p99"`
Clients []ClientSplit `json:"clients"`
}
ToolStat is one tool's health: volume, error rate, latency tail, and who's calling it.
Source Files