Documentation
¶
Overview ¶
Package quality turns a session's behavioral signals into an outcome classification and a 0-100 quality score with an A-F grade. It is pure: the store gathers the raw counts and facts from a session's projection (its messages and tool calls) and hands them here, so the scoring model can be reasoned about and unit tested without a database. The model is adapted from agentsview's penalty scheme: start at 100 and subtract for the things that make a session go badly (errors, retries, churn, an errored ending), then map the remainder to a letter.
Index ¶
- Constants
- func Classify(f Facts) (Outcome, Confidence)
- func ContextHealth(perTurnTokens []int64) (peak int64, resets int)
- func IsContextReset(prev, cur int64) bool
- func Score(s Signals) (score int, grade string, scored bool)
- type Archetype
- type ArchetypeFacts
- type Confidence
- type ContextHealthFolder
- type Facts
- type Outcome
- type PromptFacts
- type PromptHygiene
- type ScoreBreakdownItem
- type Signals
Constants ¶
const ( MarathonMinutes = 120 MarathonMessages = 200 DeepMinutes = 30 DeepMessages = 60 StandardMinutes = 5 StandardMessages = 15 )
Archetype thresholds. A session lands in the heaviest band whose duration OR message count it reaches, so a long-but-quiet session and a short-but-chatty one both read as substantial. The store builds the same banding in SQL from these constants (one numeric source), and ClassifyArchetype is the tested reference.
const PromptFactsVersion = 1
PromptFactsVersion stamps the per-message prompt-hygiene facts ClassifyPrompt materializes on each human message (see the messages.prompt_* columns and store.gatherPromptHygiene). It is deliberately separate from Version: those facts are cached ON the messages row and can only be re-derived by re-inserting the message, which happens on a reparse, whereas Version-stamped session_signals rows are re-derived by the settle pass from whatever facts the row currently holds. Reusing Version would break the settle pass's incremental re-stamp: a Version-only bump (a scoring tweak that leaves ClassifyPrompt untouched and ships no reparse) would mark every cached fact stale, and the settle pass, which cannot re-derive them, could never grade those sessions again.
Bump this ONLY when ClassifyPrompt's output changes (a new or changed prompt flag, a different normalization for the duplicate digest), and pair the bump with a parse.Epoch bump so the corpus reparses and re-derives every message's facts at the new version. Until that reparse reaches a session, gatherPromptHygiene treats its old-version facts as unmeasured and leaves the session ungraded, so a stored hygiene count is never computed from a superseded classifier.
const Version = 2
Version is the signals-and-scoring version. Bump it whenever the set of signals computed, the penalty weights, or the grade thresholds change, so a stored row records which model produced it. A reparse rebuilds every row to the running version (see store.RefreshSessionSignals), and the UI gates parsed views while a reparse runs, so a reader never mixes versions mid-rebuild.
Version 1 is the initial signal set: tool-health signals (failures, immediate retries, edit churn, the longest failure streak) and an outcome classification that together drive the score and grade, plus two informational signal sets that do not feed the score. Prompt hygiene (terse, duplicate, and no-code-context prompt counts, and an unstructured-start flag) describes the human's input, not the agent's work. Context health (peak context tokens and an inferred context-reset count; see ContextHealth) describes resource load, not whether the session went well; it reads a session's own usage turns, since a subagent is a separate session in its own transcript file and never mixes into another session's context.
Version 2 sharpens outcome classification so a settled session gets a verdict where v1 gave up. Two cases that v1 always left unknown now resolve: an automation run (no human turn) that reached a substantive assistant last word and has gone idle reads as completed, and a session that ends mid-tool once it has gone idle past the abandoned threshold reads as errored for automation (the run was killed or crashed) or abandoned for a human (someone interrupted or walked away). This lifts the "unknown outcome, and therefore unscored" share without inventing a verdict for a still-live session: a pending call or an unsettled automation run stays unknown until the settle pass sees it idle. The signal set, the penalty weights, and the grade thresholds are unchanged, so scoring of a given Signals value is identical to v1; only the outcome fed into it moved.
Variables ¶
This section is empty.
Functions ¶
func Classify ¶
func Classify(f Facts) (Outcome, Confidence)
Classify infers a session's Outcome and the Confidence in it from the facts. The order is deliberate: the strong, unambiguous signals (no human, mid-tool, an errored tail) win before the softer last-word heuristic, so a session that ended badly is never read as "completed" merely because the assistant spoke last.
func ContextHealth ¶
ContextHealth summarizes a session's context load from its ordered per-turn prompt sizes (uncached input plus cached read plus cache creation, in transcript order). Peak is the largest single-turn context the session reached: a window-independent "how heavy did it get" score in tokens, where a higher number means the session ran closer to whatever its model's limit was. Resets is the count of inferred context resets, the sharp drops that read as a compaction or a manual clear.
It is the buffered form over ContextHealthFolder, kept as the tested reference and for callers that already hold the whole slice; the store folds the streaming form instead so its memory stays bounded. An empty input (a session with no usage) yields a zero peak and zero resets; the caller distinguishes "measured as zero" from "nothing to measure" by whether it had any turns.
func IsContextReset ¶ added in v0.2.2
IsContextReset reports whether the move from a prior turn's context occupancy (prev) to this turn's (cur) reads as an inferred context reset: a compaction or a manual clear. The rule is the single tested definition two surfaces share: the ContextHealthFolder folds it into a session's reset count, and the transcript's per-message shed markers call it to decide where to draw a "context shed" divider. Both must agree on exactly which turn shed context, so neither open-codes the arithmetic.
A reset fires when this turn's occupancy falls to at most resetDropFraction of the prior turn's (the drop is at least half) AND the prior turn was already at least resetKeepFloorTokens large. The floor keeps an early swing, while the context is still warming up and small, from reading as a shed; a real compaction always sheds a substantial context, so the prior turn sits well past the floor. Both bounds are absolute token counts, deliberately independent of any model's context window, so the signal holds for a model akari has never priced. Changing either constant changes the stored reset counts, so a change must bump Version.
func Score ¶
Score returns the 0-100 quality score, its A-F grade, and whether the session is scored at all. An unknown-outcome session with no tool signal is unscored (scored=false): scoring it would invent a verdict the transcript does not support. Otherwise the score starts at 100 and subtracts the capped penalties.
Types ¶
type Archetype ¶
type Archetype string
Archetype is a coarse shape-of-session label, the kind agentsview surfaces to let a reader see at a glance whether their fleet is mostly quick lookups, standard work, or long marathons. It is inferred from cheap session facts (how many human turns, how many messages, how long it ran), not from the quality signals, so it answers "what kind of session was this" rather than "how did it go".
const ( // ArchetypeAutomation: no human turn at all, a subagent or scripted run. Checked // first so an automated job never reads as a "quick" human session. ArchetypeAutomation Archetype = "automation" // ArchetypeQuick: a short exchange, a question or a one-step fix. ArchetypeQuick Archetype = "quick" // ArchetypeStandard: an ordinary working session. ArchetypeStandard Archetype = "standard" // ArchetypeDeep: a long, involved session. ArchetypeDeep Archetype = "deep" // ArchetypeMarathon: an exceptionally long-running or message-heavy session. ArchetypeMarathon Archetype = "marathon" )
func ClassifyArchetype ¶
func ClassifyArchetype(f ArchetypeFacts) Archetype
ClassifyArchetype buckets a session by its shape. Automation wins first (no human turn). Otherwise the session takes the heaviest band whose duration or message count it reaches, so neither a long idle session nor a short burst is undersold.
type ArchetypeFacts ¶
ArchetypeFacts are the cheap session facts archetype classification reads: the human turn count (zero means automation), the total message count, and the wall-clock duration in minutes (0 when the session carries no start/end span).
type Confidence ¶
type Confidence string
Confidence grades how trustworthy an Outcome is, so the UI can mute a low-confidence guess rather than presenting every heuristic as fact.
const ( ConfHigh Confidence = "high" ConfMedium Confidence = "medium" ConfLow Confidence = "low" )
type ContextHealthFolder ¶
type ContextHealthFolder struct {
// contains filtered or unexported fields
}
ContextHealthFolder computes a session's context-health figures in one streaming pass, holding O(1) state (the running peak, the reset count, and the previous turn's size) rather than buffering the whole session. The store folds usage rows as they arrive from an ordered query, so peak memory does not grow with an arbitrarily long session. Add the turns in transcript order; Result reports the peak, the inferred reset count, and whether any turn was seen (so the caller can tell "measured as zero" from "nothing to measure").
func (*ContextHealthFolder) Add ¶
func (f *ContextHealthFolder) Add(tokens int64)
Add folds one turn's context size (uncached input plus cached read plus cache creation) into the running figures. A reset is inferred when this turn's size falls to at most resetDropFraction of the prior turn's and the prior turn was at least resetKeepFloorTokens large (see the constants above for why the floor is there).
type Facts ¶
type Facts struct {
UserMessages int // messages with role=user (any content)
LastAssistantOrd int // ordinal of the last substantive assistant message, -1 if none
LastUserOrd int // ordinal of the last substantive user message, -1 if none
ToolCallPending bool // a tool call never got a result (ended mid-tool / truncated)
TrailingFailures int // length of the run of failing tool calls at the very end
IdleLongEnough bool // the session has been inactive past the abandoned threshold
}
Facts are the raw, projection-derived inputs to outcome classification. They are gathered in the store (one set of cheap aggregates over a session's messages and tool calls) and classified here, so the rule lives in one tested place rather than in SQL. "Substantive" means a message that carries conversational content, not an empty turn that only delivered a tool result.
type Outcome ¶
type Outcome string
Outcome is how a session ended, inferred from its projection. It is a best effort: without a terminal marker in the transcript the ending is a heuristic, so every outcome carries a Confidence.
const ( // OutcomeCompleted: the last substantive turn was the assistant's, with no // unresolved tool call and no trailing failures. The agent had the last word. OutcomeCompleted Outcome = "completed" // OutcomeAbandoned: the last substantive turn was the user's, and the session has // been idle long enough that no reply is coming. The human walked away. OutcomeAbandoned Outcome = "abandoned" // OutcomeErrored: the session ends on a run of failing tool calls, so it stopped in // a broken state rather than at a clean answer. OutcomeErrored Outcome = "errored" // OutcomeUnknown: automated (no human turn), still active, truncated mid-tool, or // otherwise not classifiable. Not a verdict, an absence of one. OutcomeUnknown Outcome = "unknown" )
type PromptFacts ¶
type PromptFacts struct {
// Short is whether the prompt fell under shortPromptWords words: a terse steer that carries
// little intent on its own.
Short bool
// NoCodeContext is whether the prompt asked for a code change while pointing at no code (a
// change verb with no file path, code fence, or inline-code span).
NoCodeContext bool
// BareGreeting is whether the prompt is only greeting and pleasantry words. It matters only for
// the opening turn (an opener that just says hello is an unstructured start), but is computed
// per prompt so the store can read the opener's flag without fetching its body.
BareGreeting bool
// Digest is a normalized fingerprint of the prompt: two prompts equal after lowercasing and
// collapsing whitespace share a Digest, so the store counts session duplicates as
// count(*) - count(DISTINCT digest) over the duplicate-eligible (non-short) prompts. It is a
// 64-bit hash, not the text, so the stored key is fixed size regardless of prompt length.
Digest int64
}
PromptFacts are the fixed-size hygiene facts of a single human prompt, derived once when the message is written (see ClassifyPrompt). Materializing them at insert is what keeps the whole hygiene signal off the settle pass's memory budget: the settle pass aggregates these stored columns (booleans and one integer digest per user message) and never reads a prompt body back, so its peak memory does not track the largest prompt a session ever held. Computing them at insert costs nothing extra: the message body is already in hand there (it is the row being written), so this is the one place the text is unavoidably resident.
func ClassifyPrompt ¶
func ClassifyPrompt(prompt string) PromptFacts
ClassifyPrompt derives the fixed-size hygiene facts of one human prompt. It scans the prompt a bounded number of times (word count to the terse threshold, the change-verb and anchor regexes, the greeting scan, and the streaming digest) and allocates nothing that scales with the prompt length, so a pasted-code prompt costs O(len) time and O(1) memory. The store calls it for each real human turn (role='user', non-empty content) as the message is inserted and stores the result beside the row; the per-turn rules it applies are the same ones the Insights hygiene panel reads.
type PromptHygiene ¶
type PromptHygiene struct {
// Short is how many prompts fell under shortPromptWords words.
Short int
// Duplicate is how many prompts repeated an earlier prompt verbatim (after
// normalizing case and whitespace), counting each repeat beyond the first and only
// among prompts long enough to be a real request (see duplicateMinWords).
Duplicate int
// NoCodeContext is how many prompts asked for a code change while pointing at no
// code: no file path, no code fence, no inline-code span. Naming what to change
// without showing where makes the agent guess.
NoCodeContext int
// UnstructuredStart reports whether the opening prompt was terse or a bare greeting,
// so the session began without a clear task set.
UnstructuredStart bool
}
PromptHygiene is a session's input-quality summary, computed from its ordered human prompts. The counts feed the Insights prompt-hygiene panel and the session signals strip. They are pure functions of the prompt text, so they live here beside the scoring model and are unit tested without a database, and they do NOT feed Score: an unclear prompt is the human's to fix, not a mark against the agent.
type ScoreBreakdownItem ¶ added in v0.2.2
type ScoreBreakdownItem struct {
Label string // e.g. "errored ending", "3 tool failures"
Points int // the penalty subtracted, always > 0
}
ScoreBreakdownItem is one line of the score arithmetic: a human-readable label for a penalty and the positive number of points it subtracted. The UI stacks these to show a reader why a session scored what it did, so a low grade is explained rather than asserted.
func ScoreBreakdown ¶ added in v0.2.2
func ScoreBreakdown(s Signals) []ScoreBreakdownItem
ScoreBreakdown returns the penalty lines behind Score for the same Signals, in the same order Score applies them (outcome, failures, retries, churn, streak) and with the same caps, so the sum of the returned Points equals 100 minus the score (before the clamp at zero). It returns nil for an unscored session (the same gating as Score: an unknown outcome with no tool signal), since there is no arithmetic to explain. A scored session with no penalties (a clean completed run) also returns nil: there is nothing subtracted, so there is nothing to list.
type Signals ¶
type Signals struct {
ToolCalls int
ToolFailures int
ToolRetries int // immediate retries: a tool re-invoked with the identical input back to back
EditChurn int // repeat edits to one file beyond the first
LongestFailureStreak int
Outcome Outcome
}
Signals are the scored inputs: the tool-health counts plus the classified outcome. The store fills these from a session's projection and passes them to Score.
Source Files