Documentation
¶
Overview ¶
Package quality turns a session's behavioral signals into an outcome classification and a 0-100 quality score with an A-F grade. It is pure: the store gathers the raw counts and facts from a session's projection (its messages and tool calls) and hands them here, so the scoring model can be reasoned about and unit tested without a database. The model is adapted from agentsview's penalty scheme: start at 100 and subtract for the things that make a session go badly (errors, retries, churn, an errored ending), then map the remainder to a letter.
Index ¶
- Constants
- Variables
- func Classify(f Facts) (Outcome, Confidence)
- func ContextHealth(perTurnTokens []int64) (peak int64, resets int)
- func GPAPoints(grade string) float64
- func GradeBand(grade string) string
- func GradeFor(score int) string
- func IsContextReset(prev, cur int64) bool
- func Score(s Signals) (score int, grade string, scored bool)
- func ThinkingBytesPerToken(agent string) float64
- type Archetype
- type ArchetypeFacts
- type Confidence
- type ContextHealthFolder
- type Facts
- type Outcome
- type PromptFacts
- type PromptHygiene
- type ScoreBreakdownItem
- type Signals
- type ThinkingBucket
Constants ¶
const ( MarathonMinutes = 120 MarathonMessages = 200 DeepMinutes = 30 DeepMessages = 60 StandardMinutes = 5 StandardMessages = 15 )
Archetype thresholds. A session lands in the heaviest band whose duration OR message count it reaches, so a long-but-quiet session and a short-but-chatty one both read as substantial. The store builds the same banding in SQL from these constants (one numeric source), and ClassifyArchetype is the tested reference.
const ( ThinkingLowMaxTokens = 128 // low: (0, 128] ThinkingMediumMaxTokens = 512 // medium: (128, 512] ThinkingHighMaxTokens = 2048 // high: (512, 2048], xhigh above )
The band edges over per-turn estimated reasoning tokens. A turn (or a session's hardest-decile-mean turn) sits in the band its token count reaches. The edges are a rough log2 ladder (128, 512, 2048) chosen from the observed distribution so the per-turn population spreads across the bands rather than piling into one, the failure mode of the budget-anchored tiers (models almost never spend their configured budget, so tiers keyed to it put ~everything in the bottom band).
Variables ¶
var GradeOrder = []string{"A", "B", "C", "D", "F", ""}
GradeOrder is the canonical display order for grade distributions: best to worst, with the empty string (unscored) last. Charts and facet counts iterate this so every distribution reads the same way; it lives beside GradeFor so the letter set and its ordering change together.
Functions ¶
func Classify ¶
func Classify(f Facts) (Outcome, Confidence)
Classify infers a session's Outcome and the Confidence in it from the facts. The order is deliberate: the strong, unambiguous signals (no human, mid-tool, an errored tail) win before the softer last-word heuristic, so a session that ended badly is never read as "completed" merely because the assistant spoke last.
func ContextHealth ¶
ContextHealth summarizes a session's context load from its ordered per-turn prompt sizes (uncached input plus cached read plus cache creation, in transcript order). Peak is the largest single-turn context the session reached: a window-independent "how heavy did it get" score in tokens, where a higher number means the session ran closer to whatever its model's limit was. Resets is the count of inferred context resets, the sharp drops that read as a compaction or a manual clear.
It is the buffered form over ContextHealthFolder, kept as the tested reference and for callers that already hold the whole slice; the store folds the streaming form instead so its memory stays bounded. An empty input (a session with no usage) yields a zero peak and zero resets; the caller distinguishes "measured as zero" from "nothing to measure" by whether it had any turns.
func GPAPoints ¶ added in v0.4.0
GPAPoints is the numeric weight of each letter on the standard 4.0 scale, used to average grades into a GPA trend. Defined beside GradeFor so a change to the letter set cannot silently leave a grade with no weight.
func GradeBand ¶ added in v0.4.0
GradeBand buckets a letter grade into the coarse tier the UI colours by: A and B "good", C "watch", D and F "poor", anything else (unscored) "none". The tiering is a scoring-model judgment, so it lives here rather than in each renderer; callers map the band to their own CSS class or colour.
func GradeFor ¶ added in v0.3.0
GradeFor maps a 0-100 score to its letter on the standard banding. It is the one place the score-to-letter thresholds live, so a per-session grade and a figure derived from an average score (the project card's representative grade) band the same way rather than drifting apart.
func IsContextReset ¶ added in v0.2.2
IsContextReset reports whether the move from a prior turn's context occupancy (prev) to this turn's (cur) reads as an inferred context reset: a compaction or a manual clear. The rule is the single tested definition two surfaces share: the ContextHealthFolder folds it into a session's reset count, and the transcript's per-message shed markers call it to decide where to draw a "context shed" divider. Both must agree on exactly which turn shed context, so neither open-codes the arithmetic.
A reset fires when this turn's occupancy falls to at most resetDropFraction of the prior turn's (the drop is at least half) AND the prior turn was already at least resetKeepFloorTokens large. The floor keeps an early swing, while the context is still warming up and small, from reading as a shed; a real compaction always sheds a substantial context, so the prior turn sits well past the floor. Both bounds are absolute token counts, deliberately independent of any model's context window, so the signal holds for a model akari has never priced. Changing either constant changes the stored reset counts, so a change must bump Version.
func Score ¶
Score returns the 0-100 quality score, its A-F grade, and whether the session is scored at all. An unknown-outcome session with no tool signal is unscored (scored=false): scoring it would invent a verdict the transcript does not support. Otherwise the score starts at 100 and subtracts the capped penalties.
func ThinkingBytesPerToken ¶ added in v0.3.0
ThinkingBytesPerToken returns the bytes-per-token divisor for an agent, defaulting to the Claude factor for an unknown agent (the encrypted-signature case is the common one). The store uses it to build the per-turn token expression shared by the settle derivation and the fleet aggregate.
Types ¶
type Archetype ¶
type Archetype string
Archetype is a coarse shape-of-session label, the kind agentsview surfaces to let a reader see at a glance whether their fleet is mostly quick lookups, standard work, or long marathons. It is inferred from cheap session facts (how many human turns, how many messages, how long it ran), not from the quality signals, so it answers "what kind of session was this" rather than "how did it go".
const ( // ArchetypeAutomation: no human turn at all, a subagent or scripted run. Checked // first so an automated job never reads as a "quick" human session. ArchetypeAutomation Archetype = "automation" // ArchetypeQuick: a short exchange, a question or a one-step fix. ArchetypeQuick Archetype = "quick" // ArchetypeStandard: an ordinary working session. ArchetypeStandard Archetype = "standard" // ArchetypeDeep: a long, involved session. ArchetypeDeep Archetype = "deep" // ArchetypeMarathon: an exceptionally long-running or message-heavy session. ArchetypeMarathon Archetype = "marathon" )
func ClassifyArchetype ¶
func ClassifyArchetype(f ArchetypeFacts) Archetype
ClassifyArchetype buckets a session by its shape. Automation wins first (no human turn). Otherwise the session takes the heaviest band whose duration or message count it reaches, so neither a long idle session nor a short burst is undersold.
type ArchetypeFacts ¶
ArchetypeFacts are the cheap session facts archetype classification reads: the human turn count (zero means automation), the total message count, and the wall-clock duration in minutes (0 when the session carries no start/end span).
type Confidence ¶
type Confidence string
Confidence grades how trustworthy an Outcome is, so the UI can mute a low-confidence guess rather than presenting every heuristic as fact.
const ( ConfHigh Confidence = "high" ConfMedium Confidence = "medium" ConfLow Confidence = "low" )
type ContextHealthFolder ¶
type ContextHealthFolder struct {
// contains filtered or unexported fields
}
ContextHealthFolder computes a session's context-health figures in one streaming pass, holding O(1) state (the running peak, the reset count, and the previous turn's size) rather than buffering the whole session. The store folds usage rows as they arrive from an ordered query, so peak memory does not grow with an arbitrarily long session. Add the turns in transcript order; Result reports the peak, the inferred reset count, and whether any turn was seen (so the caller can tell "measured as zero" from "nothing to measure").
func (*ContextHealthFolder) Add ¶
func (f *ContextHealthFolder) Add(tokens int64)
Add folds one turn's context size (uncached input plus cached read plus cache creation) into the running figures. A reset is inferred when this turn's size falls to at most resetDropFraction of the prior turn's and the prior turn was at least resetKeepFloorTokens large (see the constants above for why the floor is there).
type Facts ¶
type Facts struct {
UserMessages int // messages with role=user (any content)
LastAssistantOrd int // ordinal of the last substantive assistant message, -1 if none
LastUserOrd int // ordinal of the last substantive user message, -1 if none
ToolCallPending bool // a tool call never got a result (ended mid-tool / truncated)
TrailingFailures int // length of the run of failing tool calls at the very end
IdleLongEnough bool // the session has been inactive past the abandoned threshold
}
Facts are the raw, projection-derived inputs to outcome classification. They are gathered in the store (one set of cheap aggregates over a session's messages and tool calls) and classified here, so the rule lives in one tested place rather than in SQL. "Substantive" means a message that carries conversational content, not an empty turn that only delivered a tool result.
type Outcome ¶
type Outcome string
Outcome is how a session ended, inferred from its projection. It is a best effort: without a terminal marker in the transcript the ending is a heuristic, so every outcome carries a Confidence.
const ( // OutcomeCompleted: the last substantive turn was the assistant's, with no // unresolved tool call and no trailing failures. The agent had the last word. OutcomeCompleted Outcome = "completed" // OutcomeAbandoned: the last substantive turn was the user's, and the session has // been idle long enough that no reply is coming. The human walked away. OutcomeAbandoned Outcome = "abandoned" // OutcomeErrored: the session ends on a run of failing tool calls, so it stopped in // a broken state rather than at a clean answer. OutcomeErrored Outcome = "errored" // OutcomeUnknown: automated (no human turn), still active, truncated mid-tool, or // otherwise not classifiable. Not a verdict, an absence of one. OutcomeUnknown Outcome = "unknown" )
type PromptFacts ¶
type PromptFacts struct {
// Short is whether the prompt fell under shortPromptWords words: a terse steer that carries
// little intent on its own.
Short bool
// NoCodeContext is whether the prompt asked for a code change while pointing at no code (a
// change verb with no file path, code fence, or inline-code span).
NoCodeContext bool
// BareGreeting is whether the prompt is only greeting and pleasantry words. It matters only for
// the opening turn (an opener that just says hello is an unstructured start), but is computed
// per prompt so the store can read the opener's flag without fetching its body.
BareGreeting bool
// Digest is a normalized fingerprint of the prompt: two prompts equal after lowercasing and
// collapsing whitespace share a Digest, so the store counts session duplicates as
// count(*) - count(DISTINCT digest) over the duplicate-eligible (non-short) prompts. It is a
// 64-bit hash, not the text, so the stored key is fixed size regardless of prompt length.
Digest int64
}
PromptFacts are the fixed-size hygiene facts of a single human prompt, derived once when the message is written (see ClassifyPrompt). Materializing them at insert is what keeps the whole hygiene signal off the settle pass's memory budget: the settle pass aggregates these stored columns (booleans and one integer digest per user message) and never reads a prompt body back, so its peak memory does not track the largest prompt a session ever held. Computing them at insert costs nothing extra: the message body is already in hand there (it is the row being written), so this is the one place the text is unavoidably resident.
func ClassifyPrompt ¶
func ClassifyPrompt(prompt string) PromptFacts
ClassifyPrompt derives the fixed-size hygiene facts of one human prompt. It scans the prompt a bounded number of times (word count to the terse threshold, the change-verb and anchor regexes, the greeting scan, and the streaming digest) and allocates nothing that scales with the prompt length, so a pasted-code prompt costs O(len) time and O(1) memory. The store calls it for each real human turn (role='user', non-empty content) as the message is inserted and stores the result beside the row; the per-turn rules it applies are the same ones the Insights hygiene panel reads.
type PromptHygiene ¶
type PromptHygiene struct {
// Short is how many prompts fell under shortPromptWords words.
Short int
// Duplicate is how many prompts repeated an earlier prompt verbatim (after
// normalizing case and whitespace), counting each repeat beyond the first and only
// among prompts long enough to be a real request (see duplicateMinWords).
Duplicate int
// NoCodeContext is how many prompts asked for a code change while pointing at no
// code: no file path, no code fence, no inline-code span. Naming what to change
// without showing where makes the agent guess.
NoCodeContext int
// UnstructuredStart reports whether the opening prompt was terse or a bare greeting,
// so the session began without a clear task set.
UnstructuredStart bool
}
PromptHygiene is a session's input-quality summary, computed from its ordered human prompts. The counts feed the Insights prompt-hygiene panel and the session signals strip. They are pure functions of the prompt text, so they live here beside the scoring model and are unit tested without a database, and they do NOT feed Score: an unclear prompt is the human's to fix, not a mark against the agent.
type ScoreBreakdownItem ¶ added in v0.2.2
type ScoreBreakdownItem struct {
Label string // e.g. "errored ending", "3 tool failures"
Points int // the penalty subtracted, always > 0
}
ScoreBreakdownItem is one line of the score arithmetic: a human-readable label for a penalty and the positive number of points it subtracted. The UI stacks these to show a reader why a session scored what it did, so a low grade is explained rather than asserted.
func ScoreBreakdown ¶ added in v0.2.2
func ScoreBreakdown(s Signals) []ScoreBreakdownItem
ScoreBreakdown returns the penalty lines behind Score for the same Signals, in the same order Score applies them (outcome, failures, retries, churn, streak) and with the same caps, so the sum of the returned Points equals 100 minus the score (before the clamp at zero). It returns nil for an unscored session (the same gating as Score: an unknown outcome with no tool signal), since there is no arithmetic to explain. A scored session with no penalties (a clean completed run) also returns nil: there is nothing subtracted, so there is nothing to list.
type Signals ¶
type Signals struct {
ToolCalls int
ToolFailures int
ToolRetries int // immediate retries: a tool re-invoked with the identical input back to back
EditChurn int // repeat edits to one file beyond the first
LongestFailureStreak int
Outcome Outcome
}
Signals are the scored inputs: the tool-health counts plus the classified outcome. The store fills these from a session's projection and passes them to Score.
type ThinkingBucket ¶ added in v0.3.0
type ThinkingBucket string
ThinkingBucket is the banded read of an observed-thinking volume.
const ( // ThinkingOff: no thinking at all. Decided by the caller from the turn or session's // thinking count, never from a token figure, so a turn that thought only a little // reads low rather than off. ThinkingOff ThinkingBucket = "off" ThinkingLow ThinkingBucket = "low" ThinkingMedium ThinkingBucket = "medium" ThinkingHigh ThinkingBucket = "high" ThinkingXHigh ThinkingBucket = "xhigh" )
func ThinkingBucketForTokens ¶ added in v0.3.0
func ThinkingBucketForTokens(tokens float64) ThinkingBucket
ThinkingBucketForTokens maps an estimated per-turn reasoning-token count to its band. It maps a thinking turn only (tokens from a turn that reasoned): the off case is the caller's, decided from the thinking count, so a turn that thought only a few tokens reads low rather than off. A non-positive count with a reasoning block present (a redacted-to- empty trace) still reads low, the floor of the thinking bands.
Source Files