Documentation
¶
Overview ¶
Package harness is the runnable form of the parity and differentiator eval suite (#1403): the thing dev/tools/evals invokes and CI (dev/ci/presubmits/evals) gates on.
It runs two suites, and the split is the plan's four-tier strategy (§2) rather than an implementation detail:
The corpus suite loads the 31 ported LangChain scenarios and the intent table and checks that the metrics can score them. It does not run them. Scoring a trajectory requires a model that chooses, and a scripted provider does not choose — replaying a fixture and asserting the tools match the fixture asserts that the script equals itself. Scoring the corpus is therefore the judge tier's job (W0.5), and what the free tier gates is that the measurement is not a constant function.
The differentiator suite runs the five scenarios upstream's harness structurally cannot express, each against the composed runtime, and holds each to its declared outcome.
Why the second suite has an allowlist and the first does not ¶
The differentiators are the scoreboard's red rows in executable form, so most of them fail today and must be allowed to. That declaration lives in exactly one place — differentiators.Scenario.Expect — and is checked in both directions, so a capability that lands without its entry being flipped fails the suite. A separate allowlist file was considered and rejected: two records of one bit is the drift the bidirectional check exists to prevent, and the Go declaration is already a diffable artifact whose shrinking a reviewer can see. What this package adds is making that state the report's headline instead of something a reader has to reconstruct from test output.
Index ¶
Constants ¶
const ( // TierDeterministic is the free, credential-free gate. Default. TierDeterministic = "deterministic" // TierJudge is the metered nightly report: the 31 corpus scenarios // scored against a live model over the fixture cluster (W0.5). It // reports rather than gates — see [Summary.OK]. TierJudge = "judge" )
Tiers, per docs/v0.3-plan.md §2. S (smoke) and U (UAT) have their own harnesses — `go test` and scripts/uat-v0.2.sh — so this runner covers E and J only.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type CeilingFinding ¶
type CeilingFinding struct {
ID string `json:"id"`
Ceiling float64 `json:"ceiling"`
Scored float64 `json:"scored"`
Why string `json:"why"`
}
CeilingFinding names a scenario the read-only surface cannot fully cover, with the score it reached and the best it could have. Reported beside the board rather than folded into it: adjusting a score for a known structural gap is the tuning the plan warns against, but leaving the gap unstated would let a reader read a capped row as a miss.
type Config ¶
type Config struct {
// Root is the repository root, the base for testdata paths.
Root string
// Tier selects the suite. Empty means TierDeterministic.
Tier string
// Scratch is a directory the fixtures own for the duration of the
// run. Empty means one is made under os.TempDir (house rule #5) and
// removed afterwards.
Scratch string
// Model is the model under test in the judge tier. Empty means the
// Anthropic default. Ignored by the deterministic tier, which has no
// model at all.
Model string
// Grader is the model that scores response_quality. Empty means the
// small Anthropic default: upstream grades with gpt-4o-mini for the
// same reason, and a grader is comparing two texts rather than
// diagnosing a cluster.
Grader string
// Provider selects the credential path for both models
// (`anthropic`, `anthropic-vertex`, or — with gemini-* names in
// Model/Grader — `gemini`, `vertex`). Empty lets compose pick from
// the environment.
Provider string
// Rows limits the judge tier to these corpus rows, by scenario id.
// Empty runs the whole corpus. For a cost probe before a full run on
// a new model, or re-running the one row a nightly lost. The corpus
// is still validated whole, so a subset cannot hide a broken row.
Rows []string
// PromptCache is the Claude prompt caching the judge tier's model
// under test runs with — a bundle's prompt_cache block, since an eval
// run has no bundle. Zero is the product default (on, 5 minutes). The
// grader never caches, so this changes only the row being measured,
// and an A/B against `off` stays reproducible from the flag alone.
PromptCache evalcompose.PromptCache
// Progress, when set, receives a line per scenario as the judge tier
// works through the corpus. Thirty-one live runs take minutes and the
// report is written all at once at the end, so without this the tier
// is indistinguishable from a hang — which is the state an operator
// is most likely to react to by killing a metered run halfway.
//
// Deliberately separate from the report writer: the CLI points this
// at stderr so the board on stdout stays a clean artifact.
Progress io.Writer
// contains filtered or unexported fields
}
Config is what the runner needs from its caller.
type CorpusSummary ¶
type CorpusSummary struct {
Dataset string `json:"dataset"`
Scenarios int `json:"scenarios"`
Intents int `json:"intents"`
Reach []evals.MetricReach `json:"reach"`
Dead []string `json:"dead_metrics,omitempty"`
// DeadDiagnostics names the diagnostic columns that score nothing
// anywhere. Separate from Dead because it does not gate — a
// diagnostic is not a parity claim, so a dead one is not a red run —
// and separate from silence because a constant that nobody names is
// read as a measurement. tool_coverage lives here permanently: the
// ported corpus expects upstream's tool names and mast cannot emit
// them, which is a fact about the two surfaces and not a defect to
// fix (#174).
DeadDiagnostics []string `json:"dead_diagnostics,omitempty"`
}
CorpusSummary is what the corpus suite found.
type JudgeScenario ¶
type JudgeScenario struct {
ID string `json:"id"`
Category string `json:"category"`
Tools []string `json:"tools_called"`
Answering []string `json:"answering_tools"`
Results []evals.MetricResult `json:"results"`
Response string `json:"response"`
Ceiling float64 `json:"intent_coverage_ceiling"`
Authored bool `json:"authored_fixture,omitempty"`
Error string `json:"error,omitempty"`
// LoopStop is why the in-turn loop guard ended this row's run (#514),
// empty when it did not. The row is scored all the same.
LoopStop string `json:"loop_stop,omitempty"`
// KVCache is the model server's KV-cache activity while this row's
// incident ran; see JudgeSummary.KVCache.
KVCache *kvwindow.Window `json:"kv_cache,omitempty"`
// Calls are the row's tool calls with their arguments and a digest
// of each result, and Violations what was wrong with them. Both are
// persisted rather than summarized away: Tools above answers "which
// tools", and #169 exists because that question and "did the run
// learn anything" have different answers often enough to matter.
Calls []evals.CallRecord `json:"calls,omitempty"`
Violations []evals.Violation `json:"violations,omitempty"`
// Misses is this row's unsatisfied intents, partitioned into the ones
// tool selection could have fixed and the ones the catalog cannot
// serve. Persisted per row because the board's list is an enumeration
// and a reader who wants one row's detail should not have to re-derive
// it from the score.
Misses evals.MissReport `json:"misses"`
}
JudgeScenario is one corpus row's run.
type JudgeSummary ¶
type JudgeSummary struct {
// Model is the model under test, Grader the one that scored
// response_quality. Both are named because the number is only worth
// reading next to what produced it — and because nothing can enforce
// that they differ, which they should.
Model string `json:"model"`
Grader string `json:"grader"`
// Provider is the credential path, or "auto" when compose picked it
// from the environment.
Provider string `json:"provider"`
// Authored is how many of the scenarios ran against a hand-written
// fixture rather than one derived from quoted spans of the expected
// response. Stated on the board because a reader's first question
// about any judge number is how much of the cluster the fixture
// author chose.
Authored int `json:"authored_fixtures"`
Scenes []JudgeScenario `json:"scenarios"`
Aggregate []MetricSummary `json:"aggregate"`
Notes []string `json:"notes,omitempty"`
Ceilings []CeilingFinding `json:"ceilings,omitempty"`
// Retries is how many model calls this board only got by waiting out
// a transient provider error (#239), and RetryWaitSeconds is how long
// that cost.
//
// On the board rather than swallowed by the retry wrapper, because a
// retry nobody can see is how a measurement quietly stops measuring:
// a provider under worsening pressure would keep producing complete,
// green, increasingly slow boards with nothing to point at. Zero on a
// healthy night, so the delta surfaces the first night it is not.
Retries int `json:"retries,omitempty"`
RetryWaitSeconds float64 `json:"retry_wait_seconds,omitempty"`
// Validity is #169's half of the board: not how many tools the run
// reached for, but whether the calls it made were well formed and
// whether they found anything.
Validity ValidityBoard `json:"validity"`
// Misses is #170's: of the questions intent_coverage marked
// unanswered, the ones a tool in the catalog would have answered.
Misses MissBoard `json:"misses"`
// Cost is J-cost-tier: a two-tier roster run live, then priced off
// the meter. Nil when the check could not run at all, which is
// itself reported as a Problem.
//
// It rides this board because it needs the same thing the corpus
// needs and nothing cheaper has — a real provider answering with
// real usage numbers. It is unlike every other row here in that its
// verdict is arithmetic rather than judgment, which is why it can
// gate (see [Summary.Problems]) where the scores cannot.
Cost *judge.CostBoard `json:"cost,omitempty"`
// CostSkipped is why there is no cost board, when the reason is that
// the run had no live model to price. Carried as a field rather than
// left as a nil Cost so the board states the absence instead of
// having one fewer section than the reader expected.
CostSkipped string `json:"cost_skipped,omitempty"`
// UnderTestUsage and GraderUsage are the tokens each model reported
// across the whole run, J-cost-tier's calls included. They are what a
// run costs before anyone has priced it — the number needed to try a
// model the catalog has no rate for — and what a nightly costs after.
UnderTestUsage *TokenUsage `json:"under_test_usage,omitempty"`
GraderUsage *TokenUsage `json:"grader_usage,omitempty"`
// KVCache is the model server's KV-cache activity folded over every
// incident, when the model under test is a provider profile that
// publishes KV-cache metrics. Fleet-level; never priced.
KVCache *kvwindow.Window `json:"kv_cache,omitempty"`
}
JudgeSummary is the metered tier's board: the 31 ported LangChain scenarios run against a live model over the fixture cluster, scored by the same deterministic evaluators the free tier checks the measurability of, plus the LLM-as-judge quality grade.
It reports; it does not gate. A low score is a finding to act on, not a broken build — the plan's §2 four-tier strategy puts the gate at S, U and E precisely so that a model having a bad day cannot turn main red. What the judge tier *can* fail on is its own machinery: a scenario whose run never happened says nothing about mast, so it is a Problem.
type MetricSummary ¶
type MetricSummary struct {
Metric string `json:"metric"`
// Mean is meaningless when Scored is zero; the report says so
// rather than printing 0.000, which reads as a failing grade.
Mean float64 `json:"mean"`
Scored int `json:"scored"`
Vacuous int `json:"vacuous"`
// Diagnostic mirrors evals.Result.Diagnostic: a metric printed for
// visibility that is not a parity claim. tool_coverage is the
// standing example — name-level overlap scores mast's consolidated
// read path as a regression.
Diagnostic bool `json:"diagnostic,omitempty"`
}
MetricSummary is one metric's mean across the rows that could score it. Vacuous rows are excluded from the mean rather than counted as zero — averaging in "there was nothing to check" is how a metric starts reporting the corpus's shape as the agent's performance.
type MissAttribution ¶
type MissAttribution struct {
Tool string `json:"tool"`
// Misses are the "<scenario> <intent>" keys charged to this tool.
Misses []string `json:"misses"`
// SoleSourceFor is every intent this tool alone answers.
SoleSourceFor []string `json:"sole_source_for"`
// Gates is how many corpus scenarios cannot reach full
// intent_coverage without it — the leverage half of the ranking.
Gates int `json:"gates_scenarios"`
}
MissAttribution is one tool's share of the board's consequential misses, with what skipping it costs the corpus (#171).
Every entry is sole-source by construction. A miss whose intent has an alternative is not attributable: some other tool would have answered, so nothing about this one caused it.
The finding this exists to surface: across both v0.4.0 boards, three of the four consequential misses are the same tool, k8s_resource_top, skipped independently by two unrelated model families. It is the only tool that satisfies either saturation intent, so skipping it does not risk the miss — it guarantees it. Sole-source × gated-scenario-count is what turns "the model skipped something" into a named item, and it is what makes the description A/B in judge.toolDescription worth running: the competing hypothesis is that the models are reasoning past the tool rather than failing to find it, and the same experiment separates the two.
type MissBoard ¶
type MissBoard struct {
// Scenarios is how many rows this board is drawn from — the ones that
// ran. A row that errored is not a row that missed nothing.
Scenarios int `json:"scenarios"`
// Consequential is every unanswered question the catalog could have
// answered, one entry per (row, intent).
Consequential []ScenarioMiss `json:"consequential,omitempty"`
// ByIntent tallies them by intent, because a miss that recurs across
// rows is a description-or-discoverability problem with one tool
// rather than three unrelated incidents.
ByIntent []string `json:"by_intent,omitempty"`
// OutOfCatalog is how many unsatisfied intents no lookout tool serves.
// Counted, not listed: they are the structural ceiling, already
// enumerated under [CeilingFinding], and repeating them here would
// read as the model's failure rather than the surface's.
OutOfCatalog int `json:"out_of_catalog"`
// Untabled is how many expected tool names intents.yaml has never
// seen. They deflate intent_coverage deliberately, but a gap in the
// table is not a miss by the model and is not counted as one.
Untabled int `json:"untabled"`
// Attribution is #171: the misses above grouped under the one tool
// that would have answered each, ranked by leverage.
Attribution []MissAttribution `json:"attribution,omitempty"`
// would have answered them, so no single tool is responsible.
// Counted rather than attributed — a miss charged to each of three
// tools would triple the ranking's numbers and point at none of them.
Shared int `json:"shared_misses"`
}
MissBoard is the corpus-wide view of what the runs failed to ask.
A list, not a rate, and for a sharper reason than #169's: the rate was measured and it points the wrong way. Counting scenarios that skipped a tool holding data flags a third of the Claude board and more than half the Gemini one, and the flagged rows score *higher* on response_quality than the rows that called everything — because most skipped tools were redundant with one that had already answered. See the note at the top of internal/evals/misses.go for the numbers.
What survives that filter is small enough to enumerate: four rows across both v0.4.0 boards. So the board names each one and says which tool would have served it, which is what a reader can act on.
type ScenarioMiss ¶
type ScenarioMiss struct {
Scenario string `json:"scenario"`
Intent string `json:"intent"`
// ServedBy are the catalog tools that would have answered it. Every
// one is a tool this run did not call — had it called one, the intent
// would be satisfied and this would not be a miss.
ServedBy []string `json:"served_by"`
}
ScenarioMiss is one consequential miss with the row it came from.
type ScenarioSummary ¶
type ScenarioSummary struct {
ID string `json:"id"`
Invariant string `json:"invariant"`
Expected string `json:"expected"`
Observed string `json:"observed"`
Matched bool `json:"matched"`
Blocked string `json:"blocked,omitempty"`
Rows []string `json:"rows"`
Reason string `json:"reason,omitempty"`
Tools []string `json:"tools,omitempty"`
Error string `json:"error,omitempty"`
}
ScenarioSummary is one differentiator's outcome, flattened so the report can be serialized.
type ScenarioViolation ¶
type ScenarioViolation struct {
Scenario string `json:"scenario"`
Violation evals.Violation `json:"violation"`
}
ScenarioViolation is one violation with the row it came from.
type Summary ¶
type Summary struct {
Tier string `json:"tier"`
Corpus CorpusSummary `json:"corpus"`
Scenes []ScenarioSummary `json:"differentiators"`
// ExpectedFail is the allowlist, in report form: the scenario IDs
// declared red today. Shrinking it is v0.3's progress metric, so it
// is a first-class field rather than something to count by eye.
ExpectedFail []string `json:"expected_fail"`
// Judge is the metered tier's board. Nil on the deterministic tier.
Judge *JudgeSummary `json:"judge,omitempty"`
// Problems are the reasons this run failed. Empty means green.
//
// The judge tier records a Problem only when the *measurement* is
// broken — a metric that scores nothing, a scenario whose run never
// happened. A low score is never a Problem: J reports, it does not
// gate (docs/v0.3-plan.md §2).
//
// J-cost-tier's findings are Problems, and they are not an exception
// to that rule but an instance of it. What it checks is arithmetic
// over the meter's own numbers — a specialist whose tier resolved to
// the cheap model and whose tokens were billed at the parent's rate
// is a mast bug with a live reproduction, not a model having a bad
// day. Nothing about the verdict depends on what the model said.
Problems []string `json:"problems,omitempty"`
}
Summary is the whole run.
func LoadSummary ¶
LoadSummary reads a board written by Summary.WriteJSON. The nightly uses it to compare a run against the previous one.
func Run ¶
Run executes the configured tier.
It returns an error only when the harness itself could not run — an unreadable fixture, an unknown tier, absent credentials. A scenario that fails, or a metric that scores nothing, is a Problem on the Summary, not an error: the caller wants the whole report, not the first thing that went wrong.
func (Summary) WriteDelta ¶
WriteDelta reports what changed between a previous board and this one.
It never decides anything: the nightly posts the delta and a human reads it. Both tiers are handled, because both have a state worth diffing — the deterministic tier's is the expected-fail allowlist, whose shrinking is v0.3's progress metric, and the judge tier's is the scores.
type TokenUsage ¶
type TokenUsage struct {
Calls int64 `json:"calls"`
PromptTokens int64 `json:"prompt_tokens"`
CachedTokens int64 `json:"cached_tokens"`
// CacheWriteTokens is the input a provider reported writing to its
// prompt cache (Anthropic's cache_creation_input_tokens), read from
// the usage sidecar: genai's usage struct has no field for it.
CacheWriteTokens int64 `json:"cache_write_tokens,omitempty"`
// CacheWrite1hTokens is the share of CacheWriteTokens written with a
// 1-hour breakpoint, billed at its own rate.
CacheWrite1hTokens int64 `json:"cache_write_1h_tokens,omitempty"`
OutputTokens int64 `json:"output_tokens"`
ThoughtsTokens int64 `json:"thoughts_tokens"`
}
TokenUsage is what a model reported over a run, summed from the usage metadata on each complete response.
type ValidityBoard ¶
type ValidityBoard struct {
// Calls is every recorded call across the corpus.
Calls int `json:"calls"`
// Malformed are the calls the tool could not accept as sent: an
// unknown name, a missing or invented argument, a type or enum the
// schema does not allow, an error result, a call with no completion.
// Every one is listed — they should be rare, and a rare thing
// summarized as a count is a thing nobody investigates.
Malformed []ScenarioViolation `json:"malformed,omitempty"`
// EmptyReads is how many well-formed calls came back with nothing.
// Not a defect: checking a hypothesis and ruling it out is the job.
EmptyReads int `json:"empty_reads"`
// Blind are scenarios where every completed call came back empty —
// the run read the cluster, saw nothing anywhere, and reported
// anyway. This is the shape #169 was opened for: on the old board
// these rows were indistinguishable from rows that read well.
Blind []string `json:"blind_runs,omitempty"`
// Counts is the tally by kind, for a one-line comparison between
// runs.
Counts []string `json:"counts,omitempty"`
// Arguments states how the recorded arguments and results were
// treated, so a reader who received this file second-hand is told
// rather than left to assume. See [argumentsNote].
Arguments string `json:"arguments"`
}
ValidityBoard is the corpus-wide view of how the runs used their tools.
Deliberately counts and lists, not a mean. A malformed-call *rate* over rows that call different numbers of tools is not comparable to itself between runs, and the number a reader can act on is never the average — it is which tool, which argument, which scope.