Documentation
¶
Overview ¶
Package evals is the v0.3 parity harness: it loads the scenario corpus and the intent table, and (as the workstream lands) scores a recorded run against them.
It lives under internal/ deliberately. The harness measures mast against a specific external agent; it is a repo quality gate, not one of mast's embedding contracts. Promoting it to pkg/ later is additive, whereas shipping an unproven API and retracting it is not.
Everything here is a pure function over on-disk fixtures and a recorded event log: no provider, no credentials, no cluster.
Index ¶
- Constants
- func DeadMetrics(reach []MetricReach) []string
- type Call
- type Dataset
- type Intent
- type IntentTable
- type LookoutTool
- type MetricReach
- type Provenance
- type Result
- func EffectOrdering(tr Trace) Result
- func EvaluateAll(tbl IntentTable, sc Scenario, tr Trace) []Result
- func ExactlyOnce(tr Trace) Result
- func IntentCoverage(tbl IntentTable, sc Scenario, tr Trace) Result
- func SeverityAccuracy(sc Scenario, tr Trace) Result
- func ToolCoverage(sc Scenario, tr Trace) Result
- type Scenario
- type ScenarioInputs
- type ScenarioOutputs
- type Trace
- type UpstreamTool
Constants ¶
const ( MetricIntentCoverage = "intent_coverage" MetricToolCoverage = "tool_coverage" MetricSeverityAccuracy = "severity_accuracy" MetricEffectOrdering = "effect_ordering" MetricExactlyOnce = "exactly_once" )
Metric names. These strings are the scoreboard's column keys and the W0.4 expected-fail allowlist's vocabulary, so treat them as stable.
Variables ¶
This section is empty.
Functions ¶
func DeadMetrics ¶
func DeadMetrics(reach []MetricReach) []string
DeadMetrics returns the gating metrics that score nothing anywhere. A non-empty result is a harness failure, not a red scoreboard row: the measurement is broken, so the board says nothing either way.
Types ¶
type Call ¶
type Call struct {
Name string
Args map[string]any
ID string
// Class is the mutation predicate's verdict at scan time.
Class effects.Class
// EventIndex is the call's position in the event log.
EventIndex int
// Completed reports whether a real completion was recorded. A call
// without one either never ran, ran and lost its completion to a
// crash (the ambiguous window), or was declined at a gate.
Completed bool
// ResponseIndex is the event index of the completion, or -1.
ResponseIndex int
}
Call is one recorded tool call and its completion, if it has one.
type Dataset ¶
type Dataset struct {
Meta Provenance
Scenarios []Scenario
}
Dataset is one loaded scenario file.
func LoadDataset ¶
LoadDataset reads a scenario JSONL file: a required `{"_meta": {...}}` header record followed by one Scenario per line.
Every parse failure is fatal rather than a skipped line. A silently dropped scenario would shrink the denominator of every score in the harness, which reads as progress.
type Intent ¶
type Intent struct {
ID string `yaml:"id"`
Description string `yaml:"description"`
Write bool `yaml:"write"`
}
Intent is one diagnostic question a scenario expects the agent to answer, independent of which tool answers it.
type IntentTable ¶
type IntentTable struct {
Version int `yaml:"version"`
Intents []Intent `yaml:"intents"`
UpstreamTools map[string]UpstreamTool `yaml:"upstream_tools"`
LookoutTools map[string]LookoutTool `yaml:"lookout_tools"`
}
IntentTable is testdata/evals/intents.yaml.
func LoadIntentTable ¶
func LoadIntentTable(path string) (IntentTable, error)
LoadIntentTable reads and validates the intent table. Validation is strict — an intent id referenced but never defined is a silent hole in the primary metric, not a warning.
func (IntentTable) IntentFor ¶
func (t IntentTable) IntentFor(upstreamTool string) (string, bool)
IntentFor resolves an upstream tool name to its intent id.
func (IntentTable) IntentsFor ¶
func (t IntentTable) IntentsFor(upstreamTools []string) (intents []string, unknown []string)
IntentsFor resolves a scenario's expected_tools to the deduplicated, sorted set of intents it expects. Unknown names are returned separately rather than dropped — a name the table has never seen is a gap in the table, and swallowing it would quietly inflate coverage.
func (IntentTable) SatisfiedBy ¶
func (t IntentTable) SatisfiedBy(calledTools []string) []string
SatisfiedBy returns the set of intents a recorded trace satisfies, given the lookout tool names it actually called.
func (IntentTable) Unreachable ¶
func (t IntentTable) Unreachable(upstreamTool string) bool
Unreachable reports whether an upstream tool name is one of the phantoms — present in the dataset, absent from upstream's registry.
type LookoutTool ¶
LookoutTool is one lookout MCP tool and the intents a single call to it satisfies. The sets overlap heavily and that is the point: one k8s_cluster_health call answers what upstream spends four calls on.
type MetricReach ¶
type MetricReach struct {
Metric string
// Diagnostic mirrors Result.Diagnostic: a diagnostic metric is
// reported but never gates, because it is not a claim about mast.
Diagnostic bool
// Scenarios is how many the metric was probed against.
Scenarios int
// Reaches is how many of them give it something to score.
Reaches int
}
MetricReach reports how much of the corpus one metric can actually score — the guard against a metric that is green because it never ran.
Upstream's harness is the cautionary case, twice over: its tool_coverage reads a key no scenario carries and returns 1.0 on all 31 rows, and its severity_accuracy extracts a bracketed token the corpus never writes and returns 0 on all 31. Both are constant functions, and nothing in either output says so. Result.Vacuous marks the individual case; this is the corpus-wide roll-up that turns it into a harness failure instead of a green board.
func CorpusReach ¶
func CorpusReach(tbl IntentTable, ds Dataset) []MetricReach
CorpusReach measures the corpus-side metrics against the loaded dataset and intent table.
The three metrics here are the ones whose vacuity is a property of the *expectation* rather than of any run: a scenario that declares no expected tools gives intent_coverage nothing to score no matter what the agent does, and one whose expected response carries no severity token gives severity_accuracy nothing to compare. That is why the probe passes an empty Trace — it isolates the corpus half, and it uses the real evaluators so the guard follows any change to what vacuity means. TestCorpusReach_IsTraceIndependent pins the property the probe relies on.
effect_ordering and exactly_once are deliberately absent: their vacuity is a property of the run (no mutating effects to order), not of the corpus, so there is nothing here for them to be measured against. The differentiator tier is what proves those two are not constants — E-exactly-once asserts both score 1.00 on a run that actually mutates, and would catch an evaluator that had degenerated into returning a fixed value.
func (MetricReach) Dead ¶
func (r MetricReach) Dead() bool
Dead reports a metric that can score nothing anywhere in the corpus. Its score carries no information, whatever it happens to be.
func (MetricReach) String ¶
func (r MetricReach) String() string
String renders one row of the reach table.
type Provenance ¶
type Provenance struct {
Fixture string `json:"fixture"`
UpstreamRepo string `json:"upstream_repo"`
UpstreamPath string `json:"upstream_path"`
ScenarioCount int `json:"scenario_count"`
Ported string `json:"ported"`
SourceChoice string `json:"source_choice"`
DriftDirection string `json:"drift_direction"`
ExpectedToolsNote string `json:"expected_tools_note"`
UpstreamEvaluatorNote string `json:"upstream_evaluator_note"`
}
Provenance is the fixture header — the first record of every scenario file. It is required, not optional: the corpus records a contested source choice (which of two drifted upstream copies is authoritative) and known defects in the upstream evaluators, and a fixture that can be copied without carrying that context loses it.
type Result ¶
type Result struct {
// The json tags matter: a Result is serialized into the judge
// board, which is a durable artifact the nightly diffs and a human
// reads. Untagged, it was the one PascalCase island in an otherwise
// snake_case document.
Metric string `json:"metric"`
// Score is in [0,1]. For the two invariants it is binary: an
// invariant is not partially held.
Score float64 `json:"score"`
// Comment says why, in operator-readable terms. It is the part a
// human reads when a scoreboard cell goes red, so it names the
// specific intent, tool, or call rather than reporting a bare count.
Comment string `json:"comment,omitempty"`
// Diagnostic marks a metric emitted for visibility only. A
// diagnostic score is never a comparison number and never gates
// CI — see tool_coverage, which scores mast's consolidated read
// path as a regression by construction.
Diagnostic bool `json:"diagnostic,omitempty"`
// Vacuous marks a score that is 1.0 because there was nothing to
// measure, not because anything was demonstrated. Upstream's
// tool_coverage is 1.0 on all 31 rows for exactly this reason
// (it reads a key no row has), and a harness that cannot tell the
// two apart reports a perfect score for a metric that never ran.
Vacuous bool `json:"vacuous,omitempty"`
}
Result is one evaluator's verdict on one scenario.
func EffectOrdering ¶
EffectOrdering checks the outbox invariant on the recorded log: every mutating effect that completed has its intent recorded durably first, at a strictly lower event index.
This is mast-only — upstream has no equivalent, because it has no durable record of an in-flight mutation. It is the invariant that makes crash recovery decidable: a call with an intent and no completion is the ambiguous window the operator gets asked about, while a completion with no preceding intent means the log cannot tell what ran.
Read-only calls are not scored (their re-execution is free), but an orphaned read-only completion is reported in the comment as a log-integrity signal.
func EvaluateAll ¶
func EvaluateAll(tbl IntentTable, sc Scenario, tr Trace) []Result
EvaluateAll runs every deterministic evaluator against one recorded run. Order is stable so scoreboard rows line up.
func ExactlyOnce ¶
ExactlyOnce checks that no mutating effect completed twice.
Identity is tool name plus canonicalized arguments: scaling two different deployments is two effects, scaling one deployment twice is the violation. This is the metric that catches a blind resume re-firing a mutation whose completion was lost to a crash — the failure the recorded-effect outbox exists to prevent, and the one upstream's harness has no way to observe.
func IntentCoverage ¶
func IntentCoverage(tbl IntentTable, sc Scenario, tr Trace) Result
IntentCoverage is v0.3's primary trajectory metric: of the diagnostic questions the scenario expects answered, what fraction did the run actually answer?
The denominator is the scenario's expected intents, including intents reachable only through upstream tool names that upstream itself cannot call (the 7 phantoms). Those names are an upstream-side artifact, and the intents behind them are reachable by lookout, so excluding them would hand mast credit for upstream's dataset bug. The unreachability is recorded as an annotation in intents.yaml, not folded into a score.
An expected tool name the intent table has never seen also stays in the denominator: a table gap should deflate the metric visibly rather than quietly shrink what is being measured.
func SeverityAccuracy ¶
SeverityAccuracy is an exact match on the severity the run declared against the severity the scenario expects.
The extractor is deliberately not upstream's. Upstream matches \[(CRITICAL|WARNING|INFO|OK)\] — bracketed — against data that writes a bare "CRITICAL: " prefix, so it scores 0 on all 31 rows regardless of what any agent does. Here the expected side is read the way the corpus is actually written, and the actual side accepts the formats a model plausibly emits for the same claim.
func ToolCoverage ¶
ToolCoverage is upstream's name-level trajectory metric, reimplemented with the semantics upstream intended (it reads expected_trajectory, a key no row carries, so it returns 1.0 unconditionally).
It is emitted as a diagnostic and must never be reported as a comparison number. Name-level set overlap scores a better-factored read path as a regression: LC-22 names three upstream tools that one k8s_triage_workload call answers completely, and this metric scores that 0/3. Keeping it visible keeps the consolidation penalty legible instead of scored.
type Scenario ¶
type Scenario struct {
ID string `json:"id"`
Category string `json:"category"`
Inputs ScenarioInputs `json:"inputs"`
Outputs ScenarioOutputs `json:"outputs"`
}
Scenario is one evaluation example. The inputs/outputs shape is carried through from upstream verbatim so the corpus stays diffable against its source; ID and Category are mast-side additions, needed because upstream examples are positional and an expected-fail allowlist has to name them.
type ScenarioInputs ¶
type ScenarioInputs struct {
Scenario string `json:"scenario"`
}
ScenarioInputs is the prompt side of an example: a cluster observation in prose. Scenarios are text, not live clusters, which is what lets the suite run anywhere.
type ScenarioOutputs ¶
type ScenarioOutputs struct {
ExpectedTools []string `json:"expected_tools"`
ExpectedActions []string `json:"expected_actions"`
ExpectedResponse string `json:"expected_response"`
}
ScenarioOutputs is the expectation side of an example.
type Trace ¶
type Trace struct {
// Calls are the tool calls in the order the log records them,
// engine control-flow calls and sub-agent delegations excluded (a
// dangling adk_request_input is the normal shape of a paused
// session, not an effect).
Calls []Call
// FinalText is the last non-empty model text in the log — the
// response the severity evaluator reads.
FinalText string
// StructuredSeverity is set when the run produced a typed report
// carrying an explicit severity. Empty until W1.3 lands the typed
// report contract; the severity evaluator falls back to FinalText.
StructuredSeverity string
}
Trace is the provider-free view of one recorded run that every evaluator scores against.
It is a plain struct on purpose. Evaluators take a Trace, never an event log, so their unit tests construct the adversarial shapes directly — a duplicated effect, an orphan completion — instead of having to stage a session store that produces them. TraceFromEvents is the only place that knows about ADK.
func TraceFromEvents ¶
func TraceFromEvents(events adksession.Events, pred effects.Predicate, subAgents map[string]bool) Trace
TraceFromEvents extracts a Trace from a recorded session event log.
This is the seam between the harness and a real run: everything downstream is a pure function. The walk mirrors effects.pairScan's pairing rules deliberately — same event indexing, same empty-ID skip, same treatment of a confirmation placeholder as "not a completion" — because an evaluator that paired calls differently from the outbox would score a contract the runtime does not implement.
One divergence, on purpose: pairScan defers long-running calls, because at turn start the runtime has not yet decided whether they ran. An evaluator scores a finished run, where a long-running call that completed is an effect like any other — dropping it would blind exactly_once to a re-fired blocking tool, which is the failure it exists to catch.
func (Trace) CalledTools ¶
CalledTools returns the distinct tool names the trace called, sorted.
type UpstreamTool ¶
type UpstreamTool struct {
Intent string `yaml:"intent"`
Write bool `yaml:"write"`
// LookoutExcluded marks a deliberate non-mapping — the write tool
// that lookout's read-only surface does not and should not serve.
LookoutExcluded bool `yaml:"lookout_excluded"`
ExclusionReason string `yaml:"exclusion_reason"`
// UnreachableUpstream marks a name absent from upstream's own tool
// registry. No upstream run can satisfy it, so any coverage number
// compared against upstream must normalize for it.
UnreachableUpstream bool `yaml:"unreachable_upstream"`
// RegistryNearMiss is the registry name it was probably meant to be
// (singular/plural slip); empty when there is no near miss.
RegistryNearMiss string `yaml:"registry_near_miss"`
// SidecarRepair is the real registry name the unwired .json sidecar
// substitutes for this one; empty when the sidecar drops it.
SidecarRepair string `yaml:"sidecar_repair"`
}
UpstreamTool maps one of the upstream agent's tool names onto an intent, carrying the annotations that keep upstream's own ceiling visible instead of folded into a score.
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
cmd
|
|
|
evals
command
Command evals runs the v0.3 parity eval suite (docs/v0.3-plan.md W0.4).
|
Command evals runs the v0.3 parity eval suite (docs/v0.3-plan.md W0.4). |
|
Package differentiators holds the v0.3 eval scenarios that the upstream LangChain SRE harness structurally cannot express (docs/v0.3-plan.md W0.3).
|
Package differentiators holds the v0.3 eval scenarios that the upstream LangChain SRE harness structurally cannot express (docs/v0.3-plan.md W0.3). |
|
Package harness is the runnable form of the v0.3 parity eval suite (docs/v0.3-plan.md W0.4): the thing scripts/evals.sh invokes and CI gates on.
|
Package harness is the runnable form of the v0.3 parity eval suite (docs/v0.3-plan.md W0.4): the thing scripts/evals.sh invokes and CI gates on. |
|
Package judge is the metered tier of the v0.3 parity eval suite (docs/v0.3-plan.md W0.5): the 31-scenario corpus scored against a real model, which is the only tier that produces a LangChain-comparable number.
|
Package judge is the metered tier of the v0.3 parity eval suite (docs/v0.3-plan.md W0.5): the 31-scenario corpus scored against a real model, which is the only tier that produces a LangChain-comparable number. |