evals

package
v3.1.0-dev.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 1, 2026 License: Apache-2.0 Imports: 23 Imported by: 0

Documentation

Overview

Package evals is the behavioural eval skeleton (#652): one case, run against a real provider, graded by a deterministic verifier.

The distinction this package exists to make is the one nothing else in the repository makes. Unit tests prove functions work; the recipe checks prove files have the right shape. Neither says whether the agent found the thing it was asked to find. That is what a case measures, and it is measured the only way it can honestly be measured: by reading what the run left behind, never by reading the model's account of it.

Three rules are structural rather than stylistic, and each one is enforced by code in this package rather than by review:

  • A case names its planted defect. A defect that is not written down cannot be reviewed, and a case nobody can review is a green check with no referent.
  • A case's prompt may not contain any of its fixture's facts. That is the "withhold the location" rule, and without it the case measures whether the model can copy a noun out of the prompt. Load fails if the prompt leaks a fact (see Case.Bind).
  • A check that carries no information about the run says so, and an aggregate containing one refuses to report a verdict rather than reporting green. See Vacuous in verify.go.

The corpus is #966 and the metric program is #967. Adding a case here must require no Go: a case is a JSON file, its world is a JSON file, and the shim that renders that world is shared. If a second case needs code, the schema is wrong.

Package evals is the v0.3 parity harness: it loads the scenario corpus and the intent table, and (as the workstream lands) scores a recorded run against them.

It lives under internal/ deliberately. The harness measures mast against a specific external agent; it is a repo quality gate, not one of mast's embedding contracts. Promoting it to pkg/ later is additive, whereas shipping an unproven API and retracting it is not.

Everything here is a pure function over on-disk fixtures and a recorded event log: no provider, no credentials, no cluster.

Index

Constants

View Source
const (
	MetricIntentCoverage   = "intent_coverage"
	MetricToolCoverage     = "tool_coverage"
	MetricSeverityAccuracy = "severity_accuracy"
	MetricEffectOrdering   = "effect_ordering"
	MetricExactlyOnce      = "exactly_once"
)

Metric names. These strings are the scoreboard's column keys and the W0.4 expected-fail allowlist's vocabulary, so treat them as stable.

View Source
const (
	// ViolationUnknownTool is a call to a name that is not in the
	// catalog. The runtime rejects it, so it costs a turn and teaches
	// the model nothing.
	ViolationUnknownTool = "unknown_tool"
	// ViolationMissingRequired is a required argument the call omitted.
	ViolationMissingRequired = "missing_required"
	// ViolationWrongType is an argument whose value does not match its
	// declared type.
	ViolationWrongType = "wrong_type"
	// ViolationNotInEnum is an argument whose value is outside the
	// declared enum.
	ViolationNotInEnum = "not_in_enum"
	// ViolationUndeclaredArg is an argument the schema does not
	// declare. Reported because a model inventing an argument name is
	// usually reaching for a capability the tool does not have, and
	// because what follows is not the call the model meant either way:
	// ADK's functiontool rejects the whole call ("unexpected additional
	// properties"), while a lenient MCP server drops the argument and
	// answers a narrower question than the one that was asked.
	ViolationUndeclaredArg = "undeclared_argument"

	// ViolationErrorResult is a call the tool answered with an error.
	ViolationErrorResult = "error_result"
	// ViolationEmptyResult is a well-formed call that returned nothing.
	// Not a defect on its own — a triage agent checking a hypothesis
	// and finding nothing is doing its job — but it is the shape of the
	// failure #169 exists to make visible, and per #162's
	// tool-failure-streak a run where the calls look ordinary and the
	// *results* are the story is a real mode we detect at runtime and
	// could not previously see in the eval.
	ViolationEmptyResult = "empty_result"
	// ViolationNoCompletion is a call with no recorded response at all:
	// it never ran, lost its completion to a crash, or was declined at
	// a gate.
	ViolationNoCompletion = "no_completion"
)

Violation kinds. The split is by what a reader would do about it: the first four are the model sending something the tool cannot accept (a prompt or schema problem), the last two are the call being made correctly and the run still learning nothing (a fixture, scope, or reasoning problem).

View Source
const DefaultTimeout = 5 * time.Minute

DefaultTimeout bounds one case run.

Wall clock is a design constraint from case one, not a thing to tune later: a suite that takes hours gets batched around, and a feedback loop measured in half-days is one nobody runs before pushing. A case that cannot answer in five minutes is too big to be case one.

View Source
const MetricApprovalRequested = "approval_requested"

MetricApprovalRequested is the outcome tier's `approval_requested` check type. Not a scoreboard column: see the file comment.

View Source
const SharedDir = "_shared"

SharedDir is the fixture directory whose contents are laid down under every fixture before the fixture's own files.

It is how the kubectl shim is shared without any fixture naming its path: materialize copies _shared/ first and the fixture second, so a fixture inherits the shim by default and overrides it by shipping a file at the same relative path. One rule, no new concept, and nothing in a fixture.json points at another directory.

View Source
const SourceAnswer = "answer"

SourceAnswer is the Source value naming the agent's final output.

View Source
const SourceStartup = "startup"

SourceStartup is the Source value naming the agent process's own startup report. Valid only in Case.Preconditions — a graded check may not read it, because then the process would be grading itself.

Variables

This section is empty.

Functions

func BaselineHolds

func BaselineHolds(r Result) error

BaselineHolds decides whether a no-access run clears the grader's self-test.

The bar is a score of zero, and vacuity does not rescue a check that scored: a baseline that scores anything means the check can be satisfied without tools, which means it is measuring the model's vocabulary and will keep measuring vocabulary however the agent changes. The case does not ship.

A score of zero is necessary and not sufficient. A baseline that never ran also scores zero, and it is the cheaper of the two ways to get one: mistype a flag and the process dies at startup, every check observes nothing, and the self-test reports that the corpus is sound. The first live run of this harness failed exactly that way — the no-access tier exited 2 on a flag collision — so the two conditions below are the ones that make the zero mean something.

func DeadMetrics added in v3.1.0

func DeadMetrics(reach []MetricReach) []string

DeadMetrics returns the gating metrics that score nothing anywhere. A non-empty result is a harness failure, not a red scoreboard row: the measurement is broken, so the board says nothing either way.

Diagnostics are deliberately absent, and that omission is load-bearing in both directions. A diagnostic is not a claim about mast, so a dead one must not fail the build — tool_coverage has been dead by construction since the corpus was ported and always will be. But a dead column still has to be *said*, or the reader takes a constant for a measurement, so harness.summarizeCorpus reports it on a non-gating line of its own.

func GatingBy added in v3.1.0

func GatingBy(tbl IntentTable, ds Dataset) map[string]ToolGating

GatingBy indexes the corpus by sole-source tool.

Only tools with at least one sole-source intent appear. A tool whose every intent has an alternative can be skipped without consequence by construction, so it has no leverage to report and listing it at zero would pad the ranking with rows that can never move.

func ListFixtures

func ListFixtures(root string) ([]string, error)

ListFixtures returns the fixture names under root, excluding the shared overlay.

func ViolationCounts added in v3.1.0

func ViolationCounts(vs []Violation) []string

ViolationCounts tallies violations by kind, for a board line that says how much of a run was malformed without printing every instance. Sorted by kind so two runs' summaries line up.

Types

type Answer added in v3.1.0

type Answer struct {
	// At is when the gate adjudicated. mast does not see when the
	// operator answered.
	At time.Time
	// Outcome is what the operator said: approve, reject, edit. Empty
	// when the payload was not readable.
	Outcome string
	// Authority is where the authorization came from: operator_verdict
	// for a question about this exact call, change_set_grant for one
	// answered earlier about another call in the same set.
	Authority string
	// Disposition is what the gate then did.
	Disposition string
	// Approver is the authenticated identity the daemon stamped (#194),
	// raw. Redaction is an export-time concern and an eval never
	// exports.
	Approver string
	// Origin is the call key of the question a granted call was
	// authorized by (approval.Decision.ChangeSet). Empty for an
	// ordinary verdict.
	Origin string
}

Answer is the gate's durable record of how a parked call was resolved: approval.Decision, narrowed to what a check can use.

It is the answer as mast recorded it, which is not the same as the answer the operator gave — Disposition and Outcome come apart when mast refuses a verdict a person gave.

func (Answer) Granted added in v3.1.0

func (a Answer) Granted() bool

Granted reports whether this call fired on an answer given about a different call.

type Call added in v3.1.0

type Call struct {
	Name string
	Args map[string]any
	ID   string

	// Class is the mutation predicate's verdict at scan time.
	Class effects.Class

	// EventIndex is the call's position in the event log.
	EventIndex int

	// Completed reports whether a real completion was recorded. A call
	// without one either never ran, ran and lost its completion to a
	// crash (the ambiguous window), or was declined at a gate.
	Completed bool
	// ResponseIndex is the event index of the completion, or -1.
	ResponseIndex int

	// Response is the payload the tool answered with, nil until a
	// completion is recorded. Retained for #169: the arguments say what
	// the model asked for and only the result says whether it learned
	// anything, and the difference between "never called the tool" and
	// "called it against a scope that matched nothing" is invisible
	// without it. ADK puts a handler error under the "error" key here,
	// which is what makes a failed call distinguishable from an empty
	// one.
	Response map[string]any

	// Park is the write gate's question about this call, nil when the
	// gate did not ask. Answer is its record of how that question was
	// resolved, nil when nothing answered. Two fields because they have
	// two authors and either can be absent independently — park.go
	// enumerates the four combinations and what each one means (#295).
	Park   *Park
	Answer *Answer
}

Call is one recorded tool call and its completion, if it has one.

func (Call) Mutating added in v3.1.0

func (c Call) Mutating() bool

Mutating reports whether this call is one the outbox guards. Spawning calls count: they start sub-runs whose effects this process cannot individually attribute.

type CallRecord added in v3.1.0

type CallRecord struct {
	Index int            `json:"index"`
	Tool  string         `json:"tool"`
	Args  map[string]any `json:"args,omitempty"`
	// Result is a one-line digest of the response, empty when no
	// completion was recorded.
	Result string `json:"result,omitempty"`
	// Completed distinguishes "the tool answered with nothing" from "the
	// tool never answered", which Result alone cannot.
	Completed bool `json:"completed"`
}

CallRecord is one recorded call in the form a report persists: what the model asked for, and a digest of what came back.

A digest rather than the response itself, and that is a judgement call worth stating. A judge-tier reading is several hundred words of prose; keeping 31 scenarios' worth verbatim would turn the summary JSON into a transcript nobody opens, and the transcript already exists — the session store holds the full event log. What the board needs is enough to answer "did this call see anything", which the first line and the empty/error flags carry.

func RecordCalls added in v3.1.0

func RecordCalls(calls []Call) []CallRecord

RecordCalls renders a trace's calls for persistence.

type Case

type Case struct {
	// ID is the case's stable name. It appears in the report and is
	// the handle an issue refers to.
	ID string `json:"id"`

	// PlantedDefect is prose, and it is required. It says what is
	// wrong with the fixture's world in one sentence — the answer a
	// perfect run would arrive at. A reviewer who disagrees with a
	// check has to be able to read this to know which of the two is
	// wrong.
	PlantedDefect string `json:"planted_defect"`

	// Fixture names the directory under the fixtures root that carries
	// this case's world. The case never names a path: only the fixture
	// catalog knows paths, so a fixture can move without touching the
	// corpus, and a case cannot smuggle a location into a check.
	Fixture string `json:"fixture"`

	// Prompt is what the agent is asked. It must not contain any of
	// the fixture's facts.
	Prompt string `json:"prompt"`

	// Checks are the deterministic verifiers. All of them are the same
	// type on purpose — see Check.
	Checks []Check `json:"checks"`

	// Preconditions are what the case ASSUMES about the process it is
	// grading, as opposed to what it asserts about the agent's
	// behaviour. Each must read SourceStartup, and a failed one makes
	// the run indeterminate rather than a violation. Optional: a case
	// that assumes nothing beyond the fixture's probes declares none.
	//
	// Same type as Check, because the question has the same shape and a
	// second type would be a taxonomy nobody needs. The difference that
	// matters is in what a failure MEANS, and that lives in Verdict.
	// See precondition.go for why this tier exists (#1061).
	Preconditions []Check `json:"preconditions,omitempty"`
}

A Case is one eval: a prompt, the defect planted in the fixture it runs against, and the checks that decide whether the agent found it.

func LoadCase

func LoadCase(path string) (*Case, error)

LoadCase reads a case from disk and validates its shape.

Shape only. The checks that need the fixture — fact expansion and the prompt-leak rule — happen in Bind, because they cannot be answered without knowing what the facts are.

func (*Case) Bind

func (c *Case) Bind(f *Fixture) (*Case, error)

Bind resolves ${fact.<key>} references in the case's check terms against the fixture's facts, and enforces the withhold-the-location rule.

The returned Case is a copy with the terms expanded; the receiver is untouched, so a caller can bind one case against two fixtures without the first binding contaminating the second.

Two failures are deliberate and neither is recoverable:

An unknown fact key is an error rather than an empty expansion. An empty term is found in every haystack, so a typo in a fact name would silently convert an AllOf check into one that always passes — the exact shape of failure this package exists to catch.

A prompt that contains a fact value is an error because the case is then measuring the wrong thing. If the prompt says "payments-prod", an answer saying "payments-prod" is a copy, not a finding, and the no-access baseline will happily score it. The rule is mechanical because "did I leak the location" is not a question a reviewer reliably asks on the twentieth case.

type Check

type Check struct {
	// Name identifies the check in the report. Required, because a
	// failing check the reader cannot name is a failing check nobody
	// acts on.
	Name string `json:"name"`

	// Why says what property this check is asserting, in prose. Also
	// required: the same argument as PlantedDefect, one level down.
	Why string `json:"why"`

	// Source is "answer" or "witness:<name>".
	Source string `json:"source"`

	// AllOf are fixed strings that must all be present, matched
	// case-insensitively but otherwise literally — no regex, because a
	// corpus of regexes is a corpus of bugs nobody reviews.
	AllOf []string `json:"all_of,omitempty"`

	// AnyOf are fixed strings of which at least one must be present.
	//
	// This is not a convenience. A property with several legitimate
	// expressions — "the agent looked outside the default namespace"
	// is reached by `-n <ns>`, by `--all-namespaces`, or by listing
	// namespaces first — has to be written as AnyOf or written as one
	// arbitrary form, and the arbitrary form measures whether the
	// agent guessed the author's idiom.
	AnyOf []string `json:"any_of,omitempty"`

	// NoneOf are fixed strings that must all be absent.
	//
	// A NoneOf-only check against an empty source passes without
	// learning anything, which is why Vacuous exists. See verify.go.
	NoneOf []string `json:"none_of,omitempty"`
}

A Check is the one deterministic verifier type.

One type, not a taxonomy. Every question this skeleton can ask has the same shape — "does this witness contain these strings, and not those" — and the discipline that makes that enough is the evidence-chain rule: a check that demands the observed artefact and the value it carried *together* is hard to satisfy without having read both, which a single substring match is not.

Source is either "answer" (the agent's final output) or "witness:<name>", naming one of the fixture's witnesses. A witness is a file the *world* wrote — the fixture's kubectl shim logging what it was asked, for instance — never a file this process produced. That asymmetry is the point: an eventlog we wrote is the system reporting on itself, and it also cannot see a subagent that fanned out in-process. What the shim was asked, it was asked by somebody, and the somebody does not matter.

func (Check) Verify

func (c Check) Verify(src Source) CheckResult

Verify runs one check against one source.

Matching is case-insensitive fixed-string containment. Not regex: a corpus of regexes is a corpus of bugs nobody reviews, and the evidence-chain discipline — demand the artefact and the value it carried together — is what makes plain containment strong enough.

func (Check) WitnessName

func (c Check) WitnessName() (string, bool)

WitnessName returns the witness a Source names, and whether it named one at all.

type CheckResult

type CheckResult struct {
	Name string `json:"name"`
	Why  string `json:"why"`

	// Passed is whether every AllOf term was present and every NoneOf
	// term absent. It is not the score — see Vacuous.
	Passed bool `json:"passed"`

	// Vacuous marks a result that carries no information about the
	// run. A vacuous result never scores, whatever Passed says.
	Vacuous bool `json:"vacuous"`

	// Reason is the human-readable why, always set.
	Reason string `json:"reason"`

	// Missing are AllOf terms that were not found; Found are NoneOf
	// terms that were; Alternatives is the AnyOf set when none of it
	// matched. All three are echoed so a failing check names the
	// string rather than making the reader diff two lists.
	Missing      []string `json:"missing,omitempty"`
	Found        []string `json:"found,omitempty"`
	Alternatives []string `json:"alternatives,omitempty"`
}

A CheckResult is one check's outcome.

func (CheckResult) Score

func (r CheckResult) Score() int

Score is 1 for an informative pass and 0 otherwise.

type ConsequentialMiss added in v3.1.0

type ConsequentialMiss struct {
	Intent   string   `json:"intent"`
	ServedBy []string `json:"served_by"`
}

ConsequentialMiss is one expected intent the run left unanswered that a tool in the catalog would have answered.

ServedBy is every catalog tool that satisfies the intent, and each one is by construction a tool the run did not call: an intent with a called tool behind it is a satisfied intent and never reaches here.

type Dataset added in v3.1.0

type Dataset struct {
	Meta      Provenance
	Scenarios []Scenario
}

Dataset is one loaded scenario file.

func LoadDataset added in v3.1.0

func LoadDataset(path string) (Dataset, error)

LoadDataset reads a scenario JSONL file: a required `{"_meta": {...}}` header record followed by one Scenario per line.

Every parse failure is fatal rather than a skipped line. A silently dropped scenario would shrink the denominator of every score in the harness, which reads as progress.

type Fixture

type Fixture struct {
	// Name is the fixture directory's name. Set by LoadFixture from the
	// directory, and checked against the file so a renamed directory
	// cannot quietly keep answering to its old name.
	Name string `json:"name"`

	// Description says what world this is, in one sentence.
	Description string `json:"description"`

	// Roles are the fixture's addressable parts.
	Roles map[string]Role `json:"roles"`

	// PathRole names the role whose directory is prepended to PATH.
	// That is how the agent's `bash` reaches the fixture's tools
	// instead of the host's. Optional: a fixture with no tools of its
	// own leaves it empty.
	PathRole string `json:"path_role,omitempty"`

	// WorkdirRole names the role the agent runs in. Required.
	WorkdirRole string `json:"workdir_role"`

	// FactsFrom locates the fixture's facts, as "<role>#<key>". The
	// role's file is JSON, the key holds an object, and every value in
	// it must be a string.
	//
	// Facts live inside the world file rather than beside it on
	// purpose. The shim renders the world from that same file, so the
	// value a check asserts and the value the agent can observe are one
	// string, not two that agree today. A name asserted on both sides
	// of a test is untested.
	FactsFrom string `json:"facts_from"`

	// Witnesses map a name to a path, relative to the materialized
	// root, that the *world* writes during the run.
	//
	// Deliberately not pre-created. A witness that does not exist after
	// a run is a different fact from a witness that exists and is
	// empty — the first means nothing ever reached the world, and a
	// none_of check against it is vacuous rather than satisfied.
	Witnesses map[string]string `json:"witnesses"`

	// Env is handed to the agent process. Values may contain
	// ${role.<name>} and ${witness.<name>}, expanded to absolute paths
	// in the materialized root.
	Env map[string]string `json:"env,omitempty"`

	// Facts are loaded from FactsFrom. Not serialized: they are derived.
	Facts map[string]string `json:"-"`
	// contains filtered or unexported fields
}

A Fixture is the world a case runs against, and the only thing in the corpus that knows a path.

Roles are why. A case says "the answer must name ${fact.workload}"; it never says which file holds the workload or which directory the agent starts in. That indirection is not tidiness — it is what lets the probe rule mean something. Probes are confirmed present in the materialized world before the agent starts, so a check that fails on an absence is entitled to be called a violation rather than an environment nobody provisioned.

func LoadFixture

func LoadFixture(root, name string) (*Fixture, error)

LoadFixture reads <root>/<name>/fixture.json and its facts.

func (*Fixture) Materialize

func (f *Fixture) Materialize(dst string) (*World, error)

Materialize lays the shared overlay down, then the fixture on top, into dst, and confirms every role's probes.

type Intent added in v3.1.0

type Intent struct {
	ID          string `yaml:"id"`
	Description string `yaml:"description"`
	Write       bool   `yaml:"write"`
}

Intent is one diagnostic question a scenario expects the agent to answer, independent of which tool answers it.

type IntentTable added in v3.1.0

type IntentTable struct {
	Version       int                     `yaml:"version"`
	Intents       []Intent                `yaml:"intents"`
	UpstreamTools map[string]UpstreamTool `yaml:"upstream_tools"`
	LookoutTools  map[string]LookoutTool  `yaml:"lookout_tools"`
}

IntentTable is testdata/evals/intents.yaml.

func LoadIntentTable added in v3.1.0

func LoadIntentTable(path string) (IntentTable, error)

LoadIntentTable reads and validates the intent table. Validation is strict — an intent id referenced but never defined is a silent hole in the primary metric, not a warning.

func (IntentTable) Emittable added in v3.1.0

func (t IntentTable) Emittable(name string) bool

Emittable reports whether a tool name is one a run under this table can actually call.

lookout_tools is that catalog by construction rather than by convention: judge.Rig builds its fixture tools from these keys, and judge.NewCluster refuses to start when the table names one it has no shape for. So a name absent from lookout_tools cannot appear in any trace, and an expectation written in terms of it can never be met — which is the difference between a metric that scored badly and one that never ran (#174).

func (IntentTable) IntentFor added in v3.1.0

func (t IntentTable) IntentFor(upstreamTool string) (string, bool)

IntentFor resolves an upstream tool name to its intent id.

func (IntentTable) IntentsFor added in v3.1.0

func (t IntentTable) IntentsFor(upstreamTools []string) (intents []string, unknown []string)

IntentsFor resolves a scenario's expected_tools to the deduplicated, sorted set of intents it expects. Unknown names are returned separately rather than dropped — a name the table has never seen is a gap in the table, and swallowing it would quietly inflate coverage.

func (IntentTable) SatisfiedBy added in v3.1.0

func (t IntentTable) SatisfiedBy(calledTools []string) []string

SatisfiedBy returns the set of intents a recorded trace satisfies, given the lookout tool names it actually called.

func (IntentTable) ToolsSatisfying added in v3.1.0

func (t IntentTable) ToolsSatisfying(intent string) []string

ToolsSatisfying returns the lookout tools that satisfy an intent, in name order. Empty means the read-only catalog cannot answer that question at all, which is a ceiling rather than a miss.

func (IntentTable) Unreachable added in v3.1.0

func (t IntentTable) Unreachable(upstreamTool string) bool

Unreachable reports whether an upstream tool name is one of the phantoms — present in the dataset, absent from upstream's registry.

type LookoutTool added in v3.1.0

type LookoutTool struct {
	Satisfies []string `yaml:"satisfies"`
	Note      string   `yaml:"note"`
}

LookoutTool is one lookout MCP tool and the intents a single call to it satisfies. The sets overlap heavily and that is the point: one k8s_cluster_health call answers what upstream spends four calls on.

type MetricReach added in v3.1.0

type MetricReach struct {
	Metric string
	// Diagnostic mirrors Result.Diagnostic: a diagnostic metric is
	// reported but never gates, because it is not a claim about mast.
	Diagnostic bool
	// Scenarios is how many the metric was probed against.
	Scenarios int
	// Reaches is how many of them give it something to score.
	Reaches int
}

MetricReach reports how much of the corpus one metric can actually score — the guard against a metric that is green because it never ran.

Upstream's harness is the cautionary case, twice over: its tool_coverage reads a key no scenario carries and returns 1.0 on all 31 rows, and its severity_accuracy extracts a bracketed token the corpus never writes and returns 0 on all 31. Both are constant functions, and nothing in either output says so. Result.Vacuous marks the individual case; this is the corpus-wide roll-up that turns it into a harness failure instead of a green board.

Reach is necessary and not sufficient, which severity_accuracy shows from both ends. mast's extractor reads all 31 expected responses where upstream's reads none, so the metric has reached the whole corpus since v0.3 — and it was still wrong in two ways this guard cannot see. It silently failed to read a decorated verdict on the *actual* side (7 of 31 rows of one 2026-08-19 board, 0 of 31 of another, so the published gap between two model families was partly markdown style; fixed in extractSeverity), and the corpus states no severity definition for the buckets 20 of its 31 rows are labelled with, so the score it produces cannot be acted on however completely it reaches (#179, which demoted it to diagnostic). A metric that scores every row has cleared this bar and no other.

tool_coverage is the same lesson from the other side: this table reported it 31/31 scorable while it returned 0.000 on every row of every board, because reach asked whether a scenario had *declared* an expectation rather than whether the expectation could be *met*. A guard that measures the wrong property is the defect it exists to catch, so it caught nothing about itself. Fixed in ToolCoverage (#174) rather than here, because vacuity is the evaluator's judgement and this table only counts it.

func CorpusReach added in v3.1.0

func CorpusReach(tbl IntentTable, ds Dataset) []MetricReach

CorpusReach measures the corpus-side metrics against the loaded dataset and intent table.

The three metrics here are the ones whose vacuity is a property of the *expectation* rather than of any run: a scenario that declares no expected tools gives intent_coverage nothing to score no matter what the agent does, and one whose expected response carries no severity token gives severity_accuracy nothing to compare. That is why the probe passes an empty Trace — it isolates the corpus half, and it uses the real evaluators so the guard follows any change to what vacuity means. TestCorpusReach_IsTraceIndependent pins the property the probe relies on.

Since #174 the expectation is read against the intent table's tool catalog as well as against the scenario, which is why the table is a parameter rather than only intent_coverage's business. That is still a fact about the corpus and not about the run: which names exist is fixed before any agent starts, so the empty-trace probe holds.

effect_ordering and exactly_once are deliberately absent: their vacuity is a property of the run (no mutating effects to order), not of the corpus, so there is nothing here for them to be measured against. The differentiator tier is what proves those two are not constants — E-exactly-once asserts both score 1.00 on a run that actually mutates, and would catch an evaluator that had degenerated into returning a fixed value.

func (MetricReach) Dead added in v3.1.0

func (r MetricReach) Dead() bool

Dead reports a metric that can score nothing anywhere in the corpus. Its score carries no information, whatever it happens to be.

func (MetricReach) String added in v3.1.0

func (r MetricReach) String() string

String renders one row of the reach table.

type MetricResult added in v3.1.0

type MetricResult struct {
	// The json tags matter: a MetricResult is serialized into the judge
	// board, which is a durable artifact the nightly diffs and a human
	// reads. Untagged, it was the one PascalCase island in an otherwise
	// snake_case document.
	Metric string `json:"metric"`
	// Score is in [0,1]. For the two invariants it is binary: an
	// invariant is not partially held.
	Score float64 `json:"score"`
	// Comment says why, in operator-readable terms. It is the part a
	// human reads when a scoreboard cell goes red, so it names the
	// specific intent, tool, or call rather than reporting a bare count.
	Comment string `json:"comment,omitempty"`

	// Diagnostic marks a metric emitted for visibility only. A
	// diagnostic score is never a comparison number and never gates
	// CI — see tool_coverage, which scores mast's consolidated read
	// path as a regression by construction.
	Diagnostic bool `json:"diagnostic,omitempty"`

	// Vacuous marks a score that carries no information about the run:
	// its value is fixed by the corpus and the tool surface, not by
	// anything the agent did. Upstream's tool_coverage is 1.0 on all 31
	// rows for exactly this reason (it reads a key no row has), and a
	// harness that cannot tell the two apart reports a perfect score for
	// a metric that never ran.
	//
	// The value is usually 1.0 and for a while it was only ever that.
	// It is not always: mast's own tool_coverage is pinned at 0 on a
	// scenario whose expected names the runtime cannot emit (#174). Both
	// are constants. Which direction a constant points is not the
	// property this field is about.
	Vacuous bool `json:"vacuous,omitempty"`
}

MetricResult is one evaluator's verdict on one scenario.

func ApprovalRequested added in v3.1.0

func ApprovalRequested(tr Trace, forEffect string) MetricResult

ApprovalRequested asserts that every call the run made to forEffect was put to an operator by the write gate before it could run, and that the question the gate asked was about that call and not a proxy for it.

What the predicate reads, and what it refuses to

The park, never the verdict. In an eval the harness supplies the verdict — nothing mutating can complete otherwise — so a check that passed on "an answer exists" would be asserting that the harness ran, which it does by construction. The question is the only half mast writes on its own, before any answer exists, and it is therefore the only half that is evidence about mast (design doc §9, OQ3).

One consequence worth stating because it looks like a bug: a call the operator *rejected* passes. The claim is that the change was put to a person, and it was. Whether it then reached the cluster is a cluster_resource_property check, and conflating the two would give the safeguard a second job it cannot do from the transcript.

Matching

The gate's recorded arguments are compared against the arguments the log records for the call, canonicalized the way Call.identity is. An edit does not break this: ADK re-fires a parked call verbatim, so the durable FunctionCall part carries the arguments the model proposed — the same ones the question carried — while the operator's substituted arguments live in the AppliedEdit record beside it.

Vacuity

A run that never proposed forEffect is vacuous, not passing. That is the whole reason this returns a Result rather than a bool: on the admitted roster the interesting failure is a model that talks about scaling and never calls the tool, and a safeguard that reported green for it would be §6's rung that cannot fire. Under the outcome tier's mapping a required vacuous verdict reds the board.

func EffectOrdering added in v3.1.0

func EffectOrdering(tr Trace) MetricResult

EffectOrdering checks the outbox invariant on the recorded log: every mutating effect that completed has its intent recorded durably first, at a strictly lower event index.

This is mast-only — upstream has no equivalent, because it has no durable record of an in-flight mutation. It is the invariant that makes crash recovery decidable: a call with an intent and no completion is the ambiguous window the operator gets asked about, while a completion with no preceding intent means the log cannot tell what ran.

Read-only calls are not scored (their re-execution is free), but an orphaned read-only completion is reported in the comment as a log-integrity signal.

func EvaluateAll added in v3.1.0

func EvaluateAll(tbl IntentTable, sc Scenario, tr Trace) []MetricResult

EvaluateAll runs every deterministic evaluator against one recorded run. Order is stable so scoreboard rows line up.

func ExactlyOnce added in v3.1.0

func ExactlyOnce(tr Trace) MetricResult

ExactlyOnce checks that no mutating effect completed twice.

Identity is tool name plus canonicalized arguments: scaling two different deployments is two effects, scaling one deployment twice is the violation. This is the metric that catches a blind resume re-firing a mutation whose completion was lost to a crash — the failure the recorded-effect outbox exists to prevent, and the one upstream's harness has no way to observe.

func IntentCoverage added in v3.1.0

func IntentCoverage(tbl IntentTable, sc Scenario, tr Trace) MetricResult

IntentCoverage is v0.3's primary trajectory metric: of the diagnostic questions the scenario expects answered, what fraction did the run actually answer?

The denominator is the scenario's expected intents, including intents reachable only through upstream tool names that upstream itself cannot call (the 7 phantoms). Those names are an upstream-side artifact, and the intents behind them are reachable by lookout, so excluding them would hand mast credit for upstream's dataset bug. The unreachability is recorded as an annotation in intents.yaml, not folded into a score.

An expected tool name the intent table has never seen also stays in the denominator: a table gap should deflate the metric visibly rather than quietly shrink what is being measured.

func SeverityAccuracy added in v3.1.0

func SeverityAccuracy(sc Scenario, tr Trace) MetricResult

SeverityAccuracy is an exact match on the severity the run declared against the severity the scenario expects.

The extractor is deliberately not upstream's. Upstream matches \[(CRITICAL|WARNING|INFO|OK)\] — bracketed — against data that writes a bare "CRITICAL: " prefix, so it scores 0 on all 31 rows regardless of what any agent does. Here the expected side is read the way the corpus is actually written, and the actual side accepts the formats a model plausibly emits for the same claim.

Why this is diagnostic and not a comparison number (#179)

Every board since v0.3 has read near half here, with the misses almost all one direction — expected WARNING, got CRITICAL — on two unrelated model families. Two readings fit that equally well: the corpus labels contradict the severity definitions the corpus itself ships, or mast's rubric escalates past them. #179 asked for the misses to be partitioned between the two, against the corpus's own stated definitions rather than its labels.

That partition is not derivable from this corpus. The definitions are twelve enumerated conditions (judge.systemInstruction, taken verbatim from upstream): CRITICAL is "service down, crash loops, OOM kills, 0 ready endpoints", WARNING is "no PDB, missing probes, :latest images, wildcard RBAC", INFO is "right-sizing, orphaned PVs, suspended CronJobs". Over the 31 scenarios, the CRITICAL conditions occur in 5; the WARNING and INFO conditions occur in none at all. Twenty of the 31 rows are nonetheless labelled WARNING or INFO — buckets the rubric describes with four hygiene examples the corpus never once presents. judge.TestSeverityRubricDoesNotSpanCorpus pins that measurement.

So on almost every row that misses, there is no stated definition to partition against, and the two readings are indistinguishable in principle rather than merely by this metric. Where the rubric does decide a row it contradicts the label twice in five — LC-03 is an OOMKill labelled WARNING, LC-10 has 0 endpoints and is labelled WARNING — and both families miss both rows by answering what they were told to answer.

Tuning the rubric until the number moves was rejected in v0.3 as teaching to the test, and that still holds: there is nothing here to tune against. The number is reported, and the per-row comment stays legible, but it is not a claim about mast — the same disposition tool_coverage has, for a sibling reason. Re-promoting it needs a corpus that states a basis for the buckets it labels with.

func ToolCoverage added in v3.1.0

func ToolCoverage(tbl IntentTable, sc Scenario, tr Trace) MetricResult

ToolCoverage is upstream's name-level trajectory metric, reimplemented with the semantics upstream intended (it reads expected_trajectory, a key no row carries, so it returns 1.0 unconditionally).

It is emitted as a diagnostic and must never be reported as a comparison number. Name-level set overlap scores a better-factored read path as a regression: LC-22 names three upstream tools that one k8s_triage_workload call answers completely, and this metric scores that 0/3. Keeping it visible keeps the consolidation penalty legible instead of scored.

Vacuity is satisfiability, not declaration (#174)

A scenario reaches this metric only when at least one name it expects is a name the runtime can emit. That is a stronger test than "the scenario declared something", and the difference is the whole corpus: the ported dataset names upstream's kubectl_* surface, mast's is k8s_*, and the intersection is empty on all 31 rows. Under the declaration-shaped test every row reached, so CorpusReach reported 31/31 scorable for a metric that is 0.000 by construction — the guard against a constant function calling a constant function healthy.

The neighbouring intent_coverage never had this problem, because it builds its denominator after mapping names through the intent table. Two definitions of reach sat side by side in one file and only the declaration-shaped one was wrong; that asymmetry was the bug, rather than any single line.

Note what does not change: the score. An unsatisfiable expectation still reports 0/N and still names the tools, because the consolidation penalty is the reason this metric exists. What changes is that the harness stops averaging that 0 into a board, and says out loud that the column is a constant.

func (MetricResult) Passed added in v3.1.0

func (r MetricResult) Passed() bool

Passed reports whether a non-diagnostic result is a full score.

type MissReport added in v3.1.0

type MissReport struct {
	// Consequential is the subset attributable to tool selection.
	Consequential []ConsequentialMiss `json:"consequential,omitempty"`
	// OutOfCatalog is the intents no lookout tool serves — the structural
	// ceiling, reported beside the board and deliberately not here.
	OutOfCatalog []string `json:"out_of_catalog,omitempty"`
	// Untabled is expected tool names intents.yaml has never seen. They
	// deflate intent_coverage on purpose (a table gap should be visible),
	// but they are not a tool-selection failure and are not counted as
	// one.
	Untabled []string `json:"untabled,omitempty"`
}

MissReport partitions one scenario's unsatisfied expectations by whose problem each one is: the model's, the catalog's, or the table's.

func ClassifyMisses added in v3.1.0

func ClassifyMisses(tbl IntentTable, sc Scenario, tr Trace) MissReport

ClassifyMisses explains what IntentCoverage scored: for every expected intent the run did not satisfy, whether a tool in the catalog would have satisfied it.

It reads the same coverage the score does, so a miss the number counts and this list omits is not reachable.

func (MissReport) Empty added in v3.1.0

func (m MissReport) Empty() bool

Empty reports whether the scenario left nothing unsatisfied at all.

type Park added in v3.1.0

type Park struct {
	// RequestedAt is when the gate parked, from the event's timestamp.
	RequestedAt time.Time
	// Hint is the one line the gate wrote for a human.
	Hint string
	// Tool and Args are the call as the question described it.
	Tool string
	Args map[string]any
	// Key is approval.CallKey of the same call. It is what a change-set
	// grant names its origin by, and therefore the only way to find the
	// question a granted call was authorized by.
	Key string
	// Agent is the agent whose call was parked.
	Agent string
	// ChangeSet is the set of calls the operator was offered alongside
	// this one, nil when the question carried none. A granted call is
	// authorized by appearing here, so this is read rather than
	// decorative.
	ChangeSet []approval.ProposedChange
	// Malformed is set when the confirmation call was mast's but its
	// payload would not decode. The park is still reported: an
	// unreadable question is not the same finding as no question, and
	// collapsing them would let a payload regression read as a gate
	// that never fired.
	Malformed string
}

Park is the question the write gate wrote into the durable log before a mutating call could run.

Tool and Args are what the gate recorded, not what the trace records: they are read out of the confirmation payload rather than off the FunctionCall part beside it. That is the difference a safeguard needs — "the operator was shown this call" is a claim about the question, and comparing it against the call that ran is the only way to catch a gate that asked about a proxy.

type Provenance added in v3.1.0

type Provenance struct {
	Fixture               string `json:"fixture"`
	UpstreamRepo          string `json:"upstream_repo"`
	UpstreamPath          string `json:"upstream_path"`
	ScenarioCount         int    `json:"scenario_count"`
	Ported                string `json:"ported"`
	SourceChoice          string `json:"source_choice"`
	DriftDirection        string `json:"drift_direction"`
	ExpectedToolsNote     string `json:"expected_tools_note"`
	UpstreamEvaluatorNote string `json:"upstream_evaluator_note"`
}

Provenance is the fixture header — the first record of every scenario file. It is required, not optional: the corpus records a contested source choice (which of two drifted upstream copies is authoritative) and known defects in the upstream evaluators, and a fixture that can be copied without carrying that context loses it.

type Report

type Report struct {
	Results []Result `json:"results"`
}

A Report is a set of results, and the thing a CI leg exits on.

func (Report) Verdict

func (rep Report) Verdict() Verdict

Verdict is the weakest verdict over the graded tiers: any indeterminate makes the whole report indeterminate, then any fail makes it a fail.

The no-access tier is deliberately not one of the graded tiers. Its contract is BaselineHolds, and by construction it fails and is partly vacuous — an agent with no tools cannot name the workload, and no witness is written because nothing reached the world. Folding that into the aggregate would make every report indeterminate forever, whatever the agent did, and an alarm that is always ringing is one nobody reads. Callers must consult BaselineHolds separately; the two answers together are the result, and evalrun reports the baseline first because a tools-tier pass on a case a naked model also passes is not evidence of anything.

type Result

type Result struct {
	CaseID        string        `json:"case_id"`
	Fixture       string        `json:"fixture"`
	PlantedDefect string        `json:"planted_defect"`
	Tier          Tier          `json:"tier"`
	Checks        []CheckResult `json:"checks"`

	// Preconditions are the case's assumptions about the process,
	// evaluated against its startup report. They never score. A failed
	// one makes the result indeterminate, because a case whose
	// assumptions did not hold did not test what it claims to test
	// (#1061).
	Preconditions []CheckResult `json:"preconditions,omitempty"`

	// Answer is the agent's output, kept so a failing run can be read
	// without re-running it. For a run the deadline stopped it is
	// partial: whatever the model had written by then.
	Answer string `json:"answer,omitempty"`

	// RunError is set when the agent process itself failed. The checks
	// still run — a crashed run whose checks all fail is a clearer
	// report than a bare exit code — but the verdict can never be a
	// pass.
	RunError string `json:"run_error,omitempty"`

	// TimedOut is set when RunError is the runner's own deadline rather
	// than the process failing. BaselineHolds reads it: a no-access run
	// that worked until the deadline is a measurement, a crash is not.
	TimedOut bool `json:"timed_out,omitempty"`
}

A Result is one case run at one tier.

func (Result) Score

func (r Result) Score() int

Score is the number of informative passes.

func (Result) UnmetPreconditions

func (r Result) UnmetPreconditions() []CheckResult

UnmetPreconditions returns the preconditions that did not hold — failed, or vacuous because the process printed no startup report at all.

Vacuous counts as unmet, and that is the whole point of the tier: the regression this exists to catch (skill discovery breaking, so the case's steer never loads) shows up as terms that are missing, and a process that never reported shows up as a source that is absent. Neither is a state in which the case tested anything.

func (Result) Vacuous

func (r Result) Vacuous() bool

Vacuous reports whether any check observed nothing.

func (Result) Verdict

func (r Result) Verdict() Verdict

Verdict folds the checks into one answer.

type Role

type Role struct {
	// Path is relative to the fixture root, and may be a file or a
	// directory.
	Path string `json:"path"`

	// Probes are paths relative to Path that must exist in the
	// materialized world before the agent starts. Empty means Path
	// itself is the probe.
	Probes []string `json:"probes,omitempty"`
}

A Role is one addressable part of a fixture's world.

type RunArtifacts

type RunArtifacts struct {
	// WorldRoot is the materialized fixture. Removed unless KeepWorld.
	WorldRoot string

	// Stdout is the agent's output; Stderr is kept separately so a
	// provider error does not land in the graded text.
	Stdout string
	Stderr string

	// ExitErr is the process's error, if any.
	ExitErr error
}

A RunArtifacts bundle is what one execution left behind.

type Runner

type Runner struct {
	// Binary is the core-agent executable under test.
	Binary string

	// Args are prepended to the agent's own arguments — this is where
	// the provider selection lives, because which provider the eval
	// runs against is the caller's business and not the corpus's.
	Args []string

	// Timeout bounds the agent process. Zero means DefaultTimeout.
	Timeout time.Duration

	// KeepWorld leaves the materialized world on disk and reports its
	// path, for reading a failure afterwards.
	KeepWorld bool

	// Log, when set, receives progress lines.
	Log func(format string, args ...any)
}

A Runner executes one case against one fixture.

func (*Runner) Run

func (r *Runner) Run(ctx context.Context, c *Case, f *Fixture, tier Tier) (Result, error)

Run executes one case at one tier and grades it.

The two tiers differ by exactly one flag. That is deliberate: if the baseline run differed from the real one in prompt, fixture or checks, a baseline of zero would prove something about the baseline's setup rather than about the check.

type Scenario added in v3.1.0

type Scenario struct {
	ID       string          `json:"id"`
	Category string          `json:"category"`
	Inputs   ScenarioInputs  `json:"inputs"`
	Outputs  ScenarioOutputs `json:"outputs"`
}

Scenario is one evaluation example. The inputs/outputs shape is carried through from upstream verbatim so the corpus stays diffable against its source; ID and Category are mast-side additions, needed because upstream examples are positional and an expected-fail allowlist has to name them.

type ScenarioInputs added in v3.1.0

type ScenarioInputs struct {
	Scenario string `json:"scenario"`
}

ScenarioInputs is the prompt side of an example: a cluster observation in prose. Scenarios are text, not live clusters, which is what lets the suite run anywhere.

type ScenarioOutputs added in v3.1.0

type ScenarioOutputs struct {
	ExpectedTools    []string `json:"expected_tools"`
	ExpectedActions  []string `json:"expected_actions"`
	ExpectedResponse string   `json:"expected_response"`
}

ScenarioOutputs is the expectation side of an example.

type Source

type Source struct {
	Text    string
	Present bool
}

A Source is the text a check reads, plus whether it existed at all.

func StartupSource

func StartupSource(stderr string) Source

StartupSource extracts the agent's own startup report from a run's stderr.

Present is false when stderr carried no such line at all, which is the case that matters most: a process that died before printing its summary establishes nothing, and a none_of-only precondition against the empty string would otherwise pass. Verify already marks an absent source vacuous, and a vacuous precondition is indeterminate.

type Tier

type Tier string

A Tier is which way a case was run.

const (
	// TierTools is the real run: the agent has its tools and the
	// fixture's world on PATH.
	TierTools Tier = "tools"

	// TierNoAccess is the grader's self-test. The same case, the same
	// prompt, the same checks, and no tools at all.
	//
	// Every objective check must score 0 here or the case does not
	// ship. A check a naked model satisfies is measuring vocabulary,
	// and it measures vocabulary no matter how good the agent is — so
	// this runs in CI beside the real tier rather than being audited
	// once when the case is written.
	TierNoAccess Tier = "no-access"
)

type ToolGating added in v3.1.0

type ToolGating struct {
	Tool string `json:"tool"`
	// SoleSource is every intent this tool alone satisfies — a fact
	// about the catalog, independent of any run or any scenario.
	SoleSource []string `json:"sole_source_for"`
	// Gates is the corpus scenarios that expect at least one of those
	// intents, so they cannot reach full intent_coverage unless this tool
	// is called. Scenario count is the leverage half of the ranking: an
	// unreachable-for tool that gates one row is a curiosity, and one
	// that gates a third of the corpus is the next thing to fix.
	Gates []string `json:"gates"`
}

ToolGating is what one tool's absence costs the corpus (#171).

A miss is only attributable to a tool when that tool is the only answer to the question. Fourteen of the table's nineteen intents have two or three servers, so skipping any one of those tools costs nothing — something else answers. The remaining few have no substitute, and skipping one of those *guarantees* the miss. That is the difference between a model that chose differently and a model that failed to reach for the one thing that would have worked.

type ToolSchemas added in v3.1.0

type ToolSchemas map[string]map[string]any

ToolSchemas maps a tool name to the JSON Schema object its declaration advertises — the same `properties`/`required`/`type` shape that reaches the provider. Held as generic JSON rather than a typed schema because that is what both ADK spellings decode to, and because a validator that only understood one of them would repeat #154's mistake in a new place.

type Trace added in v3.1.0

type Trace struct {
	// Calls are the tool calls in the order the log records them,
	// engine control-flow calls and sub-agent delegations excluded (a
	// dangling adk_request_input is the normal shape of a paused
	// session, not an effect).
	Calls []Call

	// FinalText is the last non-empty model text in the log — the
	// response the severity evaluator reads.
	FinalText string

	// StructuredSeverity is set when the run produced a typed report
	// carrying an explicit severity. Empty until W1.3 lands the typed
	// report contract; the severity evaluator falls back to FinalText.
	StructuredSeverity string
}

Trace is the provider-free view of one recorded run that every evaluator scores against.

It is a plain struct on purpose. Evaluators take a Trace, never an event log, so their unit tests construct the adversarial shapes directly — a duplicated effect, an orphan completion — instead of having to stage a session store that produces them. TraceFromEvents is the only place that knows about ADK.

func TraceFromEvents added in v3.1.0

func TraceFromEvents(events adksession.Events, pred effects.Predicate, subAgents map[string]bool) Trace

TraceFromEvents extracts a Trace from a recorded session event log.

This is the seam between the harness and a real run: everything downstream is a pure function. The walk mirrors effects.pairScan's pairing rules deliberately — same event indexing, same empty-ID skip, same treatment of a confirmation placeholder as "not a completion" — because an evaluator that paired calls differently from the outbox would score a contract the runtime does not implement.

One divergence, on purpose: pairScan defers long-running calls, because at turn start the runtime has not yet decided whether they ran. An evaluator scores a finished run, where a long-running call that completed is an effect like any other — dropping it would blind exactly_once to a re-fired blocking tool, which is the failure it exists to catch.

func (Trace) CalledTools added in v3.1.0

func (t Trace) CalledTools() []string

CalledTools returns the distinct tool names the trace called, sorted.

type UpstreamTool added in v3.1.0

type UpstreamTool struct {
	Intent string `yaml:"intent"`
	Write  bool   `yaml:"write"`

	// LookoutExcluded marks a deliberate non-mapping — the write tool
	// that lookout's read-only surface does not and should not serve.
	LookoutExcluded bool   `yaml:"lookout_excluded"`
	ExclusionReason string `yaml:"exclusion_reason"`

	// UnreachableUpstream marks a name absent from upstream's own tool
	// registry. No upstream run can satisfy it, so any coverage number
	// compared against upstream must normalize for it.
	UnreachableUpstream bool `yaml:"unreachable_upstream"`
	// RegistryNearMiss is the registry name it was probably meant to be
	// (singular/plural slip); empty when there is no near miss.
	RegistryNearMiss string `yaml:"registry_near_miss"`
	// SidecarRepair is the real registry name the unwired .json sidecar
	// substitutes for this one; empty when the sidecar drops it.
	SidecarRepair string `yaml:"sidecar_repair"`
}

UpstreamTool maps one of the upstream agent's tool names onto an intent, carrying the annotations that keep upstream's own ceiling visible instead of folded into a score.

type Verdict

type Verdict string

A Verdict is an aggregate's answer.

const (
	VerdictPass Verdict = "pass"
	VerdictFail Verdict = "fail"

	// VerdictIndeterminate is the one that earns its keep.
	//
	// A run containing a check that carried no information about it
	// does not get to be green. The failure mode this prevents is
	// specific and quiet: a none_of check against a witness the world
	// never wrote passes, because nothing is absent from nothing, and
	// an aggregate that counts it reports a clean run in which nothing
	// was observed. That is exactly the shape of the credential-less
	// CI green #488 was filed for, one layer up.
	VerdictIndeterminate Verdict = "indeterminate"
)

type Violation added in v3.1.0

type Violation struct {
	// CallIndex is the call's position in Trace.Calls, so a reader can
	// find it in the recorded log rather than guessing which of three
	// calls to the same tool is meant.
	CallIndex int    `json:"call_index"`
	Tool      string `json:"tool"`
	Kind      string `json:"kind"`
	// Arg is the offending argument, empty for the kinds that are about
	// the call as a whole.
	Arg    string `json:"arg,omitempty"`
	Detail string `json:"detail"`
}

Violation is one thing wrong with one recorded call.

func ValidateCalls added in v3.1.0

func ValidateCalls(schemas ToolSchemas, calls []Call, emptyResult func(Call) bool) []Violation

ValidateCalls checks every recorded call against its declared schema and returns the violations in call order, then kind order.

A tool absent from schemas is reported as unknown rather than skipped. Skipping would make an incomplete schema map look like a clean run, which is the failure mode this whole file is a reaction to.

emptyResult decides whether a completed call learned anything. It is a callback because "empty" is a property of the tool's own result shape, which this package cannot know: the fixture cluster's no-findings reading is prose, an MCP server's is likely an empty array. Nil means the caller is not making that claim, and no ViolationEmptyResult is ever reported — silence rather than a guess.

func (Violation) String added in v3.1.0

func (v Violation) String() string

type World

type World struct {
	Root    string
	Workdir string
	PathDir string
	Env     map[string]string
	// contains filtered or unexported fields
}

A World is one materialized copy of a fixture — the thing the agent actually runs against.

Materialized rather than used in place because a run mutates it: the shim appends to its witnesses, and an agent that decides to write something writes it here. Grading the committed tree would mean the first run contaminates every later one, and it would leave the checkout dirty, which is its own well-earned lesson.

func (*World) ReadWitness

func (w *World) ReadWitness(name string) (string, bool)

ReadWitness returns a witness's contents and whether it exists.

The bool is the interesting half. Absent means the world was never touched, which is a different finding from touched-and-empty, and verify.go spends that difference.

Directories

Path Synopsis
cmd
evals command
Command evals runs the v0.3 parity eval suite (docs/v0.3-plan.md W0.4).
Command evals runs the v0.3 parity eval suite (docs/v0.3-plan.md W0.4).
outcome command
Command outcome runs the O tier: the outcome evals (#297, docs/outcome-evals-design.md).
Command outcome runs the O tier: the outcome evals (#297, docs/outcome-evals-design.md).
Package differentiators holds the v0.3 eval scenarios that the upstream LangChain SRE harness structurally cannot express (docs/v0.3-plan.md W0.3).
Package differentiators holds the v0.3 eval scenarios that the upstream LangChain SRE harness structurally cannot express (docs/v0.3-plan.md W0.3).
Package evalcompose provides the runtime wiring helpers used by the judge and outcome eval tiers: model construction, per-specialist budget metering, fan-out roster assembly, and the in-turn loop guard.
Package evalcompose provides the runtime wiring helpers used by the judge and outcome eval tiers: model construction, per-specialist budget metering, fan-out roster assembly, and the in-turn loop guard.
Package harness is the runnable form of the v0.3 parity eval suite (docs/v0.3-plan.md W0.4): the thing scripts/evals.sh invokes and CI gates on.
Package harness is the runnable form of the v0.3 parity eval suite (docs/v0.3-plan.md W0.4): the thing scripts/evals.sh invokes and CI gates on.
Package judge is the metered tier of the v0.3 parity eval suite (docs/v0.3-plan.md W0.5): the 31-scenario corpus scored against a real model, which is the only tier that produces a LangChain-comparable number.
Package judge is the metered tier of the v0.3 parity eval suite (docs/v0.3-plan.md W0.5): the 31-scenario corpus scored against a real model, which is the only tier that produces a LangChain-comparable number.
Package outcome is the O tier's corpus: the fixture-role catalog, the case schema, and the loader that refuses a corpus which cannot measure what it claims to.
Package outcome is the O tier's corpus: the fixture-role catalog, the case schema, and the loader that refuses a corpus which cannot measure what it claims to.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL