evals

package
v3.0.0-dev.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 12, 2026 License: Apache-2.0 Imports: 14 Imported by: 0

Documentation

Overview

Package evals is the behavioural eval skeleton (#652): one case, run against a real provider, graded by a deterministic verifier.

The distinction this package exists to make is the one nothing else in the repository makes. Unit tests prove functions work; the recipe checks prove files have the right shape. Neither says whether the agent found the thing it was asked to find. That is what a case measures, and it is measured the only way it can honestly be measured: by reading what the run left behind, never by reading the model's account of it.

Three rules are structural rather than stylistic, and each one is enforced by code in this package rather than by review:

  • A case names its planted defect. A defect that is not written down cannot be reviewed, and a case nobody can review is a green check with no referent.
  • A case's prompt may not contain any of its fixture's facts. That is the "withhold the location" rule, and without it the case measures whether the model can copy a noun out of the prompt. Load fails if the prompt leaks a fact (see Case.Bind).
  • A check that carries no information about the run says so, and an aggregate containing one refuses to report a verdict rather than reporting green. See Vacuous in verify.go.

The corpus is #966 and the metric program is #967. Adding a case here must require no Go: a case is a JSON file, its world is a JSON file, and the shim that renders that world is shared. If a second case needs code, the schema is wrong.

Index

Constants

View Source
const DefaultTimeout = 5 * time.Minute

DefaultTimeout bounds one case run.

Wall clock is a design constraint from case one, not a thing to tune later: a suite that takes hours gets batched around, and a feedback loop measured in half-days is one nobody runs before pushing. A case that cannot answer in five minutes is too big to be case one.

View Source
const SharedDir = "_shared"

SharedDir is the fixture directory whose contents are laid down under every fixture before the fixture's own files.

It is how the kubectl shim is shared without any fixture naming its path: materialize copies _shared/ first and the fixture second, so a fixture inherits the shim by default and overrides it by shipping a file at the same relative path. One rule, no new concept, and nothing in a fixture.json points at another directory.

View Source
const SourceAnswer = "answer"

SourceAnswer is the Source value naming the agent's final output.

View Source
const SourceStartup = "startup"

SourceStartup is the Source value naming the agent process's own startup report. Valid only in Case.Preconditions — a graded check may not read it, because then the process would be grading itself.

Variables

This section is empty.

Functions

func BaselineHolds

func BaselineHolds(r Result) error

BaselineHolds decides whether a no-access run clears the grader's self-test.

The bar is a score of zero, and vacuity does not rescue a check that scored: a baseline that scores anything means the check can be satisfied without tools, which means it is measuring the model's vocabulary and will keep measuring vocabulary however the agent changes. The case does not ship.

A score of zero is necessary and not sufficient. A baseline that never ran also scores zero, and it is the cheaper of the two ways to get one: mistype a flag and the process dies at startup, every check observes nothing, and the self-test reports that the corpus is sound. The first live run of this harness failed exactly that way — the no-access tier exited 2 on a flag collision — so the two conditions below are the ones that make the zero mean something.

func ListFixtures

func ListFixtures(root string) ([]string, error)

ListFixtures returns the fixture names under root, excluding the shared overlay.

Types

type Case

type Case struct {
	// ID is the case's stable name. It appears in the report and is
	// the handle an issue refers to.
	ID string `json:"id"`

	// PlantedDefect is prose, and it is required. It says what is
	// wrong with the fixture's world in one sentence — the answer a
	// perfect run would arrive at. A reviewer who disagrees with a
	// check has to be able to read this to know which of the two is
	// wrong.
	PlantedDefect string `json:"planted_defect"`

	// Fixture names the directory under the fixtures root that carries
	// this case's world. The case never names a path: only the fixture
	// catalog knows paths, so a fixture can move without touching the
	// corpus, and a case cannot smuggle a location into a check.
	Fixture string `json:"fixture"`

	// Prompt is what the agent is asked. It must not contain any of
	// the fixture's facts.
	Prompt string `json:"prompt"`

	// Checks are the deterministic verifiers. All of them are the same
	// type on purpose — see Check.
	Checks []Check `json:"checks"`

	// Preconditions are what the case ASSUMES about the process it is
	// grading, as opposed to what it asserts about the agent's
	// behaviour. Each must read SourceStartup, and a failed one makes
	// the run indeterminate rather than a violation. Optional: a case
	// that assumes nothing beyond the fixture's probes declares none.
	//
	// Same type as Check, because the question has the same shape and a
	// second type would be a taxonomy nobody needs. The difference that
	// matters is in what a failure MEANS, and that lives in Verdict.
	// See precondition.go for why this tier exists (#1061).
	Preconditions []Check `json:"preconditions,omitempty"`
}

A Case is one eval: a prompt, the defect planted in the fixture it runs against, and the checks that decide whether the agent found it.

func LoadCase

func LoadCase(path string) (*Case, error)

LoadCase reads a case from disk and validates its shape.

Shape only. The checks that need the fixture — fact expansion and the prompt-leak rule — happen in Bind, because they cannot be answered without knowing what the facts are.

func (*Case) Bind

func (c *Case) Bind(f *Fixture) (*Case, error)

Bind resolves ${fact.<key>} references in the case's check terms against the fixture's facts, and enforces the withhold-the-location rule.

The returned Case is a copy with the terms expanded; the receiver is untouched, so a caller can bind one case against two fixtures without the first binding contaminating the second.

Two failures are deliberate and neither is recoverable:

An unknown fact key is an error rather than an empty expansion. An empty term is found in every haystack, so a typo in a fact name would silently convert an AllOf check into one that always passes — the exact shape of failure this package exists to catch.

A prompt that contains a fact value is an error because the case is then measuring the wrong thing. If the prompt says "payments-prod", an answer saying "payments-prod" is a copy, not a finding, and the no-access baseline will happily score it. The rule is mechanical because "did I leak the location" is not a question a reviewer reliably asks on the twentieth case.

type Check

type Check struct {
	// Name identifies the check in the report. Required, because a
	// failing check the reader cannot name is a failing check nobody
	// acts on.
	Name string `json:"name"`

	// Why says what property this check is asserting, in prose. Also
	// required: the same argument as PlantedDefect, one level down.
	Why string `json:"why"`

	// Source is "answer" or "witness:<name>".
	Source string `json:"source"`

	// AllOf are fixed strings that must all be present, matched
	// case-insensitively but otherwise literally — no regex, because a
	// corpus of regexes is a corpus of bugs nobody reviews.
	AllOf []string `json:"all_of,omitempty"`

	// AnyOf are fixed strings of which at least one must be present.
	//
	// This is not a convenience. A property with several legitimate
	// expressions — "the agent looked outside the default namespace"
	// is reached by `-n <ns>`, by `--all-namespaces`, or by listing
	// namespaces first — has to be written as AnyOf or written as one
	// arbitrary form, and the arbitrary form measures whether the
	// agent guessed the author's idiom.
	AnyOf []string `json:"any_of,omitempty"`

	// NoneOf are fixed strings that must all be absent.
	//
	// A NoneOf-only check against an empty source passes without
	// learning anything, which is why Vacuous exists. See verify.go.
	NoneOf []string `json:"none_of,omitempty"`
}

A Check is the one deterministic verifier type.

One type, not a taxonomy. Every question this skeleton can ask has the same shape — "does this witness contain these strings, and not those" — and the discipline that makes that enough is the evidence-chain rule: a check that demands the observed artefact and the value it carried *together* is hard to satisfy without having read both, which a single substring match is not.

Source is either "answer" (the agent's final output) or "witness:<name>", naming one of the fixture's witnesses. A witness is a file the *world* wrote — the fixture's kubectl shim logging what it was asked, for instance — never a file this process produced. That asymmetry is the point: an eventlog we wrote is the system reporting on itself, and it also cannot see a subagent that fanned out in-process. What the shim was asked, it was asked by somebody, and the somebody does not matter.

func (Check) Verify

func (c Check) Verify(src Source) CheckResult

Verify runs one check against one source.

Matching is case-insensitive fixed-string containment. Not regex: a corpus of regexes is a corpus of bugs nobody reviews, and the evidence-chain discipline — demand the artefact and the value it carried together — is what makes plain containment strong enough.

func (Check) WitnessName

func (c Check) WitnessName() (string, bool)

WitnessName returns the witness a Source names, and whether it named one at all.

type CheckResult

type CheckResult struct {
	Name string `json:"name"`
	Why  string `json:"why"`

	// Passed is whether every AllOf term was present and every NoneOf
	// term absent. It is not the score — see Vacuous.
	Passed bool `json:"passed"`

	// Vacuous marks a result that carries no information about the
	// run. A vacuous result never scores, whatever Passed says.
	Vacuous bool `json:"vacuous"`

	// Reason is the human-readable why, always set.
	Reason string `json:"reason"`

	// Missing are AllOf terms that were not found; Found are NoneOf
	// terms that were; Alternatives is the AnyOf set when none of it
	// matched. All three are echoed so a failing check names the
	// string rather than making the reader diff two lists.
	Missing      []string `json:"missing,omitempty"`
	Found        []string `json:"found,omitempty"`
	Alternatives []string `json:"alternatives,omitempty"`
}

A CheckResult is one check's outcome.

func (CheckResult) Score

func (r CheckResult) Score() int

Score is 1 for an informative pass and 0 otherwise.

type Fixture

type Fixture struct {
	// Name is the fixture directory's name. Set by LoadFixture from the
	// directory, and checked against the file so a renamed directory
	// cannot quietly keep answering to its old name.
	Name string `json:"name"`

	// Description says what world this is, in one sentence.
	Description string `json:"description"`

	// Roles are the fixture's addressable parts.
	Roles map[string]Role `json:"roles"`

	// PathRole names the role whose directory is prepended to PATH.
	// That is how the agent's `bash` reaches the fixture's tools
	// instead of the host's. Optional: a fixture with no tools of its
	// own leaves it empty.
	PathRole string `json:"path_role,omitempty"`

	// WorkdirRole names the role the agent runs in. Required.
	WorkdirRole string `json:"workdir_role"`

	// FactsFrom locates the fixture's facts, as "<role>#<key>". The
	// role's file is JSON, the key holds an object, and every value in
	// it must be a string.
	//
	// Facts live inside the world file rather than beside it on
	// purpose. The shim renders the world from that same file, so the
	// value a check asserts and the value the agent can observe are one
	// string, not two that agree today. A name asserted on both sides
	// of a test is untested.
	FactsFrom string `json:"facts_from"`

	// Witnesses map a name to a path, relative to the materialized
	// root, that the *world* writes during the run.
	//
	// Deliberately not pre-created. A witness that does not exist after
	// a run is a different fact from a witness that exists and is
	// empty — the first means nothing ever reached the world, and a
	// none_of check against it is vacuous rather than satisfied.
	Witnesses map[string]string `json:"witnesses"`

	// Env is handed to the agent process. Values may contain
	// ${role.<name>} and ${witness.<name>}, expanded to absolute paths
	// in the materialized root.
	Env map[string]string `json:"env,omitempty"`

	// Facts are loaded from FactsFrom. Not serialized: they are derived.
	Facts map[string]string `json:"-"`
	// contains filtered or unexported fields
}

A Fixture is the world a case runs against, and the only thing in the corpus that knows a path.

Roles are why. A case says "the answer must name ${fact.workload}"; it never says which file holds the workload or which directory the agent starts in. That indirection is not tidiness — it is what lets the probe rule mean something. Probes are confirmed present in the materialized world before the agent starts, so a check that fails on an absence is entitled to be called a violation rather than an environment nobody provisioned.

func LoadFixture

func LoadFixture(root, name string) (*Fixture, error)

LoadFixture reads <root>/<name>/fixture.json and its facts.

func (*Fixture) Materialize

func (f *Fixture) Materialize(dst string) (*World, error)

Materialize lays the shared overlay down, then the fixture on top, into dst, and confirms every role's probes.

type Report

type Report struct {
	Results []Result `json:"results"`
}

A Report is a set of results, and the thing a CI leg exits on.

func (Report) Verdict

func (rep Report) Verdict() Verdict

Verdict is the weakest verdict over the graded tiers: any indeterminate makes the whole report indeterminate, then any fail makes it a fail.

The no-access tier is deliberately not one of the graded tiers. Its contract is BaselineHolds, and by construction it fails and is partly vacuous — an agent with no tools cannot name the workload, and no witness is written because nothing reached the world. Folding that into the aggregate would make every report indeterminate forever, whatever the agent did, and an alarm that is always ringing is one nobody reads. Callers must consult BaselineHolds separately; the two answers together are the result, and evalrun reports the baseline first because a tools-tier pass on a case a naked model also passes is not evidence of anything.

type Result

type Result struct {
	CaseID        string        `json:"case_id"`
	Fixture       string        `json:"fixture"`
	PlantedDefect string        `json:"planted_defect"`
	Tier          Tier          `json:"tier"`
	Checks        []CheckResult `json:"checks"`

	// Preconditions are the case's assumptions about the process,
	// evaluated against its startup report. They never score. A failed
	// one makes the result indeterminate, because a case whose
	// assumptions did not hold did not test what it claims to test
	// (#1061).
	Preconditions []CheckResult `json:"preconditions,omitempty"`

	// Answer is the agent's output, kept so a failing run can be read
	// without re-running it. For a run the deadline stopped it is
	// partial: whatever the model had written by then.
	Answer string `json:"answer,omitempty"`

	// RunError is set when the agent process itself failed. The checks
	// still run — a crashed run whose checks all fail is a clearer
	// report than a bare exit code — but the verdict can never be a
	// pass.
	RunError string `json:"run_error,omitempty"`

	// TimedOut is set when RunError is the runner's own deadline rather
	// than the process failing. BaselineHolds reads it: a no-access run
	// that worked until the deadline is a measurement, a crash is not.
	TimedOut bool `json:"timed_out,omitempty"`
}

A Result is one case run at one tier.

func (Result) Score

func (r Result) Score() int

Score is the number of informative passes.

func (Result) UnmetPreconditions

func (r Result) UnmetPreconditions() []CheckResult

UnmetPreconditions returns the preconditions that did not hold — failed, or vacuous because the process printed no startup report at all.

Vacuous counts as unmet, and that is the whole point of the tier: the regression this exists to catch (skill discovery breaking, so the case's steer never loads) shows up as terms that are missing, and a process that never reported shows up as a source that is absent. Neither is a state in which the case tested anything.

func (Result) Vacuous

func (r Result) Vacuous() bool

Vacuous reports whether any check observed nothing.

func (Result) Verdict

func (r Result) Verdict() Verdict

Verdict folds the checks into one answer.

type Role

type Role struct {
	// Path is relative to the fixture root, and may be a file or a
	// directory.
	Path string `json:"path"`

	// Probes are paths relative to Path that must exist in the
	// materialized world before the agent starts. Empty means Path
	// itself is the probe.
	Probes []string `json:"probes,omitempty"`
}

A Role is one addressable part of a fixture's world.

type RunArtifacts

type RunArtifacts struct {
	// WorldRoot is the materialized fixture. Removed unless KeepWorld.
	WorldRoot string

	// Stdout is the agent's output; Stderr is kept separately so a
	// provider error does not land in the graded text.
	Stdout string
	Stderr string

	// ExitErr is the process's error, if any.
	ExitErr error
}

A RunArtifacts bundle is what one execution left behind.

type Runner

type Runner struct {
	// Binary is the core-agent executable under test.
	Binary string

	// Args are prepended to the agent's own arguments — this is where
	// the provider selection lives, because which provider the eval
	// runs against is the caller's business and not the corpus's.
	Args []string

	// Timeout bounds the agent process. Zero means DefaultTimeout.
	Timeout time.Duration

	// KeepWorld leaves the materialized world on disk and reports its
	// path, for reading a failure afterwards.
	KeepWorld bool

	// Log, when set, receives progress lines.
	Log func(format string, args ...any)
}

A Runner executes one case against one fixture.

func (*Runner) Run

func (r *Runner) Run(ctx context.Context, c *Case, f *Fixture, tier Tier) (Result, error)

Run executes one case at one tier and grades it.

The two tiers differ by exactly one flag. That is deliberate: if the baseline run differed from the real one in prompt, fixture or checks, a baseline of zero would prove something about the baseline's setup rather than about the check.

type Source

type Source struct {
	Text    string
	Present bool
}

A Source is the text a check reads, plus whether it existed at all.

func StartupSource

func StartupSource(stderr string) Source

StartupSource extracts the agent's own startup report from a run's stderr.

Present is false when stderr carried no such line at all, which is the case that matters most: a process that died before printing its summary establishes nothing, and a none_of-only precondition against the empty string would otherwise pass. Verify already marks an absent source vacuous, and a vacuous precondition is indeterminate.

type Tier

type Tier string

A Tier is which way a case was run.

const (
	// TierTools is the real run: the agent has its tools and the
	// fixture's world on PATH.
	TierTools Tier = "tools"

	// TierNoAccess is the grader's self-test. The same case, the same
	// prompt, the same checks, and no tools at all.
	//
	// Every objective check must score 0 here or the case does not
	// ship. A check a naked model satisfies is measuring vocabulary,
	// and it measures vocabulary no matter how good the agent is — so
	// this runs in CI beside the real tier rather than being audited
	// once when the case is written.
	TierNoAccess Tier = "no-access"
)

type Verdict

type Verdict string

A Verdict is an aggregate's answer.

const (
	VerdictPass Verdict = "pass"
	VerdictFail Verdict = "fail"

	// VerdictIndeterminate is the one that earns its keep.
	//
	// A run containing a check that carried no information about it
	// does not get to be green. The failure mode this prevents is
	// specific and quiet: a none_of check against a witness the world
	// never wrote passes, because nothing is absent from nothing, and
	// an aggregate that counts it reports a clean run in which nothing
	// was observed. That is exactly the shape of the credential-less
	// CI green #488 was filed for, one layer up.
	VerdictIndeterminate Verdict = "indeterminate"
)

type World

type World struct {
	Root    string
	Workdir string
	PathDir string
	Env     map[string]string
	// contains filtered or unexported fields
}

A World is one materialized copy of a fixture — the thing the agent actually runs against.

Materialized rather than used in place because a run mutates it: the shim appends to its witnesses, and an agent that decides to write something writes it here. Grading the committed tree would mean the first run contaminates every later one, and it would leave the checkout dirty, which is its own well-earned lesson.

func (*World) ReadWitness

func (w *World) ReadWitness(name string) (string, bool)

ReadWitness returns a witness's contents and whether it exists.

The bool is the interesting half. Absent means the world was never touched, which is a different finding from touched-and-empty, and verify.go spends that difference.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL