Documentation
¶
Overview ¶
Package eval implements the agent eval harness (design 009): "unit tests for agents." A fixture describes a task and structural assertions; a Result is what running the agent produced; Evaluate checks the Result against the assertions. The format is open and in-repo.
This package is the fixture format + the pure, deterministic assertion engine (no token spend). Live execution that PRODUCES a Result by running the agent (`rysh run`) is the integration half; the structural mode here runs against a recorded/produced Result artifact.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
Types ¶
type Case ¶
type Case struct {
Name string
Prompt string
Until string
Expect Expect
// Judge is the optional LLM-judge rubric (evals/<case>/judge.md, design
// 009 §3.2). Empty when the case has no judge.md: assertions alone grade
// the case. When present, the judge runs AFTER assertions and both must
// pass — the judge is a signal, structural assertions stay the hard gate.
Judge string
}
Case is a loaded fixture.
type Expect ¶
type Expect struct {
// FilesChanged: every changed file must match at least one of these globs
// (scope containment — catches out-of-scope edits). Empty ⇒ not checked.
FilesChanged []string `yaml:"files_changed"`
// CommandsAllowed: if non-empty, every executed command must match one.
CommandsAllowed []string `yaml:"commands_allowed"`
// CommandsForbidden: no executed command may match any of these.
CommandsForbidden []string `yaml:"commands_forbidden"`
// OutputMatches: every regex must match somewhere in the output.
OutputMatches []string `yaml:"output_matches"`
// MaxTokens: tokens_used must be ≤ this (0 ⇒ not checked).
MaxTokens int `yaml:"max_tokens"`
}
Expect is the assertion set for a case (evals/<case>/expect.yaml).
Click to show internal directories.
Click to hide internal directories.