Documentation
¶
Overview ¶
Package eval provides the PromptZero golden evaluation harness (roadmap P2-25). Scenarios exercise the top-level agent flows — handoff round-trip, snapshot / rewind, ATT&CK constraint, detector verdicts, prompt-injection quarantine — against mock Flipper + Marauder transports. A scenario pass means the agent glue still wires up end-to-end; a fail signals a regression that CI might not otherwise catch at the unit-test layer.
The harness is deliberately plain Go: each scenario is a func that returns an error (or nil). Runner collects timings + tool-call counts + error state per scenario. Cost metrics are computed from `cost.Tracker` when the scenario installs one.
Scenarios are also discoverable as Go tests (see `scenario_tests.go`) so they run under `go test`, `task test`, and CI without a custom binary — the Runner type just gives us a structured aggregate when the operator wants a summary.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func Summarise ¶
Summarise renders a compact, human-readable report from a Result slice. Two-column table with pass/fail marker, name, duration, and error (if any). Grouped by tag when the Runner supplied tags.
The format is stable — external tooling (the `task eval` target) can grep on "PASS:" / "FAIL:" prefixes.
Types ¶
type Result ¶
type Result struct {
Name string
Pass bool
Err error
Duration time.Duration
// ToolCalls is optional; scenarios that thread a counter through
// the mock agent can populate it for a richer summary. Scenarios
// that don't care leave it zero.
ToolCalls int
}
Result captures the outcome of one scenario pass.
type Runner ¶
type Runner struct {
// contains filtered or unexported fields
}
Runner holds a registered set of scenarios and executes them in deterministic order. Safe to reuse across calls.
func (*Runner) Register ¶
Register adds a scenario. Duplicate names are allowed — the Runner reports them verbatim — but downstream consumers should prefer unique names for clean summaries.
func (*Runner) RegisterAll ¶
RegisterAll is a convenience for batch registration.
func (*Runner) Run ¶
Run executes every registered scenario sequentially. Each scenario is independent: a panic in one does not abort the rest (the Runner recovers and reports the panic as an error).
func (*Runner) RunMatching ¶
RunMatching filters scenarios by tag and runs the subset. Empty tag list matches everything. Tags use OR semantics — a scenario runs if ANY of its tags matches ANY of the filter tags.
type Scenario ¶
type Scenario struct {
Name string
Description string
// Tags group scenarios so a later filter ("only RF scenarios")
// can subset a run. Examples: "handoff", "snapshot", "attack",
// "detector", "quarantine".
Tags []string
Run func() error
}
Scenario is a named integration-level check that exercises one or more agent features end-to-end. Run returns nil on success; any non-nil error is surfaced as a scenario fail. Description is optional operator-facing prose.
func Default ¶
Default returns the canonical scenario suite used by CI and `task eval`. Covers one end-to-end flow per critical agent layer so a PR that regresses any of them trips the gate loudly.
Adding a scenario: append a new function below (keep them pure — no network, no live Anthropic API) and wire it into this constructor.
Source Files