Documentation
¶
Overview ¶
Package evalcase loads eval cases from disk.
A case is a directory under tests/eval/cases/<id>/ containing:
case.yaml # name, description, tags, owner, freshness_days, model_baseline input/ config.yaml # config for the engine under test inputs.yaml # scripted user inputs (drive nexus.io.test) journal/ # full copy of source session journal (header + segments) assertions.yaml # mix of deterministic + semantic assertions
Cases are pure data — the runner consumes them. This package owns the schema and loader; pkg/eval/runner owns execution.
Index ¶
- type Assertion
- type AssertionResult
- type Assertions
- type Case
- type ContainsValueSpec
- type CountRange
- type EventCountBoundsSpec
- type EventEmittedSpec
- type EventSequenceDistanceSpec
- type EventSequenceStrictSpec
- type LatencySpec
- type Meta
- type ObservedEvent
- type ResponseContainsSpec
- type TokenBudgetSpec
- type ToolInvocationParitySpec
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Assertion ¶
type Assertion struct {
Kind string
EventEmitted *EventEmittedSpec
EventSequenceDistance *EventSequenceDistanceSpec
ToolInvocationParity *ToolInvocationParitySpec
EventCountBounds *EventCountBoundsSpec
EventSequenceStrict *EventSequenceStrictSpec
TokenBudget *TokenBudgetSpec
Latency *LatencySpec
ResponseContains *ResponseContainsSpec
ContainsValue *ContainsValueSpec
}
Assertion is the union type for every concrete deterministic assertion. Exactly one of the embedded *Spec pointers is non-nil.
func (Assertion) Evaluate ¶
func (a Assertion) Evaluate(observed, golden []ObservedEvent) AssertionResult
Evaluate runs the assertion against observed and (where applicable) golden event streams. Golden may be nil for assertions that do not need it (token_budget, event_emitted, latency, event_count_bounds).
type AssertionResult ¶
type AssertionResult struct {
Kind string `json:"kind"`
Pass bool `json:"pass"`
Message string `json:"message,omitempty"`
Diagnostics map[string]any `json:"diagnostics,omitempty"`
}
AssertionResult captures the outcome of evaluating one assertion.
type Assertions ¶
type Assertions struct {
Deterministic []Assertion `yaml:"-"`
Semantic []map[string]any `yaml:"-"`
}
Assertions is the in-memory schema of assertions.yaml. All deterministic kinds live under `deterministic:`. Phase 1 has no semantic kinds yet — the `semantic:` block is parsed and stored as raw maps so a future phase can add LLM-judge support without a breaking migration.
func ParseAssertions ¶
func ParseAssertions(data []byte) (Assertions, error)
ParseAssertions decodes assertions.yaml bytes into an Assertions value. Each deterministic entry must have a `kind` field; unknown kinds error out rather than silently degrade — typos in test bundles must be loud.
type Case ¶
type Case struct {
// Dir is the absolute path to the case directory.
Dir string
// ID is the case identifier (the directory's basename).
ID string
// Meta is the parsed case.yaml.
Meta Meta
// ConfigYAML is the raw bytes of input/config.yaml. The runner feeds
// these to engine.NewFromBytes — keeping it raw lets the engine own
// validation and tilde expansion.
ConfigYAML []byte
// Inputs are the scripted user inputs from input/inputs.yaml.
Inputs []string
// JournalDir is the absolute path to the journal directory.
JournalDir string
// Assertions is the parsed assertions.yaml.
Assertions Assertions
}
Case is the in-memory representation of a single eval case.
type ContainsValueSpec ¶
type ContainsValueSpec struct {
EventType string `yaml:"event_type,omitempty"`
Value float64 `yaml:"value"`
Tolerance float64 `yaml:"tolerance,omitempty"`
Unit string `yaml:"unit,omitempty"`
}
ContainsValueSpec asserts that a selected event's text content (default selection: same as ResponseContainsSpec — the final assistant response, via selectResponseText) contains at least one numeric token within Tolerance of Value.
Tolerance defaulting: when Tolerance is zero (unset — the struct's own zero value, so an explicit `tolerance: 0` in YAML is indistinguishable from omitting it and gets the same inferred default), it defaults to half the smallest decimal unit implied by how many digits were written after Value's decimal point in assertions.yaml — e.g. `value: 42.3` infers tolerance 0.05; `value: 100` infers tolerance 0.5. A plain Go float64 field cannot itself remember "how many digits were written" (42.3 and 42.30 are the same float64), and by the time ParseAssertions reaches this spec's decode step the entry has already round-tripped once through map[string]any (see rawAssertionsFile / ParseAssertions's entryBytes re-marshal) — so any "original YAML text" capture done here would already be reading a normalized, shortest-round-trip re-marshal, not the user's literal keystrokes. Given that, this implementation infers digit count directly from Value itself at evaluation time, via strconv.FormatFloat(Value, 'f', -1, 64) (Go's own shortest string that round-trips back to the identical float64) — which is provably the same string ParseAssertions's yaml.v3 re-marshal would have produced for this field, so capturing "raw text" earlier in the pipeline would infer an identical default. This is a deliberate simplification documented here rather than added machinery (a yaml.Node walk) that would only reproduce the same result through more code. Practical consequence: `value: 42.30` infers the same default tolerance as `value: 42.3` (both are the float64 42.3) — set Tolerance explicitly when that distinction matters.
Unit (e.g. "%", "$") is stripped from the selected text, along with thousands-separator commas, before numeric tokens are extracted — see extractNumericTokens.
Extraction is word-boundary-aware: a numeric token immediately preceded or followed by another digit is rejected, so `value: 4.2` never matches text containing "24.2" (see extractNumericTokens).
A candidate passes if its absolute difference from Value is <= Tolerance — the tolerance boundary is INCLUSIVE (a value exactly Tolerance away passes).
type CountRange ¶
CountRange is a closed range. A zero Min is "no minimum"; a zero Max is "no maximum" (matches plan.md semantics — count: { min: 1 } with no max reads as "at least 1").
type EventCountBoundsSpec ¶
type EventCountBoundsSpec struct {
Bounds map[string]CountRange `yaml:"bounds"`
}
EventCountBoundsSpec asserts a per-event-type count range over the observed stream.
type EventEmittedSpec ¶
type EventEmittedSpec struct {
Type string `yaml:"type"`
Where map[string]any `yaml:"where,omitempty"`
Count *CountRange `yaml:"count,omitempty"`
}
EventEmittedSpec passes when at least Count.Min and at most Count.Max envelopes match Type and (optionally) Where.
type EventSequenceDistanceSpec ¶
type EventSequenceDistanceSpec struct {
Threshold float64 `yaml:"threshold"`
// Filter, when non-empty, restricts the streams to envelopes whose Type
// is in the set. Lets a case ignore high-frequency noise (core.tick).
Filter []string `yaml:"filter,omitempty"`
}
EventSequenceDistanceSpec asserts that the Levenshtein ratio between the observed event-type stream and the journal's event-type stream is at most Threshold (0.0–1.0). 0.0 = identical, 1.0 = totally different.
type EventSequenceStrictSpec ¶
type EventSequenceStrictSpec struct {
Pattern []string `yaml:"pattern"`
Filter []string `yaml:"filter,omitempty"`
}
EventSequenceStrictSpec asserts the exact (filtered) event-type sequence equals Pattern.
type LatencySpec ¶
LatencySpec caps p50/p95 turn latency, computed from event timestamp deltas (NOT wall-clock). A turn = agent.turn.start..agent.turn.end. Values in ms.
type Meta ¶
type Meta struct {
Name string `yaml:"name"`
Description string `yaml:"description"`
Tags []string `yaml:"tags"`
Owner string `yaml:"owner"`
FreshnessDays int `yaml:"freshness_days"`
ModelBaseline string `yaml:"model_baseline"`
RecordedAt time.Time `yaml:"recorded_at,omitempty"`
}
Meta is case.yaml.
type ObservedEvent ¶
ObservedEvent is the minimal projection of a bus event the assertion engine needs. The runner converts engine.Event[any] to ObservedEvent so the case package has no engine dependency.
type ResponseContainsSpec ¶
type ResponseContainsSpec struct {
// EventType selects which event's Content field to search. Empty means
// "the final assistant response" — the last llm.response event that has
// no pending tool calls and non-empty Content (mirrors
// protocol.isFinalAssistant's "real final turn" definition).
EventType string `yaml:"event_type,omitempty"`
Contains []string `yaml:"contains,omitempty"`
ContainsAny []string `yaml:"contains_any,omitempty"`
}
ResponseContainsSpec asserts that the text content of a selected event (by default, the final assistant response) contains the required substring(s). Matching is case-insensitive.
type TokenBudgetSpec ¶
type TokenBudgetSpec struct {
MaxInputTokens int `yaml:"max_input_tokens,omitempty"`
MaxOutputTokens int `yaml:"max_output_tokens,omitempty"`
PerTurn bool `yaml:"per_turn,omitempty"`
}
TokenBudgetSpec caps tokens read off llm.response events. When PerTurn is set, each turn must stay below the limits. Unset (zero) limits are unbounded.
type ToolInvocationParitySpec ¶
type ToolInvocationParitySpec struct {
CountTolerance int `yaml:"count_tolerance,omitempty"`
ArgKeys bool `yaml:"arg_keys,omitempty"`
}
ToolInvocationParitySpec asserts per-tool invocation count parity (within CountTolerance) and, when ArgKeys is true, that the set of argument keys observed matches the journal's set per tool. Arg-value parity is intentionally not checked — values vary across model upgrades.