evalcase

package
v0.32.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 1, 2026 License: MIT Imports: 11 Imported by: 0

Documentation

Overview

Package evalcase loads eval cases from disk.

A case is a directory under tests/eval/cases/<id>/ containing:

case.yaml          # name, description, tags, owner, freshness_days, model_baseline
input/
  config.yaml      # config for the engine under test
  inputs.yaml      # scripted user inputs (drive nexus.io.test)
journal/           # full copy of source session journal (header + segments)
assertions.yaml    # mix of deterministic + semantic assertions

Cases are pure data — the runner consumes them. This package owns the schema and loader; pkg/eval/runner owns execution.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Assertion

type Assertion struct {
	Kind                  string
	EventEmitted          *EventEmittedSpec
	EventSequenceDistance *EventSequenceDistanceSpec
	ToolInvocationParity  *ToolInvocationParitySpec
	EventCountBounds      *EventCountBoundsSpec
	EventSequenceStrict   *EventSequenceStrictSpec
	TokenBudget           *TokenBudgetSpec
	Latency               *LatencySpec
	ResponseContains      *ResponseContainsSpec
	ContainsValue         *ContainsValueSpec
}

Assertion is the union type for every concrete deterministic assertion. Exactly one of the embedded *Spec pointers is non-nil.

func (Assertion) Evaluate

func (a Assertion) Evaluate(observed, golden []ObservedEvent) AssertionResult

Evaluate runs the assertion against observed and (where applicable) golden event streams. Golden may be nil for assertions that do not need it (token_budget, event_emitted, latency, event_count_bounds).

type AssertionResult

type AssertionResult struct {
	Kind        string         `json:"kind"`
	Pass        bool           `json:"pass"`
	Message     string         `json:"message,omitempty"`
	Diagnostics map[string]any `json:"diagnostics,omitempty"`
}

AssertionResult captures the outcome of evaluating one assertion.

type Assertions

type Assertions struct {
	Deterministic []Assertion      `yaml:"-"`
	Semantic      []map[string]any `yaml:"-"`
}

Assertions is the in-memory schema of assertions.yaml. All deterministic kinds live under `deterministic:`. Phase 1 has no semantic kinds yet — the `semantic:` block is parsed and stored as raw maps so a future phase can add LLM-judge support without a breaking migration.

func ParseAssertions

func ParseAssertions(data []byte) (Assertions, error)

ParseAssertions decodes assertions.yaml bytes into an Assertions value. Each deterministic entry must have a `kind` field; unknown kinds error out rather than silently degrade — typos in test bundles must be loud.

type Case

type Case struct {
	// Dir is the absolute path to the case directory.
	Dir string
	// ID is the case identifier (the directory's basename).
	ID string
	// Meta is the parsed case.yaml.
	Meta Meta
	// ConfigYAML is the raw bytes of input/config.yaml. The runner feeds
	// these to engine.NewFromBytes — keeping it raw lets the engine own
	// validation and tilde expansion.
	ConfigYAML []byte
	// Inputs are the scripted user inputs from input/inputs.yaml.
	Inputs []string
	// JournalDir is the absolute path to the journal directory.
	JournalDir string
	// Assertions is the parsed assertions.yaml.
	Assertions Assertions
}

Case is the in-memory representation of a single eval case.

func Load

func Load(dir string) (*Case, error)

Load opens a case directory and parses every artifact. Errors carry the path that failed so debugging a malformed bundle is straightforward.

type ContainsValueSpec

type ContainsValueSpec struct {
	EventType string  `yaml:"event_type,omitempty"`
	Value     float64 `yaml:"value"`
	Tolerance float64 `yaml:"tolerance,omitempty"`
	Unit      string  `yaml:"unit,omitempty"`
}

ContainsValueSpec asserts that a selected event's text content (default selection: same as ResponseContainsSpec — the final assistant response, via selectResponseText) contains at least one numeric token within Tolerance of Value.

Tolerance defaulting: when Tolerance is zero (unset — the struct's own zero value, so an explicit `tolerance: 0` in YAML is indistinguishable from omitting it and gets the same inferred default), it defaults to half the smallest decimal unit implied by how many digits were written after Value's decimal point in assertions.yaml — e.g. `value: 42.3` infers tolerance 0.05; `value: 100` infers tolerance 0.5. A plain Go float64 field cannot itself remember "how many digits were written" (42.3 and 42.30 are the same float64), and by the time ParseAssertions reaches this spec's decode step the entry has already round-tripped once through map[string]any (see rawAssertionsFile / ParseAssertions's entryBytes re-marshal) — so any "original YAML text" capture done here would already be reading a normalized, shortest-round-trip re-marshal, not the user's literal keystrokes. Given that, this implementation infers digit count directly from Value itself at evaluation time, via strconv.FormatFloat(Value, 'f', -1, 64) (Go's own shortest string that round-trips back to the identical float64) — which is provably the same string ParseAssertions's yaml.v3 re-marshal would have produced for this field, so capturing "raw text" earlier in the pipeline would infer an identical default. This is a deliberate simplification documented here rather than added machinery (a yaml.Node walk) that would only reproduce the same result through more code. Practical consequence: `value: 42.30` infers the same default tolerance as `value: 42.3` (both are the float64 42.3) — set Tolerance explicitly when that distinction matters.

Unit (e.g. "%", "$") is stripped from the selected text, along with thousands-separator commas, before numeric tokens are extracted — see extractNumericTokens.

Extraction is word-boundary-aware: a numeric token immediately preceded or followed by another digit is rejected, so `value: 4.2` never matches text containing "24.2" (see extractNumericTokens).

A candidate passes if its absolute difference from Value is <= Tolerance — the tolerance boundary is INCLUSIVE (a value exactly Tolerance away passes).

type CountRange

type CountRange struct {
	Min int `yaml:"min,omitempty"`
	Max int `yaml:"max,omitempty"`
}

CountRange is a closed range. A zero Min is "no minimum"; a zero Max is "no maximum" (matches plan.md semantics — count: { min: 1 } with no max reads as "at least 1").

type EventCountBoundsSpec

type EventCountBoundsSpec struct {
	Bounds map[string]CountRange `yaml:"bounds"`
}

EventCountBoundsSpec asserts a per-event-type count range over the observed stream.

type EventEmittedSpec

type EventEmittedSpec struct {
	Type  string         `yaml:"type"`
	Where map[string]any `yaml:"where,omitempty"`
	Count *CountRange    `yaml:"count,omitempty"`
}

EventEmittedSpec passes when at least Count.Min and at most Count.Max envelopes match Type and (optionally) Where.

type EventSequenceDistanceSpec

type EventSequenceDistanceSpec struct {
	Threshold float64 `yaml:"threshold"`
	// Filter, when non-empty, restricts the streams to envelopes whose Type
	// is in the set. Lets a case ignore high-frequency noise (core.tick).
	Filter []string `yaml:"filter,omitempty"`
}

EventSequenceDistanceSpec asserts that the Levenshtein ratio between the observed event-type stream and the journal's event-type stream is at most Threshold (0.0–1.0). 0.0 = identical, 1.0 = totally different.

type EventSequenceStrictSpec

type EventSequenceStrictSpec struct {
	Pattern []string `yaml:"pattern"`
	Filter  []string `yaml:"filter,omitempty"`
}

EventSequenceStrictSpec asserts the exact (filtered) event-type sequence equals Pattern.

type LatencySpec

type LatencySpec struct {
	P50Ms int `yaml:"p50_ms,omitempty"`
	P95Ms int `yaml:"p95_ms,omitempty"`
}

LatencySpec caps p50/p95 turn latency, computed from event timestamp deltas (NOT wall-clock). A turn = agent.turn.start..agent.turn.end. Values in ms.

type Meta

type Meta struct {
	Name          string    `yaml:"name"`
	Description   string    `yaml:"description"`
	Tags          []string  `yaml:"tags"`
	Owner         string    `yaml:"owner"`
	FreshnessDays int       `yaml:"freshness_days"`
	ModelBaseline string    `yaml:"model_baseline"`
	RecordedAt    time.Time `yaml:"recorded_at,omitempty"`
}

Meta is case.yaml.

type ObservedEvent

type ObservedEvent struct {
	Type      string
	Timestamp time.Time
	Payload   any
}

ObservedEvent is the minimal projection of a bus event the assertion engine needs. The runner converts engine.Event[any] to ObservedEvent so the case package has no engine dependency.

type ResponseContainsSpec

type ResponseContainsSpec struct {
	// EventType selects which event's Content field to search. Empty means
	// "the final assistant response" — the last llm.response event that has
	// no pending tool calls and non-empty Content (mirrors
	// protocol.isFinalAssistant's "real final turn" definition).
	EventType   string   `yaml:"event_type,omitempty"`
	Contains    []string `yaml:"contains,omitempty"`
	ContainsAny []string `yaml:"contains_any,omitempty"`
}

ResponseContainsSpec asserts that the text content of a selected event (by default, the final assistant response) contains the required substring(s). Matching is case-insensitive.

type TokenBudgetSpec

type TokenBudgetSpec struct {
	MaxInputTokens  int  `yaml:"max_input_tokens,omitempty"`
	MaxOutputTokens int  `yaml:"max_output_tokens,omitempty"`
	PerTurn         bool `yaml:"per_turn,omitempty"`
}

TokenBudgetSpec caps tokens read off llm.response events. When PerTurn is set, each turn must stay below the limits. Unset (zero) limits are unbounded.

type ToolInvocationParitySpec

type ToolInvocationParitySpec struct {
	CountTolerance int  `yaml:"count_tolerance,omitempty"`
	ArgKeys        bool `yaml:"arg_keys,omitempty"`
}

ToolInvocationParitySpec asserts per-tool invocation count parity (within CountTolerance) and, when ArgKeys is true, that the set of argument keys observed matches the journal's set per tool. Arg-value parity is intentionally not checked — values vary across model upgrades.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL