Documentation
¶
Overview ¶
Package runner — multi-case driver. RunMany executes a slice of cases in bounded parallel, applies the model override (which mutates each case's raw config bytes — not a parsed Config), and returns one combined Report.
The single-case Run remains the workhorse. RunMany is a thin coordinator: fan out, collect, fold into the Phase 1 report shape (no schema bump).
Package runner executes a single eval case. It is a thin wrapper around engine.Replay() and the journal projection: a fresh engine is constructed from the case's config bytes, the journal is loaded, the deterministic stash is seeded, the io.input events are re-fired, and observed events are collected via a side-channel subscription. Side-effecting plugins (LLM providers, tools) detect engine.Replay.Active() and pop stashed responses instead of calling out, so no API key is required.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func RenderTranscript ¶
func RenderTranscript(events []evalcase.ObservedEvent) string
RenderTranscript renders a live session's full, observed event stream as legible plain text, suitable as protocol.Request.UserInput for an LLM judge alongside a rubric (E2-S3).
v1 is deliberately minimal, per the interview's own resolved decision: every event in events is rendered, in order, with its full payload — no event-type filtering, no payload truncation. Narrowing that (dropping high-frequency noise, trimming huge payloads) is explicitly deferred until real usage shows what's actually noise; do not add it here.
Each event renders as a numbered "[n] <RFC3339 timestamp> <type>" header line, followed by its payload as indented JSON (omitted when the payload is nil). This is a rendering of the ObservedEvent projection's own Type/Timestamp/Payload fields, not a raw dump of engine- or journal-internal bookkeeping (seq, event_id, parent_seq, etc. never reach ObservedEvent in the first place).
Types ¶
type MultiOptions ¶
type MultiOptions struct {
// Parallelism caps concurrent case executions. <=0 picks a sensible
// default (4, or NumCPU when smaller).
Parallelism int
// Tags, when non-empty, restricts the run to cases whose Meta.Tags
// contains every entry. Empty means "all cases".
Tags []string
// ModelOverride, when non-empty, rewrites core.models.default in each
// case's config bytes before constructing the engine. Surgical YAML
// rewrite — does not touch other keys.
ModelOverride string
// PerCase is forwarded to every Run() call. Notably, SessionsRoot here
// is shared — RunMany subdivides it into per-case sub-tempdirs to keep
// concurrent runs isolated. Pass an empty SessionsRoot to inherit the
// case-config default (production), or t.TempDir() in tests.
PerCase Options
}
MultiOptions tunes RunMany.
type Options ¶
type Options struct {
// Logger, when non-nil, is used by the engine. Defaults to a discard
// handler so test output stays clean.
Logger *slog.Logger
// BootTimeout caps engine boot.
BootTimeout time.Duration
// ReplayTimeout caps the replay run (per-turn timeout lives in the
// journal coordinator; this is the overall ceiling).
ReplayTimeout time.Duration
// SessionsRoot, when non-empty, is the absolute path the engine should
// use as core.sessions.root. Tests pass t.TempDir(); production passes
// the empty string and accepts the case-config default.
SessionsRoot string
// ExtraPlugins registers additional plugin factories onto the engine's
// registry, keyed by plugin ID, alongside the built-ins
// allplugins.RegisterAll already provides. This is an embedder's seam
// for booting cases that reference their own (non-nexus.*) plugins —
// e.g. an eval harness embedded in another module, exercising its own
// tools or agents.
//
// Registering an ID here only makes its factory available to the
// engine's registry; it does not activate the plugin. The case's own
// config (`plugins.active`) must still name the ID for the engine to
// construct and Init it during Boot — exactly as for any built-in.
//
// An ID that collides with a built-in registered by
// allplugins.RegisterAll is not guarded against: entries here are
// registered after the built-ins, so PluginRegistry.Register's existing
// silent-overwrite behavior means an ExtraPlugins entry always wins over
// a same-ID built-in. This is deliberate, not an oversight — it lets an
// embedder shadow a built-in for testing.
//
// A nil or empty map changes nothing about existing behavior.
ExtraPlugins map[string]engine.PluginFactory
}
Options tune Run.
type Result ¶
type Result struct {
CaseID string `json:"case_id"`
StartedAt time.Time `json:"started_at"`
EndedAt time.Time `json:"ended_at"`
Pass bool `json:"pass"`
Assertions []evalcase.AssertionResult `json:"assertions"`
// Counts is a per-event-type histogram for the observed (replayed)
// stream. Useful for the Phase 2 baseline differ.
Counts map[string]int `json:"counts,omitempty"`
// JournalDir is the case's golden journal dir, echoed back for the
// reporter's convenience.
JournalDir string `json:"journal_dir,omitempty"`
// Transcript is a rendered, legible plain-text rendering of the
// session's full, unfiltered observed event stream (see
// RenderTranscript). Populated for both Run and RunLive — they share
// buildResult's tail — but it exists primarily for RunLive: a
// downstream judge call feeds it as protocol.Request.UserInput
// alongside a rubric (E2-S3).
Transcript string `json:"transcript,omitempty"`
// CustomScores is caller-populated — neither Run nor RunLive sets it.
// An external caller (e.g. after computing a correctness check or a
// judge verdict) sets it on the Result it already holds before
// passing the Result to report.Aggregate, which copies it onto the
// corresponding CaseEntry.
CustomScores map[string]float64 `json:"custom_scores,omitempty"`
}
Result is everything Run produces about a single case execution.
func Run ¶
Run executes one case end-to-end and returns its Result.
The flow:
- Decode the case config (raw YAML stored on the case so the engine owns parsing).
- Optionally override core.sessions.root (tests use t.TempDir()).
- engine.NewFromBytes → allplugins.RegisterAll → Boot.
- Subscribe a wildcard collector for the observed stream.
- Drive replay via engine.Replay (a thin wrapper around journal.NewCoordinator).
- Stop the engine. Project the case's golden journal into a stream, evaluate assertions.
func RunLive ¶
RunLive executes one case's Inputs end-to-end against a real, live engine boot — no engine.Replay stash, real LLM calls (or, in tests, a case config whose own nexus.io.test plugin mocks the LLM response itself, the same pattern pkg/eval/protocol's own tests use — see plugins/io/test's mock_responses). It is a third execution path, structurally distinct from Run's replay: instead of replaying a journal stash, it overlays the case's config so a live nexus.io.test transport fires c.Inputs and signals natural completion, then waits for that completion via the same "wait to idle" mechanics protocol.Run uses (factored into pkg/eval/internal/livedrive.WaitForIdle so neither package reimplements it).
Otherwise RunLive mirrors Run closely: identical Options/Result shapes, identical ExtraPlugins registration, and the same Assertions.Deterministic evaluation against (live-observed, golden) — so EventSequenceDistance/ToolInvocationParity still catch structural drift between a live run and the golden journal even though the LLM call itself isn't replayed from a stash.
func RunMany ¶
RunMany executes cases concurrently and returns one Result per case, in stable case-ID order. Failure of one case does not abort the others — every case runs, and a Result whose Run() returned a hard error (engine boot failure, journal load error) lands here with a synthesized failed "runner" assertion so callers can fold it into a report uniformly.
The aggregator (pkg/eval/report.Aggregate) is invoked by callers, not by this package — keeping report out of runner's import set avoids an import cycle (report depends on runner.Result).