runner

package
v0.30.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 31, 2026 License: MIT Imports: 18 Imported by: 0

Documentation

Overview

Package runner — multi-case driver. RunMany executes a slice of cases in bounded parallel, applies the model override (which mutates each case's raw config bytes — not a parsed Config), and returns one combined Report.

The single-case Run remains the workhorse. RunMany is a thin coordinator: fan out, collect, fold into the Phase 1 report shape (no schema bump).

Package runner executes a single eval case. It is a thin wrapper around engine.Replay() and the journal projection: a fresh engine is constructed from the case's config bytes, the journal is loaded, the deterministic stash is seeded, the io.input events are re-fired, and observed events are collected via a side-channel subscription. Side-effecting plugins (LLM providers, tools) detect engine.Replay.Active() and pop stashed responses instead of calling out, so no API key is required.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func RenderTranscript

func RenderTranscript(events []evalcase.ObservedEvent) string

RenderTranscript renders a live session's full, observed event stream as legible plain text, suitable as protocol.Request.UserInput for an LLM judge alongside a rubric (E2-S3).

v1 is deliberately minimal, per the interview's own resolved decision: every event in events is rendered, in order, with its full payload — no event-type filtering, no payload truncation. Narrowing that (dropping high-frequency noise, trimming huge payloads) is explicitly deferred until real usage shows what's actually noise; do not add it here.

Each event renders as a numbered "[n] <RFC3339 timestamp> <type>" header line, followed by its payload as indented JSON (omitted when the payload is nil). This is a rendering of the ObservedEvent projection's own Type/Timestamp/Payload fields, not a raw dump of engine- or journal-internal bookkeeping (seq, event_id, parent_seq, etc. never reach ObservedEvent in the first place).

Types

type MultiOptions

type MultiOptions struct {
	// Parallelism caps concurrent case executions. <=0 picks a sensible
	// default (4, or NumCPU when smaller).
	Parallelism int
	// Tags, when non-empty, restricts the run to cases whose Meta.Tags
	// contains every entry. Empty means "all cases".
	Tags []string
	// ModelOverride, when non-empty, rewrites core.models.default in each
	// case's config bytes before constructing the engine. Surgical YAML
	// rewrite — does not touch other keys.
	ModelOverride string
	// PerCase is forwarded to every Run() call. Notably, SessionsRoot here
	// is shared — RunMany subdivides it into per-case sub-tempdirs to keep
	// concurrent runs isolated. Pass an empty SessionsRoot to inherit the
	// case-config default (production), or t.TempDir() in tests.
	PerCase Options
}

MultiOptions tunes RunMany.

type Options

type Options struct {
	// Logger, when non-nil, is used by the engine. Defaults to a discard
	// handler so test output stays clean.
	Logger *slog.Logger
	// BootTimeout caps engine boot.
	BootTimeout time.Duration
	// ReplayTimeout caps the replay run (per-turn timeout lives in the
	// journal coordinator; this is the overall ceiling).
	ReplayTimeout time.Duration
	// SessionsRoot, when non-empty, is the absolute path the engine should
	// use as core.sessions.root. Tests pass t.TempDir(); production passes
	// the empty string and accepts the case-config default.
	SessionsRoot string
	// ExtraPlugins registers additional plugin factories onto the engine's
	// registry, keyed by plugin ID, alongside the built-ins
	// allplugins.RegisterAll already provides. This is an embedder's seam
	// for booting cases that reference their own (non-nexus.*) plugins —
	// e.g. an eval harness embedded in another module, exercising its own
	// tools or agents.
	//
	// Registering an ID here only makes its factory available to the
	// engine's registry; it does not activate the plugin. The case's own
	// config (`plugins.active`) must still name the ID for the engine to
	// construct and Init it during Boot — exactly as for any built-in.
	//
	// An ID that collides with a built-in registered by
	// allplugins.RegisterAll is not guarded against: entries here are
	// registered after the built-ins, so PluginRegistry.Register's existing
	// silent-overwrite behavior means an ExtraPlugins entry always wins over
	// a same-ID built-in. This is deliberate, not an oversight — it lets an
	// embedder shadow a built-in for testing.
	//
	// A nil or empty map changes nothing about existing behavior.
	ExtraPlugins map[string]engine.PluginFactory
}

Options tune Run.

type Result

type Result struct {
	CaseID     string                     `json:"case_id"`
	StartedAt  time.Time                  `json:"started_at"`
	EndedAt    time.Time                  `json:"ended_at"`
	Pass       bool                       `json:"pass"`
	Assertions []evalcase.AssertionResult `json:"assertions"`
	// Counts is a per-event-type histogram for the observed (replayed)
	// stream. Useful for the Phase 2 baseline differ.
	Counts map[string]int `json:"counts,omitempty"`
	// JournalDir is the case's golden journal dir, echoed back for the
	// reporter's convenience.
	JournalDir string `json:"journal_dir,omitempty"`
	// Transcript is a rendered, legible plain-text rendering of the
	// session's full, unfiltered observed event stream (see
	// RenderTranscript). Populated for both Run and RunLive — they share
	// buildResult's tail — but it exists primarily for RunLive: a
	// downstream judge call feeds it as protocol.Request.UserInput
	// alongside a rubric (E2-S3).
	Transcript string `json:"transcript,omitempty"`
	// CustomScores is caller-populated — neither Run nor RunLive sets it.
	// An external caller (e.g. after computing a correctness check or a
	// judge verdict) sets it on the Result it already holds before
	// passing the Result to report.Aggregate, which copies it onto the
	// corresponding CaseEntry.
	CustomScores map[string]float64 `json:"custom_scores,omitempty"`
}

Result is everything Run produces about a single case execution.

func Run

func Run(ctx context.Context, c *evalcase.Case, opts Options) (*Result, error)

Run executes one case end-to-end and returns its Result.

The flow:

  1. Decode the case config (raw YAML stored on the case so the engine owns parsing).
  2. Optionally override core.sessions.root (tests use t.TempDir()).
  3. engine.NewFromBytes → allplugins.RegisterAll → Boot.
  4. Subscribe a wildcard collector for the observed stream.
  5. Drive replay via engine.Replay (a thin wrapper around journal.NewCoordinator).
  6. Stop the engine. Project the case's golden journal into a stream, evaluate assertions.

func RunLive

func RunLive(ctx context.Context, c *evalcase.Case, opts Options) (*Result, error)

RunLive executes one case's Inputs end-to-end against a real, live engine boot — no engine.Replay stash, real LLM calls (or, in tests, a case config whose own nexus.io.test plugin mocks the LLM response itself, the same pattern pkg/eval/protocol's own tests use — see plugins/io/test's mock_responses). It is a third execution path, structurally distinct from Run's replay: instead of replaying a journal stash, it overlays the case's config so a live nexus.io.test transport fires c.Inputs and signals natural completion, then waits for that completion via the same "wait to idle" mechanics protocol.Run uses (factored into pkg/eval/internal/livedrive.WaitForIdle so neither package reimplements it).

Otherwise RunLive mirrors Run closely: identical Options/Result shapes, identical ExtraPlugins registration, and the same Assertions.Deterministic evaluation against (live-observed, golden) — so EventSequenceDistance/ToolInvocationParity still catch structural drift between a live run and the golden journal even though the LLM call itself isn't replayed from a stash.

func RunMany

func RunMany(ctx context.Context, cases []*evalcase.Case, opts MultiOptions) []*Result

RunMany executes cases concurrently and returns one Result per case, in stable case-ID order. Failure of one case does not abort the others — every case runs, and a Result whose Run() returned a hard error (engine boot failure, journal load error) lands here with a synthesized failed "runner" assertion so callers can fold it into a report uniformly.

The aggregator (pkg/eval/report.Aggregate) is invoked by callers, not by this package — keeping report out of runner's import set avoids an import cycle (report depends on runner.Result).

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL