runner

package
v0.15.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 16, 2026 License: Apache-2.0 Imports: 24 Imported by: 0

Documentation

Overview

Package runner plans and executes an evaluation: which sessions to run, in what environment, with what resumed from a previous attempt.

Planning is separated from execution and has no I/O, so the session budgets are arithmetic a test can check rather than a cost discovered afterwards.

Index

Constants

View Source
const (
	CondSkill    store.Condition = "skill"
	CondBaseline store.Condition = "baseline"
	CondTrigger  store.Condition = "trigger"
)

Conditions a session runs under. The skill/baseline pair is what the published delta compares; a trigger probe is a short session measuring only whether the skill fired.

Variables

View Source
var DefaultAllowDomains = []string{"api.anthropic.com", "statsig.anthropic.com", "sentry.io"}
View Source
var ErrEventsNotWritten = errors.New("runner: events.jsonl was not written")

ErrEventsNotWritten is separable with errors.Is because events are the one artifact scoring and resume read back: losing the transcript costs secondary evidence, losing events.jsonl means the run cannot be scored at all.

Functions

func BaselinePlanned added in v0.11.0

func BaselinePlanned(p Plan) bool

BaselinePlanned reports whether the plan runs any baseline session, which is what makes the without-skill delta measurable.

func EvalID added in v0.11.0

func EvalID(id int) string

EvalID renders an eval's id as the run key's task id. Run keys are stored as text and predate evals.json, so the conversion happens in one place.

func LoadEvents

func LoadEvents(path string) ([]agent.Event, error)

LoadEvents reads a trajectory written by WriteArtifacts.

Types

type Artifacts

type Artifacts struct {
	Dir            string
	TranscriptPath string
	EventsPath     string
	GradingPath    string
	OutputsDir     string
}

Artifacts locates the files WriteArtifacts produced for one run.

func WriteArtifacts

func WriteArtifacts(dir string, raw []byte, events []agent.Event, g Grading, workspace string, skipDirs []string) (Artifacts, error)

WriteArtifacts records one run's evidence trail into dir: transcript.raw, events.jsonl, grading.json and outputs/.

skipDirs keeps the installed bundle out of outputs/ — it is already stored once as the published bundle, and sixty copies of it would be most of the disk an evaluation uses. A baseline installs no skill and passes none.

Best-effort across all four: every failure is attempted and joined, so a caller denied its events still gets the transcript that did write.

type ExecuteInput

type ExecuteInput struct {
	Skill       string
	BundleDir   string
	SuiteDir    string
	Image       sandbox.ImageRef
	Plan        *Plan
	Healthy     map[Member]bool
	Distractors []suite.Distractor
}

ExecuteInput is everything Execute needs beyond the eval id.

type ExecuteResult

type ExecuteResult struct {
	Outcomes      []Outcome
	Probes        []ProbeOutcome
	PanelComplete bool
}

ExecuteResult is everything Execute produced.

type Grading

type Grading struct {
	Key        store.RunKey
	Meta       agent.Meta
	Status     string
	Error      string
	StartedAt  time.Time
	FinishedAt time.Time
}

Grading is the record of one run: what identifies it, the session metadata, and its terminal status. It is what makes a surprising score checkable.

func LoadGrading

func LoadGrading(path string) (*Grading, error)

LoadGrading reads a grading.json written by WriteArtifacts.

type Health

type Health struct {
	Member Member
	OK     bool
	Detail string
}

Health is one panel member's health-probe result.

type Member

type Member struct {
	Agent string
	Model string
	Class spec.ModelTier
}

Member is one entry in the model panel.

type Options

type Options struct {
	Store    *store.Store
	Driver   sandbox.Driver
	Adapters func(name string) (agent.Adapter, bool)

	Concurrency    int
	SessionTimeout time.Duration
	Untrusted      bool
	AllowDomains   []string

	// PanelExcludeEnv withholds credential variables from the sandbox. Comes
	// from provider.Config.PanelExcludeEnv — this package cannot import
	// provider (circular), so it receives the names as data.
	PanelExcludeEnv []string

	// WorkspaceRoot overrides os.TempDir() for a containerized runner whose
	// /tmp is invisible to the host daemon. Must be bind-mounted at the same
	// path on both sides or Docker silently mounts an empty directory.
	WorkspaceRoot string

	Sleep  func(time.Duration)
	Logger func(string, ...any)

	// OnComplete, when set, is called once per finished session, from that
	// session's own goroutine. It lets a caller grade a session while the
	// rest still run: a grade needs that session's artifacts and nothing
	// else. It is never called for a probe.
	OnComplete func(Outcome)

	MaxRateLimitRetries int
}

Options configures a Runner.

type Outcome

type Outcome struct {
	Key         store.RunKey
	Events      []agent.Event
	Meta        agent.Meta
	ArtifactDir string
	Status      string
	Err         error
	// MetaPartial is true when Meta was rebuilt from store columns rather than
	// the grading.json artifact. Model, NumTurns, VisibleSkills,
	// PermissionDenials, and IsError are zero in that case.
	MetaPartial       bool
	MetaPartialReason string
}

Outcome is what one planned session (skill or baseline) reported.

type Panel

type Panel []Member

Panel is the set of members every eval is run against.

func DeepPanel added in v0.11.0

func DeepPanel() Panel

DeepPanel adds the floor member.

func DefaultPanel

func DefaultPanel() Panel

DefaultPanel is the shipped panel: one member.

Sonnet rather than Opus, because it is what most people running claude-code actually run — so it is a truer measurement, not merely a cheaper one. The floor member joins only at TierDeep: "works on the weakest model" is worth measuring, and it is not worth doubling every score's cost to measure.

func PanelFor added in v0.11.0

func PanelFor(t Tier) Panel

PanelFor returns the shipped panel for a tier.

func ParsePanel

func ParsePanel(agents, models []string) (Panel, error)

ParsePanel builds a panel from caller arguments, cross-checking each agent against the adapter lookup so a typo is a refusal rather than a panel member that silently never runs. The first model is the strong member and any others are floor members.

Models alone are enough. The UI and the re-run endpoint offer a model without an agent — there is one adapter to choose from — and requiring both meant a chosen model fell back to the shipped panel with no error anywhere, producing a score against a model nobody asked for.

type Plan

type Plan struct {
	Tier   Tier
	Panel  Panel
	Runs   []store.RunKey
	Probes []Probe
	Evals  []suite.Eval
}

Plan is the complete list of work an evaluation performs.

func BuildPlan

func BuildPlan(t Tier, p Panel, set *suite.EvalSet, void map[int]bool, triggers []suite.TriggerQuery) (*Plan, error)

BuildPlan enumerates the sessions for a tier. void lists evals excluded by suite.Validate, which are planned by nobody and scored by nobody.

func (*Plan) EvalByID added in v0.11.0

func (p *Plan) EvalByID(taskID string) (suite.Eval, bool)

EvalByID returns the planned eval with the given run-key task id.

type Probe

type Probe struct {
	Index    int
	Prompt   string
	Positive bool
	Member   Member
}

Probe is one trigger measurement: a short session that answers only whether the skill fired.

type ProbeOutcome

type ProbeOutcome struct {
	Probe  Probe
	Events []agent.Event
	Meta   agent.Meta
	Caps   agent.Caps
	// Unknown is true when the probe could not be observed at all — the
	// session failed outright, or (on resume) its events could not be
	// recovered. An unknown probe is excluded from trigger-precision
	// denominators rather than counted as a miss.
	Unknown bool
	Reason  string
	Err     error
	// MetaPartial mirrors Outcome.MetaPartial: true when Meta came from the
	// store's own columns on resume rather than the probe's grading.json.
	MetaPartial       bool
	MetaPartialReason string
}

ProbeOutcome is what one trigger probe observed. No pre-computed verdict: whether the skill fired is scoring's job (score.DetectFiring).

type Runner

type Runner struct {
	// contains filtered or unexported fields
}

Runner executes a plan.

func New

func New(o Options) (*Runner, error)

New validates options and applies defaults.

func (*Runner) Execute

func (r *Runner) Execute(ctx context.Context, evalID int64, in ExecuteInput) (*ExecuteResult, error)

Execute runs in.Plan, skipping sessions already recorded.

func (*Runner) ProbePanel

func (r *Runner) ProbePanel(ctx context.Context, p Panel, image sandbox.ImageRef) ([]Health, error)

ProbePanel runs one trivial session against every panel member and reports whether each came back healthy. A member is healthy only when both the invocation and the parse succeed: an adapter that invokes and cannot parse produces a trajectory of nothing, which would otherwise score as a session in which the agent did nothing.

func (*Runner) SetOnComplete added in v0.12.0

func (r *Runner) SetOnComplete(fn func(Outcome))

SetOnComplete installs the per-session callback after construction. The runner is built before the plan is (it health-probes the panel first), and the caller's grade pool is sized from the plan, so the callback cannot be known at New time.

type Tier

type Tier string

Tier names an evaluation's depth.

const (
	// TierSmoke is immediate feedback: five evals on one member, no baseline
	// and no probes.
	TierSmoke Tier = "smoke"
	// TierFull is the reportable tier: 30 task sessions on one member plus a
	// six-query trigger smoke check.
	TierFull Tier = "full"
	// TierDeep adds the floor member and a third attempt per eval.
	TierDeep Tier = "deep"
)

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL