runner

package
v0.10.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 6, 2026 License: Apache-2.0 Imports: 24 Imported by: 0

Documentation

Overview

Package runner plans and executes an evaluation: which sessions to run, in what environment, with what resumed from a previous attempt.

Planning is separated from execution and has no I/O, so the session budgets are arithmetic a test can check rather than a cost discovered afterwards.

Index

Constants

View Source
const (
	CondSkill    store.Condition = "skill"
	CondBaseline store.Condition = "baseline"
	CondTrigger  store.Condition = "trigger"
)

Conditions a session runs under. The skill/baseline pair is what Uplift compares; a trigger probe is a short session measuring only whether the skill fired. Typed as store.Condition — the shared representation store.RunKey and report.ConditionReport both use — rather than a bare string.

Variables

View Source
var DefaultAllowDomains = []string{"api.anthropic.com", "statsig.anthropic.com", "sentry.io"}

DefaultAllowDomains is the egress an agent session needs and nothing more. A session with no network cannot authenticate; a session with full network makes "did the skill reach out" unanswerable.

View Source
var ErrEventsNotWritten = errors.New("runner: events.jsonl was not written")

ErrEventsNotWritten wraps a failure to write events.jsonl specifically, so a caller can tell that failure apart from a transcript, grading, or outputs failure with errors.Is rather than by parsing a message. Events are the one artifact scoring and resume actually read back — losing the transcript or a verifier's output files is a loss of secondary evidence, but losing events.jsonl means the run cannot be scored or resumed at all.

Functions

func LoadEvents

func LoadEvents(path string) ([]trajectory.Event, error)

LoadEvents reads a newline-delimited JSON trajectory written by WriteArtifacts.

Types

type Artifacts

type Artifacts struct {
	Dir            string
	TranscriptPath string
	EventsPath     string
	GradingPath    string
	OutputsDir     string
}

Artifacts locates the files WriteArtifacts produced for one run.

func WriteArtifacts

func WriteArtifacts(dir string, raw []byte, events []trajectory.Event, g Grading, workspace string, skipDirs []string) (Artifacts, error)

WriteArtifacts records the evidence trail for one run into dir: transcript.raw (the agent's native stream, byte for byte), events.jsonl (the normalized trajectory, one compact JSON object per line, in order), grading.json (the Grading record, indented for a human), and outputs/ (a copy of the workspace's regular files a verifier inspected, skipping any entry under a directory listed in skipDirs).

skipDirs exists so the installed skill bundle — and, for a trigger probe, the distractor pack alongside it — is never copied into outputs/: it is already stored once as the published bundle, and sixty copies of it under runs/ would be most of the disk an evaluation uses for zero evidentiary value. A baseline session installs no skill, so its caller passes no skipDirs and its real outputs are copied in full.

WriteArtifacts is best-effort across the four artifacts: it attempts every one rather than stopping at the first failure, so a caller that only cares about (say) the events failure is not also denied the transcript that did write successfully. Every failure is accumulated and returned via errors.Join; a failure to write events.jsonl specifically is wrapped in ErrEventsNotWritten so a caller can single it out with errors.Is without parsing the message.

type ExecuteInput

type ExecuteInput struct {
	Skill       string
	BundleDir   string
	SuiteDir    string
	Image       sandbox.ImageRef
	Plan        *Plan
	Healthy     map[Member]bool
	Distractors []suite.Distractor
}

ExecuteInput is everything Execute needs beyond the eval id.

type ExecuteResult

type ExecuteResult struct {
	Outcomes []Outcome
	Probes   []ProbeOutcome
	// PanelComplete is false when any planned panel member was skipped for
	// being unhealthy. A false PanelComplete means the eval's scores are
	// partial, not that the skill scored zero on that member.
	PanelComplete bool
}

ExecuteResult is everything Execute produced.

type Grading

type Grading struct {
	Key store.RunKey
	// VerifierExit is nil when the verifier never ran — see
	// store.RunOutcome.VerifierExit, which this mirrors.
	VerifierExit *int
	Meta         agent.Meta
	Status       string
	Error        string
	StartedAt    time.Time
	FinishedAt   time.Time
}

Grading is the human- and machine-readable record of how one run was graded: the key that identifies it, what the verifier reported, the adapter's session metadata, and its terminal status. It is what a report drills into and what makes a surprising score checkable.

func LoadGrading

func LoadGrading(path string) (*Grading, error)

LoadGrading reads a grading.json written by WriteArtifacts.

type Health

type Health struct {
	Member Member
	OK     bool
	Detail string
}

Health is one panel member's health-probe result.

type Member

type Member struct {
	Agent string
	Model string
	// Class is the capability tier this member stands for. RobustnessGap is the
	// difference between the strong member and the floor member, so the class
	// is part of the panel's identity rather than a label.
	Class spec.ModelTier
}

Member is one entry in the model panel.

type Options

type Options struct {
	Store    *store.Store
	Driver   sandbox.Driver
	Adapters func(name string) (agent.Adapter, bool)

	Concurrency    int
	SessionTimeout time.Duration
	Untrusted      bool
	AllowDomains   []string

	// Sleep is time.Sleep by default; tests override it so a backoff test does
	// not actually wait minutes.
	Sleep func(time.Duration)
	// Logger receives progress and retry notices. A no-op by default.
	Logger func(string, ...any)

	MaxRateLimitRetries int
}

Options configures a Runner.

type Outcome

type Outcome struct {
	Key store.RunKey
	// VerifierExit is nil when the verifier never ran (the session failed
	// before reaching it, or timed out inside it) and its exit code
	// otherwise — see store.RunOutcome.VerifierExit, which this feeds.
	VerifierExit *int
	Events       []trajectory.Event
	Meta         agent.Meta
	ArtifactDir  string
	Status       string
	Err          error
	// MetaPartial is true when Meta was rebuilt from the store's own columns
	// rather than from the run's grading.json artifact — on resume, when the
	// artifact is missing or unreadable. A partial Meta carries only what
	// those columns hold (tokens, duration, agent version, rate-limited);
	// Model, NumTurns, VisibleSkills, PermissionDenials, and IsError are
	// silently zero, so a caller that feeds Meta into anything reading those
	// fields (VisibleSkills feeds trigger precision) must check this first.
	MetaPartial       bool
	MetaPartialReason string
}

Outcome is what one planned session (skill or baseline) reported.

type Panel

type Panel []Member

Panel is the set of members every task is run against.

func DefaultPanel

func DefaultPanel() Panel

DefaultPanel is the shipped panel: one strong member and one floor member of the same vendor.

Two capability tiers rather than two vendors, for a recorded reason: the second vendor's CLI is unauthenticated on the machine this was built on, and a parser written from documentation passes its own tests and fails on real output. min-across-panel and RobustnessGap need a strong/floor pair, which this provides; the cross-vendor claim stays unmade until a real stream can be recorded.

func ParsePanel

func ParsePanel(agents, models []string) (Panel, error)

ParsePanel builds a panel from CLI arguments, cross-checking each agent against the adapter registry so a typo is a refusal rather than a panel member that silently never runs.

type Plan

type Plan struct {
	Tier   Tier
	Panel  Panel
	Runs   []store.RunKey
	Probes []Probe
	// N is runs per task per member; K is the passes required by the
	// Reliability estimator.
	N, K int
	// BaselineK is the passes-required estimator width for the baseline
	// condition — see tierBudget.BaselineK.
	BaselineK int
	Tasks     []suite.TaskPkg
}

Plan is the complete list of work an evaluation performs.

func BuildPlan

func BuildPlan(t Tier, p Panel, s *suite.Suite, void map[string]bool) (*Plan, error)

BuildPlan enumerates the sessions for a tier.

func (*Plan) DevRuns

func (p *Plan) DevRuns() []store.RunKey

DevRuns and HoldoutRuns partition the plan by split. The repair loop iterates on DevRuns and reports HoldoutRuns, and it must never see the latter.

func (*Plan) HoldoutRuns

func (p *Plan) HoldoutRuns() []store.RunKey

func (*Plan) SessionCount

func (p *Plan) SessionCount() int

SessionCount is the total number of agent sessions the plan will spend.

type Probe

type Probe struct {
	Index    int
	Prompt   string
	Positive bool
	Member   Member
}

Probe is one trigger measurement: a short session that answers only whether the skill fired.

type ProbeOutcome

type ProbeOutcome struct {
	Probe  Probe
	Events []trajectory.Event
	Meta   agent.Meta
	Caps   agent.Caps
	// Unknown is true when the probe could not be observed at all — the
	// session failed outright, or (on resume) its events could not be
	// recovered. An unknown probe is excluded from trigger-precision
	// denominators rather than counted as a miss.
	Unknown bool
	Reason  string
	Err     error
	// MetaPartial mirrors Outcome.MetaPartial: true when Meta came from the
	// store's own columns on resume rather than the probe's grading.json.
	MetaPartial       bool
	MetaPartialReason string
}

ProbeOutcome is what one trigger probe observed. It deliberately carries no pre-computed verdict: whether the skill fired, and whether that determination was explicit or inferred, depends on the adapter's capabilities and is scoring's job, not the runner's — score.DetectFiring (a later task) is the one definition of firing, working from exactly these fields.

type Runner

type Runner struct {
	// contains filtered or unexported fields
}

Runner executes a plan.

func New

func New(o Options) (*Runner, error)

New validates the options and applies defaults: concurrency 4 (§9's scheduling assumption), a 20-minute session timeout, and three rate-limit retries.

func (*Runner) Execute

func (r *Runner) Execute(ctx context.Context, evalID int64, in ExecuteInput) (*ExecuteResult, error)

Execute runs in.Plan against evalID, claiming each session first so a resumed Execute over the same eval spends nothing on work already recorded.

func (*Runner) ProbePanel

func (r *Runner) ProbePanel(ctx context.Context, p Panel, image sandbox.ImageRef) ([]Health, error)

ProbePanel runs one trivial session against every panel member and reports whether each came back healthy. A member is healthy only when both the invocation and the parse succeed: an adapter that invokes and cannot parse produces a trajectory of nothing, which would otherwise score as a session in which the agent did nothing.

type Tier

type Tier string

Tier names an evaluation's depth.

const (
	// TierSmoke is immediate feedback: five development tasks on the primary
	// member, no baselines, no probes.
	TierSmoke Tier = "smoke"
	// TierFull is the reportable tier: 60 core sessions plus 16 trigger probes.
	TierFull Tier = "full"
	// TierDeep is opt-in and takes hours.
	TierDeep Tier = "deep"
)

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL