Documentation
¶
Overview ¶
Package runner plans and executes an evaluation: which sessions to run, in what environment, with what resumed from a previous attempt.
Planning is separated from execution and has no I/O, so the session budgets are arithmetic a test can check rather than a cost discovered afterwards.
Index ¶
Constants ¶
const ( CondSkill store.Condition = "skill" CondBaseline store.Condition = "baseline" CondTrigger store.Condition = "trigger" )
Conditions a session runs under. The skill/baseline pair is what Uplift compares; a trigger probe is a short session measuring only whether the skill fired. Typed as store.Condition — the shared representation store.RunKey and report.ConditionReport both use — rather than a bare string.
Variables ¶
var DefaultAllowDomains = []string{"api.anthropic.com", "statsig.anthropic.com", "sentry.io"}
DefaultAllowDomains is the egress an agent session needs and nothing more. A session with no network cannot authenticate; a session with full network makes "did the skill reach out" unanswerable.
var ErrEventsNotWritten = errors.New("runner: events.jsonl was not written")
ErrEventsNotWritten wraps a failure to write events.jsonl specifically, so a caller can tell that failure apart from a transcript, grading, or outputs failure with errors.Is rather than by parsing a message. Events are the one artifact scoring and resume actually read back — losing the transcript or a verifier's output files is a loss of secondary evidence, but losing events.jsonl means the run cannot be scored or resumed at all.
Functions ¶
func LoadEvents ¶
func LoadEvents(path string) ([]trajectory.Event, error)
LoadEvents reads a newline-delimited JSON trajectory written by WriteArtifacts.
Types ¶
type Artifacts ¶
type Artifacts struct {
Dir string
TranscriptPath string
EventsPath string
GradingPath string
OutputsDir string
}
Artifacts locates the files WriteArtifacts produced for one run.
func WriteArtifacts ¶
func WriteArtifacts(dir string, raw []byte, events []trajectory.Event, g Grading, workspace string, skipDirs []string) (Artifacts, error)
WriteArtifacts records the evidence trail for one run into dir: transcript.raw (the agent's native stream, byte for byte), events.jsonl (the normalized trajectory, one compact JSON object per line, in order), grading.json (the Grading record, indented for a human), and outputs/ (a copy of the workspace's regular files a verifier inspected, skipping any entry under a directory listed in skipDirs).
skipDirs exists so the installed skill bundle — and, for a trigger probe, the distractor pack alongside it — is never copied into outputs/: it is already stored once as the published bundle, and sixty copies of it under runs/ would be most of the disk an evaluation uses for zero evidentiary value. A baseline session installs no skill, so its caller passes no skipDirs and its real outputs are copied in full.
WriteArtifacts is best-effort across the four artifacts: it attempts every one rather than stopping at the first failure, so a caller that only cares about (say) the events failure is not also denied the transcript that did write successfully. Every failure is accumulated and returned via errors.Join; a failure to write events.jsonl specifically is wrapped in ErrEventsNotWritten so a caller can single it out with errors.Is without parsing the message.
type ExecuteInput ¶
type ExecuteInput struct {
Skill string
BundleDir string
SuiteDir string
Image sandbox.ImageRef
Plan *Plan
Healthy map[Member]bool
Distractors []suite.Distractor
}
ExecuteInput is everything Execute needs beyond the eval id.
type ExecuteResult ¶
type ExecuteResult struct {
Outcomes []Outcome
Probes []ProbeOutcome
// PanelComplete is false when any planned panel member was skipped for
// being unhealthy. A false PanelComplete means the eval's scores are
// partial, not that the skill scored zero on that member.
PanelComplete bool
}
ExecuteResult is everything Execute produced.
type Grading ¶
type Grading struct {
Key store.RunKey
// VerifierExit is nil when the verifier never ran — see
// store.RunOutcome.VerifierExit, which this mirrors.
VerifierExit *int
Meta agent.Meta
Status string
Error string
StartedAt time.Time
FinishedAt time.Time
}
Grading is the human- and machine-readable record of how one run was graded: the key that identifies it, what the verifier reported, the adapter's session metadata, and its terminal status. It is what a report drills into and what makes a surprising score checkable.
func LoadGrading ¶
LoadGrading reads a grading.json written by WriteArtifacts.
type Member ¶
type Member struct {
Agent string
Model string
// Class is the capability tier this member stands for. RobustnessGap is the
// difference between the strong member and the floor member, so the class
// is part of the panel's identity rather than a label.
Class spec.ModelTier
}
Member is one entry in the model panel.
type Options ¶
type Options struct {
Store *store.Store
Driver sandbox.Driver
Adapters func(name string) (agent.Adapter, bool)
Concurrency int
SessionTimeout time.Duration
Untrusted bool
AllowDomains []string
// Sleep is time.Sleep by default; tests override it so a backoff test does
// not actually wait minutes.
Sleep func(time.Duration)
// Logger receives progress and retry notices. A no-op by default.
Logger func(string, ...any)
MaxRateLimitRetries int
}
Options configures a Runner.
type Outcome ¶
type Outcome struct {
Key store.RunKey
// VerifierExit is nil when the verifier never ran (the session failed
// before reaching it, or timed out inside it) and its exit code
// otherwise — see store.RunOutcome.VerifierExit, which this feeds.
VerifierExit *int
Events []trajectory.Event
Meta agent.Meta
ArtifactDir string
Status string
Err error
// MetaPartial is true when Meta was rebuilt from the store's own columns
// rather than from the run's grading.json artifact — on resume, when the
// artifact is missing or unreadable. A partial Meta carries only what
// those columns hold (tokens, duration, agent version, rate-limited);
// Model, NumTurns, VisibleSkills, PermissionDenials, and IsError are
// silently zero, so a caller that feeds Meta into anything reading those
// fields (VisibleSkills feeds trigger precision) must check this first.
MetaPartial bool
MetaPartialReason string
}
Outcome is what one planned session (skill or baseline) reported.
type Panel ¶
type Panel []Member
Panel is the set of members every task is run against.
func DefaultPanel ¶
func DefaultPanel() Panel
DefaultPanel is the shipped panel: one strong member and one floor member of the same vendor.
Two capability tiers rather than two vendors, for a recorded reason: the second vendor's CLI is unauthenticated on the machine this was built on, and a parser written from documentation passes its own tests and fails on real output. min-across-panel and RobustnessGap need a strong/floor pair, which this provides; the cross-vendor claim stays unmade until a real stream can be recorded.
func ParsePanel ¶
ParsePanel builds a panel from CLI arguments, cross-checking each agent against the adapter registry so a typo is a refusal rather than a panel member that silently never runs.
type Plan ¶
type Plan struct {
Tier Tier
Panel Panel
Runs []store.RunKey
Probes []Probe
// N is runs per task per member; K is the passes required by the
// Reliability estimator.
N, K int
// BaselineK is the passes-required estimator width for the baseline
// condition — see tierBudget.BaselineK.
BaselineK int
Tasks []suite.TaskPkg
}
Plan is the complete list of work an evaluation performs.
func (*Plan) DevRuns ¶
DevRuns and HoldoutRuns partition the plan by split. The repair loop iterates on DevRuns and reports HoldoutRuns, and it must never see the latter.
func (*Plan) HoldoutRuns ¶
func (*Plan) SessionCount ¶
SessionCount is the total number of agent sessions the plan will spend.
type Probe ¶
Probe is one trigger measurement: a short session that answers only whether the skill fired.
type ProbeOutcome ¶
type ProbeOutcome struct {
Probe Probe
Events []trajectory.Event
Meta agent.Meta
Caps agent.Caps
// Unknown is true when the probe could not be observed at all — the
// session failed outright, or (on resume) its events could not be
// recovered. An unknown probe is excluded from trigger-precision
// denominators rather than counted as a miss.
Unknown bool
Reason string
Err error
// MetaPartial mirrors Outcome.MetaPartial: true when Meta came from the
// store's own columns on resume rather than the probe's grading.json.
MetaPartial bool
MetaPartialReason string
}
ProbeOutcome is what one trigger probe observed. It deliberately carries no pre-computed verdict: whether the skill fired, and whether that determination was explicit or inferred, depends on the adapter's capabilities and is scoring's job, not the runner's — score.DetectFiring (a later task) is the one definition of firing, working from exactly these fields.
type Runner ¶
type Runner struct {
// contains filtered or unexported fields
}
Runner executes a plan.
func New ¶
New validates the options and applies defaults: concurrency 4 (§9's scheduling assumption), a 20-minute session timeout, and three rate-limit retries.
func (*Runner) Execute ¶
func (r *Runner) Execute(ctx context.Context, evalID int64, in ExecuteInput) (*ExecuteResult, error)
Execute runs in.Plan against evalID, claiming each session first so a resumed Execute over the same eval spends nothing on work already recorded.
func (*Runner) ProbePanel ¶
ProbePanel runs one trivial session against every panel member and reports whether each came back healthy. A member is healthy only when both the invocation and the parse succeed: an adapter that invokes and cannot parse produces a trajectory of nothing, which would otherwise score as a session in which the agent did nothing.
type Tier ¶
type Tier string
Tier names an evaluation's depth.
const ( // TierSmoke is immediate feedback: five development tasks on the primary // member, no baselines, no probes. TierSmoke Tier = "smoke" // TierFull is the reportable tier: 60 core sessions plus 16 trigger probes. TierFull Tier = "full" // TierDeep is opt-in and takes hours. TierDeep Tier = "deep" )