Documentation
¶
Overview ¶
Package runner plans and executes an evaluation: which sessions to run, in what environment, with what resumed from a previous attempt.
Planning is separated from execution and has no I/O, so the session budgets are arithmetic a test can check rather than a cost discovered afterwards.
Index ¶
- Constants
- Variables
- func BaselinePlanned(p Plan) bool
- func EvalID(id int) string
- func LoadEvents(path string) ([]agent.Event, error)
- type Artifacts
- type ExecuteInput
- type ExecuteResult
- type Grading
- type Health
- type Member
- type Options
- type Outcome
- type Panel
- type Plan
- type Probe
- type ProbeOutcome
- type Runner
- type Tier
Constants ¶
const ( CondSkill store.Condition = "skill" CondBaseline store.Condition = "baseline" CondTrigger store.Condition = "trigger" )
Conditions a session runs under. The skill/baseline pair is what the published delta compares; a trigger probe is a short session measuring only whether the skill fired.
Variables ¶
var DefaultAllowDomains = []string{"api.anthropic.com", "statsig.anthropic.com", "sentry.io"}
var ErrEventsNotWritten = errors.New("runner: events.jsonl was not written")
ErrEventsNotWritten is separable with errors.Is because events are the one artifact scoring and resume read back: losing the transcript costs secondary evidence, losing events.jsonl means the run cannot be scored at all.
Functions ¶
func BaselinePlanned ¶ added in v0.11.0
BaselinePlanned reports whether the plan runs any baseline session, which is what makes the without-skill delta measurable.
Types ¶
type Artifacts ¶
type Artifacts struct {
Dir string
TranscriptPath string
EventsPath string
GradingPath string
OutputsDir string
}
Artifacts locates the files WriteArtifacts produced for one run.
func WriteArtifacts ¶
func WriteArtifacts(dir string, raw []byte, events []agent.Event, g Grading, workspace string, skipDirs []string) (Artifacts, error)
WriteArtifacts records one run's evidence trail into dir: transcript.raw, events.jsonl, grading.json and outputs/.
skipDirs keeps the installed bundle out of outputs/ — it is already stored once as the published bundle, and sixty copies of it would be most of the disk an evaluation uses. A baseline installs no skill and passes none.
Best-effort across all four: every failure is attempted and joined, so a caller denied its events still gets the transcript that did write.
type ExecuteInput ¶
type ExecuteInput struct {
Skill string
BundleDir string
SuiteDir string
Image sandbox.ImageRef
Plan *Plan
Healthy map[Member]bool
Distractors []suite.Distractor
}
ExecuteInput is everything Execute needs beyond the eval id.
type ExecuteResult ¶
type ExecuteResult struct {
Outcomes []Outcome
Probes []ProbeOutcome
PanelComplete bool
}
ExecuteResult is everything Execute produced.
type Grading ¶
type Grading struct {
Key store.RunKey
Meta agent.Meta
Status string
Error string
StartedAt time.Time
FinishedAt time.Time
}
Grading is the record of one run: what identifies it, the session metadata, and its terminal status. It is what makes a surprising score checkable.
func LoadGrading ¶
LoadGrading reads a grading.json written by WriteArtifacts.
type Options ¶
type Options struct {
Store *store.Store
Driver sandbox.Driver
Adapters func(name string) (agent.Adapter, bool)
Concurrency int
SessionTimeout time.Duration
Untrusted bool
AllowDomains []string
// PanelExcludeEnv withholds credential variables from the sandbox. Comes
// from provider.Config.PanelExcludeEnv — this package cannot import
// provider (circular), so it receives the names as data.
PanelExcludeEnv []string
// WorkspaceRoot overrides os.TempDir() for a containerized runner whose
// /tmp is invisible to the host daemon. Must be bind-mounted at the same
// path on both sides or Docker silently mounts an empty directory.
WorkspaceRoot string
Sleep func(time.Duration)
Logger func(string, ...any)
// OnComplete, when set, is called once per finished session, from that
// session's own goroutine. It lets a caller grade a session while the
// rest still run: a grade needs that session's artifacts and nothing
// else. It is never called for a probe.
OnComplete func(Outcome)
MaxRateLimitRetries int
}
Options configures a Runner.
type Outcome ¶
type Outcome struct {
Key store.RunKey
Events []agent.Event
Meta agent.Meta
ArtifactDir string
Status string
Err error
// MetaPartial is true when Meta was rebuilt from store columns rather than
// the grading.json artifact. Model, NumTurns, VisibleSkills,
// PermissionDenials, and IsError are zero in that case.
MetaPartial bool
MetaPartialReason string
}
Outcome is what one planned session (skill or baseline) reported.
type Panel ¶
type Panel []Member
Panel is the set of members every eval is run against.
func DefaultPanel ¶
func DefaultPanel() Panel
DefaultPanel is the shipped panel: one member.
Sonnet rather than Opus, because it is what most people running claude-code actually run — so it is a truer measurement, not merely a cheaper one. The floor member joins only at TierDeep: "works on the weakest model" is worth measuring, and it is not worth doubling every score's cost to measure.
func ParsePanel ¶
ParsePanel builds a panel from caller arguments, cross-checking each agent against the adapter lookup so a typo is a refusal rather than a panel member that silently never runs. The first model is the strong member and any others are floor members.
Models alone are enough. The UI and the re-run endpoint offer a model without an agent — there is one adapter to choose from — and requiring both meant a chosen model fell back to the shipped panel with no error anywhere, producing a score against a model nobody asked for.
type Plan ¶
Plan is the complete list of work an evaluation performs.
type Probe ¶
Probe is one trigger measurement: a short session that answers only whether the skill fired.
type ProbeOutcome ¶
type ProbeOutcome struct {
Probe Probe
Events []agent.Event
Meta agent.Meta
Caps agent.Caps
// Unknown is true when the probe could not be observed at all — the
// session failed outright, or (on resume) its events could not be
// recovered. An unknown probe is excluded from trigger-precision
// denominators rather than counted as a miss.
Unknown bool
Reason string
Err error
// MetaPartial mirrors Outcome.MetaPartial: true when Meta came from the
// store's own columns on resume rather than the probe's grading.json.
MetaPartial bool
MetaPartialReason string
}
ProbeOutcome is what one trigger probe observed. No pre-computed verdict: whether the skill fired is scoring's job (score.DetectFiring).
type Runner ¶
type Runner struct {
// contains filtered or unexported fields
}
Runner executes a plan.
func (*Runner) Execute ¶
func (r *Runner) Execute(ctx context.Context, evalID int64, in ExecuteInput) (*ExecuteResult, error)
Execute runs in.Plan, skipping sessions already recorded.
func (*Runner) ProbePanel ¶
ProbePanel runs one trivial session against every panel member and reports whether each came back healthy. A member is healthy only when both the invocation and the parse succeed: an adapter that invokes and cannot parse produces a trajectory of nothing, which would otherwise score as a session in which the agent did nothing.
func (*Runner) SetOnComplete ¶ added in v0.12.0
SetOnComplete installs the per-session callback after construction. The runner is built before the plan is (it health-probes the panel first), and the caller's grade pool is sized from the plan, so the callback cannot be known at New time.
type Tier ¶
type Tier string
Tier names an evaluation's depth.
const ( // TierSmoke is immediate feedback: five evals on one member, no baseline // and no probes. TierSmoke Tier = "smoke" // TierFull is the reportable tier: 30 task sessions on one member plus a // six-query trigger smoke check. TierFull Tier = "full" // TierDeep adds the floor member and a third attempt per eval. TierDeep Tier = "deep" )