Documentation
¶
Overview ¶
Package runner executes an experiment: every task, repeated, against one or more subjects, and turns the resulting bundles into a comparable report.
Index ¶
- Constants
- func Preflight(ctx context.Context, entry TaskEntry) error
- func PreflightSuite(ctx context.Context, tasks []TaskEntry) error
- func Wilson(successes, trials int, z float64) (low, high float64)
- func WriteComparison(w io.Writer, baseline, candidate Result, cmp Comparison)
- func WriteReport(w io.Writer, result Result)
- type Comparison
- type Result
- type Runner
- type TaskEntry
- type TaskOutcomes
Constants ¶
const DefaultResamples = 2000
DefaultResamples is how many bootstrap draws a comparison takes. Two thousand is enough for a stable 95% percentile interval at the task counts a suite realistically holds, and cheap next to running the trials themselves.
const Z95 = 1.959963984540054
Z95 is the standard normal quantile for a 95% two-sided interval.
Variables ¶
This section is empty.
Functions ¶
func Preflight ¶
Preflight checks that a task measures what it claims to, without running a model.
Section 8.1 asks five things of a committed task. Four are checked here:
- the hidden material stays outside the trial workspace;
- the initial state does not already satisfy the required graders, because a task that starts finished passes for every subject and distinguishes none of them;
- the oracle completes the task; and
- every required grader accepts the oracle, since a task whose own reference solution fails is measuring its graders rather than a subject.
The fifth — that repeated oracle runs are deterministic enough — needs repetition policy this function does not own, and is left to the caller.
Only deterministic graders take part. A trace grader has no trace to read when nothing ran an agent, and a model grader would need a provider; running either here would report an unknown verdict as a task defect.
func PreflightSuite ¶
PreflightSuite checks every task and reports all failures rather than the first: a contributor fixing a suite wants the whole list.
func Wilson ¶
Wilson returns the Wilson score interval for successes out of trials.
It is used rather than the textbook normal approximation because evaluation runs are small. Ten passes out of ten give a normal interval of zero width, which reads as certainty from a sample that cannot support it; Wilson keeps the interval away from the boundary and stays defined when the count is zero or complete.
Trials of zero returns [0,1]: no evidence is not evidence of nothing.
func WriteComparison ¶
func WriteComparison(w io.Writer, baseline, candidate Result, cmp Comparison)
WriteComparison renders a candidate against a baseline.
func WriteReport ¶
WriteReport renders one subject's result.
The layout follows section 12: a vector, never a single number. Pass rate, consistency, harness faults, and critical failures are printed as separate lines because none of them may be averaged into another — a trust violation that a rising pass rate could absorb is exactly what section 7.5 forbids.
Types ¶
type Comparison ¶
type Comparison struct {
// Delta is the candidate's mean per-task rate minus the baseline's, over
// tasks both subjects actually ran.
Delta float64
// Low and High bound Delta. The interval is over the paired difference
// rather than over each rate separately: two rates can both move while
// their difference stays indistinguishable from noise, and a reader
// comparing two separate intervals cannot see that.
Low, High float64
// Improved and Regressed name tasks whose rate moved in each direction.
Improved, Regressed []string
// Unscorable names tasks one side or the other never scored. Section 12
// requires them shown: a task silently dropped from both arms reads as
// agreement.
Unscorable []string
// Paired is how many tasks the delta rests on.
Paired int
}
Comparison is a candidate measured against a baseline over shared tasks.
func ComparePaired ¶
func ComparePaired(baseline, candidate TaskOutcomes, seed uint64, resamples int) Comparison
ComparePaired computes the paired difference between two subjects and a percentile bootstrap interval over tasks.
Resampling is over tasks rather than over trials because tasks are what vary independently: two attempts at the same task share its difficulty, and treating them as independent draws would report an interval far narrower than the evidence supports.
The seed is a parameter so a report is reproducible. An interval that moves between two readings of the same data is not a measurement.
type Result ¶
type Result struct {
Subject contract.SubjectManifest
Bundles []contract.TrialBundle
Outcomes TaskOutcomes
Metrics contract.SuiteMetrics
}
Result is one subject's outcome across a suite.
func Summarize ¶
func Summarize(subject contract.SubjectManifest, bundles []contract.TrialBundle) Result
Summarize derives one subject's outcome vector from the trials it produced.
It takes bundles rather than being folded into the execution loop because a bundle is the contract's unit of evidence, and evidence that arrives some other way deserves the same arithmetic. An external benchmark imported from Harbor has no execution loop here at all, and a second summarizer written for it would be a second definition of what a pass rate counts — which harness faults are in the denominator, whether a timeout is scored, how consistency is measured. Those are section 12's decisions, and they get exactly one implementation.
Bundle order is the attempt order: a task's attempts are recorded as they appear, so a caller pairing on index has to hand them over in index order.
type Runner ¶
type Runner struct {
// Adapters maps each surface to the executor that runs it. A suite may hold
// tasks for more than one surface, and a task states the surface it
// measures, so dispatch is per task rather than per experiment.
Adapters map[contract.Surface]adapter.Executor
Graders grader.Registry
// BundleRoot is where trial evidence is written.
BundleRoot string
// Trials overrides each task's own repetition count when higher. Section 12
// estimates pass@1 from independent attempts, so raising it is how a
// scheduled run buys a tighter interval than a pull-request run needs.
Trials int
// KeepFailures leaves the temporary workspace of a failed trial in place so
// a contributor can look at it. The bundle records where.
KeepFailures bool
// Progress receives one line per finished trial. Nil discards them.
Progress func(string)
}
Runner executes an experiment against one subject.
func (*Runner) Run ¶
func (r *Runner) Run(ctx context.Context, tasks []TaskEntry, subject contract.SubjectManifest, experimentID string) (Result, error)
Run executes every task the given number of times and returns the subject's result. It writes each trial's bundle under BundleRoot as it goes, so an interrupted experiment leaves the evidence it already gathered.
type TaskEntry ¶
TaskEntry is a loaded task and the directory it came from. The two travel together because the directory holds what the task deliberately does not carry inline: the initial state, the graders, and the oracle.
func LoadSuite ¶
LoadSuite reads every task directory directly under root, sorted by task id.
A directory without a task file is skipped, but a task file that does not load is an error rather than a skip. Section 8.1 makes task validity part of the experiment: a suite that quietly ran nine of its ten tasks reports a score for a dataset that was never used.
type TaskOutcomes ¶
TaskOutcomes is one subject's per-task attempts: true for a scored pass, false for a scored failure. Trials the harness could not run are absent rather than false, because section 12 keeps them out of the rate.
func (TaskOutcomes) ConsistencyRate ¶
func (o TaskOutcomes) ConsistencyRate() float64
ConsistencyRate is pass^k: the share of tasks that passed every attempt.
A task with no scored attempt is excluded rather than counted as inconsistent. It was never measured, and calling that a consistency failure would let an infrastructure outage look like a flaky subject.
func (TaskOutcomes) PassRate ¶
func (o TaskOutcomes) PassRate() (passed, scored int, rate float64)
PassRate is scored passes over scored attempts.
Source Files