Documentation
¶
Overview ¶
Package harness is the R4 eval driver: a single Harness.Run entry point that sequences the five eval stages and returns an EvalRun aggregating every stage's output.
The five stages, in order:
- generate — resolve the language's generator from the injected registry and synthesise a versioned eval.TaskSpec from the knowledge graph.
- provision — create an isolated sandbox instance for the run.
- run — invoke the agent runner inside the sandbox workdir.
- verify — run the language verifier's build/test commands over the sandbox workdir.
- score — bridge the completed run into R1: emit the iteration record and persist the rubric score.
The harness is pure orchestration. Every stage's heavy lifting lives behind a seam interface injected at construction (an eval.Registry of generators, a sandbox.Sandbox, a runner.Runner, and per-language verifier.Verifiers), so a test can swap a FakeRunner or a fixture sandbox without touching the driver. The driver owns only the sequencing and the failure policy:
- An infrastructure failure — no generator or verifier for the language, a provision error, an agent launch failure, a verifier step that could not start, a scoring write error, or a sandbox cleanup failure — is surfaced as an error.
- A completed-but-wrong outcome is not an error. A failing build or test is recorded in the score: the verifier encodes it in its VerifyResult, which is what drives the rubric's verifier signal. A non-zero agent exit code is likewise not an error — the runner encodes it in its Result, which the harness captures on the returned EvalRun. The exit code itself is not a scoring input: the score is driven by the verify outcome and the agent telemetry (retries, token usage) that flow into the scoringbridge; a run that exits non-zero but still verifies clean scores as a pass.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Config ¶
type Config struct {
// Generators resolves a language to its task generator (the generate
// stage). The harness only ever reads it via Lookup.
Generators *eval.Registry
// Sandbox provisions the isolated working tree each run executes in (the
// provision stage).
Sandbox sandbox.Sandbox
// Runner invokes the agent inside a provisioned sandbox (the run stage).
Runner runner.Runner
// Verifiers maps a language to the verifier that runs its build/test
// commands (the verify stage). At least one entry is required.
Verifiers map[eval.Language]verifier.Verifier
}
Config carries the seam dependencies a Harness orchestrates. Every field is required; New rejects a nil dependency so a mis-wired harness fails at construction rather than mid-run.
type EvalRun ¶
type EvalRun struct {
// Spec is the TaskSpec the generate stage produced (stage 1).
Spec *eval.TaskSpec
// RunID, RunDir, and BaseCommit identify the provisioned sandbox instance
// (stage 2) and pin the run for reproducibility.
RunID string
RunDir string
BaseCommit string
// Run is the agent runner's output and telemetry (stage 3).
Run runner.Result
// Verify is the verifier's full build/test outcome (stage 4).
Verify *verifier.VerifyResult
// Score is everything the scoring bridge persisted and computed (stage 5).
Score scoringbridge.Result
}
EvalRun aggregates the output of every stage of one Harness.Run. It is the harness's result — distinct from the scoringbridge.EvalRun that Run assembles internally as the score stage's input.
type Harness ¶
type Harness struct {
// contains filtered or unexported fields
}
Harness sequences the five eval stages behind their seam interfaces. It is safe to reuse across runs (it holds no per-run mutable state); concurrency safety is inherited from the injected seams.
func New ¶
New validates cfg and returns a Harness. It errors when any dependency is absent so wiring bugs surface immediately.
func (*Harness) Run ¶
Run sequences the five eval stages for one task and returns the aggregated EvalRun. A stage's infrastructure failure aborts the run and is returned as an error; a completed-but-failing agent or verifier outcome is not an error — it flows into the score.
The provisioned sandbox instance is always cleaned up before Run returns. A Cleanup failure is not swallowed: on an otherwise-successful run it turns the run into an error (spec R8 — a run must not silently leak working trees), while the returned EvalRun stays fully populated (the score was already computed and persisted). A cleanup failure never masks a primary-stage error; that root cause wins.
type Options ¶
type Options struct {
// Language selects both the generator and the verifier for the run.
Language eval.Language
// Difficulty optionally constrains the generated task's band; the zero
// value lets the generator choose.
Difficulty eval.Difficulty
// TemplateID optionally selects a specific generator template; the zero
// value lets the generator pick its default.
TemplateID string
}
Options are the per-run inputs Run threads into the generate stage.
Source Files