Documentation
¶
Overview ¶
Package runner abstracts over invoking a CLI agent against a sandbox working directory. It is the R4 harness's per-platform binding: the same Runner interface drives claude, codex, and gh-copilot from the same eval pipeline.
Canonical AgentTelemetry ¶
AgentTelemetry is the canonical type for agent-runner identity and usage telemetry across the eval harness. The scoringbridge package defines its own interim AgentTelemetry shape (introduced before this package existed); a follow-up should switch scoringbridge to import runner.AgentTelemetry directly to avoid the two-type maintenance burden. Until then both shapes are structurally identical and the harness driver maps between them at the scoringbridge call site.
OQ1 ruling (v1 adapter set) ¶
v1 runners: claude, codex, AND copilot all ship (repo-owner ruling 2026-07-02), overriding R4 OQ1's "others stubbed" default (design.md:101). The owner ruling is the canonical authorization; additional platforms (cursor, others) remain deferred behind the Runner interface and can be added as follow-on per-adapter tasks without harness changes.
Exec seam ¶
Every adapter struct carries a run field of type cmdFn. In production the field is set to realExec, which wraps exec.CommandContext. Tests replace it with a deterministic fake to avoid spawning real subprocesses. Using a per-instance field rather than a package-level var keeps tests safe under -race: each test constructs its own adapter instance with its own fake, so no goroutine races on shared state.
Token telemetry ¶
Claude emits structured usage data via --output-format json; the claude adapter parses it when present. Codex and copilot do not expose machine-readable token counts in their v1 CLI surfaces; their adapters leave Telemetry.Tokens nil, which scoringbridge records as absent (the rubric renormalizes over present signals).
Index ¶
Constants ¶
This section is empty.
Variables ¶
var ErrUnknownAdapter = errors.New("runner: unknown adapter")
ErrUnknownAdapter is returned by New when the adapter name is not recognised.
Functions ¶
This section is empty.
Types ¶
type AgentStartError ¶
type AgentStartError struct {
// Agent is the adapter/harness identity (claude-code, codex, gh-copilot).
Agent string
// Binary is the CLI that was invoked or looked up.
Binary string
// Reason is why the agent did not run.
Reason AgentStartReason
// ExitCode is the CLI's exit code for the auth case; 0 when unavailable.
ExitCode int
// Detail is the auth signature that matched (auth case); empty otherwise.
Detail string
// Cause is the underlying exec error (unavailable case); nil otherwise.
Cause error
}
AgentStartError signals that an agent invocation did not produce a scorable run: the CLI was absent (ReasonUnavailable) or it failed to authenticate/configure under the sandbox's isolated HOME (ReasonAuth). It is DISTINCT from a completed agent run that produced a wrong solution — a non-zero exit with no auth signature stays a scorable Result. The harness aborts the run on this error with a clear message, so an environment/credential problem is never scored as poor model quality (dogfood #10). Match it with errors.As(err, *AgentStartError).
func (*AgentStartError) Error ¶
func (e *AgentStartError) Error() string
Error implements error with an actionable, operator-facing message that spells out the isolated-HOME cause for the auth case.
func (*AgentStartError) Unwrap ¶
func (e *AgentStartError) Unwrap() error
Unwrap exposes the underlying exec error so errors.Is/As traversal reaches it (the unavailable case wraps the original exec failure).
type AgentStartReason ¶
type AgentStartReason string
AgentStartReason classifies why an agent invocation did not yield a scorable run.
const ( ReasonUnavailable AgentStartReason = "unavailable" // ReasonAuth: the agent CLI ran but exited with an authentication or // configuration failure — the expected outcome when the eval sandbox runs it // under an isolated HOME that carries no credentials. ReasonAuth AgentStartReason = "auth" )
type AgentTelemetry ¶
type AgentTelemetry struct {
// SessionID is the platform session UUID when the runner captured one.
SessionID string
// Harness is the agent platform identifier (claude-code, codex, ...).
Harness string
// Model is the model ID the run used; populated when the CLI reports it.
Model string
// Retries is how many times the runner had to re-invoke the agent.
Retries int
// Tokens is the run's token usage; nil when the runner captured none.
// Absent token data is first-class — the rubric renormalizes.
Tokens *scoring.TokenUsage
}
AgentTelemetry is the canonical agent-runner identity and usage telemetry for one eval run (R4 spec requirement R9). All fields are optional; what is present flows into the scoringbridge's iteration record so the platform-tuner persona can diff platforms.
Scoringbridge defines its own interim AgentTelemetry shape; a follow-up should switch it to import this type. Until then the harness driver maps between them at the bridge call site (the shapes are structurally identical).
type FakeRunner ¶
type FakeRunner struct {
// Result is returned from every Run call.
Result Result
// Err is returned from every Run call; when non-nil the harness treats the
// invocation as a launch failure (Result is still returned alongside it,
// matching the real adapters' error contract).
Err error
// Calls counts how many times Run was invoked.
Calls int
// LastSpec and LastInstance capture the most recent Run arguments so a
// consumer can assert the harness wired the sandbox and task through.
LastSpec *eval.TaskSpec
LastInstance *sandbox.Instance
}
FakeRunner is an in-memory Runner for downstream tests (harness-driver and beyond) that need to inject a scripted agent without shelling out to a real CLI. It returns its canned Result and Err verbatim, and records the last call's arguments so a consumer can assert what the harness passed.
This is the injectable fake the plan's TASKS.yaml notes call for ("v1 ships a fakeRunner for tests"): a fake Runner, distinct from the per-adapter exec seam. Script success, failure, or cancellation by setting Result and Err; leave both zero for a no-op success.
func (*FakeRunner) Run ¶
func (f *FakeRunner) Run( _ context.Context, spec *eval.TaskSpec, instance *sandbox.Instance, ) (Result, error)
Run implements Runner. It records the call and returns the canned Result and Err. The context is accepted to satisfy the interface; FakeRunner performs no work, so it neither blocks on nor observes cancellation — a consumer that needs to script a cancel sets Err to a context error.
type Result ¶
type Result struct {
// Stdout is the captured standard output of the agent process.
Stdout []byte
// Stderr is the captured standard error of the agent process.
Stderr []byte
// ExitCode is the process exit code (0 = success).
ExitCode int
// Duration is wall-clock time from exec start to process exit.
Duration time.Duration
// Telemetry holds the identity and usage data the adapter extracted.
Telemetry AgentTelemetry
}
Result is the complete output of one agent run.
type Runner ¶
type Runner interface {
// Run invokes the agent with the task prompt inside instance.Workdir,
// appending instance.Env to the subprocess environment. The returned
// Result carries stdout, stderr, exit code, wall time, and any telemetry
// the adapter could extract. A non-zero exit code is NOT itself a Go
// error — it is encoded in Result.ExitCode so the harness can
// distinguish "the agent ran but produced wrong output" from "we could
// not launch the agent at all".
Run(ctx context.Context, spec *eval.TaskSpec, instance *sandbox.Instance) (Result, error)
}
Runner runs an agent against a TaskSpec inside a provisioned sandbox and returns the captured output and telemetry.