runner

package
v0.5.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 22, 2026 License: Apache-2.0 Imports: 13 Imported by: 0

Documentation

Overview

Package runner abstracts over invoking a CLI agent against a sandbox working directory. It is the R4 harness's per-platform binding: the same Runner interface drives claude, codex, and gh-copilot from the same eval pipeline.

Canonical AgentTelemetry

AgentTelemetry is the canonical type for agent-runner identity and usage telemetry across the eval harness. The scoringbridge package defines its own interim AgentTelemetry shape (introduced before this package existed); a follow-up should switch scoringbridge to import runner.AgentTelemetry directly to avoid the two-type maintenance burden. Until then both shapes are structurally identical and the harness driver maps between them at the scoringbridge call site.

OQ1 ruling (v1 adapter set)

v1 runners: claude, codex, AND copilot all ship (repo-owner ruling 2026-07-02), overriding R4 OQ1's "others stubbed" default (design.md:101). The owner ruling is the canonical authorization; additional platforms (cursor, others) remain deferred behind the Runner interface and can be added as follow-on per-adapter tasks without harness changes.

Exec seam

Every adapter struct carries a run field of type cmdFn. In production the field is set to realExec, which wraps exec.CommandContext. Tests replace it with a deterministic fake to avoid spawning real subprocesses. Using a per-instance field rather than a package-level var keeps tests safe under -race: each test constructs its own adapter instance with its own fake, so no goroutine races on shared state.

Token telemetry

Claude emits structured usage data via --output-format json; the claude adapter parses it when present. Codex and copilot do not expose machine-readable token counts in their v1 CLI surfaces; their adapters leave Telemetry.Tokens nil, which scoringbridge records as absent (the rubric renormalizes over present signals).

Index

Constants

This section is empty.

Variables

View Source
var ErrUnknownAdapter = errors.New("runner: unknown adapter")

ErrUnknownAdapter is returned by New when the adapter name is not recognised.

Functions

This section is empty.

Types

type Adapter

type Adapter string

Adapter names one of the supported agent platforms.

const (
	AdapterClaude  Adapter = "claude"
	AdapterCodex   Adapter = "codex"
	AdapterCopilot Adapter = "copilot"
)

v1 adapter set (OQ1 ruling: claude, codex, and copilot ship; others deferred).

type AgentStartError

type AgentStartError struct {
	// Agent is the adapter/harness identity (claude-code, codex, gh-copilot).
	Agent string
	// Binary is the CLI that was invoked or looked up.
	Binary string
	// Reason is why the agent did not run.
	Reason AgentStartReason
	// ExitCode is the CLI's exit code for the auth case; 0 when unavailable.
	ExitCode int
	// Detail is the auth signature that matched (auth case); empty otherwise.
	Detail string
	// Cause is the underlying exec error (unavailable case); nil otherwise.
	Cause error
}

AgentStartError signals that an agent invocation did not produce a scorable run: the CLI was absent (ReasonUnavailable) or it failed to authenticate/configure under the sandbox's isolated HOME (ReasonAuth). It is DISTINCT from a completed agent run that produced a wrong solution — a non-zero exit with no auth signature stays a scorable Result. The harness aborts the run on this error with a clear message, so an environment/credential problem is never scored as poor model quality (dogfood #10). Match it with errors.As(err, *AgentStartError).

func (*AgentStartError) Error

func (e *AgentStartError) Error() string

Error implements error with an actionable, operator-facing message that spells out the isolated-HOME cause for the auth case.

func (*AgentStartError) Unwrap

func (e *AgentStartError) Unwrap() error

Unwrap exposes the underlying exec error so errors.Is/As traversal reaches it (the unavailable case wraps the original exec failure).

type AgentStartReason

type AgentStartReason string

AgentStartReason classifies why an agent invocation did not yield a scorable run.

const (
	// ReasonUnavailable: the agent CLI could not be found on PATH.
	ReasonUnavailable AgentStartReason = "unavailable"
	// ReasonAuth: the agent CLI ran but exited with an authentication or
	// configuration failure — the expected outcome when the eval sandbox runs it
	// under an isolated HOME that carries no credentials.
	ReasonAuth AgentStartReason = "auth"
)

type AgentTelemetry

type AgentTelemetry struct {
	// SessionID is the platform session UUID when the runner captured one.
	SessionID string
	// Harness is the agent platform identifier (claude-code, codex, ...).
	Harness string
	// Model is the model ID the run used; populated when the CLI reports it.
	Model string
	// Retries is how many times the runner had to re-invoke the agent.
	Retries int
	// Tokens is the run's token usage; nil when the runner captured none.
	// Absent token data is first-class — the rubric renormalizes.
	Tokens *scoring.TokenUsage
}

AgentTelemetry is the canonical agent-runner identity and usage telemetry for one eval run (R4 spec requirement R9). All fields are optional; what is present flows into the scoringbridge's iteration record so the platform-tuner persona can diff platforms.

Scoringbridge defines its own interim AgentTelemetry shape; a follow-up should switch it to import this type. Until then the harness driver maps between them at the bridge call site (the shapes are structurally identical).

type FakeRunner

type FakeRunner struct {
	// Result is returned from every Run call.
	Result Result
	// Err is returned from every Run call; when non-nil the harness treats the
	// invocation as a launch failure (Result is still returned alongside it,
	// matching the real adapters' error contract).
	Err error

	// Calls counts how many times Run was invoked.
	Calls int
	// LastSpec and LastInstance capture the most recent Run arguments so a
	// consumer can assert the harness wired the sandbox and task through.
	LastSpec     *eval.TaskSpec
	LastInstance *sandbox.Instance
}

FakeRunner is an in-memory Runner for downstream tests (harness-driver and beyond) that need to inject a scripted agent without shelling out to a real CLI. It returns its canned Result and Err verbatim, and records the last call's arguments so a consumer can assert what the harness passed.

This is the injectable fake the plan's TASKS.yaml notes call for ("v1 ships a fakeRunner for tests"): a fake Runner, distinct from the per-adapter exec seam. Script success, failure, or cancellation by setting Result and Err; leave both zero for a no-op success.

func (*FakeRunner) Run

func (f *FakeRunner) Run(
	_ context.Context,
	spec *eval.TaskSpec,
	instance *sandbox.Instance,
) (Result, error)

Run implements Runner. It records the call and returns the canned Result and Err. The context is accepted to satisfy the interface; FakeRunner performs no work, so it neither blocks on nor observes cancellation — a consumer that needs to script a cancel sets Err to a context error.

type Result

type Result struct {
	// Stdout is the captured standard output of the agent process.
	Stdout []byte
	// Stderr is the captured standard error of the agent process.
	Stderr []byte
	// ExitCode is the process exit code (0 = success).
	ExitCode int
	// Duration is wall-clock time from exec start to process exit.
	Duration time.Duration
	// Telemetry holds the identity and usage data the adapter extracted.
	Telemetry AgentTelemetry
}

Result is the complete output of one agent run.

type Runner

type Runner interface {
	// Run invokes the agent with the task prompt inside instance.Workdir,
	// appending instance.Env to the subprocess environment. The returned
	// Result carries stdout, stderr, exit code, wall time, and any telemetry
	// the adapter could extract. A non-zero exit code is NOT itself a Go
	// error — it is encoded in Result.ExitCode so the harness can
	// distinguish "the agent ran but produced wrong output" from "we could
	// not launch the agent at all".
	Run(ctx context.Context, spec *eval.TaskSpec, instance *sandbox.Instance) (Result, error)
}

Runner runs an agent against a TaskSpec inside a provisioned sandbox and returns the captured output and telemetry.

func New

func New(adapter Adapter) (Runner, error)

New returns a Runner for the named adapter. Unrecognised adapters return ErrUnknownAdapter.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL