Documentation
¶
Overview ¶
Package verifier defines the language-agnostic verifier contract for the R4 eval harness and houses the shared run engine every per-language verifier reuses. It holds the shared types — Verifier, VerifyResult, VerifyError, and Phase with its constants — plus BaseVerifier, the generic engine that runs a TaskSpec's verification commands.
Seam and interface ¶
Verifier is the interface the R4 harness driver binds to; concrete per-language adapters (the goverifier package under golang/, plus the sibling verifier-python and verifier-typescript packages) implement it without importing one another. Each adapter embeds BaseVerifier and contributes only its Language() identity via NewBase, so the run loop lives here once instead of being duplicated per language.
Command execution model ¶
BaseVerifier.Verify runs build_cmd first when present; a non-zero build exit short-circuits the test step so the harness does not spin up a test run against unbuilt code. test_cmd always runs if the build passes. The eval.Verification.TimeoutSeconds field, when non-zero, is applied as a context deadline over the combined build + test wall time. Both commands run in the sandbox workdir with the sandbox environment appended to the host environment so the agent's HOME and USERPROFILE are pinned to the scratch directory provisioned by the sandbox.
Result and error model ¶
A non-zero exit code is NOT returned as an error; it is encoded in VerifyResult.Passed and VerifyResult.ExitCode so the scoring bridge can record failure outcomes rather than treating them as harness faults. A VerifyError is returned only when a step could not start at all (context cancelled, binary not found, OS-level failure); it carries the Phase that failed and unwraps to the underlying cause.
Testability ¶
BaseVerifier holds a runCmd seam (unexported function-variable field) so tests can inject a deterministic command runner without invoking a real toolchain. Integration tests for the runProcess runner exercise the real exec path using a test-helper subprocess pattern.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type BaseVerifier ¶
type BaseVerifier struct {
// contains filtered or unexported fields
}
BaseVerifier is the language-agnostic run engine shared by every per-language verifier. It runs build_cmd then test_cmd from a TaskSpec, short-circuiting on build failure. The TimeoutSeconds field is applied as a context deadline when non-zero; no extra deadline is applied when it is zero. Per-language adapters embed *BaseVerifier and contribute only their Language() identity via NewBase, so the run loop lives here once instead of being duplicated per language.
func NewBase ¶
func NewBase(lang eval.Language) *BaseVerifier
NewBase returns a BaseVerifier for lang backed by the real process runner and exec.LookPath as the toolchain resolver.
func (*BaseVerifier) Language ¶
func (b *BaseVerifier) Language() eval.Language
Language implements Verifier.
func (*BaseVerifier) Verify ¶
func (b *BaseVerifier) Verify(ctx context.Context, spec *eval.TaskSpec, workdir string, env []string) (*VerifyResult, error)
Verify implements Verifier. A nil spec returns an immediate error. An empty workdir is rejected with a VerifyError (PhaseValidate) before any command runs — an empty exec.Cmd.Dir would default to the current process directory, escaping the sandbox. A negative TimeoutSeconds is also rejected; 0 means no timeout (documented contract). TimeoutSeconds, when positive, becomes a context deadline spanning both build and test. A build failure short-circuits the test step.
type Phase ¶
type Phase string
Phase identifies which verification step produced the current result.
const ( // PhaseBuild is the build_cmd step; set only when build fails and the // test step is short-circuited. PhaseBuild Phase = "build" // PhaseTest is the test_cmd step; set on both test pass and test failure. PhaseTest Phase = "test" // PhaseValidate is the pre-flight validation step; it is set on VerifyErrors // returned before any command runs (empty workdir, invalid spec fields). PhaseValidate Phase = "validate" )
type ToolchainError ¶
type ToolchainError struct {
// Language is the verifier language whose toolchain is missing.
Language eval.Language
// Binary is the logical command the TaskSpec asked for (argv[0]).
Binary string
// Tried lists the candidate invocations probed on PATH, in order.
Tried []string
}
ToolchainError reports that a language's required interpreter or compiler is not installed on PATH, so verification could not even begin. It is a DISTINCT error type from VerifyError (a build/test step that started and then failed): the harness surfaces it as a "toolchain unavailable" abort instead of scoring the run, so a missing python3 / node / tsc is never mistaken for the agent producing bad code. Match it with errors.As(err, *ToolchainError).
func (*ToolchainError) Error ¶
func (e *ToolchainError) Error() string
Error implements error with an actionable, operator-facing message.
type Verifier ¶
type Verifier interface {
// Language reports the language this verifier handles.
Language() eval.Language
// Verify runs the TaskSpec's verification commands in workdir with env
// appended to the host environment. A non-zero exit populates the
// returned VerifyResult (Passed=false); a VerifyError is returned only
// when a step could not start.
Verify(ctx context.Context, spec *eval.TaskSpec, workdir string, env []string) (*VerifyResult, error)
}
Verifier runs the verification commands from an eval.TaskSpec inside a sandbox working directory and returns a VerifyResult. It is the seam the R4 harness driver uses; per-language adapters sit behind this interface so the harness is language-agnostic.
type VerifyError ¶
type VerifyError struct {
// Phase is the step that could not start.
Phase Phase
// Cause is the underlying exec or context error.
Cause error
}
VerifyError wraps a non-exit-code failure from a verification step — context cancellation, command not found, or other OS-level start failure. Callers should use errors.As to distinguish this from a clean non-zero exit (which is encoded in VerifyResult without an error return).
func (*VerifyError) Error ¶
func (e *VerifyError) Error() string
Error implements the error interface.
func (*VerifyError) Unwrap ¶
func (e *VerifyError) Unwrap() error
Unwrap returns the underlying cause so errors.Is / errors.As traversal passes through VerifyError transparently.
type VerifyResult ¶
type VerifyResult struct {
// Passed is true only when all verification steps exit with code 0.
Passed bool
// Phase is the last step that ran: PhaseBuild on build short-circuit or
// PhaseTest on full pass or test failure.
Phase Phase
// ExitCode is the exit code of the last step that ran.
ExitCode int
// Stdout is the combined standard output of all steps that ran.
Stdout string
// Stderr is the combined standard error of all steps that ran.
Stderr string
// Duration is the total elapsed wall time of all verification steps.
Duration time.Duration
}
VerifyResult is the outcome of running a TaskSpec's verification commands inside a sandbox working directory. It is the canonical shape that the sibling verifier-python and verifier-typescript packages mirror (the R4 TASKS.yaml note: "sequenced after verifier-go so the VerifyResult shape stabilizes first").
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
Package goverifier is the Go-language adapter for the R4 eval verifier (R4 spec task t-verifier-iface).
|
Package goverifier is the Go-language adapter for the R4 eval verifier (R4 spec task t-verifier-iface). |
|
Package pyverifier is the Python-language adapter for the R4 eval verifier (R4 spec task t-verifier-iface).
|
Package pyverifier is the Python-language adapter for the R4 eval verifier (R4 spec task t-verifier-iface). |
|
Package tsverifier is the TypeScript-language adapter for the R4 eval verifier (R4 spec task t-verifier-iface).
|
Package tsverifier is the TypeScript-language adapter for the R4 eval verifier (R4 spec task t-verifier-iface). |