Documentation
¶
Overview ¶
Package scoringbridge translates a completed eval run into the R1 scoring pipeline: it emits an R1-shaped iteration record and scores it with the production rubric, persisting both under the run's eval-namespaced iteration-log directory.
Two invariants from the R4 spec (.agents/workflow/specs/r4-code-task-generation-eval/design.md) anchor this package:
D4.4 — eval outcomes feed R1 unchanged. The bridge scores with scoring.DefaultRubric() through the same AssembleSignalSet → Rubric.Score → WriteIterationScore path production scoring uses. There is no eval-specific rubric, no parallel scoring path, and no new signal; eval is just another input source, so eval scores stay comparable to production scores under the same RubricVersion.
D4.6 — eval-namespaced iter-log root. Eval runs write ONLY under .agents/eval/runs/<run-id>/iteration-log/, never into the active orchestration log (.agents/active/iteration-log/): eval iterations are not real wave iterations, and mixing them would pollute orchestration history. Eval and production share the rubric but not the iter-log space. The bridge enforces the separation structurally — it derives the iteration-log directory from the run directory it is handed and never accepts an arbitrary log-dir path.
Per spec OQ2 (recommendation applied) v1 scoring is 1-shot: one eval run produces exactly one iteration entry (iter-1.yaml) and one score sidecar (iter-1.score.yaml); aggregation across repeated runs of the same TaskSpec is a v2 concern that wraps this package rather than living inside it.
The emitted record is schema_version 2 of the workflow iter-log schema (schemas/workflow-iter-log.schema.json), so the existing loaders — scoring.LoadIterationLog and the `da score iteration` CLI — read the eval space without modification (spec requirement R5).
record.go is the iteration-record emitter: it renders an EvalRun as a schema_version-2 workflow iter-log entry (schemas/workflow-iter-log. schema.json) and provides the matching in-memory translation to scoring.IterationRecord.
The emit shapes below are the write-side twin of scoring's read-side rawV2 shape: field names and YAML tags must stay aligned with the schema so that scoring.LoadIterationLog — and therefore `da score iteration` (spec R5) — parses the eval space byte-for-byte the way production entries parse. The round-trip is pinned by tests: emitRecord.toIterationRecord() must equal what scoring.ParseIterationRecord yields from the marshaled bytes.
Index ¶
Constants ¶
This section is empty.
Variables ¶
var ( // ErrEmptyRunID is returned by ScoreRun when the run carries no run ID. ErrEmptyRunID = errors.New("scoringbridge: run id is required") // ErrEmptyRunDir is returned by ScoreRun when the run names no run // directory — without it there is no eval-namespaced iter-log root to // write into. ErrEmptyRunDir = errors.New("scoringbridge: run dir is required") // ErrNilTaskSpec is returned by ScoreRun when the run carries no task // spec. ErrNilTaskSpec = errors.New("scoringbridge: task spec is nil") )
Typed errors callers can match with errors.Is.
Functions ¶
func IterationLogDir ¶
IterationLogDir returns the eval-namespaced iteration-log directory for a run dir (<runDir>/iteration-log). It is the only iter-log location this package ever writes to: deriving it from the run dir — rather than accepting an arbitrary log dir — is how the bridge structurally upholds spec invariant D4.6 (eval runs never write into the active orchestration log).
Types ¶
type AgentTelemetry ¶
type AgentTelemetry struct {
// SessionID is the platform session UUID, when the runner captured one.
SessionID string
// Harness is the agent platform harness (claude-code, codex, ...).
Harness string
// Model is the model ID the run used.
Model string
// Retries is how many times the runner had to re-invoke the agent; it
// feeds the rubric's correction-pressure signal.
Retries int
// Tokens is the run's token usage; nil when the runner captured none,
// which leaves the token-efficiency signal absent (absent is
// first-class — the rubric renormalizes).
Tokens *scoring.TokenUsage
}
AgentTelemetry is the agent-runner identity and usage telemetry for one eval run (R4 spec requirement R9: platform, model, and session identity are captured so the platform-tuner persona can diff platforms). All fields are optional; what is present flows into the iteration record's agent / session_tokens blocks and from there into the rubric's token-efficiency signal.
type EvalRun ¶
type EvalRun struct {
// RunID uniquely identifies the run (sandbox.Instance.RunID).
RunID string
// RunDir is the run's sidecar directory (sandbox.Instance.RunDir,
// .agents/eval/runs/<run-id>). The iteration record and score sidecar
// are written under <RunDir>/iteration-log/ per spec invariant D4.6.
RunDir string
// BaseCommit is the source-repo commit the sandbox was provisioned at,
// recorded on the iteration entry for reproducibility (spec R10).
BaseCommit string
// Spec is the task the run executed. Required and must validate.
Spec *eval.TaskSpec
// Agent is the runner identity + usage telemetry (spec R9).
Agent AgentTelemetry
// Verify is the sandbox verification outcome.
Verify VerifyResult
// FinishedAt timestamps the run for the record's date / checkpoint_at
// fields; the zero value falls back to the current time.
FinishedAt time.Time
}
EvalRun is the bridge's input: everything one completed eval run produced. The harness driver assembles it from the sandbox instance (RunID, RunDir, BaseCommit), the generated TaskSpec, the agent-runner telemetry, and the verifier outcome.
type Result ¶
type Result struct {
// IterLogDir is the eval-namespaced iteration-log dir the artifacts
// were written into.
IterLogDir string
// RecordPath is the persisted iteration record (iter-1.yaml).
RecordPath string
// ScorePath is the persisted score sidecar (iter-1.score.yaml).
ScorePath string
// Record is the scored iteration record, exactly as a re-load of
// RecordPath through scoring.LoadIterationLog would yield it.
Record scoring.IterationRecord
// Signals is the assembled signal set the rubric scored, including the
// parallel integrity / objective outputs that never reach the numeric
// score.
Signals scoring.SignalSet
// Score is the rubric-applied outcome, already persisted at ScorePath.
Score scoring.Score
}
Result reports everything ScoreRun persisted and computed for one run.
func ScoreRun ¶
ScoreRun bridges one completed eval run into R1: it emits the R1-shaped iteration record into <RunDir>/iteration-log/iter-1.yaml, assembles the production signal set, scores it with scoring.DefaultRubric(), and persists the score sidecar via scoring.WriteIterationScore.
A failed verification still produces a score sidecar — the failure is encoded as a 0-valued verifier/tests signal, not a missing file (spec done criterion 8). The rubric and its version are the production ones (spec invariant D4.4): nothing here forks the scoring path.
type VerifyResult ¶
type VerifyResult struct {
// BuildRan reports whether the TaskSpec's build_cmd was executed; false
// when the spec has no build_cmd or the harness never reached it.
BuildRan bool
// BuildPassed reports the build_cmd outcome; meaningful only when
// BuildRan.
BuildPassed bool
// TestRan reports whether the TaskSpec's test_cmd was executed. When
// false the test verifier is recorded with status "unknown", which the
// rubric excludes from the verifier mean — "never checked" is not the
// same as "failed".
TestRan bool
// TestPassed reports the test_cmd outcome; meaningful only when TestRan.
TestPassed bool
}
VerifyResult is the bridge's view of the sandbox verification outcome: did the TaskSpec's build and test commands run, and did they pass. It is the translation input for the record's verifiers block; the full stdout/stderr capture stays with the verifier packages and the eval-run sidecar — the rubric only consumes pass/fail.
Source Files