Documentation
¶
Overview ¶
Package adapter runs a trial against a built BuildMax artifact and returns a canonical bundle. It is the black-box half of the evaluation system: nothing here calls the agent runtime in process, because an in-process helper cannot tell whether the shipped binary behaves the way the library does.
Index ¶
- Constants
- func DigestDir(root string) (string, error)
- func Materialize(taskDir, workspace string) error
- func MaterializedRoot(workspace string, surface contract.Surface) string
- func VerifyBoundary(taskDir, workspace string) ([]string, error)
- func WriteHome(dir string, subject contract.SubjectManifest, cred Credential) error
- type CLI
- type Credential
- type Executor
- type Result
- type Trial
- type Worker
Constants ¶
const CLIAdapterVersion = 1
CLIAdapterVersion changes when this adapter changes how it invokes the subject. It is recorded on the subject manifest because an adapter change moves results without the product moving, and a comparison spanning one is not paired.
const TracesDir = "traces"
TracesDir is where the runtime writes durable traces under a home. The adapter locates a trial's trace from the run's reported path rather than by rebuilding this layout; the constant exists for the fallback in Run, which says so where it uses it.
const WorkerAdapterVersion = 1
WorkerAdapterVersion changes when this adapter changes how it dispatches a run. Like the CLI's, it lands on the subject manifest: a comparison spanning an adapter change is not paired.
Variables ¶
This section is empty.
Functions ¶
func DigestDir ¶
DigestDir returns a content identity for a directory tree: the SHA-256 over every path, mode, and body, in sorted order.
Section 7.1 makes final state the authoritative outcome, and a digest is what lets a bundle carry that fact without carrying the workspace. Sorting is what makes it comparable at all — filesystem walk order differs between machines, so an unsorted digest would report every re-run as a changed outcome.
func Materialize ¶
Materialize copies a task's visible initial state into the trial workspace.
Only contract.StateDir is copied. The grader, the oracle, and the task definition stay in the task directory, which the trial never sees, so a task cannot leak its own answer by listing the wrong path: there is no path to list. A task with no state directory yields an empty workspace, which is valid — some tasks start from nothing.
func MaterializedRoot ¶
MaterializedRoot returns where a surface places the task's initial state inside the agent's workspace.
The two surfaces differ, and the difference is the product's, not this package's: a CLI run is given the workspace directly, while a worker materializes the team's persistent files into a `home` subdirectory of the run directory it works in. Anything that has to agree with a real run about where a file will be — preflight above all — asks here rather than assuming the CLI's layout.
A task therefore writes its path assertions for the surface it names. Section 11's cross-surface parity is two tasks stating the same goal, not one task run twice, which is what makes that acceptable rather than a defect.
func VerifyBoundary ¶
VerifyBoundary reports files in the workspace whose content matches hidden task material. It is the adversarial half of section 18.4: keeping graders out of the copy is the mechanism, and this is the check that the mechanism worked — including when the leak came from a task author committing the answer into the initial state rather than from the copier.
It returns the offending workspace paths, empty when the boundary held.
func WriteHome ¶
func WriteHome(dir string, subject contract.SubjectManifest, cred Credential) error
WriteHome builds the BUILDMAX_HOME one trial runs under, from the subject alone.
Building it rather than reusing the contributor's home is what makes the result attributable. Section 2.1 lists local settings, hooks, plugins, and permissions among the things that silently change what a benchmark measures; a home containing only the subject's model has none of them to inherit, and a plugin installed on the machine cannot reach a run that never looks there.
Types ¶
type CLI ¶
type CLI struct {
// Binary is the built buildmax executable under evaluation.
Binary string
// Credential is the provider access written into each trial home.
Credential Credential
// Retention is how much free text bundles keep. Callers exporting a bundle
// lower it; the default keeps replies for local diagnosis.
Retention contract.RetentionLevel
}
CLI runs trials through a built buildmax binary. Nothing in this type reaches the agent runtime as a library: section 7.3 makes the shipped artifact the benchmark interface, so what is measured is what a user would run.
type Credential ¶
Credential is the provider access a trial needs. It travels beside the subject rather than inside it because section 8.2 keeps secrets out of the manifest: a stored trial result must never become a credential store.
type Executor ¶
type Executor interface {
Run(ctx context.Context, tr Trial, bundleRoot string) (Result, error)
// Describe reports the surface this executor runs and its own version.
// Both belong to the subject's identity: two adapters reaching the same
// build are two execution paths, and section 8.2 makes the path part of
// what was measured rather than a detail of how it was measured.
Describe() (contract.Surface, int)
}
Executor runs one trial through a built artifact and returns its evidence.
Both implementations run a shipped binary rather than the runtime as a library, which is what section 7.3 asks of an authoritative adapter. The interface exists so a suite can hold tasks for more than one surface: a cross-surface parity case is two tasks stating the same goal, and it can only be run if the runner can dispatch each to the surface its task names.
type Result ¶
type Result struct {
Bundle contract.TrialBundle
Gradable bool
// Workspace is the final state, kept until Cleanup so graders can read it.
Workspace string
// TrialDir is the bundle directory holding the trace and artifacts.
TrialDir string
Cleanup func()
}
Result is one execution, before grading.
Bundle.Status carries a terminal status only when execution itself decided the outcome. When Gradable is true the status is not yet meaningful: the caller runs the task's graders against Workspace, then sets it from contract.DecideStatus. Splitting it this way keeps the adapter out of the judgement business, which is what lets a grader failure stay distinguishable from an agent failure.
type Trial ¶
type Trial struct {
Task contract.Task
TaskDir string
Subject contract.SubjectManifest
ExperimentID string
TrialID string
Index int
}
Trial is one attempt's inputs.
type Worker ¶
type Worker struct {
// Binary is the built buildmax-worker executable under evaluation.
Binary string
// Credential is the provider access written into the run's server.yaml.
Credential Credential
// Retention is how much free text bundles keep.
Retention contract.RetentionLevel
// TeamID and UserID scope the run's directories. They are identifiers in a
// control plane no real deployment sees, so they only need to be stable.
TeamID string
UserID string
}
Worker runs trials through the built buildmax-worker binary.
The worker is dispatched the way a scheduler dispatches it — a run id on the command line, a run token in the environment, and a server to fetch the run from — against a control plane this adapter serves. What it exercises that the CLI adapter cannot is the part of the product only a worker has: materializing the team's persistent workspace into a run-scoped directory, executing with no interactive surface, and reporting an outcome over the API rather than to a terminal.
Source Files