adapter

type CLI ¶

type CLI struct {
	// Binary is the built buildmax executable under evaluation.
	Binary string
	// Credential is the provider access written into each trial home.
	Credential ModelAccess
	// Retention is how much free text bundles keep. Callers exporting a bundle
	// lower it; the default keeps replies for local diagnosis.
	Retention contract.RetentionLevel
}

CLI runs trials through a built buildmax binary. Nothing in this type reaches the agent runtime as a library: section 7.3 makes the shipped artifact the benchmark interface, so what is measured is what a user would run.

func (c *CLI) Describe() (contract.Surface, int)

func (c *CLI) Run(ctx context.Context, tr Trial, bundleRoot string) (Result, error)

type Executor ¶

type Executor interface {
	Run(ctx context.Context, tr Trial, bundleRoot string) (Result, error)
	// Describe reports the surface this executor runs and its own version.
	// Both belong to the subject's identity: two adapters reaching the same
	// build are two execution paths, and section 8.2 makes the path part of
	// what was measured rather than a detail of how it was measured.
	Describe() (contract.Surface, int)
}

Executor runs one trial through a built artifact and returns its evidence.

Both implementations run a shipped binary rather than the runtime as a library, which is what section 7.3 asks of an authoritative adapter. The interface exists so a suite can hold tasks for more than one surface: a cross-surface parity case is two tasks stating the same goal, and it can only be run if the runner can dispatch each to the surface its task names.

type ModelAccess ¶

type ModelAccess struct {
	APIURL string
	APIKey string
	// Pricing lets a run report what it spent. Absent, the trial reports its
	// cost as unavailable rather than as zero — BuildMax does not know what any
	// provider charges, and guessing would be worse than silence.
	Pricing *Pricing
}

ModelAccess is how a trial reaches its model and what that costs. It travels beside the subject rather than inside it, for two different reasons that both end in the same place.

The credential is section 8.2: a stored trial result must never become a credential store. The prices are section 12: they change what a run reports spending without changing anything it did, so two subjects at different prices are the same subject and must compare as one.

type Pricing ¶

type Pricing struct {
	Currency          string `yaml:"currency"`
	InputPerMTok      string `yaml:"input_per_mtok,omitempty"`
	CacheReadPerMTok  string `yaml:"cache_read_per_mtok,omitempty"`
	CacheWritePerMTok string `yaml:"cache_write_per_mtok,omitempty"`
	OutputPerMTok     string `yaml:"output_per_mtok,omitempty"`
}

Pricing is what one model charges, in the shape settings.yaml takes. Rates are strings for the reason internal/config keeps them as strings: a price is a decimal, and parsing one through a float loses the last digits of a rate quoted in millionths.

type Result ¶

type Result struct {
	Bundle   contract.TrialBundle
	Gradable bool
	// Workspace is the final state, kept until Cleanup so graders can read it.
	Workspace string
	// TrialDir is the bundle directory holding the trace and artifacts.
	TrialDir string
	Cleanup  func()
}

Result is one execution, before grading.

Bundle.Status carries a terminal status only when execution itself decided the outcome. When Gradable is true the status is not yet meaningful: the caller runs the task's graders against Workspace, then sets it from contract.DecideStatus. Splitting it this way keeps the adapter out of the judgement business, which is what lets a grader failure stay distinguishable from an agent failure.

type Trial ¶

type Trial struct {
	Task         contract.Task
	TaskDir      string
	Subject      contract.SubjectManifest
	ExperimentID string
	TrialID      string
	Index        int
}

Trial is one attempt's inputs.

type Worker ¶

type Worker struct {
	// Binary is the built buildmax-worker executable under evaluation.
	Binary string
	// Credential is the provider access written into the run's server.yaml.
	Credential ModelAccess
	// Retention is how much free text bundles keep.
	Retention contract.RetentionLevel
	// SpaceID and UserID scope the run's directories. They are identifiers in a
	// control plane no real deployment sees, so they only need to be stable.
	SpaceID string
	UserID  string
}

Worker runs trials through the built buildmax-worker binary.

The worker is dispatched the way a scheduler dispatches it — a run id on the command line, a run token in the environment, and a server to fetch the run from — against a control plane this adapter serves. What it exercises that the CLI adapter cannot is the part of the product only a worker has: materializing the space's persistent workspace into a run-scoped directory, executing with no interactive surface, and reporting an outcome over the API rather than to a terminal.

func (w *Worker) Describe() (contract.Surface, int)

func (w *Worker) Run(ctx context.Context, tr Trial, bundleRoot string) (Result, error)