bench

package
v0.1.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 7, 2026 License: MIT Imports: 16 Imported by: 0

Documentation

Overview

Package bench measures how a model+config combination behaves when driving bee through real coding tasks. It runs each task through the real binary's headless /goal loop, scores success/efficiency/format, and emits a JSON scoreboard. It never mutates bee config — improving bee is the caller's job.

Index

Constants

View Source 
const FinalMessageFile = ".bee_final_message"

FinalMessageFile is the sandbox file the runner writes the model's last assistant text into. Abstain/refusal tasks grep this to confirm the model said it cannot do the request instead of fabricating a result.

Variables

View Source 
var DefaultWeights = Weights{Success: 0.6, Format: 0.25, Efficiency: 0.15}

DefaultWeights — success matters most, format is the next most common small-model failure, efficiency is a tiebreaker.

Functions

func AppendLedger 

func AppendLedger(path string, rec LedgerRecord) error

AppendLedger appends one JSON line to the ledger at path, creating parent dirs as needed. Empty path disables logging. A ledger write must never fail the run, so callers report the error and continue.

func AttachHoldout 

func AttachHoldout(ctx context.Context, res *SuiteResult, dir string, opt Options) error

AttachHoldout runs the held-out task slice with the same options and folds its aggregate into res under the Holdout* fields. The held-out tasks reuse the exact scoring/checks machinery — this only relabels and segregates the scoreboard so a tuner can compare main-suite delta vs held-out delta in one run. When dir is missing or empty, res is left untouched (backward compatible).

func WriteJSON 

func WriteJSON(res SuiteResult, dir string) (string, error)

WriteJSON persists the scoreboard to dir/<label>.json for the tune skill to diff across runs. No timestamp baked into the content — file mtime carries time, keeping diffs clean.

func WriteTable 

func WriteTable(res SuiteResult, w io.Writer)

WriteTable prints a human scoreboard. With repeats it appends the spread (max−min across runs) so a reader can see how noisy each score is. When a held-out slice ran, it prints below the main suite as a separate section.

Types

type Budget 

type Budget struct {
	MaxTurns int `json:"max_turns"`
}

Budget is the denominator for the efficiency score, not a hard cap: the headless goal loop runs under its own default caps regardless. A task that expects ~6 turns sets MaxTurns:6 so a run taking more scores lower on efficiency. Token usage isn't in the session transcript the harness reads, so only turns can be scored.

type Check 

type Check struct {
	Kind       string `json:"kind"`                  // "cmd" | "grep"
	Run        string `json:"run,omitempty"`         // cmd: shell line, $SANDBOX expanded
	ExpectExit int    `json:"expect_exit,omitempty"` // cmd: required exit code
	File       string `json:"file,omitempty"`        // grep: target file, $SANDBOX expanded
	Pattern    string `json:"pattern,omitempty"`     // grep: regex that must match
}

Check is one objective success assertion run against the sandbox.

type CheckResult 

type CheckResult struct {
	Kind   string `json:"kind"`
	Passed bool   `json:"passed"`
	Detail string `json:"detail,omitempty"`
}

CheckResult records one assertion's outcome.

func RunChecks 

func RunChecks(checks []Check, sandbox string, timeout time.Duration) (results []CheckResult, allPassed bool)

RunChecks evaluates every check against sandbox. $SANDBOX in cmd/file fields is expanded to the sandbox path. allPassed is true only when every check passes (binary success — partial credit invites gaming). timeout caps each cmd check so a hung assertion (e.g. a test that loops) can't stall the suite.

type Dims 

type Dims struct {
	Success    float64 `json:"success"`
	Format     float64 `json:"format"`
	Efficiency float64 `json:"efficiency"`
}

Dims holds the three normalized 0-1 dimension scores.

func Score 

func Score(b Budget, m RunMetrics, succeeded bool, w Weights) (Dims, float64)

Score computes the per-dimension scores and the weighted total (0-100). 

success    — 1.0 if the task demonstrably worked, else 0.0 (binary).
format     — fraction of tool calls that did not error, with a clean-stop
             penalty when the loop hit a cap instead of finishing.
efficiency — how far under the turn budget the run stayed.

type LedgerRecord 

type LedgerRecord struct {
	Time             string  `json:"time"`
	Label            string  `json:"label"`
	Provider         string  `json:"provider,omitempty"`
	Model            string  `json:"model,omitempty"`
	Profile          string  `json:"profile,omitempty"`
	Config           string  `json:"config,omitempty"`
	Suite            string  `json:"suite"`
	Tasks            int     `json:"tasks"`
	Runs             int     `json:"runs"`
	Aggregate        float64 `json:"aggregate"`
	Success          float64 `json:"success"`
	Format           float64 `json:"format"`
	Efficiency       float64 `json:"efficiency"`
	HoldoutAggregate float64 `json:"holdout_aggregate,omitempty"`
	ResultsJSON      string  `json:"results_json"`
}

LedgerRecord is one row in the append-only benchmark ledger: enough to rank and audit a run without reopening its results JSON.

type Options 

type Options struct {
	BeeBin     string        // path to the bee binary under test
	Provider   string        // --provider passthrough (empty = inherit config)
	Model      string        // --model passthrough (empty = inherit config)
	ConfigFile string        // BEE_CONFIG overlay path (empty = default ~/.bee/config.toml)
	Profile    string        // BEE_PROFILE override (empty = inherit/auto)
	Label      string        // tags the result set (config-variant identifier)
	Weights    Weights       // scoring blend
	Timeout    time.Duration // per-task wall-clock cap
	Runs       int           // repeats per task for variance (0/1 = single shot)
	RolloutDir string        // when set, persist blessed (passed + clean) session jsonl here for fine-tune harvest
	ExtraEnv   []string      // extra env (offline tests inject the scripted provider here)
	Progress   io.Writer     // when set, stream timestamped per-task/per-run progress here (nil = silent)
}

Options configures a suite run.

type RunMetrics 

type RunMetrics struct {
	Turns        int // assistant messages
	ToolCalls    int // tool_use blocks
	ErroredCalls int // tool_result blocks with is_error (unknown tool, bad input, failed exec)
	StoppedClean bool
}

RunMetrics is what we mine from a run's session transcript. Pure counting — no judgement here.

func MetricsFromMessages 

func MetricsFromMessages(msgs []types.Message, stoppedClean bool) RunMetrics

MetricsFromMessages counts turns and tool activity from a transcript. stoppedClean reflects whether the goal loop ended on its own ("✓ goal achieved" / clean stop) vs hitting a cap — the caller knows that, so it is passed in rather than inferred from messages.

type SuiteResult 

type SuiteResult struct {
	Label            string       `json:"label"`
	Runs             int          `json:"runs"`
	Aggregate        float64      `json:"aggregate"`
	MeanSpread       float64      `json:"mean_spread,omitempty"`
	DimMeans         Dims         `json:"dim_means"`
	Tasks            []TaskResult `json:"tasks"`
	HoldoutAggregate float64      `json:"holdout_aggregate,omitempty"`
	HoldoutDimMeans  Dims         `json:"holdout_dim_means,omitempty"`
	HoldoutTasks     []TaskResult `json:"holdout_tasks,omitempty"`
}

SuiteResult is the full scoreboard. MeanSpread is the average per-task score spread — a single noise figure for the whole run. Holdout* mirror the main fields for the never-tuned slice; they stay zero/empty when no held-out dir is present, keeping output backward compatible.

func RunSuite 

func RunSuite(ctx context.Context, tasks []Task, opt Options) (SuiteResult, error)

RunSuite runs every task serially and scores it. Serial keeps results reproducible — small models are latency-bound, so parallelism buys little.

type Task 

type Task struct {
	ID     string  `json:"id"`
	Prompt string  `json:"prompt"`           // goal condition handed to bee as "/goal <prompt>"
	Setup  string  `json:"setup,omitempty"`  // optional shell to scaffold $SANDBOX
	Checks []Check `json:"checks,omitempty"` // objective success assertions; when empty, success falls back to the goal loop's own (self-graded) verdict
	Judge  string  `json:"judge,omitempty"`  // human-readable success criterion documenting author intent; not executed — checks are authoritative
	Budget Budget  `json:"budget"`
}

Task is one benchmark scenario.

func LoadTasks 

func LoadTasks(dir string) ([]Task, error)

LoadTasks reads every *.json task spec in dir, sorted by id for stable runs.

type TaskResult 

type TaskResult struct {
	ID        string        `json:"id"`
	Score     float64       `json:"score"`
	Spread    float64       `json:"spread,omitempty"`
	Samples   []float64     `json:"samples,omitempty"`
	Dims      Dims          `json:"dims"`
	Succeeded bool          `json:"succeeded"`
	Metrics   RunMetrics    `json:"metrics"`
	Checks    []CheckResult `json:"checks,omitempty"`
	Reason    string        `json:"reason"`
	Err       string        `json:"error,omitempty"`
}

TaskResult is one task's full outcome. With Runs>1, Score/Dims are means across repeats, Spread is max−min of the per-run scores, and Samples holds each run's score so a tuner can judge whether a delta clears the noise.

type Weights 

type Weights struct {
	Success    float64
	Format     float64
	Efficiency float64
}

Weights blend the three dimensions into a single 0-100 score. success dominates: a model that thrashes but never works should score low regardless.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.