Documentation
¶
Overview ¶
Package bench provides a Terminal Bench-style harness for measuring how well the BroCode agent loop solves real tasks: it runs a task in a throwaway sandbox project, then verifies the outcome with a script. It reports pass rate, mean task time, and mean context tokens consumed — the same axes the big players measure (quality, latency, cost).
A benchmark case is a JSON file (or inline struct) with:
{
"id": "fix-broken-import",
"prompt": "The project doesn't build. Fix it.",
"setup": "create a go file with a broken import", // shell script
"verify": "grep -q 'net/http' main.go", // shell script; exit 0 = pass
"maxIterations": 25
}
Usage (headless): build a small binary or test that constructs a Runner with a live provider adapter and calls RunCases.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func RenderReport ¶
RenderReport formats the summary as a compact table.
Types ¶
type Case ¶
type Case struct {
ID string `json:"id"`
Prompt string `json:"prompt"`
Setup string `json:"setup"` // shell script run in the sandbox before the agent
Verify string `json:"verify"` // shell script; exit 0 = pass
MaxIterations int `json:"maxIterations"`
}
Case is a single benchmark task.
type Report ¶
type Report struct {
Total int
Passed int
Failed int
PassRate float64
MeanDuration time.Duration
MeanTokens int
MeanCostUSD float64
PerCase []Result
}
Report summarizes a set of results.
type Result ¶
type Result struct {
ID string
Pass bool
Error string
Duration time.Duration
Iterations int
ToolCalls int // tool executions this turn (from persisted tool-result messages)
Tokens int // estimated context tokens consumed by the turn
CostUSD float64
Answer string
}
Result is the outcome of running one case.
type Runner ¶
type Runner struct {
Adapter provider.ProviderAdapter
Model string
// SandboxRoot is where each case gets its own temp subdirectory. Empty
// uses os.TempDir().
SandboxRoot string
// MaxIterations overrides the per-case loop cap (default 25).
MaxIterations int
// Timeout caps each case (default 10 minutes).
Timeout time.Duration
// Parallel runs cases concurrently (default sequential; keep 1 unless you
// trust the provider's rate limits).
Parallel bool
// Verbose prints per-case progress to stderr.
Verbose bool
}
Runner executes benchmark cases against a live adapter.
Source Files