perfbench

package
v0.4.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 17, 2026 License: MIT Imports: 20 Imported by: 0

Documentation

Index

Constants

View Source 
const SchemaVersion = 3
View Source 
const TaskSchemaVersion = 1

TaskSchemaVersion is the schema version of a published task-benchmark result. Bump it whenever the TaskRunResult shape changes so consumers (docs/BENCHMARK.md tooling, dashboards) can detect format drift.

View Source 
const TurnSchemaVersion = 2

TurnSchemaVersion is the schema version of a published turn-benchmark result. Bump when the TurnBenchResult shape changes so consumers can detect drift.

v2 splits pass/fail into three oracle tiers so the report cannot be misread as a blanket correctness verdict: tasksVerified/tasksPassed/correctnessPassRate cover only the positive-oracle classes (edit/fix); buildCheckedTasks/ buildPassedTasks/buildPassRate cover the non-positive build-check classes (refactor); latencyOnlyTasks covers the no-oracle classes (nav/longproc/ longctx/parallel). tasksAttempted is still every task run.

Variables

View Source 
var DefaultThresholds = Thresholds{
	ColdStartP95Ms:     300,
	FirstOutputP95Ms:   500,
	HarnessEndRssMaxMb: 256,
}

Functions

func EmitWarnings 

func EmitWarnings(w io.Writer, result Result)

func EscapeActionCommand 

func EscapeActionCommand(value string) string

func FormatCommand 

func FormatCommand(command []string) string

func FormatMetric 

func FormatMetric(value float64, unit string) string

func FormatSummary 

func FormatSummary(result Result) string

func FormatTaskSummary 

func FormatTaskSummary(result TaskRunResult) string

FormatTaskSummary renders a human-readable summary that foregrounds the honest framing: the model and the self-correct setting are printed alongside the score, because the number is model-bounded.

func FormatTurnBenchSummary added in v0.4.0 

func FormatTurnBenchSummary(result TurnBenchResult) string

FormatTurnBenchSummary renders a human-readable turn-benchmark summary that names the top controllable latency sources — the baseline's "do not proceed until" criterion.

func MeasureColdStart 

func MeasureColdStart(ctx context.Context, command []string) (float64, error)

func MeasureFirstOutput 

func MeasureFirstOutput(ctx context.Context, command []string) (firstOutputSample, error)

func ResolveBinary added in v0.4.0 

func ResolveBinary(explicit string) (string, error)

ResolveBinary locates the zero binary for a benchmark run: an explicit path when provided, else a `zero` (or zero.exe) on PATH, else a binary built into the repo root. Returns an error when none is found.

func ResolveColdStartCommand 

func ResolveColdStartCommand(rootDir string) ([]string, error)

func ResolveFirstOutputCommand 

func ResolveFirstOutputCommand(rootDir string) ([]string, error)

func RoundMetric 

func RoundMetric(value float64) float64

func WriteJSON 

func WriteJSON(w io.Writer, result Result) error

func WriteTaskJSON 

func WriteTaskJSON(w io.Writer, result TaskRunResult) error

WriteTaskJSON writes the indented JSON form of a benchmark result.

func WriteTurnBenchJSON added in v0.4.0 

func WriteTurnBenchJSON(w io.Writer, result TurnBenchResult) error

WriteTurnBenchJSON writes the indented JSON form of a turn-benchmark result.

Types

type BenchTask 

type BenchTask struct {
	ID                  string   `json:"id"`
	Name                string   `json:"name,omitempty"`
	Class               string   `json:"class,omitempty"`
	Prompt              string   `json:"prompt"`
	WorkspaceFixture    string   `json:"workspaceFixture,omitempty"`
	VerificationCommand []string `json:"verificationCommand,omitempty"`
}

BenchTask is one benchmark task. WorkspaceFixture is the relative path of the task's starting workspace; VerificationCommand (optional) is the command the default runner executes to decide pass/fail after ZERO finishes. Class groups the task for the turn-benchmark's per-group latency breakdown (e.g. "nav", "edit", "fix"); it is optional and ignored by the pass/fail runner.

type ClassSummary added in v0.4.0 

type ClassSummary struct {
	Tasks       int                `json:"tasks"`
	Verified    int                `json:"verified"`
	Passed      int                `json:"passed"`
	LatencyOnly int                `json:"latencyOnly"`
	WallMs      NumericStats       `json:"wallMs"`
	SpanTotals  map[string]float64 `json:"spanTotals"`
}

ClassSummary is the per-class (task group) roll-up.

Passed is the count that passed THIS class's oracle: a correctness pass for edit/fix, a build pass for refactor, and always 0 for the no-oracle classes (nav/longproc/longctx/parallel). Verified is how many tasks in the class carry an oracle (correctness or build); LatencyOnly is how many carry none. A latency-only class therefore reports Passed=0, Verified=0, LatencyOnly=Tasks.

type LatencySource added in v0.4.0 

type LatencySource struct {
	Span    string  `json:"span"`
	TotalMs float64 `json:"totalMs"`
	Share   float64 `json:"share"`
}

LatencySource is one of the top controllable latency sources, ranked by total exclusive time across the whole run. Share is its fraction of total exclusive span time. Because exclusive time subtracts nested children, concurrent and nested spans no longer double-count, so shares of the top sources sum to ~1 for a well-instrumented run.

type Metrics 

type Metrics struct {
	ColdStartMs          NumericStats `json:"coldStartMs"`
	FirstOutputMs        NumericStats `json:"firstOutputMs"`
	ProcessDrainMs       NumericStats `json:"processDrainMs"`
	HarnessEndRssMb      NumericStats `json:"harnessEndRssMb"`
	HarnessEndRssDeltaMb NumericStats `json:"harnessEndRssDeltaMb"`
}

type NumericStats 

type NumericStats struct {
	Samples []float64 `json:"samples"`
	Min     float64   `json:"min"`
	Median  float64   `json:"median"`
	Average float64   `json:"average"`
	P95     float64   `json:"p95"`
	Max     float64   `json:"max"`
}

func SummarizeSamples 

func SummarizeSamples(samples []float64) NumericStats

type Options 

type Options struct {
	Iterations         int
	WarmupIterations   int
	Thresholds         Thresholds
	ColdStartCommand   []string
	FirstOutputCommand []string
}

type Platform 

type Platform struct {
	OS        string `json:"os"`
	Arch      string `json:"arch"`
	GoVersion string `json:"goVersion"`
	Harness   string `json:"harness"`
}

func CurrentPlatform 

func CurrentPlatform() Platform

type Result 

type Result struct {
	SchemaVersion       int        `json:"schemaVersion"`
	Timestamp           string     `json:"timestamp"`
	Platform            Platform   `json:"platform"`
	ColdStartCommand    []string   `json:"coldStartCommand"`
	FirstOutputCommand  []string   `json:"firstOutputCommand"`
	Iterations          int        `json:"iterations"`
	WarmupIterations    int        `json:"warmupIterations"`
	Thresholds          Thresholds `json:"thresholds"`
	Metrics             Metrics    `json:"metrics"`
	BenchmarkDurationMs float64    `json:"benchmarkDurationMs"`
	Warnings            []Warning  `json:"warnings"`
}

func Run 

func Run(ctx context.Context, options Options) (Result, error)

type RunContext 

type RunContext struct {
	Model       string
	Mode        string
	SelfCorrect bool
}

RunContext is the immutable per-run context handed to the runner for each task, so a runner records the same model/mode/self-correct values the result is stamped with.

type SpanStats added in v0.4.0 

type SpanStats struct {
	Count    int     `json:"count"`
	TotalMs  float64 `json:"totalMs"`
	MedianMs float64 `json:"medianMs"`
	P95Ms    float64 `json:"p95Ms"`
	MaxMs    float64 `json:"maxMs"`
}

SpanStats summarizes one span's duration across all measured task iterations.

type TaskConfig 

type TaskConfig struct {
	Model       string
	Mode        string
	SelfCorrect bool
	Version     string
	Commit      string
	// Now overrides the clock for the recorded date (tests inject a fixed time).
	Now func() time.Time
	// Runner executes one task and reports the outcome. Required. The default
	// production runner (NewExecRunner) invokes headless `zero exec`.
	Runner TaskRunner
}

TaskConfig is the recorded configuration of a benchmark run. The honest framing matters: the model, mode, and self-correct flag are captured because the score is largely model-bounded — the point of the benchmark is to show the self-correct loop's contribution (the with/without delta), not the raw digit.

type TaskOutcome 

type TaskOutcome struct {
	Passed bool
	Detail string
	Err    error
}

TaskOutcome is what a runner reports for one task. Err is reserved for the run failing to execute (e.g. the agent process crashed); Passed reports whether the task's verification succeeded. A non-nil Err is always a non-pass.

func SkipRunner 

func SkipRunner(context.Context, BenchTask, RunContext) TaskOutcome

SkipRunner is a runner that performs no work and records every task as skipped (not passed, no error). It backs the harness's --dry-run mode, which exercises the full record path without invoking an agent or a model.

type TaskResult 

type TaskResult struct {
	ID      string `json:"id"`
	Name    string `json:"name,omitempty"`
	Passed  bool   `json:"passed"`
	Errored bool   `json:"errored,omitempty"`
	Detail  string `json:"detail,omitempty"`
}

TaskResult is the per-task entry in a TaskRunResult.

type TaskRunResult 

type TaskRunResult struct {
	SchemaVersion  int          `json:"schemaVersion"`
	Suite          string       `json:"suite"`
	SuiteName      string       `json:"suiteName,omitempty"`
	Model          string       `json:"model"`
	Mode           string       `json:"mode,omitempty"`
	SelfCorrect    bool         `json:"selfCorrect"`
	Version        string       `json:"version,omitempty"`
	Commit         string       `json:"commit,omitempty"`
	Date           string       `json:"date"`
	TasksAttempted int          `json:"tasksAttempted"`
	TasksPassed    int          `json:"tasksPassed"`
	Errors         int          `json:"errors"`
	PassRate       float64      `json:"passRate"`
	Tasks          []TaskResult `json:"tasks"`
}

TaskRunResult is the publishable benchmark record. It is intentionally self-describing: model + version + commit + self-correct flag + date are all embedded so a copy of this JSON is reproducible and auditable on its own.

func RunTasks 

func RunTasks(ctx context.Context, set TaskSet, config TaskConfig) (TaskRunResult, error)

RunTasks executes every task in the set with the configured runner and returns a self-describing result record. It never aborts on a single task failure or runner error — every task is attempted and recorded — so one bad task can't hide the rest of the run. The context cancels the whole run.

type TaskRunner 

type TaskRunner func(ctx context.Context, task BenchTask, rc RunContext) TaskOutcome

TaskRunner runs a single benchmark task and returns its outcome.

func NewExecRunner 

func NewExecRunner(binary string, extraArgs ...string) TaskRunner

NewExecRunner builds the production runner: it invokes headless `zero exec` with stream-json output for each task and decides pass/fail. binary is the path to the `zero` binary; extraArgs are appended to every invocation (e.g. sandbox flags). The self-correct flag from RunContext is translated into the exec invocation so the recorded config matches what actually ran.

Pass/fail is decided from the stream-json run_end event's exit code: a zero exit means the agent completed the task without error. When a task carries a VerificationCommand, that command is run after the agent and its exit status is authoritative (this mirrors Terminal-Bench's external verifier model).

type TaskSet 

type TaskSet struct {
	ID          string      `json:"id"`
	Name        string      `json:"name,omitempty"`
	Description string      `json:"description,omitempty"`
	Tasks       []BenchTask `json:"tasks"`
	// BuildOnlyClasses lists task classes whose verificationCommand is a
	// non-positive build check (e.g. refactor's `go build ./...`): it proves the
	// edit compiles, not that the refactor achieved its goal. The turn benchmark
	// reports these separately from correctness oracles so a build-pass cannot be
	// misread as a correctness pass. Classes with a verificationCommand that are
	// NOT listed here are treated as correctness classes; classes whose tasks
	// carry no verificationCommand are latency-only regardless of this list.
	BuildOnlyClasses []string `json:"buildOnlyClasses,omitempty"`
}

TaskSet is a reproducible benchmark task list. It mirrors the shape of a Terminal-Bench task manifest: each task is an isolated workspace plus a prompt ZERO must satisfy. The set is recorded by ID with every result so a published number is traceable to the exact tasks that produced it.

func LoadTaskSet 

func LoadTaskSet(path string) (TaskSet, error)

LoadTaskSet reads and validates a benchmark task set from a JSON file. Unknown fields are rejected so a malformed manifest fails loudly rather than silently dropping tasks.

type Thresholds 

type Thresholds struct {
	ColdStartP95Ms     float64 `json:"coldStartP95Ms"`
	FirstOutputP95Ms   float64 `json:"firstOutputP95Ms"`
	HarnessEndRssMaxMb float64 `json:"harnessEndRssMaxMb"`
}

type TurnBenchConfig added in v0.4.0 

type TurnBenchConfig struct {
	Model       string
	Mode        string
	SelfCorrect bool
	Version     string
	Commit      string
	// Iterations is how many times each task is run. The per-process `zero exec`
	// runner is inherently cold-start, so this is the sample count for per-span
	// median/P95 — a genuine warm path needs an in-process runner (future work).
	Iterations int
	// Runner executes one task iteration. Required.
	Runner TurnRunner
	// Now overrides the clock for the recorded date (tests inject a fixed time).
	Now func() time.Time
}

TurnBenchConfig configures a turn-benchmark run.

type TurnBenchResult added in v0.4.0 

type TurnBenchResult struct {
	SchemaVersion       int                     `json:"schemaVersion"`
	Suite               string                  `json:"suite"`
	Model               string                  `json:"model"`
	Mode                string                  `json:"mode,omitempty"`
	SelfCorrect         bool                    `json:"selfCorrect"`
	Version             string                  `json:"version,omitempty"`
	Commit              string                  `json:"commit,omitempty"`
	Date                string                  `json:"date"`
	TasksAttempted      int                     `json:"tasksAttempted"`
	TasksVerified       int                     `json:"tasksVerified"`
	TasksPassed         int                     `json:"tasksPassed"`
	LatencyOnlyTasks    int                     `json:"latencyOnlyTasks"`
	BuildCheckedTasks   int                     `json:"buildCheckedTasks"`
	BuildPassedTasks    int                     `json:"buildPassedTasks"`
	CorrectnessPassRate float64                 `json:"correctnessPassRate"`
	BuildPassRate       float64                 `json:"buildPassRate"`
	CorrectnessClasses  []string                `json:"correctnessClasses,omitempty"`
	BuildOnlyClasses    []string                `json:"buildOnlyClasses,omitempty"`
	LatencyOnlyClasses  []string                `json:"latencyOnlyClasses,omitempty"`
	Iterations          int                     `json:"iterations"`
	PerSpan             map[string]SpanStats    `json:"perSpan"`
	TopLatency          []LatencySource         `json:"topLatency"`
	PerClass            map[string]ClassSummary `json:"perClass"`
	Totals              TurnBenchTotals         `json:"totals"`
	Warnings            []Warning               `json:"warnings,omitempty"`
}

TurnBenchResult is the publishable turn-benchmark record.

Pass/fail is split into three oracle tiers so it cannot be misread as a blanket correctness verdict:

  • Correctness (tasksVerified / tasksPassed / correctnessPassRate): tasks with a positive oracle (edit/fix). This is the only pass rate that can move with model quality.
  • Build-only (buildCheckedTasks / buildPassedTasks / buildPassRate): tasks whose oracle is a non-positive build check (refactor `go build`). A pass means it compiles, not that the refactor is correct.
  • Latency-only (latencyOnlyTasks): tasks with no oracle (nav/longproc/ longctx/parallel). They ran for latency and span attribution only and are excluded from every pass rate.

tasksAttempted is still the total number of tasks run across all three tiers. The tier class lists are echoed so a consumer can see exactly which classes each rate is computed over.

func RunTurnBench added in v0.4.0 

func RunTurnBench(ctx context.Context, set TaskSet, cfg TurnBenchConfig) (TurnBenchResult, error)

RunTurnBench executes every task in the set with the configured runner and returns a self-describing per-turn result. It never aborts on a single task failure — every task is attempted and recorded. Per-span stats aggregate across iterations; the top three controllable latency sources are ranked by total attributed time.

type TurnBenchTotals added in v0.4.0 

type TurnBenchTotals struct {
	InputTokens       int64 `json:"inputTokens"`
	CachedInputTokens int64 `json:"cachedInputTokens"`
	OutputTokens      int64 `json:"outputTokens"`
	ModelRequests     int64 `json:"modelRequests"`
	ToolCalls         int64 `json:"toolCalls"`
	Retries           int64 `json:"retries"`
	Reconnects        int64 `json:"reconnects"`
	Compactions       int64 `json:"compactions"`
}

TurnBenchTotals aggregates token and count totals across the whole run.

type TurnRunner added in v0.4.0 

type TurnRunner func(ctx context.Context, task BenchTask, rc RunContext) TurnTaskOutcome

TurnRunner runs one benchmark task and reports its outcome plus the captured per-turn trace. A non-nil Err means the run failed to execute (process crash); Passed reflects the verification result. Trace is the parsed NDJSON trace (nil when the run errored before emitting one).

func NewTurnExecRunner added in v0.4.0 

func NewTurnExecRunner(binary string, extraArgs ...string) TurnRunner

NewTurnExecRunner builds the production turn-benchmark runner: it invokes headless `zero exec` with stream-json output AND `--trace <tmpfile>`, then parses the emitted NDJSON trace into a *trace.TurnTrace. binary is the path to the `zero` binary; extraArgs are appended to every invocation. Pass/fail is decided from the stream-json run_end exit code (and the task's VerificationCommand when present), exactly like NewExecRunner.

type TurnTaskOutcome added in v0.4.0 

type TurnTaskOutcome struct {
	Passed    bool
	VerifyErr string
	WallMs    float64
	Trace     *trace.TurnTrace
	// TraceIssue, when non-empty, explains why the per-turn trace could not be
	// parsed (file missing or malformed). The run still has a pass/fail verdict,
	// but an incomplete measurement is surfaced as a result warning so it can't
	// masquerade as a clean attribution sample.
	TraceIssue string
	Err        error
}

TurnTaskOutcome is what a TurnRunner reports for one task iteration.

type Warning 

type Warning struct {
	Metric    string  `json:"metric"`
	Statistic string  `json:"statistic"`
	Observed  float64 `json:"observed"`
	Threshold float64 `json:"threshold"`
	Unit      string  `json:"unit"`
	Message   string  `json:"message"`
}

func EvaluateWarnings 

func EvaluateWarnings(metrics Metrics, thresholds Thresholds) []Warning

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.