perfbench

package
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 2, 2026 License: MIT Imports: 18 Imported by: 0

Documentation

Index

Constants

View Source 
const SchemaVersion = 3
View Source 
const TaskSchemaVersion = 1

TaskSchemaVersion is the schema version of a published task-benchmark result. Bump it whenever the TaskRunResult shape changes so consumers (docs/BENCHMARK.md tooling, dashboards) can detect format drift.

Variables

View Source 
var DefaultThresholds = Thresholds{
	ColdStartP95Ms:     300,
	FirstOutputP95Ms:   500,
	HarnessEndRssMaxMb: 256,
}

Functions

func EmitWarnings 

func EmitWarnings(w io.Writer, result Result)

func EscapeActionCommand 

func EscapeActionCommand(value string) string

func FormatCommand 

func FormatCommand(command []string) string

func FormatMetric 

func FormatMetric(value float64, unit string) string

func FormatSummary 

func FormatSummary(result Result) string

func FormatTaskSummary 

func FormatTaskSummary(result TaskRunResult) string

FormatTaskSummary renders a human-readable summary that foregrounds the honest framing: the model and the self-correct setting are printed alongside the score, because the number is model-bounded.

func MeasureColdStart 

func MeasureColdStart(ctx context.Context, command []string) (float64, error)

func MeasureFirstOutput 

func MeasureFirstOutput(ctx context.Context, command []string) (firstOutputSample, error)

func ResolveColdStartCommand 

func ResolveColdStartCommand(rootDir string) ([]string, error)

func ResolveFirstOutputCommand 

func ResolveFirstOutputCommand(rootDir string) ([]string, error)

func RoundMetric 

func RoundMetric(value float64) float64

func WriteJSON 

func WriteJSON(w io.Writer, result Result) error

func WriteTaskJSON 

func WriteTaskJSON(w io.Writer, result TaskRunResult) error

WriteTaskJSON writes the indented JSON form of a benchmark result.

Types

type BenchTask 

type BenchTask struct {
	ID                  string   `json:"id"`
	Name                string   `json:"name,omitempty"`
	Prompt              string   `json:"prompt"`
	WorkspaceFixture    string   `json:"workspaceFixture,omitempty"`
	VerificationCommand []string `json:"verificationCommand,omitempty"`
}

BenchTask is one benchmark task. WorkspaceFixture is the relative path of the task's starting workspace; VerificationCommand (optional) is the command the default runner executes to decide pass/fail after ZERO finishes.

type Metrics 

type Metrics struct {
	ColdStartMs          NumericStats `json:"coldStartMs"`
	FirstOutputMs        NumericStats `json:"firstOutputMs"`
	ProcessDrainMs       NumericStats `json:"processDrainMs"`
	HarnessEndRssMb      NumericStats `json:"harnessEndRssMb"`
	HarnessEndRssDeltaMb NumericStats `json:"harnessEndRssDeltaMb"`
}

type NumericStats 

type NumericStats struct {
	Samples []float64 `json:"samples"`
	Min     float64   `json:"min"`
	Median  float64   `json:"median"`
	Average float64   `json:"average"`
	P95     float64   `json:"p95"`
	Max     float64   `json:"max"`
}

func SummarizeSamples 

func SummarizeSamples(samples []float64) NumericStats

type Options 

type Options struct {
	Iterations         int
	WarmupIterations   int
	Thresholds         Thresholds
	ColdStartCommand   []string
	FirstOutputCommand []string
}

type Platform 

type Platform struct {
	OS        string `json:"os"`
	Arch      string `json:"arch"`
	GoVersion string `json:"goVersion"`
	Harness   string `json:"harness"`
}

func CurrentPlatform 

func CurrentPlatform() Platform

type Result 

type Result struct {
	SchemaVersion       int        `json:"schemaVersion"`
	Timestamp           string     `json:"timestamp"`
	Platform            Platform   `json:"platform"`
	ColdStartCommand    []string   `json:"coldStartCommand"`
	FirstOutputCommand  []string   `json:"firstOutputCommand"`
	Iterations          int        `json:"iterations"`
	WarmupIterations    int        `json:"warmupIterations"`
	Thresholds          Thresholds `json:"thresholds"`
	Metrics             Metrics    `json:"metrics"`
	BenchmarkDurationMs float64    `json:"benchmarkDurationMs"`
	Warnings            []Warning  `json:"warnings"`
}

func Run 

func Run(ctx context.Context, options Options) (Result, error)

type RunContext 

type RunContext struct {
	Model       string
	Mode        string
	SelfCorrect bool
}

RunContext is the immutable per-run context handed to the runner for each task, so a runner records the same model/mode/self-correct values the result is stamped with.

type TaskConfig 

type TaskConfig struct {
	Model       string
	Mode        string
	SelfCorrect bool
	Version     string
	Commit      string
	// Now overrides the clock for the recorded date (tests inject a fixed time).
	Now func() time.Time
	// Runner executes one task and reports the outcome. Required. The default
	// production runner (NewExecRunner) invokes headless `zero exec`.
	Runner TaskRunner
}

TaskConfig is the recorded configuration of a benchmark run. The honest framing matters: the model, mode, and self-correct flag are captured because the score is largely model-bounded — the point of the benchmark is to show the self-correct loop's contribution (the with/without delta), not the raw digit.

type TaskOutcome 

type TaskOutcome struct {
	Passed bool
	Detail string
	Err    error
}

TaskOutcome is what a runner reports for one task. Err is reserved for the run failing to execute (e.g. the agent process crashed); Passed reports whether the task's verification succeeded. A non-nil Err is always a non-pass.

func SkipRunner 

func SkipRunner(context.Context, BenchTask, RunContext) TaskOutcome

SkipRunner is a runner that performs no work and records every task as skipped (not passed, no error). It backs the harness's --dry-run mode, which exercises the full record path without invoking an agent or a model.

type TaskResult 

type TaskResult struct {
	ID      string `json:"id"`
	Name    string `json:"name,omitempty"`
	Passed  bool   `json:"passed"`
	Errored bool   `json:"errored,omitempty"`
	Detail  string `json:"detail,omitempty"`
}

TaskResult is the per-task entry in a TaskRunResult.

type TaskRunResult 

type TaskRunResult struct {
	SchemaVersion  int          `json:"schemaVersion"`
	Suite          string       `json:"suite"`
	SuiteName      string       `json:"suiteName,omitempty"`
	Model          string       `json:"model"`
	Mode           string       `json:"mode,omitempty"`
	SelfCorrect    bool         `json:"selfCorrect"`
	Version        string       `json:"version,omitempty"`
	Commit         string       `json:"commit,omitempty"`
	Date           string       `json:"date"`
	TasksAttempted int          `json:"tasksAttempted"`
	TasksPassed    int          `json:"tasksPassed"`
	Errors         int          `json:"errors"`
	PassRate       float64      `json:"passRate"`
	Tasks          []TaskResult `json:"tasks"`
}

TaskRunResult is the publishable benchmark record. It is intentionally self-describing: model + version + commit + self-correct flag + date are all embedded so a copy of this JSON is reproducible and auditable on its own.

func RunTasks 

func RunTasks(ctx context.Context, set TaskSet, config TaskConfig) (TaskRunResult, error)

RunTasks executes every task in the set with the configured runner and returns a self-describing result record. It never aborts on a single task failure or runner error — every task is attempted and recorded — so one bad task can't hide the rest of the run. The context cancels the whole run.

type TaskRunner 

type TaskRunner func(ctx context.Context, task BenchTask, rc RunContext) TaskOutcome

TaskRunner runs a single benchmark task and returns its outcome.

func NewExecRunner 

func NewExecRunner(binary string, extraArgs ...string) TaskRunner

NewExecRunner builds the production runner: it invokes headless `zero exec` with stream-json output for each task and decides pass/fail. binary is the path to the `zero` binary; extraArgs are appended to every invocation (e.g. sandbox flags). The self-correct flag from RunContext is translated into the exec invocation so the recorded config matches what actually ran.

Pass/fail is decided from the stream-json run_end event's exit code: a zero exit means the agent completed the task without error. When a task carries a VerificationCommand, that command is run after the agent and its exit status is authoritative (this mirrors Terminal-Bench's external verifier model).

type TaskSet 

type TaskSet struct {
	ID    string      `json:"id"`
	Name  string      `json:"name,omitempty"`
	Tasks []BenchTask `json:"tasks"`
}

TaskSet is a reproducible benchmark task list. It mirrors the shape of a Terminal-Bench task manifest: each task is an isolated workspace plus a prompt ZERO must satisfy. The set is recorded by ID with every result so a published number is traceable to the exact tasks that produced it.

func LoadTaskSet 

func LoadTaskSet(path string) (TaskSet, error)

LoadTaskSet reads and validates a benchmark task set from a JSON file. Unknown fields are rejected so a malformed manifest fails loudly rather than silently dropping tasks.

type Thresholds 

type Thresholds struct {
	ColdStartP95Ms     float64 `json:"coldStartP95Ms"`
	FirstOutputP95Ms   float64 `json:"firstOutputP95Ms"`
	HarnessEndRssMaxMb float64 `json:"harnessEndRssMaxMb"`
}

type Warning 

type Warning struct {
	Metric    string  `json:"metric"`
	Statistic string  `json:"statistic"`
	Observed  float64 `json:"observed"`
	Threshold float64 `json:"threshold"`
	Unit      string  `json:"unit"`
	Message   string  `json:"message"`
}

func EvaluateWarnings 

func EvaluateWarnings(metrics Metrics, thresholds Thresholds) []Warning

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.