benchcmp

Overview ¶

Command benchcmp compares and aggregates `go test -bench` results using only the standard library. It is the repository's dependency-compliant stand-in for benchstat.

Comparison (two files): for every benchmark and unit present in both inputs it reports the old and new medians, the relative change and a two-sided Mann-Whitney U p-value. A change whose p-value exceeds alpha is reported as "~" (no significant difference), never as a speedup or slowdown. Rows whose old and new observations are all identical (typically 0 allocs/op) are omitted.

Summary (-summary, one file): the per-benchmark aggregation of one run, with the sample count, median, minimum, maximum and spread of each unit.

Dominance (-dominance, one file): the per-family comparison against competing libraries. Names of the form Family/library/variant are grouped by family and variant; the reference library (-ref, default decimal) is compared on every unit with each incumbent that ran the same variant. The reference dominates only when its raw samples lie wholly below every incumbent's (non-overlapping distributions); an incumbent whose samples lie wholly below the reference's is a SHORTFALL, printed as a visible exception; any overlap is "unseparated", which supports neither claim. Fewer than -min-samples observations on either side is "insufficient".

Layouts (-layouts, two files): a layout-randomized comparison of two source trees, which separates a code change from the effect of code placement (function and loop alignment) on timing. Each input holds several link layouts of one tree (test binaries built with -ldflags=-randlayout=<seed>), every result line after a "# layout ID" line belonging to layout ID. The statistical unit is the layout: each layout's samples are averaged (they are pseudo-replicates of one placement) and the two sets of layout means are compared with the Mann-Whitney U test; the delta is the change of the median layout mean. A row with fewer than -min-layouts layouts on either side is "insufficient" and never a claim. Every row gets a verdict: a time unit (ns/op) is a "claim" only when p <= alpha and the delta lies outside the placement band (-band, percent, default 16), and "inside band" when significant within it; an exact unit (B/op, allocs/op), which placement does not move, is a claim only when p <= alpha and the two sides' layout means are separated (no overlap). -null marks a null control (both inputs built from one tree): a claim is then a NULL-CONTROL FAILURE of the instrument, never a claim, and the command exits 3.

A run containing a failed benchmark (a matched-accuracy check calls b.Fatal before timing) is void and refused in every mode.

Usage:

go run ./tools/benchcmp [-alpha 0.05] old.txt new.txt
go run ./tools/benchcmp -layouts [-band 16] [-min-layouts 8] [-null] a.txt b.txt
go run ./tools/benchcmp -summary run.txt
go run ./tools/benchcmp -dominance [-ref decimal] [-min-samples 10] run.txt
go run ./tools/benchcmp -regression old.txt new.txt

Regression (-regression, two files): fails on a significant slowdown, any allocation increase, or fewer than eight samples per side.

Run each side at least 10 times, interleaved, so the test has enough power; with too few samples every row reports "~".