bench

Overview ¶

The cross-language benchmark.

It hands ONE corpus to every evaluator the differential already runs, asks each to do its parse-and-verify work over that corpus a fixed number of times, and reports how long that work took. The workload is the differential's workload: the same 208 vectors, and the same answers, computed the same way. Nothing here asks an implementation to do less.

It knows nothing about any chain, and it does not time anything itself. Each evaluator runs its own clock around its own work and prints one line to stderr:

B <impl> <vectors> <repeats> <seconds>

which is the only thing this program reads. The alternative — timing the child process from out here — would measure five process starts, five corpus reads and five output writes as if they were chain work, and the Go binary carries a 27 MB linked reference whose start alone would swamp the C++ one's entire run.

A SINGLE TIMING IS NOT A MEASUREMENT. Every evaluator is run -runs times and the spread of those runs is reported beside the median. The runs are interleaved — every evaluator once, then every evaluator again — so that a machine that slows down halfway through slows all of them down rather than one.