benchcheck

Overview ¶

benchcheck detects performance degradation by comparing the benchmark results of the current working tree with those of the base branch.

The working tree is treated as degraded if the mean of all the benchmarks is slower than the base beyond the tolerance, or if a single benchmark is slower beyond the tolerance of a single benchmark. A single benchmark of identical code differs by several percent on a shared machine, so its tolerance is large and the mean, whose noise is far smaller, is what notices a small degradation.

To make the comparison robust against the load of the machine changing over time, the base and the working tree are measured alternately for each benchmark function. While the working tree is degraded, an additional attempt measures both of them again for the benchmarks which make the mean slow. The noise of a measurement only makes it slower, so the fastest result of all the attempts is what is compared, for each side.

The measured results are cached by commit hash:

• the result of the base branch is reused as long as it is measured on the same machine.
• if the working tree is clean, the verdict for the pair of HEAD and the base commit is reused, so that the command ends immediately.
• if the working tree has uncommitted changes, the working tree is always measured.