Documentation
¶
Overview ¶
Command perfab is the statistics half of the Tier B paired A/B driver (hack/perf-ab.sh; design: .omc/plans/raw-write-bypass-elimination.md A.13.4).
perfab probe -thresholds F -rounds a.json,b.json,c.json perfab verdict -thresholds F -pairs N -schedule S -base b1.json,... -head h1.json,... [-noisy]
probe checks the A A A early-abort: the CV of every headline metric over the probe rounds must stay under probe_cv_max_pct.
verdict decides from PAIRS interleaved (base, head) rounds. Per ratio metric the primary statistic is a paired t-test on the per-round log-ratios d_i = ln(head_i / base_i); Mann-Whitney U on the two samples is the second opinion; REGRESSION needs the mean ratio past the threshold AND either test significant. Noise is measured from the pairs themselves, post hoc: the paired CV is the SD of d_i, and the MDE uses t_{N-1} quantiles. Exit codes: 0 PASS, 1 REGRESSION, 2 CONFIG MISMATCH, 3 INCONCLUSIVE, 4 UNDERPOWERED.
Source Files