Documentation
¶
Overview ¶
Package main implements `screen` — a two-level screening design over the live bench knobs, and the effect estimator that reads its result back.
go run ./bench/screen gen -factors bench/screen/factors-mux.tsv -runs 16 > variants.txt bench/run-variants.sh <exit> bench/<date>/<commit>-screen $S/pins 3 <sink> "" variants.txt go run ./bench/screen analyze variants.txt bench/<date>/<commit>-screen
WHY A SCREEN COMES BEFORE A TUNER. bench/tune spends a rig invocation per round on ONE knob against the incumbent; that is the right instrument once the handful of knobs that matter is known, and the wrong one for finding them. There are 115 router knobs and 71 proxy knobs, and a one-knob-at-a-time walk over even twelve of them at two values each is 24 rig rows before anything is known about any of them — while a Plackett-Burman design moves EVERY factor in EVERY run, so each of the twelve main effects is estimated from all sixteen runs at once. That is the whole trick of a screening design: the N-1 columns are orthogonal, so the average of the runs where a factor sat high, minus the average of the runs where it sat low, is an unbiased estimate of that factor's main effect with every other factor averaged out of it.
WHAT IT CANNOT DO, said plainly. A resolution-III design aliases each main effect with two-factor interactions: if ecf.max_window_bytes only helps while leg.starve_ratio is low, the screen will attribute some of that to whichever other factor happens to share the alias. A screen is a FILTER, not a verdict — it says which factors are worth a tuner and which are noise. The verdict is still a paired campaign under bench/verdict.sh.
THE RESPONSE IS THE A/B RATIO, not the rate: bench/run-variants.sh interleaves the baseline with every design point, so each run has its own control measured seconds away, and the ratio is what survives a link whose bar swings 2x inside an hour. The standard error is taken from the replicate scatter — the several pair ratios each run contributes — which is the only honest noise estimate available when every run is measured once per trial.
Source Files