Documentation
¶
Overview ¶
Package stats provides the statistical primitives behind the DevHub benchmark engine (S42): robust summaries plus a two-sample significance test so a benchmark delta can be called improvement / regression / no_change with a p-value instead of eyeballing a single run.
Everything here is pure and dependency-free (stdlib only — no gonum): the algorithms are implemented by hand so the tooling stays CGO-free and trivially vendorable.
Index ¶
- func BootstrapMedianDiffCI(a, b []float64) (lower, upper float64)
- func CV(x []float64) float64
- func IQROutliers(x []float64) []int
- func MannWhitneyU(a, b []float64) (u float64, pValue float64)
- func Median(x []float64) float64
- func Percentile(x []float64, p float64) float64
- func PermutationQuantileDiff(a, b []float64, p float64, resamples int) (observed float64, pValue float64)
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func BootstrapMedianDiffCI ¶
BootstrapMedianDiffCI returns the 95% confidence interval of the difference in medians (median(b) - median(a)) via the percentile bootstrap with 2000 resamples. The RNG is seeded deterministically (42) so results are reproducible across runs.
func CV ¶
CV is the coefficient of variation (sample stddev / mean). Returns 0 when the mean is 0 or there are fewer than 2 samples. Uses the sample standard deviation (n-1 denominator).
func IQROutliers ¶
IQROutliers returns the indices (into the original, unsorted x) of values that fall outside the Tukey fences: below Q1-1.5*IQR or above Q3+1.5*IQR.
func MannWhitneyU ¶
MannWhitneyU runs a two-sided Mann-Whitney U test (a.k.a. Wilcoxon rank-sum) on two independent samples, using the normal approximation with tie and continuity corrections. It returns the smaller U statistic and the two-sided p-value.
Algorithm:
- Rank the union of both samples (average ranks within tie groups).
- R1 = sum of ranks of sample a.
- U1 = R1 - n1*(n1+1)/2 ; U2 = n1*n2 - U1 ; U = min(U1, U2).
- mu = n1*n2/2.
- sigma = sqrt( (n1*n2/12) * ((n+1) - sum(t^3-t)/(n*(n-1))) ), t = tie sizes.
- z = (U - mu + 0.5) / sigma (continuity correction toward the mean).
- p = 2 * (1 - Φ(|z|)).
func Median ¶
Median returns the median of x (mean of the two central values for even n). It copies and sorts, so the input is never mutated.
func Percentile ¶
Percentile returns the p-th percentile (p in [0,100]) using linear interpolation between closest ranks — the R-7 / NumPy-default / Excel PERCENTILE.INC method. Percentile([1..100], 95) == 95.05.
func PermutationQuantileDiff ¶ added in v0.1.14
func PermutationQuantileDiff(a, b []float64, p float64, resamples int) (observed float64, pValue float64)
PermutationQuantileDiff answers the question Mann-Whitney U does NOT: "did the p-th percentile move?" (CENTINELA-C-S1, OPS-35). Mann-Whitney compares stochastic dominance — in practice the medians — so a change that shifts only the tail (p99) can pass it with p ≈ 1. This is a two-sided permutation test on the statistic of interest itself: observed = P_p(b) − P_p(a); under the null (the two samples come from one distribution) the labels are exchangeable, so the samples are pooled, re-labelled `resamples` times with a deterministic RNG, and the fraction of |permuted diff| ≥ |observed| is the p-value (with the +1 correction so it never reports exactly 0). It is exact in spirit (no normality, no variance formula) and its resolution is bounded by the sample sizes: a percentile of the tail estimated from a few hundred points is wide by construction — that is the caveat A-54 writes down, not a defect of the test.
p is the percentile (99 for p99); resamples ≥ 1000 is the practical floor.
Types ¶
This section is empty.
Source Files