Documentation
¶
Overview ¶
Package stats holds the statistics Halos uses to gate rollouts: always-valid sequential tests (mSPRT), fixed-horizon Welch tests, CUPED variance reduction, paired bootstrap for eval scorecards, and guardrail evaluation. Stdlib only.
Index ¶
- Variables
- func CUPED(y, x []float64, theta float64) []float64
- func CUPEDArms(yc, xc, yt, xt []float64) (adjC, adjT []float64, theta float64, err error)
- func PassAtK(n, c, k int) (float64, error)
- func PassHatK(n, c, k int) (float64, error)
- type BootstrapResult
- type Decision
- type GuardrailResult
- type GuardrailStatus
- type MSPRT
- type SeqResult
- type WelchResult
Constants ¶
This section is empty.
Variables ¶
var ErrTooFewSamples = errors.New("stats: need at least 2 samples per arm")
ErrTooFewSamples is returned when a test needs more observations.
Functions ¶
func CUPED ¶
CUPED adjusts y using pre-period covariate x with coefficient theta: y' = y - theta*(x - mean(x)). Mean of y is preserved, variance shrinks.
func CUPEDArms ¶
CUPEDArms adjusts both arms with a single pooled theta = cov(y,x)/var(x) (pooling keeps the treatment-effect estimate unbiased). y and x must be index-aligned per arm.
Types ¶
type BootstrapResult ¶
type BootstrapResult struct {
Mean float64
CILo float64
CIHi float64
PGreat float64 // share of resamples with mean > 0 (paired diffs: P(treatment better))
N int
}
BootstrapResult is a percentile bootstrap estimate of a mean.
func BootstrapMean ¶
BootstrapMean is a percentile bootstrap CI of the mean of x. Deterministic for a given seed.
func PairedBootstrap ¶
func PairedBootstrap(control, treatment []float64, iters int, seed uint64, alpha float64) (BootstrapResult, error)
PairedBootstrap bootstraps the mean of (treatment[i]-control[i]) over tasks, preserving the pairing. Slices must be index-aligned.
type Decision ¶
type Decision string
Decision is the outcome of a sequential test at the current look.
type GuardrailResult ¶
type GuardrailResult struct {
Status GuardrailStatus `json:"status"`
Regression float64 `json:"regression"` // point estimate, relative to control mean (absolute when that is 0)
Lower float64 `json:"lower"` // one-sided (1-alpha) bounds on Regression
Upper float64 `json:"upper"`
MaxRegression float64 `json:"maxRegression"`
}
GuardrailResult reports the relative regression of treatment vs control (positive = worse) with a one-sided (1-alpha) bound on each side.
func SeqGuardrail ¶
func SeqGuardrail(control, treatment []float64, direction string, maxRegression, alpha float64) (GuardrailResult, error)
SeqGuardrail tests whether treatment regressed against control by more than maxRegression (relative, e.g. 0.05). direction is the metric's good direction: "increase" (lower is a regression) or "decrease" (higher is a regression). It is always valid (peeking-safe): it feeds the delta-method relative regression and its variance into an mSPRT and reads the verdict off the resulting confidence sequence, which holds simultaneously over all looks: Fail if the lower bound exceeds maxRegression, Pass if the upper bound is below it. alpha bounds the probability that *any* look ever wrongly fails/passes. Result.Lower/Upper are the sequence bounds. Each call sees one look; a confidence sequence at any single look is valid, so no state is needed across polls.
type GuardrailStatus ¶
type GuardrailStatus string
GuardrailStatus is the verdict on one guardrail metric.
const ( Pass GuardrailStatus = "pass" // regression is significantly below the limit Fail GuardrailStatus = "fail" // regression is significantly above the limit Inconclusive GuardrailStatus = "inconclusive" // not enough evidence either way )
type MSPRT ¶
type MSPRT struct {
Alpha float64
// Tau2 is the mixture variance: the prior variance of the true effect.
// Pick roughly (plausible effect size)^2; power, not validity, depends on it.
Tau2 float64
// contains filtered or unexported fields
}
MSPRT is a stateful two-sample normal-mixture sequential probability ratio test (Johari et al. 2017). Feed it the cumulative data at each look. Not safe for concurrent use.
func (*MSPRT) ObserveEstimate ¶
ObserveEstimate updates with a pre-computed effect estimate d and the variance v of that estimate (e.g. a delta-method relative difference).
func (*MSPRT) ObserveMeans ¶
ObserveMeans updates with cumulative samples of a continuous metric.
type SeqResult ¶
type SeqResult struct {
PValue float64 // always-valid p-value (running min of 1/Lambda)
Decision Decision
Effect float64 // treatment - control
CILo float64 // (1-alpha) confidence sequence (running intersection)
CIHi float64
NC, NT int
}
SeqResult is an always-valid snapshot. PValue and the CI are valid under continuous peeking. mSPRT can only reject the null; it never "accepts", so callers stop on a sample/time budget instead (see internal/promote).
type WelchResult ¶
type WelchResult struct {
Effect float64 // mean(t) - mean(c)
SE float64
T, DF float64
PValue float64 // two-sided
CILo float64 // (1-alpha) two-sided CI on Effect
CIHi float64
NC, NT int
MeanC float64
MeanT float64
}
WelchResult is a fixed-horizon two-sample comparison (treatment - control).
Source Files