Documentation
¶
Overview ¶
Package stats holds the statistics Halos uses to gate rollouts: always-valid sequential tests (mSPRT), fixed-horizon Welch tests, CUPED variance reduction, paired bootstrap for eval scorecards, and guardrail evaluation. Stdlib only.
Index ¶
- Variables
- func CUPED(y, x []float64, theta float64) []float64
- func CUPEDArms(yc, xc, yt, xt []float64) (adjC, adjT []float64, theta float64, err error)
- func MDE(n int, sd, alpha, power float64) (float64, error)
- func NormalQuantile(p float64) float64
- func PassAtK(n, c, k int) (float64, error)
- func PassHatK(n, c, k int) (float64, error)
- func SampleSize(sd, mde, alpha, power float64) (int, error)
- type BootstrapResult
- func BootstrapMean(x []float64, iters int, seed uint64, alpha float64) (BootstrapResult, error)
- func ClusterBootstrapMean(x []float64, cluster []string, iters int, seed uint64, alpha float64) (BootstrapResult, error)
- func PairedBootstrap(control, treatment []float64, iters int, seed uint64, alpha float64) (BootstrapResult, error)
- func PairedClusterBootstrap(control, treatment []float64, cluster []string, iters int, seed uint64, ...) (BootstrapResult, error)
- type Cluster
- type Decision
- type DiffResult
- type GuardrailResult
- type GuardrailStatus
- type MSPRT
- type SRMResult
- type SeqResult
- type WelchResult
Constants ¶
This section is empty.
Variables ¶
var ErrTooFewSamples = errors.New("stats: need at least 2 samples per arm")
ErrTooFewSamples is returned when a test needs more observations.
Functions ¶
func CUPED ¶
CUPED adjusts y using pre-period covariate x with coefficient theta: y' = y - theta*(x - mean(x)). Mean of y is preserved, variance shrinks.
func CUPEDArms ¶
CUPEDArms adjusts both arms with a single pooled theta = cov(y,x)/var(x) (pooling keeps the treatment-effect estimate unbiased). y and x must be index-aligned per arm.
func MDE ¶ added in v0.4.0
MDE is the smallest absolute difference SampleSize's test detects with n units per arm (its inverse).
func NormalQuantile ¶ added in v0.4.0
NormalQuantile is the standard normal inverse CDF at p in (0,1).
func PassAtK ¶
PassAtK is the unbiased estimator of pass@k (Chen et al. 2021): the probability that at least one of k trials drawn without replacement from n trials with c passes succeeds, 1 - C(n-c,k)/C(n,k), computed as a product so it never overflows.
func PassHatK ¶
PassHatK is pass^k (tau-bench): the probability that all k trials drawn without replacement succeed, C(c,k)/C(n,k). It measures reliability, where pass@k measures capability.
func SampleSize ¶ added in v0.4.0
SampleSize is the per-arm sample size of a two-sided two-sample z-test that detects an absolute difference mde between arm means, with per-unit standard deviation sd, at level alpha and the given power: n = 2 (z(1-alpha/2) + z(power))^2 sd^2 / mde^2, rounded up.
Types ¶
type BootstrapResult ¶
type BootstrapResult struct {
Mean float64
CILo float64
CIHi float64
PGreat float64 // share of resamples with mean > 0 (paired diffs: P(treatment better))
N int
}
BootstrapResult is a percentile bootstrap estimate of a mean.
func BootstrapMean ¶
BootstrapMean is a percentile bootstrap CI of the mean of x. Deterministic for a given seed.
func ClusterBootstrapMean ¶ added in v0.3.0
func ClusterBootstrapMean(x []float64, cluster []string, iters int, seed uint64, alpha float64) (BootstrapResult, error)
ClusterBootstrapMean is a percentile bootstrap CI of the mean of x when observations are grouped in clusters (cluster[i] labels x[i]), e.g. the steps of one agent session. Steps of a session are correlated, so resampling them independently understates the variance; this resamples whole clusters and takes the ratio estimator sum(x)/n over the resample. With one observation per cluster it reduces to BootstrapMean.
func PairedBootstrap ¶
func PairedBootstrap(control, treatment []float64, iters int, seed uint64, alpha float64) (BootstrapResult, error)
PairedBootstrap bootstraps the mean of (treatment[i]-control[i]) over tasks, preserving the pairing. Slices must be index-aligned.
func PairedClusterBootstrap ¶ added in v0.3.0
func PairedClusterBootstrap(control, treatment []float64, cluster []string, iters int, seed uint64, alpha float64) (BootstrapResult, error)
PairedClusterBootstrap is PairedBootstrap with cluster resampling.
type Cluster ¶ added in v0.4.0
type Cluster struct{ Num, Den float64 }
Cluster is one cluster's contribution to a ratio estimator: the sums of a numerator and a denominator over its observations, e.g. one developer's total PR cycle hours (Num) over their merged PRs (Den).
type Decision ¶
type Decision string
Decision is the outcome of a sequential test at the current look.
type DiffResult ¶ added in v0.4.0
type DiffResult struct {
Control float64 `json:"control"`
Treatment float64 `json:"treatment"`
Effect float64 `json:"effect"` // Treatment - Control
CILo float64 `json:"ciLo"`
CIHi float64 `json:"ciHi"`
// Rel is Effect/Control with its CI (resamples whose control ratio is 0
// are left out of the relative CI); all 0 when Control is 0.
Rel float64 `json:"rel"`
RelCILo float64 `json:"relCiLo"`
RelCIHi float64 `json:"relCiHi"`
SE float64 `json:"se"` // bootstrap standard deviation of Effect
NC int `json:"nControl"`
NT int `json:"nTreatment"`
}
DiffResult is a two-arm estimate of ratio(treatment) - ratio(control), where an arm's ratio is sum(Num)/sum(Den) over its clusters.
func ClusterBootstrapDiff ¶ added in v0.4.0
func ClusterBootstrapDiff(control, treatment []Cluster, iters int, seed uint64, alpha float64) (DiffResult, error)
ClusterBootstrapDiff is a percentile bootstrap CI of the difference between two independent arms' ratio estimators. Whole clusters are resampled within each arm (stratified by arm), so correlated observations of one cluster never count as independent evidence. Every Den must be > 0. Deterministic for a given seed.
type GuardrailResult ¶
type GuardrailResult struct {
Status GuardrailStatus `json:"status"`
Regression float64 `json:"regression"` // point estimate, relative to control mean (absolute when that is 0)
Lower float64 `json:"lower"` // one-sided (1-alpha) bounds on Regression
Upper float64 `json:"upper"`
MaxRegression float64 `json:"maxRegression"`
}
GuardrailResult reports the relative regression of treatment vs control (positive = worse) with a one-sided (1-alpha) bound on each side.
func SeqGuardrail ¶
func SeqGuardrail(control, treatment []float64, direction string, maxRegression, alpha float64) (GuardrailResult, error)
SeqGuardrail tests whether treatment regressed against control by more than maxRegression (relative, e.g. 0.05). direction is the metric's good direction: "increase" (lower is a regression) or "decrease" (higher is a regression). It is always valid (peeking-safe): it feeds the delta-method relative regression and its variance into an mSPRT and reads the verdict off the resulting confidence sequence, which holds simultaneously over all looks: Fail if the lower bound exceeds maxRegression, Pass if the upper bound is below it. alpha bounds the probability that *any* look ever wrongly fails/passes. Result.Lower/Upper are the sequence bounds. Each call sees one look; a confidence sequence at any single look is valid, so no state is needed across polls.
type GuardrailStatus ¶
type GuardrailStatus string
GuardrailStatus is the verdict on one guardrail metric.
const ( Pass GuardrailStatus = "pass" // regression is significantly below the limit Fail GuardrailStatus = "fail" // regression is significantly above the limit Inconclusive GuardrailStatus = "inconclusive" // not enough evidence either way )
type MSPRT ¶
type MSPRT struct {
Alpha float64
// Tau2 is the mixture variance: the prior variance of the true effect.
// Pick roughly (plausible effect size)^2; power, not validity, depends on it.
Tau2 float64
// contains filtered or unexported fields
}
MSPRT is a stateful two-sample normal-mixture sequential probability ratio test (Johari et al. 2017). Feed it the cumulative data at each look. Not safe for concurrent use.
func (*MSPRT) ObserveEstimate ¶
ObserveEstimate updates with a pre-computed effect estimate d and the variance v of that estimate (e.g. a delta-method relative difference).
func (*MSPRT) ObserveMeans ¶
ObserveMeans updates with cumulative samples of a continuous metric.
type SRMResult ¶ added in v0.4.0
type SRMResult struct {
Chi2 float64 `json:"chi2"`
PValue float64 `json:"pValue"`
Mismatch bool `json:"mismatch"` // PValue < alpha
}
SRMResult is a sample ratio mismatch check: a chi-square goodness-of-fit test of the observed units per arm against the configured weights.
func SRM ¶ added in v0.4.0
SRM tests observed[i] against weights[i] (normalised). A mismatch means units were lost or gained unevenly between arms (logging, eligibility, attrition caused by the treatment), so arm differences are not trustworthy estimates of the effect. The conventional alpha is 0.001.
type SeqResult ¶
type SeqResult struct {
PValue float64 // always-valid p-value (running min of 1/Lambda)
Decision Decision
Effect float64 // treatment - control
CILo float64 // (1-alpha) confidence sequence (running intersection)
CIHi float64
NC, NT int
}
SeqResult is an always-valid snapshot. PValue and the CI are valid under continuous peeking. mSPRT can only reject the null; it never "accepts", so callers stop on a sample/time budget instead (see internal/promote).
type WelchResult ¶
type WelchResult struct {
Effect float64 // mean(t) - mean(c)
SE float64
T, DF float64
PValue float64 // two-sided
CILo float64 // (1-alpha) two-sided CI on Effect
CIHi float64
NC, NT int
MeanC float64
MeanT float64
}
WelchResult is a fixed-horizon two-sample comparison (treatment - control).