stats

package
v0.6.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 3, 2026 License: Apache-2.0 Imports: 5 Imported by: 0

Documentation

Overview

Package stats holds the statistics Halos uses to gate rollouts: always-valid sequential tests (mSPRT), fixed-horizon Welch tests, CUPED variance reduction, paired bootstrap for eval scorecards, and guardrail evaluation. Stdlib only.

Index

Constants

This section is empty.

Variables

View Source
var ErrTooFewSamples = errors.New("stats: need at least 2 samples per arm")

ErrTooFewSamples is returned when a test needs more observations.

Functions

func CUPED

func CUPED(y, x []float64, theta float64) []float64

CUPED adjusts y using pre-period covariate x with coefficient theta: y' = y - theta*(x - mean(x)). Mean of y is preserved, variance shrinks.

func CUPEDArms

func CUPEDArms(yc, xc, yt, xt []float64) (adjC, adjT []float64, theta float64, err error)

CUPEDArms adjusts both arms with a single pooled theta = cov(y,x)/var(x) (pooling keeps the treatment-effect estimate unbiased). y and x must be index-aligned per arm.

func MDE added in v0.4.0

func MDE(n int, sd, alpha, power float64) (float64, error)

MDE is the smallest absolute difference SampleSize's test detects with n units per arm (its inverse).

func NormalQuantile added in v0.4.0

func NormalQuantile(p float64) float64

NormalQuantile is the standard normal inverse CDF at p in (0,1).

func PassAtK

func PassAtK(n, c, k int) (float64, error)

PassAtK is the unbiased estimator of pass@k (Chen et al. 2021): the probability that at least one of k trials drawn without replacement from n trials with c passes succeeds, 1 - C(n-c,k)/C(n,k), computed as a product so it never overflows.

func PassHatK

func PassHatK(n, c, k int) (float64, error)

PassHatK is pass^k (tau-bench): the probability that all k trials drawn without replacement succeed, C(c,k)/C(n,k). It measures reliability, where pass@k measures capability.

func SampleSize added in v0.4.0

func SampleSize(sd, mde, alpha, power float64) (int, error)

SampleSize is the per-arm sample size of a two-sided two-sample z-test that detects an absolute difference mde between arm means, with per-unit standard deviation sd, at level alpha and the given power: n = 2 (z(1-alpha/2) + z(power))^2 sd^2 / mde^2, rounded up.

Types

type BootstrapResult

type BootstrapResult struct {
	Mean   float64
	CILo   float64
	CIHi   float64
	PGreat float64 // share of resamples with mean > 0 (paired diffs: P(treatment better))
	N      int
}

BootstrapResult is a percentile bootstrap estimate of a mean.

func BootstrapMean

func BootstrapMean(x []float64, iters int, seed uint64, alpha float64) (BootstrapResult, error)

BootstrapMean is a percentile bootstrap CI of the mean of x. Deterministic for a given seed.

func ClusterBootstrapMean added in v0.3.0

func ClusterBootstrapMean(x []float64, cluster []string, iters int, seed uint64, alpha float64) (BootstrapResult, error)

ClusterBootstrapMean is a percentile bootstrap CI of the mean of x when observations are grouped in clusters (cluster[i] labels x[i]), e.g. the steps of one agent session. Steps of a session are correlated, so resampling them independently understates the variance; this resamples whole clusters and takes the ratio estimator sum(x)/n over the resample. With one observation per cluster it reduces to BootstrapMean.

func PairedBootstrap

func PairedBootstrap(control, treatment []float64, iters int, seed uint64, alpha float64) (BootstrapResult, error)

PairedBootstrap bootstraps the mean of (treatment[i]-control[i]) over tasks, preserving the pairing. Slices must be index-aligned.

func PairedClusterBootstrap added in v0.3.0

func PairedClusterBootstrap(control, treatment []float64, cluster []string, iters int, seed uint64, alpha float64) (BootstrapResult, error)

PairedClusterBootstrap is PairedBootstrap with cluster resampling.

type Cluster added in v0.4.0

type Cluster struct{ Num, Den float64 }

Cluster is one cluster's contribution to a ratio estimator: the sums of a numerator and a denominator over its observations, e.g. one developer's total PR cycle hours (Num) over their merged PRs (Den).

type Decision

type Decision string

Decision is the outcome of a sequential test at the current look.

const (
	Continue   Decision = "continue"
	RejectNull Decision = "reject_null"
)

type DiffResult added in v0.4.0

type DiffResult struct {
	Control   float64 `json:"control"`
	Treatment float64 `json:"treatment"`
	Effect    float64 `json:"effect"` // Treatment - Control
	CILo      float64 `json:"ciLo"`
	CIHi      float64 `json:"ciHi"`
	// Rel is Effect/Control with its CI (resamples whose control ratio is 0
	// are left out of the relative CI); all 0 when Control is 0.
	Rel     float64 `json:"rel"`
	RelCILo float64 `json:"relCiLo"`
	RelCIHi float64 `json:"relCiHi"`
	SE      float64 `json:"se"` // bootstrap standard deviation of Effect
	NC      int     `json:"nControl"`
	NT      int     `json:"nTreatment"`
}

DiffResult is a two-arm estimate of ratio(treatment) - ratio(control), where an arm's ratio is sum(Num)/sum(Den) over its clusters.

func ClusterBootstrapDiff added in v0.4.0

func ClusterBootstrapDiff(control, treatment []Cluster, iters int, seed uint64, alpha float64) (DiffResult, error)

ClusterBootstrapDiff is a percentile bootstrap CI of the difference between two independent arms' ratio estimators. Whole clusters are resampled within each arm (stratified by arm), so correlated observations of one cluster never count as independent evidence. Every Den must be > 0. Deterministic for a given seed.

type GuardrailResult

type GuardrailResult struct {
	Status        GuardrailStatus `json:"status"`
	Regression    float64         `json:"regression"` // point estimate, relative to control mean (absolute when that is 0)
	Lower         float64         `json:"lower"`      // one-sided (1-alpha) bounds on Regression
	Upper         float64         `json:"upper"`
	MaxRegression float64         `json:"maxRegression"`
}

GuardrailResult reports the relative regression of treatment vs control (positive = worse) with a one-sided (1-alpha) bound on each side.

func SeqGuardrail

func SeqGuardrail(control, treatment []float64, direction string, maxRegression, alpha float64) (GuardrailResult, error)

SeqGuardrail tests whether treatment regressed against control by more than maxRegression (relative, e.g. 0.05). direction is the metric's good direction: "increase" (lower is a regression) or "decrease" (higher is a regression). It is always valid (peeking-safe): it feeds the delta-method relative regression and its variance into an mSPRT and reads the verdict off the resulting confidence sequence, which holds simultaneously over all looks: Fail if the lower bound exceeds maxRegression, Pass if the upper bound is below it. alpha bounds the probability that *any* look ever wrongly fails/passes. Result.Lower/Upper are the sequence bounds. Each call sees one look; a confidence sequence at any single look is valid, so no state is needed across polls.

type GuardrailStatus

type GuardrailStatus string

GuardrailStatus is the verdict on one guardrail metric.

const (
	Pass         GuardrailStatus = "pass"         // regression is significantly below the limit
	Fail         GuardrailStatus = "fail"         // regression is significantly above the limit
	Inconclusive GuardrailStatus = "inconclusive" // not enough evidence either way
)

type MSPRT

type MSPRT struct {
	Alpha float64
	// Tau2 is the mixture variance: the prior variance of the true effect.
	// Pick roughly (plausible effect size)^2; power, not validity, depends on it.
	Tau2 float64
	// contains filtered or unexported fields
}

MSPRT is a stateful two-sample normal-mixture sequential probability ratio test (Johari et al. 2017). Feed it the cumulative data at each look. Not safe for concurrent use.

func NewMSPRT

func NewMSPRT(alpha, tau2 float64) *MSPRT

NewMSPRT returns a test with the given alpha and mixture variance.

func (*MSPRT) ObserveEstimate

func (m *MSPRT) ObserveEstimate(d, v float64, nc, nt int) SeqResult

ObserveEstimate updates with a pre-computed effect estimate d and the variance v of that estimate (e.g. a delta-method relative difference).

func (*MSPRT) ObserveMeans

func (m *MSPRT) ObserveMeans(control, treatment []float64) SeqResult

ObserveMeans updates with cumulative samples of a continuous metric.

type SRMResult added in v0.4.0

type SRMResult struct {
	Chi2     float64 `json:"chi2"`
	PValue   float64 `json:"pValue"`
	Mismatch bool    `json:"mismatch"` // PValue < alpha
}

SRMResult is a sample ratio mismatch check: a chi-square goodness-of-fit test of the observed units per arm against the configured weights.

func SRM added in v0.4.0

func SRM(observed []int, weights []float64, alpha float64) (SRMResult, error)

SRM tests observed[i] against weights[i] (normalised). A mismatch means units were lost or gained unevenly between arms (logging, eligibility, attrition caused by the treatment), so arm differences are not trustworthy estimates of the effect. The conventional alpha is 0.001.

type SeqResult

type SeqResult struct {
	PValue   float64 // always-valid p-value (running min of 1/Lambda)
	Decision Decision
	Effect   float64 // treatment - control
	CILo     float64 // (1-alpha) confidence sequence (running intersection)
	CIHi     float64
	NC, NT   int
}

SeqResult is an always-valid snapshot. PValue and the CI are valid under continuous peeking. mSPRT can only reject the null; it never "accepts", so callers stop on a sample/time budget instead (see internal/promote).

type WelchResult

type WelchResult struct {
	Effect float64 // mean(t) - mean(c)
	SE     float64
	T, DF  float64
	PValue float64 // two-sided
	CILo   float64 // (1-alpha) two-sided CI on Effect
	CIHi   float64
	NC, NT int
	MeanC  float64
	MeanT  float64
}

WelchResult is a fixed-horizon two-sample comparison (treatment - control).

func Welch

func Welch(control, treatment []float64, alpha float64) (WelchResult, error)

Welch runs Welch's unequal-variance t-test.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL