Documentation
¶
Index ¶
Constants ¶
const DefaultBootstrapIterations = 10000
DefaultBootstrapIterations is the number of bootstrap resamples.
Variables ¶
This section is empty.
Functions ¶
func IsSignificant ¶
func IsSignificant(ci ConfidenceInterval) bool
IsSignificant returns true if the confidence interval does not contain zero, indicating statistical significance at the given confidence level. It is only meaningful for an interval on a difference (e.g. treatment minus baseline); for an interval on an absolute, non-negative score it only says the score is above zero.
func MinimumPairedClusters ¶ added in v0.39.0
MinimumPairedClusters returns the equal-weight independent cluster count needed for a declared maximum two-sided Hoeffding half-width. It is a conservative precision requirement, not power or a guarantee that any particular release threshold will be met. Always validate the actual frozen weights with FixedPairedHoeffdingBound: unequal weighting or conservative arithmetic rounding at an exact precision boundary can require more clusters. The count is minimal for an upper logarithm enclosure; an ambiguous transcendental boundary may conservatively require one additional cluster.
func NormalizedGain ¶
NormalizedGain computes Hake's normalized gain (1998):
g = (post - pre) / (1 - pre)
This controls for ceiling effects — a gain from 0.9→0.95 is harder than 0.1→0.15. Returns 0 if pre >= 1.0 (already at ceiling) or pre == post (no change). Returns 1.0 if post >= 1.0 (reached maximum).
Types ¶
type ConfidenceInterval ¶
type ConfidenceInterval struct {
Lower float64 `json:"lower"`
Upper float64 `json:"upper"`
Mean float64 `json:"mean"`
ConfidenceLevel float64 `json:"confidence_level"`
NumBootstraps int `json:"num_bootstraps,omitempty"`
}
ConfidenceInterval holds a confidence interval and its point estimate. NumBootstraps is set only for bootstrap intervals.
func BootstrapCI ¶
func BootstrapCI(scores []float64, confidenceLevel float64) ConfidenceInterval
BootstrapCI computes a bootstrap confidence interval over the given scores using the percentile method. confidenceLevel should be in (0, 1), e.g. 0.95. Returns a zero-value ConfidenceInterval when fewer than 2 data points exist.
func BootstrapCIWithSeed ¶
func BootstrapCIWithSeed(scores []float64, confidenceLevel float64, seed int64) ConfidenceInterval
BootstrapCIWithSeed is like BootstrapCI but accepts a seed for reproducibility. A negative seed uses a non-deterministic source.
func WilsonCI ¶ added in v0.39.0
func WilsonCI(successes, trials int, confidenceLevel float64) ConfidenceInterval
WilsonCI returns the Wilson score interval for a pass rate of successes out of trials at the given confidence level (e.g. 0.95). Unlike a bootstrap or the normal approximation, it stays inside [0, 1] and keeps its coverage at the 3–10 trials typical of agent evals (Bowyer et al., ICML 2025). Mean holds the observed rate. Returns a zero interval when trials < 1.
type PairedHoeffdingBound ¶ added in v0.39.0
PairedHoeffdingBound describes a fixed-sample precision bound, not a release verdict or a power calculation. EffectiveClusters conservatively approximates 1 / sum(weights^2); arithmetic rounding must not overstate evidence.
func FixedPairedHoeffdingBound ¶ added in v0.39.0
func FixedPairedHoeffdingBound(weights []float64, alpha float64, comparisons int) (PairedHoeffdingBound, error)
FixedPairedHoeffdingBound bounds the weighted expected candidate-minus-baseline outcome for a declared fixed suite under its repeated-execution mechanism. Each independent cluster contributes a difference in [-1, 1]; dependence within a cluster is unrestricted. Trials and retries within the same cluster do not supply additional independent samples.
Weights, allocation, scoring, family size and alpha must be fixed before collection. The two-sided intervals [estimate-HalfWidth, estimate+HalfWidth] have simultaneous coverage at least 1-alpha for the declared comparisons. Comparisons need not be independent, but execution clusters must be. The radius is sqrt(2*log(2*comparisons/alpha)*sum(weights^2)): signed differences have range length 2, not the range length 1 of each arm's score.
This function validates numeric inputs only. It cannot establish independence, completeness, precollection binding, golden evidence or billing availability. Missing clusters must not be dropped or their weights renormalized. Adaptive stopping, selected successful attempts and post-hoc comparisons are unsupported.
Source Files