abstats

package
v1.5.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 10, 2026 License: AGPL-3.0 Imports: 1 Imported by: 0

Documentation

Overview

Package abstats is the A/B test's decision rule, shared by the sending engine (which decides) and the report (which explains the decision), so the two can't disagree (review AN-1).

A variant wins only when every variant reached the minimum sample and the leader's lead over the runner-up is significant: a two-sided two-proportion z-test at the chosen confidence (95% unless the campaign says otherwise). Otherwise there is no clear winner, and the rest of the audience gets the default variant.

Index

Constants

View Source
const (
	DefaultMinSample  = 500
	DefaultConfidence = 0.95
)

Defaults of the rule (contract C14: campaigns.ab_test.min_sample and confidence).

View Source
const (
	// Winner: the leader won.
	Winner = "winner"
	// NotSignificant: the lead could be chance.
	NotSignificant = "not_significant"
	// TooSmall: a variant didn't reach the minimum sample.
	TooSmall = "too_small"
	// Manual: a person picked the variant.
	Manual = "manual"
)

Outcomes of a decision.

Variables

This section is empty.

Functions

func Rate

func Rate(n, d int64) (float64, bool)

Rate is n/d in percent; ok is false without a base.

func ZTest

func ZTest(x1, n1, x2, n2 int64) (lift, z, p float64, ok bool)

ZTest is a two-proportion z-test of x1/n1 against x2/n2. It returns the difference p2 - p1 in percentage points, z and the two-sided p-value. ok is false when either sample is empty or the pooled proportion is 0 or 1 (no variance, nothing to test).

Types

type Result

type Result struct {
	// Leader has the best rate, RunnerUp the best of the rest (ties go to
	// the earlier variant).
	Leader, RunnerUp string
	// Tested: the z-test could run (both had results and the rates
	// weren't all 0% or all 100%).
	Tested bool
	// Lift is the leader's lead in percentage points; P the two-sided
	// p-value; Confidence is 1 - P in percent.
	Lift, Z, P, Confidence float64
	// Enough: every variant reached the minimum sample.
	Enough bool
	// Outcome is Winner, NotSignificant or TooSmall.
	Outcome string
}

Result is the rule applied to the variants.

func Decide

func Decide(r Rule, vs []Variant) Result

Decide applies the rule to the variants (at least two).

type Rule

type Rule struct {
	MinSample  int64
	Confidence float64
}

Rule is what a winner needs.

func (Rule) Normalized

func (r Rule) Normalized() Rule

Normalized fills in the defaults.

type Variant

type Variant struct {
	Key         string
	Eligible    int64
	Conversions int64
}

Variant is one variant's result on the test's metric: the recipients whose result could be measured and how many of them converted.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL