ab

package
v1.0.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 11, 2026 License: AGPL-3.0 Imports: 15 Imported by: 0

Documentation

Overview

Package ab provides A/B testing harness for comparing algorithm implementations.

Architecture

The ab package enables controlled experiments comparing two algorithm implementations with statistical rigor: 

┌─────────────────────────────────────────────────────────────────────────┐
│                           A/B HARNESS                                    │
├─────────────────────────────────────────────────────────────────────────┤
│                                                                          │
│   Request ──► Sampler ──► ┬─► Control Algorithm ──────┐                 │
│                           │                            ├──► Comparator  │
│                           └─► Experiment Algorithm ───┘       │         │
│                                                                │         │
│   ┌────────────────────────────────────────────────────────────┘         │
│   │                                                                      │
│   ▼                                                                      │
│   Statistics ──► Decision Engine ──► Recommendation                     │
│   • t-test                          • KEEP_CONTROL                      │
│   • Cohen's d                       • SWITCH_TO_EXPERIMENT              │
│   • Confidence intervals            • NEED_MORE_DATA                    │
│   • Power analysis                                                      │
│                                                                          │
└─────────────────────────────────────────────────────────────────────────┘

Components

  • Harness: Main coordinator that runs both algorithms and collects data
  • Sampler: Controls which requests go to experiment vs control
  • Statistics: Statistical analysis (Welch's t-test, confidence intervals)
  • Decision: Automated winner selection with configurable thresholds

Usage

Basic A/B test setup: 

harness := ab.NewHarness(controlAlgo, experimentAlgo,
    ab.WithSampleRate(0.1),        // 10% of traffic to experiment
    ab.WithMinSamples(1000),       // Minimum samples before decision
    ab.WithConfidenceLevel(0.95),  // 95% confidence required
)

// Run comparison
result, err := harness.Compare(ctx, snapshot, input)

// Get statistical analysis
analysis := harness.GetResults()
if analysis.Recommendation == ab.SwitchToExperiment {
    // Experiment is significantly better
}

Statistical Methodology

The package uses established statistical methods:

  • Welch's t-test for comparing means (handles unequal variances)
  • Cohen's d for effect size measurement
  • Bootstrap confidence intervals for robustness
  • Power analysis to determine required sample sizes

Thread Safety

All types in this package are safe for concurrent use unless otherwise noted.

Integration

The ab package integrates with the eval framework:

  • Uses benchmark.Result for performance measurements
  • Implements Evaluable for self-testing
  • Exports metrics to telemetry sinks

Index

Constants

This section is empty.

Variables

View Source 
var (
	// ErrNilAlgorithm indicates a nil algorithm was provided.
	ErrNilAlgorithm = errors.New("algorithm must not be nil")

	// ErrTypeMismatch indicates input/output type mismatch between algorithms.
	ErrTypeMismatch = errors.New("algorithm type mismatch")
)
View Source 
var (
	// ErrInsufficientSamples indicates not enough samples for analysis.
	ErrInsufficientSamples = errors.New("insufficient samples for statistical analysis")

	// ErrZeroVariance indicates a sample set has zero variance.
	ErrZeroVariance = errors.New("sample set has zero variance")
)

Functions

func CalculatePower 

func CalculatePower(n1, n2 int, effectSize, alpha float64) float64

CalculatePower estimates statistical power for the current sample sizes.

Description: 

Power is the probability of correctly rejecting the null hypothesis
when there is a true effect. Higher power (e.g., 0.8 or 0.9) means
the experiment is more likely to detect real differences.

Inputs:

  • n1: Sample size for group 1.
  • n2: Sample size for group 2.
  • effectSize: Expected Cohen's d effect size.
  • alpha: Significance level (e.g., 0.05).

Outputs:

  • float64: Statistical power (0 to 1).

Thread Safety: This function is stateless and safe for concurrent use.

func EffectSize 

func EffectSize(samples1, samples2 []time.Duration) (float64, error)

EffectSize calculates Cohen's d effect size.

Description: 

Cohen's d measures the standardized difference between two means.
Uses the pooled standard deviation for the denominator.

Inputs:

  • samples1: First sample set. Must not be empty.
  • samples2: Second sample set. Must not be empty.

Outputs:

  • float64: Cohen's d value. Positive means samples1 > samples2.
  • error: Non-nil if samples are empty or have zero pooled variance.

Thread Safety: This function is stateless and safe for concurrent use.

func RequiredSampleSize 

func RequiredSampleSize(effectSize, alpha, power float64) int

RequiredSampleSize calculates samples needed for desired power.

Description: 

Determines the minimum sample size per group needed to achieve
a specified power level for detecting a given effect size.

Inputs:

  • effectSize: Expected Cohen's d effect size.
  • alpha: Significance level (e.g., 0.05).
  • power: Desired power (e.g., 0.8 for 80% power).

Outputs:

  • int: Required sample size per group.

Thread Safety: This function is stateless and safe for concurrent use.

Types

type BanditSampler 

type BanditSampler struct {
	// contains filtered or unexported fields
}

BanditSampler uses Thompson Sampling to adaptively allocate traffic.

Description: 

BanditSampler implements Thompson Sampling, a Bayesian approach
to the multi-armed bandit problem. It balances exploration (trying
both variants) with exploitation (preferring the better variant).
As evidence accumulates, traffic is automatically shifted toward
the better-performing variant.

Thread Safety: Safe for concurrent use.

func NewBanditSampler 

func NewBanditSampler(minExplorationRate float64) *BanditSampler

NewBanditSampler creates a new Thompson Sampling sampler.

Inputs:

  • minExplorationRate: Minimum rate for each variant (e.g., 0.05 for 5%).

Outputs:

  • *BanditSampler: The new sampler. Never nil.

func (*BanditSampler) Rate 

func (s *BanditSampler) Rate() float64

Rate returns the current effective experiment rate.

Description: 

Returns the probability that experiment will be selected based
on current Beta distribution parameters. This is an approximation
as actual sampling uses Thompson Sampling.

func (*BanditSampler) RecordFailure 

func (s *BanditSampler) RecordFailure(experiment bool)

RecordFailure records a failed outcome for the given variant.

Inputs:

  • experiment: true if experiment variant, false if control.

Thread Safety: Safe for concurrent use.

func (*BanditSampler) RecordSuccess 

func (s *BanditSampler) RecordSuccess(experiment bool)

RecordSuccess records a successful outcome for the given variant.

Inputs:

  • experiment: true if experiment variant, false if control.

Thread Safety: Safe for concurrent use.

func (*BanditSampler) Sample 

func (s *BanditSampler) Sample(_ string) bool

Sample returns true if experiment should be used.

Description: 

Uses Thompson Sampling to select variant. Samples from Beta
distributions for each variant and selects the one with higher sample.

Thread Safety: Safe for concurrent use.

func (*BanditSampler) SetRate 

func (s *BanditSampler) SetRate(rate float64)

SetRate adjusts the Beta distribution to achieve target rate.

Description: 

This method resets the sampler to start fresh with parameters
that will initially produce the target rate.

func (*BanditSampler) Stats 

func (s *BanditSampler) Stats() (controlAlpha, controlBeta, expAlpha, expBeta float64)

Stats returns the current Beta distribution parameters.

type ConfidenceInterval 

type ConfidenceInterval struct {
	// Lower is the lower bound.
	Lower float64

	// Upper is the upper bound.
	Upper float64

	// Level is the confidence level (e.g., 0.95).
	Level float64

	// Center is the point estimate (mean).
	Center float64
}

ConfidenceInterval represents a statistical confidence interval.

func BootstrapCI 

func BootstrapCI(samples1, samples2 []time.Duration, level float64, nBootstrap int) (*ConfidenceInterval, error)

BootstrapCI calculates a bootstrap confidence interval.

Description: 

Uses bootstrap resampling to construct a confidence interval
for the mean difference. More robust than parametric methods
when sample distributions are non-normal.

Inputs:

  • samples1: First sample set. Must have at least 2 samples.
  • samples2: Second sample set. Must have at least 2 samples.
  • level: Confidence level (e.g., 0.95).
  • nBootstrap: Number of bootstrap iterations (recommend 10000+).

Outputs:

  • *ConfidenceInterval: Bootstrap percentile interval.
  • error: Non-nil if samples are insufficient.

Thread Safety: This function is stateless and safe for concurrent use.

func CalculateCI 

func CalculateCI(samples1, samples2 []time.Duration, level float64) (*ConfidenceInterval, error)

CalculateCI calculates a confidence interval for the mean difference.

Description: 

Calculates a confidence interval for the difference between two means
using Welch's method (for unequal variances).

Inputs:

  • samples1: First sample set. Must have at least 2 samples.
  • samples2: Second sample set. Must have at least 2 samples.
  • level: Confidence level (e.g., 0.95 for 95% CI).

Outputs:

  • *ConfidenceInterval: The confidence interval for mean1 - mean2.
  • error: Non-nil if samples are insufficient.

Thread Safety: This function is stateless and safe for concurrent use.

func (*ConfidenceInterval) Contains 

func (ci *ConfidenceInterval) Contains(v float64) bool

Contains returns true if the interval contains the value.

func (*ConfidenceInterval) Width 

func (ci *ConfidenceInterval) Width() float64

Width returns the interval width.

type Decision 

type Decision struct {
	// Recommendation is the suggested action.
	Recommendation Recommendation

	// Confidence is the statistical confidence in the recommendation.
	Confidence float64

	// Reason explains the recommendation in human-readable form.
	Reason string

	// Evidence contains the supporting statistical evidence.
	Evidence *Evidence

	// Timestamp is when the decision was made.
	Timestamp time.Time
}

Decision holds the complete decision with supporting evidence.

func (*Decision) Summary 

func (d *Decision) Summary() string

Summary generates a human-readable summary of the experiment.

type DecisionConfig 

type DecisionConfig struct {
	// MinSamples is the minimum samples per variant before making a decision.
	// Default: 100
	MinSamples int

	// ConfidenceLevel is the statistical confidence required (e.g., 0.95).
	// Default: 0.95
	ConfidenceLevel float64

	// MinEffectSize is the minimum Cohen's d to consider practically significant.
	// Default: 0.2 (small effect)
	MinEffectSize float64

	// MaxPValue is the maximum p-value for statistical significance.
	// Default: 0.05
	MaxPValue float64

	// MinPower is the minimum statistical power required.
	// Default: 0.8
	MinPower float64

	// ImprovementThreshold is the minimum relative improvement to switch.
	// E.g., 0.05 means experiment must be at least 5% better.
	// Default: 0.0 (any improvement)
	ImprovementThreshold float64

	// MaxDuration is the maximum experiment duration before forcing a decision.
	// Default: 7 days
	MaxDuration time.Duration

	// RequireCorrectness requires correctness match rate >= threshold.
	// Default: 0.99 (99% correctness match)
	RequireCorrectness float64
}

DecisionConfig configures the decision engine.

func DefaultDecisionConfig 

func DefaultDecisionConfig() *DecisionConfig

DefaultDecisionConfig returns sensible defaults.

Outputs:

  • *DecisionConfig: Default configuration. Never nil.

type DecisionEngine 

type DecisionEngine struct {
	// contains filtered or unexported fields
}

DecisionEngine evaluates A/B test results and makes recommendations.

Description: 

DecisionEngine analyzes collected samples and statistical results
to determine whether to keep the control, switch to experiment,
or continue collecting data.

Thread Safety: Safe for concurrent use (stateless).

func NewDecisionEngine 

func NewDecisionEngine(config *DecisionConfig) *DecisionEngine

NewDecisionEngine creates a new decision engine.

Inputs:

  • config: Decision configuration. If nil, uses defaults.

Outputs:

  • *DecisionEngine: The new engine. Never nil.

func (*DecisionEngine) Evaluate 

func (e *DecisionEngine) Evaluate(input *DecisionInput) *Decision

Evaluate analyzes the input and returns a decision.

Inputs:

  • input: The collected experiment data. Must not be nil.

Outputs:

  • *Decision: The recommendation with evidence. Never nil.

Thread Safety: Safe for concurrent use.

type DecisionInput 

type DecisionInput struct {
	// ControlSamples are the latency samples from control.
	ControlSamples []time.Duration

	// ExperimentSamples are the latency samples from experiment.
	ExperimentSamples []time.Duration

	// CorrectnessMatches is the count of matching outputs.
	CorrectnessMatches int

	// TotalComparisons is the total number of correctness comparisons.
	TotalComparisons int

	// ExperimentStartTime is when the experiment started.
	ExperimentStartTime time.Time
}

DecisionInput contains the data needed to make a decision.

type EffectCategory 

type EffectCategory int

EffectCategory categorizes effect sizes using Cohen's conventions. 

const (
	// EffectNegligible indicates |d| < 0.2
	EffectNegligible EffectCategory = iota
	// EffectSmall indicates 0.2 <= |d| < 0.5
	EffectSmall
	// EffectMedium indicates 0.5 <= |d| < 0.8
	EffectMedium
	// EffectLarge indicates |d| >= 0.8
	EffectLarge
)

func CategorizeEffect 

func CategorizeEffect(d float64) EffectCategory

CategorizeEffect returns the category for a Cohen's d value.

func (EffectCategory) String 

func (e EffectCategory) String() string

String returns the string representation.

type Evidence 

type Evidence struct {
	// ControlSamples is the number of control observations.
	ControlSamples int

	// ExperimentSamples is the number of experiment observations.
	ExperimentSamples int

	// ControlMean is the mean latency for control (nanoseconds).
	ControlMean float64

	// ExperimentMean is the mean latency for experiment (nanoseconds).
	ExperimentMean float64

	// MeanDifference is experiment - control (negative = experiment faster).
	MeanDifference float64

	// RelativeImprovement is the percentage improvement (negative = experiment better).
	RelativeImprovement float64

	// TTest is the t-test result.
	TTest *TTestResult

	// EffectSize is Cohen's d.
	EffectSize float64

	// EffectCategory is the effect size interpretation.
	EffectCategory EffectCategory

	// ConfidenceInterval is the CI for the mean difference.
	ConfidenceInterval *ConfidenceInterval

	// Power is the statistical power.
	Power float64

	// CorrectnessMatch is the rate at which outputs matched (0 to 1).
	CorrectnessMatch float64

	// ExperimentDuration is how long the experiment has been running.
	ExperimentDuration time.Duration
}

Evidence contains the statistical support for a decision.

type Harness 

type Harness struct {
	// contains filtered or unexported fields
}

Harness runs A/B tests comparing two algorithm implementations.

Description: 

Harness manages an A/B experiment between a control and experiment
algorithm. It samples traffic, collects latency measurements,
compares outputs for correctness, and provides statistical analysis.

Thread Safety: Safe for concurrent use.

func NewHarness 

func NewHarness(control, experiment eval.Evaluable, opts ...HarnessOption) (*Harness, error)

NewHarness creates a new A/B test harness.

Inputs:

  • control: The control algorithm (baseline). Must not be nil.
  • experiment: The experiment algorithm (new implementation). Must not be nil.
  • opts: Optional configuration options.

Outputs:

  • *Harness: The new harness. Never nil.
  • error: Non-nil if algorithms are nil.

Example: 

harness, err := ab.NewHarness(controlAlgo, experimentAlgo,
    ab.WithSampleRate(0.1),
    ab.WithMinSamples(1000),
)

func (*Harness) Compare 

func (h *Harness) Compare(
	ctx context.Context,
	key string,
	controlProc, expProc Processor,
	input any,
) (any, error)

Compare runs both algorithms and compares results.

Description: 

Executes both control and experiment algorithms with the same input,
records latencies, compares outputs, and returns the control result.
Use this when you need to compare every request.

Inputs:

  • ctx: Context for cancellation. Must not be nil.
  • key: Unique key for sampling consistency.
  • controlProc: Function to process with control algorithm.
  • expProc: Function to process with experiment algorithm.
  • input: Input to pass to processors.

Outputs:

  • output: The control algorithm's output.
  • error: Error from control algorithm (experiment errors are logged).

Thread Safety: Safe for concurrent use.

func (*Harness) GetDecision 

func (h *Harness) GetDecision() *Decision

GetDecision returns the current recommendation.

Thread Safety: Safe for concurrent use.

func (*Harness) GetResults 

func (h *Harness) GetResults() *Results

GetResults returns the current statistical analysis.

Thread Safety: Safe for concurrent use.

func (*Harness) HealthCheck 

func (h *Harness) HealthCheck(ctx context.Context) error

HealthCheck implements eval.Evaluable.

func (*Harness) Metrics 

func (h *Harness) Metrics() []eval.MetricDefinition

Metrics implements eval.Evaluable.

func (*Harness) Name 

func (h *Harness) Name() string

Name implements eval.Evaluable.

func (*Harness) Properties 

func (h *Harness) Properties() []eval.Property

Properties implements eval.Evaluable.

func (*Harness) RecordCorrectness 

func (h *Harness) RecordCorrectness(matched bool)

RecordCorrectness records a correctness comparison result.

Thread Safety: Safe for concurrent use.

func (*Harness) RecordError 

func (h *Harness) RecordError(experiment bool)

RecordError records an error for the specified variant.

Thread Safety: Safe for concurrent use.

func (*Harness) RecordLatency 

func (h *Harness) RecordLatency(experiment bool, duration time.Duration)

RecordLatency records a latency measurement for the specified variant.

Inputs:

  • experiment: true for experiment variant, false for control.
  • duration: The measured latency.

Thread Safety: Safe for concurrent use.

func (*Harness) Reset 

func (h *Harness) Reset()

Reset clears all collected data and restarts the experiment.

Thread Safety: Safe for concurrent use.

func (*Harness) SelectVariant 

func (h *Harness) SelectVariant(key string) bool

SelectVariant returns which variant should be used for the given key.

Description: 

Returns true if experiment should be used, false for control.
Use this when you want sampling logic but will run only one variant.

Thread Safety: Safe for concurrent use.

func (*Harness) SetSampleRate 

func (h *Harness) SetSampleRate(rate float64)

SetSampleRate updates the experiment sample rate.

Thread Safety: Safe for concurrent use.

type HarnessConfig 

type HarnessConfig struct {
	// SampleRate is the fraction of requests to send to experiment (0.0 to 1.0).
	// Default: 0.1 (10%)
	SampleRate float64

	// MaxSamples is the maximum samples to collect per variant.
	// Default: 10000
	MaxSamples int

	// SamplerType determines the sampling strategy.
	// Default: SamplerTypeHash
	SamplerType SamplerType

	// RunBothAlways runs both algorithms for every request (for comparison).
	// Default: false
	RunBothAlways bool

	// CompareOutputs enables output comparison for correctness checking.
	// Default: true
	CompareOutputs bool

	// DecisionConfig configures automated decision making.
	// Default: DefaultDecisionConfig()
	DecisionConfig *DecisionConfig

	// Logger for debug output.
	Logger *slog.Logger

	// MetricsPrefix for exported metrics.
	// Default: "ab_harness"
	MetricsPrefix string
}

HarnessConfig configures the A/B test harness.

func DefaultHarnessConfig 

func DefaultHarnessConfig() *HarnessConfig

DefaultHarnessConfig returns sensible defaults.

Outputs:

  • *HarnessConfig: Default configuration. Never nil.

type HarnessOption 

type HarnessOption func(*HarnessConfig)

HarnessOption configures the harness.

func WithCompareOutputs 

func WithCompareOutputs(enabled bool) HarnessOption

WithCompareOutputs enables output comparison.

func WithConfidenceLevel 

func WithConfidenceLevel(level float64) HarnessOption

WithConfidenceLevel sets the statistical confidence level.

func WithDecisionConfig 

func WithDecisionConfig(config *DecisionConfig) HarnessOption

WithDecisionConfig sets the decision configuration.

func WithLogger 

func WithLogger(logger *slog.Logger) HarnessOption

WithLogger sets the logger.

func WithMaxSamples 

func WithMaxSamples(n int) HarnessOption

WithMaxSamples sets the maximum samples per variant.

func WithMinSamples 

func WithMinSamples(n int) HarnessOption

WithMinSamples sets the minimum samples for decision.

func WithRunBothAlways 

func WithRunBothAlways(enabled bool) HarnessOption

WithRunBothAlways enables running both algorithms for every request.

func WithSampleRate 

func WithSampleRate(rate float64) HarnessOption

WithSampleRate sets the experiment sample rate.

func WithSamplerType 

func WithSamplerType(t SamplerType) HarnessOption

WithSamplerType sets the sampling strategy.

type HashSampler 

type HashSampler struct {
	// contains filtered or unexported fields
}

HashSampler provides consistent sampling based on key hash.

Description: 

HashSampler uses FNV-1a hashing to consistently assign the same
key to the same variant. This ensures that repeated requests with
the same key always get the same variant, which is useful for
user-based experiments where consistency matters.

Thread Safety: Safe for concurrent use.

func NewHashSampler 

func NewHashSampler(rate float64) *HashSampler

NewHashSampler creates a hash-based sampler with the given rate.

Inputs:

  • rate: Probability of selecting experiment (0.0 to 1.0).

Outputs:

  • *HashSampler: The new sampler. Never nil.

func (*HashSampler) Rate 

func (s *HashSampler) Rate() float64

Rate returns the current sample rate.

func (*HashSampler) Sample 

func (s *HashSampler) Sample(key string) bool

Sample returns true if experiment should be used for this key.

Description: 

Uses FNV-1a hash of the key to determine variant. The same key
will always return the same result for a given rate.

Thread Safety: Safe for concurrent use.

func (*HashSampler) SetRate 

func (s *HashSampler) SetRate(rate float64)

SetRate updates the sample rate.

type PowerAnalysis 

type PowerAnalysis struct {
	// Power is the probability of detecting a true effect (1 - beta).
	Power float64

	// RequiredSampleSize is the samples needed per group for desired power.
	RequiredSampleSize int

	// EffectSize is the target effect size used in calculation.
	EffectSize float64

	// Alpha is the significance level used.
	Alpha float64

	// TargetPower is the desired power level (e.g., 0.8).
	TargetPower float64
}

PowerAnalysis holds results of statistical power calculation.

type Processor 

type Processor func(ctx context.Context, input any) (output any, duration time.Duration, err error)

Processor is a function that processes input and returns output, delta, and error.

type RampUpSampler 

type RampUpSampler struct {
	// contains filtered or unexported fields
}

RampUpSampler gradually increases experiment traffic over time.

Description: 

RampUpSampler implements a gradual rollout strategy where
experiment traffic starts at a minimum and increases to a
target over a specified duration. Useful for cautious deployments.

Thread Safety: Safe for concurrent use.

func NewRampUpSampler 

func NewRampUpSampler(startRate, endRate float64, duration time.Duration) *RampUpSampler

NewRampUpSampler creates a sampler that ramps up over time.

Inputs:

  • startRate: Initial experiment rate.
  • endRate: Target experiment rate after duration.
  • duration: Time to reach target rate.

Outputs:

  • *RampUpSampler: The new sampler. Never nil.

func (*RampUpSampler) Rate 

func (s *RampUpSampler) Rate() float64

Rate returns the current sample rate.

func (*RampUpSampler) Reset 

func (s *RampUpSampler) Reset()

Reset restarts the ramp-up from now.

func (*RampUpSampler) Sample 

func (s *RampUpSampler) Sample(key string) bool

Sample returns true if experiment should be used.

Thread Safety: Safe for concurrent use.

func (*RampUpSampler) SetRate 

func (s *RampUpSampler) SetRate(rate float64)

SetRate sets the end rate.

type RandomSampler 

type RandomSampler struct {
	// contains filtered or unexported fields
}

RandomSampler samples based on a configurable rate.

Description: 

RandomSampler uses a pseudo-random number generator seeded by
the current time to determine variant assignment. This provides
true random sampling but does not ensure consistency for the
same key across calls.

Thread Safety: Safe for concurrent use.

func NewRandomSampler 

func NewRandomSampler(rate float64) *RandomSampler

NewRandomSampler creates a random sampler with the given rate.

Inputs:

  • rate: Probability of selecting experiment (0.0 to 1.0).

Outputs:

  • *RandomSampler: The new sampler. Never nil.

func (*RandomSampler) Rate 

func (s *RandomSampler) Rate() float64

Rate returns the current sample rate.

func (*RandomSampler) Sample 

func (s *RandomSampler) Sample(_ string) bool

Sample returns true if experiment should be used.

Thread Safety: Safe for concurrent use.

func (*RandomSampler) SetRate 

func (s *RandomSampler) SetRate(rate float64)

SetRate updates the sample rate.

type Recommendation 

type Recommendation int

Recommendation indicates the suggested action based on experiment results. 

const (
	// NeedMoreData indicates insufficient samples for a decision.
	NeedMoreData Recommendation = iota

	// KeepControl indicates control variant should be kept.
	KeepControl

	// SwitchToExperiment indicates experiment variant is better.
	SwitchToExperiment

	// NoDifference indicates no statistically significant difference.
	NoDifference
)

func (Recommendation) String 

func (r Recommendation) String() string

String returns the string representation.

type Results 

type Results struct {
	// Names
	ControlName    string
	ExperimentName string

	// Sample counts
	ControlSamples    int
	ExperimentSamples int

	// Call counts
	ControlCalls     int64
	ExperimentCalls  int64
	ControlErrors    int64
	ExperimentErrors int64

	// Timing
	Duration   time.Duration
	SampleRate float64

	// Means
	ControlMean    time.Duration
	ExperimentMean time.Duration

	// Statistical analysis
	TTest              *TTestResult
	EffectSize         float64
	EffectCategory     EffectCategory
	ConfidenceInterval *ConfidenceInterval
	Power              float64
	CorrectnessMatch   float64

	// Decision
	Recommendation Recommendation
	Decision       *Decision
}

Results holds the complete A/B test results.

func (*Results) ExperimentBetter 

func (r *Results) ExperimentBetter() bool

ExperimentBetter returns true if experiment is significantly faster.

func (*Results) Significant 

func (r *Results) Significant() bool

Significant returns true if the difference is statistically significant.

type SampleCollector 

type SampleCollector struct {
	// contains filtered or unexported fields
}

SampleCollector accumulates samples for statistical analysis.

Description: 

SampleCollector provides thread-safe collection of duration samples
with support for rolling windows and summary statistics.

Thread Safety: Safe for concurrent use.

func NewSampleCollector 

func NewSampleCollector(maxSamples int, windowSize time.Duration) *SampleCollector

NewSampleCollector creates a new sample collector.

Inputs:

  • maxSamples: Maximum samples to retain. Zero means unlimited.
  • windowSize: Time window for samples. Zero means no time limit.

Outputs:

  • *SampleCollector: The new collector. Never nil.

func (*SampleCollector) Add 

func (c *SampleCollector) Add(d time.Duration)

Add records a new sample.

Thread Safety: Safe for concurrent use.

func (*SampleCollector) Count 

func (c *SampleCollector) Count() int

Count returns the number of samples.

Thread Safety: Safe for concurrent use.

func (*SampleCollector) Reset 

func (c *SampleCollector) Reset()

Reset clears all samples.

Thread Safety: Safe for concurrent use.

func (*SampleCollector) Samples 

func (c *SampleCollector) Samples() []time.Duration

Samples returns a copy of collected samples.

Thread Safety: Safe for concurrent use.

type Sampler 

type Sampler interface {
	// Sample returns true if the experiment variant should be used.
	Sample(key string) bool

	// Rate returns the current experiment sample rate.
	Rate() float64

	// SetRate updates the sample rate.
	SetRate(rate float64)
}

Sampler determines which variant to use for a given request.

Thread Safety: Implementations must be safe for concurrent use.

type SamplerType 

type SamplerType int

SamplerType determines which sampling strategy to use. 

const (
	// SamplerTypeRandom uses random sampling.
	SamplerTypeRandom SamplerType = iota
	// SamplerTypeHash uses consistent hash-based sampling.
	SamplerTypeHash
	// SamplerTypeBandit uses Thompson Sampling.
	SamplerTypeBandit
	// SamplerTypeRampUp uses gradual ramp-up.
	SamplerTypeRampUp
)

type TTestResult 

type TTestResult struct {
	// TStatistic is the computed t-statistic.
	TStatistic float64

	// PValue is the two-tailed p-value.
	PValue float64

	// DegreesOfFreedom is the Welch-Satterthwaite df.
	DegreesOfFreedom float64

	// Significant is true if PValue < significance level.
	Significant bool

	// SignificanceLevel is the alpha used (e.g., 0.05).
	SignificanceLevel float64
}

TTestResult holds the results of a t-test.

func WelchTTest 

func WelchTTest(samples1, samples2 []time.Duration, alpha float64) (*TTestResult, error)

WelchTTest performs Welch's t-test for two sample sets.

Description: 

Welch's t-test is used when the two samples may have unequal variances.
It does not assume equal population variances, making it more robust
than Student's t-test.

Inputs:

  • samples1: First sample set. Must have at least 2 samples.
  • samples2: Second sample set. Must have at least 2 samples.
  • alpha: Significance level (e.g., 0.05 for 95% confidence).

Outputs:

  • *TTestResult: Test results with t-statistic, p-value, and significance.
  • error: Non-nil if samples are insufficient.

Thread Safety: This function is stateless and safe for concurrent use.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.