flag

package
v0.68.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 18, 2026 License: MIT Imports: 15 Imported by: 0

Documentation

Overview

Package flag is feature flags for smolanalytics — boolean and multivariate, with property targeting and percentage rollouts, evaluated deterministically so the same user always lands in the same bucket. What makes it deeper than a plain flag console (a later increment): a flag flip is auto-recorded as a deploy marker, so the existing deploy-impact engine answers "did flag X move activation?" from your editor, provably. This file is the pure engine — types + evaluation — with no I/O, so it's trivially testable and shared verbatim with any SDK that copies the bucketing.

Index

Constants

View Source 
const (
	ModeSequential = "sequential"
	ModeFixed      = "fixed"
)

Inference mode. sequential is the DEFAULT: this dashboard invites daily checking, and a fixed-horizon p-value read every day has a true false-positive rate several times its nominal one. fixed exists for someone who genuinely commits to a single look at a planned sample size, and refuses to render its interval before that sample is reached.

View Source 
const (
	UnitBucketID   = "bucket_id"
	UnitDistinctID = "distinct_id"
)

Randomisation unit. bucket_id is a device key written once per browser and never rewritten by identify() or reset(), so assignment survives login; distinct_id is the fallback for clients too old to send one.

The report STATES which one it used, because they are two different estimands and mixing them is not a rounding error — it is measuring a different experiment. A user who logs in mid-test is one unit under bucket_id and two under distinct_id, and the second of those two carries only the post-login half of their behaviour.

View Source 
const (
	NTuneFromPlanned  = "n_planned" // the power calculator's answer for this experiment's own MDE
	NTuneFromDefault  = "default"   // no baseline/MDE/power declared, so a generic 5000
	NTuneFromExplicit = "explicit"  // the operator typed a number
)

How n_tune was chosen, echoed in the payload. n_tune is the sample size at which the confidence sequence is narrowest, so a wrong one silently widens every interval the user will ever see — which reads as "the tool is unsure" rather than "the tool guessed 5000".

View Source 
const (
	DefaultAlpha = 0.05
	DefaultPower = 0.80
	// DefaultNTune is the fallback only. When the experiment declares baseline + MDE + power, the
	// power calculator's n_planned is the right N* by construction, and the payload says so.
	DefaultNTune = 5000
	// DefaultCUPEDLookbackDays is the per-user pre-exposure window for the covariate.
	DefaultCUPEDLookbackDays = 7
	// CurrentHashVersion pins the bucketing construction. It exists so version 2 can never
	// retroactively re-randomise a running experiment — the defect LaunchDarkly shipped and could
	// not fix, because once users are bucketed the hash is load-bearing history.
	CurrentHashVersion = 1
)
View Source 
const (
	GuardrailNotWorse  = "not_worse"
	GuardrailNotBetter = "not_better"
)

Guardrails: the metrics you are not trying to move, and are not willing to break.

The mistake every homegrown A/B feature makes is running the SAME two-sided significance test on a guardrail as on the primary metric. That is backwards, and it fails silently in the direction that costs money. A two-sided test asks "did this change?" and answers "not significant" both for "we measured no harm across 40,000 users" and for "we have 60 users and no idea" — and the second one reads as a green light. What a guardrail actually asks is the non-inferiority question: "can I RULE OUT a degradation worse than δ?" That is a one-sided claim with a stated margin, it is only answerable with enough data, and it says so when it isn't.

Hence three states, never two. Collapsing INCONCLUSIVE into PASS is exactly how guardrails become decorative, and a decorative guardrail is worse than none: it is a check somebody is relying on that has never once been able to fail.

Guardrail directions. The names come from the metric's OWN good direction, which is the part that trips people up, so it is spelled out here rather than left to the reader:

  • GuardrailNotWorse is for a metric where UP is good (checkout, activation, retention). It forbids a relative DROP bigger than δ.
  • GuardrailNotBetter is for a metric where UP is bad ($exception, refunds, support tickets). It forbids a relative RISE bigger than δ.

An error-rate guardrail declared not_worse is an easy and real mistake — it would forbid errors FALLING — which is why the direction is part of the plan and part of the plan hash.

View Source 
const (
	GuardrailPass         = "PASS"
	GuardrailFail         = "FAIL"
	GuardrailInconclusive = "INCONCLUSIVE"
)

Guardrail verdicts. Strings, not a bool plus a "known" flag, because these travel into JSON and into a sentence a human reads, and PASS/FAIL/INCONCLUSIVE survives both without anybody having to remember which field qualifies which.

View Source 
const (
	MarginDeclared = "declared"
	MarginDefault  = "default"
)

Where δ came from, echoed in the result. A margin nobody chose and a margin somebody chose lead to different decisions, and the reader cannot tell them apart from the number alone.

View Source 
const (
	ExposureEvent = "$feature_flag_called"
	PropFlag      = "$feature_flag"
	PropVariant   = "$feature_flag_response"
)

Exposure/response property keys, mirroring PostHog's $feature_flag_called convention. The "$" prefix means the tracking-plan drift gate treats these as system events, not unplanned ones.

View Source 
const (
	HypothesisPrimary   = "primary"
	HypothesisSecondary = "secondary"
	HypothesisGuardrail = "guardrail"
)

Hypothesis kinds. Only primary hypotheses form the correction family.

View Source 
const (
	DefaultWinsorLoPct = 1.0
	DefaultWinsorHiPct = 99.0
)

Default winsorization percentiles. Recorded on the result and meant to be recorded in the plan hash by the caller, because a cap chosen after seeing the data is a researcher degree of freedom — "we winsorized at p99" and "we winsorized until it was significant" look identical afterwards.

View Source 
const (
	SeqModeSequential = "sequential"
	SeqModeFixed      = "fixed"
)

Mode names. Sequential is the DEFAULT, and that default is the whole point: a fixed-horizon interval on a live dashboard is a wrong number wearing the clothes of a right one.

View Source 
const (
	SeqDefaultAlpha = 0.05
	SeqDefaultNTune = 5000
)

Defaults. NTune is only reached when the experiment declares no planned sample size — see SeqResolveNTune for why n_planned is always the better answer when it exists.

View Source 
const (
	SeqNTuneFromPlan    = "n_planned"    // the power calculator's answer: baseline + MDE + power were declared
	SeqNTuneFromDefault = "default_5000" // nothing was declared, so this is a guess and says so
)

Where n_tune came from. It ships in the payload because k(N) is minimised at N = n_tune, so a reader who does not know which N we tuned for cannot tell a wide interval from a mistuned one.

View Source 
const AdjustedDeltaNote = "back-solved from the adjusted p-value so the interval agrees with the " +
	"decision; this is a presentation convention, not a derived interval"

AdjustedDeltaNote is shipped verbatim beside any BH-adjusted interval.

It is here because there is NO canonical BH-adjusted confidence interval. GrowthBook back-solves the standard error that would have produced the adjusted p and rebuilds the interval from it, purely so an adjusted-significant result still shows an interval excluding zero. This does the same thing — and says so in the payload rather than presenting a presentation convention as a derived quantity. Being the tool that admits this is worth more than the interval is.

View Source 
const BHMethod = "benjamini-hochberg"

BHMethod is the correction name, echoed in the payload so a reader never has to infer it from the shape of the numbers.

View Source 
const DefaultGuardrailMarginPct = 10.0

DefaultGuardrailMarginPct is the relative non-inferiority margin used when a guardrail does not state one: 10% relative.

It exists because δ=0 is not a guardrail. A non-inferiority test at margin zero IS a one-sided superiority test — it can only PASS by proving the treatment strictly better — so a zero-margin guardrail on error rate reads INCONCLUSIVE at every finite sample size, forever, on every experiment. A check that cannot pass, presented as a check that failed to pass, is the most expensive kind of wrong. See Guardrail.Validate.

View Source 
const PropBucketID = "$bucket_id"

PropBucketID is the event property carrying the browser's stable bucket key — the randomisation unit named by UnitBucketID in experiment.go. The evaluate endpoint already accepts bucket_id as a query parameter (internal/api/flags_api.go), but the SDK does not yet stamp it onto $feature_flag_called, so today most instances fall back to distinct_id. That fallback is stated in every result rather than hidden, because the two are different estimands: a user who logs in mid-test is one unit under bucket_id and two under distinct_id.

View Source 
const SRMAlpha = 0.001

Sample Ratio Mismatch: the users actually exposed to each arm do not match the split that was configured. A 50/50 experiment that delivered 5,200 / 4,800 did not randomize the way it said it did, and once that is true nothing downstream can be trusted — the arms differ by whatever mechanism broke the split, not by the change being tested.

This is the single most valuable check an experiment tool can run, because without it a broken experiment is indistinguishable from a successful one. It reports a confident number either way. Every serious platform runs it; the difference between them and a homegrown A/B feature is usually this test and nothing else.

The threshold is deliberately far stricter than the usual 0.05. Experiments are checked constantly and by many people, so a 1-in-20 false-alarm rate would cry wolf until the warning is ignored — which is worse than not having it. GrowthBook fires at 0.001 and Microsoft at 0.0005; 0.001 matches the former.

View Source 
const SuggestedMarginPP = 0.1

SuggestedMarginPP is the absolute margin the creation sites attach to a low-base error guardrail: a tenth of a percentage point.

NOT a default applied inside the evaluator. It is offered at creation, where the operator can see and change it, because no fixed percentage-point figure is right at every base rate — at a 1-in-5000 control rate even half a point would wave through a sixfold rise in errors. Absent an explicit margin the evaluator stays INCONCLUSIVE rather than guessing.

Variables

View Source 
var PlanFields = []string{
	"goal", "control", "guardrails", "secondary", "mode", "alpha", "power",
	"n_tune", "n_tune_source", "baseline_pct", "mde_pct", "n_planned",
	"randomisation_unit", "cuped", "layer", "slice", "holdout", "holdout_pct",
	"hash_version", "variant weights",
}

PlanFields are the fields the lock protects, in the order they are reported. Changing any of them after the data starts arriving means the result was chosen after seeing it.

Functions

func ChangedPlanFields added in v0.29.0 

func ChangedPlanFields(old, next Experiment, oldVariants, nextVariants []Variant) []string

ChangedPlanFields names what differs between two plans, comparing them in RESOLVED form so a caller that round-trips a plan through defaults does not trip the lock on a change it did not make. Variant weights live on the Flag, not the plan, but changing them mid-flight is the exact cause the SRM verdict blames for split mismatches, so they are locked with everything else.

func DirectionFor added in v0.51.0 

func DirectionFor(event string) string

DirectionFor is the sensible default direction for a guardrail on `event`.

It is only a default. A caller that knows better states the direction explicitly, and it is part of the plan hash either way — the point is that the common path stops being wrong by omission.

func InHoldout added in v0.29.0 

func InHoldout(key, unitID string, pct float64) bool

InHoldout reports whether a unit is being kept out of experiments entirely.

pct is the share held out, 0..100. Zero means no holdout, which is the default — a holdout that switched itself on would quietly shrink every experiment's sample.

func InSlice added in v0.29.0 

func InSlice(layer, unitID string, s LayerSlice) bool

InSlice reports whether a unit is inside an experiment's slice of its layer.

An empty slice means "not layered" and admits everyone, so every existing flag keeps its current behaviour exactly.

func LayerPosition added in v0.29.0 

func LayerPosition(layer, unitID string) float64

LayerPosition is where a unit falls in a layer's shared space, in [0,1).

Deterministic and stable: the same unit is always in the same position in the same layer, for as long as the layer exists. That stability is what lets two experiments claim disjoint slices and stay disjoint.

func MDEForSample added in v0.29.0 

func MDEForSample(baseline float64, nPlanned int, alpha, power float64, variants []Variant) (float64, error)

MDEForSample is the inverse: given a total number of exposed units you can actually get, what is the smallest relative lift the design can detect?

This is the question people actually have — they know their traffic, not their effect size — and answering it is what stops someone starting a test that could never have concluded.

func ProportionsDiffer added in v0.54.0 

func ProportionsDiffer(c1, n1, c2, n2 int, alpha float64) (bool, float64)

ProportionsDiffer is the exported two-proportion test, for callers outside this package that need to ask "did this rate actually change" with the SAME arithmetic the experiment reports use.

Exported rather than reimplemented because the recurring defect in this codebase is two surfaces computing one question differently and disagreeing in front of a customer. The weekly update's "unchanged across 23 ships" claim has to be the same statistic as everything else, or it is a fourth definition waiting to drift.

Returns whether the difference clears alpha, and the two-sided p-value. Under-powered inputs return false with a p near 1, which reads as "cannot tell" rather than "no change" — the caller must keep those apart, because they are opposite findings.

func SeqAlwaysValidP added in v0.29.0 

func SeqAlwaysValidP(diff, se float64, n, nTune int) float64

SeqAlwaysValidP is the anytime-valid p-value: the smallest alpha at which the confidence sequence would exclude zero, i.e. the alpha solving half_seq(alpha) = |diff|.

Found by bisection with a FIXED iteration count and no RNG, so it is bit-reproducible. The bracket is alpha in [1e-12, 1] as specified, walked in log-alpha because that is the only conditioning under which the answer is equally precise at p = 0.4 and p = 1e-11: linear bisection puts its entire resolution in the first decade and returns a number whose round trip through half_seq misses by orders of magnitude down at the floor.

Bisection is only legitimate because half_seq is strictly decreasing in alpha — a wider interval is what a smaller alpha buys. That monotonicity is not assumed here, it is pinned by TestSeqHalfWidthIsMonotoneInAlpha; without it bisection returns a deterministic wrong answer, which is strictly worse than a nondeterministic one because nobody ever catches it.

Note this p depends on n_tune (through rho) but NOT on the experiment's alpha. It is the evidence, not the decision.

func SeqHalfWidth added in v0.29.0 

func SeqHalfWidth(se float64, n int, alpha float64, nTune int) float64

SeqHalfWidth is the always-valid half-width for a given standard error, on the proportion scale.

Computed as se * widthFactor rather than as k * z * se — algebraically identical for every alpha we allow, but numerically safe at the edges the p-value bisection walks through: as alpha -> 1 the critical value z -> 0 and k -> infinity, and k*z*se would evaluate 0 * infinity and hand back NaN. A NaN p-value renders as "significant" in most templates, which is the worst possible failure for this particular number.

func SeqInflation added in v0.29.0 

func SeqInflation(n int, alpha float64, nTune int) float64

SeqInflation is k(N): how much wider the always-valid half-width is than the fixed-horizon one. 

k(N) = sqrt( N · ( 2·(N·rho^2 + 1) / (N^2·rho^2) ) · ln( sqrt(N·rho^2 + 1) / alpha ) ) / z_{1-alpha/2}

It is > 1 for every N — pinned as a test, because a k below 1 would mean the always-valid interval is NARROWER than the fixed one, which is impossible and would mean a sign or a bracket is wrong somewhere in the algebra.

func SeqResolveMode added in v0.29.0 

func SeqResolveMode(declared string, n, nPlanned int) (mode, reason string)

SeqResolveMode decides which interval is actually valid to draw right now, and says why.

A pre-registered fixed-horizon test is a promise to look ONCE, at n_planned. Rendering its interval before then is rendering an interval whose stated coverage is false, and the reader has no way to know: it looks exactly like the valid one, just earlier. So we refuse. We draw the always-valid interval instead — wider, correct at every N — and we name the substitution.

The nPlanned <= 0 case is the same failure with the horizon missing entirely: a fixed-horizon test with no declared horizon is peeking with extra steps, so it gets the sequential interval too.

func SeqResolveNTune added in v0.29.0 

func SeqResolveNTune(nPlanned int) (int, string)

SeqResolveNTune picks the tuning point N*, and names its source.

k(N) is minimised at N = N* and grows on both sides, so a wrong N* silently widens every interval the experiment will ever show. When the experiment declared a baseline, an MDE and a power target, the power calculator already computed the sample size at which the operator intends to decide — that number IS N*, and inventing a different one instead means shipping intervals wider than the design justifies while calling it rigour.

nPlanned <= 0 means the experiment declared no design, so 5,000 is a placeholder. The source string travels with it so the report can say "tuned for 5,000 because you never said" rather than presenting a guess as a decision.

func ValidatePlanChange added in v0.29.0 

func ValidatePlanChange(flagKey string, old, next *Experiment, oldVariants, nextVariants []Variant, exposedUsers int) error

ValidatePlanChange is the gate Store.Save calls on every write. It enforces three things:

  1. A layer can never be added, changed or removed once anyone is exposed — the layer is folded into the bucketing salt, so changing it re-randomises every user already in the experiment and re-attributes their past behaviour to whichever arm they land in next.
  2. A locked plan cannot be edited. Stop it, or amend it with a reason that ships in the report.
  3. An unlocked plan still has to be coherent.

flagKey and exposedUsers exist for the error message: "you cannot do that" is useless next to "checkout_v2 has been running since 2026-07-21 with 4,102 users exposed."

func ValidateSlices added in v0.29.0 

func ValidateSlices(layer string, named map[string]LayerSlice) error

ValidateSlices rejects a layer whose experiments overlap.

This is the whole guarantee, so it is checked rather than documented. Overlapping slices do not fail loudly at runtime — they produce two experiments that quietly share users and two results that are each a little wrong in a way no amount of staring at the numbers reveals.

`named` maps an experiment key to its claimed slice. The error names BOTH sides of a collision and the exact overlapping range, because "layer full" tells an operator nothing about what to move.

Types

type Amendment added in v0.29.0 

type Amendment struct {
	At       time.Time `json:"at"`
	Reason   string    `json:"reason"`
	Changed  []string  `json:"changed"`
	FromHash string    `json:"from_plan_hash"`
	ToHash   string    `json:"to_plan_hash"`
}

Amendment is a recorded change to a locked plan. It ships INSIDE the report payload, not only in the audit log: an amendment nobody sees is the same as no lock at all.

type BHFamily added in v0.29.0 

type BHFamily struct {
	Experiment             string   `json:"experiment"`
	Variants               []string `json:"variants"`
	Metrics                []string `json:"metrics"`
	Size                   int      `json:"size"`
	Description            string   `json:"description"`
	AccumulatesAcrossLooks bool     `json:"accumulates_across_looks"` // always false
	Excluded               string   `json:"excluded,omitempty"`       // what was deliberately left out
}

BHFamily names the set of hypotheses corrected together, and is shipped inside the result.

AccumulatesAcrossLooks is a field rather than a comment because it is the question a statistician asks first and the one a dashboard silently gets wrong: the family is the cells at ONE look, and it does NOT grow every time somebody refreshes. Peeking is already paid for by the confidence sequence — inflating the interval at every look AND growing m at every look would charge for the same sin twice, and the intervals would crawl toward useless the longer an honest experiment ran.

type BHResult added in v0.29.0 

type BHResult struct {
	Method     string   `json:"method"`
	Family     BHFamily `json:"family"`
	Q          float64  `json:"q"` // FDR level, = alpha
	Rejections int      `json:"rejections"`
	// Threshold is the largest raw p that was rejected, i.e. p_(k). Zero when nothing was
	// rejected. It is the one number that lets a reader check the step-up by eye.
	Threshold float64  `json:"threshold"`
	Tests     []BHTest `json:"tests"`
	Note      string   `json:"note"`
}

BHResult is the correction, its family, and everything needed to redo it by hand.

func BenjaminiHochberg added in v0.29.0 

func BenjaminiHochberg(experiment string, hs []Hypothesis, q float64) (BHResult, error)

BenjaminiHochberg controls the false discovery rate at q across one experiment's primary (variant × metric) cells at one look. 

sort p ascending → p_(1) … p_(m)
k = max{ i : p_(i) ≤ (i/m)·q }        // step-up; reject 1..k
adjusted p_(i) = min over j ≥ i of ( (m/j)·p_(j) ), capped at 1

The min-over-j is what makes the adjusted values monotone, and monotonicity is not cosmetic: a non-monotone adjusted column can show a SMALLER adjusted p on a LARGER raw p, and a reader who sorts the table by the adjusted column then sees the ranking of their arms silently reordered.

It refuses rather than guesses on: an unnamed experiment (the family would be unnameable), a non-primary hypothesis (see PrimaryHypotheses), a duplicate cell (two looks merged into one family, which is the one thing binding decision 10 forbids), and a p outside [0,1].

type BHTest added in v0.29.0 

type BHTest struct {
	Variant   string  `json:"variant"`
	Metric    string  `json:"metric"`
	P         float64 `json:"p_value"`
	PAdjusted float64 `json:"p_value_adjusted"`
	Rank      int     `json:"rank"` // 1-based position in the ascending p order
	Rejected  bool    `json:"rejected"`
}

BHTest is one hypothesis with its correction applied.

type Bundle added in v0.34.0 

type Bundle struct {
	V    int          `json:"v"`
	HV   int          `json:"hv"` // hash generation — a client that cannot reproduce it must not try
	ETag string       `json:"etag"`
	TTL  int          `json:"ttl"`
	Rest bool         `json:"rest"` // enabled flags exist OUTSIDE this bundle; still ask the server for those
	F    []BundleFlag `json:"f"`
}

Bundle is the published payload. Single-letter keys: this ships to every page on every load, and the field names would otherwise outweigh the data.

func BuildBundle added in v0.34.0 

func BuildBundle(flags []Flag, ttlSeconds int) Bundle

BuildBundle publishes every flag that is BOTH enabled and explicitly opted in.

Rest reports whether other enabled flags exist. Without it a client that evaluated the bundle locally would conclude the flags it did not find are off — silently disabling every server-evaluated flag on the page.

type BundleFilter added in v0.34.0 

type BundleFilter struct {
	P string `json:"p"`
	O string `json:"o"`
	V any    `json:"v,omitempty"`
	// N is the comparand PRE-PARSED as Go parsed it, for numeric ops only.
	//
	// This exists to delete a whole class of divergence rather than test for it. Go's
	// strconv.ParseFloat and JavaScript's Number() disagree on real inputs — "1_000", "0x10",
	// leading/trailing space, "" — and each disagreement silently moves one user into a different
	// arm. Shipping the number Go already computed, including its zero fallback for a
	// non-numeric comparand, means the client never parses it at all.
	N *float64 `json:"n,omitempty"`
}

type BundleFlag added in v0.34.0 

type BundleFlag struct {
	K  string          `json:"k"`            // the key — LOAD-BEARING, it is the bucketing salt
	M  int             `json:"m,omitempty"`  // measured → the client must log an exposure
	V  []BundleVariant `json:"v,omitempty"`  // ORDER PRESERVED: variant bucketing walks it in order
	L  string          `json:"l,omitempty"`  // layer
	S  []float64       `json:"s,omitempty"`  // layer slice [start,end)
	H  string          `json:"h,omitempty"`  // holdout
	HP float64         `json:"hp,omitempty"` // holdout percent
	R  []BundleRule    `json:"r,omitempty"`  // ORDER PRESERVED: first match wins
}

type BundleRule added in v0.34.0 

type BundleRule struct {
	P int            `json:"p"`           // rollout percent
	F []BundleFilter `json:"f,omitempty"` // all must pass
}

type BundleVariant added in v0.34.0 

type BundleVariant struct {
	K string `json:"k"`
	W int    `json:"w"`
}

type CUPEDCfg added in v0.29.0 

type CUPEDCfg struct {
	Enabled      bool   `json:"enabled"`
	LookbackDays int    `json:"lookback_days"`
	Covariate    string `json:"covariate,omitempty"`
}

CUPEDCfg configures variance reduction against each user's own pre-exposure behaviour. Covariate defaults to the goal event; LookbackDays is measured per user, backwards from THEIR first exposure, not from a fixed calendar date — a fixed date mixes post-exposure behaviour into the covariate for anyone who joined late, which biases the adjustment instead of only sharpening it.

type CUPEDOptions added in v0.29.0 

type CUPEDOptions struct {
	Unit         string    `json:"unit"`          // UnitBucketID or UnitDistinctID
	Covariate    string    `json:"covariate"`     // event name counted in the pre-period
	LookbackDays int       `json:"lookback_days"` // 0 → DefaultCUPEDLookbackDays
	From         time.Time `json:"from"`          // resolved report window, echoed
	To           time.Time `json:"to"`
}

CUPEDOptions describes the rows so the result can state what it did. Nothing here changes the arithmetic except the guard thresholds; it exists so no reader has to ask what the covariate was.

type CUPEDResult added in v0.29.0 

type CUPEDResult struct {
	Applied      bool   `json:"applied"`
	SkipReason   string `json:"skip_reason,omitempty"`
	Unit         string `json:"unit"`
	Covariate    string `json:"covariate,omitempty"`
	LookbackDays int    `json:"lookback_days"`

	Control   string `json:"control"`
	Treatment string `json:"treatment"`

	Units      int `json:"units"`          // units in both arms
	UnitsWithP int `json:"units_with_pre"` // units carrying a usable pre-period

	Theta float64 `json:"theta"` // within-arm pooled regression coefficient on the covariate
	Rho   float64 `json:"rho"`   // correlation of Y with X among units that have a pre-period

	RawDiff float64 `json:"raw_diff"`
	RawSE   float64 `json:"raw_se"`
	AdjDiff float64 `json:"adj_diff"`
	AdjSE   float64 `json:"adj_se"`

	// VarianceReductionPct is measured from the two standard errors actually produced
	// (1 - (AdjSE/RawSE)^2), not derived from rho. The identity 1-rho^2 is what the method
	// promises asymptotically; this is what it delivered on these rows.
	VarianceReductionPct float64 `json:"variance_reduction_pct"`
	// EquivalentExtraUnits is that reduction restated in the only currency an operator cares
	// about: how many more users they would have had to wait for to get this interval without it.
	EquivalentExtraUnits int `json:"equivalent_extra_units"`

	From time.Time `json:"from"`
	To   time.Time `json:"to"`

	Note string `json:"note"`
}

CUPEDResult is the adjustment, its cost, and — when it was skipped — why.

RawSE and AdjSE are both here on purpose. "CUPED cut your variance 31%" is an assertion; two standard errors a reader can divide is a measurement. Whoever composes this with a sequential interval must take AdjDiff and AdjSE FIRST and inflate the resulting half-width by k(N) second (binding rule 5) — inflating the raw SE and then adjusting would apply the always-valid penalty to a variance the estimator no longer has.

When Applied is false, AdjDiff and AdjSE are set to the raw values. A caller that always reads the adjusted fields is then always correct, and cannot accidentally report an unadjusted number as adjusted or branch its way into an inconsistency.

func CUPEDAdjust added in v0.29.0 

func CUPEDAdjust(units []CUPEDUnit, control, treatment string, opts CUPEDOptions) (CUPEDResult, error)

CUPEDAdjust computes the covariate-adjusted difference in means between one treatment arm and the control arm, with heteroskedasticity-robust (HC1) standard errors.

Returns an error only for a covariate that is not strictly pre-exposure — that is a broken input rather than a thin one, and silently adjusting on it would bias the estimate toward zero while making the interval narrower, which is the worst combination available. Everything else that goes wrong is a SKIP with a reason, because "we could not help here" is an answer.

func (CUPEDResult) HalfWidthAtZ added in v0.29.0 

func (r CUPEDResult) HalfWidthAtZ(z float64) float64

HalfWidthAtZ is the half-width of the adjusted interval at critical value z.

Exposed as a method so the sequential mode composes in the fixed order: adjust, then inflate. Pass z95 for a fixed-horizon interval, or z95*k(N) for the always-valid one.

type CUPEDSpec added in v0.29.0 

type CUPEDSpec struct {
	Flag         string
	Goal         string    // outcome event
	Covariate    string    // pre-period event to count; empty → Goal
	From, To     time.Time // half-open [From, To); zero To means no upper bound
	LookbackDays int       // 0 → DefaultCUPEDLookbackDays
	CountOutcome bool      // true → Y is the number of goal events; false → Y is 0/1 converted
}

CUPEDSpec describes the rows to build. Instants only — no durations resolved against a clock — so the same event log always yields the same covariate.

type CUPEDUnit added in v0.29.0 

type CUPEDUnit struct {
	Unit    string  `json:"unit"`    // randomisation-unit id
	Variant string  `json:"variant"` // arm this unit was exposed to (first exposure wins)
	Y       float64 `json:"y"`       // outcome
	X       float64 `json:"x"`       // pre-exposure covariate; meaningless when HasPre is false
	// HasPre says the unit was OBSERVABLE for the whole lookback, not that X is non-zero. A user
	// first seen an hour before exposure has X=0 because we were not watching, not because they
	// did nothing, and treating those two as the same value is how a new-user experiment gets a
	// covariate that is really a proxy for signup date.
	HasPre    bool      `json:"has_pre"`
	ExposedAt time.Time `json:"exposed_at"`
	CovFrom   time.Time `json:"cov_from"`
	CovTo     time.Time `json:"cov_to"` // exclusive; must never be after ExposedAt
}

CUPEDUnit is one randomisation unit's outcome and pre-exposure covariate.

It carries the covariate window and the exposure instant, not just the number, because "was this covariate measured strictly before treatment" is the one property that makes the whole method valid, and it has to be checkable by the code that uses the number rather than trusted of whoever built the rows. A covariate that overlaps exposure is post-treatment: adjusting on it bakes the treatment effect into the control variable and biases the answer toward zero, with a narrower interval to make it look better.

func BuildCUPEDUnits added in v0.29.0 

func BuildCUPEDUnits(evs []event.Event, spec CUPEDSpec) ([]CUPEDUnit, string, error)

BuildCUPEDUnits derives one row per randomisation unit from raw events: the outcome inside the report window, and the covariate in the lookback window that ends at that unit's OWN first exposure.

The covariate deliberately reads events from before `From`. That is the point — the pre-period sits before the experiment, so restricting it to the report window would leave every user with a covariate of zero and a method that quietly does nothing.

Returns the randomisation unit it used. It refuses when some exposures carry a bucket id and others do not: that dataset is two estimands stacked on top of each other (device-scoped for some users, account-scoped for the rest), and silently picking one produces a number that is not an estimate of anything.

type Experiment added in v0.29.0 

type Experiment struct {
	Goal              string      `json:"goal"`                   // primary metric event
	Control           string      `json:"control"`                // declared, never inferred
	Guardrails        []Guardrail `json:"guardrails,omitempty"`   // one-sided, never δ=0
	Secondary         []string    `json:"secondary,omitempty"`    // excluded from the BH family
	Mode              string      `json:"mode"`                   // sequential | fixed
	Alpha             float64     `json:"alpha"`                  // 0.05
	Power             float64     `json:"power"`                  // 0.80
	NTune             int         `json:"n_tune"`                 // N*, where the sequence is narrowest
	NTuneSource       string      `json:"n_tune_source"`          // n_planned | default | explicit
	BaselinePct       float64     `json:"baseline_pct,omitempty"` // the control rate the plan assumed
	MDEPct            float64     `json:"mde_pct,omitempty"`      // relative, e.g. 5 = "+5% or better"
	NPlanned          int         `json:"n_planned,omitempty"`    // total exposed units, from power.go
	RandomisationUnit string      `json:"randomisation_unit"`     // bucket_id | distinct_id
	CUPED             *CUPEDCfg   `json:"cuped,omitempty"`
	Layer             string      `json:"layer,omitempty"` // immutable once anyone is exposed
	// Slice is this experiment's claim on its layer's [0,1) space. Empty means unlayered, which
	// admits everyone — so a flag that never opts in is unaffected by any of this.
	Slice LayerSlice `json:"slice,omitempty"`
	// Holdout names a population kept out of experiments entirely, and HoldoutPct is its share
	// (0..100). Its purpose is the question nobody can otherwise answer: after six months of
	// individually-significant wins, did any of it add up? Held-out users are the control for
	// everything you shipped.
	Holdout     string    `json:"holdout,omitempty"`
	HoldoutPct  float64   `json:"holdout_pct,omitempty"`
	HashVersion int       `json:"hash_version"`
	Started     time.Time `json:"started"`
	Stopped     time.Time `json:"stopped,omitempty"`
	// GuardrailStatus is the last evaluation of each guardrail, and GuardrailCheckedAt is when.
	//
	// Both exist so a surface can say "not checked yet" rather than rendering silence as health.
	// The guardrails on every experiment this product created went unevaluated for months while
	// the UI said they were watched, which nobody could see precisely because there was nowhere
	// for "we have not looked" to be written down.
	// RevertedAt / RevertedReason record an AUTOMATIC pull. Observations like the guardrail
	// status, never plan — and a reverted experiment stays reverted until a human says otherwise.
	RevertedAt         time.Time         `json:"reverted_at,omitempty"`
	RevertedReason     string            `json:"reverted_reason,omitempty"`
	GuardrailStatus    []GuardrailResult `json:"guardrail_status,omitempty"`
	GuardrailCheckedAt time.Time         `json:"guardrail_checked_at,omitempty"`
	PlanHash           string            `json:"plan_hash"`
	Locked             bool              `json:"locked"`
	Amendments         []Amendment       `json:"amendments,omitempty"`
}

Experiment is the plan. Attach it to a Flag (as *Experiment) and Store.Save enforces the lock.

Started/Locked/PlanHash are set by Start, never by the caller. Stopped, Amendments, Locked and PlanHash are deliberately OUTSIDE the hash — see canonicalPlan.

func AmendExperiment added in v0.29.0 

func AmendExperiment(flagKey string, old, next *Experiment, oldVariants, nextVariants []Variant, reason string, exposedUsers int, at time.Time) (*Experiment, error)

Amend is the ONLY way past the lock, and it costs a reason that ships inside the report. This is the difference between a pre-registration and a suggestion: you may change the plan, and every reader of every result afterwards sees that you did, when, and why.

func StartExperiment added in v0.29.0 

func StartExperiment(e *Experiment, variants []Variant, exposedUsers int, at time.Time) (*Experiment, error)

Start locks the plan. at is passed in, never read from the clock here: a pure function that calls time.Now() produces a different plan hash on every call, and the receipt that hash anchors is then unverifiable by construction. Callers resolve the instant at the API/MCP boundary and echo it back in the response.

exposedUsers is how many users have already logged an exposure on this flag. It matters only for Layer: folding a layer into the bucketing salt re-randomises everyone, so a layer can only be declared while nobody is in the experiment yet.

func StopExperiment added in v0.29.0 

func StopExperiment(e *Experiment, at time.Time) (*Experiment, error)

Stop ends the experiment. It does NOT change the plan hash: what was pre-registered did not change when you stopped looking, and a receipt issued yesterday has to keep verifying today.

func (Experiment) ComputePlanHash added in v0.29.0 

func (e Experiment) ComputePlanHash() string

ComputePlanHash is the content address of the plan: sha256 over canonicalPlan, hex.

func (Experiment) Resolve added in v0.29.0 

func (e Experiment) Resolve() Experiment

Resolve returns the plan with every default filled in. This is also the ECHO form: what gets rendered in the payload, so a reader never has to know which of these numbers we chose for them. Nothing downstream should reason about a zero-valued Mode, Alpha or margin.

func (Experiment) Validate added in v0.29.0 

func (e Experiment) Validate(variants []Variant) error

Validate checks a plan is coherent enough to be locked. variants may be empty (a boolean flag or a caller that has none to hand); when present, the declared control must be one of them.

type Flag 

type Flag struct {
	Key         string    `json:"key"`
	Description string    `json:"description,omitempty"`
	Enabled     bool      `json:"enabled"`
	Variants    []Variant `json:"variants,omitempty"`
	Rules       []Rule    `json:"rules,omitempty"`
	Measured    bool      `json:"measured,omitempty"`
	// Local publishes this flag's DEFINITION — including the values inside its targeting rules —
	// so a browser or edge worker can evaluate it without a round-trip.
	//
	// Off by default and opted into per flag, because turning it on is a disclosure, not a
	// performance setting: `plan in ["enterprise","legacy_pro"]` tells anyone who views source
	// which customers are on a legacy plan. Any UI for this must describe the consequence rather
	// than the feature.
	Local bool `json:"local,omitempty"`
	// Experiment is the pre-registered analysis plan: which arm is control, what the goal is,
	// the inference mode, alpha, N*, the guardrails and their margins. Nil means "a flag, not an
	// experiment" — Measure then falls back to documented defaults and SAYS SO in the report,
	// rather than pretending a plan existed. Locked once the experiment starts, because a design
	// chosen after seeing the data is not a design.
	Experiment *Experiment `json:"experiment,omitempty"`
	Created    time.Time   `json:"created"`
	Updated    time.Time   `json:"updated"`
}

Flag is a saved feature flag. Variants empty = a boolean flag (served variant is "on"). Rules empty = on for everyone when Enabled. Measured opts this flag into exposure logging (a later increment) so it can be A/B-analysed without every flag inflating the event count.

func (Flag) Evaluate 

func (f Flag) Evaluate(distinctID string, context map[string]any) (string, bool)

Evaluate resolves the flag for one user, given their context properties. Returns the served variant ("on" for a boolean flag, "" when off) and whether the flag is on. Deterministic: the same key + distinct_id always yields the same result, computed only from a stable hash — no randomness, no state — so a client SDK that copies this bucketing agrees byte-for-byte.

type Guardrail added in v0.29.0 

type Guardrail struct {
	Event     string `json:"event"`     // the metric event, e.g. "$exception" or "checkout"
	Direction string `json:"direction"` // not_worse (default) | not_better
	// MarginPct is δ: the RELATIVE degradation, in percent, this guardrail must rule out. 2 means
	// "I will accept up to a 2% relative loss; prove it is no worse than that".
	MarginPct *float64 `json:"margin_pct"`
	// MarginPP is an ABSOLUTE margin in percentage points, and it is the right instrument for a
	// low-base metric. A relative margin needs a control rate far from zero for a ratio to mean
	// anything; a healthy product's error rate is nowhere near it, so the relative test reads
	// INCONCLUSIVE forever and the safety net is quietest exactly where it should be loudest.
	//
	// "No more than half a percentage point more of your users hitting an error" is answerable at
	// any base rate, and is what a person actually means when they set an error guardrail.
	MarginPP *float64 `json:"margin_pp,omitempty"`
}

Guardrail is one metric an experiment promises not to break, with the size of the degradation it must be able to rule out. It is part of the pre-registered plan, so it is hashed with it.

MarginPct is a POINTER on purpose. Absent means "I didn't think about it" and takes the 10% default; an explicit 0 means "prove this got no worse at all", which no finite sample can do and which has to be refused by name. A plain float64 makes those two the same value, and the tool would then silently answer a question the operator never asked.

func GuardrailFor added in v0.53.0 

func GuardrailFor(event string) Guardrail

GuardrailFor builds a sensible guardrail for `event`: the right direction, plus an absolute margin when the metric is one whose base rate is normally near zero.

The absolute margin matters because without it a low-base error guardrail reads INCONCLUSIVE at every sample size, forever — the relative test cannot form a ratio against a near-zero control, so the safety net stays silent precisely on the healthy products where an error spike is most worth catching. Both values are visible in the plan and can be changed; the point is that the common path produces a guardrail that can actually fire.

func ParseGuardrail added in v0.29.0 

func ParseGuardrail(raw []byte) (Guardrail, string, error)

ParseGuardrail decodes a guardrail from JSON, distinguishing an ABSENT margin from an explicit zero one.

The distinction cannot be made after unmarshalling into a float64 — both arrive as 0 — and it is the whole difference between "we chose 10% for you and said so" and "you asked for something impossible and we told you". Every API and MCP handler taking a user-authored guardrail should come through here rather than unmarshalling the struct directly.

func (Guardrail) Margin added in v0.29.0 

func (g Guardrail) Margin() float64

Margin resolves δ. The safe accessor for anything holding a guardrail that may not have been through ParseGuardrail.

func (Guardrail) Validate added in v0.29.0 

func (g Guardrail) Validate() error

Validate rejects a guardrail that cannot produce an honest verdict. One definition of a usable margin, called from the plan's own validation, so a plan and a report can never disagree about whether a guardrail is safe to lock.

A nil margin is ACCEPTED: unstated means the 10% default, and ParseGuardrail fills it at the wire boundary. An explicit zero is refused, with the reason, because it is the single most common way a guardrail becomes decorative.

type GuardrailInput added in v0.29.0 

type GuardrailInput struct {
	Guardrail Guardrail
	// Variant being judged against control. Carried through so a multi-arm experiment's guardrail
	// block is readable without positional guessing.
	Variant string
	// Counts on the guardrail's OWN event: users who did it, out of users exposed, per arm.
	ConvTest, ExpTest int
	ConvCtrl, ExpCtrl int
	// Alpha is the ONE-SIDED level. 0 means DefaultAlpha.
	Alpha float64
	// Mode is ModeSequential (the default) or ModeFixed. NTune is N* for the confidence sequence,
	// needed in sequential mode so k(N) here is the SAME k the primary metric got — two different
	// inflations on one screen is two different experiments.
	Mode  string
	NTune int
	// KFactor overrides the computed k(N). Left at 0 it is derived with SeqInflation, which is the
	// path that cannot drift from the rest of the report. Anything below 1 is refused.
	KFactor float64
	// Unit is the randomisation unit the counts were aggregated to: UnitBucketID or
	// UnitDistinctID. Required, never defaulted — the two are different estimands, and a guardrail
	// that does not say which one it counted cannot be checked by anybody.
	Unit string
}

GuardrailInput is everything one verdict is computed from. Every field that changes the answer is here and is echoed back out, because a guardrail result read without its α, its margin, its inference mode and its randomisation unit is four different claims wearing one word.

type GuardrailResult added in v0.29.0 

type GuardrailResult struct {
	Event        string  `json:"event"`
	Variant      string  `json:"variant,omitempty"`
	Direction    string  `json:"direction"`
	MarginPct    float64 `json:"margin_pct"`
	MarginSource string  `json:"margin_source"`
	Status       string  `json:"status"` // PASS | FAIL | INCONCLUSIVE
	// AbsoluteScale says Bound and ThresholdPct are PERCENTAGE POINTS, not relative percent.
	// Without it a reader compares a 0.4 against a 10 and concludes everything is fine.
	AbsoluteScale bool `json:"absolute_scale,omitempty"`

	Alpha    float64 `json:"alpha"`     // one-sided
	OneSided bool    `json:"one_sided"` // always true; stated so nobody assumes 1.96
	Mode     string  `json:"mode"`      // sequential | fixed
	NTune    int     `json:"n_tune,omitempty"`
	KFactor  float64 `json:"k_factor"` // the confidence-sequence inflation actually applied
	Z        float64 `json:"z"`        // the critical value used, AFTER k inflation
	Unit     string  `json:"unit"`     // randomisation unit

	ExposedTest   int `json:"exposed_test"`
	ConvertedTest int `json:"converted_test"`
	ExposedCtrl   int `json:"exposed_control"`
	ConvertedCtrl int `json:"converted_control"`

	// Bound is the one-sided interval on RELATIVE lift at the stated α, inflated by k. Present
	// only when a relative comparison is defined at all.
	Bound *Interval `json:"bound,omitempty"`
	// ThresholdPct is the number Bound is compared against: -δ for not_worse, +δ for not_better.
	ThresholdPct float64 `json:"threshold_pct"`
	// Reason names why no verdict beyond INCONCLUSIVE was possible. Empty on PASS/FAIL.
	Reason string `json:"reason,omitempty"`
	// Read is the sentence a person acts on.
	Read string `json:"read"`
}

GuardrailResult is one guardrail's verdict plus everything needed to audit it.

func EvaluateGuardrail added in v0.29.0 

func EvaluateGuardrail(in GuardrailInput) (GuardrailResult, error)

EvaluateGuardrail runs one guardrail as a one-sided non-inferiority test.

The claim is: at level α, can the data rule out a relative degradation larger than δ? That is answered by ONE bound of the relative-lift interval computed at z_{1-α} — not z_{1-α/2}. Using the two-sided critical value here would widen the bound by 19% at α=0.05 (1.96 against 1.6449), making the guardrail less sensitive precisely where sensitivity is the entire point: a guardrail is a test you WANT to fire.

FAIL uses the mirrored bound at the same α, so PASS and FAIL are each one-sided α-level claims and the gap between them is the honest INCONCLUSIVE region.

In sequential mode the bound is inflated by the same k(N) the primary metric uses. k is defined against the two-sided critical value (SeqInflation divides by z_{1-α/2}), so applying it to the one-sided value keeps this bound conservative relative to the always-valid one-sided bound at the same α — the direction that can only make a PASS harder, never easier, which is the only direction an approximation is allowed to err in on a metric whose job is catching harm.

func EvaluateGuardrails added in v0.29.0 

func EvaluateGuardrails(ins []GuardrailInput) ([]GuardrailResult, error)

EvaluateGuardrails runs a set and fails the whole set on the first invalid guardrail rather than dropping the bad one: a shorter list of results looks exactly like a set where everything passed.

func GuardrailsBlocking added in v0.29.0 

func GuardrailsBlocking(rs []GuardrailResult) []GuardrailResult

GuardrailsBlocking returns the guardrails that are not a PASS, which is the list a "is this safe to ship" answer is built from. INCONCLUSIVE is in it deliberately: an unanswered guardrail blocks a decision exactly like a failed one does, and only the sentence differs.

type Hypothesis added in v0.29.0 

type Hypothesis struct {
	Variant string  `json:"variant"` // the non-control arm being compared
	Metric  string  `json:"metric"`  // the goal/metric event
	Kind    string  `json:"kind"`    // primary (default) | secondary | guardrail
	P       float64 `json:"p_value"` // raw two-sided p against control
}

Hypothesis is one (variant, metric) cell tested at ONE look.

func PrimaryHypotheses added in v0.29.0 

func PrimaryHypotheses(hs []Hypothesis) []Hypothesis

PrimaryHypotheses filters a mixed list down to the correction family.

Guardrails are excluded on purpose, and it is not a technicality: a guardrail is a test you WANT to fire, so correcting it makes it less likely to catch the harm it exists to catch. Secondary metrics are excluded because they were declared as exploratory — they are not decision inputs, and folding them in would cost the primary metric power to protect a claim nobody is making.

type Interval added in v0.24.0 

type Interval struct {
	Point float64 `json:"point"`
	Lo    float64 `json:"lo"`
	Hi    float64 `json:"hi"`
}

Interval is a range with the point estimate that generated it. Percentages, not proportions, because every other number on the report is a percentage and mixing the two is how a reader misreads by 100x.

func AdjustedDeltaInterval added in v0.29.0 

func AdjustedDeltaInterval(deltaPct, pAdjusted, q float64) (Interval, string, error)

AdjustedDeltaInterval rebuilds a delta interval consistent with an adjusted p-value.

se_implied = |delta| / z_{1-p_adj/2}; interval = delta ± z_{1-q/2} · se_implied

At p_adj = q the bound lands exactly on zero, which is the property that makes the table self-consistent: an adjusted-significant row never shows an interval spanning zero, and an adjusted-non-significant row never shows one that excludes it.

Returns an error rather than a number in the degenerate cases, because each of them would print something that looks like an interval and is not one: a zero point estimate has no scale to back-solve from, and p_adj = 1 implies an infinite standard error.

type LayerSlice added in v0.29.0 

type LayerSlice struct {
	Start float64 `json:"start"`
	End   float64 `json:"end"`
}

LayerSlice is one experiment's claim on a layer's space, half-open [Start, End).

func NextSlice added in v0.29.0 

func NextSlice(layer string, existing map[string]LayerSlice, width float64) (LayerSlice, error)

NextSlice finds room for a new experiment of the given width in a layer.

It packs from the left into the first gap that fits, so allocations stay stable: an experiment that ends does not shuffle the ones still running, and a new one lands in freed space rather than moving anybody. Returns an error naming the free space when the layer cannot fit it, because "no room" without a number is not something an operator can act on.

func (LayerSlice) Empty added in v0.29.0 

func (s LayerSlice) Empty() bool

Empty reports whether no slice was declared — the ordinary case of a flag that is not in a layer at all, which sees ALL traffic.

func (LayerSlice) Width added in v0.29.0 

func (s LayerSlice) Width() float64

Width is the share of traffic this slice admits.

type PowerArm added in v0.29.0 

type PowerArm struct {
	Key      string  `json:"key"`
	Weight   int     `json:"weight"`
	SharePct float64 `json:"share_pct"`
	N        int     `json:"n"` // units this arm receives once NPlanned units are exposed
}

PowerArm is one arm's share of the plan.

type PowerBaseline added in v0.29.0 

type PowerBaseline struct {
	// Unit names the randomisation unit the rate was computed over — "bucket_id" when the
	// events carry one, "distinct_id" otherwise. It is stated rather than assumed because the
	// two are different estimands: bucket_id is device-scoped and survives identify(), so a
	// rate per bucket and a rate per person are different numbers and averaging them is
	// meaningless.
	Unit string `json:"unit"`
	// From and To echo the resolved half-open window [from, to) as RFC3339 UTC. Echoed, not
	// remembered: a caller who asked for "28 days" gets told exactly which 28 days answered.
	From string `json:"from"`
	To   string `json:"to"`
	Days int    `json:"days"`

	Units      int     `json:"units"`      // distinct randomisation units with any event in the window
	Converters int     `json:"converters"` // units that did the goal at least once in the window
	RatePct    float64 `json:"rate_pct"`   // p_A as a percentage, for display
	Rate       float64 `json:"-"`          // p_A as a proportion, for arithmetic

	// DailyUnits is the MEDIAN distinct units per day, not the mean: one launch day or one bot
	// crawl otherwise doubles the planned traffic and halves the estimated duration.
	DailyUnits float64 `json:"daily_units"`

	// EligibilityCaveat is present always, because DailyUnits counts ACTIVE units and an
	// experiment only randomises units that reach the flag AND pass its targeting rules. The
	// honest denominator for eligibility cannot be computed from events alone — it depends on
	// whether the code path ran — so this says so instead of quietly reporting a duration that
	// is too optimistic by whatever the targeting rules filter out.
	EligibilityCaveat string `json:"eligibility_caveat"`

	// ValueProp and Skewness are populated only for a value goal (revenue per unit, say).
	// Skewness is the sample skewness of the per-unit total, zeros included, because a unit
	// that never converted is still a randomised unit and dropping it flatters the tail.
	ValueProp string  `json:"value_prop,omitempty"`
	Skewness  float64 `json:"skewness,omitempty"`
}

PowerBaseline is what this instance's own events say about the goal, over one resolved window.

The whole point of computing it here is that nobody should have to type a baseline conversion rate they do not know. The cost of that convenience is that the number carries assumptions, so every one of them is a field: which window, which randomisation unit, how many days, and the caveat that active units are not the same thing as ELIGIBLE units.

func EstimateBaseline added in v0.29.0 

func EstimateBaseline(evs []event.Event, goal, unitProp, valueProp string, from, to time.Time) (PowerBaseline, error)

EstimateBaseline reads p_A, the daily unit rate, and (for a value goal) the skewness straight out of this instance's events, over one explicit half-open window [from, to).

Both instants are required. "Last 28 days" resolved in here against the wall clock is the same bug MeasureRange was fixed for: the plan would change between two reads of the same page, and the number a user wrote down would never reproduce. The window is resolved by the caller at the API/MCP boundary and echoed back in the result.

unitProp names the event property holding the randomisation unit (a bucket id). Empty, or absent from the events, falls back to DistinctID — and the result SAYS which was used, because a rate per device and a rate per person are different estimands and a plan built on one does not apply to the other.

type PowerInput added in v0.29.0 

type PowerInput struct {
	// Baseline is p_A as a PROPORTION in (0,1), typically PowerBaseline.Rate.
	Baseline float64 `json:"baseline"`
	// MDEPct is the RELATIVE minimum detectable effect in percent: 5 means "detect a 5%
	// relative lift", i.e. p_B = p_A * 1.05. Relative because that is how people describe a
	// win, and mixing it up with an absolute 5 percentage points is a 20x error at a 5%
	// baseline — which is why both forms are echoed in the result.
	MDEPct float64 `json:"mde_pct"`

	Alpha float64 `json:"alpha"` // 0 => 0.05
	Power float64 `json:"power"` // 0 => 0.80

	// Variants is the flag's arms in DECLARED order, control first — the same control rule
	// MeasureRange uses, so the plan and the report agree on which arm is the baseline.
	// Empty or single => a boolean flag, powered as an even two-arm split.
	Variants []Variant `json:"variants,omitempty"`

	// DailyUnits is eligible randomisation units per day. 0 => duration is not computed and
	// the result says so rather than printing a zero that reads as "ready today".
	DailyUnits float64 `json:"daily_units"`

	// Unit is the randomisation unit label to echo ("bucket_id" / "distinct_id"). Empty is
	// reported as unstated rather than defaulted, because guessing here silently changes the
	// estimand the sample size belongs to.
	Unit string `json:"unit"`

	// GoalSkewness, when non-zero, is the sample skewness of the per-unit goal value. Past
	// maxGoalSkewness ComputePower refuses: a normal-approximation sample size on a metric
	// that skewed is a confidently wrong number, which is worse than no number.
	GoalSkewness float64 `json:"goal_skewness,omitempty"`

	// HalfWidth is the always-valid (sequential) half-width of the difference interval, on the
	// proportion scale, given the fixed-horizon standard error at a per-arm N and that N. It is
	// the package's confidence-sequence half-width with alpha and n_tune already closed over.
	//
	// Passed in rather than reimplemented here so there is exactly ONE definition of the
	// sequence in this package. Two would drift apart, and the calculator would then plan for a
	// wider interval than the report renders — the user hits significance early and concludes
	// the planner lied. This function supplies the SE, since only it knows the design.
	//
	// nil => the sequential requirement is not computed and the result says why.
	//
	// The search assumes the half-width is non-increasing in N. The inflation factor k(N) does
	// grow again once N passes n_tune, but only like sqrt(log N), while the SE falls like
	// 1/sqrt(N) — so the product is decreasing everywhere past a handful of units. Stated
	// rather than detected: a non-monotone half-width makes "the smallest N that is narrow
	// enough" ill-defined, not merely hard to find.
	HalfWidth func(se float64, nPerArm int) float64 `json:"-"`
}

PowerInput is one power question. Everything optional has a stated default.

type PowerPlan added in v0.29.0 

type PowerPlan struct {
	Unit string `json:"unit"`

	BaselinePct float64 `json:"baseline_pct"` // p_A
	TargetPct   float64 `json:"target_pct"`   // p_B = p_A * (1 + r)
	MDEPct      float64 `json:"mde_pct"`      // relative, as asked
	MDEAbsPct   float64 `json:"mde_abs_pct"`  // p_B - p_A in percentage points, because people conflate the two
	Alpha       float64 `json:"alpha"`
	Power       float64 `json:"power"`

	Arms []PowerArm `json:"arms"`
	// PoweredArm is the arm the sample size was computed against control. With more than two
	// arms this is the SMALLEST non-control arm — the one that fills last, so the plan is not
	// quietly powered for the fastest comparison while the user waits on the slowest.
	PoweredArm string `json:"powered_arm"`
	// NPlanned is total units exposed across ALL arms. This is the headline number, and it is
	// also the default for the confidence sequence's tuning parameter n_tune: k(N) is
	// minimised near the N a design is actually aiming at, so tuning anywhere else
	// systematically widens every interval the experiment ever renders.
	NPlanned int `json:"n_planned"`
	// NControl and NTest are the two arms the power calculation is about, which is not the
	// same as NPlanned/len(arms) once the weights are uneven.
	NControl int `json:"n_control"`
	NTest    int `json:"n_test"`

	// NPlannedSequential is what the same design costs under always-valid inference, and
	// SequentialNote says either how much extra that is or why it could not be computed. Zero
	// with a note, never zero silently.
	NPlannedSequential int    `json:"n_planned_sequential"`
	SequentialNote     string `json:"sequential_note"`

	DurationDays    int     `json:"duration_days"`    // NPlanned / DailyUnits, rounded up; 0 when unknown
	RecommendedDays int     `json:"recommended_days"` // max(DurationDays, 14)
	DurationNote    string  `json:"duration_note"`
	DailyUnits      float64 `json:"daily_units"`

	// Formula names the closed form used, so a reader can check the arithmetic instead of
	// trusting it. Same reason every rate in this package ships its numerator and denominator.
	Formula string `json:"formula"`
	Read    string `json:"read"`
}

PowerPlan is the answer, with every input that produced it restated.

func ComputePower added in v0.29.0 

func ComputePower(in PowerInput) (PowerPlan, error)

ComputePower turns a design into a plan: how many units, split how, for how long.

Two-sided, unpooled planning variance, normal approximation. Unpooled because the planning question is "how much noise will the DIFFERENCE have if the effect is real", which is p_A q_A / n_C + p_B q_B / n_T — the pooled form answers the null's question instead and, used only for the uneven-allocation branch (as the spec draft had it), makes a 50/50 split come out to two different sample sizes depending on which code path computed it. One design, two answers, is the exact defect this package exists to eliminate.

type RatioArm added in v0.29.0 

type RatioArm struct {
	Key           string  `json:"key"`
	Units         int     `json:"units"`          // randomisation units (clusters)
	AnalysisUnits float64 `json:"analysis_units"` // sum of denominators
	Num           float64 `json:"num"`
	Den           float64 `json:"den"`
	Estimable     bool    `json:"estimable"`
	Reason        string  `json:"reason,omitempty"` // why not, when not

	Ratio   float64 `json:"ratio"`
	SE      float64 `json:"se"`       // delta method, clustered at the randomisation unit
	RatioLo float64 `json:"ratio_lo"` // in the metric's own units, not percent
	RatioHi float64 `json:"ratio_hi"`

	// SECluster is the same variance by the sandwich route. SEDeltaNoCov is the same variance with
	// the covariance term deleted — shipped so a reader can see what the classic bug would have
	// printed, rather than being told it matters.
	SECluster    float64 `json:"se_cluster"`
	SEDeltaNoCov float64 `json:"se_delta_no_covariance"`
	// SENaive pretends every analysis unit was independently randomised. ClusterInflation is
	// SE/SENaive: the factor by which a tool that forgot to aggregate would have understated this.
	SENaive          float64 `json:"se_naive,omitempty"`
	ClusterInflation float64 `json:"cluster_inflation,omitempty"`
	MeanClusterSize  float64 `json:"mean_cluster_size"`

	// Relative comparison against control. DeltaLogPoint/DeltaLogSE are the log-scale point and
	// standard error the interval is built from; they are exported so a sequential mode can
	// inflate the critical value (binding rule 5) instead of re-deriving the interval.
	DeltaDefined  bool      `json:"delta_defined"`
	DeltaPct      float64   `json:"delta_pct"`
	DeltaCI       *Interval `json:"delta_ci,omitempty"`
	DeltaLogPoint float64   `json:"delta_log_point"`
	DeltaLogSE    float64   `json:"delta_log_se"`
	PValue        float64   `json:"p_value,omitempty"`
	Significant   bool      `json:"significant"`
	Read          string    `json:"read,omitempty"`
}

RatioArm is one arm's ratio with everything needed to check it.

func (RatioArm) DeltaIntervalAtZ added in v0.29.0 

func (a RatioArm) DeltaIntervalAtZ(z float64) (Interval, bool)

DeltaIntervalAtZ rebuilds the relative-lift interval at an arbitrary critical value, so the sequential mode multiplies z by k(N) rather than inflating an already-rounded interval.

type RatioOptions added in v0.29.0 

type RatioOptions struct {
	Unit        string    `json:"unit"` // UnitBucketID or UnitDistinctID
	Numerator   string    `json:"numerator"`
	Denominator string    `json:"denominator"`
	Control     string    `json:"control"`
	WinsorLoPct float64   `json:"winsor_lo_pct"` // 0 and 0 → winsorization off, and the report says so
	WinsorHiPct float64   `json:"winsor_hi_pct"`
	From        time.Time `json:"from"`
	To          time.Time `json:"to"`
}

RatioOptions states the estimand. All of it is echoed on the report: a ratio with no stated numerator, denominator and randomisation unit is not a metric, it is a number.

type RatioReport added in v0.29.0 

type RatioReport struct {
	Unit        string     `json:"unit"`
	Numerator   string     `json:"numerator"`
	Denominator string     `json:"denominator"`
	Control     string     `json:"control"`
	Arms        []RatioArm `json:"arms"`

	Winsorized    bool    `json:"winsorized"`
	WinsorLoPct   float64 `json:"winsor_lo_pct,omitempty"`
	WinsorHiPct   float64 `json:"winsor_hi_pct,omitempty"`
	WinsorLoValue float64 `json:"winsor_lo_value,omitempty"`
	WinsorHiValue float64 `json:"winsor_hi_value,omitempty"`

	// CrossCheckMaxDivergencePct is the largest disagreement between the delta-method and
	// cluster-robust standard errors across arms, as a percentage. OK is false above 1%, which
	// means one of the two implementations is wrong — not that the reader should pick a side.
	CrossCheckMaxDivergencePct float64 `json:"cross_check_max_divergence_pct"`
	CrossCheckOK               bool    `json:"cross_check_ok"`

	From time.Time `json:"from"`
	To   time.Time `json:"to"`
	Note string    `json:"note"`
}

RatioReport is the whole read: every arm, the estimand, and whether the two variance routes agreed.

func MeasureRatio added in v0.29.0 

func MeasureRatio(units []RatioUnit, opts RatioOptions) (RatioReport, error)

MeasureRatio computes a ratio metric per arm from rows already aggregated to the randomisation unit.

It returns an error rather than a number when the estimand itself is broken — no stated randomisation unit, no control arm, rows whose Values contradict their own Num/Den. Every vendor default is to compute something anyway; a ratio computed on an undefined unit of randomisation is not a weaker estimate, it is an estimate of nothing.

type RatioSpec added in v0.29.0 

type RatioSpec struct {
	Flag             string
	Numerator        string // event whose occurrences form the numerator
	NumeratorProp    string // optional numeric property to sum instead of counting occurrences
	DenominatorEvent string
	DenominatorProp  string
	From, To         time.Time // half-open [From, To)
}

RatioSpec describes a ratio metric in terms of events.

The denominator has three shapes and they are genuinely different metrics:

  • DenominatorProp set (e.g. "session_id"): the analysis unit is a distinct value of that property, so the rows are per-session and the clustering diagnostic is available.
  • DenominatorEvent set: the denominator counts a second event (impressions against clicks). There is no per-analysis-unit decomposition, so no winsorization and no naive comparison.
  • neither: one analysis unit per randomisation unit, i.e. a plain per-user mean.

type RatioUnit added in v0.29.0 

type RatioUnit struct {
	Unit    string    `json:"unit"`
	Variant string    `json:"variant"`
	Num     float64   `json:"num"`
	Den     float64   `json:"den"`
	Values  []float64 `json:"values,omitempty"`
}

RatioUnit is one randomisation unit's contribution to a ratio metric.

Num and Den are the aggregates the estimator actually uses. Values is optional and carries the per-analysis-unit numerators when the analysis unit is enumerable (per-session revenue, say); it buys winsorization and the clustering diagnostic, and when it is absent both are reported as unavailable rather than approximated.

func BuildRatioUnits added in v0.29.0 

func BuildRatioUnits(evs []event.Event, spec RatioSpec) ([]RatioUnit, string, error)

BuildRatioUnits aggregates raw events into per-randomisation-unit rows, counting only what happened at or after each unit's first exposure — the same attribution rule the binary path uses, so the two never disagree about which events belong to the experiment.

type Report added in v0.9.7 

type Report struct {
	Flag     string          `json:"flag"`
	Goal     string          `json:"goal"`
	Days     int             `json:"days"`
	Control  string          `json:"control"`
	Variants []VariantResult `json:"variants"`
	Note     string          `json:"note"`
	// The DESIGN, echoed. Every number above is conditional on these, and a reader who cannot
	// see them cannot check the result — "significant" means nothing without the alpha it was
	// tested at or the mode it was read under. Stated rather than assumed, on every response.
	Mode        string  `json:"mode"` // sequential | fixed — what actually drew the intervals
	ModeReason  string  `json:"mode_reason,omitempty"`
	Alpha       float64 `json:"alpha"`
	NTune       int     `json:"n_tune"`
	NTuneSource string  `json:"n_tune_source"`
	NPlanned    int     `json:"n_planned,omitempty"`
	// Unit names the randomisation unit the arms were counted over. bucket_id is device-scoped
	// and survives identify(); distinct_id changes at login. Counting one and calling it the
	// other is two different estimands wearing one label.
	Unit string `json:"randomisation_unit"`
	// Planned is nil when the flag carries no pre-registered plan, and the note says so. A
	// missing plan is a fact about the experiment, not a detail to paper over with defaults.
	Planned bool `json:"pre_registered"`
	// From/To are the resolved window, echoed so a re-run is reproducible without re-deriving
	// what "30 days" meant at the moment this ran.
	From string `json:"from,omitempty"`
	To   string `json:"to,omitempty"`
}

Report is the A/B read for one measured flag: for each variant, how many exposed users converted on the goal event AFTER their first exposure, and whether the lift over the control arm is statistically significant. Pure + deterministic (same events → same report), so it is pinnable MCP==API by an agreement test, the same contract as every other report.

func Measure added in v0.9.7 

func Measure(evs []event.Event, flagKey, goal string, days int) Report

Measure computes the report from raw events. An exposure is a $feature_flag_called event tagging the user's variant for this flag; a conversion is the user doing `goal` at or after their first exposure (so we never credit behavior that predates the experiment). Only events within the last `days` (0 = all) are considered.

Deprecated in favour of MeasureRange: `days` is resolved against time.Now() INSIDE this function, so the same events return a different report as the clock moves. Kept so existing callers keep compiling; every one of them should resolve its own window and pass instants.

func MeasureRange added in v0.29.0 

func MeasureRange(evs []event.Event, f Flag, goal string, from, to time.Time) Report

MeasureRange is Measure over an explicit half-open window [from, to), told which flag it is measuring rather than only its key.

Four defects made this necessary, all of them producing a confidently wrong answer:

  1. THE CLOCK. `days` was turned into a cutoff with time.Now() inside the pure computation, so the same input produced different output every time it ran. On a product whose claim is "recomputed from raw events, byte-identical answers", the experiment report was the one thing that could not be reproduced.

  2. CONTROL WAS ALPHABETICAL. `sort.Strings(variants)[0]` — so an experiment with arms named {control, a_new} elected `a_new` as the control and INVERTED THE SIGN OF EVERY LIFT. The winning arm reads as the losing one. Control is now the flag's declared first variant.

  3. A DEAD ARM VANISHED. byVariant was built purely from observed exposures, so an arm nobody was ever exposed to simply was not in the report. That is the single most important failure to surface — it means the treatment code never ran — and it rendered as a clean one-arm experiment with nothing wrong.

  4. A GOAL NAMED LIKE THE EXPOSURE EVENT WAS UNREACHABLE. `case ExposureEvent:` and `case goal:` shared one switch, so measuring a goal of "$feature_flag_called" silently counted zero conversions forever.

  5. REPEAT CONVERTERS WERE SCORED ON THEIR FIRST GOAL EVENT EVER. Only the MINIMUM goal timestamp per user was kept, then compared to their exposure — so a user who purchased once before the experiment started and again after being exposed landed in the denominator and never in the numerator, forever. On any repeatable goal (purchase, $pageview, session_start) that is every returning customer, and it understates both arms unequally whenever exposure timing differs between them. Measured on the three-event case {goal T0, exposure T1, goal T2}: exposed=1, converted=0.

`to` zero means "no upper bound"; `from` zero means "all history".

type Rule 

type Rule struct {
	Filters    []query.Filter `json:"filters,omitempty"`
	RolloutPct int            `json:"rollout_pct"`
}

Rule is one ordered targeting clause: the user's context must pass all Filters (empty = every user), and RolloutPct (0..100) is the deterministic share of matched users served. Rules are evaluated in order, first match wins.

type SRMResult added in v0.23.0 

type SRMResult struct {
	Checked   bool               `json:"checked"`
	Detected  bool               `json:"detected"`
	PValue    float64            `json:"p_value"`
	ChiSquare float64            `json:"chi_square"`
	Observed  map[string]int     `json:"observed"`
	Expected  map[string]float64 `json:"expected"`
	Total     int                `json:"total"`
	// Culprit names the segment whose split is most skewed, when one stands out. This is the
	// difference between "your experiment is broken" and "your experiment is broken, and it is
	// iOS users: 340 in control against 91 in test".
	Culprit string `json:"culprit,omitempty"`
	// WindowDays is the window the check actually ran over (0 = all history). It exists because
	// the result was not self-describing: a caller that asks for all-time and is silently given
	// 30 days gets back "traffic split looks correct" with nothing saying which window that
	// sentence is about, so an SRM on day 40 of a long-running experiment reads as a clean bill
	// of health. Echoing the window makes the number checkable against what was requested.
	WindowDays int    `json:"window_days"`
	Verdict    string `json:"verdict"`
}

SRMResult is the health check for one experiment's traffic split.

func CheckSRM added in v0.23.0 

func CheckSRM(evs []event.Event, f Flag, days int) SRMResult

CheckSRM compares how many users were actually exposed to each arm against the flag's configured weights.

Only arms the flag actually declares are counted. An exposure tagged with a variant that no longer exists is itself a problem, but it is a different one — folding it into the chi-square would blame the split for a stale SDK.

type SeqParams added in v0.29.0 

type SeqParams struct {
	Mode        string  `json:"mode"`
	Alpha       float64 `json:"alpha"`
	NTune       int     `json:"n_tune"`
	NTuneSource string  `json:"n_tune_source"`
}

SeqParams is the pre-registered analysis configuration. Every field is echoed in SeqStats rather than assumed, because an interval whose alpha, mode and tuning point are invisible is an interval a reader has to take on faith.

func SeqDefaults added in v0.29.0 

func SeqDefaults() SeqParams

SeqDefaults is the configuration an experiment gets when it declares nothing: always-valid inference at 95%, tuned for 5,000 exposed units, and honest in the payload about the fact that 5,000 was nobody's decision.

func SeqParamsFor added in v0.29.0 

func SeqParamsFor(mode string, alpha float64, nPlanned int) SeqParams

SeqParamsFor builds the parameters from what the experiment actually declared. alpha <= 0 falls back to 0.05; an empty mode falls back to sequential, never to fixed — a missing mode is an experiment nobody configured, and the safe read for an unconfigured experiment is the one that survives peeking.

func (SeqParams) Validate added in v0.29.0 

func (p SeqParams) Validate() error

Validate rejects configurations whose intervals would be meaningless rather than merely wide.

type SeqStats added in v0.29.0 

type SeqStats struct {
	// ModeDeclared is what the experiment pre-registered. Mode is what was actually used to draw
	// this interval. They differ exactly when a fixed-horizon test is being read too early, and
	// ModeReason says so in words rather than leaving the reader to notice.
	ModeDeclared string  `json:"mode_declared"`
	Mode         string  `json:"mode"`
	ModeReason   string  `json:"mode_reason,omitempty"`
	Alpha        float64 `json:"alpha"`
	NTune        int     `json:"n_tune"`
	NTuneSource  string  `json:"n_tune_source"`
	NPlanned     int     `json:"n_planned,omitempty"`

	N int `json:"n"` // total exposed units across both arms — the N that sets the width

	// K is the inflation over the fixed-horizon half-width. Always > 1 in sequential mode, exactly
	// 1 in fixed mode. Printed because "your interval is 1.55x wider than a fixed test's" is the
	// honest price of being allowed to look whenever you want, and hiding the price reads as the
	// method being free.
	K float64 `json:"k"`
	// Z is the critical value the fixed half-width uses, and Z*K is what to hand liftInterval so
	// the relative-lift interval inflates by exactly the same factor as the absolute one. Two
	// intervals on one row disagreeing about how conservative they are is worse than either.
	Z     float64 `json:"z"`
	LiftZ float64 `json:"lift_z"`

	SE        float64 `json:"se"`         // the standard error this interval was built from
	Diff      float64 `json:"diff"`       // p_test - p_control
	HalfWidth float64 `json:"half_width"` // K * Z * SE
	Lo        float64 `json:"lo"`
	Hi        float64 `json:"hi"`

	// AlwaysValidP is the alpha at which this difference would just barely be called — the
	// anytime-valid p-value, NEVER the fixed-horizon one. Named in full because a field called
	// `p_value` next to a sequential interval would be read as the number everybody already knows,
	// and the two differ by a lot.
	AlwaysValidP float64 `json:"always_valid_p"`
	// Excludes0 is the decision: does the interval exclude no-difference. It is derived from the
	// interval actually rendered, so the sentence and the picture can never disagree.
	Excludes0 bool `json:"excludes_zero"`
	// Usable is false when there is nothing to compute from (no exposures, or an arm with no
	// variance yet). The zero interval is not a claim of "no effect" and this flag is what stops
	// it being read as one.
	Usable bool `json:"usable"`
}

SeqStats is the always-valid read, with every input that shaped it stated alongside.

Absolute quantities (Diff, SE, HalfWidth, Lo, Hi) are on the PROPORTION scale — 0.031 means 3.1 percentage points — because that is the scale the arithmetic happens on. DiffInterval() converts to the percentage convention the rest of the report renders in; mixing the two silently is how a reader misreads by 100x.

func SeqCompute added in v0.29.0 

func SeqCompute(diff, se float64, n, nPlanned int, par SeqParams) SeqStats

SeqCompute is the whole always-valid read for an absolute difference in rates.

It takes a difference and a standard error rather than four counts, and that signature is the composition rule with CUPED (spec item 12), not an accident. CUPED adjusts the estimate and shrinks the SE; the confidence sequence then inflates whatever half-width it is handed. Order: adjust FIRST, inflate SECOND. Handing this function the raw SE and applying the variance reduction afterwards inflates a quantity that no longer matches the estimate, and the interval silently stops covering. Callers echo the raw and adjusted SE side by side so the reduction is visible rather than asserted.

n is the TOTAL exposed units across both arms — the CS is a statement about the experiment, not about one arm. nPlanned is the pre-registered sample size (0 when none was declared) and is used only for the mode decision.

func (SeqStats) DiffInterval added in v0.29.0 

func (s SeqStats) DiffInterval() Interval

DiffInterval renders the absolute difference in percentage points, matching the units every other Interval on the report carries.

type Store 

type Store struct {
	// contains filtered or unexported fields
}

Store persists flags to a JSON file (atomic tmp+rename), same discipline as the cohort and deploy stores. Flags are keyed by their stable Key (e.g. "checkout_v2"), so Save is an upsert: creating or updating the flag with that key. Empty path = in-memory only.

func Open 

func Open(path string) (*Store, error)

func (*Store) Delete 

func (s *Store) Delete(key string) (found bool, err error)

Delete removes a flag by key. found is true only when a flag actually went away, so callers can report "nothing was deleted" instead of implying a removal that never happened. Deleting a key that isn't there is not an error (retries are fine).

func (*Store) Get 

func (s *Store) Get(key string) (Flag, bool)

func (*Store) List 

func (s *Store) List() []Flag

func (*Store) OnFlip added in v0.61.0 

func (s *Store) OnFlip(fn func(key string, on bool))

OnFlip registers a callback fired after a flag is successfully enabled or disabled.

It lives on the STORE rather than at each call site because there were two call sites already (the MCP tool and the auto-reverter) and no reason to think there would not be a third. The hook is optional and nil-safe: a self-hosted instance without a deploy store keeps working exactly as before.

No import of internal/deploys here — a plain function keeps the dependency pointing one way, and the wiring happens once at startup where both stores already exist.

func (*Store) Save 

func (s *Store) Save(f Flag) (Flag, error)

Save upserts by Key. A new key is created (Created stamped); an existing key is updated in place (Created preserved, Updated bumped). Validates the key and the variant weights. Save persists a flag. Plan changes go through the lock.

exposedUsers is unknown here — this store holds flags, not events — so the error message cannot say how many people are already in the experiment. Callers that DO know should use SaveWithExposure, which produces the message an operator can act on. The LOCK itself is enforced either way; only the wording degrades.

func (*Store) SaveWithExposure added in v0.29.0 

func (s *Store) SaveWithExposure(f Flag, exposedUsers int) (Flag, error)

SaveWithExposure is Save with the exposure count, for the error message and for the layer rule.

This is the enforcement point experiment.go's own comment promised — "attach it to a Flag and Store.Save enforces the lock" — and which nothing implemented. A pre-registered plan that can be silently re-aimed after the data arrives is not pre-registration, it is a comment. Verified by changing a locked plan's goal through the MCP tool and watching it succeed.

func (*Store) SetEnabled 

func (s *Store) SetEnabled(key string, on bool) (Flag, error)

SetEnabled toggles a flag on/off by key (the common flip). Returns the updated flag, and records the flip as a deploy marker when a recorder is wired.

FEATURES.md has claimed "flag → deploy marker: auto-recorded" under "Only we have" for months, while the comment here said "a future increment records this flip". It was a straight contradiction in the strongest-worded part of the comparison table. Making it true was the right resolution rather than deleting the claim: flipping a flag IS a ship, and it is the one kind of ship this tool can detect with no CI setup at all.

type Variant 

type Variant struct {
	Key    string `json:"key"`
	Weight int    `json:"weight"`
}

Variant is one arm of a multivariate flag; Weight is its relative share (need not sum to 100).

type VariantResult added in v0.9.7 

type VariantResult struct {
	Key       string `json:"key"`
	Exposed   int    `json:"exposed"`
	Converted int    `json:"converted"`
	// RatePct is the point estimate; RateCI is where the true rate plausibly sits. Wilson
	// interval, so it never reports a bound below 0% or above 100% on a small arm.
	RatePct float64  `json:"rate_pct"`
	RateCI  Interval `json:"rate_ci"`
	// DeltaPct is relative lift vs control. DeltaCI is its interval, present only when the
	// control rate is far enough from zero for a ratio to be meaningful — otherwise a handful of
	// conversions prints "+4000%" and someone ships on it.
	//
	// DeltaDefined says whether DeltaPct is a real number. It has to exist because 0 was doing
	// double duty: a control at 0 of 200 against a test arm at 60 of 200 — an infinite lift, the
	// single biggest win the tool can find — left DeltaPct at its zero value, so the row rendered
	// "0" (neutral, "no change") next to "significant at 95% against control". Undefined and
	// genuinely flat were indistinguishable to every reader and every consumer. False on the
	// control row too: an arm has no lift against itself, and 0 there is a convention, not a
	// measurement.
	DeltaPct     float64   `json:"delta_pct"`
	DeltaDefined bool      `json:"delta_defined"`
	DeltaCI      *Interval `json:"delta_ci,omitempty"`
	PValue       float64   `json:"p_value,omitempty"`
	Significant  bool      `json:"significant"`  // 95% two-proportion z-test vs control
	SmallSample  bool      `json:"small_sample"` // too few exposed to trust the rate
	// Read is the sentence a person should act on. "Significant" invites shipping; "the range
	// still spans zero" invites waiting, which is usually the right call and one a boolean
	// never prompts.
	Read string `json:"read,omitempty"`
	// Seq is the always-valid machinery behind DeltaCI: the inflation factor K, the critical
	// value it used, and the always-valid p-value. Present on every compared arm so the width
	// of the interval can be traced to the method rather than taken on faith.
	Seq *SeqStats `json:"sequential,omitempty"`
}

VariantResult is one arm of a measured flag's A/B read.

Every rate ships with the raw numerator and denominator that produced it and a 95% interval around it. A bare percentage asks to be trusted; 43 of 512, somewhere between 6.3% and 11.1%, can be checked. That is the difference this product claims to sell.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.