Documentation
¶
Overview ¶
Package experiments is A/B testing anything: a flag, an ad, a subject line, a model.
It is the unified EXPERIMENT primitive — ONE value whatever the variant KIND is — and a COMPOSITION of three planes that already exist, never a fourth engine:
ASSIGNMENT = flags — subject -> variant is a deterministic flags evaluation
(engineEvaluate, sha1 rollout hash). No 2nd bucketing.
MEASUREMENT = analytics — a subject's outcome events are already captured by
distinct_id in event.event. No 2nd event store.
EVIDENCE = research — per-variant samples land as immutable evidence rows
(kind "ab"); significance is a pure function over them.
The experiment is the VALUE that composes them:
Experiment = { id, org, name, subjectKind, variants (payload is variant-kind
AGNOSTIC), flagKey (-> the assignment def), metric/exposure events
(-> the analytics outcome grain), status, winner }
The lifecycle: create registers a multivariate flag def (flags.PutDef); assign is a flags evaluation (flags.Assign, deterministic); analyze folds analytics outcomes per variant and computes lift + significance, persisting the samples to research; decide promotes a winner by rewriting the flag's variant weights to 100% for it.
The variant KIND is orthogonal: variant.payload can be a feature config (feature experiment), an ad-creative id (campaign experiment), an email subject, a model id — the primitive does not care. apps/campaign composes THIS to run a creative A/B; it does not reinvent assignment or evidence.
Mounted into the unified cloud binary via apps.go ({Name:"experiments", Mount}).
Index ¶
- func Assign(ctx context.Context, org, project, experimentID, subject string, ...) (flags.Assignment, error)
- func Mount(app cloud.Router, deps cloud.Deps) error
- func Shutdown() error
- type Analysis
- type Experiment
- type MetricOutcome
- type MetricSource
- type Result
- type Status
- type SubjectKind
- type Variant
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func Assign ¶
func Assign(ctx context.Context, org, project, experimentID, subject string, props json.RawMessage) (flags.Assignment, error)
Assign is the in-process assignment seam: resolve the experiment's flag and return the subject's variant + payload. clients/campaign composes THIS to pick a creative (variant.payload) per subject — it never reinvents the bucketing. Org-scoped and fail-closed (an unknown experiment is an error, not a silent default).
Types ¶
type Analysis ¶
type Analysis struct {
Experiment string `json:"experiment"`
Metric string `json:"metric"`
Alpha float64 `json:"alpha"`
Results []Result `json:"results"`
Winner string `json:"winner"`
ExposedTotal int `json:"exposedTotal"`
}
Analysis is the experiment's full read: per-variant Results plus the advisory Winner (the significant treatment with the highest rate that beats control, else "" when inconclusive). The Winner is advisory — decide takes an explicit choice.
func Analyze ¶
func Analyze(ctx context.Context, org, project, experimentID string, start, end time.Time, alpha float64) (Analysis, error)
Analyze is the in-process analysis seam (campaign's scheduler / a cron composes it to refresh an experiment's significance): resolve the experiment, then run the full analytics x flags x research analysis over [start,end).
type Experiment ¶
type Experiment struct {
Project string `json:"project"`
ID string `json:"id"`
Name string `json:"name"`
SubjectKind SubjectKind `json:"subjectKind"`
FlagKey string `json:"flagKey"`
ExposureEvent string `json:"exposureEvent"`
MetricEvent string `json:"metricEvent"`
Variants []Variant `json:"variants"`
Status Status `json:"status"`
Winner string `json:"winner,omitempty"`
CreatedBy string `json:"createdBy,omitempty"`
CreatedAt string `json:"createdAt,omitempty"`
DecidedBy string `json:"decidedBy,omitempty"`
DecidedAt string `json:"decidedAt,omitempty"`
}
Experiment is the primitive: the definition + lifecycle of one controlled experiment. Project + ID are the server-stamped identity (project from the validated principal, never a client field). FlagKey links the assignment plane; MetricEvent + ExposureEvent link the measurement plane.
type MetricOutcome ¶
MetricOutcome is one subject's exposure + conversion for an experiment window — the experiments-package grain of a measurement, decoupled from the analytics wire type so the pure analysis core never imports the warehouse.
type MetricSource ¶
type MetricSource interface {
Outcomes(ctx context.Context, org, exposureEvent, metricEvent string, start, end time.Time) ([]MetricOutcome, error)
}
MetricSource is the measurement seam: per-subject outcomes for two event names over a window, org-scoped by the impl. Production is analyticsSource (the ONE analytics events plane); tests inject a fake. This is the only seam to the measurement half — there is no second event store.
type Result ¶
type Result struct {
Variant string `json:"variant"`
Control bool `json:"control"`
Exposed int `json:"exposed"`
Converted int `json:"converted"`
Rate float64 `json:"rate"`
Lift float64 `json:"lift"` // relative to control: (rate-ctrl)/ctrl
Z float64 `json:"z"` // two-proportion z vs control
PValue float64 `json:"pValue"` // two-tailed p vs control
Significant bool `json:"significant"` // pValue < alpha
}
Result is one variant's measured outcome and its comparison to the control arm. Lift/Z/PValue/Significant are 0/false for the control itself (it is its own baseline).
type Status ¶
type Status string
Status is an experiment's lifecycle state: running (assigning + measuring) or decided (a winner promoted to 100% of the rollout).
type SubjectKind ¶
type SubjectKind string
SubjectKind is the unit an experiment assigns and measures: a user, an org, a session, or a named audience. It selects the distinct_id grain the flags assignment hashes and the analytics outcomes fold on.
const ( SubjectUser SubjectKind = "user" SubjectOrg SubjectKind = "org" SubjectSession SubjectKind = "session" SubjectAudience SubjectKind = "audience" )
type Variant ¶
type Variant struct {
Key string `json:"key"`
Weight float64 `json:"weight"`
Control bool `json:"control,omitempty"`
Payload json.RawMessage `json:"payload,omitempty"`
}
Variant is one arm of an experiment: a key, its rollout Weight within the experiment (percentages across variants sum to 100), whether it is the Control (baseline) arm, and a variant-kind-AGNOSTIC Payload the assignment carries — a feature config, an ad-creative id, an email subject, a model id. The experiment primitive never interprets the payload; the consumer does.
Source Files