Documentation
¶
Overview ¶
Package roi measures the delivery impact of a policy change (ADR-0027).
It ingests pull-request outcomes (GitHub, or a generic CSV/JSONL export from any SCM or tracker), maps their authors to Halos subjects, recovers each developer's arm with the same assignment the fleet used (policy.ResolveRing and Experiment.AssignVariant, both on internal/assign), and estimates the difference between arms with a developer-clustered bootstrap. Only an experiment's variants are randomized; a comparison between rings is labelled observational and never reported as an effect.
Index ¶
- Constants
- func Open(ch *promote.ClickHouse, path string) (st Store, cost CostSource, err error)
- func ParseTime(s string) (time.Time, error)
- func PromotionSection(ctx context.Context, st Store, cost CostSource, ids *Identities, ...) (string, error)
- func Window(ctx context.Context, ch *promote.ClickHouse, e *policy.Experiment, ...) (from, to time.Time, err error)
- type Arm
- type ArmCost
- type ClickHouseStore
- type CostSource
- type Effect
- type FileStore
- type GitHub
- type Identities
- type Options
- type PR
- type Report
- type Store
Constants ¶
const ( // DesignRandomized: arms are an experiment's variants, assigned by hash. // Differences estimate causal (intent-to-treat) effects. DesignRandomized = "randomized" // DesignObservational: arms are rings. Membership is chosen (users, // groups, opt-in, rollout order), so differences are associations. DesignObservational = "observational" )
Designs: what a report may claim.
const ( StateSignificant = "significant" // CI excludes 0 StateNoDetectableEffect = "no_detectable_effect" // CI includes 0 and the study could have seen an effect of MDE StateInsufficientPower = "insufficient_power" // CI includes 0 and the minimum detectable effect is above MDE StateInsufficientData = "insufficient_data" // fewer than MinDevelopers per arm )
Effect states.
const ( StatusOK = "ok" StatusSRM = "srm" // sample ratio mismatch: do not trust the effects StatusInsufficientData = "insufficient_data" )
Report statuses.
const ( MetricCycleTime = "pr_cycle_time_hours" MetricThroughput = "merged_prs_per_dev_week" MetricReviewRounds = "review_rounds_per_pr" MetricChangeFailure = "change_failure_rate" MetricSize = "lines_changed_per_pr" )
Metric names.
const GitHubGraphQL = "https://api.github.com/graphql"
GitHubGraphQL is the public GraphQL endpoint. GitHub Enterprise Server serves it at https://<host>/api/graphql.
const Table = "halo_delivery_prs"
Table is the ClickHouse table `halo roi ingest` writes (deploy/observability/clickhouse/schema.sql).
Variables ¶
This section is empty.
Functions ¶
func Open ¶
func Open(ch *promote.ClickHouse, path string) (st Store, cost CostSource, err error)
Open picks the sources. PRs come from the JSONL file at path when set, else from ClickHouse; AI spend comes from ClickHouse's halo_metrics when ch is set (cost is nil otherwise).
func PromotionSection ¶
func PromotionSection(ctx context.Context, st Store, cost CostSource, ids *Identities, org *policy.Org, e *policy.Experiment, from, to time.Time) (string, error)
PromotionSection is the Markdown for e's promotion PR over [from, to), or "" when no rostered developer in the experiment has a PR yet. It is the only way ROI reaches internal/promote, and only as text for the reviewer.
func Window ¶
func Window(ctx context.Context, ch *promote.ClickHouse, e *policy.Experiment, since, until string, now time.Time) (from, to time.Time, err error)
Window resolves a report window from since/until (ParseTime). until defaults to now; since defaults to e's first telemetry in ClickHouse, so it is required without an experiment or ch.
Types ¶
type Arm ¶
type Arm struct {
Name string `json:"name"`
// Weight is the configured share (randomized only).
Weight float64 `json:"weight,omitempty"`
// Developers on the roster assigned to the arm; Active opened >= 1 PR.
Developers int `json:"developers"`
Active int `json:"active"`
PRs int `json:"prs"`
Merged int `json:"merged"`
// AI spend over the window and the arm-level cost per merged PR, when a
// cost source has rows for the arm. Spend and PRs are joined per arm,
// not per developer, so there is no CI on CostPerMergedPR.
SpendUSD float64 `json:"spendUSD,omitempty"`
CostUnits int `json:"costUnits,omitempty"`
CostPerMergedPR float64 `json:"costPerMergedPR,omitempty"`
}
Arm is one arm's population and volume.
type ClickHouseStore ¶
type ClickHouseStore struct{ CH *promote.ClickHouse }
ClickHouseStore keeps PRs in halo_delivery_prs and reads AI spend from halo_metrics, through the same client `halo exp analyze` uses.
func (ClickHouseStore) ArmCosts ¶
func (s ClickHouseStore) ArmCosts(ctx context.Context, experiment string, rings []string, from, to time.Time) (map[string]ArmCost, string, error)
ArmCosts sums halo.cost.usd per arm. Rows the collector stamped on its authenticated gateway receiver win; otherwise the CLI-reported rows are used and labelled promote.SourceCLI (client-supplied).
type CostSource ¶
type CostSource interface {
ArmCosts(ctx context.Context, experiment string, rings []string, from, to time.Time) (map[string]ArmCost, string, error)
}
CostSource reports AI spend per arm (variant of experiment, or ring when experiment is "") and which receiver the rows came through.
type Effect ¶
type Effect struct {
Metric string `json:"metric"`
Arm string `json:"arm"` // the treatment arm compared with the control
// Kind is "causal" (randomized) or "association" (observational).
Kind string `json:"kind"`
Better string `json:"better,omitempty"` // decrease | increase; "" = descriptive
State string `json:"state"`
// Outcome is improved | worsened for a significant effect with a direction.
Outcome string `json:"outcome,omitempty"`
stats.DiffResult
// MDERel is the relative effect detectable with 80% power at Alpha.
MDERel float64 `json:"mdeRel,omitempty"`
Slices []Effect `json:"slices,omitempty"`
Novelty bool `json:"novelty,omitempty"`
From string `json:"from,omitempty"` // set on slices
}
Effect is one metric's treatment-vs-control comparison.
type FileStore ¶
type FileStore struct{ Path string }
FileStore is a JSONL file of PRs: the local mode for tests and for teams without ClickHouse.
type GitHub ¶
type GitHub struct {
URL string // GraphQL endpoint; default GitHubGraphQL
Token string
// HotfixLabel marks a hotfix PR; a merged one marks every PR it
// references (#N) as a change failure. Default "hotfix".
HotfixLabel string
Client *http.Client // default: 60s timeout, no redirects
}
GitHub reads pull requests through the GraphQL API with a read-only query. The token is sent only in the Authorization header and scrubbed from every error; redirects are never followed.
func (*GitHub) Fetch ¶
Fetch returns the PRs of repo ("owner/name") created at or after since, with change failures marked: a merged PR whose body carries GitHub's revert line "Reverts owner/name#N", or a merged hotfix-labelled PR that references #N, marks PR N. Only PRs fetched in the same call are marked, so fetch a window that also covers the follow-ups.
type Identities ¶
type Identities struct {
// contains filtered or unexported fields
}
Identities is the roster: SCM logins mapped to Halos subjects with the IdP groups ring membership uses. Developers missing from it are unmapped and never guessed.
func LoadIdentities ¶
func LoadIdentities(path string) (*Identities, error)
LoadIdentities reads a CSV with a header: login,subject[,groups][,kind]. groups are separated by ';'. kind "workload" marks a CI or bot identity, which is excluded. Logins compare case-insensitively; subjects are canonicalised with policy.CanonicalUser.
func ParseIdentities ¶
func ParseIdentities(r io.Reader) (*Identities, error)
ParseIdentities is LoadIdentities on a reader.
func (*Identities) Resolve ¶
func (ids *Identities) Resolve(p PR) (policy.Subject, bool)
Resolve maps a PR's author to a subject: its Subject when set, else its login. ok is false when the identity is not on the roster.
func (*Identities) Subjects ¶
func (ids *Identities) Subjects() []policy.Subject
Subjects is the roster in file order.
type Options ¶
type Options struct {
Org *policy.Org
// Experiment (randomized) or Rings (observational; the first is the
// baseline); exactly one.
Experiment *policy.Experiment
Rings []string
From, To time.Time
// Slices splits the window for the novelty check (default 3; < 2 off).
Slices int
// MDE is the relative effect the study must be able to detect for a null
// result to count as "no detectable effect" (default 0.10).
MDE float64
// Alpha for CIs (default 0.05); SRMAlpha for the SRM test (default 0.001).
Alpha, SRMAlpha float64
// MinDevelopers per arm before an effect is estimated (default 10).
MinDevelopers int
Iters int // bootstrap resamples (default 2000)
Seed uint64 // bootstrap seed (default 1)
}
Options selects the arms and the window.
type PR ¶
type PR struct {
Source string `json:"source"` // github | file
Repo string `json:"repo"`
Number int64 `json:"number"`
// Author is the SCM login, lowercased. Subject, when the source knows it
// (a generic export), is the Halos subject id and wins over Author.
Author string `json:"author,omitempty"`
Subject string `json:"subject,omitempty"`
// Bot marks an author the SCM reports as a bot account; never counted.
Bot bool `json:"bot,omitempty"`
OpenedAt time.Time `json:"opened_at"`
MergedAt *time.Time `json:"merged_at,omitempty"`
ReviewRounds int `json:"review_rounds"`
Additions int `json:"additions"`
Deletions int `json:"deletions"`
// ChangeFailure is the change-failure proxy: the merged PR was later
// reverted or followed by a hotfix that references it.
ChangeFailure bool `json:"change_failure,omitempty"`
}
PR is one pull/merge request outcome. Titles, bodies and diffs are never stored: only what the estimators need.
type Report ¶
type Report struct {
Design string `json:"design"`
Claim string `json:"claim"`
Status string `json:"status"`
Experiment string `json:"experiment,omitempty"`
Rings []string `json:"rings,omitempty"`
From time.Time `json:"from"`
To time.Time `json:"to"`
Control string `json:"control"`
Arms []Arm `json:"arms"`
SRM *stats.SRMResult `json:"srm,omitempty"`
Effects []Effect `json:"effects"`
CostSource string `json:"costSource,omitempty"`
// Excluded counts PRs left out, by reason.
Excluded map[string]int `json:"excluded"`
UnmappedLogins []string `json:"unmappedLogins,omitempty"` // first 20
Warnings []string `json:"warnings"`
}
Report is a delivery-ROI analysis.
func Analyze ¶
func Analyze(ctx context.Context, store Store, cost CostSource, ids *Identities, o Options) (*Report, error)
Analyze builds the report from the store's PRs, the identity roster and, when cost is non-nil, AI spend per arm.
type Store ¶
type Store interface {
Put(ctx context.Context, prs []PR) error
// PRs returns the PRs opened in [from, to).
PRs(ctx context.Context, from, to time.Time) ([]PR, error)
}
Store holds ingested PRs. Put replaces records with the same (source, repo, number), so re-ingesting a window refreshes late reverts and merges.