roi

package
v0.6.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 3, 2026 License: Apache-2.0 Imports: 23 Imported by: 0

Documentation

Overview

Package roi measures the delivery impact of a policy change (ADR-0027).

It ingests pull-request outcomes (GitHub, or a generic CSV/JSONL export from any SCM or tracker), maps their authors to Halos subjects, recovers each developer's arm with the same assignment the fleet used (policy.ResolveRing and Experiment.AssignVariant, both on internal/assign), and estimates the difference between arms with a developer-clustered bootstrap. Only an experiment's variants are randomized; a comparison between rings is labelled observational and never reported as an effect.

Index

Constants

View Source
const (
	// DesignRandomized: arms are an experiment's variants, assigned by hash.
	// Differences estimate causal (intent-to-treat) effects.
	DesignRandomized = "randomized"
	// DesignObservational: arms are rings. Membership is chosen (users,
	// groups, opt-in, rollout order), so differences are associations.
	DesignObservational = "observational"
)

Designs: what a report may claim.

View Source
const (
	StateSignificant        = "significant"          // CI excludes 0
	StateNoDetectableEffect = "no_detectable_effect" // CI includes 0 and the study could have seen an effect of MDE
	StateInsufficientPower  = "insufficient_power"   // CI includes 0 and the minimum detectable effect is above MDE
	StateInsufficientData   = "insufficient_data"    // fewer than MinDevelopers per arm
)

Effect states.

View Source
const (
	StatusOK               = "ok"
	StatusSRM              = "srm" // sample ratio mismatch: do not trust the effects
	StatusInsufficientData = "insufficient_data"
)

Report statuses.

View Source
const (
	MetricCycleTime     = "pr_cycle_time_hours"
	MetricThroughput    = "merged_prs_per_dev_week"
	MetricReviewRounds  = "review_rounds_per_pr"
	MetricChangeFailure = "change_failure_rate"
	MetricSize          = "lines_changed_per_pr"
)

Metric names.

View Source
const GitHubGraphQL = "https://api.github.com/graphql"

GitHubGraphQL is the public GraphQL endpoint. GitHub Enterprise Server serves it at https://<host>/api/graphql.

View Source
const Table = "halo_delivery_prs"

Table is the ClickHouse table `halo roi ingest` writes (deploy/observability/clickhouse/schema.sql).

Variables

This section is empty.

Functions

func Open

func Open(ch *promote.ClickHouse, path string) (st Store, cost CostSource, err error)

Open picks the sources. PRs come from the JSONL file at path when set, else from ClickHouse; AI spend comes from ClickHouse's halo_metrics when ch is set (cost is nil otherwise).

func ParseTime

func ParseTime(s string) (time.Time, error)

ParseTime accepts RFC 3339 or a date (YYYY-MM-DD, UTC midnight).

func PromotionSection

func PromotionSection(ctx context.Context, st Store, cost CostSource, ids *Identities, org *policy.Org, e *policy.Experiment, from, to time.Time) (string, error)

PromotionSection is the Markdown for e's promotion PR over [from, to), or "" when no rostered developer in the experiment has a PR yet. It is the only way ROI reaches internal/promote, and only as text for the reviewer.

func Window

func Window(ctx context.Context, ch *promote.ClickHouse, e *policy.Experiment, since, until string, now time.Time) (from, to time.Time, err error)

Window resolves a report window from since/until (ParseTime). until defaults to now; since defaults to e's first telemetry in ClickHouse, so it is required without an experiment or ch.

Types

type Arm

type Arm struct {
	Name string `json:"name"`
	// Weight is the configured share (randomized only).
	Weight float64 `json:"weight,omitempty"`
	// Developers on the roster assigned to the arm; Active opened >= 1 PR.
	Developers int `json:"developers"`
	Active     int `json:"active"`
	PRs        int `json:"prs"`
	Merged     int `json:"merged"`
	// AI spend over the window and the arm-level cost per merged PR, when a
	// cost source has rows for the arm. Spend and PRs are joined per arm,
	// not per developer, so there is no CI on CostPerMergedPR.
	SpendUSD        float64 `json:"spendUSD,omitempty"`
	CostUnits       int     `json:"costUnits,omitempty"`
	CostPerMergedPR float64 `json:"costPerMergedPR,omitempty"`
}

Arm is one arm's population and volume.

type ArmCost

type ArmCost struct {
	USD   float64 `json:"usd"`
	Units int     `json:"units"`
}

ArmCost is one arm's AI spend over a window.

type ClickHouseStore

type ClickHouseStore struct{ CH *promote.ClickHouse }

ClickHouseStore keeps PRs in halo_delivery_prs and reads AI spend from halo_metrics, through the same client `halo exp analyze` uses.

func (ClickHouseStore) ArmCosts

func (s ClickHouseStore) ArmCosts(ctx context.Context, experiment string, rings []string, from, to time.Time) (map[string]ArmCost, string, error)

ArmCosts sums halo.cost.usd per arm. Rows the collector stamped on its authenticated gateway receiver win; otherwise the CLI-reported rows are used and labelled promote.SourceCLI (client-supplied).

func (ClickHouseStore) PRs

func (s ClickHouseStore) PRs(ctx context.Context, from, to time.Time) ([]PR, error)

PRs reads the latest version of each PR opened in [from, to).

func (ClickHouseStore) Put

func (s ClickHouseStore) Put(ctx context.Context, prs []PR) error

Put inserts prs; ReplacingMergeTree(ingested_at) keeps the newest per key.

type CostSource

type CostSource interface {
	ArmCosts(ctx context.Context, experiment string, rings []string, from, to time.Time) (map[string]ArmCost, string, error)
}

CostSource reports AI spend per arm (variant of experiment, or ring when experiment is "") and which receiver the rows came through.

type Effect

type Effect struct {
	Metric string `json:"metric"`
	Arm    string `json:"arm"` // the treatment arm compared with the control
	// Kind is "causal" (randomized) or "association" (observational).
	Kind   string `json:"kind"`
	Better string `json:"better,omitempty"` // decrease | increase; "" = descriptive
	State  string `json:"state"`
	// Outcome is improved | worsened for a significant effect with a direction.
	Outcome string `json:"outcome,omitempty"`
	stats.DiffResult
	// MDERel is the relative effect detectable with 80% power at Alpha.
	MDERel  float64  `json:"mdeRel,omitempty"`
	Slices  []Effect `json:"slices,omitempty"`
	Novelty bool     `json:"novelty,omitempty"`
	From    string   `json:"from,omitempty"` // set on slices
}

Effect is one metric's treatment-vs-control comparison.

type FileStore

type FileStore struct{ Path string }

FileStore is a JSONL file of PRs: the local mode for tests and for teams without ClickHouse.

func (FileStore) PRs

func (f FileStore) PRs(_ context.Context, from, to time.Time) ([]PR, error)

PRs returns the PRs opened in [from, to).

func (FileStore) Put

func (f FileStore) Put(_ context.Context, prs []PR) error

Put merges prs into the file (atomic rename).

type GitHub

type GitHub struct {
	URL   string // GraphQL endpoint; default GitHubGraphQL
	Token string
	// HotfixLabel marks a hotfix PR; a merged one marks every PR it
	// references (#N) as a change failure. Default "hotfix".
	HotfixLabel string
	Client      *http.Client // default: 60s timeout, no redirects
}

GitHub reads pull requests through the GraphQL API with a read-only query. The token is sent only in the Authorization header and scrubbed from every error; redirects are never followed.

func (*GitHub) Fetch

func (g *GitHub) Fetch(ctx context.Context, repo string, since time.Time) ([]PR, error)

Fetch returns the PRs of repo ("owner/name") created at or after since, with change failures marked: a merged PR whose body carries GitHub's revert line "Reverts owner/name#N", or a merged hotfix-labelled PR that references #N, marks PR N. Only PRs fetched in the same call are marked, so fetch a window that also covers the follow-ups.

type Identities

type Identities struct {
	// contains filtered or unexported fields
}

Identities is the roster: SCM logins mapped to Halos subjects with the IdP groups ring membership uses. Developers missing from it are unmapped and never guessed.

func LoadIdentities

func LoadIdentities(path string) (*Identities, error)

LoadIdentities reads a CSV with a header: login,subject[,groups][,kind]. groups are separated by ';'. kind "workload" marks a CI or bot identity, which is excluded. Logins compare case-insensitively; subjects are canonicalised with policy.CanonicalUser.

func ParseIdentities

func ParseIdentities(r io.Reader) (*Identities, error)

ParseIdentities is LoadIdentities on a reader.

func (*Identities) Resolve

func (ids *Identities) Resolve(p PR) (policy.Subject, bool)

Resolve maps a PR's author to a subject: its Subject when set, else its login. ok is false when the identity is not on the roster.

func (*Identities) Subjects

func (ids *Identities) Subjects() []policy.Subject

Subjects is the roster in file order.

type Options

type Options struct {
	Org *policy.Org
	// Experiment (randomized) or Rings (observational; the first is the
	// baseline); exactly one.
	Experiment *policy.Experiment
	Rings      []string
	From, To   time.Time
	// Slices splits the window for the novelty check (default 3; < 2 off).
	Slices int
	// MDE is the relative effect the study must be able to detect for a null
	// result to count as "no detectable effect" (default 0.10).
	MDE float64
	// Alpha for CIs (default 0.05); SRMAlpha for the SRM test (default 0.001).
	Alpha, SRMAlpha float64
	// MinDevelopers per arm before an effect is estimated (default 10).
	MinDevelopers int
	Iters         int    // bootstrap resamples (default 2000)
	Seed          uint64 // bootstrap seed (default 1)
}

Options selects the arms and the window.

type PR

type PR struct {
	Source string `json:"source"` // github | file
	Repo   string `json:"repo"`
	Number int64  `json:"number"`
	// Author is the SCM login, lowercased. Subject, when the source knows it
	// (a generic export), is the Halos subject id and wins over Author.
	Author  string `json:"author,omitempty"`
	Subject string `json:"subject,omitempty"`
	// Bot marks an author the SCM reports as a bot account; never counted.
	Bot          bool       `json:"bot,omitempty"`
	OpenedAt     time.Time  `json:"opened_at"`
	MergedAt     *time.Time `json:"merged_at,omitempty"`
	ReviewRounds int        `json:"review_rounds"`
	Additions    int        `json:"additions"`
	Deletions    int        `json:"deletions"`
	// ChangeFailure is the change-failure proxy: the merged PR was later
	// reverted or followed by a hotfix that references it.
	ChangeFailure bool `json:"change_failure,omitempty"`
}

PR is one pull/merge request outcome. Titles, bodies and diffs are never stored: only what the estimators need.

func ReadFile

func ReadFile(path string) ([]PR, error)

ReadFile parses a generic export: .csv (header row naming the PR JSON fields) or .jsonl/.ndjson (one PR object per line). Every record is checked; Source defaults to "file" and logins are lowercased.

func (PR) Check

func (p PR) Check() error

Check rejects a record the estimators cannot use.

type Report

type Report struct {
	Design     string           `json:"design"`
	Claim      string           `json:"claim"`
	Status     string           `json:"status"`
	Experiment string           `json:"experiment,omitempty"`
	Rings      []string         `json:"rings,omitempty"`
	From       time.Time        `json:"from"`
	To         time.Time        `json:"to"`
	Control    string           `json:"control"`
	Arms       []Arm            `json:"arms"`
	SRM        *stats.SRMResult `json:"srm,omitempty"`
	Effects    []Effect         `json:"effects"`
	CostSource string           `json:"costSource,omitempty"`
	// Excluded counts PRs left out, by reason.
	Excluded       map[string]int `json:"excluded"`
	UnmappedLogins []string       `json:"unmappedLogins,omitempty"` // first 20
	Warnings       []string       `json:"warnings"`
}

Report is a delivery-ROI analysis.

func Analyze

func Analyze(ctx context.Context, store Store, cost CostSource, ids *Identities, o Options) (*Report, error)

Analyze builds the report from the store's PRs, the identity roster and, when cost is non-nil, AI spend per arm.

func (*Report) Markdown

func (rep *Report) Markdown() string

Markdown renders the report for a promotion PR body. It is informational: nothing in Halos promotes on it.

type Store

type Store interface {
	Put(ctx context.Context, prs []PR) error
	// PRs returns the PRs opened in [from, to).
	PRs(ctx context.Context, from, to time.Time) ([]PR, error)
}

Store holds ingested PRs. Put replaces records with the same (source, repo, number), so re-ingesting a window refreshes late reverts and merges.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL