rollout

package
v0.6.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 3, 2026 License: Apache-2.0 Imports: 26 Imported by: 0

Documentation

Overview

Package rollout drives a policy.Rollout through its steps.

Evaluate is a pure, deterministic state machine: given the rollout, its persisted State, the Evidence gathered for the live step and the time, it decides advance | hold | rollback | pause | complete. It never acts. The controller acts on decisions asymmetrically, like internal/promote: rollback and pause happen automatically; advance and complete only ever open a PR (Plan) that a human merges. Nothing here merges.

Index

Constants

View Source
const (
	GatePass    = "pass"
	GateFail    = "fail"
	GatePending = "pending"
)

Gate statuses.

View Source
const (
	ScenarioHealthy    = "healthy"    // treatment matches control on every metric
	ScenarioRegression = "regression" // the first guardrail regresses from BreachStep on
)

Simulation scenarios.

View Source
const (
	EventEnter  = "enter"  // a merged step was observed live (Index, Step)
	EventHalt   = "halt"   // the controller rolled back or paused (Detail = action, Source, Reason)
	EventResume = "resume" // the policy left "active"; the halt is acknowledged
	EventAction = "action" // Key was done at step Index (Detail: e.g. a PR URL)
)

History events.

View Source
const DocKind = "rollout-state"

DocKind is the document kind DocStore keeps rollout state under.

Variables

View Source
var ErrTampered = errors.New("rollout state: history hash chain does not verify")

ErrTampered is returned by LoadState when the history chain does not verify.

Functions

func ApprovalKey added in v0.4.0

func ApprovalKey(org *policy.Org, r *policy.Rollout) string

ApprovalKey names the approvals of advancing r from its live step: they never carry over to another step or another change. The change is the release digest, the alias, the backing experiment and the treatment's route for the alias in org, hashed: editing any of them after an approval invalidates it.

func Body

func Body(intro string, d Decision, manual []string, patch string) string

Body renders a PR body: intro, the gate table of d, manual steps and the diff. Halos never merges it.

func Branch

func Branch(r *policy.Rollout, act Action, next int) string

Branch is the PR branch for act.

func ChangeSummary

func ChangeSummary(r *policy.Rollout) string

ChangeSummary describes what r rolls out.

func Days

func Days(d time.Duration) string

Days renders d as "3d4h" style (time.Duration stops at hours).

func Effective

func Effective(org *policy.Org, r *policy.Rollout) *policy.Rollout

Effective is r with every step's guardrails resolved against its backing experiment (policy.RolloutStep.EffectiveGuardrails). Decide on this, never on the raw document. Idempotent; r is not modified.

func Exposure

func Exposure(org *policy.Org, r *policy.Rollout, s policy.RolloutStep) string

Exposure describes what step s does to users.

func GateExperiment

func GateExperiment(exp *policy.Experiment, s policy.RolloutStep) *policy.Experiment

GateExperiment is exp as step s's gates see it: the step's effective guardrails (see Effective) and minSamples, without budgets (a rollout does not expire), so promote.EvaluateAt reports exactly the step's evidence.

func GateSummary

func GateSummary(s policy.RolloutStep) []string

GateSummary lists step s's gates compactly.

func GovernanceBody added in v0.4.0

func GovernanceBody(d Decision) string

GovernanceBody is the PR-body section recording a decision's approvals and ticket ("" when no rule covered it).

func LoadScorecard

func LoadScorecard(dir, rel string) (*eval.Scorecard, error)

LoadScorecard reads a `halo eval run --output json` scorecard at rel inside dir.

func ManualSteps

func ManualSteps(org *policy.Org, r *policy.Rollout, act Action) []string

ManualSteps lists what Plan(act) cannot edit and a human must add to the PR before merging: completing a traffic-axis rollout whose gateway alias or treatment route has multiple targets (failover tiers are a human call).

func NeedsMetrics

func NeedsMetrics(s policy.RolloutStep) bool

NeedsMetrics reports whether step s is judged on experiment evidence: every step that exposes the backing experiment's arms is (its guardrails and primary metric can roll it back), ring-wide steps are not.

func Plan

func Plan(dir string, org *policy.Org, r *policy.Rollout, act Action, next int) (*promote.Change, error)

Plan builds the policy-repo change for act: Advance (to step next), Complete, Rollback or Pause. org is the policy the decision was made on; dir is the policy repo (files are located there, and the edit re-applies to whatever version of them the PR is built from). It writes nothing.

  • canary / holdout: the backing experiment runs as a canary with the step's treatment / control share as variant weights (assignment stays sticky: internal/assign.Pick keeps a growing arm's users).
  • dark-launch: the experiment runs as a shadow at the step's sample rate.
  • progressive / blue-green: the ring's release pointer moves to the change (the experiment is paused so the whole ring gets it).
  • complete: the experiment is concluded; traffic-axis rollouts point the gateway alias at the treatment's route. In simple mode that is halos.yaml models.<alias>, and the gateway.yaml override `halo model switch --canary` generated for the rollout (see policy.LabelGeneratedBy) is deleted.
  • rollback: rollout aborted, experiment paused, every ring moved so far re-pointed at baseline.release.

func StatePath

func StatePath(dir, name string) string

StatePath is <dir>/<name>.json; name is a validated policy name.

func StepRings

func StepRings(org *policy.Org, r *policy.Rollout, s policy.RolloutStep) string

StepRings is the population step s exposes: its ring, or for percent steps without one, the backing experiment's rings.

func Target added in v0.4.0

func Target(org *policy.Org, r *policy.Rollout, from int) policy.ChangeTarget

Target is what advancing r from live step from (-1 = not started) changes: the rings of the step it enters, or the last step's on completion.

func TargetResource added in v0.5.0

func TargetResource(org *policy.Org, r *policy.Rollout, from int) policy.Resource

TargetResource is what advancing r from live step from touches, for scoped authorization (ADR-0037): the rings of Target, every unit's members.

func Title

func Title(r *policy.Rollout, act Action, next int) string

Title is the PR title for act.

Types

type Action

type Action string

Action is what should happen to a rollout now.

const (
	Advance  Action = "advance"  // propose the next step (PR)
	Hold     Action = "hold"     // stay; gates pending
	Rollback Action = "rollback" // a gate breached and onFailure is rollback
	Pause    Action = "pause"    // a gate breached and onFailure is pause
	Complete Action = "complete" // last step passed: propose GA (PR)
)

func NextAction

func NextAction(r *policy.Rollout) (Action, int)

NextAction is what a human advance proposes from r's live step: the next step, or completion after the last.

type Decision

type Decision struct {
	Action  Action       `json:"action"`
	Step    int          `json:"step"` // live step index, -1 = not started
	Next    int          `json:"next"` // step an advance proposes, else -1
	Reasons []string     `json:"reasons"`
	Gates   []GateResult `json:"gates"`
	// AwaitingApproval: everything but the approval gate passed.
	AwaitingApproval bool `json:"awaitingApproval,omitempty"`
	// Source is the evidence source of a breach: promote.SourceGateway,
	// promote.SourceCLI or "scorecard". Only gateway evidence may auto-kill.
	Source string `json:"source,omitempty"`
	// Frozen is the change freeze holding an advance or completion (ADR-0030);
	// rollback and pause are never held.
	Frozen string `json:"frozen,omitempty"`
	// Quorum is the approval quorum the advance needs, when a governance rule
	// covers it.
	Quorum *policy.QuorumResult `json:"quorum,omitempty"`
}

Decision is Evaluate's output.

func Evaluate

func Evaluate(r *policy.Rollout, st State, ev Evidence, now time.Time) Decision

Evaluate decides what to do with r now. It is pure: same inputs, same output. r must be resolved with Effective so the backing experiment's guardrails count.

Order: (1) policy status and controller halts hold; (2) not started -> advance to step 0; (3) a failed guardrail or scorecard -> the step's onFailure, at any time; (4) bake, samples, guardrails, scorecard and approval (or the approval quorum) must all pass, else hold; (5) a change freeze holds; (6) advance, or complete after the last.

type DirStore added in v0.3.0

type DirStore string

DirStore keeps <dir>/<name>.json files.

func (DirStore) Load added in v0.3.0

func (d DirStore) Load(_ context.Context, name string) (State, error)

Load implements StateStore.

func (DirStore) Save added in v0.3.0

func (d DirStore) Save(_ context.Context, s *State) error

Save implements StateStore.

type DocStore added in v0.3.0

type DocStore struct{ DB Docs }

DocStore keeps each rollout's state as one document, verified on load like a file.

func (DocStore) Load added in v0.3.0

func (d DocStore) Load(ctx context.Context, name string) (State, error)

Load implements StateStore.

func (DocStore) Save added in v0.3.0

func (d DocStore) Save(ctx context.Context, s *State) error

Save implements StateStore.

type Docs added in v0.3.0

type Docs interface {
	GetDoc(ctx context.Context, kind, name string) ([]byte, bool, error)
	PutDoc(ctx context.Context, kind, name string, body []byte) error
}

Docs is a document table (pgstore.DB).

type Entry

type Entry struct {
	Seq    int       `json:"seq"`
	At     time.Time `json:"at"`
	Event  string    `json:"event"`
	Index  int       `json:"index"` // step index for enter
	Step   string    `json:"step,omitempty"`
	Detail string    `json:"detail,omitempty"`
	Key    string    `json:"key,omitempty"`    // action entries: what was done
	Source string    `json:"source,omitempty"` // halt entries: evidence source of the breach
	Reason string    `json:"reason,omitempty"` // halt entries: why
	Prev   string    `json:"prev"`             // previous entry's Hash ("" for the first)
	Hash   string    `json:"hash"`             // sha256 over this entry with Hash empty
}

Entry is one hash-chained history record.

type Evidence

type Evidence struct {
	// Report is the backing experiment evaluated with the step's guardrails
	// and minSamples (see GateExperiment); nil = no metric evidence.
	Report *promote.Report `json:"report,omitempty"`
	// Scorecard is the step's eval scorecard; nil with ScorecardErr = unreadable.
	Scorecard    *eval.Scorecard `json:"-"`
	ScorecardErr string          `json:"scorecardError,omitempty"`
	// Approved: a human is advancing this step by hand (`halo rollout advance`).
	// It does not satisfy an approval quorum (Quorum).
	Approved bool `json:"approved,omitempty"`
	// Freeze is the change freeze covering the advance; nil = not frozen.
	Freeze *policy.FreezeStatus `json:"freeze,omitempty"`
	// Quorum is the approval quorum covering the advance (ADR-0030); nil = no
	// governance rule covers it. It replaces the approval gate.
	Quorum *policy.QuorumResult `json:"quorum,omitempty"`
}

Evidence is what the live step's gates are judged on.

func Gather

func Gather(ctx context.Context, dir string, org *policy.Org, r *policy.Rollout, st State, metrics promote.MetricSource, now time.Time) (Evidence, error)

Gather collects the live step's evidence: the backing experiment from metrics (nil = none; only queried when the step has metric gates) and the scorecard file under dir. A metrics error is returned; a missing or bad scorecard is reported in the evidence (the gate stays pending).

type GateResult

type GateResult struct {
	Gate      string `json:"gate"` // bake | samples | guardrail:<metric> | scorecard | approval
	Status    string `json:"status"`
	Value     string `json:"value,omitempty"`
	Threshold string `json:"threshold,omitempty"`
	Detail    string `json:"detail,omitempty"`
}

GateResult is one gate's outcome, with value vs threshold for humans.

type Governance added in v0.4.0

type Governance struct {
	Incident  *policy.ActiveFreeze `json:"incident,omitempty"` // nil = none
	Approvals []policy.Approval    `json:"approvals,omitempty"`
	// Requester proposes the advance ("" = the controller): their own
	// approval never counts.
	Requester string `json:"-"`
}

Governance is the change-governance input to one decision (ADR-0030): halo-server's incident freeze and the approvals recorded for advancing the live step. The policy's freeze windows and approval rules come from org.

func (Governance) Apply added in v0.4.0

func (g Governance) Apply(org *policy.Org, r *policy.Rollout, st State, ev *Evidence, now time.Time)

Apply sets ev.Freeze and ev.Quorum for advancing r from st.Step at now. A recorded approval satisfies a plain approval gate when no rule covers it.

type SimEvent

type SimEvent struct {
	At       string       `json:"at"` // elapsed since start, e.g. "T+2d4h"
	Elapsed  int64        `json:"elapsedSeconds"`
	Step     string       `json:"step"`
	Action   Action       `json:"action"`
	Reasons  []string     `json:"reasons"`
	Gates    []GateResult `json:"gates"`
	Approved bool         `json:"approved,omitempty"`
}

SimEvent is one notable moment of a simulation.

type SimOptions

type SimOptions struct {
	Scenario string        // default healthy
	Tick     time.Duration // default 1h
	Horizon  time.Duration // default 120d
	// UnitsPerHour is how many new users the experiment sees per hour at 100%; default 1000.
	UnitsPerHour int
	// BreachStep is the step index a regression starts at (default: the second
	// step with metric gates, else the first).
	BreachStep *int
	// ApproveAfter is how long a simulated human takes to approve; default 4h.
	ApproveAfter time.Duration
	Seed         uint64
	// Recorded, when set, replaces synthetic evidence: one entry per tick (the
	// last repeats). Scorecards still come from the policy dir.
	Recorded []Evidence
}

SimOptions configures Simulate. Zero values take the defaults noted.

type SimResult

type SimResult struct {
	Rollout  string     `json:"rollout"`
	Scenario string     `json:"scenario"`
	Outcome  Action     `json:"outcome"` // complete | rollback | pause | hold (horizon reached)
	Duration string     `json:"duration"`
	Events   []SimEvent `json:"events"`
}

SimResult is a whole simulated run.

func Simulate

func Simulate(ctx context.Context, dir string, org *policy.Org, r *policy.Rollout, o SimOptions) (SimResult, error)

Simulate runs the real state machine (Evaluate, with guardrails computed by promote.EvaluateAt over seeded synthetic samples, or recorded evidence) from "not started" until it completes, rolls back, pauses or reaches the horizon, assuming every proposed PR merges immediately. Deterministic for a given seed. It writes nothing.

type State

type State struct {
	Rollout   string    `json:"rollout"`
	Step      int       `json:"step"` // -1 = not started
	StepName  string    `json:"stepName,omitempty"`
	EnteredAt time.Time `json:"enteredAt"`
	// Halted is "rollback" or "pause" once the controller acted on a breach;
	// it holds the rollout until the policy leaves "active".
	Halted     string `json:"halted,omitempty"`
	HaltSource string `json:"haltSource,omitempty"`
	HaltReason string `json:"haltReason,omitempty"`
	// Done maps idempotency keys (see Key) to details such as PR URLs.
	Done    map[string]string `json:"done,omitempty"`
	Last    *Decision         `json:"last,omitempty"` // last decision: status views only
	LastAt  time.Time         `json:"lastAt"`
	History []Entry           `json:"history"`
}

State is a rollout's controller state. Everything the controller acts on (Step, StepName, EnteredAt, Halted, HaltSource, HaltReason, Done) is derived by replaying History; the header copies are only checked against it. Last is display-only and never acted on.

The hash chain is unkeyed: it detects partial edits (a changed, dropped or reordered entry, or a header that disagrees with the history), not an attacker who can rewrite the whole file and recompute every hash. Protect the data dir like the kill switch and audit log it sits next to.

func LoadState

func LoadState(dir, name string) (State, error)

LoadState reads and verifies <dir>/<name>.json; a missing file is a new state.

func NewState

func NewState(name string) State

NewState is the state of a rollout that has not started.

func ParseState added in v0.3.0

func ParseState(where, name string, b []byte) (State, error)

ParseState decodes and verifies a stored state for rollout name; where names its origin in errors. The history hash chain must verify and the header must agree with it.

func PolicyState

func PolicyState(r *policy.Rollout) State

PolicyState is the state implied by the policy alone (no controller state file): the live step, with an unknown entry time.

func (*State) Did

func (s *State) Did(action string) (string, bool)

Did reports whether action was done at the current step, and its detail.

func (*State) Halt

func (s *State) Halt(action, source, reason string, at time.Time)

Halt records that the controller rolled back or paused (action) at the current step, on evidence from source.

func (*State) Key

func (s *State) Key(action string) string

Key is the idempotency key of an action taken at the current step.

func (*State) Mark

func (s *State) Mark(action, detail string, at time.Time)

Mark records an action as done (once per step) with its detail.

func (*State) Record

func (s *State) Record(event string, index int, step, detail string, at time.Time)

Record appends a chained entry and applies it.

func (*State) Save

func (s *State) Save(dir string) error

Save writes the state atomically (0600).

func (*State) Sync

func (s *State) Sync(r *policy.Rollout, now time.Time) bool

Sync folds the merged policy into the state: a new live step is entered (now: the controller first sees the merge), and a halt is acknowledged once the policy is no longer active (a human merged the pause or rollback). It reports whether anything changed.

type StateStore added in v0.3.0

type StateStore interface {
	Load(ctx context.Context, name string) (State, error)
	Save(ctx context.Context, s *State) error
}

StateStore persists rollout state: DirStore (files under halo-server's data dir, single replica) or DocStore (a table shared by every replica).

type Timeline

type Timeline struct {
	Rollout    string         `json:"rollout"`
	Axis       string         `json:"axis"`
	Status     string         `json:"status"`
	Experiment string         `json:"experiment"`
	Change     string         `json:"change"`
	Steps      []TimelineStep `json:"steps"`
	// MinDuration is the sum of all bakes: the soonest completion.
	MinDuration string `json:"minDuration"`
}

Timeline is a rollout's plan for humans.

func BuildTimeline

func BuildTimeline(org *policy.Org, r *policy.Rollout) Timeline

BuildTimeline lays out r's steps with their earliest start.

type TimelineStep

type TimelineStep struct {
	Index      int      `json:"index"`
	Name       string   `json:"name"`
	Strategy   string   `json:"strategy"`
	Ring       string   `json:"ring"`
	Percent    float64  `json:"percent"`
	Exposure   string   `json:"exposure"`
	Bake       string   `json:"bake,omitempty"`
	MinSamples int      `json:"minSamples,omitempty"`
	Gates      []string `json:"gates"`
	OnFailure  string   `json:"onFailure"`
	// EarliestStart is the sum of earlier bakes: the soonest the step can be
	// live if every gate passes on time and every PR merges at once.
	EarliestStart string `json:"earliestStart"`
	Live          bool   `json:"live,omitempty"`
}

TimelineStep is one row of a rollout plan.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL