Documentation
¶
Overview ¶
Package rollout drives a policy.Rollout through its steps.
Evaluate is a pure, deterministic state machine: given the rollout, its persisted State, the Evidence gathered for the live step and the time, it decides advance | hold | rollback | pause | complete. It never acts. The controller acts on decisions asymmetrically, like internal/promote: rollback and pause happen automatically; advance and complete only ever open a PR (Plan) that a human merges. Nothing here merges.
Index ¶
- Constants
- Variables
- func ApprovalKey(org *policy.Org, r *policy.Rollout) string
- func Body(intro string, d Decision, manual []string, patch string) string
- func Branch(r *policy.Rollout, act Action, next int) string
- func ChangeSummary(r *policy.Rollout) string
- func Days(d time.Duration) string
- func Effective(org *policy.Org, r *policy.Rollout) *policy.Rollout
- func Exposure(org *policy.Org, r *policy.Rollout, s policy.RolloutStep) string
- func GateExperiment(exp *policy.Experiment, s policy.RolloutStep) *policy.Experiment
- func GateSummary(s policy.RolloutStep) []string
- func GovernanceBody(d Decision) string
- func LoadScorecard(dir, rel string) (*eval.Scorecard, error)
- func ManualSteps(org *policy.Org, r *policy.Rollout, act Action) []string
- func NeedsMetrics(s policy.RolloutStep) bool
- func Plan(dir string, org *policy.Org, r *policy.Rollout, act Action, next int) (*promote.Change, error)
- func StatePath(dir, name string) string
- func StepRings(org *policy.Org, r *policy.Rollout, s policy.RolloutStep) string
- func Target(org *policy.Org, r *policy.Rollout, from int) policy.ChangeTarget
- func TargetResource(org *policy.Org, r *policy.Rollout, from int) policy.Resource
- func Title(r *policy.Rollout, act Action, next int) string
- type Action
- type Decision
- type DirStore
- type DocStore
- type Docs
- type Entry
- type Evidence
- type GateResult
- type Governance
- type SimEvent
- type SimOptions
- type SimResult
- type State
- func (s *State) Did(action string) (string, bool)
- func (s *State) Halt(action, source, reason string, at time.Time)
- func (s *State) Key(action string) string
- func (s *State) Mark(action, detail string, at time.Time)
- func (s *State) Record(event string, index int, step, detail string, at time.Time)
- func (s *State) Save(dir string) error
- func (s *State) Sync(r *policy.Rollout, now time.Time) bool
- type StateStore
- type Timeline
- type TimelineStep
Constants ¶
const ( GatePass = "pass" GateFail = "fail" GatePending = "pending" )
Gate statuses.
const ( ScenarioHealthy = "healthy" // treatment matches control on every metric ScenarioRegression = "regression" // the first guardrail regresses from BreachStep on )
Simulation scenarios.
const ( EventEnter = "enter" // a merged step was observed live (Index, Step) EventHalt = "halt" // the controller rolled back or paused (Detail = action, Source, Reason) EventResume = "resume" // the policy left "active"; the halt is acknowledged EventAction = "action" // Key was done at step Index (Detail: e.g. a PR URL) )
History events.
const DocKind = "rollout-state"
DocKind is the document kind DocStore keeps rollout state under.
Variables ¶
var ErrTampered = errors.New("rollout state: history hash chain does not verify")
ErrTampered is returned by LoadState when the history chain does not verify.
Functions ¶
func ApprovalKey ¶ added in v0.4.0
ApprovalKey names the approvals of advancing r from its live step: they never carry over to another step or another change. The change is the release digest, the alias, the backing experiment and the treatment's route for the alias in org, hashed: editing any of them after an approval invalidates it.
func Body ¶
Body renders a PR body: intro, the gate table of d, manual steps and the diff. Halos never merges it.
func ChangeSummary ¶
ChangeSummary describes what r rolls out.
func Effective ¶
Effective is r with every step's guardrails resolved against its backing experiment (policy.RolloutStep.EffectiveGuardrails). Decide on this, never on the raw document. Idempotent; r is not modified.
func GateExperiment ¶
func GateExperiment(exp *policy.Experiment, s policy.RolloutStep) *policy.Experiment
GateExperiment is exp as step s's gates see it: the step's effective guardrails (see Effective) and minSamples, without budgets (a rollout does not expire), so promote.EvaluateAt reports exactly the step's evidence.
func GateSummary ¶
func GateSummary(s policy.RolloutStep) []string
GateSummary lists step s's gates compactly.
func GovernanceBody ¶ added in v0.4.0
GovernanceBody is the PR-body section recording a decision's approvals and ticket ("" when no rule covered it).
func LoadScorecard ¶
LoadScorecard reads a `halo eval run --output json` scorecard at rel inside dir.
func ManualSteps ¶
ManualSteps lists what Plan(act) cannot edit and a human must add to the PR before merging: completing a traffic-axis rollout whose gateway alias or treatment route has multiple targets (failover tiers are a human call).
func NeedsMetrics ¶
func NeedsMetrics(s policy.RolloutStep) bool
NeedsMetrics reports whether step s is judged on experiment evidence: every step that exposes the backing experiment's arms is (its guardrails and primary metric can roll it back), ring-wide steps are not.
func Plan ¶
func Plan(dir string, org *policy.Org, r *policy.Rollout, act Action, next int) (*promote.Change, error)
Plan builds the policy-repo change for act: Advance (to step next), Complete, Rollback or Pause. org is the policy the decision was made on; dir is the policy repo (files are located there, and the edit re-applies to whatever version of them the PR is built from). It writes nothing.
- canary / holdout: the backing experiment runs as a canary with the step's treatment / control share as variant weights (assignment stays sticky: internal/assign.Pick keeps a growing arm's users).
- dark-launch: the experiment runs as a shadow at the step's sample rate.
- progressive / blue-green: the ring's release pointer moves to the change (the experiment is paused so the whole ring gets it).
- complete: the experiment is concluded; traffic-axis rollouts point the gateway alias at the treatment's route. In simple mode that is halos.yaml models.<alias>, and the gateway.yaml override `halo model switch --canary` generated for the rollout (see policy.LabelGeneratedBy) is deleted.
- rollback: rollout aborted, experiment paused, every ring moved so far re-pointed at baseline.release.
func StepRings ¶
StepRings is the population step s exposes: its ring, or for percent steps without one, the backing experiment's rings.
func Target ¶ added in v0.4.0
Target is what advancing r from live step from (-1 = not started) changes: the rings of the step it enters, or the last step's on completion.
func TargetResource ¶ added in v0.5.0
TargetResource is what advancing r from live step from touches, for scoped authorization (ADR-0037): the rings of Target, every unit's members.
Types ¶
type Action ¶
type Action string
Action is what should happen to a rollout now.
const ( Advance Action = "advance" // propose the next step (PR) Hold Action = "hold" // stay; gates pending Rollback Action = "rollback" // a gate breached and onFailure is rollback Pause Action = "pause" // a gate breached and onFailure is pause Complete Action = "complete" // last step passed: propose GA (PR) )
type Decision ¶
type Decision struct {
Action Action `json:"action"`
Step int `json:"step"` // live step index, -1 = not started
Next int `json:"next"` // step an advance proposes, else -1
Reasons []string `json:"reasons"`
Gates []GateResult `json:"gates"`
// AwaitingApproval: everything but the approval gate passed.
AwaitingApproval bool `json:"awaitingApproval,omitempty"`
// Source is the evidence source of a breach: promote.SourceGateway,
// promote.SourceCLI or "scorecard". Only gateway evidence may auto-kill.
Source string `json:"source,omitempty"`
// Frozen is the change freeze holding an advance or completion (ADR-0030);
// rollback and pause are never held.
Frozen string `json:"frozen,omitempty"`
// Quorum is the approval quorum the advance needs, when a governance rule
// covers it.
Quorum *policy.QuorumResult `json:"quorum,omitempty"`
}
Decision is Evaluate's output.
func Evaluate ¶
Evaluate decides what to do with r now. It is pure: same inputs, same output. r must be resolved with Effective so the backing experiment's guardrails count.
Order: (1) policy status and controller halts hold; (2) not started -> advance to step 0; (3) a failed guardrail or scorecard -> the step's onFailure, at any time; (4) bake, samples, guardrails, scorecard and approval (or the approval quorum) must all pass, else hold; (5) a change freeze holds; (6) advance, or complete after the last.
type DirStore ¶ added in v0.3.0
type DirStore string
DirStore keeps <dir>/<name>.json files.
type DocStore ¶ added in v0.3.0
type DocStore struct{ DB Docs }
DocStore keeps each rollout's state as one document, verified on load like a file.
type Docs ¶ added in v0.3.0
type Docs interface {
GetDoc(ctx context.Context, kind, name string) ([]byte, bool, error)
PutDoc(ctx context.Context, kind, name string, body []byte) error
}
Docs is a document table (pgstore.DB).
type Entry ¶
type Entry struct {
Seq int `json:"seq"`
At time.Time `json:"at"`
Event string `json:"event"`
Index int `json:"index"` // step index for enter
Step string `json:"step,omitempty"`
Detail string `json:"detail,omitempty"`
Key string `json:"key,omitempty"` // action entries: what was done
Source string `json:"source,omitempty"` // halt entries: evidence source of the breach
Reason string `json:"reason,omitempty"` // halt entries: why
Prev string `json:"prev"` // previous entry's Hash ("" for the first)
Hash string `json:"hash"` // sha256 over this entry with Hash empty
}
Entry is one hash-chained history record.
type Evidence ¶
type Evidence struct {
// Report is the backing experiment evaluated with the step's guardrails
// and minSamples (see GateExperiment); nil = no metric evidence.
Report *promote.Report `json:"report,omitempty"`
// Scorecard is the step's eval scorecard; nil with ScorecardErr = unreadable.
Scorecard *eval.Scorecard `json:"-"`
ScorecardErr string `json:"scorecardError,omitempty"`
// Approved: a human is advancing this step by hand (`halo rollout advance`).
// It does not satisfy an approval quorum (Quorum).
Approved bool `json:"approved,omitempty"`
// Freeze is the change freeze covering the advance; nil = not frozen.
Freeze *policy.FreezeStatus `json:"freeze,omitempty"`
// Quorum is the approval quorum covering the advance (ADR-0030); nil = no
// governance rule covers it. It replaces the approval gate.
Quorum *policy.QuorumResult `json:"quorum,omitempty"`
}
Evidence is what the live step's gates are judged on.
func Gather ¶
func Gather(ctx context.Context, dir string, org *policy.Org, r *policy.Rollout, st State, metrics promote.MetricSource, now time.Time) (Evidence, error)
Gather collects the live step's evidence: the backing experiment from metrics (nil = none; only queried when the step has metric gates) and the scorecard file under dir. A metrics error is returned; a missing or bad scorecard is reported in the evidence (the gate stays pending).
type GateResult ¶
type GateResult struct {
Gate string `json:"gate"` // bake | samples | guardrail:<metric> | scorecard | approval
Status string `json:"status"`
Value string `json:"value,omitempty"`
Threshold string `json:"threshold,omitempty"`
Detail string `json:"detail,omitempty"`
}
GateResult is one gate's outcome, with value vs threshold for humans.
type Governance ¶ added in v0.4.0
type Governance struct {
Incident *policy.ActiveFreeze `json:"incident,omitempty"` // nil = none
Approvals []policy.Approval `json:"approvals,omitempty"`
// Requester proposes the advance ("" = the controller): their own
// approval never counts.
Requester string `json:"-"`
}
Governance is the change-governance input to one decision (ADR-0030): halo-server's incident freeze and the approvals recorded for advancing the live step. The policy's freeze windows and approval rules come from org.
type SimEvent ¶
type SimEvent struct {
At string `json:"at"` // elapsed since start, e.g. "T+2d4h"
Elapsed int64 `json:"elapsedSeconds"`
Step string `json:"step"`
Action Action `json:"action"`
Reasons []string `json:"reasons"`
Gates []GateResult `json:"gates"`
Approved bool `json:"approved,omitempty"`
}
SimEvent is one notable moment of a simulation.
type SimOptions ¶
type SimOptions struct {
Scenario string // default healthy
Tick time.Duration // default 1h
Horizon time.Duration // default 120d
// UnitsPerHour is how many new users the experiment sees per hour at 100%; default 1000.
UnitsPerHour int
// BreachStep is the step index a regression starts at (default: the second
// step with metric gates, else the first).
BreachStep *int
// ApproveAfter is how long a simulated human takes to approve; default 4h.
ApproveAfter time.Duration
Seed uint64
// Recorded, when set, replaces synthetic evidence: one entry per tick (the
// last repeats). Scorecards still come from the policy dir.
Recorded []Evidence
}
SimOptions configures Simulate. Zero values take the defaults noted.
type SimResult ¶
type SimResult struct {
Rollout string `json:"rollout"`
Scenario string `json:"scenario"`
Outcome Action `json:"outcome"` // complete | rollback | pause | hold (horizon reached)
Duration string `json:"duration"`
Events []SimEvent `json:"events"`
}
SimResult is a whole simulated run.
func Simulate ¶
func Simulate(ctx context.Context, dir string, org *policy.Org, r *policy.Rollout, o SimOptions) (SimResult, error)
Simulate runs the real state machine (Evaluate, with guardrails computed by promote.EvaluateAt over seeded synthetic samples, or recorded evidence) from "not started" until it completes, rolls back, pauses or reaches the horizon, assuming every proposed PR merges immediately. Deterministic for a given seed. It writes nothing.
type State ¶
type State struct {
Rollout string `json:"rollout"`
Step int `json:"step"` // -1 = not started
StepName string `json:"stepName,omitempty"`
EnteredAt time.Time `json:"enteredAt"`
// Halted is "rollback" or "pause" once the controller acted on a breach;
// it holds the rollout until the policy leaves "active".
Halted string `json:"halted,omitempty"`
HaltSource string `json:"haltSource,omitempty"`
HaltReason string `json:"haltReason,omitempty"`
// Done maps idempotency keys (see Key) to details such as PR URLs.
Done map[string]string `json:"done,omitempty"`
Last *Decision `json:"last,omitempty"` // last decision: status views only
LastAt time.Time `json:"lastAt"`
History []Entry `json:"history"`
}
State is a rollout's controller state. Everything the controller acts on (Step, StepName, EnteredAt, Halted, HaltSource, HaltReason, Done) is derived by replaying History; the header copies are only checked against it. Last is display-only and never acted on.
The hash chain is unkeyed: it detects partial edits (a changed, dropped or reordered entry, or a header that disagrees with the history), not an attacker who can rewrite the whole file and recompute every hash. Protect the data dir like the kill switch and audit log it sits next to.
func ParseState ¶ added in v0.3.0
ParseState decodes and verifies a stored state for rollout name; where names its origin in errors. The history hash chain must verify and the header must agree with it.
func PolicyState ¶
PolicyState is the state implied by the policy alone (no controller state file): the live step, with an unknown entry time.
func (*State) Halt ¶
Halt records that the controller rolled back or paused (action) at the current step, on evidence from source.
type StateStore ¶ added in v0.3.0
type StateStore interface {
Load(ctx context.Context, name string) (State, error)
Save(ctx context.Context, s *State) error
}
StateStore persists rollout state: DirStore (files under halo-server's data dir, single replica) or DocStore (a table shared by every replica).
type Timeline ¶
type Timeline struct {
Rollout string `json:"rollout"`
Axis string `json:"axis"`
Status string `json:"status"`
Experiment string `json:"experiment"`
Change string `json:"change"`
Steps []TimelineStep `json:"steps"`
// MinDuration is the sum of all bakes: the soonest completion.
MinDuration string `json:"minDuration"`
}
Timeline is a rollout's plan for humans.
type TimelineStep ¶
type TimelineStep struct {
Index int `json:"index"`
Name string `json:"name"`
Strategy string `json:"strategy"`
Ring string `json:"ring"`
Percent float64 `json:"percent"`
Exposure string `json:"exposure"`
Bake string `json:"bake,omitempty"`
MinSamples int `json:"minSamples,omitempty"`
Gates []string `json:"gates"`
OnFailure string `json:"onFailure"`
// EarliestStart is the sum of earlier bakes: the soonest the step can be
// live if every gate passes on time and every PR merges at once.
EarliestStart string `json:"earliestStart"`
Live bool `json:"live,omitempty"`
}
TimelineStep is one row of a rollout plan.