Documentation
¶
Overview ¶
Package rollout is billet's durable fleet decision: one immutable target, and where every controller and node has got to on the way to it.
WHY A STATE MACHINE AND NOT A LOOP. Updating a fleet is a sequence of irreversible acts on machines that are running somebody's builds — draining a host, replacing a binary, migrating a ledger — and each of them has to be resumable by a process that did not start it. A control plane restart, a leadership handoff or an operator's second `billet rollout status` must all find the same answer, so where each component has got to is a durable phase rather than a position in a function.
Index ¶
- Constants
- Variables
- func CanTransition(from, to Phase) bool
- func KnownPhase(p Phase) bool
- func Transition(from, to Phase) error
- type AdvanceRequest
- type Coordinator
- type CoordinatorOption
- type Dispatcher
- type Fleet
- type Host
- type Node
- type Option
- type Phase
- type Policy
- type Resolver
- type Rollout
- type StartPolicy
- type StartRequest
- type Starter
- type StarterOption
- type Store
- func (s *Store) Advance(ctx context.Context, req AdvanceRequest) error
- func (s *Store) Finish(ctx context.Context, rolloutID, outcome, reason string) error
- func (s *Store) History(ctx context.Context, limit int) ([]Rollout, error)
- func (s *Store) NewestForTarget(ctx context.Context, digest string) (*Rollout, bool, error)
- func (s *Store) Nodes(ctx context.Context, rolloutID string) ([]Node, error)
- func (s *Store) Open(ctx context.Context) (*Rollout, error)
- func (s *Store) Start(ctx context.Context, req StartRequest) (*Rollout, error)
- type Target
Constants ¶
const ( StateOpen = "open" StateCompleted = "completed" StateAborted = "aborted" )
State is whether a rollout is still being worked on.
const DefaultStartInterval = time.Hour
DefaultStartInterval is how often the starter looks at the channel.
HOURLY, because a channel advances when a release is cut and nothing about it becomes true faster for being asked more often. The statement it reads is served from a branch rather than the API, so the cost of asking is a small fetch and not a share of anybody's rate limit.
Variables ¶
var ErrAbortedTarget = errors.New("rollout: the newest rollout to this target was aborted")
ErrAbortedTarget means the newest rollout to the requested bytes was aborted by an operator, and the request said not to overrule that.
var ErrBadTransition = fmt.Errorf("rollout: that phase change is not one the state machine allows")
ErrBadTransition means a phase change the state machine does not allow.
var ErrNoRollout = errors.New("rollout: no rollout is running")
ErrNoRollout means nothing is running.
var ErrOpen = errors.New("rollout: a rollout is already running")
ErrOpen means a rollout is already running.
A SENTINEL, because the caller's answer is not an error message. `billet rollout start` run twice must report the rollout that is running rather than creating a second one — which is the issue's requirement that a repeated instruction does not duplicate or retarget a decision.
var ErrOutstanding = errors.New("rollout: this rollout has not converged")
ErrOutstanding means a rollout cannot be completed yet because something is still unresolved.
A SENTINEL, BECAUSE IT IS THE ORDINARY ANSWER AND A BROKEN LEDGER IS NOT. The coordinator attempts completion on every pass, so "a host has not converged yet" is what it hears on every tick but the last and must not be reported as a failed pass. Without a way to tell that from a storage error, the coordinator swallowed both — and a Finish failing for a real reason was retried forever, reported as a successful pass, and visible nowhere.
Functions ¶
func CanTransition ¶
CanTransition reports whether one phase may follow another.
A REPEAT IS ALLOWED AND CHANGES NOTHING. Instructions are delivered over a network and retried, so a component reporting the phase it is already in is the ordinary case rather than an error — and refusing it would turn a redelivery into a blocked host.
func KnownPhase ¶
KnownPhase reports whether a string is a phase this build understands.
A ROW THIS BINARY CANNOT CLASSIFY IS NOT ONE IT MAY ACT ON. A newer binary can write a phase an older one has never heard of, and treating that as pending would restart an install on a component already midway through one.
func Transition ¶
Transition validates one phase change and explains a refusal.
Types ¶
type AdvanceRequest ¶
type AdvanceRequest struct {
RolloutID string
// Node is the host, or empty for the controller itself.
Node string
To Phase
// Blocker explains a phase that needs explaining. Required for blocked,
// because a cordoned host with no reason recorded is one nobody can clear.
Blocker string
// RollbackResult records what a rollback proved, or could not.
RollbackResult string
// ExemptReason is the operator's decision, required for exempt and
// decommissioned.
ExemptReason string
// PriorRelease is what the component was running before this rollout touched
// it, recorded once so a rollback has somewhere to go.
PriorRelease string
// Backoff, when positive, is how long before this component may be tried
// again. It also increments the attempt count.
Backoff time.Duration
// DispatchEpoch, when positive, records the host's registration epoch at the
// moment it was told to upgrade. See Node.DispatchEpoch.
DispatchEpoch int64
// ConvergedDigest, when non-empty, records the release manifest that proved a
// host converged. See Node.ConvergedDigest.
ConvergedDigest string
}
AdvanceRequest is one component moving to a new phase.
type Coordinator ¶
type Coordinator struct {
// contains filtered or unexported fields
}
Coordinator drives one durable rollout to convergence.
WITHOUT IT THE ROLLOUT IS A RECORD NOBODY ACTS ON. `billet rollout start` says the control plane picks the decision up and converges the fleet; a review pointed out that nothing did, so every started rollout stayed open forever and blocked the next one. This is what makes that sentence true.
IT DOES NOT REPLACE THE CONTROL PLANE. A process cannot install its own successor, and this one runs unprivileged — that is `billet host-upgrade --from-rollout`, run on the controller's host as root by the packaged timer or an operator. What the coordinator does about the controller is OBSERVE: when the running binary is the target, it records that, and only then does it begin on the nodes. That is what makes the rollout server-first, and it is an observation rather than an action precisely because the acting half cannot live in the process being replaced.
func NewCoordinator ¶
func NewCoordinator(store *Store, fleet Fleet, dispatch Dispatcher, ourVersion string, minWire int, opts ...CoordinatorOption, ) *Coordinator
NewCoordinator builds the driver for a rollout.
ourVersion is what this binary reports itself as, and minWire is the node-wire version below which a host cannot be told to upgrade. Both are passed in rather than read here, for the reason releasesource.Current is: a package that read them itself could only ever be tested against the build running the test.
func (*Coordinator) Run ¶
func (c *Coordinator) Run(ctx context.Context, every time.Duration)
Run drives the open rollout until the context ends.
func (*Coordinator) Tick ¶
func (c *Coordinator) Tick(ctx context.Context) error
Tick advances the open rollout by as much as it can prove.
EVERY TRANSITION IS DRIVEN BY AN OBSERVATION, never by elapsed time. A host is converged because it registered reporting the target release; it is rolled back because it came back reporting the one before. Nothing here times a host out — a drain has no bound, and a coordinator that gave up on a slow one would be the timer this whole area refuses.
type CoordinatorOption ¶
type CoordinatorOption func(*Coordinator)
CoordinatorOption configures a Coordinator.
func WithCoordinatorClock ¶
func WithCoordinatorClock(now func() time.Time) CoordinatorOption
WithCoordinatorClock replaces the clock, so a test can drive backoff.
func WithCoordinatorLogger ¶
func WithCoordinatorLogger(log *slog.Logger) CoordinatorOption
WithCoordinatorLogger sets where the coordinator reports.
type Dispatcher ¶
type Dispatcher interface {
Upgrade(ctx context.Context, node, version, manifestSHA256, rolloutID string,
generation int64) error
}
Dispatcher tells one node to replace its own billet.
IT RETURNS WHEN THE UPDATER HAS STARTED, not when the upgrade is done. The node execs a detached transaction that outlives the command, so there is nothing to wait for — and waiting would hold the node's single command slot for the length of a drain, which has no bound.
type Host ¶
type Host struct {
Name string
// Release is what the host said it was running at its last registration.
//
// EMPTY IS NOT "OLD". A build below VersionNodeRelease has no release to give,
// which is the entire installed fleet on the day this ships. A host that says
// nothing is one the coordinator cannot prove converged, so it stays pending
// and blocks completion rather than being guessed about in either direction.
Release string
// Digest is the signed release manifest that produced this host's binary, or
// empty when nothing on that machine could say.
//
// THE ONLY THING THAT DISTINGUISHES BYTES FROM A NAME. Release is the version
// the host's binary was BUILT as, which two builds can share and which a moved
// tag makes identical — so a rollout comparing versions alone converges on
// evidence weaker than the decision it is converging. Empty is the ordinary
// case and is read as "cannot tell"; a value that DISAGREES is a fact that
// could not previously exist.
Digest string
// Wire is the protocol version its registration settled on.
Wire int
// Live is whether this deployment is currently in contact with it.
Live bool
// Epoch is the fencing token this host's CURRENT registration holds.
//
// THE ONLY THING THAT PROVABLY POSTDATES AN INSTRUCTION. Release and Live are
// both true of a host that never left and of one that went away and came back
// on the same binary, so without this the coordinator cannot tell a host still
// draining from one that rolled itself back — and would leave the second in
// the cohort forever.
Epoch int64
}
Host is what the coordinator knows about one machine in the fleet.
READ FROM THE LEDGER, NOT FROM THE PLANE'S MEMORY. The release a host reports and the wire it negotiated are recorded at registration and survive a control plane restart; the plane's in-memory view does not, and a successor that had to wait for every host to reconnect before it could resume a rollout would stall for as long as the quietest node's poll interval.
type Node ¶
type Node struct {
Node string
Phase Phase
Attempts int
NextAttemptAt string
Blocker string
PriorRelease string
RollbackResult string
ExemptReason string
UpdatedAt string
// ConvergedDigest is the release manifest that proved this host converged, or
// empty for one that converged on its version alone.
//
// EMPTY IS A CONVERGED HOST NOTHING PROVED, not a host that failed. Read
// beside the phase: `committed` with no digest is a host that reached the
// target version and could not say which bytes it installed, which is every
// host in the field before one billet-driven upgrade has run.
ConvergedDigest string
// DispatchEpoch is the host's registration epoch when it was told to upgrade.
//
// A CAUSAL FENCE, and the only one available. After the instruction, a host
// that has not started yet and one that upgraded, failed and rolled itself
// back are identical in every other field — both live, both reporting the
// previous release. A registration bumps the epoch and nothing else does, so
// a HIGHER one provably postdates the instruction. Zero means nothing was
// recorded, and nothing is concluded from it.
DispatchEpoch int64
}
Node is where one host has got to.
type Phase ¶
type Phase string
Phase is where one component has got to.
ONE VOCABULARY FOR CONTROLLERS AND NODES, because the sequence is the same thing happening to a different kind of machine and two enums would be two places to get the ordering right. What differs is who carries out each step, not what the steps are.
const ( // PhasePending is a component the rollout has not started on. // // WHERE A DISCONNECTED NODE STAYS. It is not gone: its compute may be running // and it will come back speaking whatever it spoke before, so it holds the // rollout open rather than being written off. Only an operator proving compute // absence moves it to decommissioned. PhasePending Phase = "pending" // PhaseDraining is a component that has stopped taking new work and is // waiting for what it already has. // // UNBOUNDED, AND THAT IS THE WHOLE POINT. A job may run for days; elapsed time // is not evidence that one stopped making progress, and a rollout that // installed anyway would fail a build GitHub does not requeue. Nothing in this // package times this phase out. PhaseDraining Phase = "draining" // PhaseReadyToInstall is a component with no active workload obligations. // // A PHASE RATHER THAN AN INSTANT, because reaching it is the proof that // authorises the next step and that proof has to be recorded. An installer // that checked and acted would be checking on one side of a restart and acting // on the other. PhaseReadyToInstall Phase = "ready_to_install" // PhaseInstalling is a component whose binary is being replaced. PhaseInstalling Phase = "installing" // PhaseVerifying is a replaced component proving it works. // // SEPARATE FROM INSTALLING because they fail differently. An install that // fails has changed less than a verification that fails, and the recovery // journal has to say which side of the switch the failure landed on. PhaseVerifying Phase = "verifying" // PhaseCommitted is a component running the target and healthy. Terminal. PhaseCommitted Phase = "committed" // PhaseRollingBack is a failed component restoring its previous release. PhaseRollingBack Phase = "rolling_back" // PhaseRolledBack is a component proved healthy on its previous release. // // NOT TERMINAL. A successfully rolled-back node may return to service when its // old release remains compatible and policy permits, so this is a state the // rollout can leave — which is exactly why it is distinct from blocked. PhaseRolledBack Phase = "rolled_back" // PhaseBlocked is a component nothing can safely act on. // // WHAT AN UNPROVABLE ROLLBACK LEAVES BEHIND, and it advertises no capacity. The // alternative to a cordon is guessing, and the two guesses are "assume the old // binary is fine" (which may be running against a migrated ledger) and "assume // the compute is gone" (which sells a running job's slot twice). PhaseBlocked Phase = "blocked" // PhaseDecommissioned is a component an operator has removed from the fleet // after proving its compute is gone. Terminal. PhaseDecommissioned Phase = "decommissioned" // PhaseExempt is a component an operator has recorded a decision to skip. // // DISTINCT FROM DECOMMISSIONED, because the machine is still there. An // exemption says "this host is not part of this rollout"; a decommission says // "this host is gone". Collapsing them would let a rollout complete while a // live host kept an old protocol open with nothing recording that anybody // decided so. PhaseExempt Phase = "exempt" )
func (Phase) Converged ¶
Converged reports whether a component is running the target.
NARROWER THAN Terminal, DELIBERATELY. A rollout is complete when every required component is converged OR an operator has recorded a decision about it, and those are different facts: an exempted host is still running the old release. "Most nodes updated" is not success, and neither is "nothing left to do".
type Policy ¶
type Policy struct {
// Cohort is how many nodes may be past `pending` at once. One is the default
// and the safe answer: a fleet updated one host at a time never loses more
// capacity than one host's worth.
Cohort int `json:"cohort"`
// FailureBudget is how many nodes may end blocked or rolled back before the
// rollout stops starting new ones.
//
// EXPLICIT, because "keep going" and "stop on the first failure" are both
// wrong as a default. A fleet of fifty should not stop for one bad host; a
// fleet of two should not lose both.
FailureBudget int `json:"failure_budget"`
// AllowDowngrade records that an operator asked for a target older than the
// release the fleet is running, by name.
//
// IN THE POLICY BECAUSE IT TRAVELS WITH THE DECISION. The controller host's
// updater reads it out of the ledger to lower the release watermark before
// the older candidate is probed; without it a downgrade is refused at that
// probe and rolls back, which is the safe answer for a decision nobody made.
// The automatic starter never sets it.
AllowDowngrade bool `json:"allow_downgrade,omitempty"`
}
Policy is how much of the fleet a rollout may disturb at once.
PERSISTED AS JSON RATHER THAN AS COLUMNS, because it is the operator's instruction rather than billet's bookkeeping: a field added here should not be a migration, and nothing in the scheduler keys on it.
func DefaultPolicy ¶
func DefaultPolicy() Policy
DefaultPolicy is one host at a time, stopping after one failure.
type Resolver ¶ added in v0.6.0
Resolver turns the deployment's channel or pin into one target that this build could install.
OUTSIDE THIS PACKAGE, AND THAT IS THE LAYERING. Resolving a channel reaches the network and verifying a manifest reaches a signature library, and a ledger writer may do neither (the ledgerwriters rule in .golangci.yml says why). What crosses the seam is the answer: a target that has already passed the same compatibility preflight `billet rollout start` runs, or an error saying why there is none this tick.
type Rollout ¶
type Rollout struct {
ID string
// Generation fences instructions. A node that has acted on a newer one
// refuses an older delivery rather than installing a release the rollout has
// moved past.
Generation int64
// Channel is the channel this resolved from, or empty for an exact pin. Kept
// so a report can say how the target was chosen; nothing re-resolves it.
Channel string
// TargetVersion is what an operator reads. TargetDigest is the identity.
TargetVersion string
TargetDigest string
// PriorVersion is what the controller was running when this began, so a
// rollback has somewhere to go without re-deriving it.
PriorVersion string
Policy Policy
ControllerPhase Phase
State string
CreatedBy string
CreatedAt string
FinishedAt string
TerminalReason string
}
Rollout is one durable fleet decision.
type StartPolicy ¶ added in v0.6.0
type StartPolicy struct {
// Enabled is release.automatic, read through its accessor.
Enabled bool
// OpenAt says whether a rollout may begin at a moment, or is nil for a
// deployment with no maintenance window.
OpenAt func(time.Time) bool
// Channel is the channel followed, or empty for a deployment pinned to Pin.
Channel string
Pin string
// Rollout is the policy every automatic rollout is recorded with. Its
// AllowDowngrade is ignored: an automatic start never downgrades.
Rollout Policy
}
StartPolicy is what the deployment's config says about starting rollouts by itself.
type StartRequest ¶
type StartRequest struct {
Channel string
TargetVersion string
TargetDigest string
PriorVersion string
Policy Policy
CreatedBy string
// Nodes is every host this rollout must converge, as the fleet stands now.
//
// SNAPSHOT AT START, and that is deliberate. A host that registers later is
// running whatever it was installed with and is not part of a decision taken
// before it existed; a host that has disappeared still holds the rollout open,
// because its compute may be running. Both are wrong if the set is recomputed
// on every pass.
Nodes []string
// RefuseAbortedTarget refuses to start when the newest rollout to this digest
// was aborted, INSIDE THE SAME TRANSACTION THAT WOULD INSERT, so a start and
// an abort that land between an automatic starter's read and its write cannot
// be overruled by it. An operator's start leaves it false: `billet rollout
// start` after an abort is the decision to try again.
RefuseAbortedTarget bool
}
StartRequest is one operator decision to move the fleet to a release.
type Starter ¶ added in v0.6.0
type Starter struct {
// contains filtered or unexported fields
}
Starter begins a rollout when the channel names a release the fleet is not on.
THE OTHER HALF OF `release.automatic`. The Coordinator converges a rollout that exists; this is what makes one exist without an operator typing `billet rollout start`. Everything it decides is an observation — the ledger has no open rollout, the channel names something newer, the fleet is not already on it, nobody aborted that exact target — and the one write it makes is the same Store.Start the command makes.
func NewStarter ¶ added in v0.6.0
func NewStarter(store *Store, fleet Fleet, resolve Resolver, policy StartPolicy, ourVersion string, opts ...StarterOption, ) *Starter
NewStarter builds the driver that begins rollouts.
ourVersion is what this binary reports itself as, passed in for the reason the Coordinator's is: a package that read it itself could only be tested against the build running the test.
func (*Starter) Tick ¶ added in v0.6.0
Tick starts a rollout if, and only if, everything observable says one is due.
EVERY STEP IS AN OBSERVATION, IN THE ORDER THAT COSTS LEAST. The config and the window are read first because they need nothing; the ledger next because an open rollout makes the rest moot; the channel after that because it is the one step that leaves the machine. An error is returned only for a ledger that could not be read — every other reason not to start is logged and waited out, because "the channel is unreachable" is a condition the next tick may find changed and not a failure of this process.
type StarterOption ¶ added in v0.6.0
type StarterOption func(*Starter)
StarterOption configures a Starter.
func WithStarterClock ¶ added in v0.6.0
func WithStarterClock(now func() time.Time) StarterOption
WithStarterClock replaces the clock, so a test can drive the window and the rate limit.
func WithStarterLogger ¶ added in v0.6.0
func WithStarterLogger(log *slog.Logger) StarterOption
WithStarterLogger sets where the starter reports.
type Store ¶
type Store struct {
// contains filtered or unexported fields
}
Store is the durable half of a rollout.
func (*Store) Advance ¶
func (s *Store) Advance(ctx context.Context, req AdvanceRequest) error
Advance moves one component through the state machine.
THE CURRENT PHASE IS READ INSIDE THE WRITE TRANSACTION, against the row the write acts on. A caller that read the phase, decided, and wrote would be deciding on a snapshot — and the thing that lands in between is another process blocking the host it is about to install on.
func (*Store) Finish ¶
Finish closes a rollout.
COMPLETION IS NOT "NOTHING LEFT TO DO". A rollout is complete when every required component reports the exact target and a healthy contract, or an operator has recorded an exemption or a decommission after proving compute absence. This refuses to call a rollout complete while any component is still live and unconverged, because "most nodes updated" is the failure mode the issue names.
func (*Store) NewestForTarget ¶ added in v0.6.0
NewestForTarget is the newest rollout, in any state, to one manifest digest, and whether there is one.
func (*Store) Open ¶
Open reads the rollout that is running, if any.
ON THE READ-ONLY POOL, because a control plane asks this on a cadence and a question must not reserve the single writer slot to answer itself.
func (*Store) Start ¶
Start records one fleet decision, or reports the one already running.
IDEMPOTENT ON THE TARGET. `billet rollout start` run twice against the same release must find the rollout that is running rather than creating a second one — the issue's requirement that a repeated instruction does not duplicate a decision. A DIFFERENT target while one is open is refused rather than silently retargeting: work is already underway against the first.