Documentation
¶
Overview ¶
Package hostupgrade replaces billet on one machine, transactionally.
WHY AN EXECUTOR OUTSIDE THE PROCESS BEING REPLACED. A control plane cannot install its own successor: the moment it stops, whatever was going to finish the job has stopped too, and a machine left with the old binary hidden and the new one half-installed has no process on it that knows what was happening. So this runs as its own short-lived program, and everything it is midway through is on the disk rather than in its memory.
WHY A JOURNAL AND NOT A SCRIPT. Every step here is irreversible in a different way — a drained node, a hidden binary, a migrated ledger — and the recovery for each is different too. A script that fails partway leaves a machine in a state only the person reading the script can classify; a journal says which step completed, which makes the next run's decision mechanical.
Index ¶
Constants ¶
const JournalName = "journal.json"
JournalName is the file inside the recovery directory.
const SnapshotName = "ledger.db"
SnapshotName is the ledger snapshot inside the recovery directory.
Variables ¶
var ErrCordoned = errors.New("hostupgrade: the upgrade failed and the rollback could not be proved")
ErrCordoned means the upgrade failed AND the rollback could not be proved.
THE WORST OUTCOME, AND IT HAS TO BE ITS OWN. Nothing about this machine is known: it may be on either release, its ledger may be either schema, and its compute may or may not exist. The only safe act is to leave it alone with its recovery journal intact and tell a person.
var ErrNoJournal = errors.New("hostupgrade: no upgrade is in progress")
ErrNoJournal means there is no upgrade in progress.
var ErrRolledBack = errors.New("hostupgrade: the upgrade failed and the previous release was restored")
ErrRolledBack means the upgrade failed and the previous state was restored.
A SENTINEL BECAUSE IT IS NOT THE SAME OUTCOME AS A FAILED ROLLBACK. This host is healthy on its old release and can be tried again; the other leaves it cordoned. A caller that could not tell them apart would either retry a machine nobody has looked at or cordon one that is fine.
Functions ¶
func Run ¶
Run carries out one upgrade, or resumes one.
THE ORDER IS NOT NEGOTIABLE and every step is placed by what would go wrong if it moved:
- Everything that can fail without consequence happens before anything stops.
- The node stops before the server, so compute drains while the control plane is still there to record what happened to it.
- The binary is hidden before the ledger is touched, so a concurrent operator command cannot enter through either version mid-swap.
- The snapshot is taken before the migration, because a migration is the one step putting the old binary back cannot undo.
- Readiness is PROVED, not assumed from a process being alive.
- The commit record is written last, and after it nothing may restore.
A RESUMED RUN SKIPS WHAT THE JOURNAL SAYS IS DONE. Each step is idempotent anyway, but skipping is what makes a resume cheap enough to be the ordinary recovery rather than something an operator avoids.
Types ¶
type Host ¶
type Host interface {
// StopNode stops the node so its compute drains. UNBOUNDED: a drain waits for
// the work already running for as long as it runs, and no elapsed time here
// may end a job.
StopNode(ctx context.Context) error
// StopServer stops the control plane, after custody has settled.
StopServer(ctx context.Context) error
// PreserveCurrent copies the installed binary, units and configuration into
// the recovery directory, so a rollback restores what was actually there
// rather than what a package would reinstall.
PreserveCurrent(ctx context.Context, dir string) error
// HideBinary removes the installed executable so a concurrent operator
// cannot enter through either version while the swap is underway.
HideBinary(ctx context.Context) error
// Fence makes already-open operator handles refuse transactions, and waits
// for any transaction that began before it to finish.
Fence(ctx context.Context, reason string) error
// ClearFence reopens the ledger to ordinary callers.
ClearFence(ctx context.Context, reason string) error
// SnapshotLedger writes a complete, verified copy of the ledger.
SnapshotLedger(ctx context.Context, dest string) error
// RestoreLedger puts a snapshot back. Only ever called before the commit
// record exists.
RestoreLedger(ctx context.Context, from string) error
// InstallCandidate puts the staged binary in place.
InstallCandidate(ctx context.Context) error
// RecordInstalled writes which release manifest produced the binary now in
// place, bound to the bytes of that binary.
//
// WHAT A VERSION STRING CANNOT SAY. A node's registration names the version it
// was BUILT as; two builds can share one and a moved tag makes them identical,
// so a rollout comparing versions converges on evidence weaker than the
// decision it is converging. This is the record that lets a host say which
// bytes it is actually running.
RecordInstalled(ctx context.Context, digest, version string) error
// Migrate runs the candidate's migrations with it as the only ledger writer.
Migrate(ctx context.Context) error
// ProbeReady starts probe units that open what they inherited and announce
// readiness, while polling nothing and accepting no workload.
ProbeReady(ctx context.Context) error
// StartServices installs the steady-state units and starts them.
StartServices(ctx context.Context) error
// RestorePreserved puts the preserved binary, units and configuration back.
RestorePreserved(ctx context.Context, dir string) error
// ProveStable reports whether the services stayed up long enough to believe.
ProveStable(ctx context.Context) error
}
Host is everything this package does to a machine.
AN INTERFACE BECAUSE THE ORDER IS THE THING WORTH TESTING. Every one of these operations is a real, irreversible act on a real host, and none of them can be exercised in a unit test — but the SEQUENCE is where every defect in this area lives, and a fake makes the sequence assertable. The real implementation composes what already exists: lifeops for the services, internal/state for the fence and the snapshot, releasesource for the candidate.
NOTHING HERE RETURNS PARTIAL SUCCESS. Each method either did its whole job or returns an error, because a recovery that has to reason about half-completed steps is a recovery nobody can write correctly.
type Journal ¶
type Journal struct {
// Dir is the unique recovery directory this upgrade owns.
Dir string `json:"dir"`
// FromVersion and ToVersion are what an operator reads when they find a
// machine mid-upgrade.
FromVersion string `json:"from_version"`
ToVersion string `json:"to_version"`
// TargetDigest is the manifest this candidate came from, so a resuming run
// can tell whether the staged bytes belong to the decision it is resuming.
TargetDigest string `json:"target_digest"`
// RolloutID and Generation tie this to the fleet decision that asked for it,
// or are empty for an operator running it by hand.
//
// RECORDED RATHER THAN ENFORCED HERE, and the distinction matters. What stops a
// delayed instruction from a superseded rollout starting a second transaction
// on this machine is the CLAIM, which refuses while one is in progress at all —
// the generation cannot do that job, because the two instructions may arrive
// with nothing of the first left to compare against. What it does is answer
// "which decision does this machine's half-finished upgrade belong to", for an
// operator reading a journal and for a coordinator reconciling one.
RolloutID string `json:"rollout_id,omitempty"`
Generation int64 `json:"generation,omitempty"`
// PID is the process that claimed this transaction.
//
// A NAME FOR THE HOLDER, NOT A HANDLE ON IT. The transaction lock already says
// whether somebody is working right now — the kernel drops it when the holder
// dies — but it cannot say WHICH process, and an operator looking at a host
// that has been draining for an hour needs something to look at. Nothing acts
// on this: a pid is a number the kernel reuses, and killing an updater that may
// already be installing is exactly what this command must not do.
PID int `json:"pid,omitempty"`
Step Step `json:"step"`
StartedAt string `json:"started_at"`
UpdatedAt string `json:"updated_at"`
// Failure is why this upgrade stopped, when it did.
Failure string `json:"failure,omitempty"`
}
Journal is the durable record of one upgrade.
func ReadJournal ¶
ReadJournal loads the record for an upgrade in progress.
func (*Journal) Reached ¶
Reached reports whether an upgrade got at least as far as a step.
THE RESUME DECISION IN ONE PLACE. Everything a recovery does is keyed on this: what has to be unwound is exactly what was reached, and comparing positions in the ordered list is the only reading of that which cannot disagree with itself.
func (*Journal) SnapshotPath ¶
SnapshotPath is where this upgrade's ledger snapshot lives.
func (*Journal) Write ¶
Write makes the journal durable.
FSYNCED, INCLUDING THE DIRECTORY. The whole point of this file is to survive the crash that happens between two steps, and a record still in the page cache when the machine loses power records a step that did happen as one that did not — after which recovery unwinds work that was already committed.
type Step ¶
type Step string
Step is how far one upgrade has got.
ORDERED, AND THE ORDER IS THE SAFETY CONTENT. Each step is safe to repeat and safe to unwind only because of what has and has not happened before it, so the sequence is written down once, here, rather than implied by the order of calls in a function somebody may reorder.
const ( // StepClaimed is the exclusion taken and nothing else done. StepClaimed Step = "claimed" // StepStaged is a verified candidate on the disk, with nothing stopped. // // EVERYTHING THAT CAN FAIL WITHOUT CONSEQUENCE HAPPENS BEFORE THIS. Resolving // a channel, verifying a signature, downloading an archive and checking a // digest are all things that can go wrong, and all of them go wrong here — // while the deployment is still running normally and the recovery is to do // nothing. StepStaged Step = "staged" // StepStopped is both services stopped and the old binary hidden. StepStopped Step = "stopped" // StepFenced is the maintenance fence written and flushed. StepFenced Step = "fenced" // StepSnapshotted is a complete ledger snapshot in the recovery directory. // // THE POINT BEFORE WHICH NOTHING MAY MIGRATE. A migration is the one step that // cannot be undone by putting the old binary back, because the old binary // refuses a schema it has never heard of — so the snapshot is what makes the // rest of this reversible at all. StepSnapshotted Step = "snapshotted" // StepInstalled is the candidate binary in place. StepInstalled Step = "installed" // StepMigrated is the ledger migrated by the candidate, as the only writer. StepMigrated Step = "migrated" // StepProbed is the candidate proved able to open what it inherited, under // units that poll nothing and accept no workload. StepProbed Step = "probed" // StepCommitted is the durable decision that this upgrade succeeded. // // THE POINT AFTER WHICH RECOVERY MUST NEVER RESTORE THE SNAPSHOT. The fence // may already have opened and admitted operator writes, so putting the old // ledger back would discard work committed against the new one. A crash after // this retries the startup, never the rollback. StepCommitted Step = "committed" // StepRolledBack is the durable decision that this upgrade failed and the // previous state was restored. StepRolledBack Step = "rolled_back" )