hostupgrade

package
v0.10.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 11, 2026 License: Apache-2.0 Imports: 11 Imported by: 0

Documentation

Overview

Package hostupgrade replaces billet on one machine, transactionally.

WHY AN EXECUTOR OUTSIDE THE PROCESS BEING REPLACED. A control plane cannot install its own successor: the moment it stops, whatever was going to finish the job has stopped too, and a machine left with the old binary hidden and the new one half-installed has no process on it that knows what was happening. So this runs as its own short-lived program, and everything it is midway through is on the disk rather than in its memory.

WHY A JOURNAL AND NOT A SCRIPT. Every step here is irreversible in a different way — a drained node, a hidden binary, a migrated ledger — and the recovery for each is different too. A script that fails partway leaves a machine in a state only the person reading the script can classify; a journal says which step completed, which makes the next run's decision mechanical.

Index

Constants

View Source
const JournalName = "journal.json"

JournalName is the file inside the recovery directory.

View Source
const LedgerExternal = "external"

LedgerExternal marks a journal for a host whose ledger is a database billet does not hold.

View Source
const SnapshotName = "ledger.db"

SnapshotName is the ledger snapshot inside the recovery directory.

Variables

View Source
var ErrCordoned = errors.New("hostupgrade: the upgrade failed and the rollback could not be proved")

ErrCordoned means the upgrade failed AND the rollback could not be proved.

THE WORST OUTCOME, AND IT HAS TO BE ITS OWN. Nothing about this machine is known: it may be on either release, its ledger may be either schema, and its compute may or may not exist. The only safe act is to leave it alone with its recovery journal intact and tell a person.

View Source
var ErrNoJournal = errors.New("hostupgrade: no upgrade is in progress")

ErrNoJournal means there is no upgrade in progress.

View Source
var ErrRolledBack = errors.New("hostupgrade: the upgrade failed and the previous release was restored")

ErrRolledBack means the upgrade failed and the previous state was restored.

A SENTINEL BECAUSE IT IS NOT THE SAME OUTCOME AS A FAILED ROLLBACK. This host is healthy on its old release and can be tried again; the other leaves it cordoned. A caller that could not tell them apart would either retry a machine nobody has looked at or cordon one that is fine.

View Source
var ErrUnsafeToRestore = errors.New("hostupgrade: the candidate could not be proved gone")

ErrUnsafeToRestore is what a step wraps when it failed in a way that leaves the candidate possibly still running, so that restoring the previous release and ledger under it would be worse than the failure.

A ROLLBACK OVER A LIVE CANDIDATE IS THE ONE THING WORSE THAN A HUNG PROBE. The probe step found this: a candidate sent SIGKILL whose exit never arrived is one the kernel is holding, and a rollback that restored the ledger while that process still had it open would be corrupting the very thing the snapshot exists to protect. A step that returns this is cordoned with its journal intact; nothing is restored until a person has looked.

Functions

func KnownStep

func KnownStep(s Step) bool

KnownStep reports whether a step is one this build understands.

func Run

func Run(ctx context.Context, req Request) error

Run carries out one upgrade, or resumes one.

THE ORDER IS NOT NEGOTIABLE and every step is placed by what would go wrong if it moved:

  • Everything that can fail without consequence happens before anything stops.
  • The node stops before the server, so compute drains while the control plane is still there to record what happened to it.
  • The binary is hidden before the ledger is touched, so a concurrent operator command cannot enter through either version mid-swap.
  • The snapshot is taken before the migration, because a migration is the one step putting the old binary back cannot undo.
  • Readiness is PROVED, not assumed from a process being alive.
  • The commit record is written last, and after it nothing may restore.

A RESUMED RUN SKIPS WHAT THE JOURNAL SAYS IS DONE. Each step is idempotent anyway, but skipping is what makes a resume cheap enough to be the ordinary recovery rather than something an operator avoids.

Types

type Host

type Host interface {
	// StopNode stops the node so its compute drains. UNBOUNDED: a drain waits for
	// the work already running for as long as it runs, and no elapsed time here
	// may end a job.
	StopNode(ctx context.Context) error

	// StopServer stops the control plane, after custody has settled.
	StopServer(ctx context.Context) error

	// PreserveCurrent copies the installed binary, units and configuration into
	// the recovery directory, so a rollback restores what was actually there
	// rather than what a package would reinstall.
	PreserveCurrent(ctx context.Context, dir string) error

	// HideBinary removes the installed executable so a concurrent operator
	// cannot enter through either version while the swap is underway.
	HideBinary(ctx context.Context) error

	// PrepareImages puts a guest generation the candidate's contract accepts in
	// the cluster for every image this host's microVM tiers boot, or does nothing
	// on a host that boots none. A host whose new binary refuses every image it
	// holds launches nothing until somebody pulls, which is not an upgraded host.
	PrepareImages(ctx context.Context) error

	// Fence makes already-open operator handles refuse transactions, and waits
	// for any transaction that began before it to finish.
	Fence(ctx context.Context, reason string) error

	// ClearFence reopens the ledger to ordinary callers.
	ClearFence(ctx context.Context, reason string) error

	// SnapshotLedger writes a complete, verified copy of the ledger.
	SnapshotLedger(ctx context.Context, dest string) error

	// RestoreLedger puts a snapshot back. Only ever called before the commit
	// record exists.
	RestoreLedger(ctx context.Context, from string) error

	// InstallCandidate puts the staged binary in place.
	InstallCandidate(ctx context.Context) error

	// RecordInstalled writes which release manifest produced the binary now in
	// place, bound to the bytes of that binary.
	//
	// WHAT A VERSION STRING CANNOT SAY. A node's registration names the version it
	// was BUILT as; two builds can share one and a moved tag makes them identical,
	// so a rollout comparing versions converges on evidence weaker than the
	// decision it is converging. This is the record that lets a host say which
	// bytes it is actually running.
	RecordInstalled(ctx context.Context, digest, version string) error

	// Migrate runs the candidate's migrations with it as the only ledger writer.
	Migrate(ctx context.Context) error

	// ProbeReady starts probe units that open what they inherited and announce
	// readiness, while polling nothing and accepting no workload.
	ProbeReady(ctx context.Context) error

	// StartServices installs the steady-state units and starts them.
	StartServices(ctx context.Context) error

	// RestorePreserved puts the preserved binary, units and configuration back.
	RestorePreserved(ctx context.Context, dir string) error

	// ProveStable reports whether the services stayed up long enough to believe.
	ProveStable(ctx context.Context) error
}

Host is everything this package does to a machine.

AN INTERFACE BECAUSE THE ORDER IS THE THING WORTH TESTING. Every one of these operations is a real, irreversible act on a real host, and none of them can be exercised in a unit test — but the SEQUENCE is where every defect in this area lives, and a fake makes the sequence assertable. The real implementation composes what already exists: lifeops for the services, internal/state for the fence and the snapshot, releasesource for the candidate.

NOTHING HERE RETURNS PARTIAL SUCCESS. Each method either did its whole job or returns an error, because a recovery that has to reason about half-completed steps is a recovery nobody can write correctly.

type Journal

type Journal struct {
	// Dir is the unique recovery directory this upgrade owns.
	Dir string `json:"dir"`

	// FromVersion and ToVersion are what an operator reads when they find a
	// machine mid-upgrade.
	FromVersion string `json:"from_version"`
	ToVersion   string `json:"to_version"`

	// TargetDigest is the manifest this candidate came from, so a resuming run
	// can tell whether the staged bytes belong to the decision it is resuming.
	TargetDigest string `json:"target_digest"`

	// RolloutID and Generation tie this to the fleet decision that asked for it,
	// or are empty for an operator running it by hand.
	//
	// RECORDED RATHER THAN ENFORCED HERE, and the distinction matters. What stops a
	// delayed instruction from a superseded rollout starting a second transaction
	// on this machine is the CLAIM, which refuses while one is in progress at all —
	// the generation cannot do that job, because the two instructions may arrive
	// with nothing of the first left to compare against. What it does is answer
	// "which decision does this machine's half-finished upgrade belong to", for an
	// operator reading a journal and for a coordinator reconciling one.
	RolloutID  string `json:"rollout_id,omitempty"`
	Generation int64  `json:"generation,omitempty"`

	// SettlesThrough is the newest fleet decision that had completed when an
	// operator's own run began, so the host settles on it at the commit and a
	// timer does not move the host back to a rollout the person knew about.
	// Zero for a run that serves a rollout, whose own Generation is settled on.
	SettlesThrough int64 `json:"settles_through,omitempty"`

	// AllowDowngrade records that a person asked for a release older than the
	// one this ledger has been served by, and that the transaction may lower the
	// ledger's release watermark to admit it.
	//
	// ON THE JOURNAL BECAUSE A RESUME HAS TO KNOW. The mark is lowered inside the
	// migrate step, and a resumed run that had forgotten the permission would
	// have the candidate refused by a ledger this same transaction had already
	// snapshotted and was about to hand it.
	AllowDowngrade bool `json:"allow_downgrade,omitempty"`

	// Ledger says where this host's ledger lives: empty for the SQLite file in
	// the state directory, LedgerExternal for a database billet does not hold.
	//
	// ON THE JOURNAL BECAUSE IT DECIDES WHICH STEPS EXIST. An external ledger is
	// neither fenced, snapshotted nor migrated by the updater — billet copies no
	// PostgreSQL database, and the candidate migrates when it takes the
	// controller claim — and a resumed run has to skip the same steps the
	// interrupted one did, or it would try to restore a snapshot nothing took.
	Ledger string `json:"ledger,omitempty"`

	// PID is the process that claimed this transaction.
	//
	// A NAME FOR THE HOLDER, NOT A HANDLE ON IT. The transaction lock already says
	// whether somebody is working right now — the kernel drops it when the holder
	// dies — but it cannot say WHICH process, and an operator looking at a host
	// that has been draining for an hour needs something to look at. Nothing acts
	// on this: a pid is a number the kernel reuses, and killing an updater that may
	// already be installing is exactly what this command must not do.
	PID int `json:"pid,omitempty"`

	Step      Step   `json:"step"`
	StartedAt string `json:"started_at"`
	UpdatedAt string `json:"updated_at"`

	// Failure is why this upgrade stopped, when it did.
	Failure string `json:"failure,omitempty"`
}

Journal is the durable record of one upgrade.

func ReadJournal

func ReadJournal(dir string) (*Journal, error)

ReadJournal loads the record for an upgrade in progress.

func (*Journal) Advance

func (j *Journal) Advance(s Step) error

Advance records that one step completed.

func (*Journal) ExternalLedger added in v0.6.0

func (j *Journal) ExternalLedger() bool

ExternalLedger reports whether this transaction has no ledger file to fence, snapshot or migrate.

func (*Journal) Fail

func (j *Journal) Fail(cause error) error

Fail records why an upgrade stopped, without deciding what to do about it.

func (*Journal) Reached

func (j *Journal) Reached(s Step) bool

Reached reports whether an upgrade got at least as far as a step.

THE RESUME DECISION IN ONE PLACE. Everything a recovery does is keyed on this: what has to be unwound is exactly what was reached, and comparing positions in the ordered list is the only reading of that which cannot disagree with itself.

func (*Journal) SnapshotPath

func (j *Journal) SnapshotPath() string

SnapshotPath is where this upgrade's ledger snapshot lives.

func (*Journal) Write

func (j *Journal) Write() error

Write makes the journal durable.

FSYNCED, INCLUDING THE DIRECTORY. The whole point of this file is to survive the crash that happens between two steps, and a record still in the page cache when the machine loses power records a step that did happen as one that did not — after which recovery unwinds work that was already committed.

type Request

type Request struct {
	Journal *Journal
	Host    Host
	Log     *slog.Logger
}

Request is one machine's upgrade.

type Step

type Step string

Step is how far one upgrade has got.

ORDERED, AND THE ORDER IS THE SAFETY CONTENT. Each step is safe to repeat and safe to unwind only because of what has and has not happened before it, so the sequence is written down once, here, rather than implied by the order of calls in a function somebody may reorder.

const (
	// StepClaimed is the exclusion taken and nothing else done.
	StepClaimed Step = "claimed"

	// StepStaged is a verified candidate on the disk, with nothing stopped.
	//
	// EVERYTHING THAT CAN FAIL WITHOUT CONSEQUENCE HAPPENS BEFORE THIS. Resolving
	// a channel, verifying a signature, downloading an archive and checking a
	// digest are all things that can go wrong, and all of them go wrong here —
	// while the deployment is still running normally and the recovery is to do
	// nothing.
	StepStaged Step = "staged"

	// StepStopped is both services stopped and the old binary hidden.
	StepStopped Step = "stopped"
	// StepImaged is a guest generation the candidate's contract accepts in the
	// cluster, for every image this host's microVM tiers boot.
	//
	// BEFORE THE FENCE, because a pull records nothing in the ledger and
	// everything in the cluster and on this host's disk, and AFTER THE STOP,
	// because nothing may launch against a generation while it is being imported
	// and verified. It has no unwinding: a generation the old binary does not
	// accept is inert to it, and the next upgrade finds it already there.
	StepImaged Step = "imaged"

	// StepFenced is the maintenance fence written and flushed.
	StepFenced Step = "fenced"

	// StepSnapshotted is a complete ledger snapshot in the recovery directory.
	//
	// THE POINT BEFORE WHICH NOTHING MAY MIGRATE. A migration is the one step that
	// cannot be undone by putting the old binary back, because the old binary
	// refuses a schema it has never heard of — so the snapshot is what makes the
	// rest of this reversible at all.
	StepSnapshotted Step = "snapshotted"

	// StepInstalled is the candidate binary in place.
	StepInstalled Step = "installed"

	// StepMigrated is the ledger migrated by the candidate, as the only writer.
	StepMigrated Step = "migrated"

	// StepProbed is the candidate proved able to open what it inherited, under
	// units that poll nothing and accept no workload.
	StepProbed Step = "probed"

	// StepCommitted is the durable decision that this upgrade succeeded.
	//
	// THE POINT AFTER WHICH RECOVERY MUST NEVER RESTORE THE SNAPSHOT. The fence
	// may already have opened and admitted operator writes, so putting the old
	// ledger back would discard work committed against the new one. A crash after
	// this retries the startup, never the rollback.
	StepCommitted Step = "committed"

	// StepRolledBack is the durable decision that this upgrade failed and the
	// previous state was restored.
	StepRolledBack Step = "rolled_back"
)

func (Step) Decided

func (s Step) Decided() bool

Decided reports whether a step is a durable terminal decision.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL