selfupgrade

package
v0.8.8 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 30, 2026 License: Apache-2.0 Imports: 28 Imported by: 0

Documentation

Overview

Package selfupgrade replaces the control plane binary with a verified newer release and rolls binary and data back if it does not become healthy. It runs from a host command, never inside the process being replaced.

Index

Constants

View Source
const (
	CauseDiskFull       = "disk_full"
	CauseNetwork        = "network"
	CauseChecksum       = "checksum_mismatch"
	CauseSignature      = "signature_invalid"
	CauseBinaryUnusable = "binary_unusable"
	CauseBackupFailed   = "backup_failed"
	CauseMigration      = "migration_incompatible"
	CauseServiceControl = "service_control"
	CauseSwapFailed     = "swap_failed"
	CauseNotHealthy     = "not_healthy"
	CauseGuardRefused   = "guard_refused"
	CauseUnknown        = "unknown"
)

Failure causes, the stable ids the dashboard and CLI key their copy on.

View Source
const (
	FixActionCleanup  = "cleanup"
	FixActionRetry    = "retry"
	FixActionRollback = "rollback"
	FixActionNone     = "none"
)

Primary fix actions.

View Source
const (
	StepGuard          = "guard"
	StepDownload       = "download"
	StepChecksum       = "verify_checksum"
	StepSignature      = "verify_signature"
	StepProbe          = "probe_binary"
	StepBackup         = "backup"
	StepMigrationCheck = "migration_check"
	StepStop           = "stop"
	StepSwap           = "swap"
	StepStart          = "start"
	StepHealth         = "health"
	StepRollback       = "rollback"
)

Step names, in the order Apply runs them.

View Source
const (
	StatusOK      = "ok"
	StatusFailed  = "failed"
	StatusSkipped = "skipped"
)

Step statuses.

View Source
const (
	OutcomeRunning    = "running"
	OutcomeSucceeded  = "succeeded"
	OutcomeRolledBack = "rolled_back"
	OutcomeRefused    = "refused"
	OutcomeFailed     = "failed"
)

Attempt outcomes. Running is only ever seen in a journal.

View Source
const (
	SourceMetadata = "metadata"
	SourceHeading  = "heading"
)

Breaking change sources.

View Source
const (
	DefaultRestorePoints = 3
	DefaultRestoreMaxMB  = 2048
)

Restore point file names and retention defaults.

View Source
const (
	TrustRootReady       = "ready"
	TrustRootCached      = "cached"
	TrustRootChecking    = "checking"
	TrustRootUnavailable = "unavailable"
)

Trust root states reported to the preflight.

View Source
const (
	VerifyAuto    = "auto"
	VerifyRequire = "require"
	VerifyOff     = "off"
)

Signature modes, the same vocabulary as the installer's APP_INSTALL_VERIFY.

View Source
const CosignOIDCIssuer = "https://token.actions.githubusercontent.com"

CosignOIDCIssuer is the issuer of the keyless certificates release.yml signs with.

View Source
const DefaultHealthTimeout = 90 * time.Second

DefaultHealthTimeout bounds how long a new release may take to become healthy before the previous binary and data are put back.

View Source
const DetailOffWhilePinned = "signature check disabled by the operator although this host is pinned to signed releases"

DetailOffWhilePinned is the timeline detail for an operator override.

View Source
const DiskReclaimFixID = "disk.reclaim"

DiskReclaimFixID is the fixit fix a disk_full failure leads with.

View Source
const ExitDowngradeRefused = 78

ExitDowngradeRefused is the process exit status when a binary refuses to start on a database written by a newer release. systemd can be told not to restart on it (RestartPreventExitStatus).

View Source
const JournalDir = "self-upgrade"

JournalDir is the directory under the data dir that holds attempt files.

View Source
const KeepFinished = 5

KeepFinished is how many finished attempt files stay on disk so the progress view can still show a just-finished run's live log.

View Source
const MaxLogLines = 300

MaxLogLines bounds Attempt.Log; the oldest lines are dropped first.

View Source
const MetaMarker = "levelrail-upgrade:"

MetaMarker opens the machine-readable upgrade metadata comment a release body may carry. The JSON inside lists breaking changes explicitly.

View Source
const PinEscapeHint = "" /* 151-byte string literal not displayed */

PinEscapeHint names the documented, explicit ways past a pinned refusal.

View Source
const PinFileName = "release-signature.pin"

PinFileName is the marker, under the data dir, that pins this host to signed releases. install.sh writes the same key=value format.

View Source
const RestorePointDir = "restore-points"

RestorePointDir is the directory under the data dir that holds restore points.

View Source
const StepRestorePoint = "restore_point"

StepRestorePoint names the single step of a manual restore.

Variables

View Source
var (
	ErrChecksumMismatch  = errors.New("downloaded binary does not match its published checksum")
	ErrChecksumMissing   = errors.New("release publishes no checksum for this binary")
	ErrSignatureInvalid  = errors.New("release signature verification failed")
	ErrSignatureRequired = errors.New("signature verification is required but unavailable")
	ErrSignaturePinned   = errors.New("this host only accepts signed releases")
	ErrBreakingNotAcked  = errors.New("breaking changes need acknowledgement")
	ErrDowngrade         = errors.New(upgrade.DowngradeRefusal)
	ErrVersionMismatch   = errors.New("downloaded binary reports a different version than requested")
	ErrMigrationCheck    = errors.New("migration dry-run failed")
	ErrNotHealthy        = errors.New("new release did not become healthy")
	// ErrDryRunUnsupported is returned by a migration check when the target
	// binary has no dry-run subcommand, which is not a failure.
	ErrDryRunUnsupported = errors.New("target binary has no migration dry-run")
)

Refusals that change nothing on the host.

View Source
var ErrManifestInvalid = errors.New("release manifest is not valid")

ErrManifestInvalid means the release manifest parsed but does not describe the release that was asked for.

View Source
var ErrNoRestorePoint = errors.New("restore point not found")

ErrNoRestorePoint means the id names no kept restore point.

View Source
var ErrUpgradeRunning = errors.New("an upgrade is still running or did not finish")

ErrUpgradeRunning means an upgrade has not finished, so nothing may be restored.

Functions

func AckError

func AckError(missing []Breaking) error

AckError names the unacknowledged breaking changes in a refusal message.

func DowngradeMessage

func DowngradeMessage(in GuardInput) string

DowngradeMessage explains the refusal and gives the exact recovery commands. Migrations only go forward, so the database cannot be opened by an older binary, and the message says which release to run instead.

func ImportJournal

func ImportJournal(ctx context.Context, j *Journal, db AttemptStore) (int, error)

ImportJournal copies every journaled attempt into the database and prunes all but the newest finished files. Running attempts stay on disk so a recovery can still read their paths. It returns how many rows were written.

func ParseChecksums

func ParseChecksums(r io.Reader) (map[string]string, error)

ParseChecksums reads sha256sum output into name to hex digest.

func PinPath added in v0.3.0

func PinPath(dataDir string) string

PinPath is the pin file's location under dataDir.

func PrunePlan

func PrunePlan(pts []RestorePoint, maxCount int, maxBytes int64) []string

PrunePlan returns the ids to delete from pts (newest first) so at most maxCount points and maxBytes bytes remain. The newest point always stays, even when it alone exceeds maxBytes. A limit of zero or less is not applied.

func ReleaseIdentityRegexp

func ReleaseIdentityRegexp(repo string) string

ReleaseIdentityRegexp matches the release workflow of repo ("owner/name"), the only signer identity a release signature may carry.

func RemovePin added in v0.3.0

func RemovePin(dataDir string) (bool, error)

RemovePin clears the pin and reports whether one existed.

func SyncRestorePoints

func SyncRestorePoints(ctx context.Context, rp *RestorePoints, db RestorePointStore) error

SyncRestorePoints makes the database table mirror the points kept on disk.

func TUFCacheDir

func TUFCacheDir(dataDir string) string

TUFCacheDir is where Sigstore trust metadata is kept under the data dir.

func ToStore

func ToStore(a Attempt) store.SelfUpgradeAttempt

ToStore converts an attempt to its database row.

func ValidTag

func ValidTag(tag string) bool

ValidTag accepts release tags such as v1.2.3 or v1.2.3-beta.4 and nothing that could escape a URL path or a file name.

func VerifyChecksum

func VerifyChecksum(path, checksumsPath, asset string) error

VerifyChecksum checks the file at path against asset's entry in the checksums file.

func WritePin added in v0.3.0

func WritePin(dataDir string, p Pin) error

WritePin atomically records p. World-readable so a non-root control plane can report it; only root can write the data dir.

Types

type Attempt

type Attempt struct {
	ID          string       `json:"id"`
	FromVersion string       `json:"from_version"`
	ToVersion   string       `json:"to_version"`
	FromSchema  int          `json:"from_schema"`
	ToSchema    int          `json:"to_schema"`
	Initiator   string       `json:"initiator"`
	Outcome     string       `json:"outcome"`
	FailedStep  string       `json:"failed_step,omitempty"`
	Error       string       `json:"error,omitempty"`
	BackupName  string       `json:"backup_name,omitempty"`
	Acked       []string     `json:"acked,omitempty"`
	Steps       []StepRecord `json:"steps"`
	// Current is the step in flight, so a watcher sees it before it ends.
	Current string `json:"current_step,omitempty"`
	// Log is a bounded stream of step events for the live progress view.
	Log        []LogLine `json:"log,omitempty"`
	StartedAt  time.Time `json:"started_at"`
	FinishedAt time.Time `json:"finished_at,omitempty"`

	// DatabaseRestored is set when a rollback put the snapshot back, and
	// RolledBackMinutes is how long the discarded changes span.
	DatabaseRestored  bool      `json:"database_restored,omitempty"`
	RolledBackMinutes int       `json:"rolled_back_minutes,omitempty"`
	RestorePointID    string    `json:"restore_point_id,omitempty"`
	AgentsBefore      int       `json:"agents_before,omitempty"`
	SnapshotAt        time.Time `json:"snapshot_at,omitempty"`

	PrevBinary    string `json:"prev_binary,omitempty"`
	SnapshotPath  string `json:"snapshot_path,omitempty"`
	SwappedBinary bool   `json:"swapped_binary,omitempty"`
}

Attempt is one upgrade run. The unexported-to-the-database fields at the end (paths) only live in the journal so a restart in the middle can recover.

func Apply

func Apply(ctx context.Context, d Deps, req Request) (Attempt, error)

Apply runs the upgrade: guard, download, verify, back up, migration dry run, swap, restart, health check, and rollback on failure. Nothing on the host changes before the stop step. err is nil only on success.

func Recover

func Recover(ctx context.Context, d Deps, a Attempt, healthTimeout time.Duration) (Attempt, error)

Recover finishes an attempt whose host command died mid-way: closes one that never stopped the service, rolls back one that did, confirms a healthy one.

func RestoreFromPoint

func RestoreFromPoint(ctx context.Context, d Deps, rp *RestorePoints, id, attemptID, initiator, currentVersion string) (Attempt, error)

RestoreFromPoint puts a restore point's binary (and unit file) back and, when the live database was migrated past the point's schema, the database copy too. It never restores a database that is still compatible, so nothing is lost silently. The result is journaled like any attempt.

func (Attempt) AckedJSON

func (a Attempt) AckedJSON() string

AckedJSON encodes the acknowledged breaking-change ids for the database.

func (*Attempt) AddLog

func (a *Attempt) AddLog(at time.Time, step, line string)

AddLog appends a line, dropping the oldest beyond MaxLogLines.

func (Attempt) StepsJSON

func (a Attempt) StepsJSON() string

StepsJSON encodes the timeline for the database column.

type AttemptStore

type AttemptStore interface {
	UpsertSelfUpgradeAttempt(ctx context.Context, a store.SelfUpgradeAttempt) error
}

AttemptStore is the persistence ImportJournal needs; *store.DB satisfies it.

type BackupInfo

type BackupInfo struct {
	Name         string
	SnapshotPath string
}

BackupInfo names the pre-upgrade restore point.

type Breaking

type Breaking struct {
	ID          string `json:"id"`
	Version     string `json:"version"`
	Summary     string `json:"summary"`
	RequiresAck bool   `json:"requires_ack"`
	Source      string `json:"source"`
}

Breaking is one breaking change between two versions. RequiresAck means the upgrade refuses to start until the operator acknowledges it by id.

func BreakingBetween

func BreakingBetween(releases []ReleaseNotes, current, target string) []Breaking

BreakingBetween collects the breaking changes of every release newer than current and no newer than target, oldest first. Versions that do not order are skipped: an unknown tag cannot be said to sit between the two.

func MissingAcks

func MissingAcks(breaking []Breaking, acked []string) []Breaking

MissingAcks returns the breaking changes that require acknowledgement and were not acknowledged.

func ParseBreaking

func ParseBreaking(tag, body string) []Breaking

ParseBreaking extracts the breaking changes one release body declares. The explicit metadata comment wins; without it, bullets under a "Breaking changes" heading count and always require acknowledgement.

type Deps

type Deps struct {
	BinaryPath string
	WorkDir    string
	Asset      string
	Journal    *Journal
	Log        *slog.Logger
	Now        func() time.Time

	Fetch       func(ctx context.Context, tag, asset, dir string) (Fetched, error)
	VerifySig   func(ctx context.Context, checksumsPath, bundlePath string) (string, error)
	Probe       func(ctx context.Context, binary string) (ProbeInfo, error)
	DBSchema    func(ctx context.Context) (int, error)
	Backup      func(ctx context.Context) (BackupInfo, error)
	MigrateTest func(ctx context.Context, binary, snapshotPath string) (int, error)
	Stop        func(ctx context.Context) error
	Start       func(ctx context.Context) error
	WaitHealthy func(ctx context.Context, timeout time.Duration) error
	// RestorePoint keeps the snapshot, binary and unit file as a restore point
	// and returns its id. Optional.
	RestorePoint func(ctx context.Context, a Attempt, snapshotPath string) (string, error)
	// AgentsOnline counts connected agents before the stop, for Verify. Optional.
	AgentsOnline func(ctx context.Context) (int, error)
	// Verify judges the new release inside the health window. When nil,
	// WaitHealthy alone decides.
	Verify func(ctx context.Context, window time.Duration, want VerifyExpect) (string, error)
	// RestoreUnit puts a saved unit file back and reloads the service manager.
	RestoreUnit func(ctx context.Context, savedPath string) error
	// RestoreDB puts the snapshot in place of the live database and returns
	// where the replaced database was kept.
	RestoreDB   func(ctx context.Context, snapshotPath string) (string, error)
	WriteMarker func(ctx context.Context, a Attempt) error
	ClearMarker func()
	Audit       func(ctx context.Context, detail string) error
}

Deps are the host operations Apply drives. Tests fake them; the host command wires real ones (HTTPS, cosign, systemd, sqlite, filesystem).

type Fetched

type Fetched struct {
	BinaryPath    string
	ChecksumsPath string
	BundlePath    string
}

Fetched is a downloaded release: the binary, its checksums file and the signature bundle (empty when the release publishes none).

type Fetcher

type Fetcher struct {
	// BaseURL is the release download prefix with the tag as its last path
	// segment appended by Fetch, for example https://host/org/repo/releases/download.
	BaseURL string
	Client  *http.Client
	// AllowHTTP permits a plain-http BaseURL, for tests against a local server.
	AllowHTTP bool
	MaxBytes  int64
	// RedirectHosts, when set, limits redirects to these hosts and their
	// subdomains, so a release asset cannot bounce the server to an internal address.
	RedirectHosts []string
}

Fetcher downloads release assets over HTTPS.

func (Fetcher) Fetch

func (f Fetcher) Fetch(ctx context.Context, tag, asset, dir string) (Fetched, error)

Fetch downloads asset, checksums.txt and the signature bundle of tag into dir. The checksums file is required, the bundle is optional.

type GuardInput

type GuardInput struct {
	BinaryVersion string
	BinarySchema  int
	DBSchema      int
	// DBVersion is the release that last ran on the database, "" if unknown.
	DBVersion string
	// Program is the executable's name, used in the recovery commands.
	Program string
	// InstallerCommand is the verified installer invocation prefix, for
	// example "curl -fsSL <url> | sudo".
	InstallerCommand string
}

GuardInput describes a binary that found a database newer than itself.

type Journal

type Journal struct {
	Dir string
}

Journal persists attempts as one JSON file each. It is the source of truth while an attempt runs, because the database may be swapped underneath it.

func NewJournal

func NewJournal(dataDir string) *Journal

NewJournal returns the journal under dataDir.

func (*Journal) List

func (j *Journal) List() ([]Attempt, error)

List returns every readable attempt, oldest first. Unreadable files are skipped: a half-written journal must not block an upgrade.

func (*Journal) Remove

func (j *Journal) Remove(id string)

Remove deletes an attempt's file.

func (*Journal) Running

func (j *Journal) Running() ([]Attempt, error)

Running returns attempts that never reached a final outcome.

func (*Journal) Save

func (j *Journal) Save(a Attempt) error

Save writes a atomically with owner-only permissions.

type LogLine

type LogLine struct {
	At   time.Time `json:"at"`
	Step string    `json:"step,omitempty"`
	Line string    `json:"line"`
}

LogLine is one event in an attempt's live log.

type Pin added in v0.3.0

type Pin struct {
	Version    string
	VerifiedAt time.Time
}

Pin records the first release whose signature this host verified.

func ReadPin added in v0.3.0

func ReadPin(dataDir string) (Pin, bool, error)

ReadPin reports whether dataDir holds a pin. A present but unreadable or empty file still pins: failing open would let a truncated file unpin.

type ProbeInfo

type ProbeInfo struct {
	Version string
	Schema  int
}

ProbeInfo is what a downloaded binary reports about itself.

type ReleaseNotes

type ReleaseNotes struct {
	Tag  string
	Body string
}

ReleaseNotes is one release's tag and notes body.

type Request

type Request struct {
	ID             string
	CurrentVersion string
	TargetVersion  string
	Initiator      string
	Breaking       []Breaking
	Acked          []string
	HealthTimeout  time.Duration
}

Request is one upgrade the operator asked for.

type RestorePoint

type RestorePoint struct {
	ID        string    `json:"id"`
	Version   string    `json:"version"`
	Schema    int       `json:"schema"`
	AttemptID string    `json:"attempt_id,omitempty"`
	CreatedAt time.Time `json:"created_at"`
	SizeBytes int64     `json:"size_bytes"`
	HasBinary bool      `json:"has_binary"`
	HasUnit   bool      `json:"has_unit"`
}

RestorePoint is a consistent copy of the database, the binary and the unit file taken just before an upgrade.

type RestorePointStore

type RestorePointStore interface {
	ReplaceUpgradeRestorePoints(ctx context.Context, pts []store.UpgradeRestorePoint) error
}

RestorePointStore is the persistence SyncRestorePoints needs; *store.DB satisfies it.

type RestorePoints

type RestorePoints struct{ Dir string }

RestorePoints keeps restore points as one directory each, so they stay readable while the database itself is being swapped.

func NewRestorePoints

func NewRestorePoints(dataDir string) *RestorePoints

NewRestorePoints returns the restore points under dataDir.

func (*RestorePoints) BinaryPath

func (s *RestorePoints) BinaryPath(id string) string

BinaryPath locates a point's binary copy.

func (*RestorePoints) Create

func (s *RestorePoints) Create(meta RestorePoint, snapshotPath, binaryPath, unitPath string) (RestorePoint, error)

Create copies the database snapshot and, when the paths are set and exist, the binary and unit file into a new point described by meta. Files are 0600.

func (*RestorePoints) DBPath

func (s *RestorePoints) DBPath(id string) string

DBPath locates a point's database copy.

func (*RestorePoints) Get

func (s *RestorePoints) Get(id string) (RestorePoint, error)

Get returns the point with id.

func (*RestorePoints) List

func (s *RestorePoints) List() ([]RestorePoint, error)

List returns every readable point, newest first.

func (*RestorePoints) Prune

func (s *RestorePoints) Prune(maxCount int, maxBytes int64) ([]string, error)

Prune applies PrunePlan to the stored points and returns the removed ids.

func (*RestorePoints) Remove

func (s *RestorePoints) Remove(id string)

Remove deletes a point's directory.

func (*RestorePoints) UnitPath

func (s *RestorePoints) UnitPath(id string) string

UnitPath locates a point's unit file copy.

type SchemaResult

type SchemaResult struct {
	Version string
	Schema  int
	// Detail is the signature verification note.
	Detail string
}

SchemaResult is what a verified release manifest says about the release.

func FetchReleaseSchema

func FetchReleaseSchema(ctx context.Context, f Fetcher, s Signer, tag, parent string) (SchemaResult, error)

FetchReleaseSchema downloads the release manifest of tag with checksums.txt and its signature, verifies the manifest against the signed checksums, and returns the highest migration version the release supports. It never runs the release binary. Size, scheme and redirect limits come from f.

type Signer

type Signer struct {
	Mode           string
	CosignPath     string
	IdentityRegexp string
	Issuer         string
	// Pinned is set once this host has verified a release signature: from
	// then on a missing bundle or cosign is refused unless Mode is off.
	Pinned bool
	// OnVerified runs after a successful verification, to write the pin.
	OnVerified func()
	// TUFCacheDir keeps Sigstore trust metadata between runs; empty uses the default.
	TUFCacheDir string
	// Run executes an external cosign; when it and CosignPath are both unset
	// the signature is verified in process.
	Run func(ctx context.Context, name string, args ...string) ([]byte, error)
}

Signer verifies a cosign keyless bundle over a checksums file.

func (Signer) Verify

func (s Signer) Verify(ctx context.Context, checksumsPath, bundlePath string) (string, error)

Verify returns a human detail for the timeline. A failed verification is always an error. A missing cosign or bundle is an error in require mode and once the host is pinned.

type StepFailure

type StepFailure struct {
	Cause string `json:"cause"`
	// Primary is the one button to lead with: cleanup, retry, rollback or none.
	Primary string `json:"primary"`
	// FixID is the fixit fix behind a cleanup primary.
	FixID string `json:"fix_id,omitempty"`
	// Retryable and Rollbackable say which buttons make sense.
	Retryable    bool `json:"retryable"`
	Rollbackable bool `json:"rollbackable"`
	// CLI is the host-side equivalent, shown collapsed.
	CLI string `json:"cli,omitempty"`
}

StepFailure is the typed reading of one failed upgrade step.

func ClassifyFailure

func ClassifyFailure(step, reason string) StepFailure

ClassifyFailure maps a failed step and its reason to a typed cause and fix. Message matches win over step defaults, so a full disk is cleanup anywhere.

type StepRecord

type StepRecord struct {
	Name       string    `json:"name"`
	Status     string    `json:"status"`
	Detail     string    `json:"detail,omitempty"`
	At         time.Time `json:"at"`
	DurationMS int64     `json:"duration_ms"`
}

StepRecord is one step's result in an attempt's timeline.

type TrustRoot

type TrustRoot struct {
	// contains filtered or unexported fields
}

TrustRoot tracks whether the Sigstore trust root can be loaded, refreshing it in the background so callers never wait on the network.

func NewTrustRoot

func NewTrustRoot(cacheDir string, logger *slog.Logger) *TrustRoot

NewTrustRoot returns a tracker backed by the loader verification uses.

func (*TrustRoot) State

func (t *TrustRoot) State() string

State returns the current state without blocking, and starts a background refresh when the last result is missing or stale.

func (*TrustRoot) Warm

func (t *TrustRoot) Warm()

Warm starts a background refresh unless one is running or the last result is fresh.

type VerifyExpect

type VerifyExpect struct {
	// Schema is the database schema head of the new release.
	Schema int
	// AgentsOnline is how many agents were connected before the upgrade.
	AgentsOnline int
}

VerifyExpect is what a healthy new release must show.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL