Documentation
¶
Overview ¶
Package selfupgrade replaces the control plane binary with a verified newer release and rolls binary and data back if it does not become healthy. It runs from a host command, never inside the process being replaced.
Index ¶
- Constants
- Variables
- func AckError(missing []Breaking) error
- func DowngradeMessage(in GuardInput) string
- func ImportJournal(ctx context.Context, j *Journal, db AttemptStore) (int, error)
- func ParseChecksums(r io.Reader) (map[string]string, error)
- func PinPath(dataDir string) string
- func PrunePlan(pts []RestorePoint, maxCount int, maxBytes int64) []string
- func ReleaseIdentityRegexp(repo string) string
- func RemovePin(dataDir string) (bool, error)
- func SyncRestorePoints(ctx context.Context, rp *RestorePoints, db RestorePointStore) error
- func TUFCacheDir(dataDir string) string
- func ToStore(a Attempt) store.SelfUpgradeAttempt
- func ValidTag(tag string) bool
- func VerifyChecksum(path, checksumsPath, asset string) error
- func WritePin(dataDir string, p Pin) error
- type Attempt
- type AttemptStore
- type BackupInfo
- type Breaking
- type Deps
- type Fetched
- type Fetcher
- type GuardInput
- type Journal
- type LogLine
- type Pin
- type ProbeInfo
- type ReleaseNotes
- type Request
- type RestorePoint
- type RestorePointStore
- type RestorePoints
- func (s *RestorePoints) BinaryPath(id string) string
- func (s *RestorePoints) Create(meta RestorePoint, snapshotPath, binaryPath, unitPath string) (RestorePoint, error)
- func (s *RestorePoints) DBPath(id string) string
- func (s *RestorePoints) Get(id string) (RestorePoint, error)
- func (s *RestorePoints) List() ([]RestorePoint, error)
- func (s *RestorePoints) Prune(maxCount int, maxBytes int64) ([]string, error)
- func (s *RestorePoints) Remove(id string)
- func (s *RestorePoints) UnitPath(id string) string
- type SchemaResult
- type Signer
- type StepFailure
- type StepRecord
- type TrustRoot
- type VerifyExpect
Constants ¶
const ( CauseDiskFull = "disk_full" CauseNetwork = "network" CauseChecksum = "checksum_mismatch" CauseSignature = "signature_invalid" CauseBinaryUnusable = "binary_unusable" CauseBackupFailed = "backup_failed" CauseMigration = "migration_incompatible" CauseServiceControl = "service_control" CauseSwapFailed = "swap_failed" CauseNotHealthy = "not_healthy" CauseGuardRefused = "guard_refused" CauseUnknown = "unknown" )
Failure causes, the stable ids the dashboard and CLI key their copy on.
const ( FixActionCleanup = "cleanup" FixActionRetry = "retry" FixActionRollback = "rollback" FixActionNone = "none" )
Primary fix actions.
const ( StepGuard = "guard" StepDownload = "download" StepChecksum = "verify_checksum" StepSignature = "verify_signature" StepProbe = "probe_binary" StepBackup = "backup" StepMigrationCheck = "migration_check" StepStop = "stop" StepSwap = "swap" StepStart = "start" StepHealth = "health" StepRollback = "rollback" )
Step names, in the order Apply runs them.
const ( StatusOK = "ok" StatusFailed = "failed" StatusSkipped = "skipped" )
Step statuses.
const ( OutcomeRunning = "running" OutcomeSucceeded = "succeeded" OutcomeRolledBack = "rolled_back" OutcomeRefused = "refused" OutcomeFailed = "failed" )
Attempt outcomes. Running is only ever seen in a journal.
const ( SourceMetadata = "metadata" SourceHeading = "heading" )
Breaking change sources.
const ( DefaultRestorePoints = 3 DefaultRestoreMaxMB = 2048 )
Restore point file names and retention defaults.
const ( TrustRootReady = "ready" TrustRootCached = "cached" TrustRootChecking = "checking" )
Trust root states reported to the preflight.
const ( VerifyAuto = "auto" VerifyRequire = "require" VerifyOff = "off" )
Signature modes, the same vocabulary as the installer's APP_INSTALL_VERIFY.
const CosignOIDCIssuer = "https://token.actions.githubusercontent.com"
CosignOIDCIssuer is the issuer of the keyless certificates release.yml signs with.
const DefaultHealthTimeout = 90 * time.Second
DefaultHealthTimeout bounds how long a new release may take to become healthy before the previous binary and data are put back.
const DetailOffWhilePinned = "signature check disabled by the operator although this host is pinned to signed releases"
DetailOffWhilePinned is the timeline detail for an operator override.
const DiskReclaimFixID = "disk.reclaim"
DiskReclaimFixID is the fixit fix a disk_full failure leads with.
const ExitDowngradeRefused = 78
ExitDowngradeRefused is the process exit status when a binary refuses to start on a database written by a newer release. systemd can be told not to restart on it (RestartPreventExitStatus).
const JournalDir = "self-upgrade"
JournalDir is the directory under the data dir that holds attempt files.
const KeepFinished = 5
KeepFinished is how many finished attempt files stay on disk so the progress view can still show a just-finished run's live log.
const MaxLogLines = 300
MaxLogLines bounds Attempt.Log; the oldest lines are dropped first.
const MetaMarker = "levelrail-upgrade:"
MetaMarker opens the machine-readable upgrade metadata comment a release body may carry. The JSON inside lists breaking changes explicitly.
const PinEscapeHint = "" /* 151-byte string literal not displayed */
PinEscapeHint names the documented, explicit ways past a pinned refusal.
const PinFileName = "release-signature.pin"
PinFileName is the marker, under the data dir, that pins this host to signed releases. install.sh writes the same key=value format.
const RestorePointDir = "restore-points"
RestorePointDir is the directory under the data dir that holds restore points.
const StepRestorePoint = "restore_point"
StepRestorePoint names the single step of a manual restore.
Variables ¶
var ( ErrChecksumMismatch = errors.New("downloaded binary does not match its published checksum") ErrChecksumMissing = errors.New("release publishes no checksum for this binary") ErrSignatureInvalid = errors.New("release signature verification failed") ErrSignatureRequired = errors.New("signature verification is required but unavailable") ErrSignaturePinned = errors.New("this host only accepts signed releases") ErrBreakingNotAcked = errors.New("breaking changes need acknowledgement") ErrDowngrade = errors.New(upgrade.DowngradeRefusal) ErrVersionMismatch = errors.New("downloaded binary reports a different version than requested") ErrMigrationCheck = errors.New("migration dry-run failed") ErrNotHealthy = errors.New("new release did not become healthy") // ErrDryRunUnsupported is returned by a migration check when the target // binary has no dry-run subcommand, which is not a failure. ErrDryRunUnsupported = errors.New("target binary has no migration dry-run") )
Refusals that change nothing on the host.
var ErrManifestInvalid = errors.New("release manifest is not valid")
ErrManifestInvalid means the release manifest parsed but does not describe the release that was asked for.
var ErrNoRestorePoint = errors.New("restore point not found")
ErrNoRestorePoint means the id names no kept restore point.
var ErrUpgradeRunning = errors.New("an upgrade is still running or did not finish")
ErrUpgradeRunning means an upgrade has not finished, so nothing may be restored.
Functions ¶
func DowngradeMessage ¶
func DowngradeMessage(in GuardInput) string
DowngradeMessage explains the refusal and gives the exact recovery commands. Migrations only go forward, so the database cannot be opened by an older binary, and the message says which release to run instead.
func ImportJournal ¶
ImportJournal copies every journaled attempt into the database and prunes all but the newest finished files. Running attempts stay on disk so a recovery can still read their paths. It returns how many rows were written.
func ParseChecksums ¶
ParseChecksums reads sha256sum output into name to hex digest.
func PrunePlan ¶
func PrunePlan(pts []RestorePoint, maxCount int, maxBytes int64) []string
PrunePlan returns the ids to delete from pts (newest first) so at most maxCount points and maxBytes bytes remain. The newest point always stays, even when it alone exceeds maxBytes. A limit of zero or less is not applied.
func ReleaseIdentityRegexp ¶
ReleaseIdentityRegexp matches the release workflow of repo ("owner/name"), the only signer identity a release signature may carry.
func SyncRestorePoints ¶
func SyncRestorePoints(ctx context.Context, rp *RestorePoints, db RestorePointStore) error
SyncRestorePoints makes the database table mirror the points kept on disk.
func TUFCacheDir ¶
TUFCacheDir is where Sigstore trust metadata is kept under the data dir.
func ToStore ¶
func ToStore(a Attempt) store.SelfUpgradeAttempt
ToStore converts an attempt to its database row.
func ValidTag ¶
ValidTag accepts release tags such as v1.2.3 or v1.2.3-beta.4 and nothing that could escape a URL path or a file name.
func VerifyChecksum ¶
VerifyChecksum checks the file at path against asset's entry in the checksums file.
Types ¶
type Attempt ¶
type Attempt struct {
ID string `json:"id"`
FromVersion string `json:"from_version"`
ToVersion string `json:"to_version"`
FromSchema int `json:"from_schema"`
ToSchema int `json:"to_schema"`
Initiator string `json:"initiator"`
Outcome string `json:"outcome"`
FailedStep string `json:"failed_step,omitempty"`
Error string `json:"error,omitempty"`
BackupName string `json:"backup_name,omitempty"`
Acked []string `json:"acked,omitempty"`
Steps []StepRecord `json:"steps"`
// Current is the step in flight, so a watcher sees it before it ends.
Current string `json:"current_step,omitempty"`
// Log is a bounded stream of step events for the live progress view.
Log []LogLine `json:"log,omitempty"`
StartedAt time.Time `json:"started_at"`
FinishedAt time.Time `json:"finished_at,omitempty"`
// DatabaseRestored is set when a rollback put the snapshot back, and
// RolledBackMinutes is how long the discarded changes span.
DatabaseRestored bool `json:"database_restored,omitempty"`
RolledBackMinutes int `json:"rolled_back_minutes,omitempty"`
RestorePointID string `json:"restore_point_id,omitempty"`
AgentsBefore int `json:"agents_before,omitempty"`
SnapshotAt time.Time `json:"snapshot_at,omitempty"`
PrevBinary string `json:"prev_binary,omitempty"`
SnapshotPath string `json:"snapshot_path,omitempty"`
SwappedBinary bool `json:"swapped_binary,omitempty"`
}
Attempt is one upgrade run. The unexported-to-the-database fields at the end (paths) only live in the journal so a restart in the middle can recover.
func Apply ¶
Apply runs the upgrade: guard, download, verify, back up, migration dry run, swap, restart, health check, and rollback on failure. Nothing on the host changes before the stop step. err is nil only on success.
func Recover ¶
Recover finishes an attempt whose host command died mid-way: closes one that never stopped the service, rolls back one that did, confirms a healthy one.
func RestoreFromPoint ¶
func RestoreFromPoint(ctx context.Context, d Deps, rp *RestorePoints, id, attemptID, initiator, currentVersion string) (Attempt, error)
RestoreFromPoint puts a restore point's binary (and unit file) back and, when the live database was migrated past the point's schema, the database copy too. It never restores a database that is still compatible, so nothing is lost silently. The result is journaled like any attempt.
type AttemptStore ¶
type AttemptStore interface {
UpsertSelfUpgradeAttempt(ctx context.Context, a store.SelfUpgradeAttempt) error
}
AttemptStore is the persistence ImportJournal needs; *store.DB satisfies it.
type BackupInfo ¶
BackupInfo names the pre-upgrade restore point.
type Breaking ¶
type Breaking struct {
ID string `json:"id"`
Version string `json:"version"`
Summary string `json:"summary"`
RequiresAck bool `json:"requires_ack"`
Source string `json:"source"`
}
Breaking is one breaking change between two versions. RequiresAck means the upgrade refuses to start until the operator acknowledges it by id.
func BreakingBetween ¶
func BreakingBetween(releases []ReleaseNotes, current, target string) []Breaking
BreakingBetween collects the breaking changes of every release newer than current and no newer than target, oldest first. Versions that do not order are skipped: an unknown tag cannot be said to sit between the two.
func MissingAcks ¶
MissingAcks returns the breaking changes that require acknowledgement and were not acknowledged.
func ParseBreaking ¶
ParseBreaking extracts the breaking changes one release body declares. The explicit metadata comment wins; without it, bullets under a "Breaking changes" heading count and always require acknowledgement.
type Deps ¶
type Deps struct {
BinaryPath string
WorkDir string
Asset string
Journal *Journal
Log *slog.Logger
Now func() time.Time
Fetch func(ctx context.Context, tag, asset, dir string) (Fetched, error)
VerifySig func(ctx context.Context, checksumsPath, bundlePath string) (string, error)
Probe func(ctx context.Context, binary string) (ProbeInfo, error)
DBSchema func(ctx context.Context) (int, error)
Backup func(ctx context.Context) (BackupInfo, error)
MigrateTest func(ctx context.Context, binary, snapshotPath string) (int, error)
Stop func(ctx context.Context) error
Start func(ctx context.Context) error
WaitHealthy func(ctx context.Context, timeout time.Duration) error
// RestorePoint keeps the snapshot, binary and unit file as a restore point
// and returns its id. Optional.
RestorePoint func(ctx context.Context, a Attempt, snapshotPath string) (string, error)
// AgentsOnline counts connected agents before the stop, for Verify. Optional.
AgentsOnline func(ctx context.Context) (int, error)
// Verify judges the new release inside the health window. When nil,
// WaitHealthy alone decides.
Verify func(ctx context.Context, window time.Duration, want VerifyExpect) (string, error)
// RestoreUnit puts a saved unit file back and reloads the service manager.
RestoreUnit func(ctx context.Context, savedPath string) error
// RestoreDB puts the snapshot in place of the live database and returns
// where the replaced database was kept.
RestoreDB func(ctx context.Context, snapshotPath string) (string, error)
WriteMarker func(ctx context.Context, a Attempt) error
ClearMarker func()
Audit func(ctx context.Context, detail string) error
}
Deps are the host operations Apply drives. Tests fake them; the host command wires real ones (HTTPS, cosign, systemd, sqlite, filesystem).
type Fetched ¶
Fetched is a downloaded release: the binary, its checksums file and the signature bundle (empty when the release publishes none).
type Fetcher ¶
type Fetcher struct {
// BaseURL is the release download prefix with the tag as its last path
// segment appended by Fetch, for example https://host/org/repo/releases/download.
BaseURL string
Client *http.Client
// AllowHTTP permits a plain-http BaseURL, for tests against a local server.
AllowHTTP bool
MaxBytes int64
// RedirectHosts, when set, limits redirects to these hosts and their
// subdomains, so a release asset cannot bounce the server to an internal address.
RedirectHosts []string
}
Fetcher downloads release assets over HTTPS.
type GuardInput ¶
type GuardInput struct {
BinaryVersion string
BinarySchema int
DBSchema int
// DBVersion is the release that last ran on the database, "" if unknown.
DBVersion string
// Program is the executable's name, used in the recovery commands.
Program string
// InstallerCommand is the verified installer invocation prefix, for
// example "curl -fsSL <url> | sudo".
InstallerCommand string
}
GuardInput describes a binary that found a database newer than itself.
type Journal ¶
type Journal struct {
Dir string
}
Journal persists attempts as one JSON file each. It is the source of truth while an attempt runs, because the database may be swapped underneath it.
func NewJournal ¶
NewJournal returns the journal under dataDir.
func (*Journal) List ¶
List returns every readable attempt, oldest first. Unreadable files are skipped: a half-written journal must not block an upgrade.
type LogLine ¶
type LogLine struct {
At time.Time `json:"at"`
Step string `json:"step,omitempty"`
Line string `json:"line"`
}
LogLine is one event in an attempt's live log.
type ReleaseNotes ¶
ReleaseNotes is one release's tag and notes body.
type Request ¶
type Request struct {
ID string
CurrentVersion string
TargetVersion string
Initiator string
Breaking []Breaking
Acked []string
HealthTimeout time.Duration
}
Request is one upgrade the operator asked for.
type RestorePoint ¶
type RestorePoint struct {
ID string `json:"id"`
Version string `json:"version"`
Schema int `json:"schema"`
AttemptID string `json:"attempt_id,omitempty"`
CreatedAt time.Time `json:"created_at"`
SizeBytes int64 `json:"size_bytes"`
HasBinary bool `json:"has_binary"`
HasUnit bool `json:"has_unit"`
}
RestorePoint is a consistent copy of the database, the binary and the unit file taken just before an upgrade.
type RestorePointStore ¶
type RestorePointStore interface {
ReplaceUpgradeRestorePoints(ctx context.Context, pts []store.UpgradeRestorePoint) error
}
RestorePointStore is the persistence SyncRestorePoints needs; *store.DB satisfies it.
type RestorePoints ¶
type RestorePoints struct{ Dir string }
RestorePoints keeps restore points as one directory each, so they stay readable while the database itself is being swapped.
func NewRestorePoints ¶
func NewRestorePoints(dataDir string) *RestorePoints
NewRestorePoints returns the restore points under dataDir.
func (*RestorePoints) BinaryPath ¶
func (s *RestorePoints) BinaryPath(id string) string
BinaryPath locates a point's binary copy.
func (*RestorePoints) Create ¶
func (s *RestorePoints) Create(meta RestorePoint, snapshotPath, binaryPath, unitPath string) (RestorePoint, error)
Create copies the database snapshot and, when the paths are set and exist, the binary and unit file into a new point described by meta. Files are 0600.
func (*RestorePoints) DBPath ¶
func (s *RestorePoints) DBPath(id string) string
DBPath locates a point's database copy.
func (*RestorePoints) Get ¶
func (s *RestorePoints) Get(id string) (RestorePoint, error)
Get returns the point with id.
func (*RestorePoints) List ¶
func (s *RestorePoints) List() ([]RestorePoint, error)
List returns every readable point, newest first.
func (*RestorePoints) Prune ¶
func (s *RestorePoints) Prune(maxCount int, maxBytes int64) ([]string, error)
Prune applies PrunePlan to the stored points and returns the removed ids.
func (*RestorePoints) Remove ¶
func (s *RestorePoints) Remove(id string)
Remove deletes a point's directory.
func (*RestorePoints) UnitPath ¶
func (s *RestorePoints) UnitPath(id string) string
UnitPath locates a point's unit file copy.
type SchemaResult ¶
type SchemaResult struct {
Version string
Schema int
// Detail is the signature verification note.
Detail string
}
SchemaResult is what a verified release manifest says about the release.
func FetchReleaseSchema ¶
func FetchReleaseSchema(ctx context.Context, f Fetcher, s Signer, tag, parent string) (SchemaResult, error)
FetchReleaseSchema downloads the release manifest of tag with checksums.txt and its signature, verifies the manifest against the signed checksums, and returns the highest migration version the release supports. It never runs the release binary. Size, scheme and redirect limits come from f.
type Signer ¶
type Signer struct {
Mode string
CosignPath string
IdentityRegexp string
Issuer string
// Pinned is set once this host has verified a release signature: from
// then on a missing bundle or cosign is refused unless Mode is off.
Pinned bool
// OnVerified runs after a successful verification, to write the pin.
OnVerified func()
// TUFCacheDir keeps Sigstore trust metadata between runs; empty uses the default.
TUFCacheDir string
// Run executes an external cosign; when it and CosignPath are both unset
// the signature is verified in process.
Run func(ctx context.Context, name string, args ...string) ([]byte, error)
}
Signer verifies a cosign keyless bundle over a checksums file.
type StepFailure ¶
type StepFailure struct {
Cause string `json:"cause"`
// Primary is the one button to lead with: cleanup, retry, rollback or none.
Primary string `json:"primary"`
// FixID is the fixit fix behind a cleanup primary.
FixID string `json:"fix_id,omitempty"`
// Retryable and Rollbackable say which buttons make sense.
Retryable bool `json:"retryable"`
Rollbackable bool `json:"rollbackable"`
// CLI is the host-side equivalent, shown collapsed.
CLI string `json:"cli,omitempty"`
}
StepFailure is the typed reading of one failed upgrade step.
func ClassifyFailure ¶
func ClassifyFailure(step, reason string) StepFailure
ClassifyFailure maps a failed step and its reason to a typed cause and fix. Message matches win over step defaults, so a full disk is cleanup anywhere.
type StepRecord ¶
type StepRecord struct {
Name string `json:"name"`
Status string `json:"status"`
Detail string `json:"detail,omitempty"`
At time.Time `json:"at"`
DurationMS int64 `json:"duration_ms"`
}
StepRecord is one step's result in an attempt's timeline.
type TrustRoot ¶
type TrustRoot struct {
// contains filtered or unexported fields
}
TrustRoot tracks whether the Sigstore trust root can be loaded, refreshing it in the background so callers never wait on the network.
func NewTrustRoot ¶
NewTrustRoot returns a tracker backed by the loader verification uses.
type VerifyExpect ¶
type VerifyExpect struct {
// Schema is the database schema head of the new release.
Schema int
// AgentsOnline is how many agents were connected before the upgrade.
AgentsOnline int
}
VerifyExpect is what a healthy new release must show.