Documentation
¶
Overview ¶
Package supervisor runs the hatamoto server as a child process, picks which binary it runs from $HATAMOTO_DATA/releases, health-gates every switch and rolls back on its own. It has no database access and no network client other than loopback calls to its own child.
Index ¶
- Constants
- func AppendEvent(path string, e Event) error
- func BaselineOf(r SmokeResult) map[string]int
- func BootstrapVersionOf(st *State, fallback string) string
- func CheckOwner(l Layout) error
- func ChildEnv(base []string, data, smokeAddr, token string) []string
- func FreeLoopbackAddr() (string, error)
- func NewEventID() string
- func NewToken() (string, error)
- func Run(ctx context.Context, c Config) error
- func Running(c Config) bool
- func RunningVersion(l Layout, st *State, bootstrapVersion string) string
- func SaveState(path string, s *State) error
- func StoredRecord(c Config, version string) (*release.Record, error)
- func VerifyStored(l Layout, version string, o release.Options) error
- func WithLock(path string, fn func() error) error
- type ApplyOpts
- type BadEntry
- type Child
- type Choice
- type ChooseInput
- type Config
- type CorruptStateError
- type Event
- type GateConfig
- type Layout
- func (l Layout) Binary(version string) string
- func (l Layout) EventsPath() string
- func (l Layout) Link(name string) string
- func (l Layout) LockPath() string
- func (l Layout) PIDPath() string
- func (l Layout) RemoveLink(name string) error
- func (l Layout) StatePath() string
- func (l Layout) SwapSymlink(name, target string) error
- func (l Layout) Valid(version string) bool
- func (l Layout) Writable() error
- type LinkSnapshot
- type Pending
- type Result
- type SmokeCheck
- type SmokeResult
- type State
Constants ¶
const ( RunBootstrap = "bootstrap" RunRelease = "release" )
Names of what can run.
const ( PhaseSuccess = "success" PhaseRolledBack = "rolled_back" PhaseFailed = "failed" PhaseManualRollback = "manual_rollback" // PhaseStagedOffline is an apply made while no supervisor ran: nothing has // been gated yet. Only the gate at the next start records success. PhaseStagedOffline = "staged-offline" // PhaseRolledBackOffline is a rollback made while no supervisor ran, from // a state that is not a release (nothing to bad-list). The next start's // gate logs how the target came up. PhaseRolledBackOffline = "rolled-back-offline" )
Event phases.
Variables ¶
This section is empty.
Functions ¶
func AppendEvent ¶
AppendEvent writes e as one JSON line with O_APPEND and fsync. An empty ID is filled in.
func BaselineOf ¶
func BaselineOf(r SmokeResult) map[string]int
BaselineOf is the record of a good run: check name to HTTP status. Skipped checks (status 0) are left out.
func BootstrapVersionOf ¶
BootstrapVersionOf is the bootstrap binary's version: the stamp the supervisor recorded in st, else fallback (the caller's own stamp, right only when the caller is the bootstrap binary).
func CheckOwner ¶
CheckOwner refuses to run the CLI as a uid other than the owner of releases/ (or of the data dir when releases/ does not exist yet). Running as someone else (typically root) leaves root-owned state files the service user cannot read and opens symlink races inside a directory the service user controls.
func ChildEnv ¶
ChildEnv is base with HATAMOTO_DATA and the smoke settings replaced by the given values; any inherited HATAMOTO_SMOKE_* is dropped. The token travels only through the environment, never argv.
func FreeLoopbackAddr ¶
FreeLoopbackAddr returns a 127.0.0.1 address whose port was free a moment ago.
func Run ¶
Run supervises the child until ctx is cancelled or SIGTERM/SIGINT arrives, then stops it and returns nil. SIGHUP means "look at state.json" and is never forwarded to the child.
func RunningVersion ¶
RunningVersion is the version the child runs now: the current release when the supervisor runs one, else the bootstrap's own stamp. Everything that reports or decides on "the running version" uses it, so the CLI and serve agree with the supervisor.
func SaveState ¶
SaveState writes state.json atomically: temp file in the same directory, fsync, rename, fsync the directory. Mode 0600.
func StoredRecord ¶
StoredRecord returns the verified record of a staged release.
func VerifyStored ¶
VerifyStored re-checks a staged release before it is run: the stored record's signature against the build's trust, the record's version against the directory name, and the binary's sha256 against the record. The binary is opened without following symlinks and read once, with a size bound. Anything that can write the data volume could otherwise plant a binary here.
Types ¶
type ApplyOpts ¶
type ApplyOpts struct {
File string // the binary
Record string // release record (signed bytes)
Sig string // "<key_id>:<base64>" file; default Record + ".sig"
Release release.Options
// Platform overrides runtime.GOOS-runtime.GOARCH (tests).
Platform string
}
ApplyOpts names the files `hatamoto update apply` verifies and stages.
type BadEntry ¶
type BadEntry struct {
Version string `json:"version"`
Reason string `json:"reason"`
At time.Time `json:"when"`
}
BadEntry is a version that was rolled back and is never retried.
type Child ¶
type Child struct {
// contains filtered or unexported fields
}
Child is one running `hatamoto serve` process.
func StartChild ¶
StartChild runs `<path> serve` with env. Stdout and stderr are inherited.
Pdeathsig fires when the OS thread that forked the child exits, so the goroutine that calls Start also waits for the child and stays pinned to its thread until the child is dead.
func (*Child) Done ¶
func (c *Child) Done() <-chan struct{}
Done is closed once the child has exited and been reaped.
type Choice ¶
Choice is the binary to run. Name is "bootstrap" or a version.
func Choose ¶
func Choose(in ChooseInput, st *State) Choice
Choose decides which binary to run and updates st (Run, CurrentSeen, BootstrapSHA256) to match. Rules, in order:
- releases dir unusable: bootstrap, st untouched.
- bootstrap file changed since last seen (or first run): bootstrap wins.
- current missing or invalid: bootstrap.
- bootstrap won earlier and current is unchanged: stay on bootstrap; a changed current (apply, or hand-flipped per RUNBOOK §8) is honoured.
- a semver-newer bootstrap beats current; non-semver never counts as newer.
- otherwise current.
type ChooseInput ¶
type ChooseInput struct {
WritableErr error // non-nil: releases dir not usable
BootstrapSHA string // sha256 of the bootstrap binary on disk now
BootstrapVersion string
Current string // what releases/current points at ("" if none)
CurrentOK bool // Current names a valid executable release
}
ChooseInput is everything Choose needs; the caller gathers it from disk so the decision stays a pure function.
type Config ¶
type Config struct {
Data string // absolute data dir; releases live in Data/releases
Bootstrap string // path of the binary the supervisor itself runs from
BootstrapVersion string // its version stamp (may be a non-semver build string)
ReadyURL string // the child's /ready, e.g. http://127.0.0.1:8080/ready
StopGrace time.Duration // SIGTERM to SIGKILL; must exceed the child's own shutdown (default 40s)
ReadyTimeout time.Duration // default 120s
ObserveWindow time.Duration // default 60s
SmokePoll time.Duration // default 5s
BootWindow time.Duration // a child that lives this long is not crash-looping (default 120s)
BackoffBase time.Duration // restart delay base, doubled per crash up to 30s (default 500ms)
// Release is the trust used to re-verify staged releases before they run
// (zero value: the keys built into this binary).
Release release.Options
Force bool // CLI: let `update rollback` target a bad-listed release
Env []string // base environment for the child (default os.Environ())
Log *slog.Logger
// contains filtered or unexported fields
}
Config is what Run needs. Timings are shrinkable for tests; zero values take the defaults.
type CorruptStateError ¶
type CorruptStateError struct{ Err error }
CorruptStateError means state.json was read but is not valid JSON, as opposed to an I/O or permission error. Only this kind may be set aside.
func (*CorruptStateError) Error ¶
func (e *CorruptStateError) Error() string
func (*CorruptStateError) Unwrap ¶
func (e *CorruptStateError) Unwrap() error
type Event ¶
type Event struct {
ID string `json:"id"`
From string `json:"from"`
To string `json:"to"`
Phase string `json:"phase"`
Error string `json:"error"`
StartedAt time.Time `json:"started_at"`
FinishedAt time.Time `json:"finished_at"`
}
Event is one line of releases/events.jsonl. The child imports unseen lines into hatamoto_updates on boot.
type GateConfig ¶
type GateConfig struct {
ReadyURL string // child's /ready
SmokeAddr string // child's loopback smoke listener, host:port
Token string
Started time.Time // child start; the observation window runs from here (default: now)
ReadyTimeout time.Duration
ReadyPoll time.Duration // default 500ms
ObserveWindow time.Duration
SmokePoll time.Duration
// KnownPluginErrors are the plugins that already failed in the last good
// run; only errors outside this list fail the gate.
KnownPluginErrors []string
// Baseline is the per-check status of the last good run (see
// BaselineOf). Nil means there is no record yet: the gate then only
// demands no 5xx anywhere and "GET /" at 200 or 404.
Baseline map[string]int
// StaleBaseline marks a Baseline older than baselineMaxAge that could not
// be refreshed (an offline apply has no running child to probe). Editors
// can unpublish the home page or the sampled page in a day, so a 404 on
// "GET /" or "published page" is not held against the release. Every other
// check stays strict, as do boot faults, plugin errors and render errors.
StaleBaseline bool
// RenderBaseline is FirstRender of the last good run. Render errors fail
// the gate only when a smoke call adds more than this.
RenderBaseline int
// BootFaultBaseline is the boot fault identifiers of the last good run.
// Any fault outside it fails the gate: the release broke something at
// boot that the last one did not.
BootFaultBaseline []string
// Initial marks the first smoke of a child that was not switched to: it
// only decides whether the result is fit to become the baseline, so
// render errors and plugin errors do not fail it.
Initial bool
}
GateConfig drives RunGate. Timings are test-shrinkable.
type Layout ¶
type Layout struct{ Data, Releases string }
Layout is the on-disk release directory:
releases/v1.4.2/hatamoto release.json release.json.sig releases/current -> v1.4.2 releases/previous -> v1.4.1 (relative) releases/state.json state.lock events.jsonl supervisor.pid
func (Layout) EventsPath ¶
func (Layout) Link ¶
Link returns the version a symlink ("current" or "previous") points at, or "" when it is missing or not a bare version name.
func (Layout) RemoveLink ¶
RemoveLink deletes releases/<name> (a symlink); a missing link is fine.
func (Layout) SwapSymlink ¶
SwapSymlink atomically points releases/<name> at the bare directory name target: create <name>.new, rename over <name>, fsync the directory. This is the same move RUNBOOK §8 does by hand.
type LinkSnapshot ¶
type LinkSnapshot struct {
Current string `json:"current"`
Previous string `json:"previous"`
PreviousIsBootstrap bool `json:"previous_is_bootstrap"`
Run string `json:"run"`
CurrentSeen string `json:"current_seen"`
}
LinkSnapshot is the release links and run choice as they were before a switch.
type Pending ¶
type Pending struct {
Kind string `json:"kind"` // apply | rollback
Version string `json:"version"`
From string `json:"from"`
PrevBefore string `json:"prev_before"`
// Before is taken by the first pass of the switch and is what a rollback
// restores; a replay after a restart reuses it.
Before *LinkSnapshot `json:"before,omitempty"`
RequestedAt time.Time `json:"requested_at"`
StartedAt time.Time `json:"started_at,omitempty"`
}
Pending is an apply or rollback that has no outcome yet.
type Result ¶
type Result struct {
Version string
Notified bool // the running supervisor was asked to do the gated switch
Message string
}
Result says what an Apply or Rollback did; Message is for the operator.
func Apply ¶
Apply verifies the artifact against its signed record, stages it under releases/<version> and asks the supervisor (or, when none is running, switches the links itself) to run it. It fetches nothing.
type SmokeCheck ¶
type SmokeCheck struct {
Name string `json:"name"`
OK bool `json:"ok"`
Status int `json:"status"` // HTTP status of the check; 0 when it was skipped
Detail string `json:"detail"`
}
SmokeCheck is one in-process request the child made against itself.
type SmokeResult ¶
type SmokeResult struct {
Pass bool `json:"pass"`
Checks []SmokeCheck `json:"checks"`
RenderErrors int `json:"render_errors"` // caused by this pass's own requests
PluginErrors []string `json:"plugin_errors"`
// BootFaults are release-level render faults the child logged outside any
// request: its theme fell back to the default, a rebuild failed, plugin
// templates were rejected.
BootFaults []string `json:"boot_faults"`
// FirstRender is RenderErrors at the first smoke call of a gated child
// (set by RunGate, not by the child): what one boot plus one smoke pass
// costs, the figure later runs are compared with.
FirstRender int `json:"-"`
}
SmokeResult is the JSON the child's smoke endpoint returns.
func RunGate ¶
func RunGate(ctx context.Context, g GateConfig, crashed <-chan struct{}) (SmokeResult, error)
RunGate decides whether a freshly started child is healthy: /ready answers 200 within ReadyTimeout, one smoke call passes, and the smoke endpoint keeps passing until ObserveWindow has elapsed since the child started. crashed is closed when the child exits. The error text is the human reason for failing. The last smoke result is returned either way when one was received.
type State ¶
type State struct {
V int `json:"v"`
Phase string `json:"phase"` // idle | probation
Run string `json:"run"` // bootstrap | release
CurrentSeen string `json:"current_seen"`
PreviousIsBootstrap bool `json:"previous_is_bootstrap"`
Attempts int `json:"attempts"`
LastError string `json:"last_error"`
Bad []BadEntry `json:"bad"`
BootstrapSHA256 string `json:"bootstrap_sha256"`
// BootstrapVersion is the version stamp of the supervisor's own binary,
// recorded by the supervisor so a child that runs a release (whose own
// stamp is the release's) can still tell what a rollback would return to.
BootstrapVersion string `json:"bootstrap_version,omitempty"`
SmokeToken string `json:"smoke_token"`
PluginErrors []string `json:"plugin_errors"`
// SmokeBaseline is the per-check HTTP status of the last good child (one
// that passed its gate, or was running without a switch). Nil: no record.
SmokeBaseline map[string]int `json:"smoke_baseline,omitempty"`
// BaselineAt is when SmokeBaseline was recorded.
BaselineAt time.Time `json:"baseline_at,omitempty"`
// RenderBaseline is the render errors the last good run logged by its
// first smoke call; a later gate fails only on an increase.
RenderBaseline int `json:"render_baseline,omitempty"`
// BootFaultBaseline is the boot fault identifiers (theme fallback and the
// like) the last good run reported; a later gate fails on any other.
BootFaultBaseline []string `json:"known_boot_faults,omitempty"`
Pending *Pending `json:"pending,omitempty"`
// Unproven is a release switched to that has not yet both passed its
// gate and lived a full boot window. Only an unproven release's crash loop
// may roll back; any other crash loop just restarts with backoff.
Unproven string `json:"unproven,omitempty"`
// OfflineRollback is the target of a rollback made with no supervisor
// running. The startup gate logs its outcome, then clears this.
OfflineRollback string `json:"offline_rollback,omitempty"`
// Ungated is a release applied while no supervisor was running. The
// supervisor runs the normal health gate on it at its next start, using
// Before to roll back if it fails.
Ungated *Pending `json:"ungated,omitempty"`
}
State is releases/state.json, shared by the supervisor and the CLI under WithLock.