Help us shape the future of Go! Take the 2026 Go Developer Survey

supervisor

package
v1.0.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 28, 2026 License: Apache-2.0 Imports: 27 Imported by: 0

Documentation

Overview

Package supervisor runs the hatamoto server as a child process, picks which binary it runs from $HATAMOTO_DATA/releases, health-gates every switch and rolls back on its own. It has no database access and no network client other than loopback calls to its own child.

Index

Constants

View Source
const (
	RunBootstrap = "bootstrap"
	RunRelease   = "release"
)

Names of what can run.

View Source
const (
	PhaseSuccess        = "success"
	PhaseRolledBack     = "rolled_back"
	PhaseFailed         = "failed"
	PhaseManualRollback = "manual_rollback"
	// PhaseStagedOffline is an apply made while no supervisor ran: nothing has
	// been gated yet. Only the gate at the next start records success.
	PhaseStagedOffline = "staged-offline"
	// PhaseRolledBackOffline is a rollback made while no supervisor ran, from
	// a state that is not a release (nothing to bad-list). The next start's
	// gate logs how the target came up.
	PhaseRolledBackOffline = "rolled-back-offline"
)

Event phases.

Variables

This section is empty.

Functions

func AppendEvent

func AppendEvent(path string, e Event) error

AppendEvent writes e as one JSON line with O_APPEND and fsync. An empty ID is filled in.

func BaselineOf

func BaselineOf(r SmokeResult) map[string]int

BaselineOf is the record of a good run: check name to HTTP status. Skipped checks (status 0) are left out.

func BootstrapVersionOf

func BootstrapVersionOf(st *State, fallback string) string

BootstrapVersionOf is the bootstrap binary's version: the stamp the supervisor recorded in st, else fallback (the caller's own stamp, right only when the caller is the bootstrap binary).

func CheckOwner

func CheckOwner(l Layout) error

CheckOwner refuses to run the CLI as a uid other than the owner of releases/ (or of the data dir when releases/ does not exist yet). Running as someone else (typically root) leaves root-owned state files the service user cannot read and opens symlink races inside a directory the service user controls.

func ChildEnv

func ChildEnv(base []string, data, smokeAddr, token string) []string

ChildEnv is base with HATAMOTO_DATA and the smoke settings replaced by the given values; any inherited HATAMOTO_SMOKE_* is dropped. The token travels only through the environment, never argv.

func FreeLoopbackAddr

func FreeLoopbackAddr() (string, error)

FreeLoopbackAddr returns a 127.0.0.1 address whose port was free a moment ago.

func NewEventID

func NewEventID() string

NewEventID is "<unixnano>-<8 hex>".

func NewToken

func NewToken() (string, error)

NewToken returns a fresh 32-byte hex token.

func Run

func Run(ctx context.Context, c Config) error

Run supervises the child until ctx is cancelled or SIGTERM/SIGINT arrives, then stops it and returns nil. SIGHUP means "look at state.json" and is never forwarded to the child.

func Running

func Running(c Config) bool

Running reports whether a supervisor holds the pid lock under c.Data.

func RunningVersion

func RunningVersion(l Layout, st *State, bootstrapVersion string) string

RunningVersion is the version the child runs now: the current release when the supervisor runs one, else the bootstrap's own stamp. Everything that reports or decides on "the running version" uses it, so the CLI and serve agree with the supervisor.

func SaveState

func SaveState(path string, s *State) error

SaveState writes state.json atomically: temp file in the same directory, fsync, rename, fsync the directory. Mode 0600.

func StoredRecord

func StoredRecord(c Config, version string) (*release.Record, error)

StoredRecord returns the verified record of a staged release.

func VerifyStored

func VerifyStored(l Layout, version string, o release.Options) error

VerifyStored re-checks a staged release before it is run: the stored record's signature against the build's trust, the record's version against the directory name, and the binary's sha256 against the record. The binary is opened without following symlinks and read once, with a size bound. Anything that can write the data volume could otherwise plant a binary here.

func WithLock

func WithLock(path string, fn func() error) error

WithLock runs fn holding an exclusive flock on the file at path (created if needed). The CLI and the supervisor both take it around state.json read-modify-write.

Types

type ApplyOpts

type ApplyOpts struct {
	File    string // the binary
	Record  string // release record (signed bytes)
	Sig     string // "<key_id>:<base64>" file; default Record + ".sig"
	Release release.Options
	// Platform overrides runtime.GOOS-runtime.GOARCH (tests).
	Platform string
}

ApplyOpts names the files `hatamoto update apply` verifies and stages.

type BadEntry

type BadEntry struct {
	Version string    `json:"version"`
	Reason  string    `json:"reason"`
	At      time.Time `json:"when"`
}

BadEntry is a version that was rolled back and is never retried.

type Child

type Child struct {
	// contains filtered or unexported fields
}

Child is one running `hatamoto serve` process.

func StartChild

func StartChild(path string, env []string) (*Child, error)

StartChild runs `<path> serve` with env. Stdout and stderr are inherited.

Pdeathsig fires when the OS thread that forked the child exits, so the goroutine that calls Start also waits for the child and stays pinned to its thread until the child is dead.

func (*Child) Done

func (c *Child) Done() <-chan struct{}

Done is closed once the child has exited and been reaped.

func (*Child) Err

func (c *Child) Err() error

Err is the child's exit status error (nil for exit 0); valid after Done.

func (*Child) Pid

func (c *Child) Pid() int

Pid is the child's process id.

func (*Child) Stop

func (c *Child) Stop(grace time.Duration) (killed bool)

Stop sends SIGTERM and waits up to grace for the child to exit, then SIGKILLs its process group. It reports whether the kill was needed and returns only once the child is gone.

type Choice

type Choice struct {
	Bootstrap bool
	Name      string
	Reason    string
}

Choice is the binary to run. Name is "bootstrap" or a version.

func Choose

func Choose(in ChooseInput, st *State) Choice

Choose decides which binary to run and updates st (Run, CurrentSeen, BootstrapSHA256) to match. Rules, in order:

  1. releases dir unusable: bootstrap, st untouched.
  2. bootstrap file changed since last seen (or first run): bootstrap wins.
  3. current missing or invalid: bootstrap.
  4. bootstrap won earlier and current is unchanged: stay on bootstrap; a changed current (apply, or hand-flipped per RUNBOOK §8) is honoured.
  5. a semver-newer bootstrap beats current; non-semver never counts as newer.
  6. otherwise current.

type ChooseInput

type ChooseInput struct {
	WritableErr      error  // non-nil: releases dir not usable
	BootstrapSHA     string // sha256 of the bootstrap binary on disk now
	BootstrapVersion string
	Current          string // what releases/current points at ("" if none)
	CurrentOK        bool   // Current names a valid executable release
}

ChooseInput is everything Choose needs; the caller gathers it from disk so the decision stays a pure function.

type Config

type Config struct {
	Data             string // absolute data dir; releases live in Data/releases
	Bootstrap        string // path of the binary the supervisor itself runs from
	BootstrapVersion string // its version stamp (may be a non-semver build string)
	ReadyURL         string // the child's /ready, e.g. http://127.0.0.1:8080/ready

	StopGrace     time.Duration // SIGTERM to SIGKILL; must exceed the child's own shutdown (default 40s)
	ReadyTimeout  time.Duration // default 120s
	ObserveWindow time.Duration // default 60s
	SmokePoll     time.Duration // default 5s
	BootWindow    time.Duration // a child that lives this long is not crash-looping (default 120s)
	BackoffBase   time.Duration // restart delay base, doubled per crash up to 30s (default 500ms)

	// Release is the trust used to re-verify staged releases before they run
	// (zero value: the keys built into this binary).
	Release release.Options

	Force bool // CLI: let `update rollback` target a bad-listed release

	Env []string // base environment for the child (default os.Environ())
	Log *slog.Logger
	// contains filtered or unexported fields
}

Config is what Run needs. Timings are shrinkable for tests; zero values take the defaults.

type CorruptStateError

type CorruptStateError struct{ Err error }

CorruptStateError means state.json was read but is not valid JSON, as opposed to an I/O or permission error. Only this kind may be set aside.

func (*CorruptStateError) Error

func (e *CorruptStateError) Error() string

func (*CorruptStateError) Unwrap

func (e *CorruptStateError) Unwrap() error

type Event

type Event struct {
	ID         string    `json:"id"`
	From       string    `json:"from"`
	To         string    `json:"to"`
	Phase      string    `json:"phase"`
	Error      string    `json:"error"`
	StartedAt  time.Time `json:"started_at"`
	FinishedAt time.Time `json:"finished_at"`
}

Event is one line of releases/events.jsonl. The child imports unseen lines into hatamoto_updates on boot.

type GateConfig

type GateConfig struct {
	ReadyURL  string // child's /ready
	SmokeAddr string // child's loopback smoke listener, host:port
	Token     string

	Started       time.Time // child start; the observation window runs from here (default: now)
	ReadyTimeout  time.Duration
	ReadyPoll     time.Duration // default 500ms
	ObserveWindow time.Duration
	SmokePoll     time.Duration

	// KnownPluginErrors are the plugins that already failed in the last good
	// run; only errors outside this list fail the gate.
	KnownPluginErrors []string

	// Baseline is the per-check status of the last good run (see
	// BaselineOf). Nil means there is no record yet: the gate then only
	// demands no 5xx anywhere and "GET /" at 200 or 404.
	Baseline map[string]int

	// StaleBaseline marks a Baseline older than baselineMaxAge that could not
	// be refreshed (an offline apply has no running child to probe). Editors
	// can unpublish the home page or the sampled page in a day, so a 404 on
	// "GET /" or "published page" is not held against the release. Every other
	// check stays strict, as do boot faults, plugin errors and render errors.
	StaleBaseline bool

	// RenderBaseline is FirstRender of the last good run. Render errors fail
	// the gate only when a smoke call adds more than this.
	RenderBaseline int

	// BootFaultBaseline is the boot fault identifiers of the last good run.
	// Any fault outside it fails the gate: the release broke something at
	// boot that the last one did not.
	BootFaultBaseline []string

	// Initial marks the first smoke of a child that was not switched to: it
	// only decides whether the result is fit to become the baseline, so
	// render errors and plugin errors do not fail it.
	Initial bool
}

GateConfig drives RunGate. Timings are test-shrinkable.

type Layout

type Layout struct{ Data, Releases string }

Layout is the on-disk release directory:

releases/v1.4.2/hatamoto  release.json  release.json.sig
releases/current -> v1.4.2     releases/previous -> v1.4.1  (relative)
releases/state.json  state.lock  events.jsonl  supervisor.pid

func NewLayout

func NewLayout(data string) Layout

NewLayout builds the layout for an absolute data directory.

func (Layout) Binary

func (l Layout) Binary(version string) string

Binary is the path of a version's executable.

func (Layout) EventsPath

func (l Layout) EventsPath() string
func (l Layout) Link(name string) string

Link returns the version a symlink ("current" or "previous") points at, or "" when it is missing or not a bare version name.

func (Layout) LockPath

func (l Layout) LockPath() string

func (Layout) PIDPath

func (l Layout) PIDPath() string
func (l Layout) RemoveLink(name string) error

RemoveLink deletes releases/<name> (a symlink); a missing link is fine.

func (Layout) StatePath

func (l Layout) StatePath() string

StatePath, LockPath, EventsPath and PIDPath name the bookkeeping files.

func (l Layout) SwapSymlink(name, target string) error

SwapSymlink atomically points releases/<name> at the bare directory name target: create <name>.new, rename over <name>, fsync the directory. This is the same move RUNBOOK §8 does by hand.

func (Layout) Valid

func (l Layout) Valid(version string) bool

Valid reports whether version names a release directory holding a regular executable hatamoto file.

func (Layout) Writable

func (l Layout) Writable() error

Writable creates the releases directory if needed and proves it accepts files. The error says why not.

type LinkSnapshot

type LinkSnapshot struct {
	Current             string `json:"current"`
	Previous            string `json:"previous"`
	PreviousIsBootstrap bool   `json:"previous_is_bootstrap"`
	Run                 string `json:"run"`
	CurrentSeen         string `json:"current_seen"`
}

LinkSnapshot is the release links and run choice as they were before a switch.

type Pending

type Pending struct {
	Kind       string `json:"kind"` // apply | rollback
	Version    string `json:"version"`
	From       string `json:"from"`
	PrevBefore string `json:"prev_before"`
	// Before is taken by the first pass of the switch and is what a rollback
	// restores; a replay after a restart reuses it.
	Before      *LinkSnapshot `json:"before,omitempty"`
	RequestedAt time.Time     `json:"requested_at"`
	StartedAt   time.Time     `json:"started_at,omitempty"`
}

Pending is an apply or rollback that has no outcome yet.

type Result

type Result struct {
	Version  string
	Notified bool // the running supervisor was asked to do the gated switch
	Message  string
}

Result says what an Apply or Rollback did; Message is for the operator.

func Apply

func Apply(c Config, o ApplyOpts) (Result, error)

Apply verifies the artifact against its signed record, stages it under releases/<version> and asks the supervisor (or, when none is running, switches the links itself) to run it. It fetches nothing.

func Rollback

func Rollback(c Config) (Result, error)

Rollback switches current back to previous (or the bootstrap when that was what ran before), health-gated when a supervisor is running. It refuses a previous release that is on the bad list unless c.Force is set.

func Stage

func Stage(c Config, o ApplyOpts) (Result, error)

Stage verifies and stages the artifact exactly as Apply does, but never records a pending switch and never moves a link: the release is only made ready under releases/<version>.

type SmokeCheck

type SmokeCheck struct {
	Name   string `json:"name"`
	OK     bool   `json:"ok"`
	Status int    `json:"status"` // HTTP status of the check; 0 when it was skipped
	Detail string `json:"detail"`
}

SmokeCheck is one in-process request the child made against itself.

type SmokeResult

type SmokeResult struct {
	Pass         bool         `json:"pass"`
	Checks       []SmokeCheck `json:"checks"`
	RenderErrors int          `json:"render_errors"` // caused by this pass's own requests
	PluginErrors []string     `json:"plugin_errors"`
	// BootFaults are release-level render faults the child logged outside any
	// request: its theme fell back to the default, a rebuild failed, plugin
	// templates were rejected.
	BootFaults []string `json:"boot_faults"`

	// FirstRender is RenderErrors at the first smoke call of a gated child
	// (set by RunGate, not by the child): what one boot plus one smoke pass
	// costs, the figure later runs are compared with.
	FirstRender int `json:"-"`
}

SmokeResult is the JSON the child's smoke endpoint returns.

func RunGate

func RunGate(ctx context.Context, g GateConfig, crashed <-chan struct{}) (SmokeResult, error)

RunGate decides whether a freshly started child is healthy: /ready answers 200 within ReadyTimeout, one smoke call passes, and the smoke endpoint keeps passing until ObserveWindow has elapsed since the child started. crashed is closed when the child exits. The error text is the human reason for failing. The last smoke result is returned either way when one was received.

type State

type State struct {
	V                   int        `json:"v"`
	Phase               string     `json:"phase"` // idle | probation
	Run                 string     `json:"run"`   // bootstrap | release
	CurrentSeen         string     `json:"current_seen"`
	PreviousIsBootstrap bool       `json:"previous_is_bootstrap"`
	Attempts            int        `json:"attempts"`
	LastError           string     `json:"last_error"`
	Bad                 []BadEntry `json:"bad"`
	BootstrapSHA256     string     `json:"bootstrap_sha256"`
	// BootstrapVersion is the version stamp of the supervisor's own binary,
	// recorded by the supervisor so a child that runs a release (whose own
	// stamp is the release's) can still tell what a rollback would return to.
	BootstrapVersion string   `json:"bootstrap_version,omitempty"`
	SmokeToken       string   `json:"smoke_token"`
	PluginErrors     []string `json:"plugin_errors"`
	// SmokeBaseline is the per-check HTTP status of the last good child (one
	// that passed its gate, or was running without a switch). Nil: no record.
	SmokeBaseline map[string]int `json:"smoke_baseline,omitempty"`
	// BaselineAt is when SmokeBaseline was recorded.
	BaselineAt time.Time `json:"baseline_at,omitempty"`
	// RenderBaseline is the render errors the last good run logged by its
	// first smoke call; a later gate fails only on an increase.
	RenderBaseline int `json:"render_baseline,omitempty"`
	// BootFaultBaseline is the boot fault identifiers (theme fallback and the
	// like) the last good run reported; a later gate fails on any other.
	BootFaultBaseline []string `json:"known_boot_faults,omitempty"`
	Pending           *Pending `json:"pending,omitempty"`
	// Unproven is a release switched to that has not yet both passed its
	// gate and lived a full boot window. Only an unproven release's crash loop
	// may roll back; any other crash loop just restarts with backoff.
	Unproven string `json:"unproven,omitempty"`
	// OfflineRollback is the target of a rollback made with no supervisor
	// running. The startup gate logs its outcome, then clears this.
	OfflineRollback string `json:"offline_rollback,omitempty"`
	// Ungated is a release applied while no supervisor was running. The
	// supervisor runs the normal health gate on it at its next start, using
	// Before to roll back if it fails.
	Ungated *Pending `json:"ungated,omitempty"`
}

State is releases/state.json, shared by the supervisor and the CLI under WithLock.

func LoadState

func LoadState(path string) (*State, error)

LoadState reads state.json; a missing file is a fresh state.

func (*State) AddBad

func (s *State) AddBad(version, reason string)

AddBad records version once.

func (*State) IsBad

func (s *State) IsBad(version string) bool

IsBad reports whether version is on the bad list.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL