upgrade

package
v1.16.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 16, 2026 License: MIT Imports: 18 Imported by: 0

Documentation

Overview

Package upgrade implements zero-downtime restarts for the wick daemon: a new process starts, inherits the listening socket from the old one, and the old process keeps running until every in-flight agent turn has finished. Nothing is killed, no request is refused, and no inbound Slack event is dropped in between.

Shape of a handoff (parent = old binary, child = new binary):

  1. Operator drops a new binary in place and sends SIGHUP (directly, via `systemctl reload`, or via the tray).
  2. Parent forks the child, handing over the listener file descriptor.
  3. Child boots, restores its registry, starts serving HTTP on the SAME socket, then reports Ready.
  4. Parent stops serving HTTP (the child now answers every new request) and begins DRAINING: it keeps its channel listeners and its agent pool alive so running turns finish and their replies still land in Slack.
  5. When the pool is idle the parent releases the intake baton and exits. The child, which has been parked waiting for that baton, starts its own channel listeners, cron and schedule runner.

The baton in steps 4–5 is why inbound events are never lost: intake stays with the process that owns the running sessions until it is completely done, instead of being handed over in a window where neither process is listening. See IntakeBaton.

Platform + deployment support:

  • Linux / macOS / BSD, run as a plain daemon or under systemd: full graceful upgrade. Under systemd the child announces itself with sd_notify MAINPID (needs Type=notify + NotifyAccess=all) so the unit follows the new process instead of treating the parent's exit as a crash.
  • Windows, or wick running under the tray's in-process supervisor (`support-tools status` / processctl managed mode): descriptor passing is not available, so the upgrader degrades to a plain listener and the old stop/start path is used. Everything here stays callable — the methods simply report "not supported", so no caller needs a build tag.

Index

Constants

This section is empty.

Variables

View Source
var BeforeSpawn func()

BeforeSpawn runs in this process immediately before a successor is forked. It is where state that only lives in memory is handed across — the scoped MCP tokens the running agents authenticate with, which the successor would otherwise reject with a 401 the moment it owns the socket.

A package var rather than a constructor argument, for the same reason as SpawnWrapper: the upgrader is built deep inside server startup, while what it needs to persist is owned elsewhere.

View Source
var Default = &Tracker{}

Default is the process-wide tracker. Registration is global on purpose: a subsystem should be able to opt in from wherever it is constructed, without every constructor growing a parameter it just forwards.

View Source
var ErrUnsupported = errors.New("graceful upgrade not supported here")

ErrUnsupported is returned by Upgrade when this build or this platform cannot hand a listener to a new process.

View Source
var SpawnWrapper func(func() error) error

SpawnWrapper wraps the fork of a successor process. wick sets it to logfiles.WithOriginalStdio so the child inherits the real stdio instead of this process's log pipes — inheriting a pipe whose reader dies with the parent kills the successor with SIGPIPE moments after a clean handoff.

A package var rather than a constructor argument: the upgrader is built deep inside server startup, while the log plumbing is set up in main.

Functions

func Armed

func Armed() bool

Armed reports what MarkArmed recorded: true when a reload hands over, false when it would be a stop/start.

func Busy

func Busy() []string

Busy / Wait / Snapshot on the default tracker.

func BusyHuman

func BusyHuman() []string

func BusyKind

func BusyKind(resumable bool) []string

func ClearDrainState

func ClearDrainState(dir string)

ClearDrainState removes the record. Called as the process exits, so a successor does not report a drain that finished.

func ClearForce

func ClearForce()

ClearForce resets the flag. Used by tests, and after a handover attempt that never started, so a stale force cannot ambush the next upgrade.

func DrainQuiet

func DrainQuiet() time.Duration

DrainQuiet is the SETTLE WINDOW the whole drain switches on. It is not a deadline on anything: no work is ever interrupted because it expired.

The condition it completes is "this process has stopped", not "enough time has passed" — every registered subsystem at zero, and still at zero this long afterwards. The window exists because work going quiet for an instant is not work ending: a tool result, a queued message, the next node of a workflow or a sub-agent reporting back lands moments later, and handing over inside that gap cuts off work that had merely paused.

It applies to agent turns, workflow runs, cron jobs, connector and plugin calls, delegations and scheduled deliveries alike — one rule, so a subsystem added later is covered by registering itself and nothing else.

WICK_DRAIN_QUIET overrides it (any Go duration); WICK_DRAIN_AGENT_QUIET is still read for continuity with the agent-only window it replaces.

func DrainTimeout

func DrainTimeout() time.Duration

DrainTimeout is an OPTIONAL hard ceiling on the whole drain, for an unattended deploy that must finish inside a known time. Unset (0) by default, because a ceiling here does not stop work — it kills it: a workflow run cut off mid-node, a cron job cut off mid-write, an agent reply cut off mid-sentence. The drain otherwise waits for the work.

WICK_DRAIN_TIMEOUT sets it (any Go duration).

func Draining

func Draining() bool

Draining reports whether this process is on its way out.

func ForceDrain

func ForceDrain()

ForceDrain tells the next (or current) drain to stop waiting. Set it BEFORE triggering the handover: once a process is draining it no longer answers HTTP, so nothing can reach it to change its mind.

func Forced

func Forced() bool

Forced reports whether a human asked for the wait to be cut short.

func HandingOver

func HandingOver() bool

HandingOver reports whether this process is currently starting a successor. It is what lets a page say "a reload is happening" while it happens, rather than only noticing afterwards that the version it was drawn with is gone.

func HandoverStatePath

func HandoverStatePath(baseDir string) string

HandoverStatePath is where the last attempt is recorded. Exported so the CLI reads the same path the daemon writes.

func HandoverTimeout

func HandoverTimeout() time.Duration

HandoverTimeout is how long a process waits for its successor to report ready before giving up on it and continuing to serve. 0 means no limit.

func Inherited

func Inherited() bool

Inherited reports whether this process took over from a predecessor.

func LastHandover

func LastHandover() (at time.Time, ok bool, errMsg string)

LastHandover returns the outcome of the last successor this process tried to start: zero time when it has never tried.

func LoadBootStats

func LoadBootStats(dir string)

LoadBootStats reads the recorded boot duration. Call once at startup.

func MarkArmed

func MarkArmed(v bool)

MarkArmed records whether graceful upgrade is available in this process.

func MarkDraining

func MarkDraining()

MarkDraining records that this process has been superseded.

func MarkServing

func MarkServing(fromHandover bool)

MarkServing records that this process now answers on the socket, and whether it got there by taking over from a predecessor.

func Notify

func Notify(state string) error

Notify sends one sd_notify datagram to systemd. No-op (nil) when the process was not started by systemd with NotifyAccess, so callers can fire and forget on every platform.

Written by hand rather than pulled from a dependency: the protocol is a newline-separated KEY=VALUE blob on a unix datagram socket, and wick only ever sends four of them.

func NotifyReady

func NotifyReady() error

NotifyReady tells systemd this process is serving AND that it is now the unit's main process. The MAINPID line is what makes a tableflip upgrade survivable under systemd: after the handoff the unit must follow the CHILD, otherwise the parent's exit looks like the service dying and systemd kills the cgroup — taking the freshly-started process with it.

Requires NotifyAccess=all in the unit, because the notification comes from a process systemd did not itself fork.

func NotifyStatus

func NotifyStatus(msg string) error

NotifyStatus sets the one-line status shown by `systemctl status`.

func NotifyStopping

func NotifyStopping() error

NotifyStopping declares that the whole SERVICE is going down. Only correct from the process systemd currently tracks as MAINPID, and never on the graceful-upgrade path: a draining predecessor is not the service, and this there makes systemd deactivate the unit and kill the successor.

func PredecessorHolds

func PredecessorHolds(dir, sessionID string) bool

PredecessorHolds reports whether a still-living, still-draining previous process is running sessionID's turn.

A process never answers for itself here: its own pool is the authority on its own agents, and the record it publishes is for the OTHER generation.

func PublishDrainState

func PublishDrainState(dir string, pid int, since time.Time, outstanding []string, sessions ...string)

PublishDrainState writes what this process is still waiting for.

func PublishDrainStateAt

func PublishDrainStateAt(dir string, pid int, since time.Time, outstanding []string, at time.Time, sessions ...string)

PublishDrainStateAt is PublishDrainState with an explicit timestamp, so a test can write a record that is already stale.

func RecordBootComplete

func RecordBootComplete(dir string)

RecordBootComplete stores how long THIS process took to become ready, so the next handover can say "usually about this long" instead of leaving the operator guessing whether a minute of silence is normal.

func Register

func Register(name string, inFlight func() int)

Register adds a subsystem to the default tracker.

func RegisterDetailed

func RegisterDetailed(name string, inFlight func() int, detail func() []string)

RegisterDetailed adds a subsystem that can also name its outstanding work.

func RegisterResumable

func RegisterResumable(name string, inFlight func() int, detail func() []string)

RegisterResumable adds a subsystem whose work survives interruption (see Work.Resumable) — it is waited for only briefly.

func ServingSince

func ServingSince() time.Time

ServingSince reports when this process started serving (zero if unknown).

func SetAutoSwap

func SetAutoSwap(s AutoSwap)

SetAutoSwap publishes the watcher's current state. Entering a DIFFERENT state restarts its clock; repeating the same one does not, so the elapsed time survives both the 15s tick and a browser refresh.

func SetTrigger

func SetTrigger(fn func() error)

SetTrigger registers how a handover is started in this process.

func Snapshot

func Snapshot() map[string]int

func Total

func Total() int

func Trigger

func Trigger() error

Trigger starts a successor, exactly as SIGHUP would. ErrUnsupported when graceful upgrade is not available here.

The outcome is recorded like a SIGHUP's, so a click or the binary watcher that failed leaves "last attempt failed: …" on the page instead of nothing. A second request racing one already underway is not an attempt of its own and must not overwrite the real result.

func TypicalBootSeconds

func TypicalBootSeconds() int

TypicalBootSeconds is how long the previous boot took here — an estimate grounded in this host's own history rather than a guess. 0 when unknown.

func Wait

func Wait(ctx context.Context) []string

func WaitKind

func WaitKind(ctx context.Context, resumable bool) []string

func WaitSettled

func WaitSettled(ctx context.Context, quiet time.Duration) []string

func Wanted

func Wanted() bool

Wanted reports whether the operator asked for graceful upgrades.

Opt-in (WICK_GRACEFUL_UPGRADE=1) rather than default-on: it changes how the process exits, and a deployment that never sends SIGHUP gains nothing from it. Set it in the unit / launchd plist once the handoff has been verified on that host.

Types

type AutoSwap

type AutoSwap struct {
	// State: "" (nothing waiting), "settling" (a new file was just seen and
	// has to hold still before it is trusted), "waiting" (stable, but work is
	// in flight), "previous_draining" (an earlier generation has not exited,
	// so only two processes may exist at once), "handing_over" (the successor
	// has been started), "blocked" (a build is installed that must NOT be
	// applied — Note says why).
	State string `json:"state"`
	To    string `json:"to,omitempty"`
	// Note explains a state the operator would otherwise have to guess at.
	// Carried for "blocked", where a silent refusal is indistinguishable from
	// a watcher that simply stopped working.
	Note string `json:"note,omitempty"`
	// Want is the version that WOULD be accepted, so the note ends in an
	// instruction rather than a complaint.
	Want string `json:"want,omitempty"`
	// NextCheckIn is seconds until the next check, so a countdown can run.
	NextCheckIn int `json:"next_check_in,omitempty"`
	// ElapsedSeconds is how long this state has been in force. Counted on the
	// SERVER, because a page refresh must not restart the clock on a handover
	// that has been underway for a minute.
	ElapsedSeconds int `json:"elapsed_seconds,omitempty"`
	// GivesUpInSeconds is how long the predecessor waits for a successor
	// before giving up and carrying on — the honest answer to "until when?".
	GivesUpInSeconds int `json:"gives_up_in_seconds,omitempty"`
	// contains filtered or unexported fields
}

AutoSwap is what the binary watcher is doing right now, published so the UI can show it. Without this the watcher is invisible: a build sits on disk, nothing appears to happen for up to two check intervals, and an operator reasonably concludes it is stuck.

func AutoSwapStatus

func AutoSwapStatus() AutoSwap

AutoSwapStatus reports the watcher's last published state, with the elapsed and remaining seconds filled in as of now.

type DrainState

type DrainState struct {
	PID         int       `json:"pid"`
	Since       time.Time `json:"since"`
	Outstanding []string  `json:"outstanding"`
	// Sessions lists the sessions with an agent turn still in flight here.
	//
	// The delegation table is shared by both generations, so the successor's
	// sweep has to know which sub-agents are alive in a process whose pool it
	// cannot see — otherwise it reads them as exited and closes their runs
	// with an empty result. Outstanding carries the same ids inside prose;
	// this is the list a reader can rely on without parsing it. Absent on a
	// record written by an older binary.
	Sessions []string `json:"sessions,omitempty"`
	// UpdatedAt lets a reader ignore a record left behind by a process that
	// died without cleaning up.
	UpdatedAt time.Time `json:"updated_at"`
}

DrainState is what a draining process publishes about itself.

func ReadDrainState

func ReadDrainState(dir string) (DrainState, bool)

ReadDrainState returns the record left by a still-living predecessor.

Two things make a record stale: the process is gone, or nothing has refreshed it recently. Both mean the drain is over and the file is litter — reporting it would leave the UI blaming a process that no longer exists.

func (DrainState) HoldsSession

func (st DrainState) HoldsSession(sessionID string) bool

HoldsSession reports whether the draining process may still be running an agent turn for sessionID.

Errs towards yes. The caller is deciding whether to close a run as abandoned, and closing a live one throws its work away, while leaving a dead one open costs a sweep or two until the record goes stale. So a record from an older binary, which has no Sessions list, counts any session it names — and, when it reports agent turns without naming this one, still counts it: the ids there may be shortened, and "cannot tell" is not "gone".

type IntakeBaton

type IntakeBaton struct {
	// contains filtered or unexported fields
}

IntakeBaton is the right to accept NEW work: channel listeners (Slack socket mode, Telegram polling), the workflow cron/router, and the scheduled-message runner. Exactly one process may hold it.

It exists because a graceful upgrade means two wick processes are briefly alive, and those subsystems are singletons in a way HTTP is not. Two Slack socket-mode connections split inbound events between the old and the new binary at random; two cron loops fire every job twice; two schedule runners deliver every scheduled message twice. None of that shows up in a test that only checks the port stayed open — it shows up in production as duplicate work, which is worse than a short restart.

The lock is an flock, so the kernel releases it even if the holder is SIGKILLed: a crashed process never wedges the baton.

func (*IntakeBaton) Release

func (b *IntakeBaton) Release()

Release hands the baton to whoever is waiting. Idempotent.

type Tracker

type Tracker struct {
	// contains filtered or unexported fields
}

Tracker holds the registered subsystems. Use the package-level Register / Wait unless you are testing.

func (*Tracker) Add

func (t *Tracker) Add(w Work)

Add registers one subsystem. Nil InFlight is ignored rather than panicking: a registration site is often optional wiring, and a nil there must not take the daemon down at boot.

func (*Tracker) Busy

func (t *Tracker) Busy() []string

Busy renders the non-zero subsystems as "name=n" strings, sorted, with each one's Detail appended when it has any. Empty slice means fully drained.

func (*Tracker) BusyHuman

func (t *Tracker) BusyHuman() []string

BusyHuman renders the outstanding work the way a person would say it: counts and kinds, no identifiers. Busy() stays as it is for logs, where a session id is exactly what you want at 3am; a status panel is not that.

Detail is kept only where it is a NAME someone recognises — a cron job, a workflow — never a session or request id, which wrap across three lines and tell a reader nothing they can act on.

func (*Tracker) BusyKind

func (t *Tracker) BusyKind(resumable bool) []string

BusyKind renders only the non-zero subsystems whose Resumable flag matches.

func (*Tracker) Snapshot

func (t *Tracker) Snapshot() map[string]int

Snapshot reports the current in-flight count per subsystem, including zeros, so a caller can render a full picture.

func (*Tracker) Total

func (t *Tracker) Total() int

Total is the sum of every subsystem's in-flight count.

func (*Tracker) Wait

func (t *Tracker) Wait(ctx context.Context) []string

Wait blocks until every registered subsystem reports zero, or ctx is done. It logs what is still outstanding every 15s so a long drain is legible rather than a silent hang — the operator can see it is finishing a debug run, not stuck.

Returns the still-busy descriptions: empty means a clean drain.

func (*Tracker) WaitKind

func (t *Tracker) WaitKind(ctx context.Context, resumable bool) []string

WaitKind waits only for the subsystems whose Resumable flag matches, so a caller can give resumable and non-resumable work different deadlines.

func (*Tracker) WaitSettled

func (t *Tracker) WaitSettled(ctx context.Context, quiet time.Duration) []string

WaitSettled blocks until EVERY registered subsystem has reported zero continuously for quiet, or ctx is done. It is the wait a handover uses.

Two properties, and both matter:

  • No deadline of its own. Whatever is running — an agent turn, a workflow run mid-node, a cron job mid-write — is waited for until it is done, however long that takes. A caller that genuinely must bound the wait passes a ctx with a deadline and accepts that the work is cut off.
  • The window is a SETTLE, not a grace. Work finishing rarely means work ending: a tool result, a queued message, the next node of a workflow or a sub-agent reporting back lands moments later. Going quiet for a moment is not the same as being done, so the counter has to stay at zero for quiet before this returns, and any new work restarts it.

Returns the still-busy descriptions: empty means everything really settled.

type Upgrader

type Upgrader struct {
	// contains filtered or unexported fields
}

Upgrader owns the listener and the handoff. A nil-safe zero value behaves exactly like the "unsupported" case, so callers can hold one unconditionally.

func New

func New(baseDir string) (*Upgrader, string)

New builds an Upgrader. It never fails the caller: when graceful upgrade is unavailable — unsupported platform, not requested, or descriptor passing refused — it returns a working Upgrader that simply listens normally, plus the reason, so the daemon boots either way.

func (*Upgrader) AcquireIntake

func (u *Upgrader) AcquireIntake(ctx context.Context) (*IntakeBaton, error)

AcquireIntake blocks until this process owns the intake baton (see IntakeBaton) or ctx is done. It is safe on every platform: where flock is unavailable the baton is a no-op and returns immediately.

func (*Upgrader) Enabled

func (u *Upgrader) Enabled() bool

Enabled reports whether this process can actually hand off.

func (*Upgrader) Exit

func (u *Upgrader) Exit() <-chan struct{}

Exit closes when a successor has taken over and this process should drain and go away. Never closes when graceful upgrade is off.

func (*Upgrader) IsChild

func (u *Upgrader) IsChild() bool

IsChild reports whether this process was started BY an upgrade — i.e. it inherited its listener from a parent that is still draining.

func (*Upgrader) Listen

func (u *Upgrader) Listen(network, addr string) (net.Listener, error)

Listen returns the listening socket: inherited from the parent during an upgrade, freshly bound otherwise.

func (*Upgrader) NotifyServing

func (u *Upgrader) NotifyServing()

NotifyServing tells the SERVICE MANAGER that this process is now the one answering on the socket. Call it the moment the serve loop starts.

Split from Ready on purpose. systemd's Type=notify has a start timeout, and wick's boot gate can legitimately take minutes to restore a big registry — waiting for that before the first READY=1 would get a healthy daemon killed as a failed start. The MAINPID in here is also what makes an upgrade survivable: after a handoff the unit must follow the child, or the parent's exit looks like the service dying and systemd kills the whole cgroup.

No-op when NOTIFY_SOCKET is unset, i.e. everywhere except systemd.

func (*Upgrader) Ready

func (u *Upgrader) Ready()

Ready announces that this process is fully warmed up. It closes the PARENT's Exit channel, which is what starts the drain over there — so call it only once this process can genuinely do the work, not merely accept connections.

func (*Upgrader) Stop

func (u *Upgrader) Stop()

Stop releases the upgrader's own resources. Safe to defer unconditionally.

func (*Upgrader) Upgrade

func (u *Upgrader) Upgrade() error

Upgrade starts a successor process. Returns ErrUnsupported when this build cannot.

func (*Upgrader) WatchSignal

func (u *Upgrader) WatchSignal(ctx context.Context)

WatchSignal starts a successor whenever SIGHUP arrives — the conventional "reload" signal, so `systemctl reload`, `kill -HUP`, and `support-tools upgrade` all work through one path. Returns immediately; the watcher lives until ctx is done.

SIGHUP is ignored (with a log line) when graceful upgrade is off, which is strictly better than the Go default of terminating the process.

type Work

type Work struct {
	// Name is what the operator sees in the drain log. Short, lowercase.
	Name string
	// InFlight returns how many units of work are running right now.
	InFlight func() int
	// Detail optionally identifies the outstanding units (session ids, run
	// ids). Called only while the subsystem is non-zero, and capped when
	// logged, so it may be moderately expensive.
	Detail func() []string
	// Resumable marks work that SURVIVES being interrupted because its state
	// is on disk and it picks up where it left off — an agent turn is resumed
	// with --resume on the next message. Everything else (a workflow run
	// mid-node, a cron job mid-write, a connector call) loses work when it is
	// cut off.
	//
	// The drain no longer treats the two differently: it waits for all of it
	// (see WaitSettled), because "resumable" still means somebody watches a
	// reply stop mid-sentence. The flag survives for reporting — BusyKind
	// separates them so a forced swap can say which work merely resumes and
	// which is actually lost.
	Resumable bool
}

A wick process runs a lot of work in the background: agent turns in the pool, workflow runs, cron jobs, scheduled messages, sub-agent delegations. Draining means waiting for all of it — and the only way that stays true as features are added is if every new subsystem declares itself here instead of the drain path hard-coding a list it will silently fall behind.

Contract for anything that owns background work:

upgrade.Register("workflow runs", eng.ActiveRuns)

InFlight must be cheap (read a counter under a mutex) and must return 0 once the subsystem has nothing outstanding. Optionally add Detail to name WHAT is still busy, which is what turns "1 still running" into a log line an operator can act on.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL