Documentation
¶
Overview ¶
Package upgrade implements zero-downtime restarts for the wick daemon: a new process starts, inherits the listening socket from the old one, and the old process keeps running until every in-flight agent turn has finished. Nothing is killed, no request is refused, and no inbound Slack event is dropped in between.
Shape of a handoff (parent = old binary, child = new binary):
- Operator drops a new binary in place and sends SIGHUP (directly, via `systemctl reload`, or via the tray).
- Parent forks the child, handing over the listener file descriptor.
- Child boots, restores its registry, starts serving HTTP on the SAME socket, then reports Ready.
- Parent stops serving HTTP (the child now answers every new request) and begins DRAINING: it keeps its channel listeners and its agent pool alive so running turns finish and their replies still land in Slack.
- When the pool is idle the parent releases the intake baton and exits. The child, which has been parked waiting for that baton, starts its own channel listeners, cron and schedule runner.
The baton in steps 4–5 is why inbound events are never lost: intake stays with the process that owns the running sessions until it is completely done, instead of being handed over in a window where neither process is listening. See IntakeBaton.
Platform + deployment support:
- Linux / macOS / BSD, run as a plain daemon or under systemd: full graceful upgrade. Under systemd the child announces itself with sd_notify MAINPID (needs Type=notify + NotifyAccess=all) so the unit follows the new process instead of treating the parent's exit as a crash.
- Windows, or wick running under the tray's in-process supervisor (`support-tools status` / processctl managed mode): descriptor passing is not available, so the upgrader degrades to a plain listener and the old stop/start path is used. Everything here stays callable — the methods simply report "not supported", so no caller needs a build tag.
Index ¶
- Variables
- func Armed() bool
- func Busy() []string
- func BusyHuman() []string
- func BusyKind(resumable bool) []string
- func ClearDrainState(dir string)
- func ClearForce()
- func DrainQuiet() time.Duration
- func DrainTimeout() time.Duration
- func Draining() bool
- func ForceDrain()
- func Forced() bool
- func HandingOver() bool
- func HandoverStatePath(baseDir string) string
- func HandoverTimeout() time.Duration
- func Inherited() bool
- func LastHandover() (at time.Time, ok bool, errMsg string)
- func LoadBootStats(dir string)
- func MarkArmed(v bool)
- func MarkDraining()
- func MarkServing(fromHandover bool)
- func Notify(state string) error
- func NotifyReady() error
- func NotifyStatus(msg string) error
- func NotifyStopping() error
- func PublishDrainState(dir string, pid int, since time.Time, outstanding []string)
- func PublishDrainStateAt(dir string, pid int, since time.Time, outstanding []string, at time.Time)
- func RecordBootComplete(dir string)
- func Register(name string, inFlight func() int)
- func RegisterDetailed(name string, inFlight func() int, detail func() []string)
- func RegisterResumable(name string, inFlight func() int, detail func() []string)
- func ServingSince() time.Time
- func SetAutoSwap(s AutoSwap)
- func SetTrigger(fn func() error)
- func Snapshot() map[string]int
- func Total() int
- func Trigger() error
- func TypicalBootSeconds() int
- func Wait(ctx context.Context) []string
- func WaitKind(ctx context.Context, resumable bool) []string
- func WaitSettled(ctx context.Context, quiet time.Duration) []string
- func Wanted() bool
- type AutoSwap
- type DrainState
- type IntakeBaton
- type Tracker
- func (t *Tracker) Add(w Work)
- func (t *Tracker) Busy() []string
- func (t *Tracker) BusyHuman() []string
- func (t *Tracker) BusyKind(resumable bool) []string
- func (t *Tracker) Snapshot() map[string]int
- func (t *Tracker) Total() int
- func (t *Tracker) Wait(ctx context.Context) []string
- func (t *Tracker) WaitKind(ctx context.Context, resumable bool) []string
- func (t *Tracker) WaitSettled(ctx context.Context, quiet time.Duration) []string
- type Upgrader
- func (u *Upgrader) AcquireIntake(ctx context.Context) (*IntakeBaton, error)
- func (u *Upgrader) Enabled() bool
- func (u *Upgrader) Exit() <-chan struct{}
- func (u *Upgrader) IsChild() bool
- func (u *Upgrader) Listen(network, addr string) (net.Listener, error)
- func (u *Upgrader) NotifyServing()
- func (u *Upgrader) Ready()
- func (u *Upgrader) Stop()
- func (u *Upgrader) Upgrade() error
- func (u *Upgrader) WatchSignal(ctx context.Context)
- type Work
Constants ¶
This section is empty.
Variables ¶
var BeforeSpawn func()
BeforeSpawn runs in this process immediately before a successor is forked. It is where state that only lives in memory is handed across — the scoped MCP tokens the running agents authenticate with, which the successor would otherwise reject with a 401 the moment it owns the socket.
A package var rather than a constructor argument, for the same reason as SpawnWrapper: the upgrader is built deep inside server startup, while what it needs to persist is owned elsewhere.
var Default = &Tracker{}
Default is the process-wide tracker. Registration is global on purpose: a subsystem should be able to opt in from wherever it is constructed, without every constructor growing a parameter it just forwards.
var ErrUnsupported = errors.New("graceful upgrade not supported here")
ErrUnsupported is returned by Upgrade when this build or this platform cannot hand a listener to a new process.
var SpawnWrapper func(func() error) error
SpawnWrapper wraps the fork of a successor process. wick sets it to logfiles.WithOriginalStdio so the child inherits the real stdio instead of this process's log pipes — inheriting a pipe whose reader dies with the parent kills the successor with SIGPIPE moments after a clean handoff.
A package var rather than a constructor argument: the upgrader is built deep inside server startup, while the log plumbing is set up in main.
Functions ¶
func Armed ¶
func Armed() bool
Armed reports what MarkArmed recorded: true when a reload hands over, false when it would be a stop/start.
func ClearDrainState ¶
func ClearDrainState(dir string)
ClearDrainState removes the record. Called as the process exits, so a successor does not report a drain that finished.
func ClearForce ¶
func ClearForce()
ClearForce resets the flag. Used by tests, and after a handover attempt that never started, so a stale force cannot ambush the next upgrade.
func DrainQuiet ¶
DrainQuiet is the SETTLE WINDOW the whole drain switches on. It is not a deadline on anything: no work is ever interrupted because it expired.
The condition it completes is "this process has stopped", not "enough time has passed" — every registered subsystem at zero, and still at zero this long afterwards. The window exists because work going quiet for an instant is not work ending: a tool result, a queued message, the next node of a workflow or a sub-agent reporting back lands moments later, and handing over inside that gap cuts off work that had merely paused.
It applies to agent turns, workflow runs, cron jobs, connector and plugin calls, delegations and scheduled deliveries alike — one rule, so a subsystem added later is covered by registering itself and nothing else.
WICK_DRAIN_QUIET overrides it (any Go duration); WICK_DRAIN_AGENT_QUIET is still read for continuity with the agent-only window it replaces.
func DrainTimeout ¶
DrainTimeout is an OPTIONAL hard ceiling on the whole drain, for an unattended deploy that must finish inside a known time. Unset (0) by default, because a ceiling here does not stop work — it kills it: a workflow run cut off mid-node, a cron job cut off mid-write, an agent reply cut off mid-sentence. The drain otherwise waits for the work.
WICK_DRAIN_TIMEOUT sets it (any Go duration).
func ForceDrain ¶
func ForceDrain()
ForceDrain tells the next (or current) drain to stop waiting. Set it BEFORE triggering the handover: once a process is draining it no longer answers HTTP, so nothing can reach it to change its mind.
func HandingOver ¶
func HandingOver() bool
HandingOver reports whether this process is currently starting a successor. It is what lets a page say "a reload is happening" while it happens, rather than only noticing afterwards that the version it was drawn with is gone.
func HandoverStatePath ¶
HandoverStatePath is where the last attempt is recorded. Exported so the CLI reads the same path the daemon writes.
func HandoverTimeout ¶
HandoverTimeout is how long a process waits for its successor to report ready before giving up on it and continuing to serve.
func Inherited ¶
func Inherited() bool
Inherited reports whether this process took over from a predecessor.
func LastHandover ¶
LastHandover returns the outcome of the last successor this process tried to start: zero time when it has never tried.
func LoadBootStats ¶
func LoadBootStats(dir string)
LoadBootStats reads the recorded boot duration. Call once at startup.
func MarkArmed ¶
func MarkArmed(v bool)
MarkArmed records whether graceful upgrade is available in this process.
func MarkServing ¶
func MarkServing(fromHandover bool)
MarkServing records that this process now answers on the socket, and whether it got there by taking over from a predecessor.
func Notify ¶
Notify sends one sd_notify datagram to systemd. No-op (nil) when the process was not started by systemd with NotifyAccess, so callers can fire and forget on every platform.
Written by hand rather than pulled from a dependency: the protocol is a newline-separated KEY=VALUE blob on a unix datagram socket, and wick only ever sends four of them.
func NotifyReady ¶
func NotifyReady() error
NotifyReady tells systemd this process is serving AND that it is now the unit's main process. The MAINPID line is what makes a tableflip upgrade survivable under systemd: after the handoff the unit must follow the CHILD, otherwise the parent's exit looks like the service dying and systemd kills the cgroup — taking the freshly-started process with it.
Requires NotifyAccess=all in the unit, because the notification comes from a process systemd did not itself fork.
func NotifyStatus ¶
NotifyStatus sets the one-line status shown by `systemctl status`.
func NotifyStopping ¶
func NotifyStopping() error
NotifyStopping declares that the whole SERVICE is going down. Only correct from the process systemd currently tracks as MAINPID, and never on the graceful-upgrade path: a draining predecessor is not the service, and this there makes systemd deactivate the unit and kill the successor.
func PublishDrainState ¶
PublishDrainState writes what this process is still waiting for.
func PublishDrainStateAt ¶
PublishDrainStateAt is PublishDrainState with an explicit timestamp, so a test can write a record that is already stale.
func RecordBootComplete ¶
func RecordBootComplete(dir string)
RecordBootComplete stores how long THIS process took to become ready, so the next handover can say "usually about this long" instead of leaving the operator guessing whether a minute of silence is normal.
func RegisterDetailed ¶
RegisterDetailed adds a subsystem that can also name its outstanding work.
func RegisterResumable ¶
RegisterResumable adds a subsystem whose work survives interruption (see Work.Resumable) — it is waited for only briefly.
func ServingSince ¶
ServingSince reports when this process started serving (zero if unknown).
func SetAutoSwap ¶
func SetAutoSwap(s AutoSwap)
SetAutoSwap publishes the watcher's current state. Entering a DIFFERENT state restarts its clock; repeating the same one does not, so the elapsed time survives both the 15s tick and a browser refresh.
func SetTrigger ¶
func SetTrigger(fn func() error)
SetTrigger registers how a handover is started in this process.
func Trigger ¶
func Trigger() error
Trigger starts a successor, exactly as SIGHUP would. ErrUnsupported when graceful upgrade is not available here.
func TypicalBootSeconds ¶
func TypicalBootSeconds() int
TypicalBootSeconds is how long the previous boot took here — an estimate grounded in this host's own history rather than a guess. 0 when unknown.
func Wanted ¶
func Wanted() bool
Wanted reports whether the operator asked for graceful upgrades.
Opt-in (WICK_GRACEFUL_UPGRADE=1) rather than default-on: it changes how the process exits, and a deployment that never sends SIGHUP gains nothing from it. Set it in the unit / launchd plist once the handoff has been verified on that host.
Types ¶
type AutoSwap ¶
type AutoSwap struct {
// State: "" (nothing waiting), "settling" (a new file was just seen and
// has to hold still before it is trusted), "waiting" (stable, but work is
// in flight), "previous_draining" (an earlier generation has not exited,
// so only two processes may exist at once), "handing_over" (the successor
// has been started).
State string `json:"state"`
To string `json:"to,omitempty"`
// NextCheckIn is seconds until the next check, so a countdown can run.
NextCheckIn int `json:"next_check_in,omitempty"`
// ElapsedSeconds is how long this state has been in force. Counted on the
// SERVER, because a page refresh must not restart the clock on a handover
// that has been underway for a minute.
ElapsedSeconds int `json:"elapsed_seconds,omitempty"`
// GivesUpInSeconds is how long the predecessor waits for a successor
// before giving up and carrying on — the honest answer to "until when?".
GivesUpInSeconds int `json:"gives_up_in_seconds,omitempty"`
// contains filtered or unexported fields
}
AutoSwap is what the binary watcher is doing right now, published so the UI can show it. Without this the watcher is invisible: a build sits on disk, nothing appears to happen for up to two check intervals, and an operator reasonably concludes it is stuck.
func AutoSwapStatus ¶
func AutoSwapStatus() AutoSwap
AutoSwapStatus reports the watcher's last published state, with the elapsed and remaining seconds filled in as of now.
type DrainState ¶
type DrainState struct {
PID int `json:"pid"`
Since time.Time `json:"since"`
Outstanding []string `json:"outstanding"`
// UpdatedAt lets a reader ignore a record left behind by a process that
// died without cleaning up.
UpdatedAt time.Time `json:"updated_at"`
}
DrainState is what a draining process publishes about itself.
func ReadDrainState ¶
func ReadDrainState(dir string) (DrainState, bool)
ReadDrainState returns the record left by a still-living predecessor.
Two things make a record stale: the process is gone, or nothing has refreshed it recently. Both mean the drain is over and the file is litter — reporting it would leave the UI blaming a process that no longer exists.
type IntakeBaton ¶
type IntakeBaton struct {
// contains filtered or unexported fields
}
IntakeBaton is the right to accept NEW work: channel listeners (Slack socket mode, Telegram polling), the workflow cron/router, and the scheduled-message runner. Exactly one process may hold it.
It exists because a graceful upgrade means two wick processes are briefly alive, and those subsystems are singletons in a way HTTP is not. Two Slack socket-mode connections split inbound events between the old and the new binary at random; two cron loops fire every job twice; two schedule runners deliver every scheduled message twice. None of that shows up in a test that only checks the port stayed open — it shows up in production as duplicate work, which is worse than a short restart.
The lock is an flock, so the kernel releases it even if the holder is SIGKILLed: a crashed process never wedges the baton.
func (*IntakeBaton) Release ¶
func (b *IntakeBaton) Release()
Release hands the baton to whoever is waiting. Idempotent.
type Tracker ¶
type Tracker struct {
// contains filtered or unexported fields
}
Tracker holds the registered subsystems. Use the package-level Register / Wait unless you are testing.
func (*Tracker) Add ¶
Add registers one subsystem. Nil InFlight is ignored rather than panicking: a registration site is often optional wiring, and a nil there must not take the daemon down at boot.
func (*Tracker) Busy ¶
Busy renders the non-zero subsystems as "name=n" strings, sorted, with each one's Detail appended when it has any. Empty slice means fully drained.
func (*Tracker) BusyHuman ¶
BusyHuman renders the outstanding work the way a person would say it: counts and kinds, no identifiers. Busy() stays as it is for logs, where a session id is exactly what you want at 3am; a status panel is not that.
Detail is kept only where it is a NAME someone recognises — a cron job, a workflow — never a session or request id, which wrap across three lines and tell a reader nothing they can act on.
func (*Tracker) BusyKind ¶
BusyKind renders only the non-zero subsystems whose Resumable flag matches.
func (*Tracker) Snapshot ¶
Snapshot reports the current in-flight count per subsystem, including zeros, so a caller can render a full picture.
func (*Tracker) Wait ¶
Wait blocks until every registered subsystem reports zero, or ctx is done. It logs what is still outstanding every 15s so a long drain is legible rather than a silent hang — the operator can see it is finishing a debug run, not stuck.
Returns the still-busy descriptions: empty means a clean drain.
func (*Tracker) WaitKind ¶
WaitKind waits only for the subsystems whose Resumable flag matches, so a caller can give resumable and non-resumable work different deadlines.
func (*Tracker) WaitSettled ¶
WaitSettled blocks until EVERY registered subsystem has reported zero continuously for quiet, or ctx is done. It is the wait a handover uses.
Two properties, and both matter:
- No deadline of its own. Whatever is running — an agent turn, a workflow run mid-node, a cron job mid-write — is waited for until it is done, however long that takes. A caller that genuinely must bound the wait passes a ctx with a deadline and accepts that the work is cut off.
- The window is a SETTLE, not a grace. Work finishing rarely means work ending: a tool result, a queued message, the next node of a workflow or a sub-agent reporting back lands moments later. Going quiet for a moment is not the same as being done, so the counter has to stay at zero for quiet before this returns, and any new work restarts it.
Returns the still-busy descriptions: empty means everything really settled.
type Upgrader ¶
type Upgrader struct {
// contains filtered or unexported fields
}
Upgrader owns the listener and the handoff. A nil-safe zero value behaves exactly like the "unsupported" case, so callers can hold one unconditionally.
func New ¶
New builds an Upgrader. It never fails the caller: when graceful upgrade is unavailable — unsupported platform, not requested, or descriptor passing refused — it returns a working Upgrader that simply listens normally, plus the reason, so the daemon boots either way.
func (*Upgrader) AcquireIntake ¶
func (u *Upgrader) AcquireIntake(ctx context.Context) (*IntakeBaton, error)
AcquireIntake blocks until this process owns the intake baton (see IntakeBaton) or ctx is done. It is safe on every platform: where flock is unavailable the baton is a no-op and returns immediately.
func (*Upgrader) Exit ¶
func (u *Upgrader) Exit() <-chan struct{}
Exit closes when a successor has taken over and this process should drain and go away. Never closes when graceful upgrade is off.
func (*Upgrader) IsChild ¶
IsChild reports whether this process was started BY an upgrade — i.e. it inherited its listener from a parent that is still draining.
func (*Upgrader) Listen ¶
Listen returns the listening socket: inherited from the parent during an upgrade, freshly bound otherwise.
func (*Upgrader) NotifyServing ¶
func (u *Upgrader) NotifyServing()
NotifyServing tells the SERVICE MANAGER that this process is now the one answering on the socket. Call it the moment the serve loop starts.
Split from Ready on purpose. systemd's Type=notify has a start timeout, and wick's boot gate can legitimately take minutes to restore a big registry — waiting for that before the first READY=1 would get a healthy daemon killed as a failed start. The MAINPID in here is also what makes an upgrade survivable: after a handoff the unit must follow the child, or the parent's exit looks like the service dying and systemd kills the whole cgroup.
No-op when NOTIFY_SOCKET is unset, i.e. everywhere except systemd.
func (*Upgrader) Ready ¶
func (u *Upgrader) Ready()
Ready announces that this process is fully warmed up. It closes the PARENT's Exit channel, which is what starts the drain over there — so call it only once this process can genuinely do the work, not merely accept connections.
func (*Upgrader) Stop ¶
func (u *Upgrader) Stop()
Stop releases the upgrader's own resources. Safe to defer unconditionally.
func (*Upgrader) Upgrade ¶
Upgrade starts a successor process. Returns ErrUnsupported when this build cannot.
func (*Upgrader) WatchSignal ¶
WatchSignal starts a successor whenever SIGHUP arrives — the conventional "reload" signal, so `systemctl reload`, `kill -HUP`, and `support-tools upgrade` all work through one path. Returns immediately; the watcher lives until ctx is done.
SIGHUP is ignored (with a log line) when graceful upgrade is off, which is strictly better than the Go default of terminating the process.
type Work ¶
type Work struct {
// Name is what the operator sees in the drain log. Short, lowercase.
Name string
// InFlight returns how many units of work are running right now.
InFlight func() int
// Detail optionally identifies the outstanding units (session ids, run
// ids). Called only while the subsystem is non-zero, and capped when
// logged, so it may be moderately expensive.
Detail func() []string
// Resumable marks work that SURVIVES being interrupted because its state
// is on disk and it picks up where it left off — an agent turn is resumed
// with --resume on the next message. Everything else (a workflow run
// mid-node, a cron job mid-write, a connector call) loses work when it is
// cut off.
//
// The drain no longer treats the two differently: it waits for all of it
// (see WaitSettled), because "resumable" still means somebody watches a
// reply stop mid-sentence. The flag survives for reporting — BusyKind
// separates them so a forced swap can say which work merely resumes and
// which is actually lost.
Resumable bool
}
A wick process runs a lot of work in the background: agent turns in the pool, workflow runs, cron jobs, scheduled messages, sub-agent delegations. Draining means waiting for all of it — and the only way that stays true as features are added is if every new subsystem declares itself here instead of the drain path hard-coding a list it will silently fall behind.
Contract for anything that owns background work:
upgrade.Register("workflow runs", eng.ActiveRuns)
InFlight must be cheap (read a counter under a mutex) and must return 0 once the subsystem has nothing outstanding. Optionally add Detail to name WHAT is still busy, which is what turns "1 still running" into a log line an operator can act on.