Documentation
¶
Overview ¶
Package upgrade implements zero-downtime restarts for the wick daemon: a new process starts, inherits the listening socket from the old one, and the old process keeps running until every in-flight agent turn has finished. Nothing is killed, no request is refused, and no inbound Slack event is dropped in between.
Shape of a handoff (parent = old binary, child = new binary):
- Operator drops a new binary in place and sends SIGHUP (directly, via `systemctl reload`, or via the tray).
- Parent forks the child, handing over the listener file descriptor.
- Child boots, restores its registry, starts serving HTTP on the SAME socket, then reports Ready.
- Parent stops serving HTTP (the child now answers every new request) and begins DRAINING: it keeps its channel listeners and its agent pool alive so running turns finish and their replies still land in Slack.
- When the pool is idle the parent releases the intake baton and exits. The child, which has been parked waiting for that baton, starts its own channel listeners, cron and schedule runner.
The baton in steps 4–5 is why inbound events are never lost: intake stays with the process that owns the running sessions until it is completely done, instead of being handed over in a window where neither process is listening. See IntakeBaton.
Platform + deployment support:
- Linux / macOS / BSD, run as a plain daemon or under systemd: full graceful upgrade. Under systemd the child announces itself with sd_notify MAINPID (needs Type=notify + NotifyAccess=all) so the unit follows the new process instead of treating the parent's exit as a crash.
- Windows, or wick running under the tray's in-process supervisor (`support-tools status` / processctl managed mode): descriptor passing is not available, so the upgrader degrades to a plain listener and the old stop/start path is used. Everything here stays callable — the methods simply report "not supported", so no caller needs a build tag.
Index ¶
- Variables
- func AgentGrace() time.Duration
- func Busy() []string
- func BusyKind(resumable bool) []string
- func DrainTimeout() time.Duration
- func Notify(state string) error
- func NotifyReady() error
- func NotifyStatus(msg string) error
- func NotifyStopping() error
- func Register(name string, inFlight func() int)
- func RegisterDetailed(name string, inFlight func() int, detail func() []string)
- func RegisterResumable(name string, inFlight func() int, detail func() []string)
- func Snapshot() map[string]int
- func Total() int
- func Wait(ctx context.Context) []string
- func WaitKind(ctx context.Context, resumable bool) []string
- func Wanted() bool
- type IntakeBaton
- type Tracker
- func (t *Tracker) Add(w Work)
- func (t *Tracker) Busy() []string
- func (t *Tracker) BusyKind(resumable bool) []string
- func (t *Tracker) Snapshot() map[string]int
- func (t *Tracker) Total() int
- func (t *Tracker) Wait(ctx context.Context) []string
- func (t *Tracker) WaitKind(ctx context.Context, resumable bool) []string
- type Upgrader
- func (u *Upgrader) AcquireIntake(ctx context.Context) (*IntakeBaton, error)
- func (u *Upgrader) Enabled() bool
- func (u *Upgrader) Exit() <-chan struct{}
- func (u *Upgrader) IsChild() bool
- func (u *Upgrader) Listen(network, addr string) (net.Listener, error)
- func (u *Upgrader) NotifyServing()
- func (u *Upgrader) Ready()
- func (u *Upgrader) Stop()
- func (u *Upgrader) Upgrade() error
- func (u *Upgrader) WatchSignal(ctx context.Context)
- type Work
Constants ¶
This section is empty.
Variables ¶
var Default = &Tracker{}
Default is the process-wide tracker. Registration is global on purpose: a subsystem should be able to opt in from wherever it is constructed, without every constructor growing a parameter it just forwards.
var ErrUnsupported = errors.New("graceful upgrade not supported here")
ErrUnsupported is returned by Upgrade when this build or this platform cannot hand a listener to a new process.
var SpawnWrapper func(func() error) error
SpawnWrapper wraps the fork of a successor process. wick sets it to logfiles.WithOriginalStdio so the child inherits the real stdio instead of this process's log pipes — inheriting a pipe whose reader dies with the parent kills the successor with SIGPIPE moments after a clean handoff.
A package var rather than a constructor argument: the upgrader is built deep inside server startup, while the log plumbing is set up in main.
Functions ¶
func AgentGrace ¶ added in v1.9.0
AgentGrace caps how long a draining parent waits for RESUMABLE work — agent turns. Short by default, and that is a deliberate trade:
An interactive Slack session is "in flight" for as long as somebody keeps talking to it, so waiting for it kept the old process alive for hours. Two processes at once is visible and confusing (only one holds intake, so the browser talks to one while the agents run in the other), and it also blocks the NEXT upgrade — tableflip refuses while a parent is still alive.
A turn interrupted here is not lost work: the session lives on disk and the next message resumes it. Set WICK_DRAIN_AGENT_GRACE higher (e.g. 20m) to let long debug runs finish instead, accepting the longer overlap; set 0 to hand over immediately.
func DrainTimeout ¶
DrainTimeout caps how long a draining parent waits for work that CANNOT be resumed — a workflow run mid-node, a cron job mid-write, a connector call. Minutes are normal for those, so the default is generous; WICK_DRAIN_TIMEOUT overrides it (any Go duration).
func Notify ¶
Notify sends one sd_notify datagram to systemd. No-op (nil) when the process was not started by systemd with NotifyAccess, so callers can fire and forget on every platform.
Written by hand rather than pulled from a dependency: the protocol is a newline-separated KEY=VALUE blob on a unix datagram socket, and wick only ever sends four of them.
func NotifyReady ¶
func NotifyReady() error
NotifyReady tells systemd this process is serving AND that it is now the unit's main process. The MAINPID line is what makes a tableflip upgrade survivable under systemd: after the handoff the unit must follow the CHILD, otherwise the parent's exit looks like the service dying and systemd kills the cgroup — taking the freshly-started process with it.
Requires NotifyAccess=all in the unit, because the notification comes from a process systemd did not itself fork.
func NotifyStatus ¶
NotifyStatus sets the one-line status shown by `systemctl status`.
func NotifyStopping ¶
func NotifyStopping() error
NotifyStopping declares that the whole SERVICE is going down. Only correct from the process systemd currently tracks as MAINPID, and never on the graceful-upgrade path: a draining predecessor is not the service, and this there makes systemd deactivate the unit and kill the successor.
func RegisterDetailed ¶
RegisterDetailed adds a subsystem that can also name its outstanding work.
func RegisterResumable ¶
RegisterResumable adds a subsystem whose work survives interruption (see Work.Resumable) — it is waited for only briefly.
func Wanted ¶
func Wanted() bool
Wanted reports whether the operator asked for graceful upgrades.
Opt-in (WICK_GRACEFUL_UPGRADE=1) rather than default-on: it changes how the process exits, and a deployment that never sends SIGHUP gains nothing from it. Set it in the unit / launchd plist once the handoff has been verified on that host.
Types ¶
type IntakeBaton ¶
type IntakeBaton struct {
// contains filtered or unexported fields
}
IntakeBaton is the right to accept NEW work: channel listeners (Slack socket mode, Telegram polling), the workflow cron/router, and the scheduled-message runner. Exactly one process may hold it.
It exists because a graceful upgrade means two wick processes are briefly alive, and those subsystems are singletons in a way HTTP is not. Two Slack socket-mode connections split inbound events between the old and the new binary at random; two cron loops fire every job twice; two schedule runners deliver every scheduled message twice. None of that shows up in a test that only checks the port stayed open — it shows up in production as duplicate work, which is worse than a short restart.
The lock is an flock, so the kernel releases it even if the holder is SIGKILLed: a crashed process never wedges the baton.
func (*IntakeBaton) Release ¶
func (b *IntakeBaton) Release()
Release hands the baton to whoever is waiting. Idempotent.
type Tracker ¶
type Tracker struct {
// contains filtered or unexported fields
}
Tracker holds the registered subsystems. Use the package-level Register / Wait unless you are testing.
func (*Tracker) Add ¶
Add registers one subsystem. Nil InFlight is ignored rather than panicking: a registration site is often optional wiring, and a nil there must not take the daemon down at boot.
func (*Tracker) Busy ¶
Busy renders the non-zero subsystems as "name=n" strings, sorted, with each one's Detail appended when it has any. Empty slice means fully drained.
func (*Tracker) BusyKind ¶
BusyKind renders only the non-zero subsystems whose Resumable flag matches.
func (*Tracker) Snapshot ¶
Snapshot reports the current in-flight count per subsystem, including zeros, so a caller can render a full picture.
func (*Tracker) Wait ¶
Wait blocks until every registered subsystem reports zero, or ctx is done. It logs what is still outstanding every 15s so a long drain is legible rather than a silent hang — the operator can see it is finishing a debug run, not stuck.
Returns the still-busy descriptions: empty means a clean drain.
type Upgrader ¶
type Upgrader struct {
// contains filtered or unexported fields
}
Upgrader owns the listener and the handoff. A nil-safe zero value behaves exactly like the "unsupported" case, so callers can hold one unconditionally.
func New ¶
New builds an Upgrader. It never fails the caller: when graceful upgrade is unavailable — unsupported platform, not requested, or descriptor passing refused — it returns a working Upgrader that simply listens normally, plus the reason, so the daemon boots either way.
func (*Upgrader) AcquireIntake ¶
func (u *Upgrader) AcquireIntake(ctx context.Context) (*IntakeBaton, error)
AcquireIntake blocks until this process owns the intake baton (see IntakeBaton) or ctx is done. It is safe on every platform: where flock is unavailable the baton is a no-op and returns immediately.
func (*Upgrader) Exit ¶
func (u *Upgrader) Exit() <-chan struct{}
Exit closes when a successor has taken over and this process should drain and go away. Never closes when graceful upgrade is off.
func (*Upgrader) IsChild ¶
IsChild reports whether this process was started BY an upgrade — i.e. it inherited its listener from a parent that is still draining.
func (*Upgrader) Listen ¶
Listen returns the listening socket: inherited from the parent during an upgrade, freshly bound otherwise.
func (*Upgrader) NotifyServing ¶
func (u *Upgrader) NotifyServing()
NotifyServing tells the SERVICE MANAGER that this process is now the one answering on the socket. Call it the moment the serve loop starts.
Split from Ready on purpose. systemd's Type=notify has a start timeout, and wick's boot gate can legitimately take minutes to restore a big registry — waiting for that before the first READY=1 would get a healthy daemon killed as a failed start. The MAINPID in here is also what makes an upgrade survivable: after a handoff the unit must follow the child, or the parent's exit looks like the service dying and systemd kills the whole cgroup.
No-op when NOTIFY_SOCKET is unset, i.e. everywhere except systemd.
func (*Upgrader) Ready ¶
func (u *Upgrader) Ready()
Ready announces that this process is fully warmed up. It closes the PARENT's Exit channel, which is what starts the drain over there — so call it only once this process can genuinely do the work, not merely accept connections.
func (*Upgrader) Stop ¶
func (u *Upgrader) Stop()
Stop releases the upgrader's own resources. Safe to defer unconditionally.
func (*Upgrader) Upgrade ¶
Upgrade starts a successor process. Returns ErrUnsupported when this build cannot.
func (*Upgrader) WatchSignal ¶
WatchSignal starts a successor whenever SIGHUP arrives — the conventional "reload" signal, so `systemctl reload`, `kill -HUP`, and `support-tools upgrade` all work through one path. Returns immediately; the watcher lives until ctx is done.
SIGHUP is ignored (with a log line) when graceful upgrade is off, which is strictly better than the Go default of terminating the process.
type Work ¶
type Work struct {
// Name is what the operator sees in the drain log. Short, lowercase.
Name string
// InFlight returns how many units of work are running right now.
InFlight func() int
// Detail optionally identifies the outstanding units (session ids, run
// ids). Called only while the subsystem is non-zero, and capped when
// logged, so it may be moderately expensive.
Detail func() []string
// Resumable marks work that SURVIVES being interrupted because its state
// is on disk and it picks up where it left off — an agent turn is resumed
// with --resume on the next message. Such work gets a short grace period,
// not the full drain: an interactive Slack session can be "in flight" for
// hours, and waiting for it kept the old process alive that whole time.
// Two wick processes coexisting is not free — only one of them holds
// intake, so the browser talks to one while the agents run in the other,
// and tableflip refuses the NEXT upgrade while a parent is still alive.
//
// Everything else (a workflow run mid-node, a cron job mid-write, a
// connector call) is NOT resumable: interrupting it loses the work, so
// the drain waits for it properly.
Resumable bool
}
A wick process runs a lot of work in the background: agent turns in the pool, workflow runs, cron jobs, scheduled messages, sub-agent delegations. Draining means waiting for all of it — and the only way that stays true as features are added is if every new subsystem declares itself here instead of the drain path hard-coding a list it will silently fall behind.
Contract for anything that owns background work:
upgrade.Register("workflow runs", eng.ActiveRuns)
InFlight must be cheap (read a counter under a mutex) and must return 0 once the subsystem has nothing outstanding. Optionally add Detail to name WHAT is still busy, which is what turns "1 still running" into a log line an operator can act on.