upgrade

package
v1.9.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 10, 2026 License: MIT Imports: 14 Imported by: 0

Documentation

Overview

Package upgrade implements zero-downtime restarts for the wick daemon: a new process starts, inherits the listening socket from the old one, and the old process keeps running until every in-flight agent turn has finished. Nothing is killed, no request is refused, and no inbound Slack event is dropped in between.

Shape of a handoff (parent = old binary, child = new binary):

  1. Operator drops a new binary in place and sends SIGHUP (directly, via `systemctl reload`, or via the tray).
  2. Parent forks the child, handing over the listener file descriptor.
  3. Child boots, restores its registry, starts serving HTTP on the SAME socket, then reports Ready.
  4. Parent stops serving HTTP (the child now answers every new request) and begins DRAINING: it keeps its channel listeners and its agent pool alive so running turns finish and their replies still land in Slack.
  5. When the pool is idle the parent releases the intake baton and exits. The child, which has been parked waiting for that baton, starts its own channel listeners, cron and schedule runner.

The baton in steps 4–5 is why inbound events are never lost: intake stays with the process that owns the running sessions until it is completely done, instead of being handed over in a window where neither process is listening. See IntakeBaton.

Platform + deployment support:

  • Linux / macOS / BSD, run as a plain daemon or under systemd: full graceful upgrade. Under systemd the child announces itself with sd_notify MAINPID (needs Type=notify + NotifyAccess=all) so the unit follows the new process instead of treating the parent's exit as a crash.
  • Windows, or wick running under the tray's in-process supervisor (`support-tools status` / processctl managed mode): descriptor passing is not available, so the upgrader degrades to a plain listener and the old stop/start path is used. Everything here stays callable — the methods simply report "not supported", so no caller needs a build tag.

Index

Constants

This section is empty.

Variables

View Source
var Default = &Tracker{}

Default is the process-wide tracker. Registration is global on purpose: a subsystem should be able to opt in from wherever it is constructed, without every constructor growing a parameter it just forwards.

View Source
var ErrUnsupported = errors.New("graceful upgrade not supported here")

ErrUnsupported is returned by Upgrade when this build or this platform cannot hand a listener to a new process.

View Source
var SpawnWrapper func(func() error) error

SpawnWrapper wraps the fork of a successor process. wick sets it to logfiles.WithOriginalStdio so the child inherits the real stdio instead of this process's log pipes — inheriting a pipe whose reader dies with the parent kills the successor with SIGPIPE moments after a clean handoff.

A package var rather than a constructor argument: the upgrader is built deep inside server startup, while the log plumbing is set up in main.

Functions

func AgentGrace added in v1.9.0

func AgentGrace() time.Duration

AgentGrace caps how long a draining parent waits for RESUMABLE work — agent turns. Short by default, and that is a deliberate trade:

An interactive Slack session is "in flight" for as long as somebody keeps talking to it, so waiting for it kept the old process alive for hours. Two processes at once is visible and confusing (only one holds intake, so the browser talks to one while the agents run in the other), and it also blocks the NEXT upgrade — tableflip refuses while a parent is still alive.

A turn interrupted here is not lost work: the session lives on disk and the next message resumes it. Set WICK_DRAIN_AGENT_GRACE higher (e.g. 20m) to let long debug runs finish instead, accepting the longer overlap; set 0 to hand over immediately.

func Busy

func Busy() []string

Busy / Wait / Snapshot on the default tracker.

func BusyKind

func BusyKind(resumable bool) []string

func DrainTimeout

func DrainTimeout() time.Duration

DrainTimeout caps how long a draining parent waits for work that CANNOT be resumed — a workflow run mid-node, a cron job mid-write, a connector call. Minutes are normal for those, so the default is generous; WICK_DRAIN_TIMEOUT overrides it (any Go duration).

func Notify

func Notify(state string) error

Notify sends one sd_notify datagram to systemd. No-op (nil) when the process was not started by systemd with NotifyAccess, so callers can fire and forget on every platform.

Written by hand rather than pulled from a dependency: the protocol is a newline-separated KEY=VALUE blob on a unix datagram socket, and wick only ever sends four of them.

func NotifyReady

func NotifyReady() error

NotifyReady tells systemd this process is serving AND that it is now the unit's main process. The MAINPID line is what makes a tableflip upgrade survivable under systemd: after the handoff the unit must follow the CHILD, otherwise the parent's exit looks like the service dying and systemd kills the cgroup — taking the freshly-started process with it.

Requires NotifyAccess=all in the unit, because the notification comes from a process systemd did not itself fork.

func NotifyStatus

func NotifyStatus(msg string) error

NotifyStatus sets the one-line status shown by `systemctl status`.

func NotifyStopping

func NotifyStopping() error

NotifyStopping declares that the whole SERVICE is going down. Only correct from the process systemd currently tracks as MAINPID, and never on the graceful-upgrade path: a draining predecessor is not the service, and this there makes systemd deactivate the unit and kill the successor.

func Register

func Register(name string, inFlight func() int)

Register adds a subsystem to the default tracker.

func RegisterDetailed

func RegisterDetailed(name string, inFlight func() int, detail func() []string)

RegisterDetailed adds a subsystem that can also name its outstanding work.

func RegisterResumable

func RegisterResumable(name string, inFlight func() int, detail func() []string)

RegisterResumable adds a subsystem whose work survives interruption (see Work.Resumable) — it is waited for only briefly.

func Snapshot

func Snapshot() map[string]int

func Total

func Total() int

func Wait

func Wait(ctx context.Context) []string

func WaitKind

func WaitKind(ctx context.Context, resumable bool) []string

func Wanted

func Wanted() bool

Wanted reports whether the operator asked for graceful upgrades.

Opt-in (WICK_GRACEFUL_UPGRADE=1) rather than default-on: it changes how the process exits, and a deployment that never sends SIGHUP gains nothing from it. Set it in the unit / launchd plist once the handoff has been verified on that host.

Types

type IntakeBaton

type IntakeBaton struct {
	// contains filtered or unexported fields
}

IntakeBaton is the right to accept NEW work: channel listeners (Slack socket mode, Telegram polling), the workflow cron/router, and the scheduled-message runner. Exactly one process may hold it.

It exists because a graceful upgrade means two wick processes are briefly alive, and those subsystems are singletons in a way HTTP is not. Two Slack socket-mode connections split inbound events between the old and the new binary at random; two cron loops fire every job twice; two schedule runners deliver every scheduled message twice. None of that shows up in a test that only checks the port stayed open — it shows up in production as duplicate work, which is worse than a short restart.

The lock is an flock, so the kernel releases it even if the holder is SIGKILLed: a crashed process never wedges the baton.

func (*IntakeBaton) Release

func (b *IntakeBaton) Release()

Release hands the baton to whoever is waiting. Idempotent.

type Tracker

type Tracker struct {
	// contains filtered or unexported fields
}

Tracker holds the registered subsystems. Use the package-level Register / Wait unless you are testing.

func (*Tracker) Add

func (t *Tracker) Add(w Work)

Add registers one subsystem. Nil InFlight is ignored rather than panicking: a registration site is often optional wiring, and a nil there must not take the daemon down at boot.

func (*Tracker) Busy

func (t *Tracker) Busy() []string

Busy renders the non-zero subsystems as "name=n" strings, sorted, with each one's Detail appended when it has any. Empty slice means fully drained.

func (*Tracker) BusyKind

func (t *Tracker) BusyKind(resumable bool) []string

BusyKind renders only the non-zero subsystems whose Resumable flag matches.

func (*Tracker) Snapshot

func (t *Tracker) Snapshot() map[string]int

Snapshot reports the current in-flight count per subsystem, including zeros, so a caller can render a full picture.

func (*Tracker) Total

func (t *Tracker) Total() int

Total is the sum of every subsystem's in-flight count.

func (*Tracker) Wait

func (t *Tracker) Wait(ctx context.Context) []string

Wait blocks until every registered subsystem reports zero, or ctx is done. It logs what is still outstanding every 15s so a long drain is legible rather than a silent hang — the operator can see it is finishing a debug run, not stuck.

Returns the still-busy descriptions: empty means a clean drain.

func (*Tracker) WaitKind

func (t *Tracker) WaitKind(ctx context.Context, resumable bool) []string

WaitKind waits only for the subsystems whose Resumable flag matches, so a caller can give resumable and non-resumable work different deadlines.

type Upgrader

type Upgrader struct {
	// contains filtered or unexported fields
}

Upgrader owns the listener and the handoff. A nil-safe zero value behaves exactly like the "unsupported" case, so callers can hold one unconditionally.

func New

func New(baseDir string) (*Upgrader, string)

New builds an Upgrader. It never fails the caller: when graceful upgrade is unavailable — unsupported platform, not requested, or descriptor passing refused — it returns a working Upgrader that simply listens normally, plus the reason, so the daemon boots either way.

func (*Upgrader) AcquireIntake

func (u *Upgrader) AcquireIntake(ctx context.Context) (*IntakeBaton, error)

AcquireIntake blocks until this process owns the intake baton (see IntakeBaton) or ctx is done. It is safe on every platform: where flock is unavailable the baton is a no-op and returns immediately.

func (*Upgrader) Enabled

func (u *Upgrader) Enabled() bool

Enabled reports whether this process can actually hand off.

func (*Upgrader) Exit

func (u *Upgrader) Exit() <-chan struct{}

Exit closes when a successor has taken over and this process should drain and go away. Never closes when graceful upgrade is off.

func (*Upgrader) IsChild

func (u *Upgrader) IsChild() bool

IsChild reports whether this process was started BY an upgrade — i.e. it inherited its listener from a parent that is still draining.

func (*Upgrader) Listen

func (u *Upgrader) Listen(network, addr string) (net.Listener, error)

Listen returns the listening socket: inherited from the parent during an upgrade, freshly bound otherwise.

func (*Upgrader) NotifyServing

func (u *Upgrader) NotifyServing()

NotifyServing tells the SERVICE MANAGER that this process is now the one answering on the socket. Call it the moment the serve loop starts.

Split from Ready on purpose. systemd's Type=notify has a start timeout, and wick's boot gate can legitimately take minutes to restore a big registry — waiting for that before the first READY=1 would get a healthy daemon killed as a failed start. The MAINPID in here is also what makes an upgrade survivable: after a handoff the unit must follow the child, or the parent's exit looks like the service dying and systemd kills the whole cgroup.

No-op when NOTIFY_SOCKET is unset, i.e. everywhere except systemd.

func (*Upgrader) Ready

func (u *Upgrader) Ready()

Ready announces that this process is fully warmed up. It closes the PARENT's Exit channel, which is what starts the drain over there — so call it only once this process can genuinely do the work, not merely accept connections.

func (*Upgrader) Stop

func (u *Upgrader) Stop()

Stop releases the upgrader's own resources. Safe to defer unconditionally.

func (*Upgrader) Upgrade

func (u *Upgrader) Upgrade() error

Upgrade starts a successor process. Returns ErrUnsupported when this build cannot.

func (*Upgrader) WatchSignal

func (u *Upgrader) WatchSignal(ctx context.Context)

WatchSignal starts a successor whenever SIGHUP arrives — the conventional "reload" signal, so `systemctl reload`, `kill -HUP`, and `support-tools upgrade` all work through one path. Returns immediately; the watcher lives until ctx is done.

SIGHUP is ignored (with a log line) when graceful upgrade is off, which is strictly better than the Go default of terminating the process.

type Work

type Work struct {
	// Name is what the operator sees in the drain log. Short, lowercase.
	Name string
	// InFlight returns how many units of work are running right now.
	InFlight func() int
	// Detail optionally identifies the outstanding units (session ids, run
	// ids). Called only while the subsystem is non-zero, and capped when
	// logged, so it may be moderately expensive.
	Detail func() []string
	// Resumable marks work that SURVIVES being interrupted because its state
	// is on disk and it picks up where it left off — an agent turn is resumed
	// with --resume on the next message. Such work gets a short grace period,
	// not the full drain: an interactive Slack session can be "in flight" for
	// hours, and waiting for it kept the old process alive that whole time.
	// Two wick processes coexisting is not free — only one of them holds
	// intake, so the browser talks to one while the agents run in the other,
	// and tableflip refuses the NEXT upgrade while a parent is still alive.
	//
	// Everything else (a workflow run mid-node, a cron job mid-write, a
	// connector call) is NOT resumable: interrupting it loses the work, so
	// the drain waits for it properly.
	Resumable bool
}

A wick process runs a lot of work in the background: agent turns in the pool, workflow runs, cron jobs, scheduled messages, sub-agent delegations. Draining means waiting for all of it — and the only way that stays true as features are added is if every new subsystem declares itself here instead of the drain path hard-coding a list it will silently fall behind.

Contract for anything that owns background work:

upgrade.Register("workflow runs", eng.ActiveRuns)

InFlight must be cheap (read a counter under a mutex) and must return 0 once the subsystem has nothing outstanding. Optionally add Detail to name WHAT is still busy, which is what turns "1 still running" into a log line an operator can act on.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL