update

package
v0.0.0-...-4c3468c Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 2, 2026 License: AGPL-3.0 Imports: 32 Imported by: 0

Documentation

Overview

Package update implements the M3 bulk plugin/theme/core update feature: an operator creates an update run targeting a selection of sites and items; the control plane fans the work out into per-(site,item) tasks executed by a River worker that snapshots, applies the update via a signed CP->agent command, health-probes the site, and auto-rolls-back on failure. Live progress is streamed over SSE from an in-process pub/sub hub.

Every query is tenant-scoped both explicitly (tenant_id in the WHERE clause) and by Postgres RLS (the app.tenant_id policy on update_runs/update_tasks).

Index

Constants

View Source
const (
	// ActionRunHalted records the wave gate stopping a run. Emitted exactly
	// once per run (see AgentRunEvaluation.Changed).
	ActionRunHalted = "update.run.halted"
	// ActionAgentSelfUpdateArmed records a successful beat 1: the agent
	// verified a newer build and began applying it inside the same request.
	// It is NOT a success record, the task is still running at this point.
	ActionAgentSelfUpdateArmed = "update.agent.armed"
	// ActionAgentSelfUpdateArmUncertain records a beat-1 request whose own
	// HTTP round trip timed out before this control plane read an answer. It
	// is deliberately a DIFFERENT action from ActionAgentSelfUpdateArmed: it
	// is not evidence the agent armed anything, only that this request could
	// not wait for the answer (see armUncertain). The confirm poll enqueued
	// alongside it is what actually decides the task's outcome.
	ActionAgentSelfUpdateArmUncertain = "update.agent.arm_uncertain"
)

Audit actions for the agent self-update channel.

View Source
const (
	// ActionRunDispatched records that a scheduled run fired. It carries the
	// dispatched and skipped counts — and, within skipped, how many were
	// declined because the operator had paused the site — so "the run updated
	// fewer sites than it named" has an answer that does not depend on reading
	// every task row.
	ActionRunDispatched = "update.run.dispatched"
	// ActionRunExpired records that a scheduled run passed its grace window
	// undispatched and no site was contacted. This is the record the #463
	// design asks for by name: an expired run is a bulk update that silently
	// never happened, and without an audit entry the only trace is a status
	// nobody was watching.
	ActionRunExpired = "update.run.expired"
)

Audit actions for the deferred-dispatch lifecycle.

View Source
const (
	RunPending   = "pending"
	RunRunning   = "running"
	RunCompleted = "completed"
	// RunHalted is terminal and specific to an agent self-update run: a wave
	// gate refused to advance, so every task the run had not already dispatched
	// was cancelled. It is deliberately distinct from RunCompleted, which would
	// erase the fact that the run was stopped rather than finished.
	RunHalted = "halted"

	// RunScheduled is a run created with a future scheduled_at that has not
	// fired. NOTHING HAS BEEN SENT TO ANY SITE, and that is the property the
	// rest of the system reads off this value: its tasks are TaskScheduled and
	// therefore outside update_tasks_inflight_target_idx (so they reserve
	// nothing) and outside the stale-task reaper's sweep (so they are not
	// failed for not having progressed). It is the only status
	// ListDueUpdateRuns scans for, and the only precondition
	// ClaimUpdateRunForDispatch, ExpireDueUpdateRun and
	// CancelScheduledUpdateRun accept.
	RunScheduled = "scheduled"
	// RunDispatching is TRANSIENT AND HELD ONLY INSIDE ONE TRANSACTION: the
	// dispatcher claims the run into it, enqueues, and leaves it again before
	// committing. It is deliberately absent from update_runs_due_idx, so a run
	// left sitting here is never scanned again — which is exactly why the
	// claim and the FinishUpdateRunDispatch that closes it must share a
	// transaction, so a crash rolls the claim back rather than stranding the
	// run forever. A reader that observes it should re-read.
	RunDispatching = "dispatching"
	// RunExpired is terminal: the run came due while the control plane was
	// unavailable and was still undispatched more than dispatchGraceWindow
	// later, so running it now is no longer what the operator asked for. No
	// site was contacted. Distinct from RunHalted, which stops a rollout that
	// already has commands out on real sites, and from RunCompleted.
	RunExpired = "expired"
)

Run statuses.

View Source
const (
	TaskPending    = "pending"
	TaskRunning    = "running"
	TaskSucceeded  = "succeeded"
	TaskFailed     = "failed"
	TaskRolledBack = "rolled_back"
	TaskSkipped    = "skipped"
	// TaskCancelled is terminal: the task was never dispatched because its run
	// halted first, or because an operator cancelled the scheduled run before
	// it fired. Distinct from TaskSkipped (the control plane declined this
	// particular target) and from TaskFailed (the site was contacted and did
	// not come back): nothing was ever sent to this site.
	TaskCancelled = "cancelled"

	// TaskScheduled is the birth status of every task of a deferred run, and
	// it is NOT terminal — see terminal(). Two exclusions are load-bearing and
	// both are by construction rather than by a second predicate anybody has
	// to remember to keep in sync:
	//
	//   update_tasks_inflight_target_idx is partial on ('pending','running'),
	//   so a task waiting for 02:00 does NOT hold the (tenant, site, target)
	//   slot and the operator's urgent 10:00 update of that same plugin is
	//   still accepted rather than 409 targets_in_flight.
	//
	//   ListStaleUpdateTasks sweeps the same two statuses, so a task waiting
	//   longer than staleTaskThreshold is not reaped as "stale: task exceeded
	//   max runtime" hours before it was ever meant to run.
	//
	// The price of both is that a scheduled task cannot pre-verify its target
	// will be free when it fires; MarkScheduledUpdateTaskPending resolves that
	// at dispatch time, and the loser becomes TaskSkipped.
	TaskScheduled = "scheduled"
	// TaskExpired is terminal: the parent run expired without dispatching, so
	// this task was never attempted and nothing was sent to its site.
	//
	// IT IS NOT A SPELLING OF TaskCancelled, and collapsing the two would tell
	// an operator that somebody stopped their run when in fact the control
	// plane was simply not up in time to start it. 'cancelled' records a
	// decision a human made; 'expired' records that the window closed.
	// schema.sql declares both, so neither needs a migration.
	TaskExpired = "expired"
)

Task statuses.

View Source
const (
	TargetPlugin = "plugin"
	TargetTheme  = "theme"
	TargetCore   = "core"
	// TargetAgent is the agent's OWN upgrade, shipped over the dedicated
	// three-beat signed channel (agentcmd.AgentSelfUpdateRequest), never over
	// the plugin update path. It is a separate target type rather than a
	// plugin slug precisely so no code that handles a plugin update can ever
	// reach it: the snapshot/rollback that guards every plugin update cannot be
	// delivered by the process being replaced.
	//
	// update_tasks.target_type is plain text NOT NULL (m3), so this value needs
	// no migration.
	TargetAgent = "agent"
)

Target types. plugin/theme/core mirror agentcmd.TargetPlugin/Theme/Core.

View Source
const (
	// RetryClassNeverRan: the task reached a terminal state without the control
	// plane ever sending anything to the site. Two statuses qualify, and they
	// differ only in WHY nothing was sent:
	//
	//   cancelled  a decision — its run halted, or an operator stopped it.
	//   expired    a missed window (#463) — the run came due while the control
	//              plane was unavailable and stayed due past the grace window.
	//
	// Nothing on the site changed in either case, which makes this the
	// LOWEST-risk thing a retry can pick up — and, for `expired` especially,
	// the one an operator most wants back: the work did not happen at all, and
	// not because anything went wrong with it.
	RetryClassNeverRan = "never_ran"
	// RetryClassFailed: the site was contacted and the update did not succeed.
	// The primary retry case.
	RetryClassFailed = "failed"
	// RetryClassReverted: the update APPLIED and was then rolled back, because
	// the post-update health probe or the wave gate refused it. Retrying walks
	// the identical path, so it may reproduce the identical break. Offered, but
	// never part of a default set.
	RetryClassReverted = "reverted"
	// RetryClassSkipped: the control plane declined this target (already up to
	// date, the site stayed busy, the agent's own plugin, an agent build this
	// channel cannot upgrade). Usually correct, but not always: a stale
	// WordPress transient can report "already current" when it is not (GH
	// #211/#212), which is why a skip is selectable rather than hidden.
	RetryClassSkipped = "skipped"
	// RetryClassNotApplicable: retrying is not a meaningful action. Either the
	// task succeeded (a retry would re-touch a working site for nothing) or it
	// has not finished yet (pending/running), in which case there is nothing to
	// retry, only something to wait for.
	RetryClassNotApplicable = "not_applicable"
)

Retry classes. A task's retry class is the SERVER's answer to "what happened to this task, and what would retrying it mean" (GH #336). It exists so no client has to infer that answer from prose: the two things a retry decision actually turns on, whether the update was ever attempted and whether it was applied and then taken back, are recorded as machine values here and read as machine values everywhere else.

It is deliberately NOT the same axis as `status`: `cancelled` and `pending` are different statuses that both mean "nothing was sent to this site", but only one of them is over.

View Source
const (
	// ExcludeNotInRun: the requested task id is not a task of this run (or not
	// of this tenant, which is indistinguishable and deliberately so).
	ExcludeNotInRun = "not_in_run"
	// ExcludeNotRetryable: the task succeeded, or has not finished yet. See
	// retryClassify.
	ExcludeNotRetryable = "not_retryable"
	// ExcludeSiteNotFound: the site no longer resolves.
	ExcludeSiteNotFound = "site_not_found"
	// ExcludeSiteNotEnrolled: the site is no longer enrolled, so no signed
	// command can be delivered to it. Service.CreateRun applies the same gate
	// (resolveSites) but drops such a site silently; a retry reports it.
	ExcludeSiteNotEnrolled = "site_not_enrolled"
	// ExcludeAgentCurrent: the site already runs the published agent build, so
	// there is nothing to upgrade it to. This is the case that fires when an
	// operator REVERTED the release mid-incident, which is exactly what they
	// are expected to do; without this reason the retry would simply produce
	// nothing and say nothing.
	ExcludeAgentCurrent = "agent_current"
	// ExcludeAgentIneligible: the site runs a build with no self-updater (the
	// plugin-directory distribution), which this channel never upgrades.
	ExcludeAgentIneligible = "agent_ineligible"
	// ExcludeAgentVersionUnknown: the site's reported version or the published
	// version is missing or not well-formed. Never guess: an unreadable version
	// must not become an upgrade.
	ExcludeAgentVersionUnknown = "agent_version_unknown"
	// ExcludeTargetInFlight: this (site, target) already has a pending or
	// running task in another run. Creating a second one would race the agent's
	// own snapshot pruning and run concurrent upgraders against the same
	// directory, so it is refused (#131 hardening).
	ExcludeTargetInFlight = "target_in_flight"
	// ExcludeDuplicateTarget: two selected tasks name the SAME (site, target),
	// so retrying both would mean doing one thing twice.
	ExcludeDuplicateTarget = "duplicate_target"
)

Exclusion reasons. Machine values, stable, safe to group and count on. Every one of them is paired with a server-authored sentence in the response so a client can render the truth without maintaining its own copy table.

View Source
const (
	ActionRunCreated = "update.run.created"
	// ActionRunRetried records a retry (GH #336) against the SOURCE run, so the
	// run that failed carries the evidence that someone acted on it. The new run
	// it produced is in the metadata, alongside the requested/created counts.
	ActionRunRetried = "update.run.retried"
	// ActionRunCancelled records an operator calling back a SCHEDULED run
	// before it fired (#463). Distinct from a halt: nothing had been sent to
	// any site, which is why its metadata asserts sites_contacted = 0.
	ActionRunCancelled   = "update.run.cancelled"
	ActionTaskSucceeded  = "update.task.succeeded"
	ActionTaskFailed     = "update.task.failed"
	ActionTaskRolledBack = "update.task.rolled_back"
)

Audit action names for the update lifecycle.

View Source
const AgentTargetSlug = agentplugin.SlugSelfHosted

AgentTargetSlug is the fixed target_slug every agent self-update task carries. The agent target names exactly one thing, this control plane's own plugin, so the operator never supplies a slug and the planner never derives one from a site's inventory (which is what a renamed plugin directory or a stale advisory could poison). It is pinned to the self-hosted distribution slug because that is the build the release channel publishes for; the plugin-directory build has no self-updater and is excluded at planning time.

View Source
const SweepInterval = time.Minute

SweepInterval is how often the safety net runs, exported so cmd/wpmgr can register the periodic job and its UniqueOpts window with the SAME value. They must match: ByPeriod is what stops a rolling deploy's RunOnStart from firing one pass per replica, and a window narrower than the interval would let them through.

Variables

View Source
var ErrRunNotOpen = errors.New("update run is not open")

ErrRunNotOpen is the run-level twin of ErrTaskNotOpen: SetRunStatus matched no row because the run is already TERMINAL ('halted' or 'expired') and the caller asked for a different status. The run is returned unchanged alongside it, exactly as FinishTask returns the recorded task.

It is NOT an error condition, and specifically not domain.NotFound, which this path returned for the same condition before #482 gave SetUpdateRunStatus its precondition. The run virtually always still exists; it is simply not accepting the transition. The caller that meets this in normal operation is Worker.maybeCompleteRun: a halt cancels every pending task and DELIBERATELY leaves running ones alone, so the last of those finishing counts zero unfinished siblings and tries to write 'completed' over 'halted'. Refusing that write is the whole point of the guard, so treating the refusal as a failure would log a warning on every correct halt and teach an operator to ignore the one line that matters. See Worker.maybeCompleteRun.

View Source
var ErrTaskNotClaimed = errors.New("update task was not claimed")

ErrTaskNotClaimed reports that MarkTaskRunning's compare-and-swap matched no row: the caller did NOT get the claim and must not dispatch. It is deliberately distinct from domain.NotFound, which this path used to return for the same condition — the row is virtually always still there, held by another worker, and reporting "update task not found" sends an operator hunting for a deleted row that exists. It is also distinct from ErrTaskNotOpen: that one means a TERMINAL outcome is already recorded and the job is done, whereas this one carries no verdict at all. Zero rows does not say why (see MarkUpdateTaskRunning in db/query/updates.sql), so the caller re-reads the row and decides: terminal => stop; otherwise another worker holds it => give the row up WITHOUT consuming a retry attempt (river.JobSnooze), because that holder may still die and the row must stay reclaimable. See Worker.yieldContendedClaim.

View Source
var ErrTaskNotOpen = errors.New("update task is not open")

ErrTaskNotOpen reports that a task could not be finished because it was no longer open (pending|running): something else already recorded its terminal state. The overwhelmingly common case is a run that was halted while this worker was in flight, the halt cancelled the task, and the worker returning afterwards must not overwrite 'cancelled' with 'succeeded', which would make the kill switch look like it stopped a rollout that then reported itself a success. It is not an error condition for the caller: the recorded outcome stands and the job is done. See Worker.finish.

View Source
var InFlightTaskStatuses = [...]string{TaskPending, TaskRunning}

InFlightTaskStatuses is the ONE definition of "in flight": the task statuses for which the control plane considers an update to be actively occupying its (tenant_id, site_id, target_type, target_slug) slot. Two guards key on exactly this set, in SQL, in two files, and neither can import this declaration:

  • update_tasks_inflight_target_idx (db/schema.sql, m88), the partial unique index that is the authoritative cross-run dedup guard. Its predicate is also CreateUpdateTask's ON CONFLICT arbiter, so Postgres itself requires the two to agree or the insert cannot resolve an arbiter at all.
  • ListStaleUpdateTasks (db/query/updates.sql), the reaper's sweep, which terminalizes anything in this set that has not moved in staleTaskThreshold. RunOnStart means a deploy runs it fleet-wide within seconds of boot.

Membership is therefore load-bearing in BOTH directions and silently so. Widening it strands legitimate immediate work behind the dedup index; widening the reaper's copy destroys work that was never stuck. A status that merely has not finished — one that is planned, deferred or waiting for a clock — is NOT in flight and must stay out of this set.

It is deliberately NOT defined as the complement of terminal(). Today the two happen to partition every status, but that is a coincidence of the current status list, not a property worth encoding: a future status can be neither terminal nor in flight, and defining one from the other would silently sweep it into both guards.

An array rather than a slice so the package's own definition cannot be appended to through the exported name.

Functions

func ClaimStaleAfter

func ClaimStaleAfter(applyJobTimeout time.Duration) time.Duration

ClaimStaleAfter is the staleness bound the update-task claim (MarkUpdateTaskRunning's reclaim arm) is given, derived from THIS install's configured apply job budget rather than fixed.

It is deliberately NOT siteWriterHoldMax, even though the two are equal on default configuration. They are different judgements that happen to agree today, and the reasoning does not transfer between them:

  • For the per-site GATE (SiteHasRunningUpdateTask), being over-permissive is SAFE. Ignoring an over-age row only lets a sibling send a command the gate would otherwise have deferred, and the agent's own site-update lock is the authoritative bound. See siteWriterHoldMax's own comment.
  • For the CLAIM, being over-permissive is THE DEFECT. A bound shorter than the time a worker can legitimately still be applying lets a second worker claim a row whose holder is alive, and both dispatch. The claim therefore has to track the apply budget; a constant cannot.

The apply budget is config-driven (DeriveApplyJobTimeout over cfg.Update.ApplyHTTPTimeout and cfg.Update.HTTPTimeout, both env-settable), so a constant 20m silently stops exceeding it once ApplyHTTPTimeout is raised past ~14m. Deriving from the same number that produced the job timeout keeps the relationship true at any configuration.

The siteWriterHoldMax floor keeps the historical value on default configuration (13m52s + 6m = 19m52s rounds up to 20m) so this is not also a silent retune of production behaviour, and makes the bound monotone in the apply budget.

applyJobTimeout of 0 means "no derived budget; River's own default applies" (see DeriveApplyJobTimeout), which the floor covers.

ValidateClaimTimings must hold for the result; call it at boot.

func DeriveApplyJobTimeout

func DeriveApplyJobTimeout(applyHTTPTimeout, probeHTTPTimeout time.Duration) time.Duration

DeriveApplyJobTimeout computes the River job-level Timeout() budget for the update-apply worker (see Worker.Timeout and NewWorker). It reads the actual ladder schedules (probeRetryDelays, agentVerifyTimeout) rather than assuming them, so this stays correct if either is ever tuned.

runApply's worst-case wall-clock budget, in the order the work happens:

  1. applyHTTPTimeout: the ONE apply/rollback command round trip (cfg.Update.ApplyHTTPTimeout; see UpdateConfig's doc comment in internal/config for how its 8m default was picked).
  2. the agent-first reachability ladder (verifyAgentHealthWithRetry, GH #291 Phase 4): up to len(probeRetryDelays)+1 attempts, each hard-capped at agentVerifyTimeout via context.WithTimeout, plus the sleep between attempts: (len(probeRetryDelays)+1)*agentVerifyTimeout + sum(probeRetryDelays)
  3. the public homepage probe ladder (probeHealthWithRetry), which reuses the SAME probeRetryDelays schedule; each attempt is one prober.Get call capped at probeHTTPTimeout (cfg.Update.HTTPTimeout, the timeout the shared SSRF client that backs the prober is built with): (len(probeRetryDelays)+1)*probeHTTPTimeout + sum(probeRetryDelays)
  4. headroom, mirroring the backup worker's `+2*time.Minute` (see NewBackupWorker/backupJobTimeout in cmd/wpmgr/main.go) for scheduling jitter and the small per-attempt backoff the shared HTTP client's own retry-on-5xx layer can add on top of a single probeHTTPTimeout window (internal/httpclient.Client.Do; not separately itemized above to keep this arithmetic reviewable).

With the production defaults (applyHTTPTimeout=8m, probeHTTPTimeout=30s, agentVerifyTimeout=8s, probeRetryDelays summing to ~21s across 4 delays) this computes to 8m + 61s + 171s + 2m = 13m52s, comfortably under staleTaskThreshold (45m), so the periodic reaper still cannot terminalize a task that is legitimately still running.

Returns 0 (defer to river.Config.JobTimeout) when applyHTTPTimeout is not positive, so a zero/misconfigured input never produces a misleadingly small nonzero timeout that omits the apply round trip it is meant to cover.

func QueueForTenant

func QueueForTenant(tenantID uuid.UUID) string

QueueForTenant maps a tenant to its River queue name. Deterministic so the enqueuer and the worker pool agree.

func QueueNames

func QueueNames() []string

QueueNames returns every per-tenant queue shard name (for River client config).

func ValidateClaimTimings

func ValidateClaimTimings(applyJobTimeout time.Duration) error

ValidateClaimTimings reports whether the claim's staleness bound still sits in the window it must occupy for this install's configuration:

applyJobTimeout  <  ClaimStaleAfter(applyJobTimeout)  <  staleTaskThreshold

The left inequality is what stops a second worker reclaiming a row whose holder is still legitimately applying. The right one keeps the periodic reaper from terminalizing a row the claim would still treat as live, which would let the reaper and a retrying worker act on one task at once.

Both are config-dependent, and neither is checked by the compiler or by any test that runs on default configuration, so this is asserted AT BOOT and the process refuses to start when it fails. That is the intended severity: a configuration that re-opens a double-dispatch window must not run. The remedy is in the message, because the operator who raised the timeout is the one who has to read it.

Types

type AgentApplyResult

type AgentApplyResult struct {
	Status      string
	FromVersion string
	ToVersion   string
	Detail      string
	// At is when the agent stamped the record; the zero time when it did not say.
	At time.Time
	// ApplyID is the opaque per-apply identifier the agent stamped this record
	// with; "" for a record written by an agent that predates it, or one that
	// simply never said. Compared whole against AgentConfirmArgs.ApplyID by
	// agentApplyAttributed; never parsed as a version or a time, and never
	// compared to either.
	ApplyID string
}

AgentApplyResult is the agent's OWN account of the apply. The agent records it on disk as the apply finishes and replays it on its next signed metadata push.

It is the only way a failed apply reaches the control plane at all: the apply releases its response before the swap begins, so nothing else carries the outcome back. Status is one of applied|failed|expired| already_applied, kept as a plain string so a status a newer agent introduces reaches an operator verbatim.

type AgentApplyResultLookup

type AgentApplyResultLookup interface {
	AgentSelfUpdateResult(ctx context.Context, tenantID, siteID uuid.UUID) (AgentApplyResult, bool, error)
}

AgentApplyResultLookup reads the last apply-beat record a site reported. Optional throughout: an unwired lookup, a site that never ran an upgrade, and an agent too old to send the record all produce (zero, false, nil), and the channel behaves exactly as it did before the record existed.

type AgentConfirmArgs

type AgentConfirmArgs struct {
	TenantID uuid.UUID `json:"tenant_id"`
	RunID    uuid.UUID `json:"run_id"`
	TaskID   uuid.UUID `json:"task_id"`
	SiteID   uuid.UUID `json:"site_id"`
	// FromVersion is what the site ran when it armed.
	FromVersion string `json:"from_version,omitempty"`
	// ExpectVersion is the version the agent said it would install. Empty
	// when the agent did not say, in which case any strictly newer well-formed
	// version confirms.
	ExpectVersion string `json:"expect_version,omitempty"`
	// CronMode is carried for the operator-facing detail on a timeout, so a
	// site whose cron is disabled explains itself in the failure text.
	CronMode string `json:"cron_mode,omitempty"`
	// DeadlineAt is when an unconfirmed upgrade becomes a failure.
	DeadlineAt time.Time `json:"deadline_at"`
	// ApplyID is the opaque per-apply identifier carried unmodified from the
	// beat-1 answer that armed this task (agentcmd.AgentSelfUpdateResponse.
	// ApplyID), on BOTH the "scheduled" and "already_scheduled" paths. "" for
	// an agent that predates this field. It is compared WHOLE against the
	// ApplyID on the site's own stored apply record: only a match lets a
	// version movement be credited to THIS run rather than to something else
	// that happened to move the site's version. See agentApplyAttributed.
	ApplyID string `json:"apply_id,omitempty"`
}

AgentConfirmArgs is the River job payload for the beat-2 confirmation poll. The deadline is an ABSOLUTE time carried in the args, so it survives the job snoozing itself repeatedly and cannot be reset by a retry.

func (AgentConfirmArgs) InsertOpts

func (a AgentConfirmArgs) InsertOpts() river.InsertOpts

InsertOpts reuses the per-tenant queue shard so confirmation polls are bounded by the same per-tenant parallelism limit as the tasks they belong to.

func (AgentConfirmArgs) Kind

func (AgentConfirmArgs) Kind() string

Kind implements river.JobArgs.

type AgentConfirmEnqueuer

type AgentConfirmEnqueuer interface {
	EnqueueAgentConfirm(ctx context.Context, args AgentConfirmArgs) error
}

AgentConfirmEnqueuer schedules the beat-2 confirmation poll job.

type AgentConfirmWorker

type AgentConfirmWorker struct {
	river.WorkerDefaults[AgentConfirmArgs]
	// contains filtered or unexported fields
}

AgentConfirmWorker is beat 2: it waits for the NEW agent code to phone home.

This is the only place an agent self-update is allowed to become TaskSucceeded. The ARM acknowledgement from beat 1 says only that a cron event was scheduled; nothing had been applied when it was sent, and on a site whose loopback cron is broken nothing ever will be. Recording that ack as success would report a fleet as upgraded while an unknown share of it still runs the old build.

func NewAgentConfirmWorker

func NewAgentConfirmWorker(w *Worker) *AgentConfirmWorker

NewAgentConfirmWorker builds the confirmation worker on top of the same Worker (and therefore the same repo/hub/audit/logger and the same wave gate) that dispatches the tasks.

func (*AgentConfirmWorker) SetClock

func (c *AgentConfirmWorker) SetClock(now func() time.Time)

SetClock overrides the time source (tests).

func (*AgentConfirmWorker) SetPollInterval

func (c *AgentConfirmWorker) SetPollInterval(d time.Duration)

SetPollInterval overrides the re-check interval. Exposed for tests so the poll loop does not actually wait the production 30s; production code should rely on the default.

func (*AgentConfirmWorker) Work

Work runs one confirmation poll.

type AgentReleaseReader

type AgentReleaseReader interface {
	LatestVersion(ctx context.Context) string
}

AgentReleaseReader reports the currently published agent version, or "" when it cannot be determined. Satisfied by *agentrelease.Reader.

type AgentRunEvaluation

type AgentRunEvaluation struct {
	// Halted is true when the run is halted, whether this call halted it or
	// found it already halted.
	Halted bool
	// Reason explains the halt in operator-readable terms; empty when !Halted.
	Reason string
	// Cancelled counts the tasks THIS call cancelled. Always 0 when Refused:
	// haltLocked asks for the status transition before it cancels anything, so
	// a declined halt leaves every task exactly as it found it.
	Cancelled int
	// Changed is true only for the call that performed the halt transition, so
	// exactly one caller records the audit event and logs the incident.
	Changed bool
	// Refused is true when the run REFUSED the halt: SetUpdateRunStatus's #482
	// precondition declined it because the run is already terminal in some
	// OTHER status (today only 'expired' — an already-'halted' run matches the
	// statement's idempotent `status = $3` arm and returns its row).
	//
	// It exists so a refusal cannot be mistaken for a halt. The refusal comes
	// back from SQL as zero rows, which used to be swallowed as "run not
	// found"; now that the same zero rows also mean "terminal, and you asked
	// for a different status", swallowing it would let a caller announce a halt
	// that never landed. When Refused is true, Halted and Changed are both
	// false, Cancelled is 0, and NO update.run.halted audit event may be
	// written. A REFUSED HALT IS A NO-OP TRANSACTION, not a partial one:
	// haltLocked attempts the status write before it cancels a single task, so
	// there is no half-halt to describe.
	Refused bool
	// Dispatchable is the tasks of a wave that JUST opened: the set the caller
	// should enqueue now, and nothing it has already enqueued. Empty when
	// Halted, and empty on the transitions that open no new wave, which is
	// what keeps a fleet-wide rollout's enqueue work linear rather than
	// quadratic (see AgentWaveState.DispatchableTasks). Safe to enqueue
	// unconditionally: the claim path only ever hands one caller the
	// pending -> running transition.
	Dispatchable []Task
}

AgentRunEvaluation is the result of re-judging a run's wave gate after one of its tasks reached a terminal state.

type AgentSelfUpdateCommander

type AgentSelfUpdateCommander interface {
	AgentSelfUpdate(ctx context.Context, siteID uuid.UUID, siteURL string, req agentcmd.AgentSelfUpdateRequest) (agentcmd.AgentSelfUpdateResponse, error)
}

AgentSelfUpdateCommander sends beat 1 (ARM) of the agent self-update protocol. Deliberately a DIFFERENT interface from Commander: nothing that can send an `update` or `rollback` command is reachable from the agent target's code path, and nothing on this interface can be used to update an ordinary plugin.

type AgentSelfUpdateDeps

type AgentSelfUpdateDeps struct {
	// Enabled is the fleet-wide kill switch. FALSE BY DEFAULT. While false no
	// agent self-update command is sent to any site, whatever is already
	// enqueued and whatever the release manifest says.
	Enabled bool
	// Cmd sends beat 1. A nil Cmd disables the channel exactly like Enabled
	// being false (no signing key configured means no signed command can be
	// minted, and an unsigned one must never be sent).
	Cmd AgentSelfUpdateCommander
	// Versions reads sites.agent_version for beat 2.
	Versions AgentVersionLookup
	// Waves is the run-scoped wave gate persistence.
	Waves AgentWaveRepo
	// Tasks enqueues a wave's tasks once the gate opens for it.
	Tasks Enqueuer
	// Confirms enqueues the beat-2 poll job after a successful arm.
	Confirms AgentConfirmEnqueuer
	// Releases reports the currently published agent version. It is what makes
	// an "up_to_date" answer checkable: the control plane only ever creates a
	// task for a site it classified as OUTDATED, so a site claiming to be up to
	// date means the CP and the agent's manifest disagree, and the disagreement
	// has to be resolved against the published version rather than believed.
	// A nil Releases (or an unknown published version) is fail-closed: the
	// answer confirms nothing. See Worker.upToDate.
	Releases AgentReleaseReader
	// Results reads the agent's own account of its last apply beat, used to
	// explain a confirmation TIMEOUT. Entirely optional: nil leaves the timeout
	// detail exactly as it was, and ready() deliberately does not require it, so
	// a control plane that never wires it still runs the channel.
	Results AgentApplyResultLookup
}

AgentSelfUpdateDeps bundles everything the agent self-update channel needs. Wired once at boot via Worker.SetAgentSelfUpdate; the zero value leaves the channel disabled, which is the shipped default.

type AgentVersionLookup

type AgentVersionLookup interface {
	AgentVersion(ctx context.Context, tenantID, siteID uuid.UUID) (string, error)
}

AgentVersionLookup reads a site's last self-reported agent version (sites.agent_version), written by the agent's signed metadata push. This is the ONLY channel beat 2 trusts: it is the new code speaking for itself.

type AgentWaveClaim

type AgentWaveClaim int

AgentWaveClaim is the outcome of asking to dispatch one agent self-update task.

const (
	// ClaimWait: an earlier wave has not finished proving itself. Nothing was
	// written; the caller must snooze and ask again.
	ClaimWait AgentWaveClaim = iota
	// ClaimProceed: THIS caller moved the task pending -> running and is the
	// only caller that will. It, and only it, may contact the site.
	ClaimProceed
	// ClaimHalted: the run is halted (already, or as of this call). The caller
	// must not contact the site. A task that was still pending has been
	// cancelled by the halt; one that was already running is left alone for its
	// own confirmation job to resolve (see haltLocked), and this claim is
	// simply a duplicate job arriving late.
	ClaimHalted
	// ClaimAlreadyClaimed: another job already took this task past pending, or
	// it is already terminal. Nothing was written and the caller must not
	// dispatch, this is the duplicate-enqueue no-op.
	ClaimAlreadyClaimed
)

type AgentWaveRepo

type AgentWaveRepo interface {
	// ClaimAgentWaveTask atomically decides whether one agent task may be
	// dispatched and, if so, claims it. Everything happens inside one
	// transaction under the per-run advisory lock.
	ClaimAgentWaveTask(ctx context.Context, tenantID, runID, taskID uuid.UUID) (AgentWaveClaim, Task, error)
	// EvaluateAgentRun re-judges the wave gate after a terminal transition,
	// halting the run (and cancelling every task that was never dispatched)
	// when a completed wave failed its gate. Idempotent: only the first caller
	// to halt a given run reports Changed.
	EvaluateAgentRun(ctx context.Context, tenantID, runID uuid.UUID) (AgentRunEvaluation, error)
	// HaltAgentRun halts a run for a reason the wave gate itself cannot see
	// (today: the fleet-wide kill switch), cancelling every task that was never
	// dispatched and leaving running tasks to their own confirmation job (see
	// haltLocked). Idempotent in the same way as EvaluateAgentRun.
	HaltAgentRun(ctx context.Context, tenantID, runID uuid.UUID, reason string) (AgentRunEvaluation, error)
}

AgentWaveRepo is the extra persistence the agent self-update wave machine needs, kept separate from Repo so the ordinary update path (and every fake that implements Repo) is untouched by it.

func NewAgentWaveRepo

func NewAgentWaveRepo(pool *db.Pool) AgentWaveRepo

NewAgentWaveRepo builds the agent self-update wave repo, backed by the same pgx pool and the same tenant-scoped RLS transactions as NewRepo.

type AgentWaveState

type AgentWaveState struct {
	// Order is the run's agent tasks in wave order.
	Order []Task
	// Waves is the wave plan over Order.
	Waves []WaveRange
	// Halt is true when a completed wave forbids the rollout continuing.
	Halt bool
	// HaltReason explains Halt in operator-readable terms; empty when !Halt.
	HaltReason string
	// OpenThrough is the number of waves cleared to dispatch: a task in wave
	// index i may be sent to its site iff i < OpenThrough. Wave 0 is always
	// open (nothing precedes it); each later wave opens only once every
	// preceding wave is fully terminal and passed its gate. Always 0 when Halt.
	OpenThrough int
}

AgentWaveState is a run's whole wave machine, derived from its task rows.

func DeriveAgentWaveState

func DeriveAgentWaveState(tasks []Task) AgentWaveState

DeriveAgentWaveState computes the wave machine from the run's current task rows. It is deliberately a pure function with no clock, no I/O and no hidden state, so two workers finishing concurrently derive the identical verdict and the repo's advisory lock only has to make the resulting write happen once.

func (AgentWaveState) DispatchableTasks

func (s AgentWaveState) DispatchableTasks() []Task

DispatchableTasks returns the tasks that must be enqueued NOW: the pending tasks of a wave that has JUST opened, and nothing else.

"Just opened" is a property of the rows, not of a remembered transition: the newest open wave (index OpenThrough-1) is newly open exactly when every one of its tasks is still pending. A task can only leave pending by being claimed (pgRepo.ClaimAgentWaveTask moves it pending -> running, and every terminal state for an agent task is reached through a claim), so an all-pending open wave is one that has never had a job dispatched from it. Every wave BEFORE it is fully terminal by construction, that is the gate's opening condition, so there is never a pending task in an earlier open wave to pick up here.

Returning the whole open-wave pending set on EVERY terminal transition, which is what this used to do, is quadratic: each of a 950-site final wave's completions re-returned the ~950 still-pending siblings, so a fleet-wide rollout enqueued on the order of n^2/2 River jobs and melted the control plane at exactly the scale this feature exists for. Enqueueing each task once as its wave opens is the same dispatch set spread over the same waves, in linear work.

The gating semantics are untouched: which tasks may run, and when, is still decided by GateOpenFor at claim time. Duplicate enqueues remain harmless (two finishers of the previous wave may both observe the same newly-opened wave, and the claim path only ever hands one caller the pending -> running transition), and a job whose wave has since shut simply snoozes.

A task whose enqueue is LOST is therefore no longer re-dispatched by the next sibling's completion. That is deliberate: it stays pending, its wave never completes, and the stale-task reaper (agentStaleTaskThreshold) terminalizes it, which the wave gate reads as a wave that confirmed less than it should have and halts the run. For this channel, stopping is the safe direction.

func (AgentWaveState) GateOpenFor

func (s AgentWaveState) GateOpenFor(taskID uuid.UUID) bool

GateOpenFor reports whether a task is cleared to dispatch right now.

func (AgentWaveState) WaveIndexOf

func (s AgentWaveState) WaveIndexOf(taskID uuid.UUID) (int, bool)

WaveIndexOf returns the wave a task belongs to, and whether it was found in the run's wave order at all.

type CancelRunResult

type CancelRunResult struct {
	Run Run
	// CancelledTasks is how many tasks were terminalized alongside the run.
	// Zero is legitimate: a run whose tasks had already left 'scheduled'
	// cancels with none.
	CancelledTasks int
}

CancelRunResult is what a successful cancellation reports back.

type Commander

type Commander interface {
	Update(ctx context.Context, siteID uuid.UUID, siteURL string, req agentcmd.UpdateRequest) (agentcmd.UpdateResponse, error)
	Rollback(ctx context.Context, siteID uuid.UUID, siteURL string, req agentcmd.RollbackRequest) (agentcmd.RollbackResponse, error)
}

Commander sends signed CP->agent update/rollback commands. siteID is the target site's stable enrollment UUID, bound into the command JWT's aud claim so a captured token cannot be replayed against a different tenant's site.

type Component

type Component struct {
	Type string // "plugin" | "theme"
	Slug string
	// Name is the component's plugin-header name, carried from the site
	// inventory purely so the planner can identify the agent's own plugin by
	// something the directory name cannot forge (see agentplugin). Empty when
	// the source inventory did not record one.
	Name    string
	Version string

	UpdateAvailable bool
	NewVersion      string
}

Component is one installed plugin/theme with its current version and its OWN pending-update advisory (mirrors site.Component.AvailableUpdate). UpdateAvailable/NewVersion are the per-site authority planTasks intersects a run's requested items against (#126: a target requested because another site in the run has it pending must not spawn a task on a site that does not).

type CreateRunInput

type CreateRunInput struct {
	TenantID    uuid.UUID
	CreatedBy   uuid.UUID
	SiteIDs     []uuid.UUID
	Tag         string
	Items       []Item `validate:"required,min=1,max=200,dive"`
	DryRun      bool
	ScheduledAt *time.Time
}

CreateRunInput is the validated input for creating an update run. Exactly one of SiteIDs or Tag selects the target sites.

type DeferTaskInput

type DeferTaskInput struct {
	TenantID uuid.UUID
	TaskID   uuid.UUID
	// Detail is the operator-facing "waiting: ..." sentence recorded on the
	// task while it waits for its site.
	Detail string
}

DeferTaskInput is the input to DeferTaskToPending.

type DispatchEnqueuer

type DispatchEnqueuer interface {
	// EnqueueDispatch inserts a dispatch job to become runnable at `at`. The
	// job carries no task ids: what is dispatchable at fire time is not
	// knowable now.
	EnqueueDispatch(ctx context.Context, tenantID, runID uuid.UUID, at time.Time) error
}

DispatchEnqueuer schedules the one-shot job that fires a deferred run at its scheduled_at (GH #463). Separate from Enqueuer because it is optional: a Service without one still CREATES scheduled runs correctly (they are committed and due-scannable), it simply has nothing pushing them but the Phase 2 safety-net sweeper.

type DispatchOutcome

type DispatchOutcome struct {
	// Claimed is false when ClaimUpdateRunForDispatch matched no row, which
	// means someone else owns this run. NOT an error, and on a two-replica
	// install it is the normal outcome for about half of every contested tick.
	Claimed bool
	// Dispatched is how many tasks moved 'scheduled' -> 'pending' and had a
	// job enqueued.
	Dispatched int
	// Skipped is how many tasks this pass did not dispatch: their (site,
	// target) was in flight from another run, or their site's monitoring was
	// paused, and they were terminalized 'skipped'. This does NOT fail the run.
	Skipped int
	// PausedSkipped is how many of Skipped were declined specifically because
	// the site's monitoring was paused at fire time.
	//
	// It is a SUBSET of Skipped and not a sibling of it, so the two must never
	// be summed. Skipped stays the answer to "how many tasks did this pass not
	// dispatch"; this one exists because the two causes are different stories
	// to put in front of an operator — "another run was already updating this
	// target" is transient and self-correcting, while "you paused this site" is
	// a state the operator themselves created and is the only one of the two
	// they can act on. A run reporting "8 of 10 applied" with no separation
	// between them reads as broken.
	PausedSkipped int
	// Status is the run status this pass left behind.
	Status string
}

DispatchOutcome reports what one pass over one due run actually did. Every field is a count the caller can log, and the caller is expected to log them: a pass that finds due work and dispatches none is the signature of both previous silent-scheduler defects in this repo.

type DispatchRunArgs

type DispatchRunArgs struct {
	TenantID uuid.UUID `json:"tenant_id"`
	RunID    uuid.UUID `json:"run_id"`
}

DispatchRunArgs is the River job payload for firing ONE deferred run.

It carries no task ids and no scheduled_at, on purpose. What is dispatchable at fire time is not knowable at create time — a target may have gone in flight, the run may have been cancelled — so the worker re-reads everything. Carrying the schedule would let a stale job argue with the row about when the run was due.

func (DispatchRunArgs) InsertOpts

func (a DispatchRunArgs) InsertOpts() river.InsertOpts

InsertOpts pins the dispatch job to the tenant's queue shard, the same shard its tasks will land on, so a tenant's dispatch cannot starve another tenant's.

func (DispatchRunArgs) Kind

func (DispatchRunArgs) Kind() string

Kind implements river.JobArgs.

type DispatchWorker

type DispatchWorker struct {
	river.WorkerDefaults[DispatchRunArgs]
	// contains filtered or unexported fields
}

DispatchWorker fires deferred update runs: River's timer wakes it at the run's scheduled_at, and it re-reads, checks the grace window, claims, and enqueues.

func NewDispatchWorker

func NewDispatchWorker(repo Repo, enq TxEnqueuer, rec *audit.Recorder, logger *slog.Logger) *DispatchWorker

NewDispatchWorker builds the deferred-run dispatcher.

func (*DispatchWorker) SetClock

func (w *DispatchWorker) SetClock(fn func() time.Time)

SetClock overrides the worker's notion of now. Tests only; production leaves it nil and reads the wall clock.

What this clock decides is the DISPATCH/EXPIRE SPLIT and the early-fire snooze, both compared against scheduled_at. That has always been a wall-clock comparison — an earlier version of this comment credited ListDueUpdateRuns' now() with deciding due-ness, which was true of the scan but was never true of the split, and Work no longer calls that scan at all.

The DB clock still gates the path, one step earlier and outside this type: River decides when the job becomes available, from the database. This clock therefore chooses between dispatching and expiring a run that River has already judged due, and the early-fire snooze is the backstop for the two disagreeing.

func (*DispatchWorker) SetTxEnqueuer

func (w *DispatchWorker) SetTxEnqueuer(enq TxEnqueuer)

SetTxEnqueuer wires the transactional job inserter. It is a setter rather than a constructor argument because of a genuine cycle in main: the enqueuer needs the River client, and the River client needs the worker set this worker belongs to. Same shape as Worker.SetRefreshEnqueuer, and it must be called at boot — Work refuses loudly without it (see below) rather than dispatching a run whose tasks would then have no jobs.

func (*DispatchWorker) Work

Work fires one deferred run.

The job is a TRIGGER, not an authority. Everything that decides what happens is re-read from the row: whether the run is still 'scheduled', whether it is inside its grace window, and which of its tasks are still dispatchable. A River job that fires twice, fires late, or fires for a run an operator cancelled ten minutes ago all reach the same CAS statements and lose harmlessly.

THE GRACE WINDOW IS CHECKED HERE, AT FIRE TIME, and not at the point that scheduled the job. That placement is the reason it holds for every trigger path — River's timer, the Phase 2 safety-net sweeper, an operator's manual re-drive — rather than only for the one that happens to have been written first.

type Enqueuer

type Enqueuer interface {
	EnqueueTask(ctx context.Context, tenantID, runID, taskID uuid.UUID, dryRun bool) error
}

Enqueuer schedules the per-task background jobs (River, wired in main).

type Event

type Event struct {
	RunID       uuid.UUID `json:"run_id"`
	TaskID      uuid.UUID `json:"task_id"`
	SiteID      uuid.UUID `json:"site_id"`
	TargetType  string    `json:"target_type"`
	TargetSlug  string    `json:"target_slug"`
	Status      string    `json:"status"`
	FromVersion string    `json:"from_version,omitempty"`
	ToVersion   string    `json:"to_version,omitempty"`
	Detail      string    `json:"detail,omitempty"`
	// RunStatus is the parent run's status at publish time, so a subscriber can
	// detect completion and close the stream.
	RunStatus string `json:"run_status"`
}

Event is a task-status transition published to SSE subscribers of a run. It is intentionally small and carries no secrets.

type FinishTaskInput

type FinishTaskInput struct {
	TenantID    uuid.UUID
	TaskID      uuid.UUID
	Status      string
	FromVersion string
	ToVersion   string
	Detail      string
	Error       string
}

FinishTaskInput records a terminal task state.

type Handler

type Handler struct {
	// contains filtered or unexported fields
}

Handler serves the update endpoints under /api/v1/updates.

func NewHandler

func NewHandler(svc *Service, hub *Hub, rec *audit.Recorder) *Handler

NewHandler builds an update Handler.

func (*Handler) Register

func (h *Handler) Register(r *gin.RouterGroup)

Register mounts the update routes on /api/v1. Mutations require operator+; reads (including the SSE stream) require viewer+.

type HealthProber

type HealthProber interface {
	Get(ctx context.Context, targetURL string) (agentcmd.ProbeResult, error)
}

HealthProber probes a site's homepage for post-update health.

type Hub

type Hub struct {
	// contains filtered or unexported fields
}

Hub is an in-process pub/sub fan-out for update progress, keyed by run ID. River workers Publish transitions; SSE handlers Subscribe to a run. It is safe for concurrent use. Delivery is best-effort: a slow subscriber whose buffer is full drops the event rather than blocking the worker (the SSE handler also re-reads authoritative state from the DB, so a dropped in-flight event is not a correctness problem — it only affects live smoothness).

func NewHub

func NewHub() *Hub

NewHub builds an empty Hub.

func (*Hub) Publish

func (h *Hub) Publish(ev Event)

Publish delivers ev to every current subscriber of ev.RunID. Non-blocking: a subscriber whose buffer is full drops this event.

func (*Hub) Subscribe

func (h *Hub) Subscribe(runID uuid.UUID) (<-chan Event, func())

Subscribe registers a subscriber for the given run and returns its event channel plus an unsubscribe func. The channel is buffered; the caller must drain it and call unsubscribe (e.g. via defer) on disconnect.

func (*Hub) SubscriberCount

func (h *Hub) SubscriberCount(runID uuid.UUID) int

SubscriberCount returns the number of active subscribers for a run (test aid).

type InFlightKey

type InFlightKey struct {
	SiteID     uuid.UUID
	TargetType string
	TargetSlug string
}

InFlightKey identifies one (site, target) pair that currently has a pending/running update task, keyed the same way tasks are planned ("site + type + slug"). Used by ListInFlightTargets/planTasks for the cross-run dedup guard (#131 hardening).

type Item

type Item struct {
	Type    string `json:"type" validate:"required,oneof=plugin theme core agent"`
	Slug    string `json:"slug" validate:"max=200"`
	Version string `json:"version" validate:"max=64"`
}

Item is one requested update target within a CreateRunInput.

type NewTask

type NewTask struct {
	SiteID         uuid.UUID
	TargetType     string
	TargetSlug     string
	DesiredVersion string
	FromVersion    string
}

NewTask is the slim per-(site,item) row to insert when creating a run.

type ReapStaleTasksArgs

type ReapStaleTasksArgs struct{}

ReapStaleTasksArgs is the River job payload for the periodic stale-task reaper sweep. No fields — it enumerates stale rows itself.

func (ReapStaleTasksArgs) Kind

func (ReapStaleTasksArgs) Kind() string

Kind implements river.JobArgs.

type ReaperWorker

type ReaperWorker struct {
	river.WorkerDefaults[ReapStaleTasksArgs]
	// contains filtered or unexported fields
}

ReaperWorker is the periodic River job that sweeps update_tasks for rows stuck in pending/running past staleTaskThreshold and terminalizes them as failed, freeing the (tenant, site, target_type, target_slug) slot the update_tasks_inflight_target_idx partial unique index (m88) would otherwise hold forever. It wraps the same Worker used to execute update tasks and reuses Worker.finish, so a reaped task gets the exact same treatment as a task the agent itself finished (publish to the SSE hub + audit record + the run-completion check that would otherwise never re-run for an orphaned run whose last outstanding task never called finish()).

func NewReaperWorker

func NewReaperWorker(w *Worker) *ReaperWorker

NewReaperWorker builds the stale-task reaper, backed by the same Worker (and therefore the same repo/hub/audit/logger) used to execute update tasks.

func (*ReaperWorker) Work

Work runs one reaper sweep: list stale tasks and terminalize each one. Errors are logged and swallowed — the reaper must not fail the periodic River job (that would just retry the exact same stale rows sooner).

type RefreshCommander

type RefreshCommander interface {
	RefreshInventory(ctx context.Context, siteID uuid.UUID, siteURL string, req agentcmd.RefreshInventoryRequest) (agentcmd.RefreshInventoryResponse, error)
}

RefreshCommander is the subset of the agentcmd.Client used by the refresh worker. Defined here so the worker can be tested with a fake.

type RefreshDebouncer

type RefreshDebouncer struct {
	// contains filtered or unexported fields
}

RefreshDebouncer is a tiny in-process LRU-ish guard that suppresses redundant refresh enqueues for the same site within a short window. The post-update worker enqueues one refresh per task-terminal-state; a 10-plugin bulk run against one site would otherwise enqueue 10 refresh jobs in a few seconds. Debounce window: 30 seconds, matching the spec.

func NewRefreshDebouncer

func NewRefreshDebouncer(window time.Duration) *RefreshDebouncer

NewRefreshDebouncer builds a debouncer. window is the per-site suppression window (the worker calls Allow before enqueueing; consecutive calls within the window short-circuit). maxKeys caps the map size to avoid an unbounded per-site key set — when full, the oldest 25% of entries are evicted.

func (*RefreshDebouncer) Allow

func (d *RefreshDebouncer) Allow(siteID uuid.UUID) bool

Allow reports whether a refresh for siteID should be enqueued NOW, recording the decision when allowed. Concurrent callers race fairly: only one wins within a window.

type RefreshEnqueuer

type RefreshEnqueuer interface {
	EnqueueRefresh(ctx context.Context, args RefreshInventoryArgs) error
}

RefreshEnqueuer schedules refresh-inventory River jobs. Implemented by the River-backed enqueuer (wired in main).

type RefreshInventoryArgs

type RefreshInventoryArgs struct {
	TenantID uuid.UUID `json:"tenant_id"`
	SiteID   uuid.UUID `json:"site_id"`
	SiteURL  string    `json:"site_url"`
	// Source is a short tag for audit/observability: "api" (operator-triggered
	// over /sites/:id/updates/refresh) or "post_update" (the update worker
	// enqueued it after a task reached a terminal state).
	Source string `json:"source,omitempty"`
}

RefreshInventoryArgs is the River job payload for an inventory-refresh request. It carries only the minimum identity needed to mint a signed CP->agent command: the agent re-reads inventory and update transients and pushes the result back over /agent/v1/metadata asynchronously, so we don't persist or correlate a refresh row server-side.

func (RefreshInventoryArgs) InsertOpts

func (a RefreshInventoryArgs) InsertOpts() river.InsertOpts

InsertOpts reuses the per-tenant queue shard so a tenant's refresh jobs are bounded by the same per-tenant parallelism limit as their update tasks.

func (RefreshInventoryArgs) Kind

Kind implements river.JobArgs.

type RefreshInventoryWorker

type RefreshInventoryWorker struct {
	river.WorkerDefaults[RefreshInventoryArgs]
	// contains filtered or unexported fields
}

RefreshInventoryWorker dispatches a signed `refresh_inventory` command to a site's agent. On an OLD-agent 404 (the route doesn't exist on the site yet), the worker records an audit warning and returns nil so River won't retry to death; on any other transport/agent error it returns the error and River retries with its default backoff.

A 200 response is NOT by itself success: the agent reports its own verdict in the body's ok field, and Client.RefreshInventory deliberately leaves that flag for this caller to read (see the note on agentcmd.Client.post). An ok=false is treated as a failure here.

func NewRefreshInventoryWorker

func NewRefreshInventoryWorker(cmd RefreshCommander, rec *audit.Recorder, logger *slog.Logger) *RefreshInventoryWorker

NewRefreshInventoryWorker builds a RefreshInventoryWorker.

func (*RefreshInventoryWorker) Work

Work runs one refresh job.

type Repo

type Repo interface {
	// CreateRunWithTasks atomically creates a run and its tasks in one tx.
	CreateRunWithTasks(ctx context.Context, in CreateRunInput, tasks []NewTask) (Run, []Task, error)

	// CreateScheduledRunWithTasks atomically creates a DEFERRED run (GH #463):
	// the run and every task are born 'scheduled', nothing enters the
	// in-flight index and nothing is enqueued. Requires in.ScheduledAt.
	CreateScheduledRunWithTasks(ctx context.Context, in CreateRunInput, tasks []NewTask) (Run, []Task, error)
	// ListDueRuns returns every run whose scheduled_at has arrived, across ALL
	// tenants, capped at limit. Cross-tenant, under InAgentTx. It does NOT
	// apply the grace window: the caller splits the result into dispatch and
	// expiry, because a scan that filtered by the window would leave a run
	// that fell past it unreachable by any query at all.
	ListDueRuns(ctx context.Context, limit int32) ([]Run, error)
	// DispatchDueRun claims one due run and, in the SAME transaction, moves
	// each of its scheduled tasks to 'pending' and enqueues its job. A task
	// whose target went in flight meanwhile is recorded 'skipped' and does not
	// fail the run. Claimed=false means another writer owns the run: skip it,
	// it is not an error.
	DispatchDueRun(ctx context.Context, enq TxEnqueuer, run Run) (DispatchOutcome, error)
	// ExpireDueRun terminalizes a run past the grace window along with its
	// undispatched tasks (as 'expired'), in ONE transaction. Reports whether
	// the run was expirable and how many tasks it terminalized.
	ExpireDueRun(ctx context.Context, tenantID, runID uuid.UUID, expireBefore time.Time, detail string) (bool, int, error)
	// CancelScheduledRun is the operator's call-back of a run that has not yet
	// fired: run and every still-scheduled task go terminal in ONE
	// transaction, tasks as 'cancelled'. Valid from 'scheduled' ONLY, so it can
	// never race a dispatch into a half-cancelled state. The bool reports
	// whether the run was cancellable; false means "too late", not an error.
	CancelScheduledRun(ctx context.Context, tenantID, runID uuid.UUID, detail string) (Run, int, bool, error)
	GetRun(ctx context.Context, tenantID, runID uuid.UUID) (Run, error)
	ListRuns(ctx context.Context, tenantID uuid.UUID, limit, offset int32) ([]Run, error)
	// ListRunSummaries returns runs with pre-computed task aggregate counts
	// (task_count, succeeded_count, failed_count, site_count) in a single query.
	// Used by the list endpoint to avoid N+1 per-run task fetches.
	ListRunSummaries(ctx context.Context, tenantID uuid.UUID, limit, offset int32) ([]RunSummary, error)
	ListTasks(ctx context.Context, tenantID, runID uuid.UUID) ([]Task, error)
	GetTask(ctx context.Context, tenantID, taskID uuid.UUID) (Task, error)

	// MarkTaskRunning is the claim: a compare-and-swap that transitions a task
	// to 'running' only from 'pending', or from an ABANDONED 'running' (a
	// non-agent row whose command went out longer ago than staleAfter). It
	// returns ErrTaskNotClaimed, never domain.NotFound, when it matches no row.
	//
	// staleAfter must be POSITIVE and is refused otherwise. It is NOT the
	// per-site gate's siteWriterHoldMax: the gate and the claim hold
	// deliberately separate values with opposite safe directions (see
	// ClaimStaleAfter). Production callers pass the worker's own derived
	// Worker.claimStaleAfter.
	//
	// A zero duration is NOT a NULL interval and does NOT fail closed. It is
	// interval '0', which makes every non-agent 'running' row instantly
	// reclaimable — the fail-OPEN direction. Hence the refusal.
	MarkTaskRunning(ctx context.Context, tenantID, taskID uuid.UUID, staleAfter time.Duration) (Task, error)
	// FinishTask records a terminal state for a task that is still OPEN
	// (pending|running). A task that already reached a terminal state is
	// returned unchanged alongside ErrTaskNotOpen: its recorded outcome wins.
	FinishTask(ctx context.Context, in FinishTaskInput) (Task, error)
	// SetRunStatus writes a run's status. A TERMINAL run ('halted'|'expired')
	// is returned unchanged alongside ErrRunNotOpen when a DIFFERENT status is
	// asked for; re-writing a terminal run's own status stays a no-op success,
	// so haltLocked's deliberate re-assert and two tasks racing to 'completed'
	// both still succeed. See ErrRunNotOpen and SetUpdateRunStatus in
	// db/query/updates.sql.
	SetRunStatus(ctx context.Context, tenantID, runID uuid.UUID, status string) (Run, error)
	CountUnfinishedTasks(ctx context.Context, tenantID, runID uuid.UUID) (int64, error)
	CountRunningTasksForTenant(ctx context.Context, tenantID uuid.UUID) (int64, error)
	// ListInFlightTargets returns the set of (site_id, target_type, target_slug)
	// keys that currently have a pending or running task for the tenant, across
	// ANY run, restricted to siteIDs. Used by Service.planTasks to skip creating
	// a duplicate task for a target that already has an update in flight from
	// another run (#131 hardening): without this, a scheduled auto-update, an
	// operator bulk update, and a portal trigger can all create tasks for the
	// SAME (site, plugin) within the same window, and several run concurrently, // racing the agent's own rollback-snapshot pruning and running concurrent
	// WordPress Plugin_Upgrader instances against the same plugin directory. The
	// authoritative guard is the update_tasks_inflight_target_idx partial unique
	// index (see CreateUpdateTask's ON CONFLICT); this pre-check just avoids
	// planning doomed tasks and lets CreateRun report a clean "already in
	// progress" error instead of a partially-empty run.
	ListInFlightTargets(ctx context.Context, tenantID uuid.UUID, siteIDs []uuid.UUID) (map[InFlightKey]struct{}, error)
	// ListStaleUpdateTasks returns tasks stuck in pending/running for longer
	// than threshold, across ALL tenants, capped at limit rows. Used by the
	// periodic reaper (see ReaperWorker) to terminalize a task a worker crash
	// or a failed enqueue left permanently occupying its (tenant, site,
	// target_type, target_slug) slot in the update_tasks_inflight_target_idx
	// partial unique index (m88) — without this, every future update attempt
	// for that target would 409 targets_in_flight forever.
	ListStaleUpdateTasks(ctx context.Context, threshold time.Duration, limit int32) ([]Task, error)

	// SiteHasRunningTask is the GH #328 per-site serialisation pre-check
	// (INVARIANT R): does SOME OTHER task for this site currently have a
	// command dispatched and unresolved? Best-effort — the agent's own
	// site-update lock is the authoritative bound; a missed collision here
	// just costs one HTTP round trip the agent itself refuses, which the
	// worker then defers. excludeTaskID excludes the caller's own row (a task
	// re-checking the gate against itself would always see itself as busy).
	// holdMax is the staleness bound (see SiteHasRunningUpdateTask's own
	// comment in db/query/updates.sql for why an over-age 'running' row is
	// ignored rather than trusted).
	SiteHasRunningTask(ctx context.Context, tenantID, siteID, excludeTaskID uuid.UUID, holdMax time.Duration) (bool, error)

	// DeferTaskToPending records that a task is WAITING for its site (busy
	// with another update) rather than talking to it: moves it back to
	// 'pending' (see DeferUpdateTaskToPending). Returns ErrTaskNotOpen when
	// the task already reached a terminal state (e.g. a halt raced the
	// defer); the caller must stop rather than snooze in that case, exactly
	// like FinishTask.
	DeferTaskToPending(ctx context.Context, in DeferTaskInput) (Task, error)
}

Repo is the tenant-scoped persistence interface for update runs/tasks. Every method runs inside a tenant-scoped transaction so RLS enforces isolation even if a query omitted its tenant filter.

func NewRepo

func NewRepo(pool *db.Pool) Repo

NewRepo builds a Repo backed by the pgx pool with RLS enforcement.

type RetryExclusion

type RetryExclusion struct {
	TaskID uuid.UUID
	// Reason is the machine value (one of the Exclude* constants).
	Reason string
	// Message is the server-authored sentence for that reason, including the
	// specifics (which versions, which status), so the client renders the truth
	// instead of reconstructing it.
	Message string
}

RetryExclusion is one requested task that produced no work, and why.

type RetryResult

type RetryResult struct {
	// RunID is the new run. uuid.Nil when nothing was created, which is a
	// legitimate outcome (everything was excluded) and not an error: the
	// exclusions say why.
	RunID uuid.UUID
	// Requested is the number of DISTINCT task ids in the request.
	Requested int
	// Created is the number of tasks in the new run.
	Created int
	// Excluded names every requested task that produced no task in the new run.
	Excluded []RetryExclusion
	// Warning is set when the run exists but something after the commit did not
	// go to plan (today: a background enqueue failed, so its tasks sit pending
	// until the reaper). Empty on the ordinary path.
	Warning string
}

RetryResult accounts for EVERY requested task: Created + len(Excluded) always equals Requested. That identity is the whole point of the type.

type RetryRunInput

type RetryRunInput struct {
	TenantID  uuid.UUID
	CreatedBy uuid.UUID
	// RunID is the run the selected tasks belong to.
	RunID uuid.UUID
	// TaskIDs is the operator's explicit selection. There is no implicit
	// default set on the server: the client knows every task's retry_class from
	// the run detail response and names the ones it means. A server-side
	// default would be a second policy that could disagree with the checkboxes
	// the operator actually saw.
	TaskIDs []uuid.UUID
}

RetryRunInput is the validated input for retrying a run's tasks.

type RiverEnqueuer

type RiverEnqueuer struct {
	// contains filtered or unexported fields
}

RiverEnqueuer enqueues update-task jobs onto River. It satisfies the service's Enqueuer interface.

func NewRiverEnqueuer

func NewRiverEnqueuer(client *river.Client[pgx.Tx]) *RiverEnqueuer

NewRiverEnqueuer builds an enqueuer around the River client.

func (*RiverEnqueuer) EnqueueAgentConfirm

func (e *RiverEnqueuer) EnqueueAgentConfirm(ctx context.Context, args AgentConfirmArgs) error

EnqueueAgentConfirm inserts one agent self-update confirmation-poll job (beat 2). The job's InsertOpts pin it to the tenant's queue shard. Satisfies AgentConfirmEnqueuer.

func (*RiverEnqueuer) EnqueueDispatch

func (e *RiverEnqueuer) EnqueueDispatch(ctx context.Context, tenantID, runID uuid.UUID, at time.Time) error

EnqueueDispatch inserts the one-shot job that fires a deferred run, using River's NATIVE scheduled insert (InsertOpts.ScheduledAt) rather than a polling loop. Satisfies DispatchEnqueuer.

River holds the job in 'scheduled' and makes it available at `at`, which is exactly the semantics wanted and is why no new timer machinery exists here. The job is still only a TRIGGER: DispatchWorker re-reads the run, so a job that fires twice, late, or for a cancelled run loses harmlessly against the CAS statements.

The job's own retention against a 30-day-out schedule is reasoned but not measured, which is what the #463 Phase 2 safety-net sweeper exists to cover: a run whose queue insert was lost or reaped is still 'scheduled' and still due-scannable, so the sweeper finds it. Until that lands, this insert is the only thing that fires a run.

func (*RiverEnqueuer) EnqueueDispatchIfAbsent

func (e *RiverEnqueuer) EnqueueDispatchIfAbsent(ctx context.Context, tenantID, runID uuid.UUID, at time.Time) (bool, error)

EnqueueDispatchIfAbsent inserts a dispatch job only if the run does not already have a live one, reporting whether it inserted. Satisfies SweepEnqueuer.

THE IDEMPOTENCY IS RIVER'S, NOT A PRE-CHECK OF OURS. dispatchInsertOpts sets UniqueOpts{ByArgs: true} and DispatchRunArgs is exactly (tenant_id, run_id), so River refuses a second job for a run that already has one pending, scheduled, available, running or retryable — atomically, against every replica at once. A read-then-insert here would have a window between the two wide enough for a second sweeper to drive through.

The false return is the HEALTHY steady state (the run's original job is still sitting there waiting for its time) and the caller must stay quiet about it. A true return means the safety net actually caught something.

func (*RiverEnqueuer) EnqueueRefresh

func (e *RiverEnqueuer) EnqueueRefresh(ctx context.Context, args RefreshInventoryArgs) error

EnqueueRefresh inserts one refresh-inventory job. The job's InsertOpts pin it to the tenant's queue shard. Satisfies RefreshEnqueuer.

func (*RiverEnqueuer) EnqueueTask

func (e *RiverEnqueuer) EnqueueTask(ctx context.Context, tenantID, runID, taskID uuid.UUID, dryRun bool) error

EnqueueTask inserts one update-task job. The job's InsertOpts pin it to the tenant's queue shard (per-tenant parallelism). dryRun is carried in the args.

func (*RiverEnqueuer) EnqueueTaskTx

func (e *RiverEnqueuer) EnqueueTaskTx(ctx context.Context, tx pgx.Tx, tenantID, runID, taskID uuid.UUID, dryRun bool) error

EnqueueTaskTx inserts one update-task job INSIDE an existing transaction, so the job becomes visible if and only if that transaction commits. Satisfies TxEnqueuer.

This is what makes deferred dispatch atomic. pgRepo.DispatchDueRun claims the run into the transient 'dispatching' status, moves each task to 'pending' and enqueues, all in one transaction: a crash anywhere rolls the claim back and the next tick finds the run still 'scheduled'. Using the non-transactional Insert here would commit jobs for a claim that then rolled back, and — worse in the other direction — commit a claim whose jobs were never inserted, which strands the run permanently ('dispatching' is outside update_runs_due_idx, so nothing ever scans it again).

type Run

type Run struct {
	ID          uuid.UUID
	TenantID    uuid.UUID
	CreatedBy   *uuid.UUID
	Status      string
	DryRun      bool
	ScheduledAt *time.Time
	CreatedAt   time.Time
	UpdatedAt   time.Time
}

Run is an update run: a tenant-scoped unit grouping per-(site,item) tasks.

type RunSummary

type RunSummary struct {
	Run
	TaskCount      int64
	SucceededCount int64
	FailedCount    int64
	SiteCount      int64
}

RunSummary is a Run with pre-computed task aggregate counts, returned by the list endpoint so callers do not need a separate per-run tasks fetch.

type Service

type Service struct {
	// contains filtered or unexported fields
}

Service holds the update orchestration logic.

func NewService

func NewService(repo Repo, sites SiteLookup, enqueuer Enqueuer, v *domain.Validator, clock domain.Clock) *Service

NewService builds an update Service.

func (*Service) CancelScheduledRun

func (s *Service) CancelScheduledRun(ctx context.Context, tenantID, runID uuid.UUID) (CancelRunResult, error)

CancelScheduledRun calls back a deferred run before it fires.

The 409 on a non-scheduled run is the WHOLE FEATURE, not an edge case. A cancel that "succeeded" on a run whose commands were already out would tell an operator their fleet was safe while it was being updated, which is worse than refusing outright. The repo's CAS is what decides, and a false from it becomes a typed conflict here so httpx renders 409 and the client can distinguish "too late" from a server fault.

The NotFound check runs first and separately so the two answers stay distinct: a run that never existed is a 404, and a run that exists but has already fired is a 409 with a status the operator can act on.

func (*Service) CreateRun

func (s *Service) CreateRun(ctx context.Context, in CreateRunInput) (Run, []Task, error)

CreateRun validates the input, resolves the target sites, creates the run and its per-(site,item) tasks atomically, then enqueues a background job per task. from_version is seeded from the site's known component versions so the run records the prior state even before the worker contacts the agent.

func (*Service) GetRun

func (s *Service) GetRun(ctx context.Context, tenantID, runID uuid.UUID) (Run, []Task, error)

GetRun returns a run with its tasks.

func (*Service) ListRunSummaries

func (s *Service) ListRunSummaries(ctx context.Context, tenantID uuid.UUID, limit, offset int32) ([]RunSummary, error)

ListRunSummaries returns a page of the tenant's runs with pre-computed per-run task aggregate counts (task_count, succeeded_count, failed_count, site_count) in a single query. Used by the list endpoint.

func (*Service) ListRuns

func (s *Service) ListRuns(ctx context.Context, tenantID uuid.UUID, limit, offset int32) ([]Run, error)

ListRuns returns a page of the tenant's runs.

func (*Service) RetryRun

func (s *Service) RetryRun(ctx context.Context, in RetryRunInput) (RetryResult, error)

RetryRun creates a NEW run repeating the selected tasks of an existing one.

It returns a result that accounts for every requested task even when it also returns an error: a non-nil RunID means the new run IS committed (its tasks exist and are visible) and only a best-effort step after the commit failed, exactly like Service.CreateRun. A caller must report the run in that case rather than telling the operator nothing happened.

func (*Service) SetAgentSelfUpdate

func (s *Service) SetAgentSelfUpdate(enabled bool, releases AgentReleaseReader)

SetAgentSelfUpdate wires the agent self-update channel's kill switch and the published-version reader. Call once at boot; not calling it leaves the channel off.

func (*Service) SetDispatchEnqueuer

func (s *Service) SetDispatchEnqueuer(d DispatchEnqueuer)

SetDispatchEnqueuer wires the deferred-dispatch job inserter. Call once at boot.

type SiteInfo

type SiteInfo struct {
	ID         uuid.UUID
	URL        string
	Name       string
	Enrolled   bool
	Components []Component

	// CoreUpdateAvailable reports whether THIS site's own inventory advertises
	// a pending WordPress core update. CoreCurrentVersion/CoreNewVersion carry
	// the versions from that advisory when set.
	CoreUpdateAvailable bool
	CoreCurrentVersion  string
	CoreNewVersion      string

	// AgentVersion is the agent version the site last reported over its signed
	// metadata push (sites.agent_version). It is the ONLY input the agent
	// self-update planner uses to decide whether a site is behind, precisely
	// because it does not come from the site's plugin inventory: that inventory
	// is built from the site's own WordPress update-check cache, which for the
	// agent is a stale advisory the control plane deliberately suppresses
	// everywhere else (see indexPending). Empty when the site has never
	// reported one.
	AgentVersion string
}

SiteInfo is the minimal site projection the update service needs to plan and execute a run: identity, the agent target URL, the current component versions (used to seed from_version without a round-trip to the agent), and the site's OWN pending WordPress-core update advisory (CoreUpdateAvailable etc). planTasks intersects a run's requested items against each site's own pending set (Components[].UpdateAvailable / CoreUpdateAvailable) — see #126.

type SiteLookup

type SiteLookup interface {
	// GetSiteInfo returns one tenant-scoped site, or a NotFound domain error.
	GetSiteInfo(ctx context.Context, tenantID, siteID uuid.UUID) (SiteInfo, error)
	// ListSiteInfoByTag returns the tenant's enrolled sites carrying tag (empty
	// tag ⇒ all enrolled sites).
	ListSiteInfoByTag(ctx context.Context, tenantID uuid.UUID, tag string) ([]SiteInfo, error)
}

SiteLookup resolves the target sites for a run. Implemented by the site service (wired in main) so the update package needs no site import.

type SweepDueRunsArgs

type SweepDueRunsArgs struct{}

SweepDueRunsArgs is the periodic River job that re-enqueues dispatch jobs for due runs that have none. Cross-tenant; carries no payload.

func (SweepDueRunsArgs) Kind

func (SweepDueRunsArgs) Kind() string

Kind implements river.JobArgs.

type SweepEnqueuer

type SweepEnqueuer interface {
	// EnqueueDispatchIfAbsent inserts a dispatch job for a run that has none,
	// reporting whether it actually inserted one. False means a live job
	// already exists and nothing was done.
	EnqueueDispatchIfAbsent(ctx context.Context, tenantID, runID uuid.UUID, at time.Time) (bool, error)
}

SweepEnqueuer is the sweeper's half of the enqueuer contract, kept separate from DispatchEnqueuer because the two want opposite things from a duplicate.

Create-time enqueue wants the job inserted, full stop. The sweeper wants it inserted ONLY IF ABSENT, and needs to be told which happened — "already live" is the healthy steady state it must stay quiet about, and "inserted" is a safety net catching a real miss, which is worth a warning every time.

type SweepStats

type SweepStats struct {
	// Due is how many due runs the scan returned, capped at dueRunScanLimit.
	Due int
	// Enqueued is how many dispatch jobs this pass actually inserted.
	Enqueued int
	// AlreadyLive is how many runs already had a dispatch job, so River's
	// unique guard skipped the insert. On a healthy install this is where
	// nearly every due run lands.
	AlreadyLive int
	// Overdue is how many of the scanned runs are already past the grace
	// window. See the gauge discussion in Work: this is a LOWER BOUND.
	Overdue int
	// Capped reports that the scan filled dueRunScanLimit, so Due is a floor
	// and not a total.
	Capped bool
	// Skipped reports that another replica held the lock and this pass did
	// nothing at all. Distinct from a pass that ran and found nothing.
	Skipped bool
}

SweepStats is one pass's outcome, returned so a test can assert on it directly rather than parsing log lines.

type SweepWorker

type SweepWorker struct {
	river.WorkerDefaults[SweepDueRunsArgs]
	// contains filtered or unexported fields
}

SweepWorker is the safety-net sweeper.

func NewSweepWorker

func NewSweepWorker(repo Repo, enq SweepEnqueuer, pool *pgxpool.Pool, logger *slog.Logger) *SweepWorker

NewSweepWorker builds the dispatch sweeper. pool is used only for the single-flight advisory lock and may be nil (the pass then runs unlocked, which is safe but wasteful — see Work).

func (*SweepWorker) SetClock

func (w *SweepWorker) SetClock(fn func() time.Time)

SetClock overrides the sweeper's notion of now. Tests only.

func (*SweepWorker) SetEnqueuer

func (w *SweepWorker) SetEnqueuer(enq SweepEnqueuer)

SetEnqueuer wires the dispatch-job inserter. A setter for the same cycle reason as DispatchWorker.SetTxEnqueuer: the enqueuer needs the River client, and the client needs this worker. A nil enqueuer makes every pass a no-op that still logs, which sweep() reports rather than hiding.

func (*SweepWorker) Work

Work runs one sweeper pass.

Errors are logged and swallowed rather than returned, matching ReaperWorker: failing the periodic job would only retry the identical scan sooner, and a dead-lettered sweeper is a silently absent safety net.

type Task

type Task struct {
	ID       uuid.UUID
	RunID    uuid.UUID
	TenantID uuid.UUID
	SiteID   uuid.UUID
	// SiteName is the site's display name, carried on the task row itself so a
	// caller never has to join a task list against a separately-fetched (and
	// therefore paginated) site list to say which site a task belongs to. Only
	// the DETAIL read populates it (Repo.ListTasks, via
	// ListUpdateTasksForRunWithSiteName); the wave machine and the per-task
	// worker paths do not need it and leave it empty.
	SiteName       string
	TargetType     string
	TargetSlug     string
	DesiredVersion string
	FromVersion    string
	ToVersion      string
	Status         string
	Detail         string
	Error          string
	StartedAt      *time.Time
	FinishedAt     *time.Time
	CreatedAt      time.Time
	UpdatedAt      time.Time
}

Task is one unit of work: apply one item on one site.

type TaskArgs

type TaskArgs struct {
	TenantID uuid.UUID `json:"tenant_id"`
	RunID    uuid.UUID `json:"run_id"`
	TaskID   uuid.UUID `json:"task_id"`
	DryRun   bool      `json:"dry_run"`
}

TaskArgs is the River job payload for one update task. It carries only IDs; the worker re-reads authoritative state (tenant-scoped) from the DB.

func (TaskArgs) InsertOpts

func (a TaskArgs) InsertOpts() river.InsertOpts

InsertOpts pins each task to a per-tenant queue so River's per-queue MaxWorkers bounds a single tenant's concurrency — one tenant cannot starve others. The queue name is derived from the tenant id; the worker pool registers a bounded number of these queues (see QueueForTenant).

func (TaskArgs) Kind

func (TaskArgs) Kind() string

Kind implements river.JobArgs.

type TxEnqueuer

type TxEnqueuer interface {
	EnqueueTaskTx(ctx context.Context, tx pgx.Tx, tenantID, runID, taskID uuid.UUID, dryRun bool) error
}

TxEnqueuer enqueues a task job INSIDE an existing transaction (River's InsertTx). The deferred-dispatch path needs this and cannot use the ordinary Enqueuer: the claim, the per-task transitions and the enqueue must commit together or not at all. See pgRepo.DispatchDueRun for why splitting them strands a run permanently.

type WaveRange

type WaveRange struct {
	Start int
	End   int
}

WaveRange is one wave's half-open span [Start, End) over a run's ordered task list.

func PlanWaves

func PlanWaves(n int) []WaveRange

PlanWaves splits n ordered tasks into the canary/pilot/remainder waves. Empty trailing waves are dropped, so a 1-site run has exactly one wave and a 3-site run has two. n <= 0 yields no waves.

func (WaveRange) Len

func (w WaveRange) Len() int

Len returns how many tasks the wave covers.

type Worker

type Worker struct {
	river.WorkerDefaults[TaskArgs]
	// contains filtered or unexported fields
}

Worker executes one update task end-to-end.

func NewWorker

func NewWorker(repo Repo, sites SiteLookup, cmd Commander, prober HealthProber, hub *Hub, rec *audit.Recorder, logger *slog.Logger, perTenantLimit int, jobTimeout time.Duration) *Worker

NewWorker builds the update task worker. jobTimeout overrides River's default 60s per-job deadline; pass update.DeriveApplyJobTimeout(cfg.Update. ApplyHTTPTimeout, cfg.Update.HTTPTimeout) (see that function's doc comment for the arithmetic). Zero keeps River's default.

func (*Worker) SetAgentSelfUpdate

func (w *Worker) SetAgentSelfUpdate(d AgentSelfUpdateDeps)

SetAgentSelfUpdate wires the agent self-update channel. Call once at boot. Not calling it at all leaves the channel off, so a deployment that never opts in behaves exactly as it did before this channel existed.

func (*Worker) SetProbeRetryDelays

func (w *Worker) SetProbeRetryDelays(delays []time.Duration)

SetProbeRetryDelays overrides the post-update health-probe backoff schedule. Exposed for tests so the retry loop does not actually sleep the production ~21s window; production code should rely on the default (probeRetryDelays). Passing nil restores the default; pass a non-nil empty slice ([]time.Duration{}) to disable retries entirely (single probe).

func (*Worker) SetRefreshEnqueuer

func (w *Worker) SetRefreshEnqueuer(r RefreshEnqueuer, d *RefreshDebouncer)

SetRefreshEnqueuer wires the post-update inventory-refresh enqueuer + its per-site debouncer. Call once at boot after the River client is up. A nil refresher disables the post-update refresh entirely.

func (*Worker) SetSiteBusyBackoff

func (w *Worker) SetSiteBusyBackoff(f func(Task) time.Duration)

SetSiteBusyBackoff overrides the busy-site retry backoff schedule. Exposed for tests so a bulk-update test does not actually sleep the production 5s-60s window; production code should leave this unset (nil restores the default, siteBusyBackoff).

func (*Worker) Timeout

func (w *Worker) Timeout(*river.Job[TaskArgs]) time.Duration

Timeout overrides River's default per-job context deadline (60s) for the update-task worker, the same way BackupWorker.Timeout does. Returning a positive duration makes River use it instead of river.Config.JobTimeout; returning 0 keeps the default (see cmp.Or in river's job executor, which falls through to river.Config.JobTimeout, itself defaulting to River's own 60s river.JobTimeoutDefault when that is also unset, exactly as it is in this codebase's river.Config today). River documents that returning -1 disables the deadline entirely; we intentionally do NOT do that: a wedged update task must eventually error out so River can retry/reap it.

func (*Worker) Work

func (w *Worker) Work(ctx context.Context, job *river.Job[TaskArgs]) error

Work runs one update task. It is idempotent-ish: a task already in a terminal state is skipped. On a transient infrastructure error it returns the error so River retries; per-item update failures are recorded as terminal task states (not job errors).

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL