runtime

package
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 5, 2026 License: Apache-2.0 Imports: 20 Imported by: 0

Documentation

Overview

Package runtime is the per-pod runtime that drives providers. It is the analogue of K8s controller-runtime's manager + worker-queue machinery: kind-agnostic plumbing that runs on every pod and turns the persistent state in `resources` / `work_queue` / `work_outbox` / `lifecycle_outbox` into actual provider invocations. It is manifest-driven: it dispatches off a kind's declared reactions (read from the KindManifestCache), never off a kind name.

The drivers here are independent goroutine loops; a pod runs the subset its duties (see internal/engine) select. Grouped by who owns them:

Dispatcher (claim loop — the broker's, via internal/broker) — loop.go
  Round-robins the per-(kind,task_type) work_queue claim (FOR UPDATE SKIP
  LOCKED, equality probe + partition pruning + per-kind global cap), keeps
  each pair filled to its cap, runs ONE heartbeat over all in-flight tasks,
  and feeds each claimed task to a StageDispatcher. The reconcile task runs
  a kind's compose?/work?/rollup? reactions in one invocation; operate runs
  the operation reaction; the shared reaction-application body is reaction_engine.go.

Drainer (control) — drainer.go
  Pops batches from work_outbox and applies the deltas to resources +
  work_queue in one tx (synced_gen advance + health_ok — which flip the
  generated is_ready — status write, resource_conditions upsert, reactive dep
  cascade). Compose results bypass it (children commit inline).

Reaper (control) — reaper.go
  Time-driven sweeper. Re-arms rows the trigger machinery missed (deps just
  ready, failed past retry, stale claims) and sweeps expired orphans. Calls
  the plpgsql reap_stale_work / requeue_failed_and_pending / sweep_expired_orphans.

Resyncer (control, only for kinds that opted in) — resyncer.go
  Drift-detection sweeper: re-pends settled resources WITHOUT bumping
  generation so the provider re-observes health. Calls requeue_for_resync.

ReactorDispatcher (broker) — reactor.go
  Drains lifecycle_outbox and ships each matched transition's reaction
  (STAGE_REACT) through the same StageDispatcher as the claim loop, acking
  only after a successful delivery (at-least-once).

KindManifestCache — kind_manifest_cache.go
  Lock-free snapshot of kind_manifest (the CRDs), refreshed via a
  NotifyRefresher on kind_manifest_changed; the claim/dispatch reads a kind's
  reactions + policy from it (the API validates schemas off a separate view).

ProviderConfigCache — provider_config_cache.go
  Boot + live per-kind default providerconfig, refreshed via a NotifyRefresher
  on providerconfig_changed; the broker serves it to dumb workers.

NotifyRefresher — notify_refresher.go
  The shared "keep an in-memory view convergent with a DB table" driver: a
  LISTEN on a NOTIFY channel (prompt) + a failsafe reload tick (a dropped
  NOTIFY still converges). KindManifestCache + ProviderConfigCache build on it, as does
  the API pod's kind-schema surface (cmd/converge).

Resharder — resharder.go
  Recomputes this pod's owned shard range on cluster-membership change and
  swaps the shared ShardSet the other drivers read each tick.

ClusterMemberReporter — the running-fleet heartbeat (one per process).

All are idempotent and shard-aware. If a pod dies mid-call the Reaper re-pends and the next claim re-runs; if Postgres dies the reaper's full-table sweep recovers the queue from `resources` (so work_queue / work_outbox can be UNLOGGED).

What this package is NOT

  • Not the model. The reaction model (ReactionDecl, StageDispatcher, Outcome) lives in internal/model/, and provider handler types in sdk-go/converge/. This package consumes them; it does not define them.
  • Not the providers. Concrete kinds live in their own provider packages (see examples/demos/ for reference providers). A provider author never needs to read this package.
  • Not boot/wiring. internal/engine/ assembles a pod's duty list from these drivers; main.go calls into engine, never directly into runtime.

Index

Constants

View Source
const (
	// DefaultHeartbeatEvery is the claimed-task LEASE renewal cadence (keeps a
	// live task from looking stale to the reaper).
	DefaultHeartbeatEvery = 5 * time.Second
	// DefaultRequireWithin bounds how stale a worker's per-task liveness attestation may
	// be for the dispatcher to keep heartbeating a NON-LOCAL (forwarded mesh) task's lease.
	// Sized between 2×DefaultHeartbeatEvery (a couple of missed frames tolerated) and half
	// the reaper's StaleAfter (broker-clock skew can't cross the stale boundary): a silent
	// peer's forwarded lease goes stale within a bounded number of missed frames, never
	// indefinitely. It does NOT gate a LOCAL task (one whose post-Complete apply phase is
	// running on this pod's own goroutine — flipped local via markLocal): that lease is
	// heartbeated by the live goroutine itself (see inFlightRecord.local). A task still
	// PARKED awaiting a worker's Complete is non-local and gated here. A broker sets this;
	// the in-process path leaves it 0 (the executor IS the worker → every task refreshed).
	DefaultRequireWithin = 15 * time.Second
	// DefaultPollMin is the claim-poll floor (busy/hot cadence).
	DefaultPollMin = 50 * time.Millisecond
	// DefaultPollMax is the deep-idle claim-poll ceiling.
	DefaultPollMax = 30 * time.Second
)

Dispatcher default cadences — the SINGLE source of truth for these knobs. NewWithBrokerID seeds a fresh Dispatcher with them, cmd/converge's Config.applyDefaults fills the same values so the reported cluster config shows what actually runs, and the worker node overrides each from its env var only when set. Keep the two in lockstep by referencing THESE constants, never re-typing the literals.

View Source
const (
	// DefaultSweeperStaleAfter is the lease-expiry crash backstop: a claimed task
	// whose heartbeat stops for this long is reaped + requeued.
	DefaultSweeperStaleAfter = 30 * time.Second
	// DefaultSweeperUnclaimedDeleteAfter is the abandoned-UNCLAIMED-work GC window:
	// a work_queue row no worker claimed for this long is dropped.
	DefaultSweeperUnclaimedDeleteAfter = 24 * time.Hour
)

Reaper default windows — the SINGLE source of truth. NewReaper seeds them and cmd/converge's Config.applyDefaults fills the SAME values so the reported cluster config matches what runs. The control plane overrides each from its SWEEPER_* env only when set (>0). Reference THESE constants, never the literals.

View Source
const ClusterChangedChannel = "cluster_changed"

ClusterChangedChannel is the LISTEN/NOTIFY channel the cluster_members INSERT/DELETE trigger fires (gated, via notify_cluster_changed → notify_gated) when a member JOINS or LEAVES. The TopologyWatcher LISTENs on it (once, for all its reactors) to react to a membership change promptly instead of waiting out a failsafe interval. Exported so the mesh references the ONE channel name.

View Source
const DefaultMemberHeartbeatEvery = 10 * time.Second

DefaultMemberHeartbeatEvery is the cadence at which every process UPSERTs its cluster_members row (the running-fleet view). It is the ONE authoritative member-liveness cadence: DefaultMeshLivenessWindow (mesh peer eviction) and the Resharder's shard-tiling liveness window both derive from 3× this, and the API marks a member NotReady after ~3 missed beats. Short enough that a crashed member is excluded from routing + tiling within ~one window (3× = 30s), not the minutes-long ClusterMemberGC TTL; the UPSERT is a single PK write on a tiny table, so a 10s cadence is negligible even at fleet scale. cmd/converge's MEMBER_HEARTBEAT_EVERY overrides it.

View Source
const DefaultMeshLivenessWindow = 3 * DefaultMemberHeartbeatEvery

DefaultMeshLivenessWindow is how recent a member's heartbeat must be to count as LIVE for routing/tiling — 3× the heartbeat cadence, so a single slow/lost beat never drops a live member (which would churn the mesh + tiling) but a genuinely crashed one is excluded within one window. The SINGLE liveness window the mesh (peer eviction) and the Resharder (shard tiling) share, so they never disagree on who is alive. Kept well below the ClusterMemberGC TTL so an excluded member is also on its way to being reclaimed from the registry.

View Source
const KindManifestChannel = "kind_manifest_changed"

KindManifestChannel is the LISTEN/NOTIFY channel the kind_manifest sync trigger fires (payload = kind) on every operator apply. Every in-memory view of kind_manifest LISTENs on it to refresh promptly — the KindManifestCache (reactions/policy) here, and the API's schema surface (spec/config schemas) in cmd/converge; a failsafe re-read converges even if a NOTIFY is dropped. Exported so those other consumers reference the ONE channel name, not a duplicated literal.

View Source
const ManifestReloadInterval = 60 * time.Second

ManifestReloadInterval is the failsafe cadence for kind_manifest views (mirrors configReloadInterval). Exported for the API schema-surface refresher.

View Source
const NumShards = shardutil.NumShards

NumShards aliases shardutil.NumShards for downstream wiring.

Variables

This section is empty.

Functions

func AllShards

func AllShards() []int16

AllShards is the default sharding domain.

func Debounce

func Debounce(ctx context.Context, window time.Duration, raw <-chan struct{}, out chan struct{})

Debounce forwards a SINGLE coalescing wake to out once raw has been QUIET for window, collapsing a BURST of source events into one downstream action on the SETTLED state. Trailing-edge: each raw event during the window RESETS the quiet timer, so the wake fires `window` after the LAST event, not the first. Runs until ctx is cancelled. The single debounce primitive shared by the membership-reactive loops (the Resharder's re-tile, the broker mesh's peer reconcile) — a scale event fires one INSERT/DELETE per pod, and without this each would act N times mid-flux instead of once on the final topology. It can only DELAY a reaction by at most window, never drop one; a failsafe tick always backstops it independently. Exported for the broker mesh (a different package).

Types

type ClusterMemberGC

type ClusterMemberGC struct {
	Repo ClusterMemberGCRepo

	// TTL: a member unheard-from for longer than this is reclaimed.
	TTL time.Duration

	Interval     time.Duration
	IdleInterval time.Duration

	// OnSwept is an optional metrics observer (sweep, n>0). nil → no-op. Off the hot path.
	OnSwept func(sweep string, n int64)
}

ClusterMemberGC is the delayed garbage collector for the cluster_members running-fleet registry: it hard-deletes rows whose last heartbeat is older than TTL (a member that crashed, or whose clean-shutdown deregister timed out). It runs on the ControlPlane only — like the Reaper/SpecGC sweepers — and is the Postgres-side cron the user asked for, implemented as a Go-driven sweeper calling a SQL function (gc_stale_cluster_members) rather than pg_cron, which isn't enabled here.

It is the FALLBACK to a clean shutdown's own deregister (the ClusterMemberReporter deletes its row on exit). k8s-style soft liveness with delayed GC: a member that vanished without deregistering is shown NotReady in the cluster view (the API derives that from heartbeat age, ~3 missed beats) for a while BEFORE this sweeper reclaims it at the longer TTL — so a just-crashed member stays visible (greyed) instead of vanishing. Cadence invariant: beat (DefaultMemberHeartbeatEvery) ≤ NotReady/3 ≤ TTL 5m.

THIS IS UI/HISTORY CLEANUP ONLY — it is NOT a routing signal. Routing liveness (which brokers reshard onto, which mesh peers to dial) is the SHORT liveness window (3× the heartbeat cadence) the TopologyWatcher applies when it re-reads the LIVE members; a crashed member drops out of routing there, within one window, long before this GC deletes its row. The GC only decides when the greyed row disappears.

Entirely off the hot path: a coarse cadence, a single-statement DELETE over a dozens-of-rows table, no shard scoping, no work_queue/outbox coupling.

func NewClusterMemberGC

func NewClusterMemberGC(repo ClusterMemberGCRepo) *ClusterMemberGC

NewClusterMemberGC builds the sweeper with production-sane defaults: a 5m TTL (10 missed 30s beats) and a coarse 60s cadence.

func (*ClusterMemberGC) Run

func (g *ClusterMemberGC) Run(ctx context.Context) error

type ClusterMemberGCRepo

type ClusterMemberGCRepo interface {
	GCStaleClusterMembers(ctx context.Context, ttl time.Duration) (int, error)
}

ClusterMemberGCRepo is what the ControlPlane's ClusterMemberGC sweeper calls each tick: hard-delete registry rows whose heartbeat is older than ttl.

type ClusterMemberReporter

type ClusterMemberReporter struct {
	Repo ClusterMemberReporterRepo

	// Info is the static per-process payload, gathered once at construction.
	Info store.ClusterMemberInfo

	// InFlight returns the member's current in-flight task count for the beat.
	// nil on a control-only member (no dispatcher) → reported as 0.
	InFlight func() int

	// Shards, when set, is the live source of this member's owned span: each
	// beat reads its current [lo, hi] bounds and stamps Info.Shards before the
	// UPSERT, so the registry's `shards` column tracks DYNAMIC reassignment
	// (the Resharder swaps the same ShardSet). nil → Info.Shards is reported
	// as-is (static/explicit-override members, and the unit tests).
	Shards *ShardSet

	// Workers, when set, is the live source of this member's connected-worker
	// snapshot: each beat calls it and writes the result to the workers JSONB
	// column, so the cluster view tracks dumb workers connecting/dropping. A
	// broker wires it to its ConnectedWorkers snapshot; nil (control/react members
	// and the unit tests) → the column is written "[]". Distinct from the STATIC
	// Info.Config (connect_addr + runtime knobs), which never changes across beats.
	Workers func() json.RawMessage

	// Interval is the beat cadence (default 30s — 3 beats inside the API's 90s
	// NotReady window, well under the GC's 5m TTL).
	Interval time.Duration
	// contains filtered or unexported fields
}

ClusterMemberReporter is the per-PROCESS heartbeat into the cluster_members registry — the running-fleet view ("like kubectl get nodes", but each row is a cluster MEMBER: one instance of the app identified by its role). Exactly ONE runs per process (constructed in cmd/converge/main.go, NOT in the engine), so a role=all process — which runs both the control duties and the broker duty — still writes a SINGLE row.

Each beat UPSERTs the member's STATIC identity (Info: id/role/shards/ config/version/host/pid/started_at) plus the LIVE in-flight count read via the InFlight hook (a BROKER concept; nil → 0 on control/react). The UPSERT stamps last_heartbeat with the DB clock (now()), so liveness is skew-free; started_at is preserved across beats.

Off every hot path: a single PK UPSERT on a dozens-of-rows table on a 30s cadence from this one goroutine — it touches no work_queue/outbox/resources row and is never on a per-task path.

func NewClusterMemberReporter

func NewClusterMemberReporter(repo ClusterMemberReporterRepo, info store.ClusterMemberInfo) *ClusterMemberReporter

NewClusterMemberReporter builds a reporter for the given static identity. The caller sets InFlight (and may override Interval) before Start.

func (*ClusterMemberReporter) Deregister

func (r *ClusterMemberReporter) Deregister(ctx context.Context) error

Deregister deletes this member's registry row so the cluster view drops it immediately on a clean shutdown (instead of waiting out the GC TTL). Call it AFTER Stop, as the last shutdown step, with a deadline-bounded ctx: a shutting-down member must never hang waiting on the DB if there's a network blip — on error/timeout we just exit and let the ClusterMemberGC reclaim the row. Best-effort by contract; the caller logs the error.

func (*ClusterMemberReporter) Start

func (r *ClusterMemberReporter) Start(ctx context.Context)

Start fires one beat synchronously (so the row appears immediately, not after the first interval), then runs the periodic beat in a tracked goroutine until Stop. The synchronous first beat is best-effort — a failure is logged, not fatal; the next tick retries.

func (*ClusterMemberReporter) Stop

func (r *ClusterMemberReporter) Stop()

Stop cancels the beat loop and joins the goroutine. It does NOT delete the registry row — call Deregister for that. Stopping the beat FIRST guarantees no beat can re-UPSERT the row after a subsequent Deregister wins the race.

func (*ClusterMemberReporter) Trigger

func (r *ClusterMemberReporter) Trigger()

Trigger requests an immediate off-cadence beat (non-blocking, coalescing). Called by the Resharder's OnChange so a reshard's new shard span lands in the registry promptly rather than after the next Interval. Safe before Start (the buffered slot just holds the request until the loop runs).

type ClusterMemberReporterRepo

type ClusterMemberReporterRepo interface {
	UpsertClusterMember(ctx context.Context, info store.ClusterMemberInfo, inFlight int, workers json.RawMessage) error
	DeleteClusterMember(ctx context.Context, memberID string) error
}

ClusterMemberReporterRepo is what the per-process ClusterMemberReporter calls: each beat UPSERTs this member's row in the cluster_members registry (static identity plus the LIVE in-flight count and connected-worker snapshot), and a clean shutdown deletes it (deregister).

type Dispatcher

type Dispatcher struct {
	// Loop bundles the shared execution deps (Pool/Repo/Registry/BrokerID)
	// the runOne… methods read; embedded so the dispatcher reuses Repo for
	// claims + heartbeats and BrokerID/Pool for the pod's single identity.
	*Loop

	HeartbeatEvery time.Duration
	// RequireWithin bounds how stale a worker's per-task liveness attestation may be
	// for the dispatcher to keep heartbeating that task's lease. A task whose worker
	// stopped attesting (silent connection or a wedged handler on a live one) past
	// this window drops out of the refresh set, so its heartbeat_at goes stale and
	// the reaper reclaims it — making heartbeat_at attest live-WORKER execution, not
	// mere broker liveness. Kept comfortably under the reaper's StaleAfter (≥ 2×) so
	// broker-clock skew can't cross it. 0 (the in-process default, where the executor
	// IS the worker) disables the gate: every in-flight task is refreshed.
	RequireWithin time.Duration

	PollMin time.Duration
	// PollMax is the dispatcher's DEEP-IDLE backoff ceiling, shared by every
	// pair: the work_ready NOTIFY carries cascade-re-pend latency, so the poll is
	// only a failsafe sweeper. One pod-level knob (not per pair) — every pair
	// backs off to the same ceiling; a real per-pair need has never appeared.
	PollMax time.Duration
	// DrainGrace is the graceful in-flight drain window on shutdown: when Run's ctx
	// is cancelled, claiming stops but already-claimed tasks (and a broker's parked
	// dispatchStage awaiting a worker's Complete) finish for up to this long before
	// being cancelled. 0 → cancel in-flight at once with ctx.
	DrainGrace time.Duration
	// HotWindow keeps an idle dispatcher sweeping at PollMin for this long after
	// the last activity (claim/completion/wake) before the idle backoff is
	// allowed to grow toward PollMax. Bounds the cost of a coalesced/dropped
	// work_ready NOTIFY to one PollMin instead of the full failsafe backoff.
	HotWindow time.Duration

	// Shards is the pod's CURRENT claim range, read lock-free every sweep via
	// Shards.Snapshot(). A *ShardSet (not a plain []int16) so the Resharder can
	// swap the range at runtime when cluster membership changes; the dispatcher
	// picks up the new range on its next sweep with no restart.
	Shards *ShardSet

	// ClaimGate, when non-nil, is consulted per (kind, kindVersion, task_type) each
	// sweep: a (kind, kindVersion) it rejects is NOT claimed this sweep. The BROKER tier
	// sets it to "is a worker for this (kind, kindVersion) connected?", so a broker
	// never pulls work it can't deliver (the rows stay worker_id IS NULL for
	// another broker / a later poll — no buffer fill, no stranding). A pinned kindVersion
	// whose worker fleet retired parks safely (visible via the zero-worker signal),
	// never mis-routes. nil → claim always (in-process default).
	ClaimGate func(model.Kind, int) bool
	// contains filtered or unexported fields
}

Dispatcher is the pod's single work-claiming loop. It replaces the former one-goroutine-per-(kind,task_type) model: ONE goroutine round-robins the proven single-kind WorkQueueTakeBatch over every pair the pod runs, and ONE heartbeat goroutine refreshes the union of all in-flight task ids. This collapses ~3×kinds goroutines + timers + poll cadences into 1 + 1, while the claim SQL (per-kind equality probe + partition pruning + the per-kind global cap) stays byte-for-byte the hot path.

Concurrency model — single-writer claims, atomic completions:

  • The dispatcher goroutine is the ONLY INCREMENTER of every pair.inFlight and the only broker, so per-kind free-slot accounting has no overfill: limits are computed (atomic Load) and claims issued sequentially.
  • Each finishing TASK goroutine DECREMENTS its own pair (atomic Add(-1)) and Triggers a re-sweep — no completion channel to size or overflow, so the count of live kinds is unbounded. The heartbeat reads a locked snapshot of the in-flight id set.

The execution path (runOne → dispatch → runReaction → reactReconcile/ reactDelete/reactOperate, or recordFailed) dispatches off the CLAIMED row's kind/task_type, never the pair, so the one set of methods on the embedded *Loop serves every pair.

func New

func New(pool *pgxpool.Pool) *Dispatcher

New constructs a pod-scoped Dispatcher with one stable BrokerID for the whole pod (the heartbeat/reaper guard is keyed on the pod, not per kind). The reaction executor is unset here — the caller installs it with SetDispatcher before Run (a broker its remote fanout; the in-process test harness its registry executor). Set the manifest cache with SetManifests and add pairs with AddPair before Run.

func NewWithBrokerID

func NewWithBrokerID(pool *pgxpool.Pool, brokerID string) *Dispatcher

NewWithBrokerID is New with an explicit lease identity. The broker tier uses it so a broker's BrokerID (the value stamped at claim and checked by the AppendOutbox fence) is a stable, externally-chosen id rather than a fresh per-process one — letting the same logical broker be addressed across the cluster_members registry and the reaper.

func (*Dispatcher) AddPair

func (d *Dispatcher) AddPair(kind model.Kind, kindVersion int, taskType store.TaskType)

AddPair registers a (kind, kindVersion, task_type) for the dispatcher to claim (its concurrency ceiling is the pod-level d.maxParallel; the idle-backoff ceiling is the pod-level d.PollMax). Use BEFORE Run (boot wiring); for a kind that comes online after Run started, use AddPairLive.

func (*Dispatcher) AddPairLive

func (d *Dispatcher) AddPairLive(kind model.Kind, kindVersion int, taskType store.TaskType)

AddPairLive registers a (kind, kindVersion, task_type) AFTER Run has started — used when a provider's Setup, retried in the background, finally succeeds so the kind starts being claimed with NO restart. Safe to call from any goroutine: it hands the pair to the Run goroutine (the sole writer of d.pairs) over a channel and wakes it, so the next sweep includes the new pair. Never blocks past shutdown: once Run has returned (d.stopped closed) there is no drainer, so the send is abandoned rather than parking the caller forever on a full buffer.

func (*Dispatcher) Attest

func (d *Dispatcher) Attest(workIDs []uuid.UUID)

Attest records that a live worker reported the given in-flight tasks' handlers still progressing — the broker's WorkHeartbeat relay calls it (having mapped each lease_token to its work_id). It refreshes each task's last-attested time so the heartbeat keeps its lease fresh; a task NOT attested within RequireWithin drops out of the heartbeat set and the reaper reclaims it. A no-op before Run starts (inFlight nil) or for ids this pod no longer holds.

func (*Dispatcher) InFlight

func (d *Dispatcher) InFlight() int

InFlight returns the pod's last-published total in-flight task count. Safe to call from any goroutine (lock-free atomic Load); used by the ClusterMemberReporter to stamp the registry row. 0 until the first sweep.

func (*Dispatcher) PairCount

func (d *Dispatcher) PairCount() int

PairCount returns how many (kind, task_type) pairs the dispatcher is claiming. Lock-free; 0 means it claims nothing (e.g. a broker booted before any CRD was applied). For tests + the cluster view.

func (*Dispatcher) ReleaseClaims

func (d *Dispatcher) ReleaseClaims(ctx context.Context) (int64, error)

ReleaseClaims frees every work_queue row this pod still holds, so a surviving pod can re-claim the work IMMEDIATELY instead of waiting out the reaper's stale window. Call ONLY after Run has returned (the dispatcher is quiesced — no goroutine can re-claim or re-heartbeat after release): the shutdown path runs it between eng.Stop (joins Run) and Deregister. Returns the count freed. Best-effort: a DB error just falls back to the reaper backstop.

func (*Dispatcher) Run

func (d *Dispatcher) Run(ctx context.Context) error

Run drives the continuous-fill dispatcher over all registered pairs.

Invariant: each pair runs at most the pod-uniform d.maxParallel tasks concurrently, and the dispatcher keeps every pair topped up whenever it has eligible work. Each sweep:

  • walks the pairs ring from a rotating cursor (fairness), and for each pair with free slots issues the per-kind claim for up to its free count, launching one goroutine per claimed task.
  • whenever a slot frees (a finishing task decrements its pair and Triggers a wake) it loops again right away to refill — it does NOT wait for the rest of the cohort, so a slow task holds only its own slot.
  • when a FULL ring sweep claims nothing AND nothing is in flight, it cools off PollMin→PollMax; a freed slot or a `work_ready` NOTIFY (both arrive as a `wakeup` Trigger) collapses the backoff back to PollMin.

ONE heartbeat goroutine refreshes heartbeat_at for the union of all in-flight ids (across every pair), so every task's claim stays fresh for its whole lifetime regardless of which kind it is or how long it runs. Run drives the dispatcher until ctx is cancelled.

GRACEFUL DRAIN: when ctx is cancelled (SIGTERM), the claim loop, heartbeat, and listener stop (no new work is claimed), but already-claimed tasks — and, for a BROKER, the parked dispatchStage awaiting a remote worker's Complete — keep running for up to DrainGrace so in-flight work lands its result instead of being abandoned + re-dispatched. A straggler past the grace is cancelled. DrainGrace 0 → in-flight cancelled at once with ctx. Run returns once the claim loop exits AND every in-flight task goroutine has drained (wg.Wait), bounded by the drain window, so no goroutine outlives it.

DRAIN INVARIANT: the heartbeat stops at ctx cancel (hbCtx below) while tasks run on under drainCtx for up to DrainGrace WITHOUT a lease refresh. That is safe only while DrainGrace stays BELOW the reaper's StaleAfter: a task draining past StaleAfter has its lease reaped and re-dispatched to a peer while it still runs here. The epoch fence keeps the eventual write CORRECT (the stale pod's commit no-ops), so this is wasted re-execution during a rolling deploy, never a double-apply — but the whole point of the drain is to AVOID that waste, so keep DrainGrace < StaleAfter (the defaults do: 45s < the reaper's StaleAfter, and RequireWithin ≤ half of it). If an operator raises SHUTDOWN_DRAIN above StaleAfter this margin inverts. (Same shape as the reactor dispatcher's heartbeat/LifecycleStaleAfter invariant.)

func (*Dispatcher) SetDispatcher

func (d *Dispatcher) SetDispatcher(exec model.StageDispatcher)

SetDispatcher swaps the reaction executor (a broker installs its remote fanout here so reactions ship to dumb workers instead of running in-process).

func (*Dispatcher) SetManifests

func (d *Dispatcher) SetManifests(cache *KindManifestCache)

SetManifests wires the kind_manifest cache the dispatch reads to select each claimed task's reactions (by Trigger+Emits). Required before Run — the reaction engine is the only execution path. Call before Run.

func (*Dispatcher) SetMaxParallel

func (d *Dispatcher) SetMaxParallel(n int)

SetMaxParallel sets the pod's UNIFORM per-pair concurrency ceiling. Call once at boot (the broker does) before Run; the value applies to every pair the dispatcher serves, including those added live. 0 leaves it unbounded per pair (the SQL global cap still applies).

func (*Dispatcher) Trigger

func (d *Dispatcher) Trigger()

Trigger wakes the dispatcher from its idle backoff (e.g. a same-pod cascade re-pend). Non-blocking and coalescing.

type DispatcherRepo

type DispatcherRepo interface {
	// Claim / lease lifecycle.
	WorkQueueTakeBatch(ctx context.Context, kind model.Kind, kindVersion int, taskType store.TaskType, brokerID string, limit int, shards []int16) ([]store.WorkTask, error)
	WorkQueueHeartbeat(ctx context.Context, ids []uuid.UUID, epochs []int64, lo, hi int16) error
	WorkQueueReleaseBroker(ctx context.Context, brokerID string) (int64, error)
	WorkQueueIncrementAttempts(ctx context.Context, id uuid.UUID, shardID int16) error

	// Reaction reads + rollup gate.
	GetComposedGen(ctx context.Context, id uuid.UUID) (int64, error)
	ListDescendants(ctx context.Context, rootID uuid.UUID) ([]store.ResourceRow, error)
	DescendantsSettled(ctx context.Context, rootID uuid.UUID) (bool, error)
	GetOperation(ctx context.Context, id uuid.UUID) (store.ResourceOperation, error)

	// Writes.
	AppendOutbox(ctx context.Context, a store.OutboxAppend) error
	EmitEvent(ctx context.Context, e store.Event) error
	ApplyComposeResult(ctx context.Context, parentID uuid.UUID, parentGeneration int64, workID uuid.UUID, claimEpoch int64, manifestVersion int64, children []model.ChildSpec, edges []model.DepEdge, configs []model.ProviderConfigSpec, pol store.ComposePolicy) (store.ComposeCounts, error)
	ArmScheduleRecheck(ctx context.Context, candidateIDs []uuid.UUID) error
	AnalyzeComposeTables(ctx context.Context) error

	// WithTx returns a DispatcherRepo whose queries run inside tx (the compose
	// fence). The concrete store's WithTx returns *store.Store; dispatcherRepo
	// (loop.go) adapts it to this interface.
	WithTx(tx pgx.Tx) DispatcherRepo
}

DispatcherRepo is the data surface the work-dispatch Loop uses — the full set of reads/writes a task's lifecycle touches (claim/heartbeat/release, compose apply, outbox append, event/trace emit, rollup-readiness gate). WithTx returns a DispatcherRepo bound to tx so the compose fence stays a single transaction. *store.Store satisfies it (see the conformance block below), but the Loop depends only on this interface — the dispatch core carries no concrete store or pgx pool.

func NewDispatcherRepo

func NewDispatcherRepo(s *store.Store) DispatcherRepo

NewDispatcherRepo wraps a store as a DispatcherRepo for the work-dispatch Loop. The root builds the store once and hands the Loop this interface, so the dispatch core carries no concrete store or pgx pool.

type Drainer

type Drainer struct {
	Repo      DrainerRepo
	Listener  Listener // outbox_ready + rollup_recheck wake subscriptions
	Interval  time.Duration
	BatchSize int

	IdleInterval time.Duration
	HotWindow    time.Duration

	// Shards is the control pod's CURRENT drain range, read lock-free each tick
	// via Shards.Snapshot(). A *ShardSet so the Resharder can swap it at runtime
	// on membership change; each band recomputes its sub-slice from the new
	// snapshot on its next tick (snapshot-pointer-cached so an unchanged range
	// costs one pointer compare).
	Shards      *ShardSet
	WorkerCount int
}

Drainer pops work_outbox rows in batches and applies them to resources + work_queue via the plpgsql drain_outbox_batch function.

func NewDrainer

func NewDrainer(repo DrainerRepo, listener Listener) *Drainer

func (*Drainer) Run

func (d *Drainer) Run(ctx context.Context) error

type DrainerRepo

type DrainerRepo interface {
	DrainOutboxBatch(ctx context.Context, limit int, shards []int16) (int, error)
	// DrainRollupRechecks re-pends, at a fresh snapshot, the straddled rollup
	// roots the cascade queued in rollup_recheck — the deferred half that closes
	// the rollup straddle race. Empty + cheap on the common no-straddle path.
	DrainRollupRechecks(ctx context.Context, limit int, shards []int16) (int, error)
	// DrainScheduleRechecks schedules a band-scoped batch of the children a wide
	// compose armed in schedule_recheck (OPT-A) — the deferred, drain-paced
	// replacement for the post-commit ScheduleEligible that deadlocked. Empty +
	// cheap once a compose's children are all scheduled.
	DrainScheduleRechecks(ctx context.Context, limit int, shards []int16) (int, error)
}

DrainerRepo is what the Drainer calls each tick. The cascade trigger in 00001_schema.sql handles downstream scheduling inside the drainer transaction, so the Go side has no per-batch follow-up work.

type KindManifestCache

type KindManifestCache struct {
	// contains filtered or unexported fields
}

KindManifestCache holds every kind's FULL model.KindManifest in memory, keyed by (kind, kind_version), so the DISPATCH + COMPOSE paths resolve a manifest with a LOCK-FREE read (an atomic pointer load + map index), not a per-task mutex or DB round-trip. Consumers pull the SLICE they need out of the returned manifest:

  • the dispatch loop reads its REACTIONS (which reaction to run for a task);
  • the composer/reaper read its POLICY (OrphanGraceForKind / FinalizerForKind).

It is NOT the API's schema-validation view: a resource apply is validated against the schemas by a SEPARATE surface on the API pod (cmd/converge's rebuildKindSchemaSurface), which reads the same kind_manifest rows for their spec/config JSON Schemas. Two independent views of one table, each for its own consumer; both stay current via the same KindManifestChannel.

Refresh (multi-pod convergent — same discipline as ProviderConfigCache):

  • loaded fully at boot (Load) so the first reconcile already has its kind;
  • refreshed on a kind_manifest_changed NOTIFY (prompt) AND a 60s failsafe re-read (a dropped NOTIFY still converges) — the shared NotifyRefresher;
  • DB-read-on-MISS (Get falls back to a single GetKindManifest), so a brand- new kind applied between refreshes is never silently treated as absent.

The published snapshot is immutable; every change swaps a fresh one in via atomic.Pointer. Manifests change only on control-plane events (a CRD apply), so the copy-on-write cost is rare while the per-task read is contention-free — the hot path (kind already resolved) takes no lock and does no DB read.

func NewKindManifestCache

func NewKindManifestCache(pool *pgxpool.Pool) *KindManifestCache

NewKindManifestCache builds the cache over the pool's store.

func (*KindManifestCache) All

All returns every manifest currently in the cache (a fresh slice over the immutable snapshot — lock-free). A broker uses it to register the claim pairs of already-applied kinds at boot; kinds applied later arrive via the SetOnReplace reload callback. Order is unspecified.

func (*KindManifestCache) FinalizerForKind

func (c *KindManifestCache) FinalizerForKind(kind model.Kind, kindVersion int) string

FinalizerForKind returns a (kind, kindVersion)'s finalizer from its manifest (empty if none). Satisfies store.ComposePolicy. nil-safe.

func (*KindManifestCache) Get

func (c *KindManifestCache) Get(ctx context.Context, kind model.Kind, kindVersion int) (model.KindManifest, bool)

Get returns a kind's manifest with a LOCK-FREE read on the hot path: an atomic pointer load + a map index, no mutex, no DB. On a cache MISS for a never-checked kind it does a single DB read and copy-on-write-fills the entry (so a just- applied kind is never wrongly reported absent). ok=false means the kind has no model. Safe for concurrent use.

func (*KindManifestCache) Load

func (c *KindManifestCache) Load(ctx context.Context) error

Load reads every manifest into the cache synchronously (boot). A load error is returned so the caller can decide whether to proceed (the engine can run with an empty cache and miss-fill per kind, so a transient boot error is non-fatal).

func (*KindManifestCache) OrphanGraceForKind

func (c *KindManifestCache) OrphanGraceForKind(kind model.Kind, kindVersion int) time.Duration

OrphanGraceForKind returns a (kind, kindVersion)'s orphan-grace window from its manifest (0 if no manifest / no grace). Satisfies store.ComposePolicy so ApplyComposeResult can decide soft-vs-hard prune. nil-safe.

func (*KindManifestCache) SetOnReplace

func (c *KindManifestCache) SetOnReplace(fn func([]model.KindManifest))

SetOnReplace registers a callback fired with the full manifest set after each whole-set refresh (Load + every reload). Set before Start.

func (*KindManifestCache) Start

func (c *KindManifestCache) Start(ctx context.Context)

Start launches the kind_manifest_changed listener + failsafe reload loop (the shared NotifyRefresher): the listener's onReady catch-up closes the subscribe- vs-apply race (a manifest inserted between boot Load and the LISTEN taking effect), the NOTIFY makes an operator's apply prompt, and the 60s failsafe converges even a dropped NOTIFY.

func (*KindManifestCache) Stop

func (c *KindManifestCache) Stop()

Stop cancels the listener + loop and joins them.

type Lease

type Lease struct {
	WorkID   uuid.UUID
	ShardID  int16
	BrokerID string
	// KindVersion is the claimed row's work_queue.kindVersion — denormalized from the
	// NOT-NULL resources.kind_version at schedule time, so it is always >= 1 (there is
	// no normalize-to-1 step). The broker reads it to route the resulting StageTask to a
	// worker/peer advertising the EXACT (kind, kindVersion); in-process executors ignore it.
	KindVersion int
	// ClaimEpoch is the strict monotonic fencing token this claim carries
	// (work_queue.claim_epoch). It rides the StageTask to the worker and is echoed on
	// the result; the fenced result write (AppendOutbox / StampComposedGen) lands only
	// if the row still bears this epoch. The broker carries it here so the fenced write
	// is owner-independent — the epoch, not the broker identity, is the gate.
	ClaimEpoch int64
	// ManifestVersion is the kind_manifest content hash this claim was enqueued under
	// (work_queue.manifest_version), carried so the compose/result fence checks it
	// without re-reading the lease. 0 = pre-manifest (inert).
	ManifestVersion int64
}

Lease carries a claimed task's work_queue identity — the row id, its shard, the lease-holder's worker_id, and the web-API KIND VERSION it was claimed for — so a remote StageDispatcher (the broker fanout) can scoped-release the DB lease the instant it abandons the task (WorkID/ShardID/BrokerID) AND stamp the task's kindVersion on the wire for STRICT (kind, kindVersion) routing (KindVersion). It rides on ctx (set by runOne via WithLease), NOT in the pure manifest.StageDispatcher signature, so the SDK-facing seam stays free of work_queue internals.

func LeaseFromContext

func LeaseFromContext(ctx context.Context) (Lease, bool)

LeaseFromContext returns the task's lease identity if present (set by the dispatcher's runOne). The broker's fanout reads it to scoped-release on abandonment; ok is false on the in-process path (no remote lease to release).

type Listener

type Listener interface {
	// Listen runs a resilient LISTEN loop on channel until ctx is cancelled,
	// calling onNotify for each notification and reconnecting (with backoff) on
	// any connection error. name tags the reconnect log line. onReady (optional;
	// pass 0 or 1) fires ONCE per successful (re)subscribe, right after the LISTEN
	// registers and BEFORE any notification is awaited — callers use it for a
	// catch-up read that closes the subscribe-vs-read race.
	Listen(ctx context.Context, name, channel string, onNotify func(), onReady ...func())
}

Listener is the resilient LISTEN/NOTIFY capability the runtime's reactive duties (drainer, reactor dispatcher, resharder) depend on — the ONE Postgres-facing dependency that isn't already behind a Repo role interface. Injecting it (rather than a concrete *pgxpool.Pool) keeps those duties testable against a fake wake source and keeps pgx out of their signatures; the production implementation is PgxListener, a thin adapter over the pool.

It is a pure LATENCY optimisation over each caller's failsafe poll: a window with no listener just means wakes fall back to the periodic tick, never a stuck task — so Listen never returns an error and exits cleanly only on ctx cancel.

type Loop

type Loop struct {
	// Tx opens the compose-fence transaction (ApplyComposeResult in one tx via
	// Repo.WithTx). A narrow txBeginner, not the concrete pool — the dispatch
	// core depends on the transaction capability, not the substrate.
	Tx       txBeginner
	Repo     DispatcherRepo
	Listener Listener // work_ready wake subscription
	BrokerID string

	// Dispatch runs a kind's reaction handler: a broker ships it to a dumb worker
	// via its remote fanout executor; the integration test harness injects an
	// in-process executor (test/internal/inproc) that runs handlers directly from a
	// ReactionRegistry. The DB reads/writes around every reaction (gate reads,
	// ApplyComposeResult, ListDescendants, AppendOutbox, …) stay on this
	// lease-holder. nil → the fail-closed noHandlerExecutor (a task fails
	// transiently and re-arms; production always installs the fanout via SetDispatcher).
	Dispatch model.StageDispatcher

	// Manifests is the kind_manifest cache: dispatch reads a kind's reactions
	// from it (selected by Trigger+Emits) to drive the reaction engine. Required —
	// a kind with no manifest can't be dispatched (the task fails loudly).
	Manifests *KindManifestCache

	// MarkLocal, when set, flips a task's in-flight record to local at the moment this
	// pod's OWN goroutine starts the post-Complete apply phase (ApplyComposeResult) — so
	// the long local apply is exempt from the worker-attestation staleness gate and can't
	// reap its own lease mid-apply. Until then the task is worker-attested (kept fresh by
	// the remote worker's WorkHeartbeat), so a never-delivered task correctly lapses and
	// the reaper re-dispatches it. Wired by the Dispatcher to inFlight.markLocal; nil on
	// the in-process test path (whose staleness gate is disabled anyway).
	MarkLocal func(id uuid.UUID)

	// RemoteFanoutCeiling bounds a task whose kind set NO TaskDeadline (the common
	// default) when the executor is REMOTE (a broker fanning out to a dumb
	// worker): such a task otherwise parks the dispatchStage goroutine + its
	// pair.inFlight slot indefinitely if the worker wedges-but-stays-connected (the
	// stream never drops, so the abandon-on-disconnect path can't fire and the DB
	// heartbeat keeps the lease un-reapable). A broker sets this to a generous
	// ceiling (well above the slowest legitimate handler) so a wedged remote task
	// is eventually cancelled via the proven WithTimeout/orphan path below and its
	// slot freed. 0 (the default, and the in-process executor) = unbounded, exactly
	// as before — the in-process dispatch is synchronous and frees its own slot, so
	// it never needs this.
	RemoteFanoutCeiling time.Duration
}

Loop is the execution handle for the dispatch path. With the unified dispatcher it is no longer a per-pair goroutine driver — it just bundles the shared deps the runOne… methods read. Kept as the method receiver so the (large, well-tested) execution path is untouched.

The wakes the dispatcher reacts to (work_ready) and emits indirectly are fired ENTIRELY in the DB, so no Go-side notifier lives here (a Go-side per-completion pg_notify storm was an earlier regression). The two channels use DIFFERENT mechanisms on purpose:

  • outbox_ready: per-row notify_gated in AppendOutbox — globally rate-limited via the wake_state advisory lock (a result-drain wake can tolerate coalescing).
  • work_ready: a DIRECT, intentionally UNGATED pg_notify in schedule_eligible / the cascade + resync paths. Gating it was actively harmful: notify_gated's pg_try_advisory_XACT_lock is held to COMMIT, so a wake fired inside the drainer's seconds-long drain tx was lost, stranding rollups.

type NotifyRefresher

type NotifyRefresher struct {
	// contains filtered or unexported fields
}

NotifyRefresher is the shared "keep an in-memory view convergent with a DB table" driver: it runs a supplied reload on a Postgres LISTEN/NOTIFY edge (prompt) AND on a periodic failsafe tick (so a dropped NOTIFY still converges within the interval). It is the mechanism every reactive cache in the runtime uses to stay current across a multi-pod deployment — the caller owns WHAT it caches and HOW reload rebuilds it; this owns only the resilient refresh loop (subscribe, catch-up, coalesce, tick, reconnect-via-Listener).

The contract mirrors the Listener it wraps: the NOTIFY is a pure latency optimisation over the failsafe interval. A window with no listener (or a lost NOTIFY) just means the reload falls back to the next tick — never a stuck view. reload must be idempotent and tolerate being called concurrently with nothing else (it runs on a single dedicated goroutine, serialized).

Refresh guarantees, once Start returns:

  • an IMMEDIATE reload, so a change made between the caller's boot load and Start is observed at once (not after the first interval);
  • a reload on every LISTEN (re)subscribe (onReady catch-up), closing the subscribe-vs-change race and re-syncing after a listener reconnect;
  • a reload on every NOTIFY on channel (coalesced — a burst folds into one);
  • a reload every interval regardless, the authoritative backstop.

func NewNotifyRefresher

func NewNotifyRefresher(listener Listener, name, channel string, interval time.Duration, reload func(ctx context.Context)) *NotifyRefresher

NewNotifyRefresher builds a refresher that calls reload on a `channel` NOTIFY and every `interval`. name tags the listener's reconnect log line. A non-positive interval falls back to a safe default so a mis-wired caller still converges.

func (*NotifyRefresher) Start

func (r *NotifyRefresher) Start(ctx context.Context)

Start launches the LISTEN subscription + failsafe loop. Both run until Stop (or the parent ctx) cancels. Call once; not re-entrant.

func (*NotifyRefresher) Stop

func (r *NotifyRefresher) Stop()

Stop cancels the listener + loop and joins them, releasing the held LISTEN connection before the pool closes.

type PgxListener

type PgxListener struct {
	// contains filtered or unexported fields
}

PgxListener is the production Listener: a resilient LISTEN/NOTIFY loop over a pgx pool. Every runtime component that reacts to a DB NOTIFY shares this one implementation: the drainer (outbox_ready), the reactor dispatcher (lifecycle_ready), and the resharder (cluster_changed).

func NewPgxListener

func NewPgxListener(pool *pgxpool.Pool) *PgxListener

NewPgxListener wraps a pool as a Listener. The pool is the substrate — a LISTEN registration lives on a session, so each Listen call holds one dedicated pooled connection for its lifetime.

func (*PgxListener) Listen

func (l *PgxListener) Listen(ctx context.Context, name, channel string, onNotify func(), onReady ...func())

Listen implements Listener over the pgx pool: it re-establishes the subscription on any connection error until ctx is cancelled.

type ProviderConfigCache

type ProviderConfigCache struct {

	// OnDefaultChange, if set, is invoked (once) after a reload in which at least
	// one default config CHANGED (spec OR data). Called from the plane's loop
	// goroutine, so it must not block.
	OnDefaultChange func()

	// OnKindConfigChange, if set, is invoked ONCE PER (kind, kindVersion) whose default
	// providerconfig changed in a reload — on EITHER axis (spec or bundle) — with the
	// (kind, kindVersion) and its NEW spec + data (nil = that half absent/deleted). A
	// providerconfig is a MONOLITH of {spec, data}, so it changes and pushes as one: the
	// broker wires this to push the whole thing down every connected worker serving that
	// exact (kind, kindVersion) (the PUSH half of worker config delivery), so an
	// operator's edit reaches workers immediately rather than on their poll — and a v2
	// edit never lands on a v1 worker. Called from the plane's loop goroutine, so it must
	// not block.
	OnKindConfigChange func(kind model.Kind, kindVersion int, spec, data []byte)
	// contains filtered or unexported fields
}

ProviderConfigCache is the runtime half of RUNTIME-EDITABLE CONFIG. It owns the ONE authoritative copy of each (kind, kindVersion)'s DEFAULT config — the providerconfigs row with kind=K AND kindVersion=M AND is_default — loaded once at boot (so a worker's boot GetProviderConfig sees it) and kept current thereafter: LISTENing on providerconfig_changed (plus a failsafe re-read) it refreshes its in-memory map whenever an operator edits a default. No restart, no env var per cloud — the Crossplane ProviderConfig model, now VERSIONED per web-API kindVersion.

Delivery is PULL, not push: the broker serves cp.Config(kind, kindVersion) / cp.Bundle(kind, kindVersion) over GetProviderConfig, and re-reads through a single RLock, so a dumb worker always observes the plane's current per-kindVersion document. The plane's only job is to keep `current` fresh and fire the per-(kind, kindVersion) change callbacks the broker wires to its live worker pushes.

Bootstrap vs non-bootstrap is a per-PROVIDER distinction, not a plane one: the document carries both tiers (same structure), and Setup decides which fields are boot-critical (a missing broker URL can fail Setup → the pod won't start) vs work-time-only (a missing topic boots fine and only a task fails). The plane just owns the document.

PER-KIND-VERSION keying: `current` is keyed by (kind, kindVersion) because a default is now per-(kind, kindVersion). A single kind that publishes v1 AND v2 holds two entries, so a v2 worker's GetProviderConfig(vpc, 2) never sees v1's document and vice-versa.

func NewProviderConfigCache

func NewProviderConfigCache(ctx context.Context, pool *pgxpool.Pool, kinds []model.Kind) *ProviderConfigCache

NewProviderConfigCache builds the plane and SYNCHRONOUSLY loads each tracked (kind, kindVersion)'s boot config, so a reader observes it immediately. Call Start afterwards for live refresh. A non-empty kinds list tracks exactly those at kindVersion 1 (a kind-scoped pod / test); an EMPTY/nil list is DYNAMIC — every reload discovers the live (kind, kindVersion) set from the DB (the broker's mode: it carries no providers, so it serves worker configs for whatever pairs have an applied manifest, including a post-boot v2). A per-(kind, kindVersion) load error at boot is logged and leaves that pair with a nil document.

func (*ProviderConfigCache) Bundle

func (s *ProviderConfigCache) Bundle(kind model.Kind, kindVersion int) []byte

Bundle returns (kind, kindVersion)'s current default BUNDLE bytes (the opaque provider data — e.g. a zip of Starlark .star files), or nil if absent. Live, like Config.

func (*ProviderConfigCache) Config

func (s *ProviderConfigCache) Config(kind model.Kind, kindVersion int) json.RawMessage

Config returns (kind, kindVersion)'s current default config SPEC document, or nil if that pair has no providerconfig. Live: a later call reflects an operator's edit. The broker reads it to serve GetProviderConfig / build a push.

func (*ProviderConfigCache) Start

func (cp *ProviderConfigCache) Start(ctx context.Context)

Start launches the providerconfig_changed listener + failsafe reload loop (the shared NotifyRefresher): the listener handles prompt edits, the interval catches a missed NOTIFY, and an immediate reload observes a default created between the boot snapshot (NewProviderConfigCache) and Start.

func (*ProviderConfigCache) Stop

func (cp *ProviderConfigCache) Stop()

Stop cancels the listener + loop and joins them, releasing the held LISTEN connection before the pool closes.

type ReactorDispatcher

type ReactorDispatcher struct {
	Repo     ReactorRepo
	Listener Listener // lifecycle_ready wake subscription
	// Dispatch is the seam that runs a kind's lifecycle reaction. In production a
	// broker installs its remote executor that ships the reaction to a worker; the
	// integration test harness injects an in-process executor (test/internal/inproc)
	// that runs handlers directly from a ReactionRegistry. Nil falls back to the
	// fail-closed noHandlerExecutor via dispatcher() — a delivery then fails
	// transiently and is retried, never lost.
	Dispatch model.StageDispatcher
	BrokerID string

	Interval     time.Duration
	IdleInterval time.Duration
	HotWindow    time.Duration
	// DrainGrace is the graceful in-flight drain window on shutdown: when Run's ctx
	// is cancelled, claiming stops but in-flight deliveries finish for up to this
	// long before cancellation. 0 → cancel in-flight at once with ctx.
	DrainGrace time.Duration
	BatchSize  int

	// MaxParallel bounds concurrent reactor invocations per tick — an external
	// upload/POST holds a goroutine, so cap the blast radius (the high-risk
	// outbound path). 0 → a sensible default.
	MaxParallel int

	// HeartbeatEvery refreshes this pod's in-flight claims so a slow reactor
	// (a long upload/POST) isn't reaped mid-delivery and double-fired. Pairs with
	// the Reaper's short LifecycleStaleAfter: the stale window must exceed this
	// cadence. 0 → a sensible default.
	HeartbeatEvery time.Duration

	// Shards is the pod's CURRENT claim range, read lock-free each tick via
	// Snapshot(); the Resharder swaps it on membership change.
	Shards *ShardSet

	// ReadyToDispatch, when set, reports whether a reactor KIND can be delivered
	// right now — on a broker that fans deliveries out to dumb workers, this is
	// the fanout's HasSubscriber, so the dispatcher skips a delivery whose reactor
	// kind has NO connected worker instead of parking it (which would block the
	// per-delivery goroutine until one connects). A skipped delivery is left
	// claimed; its heartbeat goes stale and the reaper re-arms it once a worker is
	// up (at-least-once). nil → always ready (the in-process executor case, where a
	// missing handler surfaces as a Terminal error instead).
	ReadyToDispatch func(kind model.Kind) bool
	// contains filtered or unexported fields
}

ReactorDispatcher drains lifecycle_outbox: it claims due transitions that match an enabled binding, runs the kind's lifecycle reaction through the shared model.StageDispatcher (the SAME seam the main engine uses), and acks (deletes) the row on success. It is a structural clone of the Drainer — listen('lifecycle_ready') + pollLoop failsafe — so it inherits the same wake/backstop characteristics:

  • reactive: a committed transition fires lifecycle_ready; the dispatcher wakes and claims within a window.
  • durable: lifecycle_outbox is LOGGED, so a dropped NOTIFY costs at most one IdleInterval (the poll backstop), never the reaper's StaleAfter.
  • at-least-once: a delivery that errors is NOT acked; its claimed row's heartbeat goes stale and reap_stale_lifecycle (on the Reaper tick) frees it for another attempt.

Sharded with NO leader: every react pod runs one, claiming its shard range (shard_id BETWEEN lo AND hi). A resource's transition lands on the same pod-range that drains its work (same shard_of hash), so there's no cross-pod handoff; FOR UPDATE SKIP LOCKED makes a membership-overlap window safe.

func NewReactorDispatcher

func NewReactorDispatcher(repo ReactorRepo, listener Listener, dispatcher model.StageDispatcher, brokerID string) *ReactorDispatcher

NewReactorDispatcher returns a dispatcher with the Drainer's tuned cadence: 25ms hot poll, 5s hot window, 2s cold failsafe. BatchSize is small because a delivery does external I/O (unlike a cheap DB drain), so we keep the per-tick claim's lock-hold short and let the work-conserving re-poll burst a backlog.

func (*ReactorDispatcher) Run

func (d *ReactorDispatcher) Run(ctx context.Context) error

Run drives the reactor until ctx is cancelled.

GRACEFUL DRAIN (mirrors the work dispatcher): when ctx is cancelled the poll loop / listener / heartbeat stop claiming new lifecycle deliveries, but in-flight deliveries (a worker running the lifecycle reaction, awaiting its Complete) finish for up to DrainGrace before being cancelled — so a graceful broker shutdown doesn't abandon a reaction a worker is mid-way through. DrainGrace 0 → in-flight cancelled at once with ctx.

type ReactorRepo

type ReactorRepo interface {
	ClaimReactorDeliveries(ctx context.Context, brokerID string, limit int, shards []int16) ([]store.ReactorDelivery, error)
	AckReactorDelivery(ctx context.Context, resourceID uuid.UUID, transition string, generation int64, bindingName string, claimEpoch int64) error
	// HeartbeatReactorClaims refreshes heartbeat_at on the SPECIFIC in-flight deliveries
	// the dispatcher holds (keyed by identity + the claim_epoch each was claimed under)
	// so a slow reactor isn't reaped mid-delivery. Epoch-fenced like the work
	// dispatcher's heartbeat — a row a concurrent claim/reap re-issued (bumped epoch)
	// drops out, so the refresh never 40P01-cycles with the re-claimer. Scoped to the
	// pod's shard range for partition pruning.
	HeartbeatReactorClaims(ctx context.Context, keys []store.ReactorLeaseKey, shards []int16) error
}

ReactorRepo is what the ReactorDispatcher calls each tick: claim due lifecycle transitions with matching bindings, and ack a delivered one. The claim_reactor_deliveries plpgsql function does the FOR UPDATE SKIP LOCKED + binding JOIN + resource/config resolution IN-DB, so the Go side just dispatches to the reactor and acks.

type Reaper

type Reaper struct {
	Repo       ReaperRepo
	StaleAfter time.Duration
	RetryAfter time.Duration
	Interval   time.Duration
	BatchSize  int

	// LifecycleStaleAfter is the stale-claim window for lifecycle_outbox reactor
	// deliveries — its own knob so a failed or crashed delivery re-arms for
	// another at-least-once attempt. Safe to be short because the
	// ReactorDispatcher heartbeats its in-flight claims (HeartbeatEvery), so a
	// legitimately slow reactor keeps its lease; only a truly dead/failed
	// delivery goes stale. Must exceed the dispatcher's HeartbeatEvery.
	LifecycleStaleAfter time.Duration

	// UnclaimedDeleteAfter is the GC window for abandoned work: a work_queue row
	// that has sat UNCLAIMED (worker_id NULL) this long is hard-deleted, because
	// nothing ever picked it up — its kind has no worker (provider removed /
	// decommissioned). Distinct from StaleAfter, which RECOVERS a stale CLAIMED
	// row (worker died mid-task); this DROPS a never-claimed one. Deliberately
	// long (default 24h) so a transiently-down fleet has ample time to come back
	// and drain its backlog before anything is reclaimed. 0 disables the pass.
	UnclaimedDeleteAfter time.Duration

	IdleInterval time.Duration

	// OrphanSweepEvery throttles the orphan-grace sweep: it runs every Nth tick,
	// not every tick. frozen_until is an ABSOLUTE deadline baked at stamp time,
	// so a coarse cadence only delays a grace-expired teardown by at most one
	// sweep interval — fine for a window measured in tens of seconds to minutes,
	// and it keeps the sweep's per-tick lock-manager footprint off the hot drain
	// path at 1M scale (where LockManager is already the dominant wait). Default
	// 6 → with Interval=5s that's a sweep ~every 30s. 0/1 = every tick.
	OrphanSweepEvery int

	// Shards is the control pod's CURRENT sweep range, read lock-free each tick
	// via Shards.Snapshot(). A *ShardSet so the Resharder can swap it at runtime
	// on membership change; the reaper picks up the new range on its next tick.
	Shards *ShardSet

	// OnSwept is an optional metrics observer: called with (sweep, n) after a sweep
	// that acted on n rows (n>0). nil → no-op (the generic reaper stays metrics-free;
	// the engine wires this to the OTel sweeper counter). Off the hot path.
	OnSwept func(sweep string, n int64)
	// contains filtered or unexported fields
}

Reaper drives the two periodic Postgres calls the trigger-based scheduler can't do on its own. Same shape as today's reaper.WorkReaper.

func NewReaper

func NewReaper(repo ReaperRepo) *Reaper

func (*Reaper) Run

func (r *Reaper) Run(ctx context.Context) error

type ReaperRepo

type ReaperRepo interface {
	ReapStaleWork(ctx context.Context, staleAfter time.Duration, limit int, shards []int16) (int, error)
	RequeueFailedAndPending(ctx context.Context, retryAfter time.Duration, limit int, shards []int16) (int, error)
	// RecountInflight rebuilds the per-kind concurrency-cap tally for the
	// reaper's shard range — the self-healing edge of the cap.
	RecountInflight(ctx context.Context, shards []int16) error
	// ReapStaleLifecycle frees lifecycle_outbox claims whose dispatcher stopped
	// heartbeating — the at-least-once backstop for reactor deliveries (the
	// dispatcher's own poll is the primary path).
	ReapStaleLifecycle(ctx context.Context, staleAfter time.Duration, limit int, shards []int16) error
	// DeleteUnclaimedWork hard-deletes work_queue rows unclaimed past deleteAfter
	// — abandoned tasks no worker ever picked up. GC, not recovery: a returning
	// worker's fleet gets a fresh row from requeue_failed_and_pending.
	DeleteUnclaimedWork(ctx context.Context, deleteAfter time.Duration, limit int, shards []int16) (int, error)
	// SweepExpiredOrphans tears down composer-dropped children whose orphan-grace
	// window elapsed without a re-emit (frozen_until < now()): finalizer kinds →
	// soft-delete, leaf kinds → hard delete. The deadline is baked into
	// frozen_until by the composer, so there is no duration arg.
	SweepExpiredOrphans(ctx context.Context, limit int, shards []int16) (int, error)
	// SweepDeletable is the level-triggered backstop for reverse-dependency cascade
	// delete: hard-deletes any deletion_requested row whose finalizers are empty AND
	// that has no remaining owned child AND no remaining dependent, so a marked
	// teardown tree collapses bottom-up (a parent waits for its children/dependents).
	SweepDeletable(ctx context.Context, limit int, shards []int16) (int, error)
}

ReaperRepo is what the Reaper calls each tick.

type Resharder

type Resharder struct {
	Repo   ResharderRepo
	Shards *ShardSet

	// MemberID is THIS process's cluster_members id — the row the assignment is
	// computed for. Must match the ClusterMemberReporter's MemberID so the member
	// ranks itself within its registered role.
	MemberID string

	// Total is the shard-space cardinality the range partitions (NumShards).
	Total int

	// LivenessWindow bounds how recent a peer's heartbeat must be to count as a live
	// member in the tiling — the SAME window the mesh uses for peer routing (both
	// set from the TopologyWatcher), so shard ownership and mesh routing never
	// disagree on who is live. Must be ≫ the member-heartbeat cadence so a slow beat
	// never drops a live peer, and ≤ the GC TTL so an excluded member is also on its
	// way to being reclaimed.
	LivenessWindow time.Duration
}

Resharder is the per-PROCESS dynamic-sharding COMPUTE unit: given the live cluster_members view it computes this member's contiguous shard range (store. AssignMemberShards → assign_member_shards(), which ranks this member among the LIVE members of its OWN role and range-partitions [0,total)) and atomically swaps it into the ShardSet the drivers read. It carries NO loop of its own — the TopologyWatcher drives it (via Reactor) on the shared cluster_changed wake + failsafe poll, alongside the mesh, so both react to the same membership snapshot.

CORRECTNESS WITHOUT CONSENSUS: every node computes its OWN range from the same shared registry with the same deterministic math, so in steady state the ranges tile [0,total) with no overlap and no gap. DURING a membership change different nodes observe it at slightly different instants and may briefly compute overlapping ranges — SAFE: work_queue claims are FOR UPDATE SKIP LOCKED (two pods racing a shard take disjoint rows), the drainer is idempotent, and the next tick reconciles everyone onto the new tiling. The explicit "transient overlap is fine" design — no lock, no leader.

func NewResharder

func NewResharder(repo ResharderRepo, memberID string, shards *ShardSet) *Resharder

NewResharder builds a Resharder for the given member id and the ShardSet the drivers read. Total defaults to NumShards; the caller sets LivenessWindow (from the TopologyWatcher's shared window) before registering the reactor.

func (*Resharder) Reactor

func (r *Resharder) Reactor() TopologyReactor

Reactor returns this resharder's TopologyReactor — the tick the TopologyWatcher runs on every cluster_changed wake + failsafe poll. It recomputes this member's shard span and swaps it into the ShardSet, reporting whether the span CHANGED (so the watcher can fire the resharder's onChange — wired to the reporter's Trigger so the new span lands in the registry promptly).

type ResharderRepo

type ResharderRepo interface {
	AssignMemberShards(ctx context.Context, memberID string, total int, livenessWindow time.Duration) (lo, hi int16, ok bool, err error)
}

ResharderRepo is what the per-process Resharder calls each tick: compute this member's contiguous shard span from the live cluster_members view (its rank among same-role live members × range-partition of [0,total)). ok=false means the member owns nothing this tick (not yet registered, beat aged out, or more live members than shards).

type ResyncKind

type ResyncKind struct {
	Interval  time.Duration
	Recompose bool
}

ResyncKind is the per-kind resync configuration the sweeper carries: how often to re-pend, and whether the re-pend also forces a composer re-run (drift-correcting composed children). Built from kind_config by the control plane.

type Resyncer

type Resyncer struct {
	Repo ResyncerRepo

	// Interval is how often the sweeper wakes to look for due rows. It
	// is the resolution of resync timing, not the interval itself — a
	// kind with ResyncInterval=5m is swept on whichever tick first finds
	// its rows older than 5m. Kept coarse so the due-row scan (one
	// indexed query per kind) is cheap relative to real work.
	Interval  time.Duration
	BatchSize int

	// Shards is the control pod's CURRENT sweep range, read lock-free each tick
	// via Shards.Snapshot(). A *ShardSet so the Resharder can swap it at runtime.
	Shards *ShardSet

	// OnSwept is an optional metrics observer (sweep, n>0). nil → no-op. Off the hot path.
	OnSwept func(sweep string, n int64)
	// contains filtered or unexported fields
}

func NewResyncer

func NewResyncer(repo ResyncerRepo, kinds map[model.Kind]ResyncKind) *Resyncer

NewResyncer constructs a Resyncer for the given opted-in kinds. Returns nil when no kind opted in, so callers can skip starting a goroutine.

func (*Resyncer) Run

func (r *Resyncer) Run(ctx context.Context) error

func (*Resyncer) SetKinds

func (r *Resyncer) SetKinds(kinds map[model.Kind]ResyncKind)

SetKinds atomically replaces the resync kind-map — the control plane calls it on its tick after re-reading kind_config, so an operator's resync edit (interval/recompose) takes effect on a running Resyncer with no restart. A kind dropped from the map simply stops being re-pended; one added (that was already resync-enabled at boot, hence this Resyncer exists) starts on the next tick. Safe to call concurrently with tick().

type ResyncerRepo

type ResyncerRepo interface {
	RequeueForResync(ctx context.Context, kind model.Kind, resyncAfter time.Duration, limit int, shards []int16, recompose bool) (int, error)
}

ResyncerRepo is what the Resyncer calls each tick: re-pend settled resources of one kind for a drift re-check.

type ShardSet

type ShardSet struct {
	// contains filtered or unexported fields
}

ShardSet is a lock-free, atomically-swappable holder of the shard range a process owns. It is the seam that lets the Resharder swap the shard range at runtime when cluster membership changes.

The hot loops — the dispatcher's claim sweep, the drainer bands, and the reaper/specgc/resyncer ticks — read the current snapshot every iteration via Snapshot() (a single atomic.Pointer load, no lock, no allocation). The Resharder publishes a new range with Store(). Because each snapshot is immutable and swapped with one atomic store, a reader either sees the whole old range or the whole new one — never a mix — so a reshard can overlap a claim/drain without corrupting ownership (and overlap is safe anyway: the work_queue claim is FOR UPDATE SKIP LOCKED, so two pods racing the same shard each take disjoint rows).

func NewShardSet

func NewShardSet(shards []int16) *ShardSet

NewShardSet builds a set initialised to the given contiguous shard slice (nil/empty → owns nothing until the first Store, the dynamic-bootstrap case). Takes ownership of shards; the caller must not mutate it afterwards.

func (*ShardSet) Bounds

func (s *ShardSet) Bounds() (lo, hi int16)

Bounds returns the current inclusive [lo, hi] span (hi < lo when empty), for the cluster-view reporter that records each member's owned span.

func (*ShardSet) Snapshot

func (s *ShardSet) Snapshot() *ShardSnapshot

Snapshot returns the current immutable shard view. Lock-free (one atomic load). Callers MUST treat the returned snapshot and its Shards slice as read-only — it may be shared with other goroutines.

func (*ShardSet) Store

func (s *ShardSet) Store(shards []int16) bool

Store publishes a new contiguous shard range and reports whether it actually changed. A contiguous set is fully determined by its [lo, hi] bounds, so the compare is two ints: an unchanged tick is a no-op that neither swaps the pointer nor invalidates the drainer bands' snapshot-keyed split cache. Takes ownership of shards; the caller must not mutate it afterwards.

type ShardSnapshot

type ShardSnapshot struct {
	Shards []int16
	Lo, Hi int16
}

ShardSnapshot is an immutable view of the shard range a process currently owns. Shards is a contiguous ascending slice; Lo/Hi are its inclusive bounds (Hi < Lo — i.e. (0, -1) — means "owns nothing", matching store.shardBounds' empty range that prunes to no partition). Once published a snapshot is never mutated, so every reader holds it lock-free and a reshard is a single atomic pointer store with no torn reads.

type SpecGC

type SpecGC struct {
	Repo      SpecGCRepo
	KeepN     int
	Interval  time.Duration
	BatchSize int

	IdleInterval time.Duration

	// Shards is the control pod's CURRENT sweep range, read lock-free each tick
	// via Shards.Snapshot(). A *ShardSet so the Resharder can swap it at runtime.
	Shards *ShardSet

	// OnSwept is an optional metrics observer (sweep, n>0). nil → no-op. Off the hot path.
	OnSwept func(sweep string, n int64)
}

SpecGC bounds the spec_versions root-history log. The live spec is inline on resources.spec; only ROOT Apply events append a copy to spec_versions (the user-facing rollback/checkout history). Composed children carry spec inline and are never versioned, so they never appear here. This sweeper trims each root's log to KeepN newest authored revisions.

Entirely off the hot path: a coarse cadence, shard-scoped to this pod's slice, LIMIT-bounded per tick. It needs NO work_queue/work_outbox coupling and no live-body guard — the live spec is a separate inline column, so trimming an aged-out log row never touches what a worker is reconciling. A version that a checkout previously copied into the live spec keeps working even if its log row ages out (the bytes already live on resources.spec); only the navigable log entry is reclaimed.

func NewSpecGC

func NewSpecGC(repo SpecGCRepo) *SpecGC

NewSpecGC builds the sweeper with production-sane defaults. KeepN=20 matches the agreed root retention depth.

func (*SpecGC) Run

func (g *SpecGC) Run(ctx context.Context) error

type SpecGCRepo

type SpecGCRepo interface {
	GCSpecHistory(ctx context.Context, keepN, limit int, shards []int16) (int, error)
}

SpecGCRepo is what the SpecGC sweeper calls each tick: reclaim superseded immutable spec bodies (roots keep keepN, children keep latest-1).

type TopologyReactor

type TopologyReactor func(ctx context.Context) (changed bool, err error)

TopologyReactor recomputes one subsystem's view from the current membership and reports whether it CHANGED (for logging / an optional onChange). It must be idempotent and respect ctx — the watcher runs it on every wake + failsafe tick.

type TopologyWatcher

type TopologyWatcher struct {
	Listener Listener // cluster_changed wake subscription (ONE for all reactors)

	// LivenessWindow bounds how recent a member's heartbeat must be to count as
	// live. Reactors that filter by liveness read it (via their own store calls);
	// the watcher itself just hands it to them at registration. Must be ≫ the
	// member-heartbeat cadence so a slow beat never drops a live peer, and ≤ the GC
	// TTL so an excluded member is also on its way to being reclaimed.
	LivenessWindow time.Duration

	// Interval is the failsafe re-run cadence (the NOTIFY handles prompt joins/
	// leaves; this catches a crashed peer whose beat aged out without a DELETE).
	Interval time.Duration

	// DebounceWindow coalesces a BURST of cluster_changed NOTIFYs (a scale event's
	// per-pod INSERT/DELETE) into a SINGLE run on the settled topology, so a 1→50
	// scale-out re-tiles + re-dials once, not N times mid-flux. The failsafe Interval
	// still backstops a crashed peer regardless. 0 ⇒ no debounce (every NOTIFY wakes
	// the loop directly; used by tests).
	DebounceWindow time.Duration
	// contains filtered or unexported fields
}

TopologyWatcher is the SINGLE reactor to a cluster-topology change: it owns ONE LISTEN on cluster_changed + ONE debounce + ONE failsafe poll, and on each wake runs a set of registered reactors. The resharder (recompute this pod's shard tile) and the broker mesh (dial/drop peer routes) are both reactors — so both react to the SAME membership snapshot on the SAME schedule, instead of each opening its own LISTEN connection and re-reading cluster_members independently.

REACTIVITY (two layers, the shared wake model):

  • cluster_changed NOTIFY — a JOIN (INSERT) or LEAVE (DELETE) wakes every node's watcher, so a scale-up/down/Deregister is reflected within ~one NOTIFY window (plus DebounceWindow).
  • Interval failsafe — a CRASHED peer does NOT DELETE its row (no NOTIFY fires), so a periodic re-run catches it once its heartbeat ages past LivenessWindow. This is why BOTH reactors must key off the liveness window, not the NOTIFY alone: a hard crash is a silent heartbeat lapse, only the poll sees it.

LivenessWindow is the ONE definition of "member alive" the reactors share (the resharder passes it to assign_member_shards; the mesh reads ListLiveClusterMembers with it), so shard ownership and mesh routing never disagree on who is live. The ClusterMemberGC (row hard-delete at a much longer TTL) is a SEPARATE, orthogonal UI/history sweeper — NOT a routing signal, so it is not a reactor here.

func NewTopologyWatcher

func NewTopologyWatcher(listener Listener) *TopologyWatcher

NewTopologyWatcher builds a watcher with production-sane cadences; the caller sets LivenessWindow (and may override Interval/DebounceWindow) and registers reactors before Start.

func (*TopologyWatcher) Register

func (w *TopologyWatcher) Register(name string, react TopologyReactor, onChange func())

Register adds a reactor run on every wake + failsafe tick. onChange (may be nil) fires from the watcher goroutine right after a run of THIS reactor reports a change — e.g. the resharder wires it to the ClusterMemberReporter's Trigger so a new shard span lands in the registry promptly. Call before Start.

func (*TopologyWatcher) Start

func (w *TopologyWatcher) Start(ctx context.Context)

Start runs every reactor once SYNCHRONOUSLY (so the drivers see this member's shard range + the mesh has its routes before the first sweep/forward), then runs the reactive loop until Stop. The synchronous run is best-effort — a failure (e.g. the member's first heartbeat hasn't landed yet) just leaves each reactor's state at its prior value and the loop retries on the next NOTIFY/tick.

func (*TopologyWatcher) Stop

func (w *TopologyWatcher) Stop()

Stop cancels the loop + listener and joins them, releasing the held LISTEN connection before the pool is closed. It does NOT touch any reactor's published state (the ShardSet / mesh peer set) — those keep their last value until the process exits.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL