Documentation
¶
Overview ¶
Package engine assembles the process from composable DUTIES over one shared Postgres.
A pod is not a ROLE; it is a SET OF DUTIES. Each Duty is one unit of work the process performs — the broker, the outbox drainer, the reaper, the spec GC, the resync sweeper, the cluster-member GC, the reactor dispatcher. main.go resolves the configured role (or an explicit DUTIES= list) into a []Duty via dutiesFromConfig and hands it to one Engine, which Start/Stops them through a single uniform loop. "What does this pod do?" is read off its duty list.
One duty — claim — drives provider execution, fanning each task's selected reaction out to dumb workers. Every other duty is kind-agnostic plumbing that operates on work_queue / work_outbox / resources rows directly and reads any per-kind operational settings it needs (cap, resync) from the kind_config DB table — never a constructed provider. converge carries no provider code at all: the broker fans each claimed task out to a dumb Connect worker that holds the handlers, so a converge pod dials no downstream. The duties coordinate entirely through the database, never via in-process state; the real runtime atoms are the drivers they wrap (Dispatcher, Drainer, Reaper, Resyncer); see internal/runtime/doc.go.
Control-side duties are grouped behind one ControlPlane bundle (it runs migrations + verifies the shard modulus once, then starts drain + reap + specgc + resync + membergc together) because every deployment runs those sweepers as a unit on a control pod; the bundle is an internal construction detail, not a tier — the Engine sees it as one controlDuty among the list.
Eligibility scheduling (deciding which pending rows are ready to dispatch and inserting work_queue rows) is not a Go duty — it lives in Postgres triggers + the schedule_eligible() function (a DB-hosted reactor). The cascade_on_ready_change AFTER-STATEMENT trigger fires reactively when a row's synced_gen catches up to generation; value-flow re-substitution happens inside drain_outbox_batch. The reaper (requeue_failed_and_pending) is the backstop for rows the cascade misses.
Index ¶
- Constants
- type BrokerConfig
- type ControlPlane
- type ControlPlaneConfig
- type Deps
- type Duty
- type Engine
- func (e *Engine) AddKindLive(m model.KindManifest)
- func (e *Engine) BroadcastConfig(kind model.Kind, kindVersion int, spec, data []byte)
- func (e *Engine) ClaimHandler(opts ...connect.HandlerOption) (path string, h http.Handler, ok bool)
- func (e *Engine) ConnectedWorkerCount() int
- func (e *Engine) ConnectedWorkers() []broker.ConnectedWorker
- func (e *Engine) InFlight() int
- func (e *Engine) MeshCounters() (peerGoneWakes, foreignDropSignals, syntheticDelivered int64)
- func (e *Engine) ReleaseClaims(ctx context.Context) (int64, error)
- func (e *Engine) RunsBroker() bool
- func (e *Engine) Start(ctx context.Context) error
- func (e *Engine) Stop(ctx context.Context) error
- func (e *Engine) WorkerKindsSnapshot() []model.Kind
- type EngineConfig
- type HandlerProvider
Constants ¶
const ( // DefaultRetryAfter is the delay before a failed task is re-pended by the reaper. DefaultRetryAfter = 2 * time.Second // DefaultSweeperInterval is the control-plane sweep cadence. DefaultSweeperInterval = 10 * time.Second )
Sweeper cadence defaults — the SINGLE source of truth; other code references THESE constants, never re-typing the literals. (StaleAfter and UnclaimedDeleteAfter default in the Reaper itself — see runtime.DefaultSweeper*.)
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type BrokerConfig ¶
type BrokerConfig struct {
Pool *pgxpool.Pool
// WorkerMaxParallel caps how many tasks of EACH (kind, task_type) are in
// flight (claimed + fanning out to workers) on this broker. Left 0 by
// DutiesFromConfig → broker.NewDispatch applies its 100 default (no env knob;
// the per-worker cap lives on the worker binary via WORKER_MAX_PARALLEL).
WorkerMaxParallel int
HeartbeatEvery time.Duration
WorkerPollMin time.Duration
WorkerPollMax time.Duration
// DrainGrace is the graceful in-flight drain window on Stop: the dispatcher +
// reactor stop claiming immediately but let already-fanned-out tasks land their
// worker Complete for up to this long before abandoning them. 0 → no drain.
DrainGrace time.Duration
// Shards is the broker's claim range: a *runtime.ShardSet so the Resharder
// can swap it at runtime on membership change (read lock-free each sweep).
// nil → the dispatcher keeps its AllShards default.
Shards *runtime.ShardSet
// BrokerID is the lease identity stamped at claim — the broker's stable,
// externally-chosen id so the AppendOutbox fence + cluster_members + reaper
// attribute its leases.
BrokerID string
// RelayHTTPClient is the HTTP/2 (h2c/ALPN) transport this broker dials PEER
// brokers' persistent Route stream with. It advertises this broker's live worker
// CREDIT to peers and pushes a claimed task it has no local worker for to a
// credited peer, so any worker can run work claimed by any broker (decouples
// placement from consumption). Set for every broker; the mesh is a no-op with zero
// peers (single-broker fleet). nil → mesh disabled. See internal/broker/relay.go.
RelayHTTPClient connect.HTTPClient
// MeshLivenessWindow is how recent a peer's heartbeat must be for the mesh to
// route to it — the SAME window the Resharder uses for shard tiling (both 3× the
// member-heartbeat cadence), so a crashed peer drops out of both within one window
// rather than lingering to the ClusterMemberGC TTL. 0 → the broker duty derives it
// from the member-heartbeat cadence.
MeshLivenessWindow time.Duration
// Claimed / Completed are the OTel dispatch counters, wired onto the broker
// server AT CONSTRUCTION (before it serves → no data race). When metrics are on
// they're non-nil (real or no-op) instruments; left nil here (the zero value, e.g.
// a directly-built config) the broker's call sites nil-guard the Add.
Claimed metric.Int64Counter
Completed metric.Int64Counter
// WorkerAuthz / MeshAuthz are the per-service SPIFFE-ID allowlists the broker's
// Handler installs as Connect interceptors — WorkerAuthz on WorkerService,
// MeshAuthz on MeshService — narrowing the shared listener's union-admitted peer to
// the right audience per RPC. nil = no allowlist for that service (chain trust).
WorkerAuthz spiffeauthz.Matcher
MeshAuthz spiffeauthz.Matcher
// PeerIdentitySource is the raw PEER_IDENTITY_SOURCE value ("" / "mtls" /
// "mesh-header") the broker resolves connected-client identity by. See SetAuthz.
PeerIdentitySource string
}
BrokerConfig configures the claim duty (the server-side dispatch surface). It is the knob set a broker pod's runtime.Dispatcher reads: the per-(kind,task_type) concurrency ceiling, heartbeat/poll cadences, and its shard range + lease identity. A broker always claims EVERY manifested kind (learned live from kind_manifest); there is no kind subset.
type ControlPlane ¶
type ControlPlane struct {
// contains filtered or unexported fields
}
func NewControlPlane ¶
func NewControlPlane(ctx context.Context, cfg *ControlPlaneConfig) (*ControlPlane, error)
type ControlPlaneConfig ¶
type ControlPlaneConfig struct {
Pool *pgxpool.Pool
// Shards is the control pod's shard range, SHARED by every control sweeper
// (drainer, reaper, specgc, resyncer) — a single *runtime.ShardSet so the
// Resharder can swap the whole pod's range atomically on membership change
// and all four sweepers pick it up together. nil → defaults to AllShards
// (single-pod / unsharded: sweep everything). A member owns ONE contiguous
// range for its role, used uniformly by all its sweepers.
Shards *runtime.ShardSet
SweeperStaleAfter time.Duration
SweeperInterval time.Duration
RetryAfter time.Duration
// SweeperUnclaimedDeleteAfter overrides the reaper's abandoned-work GC window
// (default 24h in runtime.NewReaper): a work_queue row unclaimed this long is
// hard-deleted because no worker for its kind ever claimed it. Distinct from
// SweeperStaleAfter (which recovers a dead CLAIMED task). 0 keeps the default.
SweeperUnclaimedDeleteAfter time.Duration
// ResyncSweepInterval overrides how often the drift Resyncer wakes to
// look for due rows (default 30s). Test-only (not reachable from boot
// config): lower it to observe drift detection within a short deadline;
// leave zero in production.
ResyncSweepInterval time.Duration
// KindConfigRefreshInterval overrides how often the control plane re-reads
// kind_config to live-refresh the Resyncer's kind-map (default 15s, the
// informer relist). Test-only: lower it to observe a runtime resync edit
// converge within a short deadline; leave zero in production.
KindConfigRefreshInterval time.Duration
// Swept is the optional OTel sweeper-rows counter, wired onto every sweeper AT
// CONSTRUCTION (before Start → before any tick → no data race). nil → sweepers
// stay metrics-free. Set from cmd/converge when metrics are enabled.
Swept metric.Int64Counter
}
ControlPlaneConfig configures the control sweeper bundle. Pool is required; everything else has a sane default. There is deliberately NO Registry — the control plane reads its per-kind resync policy from the kind_config DB table, so it constructs no provider and dials no client.
type Deps ¶
type Deps struct {
Pool *pgxpool.Pool
Shards *runtime.ShardSet
// Configs is the live default-config source the CLAIM duty serves to dumb
// workers via GetProviderConfig (the ProviderConfigCache satisfies it). nil-safe: a
// broker with no source serves no defaults (workers get only the per-resource
// override). A non-broker pod leaves it nil.
Configs broker.ConfigReader
}
Deps is the shared bag of dependencies wired ONCE at boot and handed to every duty's Start. A duty reads only the fields it needs. The API server's read-replica pool is NOT here — serving the API stays in cmd/converge, not a duty.
type Duty ¶
type Duty interface {
// Name is a short identifier for logs and the cluster view ("control",
// "claim").
Name() string
// Start launches the duty's goroutines against the shared Deps and returns
// once they are running (it does not block until they finish). The duty owns
// its own lifecycle off the passed ctx; cancelling ctx stops it.
Start(ctx context.Context, deps Deps) error
// Stop joins the duty's goroutines (idempotent; a no-op before Start). The
// Engine calls it in reverse start order on shutdown.
Stop(ctx context.Context) error
}
A Duty is one unit of work a process performs. A process is simply the SET of duties it was asked to run (see DutiesFromConfig). Every sweeper and the broker are peer duties behind this one interface, so "what does this pod do?" is answered by reading its duty list.
The concrete duties (controlDuty / claimDuty) are thin wrappers around the ControlPlane sweeper bundle and the Connect broker (which itself also drains lifecycle_outbox and fans reactors out to its workers); the Engine runs the list uniformly. converge (control/broker) carries NO provider code and runs no in-process dispatch — every kind's handlers live in a dumb Connect worker, and the broker learns which kinds exist LIVE from kind_manifest, not from the duty objects.
func ClaimDuty ¶
func ClaimDuty(cfg *BrokerConfig) Duty
ClaimDuty builds a claim duty from a BrokerConfig. cfg.BrokerID is the broker's lease identity.
func ControlDuty ¶
func ControlDuty(cfg *ControlPlaneConfig) Duty
ControlDuty builds the control sweepers from cfg. cfg.Pool is required; the rest default as in NewControlPlane. It takes NO registry — the control plane reads resync policy from the kind_config DB table, never a constructed provider.
func DutiesFromConfig ¶
func DutiesFromConfig(cfg EngineConfig) ([]Duty, error)
DutiesFromConfig resolves an EngineConfig into the ordered duty list the Engine runs, derived from the ROLE-set RunControl/RunBroker flags. The control bundle (if present) comes FIRST and the claim duty LAST, so on shutdown the broker (reverse order) drains before the sweepers that feed it stop:
RunControl → controlDuty RunBroker → claimDuty (claims work AND lifecycle_outbox; every manifested kind)
A config that yields no duties is legal (the pod does nothing but heartbeat) — the caller decides whether that's acceptable.
type Engine ¶
type Engine struct {
// contains filtered or unexported fields
}
Engine is the process's run unit: the ordered SET OF DUTIES this pod performs, plus the shared dependencies they run against. main.go builds one duty list (dutiesFromConfig) and hands it to one Engine; Start and Stop are a single uniform loop over that list.
The Engine owns NO policy of its own — every behavior lives in a Duty. It only folds the shared Deps onto each duty at Start and joins them at Stop, and surfaces the cross-cutting hooks main.go needs from the dispatch duty: AddKindLive (for the ProviderRetrier) and InFlight (for the cluster-view reporter).
func NewEngine ¶
NewEngine builds the run unit from a duty list and the shared dependencies every duty draws from (the pool, the read pool, the Setup'd registry, the pod's shard range). The duties are typically produced by dutiesFromConfig.
func (*Engine) AddKindLive ¶
func (e *Engine) AddKindLive(m model.KindManifest)
AddKindLive forwards a late-applied kind's manifest to the dispatch duty so it starts claiming with no restart. No-op on a pod with no dispatch duty. The ProviderRetrier / a manifest apply calls this.
func (*Engine) BroadcastConfig ¶
BroadcastConfig pushes a (kind, kindVersion)'s new default providerconfig — the whole monolith, spec + data/bundle — to every connected worker serving that exact (kind, kindVersion) (via the claim duty's broker). No-op on a control-only pod (no broker to broadcast through). main wires the ProviderConfigCache's OnKindConfigChange here.
func (*Engine) ClaimHandler ¶
ClaimHandler returns the broker Connect service route to mount on the broker's listener, or ok=false when this pod runs no claim duty. Call AFTER Start (the claim duty builds its server in Start). main.go mounts it on the broker listener, reusing the TLS/mTLS scaffold.
func (*Engine) ConnectedWorkerCount ¶
ConnectedWorkerCount is the live worker-stream count (metrics gauge source); 0 on a pod with no dispatch duty.
func (*Engine) ConnectedWorkers ¶
func (e *Engine) ConnectedWorkers() []broker.ConnectedWorker
ConnectedWorkers snapshots the workers connected to this pod's broker for the connected-worker cluster view (one row per WorkStream stream). nil on a pod with no dispatch duty. main wires it to the reporter's Workers hook so the cluster_members.workers column tracks streams connecting/dropping.
func (*Engine) InFlight ¶
InFlight returns the pod's live in-flight task count for the cluster-view reporter, or 0 on a pod with no dispatch duty.
func (*Engine) MeshCounters ¶
MeshCounters returns the cumulative broker-mesh fast-recovery counts (metrics); zeros on a pod with no dispatch duty.
func (*Engine) ReleaseClaims ¶
ReleaseClaims frees every work_queue row this pod's dispatcher holds, so a surviving pod re-claims the work immediately on graceful shutdown instead of waiting out the reaper's stale window. Call AFTER Stop (the dispatcher is then quiesced). Returns the count freed; 0 on a pod with no dispatch duty.
func (*Engine) RunsBroker ¶
RunsBroker reports whether this pod runs the broker (dispatch) duty — the gate for registering broker metrics.
func (*Engine) Start ¶
Start launches every duty against the shared Deps, in list order. It stops at the first duty that fails to start, UNWINDS the duties already started (reverse order, best-effort), and returns the error — so a partial-start abort leaves no orphaned sweepers running (previously the started duties lingered until the caller cancelled the ctx, skipping their ordered drain). Each duty's Start is non-blocking (it spawns goroutines and returns), so on success this returns once the whole pod is running.
func (*Engine) Stop ¶
Stop joins every duty in REVERSE start order (so the dispatch duty, usually last, drains before the control sweepers it fed stop). Best-effort: it stops them all even if one errors, returning the first error seen.
func (*Engine) WorkerKindsSnapshot ¶
WorkerKindsSnapshot returns the pod's LIVE connected-worker kind set, or nil on a pod with no dispatch duty.
type EngineConfig ¶
type EngineConfig struct {
// RunControl requests the control sweeper bundle (drain + reap + specgc +
// resync + membergc). False on a pure worker pod.
RunControl bool
// Control sweeper knobs (passed to the control bundle; zero = driver default).
RetryAfter time.Duration
SweeperStaleAfter time.Duration
SweeperInterval time.Duration
SweeperUnclaimedDeleteAfter time.Duration
// Claim knobs (passed to the claim duty; zero = driver default).
HeartbeatEvery time.Duration
WorkerPollMin time.Duration
WorkerPollMax time.Duration
// ShutdownDrain is the graceful in-flight drain window the claim duty's
// dispatcher + reactor honor on Stop (let fanned-out work land its result
// before abandoning it). 0 → no drain.
ShutdownDrain time.Duration
// Shards is the pod's shard range, shared by every duty (control sweepers and
// the broker). nil → each driver keeps its AllShards default.
Shards *runtime.ShardSet
// RunBroker requests the CLAIM duty: the pod owns its shard tile, claims
// work_queue rows AND drains lifecycle_outbox, and fans each reaction (work or
// lifecycle STAGE_REACT) out to dumb workers (the universal claim tier).
// A broker claims EVERY manifested kind (learned live from kind_manifest).
// BrokerID is the broker's stable lease identity (also the reactor
// dispatcher's claim id).
RunBroker bool
BrokerID string
// Broker-to-broker MESH, passed to the claim duty. Enabled whenever
// RelayHTTPClient is set (always, for a broker); a no-op with zero peers
// (single-broker fleet). RelayHTTPClient is the HTTP/2 (h2c/ALPN) route transport.
RelayHTTPClient connect.HTTPClient
// MeshLivenessWindow is how recent a peer's heartbeat must be for the mesh to
// route to it — the SAME window the Resharder uses for shard tiling, so a crashed
// peer drops out of both within one window (not the ClusterMemberGC TTL). Derived
// from 3× the member-heartbeat cadence at the composition root.
MeshLivenessWindow time.Duration
// WorkerAuthz / MeshAuthz are the parsed SPIFFE-ID allowlists for the broker's
// two Connect services, which share ONE mTLS listener. The handshake admits the
// UNION (main.go); these narrow each RPC per-service — WorkerAuthz gates
// WorkStream/GetProviderConfig (the workers that pull work), MeshAuthz gates
// Route (the PEER BROKERS on the internal mesh) — so a worker cert
// can't reach a mesh RPC and vice versa. nil = no allowlist for that service.
WorkerAuthz spiffeauthz.Matcher
MeshAuthz spiffeauthz.Matcher
// PeerIdentitySource is the raw PEER_IDENTITY_SOURCE value ("" / "mtls" /
// "mesh-header") — how the broker derives a connected client's OBSERVED identity
// for the cluster view + attribution (never a self-report). See cmd/converge config.
PeerIdentitySource string
// OTel metric counters, wired into the duties AT CONSTRUCTION (before Start → no
// data race). When metrics are on they're non-nil (real or no-op) instruments;
// left nil (the zero value) the duties' Add call sites nil-guard. SweptCounter →
// control sweepers; BrokerClaimed/BrokerCompleted → the broker dispatch path.
SweptCounter metric.Int64Counter
BrokerClaimed metric.Int64Counter
BrokerCompleted metric.Int64Counter
}
EngineConfig is the resolved per-process boot configuration DutiesFromConfig turns into a duty list. It carries the role decision (RunControl/RunBroker + the dispatch identity) plus the tunable knobs the duties pass through to their runtime drivers. main.go fills it from env; tests fill it directly.
type HandlerProvider ¶
type HandlerProvider interface {
Handler(opts ...connect.HandlerOption) (path string, h http.Handler, ok bool)
}
HandlerProvider is an OPTIONAL capability a Duty may also implement: it publishes an HTTP route to mount on a pod's listener (the claim duty exposes the broker Connect service route this way). Engine.ClaimHandler ranges the duty list for the first implementer, so a new route-publishing duty needs no Engine edit — it just implements this. ok=false means the duty has no route to mount right now (e.g. its server isn't built until Start). Called AFTER Start.