Documentation
¶
Overview ¶
Package jobs is the single River composition root for e2a's background work. All durable background jobs — outbound sends, webhook delivery, HITL hold resolution, sender-identity provisioning, janitors — run on ONE shared river.Client against ONE river_job table, instead of a bespoke Postgres queue per subsystem. Each domain contributes its workers as a Registrar; queues are central (queues.go); business code enqueues through the Enqueuer interface so it stays testable and River lives at the edges.
This replaces the hand-rolled claim/lease/retry/sweep machinery: River's SKIP LOCKED claim, per-job MaxAttempts + retry policy, and built-in reaper do the plumbing, so domains only write job args + a Work() handler.
Index ¶
Constants ¶
const ( // QueueOutbound carries outbound send jobs (API → SES). QueueOutbound = "outbound" // QueueInbound carries inbound message-processing jobs (accepted raw MIME → // parse/screen/persist/deliver). Isolated so an inbound spike (incl. the // per-message Gemini screen) can't starve outbound sends or webhook delivery. QueueInbound = "inbound" // QueueWebhook carries customer webhook-delivery jobs. Isolated from outbound // so a slow/failing endpoint's backlog never delays sends. QueueWebhook = "webhook" // QueueMaintenance carries low-urgency periodic/janitor work (reapers, // hold-TTL resolution, auto-disable sweeps). QueueMaintenance = "maintenance" // QueueNotify carries HITL approval-notification emails (the owner's review // alert when an outbound message enters pending_review). Small, isolated pool // so a burst of held sends notifying — or a stuck notification to a bad owner // address retrying — never competes with customer outbound delivery. QueueNotify = "notify" // QueueDefault is River's built-in default — anything not explicitly routed. QueueDefault = river.QueueDefault )
Named queues. Jobs are assigned a queue at enqueue time (InsertOpts.Queue); the shared client works all of these. Separate queues give INDEPENDENT concurrency per lane, so a backlog in one (e.g. a slow customer webhook endpoint) can never starve another (e.g. outbound sends) — the isolation the hand-rolled queues couldn't express cleanly. Keep these names stable: they are persisted on river_job rows.
const DefaultReconcileBatch = 1000
DefaultReconcileBatch bounds one reconcile scan. In steady state the stranded set is ~empty; under a systemic enqueue failure it caps how many rows one pass re-drives (one tx each) so an unhealthy River isn't amplified by fanning the whole backlog every tick — the remainder is picked up on the next pass.
Variables ¶
This section is empty.
Functions ¶
func Migrate ¶
Migrate applies River's own schema (river_job et al.) to the pool. River tracks its migrations in its own river_migration table, separate from e2a's schema_migrations, so this is idempotent and safe alongside identity migrations. Call once at startup, after the e2a migrations. (Generalized from the per-manager Migrate senderidentity used to own.)
Types ¶
type Client ¶
Client is the shared River client plus e2a's lifecycle helpers. It embeds *river.Client[pgx.Tx], so it IS an Enqueuer and exposes Start/Stop directly.
type Config ¶
type Config struct {
OutboundWorkers int // QueueOutbound concurrency (default 8)
InboundWorkers int // QueueInbound concurrency (default 8)
WebhookWorkers int // QueueWebhook concurrency (default 16)
MaintenanceWorkers int // QueueMaintenance concurrency (default 2)
NotifyWorkers int // QueueNotify concurrency (default 4)
DefaultWorkers int // QueueDefault concurrency (default 5)
}
Config sizes the per-queue worker pools. Zero values get sane defaults.
type Enqueuer ¶
type Enqueuer interface {
Insert(ctx context.Context, args river.JobArgs, opts *river.InsertOpts) (*rivertype.JobInsertResult, error)
InsertTx(ctx context.Context, tx pgx.Tx, args river.JobArgs, opts *river.InsertOpts) (*rivertype.JobInsertResult, error)
}
Enqueuer is the narrow insert surface business code depends on, rather than the whole river.Client — keeps the store/agent layer testable with a fake and River swappable. Insert enqueues immediately; InsertTx enqueues WITHIN the caller's transaction (the outbox pattern: the job commits atomically with the business write and can never be lost). *river.Client[pgx.Tx] satisfies this directly.
type QueueStatsArgs ¶ added in v1.4.1
type QueueStatsArgs struct{}
QueueStatsArgs drives one gauge sampling pass.
func (QueueStatsArgs) Kind ¶ added in v1.4.1
func (QueueStatsArgs) Kind() string
type QueueStatsJobs ¶ added in v1.4.1
type QueueStatsJobs struct {
// contains filtered or unexported fields
}
QueueStatsJobs contributes the periodic queue-gauge sampler to the shared client. Implements Registrar (same shape as sendramp.MaintenanceJobs).
func NewQueueStatsJobs ¶ added in v1.4.1
func NewQueueStatsJobs(pool *pgxpool.Pool, metrics QueueStatsMetrics) *QueueStatsJobs
NewQueueStatsJobs builds the sampler registrar. nil metrics degrades to a no-op sampler rather than a nil-panic.
func (*QueueStatsJobs) RegisterJobs ¶ added in v1.4.1
func (q *QueueStatsJobs) RegisterJobs(w *river.Workers) []*river.PeriodicJob
type QueueStatsMetrics ¶ added in v1.4.1
type QueueStatsMetrics interface {
SetQueueDepth(queue, state string, n int)
SetQueueOldestAge(queue string, seconds float64)
}
QueueStatsMetrics is the narrow gauge surface the sampler sets. Satisfied by telemetry.Metrics; injectable so tests assert with a fake.
type QueueStatsWorker ¶ added in v1.4.1
type QueueStatsWorker struct {
river.WorkerDefaults[QueueStatsArgs]
// contains filtered or unexported fields
}
QueueStatsWorker samples river_job depth and oldest-runnable-age gauges per queue. Read-only and bounded (two single-round-trip aggregates over the indexed state column), so River's default JobTimeout is ample.
func NewQueueStatsWorker ¶ added in v1.4.1
func NewQueueStatsWorker(pool *pgxpool.Pool, metrics QueueStatsMetrics) *QueueStatsWorker
NewQueueStatsWorker builds the sampler. nil metrics degrades to a no-op.
func (*QueueStatsWorker) Sample ¶ added in v1.4.1
func (w *QueueStatsWorker) Sample(ctx context.Context) error
Sample runs one gauge pass: depth per (queue, state) and the oldest RUNNABLE (available, scheduled_at due) job age per queue. Every known queue × state pair is set on every pass — pairs absent from the query zero-fill, so gauges drop to 0 when a queue drains. Rows for queues outside the known set are ignored: they cannot be zero-filled once they empty, so reporting them would leave stuck gauges.
func (*QueueStatsWorker) Work ¶ added in v1.4.1
func (w *QueueStatsWorker) Work(ctx context.Context, _ *river.Job[QueueStatsArgs]) error
type ReconcileResult ¶ added in v1.4.1
ReconcileResult reports one ReconcilePending pass, split by arm so callers can observe them separately: Enqueued counts rows re-driven from the JobColumn IS NULL arm (work that never got a job); Rescued counts rows re-driven from the dead-job arm (RescueDeadJobs — a fresh job replacing a terminal/pruned one). A monotonically climbing Rescued rate is the poison-row signal: a deterministically failing row burns a full job envelope per rescue, forever, and only this count makes that loop visible.
func ReconcilePending ¶
func ReconcilePending(ctx context.Context, pool *pgxpool.Pool, spec ReconcileSpec, enqueueTx func(ctx context.Context, tx pgx.Tx, id string) (int64, error)) (ReconcileResult, error)
ReconcilePending re-drives every stranded row matching spec.Where: rows whose spec.JobColumn IS NULL and — when spec.RescueDeadJobs is set — rows whose stamped river_job is missing or terminal (narrowed by spec.RescueWhere when given). It scans up to spec.Batch ids, then per id opens a tx, re-checks strandedness under FOR UPDATE (skipping rows another process or a prior pass already gave a live job), calls enqueueTx to insert the River job in that tx, and stamps the returned job id back onto the row — all atomically. A per-row failure is logged and skipped (the next pass retries it); the returned result counts the rows re-driven, split by arm.
Idempotency, per arm: the IS NULL arm never double-enqueues — the FOR UPDATE re-check reads the committed job column under the row lock, so a concurrent enqueuer's stamp is always seen. The dead-job arm is AT-LEAST-ONCE by design under concurrent reconcilers: when the loser of the row-lock race resumes, READ COMMITTED's EvalPlanQual re-evaluates the quals against the updated row version but keeps the outer-joined river_job tuple from its original snapshot, so the winner's freshly stamped (live) job can still read as dead to the loser and a duplicate job is enqueued. That duplicate is within the webhook pipeline's at-least-once contract (worker-side status guards make it a no-op or a tolerated duplicate delivery).
This is the shared body behind every domain's startup cutover + live reconcile worker (outboundsend, inboundprocess, hitlnotify, webhookdelivery, webhookpub fan-out). Each domain supplies a ReconcileSpec + its own EnqueueXTx.
func (ReconcileResult) Total ¶ added in v1.4.1
func (r ReconcileResult) Total() int
Total is the number of rows this pass re-drove across both arms.
type ReconcileSpec ¶
type ReconcileSpec struct {
// Table is the row table (e.g. "messages", "webhook_events").
Table string
// JobColumn is the nullable bigint column holding the enqueued River job id
// (e.g. "send_job_id", "fanout_job_id"). A row is "stranded" when it matches
// Where AND JobColumn IS NULL (plus, with RescueDeadJobs, when its stamped
// job is dead — see below).
JobColumn string
// Where is the domain predicate identifying rows that SHOULD carry a job
// (e.g. "direction='outbound' AND delivery_status='accepted'"). ANDed with
// the strandedness condition; pass no leading/trailing "AND". Every query
// binds the row table to the alias `t`, so unqualified column names resolve
// to it — but with RescueDeadJobs the queries also join river_job as `r`, so
// qualify any column whose name river_job shares (id, state, kind, queue,
// attempt, ...) as t.<col> to avoid ambiguity.
Where string
// LogPrefix tags per-row enqueue-failure logs (e.g. "[outbound-reconcile]").
LogPrefix string
// Batch caps rows scanned per pass; 0 uses DefaultReconcileBatch.
Batch int
// RescueDeadJobs additionally treats a row as stranded when JobColumn IS NOT
// NULL but the stamped river_job no longer exists (pruned by River's reaper)
// or is in a terminal state (cancelled/discarded/completed) — a job that can
// never run again while the row still matches Where. Same "job is dead,
// reconcile the row" test as outboundsend's TerminalReconcileWorker, but the
// remedy here is RE-DRIVING (fresh job, re-stamped id), so it is OPT-IN:
// only set it when re-running the work is both idempotent and desired.
// outboundsend, for example, must NOT re-enqueue a discarded send job (its
// terminal reconciler settles those rows as sent/failed instead — re-driving
// would resend email); the webhook pipeline's fan-out and delivery rows DO
// want re-driving, because their at-least-once contract already tolerates it.
//
// Cost note: the dead-job scan cannot use the JobColumn-IS-NULL partial
// indexes; it walks the rows matching Where with a stamped job (the in-flight
// set) and probes river_job by primary key per row. Callers' Where predicates
// keep that set small and TTL-bounded; the plain IS NULL scan still runs
// first and stays index-backed. That ordering is also a prioritization: the
// IS NULL strand (work that never got a job) fills the batch before dead-job
// strands are even scanned, so under an IS-NULL backlog that saturates every
// tick's batch, dead-job rescues wait until that backlog drains.
RescueDeadJobs bool
// RescueWhere optionally narrows the dead-job arm ONLY (scan + per-row
// re-check); the IS NULL arm is unaffected. ANDed with the dead-job
// condition; empty = no extra gate. Use it to age-gate the rescue so the
// per-tick scan doesn't churn on rows whose jobs are plausibly still live —
// e.g. webhookdelivery gates on quiet-for-longer-than-the-retry-envelope.
// Same alias rules and SECURITY contract as Where.
RescueWhere string
}
ReconcileSpec describes one table's stranded-row reconcile.
SECURITY: Table, JobColumn, Where, and RescueWhere are COMPILE-TIME CONSTANTS supplied by the calling package — never runtime or user input. They are interpolated directly into SQL (there is no parameter form for identifiers), so callers must pass only literal strings. The only runtime value (the row id) is always a bound parameter.
type Registrar ¶
type Registrar interface {
RegisterJobs(w *river.Workers) []*river.PeriodicJob
}
Registrar is what each domain implements to contribute its jobs to the shared client. RegisterJobs adds the domain's workers to the shared bundle (via river.AddWorker) and returns any periodic jobs it wants scheduled. Called once, at client construction — River requires all workers registered before Start.