queue

package
v0.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 11, 2026 License: MIT Imports: 14 Imported by: 0

README

queue

Postgres-backed job queue with batched claims and lease fencing.

A claim is one statement, and therefore one transaction:

UPDATE job AS t
SET state = 'claimed', lease_epoch = t.lease_epoch + 1
WHERE t.id IN (
    SELECT d.id
    FROM job AS d
    WHERE d.state = 'pending' AND d.next_due <= now()
    ORDER BY d.priority, d.next_due, d.id
    LIMIT $1
    FOR UPDATE SKIP LOCKED
)
RETURNING t.id, t.lease_epoch

The row locks taken by the subquery are held until that statement commits, and SKIP LOCKED steps over rows another claimer holds, so concurrent claimers get disjoint batches without waiting on each other. Claim order is lowest priority value, then earliest next_due, then lowest id. The index in schema.sql serves both the pending range and that order.

ClaimN clamps n to 1..MaxBatch (1000) and returns fewer than n rows, including zero, when fewer are due.

Fencing

Every claim increments lease_epoch. Settle and Fail only touch a row that is still claimed at the epoch the caller holds. If another owner reclaimed the row, the epoch no longer matches, no row is updated, and the call returns ErrStale. A stale owner can therefore neither settle nor reschedule a job it no longer owns.

Settle marks the row done. Fail returns it to pending with next_due moved forward by the given backoff (a negative backoff means zero).

Usage

q, err := queue.New(db) // or queue.NewWithSchema for other column names
if err != nil {
    return err
}

jobs, err := q.ClaimN(ctx, 32)
if err != nil {
    return err
}
for _, job := range jobs {
    if err := handle(ctx, job); err != nil {
        _ = q.Fail(ctx, job.ID, job.Epoch, time.Minute)
        continue
    }
    if err := q.Settle(ctx, job.ID, job.Epoch); err != nil {
        // ErrStale: the lease was reclaimed while the job was in hand.
    }
}

The returned slice is not ordered; order only decides which rows are claimed.

Jobs: leases, retries and dead letters

Jobs is a complete job queue on top of the same claim technique. It owns its table; create it from JobsDDL(table) (also in jobs_schema.sql) in a migration.

j, err := queue.NewJobs(db, queue.JobsConfig{LeaseTTL: 30 * time.Second})

// Producer. The key makes a retried request add the job once.
job, created, err := j.Enqueue(ctx, queue.NewJob{
    Kind: "email.send", Payload: payload, IdempotencyKey: "order-1842",
})

// Worker.
jobs, err := j.Claim(ctx, 16, queue.Filter{Kinds: []string{"email.send"}})
for _, job := range jobs {
    res, err := handle(ctx, job) // call j.Heartbeat(ctx, job.ID, job.Epoch) while it runs
    if err != nil {
        state, _ := j.Fail(ctx, job.ID, job.Epoch, err) // "pending" (retry) or "dead"
        continue
    }
    err = j.Complete(ctx, job.ID, job.Epoch, res) // ErrStale: lease was lost
}
  • Claim leases up to N due jobs in one FOR UPDATE SKIP LOCKED statement. Due means pending with run_at passed, or running with an expired lease and attempts left, so a crashed worker's jobs come back without a separate reaper. Each claim counts one attempt.
  • Fencing. Heartbeat, Complete and Fail take the Epoch from Claim and return ErrStale once the lease was lost, completed or failed. This is why they take the epoch as well as the job id.
  • Heartbeat pushes the lease out by LeaseTTL; call it at a fraction of the TTL while a job runs.
  • Retries. Fail reschedules after BackoffBase * 2^(attempts-1), capped at BackoffMax. When Attempts reaches MaxAttempts (per job, default 5) the job becomes dead, keeping LastError. A job whose lease expires on its final attempt is dead-lettered by Reap, which Claim runs first.
  • Dead letters. DeadLetters lists dead jobs; Requeue gives one a fresh set of attempts.
  • Idempotency. (kind, idempotency_key) is unique; Enqueue returns the existing job with created == false, including under concurrent enqueues.
  • Admin. Depth (ready, delayed, running, done, dead), Stuck (running with a lease expired longer than a grace period, unclaimed) and Throughput (done and dead in a window).

Finished rows are not deleted; prune done rows with a retention job of your own.

Measured ceiling

One million due rows, one connection, Postgres 18:

batch claims/s rows/s p99
1 700.9 700.9 12.3 ms
32 514.5 16,464 6.8 ms

The single-row ceiling is about 700 claims/s on one connection, i.e. 700 jobs/s: the statement cost is dominated by round trip and commit, not by the row.

A batch of 32 raises row throughput to about 16.5k/s, roughly 23x, while the p99 of a claim call stays under 10 ms. That is the batch size this package recommends. Larger batches amortize the same fixed cost over more rows but hold row locks and connections longer for little further gain; MaxBatch caps a careless request at 1000.

Reproducing

The numbers come from BenchmarkClaimN, over a disposable database:

../tools/testdb.sh go test ./queue/... -run '^$' -bench BenchmarkClaimN -benchtime=2s

The benchmark seeds dueRows = 1_000_000 due rows with ten distinct priorities and next_due spread across the last hour, builds the claim index after the load, and reports claims/s, rows/s and the p99 of each claim call. Absolute numbers depend on the host; the ratio between the two batch sizes is the point.

River client

Client adds a durable, retrying queue on top of River over database/sql. Use the claim API above for a table you own; use Client for background work enqueued from application transactions.

c, err := queue.NewClient(queue.Config{DB: db, Workers: workers})
if err != nil {
    return err
}
if err := c.Start(ctx); err != nil { // applies River's migrations first
    return err
}
defer c.Stop(ctx)

// In a request: the job exists only if the transaction commits.
_, err = c.InsertTx(ctx, tx, SendEmail{To: addr, Key: msgID}, queue.Idempotent(nil))
  • Idempotent inserts. Idempotent skips a second insert with the same args. Tag the key field of the args with river:"unique". InsertMany and InsertManyTx report skipped duplicates per job; one batch must not contain the same key twice.
  • Periodic jobs. Config.Periodic or AddPeriodic take a schedule (Every or any river.PeriodicSchedule, such as a cron parser) and a constructor. Only the leader enqueues them.
  • One leader. River elects one client through the river_leader table; that client runs the scheduler, the rescuer and the cleaner. NoLeader keeps a client with a partial worker set out of the election.
  • Rescue. A job still running after Lease is treated as abandoned by a dead worker: the leader reschedules it with backoff, or marks it Dead once MaxAttempts runs are used. The per-job timeout is nine tenths of the lease. A job that outlives its lease can run twice, so handlers must be idempotent.
  • Backoff. Backoff{Base, Max} doubles the delay per attempt and never exceeds Max.

Rescue and retry timing is bounded by River's maintenance intervals (the scheduler runs every 5 seconds and the rescuer every 30), so a rescued job resumes some seconds after its lease ends, not at the instant.

The client uses the database/sql driver, which has no LISTEN/NOTIFY: fetching and leader election poll. Tune FetchPollInterval for latency.

Tests run against Postgres through ../tools/testdb.sh go test ./queue/... and skip without KEEL_TEST_DSN.

Documentation

Overview

Package queue claims due rows from a PostgreSQL job table in batches and fences the owner with a lease epoch. Client, in client.go, is a River-backed durable queue for work enqueued from application transactions.

A job table claimed by Queue has these columns:

id          bigint      primary key
priority    integer     lower is claimed first
next_due    timestamptz the row is due when next_due <= now()
state       text        'pending' before a claim, 'claimed' after
lease_epoch bigint      bumped on every claim; the owner's fencing token

ClaimN runs one statement, so it is its own transaction: the subquery locks the rows it picks with FOR UPDATE SKIP LOCKED and the update claims exactly those rows. Settle and Fail run on the caller's *sql.DB, each on a pooled connection.

Fencing: ClaimN returns each row's new lease_epoch. Settle and Fail only update the row while state = 'claimed' AND lease_epoch = the caller's epoch, so an owner whose lease was reclaimed (or who settles twice) reports ErrStale and must stop touching the row.

Index

Constants

View Source
const (
	// DefaultLease is how long a job may stay running before the leader
	// treats its worker as dead and reschedules it.
	DefaultLease = 5 * time.Minute
	// DefaultMaxAttempts is how many times a job runs before it is dead.
	DefaultMaxAttempts = 8
	// DefaultBackoffBase is the delay after the first failed attempt.
	DefaultBackoffBase = 5 * time.Second
	// DefaultBackoffMax caps the delay between attempts.
	DefaultBackoffMax = 15 * time.Minute
	// DefaultWorkers is the concurrent jobs per client on the default queue.
	DefaultWorkers = 10
)

Default tuning for Client. Every value can be overridden in Config.

View Source
const (
	StatePending = "pending"
	StateRunning = "running"
	StateDone    = "done"
	StateDead    = "dead"
)

Job states. Dead is the dead-letter state.

Dead is the state of a job that used all its attempts. Dead jobs are kept for inspection and are never run again.

View Source
const MaxBatch = 1000

MaxBatch caps ClaimN's n. Rows claimed together share one owner, so larger batches amortize the round trip at the cost of a coarser split between claimers.

Variables

View Source
var Default = Schema{
	Table:    "job",
	ID:       "id",
	Priority: "priority",
	NextDue:  "next_due",
	State:    "state",
	Epoch:    "lease_epoch",
	Pending:  "pending",
	Claimed:  "claimed",
}

Default is the schema documented on the package, on table "job".

View Source
var ErrStale = errors.New("queue: lease is stale; the row was reclaimed or settled")

ErrStale reports that Settle or Fail no longer owns the row: it was reclaimed and claimed by another owner, settled, or already failed.

Functions

func Every

Every is a periodic schedule that fires every d after the previous run.

func Idempotent

func Idempotent(base *river.InsertOpts) *river.InsertOpts

Idempotent returns insert options under which a second insert of the same job is skipped for as long as the first one is pending, running, retrying, scheduled or completed (completed rows live until River's cleaner removes them, 24 hours by default).

A job is identified by its args: tag the field that carries the idempotency key with `river:"unique"` and only tagged fields are compared; with no tag the whole args value is. Dead and cancelled jobs do not block a new insert. base may be nil.

func JobsDDL

func JobsDDL(table string) (string, error)

JobsDDL returns the statements that create the job table and its indexes for the named relation, for use in a migration. The table may be schema-qualified.

func Migrate

func Migrate(ctx context.Context, db *sql.DB, schema string) error

Migrate brings River's tables in schema (empty for the search path) up to date. It is idempotent and safe to run from several processes at once. Client.Start calls it unless SkipMigrate is set.

func PeriodicJob

func PeriodicJob(p Periodic) *river.PeriodicJob

PeriodicJob converts p for Config.Periodic.

Types

type Backoff

type Backoff struct {
	Base time.Duration
	Max  time.Duration
}

Backoff is a bounded exponential retry policy: the delay after attempt n is Base * 2^(n-1), never more than Max. It plugs into River as the client's retry policy.

func (Backoff) Delay

func (b Backoff) Delay(attempt int) time.Duration

Delay returns the wait before the run that follows the given failed attempt (1 for the first failure).

func (Backoff) NextRetry

func (b Backoff) NextRetry(job *rivertype.JobRow) time.Time

NextRetry implements river.ClientRetryPolicy.

type Claimed

type Claimed struct {
	// ID is the row's key.
	ID int64
	// Epoch is the row's lease_epoch after the claim; pass it to Settle and
	// Fail. It is strictly larger than any epoch a previous owner held.
	Epoch int64
}

Claimed is one row handed to a claimer by ClaimN.

type Client

type Client struct {
	// contains filtered or unexported fields
}

Client is a River client over database/sql. Jobs are inserted with the caller's transaction, workers claim them with row locks that never block other workers, and the elected leader runs maintenance: scheduling, rescuing stuck jobs, and cleaning finished ones. It is safe for concurrent use.

func NewClient

func NewClient(cfg Config) (*Client, error)

NewClient validates cfg and builds a client. It does not reach the database.

func (*Client) AddPeriodic

func (c *Client) AddPeriodic(p Periodic) error

AddPeriodic registers p on a running or stopped client. Only the leader enqueues it.

func (*Client) InsertMany

func (c *Client) InsertMany(ctx context.Context, jobs []river.InsertManyParams) ([]*rivertype.JobInsertResult, error)

InsertMany enqueues jobs in one transaction of its own. Use InsertManyTx when the jobs must commit with other changes.

func (*Client) InsertManyTx

func (c *Client) InsertManyTx(ctx context.Context, tx *sql.Tx, jobs []river.InsertManyParams) ([]*rivertype.JobInsertResult, error)

InsertManyTx enqueues jobs in the caller's transaction. Jobs whose opts set UniqueOpts and match an existing job are skipped; the matching result has UniqueSkippedAsDuplicate set. Results are in input order. One batch must not contain the same unique key twice: PostgreSQL rejects the statement.

func (*Client) InsertTx

func (c *Client) InsertTx(ctx context.Context, tx *sql.Tx, args river.JobArgs, opts *river.InsertOpts) (*rivertype.JobInsertResult, error)

InsertTx enqueues a job in the caller's transaction. The job exists only if tx commits, and workers cannot see it before then.

func (*Client) Start

func (c *Client) Start(ctx context.Context) error

Start applies River's migrations (unless skipped) and begins working jobs, electing a leader for maintenance and periodic jobs. It returns once the client is running; call Stop to shut down.

func (*Client) Stop

func (c *Client) Stop(ctx context.Context) error

Stop stops fetching new jobs and waits for running ones to finish or ctx to end, whichever comes first.

type Config

type Config struct {
	// DB is the pool River works on and migrates. It must be a PostgreSQL
	// database/sql pool; the caller owns it and closes it after Stop.
	DB *sql.DB
	// Workers registers the job kinds this client runs. Leave nil for an
	// insert-only client that is never started.
	Workers *river.Workers
	// Workers per queue. Nil means river.QueueDefault with DefaultWorkers.
	Queues map[string]river.QueueConfig
	// Lease is how long a running job may go without finishing before it is
	// rescued: retried with backoff, or dead when attempts are used up. Set
	// it above the longest legitimate job; a job that outlives its lease can
	// run twice. The per-job timeout is set to nine tenths of the lease, so a
	// job is cancelled before it can be rescued. Zero means DefaultLease.
	Lease time.Duration
	// MaxAttempts per job, including the first run. Zero means
	// DefaultMaxAttempts. A job's own InsertOpts.MaxAttempts wins.
	MaxAttempts int
	// BackoffBase and BackoffMax shape the retry delay, see Backoff. Zero
	// means DefaultBackoffBase and DefaultBackoffMax.
	BackoffBase time.Duration
	BackoffMax  time.Duration
	// Periodic jobs are enqueued by the leader only, so a fleet of clients
	// inserts each occurrence once.
	Periodic []*river.PeriodicJob
	// NoLeader keeps this client out of leader election: it works jobs but
	// never runs maintenance, rescue or periodic enqueues. Use it for clients
	// with a partial worker set. Periodic must be empty when it is set.
	NoLeader bool
	// SkipMigrate stops Start from applying River's schema migrations. Set it
	// when the role keel runs as cannot create tables and Migrate runs
	// elsewhere.
	SkipMigrate bool
	// Schema is the PostgreSQL schema holding River's tables. It must already
	// exist. Empty means the connection's search path, normally public.
	Schema string
	// FetchPollInterval and FetchCooldown tune how often idle workers look
	// for new jobs. Zero keeps River's defaults.
	FetchPollInterval time.Duration
	FetchCooldown     time.Duration
	// Logger receives River's logs. Nil discards them.
	Logger *slog.Logger
}

Config configures a Client. Only DB is required.

type Depth

type Depth struct {
	// Ready are pending jobs whose run_at has passed.
	Ready int64
	// Delayed are pending jobs waiting for run_at, including retry backoff.
	Delayed int64
	Running int64
	Done    int64
	Dead    int64
}

Depth counts jobs by state.

type Filter

type Filter struct {
	// Kinds limits the claim to these kinds; empty means every kind.
	Kinds []string
}

Filter narrows what Claim considers.

type Job

type Job struct {
	ID             int64
	Kind           string
	Payload        json.RawMessage
	IdempotencyKey string
	State          string
	// Attempts counts claims, including the current one.
	Attempts    int
	MaxAttempts int
	// Epoch is the lease epoch; pass it to Heartbeat, Complete and Fail.
	Epoch          int64
	RunAt          time.Time
	LeaseExpiresAt time.Time
	Result         json.RawMessage
	LastError      string
	CreatedAt      time.Time
	FinishedAt     time.Time
}

Job is one row of the job table.

type Jobs

type Jobs struct {
	// contains filtered or unexported fields
}

Jobs is a job queue over one table: enqueue with idempotency, batched claims with SKIP LOCKED, leases with heartbeats, retry with exponential backoff and a dead-letter state. Every transition after a claim is fenced on the lease epoch, so a worker that lost its lease gets ErrStale. It is safe for concurrent use.

func NewJobs

func NewJobs(db *sql.DB, cfg JobsConfig) (*Jobs, error)

NewJobs returns Jobs over db. It does not reach the database.

func (*Jobs) Claim

func (j *Jobs) Claim(ctx context.Context, limit int, f Filter) ([]Job, error)

Claim leases up to max due jobs for LeaseTTL: pending jobs whose run_at has passed, and running jobs whose lease expired (a crashed or stalled worker), provided attempts remain. It first dead-letters expired jobs that have none left. Concurrent claimers get disjoint sets. max is clamped to 1..MaxBatch.

func (*Jobs) Complete

func (j *Jobs) Complete(ctx context.Context, id, epoch int64, result json.RawMessage) error

Complete marks the job done and stores result, which must be valid JSON or empty. It returns ErrStale if the caller no longer holds the lease; a second Complete of the same lease is stale.

func (*Jobs) DeadLetters

func (j *Jobs) DeadLetters(ctx context.Context, limit int) ([]Job, error)

DeadLetters lists up to limit dead jobs, most recently dead first.

func (*Jobs) Depth

func (j *Jobs) Depth(ctx context.Context) (Depth, error)

Depth returns the queue depth.

func (*Jobs) Enqueue

func (j *Jobs) Enqueue(ctx context.Context, n NewJob) (job Job, created bool, err error)

Enqueue adds a job. With an IdempotencyKey, a job of the same Kind and key that already exists is returned unchanged and created is false; concurrent enqueues of one key add exactly one row.

func (*Jobs) Fail

func (j *Jobs) Fail(ctx context.Context, id, epoch int64, cause error) (string, error)

Fail records cause and releases the job. While attempts remain it is retried after BackoffBase * 2^(attempts-1), capped at BackoffMax; once Attempts reaches MaxAttempts it becomes dead. It returns the new state, StatePending or StateDead, or ErrStale if the caller no longer holds the lease.

func (*Jobs) Heartbeat

func (j *Jobs) Heartbeat(ctx context.Context, id, epoch int64) error

Heartbeat extends the lease of a claimed job by LeaseTTL. It returns ErrStale if the job was reclaimed, completed or failed.

func (*Jobs) Reap

func (j *Jobs) Reap(ctx context.Context) (int64, error)

Reap moves running jobs whose lease expired on their final attempt to the dead-letter state and returns how many it moved. Claim calls it; call it on a timer too if nothing claims for long stretches.

func (*Jobs) Requeue

func (j *Jobs) Requeue(ctx context.Context, id int64) error

Requeue returns a dead job to pending with its attempts reset. It returns ErrStale if the job is not dead.

func (*Jobs) Stuck

func (j *Jobs) Stuck(ctx context.Context, grace time.Duration, limit int) ([]Job, error)

Stuck lists up to limit running jobs whose lease expired more than grace ago and that nothing has reclaimed: the workers are gone and no claimer is polling, or the job has no attempts left and Reap has not run.

func (*Jobs) Throughput

func (j *Jobs) Throughput(ctx context.Context, window time.Duration) (Throughput, error)

Throughput counts jobs completed or dead-lettered in the last window.

type JobsConfig

type JobsConfig struct {
	// Table is the relation name; default "queue_job".
	Table string
	// LeaseTTL is how long a claim or heartbeat keeps a job; default 30s. A
	// running job whose lease has expired is reclaimed by the next Claim.
	LeaseTTL time.Duration
	// DefaultMaxAttempts applies to jobs enqueued without MaxAttempts; default 5.
	DefaultMaxAttempts int
	// BackoffBase is the retry delay after the first failed attempt; each
	// further attempt doubles it. Default 1s.
	BackoffBase time.Duration
	// BackoffMax caps the retry delay; default 1h.
	BackoffMax time.Duration
}

JobsConfig configures Jobs. Zero fields take the documented default.

type NewJob

type NewJob struct {
	Kind    string
	Payload json.RawMessage // JSON; empty means {}
	// IdempotencyKey, when set, makes Enqueue return the existing job for
	// the same Kind and key instead of adding another.
	IdempotencyKey string
	// MaxAttempts overrides JobsConfig.DefaultMaxAttempts when positive.
	MaxAttempts int
	// RunAt delays the first claim; zero means now.
	RunAt time.Time
}

NewJob describes a job to enqueue.

type Periodic

type Periodic struct {
	// ID names the schedule; it must be unique per client.
	ID string
	// Schedule says when to fire. Use Every, or any river.PeriodicSchedule
	// such as one backed by a cron parser.
	Schedule river.PeriodicSchedule
	// Make builds the job for one occurrence.
	Make func() (river.JobArgs, *river.InsertOpts)
	// RunOnStart also enqueues one occurrence when a leader starts, so a long
	// interval does not wait for the first tick after a failover.
	RunOnStart bool
}

Periodic describes cron-style work: Make is called each time the schedule fires and returns the job to enqueue.

type Queue

type Queue struct {
	// contains filtered or unexported fields
}

Queue claims, settles and fails rows of one job table on one pool. It is safe for concurrent use.

func New

func New(db *sql.DB) (*Queue, error)

New returns a Queue over db that uses Default. It does not reach the database; the table is checked by the first statement.

func NewWithSchema

func NewWithSchema(db *sql.DB, schema Schema) (*Queue, error)

NewWithSchema returns a Queue over db for the named schema. Fields left zero take their Default value.

func (*Queue) ClaimN

func (q *Queue) ClaimN(ctx context.Context, n int) ([]Claimed, error)

ClaimN claims up to n due rows, in one statement and therefore one transaction: highest priority first (lowest priority value, then earliest next_due, then lowest id). Each returned row has state = Claimed and an epoch one larger than before.

Rows another claimer holds at the same moment are skipped, so concurrent claimers on one table get disjoint batches without waiting on each other. ClaimN returns fewer than n rows, possibly zero, when fewer than n rows are due. n is clamped to 1..MaxBatch; the returned slice is not ordered.

func (*Queue) Fail

func (q *Queue) Fail(ctx context.Context, id, epoch int64, backoff time.Duration) error

Fail returns the row to pending so it is claimed again after backoff, and retires the lease. It succeeds only while the row is still claimed at epoch; otherwise it returns ErrStale and changes nothing. A negative backoff is treated as zero, which makes the row due immediately.

func (*Queue) Settle

func (q *Queue) Settle(ctx context.Context, id, epoch int64) error

Settle marks the row done. It succeeds only while the row is still claimed at epoch; otherwise it returns ErrStale and changes nothing. A successful Settle retires the lease, so a second Settle of the same id and epoch is ErrStale.

type Schema

type Schema struct {
	// Table is the relation name, optionally schema-qualified ("schema.job").
	Table string
	// ID is the bigint key column.
	ID string
	// Priority orders claims; lower is claimed first.
	Priority string
	// NextDue is the timestamptz column; a row is due once it has passed.
	NextDue string
	// State is the text column holding Pending or Claimed.
	State string
	// Epoch is the bigint lease column; every claim adds one.
	Epoch string
	// Pending is the state of a row waiting to be claimed. ClaimN only
	// considers rows in this state.
	Pending string
	// Claimed is the state ClaimN sets and Settle and Fail require.
	Claimed string
}

Schema names the table, columns and states Queue reads and writes. The zero value is the schema in the package documentation, on table "job".

type Throughput

type Throughput struct {
	Window time.Duration
	Done   int64
	Dead   int64
	// PerSecond is Done divided by the window.
	PerSecond float64
}

Throughput is the number of jobs that finished within a window.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL