hopper

package module
v0.9.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 28, 2026 License: Apache-2.0 Imports: 28 Imported by: 1

README

hopper

A high-performance job queue and message broker for Go, built on Postgres.

hopper runs background jobs and delivers messages using the PostgreSQL database your application already has. There's no broker to operate and no separate worker service: it is a library that runs inside your process and scales out with it. Postgres is the first engine, behind a driver interface designed for more; SQLite is the second, for an application that runs on one machine.

  • Fast: set-based claim, finalize and insert (including COPY), a live table that stays small however much history accumulates, and retention by dropping partitions. Throughput and latency targets are release gates.
  • Transactional enqueue and completion: jobs and messages commit or roll back with your data, and a worker can complete its job in the transaction that does its work.
  • Scale by adding replicas: work is claimed with FOR UPDATE SKIP LOCKED, and periodic jobs and maintenance run on one elected leader.
  • Reliable: retries with backoff, dead-lettering and replay, unique jobs, and lease-based rescue of jobs from crashed processes in seconds.
  • Complete: priorities, cron with time zones, snooze, cancellation, timeouts, awaitable results, global concurrency and rate limits, batches and workflows.
  • Messaging: topics, subscriptions and fan-out with AMQP-style routing, ordering keys, TTLs, idempotent publish and request/reply.
  • Idiomatic Go: generic, type-safe workers and messages, context, slog, and iterators. The core depends only on the standard library and pgx, and it runs under GODEBUG=fips140=only.

Status: v0.1.0. The first release covers the job queue (schema and migrations, the pgx and database/sql drivers, transactional and bulk inserts, batched claim and finalize, typed workers, retries with backoff, graceful shutdown, lease-based rescue of jobs from crashed processes, leader election, partition-drop retention, LISTEN/NOTIFY wake-ups, unique jobs, cron and interval schedules, cancellation, retry from the dead-letter queue, TTLs, pause and runtime queues, middleware, awaitable results, listing, events and stats), messaging (subscriptions with topic patterns, fan-out, dedup and ordering keys, streams with consumers, a SQL contract for producers in other languages), flow control, batches and workflows, plus the hopper CLI, hoppertest, hopperotel, the hopperui web UI and the drivertest conformance suite, and a SQLite driver (driver/hoppersqlite, pure Go, a module of its own) that passes it. docs/PLAN.md is the plan of record and docs/getting-started.md the place to start. Until v1.0.0, minor versions may change the API. Feedback is welcome through issues and PRs.

_, err := client.InsertTx(ctx, tx, SendEmail{UserID: 42}, nil)        // commits with tx
_, err = client.PublishTx(ctx, tx, AllocationCreated{ID: 42}, nil)  // fans out to subscribers

Inside a worker or subscriber, the client running the job comes from the context, so follow-up work needs no reference wired in after construction:

client, _ := hopper.ClientFromContext[pgx.Tx](ctx)
_, err := client.Insert(ctx, Next{ID: job.Args.ID}, nil)

The name

A feed hopper releases work into a machine one piece at a time. The name is also a tribute to RADM Grace Hopper, and it's a good name for something meant to replace a rabbit.

License

Apache-2.0

Documentation

Overview

Package hopper is a job queue and message broker for Go, built on the database the application already runs. PostgreSQL is the first engine.

A Client inserts jobs, claims them in batches, runs them on registered workers and finalizes results in batches. Any number of clients run against one database; work is claimed with FOR UPDATE SKIP LOCKED, so adding replicas adds throughput.

workers := hopper.NewWorkers()
hopper.AddWorker(workers, &SendEmailWorker{})

client, err := hopper.NewClient(hopperpgx.New(pool), &hopper.Config{
	Queues:  map[string]hopper.QueueConfig{hopper.QueueDefault: {MaxWorkers: 100}},
	Workers: workers,
})
g.Go(func() error { return client.Run(ctx) })

_, err = client.InsertTx(ctx, tx, SendEmail{UserID: 42}, nil) // commits with tx

Jobs run at least once. A job runs twice only when the process running it is lost mid-job, so workers must be idempotent.

The database schema is installed with the hoppermigrate package.

Index

Constants

View Source
const (
	DefaultJobTimeout         = time.Minute
	DefaultMaxAttempts        = 25
	DefaultStopTimeout        = 30 * time.Second
	DefaultPollInterval       = time.Second
	DefaultLeaseTTL           = 15 * time.Second
	DefaultCompletedRetention = 24 * time.Hour
	DefaultFailedRetention    = 7 * 24 * time.Hour
	DefaultStreamRetention    = 7 * 24 * time.Hour
)

Defaults for Config fields.

View Source
const (
	JobStatePending   = driver.JobStatePending
	JobStateAvailable = driver.JobStateAvailable
	JobStateScheduled = driver.JobStateScheduled
	JobStateRunning   = driver.JobStateRunning
	JobStateRetryable = driver.JobStateRetryable
	JobStateCompleted = driver.JobStateCompleted
	JobStateCanceled  = driver.JobStateCanceled
	JobStateDiscarded = driver.JobStateDiscarded
)

Job states. Available, scheduled and retryable all mean "claim once scheduled_at has passed"; they are distinct only so operators can see why a job is waiting.

View Source
const (
	// RescueClientLost is the reason when the client running the attempt
	// lost its lease.
	RescueClientLost = driver.RescueClientLost
	// RescueStuck is the reason when the attempt ran past
	// Config.RescueStuckAfter.
	RescueStuck = driver.RescueStuck
	// RescueLeaseExpired is the reason when the attempt's job lease was not
	// renewed in time.
	RescueLeaseExpired = driver.RescueLeaseExpired
)

The reasons hopper ends attempts.

View Source
const (
	DefaultJobLogBytes    = 32 << 10
	DefaultJobLogAttempts = 10
)

Defaults for JobLogsConfig fields.

View Source
const (
	MaxJobLogBytes      = 1 << 20
	MaxJobLogAttempts   = 100
	MaxJobLogTotalBytes = 4 << 20
)

Limits of JobLogsConfig fields. A finalize that writes a log rewrites all of the job's, so MaxJobLogTotalBytes bounds MaxBytes times MaxAttempts, and with it what one write costs: about that much, as each entry may pass MaxBytes by its dropped-lines line and JSON escaping.

View Source
const (
	// StreamStartLatest delivers events appended from now on. It is the
	// default.
	StreamStartLatest = driver.StreamStartLatest
	// StreamStartEarliest delivers every retained event first.
	StreamStartEarliest = driver.StreamStartEarliest
)
View Source
const (
	MaxTagBytes = jobtags.MaxBytes
	MaxTags     = jobtags.Max
)

MaxTagBytes (255) is the longest tag InsertOpts.Tags accepts, and MaxTags (64) the most distinct tags a job may have.

View Source
const (
	// DependencyCancel cancels the dependent step, and its dependents in
	// turn. It is the default.
	DependencyCancel = driver.DependencyCancel
	// DependencyIgnore runs the dependent step once the steps it depends on
	// have all finished, whatever their outcome.
	DependencyIgnore = driver.DependencyIgnore
)
View Source
const ChannelControl = driver.ChannelControl

ChannelControl and the other channel names are the Postgres NOTIFY channels hopper uses; see the driver package.

View Source
const DefaultJobLeaseTTL = time.Minute

DefaultJobLeaseTTL is a job lease's time to live when LeaseOpts.TTL is zero.

View Source
const QueueDefault = "default"

QueueDefault is the queue jobs go to when none is given.

Variables

View Source
var (
	// ErrJobRunning is returned by JobRetry and JobDelete for a running
	// job, and by Leases.Claim for a job running at the fence it names.
	ErrJobRunning = driver.ErrJobRunning
	// ErrUniqueConflict is returned by JobRetry when a live job holds the
	// same unique key.
	ErrUniqueConflict = driver.ErrUniqueConflict
	// ErrJobCanceled is the cause of a job's context cancellation after
	// JobCancel. Check it with context.Cause.
	ErrJobCanceled = errors.New("hopper: job canceled")
)

Sentinel errors from control operations.

View Source
var (
	ErrClientStarted = errors.New("hopper: client already started")
	ErrClientStopped = errors.New("hopper: client stopped")
	// ErrSchemaOutdated is returned by Start when the database's schema is
	// older than this release needs. Apply the missing migrations with
	// hoppermigrate.Up or `hopper migrate up` first.
	ErrSchemaOutdated = errors.New("hopper: database schema is out of date")
)

Sentinel errors from Client lifecycle methods.

View Source
var ErrInvalidFilter = driver.ErrInvalidFilter

ErrInvalidFilter is the error Jobs yields for a JobFilter that it cannot list by, such as an out-of-range priority, Metadata that is not a JSON object, an ordering key without a queue, a tag no job could have, more distinct TagsAny or TagsAll than MaxTags, Metadata that Postgres's jsonb refuses, such as a number beyond its numeric's range or a \u0000 escape, or a queue, kind, state or tag in characters the database's encoding lacks.

View Source
var ErrJobFenced = driver.ErrJobFenced

ErrJobFenced is returned by CompleteTx, RecordStep and RecordCursor when the attempt no longer owns its job: the job was rescued, after this client lost its lease or for RescueStuckAfter, and perhaps claimed again, or it was already finalized. It is also the cause of the job's context cancellation (check it with context.Cause) when the job, rescued while the attempt still ran, is claimed again on the same client. The Leases calls return it once the job lease they name is no longer the job's, and Leases.Claim when the live job's fence is not the one expected.

View Source
var ErrJobLocked = driver.ErrJobLocked

ErrJobLocked is returned by RecordStep and RecordCursor when a transaction of the worker's that wrote the job's row (RecordStepTx, RecordCursorTx, CompleteTx) has not ended: a write on the pool would wait for the worker itself. End that transaction before recording on the pool, and before a step run by RunSteps returns.

View Source
var ErrJobNotClaimable = driver.ErrJobNotClaimable

ErrJobNotClaimable is returned by Leases.Claim for a job that cannot be claimed now: pending, finalized, past its TTL, held back by its ordering key, or in a paused queue.

View Source
var ErrJobTimeout = errors.New("hopper: job timed out")

ErrJobTimeout is the cause of a job's context cancellation when the job ran past its timeout. Check it with context.Cause.

View Source
var ErrNotFound = driver.ErrNotFound

ErrNotFound is returned when a job does not exist, live or in history.

View Source
var ErrNotInJob = errors.New("hopper: context does not belong to a running job")

ErrNotInJob is returned by SetOutput, CompleteTx, RecordStep and RecordCursor outside a worker.

Functions

func AddWorkFunc

func AddWorkFunc[T JobArgs](workers *Workers, fn func(ctx context.Context, job *Job[T]) error)

AddWorkFunc registers a function as the worker for the kind of T, with default timeout and retry behavior.

func AddWorker

func AddWorker[T JobArgs](workers *Workers, worker Worker[T])

AddWorker registers worker for the kind of T and for its KindAliases. It panics if one of them is already registered, like http.Handle.

func Await

func Await[T any, TTx any](ctx context.Context, c *Client[TTx], id JobID) (T, error)

Await blocks until the job is finalized and returns its output decoded into T, or a *JobFailedError if it was canceled or discarded. Insert the job with InsertOpts.Await so that the finalize is announced and Await returns as soon as it commits; otherwise, and on a client that is not started, Await polls every Config.PollInterval. A completed job without output yields the zero T. A job that does not exist, or no longer does (JobDelete), yields ErrNotFound.

func Cancel

func Cancel(err error) error

Cancel returns an error a worker can return to discard the job now, with no retry. The job is archived as discarded with err recorded.

func CompleteTx added in v0.7.0

func CompleteTx[TTx any, T JobArgs](ctx context.Context, tx TTx, job *Job[T]) error

CompleteTx completes the running job inside tx, so that the worker's own writes and the job's completion commit together or not at all. ctx must be the context the worker was given. The job moves to history (unless its queue has DeleteCompleted) with the output SetOutput recorded before the call; awaited jobs, batches and workflow dependents are handled as on any completion, once tx commits.

The completion is fenced like every finalize: if the attempt no longer owns the job, nothing is written and ErrJobFenced is returned, and the worker should roll tx back. Once the worker returns, the client reads the job to learn whether tx committed. If it did, the job is finished, whatever the worker returned, and the job.completed event fires. If it did not, the attempt fails with the worker's error, or with a failure if the worker returned nil, and the job is retried. A worker that returns before tx ends holds up only its own job's result, until tx ends, on an engine that locks rows.

Call it last, just before the commit. From CompleteTx until tx ends, tx holds the job's row and those of workflow dependents it promotes or cancels, and holds off the creation and drop of history partitions. The job's siblings in its batch finalize without waiting for it.

On an engine that runs one writer at a time (driver.Capabilities.SingleWriter, SQLite), tx holds the whole database for writes from its first statement until it ends: every other write waits for it, the results of other jobs and lease renewals included, and a write of the worker's own on the pool fails rather than wait for tx, provided tx was begun with the worker's ctx (or one derived from it), which names the job's attempt.

func (w *Worker) Work(ctx context.Context, job *hopper.Job[Args]) error {
	tx, err := w.pool.Begin(ctx)
	if err != nil {
		return err
	}
	defer tx.Rollback(ctx)
	if err := apply(ctx, tx, job.Args); err != nil {
		return err
	}
	if err := hopper.CompleteTx(ctx, tx, job); err != nil {
		return err
	}
	return tx.Commit(ctx)
}

func Consume

func Consume[T any](workers *Workers, consumer Consumer, fn func(ctx context.Context, msg *Message[T]) error)

Consume registers a handler for a consumer's deliveries. The consumer is created (at its Start) or updated when a client using workers starts, and the leader delivers its events in order as jobs; fn receives them as Subscribe's handler does, with Message.Stream and Message.Position set. An event's key becomes the delivery's ordering key, so events with the same key are handled one at a time, in order.

It panics on an invalid consumer or a name registered twice, like AddWorker.

func ContextWithClient added in v0.2.0

func ContextWithClient[TTx any](ctx context.Context, client *Client[TTx]) context.Context

ContextWithClient returns a context from which ClientFromContext returns client. Jobs get this from the client that runs them; it is for tests that call a worker directly, as with hoppertest.Work.

func Logger added in v0.8.0

func Logger(ctx context.Context) *slog.Logger

Logger returns the logger of the job whose context ctx is. Its lines go to Config.Logger, with the job's ID, kind and attempt, and are kept with the job, in text form, as the logs of the attempt: Client.JobLogs returns them, and hopper jobs get and hopperui show them. They are written with the attempt's result, so they cost no writes of their own; a job keeps those of its latest attempts, bounded by Config.JobLogs, through retries and into history, and they leave with the job. With Config.JobLogs.Store, the lines go to the store as they are logged instead, and nothing of them is written to the database.

func (w *Importer) Work(ctx context.Context, job *hopper.Job[ImportArgs]) error {
	log := hopper.Logger(ctx)
	log.Info("downloading", "url", job.Args.URL)
	...
}

Without a store, the lines of an attempt whose result is not written are lost with it: one whose client crashes or loses its lease, or that is rescued for running past Config.RescueStuckAfter. The rescue's error says which. A job completed with CompleteTx keeps the lines logged before that call. A worker run with Workers.Work logs as the attempt that ran it, if any. Outside a job, Logger returns slog.Default().

func PerSecond

func PerSecond(n float64) float64

PerSecond is a readable RateLimit value.

func RecordCursor added in v0.8.0

func RecordCursor[T JobArgs](ctx context.Context, job *Job[T], cursor any) error

RecordCursor records a cursor within the step under way, keeping the last finished step, for a step that works through many items: a later attempt resumes the step at the cursor (StepCursor) instead of from its start. A nil cursor clears it.

func RecordCursorTx added in v0.8.0

func RecordCursorTx[TTx any, T JobArgs](ctx context.Context, tx TTx, job *Job[T], cursor any) error

RecordCursorTx is RecordCursor inside tx, like RecordStepTx.

func RecordStep added in v0.8.0

func RecordStep[T JobArgs](ctx context.Context, job *Job[T], step string, cursor any) error

RecordStep records that the running job finished step, with an optional cursor marking a position within the work that follows: any value encoding/json can encode, or nil. ctx must be the context the worker was given. The progress is written at once and kept through retries, snoozes and a JobRetry from history (unless FromStart), so a later attempt finds it in job.Progress and can resume after step; job.Progress is updated too. RunSteps uses it to resume a job at its first unfinished step.

The write is fenced like a finalize: if the attempt no longer owns the job (it was rescued, or finalized), nothing is written and ErrJobFenced is returned.

RecordStep, RecordCursor and their Tx forms may be called from several goroutines: an attempt's records are written one at a time, and job.Progress ends as the last one stored. Reading job.Progress while another goroutine records is a data race.

func RecordStepTx added in v0.8.0

func RecordStepTx[TTx any, T JobArgs](ctx context.Context, tx TTx, job *Job[T], step string, cursor any) error

RecordStepTx is RecordStep inside tx, so that a step's own writes and the record that it finished commit together or not at all, and a later attempt neither repeats a step that committed nor skips one that did not. job.Progress is updated as if tx commits; if it rolls back, the worker should return an error. Until tx ends it holds the job's row: end tx before the step returns, since RunSteps records the next finished step on the pool, and a record on the pool while tx is open returns ErrJobLocked. A job's finalize skips a row tx still holds, and is retried once tx ends. On an engine that runs one writer at a time (driver.Capabilities.SingleWriter, SQLite), tx holds the whole database for writes from its first statement until it ends: every other write waits for it, lease renewals included, and a write of the worker's own on the pool fails rather than wait for tx, provided tx was begun with the worker's ctx.

func RunSteps added in v0.8.0

func RunSteps[T JobArgs](ctx context.Context, job *Job[T], steps ...JobStep[T]) error

RunSteps runs a job's steps in order, starting after the last one its progress records as finished, and records each step as finished once it returns nil (RecordStep), unless the step recorded that itself with RecordStepTx or the job was completed with CompleteTx. It returns the first error a step returns, which fails the attempt as any worker error does; the next attempt starts at that step, with the cursor it recorded, if any, in job.Progress.

func (w *Importer) Work(ctx context.Context, job *hopper.Job[ImportArgs]) error {
	return hopper.RunSteps(ctx, job,
		hopper.NamedStep("download", w.download),
		hopper.NamedStep("load", w.load),
		hopper.NamedStep("notify", w.notify),
	)
}

A job whose progress names a step that is not among steps is discarded (Cancel), since every attempt would fail the same way: steps were renamed or removed while it waited. So is one run with a step that has no name or is listed twice.

func SetOutput

func SetOutput(ctx context.Context, v any) error

SetOutput records the job's result, which is stored with the finalized job, returned by JobGet and delivered to Await. The last call wins. Output is kept only for archived jobs, so it is lost on queues with DeleteCompleted. A job finalized with CompleteTx stores the output set before that call.

func Snooze

func Snooze(d time.Duration) error

Snooze returns an error a worker can return to reschedule the job after d without using up an attempt.

func StepCursor added in v0.8.0

func StepCursor[C any](job *JobRow) (cursor C, ok bool, err error)

StepCursor decodes the cursor of the job's progress into a C. ok is false when the job has no cursor.

func Subscribe

func Subscribe[T any](workers *Workers, sub Subscription, fn func(ctx context.Context, msg *Message[T]) error)

Subscribe registers a consumer for a subscription. The subscription row is created or updated when a client using workers starts. fn is called with each delivery: returning nil acks it, an error nacks it with backoff (capped at five minutes for ordered deliveries), and Snooze and Cancel work as for jobs. Deliveries out of attempts are dead-lettered in history, where ReplayDiscarded can re-drive them.

It panics on an invalid subscription or a name registered twice, like AddWorker.

Types

type AppendOpts

type AppendOpts struct {
	// Key orders the event's deliveries with others of the same key: a
	// consumer handles them one at a time, in order.
	Key string
	// Headers travel with the event and are available on Message.
	Headers map[string]string
}

AppendOpts customizes an append. A nil *AppendOpts means the defaults.

type AttemptError

type AttemptError = driver.AttemptError

AttemptError records one failed attempt of a job.

type AttemptLog added in v0.8.0

type AttemptLog = driver.AttemptLog

AttemptLog is the log output of one attempt of a job. See Logger.

type Batch

type Batch[TTx any] struct {
	// contains filtered or unexported fields
}

Batch collects jobs to insert together with completion callbacks.

func (*Batch[TTx]) Add

func (b *Batch[TTx]) Add(args JobArgs, opts *InsertOpts)

Add queues a job for the batch.

func (*Batch[TTx]) Insert

func (b *Batch[TTx]) Insert(ctx context.Context) (*BatchResult, error)

Insert inserts the batch and its jobs.

func (*Batch[TTx]) InsertTx

func (b *Batch[TTx]) InsertTx(ctx context.Context, tx TTx) (*BatchResult, error)

InsertTx inserts the batch and its jobs in the caller's transaction.

type BatchOpts

type BatchOpts struct {
	OnSuccess  JobArgs
	OnFailure  JobArgs
	OnComplete JobArgs
	// CallbackOpts applies to every callback job.
	CallbackOpts *InsertOpts
	// Metadata is stored with the batch.
	Metadata []byte
}

BatchOpts sets a batch's callbacks. Each is inserted as a job when the batch completes: OnSuccess if no job of the batch was canceled or discarded, OnFailure otherwise, and OnComplete either way. Callback metadata gains "batch_id" and "batch_failed" (the number of failed jobs).

type BatchResult

type BatchResult struct {
	ID   JobID
	Jobs []*InsertResult
}

BatchResult reports an inserted batch.

type BatchRow

type BatchRow = driver.BatchRow

BatchRow is a batch's progress.

type CancelError

type CancelError struct {
	Err error
}

CancelError is returned by a worker to discard the job immediately, with no retry. Use Cancel to create one.

func (*CancelError) Error

func (e *CancelError) Error() string

func (*CancelError) Unwrap

func (e *CancelError) Unwrap() error

type Client

type Client[TTx any] struct {
	// contains filtered or unexported fields
}

Client is the per-process runtime. It inserts, claims, works and finalizes jobs, and holds a liveness lease while running. TTx is the engine's transaction type, fixed by the driver.

func ClientFromContext added in v0.2.0

func ClientFromContext[TTx any](ctx context.Context) (*Client[TTx], bool)

ClientFromContext returns the client that is running the job whose context ctx is, so a worker or subscriber can insert follow-up jobs, publish, or query without holding a client reference. That matters for code that is wired before the client exists: the workers are registered first and the client is built from them. The transaction type must match the client's; a mismatch, or a context from outside a job, returns false.

func (w *Worker) Work(ctx context.Context, job *hopper.Job[Args]) error {
	client, ok := hopper.ClientFromContext[pgx.Tx](ctx)
	if !ok {
		return hopper.ErrNotInJob
	}
	_, err := client.Insert(ctx, Next{ID: job.Args.ID}, nil)
	return err
}

func NewClient

func NewClient[TTx any](d driver.Driver[TTx], cfg *Config) (*Client[TTx], error)

NewClient validates cfg and returns a client. Nothing runs until Start.

func (*Client[TTx]) BatchGet

func (c *Client[TTx]) BatchGet(ctx context.Context, id JobID) (*BatchRow, error)

BatchGet returns a batch's progress.

func (*Client[TTx]) Clients

func (c *Client[TTx]) Clients(ctx context.Context) ([]*ClientRow, error)

Clients lists every client lease, live or expired (expired rows are pruned by the leader).

func (*Client[TTx]) Events

func (c *Client[TTx]) Events(ctx context.Context, kinds ...EventKind) iter.Seq[Event]

Events streams events of the given kinds (all kinds if none) until ctx is done. Events are delivered best-effort: a consumer that falls behind by more than a buffer's worth misses some.

func (*Client[TTx]) Insert

func (c *Client[TTx]) Insert(ctx context.Context, args JobArgs, opts *InsertOpts) (*InsertResult, error)

Insert inserts one job. It can be called before Start and on an insert-only client. When this client works the job's queue, its producer is woken directly, with no database round trip.

func (*Client[TTx]) InsertMany

func (c *Client[TTx]) InsertMany(ctx context.Context, params []InsertParams) ([]*InsertResult, error)

InsertMany inserts a batch of jobs in one statement, or with COPY for large batches without unique keys. Results are in input order.

func (*Client[TTx]) InsertManyTx

func (c *Client[TTx]) InsertManyTx(ctx context.Context, tx TTx, params []InsertParams) ([]*InsertResult, error)

InsertManyTx is InsertMany in the caller's transaction.

func (*Client[TTx]) InsertTx

func (c *Client[TTx]) InsertTx(ctx context.Context, tx TTx, args JobArgs, opts *InsertOpts) (*InsertResult, error)

InsertTx inserts one job in the caller's transaction. The job exists only if tx commits, and becomes visible to workers when it does.

func (*Client[TTx]) InsertWorkflow

func (c *Client[TTx]) InsertWorkflow(ctx context.Context, wf *Workflow) (*WorkflowResult, error)

InsertWorkflow inserts a workflow's steps and dependencies atomically.

func (*Client[TTx]) InsertWorkflowTx

func (c *Client[TTx]) InsertWorkflowTx(ctx context.Context, tx TTx, wf *Workflow) (*WorkflowResult, error)

InsertWorkflowTx is InsertWorkflow in the caller's transaction.

func (*Client[TTx]) JobCancel

func (c *Client[TTx]) JobCancel(ctx context.Context, id JobID) (*JobRow, error)

JobCancel cancels a job. A waiting job moves to history as canceled. A running job, on any client, has its context canceled (with cause ErrJobCanceled) and finalizes as canceled unless it completes anyway. A finalized job is returned unchanged. The job as it is afterwards is returned.

func (*Client[TTx]) JobCancelTx added in v0.8.0

func (c *Client[TTx]) JobCancelTx(ctx context.Context, tx TTx, id JobID) (*JobRow, error)

JobCancelTx is JobCancel inside the caller's transaction, so the cancel commits or rolls back with the caller's own changes. A running job's client is told when the transaction commits.

Until tx ends it holds the job's row, so a running job that finishes meanwhile has its result wait for tx; its client's other results do not. Call it late in tx, as CompleteTx. On an engine that runs one writer at a time, tx holds every write in the database up until it ends.

func (*Client[TTx]) JobDelete added in v0.8.0

func (c *Client[TTx]) JobDelete(ctx context.Context, id JobID) (*JobRow, error)

JobDelete removes a job that is not running, from the live table or from history, and returns it as it was last stored. A waiting job is canceled as JobCancel does, but keeps no history row: its batch counts it as failed, its dependents react as they do to a cancel, and Await on it returns ErrNotFound rather than waiting forever. It returns ErrJobRunning for a running job (cancel it, then delete it once it has finished) and ErrNotFound for an unknown one.

func (*Client[TTx]) JobDeleteTx added in v0.8.0

func (c *Client[TTx]) JobDeleteTx(ctx context.Context, tx TTx, id JobID) (*JobRow, error)

JobDeleteTx is JobDelete inside the caller's transaction. Until tx ends it holds the job's row and, for a waiting job, those of the workflow dependents it cancels or promotes; on an engine that runs one writer at a time, every write in the database.

func (*Client[TTx]) JobGet

func (c *Client[TTx]) JobGet(ctx context.Context, id JobID) (*JobRow, error)

JobGet returns a job by ID, live or from history. It returns ErrNotFound if the job does not exist or its history has been dropped.

func (*Client[TTx]) JobLogs added in v0.8.0

func (c *Client[TTx]) JobLogs(ctx context.Context, id JobID) ([]AttemptLog, error)

JobLogs returns the logs a job's worker kept with Logger, of its latest attempts, oldest first, live or from history. It returns ErrNotFound if the job does not exist or its history has been dropped.

With Config.JobLogs.Store, the logs the job's row holds, from before the store or from clients without it, come first, then the store's, in attempt order. A store log that a later one of the same attempt number follows, or whose number is past the job's, was a snooze, which gave its attempt back; as in the row, only the latest snooze of each attempt number is returned, and only the latest JobLogs.MaxAttempts logs. A job that is gone yields the store's logs of it, if it has any. If the store fails, the row's logs are returned with the error.

func (*Client[TTx]) JobRetry

func (c *Client[TTx]) JobRetry(ctx context.Context, id JobID, opts ...RetryOption) (*JobRow, error)

JobRetry makes a job run again as soon as possible: a canceled or discarded job is re-driven from history with one more attempt allowed if it had run out, and a waiting job has its schedule brought forward. An explicit retry overrides the wait, so the job's TTL no longer applies: a job past its deadline still runs. The job resumes from its progress (RecordStep), unless FromStart is given. It returns ErrJobRunning for a running job.

func (*Client[TTx]) JobRetryOrderingKey added in v0.7.0

func (c *Client[TTx]) JobRetryOrderingKey(ctx context.Context, queue, orderingKey string) (int, error)

JobRetryOrderingKey wakes an ordering key: its jobs waiting out a snooze, a retry delay or their schedule become runnable now, and still run one at a time, oldest first. Unlike JobRetry it keeps each job's TTL, and leaves a job already past it to expire. A job of the key that is running is not interrupted; if its attempt ends in a snooze or a retry, it runs again at once instead of after its delay. It returns how many waiting jobs it woke.

func (*Client[TTx]) JobRetryOrderingKeyTx added in v0.8.0

func (c *Client[TTx]) JobRetryOrderingKeyTx(ctx context.Context, tx TTx, queue, orderingKey string) (int, error)

JobRetryOrderingKeyTx is JobRetryOrderingKey inside the caller's transaction. The queue is woken when the transaction commits.

Until tx ends it holds the rows of the jobs it woke and of the key's running job, whose result waits for tx; its client's other results do not. Call it late in tx, as CompleteTx. On an engine that runs one writer at a time, tx holds every write in the database up until it ends.

func (*Client[TTx]) JobRetryTx added in v0.8.0

func (c *Client[TTx]) JobRetryTx(ctx context.Context, tx TTx, id JobID, opts ...RetryOption) (*JobRow, error)

JobRetryTx is JobRetry inside the caller's transaction. The job's queue is woken when the transaction commits.

func (*Client[TTx]) Jobs

func (c *Client[TTx]) Jobs(ctx context.Context, filter JobFilter) iter.Seq2[*JobRow, error]

Jobs lists jobs matching the filter, live and from history, in ID order (which is insertion order), or newest first with NewestFirst. It pages transparently; stop iterating to stop fetching.

func (*Client[TTx]) JobsTx

func (c *Client[TTx]) JobsTx(ctx context.Context, tx TTx, filter JobFilter) iter.Seq2[*JobRow, error]

JobsTx is Jobs inside the caller's transaction, so that jobs inserted with InsertTx can be listed before the transaction commits. A filter the database refuses (ErrInvalidFilter) leaves the transaction usable.

func (*Client[TTx]) Leases added in v0.9.0

func (c *Client[TTx]) Leases() *Leases[TTx]

Leases returns the controller of job leases: the attempts of jobs claimed by ID, which run outside any client, such as on a remote worker that reports through whichever client it reaches. The calls work on a client that is not started.

func (*Client[TTx]) NewBatch

func (c *Client[TTx]) NewBatch(opts BatchOpts) *Batch[TTx]

NewBatch starts a batch. Add jobs to it, then Insert or InsertTx.

func (*Client[TTx]) PeriodicJobs added in v0.8.0

func (c *Client[TTx]) PeriodicJobs() *PeriodicJobs

PeriodicJobs returns the client's periodic jobs.

func (*Client[TTx]) Publish

func (c *Client[TTx]) Publish(ctx context.Context, msg Topic, opts *PublishOpts) (*PublishResult, error)

Publish delivers a message to every subscription matching its topic, as one job each, in one statement. With no matching subscription nothing is inserted, as with a broker.

func (*Client[TTx]) PublishTx

func (c *Client[TTx]) PublishTx(ctx context.Context, tx TTx, msg Topic, opts *PublishOpts) (*PublishResult, error)

PublishTx is Publish in the caller's transaction: the deliveries exist only if tx commits.

func (*Client[TTx]) Queues

func (c *Client[TTx]) Queues() *QueueControl[TTx]

Queues returns the queue controller.

func (*Client[TTx]) ReplayDiscarded

func (c *Client[TTx]) ReplayDiscarded(ctx context.Context, subscription string) (int, error)

ReplayDiscarded re-drives every dead-lettered delivery of a subscription and returns how many.

func (*Client[TTx]) Run

func (c *Client[TTx]) Run(ctx context.Context) error

Run starts the client and blocks until ctx is canceled, then stops it, waiting up to Config.StopTimeout for running jobs. It returns nil after a clean drain, so it fits errgroup.

func (*Client[TTx]) Start

func (c *Client[TTx]) Start(ctx context.Context) error

Start registers the client's lease and begins claiming and working jobs. ctx bounds the startup itself; the client keeps running after it ends. Use Run for a lifecycle tied to a context. It returns ErrSchemaOutdated, naming the missing migrations, if the database's schema is older than this release needs.

func (*Client[TTx]) Stats

func (c *Client[TTx]) Stats(ctx context.Context) (*Stats, error)

Stats returns queue depths by state, the age of the oldest claimable job, recent throughput, the running count by client and the current leader.

func (*Client[TTx]) Stop

func (c *Client[TTx]) Stop(ctx context.Context) error

Stop shuts the client down in three steps: stop claiming and give up leadership; wait for running jobs to finish and flush their results; release the lease. If ctx expires during the wait, every job's context is canceled, whatever finishes within a short grace period is flushed, and ctx's error is returned. Jobs still running then are rescued by another client once this one's lease expires.

func (*Client[TTx]) Streams

func (c *Client[TTx]) Streams() *Streams[TTx]

Streams returns the client's stream log.

func (*Client[TTx]) Subscriptions

func (c *Client[TTx]) Subscriptions(ctx context.Context) ([]*SubscriptionRow, error)

Subscriptions lists every subscription known to the cluster.

func (*Client[TTx]) WorkflowGet

func (c *Client[TTx]) WorkflowGet(ctx context.Context, id JobID) (*WorkflowRow, error)

WorkflowGet returns a workflow's progress, its jobs (live and finished) and its edges.

type ClientRow

type ClientRow = driver.ClientRow

ClientRow is a client process's lease.

type Codec

type Codec interface {
	Marshal(v any) ([]byte, error)
	Unmarshal(data []byte, v any) error
}

Codec encodes job args and outputs for storage. The encoded form must be valid JSON, because it is stored in a jsonb column; a codec that encrypts or compresses wraps its output in a JSON string. The default is encoding/json.

type Config

type Config struct {
	// Queues lists the queues this client works, with per-queue settings. A
	// client with no queues only inserts jobs.
	Queues map[string]QueueConfig
	// Workers is the registry of workers. Required when Queues is set.
	Workers *Workers
	// Periodic lists jobs inserted on a schedule by the leader. See Cron
	// and Every. It is the initial set, which Client.PeriodicJobs changes
	// at runtime. NewClient builds each job's insert and fails if it cannot
	// be built, apart from StrictKinds: a StrictKinds client records only
	// jobs of kinds it has a worker for, checking again as workers are
	// registered, and its own inserts of the others fail.
	Periodic []PeriodicJob
	// Middleware wraps job execution and inserts, outermost first.
	Middleware []Middleware
	// Logger defaults to slog.Default().
	Logger *slog.Logger
	// JobLogs bounds the log lines kept with each job, from its worker's
	// Logger.
	JobLogs JobLogsConfig
	// Codec encodes args and outputs. Defaults to JSONCodec.
	Codec Codec
	// JobTimeout is how long a job may run before its context is canceled,
	// unless its worker's Timeout says otherwise. Defaults to one minute.
	JobTimeout time.Duration
	// MaxAttempts is the default number of attempts before a job is
	// discarded. Defaults to 25.
	MaxAttempts int
	// OnError is called for every failed attempt this client runs, of any
	// kind, with the worker's error: a returned error, a recovered panic
	// (*PanicError), a timeout (ErrJobTimeout), a Cancel, or a kind with no
	// worker (*UnknownKindError). It is not called for a Snooze, nor when the
	// job was canceled with JobCancel or the client is stopping, nor for a
	// job the leader rescues from a lost client or RescueStuckAfter. What it
	// returns overrides the outcome; the zero ErrorOutcome keeps it. It runs
	// on the worker's goroutine, concurrently with other jobs, before the
	// result is written, so it should return quickly; a panic in it is
	// logged and keeps the outcome. Its context carries the client
	// (ClientFromContext) and outlives the job's. It runs after Middleware,
	// which sees the worker's error, not the outcome OnError chooses; the
	// job's events report the outcome.
	OnError func(ctx context.Context, job *JobRow, err error) ErrorOutcome
	// RetryPolicy chooses when a failed job runs again, for workers whose
	// NextRetry returns the zero time, unless the queue's RetryPolicy does.
	// Defaults to attempt^4 + 5 seconds with ±10% jitter, capped at 24
	// hours.
	RetryPolicy RetryPolicy
	// StopTimeout bounds how long Run waits for running jobs after its
	// context is canceled, before canceling them. Defaults to 30 seconds.
	StopTimeout time.Duration
	// PollInterval is how often an idle producer checks its queue for work
	// that arrived without a wake-up. Defaults to one second.
	PollInterval time.Duration
	// StrictKinds rejects inserts whose kind has no worker on this client.
	// By default any kind can be inserted, since another service may work it.
	StrictKinds bool
	// Hostname identifies this process in the clients table. Defaults to
	// os.Hostname().
	Hostname string
	// CompletedRetention is how long completed jobs stay in history.
	// Defaults to 24 hours; negative keeps them forever. Retention is
	// enforced by the leader, so the leader's setting applies cluster-wide.
	CompletedRetention time.Duration
	// FailedRetention is how long canceled and discarded jobs stay in
	// history, which is the dead-letter queue. Defaults to 7 days; negative
	// keeps them forever.
	FailedRetention time.Duration
	// StreamRetention is how long stream events stay in the log, for
	// consumers that start from the earliest event or are moved back.
	// Defaults to 7 days; negative keeps them forever.
	StreamRetention time.Duration
	// NeverLead keeps this client out of leader elections. It still claims
	// and works jobs, but never runs the leader's duties: rescuing jobs from
	// lost clients, retention, maintenance, periodic jobs and stream
	// delivery. Use it for clients that share an install but must not run it,
	// such as a developer's process against a shared environment. At least
	// one client of every install must be able to lead. Its periodic jobs are
	// still recorded, and the leader schedules them while it runs, so such a
	// process registers none it should not run.
	NeverLead bool
	// LeaseTTL is how long this client's lease lasts without a renewal. It
	// is renewed three times per TTL; when it lapses, the client's running
	// jobs are rescued and the client cancels them. A longer lease rides
	// out a longer database outage before interrupting jobs, and delays the
	// rescue of a crashed client's jobs by as much. Defaults to 15 seconds.
	LeaseTTL time.Duration
	// RescueStuckAfter, if set, rescues jobs that have been running longer
	// than this even though their client's lease is live, for workers that
	// ignore their context. Off by default; timeouts cover most cases.
	RescueStuckAfter time.Duration
	// JobLeaseGrace is the least time a job lease is given after a cluster
	// restart, for holders that could not renew while no client ran. A
	// lease whose last renewal predates the start of a client whose own
	// lease is live is due no earlier than the first such start plus the
	// larger of its TTL and JobLeaseGrace, by database time. Zero gives
	// such a lease its TTL from that start; set it to how long remote
	// workers may take to reach the cluster again, such as their reconnect
	// backoff. The start of any client that works a queue counts (one that
	// only inserts registers none), as does one that re-registers after its
	// lease lapsed, so it also delays expiries across a rolling deploy, and
	// clients that each stop within the grace of their start, the next
	// starting before the last stops, keep a lease that is never renewed
	// again from expiring. The leader's setting applies.
	JobLeaseGrace time.Duration
}

Config configures a Client. Every field has a production default; the zero Config is an insert-only client.

type Consumer

type Consumer struct {
	// Name identifies the consumer; the delivery job kind is derived from
	// it.
	Name    string
	Pattern string
	// Queue is where deliveries go. Defaults to QueueDefault.
	Queue string
	// Start applies when the consumer is created. Defaults to
	// StreamStartLatest.
	Start StreamStart
	// MaxAttempts is the retry budget of deliveries. Zero means 25, or 10
	// for events with a key, whose retries block their key.
	MaxAttempts int
	// Metadata is merged into every delivery's metadata.
	Metadata json.RawMessage
}

Consumer reads the stream log from a position of its own and delivers the events whose topic matches Pattern to a queue, as jobs of kind "stream:<Name>". Unlike a Subscription, which is fanned out to at publish time, a consumer can start anywhere in the retained log and be moved (Streams.Seek), so late subscribers and replays see past events.

func (Consumer) Kind

func (c Consumer) Kind() string

Kind returns the job kind of the consumer's deliveries.

type ConsumerRow

type ConsumerRow = driver.StreamConsumerRow

ConsumerRow is a consumer as recorded in the database, with its position.

type CronSchedule

type CronSchedule struct {
	// contains filtered or unexported fields
}

CronSchedule is a parsed cron expression. Use ParseCron.

func ParseCron

func ParseCron(spec string) (*CronSchedule, error)

ParseCron parses a cron expression.

Standard five fields (minute, hour, day of month, month, day of week) or six with a leading seconds field. Each field takes "*", a value, a range ("1-5"), a step ("*/15", "1-30/5"), a list ("1,15") and, for months and days of the week, three-letter names ("JAN", "MON"). Day of week is 0-6 with 7 also meaning Sunday. When both day fields are restricted, a day matches if either does, as in Vixie cron.

The descriptors @yearly (@annually), @monthly, @weekly, @daily (@midnight) and @hourly are accepted, as is "@every <duration>" for an IntervalSchedule-like cadence.

Times are evaluated in UTC unless the expression starts with "CRON_TZ=" or "TZ=" and an IANA zone ("CRON_TZ=America/Chicago 0 9 * * MON-FRI"), or the schedule is given a location with In. Around a daylight-saving change, a time in the skipped hour does not occur that day, and a time in the repeated hour occurs twice, as in Vixie cron.

func (*CronSchedule) In

func (s *CronSchedule) In(loc *time.Location) *CronSchedule

In returns a copy of the schedule evaluated in loc.

func (*CronSchedule) Next

func (s *CronSchedule) Next(t time.Time) time.Time

Next implements Schedule. It advances field by field, from months down to seconds, resetting the lower fields whenever a higher one moves.

func (*CronSchedule) String

func (s *CronSchedule) String() string

String returns the expression the schedule was parsed from.

type DependencyFailure

type DependencyFailure = driver.DependencyFailure

DependencyFailure says what happens to a step when a step it depends on is canceled or discarded.

type ErrorOutcome added in v0.8.0

type ErrorOutcome struct {
	// RetryAt retries the job at this time, using up the attempt as any
	// failure does: on its last attempt the job is discarded all the same.
	RetryAt time.Time
	// Snooze reschedules the job after this long without using up the
	// attempt, as if the worker had returned Snooze. The error is kept as the
	// snooze reason rather than in the job's errors.
	Snooze time.Duration
	// Discard discards the job now, with the error recorded, as if the
	// worker had returned Cancel.
	Discard bool
}

ErrorOutcome is what Config.OnError decides for a failed attempt. The zero ErrorOutcome keeps the outcome the client would otherwise choose. If more than one field is set, Discard wins over Snooze, and Snooze over RetryAt.

type Event

type Event struct {
	Kind EventKind
	At   time.Time
	// Job is set for job events: the row as claimed, before the outcome;
	// for the EventJobRescued of a job lease's expiry, the row after it.
	Job *JobRow
	// Error is the attempt's error text for failed, discarded and
	// canceled jobs, and the snooze reason, if any, for snoozed jobs.
	Error string
	// ClientID is the client the event happened on.
	ClientID int64
}

Event is one occurrence on a client.

type EventKind

type EventKind string

EventKind names an event stream.

const (
	EventJobCompleted  EventKind = "job.completed"
	EventJobFailed     EventKind = "job.failed" // an attempt failed; the job will retry
	EventJobSnoozed    EventKind = "job.snoozed"
	EventJobCanceled   EventKind = "job.canceled"
	EventJobDiscarded  EventKind = "job.discarded"
	EventJobRescued    EventKind = "job.rescued"
	EventLeaderElected EventKind = "leader.elected"
	EventLeaderLost    EventKind = "leader.lost"
	EventLeaseLost     EventKind = "lease.lost"
)

Event kinds.

type ExponentialRetry added in v0.8.0

type ExponentialRetry struct {
	Base time.Duration
	Max  time.Duration
}

ExponentialRetry is a RetryPolicy that waits Base, doubling with each attempt, with ±10% jitter, up to Max. Base defaults to one second and Max to 24 hours.

func (ExponentialRetry) NextRetry added in v0.8.0

func (r ExponentialRetry) NextRetry(job *JobRow) time.Time

NextRetry implements RetryPolicy.

type InsertOpts

type InsertOpts struct {
	// Queue defaults to QueueDefault.
	Queue string
	// Priority defaults to PriorityNormal.
	Priority Priority
	// MaxAttempts defaults to Config.MaxAttempts.
	MaxAttempts int
	// ScheduledAt delays the job until the given time. Zero runs it as soon
	// as possible.
	ScheduledAt time.Time
	// Metadata is a JSON object stored with the job, for trace context and
	// similar. It is not passed to the worker as args.
	Metadata json.RawMessage
	// Unique makes the job unique among jobs of its kind. See UniqueOpts.
	Unique *UniqueOpts
	// TTL bounds how long the job may wait to run, measured from its
	// insert: before its first attempt, between retries, and while snoozed.
	// A job still waiting when it passes is discarded without running again.
	// A running job is not interrupted. Zero means no limit.
	TTL time.Duration
	// Await announces the job's finalize, so that Await returns as soon as
	// it commits instead of on its next poll.
	Await bool
	// OrderingKey serializes the job with others of the same key in its
	// queue: at most one runs at a time, oldest first, and a failing job
	// blocks its key while it retries. See PublishOpts.OrderingKey.
	OrderingKey string
	// PartitionKey groups the job for the queue's PartitionLimit, for
	// example a customer ID.
	PartitionKey string
	// Tags label the job, for finding it with JobFilter.TagsAny and
	// TagsAll, for example a tenant or a release. Each is a non-empty
	// string of valid UTF-8, at most MaxTagBytes, without control
	// characters. Repeats are dropped, keeping the first; more than MaxTags
	// distinct tags fail the insert. They replace, not add to, the tags of
	// the JobArgs' own InsertOpts.
	Tags []string
}

InsertOpts customizes one insert. A nil *InsertOpts means the defaults, which come from the args' InsertOpts method if it has one, then from the client's Config.

type InsertParams

type InsertParams struct {
	Args JobArgs
	Opts *InsertOpts
	// contains filtered or unexported fields
}

InsertParams is one job in an InsertMany call.

type InsertResult

type InsertResult struct {
	Job *JobRow
	// Duplicate reports that a job with the same unique key already
	// existed: a live one, or a finished one UniqueOpts.Finished counts. Job
	// is then that existing job.
	Duplicate bool
}

InsertResult is the outcome of inserting one job.

type IntervalSchedule

type IntervalSchedule struct {
	Interval time.Duration
}

IntervalSchedule fires every Interval, at slots aligned to the interval (every 30 seconds fires at :00 and :30), so slots are the same whichever leader computes them.

func (IntervalSchedule) Next

func (s IntervalSchedule) Next(t time.Time) time.Time

Next implements Schedule.

type JSONCodec

type JSONCodec struct{}

JSONCodec is the default Codec, using encoding/json.

func (JSONCodec) Marshal

func (JSONCodec) Marshal(v any) ([]byte, error)

Marshal implements Codec.

func (JSONCodec) Unmarshal

func (JSONCodec) Unmarshal(data []byte, v any) error

Unmarshal implements Codec.

type Job

type Job[T JobArgs] struct {
	*JobRow
	Args T
}

Job is a claimed job with its decoded args.

type JobArgs

type JobArgs interface {
	Kind() string
}

JobArgs is the payload of a job. Kind names the job type and maps to exactly one worker. Args must be a value type (not a pointer), encodable by the client's Codec, and Kind must not depend on the value's fields.

type JobArgsWithInsertOpts

type JobArgsWithInsertOpts interface {
	JobArgs
	InsertOpts() InsertOpts
}

JobArgsWithInsertOpts is implemented by args that carry per-kind insert defaults. Options passed to Insert override them field by field. Args are inserted as values, so InsertOpts needs a value receiver to be seen.

type JobArgsWithKindAliases added in v0.8.0

type JobArgsWithKindAliases interface {
	JobArgs
	KindAliases() []string
}

JobArgsWithKindAliases is implemented by args whose kind was renamed. KindAliases returns the earlier kinds, and the worker registered for the args also works jobs stored under them, decoding their stored args into these args, so the encoding must stay compatible. A job keeps the kind it was inserted with, and unique keys, periodic job names and job filters match that stored kind. KindAliases is read once, from the zero value, when the worker is registered, so like Kind it must not depend on the value's fields. An alias must not be empty, repeated, the args' own kind, or another registered kind or alias.

type JobFailedError

type JobFailedError struct {
	Job *JobRow
}

JobFailedError is returned by Await when the job ended other than completed.

func (*JobFailedError) Error

func (e *JobFailedError) Error() string

type JobFilter

type JobFilter struct {
	Queue  string
	Kinds  []string
	States []JobState
	// OrderingKey lists one ordering key's jobs, its lane. Keys are scoped
	// to a queue, so it needs Queue; the listing fails without one.
	OrderingKey string
	// NewestFirst lists in descending ID order: the latest inserted jobs
	// first, though IDs from the same millisecond can sort either way.
	NewestFirst bool
	// After starts the listing after the given job, for paging: pass the
	// last ID of one page to get the next.
	After JobID
	// IDs lists only the jobs with one of the given IDs.
	IDs []JobID
	// Priorities lists only the jobs with one of the given priorities.
	Priorities []Priority
	// TagsAny lists jobs with at least one of the tags, and TagsAll jobs
	// with all of them. Both can be set, each with at most MaxTags distinct
	// tags.
	TagsAny []string
	TagsAll []string
	// Metadata, a JSON object, lists jobs whose metadata contains it: every
	// key it has, with an equal value, where an object value contains the
	// same way and an array value contains each of its elements, as
	// Postgres's jsonb containment (@>) does. {"tenant": "acme"} lists the
	// jobs with that tenant, whatever else their metadata holds. Unlike
	// tags, it has no index: it narrows what the other filters find, and
	// alone reads every job.
	Metadata json.RawMessage
}

JobFilter selects jobs for Jobs. Empty fields match everything.

type JobID

type JobID = driver.JobID

JobID identifies a job. It is a UUIDv7 generated by the database.

func ParseJobID

func ParseJobID(s string) (JobID, error)

ParseJobID parses the canonical 8-4-4-4-12 text form of a JobID.

type JobLease added in v0.9.0

type JobLease struct {
	// Job is the job as claimed: running with no owning client, its attempt
	// and fence incremented.
	Job   *JobRow
	Token LeaseToken
	// ExpiresAt is when the lease expires unless renewed, by database time.
	ExpiresAt time.Time
}

JobLease is a job claimed by ID and its lease.

type JobLogsConfig added in v0.8.0

type JobLogsConfig struct {
	// Store, if set, receives each attempt's lines as they are logged
	// instead of the job's row in the database, and Client.JobLogs reads
	// them from it, after any the job's row holds. MaxBytes then neither
	// applies nor is checked, and MaxAttempts bounds what Client.JobLogs
	// returns: what the store keeps is its own choice.
	Store LogStore
	// MaxBytes is how much of an attempt's log is kept. Once a line would
	// pass it, that line and the rest are dropped, and the log ends with a
	// line that counts them. Defaults to 32 KiB, and is at most 1 MiB,
	// and MaxBytes times MaxAttempts at most 4 MiB; negative keeps no
	// lines. For larger logs, use a Store. A finished attempt's lines
	// wait in memory with its result until it is written, so a client
	// holds at most 64 MiB of them; past that, finished jobs wait to hand
	// in their results, as they do when the finalizer's buffer is full.
	MaxBytes int
	// MaxAttempts is how many attempts' logs a job keeps, the latest. Each
	// snooze keeps its own until the next snooze of the same attempt
	// number, which replaces it, or removes it if that snooze logged
	// nothing. A store keeps what it is given: a snooze that logs nothing
	// writes nothing to it, so it removes only a snooze's log that the
	// job's row holds. Defaults to 10, and is at most 100.
	MaxAttempts int
	// Level is the lowest level kept. Defaults to slog.LevelInfo. Lines
	// go to Config.Logger at whatever level its handler takes.
	Level slog.Leveler
	// ReplaceAttr, if set, rewrites or drops each attribute of a kept line,
	// as slog.HandlerOptions.ReplaceAttr does. The options of
	// Config.Logger's handler do not apply to the kept lines, so whatever
	// that handler redacts is redacted here too, or it is stored with the
	// job and shown by hopper jobs get and hopperui.
	ReplaceAttr func(groups []string, a slog.Attr) slog.Attr
}

JobLogsConfig bounds the log lines Logger keeps with each job, or sends them to a LogStore.

type JobRow

type JobRow = driver.JobRow

JobRow is a job as stored, with its encoded args. Workers receive a typed Job[T] that embeds it.

type JobState

type JobState = driver.JobState

JobState is a job's position in the state machine.

type JobStep added in v0.8.0

type JobStep[T JobArgs] struct {
	// contains filtered or unexported fields
}

JobStep is one named step of a job, for RunSteps. NamedStep makes one.

func NamedStep added in v0.8.0

func NamedStep[T JobArgs](name string, run func(ctx context.Context, job *Job[T]) error) JobStep[T]

NamedStep names one step of a job's own work for RunSteps (unlike a workflow's Step, which is a job of its own). A job resumes after the step its progress names, so names must be unique within the job and keep their meaning across releases.

type LeaseOpts added in v0.9.0

type LeaseOpts struct {
	// TTL is how long the lease lasts from the claim and from each renewal,
	// by database time: DefaultJobLeaseTTL if zero. Renew about every TTL/3.
	TTL time.Duration
}

LeaseOpts are the options of a claim by ID.

type LeaseResult added in v0.9.0

type LeaseResult struct {
	Err error
	// Output is the completed job's output, encoded with Config.Codec and
	// kept as SetOutput's is. It is ignored unless the attempt ends the
	// job.
	Output any
}

LeaseResult ends a leased attempt as if the job's worker had returned Err: nil completes the job, with Output; a *SnoozeError (Snooze) snoozes it, which gives the attempt back and keeps the fence; a *CancelError (Cancel) discards it; any other error retries it, with Config.OnError, the kind's worker's NextRetry (when this client has the worker) and the retry policies applied as for a worker's error, or discards it on its last attempt. After JobCancel, an error cancels the job.

type LeaseStatus added in v0.9.0

type LeaseStatus struct {
	// ExpiresAt is when the lease now expires, by database time.
	ExpiresAt time.Time
	// CancelRequested reports that JobCancel was called on the job. The
	// holder should stop, and Finish with an error, which cancels the job.
	CancelRequested bool
}

LeaseStatus is a job lease as renewed.

type LeaseToken added in v0.9.0

type LeaseToken struct {
	// contains filtered or unexported fields
}

LeaseToken is the capability for one leased attempt: the job's ID, the claim's fence and a random value the database generated. It is opaque to the worker that carries it: String and ParseLeaseToken, and the text and JSON forms, move it between processes. hopper does not authenticate the holder; a token is as good as the worker's channel to the coordinator, which should authenticate the worker.

func ParseLeaseToken added in v0.9.0

func ParseLeaseToken(s string) (LeaseToken, error)

ParseLeaseToken parses a token's text form.

func (LeaseToken) Fence added in v0.9.0

func (t LeaseToken) Fence() int64

Fence returns the fence of the claim the token belongs to. The attempt's snooze keeps it, so the next claim by ID of the snoozed job names it.

func (LeaseToken) IsZero added in v0.9.0

func (t LeaseToken) IsZero() bool

IsZero reports whether t is the zero token, which names no lease.

func (LeaseToken) JobID added in v0.9.0

func (t LeaseToken) JobID() JobID

JobID returns the ID of the token's job.

func (LeaseToken) MarshalText added in v0.9.0

func (t LeaseToken) MarshalText() ([]byte, error)

MarshalText implements encoding.TextMarshaler.

func (LeaseToken) String added in v0.9.0

func (t LeaseToken) String() string

String returns the token's text form.

func (*LeaseToken) UnmarshalText added in v0.9.0

func (t *LeaseToken) UnmarshalText(b []byte) error

UnmarshalText implements encoding.TextUnmarshaler.

type Leases added in v0.9.0

type Leases[TTx any] struct {
	// contains filtered or unexported fields
}

Leases claims jobs by ID under job leases, and renews, records the progress of and finishes their attempts. See Client.Leases.

A job claimed by ID runs with no owning client. Its lease owns it: a token that names the job, the claim's fence and a random value the database generates. Every call on the attempt names the token and is fenced to it, from any client of the cluster: once the lease has expired, or the attempt has ended and the job been claimed again, the call returns ErrJobFenced and changes nothing. A lease that is not renewed within its TTL expires, though a lease last renewed before the start of a client whose own lease is live is given the larger of its TTL and Config.JobLeaseGrace from the first such start, so that leases outlast a restart of the whole cluster. The leader then makes the job retryable at once, with an entry in its errors whose Rescue is RescueLeaseExpired and whose Fence is the lease's, or, on its last attempt, discards it, or cancels it if JobCancel was called. hopper never runs a leased attempt's work itself. Only a client with queues, and without Config.NeverLead, can lead, so a cluster whose clients only insert and take reports never expires a lease.

The queue's pause applies to a claim by ID; its limits (GlobalLimit, PartitionLimit, RateLimit) do not, and leased jobs do not count toward them. The job's TTL and ordering key apply.

Finish applies the calling client's configuration as the job's worker would: Config.OnError, the queue's configuration from Config.Queues or Queues().Add (its RetryPolicy and DeleteCompleted) and, if the client has the kind's worker, its NextRetry. A worker may report through any client, so give every client that takes reports the same configuration for those queues and kinds.

func (*Leases[TTx]) Claim added in v0.9.0

func (l *Leases[TTx]) Claim(ctx context.Context, id JobID, fence int64, opts *LeaseOpts) (*JobLease, error)

Claim claims a waiting job (available, scheduled or retryable) by ID under a new job lease, whatever its schedule, so a job snoozed while it waits for its worker can be claimed early. The job runs with no owning client until the lease's attempt ends or expires.

fence is the job's fence the caller expects, such as the Fence of the handler's attempt that placed the job on a worker, recorded with the placement, or 0 for a job never claimed. The claim succeeds only at that fence, so a superseded placement, one made before another attempt ran, cannot claim the job.

It returns the first of these that applies to the job:

neither live nor in history (never inserted, or deleted
  by DeleteCompleted, JobDelete or retention)            ErrNotFound
in history, at any fence                                 ErrJobNotClaimable
live at another fence than fence, running or not         ErrJobFenced
running at fence, by a client or another lease           ErrJobRunning
pending, past its TTL, in a paused queue, or held back
  by its ordering key                                    ErrJobNotClaimable
available, scheduled or retryable                        claimed

ErrJobFenced means that the job is not at the fence the caller expects, as when another attempt ran, or runs, since the one that fence names: the placement is superseded. ErrJobRunning is transient, such as for the placing attempt whose snooze is still being written: try again shortly. A claim of a waiting job waits for its row if another transaction holds it.

func (*Leases[TTx]) ClaimTx added in v0.9.0

func (l *Leases[TTx]) ClaimTx(ctx context.Context, tx TTx, id JobID, fence int64, opts *LeaseOpts) (*JobLease, error)

ClaimTx is Claim inside the caller's transaction, so that the claim commits or rolls back with the caller's own writes, such as a placement record locked in the same transaction. Until tx ends it holds the job's row. On PostgreSQL a claim that loses a race with another claim of a job with the same ordering key returns ErrJobNotClaimable once the database has aborted tx, which must then be rolled back.

func (*Leases[TTx]) Finish added in v0.9.0

func (l *Leases[TTx]) Finish(ctx context.Context, token LeaseToken, res LeaseResult) error

Finish ends the leased attempt with res, as if the job's worker had returned (see LeaseResult), and releases the lease. The result is written before Finish returns, so a worker's report is durable once acknowledged. It returns ErrJobFenced, and changes nothing, once the lease is no longer the job's. The job's batch and workflow dependents are updated as for any finished job, and the client's events and Await see the result. Config.OnError runs before the result is written, as for a worker, so it can run for a Finish that then returns ErrJobFenced, such as one whose token is stale or made up but names the current fence.

func (*Leases[TTx]) FinishTx added in v0.9.0

func (l *Leases[TTx]) FinishTx(ctx context.Context, tx TTx, token LeaseToken, res LeaseResult) error

FinishTx is Finish inside the caller's transaction, so that the attempt's result commits or rolls back with the caller's own writes. Until tx ends it holds the job's row. The job's batch and dependents are updated as for any finished job, the batch's completion and the dependents' promotion within a leader interval of the commit. As the client cannot know whether tx commits, it emits no event for the result.

func (*Leases[TTx]) RecordCursor added in v0.9.0

func (l *Leases[TTx]) RecordCursor(ctx context.Context, token LeaseToken, cursor any) (*Progress, error)

RecordCursor records a cursor within the step under way, keeping the last finished step, as RecordCursor does for a worker.

func (*Leases[TTx]) RecordCursorTx added in v0.9.0

func (l *Leases[TTx]) RecordCursorTx(ctx context.Context, tx TTx, token LeaseToken, cursor any) (*Progress, error)

RecordCursorTx is RecordCursor inside the caller's transaction.

func (*Leases[TTx]) RecordStep added in v0.9.0

func (l *Leases[TTx]) RecordStep(ctx context.Context, token LeaseToken, step string, cursor any) (*Progress, error)

RecordStep records that the leased attempt finished step, with an optional cursor, as RecordStep does for a worker, and returns the progress as stored. It returns ErrJobFenced once the lease is no longer the job's.

func (*Leases[TTx]) RecordStepTx added in v0.9.0

func (l *Leases[TTx]) RecordStepTx(ctx context.Context, tx TTx, token LeaseToken, step string, cursor any) (*Progress, error)

RecordStepTx is RecordStep inside the caller's transaction. Until tx ends it holds the job's row, which the calls on the pool for the attempt wait for.

func (*Leases[TTx]) Renew added in v0.9.0

func (l *Leases[TTx]) Renew(ctx context.Context, token LeaseToken) (*LeaseStatus, error)

Renew extends the lease by its TTL from database now(), and reports a cancel request. It returns ErrJobFenced once the lease is no longer the job's: it expired, or its attempt ended. A renewal succeeds until the leader applies the expiry, not only until ExpiresAt, so a holder learns that it lost the lease from ErrJobFenced, never from its own clock.

type LogStore added in v0.8.0

type LogStore interface {
	// Writer returns where the lines of the attempt of job that is running
	// go, as Logger logs them: one line per Write, in slog's text format,
	// in the order they were logged. job is the row as it was claimed.
	//
	// Logging never waits for the store: lines are queued, and one
	// goroutine per attempt opens the writer with the first line and hands
	// the lines to it. Up to 256 KiB of lines wait for a store that falls
	// behind; lines past that are dropped, and a line counting them follows.
	// Writer's context carries the values of the context the worker first
	// called Logger with, such as a trace span its middleware started, but
	// not its cancellation, deadline or attempt (Logger called with it logs
	// as outside a job). It is canceled once the writer is closed, or once
	// hopper stops waiting for it.
	//
	// Hopper closes the writer once the worker returns and waits for the
	// queued lines and Close for up to 5 seconds before it hands in the
	// attempt's result, so a store that saves on Close has the log by the
	// time the job is finished; past that, the context is canceled and the
	// result goes ahead. A client that stops hard waits for at most half its
	// stop grace, so the result of the job it canceled is still written. A
	// job completed with CompleteTx is finished when the worker's
	// transaction commits, which may be before. Lines logged after the
	// close go to Config.Logger only.
	//
	// A snooze gives the attempt back, so the run after it calls Writer
	// again with the same job ID and attempt: keep both, in order. A failed
	// or panicking Writer, Write or Close, or a nil writer, is logged to
	// Config.Logger once and drops the attempt's remaining lines; the job's
	// outcome does not depend on it.
	Writer(ctx context.Context, job *JobRow) (io.WriteCloser, error)
	// Logs returns the logs of a job's attempts, oldest first, for
	// Client.JobLogs and hopperui, with no entry for a job it has none of.
	Logs(ctx context.Context, id JobID) ([]AttemptLog, error)
}

LogStore keeps jobs' log lines outside hopper's tables, for logs that are large, must be read while the attempt runs, or are written by a process that runs a job's work elsewhere. Logs are keyed by job ID and attempt: hopper writes nothing of them to the database, so they survive the client's crash, and a remote process writes an attempt's lines under the ID and attempt its job was claimed with. The store keeps them for as long as it chooses; hopper does not remove them with the job.

Its methods are called from many goroutines at once.

type Message

type Message[T any] struct {
	*JobRow
	Payload T
	Topic   string
	// ID identifies the published message; every subscription's delivery of
	// it shares the ID.
	ID          string
	Headers     map[string]string
	OrderingKey string
	// Stream and Position are set for a consumer's deliveries: the
	// consumer's name and the event's place in the log.
	Stream   string
	Position StreamPosition
}

Message is one delivery of a published message to a subscription.

type Middleware

type Middleware interface {
	// Work wraps one attempt of a job. next runs the worker (and any inner
	// middleware); its error is the worker's return value.
	Work(ctx context.Context, job *JobRow, next func(ctx context.Context) error) error
	// Insert wraps an insert batch. next performs the insert; middleware may
	// pass it a modified copy of params, for example with trace context in
	// Metadata.
	Insert(ctx context.Context, params []InsertParams, next func(ctx context.Context, params []InsertParams) ([]*InsertResult, error)) ([]*InsertResult, error)
}

Middleware wraps job execution and inserts, for logging, metrics, tracing and custom policy. Embed MiddlewareDefaults to implement only one side, or use WorkMiddleware and InsertMiddleware for a function.

Middleware runs in Config.Middleware order: the first wraps the second, and so on down to the worker or the insert.

func InsertMiddleware

func InsertMiddleware(fn func(ctx context.Context, params []InsertParams, next func(ctx context.Context, params []InsertParams) ([]*InsertResult, error)) ([]*InsertResult, error)) Middleware

InsertMiddleware makes a Middleware from a function that wraps inserts.

func WorkMiddleware

func WorkMiddleware(fn func(ctx context.Context, job *JobRow, next func(ctx context.Context) error) error) Middleware

WorkMiddleware makes a Middleware from a function that wraps execution.

type MiddlewareDefaults

type MiddlewareDefaults struct{}

MiddlewareDefaults implements Middleware as pass-through.

func (MiddlewareDefaults) Insert

func (MiddlewareDefaults) Insert(ctx context.Context, params []InsertParams, next func(ctx context.Context, params []InsertParams) ([]*InsertResult, error)) ([]*InsertResult, error)

Insert implements Middleware.

func (MiddlewareDefaults) Work

func (MiddlewareDefaults) Work(ctx context.Context, _ *JobRow, next func(ctx context.Context) error) error

Work implements Middleware.

type PanicError

type PanicError struct {
	Value any
	Stack string
}

PanicError records a worker panic. The job is treated as failed.

func (*PanicError) Error

func (e *PanicError) Error() string

type PeriodicJob

type PeriodicJob struct {
	// Name identifies the job in the periodic table. It defaults to the
	// args' kind and must be unique among a client's periodic jobs.
	Name     string
	Schedule Schedule
	Args     JobArgs
	Opts     *PeriodicOpts
	// contains filtered or unexported fields
}

PeriodicJob inserts a job on a schedule. The leader computes the slots and inserts each at most once cluster-wide, even across leader changes. Build one with Cron or Every.

func Cron

func Cron(spec string, args JobArgs, opts *PeriodicOpts) PeriodicJob

Cron returns a periodic job on a cron expression (see ParseCron). An invalid expression is reported by NewClient or PeriodicJobs.Add.

func Every

func Every(interval time.Duration, args JobArgs, opts *PeriodicOpts) PeriodicJob

Every returns a periodic job that fires every interval, at slots aligned to the interval.

type PeriodicJobs added in v0.8.0

type PeriodicJobs struct {
	// contains filtered or unexported fields
}

PeriodicJobs is a client's set of periodic jobs, which starts as Config.Periodic. The client records the set under its lease, and the leader schedules the jobs of every live client, for each name the newest client's definition: a job runs while any live client registers it. A change applies at once on the leader, and within a leader interval on another client. A job whose schedule cannot be recorded, one other than Cron's or Every's, or a Cron in time.Local or in a zone that does not load by its name, such as a time.FixedZone, is scheduled only from its own client, while that client leads and no other client records its name. So is a job whose insert differs each time it is built, as with a Codec that encrypts with a random nonce, a job whose insert sets ScheduledAt, and one whose name, args or options hold a NUL or invalid UTF-8. A Cron in a named zone runs only while a client that can load that zone leads, so every client that can lead needs the zone database. A client with no queues holds no lease and records nothing. It is safe for concurrent use.

func (*PeriodicJobs) Add added in v0.8.0

func (p *PeriodicJobs) Add(job PeriodicJob) error

Add registers a periodic job. It fails if the job is invalid, including if the job it inserts could not be built (as Insert would fail with its args and InsertOpts), or if its name (PeriodicOpts.Name, or else the args' kind) is already registered; to change a job, Remove it and Add it again. The leader starts scheduling an added job at once when it is added on the leader, and within a leader interval when added on another client, as if its term had just started: RunOnStart inserts a job and CatchUp back-fills missed slots. A job the leader already schedules with the same definition, from this client or another, goes on as it was.

func (*PeriodicJobs) AddMany added in v0.8.0

func (p *PeriodicJobs) AddMany(jobs []PeriodicJob) error

AddMany registers every job, or none if any is invalid or its name is taken, including by another of jobs.

func (*PeriodicJobs) All added in v0.8.0

func (p *PeriodicJobs) All() []PeriodicJob

All returns the registered jobs, in the order they were added, with their names filled in.

func (*PeriodicJobs) Clear added in v0.8.0

func (p *PeriodicJobs) Clear()

Clear unregisters every job, including those from Config.Periodic.

func (*PeriodicJobs) Remove added in v0.8.0

func (p *PeriodicJobs) Remove(name string) error

Remove unregisters the job with this name on this client. The job stops once no live client registers it. A slot being inserted as it is removed may still be inserted. The job's last slot stays in the database, so a job registered again does not insert a slot twice.

type PeriodicOpts

type PeriodicOpts struct {
	// Name overrides the periodic job's name.
	Name string
	// RunOnStart inserts a job as soon as a leader starts scheduling the
	// periodic job, at the start of its term or when the job is added on
	// it, in addition to the regular slots. Its slot is the database time,
	// so an insert that committed but reported an error, as when the
	// connection drops before the reply, is inserted again when retried.
	RunOnStart bool
	// CatchUp inserts up to this many slots missed while no leader was
	// running (for example during a full outage), oldest first, at most
	// 10000. By default missed slots are skipped, except those due since the
	// last one inserted within the last lease TTL, which covers a leader
	// failover and a job removed and added again; CatchUp never inserts
	// fewer than those. Either way at most the 10000 most recent missed
	// slots are inserted.
	CatchUp int
	// Location evaluates a cron expression in this zone when the
	// expression does not name one. Defaults to UTC.
	Location *time.Location
	// InsertOpts are applied to every inserted job. A UniqueOpts.OnConflict
	// of UniqueReplace is treated as UniqueSkip: a slot whose job is a
	// duplicate keeps the existing job.
	InsertOpts *InsertOpts
}

PeriodicOpts customizes a periodic job.

type Priority

type Priority int

Priority orders jobs within a queue. Lower runs first.

const (
	PriorityHigh   Priority = 1
	PriorityNormal Priority = 2 // the default
	PriorityLow    Priority = 3
	PriorityLowest Priority = 4
)

The four priority levels.

type Progress added in v0.8.0

type Progress = driver.Progress

Progress is how far a job's worker got: the last step it finished and a cursor within the work that follows. See RecordStep.

type PublishOpts

type PublishOpts struct {
	// OrderingKey makes delivery FIFO per key, per queue: at most one
	// delivery with the key runs at a time, oldest first, and a failing
	// delivery blocks its key while it retries. Keys are independent.
	OrderingKey string
	// DedupKey makes the publish idempotent: while a delivery with the key
	// is live, a repeated publish inserts nothing and reports Duplicate.
	DedupKey string
	// Headers travel with the message and are available on Message.
	Headers map[string]string
	// Delay defers delivery.
	Delay time.Duration
	// TTL bounds how long each delivery may wait to run, as
	// InsertOpts.TTL does: one still waiting when it passes, whether before
	// its first attempt, between retries or while snoozed, is discarded.
	TTL time.Duration
	// Priority defaults to PriorityNormal.
	Priority Priority
	// Await announces each delivery's finalize, for Await.
	Await bool
}

PublishOpts customizes a publish. A nil *PublishOpts means the defaults.

type PublishResult

type PublishResult struct {
	// MessageID identifies the message across its deliveries. It is empty
	// when no subscription matched.
	MessageID  string
	Deliveries []*InsertResult
}

PublishResult reports the deliveries a publish produced, one per matching subscription.

type QueueConfig

type QueueConfig struct {
	// MaxWorkers is the number of jobs this client runs at once from the
	// queue. Required.
	MaxWorkers int
	// DeleteCompleted deletes completed jobs instead of archiving them to
	// history, for maximum throughput. Canceled and discarded jobs are
	// always archived, because history is the dead-letter queue.
	DeleteCompleted bool
	// RetryPolicy overrides Config.RetryPolicy for the queue's jobs on this
	// client.
	RetryPolicy RetryPolicy

	// GlobalLimit caps the queue's running jobs across all clients.
	GlobalLimit int
	// RateLimit caps claims per second across all clients, in a token
	// bucket of RateBurst tokens (defaulting to one second's worth).
	RateLimit float64
	RateBurst int
	// PartitionLimit caps running jobs per InsertOpts.PartitionKey.
	PartitionLimit int
	// PriorityAging raises a waiting job's priority by one level each time
	// it has waited this long, so low priorities cannot starve.
	PriorityAging time.Duration
}

QueueConfig configures one queue on a client.

type QueueControl

type QueueControl[TTx any] struct {
	// contains filtered or unexported fields
}

QueueControl pauses, resumes, lists and adds queues.

func (*QueueControl[TTx]) Add

func (q *QueueControl[TTx]) Add(ctx context.Context, name string, cfg QueueConfig) error

Add starts working a queue on this client at runtime, for example a tenant's queue. The client must be started.

func (*QueueControl[TTx]) List

func (q *QueueControl[TTx]) List(ctx context.Context) ([]*QueueRow, error)

List returns every queue known to the cluster.

func (*QueueControl[TTx]) Pause

func (q *QueueControl[TTx]) Pause(ctx context.Context, name string) error

Pause stops every client from claiming from the queue. Running jobs finish. Clients learn of it through a notification and on every lease renewal.

func (*QueueControl[TTx]) Remove

func (q *QueueControl[TTx]) Remove(ctx context.Context, name string) error

Remove stops working a queue on this client, waiting for its running jobs until ctx is done.

func (*QueueControl[TTx]) Resume

func (q *QueueControl[TTx]) Resume(ctx context.Context, name string) error

Resume undoes Pause.

func (*QueueControl[TTx]) SetLimits

func (q *QueueControl[TTx]) SetLimits(ctx context.Context, name string, limits QueueLimits) error

SetLimits replaces a queue's cluster-wide limits. Zero values remove a limit. Clients learn of the change through a notification and on their next lease renewal.

type QueueLimits

type QueueLimits = driver.QueueLimits

QueueLimits are a queue's cluster-wide limits; see QueueConfig.

type QueueRow

type QueueRow = driver.QueueRow

QueueRow is a queue's cluster-wide state.

type QueueStats

type QueueStats = driver.QueueStats

QueueStats is one queue's depth by state.

type Raw

type Raw = json.RawMessage

Raw is a message payload left undecoded, for a subscription that matches several topics with different payload types; dispatch on Message.Topic.

type RescueReason added in v0.9.0

type RescueReason = driver.RescueReason

RescueReason says why hopper, not the attempt, ended an attempt. It is set on the attempt's entry in JobRow.Errors.

type RetryOption added in v0.8.0

type RetryOption func(*driver.JobRetryParams)

RetryOption changes what JobRetry and JobRetryTx do.

func FromStart added in v0.8.0

func FromStart() RetryOption

FromStart clears the job's progress, so that it runs all its steps again instead of resuming after the last one it finished: to run a finished job over, or one whose recorded step its worker no longer has.

type RetryPolicy added in v0.8.0

type RetryPolicy interface {
	NextRetry(job *JobRow) time.Time
}

RetryPolicy chooses when a failed job runs again, for workers whose NextRetry returns the zero time. job.Attempt is the attempt that just failed, starting at 1. Returning the zero time passes the choice on: from a queue's policy to the client's, and from the client's to the default backoff. It is called from job goroutines, concurrently; a panic in it is logged and also passes the choice on.

type Schedule

type Schedule interface {
	// Next returns the first slot strictly after t, or the zero time if
	// there is none within a reasonable horizon.
	Next(t time.Time) time.Time
}

Schedule computes the slots of a periodic job.

type SeekOpts

type SeekOpts struct {
	// Position makes the event after it the next delivered.
	Position StreamPosition
	// Time replays from the first event appended at or after it.
	Time time.Time
	// Earliest replays every retained event.
	Earliest bool
	// Latest skips to the events appended from now on.
	Latest bool
}

SeekOpts says where to move a consumer. Exactly one field applies, in this order of precedence.

type SnoozeError

type SnoozeError struct {
	Duration time.Duration
	// Reason is stored on the job while it waits, as JobRow.SnoozeReason,
	// for operators, and is the Error of the job.snoozed event. It is
	// cleared when the job runs again. A job that expires or is canceled
	// while it waits keeps the reason in the error it is finalized with.
	Reason string
}

SnoozeError is returned by a worker to run the job again later without using up an attempt. Use Snooze to create one, or return a SnoozeError with a Reason to record why the job is waiting.

func (*SnoozeError) Error

func (e *SnoozeError) Error() string

type Stats

type Stats = driver.Stats

Stats is a snapshot of queue depths and cluster state.

type Step

type Step struct {
	// contains filtered or unexported fields
}

Step identifies a step of a workflow, for other steps to depend on.

type StepOpts

type StepOpts struct {
	// After lists the steps this one waits for. It is inserted pending and
	// becomes available once they have all finished.
	After []Step
	// OnDependencyFailure defaults to DependencyCancel.
	OnDependencyFailure DependencyFailure
	// Insert holds the job's own options.
	Insert *InsertOpts
}

StepOpts sets a step's dependencies and insert options.

func After

func After(steps ...Step) *StepOpts

After returns options for a step that waits for the given steps.

type StreamEvent

type StreamEvent = driver.StreamEvent

StreamEvent is one event of the stream log.

type StreamPosition

type StreamPosition = driver.StreamPosition

StreamPosition is a place in the stream log.

func ParseStreamPosition

func ParseStreamPosition(s string) (StreamPosition, error)

ParseStreamPosition parses a position's String form ("xid:seq").

type StreamStart

type StreamStart = driver.StreamStart

StreamStart says where a new consumer starts.

type Streams

type Streams[TTx any] struct {
	// contains filtered or unexported fields
}

Streams is the stream log: an append-only, retained record of events that consumers read from positions of their own.

func (*Streams[TTx]) Append

func (s *Streams[TTx]) Append(ctx context.Context, event Topic, opts *AppendOpts) (*StreamEvent, error)

Append appends an event to the log. Consumers whose pattern matches its topic deliver it, in order, after everything appended before it.

func (*Streams[TTx]) AppendTx

func (s *Streams[TTx]) AppendTx(ctx context.Context, tx TTx, event Topic, opts *AppendOpts) (*StreamEvent, error)

AppendTx is Append in the caller's transaction: the event exists only if tx commits, and is delivered after events of every transaction that started before tx, however they commit.

func (*Streams[TTx]) Consumers

func (s *Streams[TTx]) Consumers(ctx context.Context) ([]*ConsumerRow, error)

Consumers lists every consumer known to the cluster.

func (*Streams[TTx]) Read

func (s *Streams[TTx]) Read(ctx context.Context, pattern string, after StreamPosition, limit int) iter.Seq2[*StreamEvent, error]

Read returns the events after a position whose topic matches pattern (empty for every topic), oldest first, in pages of limit. It sees only events that every earlier transaction has finished around, as consumers do, so paging by the last position never skips an event.

func (*Streams[TTx]) Seek

func (s *Streams[TTx]) Seek(ctx context.Context, consumer string, opts SeekOpts) error

Seek moves a consumer's position; the events after it are delivered (again) from the next pump. Deliveries already inserted are unaffected.

type Subscription

type Subscription struct {
	// Name identifies the subscription; the delivery job kind is derived
	// from it.
	Name    string
	Pattern string
	// Queue is where deliveries go. Defaults to QueueDefault.
	Queue string
	// MaxAttempts is the retry budget of deliveries. Zero means 25, or 10
	// for deliveries with an ordering key, whose retries block their key.
	MaxAttempts int
	// Metadata is merged into every delivery's metadata.
	Metadata json.RawMessage
}

Subscription routes messages whose topic matches Pattern to a queue, as jobs of kind "sub:<Name>".

Pattern uses AMQP topic syntax: dot-separated words, where "*" matches one word and "#" zero or more. "#" alone receives everything, and a literal topic exactly that topic. Fan-out happens at publish time, so a subscription created later does not receive earlier messages.

func (Subscription) Kind

func (s Subscription) Kind() string

Kind returns the job kind of the subscription's deliveries.

type SubscriptionRow

type SubscriptionRow = driver.SubscriptionRow

SubscriptionRow is a subscription as recorded in the database.

type Topic

type Topic interface {
	Topic() string
}

Topic is implemented by message payloads: Topic names the topic the message is published to, in dot-separated words ("allocation.created").

type UniqueConflict

type UniqueConflict int

UniqueConflict says what happens when a unique job already exists.

const (
	// UniqueSkip keeps the existing job. Insert reports Duplicate: true with
	// the existing job.
	UniqueSkip UniqueConflict = iota
	// UniqueReplace updates the existing job's args, metadata, priority,
	// max attempts and schedule, unless it is running. This gives
	// debouncing: inserting again pushes the job back. Insert still reports
	// Duplicate: true.
	UniqueReplace
)

Unique conflict actions.

type UniqueOpts

type UniqueOpts struct {
	// ByArgs includes the args in the key: the fields tagged
	// `hopper:"unique"`, or the whole args if no field is tagged.
	ByArgs bool
	// ByQueue includes the queue in the key.
	ByQueue bool
	// ByPeriod includes the current time, truncated to this period, so that
	// at most one job exists per period. For example 24 * time.Hour allows
	// one job per UTC day. A periodic job's key uses its slot instead.
	ByPeriod time.Duration
	// OnConflict is what to do when the job exists.
	OnConflict UniqueConflict
	// Finished also counts finished jobs with the key as duplicates when
	// their state is one of these (JobStateCompleted, JobStateCanceled,
	// JobStateDiscarded), for as long as history keeps them
	// (Config.CompletedRetention, Config.FailedRetention). With ByPeriod a
	// finished job counts only for the rest of its period, which is part of
	// the key. A finished duplicate is reported like a live one, and is
	// never replaced. Looking in history costs an index lookup per history
	// partition, so it grows with retention (docs/operations.md).
	//
	// A duplicate that finalizes while the insert runs is still found,
	// except in a caller's transaction (InsertTx, InsertManyTx) at
	// repeatable read or serializable, which does not see a duplicate
	// finalized after its snapshot.
	Finished []JobState
}

UniqueOpts makes a job unique among live jobs (those not yet finalized) of its kind, and among its finished jobs in the states Finished names. The zero value is unique by kind alone: at most one live job of the kind at a time.

type UnknownKindError

type UnknownKindError struct {
	Kind string
}

UnknownKindError reports a job kind with no registered worker.

func (*UnknownKindError) Error

func (e *UnknownKindError) Error() string

type Worker

type Worker[T JobArgs] interface {
	// Work runs the job. Returning nil completes it. Returning an error
	// schedules a retry with backoff, or discards the job when it is out of
	// attempts. Return Snooze(d) to run again later without using an attempt
	// and Cancel(err) to discard now.
	Work(ctx context.Context, job *Job[T]) error
	// Timeout returns how long the job may run before its context is
	// canceled: 0 for the client's default, or a negative duration for no
	// timeout.
	Timeout(job *Job[T]) time.Duration
	// NextRetry returns when a failed job runs again, or the zero time for
	// the default backoff.
	NextRetry(job *Job[T]) time.Time
}

Worker performs one kind of job. Embed WorkerDefaults to get default Timeout and NextRetry implementations.

type WorkerDefaults

type WorkerDefaults[T JobArgs] struct{}

WorkerDefaults provides default Timeout and NextRetry methods. Embed it in a worker to implement only Work.

func (WorkerDefaults[T]) NextRetry

func (WorkerDefaults[T]) NextRetry(*Job[T]) time.Time

NextRetry returns the zero time, which selects the default backoff.

func (WorkerDefaults[T]) Timeout

func (WorkerDefaults[T]) Timeout(*Job[T]) time.Duration

Timeout returns 0, which selects the client's default.

type Workers

type Workers struct {
	// contains filtered or unexported fields
}

Workers is a registry of workers by kind. Register at startup with AddWorker and AddWorkFunc, then pass it to NewClient.

func NewWorkers

func NewWorkers() *Workers

NewWorkers returns an empty registry.

func (*Workers) Work

func (w *Workers) Work(ctx context.Context, job *JobRow, codec Codec) error

Work runs the registered worker for job.Kind inline, decoding job.Args with codec and honoring the worker's Timeout. It exists for tests (see the hoppertest package): nothing is claimed or finalized, and middleware does not run. RecordStep and RecordCursor update job.Progress without storing it. It returns an UnknownKindError for an unregistered kind.

type Workflow

type Workflow struct {
	// contains filtered or unexported fields
}

Workflow collects steps with dependencies between them, to insert together. Steps are jobs; a step with dependencies waits, pending, until they have finished. Because a step can only depend on steps added before it, a workflow is always a DAG.

func NewWorkflow

func NewWorkflow(name string, opts *WorkflowOpts) *Workflow

NewWorkflow starts a workflow. Add steps to it, then insert it with Client.InsertWorkflow or InsertWorkflowTx.

func (*Workflow) Add

func (w *Workflow) Add(args JobArgs, opts *StepOpts) Step

Add adds a step. opts may be nil for a step with no dependencies; After(steps...) is the usual way to declare them.

type WorkflowOpts

type WorkflowOpts struct {
	OnSuccess  JobArgs
	OnFailure  JobArgs
	OnComplete JobArgs
	// CallbackOpts applies to every callback job.
	CallbackOpts *InsertOpts
	// Metadata is stored with the workflow.
	Metadata json.RawMessage
}

WorkflowOpts sets a workflow's completion callbacks and metadata. A workflow is a batch, so the callbacks are those of BatchOpts: OnSuccess when every step completed, OnFailure when any was canceled or discarded, OnComplete either way.

type WorkflowResult

type WorkflowResult struct {
	ID JobID
	// Jobs are in step order.
	Jobs []*InsertResult
}

WorkflowResult reports an inserted workflow.

type WorkflowRow

type WorkflowRow = driver.WorkflowRow

WorkflowRow is a workflow's progress, jobs and graph.

Directories

Path Synopsis
cmd
hopper command
Command hopper administers a hopper installation from the shell.
Command hopper administers a hopper installation from the shell.
hopperbench command
Command hopperbench measures hopper's hot paths against a real database and prints machine-readable results, one JSON object per scenario.
Command hopperbench measures hopper's hot paths against a real database and prints machine-readable results, one JSON object per scenario.
Package driver defines the contract between the hopper core and a storage engine.
Package driver defines the contract between the hopper core and a storage engine.
hopperpgx
Package hopperpgx is hopper's reference driver: PostgreSQL through pgx v5.
Package hopperpgx is hopper's reference driver: PostgreSQL through pgx v5.
hoppersql
Package hoppersql is hopper's PostgreSQL driver for database/sql, for applications that already use it with lib/pq or pgx's stdlib adapter.
Package hoppersql is hopper's PostgreSQL driver for database/sql, for applications that already use it with lib/pq or pgx's stdlib adapter.
internal/jobcodec
Package jobcodec encodes and decodes the JSON values a job row keeps beside its args and metadata: its errors, progress, logs and tags.
Package jobcodec encodes and decodes the JSON values a job row keeps beside its args and metadata: its errors, progress, logs and tags.
internal/pgsql
Package pgsql implements hopper's driver operations for PostgreSQL over a small transport interface, so that the pgx driver (hopperpgx) and the database/sql driver (hoppersql) share one implementation of the SQL and the logic around it.
Package pgsql implements hopper's driver operations for PostgreSQL over a small transport interface, so that the pgx driver (hopperpgx) and the database/sql driver (hoppersql) share one implementation of the SQL and the logic around it.
hoppersqlite module
Package drivertest is the conformance suite every hopper driver must pass.
Package drivertest is the conformance suite every hopper driver must pass.
examples
basic command
Command basic is a small, complete hopper application: it installs the schema, registers a worker, starts a client, inserts a few jobs (one of them inside a transaction) and waits for a result.
Command basic is a small, complete hopper application: it installs the schema, registers a worker, starts a client, inserts a few jobs (one of them inside a transaction) and waits for a result.
Package hoppermigrate installs and upgrades hopper's database schema.
Package hoppermigrate installs and upgrades hopper's database schema.
hopperotel module
Package hoppertest has helpers for testing code that uses hopper.
Package hoppertest has helpers for testing code that uses hopper.
hopperui module
internal
jobtags
Package jobtags checks a job's tags.
Package jobtags checks a job's tags.
testdb
Package testdb gives integration tests a Postgres database of their own.
Package testdb gives integration tests a Postgres database of their own.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL