queue

package
v0.3.3 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 5, 2026 License: GPL-3.0 Imports: 8 Imported by: 0

Documentation

Overview

Package queue declares durable background jobs that run against the application's own database.

A job type is declared once with Register, which binds a payload type to a handler and returns the only value able to construct work for that type. The package is a contract: it holds no database handle, starts no goroutine, and imports nothing from the rest of the module. The worker and the enqueue entry points live in the runtime.

Index

Constants

View Source
const InvalidCode = "invalid_code"

InvalidCode replaces a recorded code a handler supplied in non-canonical form. The error itself is never discarded.

View Source
const MaximumKeyBytes = 256

MaximumKeyBytes is the ceiling on a dedupe or exclusivity key.

View Source
const MaximumOperatorBatch = 256

MaximumOperatorBatch bounds one operator page or bulk control action.

View Source
const MaximumPayloadBytes = 1 << 20

MaximumPayloadBytes is the absolute ceiling on one marshalled payload. It bounds both Type.New and Limits.MaxPayloadBytes.

View Source
const MaximumRetentionRows = 4096

MaximumRetentionRows bounds one automatic or operator retention batch.

View Source
const RetentionDisabled = time.Duration(-1)

RetentionDisabled turns automatic retention off. Assigned to Limits.RetentionEvery it stops a worker deleting terminal job rows at all, so job history survives until an operator calls RunRetention. It is the only negative duration Limits accepts.

Variables

This section is empty.

Functions

func CompletedWith

func CompletedWith(code string, err error) error

CompletedWith terminates the job successfully while durably recording a code describing what degraded.

func Fail

func Fail(code ErrorCode, format string, arguments ...any) error

Fail builds a classified queue error. It exists so the worker and runtime report the same closed vocabulary this package defines.

func RetryIn

func RetryIn(delay time.Duration, err error) error

RetryIn overrides the scheduled delay of an ordinary retry. The attempt is still consumed.

func RetryInWithoutAttempt

func RetryInWithoutAttempt(delay time.Duration, err error) error

RetryInWithoutAttempt reschedules work without spending the claimed attempt. It is intended for external capacity deferrals, not handler failures. A handler that always returns it produces a job that never exhausts its attempts and so never dead-letters: the row stays live, and automatic retention only deletes terminal rows, so nothing ages it out. Bound the deferral on something other than the attempt count.

func Terminal

func Terminal(err error) error

Terminal marks a handler failure as permanent. The job moves to failed with its remaining attempts unspent.

Types

type Backoff

type Backoff struct {
	Base time.Duration
	Cap  time.Duration
}

Backoff schedules retries as exponential full jitter under a ceiling.

func (Backoff) Delay

func (backoff Backoff) Delay(attempt int) time.Duration

Delay returns the wait before the given one-based attempt is retried, as exponential full jitter under the configured ceiling.

type CountQuery

type CountQuery struct {
	Types []string
}

CountQuery selects the job types included in state counts. An empty Types list includes every stored type.

type Definition

type Definition[T any] struct {
	Type          string
	Handle        func(context.Context, Job[T]) error
	MaxAttempts   int
	Timeout       time.Duration
	MaxConcurrent int
	Backoff       Backoff
	ExclusiveBy   func(T) string
}

Definition declares one job type. Zero-valued MaxAttempts, Timeout and Backoff fields take the package defaults.

type ErrorCode

type ErrorCode string

ErrorCode is the closed set of queue failure classifications.

const (
	// CodeConfigInvalid reports a registration or limits value refused before
	// any background work begins.
	CodeConfigInvalid ErrorCode = "QUEUE_CONFIG_INVALID"
	// CodePayloadInvalid reports a payload that cannot be marshalled or that
	// exceeds the payload ceiling.
	CodePayloadInvalid ErrorCode = "QUEUE_PAYLOAD_INVALID"
	// CodeJobNotFound reports an operator lookup for an absent job.
	CodeJobNotFound ErrorCode = "QUEUE_JOB_NOT_FOUND"
	// CodeWorkerRunning reports a second concurrent worker on one application.
	CodeWorkerRunning ErrorCode = "QUEUE_WORKER_RUNNING"
	// CodeStoreFailure reports a durable store failure.
	CodeStoreFailure ErrorCode = "QUEUE_STORE_FAILURE"
)

func CodeOf

func CodeOf(err error) (ErrorCode, bool)

CodeOf reports the classification of a queue failure. It recognises wrapped errors and never classifies from message text.

type FailedCursor

type FailedCursor struct {
	FinishedAt time.Time
	ID         JobID
}

FailedCursor is the stable position immediately after one failed job.

type FailedPage

type FailedPage struct {
	Jobs []Status
	Next *FailedCursor
}

FailedPage is one payload-redacted page of failed jobs.

type FailedQuery

type FailedQuery struct {
	Types  []string
	Limit  int
	Before *FailedCursor
}

FailedQuery selects one bounded page of failed jobs. An empty Types list includes every stored type.

type Handler

type Handler func(ctx context.Context, payload []byte, meta Meta) error

Handler is the type-erased handler the worker invokes. Register builds one per definition; applications never write one.

type Job

type Job[T any] struct {
	ID          JobID
	Payload     T
	Attempt     int
	MaxAttempts int
	EnqueuedAt  time.Time
}

Job is one claimed unit of work handed to a handler.

type JobCursor

type JobCursor struct {
	EnqueuedAt time.Time
	ID         JobID
}

JobCursor is the stable position immediately after one listed job.

type JobID

type JobID string

JobID is the durable identity of one enqueued job.

type JobPage

type JobPage struct {
	Jobs []Status
	Next *JobCursor
}

JobPage is one payload-redacted page of jobs.

type JobQuery

type JobQuery struct {
	Types  []string
	States []State
	Limit  int
	Before *JobCursor
}

JobQuery selects one bounded payload-free page of jobs. Empty Types and States lists include every stored type and state.

type Limits

type Limits struct {
	Concurrency   int
	ClaimBatch    int
	LeaseDuration time.Duration
	PollInterval  time.Duration
	ShutdownGrace time.Duration
	// AbandonGrace is how long the worker waits for a handler to return after
	// cancelling it, on shutdown and on a per-type Timeout. Go cannot kill a
	// goroutine: once the grace elapses the worker stops waiting and records
	// the durable outcome while the handler may still be running, so an
	// abandoned timed-out job can execute concurrently with its own retry.
	// Handlers that must not run twice have to honour their context.
	AbandonGrace    time.Duration
	MaxPayloadBytes int
	// RetentionAge is how long a terminal job row survives automatic
	// retention. It bounds only succeeded, failed and canceled rows; live work
	// is never deleted.
	RetentionAge time.Duration
	// RetentionEvery is how often a worker runs automatic retention.
	// RetentionDisabled turns it off entirely and keeps every terminal row
	// until an operator calls RunRetention.
	RetentionEvery time.Duration
	RetentionRows  int
}

Limits bound one worker. Zero-valued fields take the package defaults.

func DefaultLimits

func DefaultLimits() Limits

DefaultLimits returns the documented worker defaults. Automatic retention is on: a worker deletes terminal job rows older than thirty days every minute. Applications that keep job history for longer must set RetentionAge, or RetentionEvery to RetentionDisabled.

func (Limits) Resolved

func (limits Limits) Resolved() Limits

Resolved returns the limits with every zero-valued field replaced by its default.

func (Limits) RetentionEnabled

func (limits Limits) RetentionEnabled() bool

RetentionEnabled reports whether a worker deletes terminal job rows on its own. It is false only when RetentionEvery is RetentionDisabled.

func (Limits) Validate

func (limits Limits) Validate() error

Validate refuses limits a worker cannot honour. Zero-valued fields are the defaults and always pass.

type Meta

type Meta struct {
	ID          JobID
	Attempt     int
	MaxAttempts int
	EnqueuedAt  time.Time
}

Meta is the durable job state a type-erased handler receives alongside the undecoded payload.

type Operator

type Operator interface {
	Inspect(ctx context.Context, id JobID) (Status, error)
	List(ctx context.Context, query JobQuery) (JobPage, error)
	ListFailed(ctx context.Context, query FailedQuery) (FailedPage, error)
	CountByState(ctx context.Context, query CountQuery) (StateCounts, error)
	Cancel(ctx context.Context, id JobID) (bool, error)
	CancelMany(ctx context.Context, ids []JobID) (int, error)
	Requeue(ctx context.Context, id JobID) (bool, error)
	RequeueFailed(ctx context.Context, ids []JobID) (int, error)
	RunRetention(ctx context.Context, policy RetentionPolicy) (int, error)
}

Operator is the durable job control surface. Every method acts on state the database owns; none of them reaches into a running handler. Cancellation is immediate for pending jobs and cooperative for leased jobs, so completion may win before a worker observes the request. RequeueFailed is retry-safe and may return a nonzero changed count together with an error. CancelMany has the same partial-progress contract.

type Option

type Option func(*enqueueOptions)

Option adjusts one enqueue.

func After

func After(delay time.Duration) Option

After schedules the job to become claimable only once the delay has passed.

func Dedupe

func Dedupe(key string) Option

Dedupe coalesces this enqueue with any active job already carrying the key.

type Outcome

type Outcome struct {
	Resolution Resolution
	Code       string
	Delay      time.Duration
	Scheduled  bool
	Uncounted  bool
	Err        error
}

Outcome is the classification of one handler return. Scheduled reports whether Delay was stated by the handler rather than left to the backoff.

func Classify

func Classify(err error) Outcome

Classify maps one handler return onto the durable outcome vocabulary. A nil error succeeds, a plain error retries, and the sentinels override both.

type Pending

type Pending struct {
	// contains filtered or unexported fields
}

Pending is one validated job awaiting a durable enqueue. It can only be produced by the Type its payload belongs to.

func (Pending) DedupeKey

func (pending Pending) DedupeKey() string

DedupeKey reports the coalescing key, empty when unset.

func (Pending) Delay

func (pending Pending) Delay() time.Duration

Delay reports how long after enqueue the job becomes claimable.

func (Pending) ExclusiveKey

func (pending Pending) ExclusiveKey() string

ExclusiveKey reports the exclusivity key, empty when the type is not exclusive.

func (Pending) MaxAttempts

func (pending Pending) MaxAttempts() int

MaxAttempts reports the attempt ceiling recorded with the durable row.

func (Pending) Payload

func (pending Pending) Payload() []byte

Payload returns a private copy of the marshalled payload.

func (Pending) TypeName

func (pending Pending) TypeName() string

TypeName reports the registered job type this work belongs to.

type Registration

type Registration struct {
	Type          string
	MaxAttempts   int
	Timeout       time.Duration
	MaxConcurrent int
	Backoff       Backoff
	Handle        Handler
}

Registration is one resolved definition with its defaults applied. It is the worker's view of a registered type.

type Registry

type Registry struct {
	// contains filtered or unexported fields
}

Registry holds every registered job type for one application.

func NewRegistry

func NewRegistry() *Registry

NewRegistry returns an empty registry.

func (*Registry) Clone

func (registry *Registry) Clone() *Registry

Clone returns an independent registry containing the same registrations.

func (*Registry) Lookup

func (registry *Registry) Lookup(name string) (Registration, bool)

Lookup returns the registration for a type name.

func (*Registry) Registrations

func (registry *Registry) Registrations() []Registration

Registrations returns every registration in declaration order.

type Resolution

type Resolution string

Resolution is the durable disposition one handler return maps to.

const (
	// ResolutionSucceeded terminates the job successfully.
	ResolutionSucceeded Resolution = "succeeded"
	// ResolutionRetry returns the job to pending and consumes an attempt.
	ResolutionRetry Resolution = "retry"
	// ResolutionFailed terminates the job as a permanent refusal.
	ResolutionFailed Resolution = "failed"
)

type Resource

type Resource struct {
	Name        string
	Concurrency int
	Costs       map[string]int
}

Resource is one fleet-wide weighted concurrency budget. Costs maps locally registered job types to the capacity consumed by one active lease. Workers with disjoint registries may share a name, but every active worker must agree on its meaning. Drain workers before changing capacity or costs.

type RetentionPolicy

type RetentionPolicy struct {
	OlderThan time.Time
	MaxRows   int
	States    []State
}

RetentionPolicy bounds one retention run over terminal rows.

type State

type State string

State is the durable job state.

const (
	// StatePending is enqueued work waiting for its availability time.
	StatePending State = "pending"
	// StateLeased is work owned by one worker until its lease expires.
	StateLeased State = "leased"
	// StateSucceeded is terminal successful completion.
	StateSucceeded State = "succeeded"
	// StateFailed is terminal refusal or attempt exhaustion.
	StateFailed State = "failed"
	// StateCanceled is terminal external cancellation.
	StateCanceled State = "canceled"
)

type StateCounts

type StateCounts struct {
	Pending   int64
	Leased    int64
	Succeeded int64
	Failed    int64
	Canceled  int64
}

StateCounts is the number of stored jobs in each durable state.

type Status

type Status struct {
	ID              JobID
	Type            string
	State           State
	Attempt         int
	MaxAttempts     int
	AvailableAt     time.Time
	LastCode        string
	CancelRequested bool
	EnqueuedAt      time.Time
	FinishedAt      *time.Time
}

Status is the sanitized operator view of one job. It carries no payload.

type Type

type Type[T any] struct {
	// contains filtered or unexported fields
}

Type is the sole constructor of work for one registered job type.

func Register

func Register[T any](registry *Registry, definition Definition[T]) (Type[T], error)

Register binds a payload type to a handler and returns the constructor for that job type. Every refusal is CodeConfigInvalid and happens before any background work exists.

func (Type[T]) New

func (jobType Type[T]) New(payload T, options ...Option) (Pending, error)

New marshals the payload, resolves the exclusivity key, and returns work ready to enqueue. An unmarshalable or oversized payload is refused here.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL