hopper

package module
v0.7.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 14, 2026 License: Apache-2.0 Imports: 22 Imported by: 1

README

hopper

A high-performance job queue and message broker for Go, built on Postgres.

hopper runs background jobs and delivers messages using the PostgreSQL database your application already has. There's no broker to operate and no separate worker service: it is a library that runs inside your process and scales out with it. Postgres is the first engine, behind a driver interface designed for more.

  • Fast: set-based claim, finalize and insert (including COPY), a live table that stays small however much history accumulates, and retention by dropping partitions. Throughput and latency targets are release gates.
  • Transactional enqueue and completion: jobs and messages commit or roll back with your data, and a worker can complete its job in the transaction that does its work.
  • Scale by adding replicas: work is claimed with FOR UPDATE SKIP LOCKED, and periodic jobs and maintenance run on one elected leader.
  • Reliable: retries with backoff, dead-lettering and replay, unique jobs, and lease-based rescue of jobs from crashed processes in seconds.
  • Complete: priorities, cron with time zones, snooze, cancellation, timeouts, awaitable results, global concurrency and rate limits, batches and workflows.
  • Messaging: topics, subscriptions and fan-out with AMQP-style routing, ordering keys, TTLs, idempotent publish and request/reply.
  • Idiomatic Go: generic, type-safe workers and messages, context, slog, and iterators. The core depends only on the standard library and pgx, and it runs under GODEBUG=fips140=only.

Status: v0.1.0. The first release covers the job queue (schema and migrations, the pgx and database/sql drivers, transactional and bulk inserts, batched claim and finalize, typed workers, retries with backoff, graceful shutdown, lease-based rescue of jobs from crashed processes, leader election, partition-drop retention, LISTEN/NOTIFY wake-ups, unique jobs, cron and interval schedules, cancellation, retry from the dead-letter queue, TTLs, pause and runtime queues, middleware, awaitable results, listing, events and stats), messaging (subscriptions with topic patterns, fan-out, dedup and ordering keys, streams with consumers, a SQL contract for producers in other languages), flow control, batches and workflows, plus the hopper CLI, hoppertest, hopperotel, the hopperui web UI and the drivertest conformance suite. docs/PLAN.md is the plan of record and docs/getting-started.md the place to start. Until v1.0.0, minor versions may change the API. Feedback is welcome through issues and PRs.

_, err := client.InsertTx(ctx, tx, SendEmail{UserID: 42}, nil)        // commits with tx
_, err = client.PublishTx(ctx, tx, AllocationCreated{ID: 42}, nil)  // fans out to subscribers

Inside a worker or subscriber, the client running the job comes from the context, so follow-up work needs no reference wired in after construction:

client, _ := hopper.ClientFromContext[pgx.Tx](ctx)
_, err := client.Insert(ctx, Next{ID: job.Args.ID}, nil)

The name

A feed hopper releases work into a machine one piece at a time. The name is also a tribute to RADM Grace Hopper, and it's a good name for something meant to replace a rabbit.

License

Apache-2.0

Documentation

Overview

Package hopper is a job queue and message broker for Go, built on the database the application already runs. PostgreSQL is the first engine.

A Client inserts jobs, claims them in batches, runs them on registered workers and finalizes results in batches. Any number of clients run against one database; work is claimed with FOR UPDATE SKIP LOCKED, so adding replicas adds throughput.

workers := hopper.NewWorkers()
hopper.AddWorker(workers, &SendEmailWorker{})

client, err := hopper.NewClient(hopperpgx.New(pool), &hopper.Config{
	Queues:  map[string]hopper.QueueConfig{hopper.QueueDefault: {MaxWorkers: 100}},
	Workers: workers,
})
g.Go(func() error { return client.Run(ctx) })

_, err = client.InsertTx(ctx, tx, SendEmail{UserID: 42}, nil) // commits with tx

Jobs run at least once. A job runs twice only when the process running it is lost mid-job, so workers must be idempotent.

The database schema is installed with the hoppermigrate package.

Index

Constants

View Source
const (
	DefaultJobTimeout         = time.Minute
	DefaultMaxAttempts        = 25
	DefaultStopTimeout        = 30 * time.Second
	DefaultPollInterval       = time.Second
	DefaultLeaseTTL           = 15 * time.Second
	DefaultCompletedRetention = 24 * time.Hour
	DefaultFailedRetention    = 7 * 24 * time.Hour
	DefaultStreamRetention    = 7 * 24 * time.Hour
)

Defaults for Config fields.

View Source
const (
	JobStatePending   = driver.JobStatePending
	JobStateAvailable = driver.JobStateAvailable
	JobStateScheduled = driver.JobStateScheduled
	JobStateRunning   = driver.JobStateRunning
	JobStateRetryable = driver.JobStateRetryable
	JobStateCompleted = driver.JobStateCompleted
	JobStateCanceled  = driver.JobStateCanceled
	JobStateDiscarded = driver.JobStateDiscarded
)

Job states. Available, scheduled and retryable all mean "claim once scheduled_at has passed"; they are distinct only so operators can see why a job is waiting.

View Source
const (
	// StreamStartLatest delivers events appended from now on. It is the
	// default.
	StreamStartLatest = driver.StreamStartLatest
	// StreamStartEarliest delivers every retained event first.
	StreamStartEarliest = driver.StreamStartEarliest
)
View Source
const (
	// DependencyCancel cancels the dependent step, and its dependents in
	// turn. It is the default.
	DependencyCancel = driver.DependencyCancel
	// DependencyIgnore runs the dependent step once the steps it depends on
	// have all finished, whatever their outcome.
	DependencyIgnore = driver.DependencyIgnore
)
View Source
const ChannelControl = driver.ChannelControl

ChannelControl and the other channel names are the Postgres NOTIFY channels hopper uses; see the driver package.

View Source
const QueueDefault = "default"

QueueDefault is the queue jobs go to when none is given.

Variables

View Source
var (
	// ErrJobRunning is returned by JobRetry for a running job.
	ErrJobRunning = driver.ErrJobRunning
	// ErrUniqueConflict is returned by JobRetry when a live job holds the
	// same unique key.
	ErrUniqueConflict = driver.ErrUniqueConflict
	// ErrJobCanceled is the cause of a job's context cancellation after
	// JobCancel. Check it with context.Cause.
	ErrJobCanceled = errors.New("hopper: job canceled")
)

Sentinel errors from control operations.

View Source
var (
	ErrClientStarted = errors.New("hopper: client already started")
	ErrClientStopped = errors.New("hopper: client stopped")
	// ErrSchemaOutdated is returned by Start when the database's schema is
	// older than this release needs. Apply the missing migrations with
	// hoppermigrate.Up or `hopper migrate up` first.
	ErrSchemaOutdated = errors.New("hopper: database schema is out of date")
)

Sentinel errors from Client lifecycle methods.

View Source
var ErrJobFenced = errors.New("hopper: job is no longer running in this attempt")

ErrJobFenced is returned by CompleteTx when the attempt no longer owns its job: the job was rescued after this client lost its lease, or it was already finalized.

View Source
var ErrJobTimeout = errors.New("hopper: job timed out")

ErrJobTimeout is the cause of a job's context cancellation when the job ran past its timeout. Check it with context.Cause.

View Source
var ErrNotFound = driver.ErrNotFound

ErrNotFound is returned when a job does not exist, live or in history.

View Source
var ErrNotInJob = errors.New("hopper: context does not belong to a running job")

ErrNotInJob is returned by SetOutput and CompleteTx outside a worker.

Functions

func AddWorkFunc

func AddWorkFunc[T JobArgs](workers *Workers, fn func(ctx context.Context, job *Job[T]) error)

AddWorkFunc registers a function as the worker for the kind of T, with default timeout and retry behavior.

func AddWorker

func AddWorker[T JobArgs](workers *Workers, worker Worker[T])

AddWorker registers worker for the kind of T. It panics if the kind is already registered, like http.Handle.

func Await

func Await[T any, TTx any](ctx context.Context, c *Client[TTx], id JobID) (T, error)

Await blocks until the job is finalized and returns its output decoded into T, or a *JobFailedError if it was canceled or discarded. Insert the job with InsertOpts.Await so that the finalize is announced and Await returns as soon as it commits; otherwise, and on a client that is not started, Await polls every Config.PollInterval. A completed job without output yields the zero T.

func Cancel

func Cancel(err error) error

Cancel returns an error a worker can return to discard the job now, with no retry. The job is archived as discarded with err recorded.

func CompleteTx added in v0.7.0

func CompleteTx[TTx any, T JobArgs](ctx context.Context, tx TTx, job *Job[T]) error

CompleteTx completes the running job inside tx, so that the worker's own writes and the job's completion commit together or not at all. ctx must be the context the worker was given. The job moves to history (unless its queue has DeleteCompleted) with the output SetOutput recorded before the call; awaited jobs, batches and workflow dependents are handled as on any completion, once tx commits.

The completion is fenced like every finalize: if the attempt no longer owns the job, nothing is written and ErrJobFenced is returned, and the worker should roll tx back. Once the worker returns, the client reads the job to learn whether tx committed. If it did, the job is finished, whatever the worker returned, and the job.completed event fires. If it did not, the attempt fails with the worker's error, or with a failure if the worker returned nil, and the job is retried. A worker that returns before tx ends holds up only its own job's result, until tx ends.

Call it last, just before the commit. From CompleteTx until tx ends, tx holds the job's row and those of workflow dependents it promotes or cancels, and holds off the creation and drop of history partitions. The job's siblings in its batch finalize without waiting for it.

func (w *Worker) Work(ctx context.Context, job *hopper.Job[Args]) error {
	tx, err := w.pool.Begin(ctx)
	if err != nil {
		return err
	}
	defer tx.Rollback(ctx)
	if err := apply(ctx, tx, job.Args); err != nil {
		return err
	}
	if err := hopper.CompleteTx(ctx, tx, job); err != nil {
		return err
	}
	return tx.Commit(ctx)
}

func Consume

func Consume[T any](workers *Workers, consumer Consumer, fn func(ctx context.Context, msg *Message[T]) error)

Consume registers a handler for a consumer's deliveries. The consumer is created (at its Start) or updated when a client using workers starts, and the leader delivers its events in order as jobs; fn receives them as Subscribe's handler does, with Message.Stream and Message.Position set. An event's key becomes the delivery's ordering key, so events with the same key are handled one at a time, in order.

It panics on an invalid consumer or a name registered twice, like AddWorker.

func ContextWithClient added in v0.2.0

func ContextWithClient[TTx any](ctx context.Context, client *Client[TTx]) context.Context

ContextWithClient returns a context from which ClientFromContext returns client. Jobs get this from the client that runs them; it is for tests that call a worker directly, as with hoppertest.Work.

func PerSecond

func PerSecond(n float64) float64

PerSecond is a readable RateLimit value.

func SetOutput

func SetOutput(ctx context.Context, v any) error

SetOutput records the job's result, which is stored with the finalized job, returned by JobGet and delivered to Await. The last call wins. Output is kept only for archived jobs, so it is lost on queues with DeleteCompleted. A job finalized with CompleteTx stores the output set before that call.

func Snooze

func Snooze(d time.Duration) error

Snooze returns an error a worker can return to reschedule the job after d without using up an attempt.

func Subscribe

func Subscribe[T any](workers *Workers, sub Subscription, fn func(ctx context.Context, msg *Message[T]) error)

Subscribe registers a consumer for a subscription. The subscription row is created or updated when a client using workers starts. fn is called with each delivery: returning nil acks it, an error nacks it with backoff (capped at five minutes for ordered deliveries), and Snooze and Cancel work as for jobs. Deliveries out of attempts are dead-lettered in history, where ReplayDiscarded can re-drive them.

It panics on an invalid subscription or a name registered twice, like AddWorker.

Types

type AppendOpts

type AppendOpts struct {
	// Key orders the event's deliveries with others of the same key: a
	// consumer handles them one at a time, in order.
	Key string
	// Headers travel with the event and are available on Message.
	Headers map[string]string
}

AppendOpts customizes an append. A nil *AppendOpts means the defaults.

type AttemptError

type AttemptError = driver.AttemptError

AttemptError records one failed attempt of a job.

type Batch

type Batch[TTx any] struct {
	// contains filtered or unexported fields
}

Batch collects jobs to insert together with completion callbacks.

func (*Batch[TTx]) Add

func (b *Batch[TTx]) Add(args JobArgs, opts *InsertOpts)

Add queues a job for the batch.

func (*Batch[TTx]) Insert

func (b *Batch[TTx]) Insert(ctx context.Context) (*BatchResult, error)

Insert inserts the batch and its jobs.

func (*Batch[TTx]) InsertTx

func (b *Batch[TTx]) InsertTx(ctx context.Context, tx TTx) (*BatchResult, error)

InsertTx inserts the batch and its jobs in the caller's transaction.

type BatchOpts

type BatchOpts struct {
	OnSuccess  JobArgs
	OnFailure  JobArgs
	OnComplete JobArgs
	// CallbackOpts applies to every callback job.
	CallbackOpts *InsertOpts
	// Metadata is stored with the batch.
	Metadata []byte
}

BatchOpts sets a batch's callbacks. Each is inserted as a job when the batch completes: OnSuccess if no job of the batch was canceled or discarded, OnFailure otherwise, and OnComplete either way. Callback metadata gains "batch_id" and "batch_failed" (the number of failed jobs).

type BatchResult

type BatchResult struct {
	ID   JobID
	Jobs []*InsertResult
}

BatchResult reports an inserted batch.

type BatchRow

type BatchRow = driver.BatchRow

BatchRow is a batch's progress.

type CancelError

type CancelError struct {
	Err error
}

CancelError is returned by a worker to discard the job immediately, with no retry. Use Cancel to create one.

func (*CancelError) Error

func (e *CancelError) Error() string

func (*CancelError) Unwrap

func (e *CancelError) Unwrap() error

type Client

type Client[TTx any] struct {
	// contains filtered or unexported fields
}

Client is the per-process runtime. It inserts, claims, works and finalizes jobs, and holds a liveness lease while running. TTx is the engine's transaction type, fixed by the driver.

func ClientFromContext added in v0.2.0

func ClientFromContext[TTx any](ctx context.Context) (*Client[TTx], bool)

ClientFromContext returns the client that is running the job whose context ctx is, so a worker or subscriber can insert follow-up jobs, publish, or query without holding a client reference. That matters for code that is wired before the client exists: the workers are registered first and the client is built from them. The transaction type must match the client's; a mismatch, or a context from outside a job, returns false.

func (w *Worker) Work(ctx context.Context, job *hopper.Job[Args]) error {
	client, ok := hopper.ClientFromContext[pgx.Tx](ctx)
	if !ok {
		return hopper.ErrNotInJob
	}
	_, err := client.Insert(ctx, Next{ID: job.Args.ID}, nil)
	return err
}

func NewClient

func NewClient[TTx any](d driver.Driver[TTx], cfg *Config) (*Client[TTx], error)

NewClient validates cfg and returns a client. Nothing runs until Start.

func (*Client[TTx]) BatchGet

func (c *Client[TTx]) BatchGet(ctx context.Context, id JobID) (*BatchRow, error)

BatchGet returns a batch's progress.

func (*Client[TTx]) Clients

func (c *Client[TTx]) Clients(ctx context.Context) ([]*ClientRow, error)

Clients lists every client lease, live or expired (expired rows are pruned by the leader).

func (*Client[TTx]) Events

func (c *Client[TTx]) Events(ctx context.Context, kinds ...EventKind) iter.Seq[Event]

Events streams events of the given kinds (all kinds if none) until ctx is done. Events are delivered best-effort: a consumer that falls behind by more than a buffer's worth misses some.

func (*Client[TTx]) Insert

func (c *Client[TTx]) Insert(ctx context.Context, args JobArgs, opts *InsertOpts) (*InsertResult, error)

Insert inserts one job. It can be called before Start and on an insert-only client. When this client works the job's queue, its producer is woken directly, with no database round trip.

func (*Client[TTx]) InsertMany

func (c *Client[TTx]) InsertMany(ctx context.Context, params []InsertParams) ([]*InsertResult, error)

InsertMany inserts a batch of jobs in one statement, or with COPY for large batches without unique keys. Results are in input order.

func (*Client[TTx]) InsertManyTx

func (c *Client[TTx]) InsertManyTx(ctx context.Context, tx TTx, params []InsertParams) ([]*InsertResult, error)

InsertManyTx is InsertMany in the caller's transaction.

func (*Client[TTx]) InsertTx

func (c *Client[TTx]) InsertTx(ctx context.Context, tx TTx, args JobArgs, opts *InsertOpts) (*InsertResult, error)

InsertTx inserts one job in the caller's transaction. The job exists only if tx commits, and becomes visible to workers when it does.

func (*Client[TTx]) InsertWorkflow

func (c *Client[TTx]) InsertWorkflow(ctx context.Context, wf *Workflow) (*WorkflowResult, error)

InsertWorkflow inserts a workflow's steps and dependencies atomically.

func (*Client[TTx]) InsertWorkflowTx

func (c *Client[TTx]) InsertWorkflowTx(ctx context.Context, tx TTx, wf *Workflow) (*WorkflowResult, error)

InsertWorkflowTx is InsertWorkflow in the caller's transaction.

func (*Client[TTx]) JobCancel

func (c *Client[TTx]) JobCancel(ctx context.Context, id JobID) (*JobRow, error)

JobCancel cancels a job. A waiting job moves to history as canceled. A running job, on any client, has its context canceled (with cause ErrJobCanceled) and finalizes as canceled unless it completes anyway. A finalized job is returned unchanged. The job as it is afterwards is returned.

func (*Client[TTx]) JobGet

func (c *Client[TTx]) JobGet(ctx context.Context, id JobID) (*JobRow, error)

JobGet returns a job by ID, live or from history. It returns ErrNotFound if the job does not exist or its history has been dropped.

func (*Client[TTx]) JobRetry

func (c *Client[TTx]) JobRetry(ctx context.Context, id JobID) (*JobRow, error)

JobRetry makes a job run again as soon as possible: a canceled or discarded job is re-driven from history with one more attempt allowed if it had run out, and a waiting job has its schedule brought forward. An explicit retry overrides the wait, so the job's TTL no longer applies: a job past its deadline still runs. It returns ErrJobRunning for a running job.

func (*Client[TTx]) JobRetryOrderingKey added in v0.7.0

func (c *Client[TTx]) JobRetryOrderingKey(ctx context.Context, queue, orderingKey string) (int, error)

JobRetryOrderingKey wakes an ordering key: its jobs waiting out a snooze, a retry delay or their schedule become runnable now, and still run one at a time, oldest first. Unlike JobRetry it keeps each job's TTL, and leaves a job already past it to expire. A job of the key that is running is not interrupted; if its attempt ends in a snooze or a retry, it runs again at once instead of after its delay. It returns how many waiting jobs it woke.

func (*Client[TTx]) Jobs

func (c *Client[TTx]) Jobs(ctx context.Context, filter JobFilter) iter.Seq2[*JobRow, error]

Jobs lists jobs matching the filter, live and from history, in ID order (which is insertion order), or newest first with NewestFirst. It pages transparently; stop iterating to stop fetching.

func (*Client[TTx]) JobsTx

func (c *Client[TTx]) JobsTx(ctx context.Context, tx TTx, filter JobFilter) iter.Seq2[*JobRow, error]

JobsTx is Jobs inside the caller's transaction, so that jobs inserted with InsertTx can be listed before the transaction commits.

func (*Client[TTx]) NewBatch

func (c *Client[TTx]) NewBatch(opts BatchOpts) *Batch[TTx]

NewBatch starts a batch. Add jobs to it, then Insert or InsertTx.

func (*Client[TTx]) Publish

func (c *Client[TTx]) Publish(ctx context.Context, msg Topic, opts *PublishOpts) (*PublishResult, error)

Publish delivers a message to every subscription matching its topic, as one job each, in one statement. With no matching subscription nothing is inserted, as with a broker.

func (*Client[TTx]) PublishTx

func (c *Client[TTx]) PublishTx(ctx context.Context, tx TTx, msg Topic, opts *PublishOpts) (*PublishResult, error)

PublishTx is Publish in the caller's transaction: the deliveries exist only if tx commits.

func (*Client[TTx]) Queues

func (c *Client[TTx]) Queues() *QueueControl[TTx]

Queues returns the queue controller.

func (*Client[TTx]) ReplayDiscarded

func (c *Client[TTx]) ReplayDiscarded(ctx context.Context, subscription string) (int, error)

ReplayDiscarded re-drives every dead-lettered delivery of a subscription and returns how many.

func (*Client[TTx]) Run

func (c *Client[TTx]) Run(ctx context.Context) error

Run starts the client and blocks until ctx is canceled, then stops it, waiting up to Config.StopTimeout for running jobs. It returns nil after a clean drain, so it fits errgroup.

func (*Client[TTx]) Start

func (c *Client[TTx]) Start(ctx context.Context) error

Start registers the client's lease and begins claiming and working jobs. ctx bounds the startup itself; the client keeps running after it ends. Use Run for a lifecycle tied to a context. It returns ErrSchemaOutdated, naming the missing migrations, if the database's schema is older than this release needs.

func (*Client[TTx]) Stats

func (c *Client[TTx]) Stats(ctx context.Context) (*Stats, error)

Stats returns queue depths by state, the age of the oldest claimable job, recent throughput, the running count by client and the current leader.

func (*Client[TTx]) Stop

func (c *Client[TTx]) Stop(ctx context.Context) error

Stop shuts the client down in three steps: stop claiming and give up leadership; wait for running jobs to finish and flush their results; release the lease. If ctx expires during the wait, every job's context is canceled, whatever finishes within a short grace period is flushed, and ctx's error is returned. Jobs still running then are rescued by another client once this one's lease expires.

func (*Client[TTx]) Streams

func (c *Client[TTx]) Streams() *Streams[TTx]

Streams returns the client's stream log.

func (*Client[TTx]) Subscriptions

func (c *Client[TTx]) Subscriptions(ctx context.Context) ([]*SubscriptionRow, error)

Subscriptions lists every subscription known to the cluster.

func (*Client[TTx]) WorkflowGet

func (c *Client[TTx]) WorkflowGet(ctx context.Context, id JobID) (*WorkflowRow, error)

WorkflowGet returns a workflow's progress, its jobs (live and finished) and its edges.

type ClientRow

type ClientRow = driver.ClientRow

ClientRow is a client process's lease.

type Codec

type Codec interface {
	Marshal(v any) ([]byte, error)
	Unmarshal(data []byte, v any) error
}

Codec encodes job args and outputs for storage. The encoded form must be valid JSON, because it is stored in a jsonb column; a codec that encrypts or compresses wraps its output in a JSON string. The default is encoding/json.

type Config

type Config struct {
	// Queues lists the queues this client works, with per-queue settings. A
	// client with no queues only inserts jobs.
	Queues map[string]QueueConfig
	// Workers is the registry of workers. Required when Queues is set.
	Workers *Workers
	// Periodic lists jobs inserted on a schedule by the leader. See Cron
	// and Every.
	Periodic []PeriodicJob
	// Middleware wraps job execution and inserts, outermost first.
	Middleware []Middleware
	// Logger defaults to slog.Default().
	Logger *slog.Logger
	// Codec encodes args and outputs. Defaults to JSONCodec.
	Codec Codec
	// JobTimeout is how long a job may run before its context is canceled,
	// unless its worker's Timeout says otherwise. Defaults to one minute.
	JobTimeout time.Duration
	// MaxAttempts is the default number of attempts before a job is
	// discarded. Defaults to 25.
	MaxAttempts int
	// StopTimeout bounds how long Run waits for running jobs after its
	// context is canceled, before canceling them. Defaults to 30 seconds.
	StopTimeout time.Duration
	// PollInterval is how often an idle producer checks its queue for work
	// that arrived without a wake-up. Defaults to one second.
	PollInterval time.Duration
	// StrictKinds rejects inserts whose kind has no worker on this client.
	// By default any kind can be inserted, since another service may work it.
	StrictKinds bool
	// Hostname identifies this process in the clients table. Defaults to
	// os.Hostname().
	Hostname string
	// CompletedRetention is how long completed jobs stay in history.
	// Defaults to 24 hours; negative keeps them forever. Retention is
	// enforced by the leader, so the leader's setting applies cluster-wide.
	CompletedRetention time.Duration
	// FailedRetention is how long canceled and discarded jobs stay in
	// history, which is the dead-letter queue. Defaults to 7 days; negative
	// keeps them forever.
	FailedRetention time.Duration
	// StreamRetention is how long stream events stay in the log, for
	// consumers that start from the earliest event or are moved back.
	// Defaults to 7 days; negative keeps them forever.
	StreamRetention time.Duration
	// NeverLead keeps this client out of leader elections. It still claims
	// and works jobs, but never runs the leader's duties: rescuing jobs from
	// lost clients, retention, maintenance, periodic jobs and stream
	// delivery. Use it for clients that share an install but must not run it,
	// such as a developer's process against a shared environment. At least
	// one client of every install must be able to lead.
	NeverLead bool
	// LeaseTTL is how long this client's lease lasts without a renewal. It
	// is renewed three times per TTL; when it lapses, the client's running
	// jobs are rescued and the client cancels them. A longer lease rides
	// out a longer database outage before interrupting jobs, and delays the
	// rescue of a crashed client's jobs by as much. Defaults to 15 seconds.
	LeaseTTL time.Duration
	// RescueStuckAfter, if set, rescues jobs that have been running longer
	// than this even though their client's lease is live, for workers that
	// ignore their context. Off by default; timeouts cover most cases.
	RescueStuckAfter time.Duration
}

Config configures a Client. Every field has a production default; the zero Config is an insert-only client.

type Consumer

type Consumer struct {
	// Name identifies the consumer; the delivery job kind is derived from
	// it.
	Name    string
	Pattern string
	// Queue is where deliveries go. Defaults to QueueDefault.
	Queue string
	// Start applies when the consumer is created. Defaults to
	// StreamStartLatest.
	Start StreamStart
	// MaxAttempts is the retry budget of deliveries. Zero means 25, or 10
	// for events with a key, whose retries block their key.
	MaxAttempts int
	// Metadata is merged into every delivery's metadata.
	Metadata json.RawMessage
}

Consumer reads the stream log from a position of its own and delivers the events whose topic matches Pattern to a queue, as jobs of kind "stream:<Name>". Unlike a Subscription, which is fanned out to at publish time, a consumer can start anywhere in the retained log and be moved (Streams.Seek), so late subscribers and replays see past events.

func (Consumer) Kind

func (c Consumer) Kind() string

Kind returns the job kind of the consumer's deliveries.

type ConsumerRow

type ConsumerRow = driver.StreamConsumerRow

ConsumerRow is a consumer as recorded in the database, with its position.

type CronSchedule

type CronSchedule struct {
	// contains filtered or unexported fields
}

CronSchedule is a parsed cron expression. Use ParseCron.

func ParseCron

func ParseCron(spec string) (*CronSchedule, error)

ParseCron parses a cron expression.

Standard five fields (minute, hour, day of month, month, day of week) or six with a leading seconds field. Each field takes "*", a value, a range ("1-5"), a step ("*/15", "1-30/5"), a list ("1,15") and, for months and days of the week, three-letter names ("JAN", "MON"). Day of week is 0-6 with 7 also meaning Sunday. When both day fields are restricted, a day matches if either does, as in Vixie cron.

The descriptors @yearly (@annually), @monthly, @weekly, @daily (@midnight) and @hourly are accepted, as is "@every <duration>" for an IntervalSchedule-like cadence.

Times are evaluated in UTC unless the expression starts with "CRON_TZ=" or "TZ=" and an IANA zone ("CRON_TZ=America/Chicago 0 9 * * MON-FRI"), or the schedule is given a location with In. Around a daylight-saving change, a time in the skipped hour does not occur that day, and a time in the repeated hour occurs twice, as in Vixie cron.

func (*CronSchedule) In

func (s *CronSchedule) In(loc *time.Location) *CronSchedule

In returns a copy of the schedule evaluated in loc.

func (*CronSchedule) Next

func (s *CronSchedule) Next(t time.Time) time.Time

Next implements Schedule. It advances field by field, from months down to seconds, resetting the lower fields whenever a higher one moves.

func (*CronSchedule) String

func (s *CronSchedule) String() string

String returns the expression the schedule was parsed from.

type DependencyFailure

type DependencyFailure = driver.DependencyFailure

DependencyFailure says what happens to a step when a step it depends on is canceled or discarded.

type Event

type Event struct {
	Kind EventKind
	At   time.Time
	// Job is set for job events: the row as claimed, before the outcome.
	Job *JobRow
	// Error is the attempt's error text for failed, discarded and
	// canceled jobs, and the snooze reason, if any, for snoozed jobs.
	Error string
	// ClientID is the client the event happened on.
	ClientID int64
}

Event is one occurrence on a client.

type EventKind

type EventKind string

EventKind names an event stream.

const (
	EventJobCompleted  EventKind = "job.completed"
	EventJobFailed     EventKind = "job.failed" // an attempt failed; the job will retry
	EventJobSnoozed    EventKind = "job.snoozed"
	EventJobCanceled   EventKind = "job.canceled"
	EventJobDiscarded  EventKind = "job.discarded"
	EventJobRescued    EventKind = "job.rescued"
	EventLeaderElected EventKind = "leader.elected"
	EventLeaderLost    EventKind = "leader.lost"
	EventLeaseLost     EventKind = "lease.lost"
)

Event kinds.

type InsertOpts

type InsertOpts struct {
	// Queue defaults to QueueDefault.
	Queue string
	// Priority defaults to PriorityNormal.
	Priority Priority
	// MaxAttempts defaults to Config.MaxAttempts.
	MaxAttempts int
	// ScheduledAt delays the job until the given time. Zero runs it as soon
	// as possible.
	ScheduledAt time.Time
	// Metadata is a JSON object stored with the job, for trace context and
	// similar. It is not passed to the worker as args.
	Metadata json.RawMessage
	// Unique makes the job unique among live jobs of its kind. See
	// UniqueOpts.
	Unique *UniqueOpts
	// TTL bounds how long the job may wait to run, measured from its
	// insert: before its first attempt, between retries, and while snoozed.
	// A job still waiting when it passes is discarded without running again.
	// A running job is not interrupted. Zero means no limit.
	TTL time.Duration
	// Await announces the job's finalize, so that Await returns as soon as
	// it commits instead of on its next poll.
	Await bool
	// OrderingKey serializes the job with others of the same key in its
	// queue: at most one runs at a time, oldest first, and a failing job
	// blocks its key while it retries. See PublishOpts.OrderingKey.
	OrderingKey string
	// PartitionKey groups the job for the queue's PartitionLimit, for
	// example a customer ID.
	PartitionKey string
}

InsertOpts customizes one insert. A nil *InsertOpts means the defaults, which come from the args' InsertOpts method if it has one, then from the client's Config.

type InsertParams

type InsertParams struct {
	Args JobArgs
	Opts *InsertOpts
	// contains filtered or unexported fields
}

InsertParams is one job in an InsertMany call.

type InsertResult

type InsertResult struct {
	Job *JobRow
	// Duplicate reports that a live job with the same unique key already
	// existed. Job is then that existing job.
	Duplicate bool
}

InsertResult is the outcome of inserting one job.

type IntervalSchedule

type IntervalSchedule struct {
	Interval time.Duration
}

IntervalSchedule fires every Interval, at slots aligned to the interval (every 30 seconds fires at :00 and :30), so slots are the same whichever leader computes them.

func (IntervalSchedule) Next

func (s IntervalSchedule) Next(t time.Time) time.Time

Next implements Schedule.

type JSONCodec

type JSONCodec struct{}

JSONCodec is the default Codec, using encoding/json.

func (JSONCodec) Marshal

func (JSONCodec) Marshal(v any) ([]byte, error)

Marshal implements Codec.

func (JSONCodec) Unmarshal

func (JSONCodec) Unmarshal(data []byte, v any) error

Unmarshal implements Codec.

type Job

type Job[T JobArgs] struct {
	*JobRow
	Args T
}

Job is a claimed job with its decoded args.

type JobArgs

type JobArgs interface {
	Kind() string
}

JobArgs is the payload of a job. Kind names the job type and maps to exactly one worker. Args must be a value type (not a pointer), encodable by the client's Codec, and Kind must not depend on the value's fields.

type JobArgsWithInsertOpts

type JobArgsWithInsertOpts interface {
	JobArgs
	InsertOpts() InsertOpts
}

JobArgsWithInsertOpts is implemented by args that carry per-kind insert defaults. Options passed to Insert override them field by field.

type JobFailedError

type JobFailedError struct {
	Job *JobRow
}

JobFailedError is returned by Await when the job ended other than completed.

func (*JobFailedError) Error

func (e *JobFailedError) Error() string

type JobFilter

type JobFilter struct {
	Queue  string
	Kinds  []string
	States []JobState
	// OrderingKey lists one ordering key's jobs, its lane. Keys are scoped
	// to a queue, so it needs Queue; the listing fails without one.
	OrderingKey string
	// NewestFirst lists in descending ID order: the latest inserted jobs
	// first, though IDs from the same millisecond can sort either way.
	NewestFirst bool
	// After starts the listing after the given job, for paging: pass the
	// last ID of one page to get the next.
	After JobID
}

JobFilter selects jobs for Jobs. Empty fields match everything.

type JobID

type JobID = driver.JobID

JobID identifies a job. It is a UUIDv7 generated by the database.

func ParseJobID

func ParseJobID(s string) (JobID, error)

ParseJobID parses the canonical 8-4-4-4-12 text form of a JobID.

type JobRow

type JobRow = driver.JobRow

JobRow is a job as stored, with its encoded args. Workers receive a typed Job[T] that embeds it.

type JobState

type JobState = driver.JobState

JobState is a job's position in the state machine.

type Message

type Message[T any] struct {
	*JobRow
	Payload T
	Topic   string
	// ID identifies the published message; every subscription's delivery of
	// it shares the ID.
	ID          string
	Headers     map[string]string
	OrderingKey string
	// Stream and Position are set for a consumer's deliveries: the
	// consumer's name and the event's place in the log.
	Stream   string
	Position StreamPosition
}

Message is one delivery of a published message to a subscription.

type Middleware

type Middleware interface {
	// Work wraps one attempt of a job. next runs the worker (and any inner
	// middleware); its error is the worker's return value.
	Work(ctx context.Context, job *JobRow, next func(ctx context.Context) error) error
	// Insert wraps an insert batch. next performs the insert; middleware may
	// pass it a modified copy of params, for example with trace context in
	// Metadata.
	Insert(ctx context.Context, params []InsertParams, next func(ctx context.Context, params []InsertParams) ([]*InsertResult, error)) ([]*InsertResult, error)
}

Middleware wraps job execution and inserts, for logging, metrics, tracing and custom policy. Embed MiddlewareDefaults to implement only one side, or use WorkMiddleware and InsertMiddleware for a function.

Middleware runs in Config.Middleware order: the first wraps the second, and so on down to the worker or the insert.

func InsertMiddleware

func InsertMiddleware(fn func(ctx context.Context, params []InsertParams, next func(ctx context.Context, params []InsertParams) ([]*InsertResult, error)) ([]*InsertResult, error)) Middleware

InsertMiddleware makes a Middleware from a function that wraps inserts.

func WorkMiddleware

func WorkMiddleware(fn func(ctx context.Context, job *JobRow, next func(ctx context.Context) error) error) Middleware

WorkMiddleware makes a Middleware from a function that wraps execution.

type MiddlewareDefaults

type MiddlewareDefaults struct{}

MiddlewareDefaults implements Middleware as pass-through.

func (MiddlewareDefaults) Insert

func (MiddlewareDefaults) Insert(ctx context.Context, params []InsertParams, next func(ctx context.Context, params []InsertParams) ([]*InsertResult, error)) ([]*InsertResult, error)

Insert implements Middleware.

func (MiddlewareDefaults) Work

func (MiddlewareDefaults) Work(ctx context.Context, _ *JobRow, next func(ctx context.Context) error) error

Work implements Middleware.

type PanicError

type PanicError struct {
	Value any
	Stack string
}

PanicError records a worker panic. The job is treated as failed.

func (*PanicError) Error

func (e *PanicError) Error() string

type PeriodicJob

type PeriodicJob struct {
	// Name identifies the job in the periodic table. It defaults to the
	// args' kind and must be unique among a client's periodic jobs.
	Name     string
	Schedule Schedule
	Args     JobArgs
	Opts     *PeriodicOpts
	// contains filtered or unexported fields
}

PeriodicJob inserts a job on a schedule. The leader computes the slots and inserts each at most once cluster-wide, even across leader changes. Build one with Cron or Every.

func Cron

func Cron(spec string, args JobArgs, opts *PeriodicOpts) PeriodicJob

Cron returns a periodic job on a cron expression (see ParseCron). An invalid expression is reported by NewClient.

func Every

func Every(interval time.Duration, args JobArgs, opts *PeriodicOpts) PeriodicJob

Every returns a periodic job that fires every interval, at slots aligned to the interval.

type PeriodicOpts

type PeriodicOpts struct {
	// Name overrides the periodic job's name.
	Name string
	// RunOnStart inserts a job as soon as a leader starts scheduling, in
	// addition to the regular slots.
	RunOnStart bool
	// CatchUp inserts up to this many slots missed while no leader was
	// running (for example during a full outage), oldest first. By default
	// missed slots are skipped, except a slot due in the last few seconds,
	// which covers a leader failover.
	CatchUp int
	// Location evaluates a cron expression in this zone when the
	// expression does not name one. Defaults to UTC.
	Location *time.Location
	// InsertOpts are applied to every inserted job.
	InsertOpts *InsertOpts
}

PeriodicOpts customizes a periodic job.

type Priority

type Priority int

Priority orders jobs within a queue. Lower runs first.

const (
	PriorityHigh   Priority = 1
	PriorityNormal Priority = 2 // the default
	PriorityLow    Priority = 3
	PriorityLowest Priority = 4
)

The four priority levels.

type PublishOpts

type PublishOpts struct {
	// OrderingKey makes delivery FIFO per key, per queue: at most one
	// delivery with the key runs at a time, oldest first, and a failing
	// delivery blocks its key while it retries. Keys are independent.
	OrderingKey string
	// DedupKey makes the publish idempotent: while a delivery with the key
	// is live, a repeated publish inserts nothing and reports Duplicate.
	DedupKey string
	// Headers travel with the message and are available on Message.
	Headers map[string]string
	// Delay defers delivery.
	Delay time.Duration
	// TTL bounds how long each delivery may wait to run, as
	// InsertOpts.TTL does: one still waiting when it passes, whether before
	// its first attempt, between retries or while snoozed, is discarded.
	TTL time.Duration
	// Priority defaults to PriorityNormal.
	Priority Priority
	// Await announces each delivery's finalize, for Await.
	Await bool
}

PublishOpts customizes a publish. A nil *PublishOpts means the defaults.

type PublishResult

type PublishResult struct {
	// MessageID identifies the message across its deliveries. It is empty
	// when no subscription matched.
	MessageID  string
	Deliveries []*InsertResult
}

PublishResult reports the deliveries a publish produced, one per matching subscription.

type QueueConfig

type QueueConfig struct {
	// MaxWorkers is the number of jobs this client runs at once from the
	// queue. Required.
	MaxWorkers int
	// DeleteCompleted deletes completed jobs instead of archiving them to
	// history, for maximum throughput. Canceled and discarded jobs are
	// always archived, because history is the dead-letter queue.
	DeleteCompleted bool

	// GlobalLimit caps the queue's running jobs across all clients.
	GlobalLimit int
	// RateLimit caps claims per second across all clients, in a token
	// bucket of RateBurst tokens (defaulting to one second's worth).
	RateLimit float64
	RateBurst int
	// PartitionLimit caps running jobs per InsertOpts.PartitionKey.
	PartitionLimit int
	// PriorityAging raises a waiting job's priority by one level each time
	// it has waited this long, so low priorities cannot starve.
	PriorityAging time.Duration
}

QueueConfig configures one queue on a client.

type QueueControl

type QueueControl[TTx any] struct {
	// contains filtered or unexported fields
}

QueueControl pauses, resumes, lists and adds queues.

func (*QueueControl[TTx]) Add

func (q *QueueControl[TTx]) Add(ctx context.Context, name string, cfg QueueConfig) error

Add starts working a queue on this client at runtime, for example a tenant's queue. The client must be started.

func (*QueueControl[TTx]) List

func (q *QueueControl[TTx]) List(ctx context.Context) ([]*QueueRow, error)

List returns every queue known to the cluster.

func (*QueueControl[TTx]) Pause

func (q *QueueControl[TTx]) Pause(ctx context.Context, name string) error

Pause stops every client from claiming from the queue. Running jobs finish. Clients learn of it through a notification and on every lease renewal.

func (*QueueControl[TTx]) Remove

func (q *QueueControl[TTx]) Remove(ctx context.Context, name string) error

Remove stops working a queue on this client, waiting for its running jobs until ctx is done.

func (*QueueControl[TTx]) Resume

func (q *QueueControl[TTx]) Resume(ctx context.Context, name string) error

Resume undoes Pause.

func (*QueueControl[TTx]) SetLimits

func (q *QueueControl[TTx]) SetLimits(ctx context.Context, name string, limits QueueLimits) error

SetLimits replaces a queue's cluster-wide limits. Zero values remove a limit. Clients learn of the change through a notification and on their next lease renewal.

type QueueLimits

type QueueLimits = driver.QueueLimits

QueueLimits are a queue's cluster-wide limits; see QueueConfig.

type QueueRow

type QueueRow = driver.QueueRow

QueueRow is a queue's cluster-wide state.

type QueueStats

type QueueStats = driver.QueueStats

QueueStats is one queue's depth by state.

type Raw

type Raw = json.RawMessage

Raw is a message payload left undecoded, for a subscription that matches several topics with different payload types; dispatch on Message.Topic.

type Schedule

type Schedule interface {
	// Next returns the first slot strictly after t, or the zero time if
	// there is none within a reasonable horizon.
	Next(t time.Time) time.Time
}

Schedule computes the slots of a periodic job.

type SeekOpts

type SeekOpts struct {
	// Position makes the event after it the next delivered.
	Position StreamPosition
	// Time replays from the first event appended at or after it.
	Time time.Time
	// Earliest replays every retained event.
	Earliest bool
	// Latest skips to the events appended from now on.
	Latest bool
}

SeekOpts says where to move a consumer. Exactly one field applies, in this order of precedence.

type SnoozeError

type SnoozeError struct {
	Duration time.Duration
	// Reason is stored on the job while it waits, as JobRow.SnoozeReason,
	// for operators, and is the Error of the job.snoozed event. It is
	// cleared when the job runs again. A job that expires or is canceled
	// while it waits keeps the reason in the error it is finalized with.
	Reason string
}

SnoozeError is returned by a worker to run the job again later without using up an attempt. Use Snooze to create one, or return a SnoozeError with a Reason to record why the job is waiting.

func (*SnoozeError) Error

func (e *SnoozeError) Error() string

type Stats

type Stats = driver.Stats

Stats is a snapshot of queue depths and cluster state.

type Step

type Step struct {
	// contains filtered or unexported fields
}

Step identifies a step of a workflow, for other steps to depend on.

type StepOpts

type StepOpts struct {
	// After lists the steps this one waits for. It is inserted pending and
	// becomes available once they have all finished.
	After []Step
	// OnDependencyFailure defaults to DependencyCancel.
	OnDependencyFailure DependencyFailure
	// Insert holds the job's own options.
	Insert *InsertOpts
}

StepOpts sets a step's dependencies and insert options.

func After

func After(steps ...Step) *StepOpts

After returns options for a step that waits for the given steps.

type StreamEvent

type StreamEvent = driver.StreamEvent

StreamEvent is one event of the stream log.

type StreamPosition

type StreamPosition = driver.StreamPosition

StreamPosition is a place in the stream log.

func ParseStreamPosition

func ParseStreamPosition(s string) (StreamPosition, error)

ParseStreamPosition parses a position's String form ("xid:seq").

type StreamStart

type StreamStart = driver.StreamStart

StreamStart says where a new consumer starts.

type Streams

type Streams[TTx any] struct {
	// contains filtered or unexported fields
}

Streams is the stream log: an append-only, retained record of events that consumers read from positions of their own.

func (*Streams[TTx]) Append

func (s *Streams[TTx]) Append(ctx context.Context, event Topic, opts *AppendOpts) (*StreamEvent, error)

Append appends an event to the log. Consumers whose pattern matches its topic deliver it, in order, after everything appended before it.

func (*Streams[TTx]) AppendTx

func (s *Streams[TTx]) AppendTx(ctx context.Context, tx TTx, event Topic, opts *AppendOpts) (*StreamEvent, error)

AppendTx is Append in the caller's transaction: the event exists only if tx commits, and is delivered after events of every transaction that started before tx, however they commit.

func (*Streams[TTx]) Consumers

func (s *Streams[TTx]) Consumers(ctx context.Context) ([]*ConsumerRow, error)

Consumers lists every consumer known to the cluster.

func (*Streams[TTx]) Read

func (s *Streams[TTx]) Read(ctx context.Context, pattern string, after StreamPosition, limit int) iter.Seq2[*StreamEvent, error]

Read returns the events after a position whose topic matches pattern (empty for every topic), oldest first, in pages of limit. It sees only events that every earlier transaction has finished around, as consumers do, so paging by the last position never skips an event.

func (*Streams[TTx]) Seek

func (s *Streams[TTx]) Seek(ctx context.Context, consumer string, opts SeekOpts) error

Seek moves a consumer's position; the events after it are delivered (again) from the next pump. Deliveries already inserted are unaffected.

type Subscription

type Subscription struct {
	// Name identifies the subscription; the delivery job kind is derived
	// from it.
	Name    string
	Pattern string
	// Queue is where deliveries go. Defaults to QueueDefault.
	Queue string
	// MaxAttempts is the retry budget of deliveries. Zero means 25, or 10
	// for deliveries with an ordering key, whose retries block their key.
	MaxAttempts int
	// Metadata is merged into every delivery's metadata.
	Metadata json.RawMessage
}

Subscription routes messages whose topic matches Pattern to a queue, as jobs of kind "sub:<Name>".

Pattern uses AMQP topic syntax: dot-separated words, where "*" matches one word and "#" zero or more. "#" alone receives everything, and a literal topic exactly that topic. Fan-out happens at publish time, so a subscription created later does not receive earlier messages.

func (Subscription) Kind

func (s Subscription) Kind() string

Kind returns the job kind of the subscription's deliveries.

type SubscriptionRow

type SubscriptionRow = driver.SubscriptionRow

SubscriptionRow is a subscription as recorded in the database.

type Topic

type Topic interface {
	Topic() string
}

Topic is implemented by message payloads: Topic names the topic the message is published to, in dot-separated words ("allocation.created").

type UniqueConflict

type UniqueConflict int

UniqueConflict says what happens when a unique job already exists.

const (
	// UniqueSkip keeps the existing job. Insert reports Duplicate: true with
	// the existing job.
	UniqueSkip UniqueConflict = iota
	// UniqueReplace updates the existing job's args, metadata, priority,
	// max attempts and schedule, unless it is running. This gives
	// debouncing: inserting again pushes the job back. Insert still reports
	// Duplicate: true.
	UniqueReplace
)

Unique conflict actions.

type UniqueOpts

type UniqueOpts struct {
	// ByArgs includes the args in the key: the fields tagged
	// `hopper:"unique"`, or the whole args if no field is tagged.
	ByArgs bool
	// ByQueue includes the queue in the key.
	ByQueue bool
	// ByPeriod includes the current time, truncated to this period, so that
	// at most one job exists per period. For example 24 * time.Hour allows
	// one job per UTC day.
	ByPeriod time.Duration
	// OnConflict is what to do when the job exists.
	OnConflict UniqueConflict
}

UniqueOpts makes a job unique among live jobs (those not yet finalized) of its kind. The zero value is unique by kind alone: at most one live job of the kind at a time.

type UnknownKindError

type UnknownKindError struct {
	Kind string
}

UnknownKindError reports a job kind with no registered worker.

func (*UnknownKindError) Error

func (e *UnknownKindError) Error() string

type Worker

type Worker[T JobArgs] interface {
	// Work runs the job. Returning nil completes it. Returning an error
	// schedules a retry with backoff, or discards the job when it is out of
	// attempts. Return Snooze(d) to run again later without using an attempt
	// and Cancel(err) to discard now.
	Work(ctx context.Context, job *Job[T]) error
	// Timeout returns how long the job may run before its context is
	// canceled: 0 for the client's default, or a negative duration for no
	// timeout.
	Timeout(job *Job[T]) time.Duration
	// NextRetry returns when a failed job runs again, or the zero time for
	// the default backoff.
	NextRetry(job *Job[T]) time.Time
}

Worker performs one kind of job. Embed WorkerDefaults to get default Timeout and NextRetry implementations.

type WorkerDefaults

type WorkerDefaults[T JobArgs] struct{}

WorkerDefaults provides default Timeout and NextRetry methods. Embed it in a worker to implement only Work.

func (WorkerDefaults[T]) NextRetry

func (WorkerDefaults[T]) NextRetry(*Job[T]) time.Time

NextRetry returns the zero time, which selects the default backoff.

func (WorkerDefaults[T]) Timeout

func (WorkerDefaults[T]) Timeout(*Job[T]) time.Duration

Timeout returns 0, which selects the client's default.

type Workers

type Workers struct {
	// contains filtered or unexported fields
}

Workers is a registry of workers by kind. Register at startup with AddWorker and AddWorkFunc, then pass it to NewClient.

func NewWorkers

func NewWorkers() *Workers

NewWorkers returns an empty registry.

func (*Workers) Work

func (w *Workers) Work(ctx context.Context, job *JobRow, codec Codec) error

Work runs the registered worker for job.Kind inline, decoding job.Args with codec and honoring the worker's Timeout. It exists for tests (see the hoppertest package): nothing is claimed or finalized, and middleware does not run. It returns an UnknownKindError for an unregistered kind.

type Workflow

type Workflow struct {
	// contains filtered or unexported fields
}

Workflow collects steps with dependencies between them, to insert together. Steps are jobs; a step with dependencies waits, pending, until they have finished. Because a step can only depend on steps added before it, a workflow is always a DAG.

func NewWorkflow

func NewWorkflow(name string, opts *WorkflowOpts) *Workflow

NewWorkflow starts a workflow. Add steps to it, then insert it with Client.InsertWorkflow or InsertWorkflowTx.

func (*Workflow) Add

func (w *Workflow) Add(args JobArgs, opts *StepOpts) Step

Add adds a step. opts may be nil for a step with no dependencies; After(steps...) is the usual way to declare them.

type WorkflowOpts

type WorkflowOpts struct {
	OnSuccess  JobArgs
	OnFailure  JobArgs
	OnComplete JobArgs
	// CallbackOpts applies to every callback job.
	CallbackOpts *InsertOpts
	// Metadata is stored with the workflow.
	Metadata json.RawMessage
}

WorkflowOpts sets a workflow's completion callbacks and metadata. A workflow is a batch, so the callbacks are those of BatchOpts: OnSuccess when every step completed, OnFailure when any was canceled or discarded, OnComplete either way.

type WorkflowResult

type WorkflowResult struct {
	ID JobID
	// Jobs are in step order.
	Jobs []*InsertResult
}

WorkflowResult reports an inserted workflow.

type WorkflowRow

type WorkflowRow = driver.WorkflowRow

WorkflowRow is a workflow's progress, jobs and graph.

Directories

Path Synopsis
cmd
hopper command
Command hopper administers a hopper installation from the shell.
Command hopper administers a hopper installation from the shell.
hopperbench command
Command hopperbench measures hopper's hot paths against a real database and prints machine-readable results, one JSON object per scenario.
Command hopperbench measures hopper's hot paths against a real database and prints machine-readable results, one JSON object per scenario.
Package driver defines the contract between the hopper core and a storage engine.
Package driver defines the contract between the hopper core and a storage engine.
hopperpgx
Package hopperpgx is hopper's reference driver: PostgreSQL through pgx v5.
Package hopperpgx is hopper's reference driver: PostgreSQL through pgx v5.
hoppersql
Package hoppersql is hopper's PostgreSQL driver for database/sql, for applications that already use it with lib/pq or pgx's stdlib adapter.
Package hoppersql is hopper's PostgreSQL driver for database/sql, for applications that already use it with lib/pq or pgx's stdlib adapter.
internal/pgsql
Package pgsql implements hopper's driver operations for PostgreSQL over a small transport interface, so that the pgx driver (hopperpgx) and the database/sql driver (hoppersql) share one implementation of the SQL and the logic around it.
Package pgsql implements hopper's driver operations for PostgreSQL over a small transport interface, so that the pgx driver (hopperpgx) and the database/sql driver (hoppersql) share one implementation of the SQL and the logic around it.
hoppersqlite module
Package drivertest is the conformance suite every hopper driver must pass.
Package drivertest is the conformance suite every hopper driver must pass.
examples
basic command
Command basic is a small, complete hopper application: it installs the schema, registers a worker, starts a client, inserts a few jobs (one of them inside a transaction) and waits for a result.
Command basic is a small, complete hopper application: it installs the schema, registers a worker, starts a client, inserts a few jobs (one of them inside a transaction) and waits for a result.
Package hoppermigrate installs and upgrades hopper's database schema.
Package hoppermigrate installs and upgrades hopper's database schema.
hopperotel module
Package hoppertest has helpers for testing code that uses hopper.
Package hoppertest has helpers for testing code that uses hopper.
hopperui module
internal
testdb
Package testdb gives integration tests a Postgres database of their own.
Package testdb gives integration tests a Postgres database of their own.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL