Documentation
¶
Overview ¶
Package queue runs typed background jobs. A job is a struct with a Kind; its handler is a plain function registered explicitly:
type SendInvoice struct{ OrderID int64 }
func (SendInvoice) Kind() string { return "send_invoice" }
q.Register(queue.Handle(func(ctx context.Context, j SendInvoice) error { ... },
queue.Retries(5), queue.Backoff(queue.Exponential(time.Second)), queue.Timeout(time.Minute)))
err := q.Dispatch(ctx, SendInvoice{OrderID: 1}, queue.OnQueue("billing"), queue.Delay(5*time.Minute))
Payloads are JSON. Drivers store messages: Sync runs jobs inline, Memory keeps them in process, the database driver (and Redis) persist them. Queue.Work runs a worker (vukem queue:work); Fake records dispatched jobs in tests. Apps get a queue from app.Queue(), configured by QUEUE_CONNECTION.
See https://github.com/AlexGalhardo/vukem/blob/main/docs/subsystems.md#queue.
Example ¶
package main
import (
"context"
"fmt"
"log/slog"
"time"
"github.com/AlexGalhardo/vukem/queue"
)
type SendInvoice struct{ OrderID int64 }
func (SendInvoice) Kind() string { return "send_invoice" }
func main() {
ctx := context.Background()
// Sync runs jobs inline; apps use the database or Redis driver and a
// worker (vukem queue:work).
q := queue.New(queue.Sync{}, slog.New(slog.DiscardHandler))
err := q.Register(queue.Handle(func(_ context.Context, j SendInvoice) error {
fmt.Println("sending invoice for order", j.OrderID)
return nil
}, queue.Retries(5), queue.Backoff(queue.Exponential(time.Second))))
if err != nil {
fmt.Println(err)
return
}
err = q.Dispatch(ctx, SendInvoice{OrderID: 1}, queue.OnQueue("billing"))
fmt.Println(err)
}
Output: sending invoice for order 1 <nil>
Index ¶
- Variables
- func Constant(d time.Duration) func(int) time.Duration
- func Dispatched[J Job](f *Fake) []J
- func Exponential(base time.Duration) func(int) time.Duration
- type Database
- func (d *Database) Ack(ctx context.Context, m *Message) error
- func (d *Database) CreateTables(ctx context.Context) error
- func (d *Database) Fail(ctx context.Context, m *Message) error
- func (d *Database) Failed(ctx context.Context) ([]FailedJob, error)
- func (d *Database) Forget(ctx context.Context, ids ...string) (int, error)
- func (d *Database) Pending(ctx context.Context) (int, error)
- func (d *Database) Pop(ctx context.Context, queues []string) (*Message, error)
- func (d *Database) Push(ctx context.Context, m *Message) error
- func (d *Database) PushTx(ctx context.Context, ex database.Executor, m *Message) error
- func (d *Database) Release(ctx context.Context, m *Message) error
- func (d *Database) Retry(ctx context.Context, ids ...string) (int, error)
- type DatabaseOptions
- type Driver
- type FailedJob
- type FailedStore
- type Fake
- type Inline
- type Job
- type JobEvent
- type JobStatus
- type Memory
- func (d *Memory) Ack(_ context.Context, m *Message) error
- func (d *Memory) Fail(_ context.Context, m *Message) error
- func (d *Memory) Failed() []Message
- func (d *Memory) Pending() int
- func (d *Memory) Pop(_ context.Context, queues []string) (*Message, error)
- func (d *Memory) Push(_ context.Context, m *Message) error
- func (d *Memory) Release(_ context.Context, m *Message) error
- type Message
- type Observer
- type Option
- type Queue
- func (q *Queue) Dispatch(ctx context.Context, j Job, opts ...Option) error
- func (q *Queue) DispatchKind(ctx context.Context, kind string, payload any, opts ...Option) error
- func (q *Queue) DispatchTx(ctx context.Context, ex database.Executor, j Job, opts ...Option) error
- func (q *Queue) Driver() Driver
- func (q *Queue) Message(j Job, opts ...Option) (*Message, error)
- func (q *Queue) Process(ctx context.Context, m *Message) error
- func (q *Queue) Register(regs ...Registration) error
- func (q *Queue) SetObserver(obs Observer)
- func (q *Queue) Work(ctx context.Context, o WorkOptions) error
- type Registration
- type Sync
- type TxPusher
- type WorkOptions
Examples ¶
Constants ¶
This section is empty.
Variables ¶
var ErrDuplicateKind = errors.New("queue: job kind already registered")
ErrDuplicateKind is returned by Register for a kind registered twice.
var ErrNoTx = errors.New("queue: driver does not support DispatchTx")
ErrNoTx is returned by DispatchTx when the driver cannot enqueue inside a transaction.
var ErrUnknownKind = errors.New("queue: no handler for job kind")
ErrUnknownKind is returned by Dispatch for a kind without a handler.
Functions ¶
func Dispatched ¶
Dispatched returns the dispatched jobs of type J, decoded, in order.
Types ¶
type Database ¶
type Database struct {
// contains filtered or unexported fields
}
Database stores jobs in SQL tables (SQLite, Postgres or MySQL). Workers reserve jobs atomically (Postgres and MySQL use FOR UPDATE SKIP LOCKED), so any number of worker processes can share the tables. A job whose worker died is reserved again once its timeout plus a grace period has passed: delivery is at least once, so handlers should be idempotent.
func NewDatabase ¶
func NewDatabase(db *database.DB, o DatabaseOptions) (*Database, error)
NewDatabase returns a database driver; call CreateTables (or a migration with the same schema) before use.
func (*Database) CreateTables ¶
CreateTables creates the jobs and failed jobs tables if missing.
func (*Database) Forget ¶
Forget implements FailedStore: it deletes failed jobs (all when ids is empty).
func (*Database) PushTx ¶
PushTx inserts m with ex, so it is enqueued only if ex's transaction commits (Queue.DispatchTx).
type DatabaseOptions ¶
type DatabaseOptions struct {
// Table holds pending jobs (default "jobs").
Table string
// FailedTable holds failed jobs (default "failed_jobs").
FailedTable string
// Grace is added to a job's timeout before an abandoned reservation is
// taken over (default 1 minute).
Grace time.Duration
// Now is the clock (default time.Now); tests set it.
Now func() time.Time
}
DatabaseOptions name the tables.
type Driver ¶
type Driver interface {
// Push stores m; it becomes available at m.AvailableAt.
Push(ctx context.Context, m *Message) error
// Pop reserves the next available message on one of queues (in
// order), or returns nil when there is none.
Pop(ctx context.Context, queues []string) (*Message, error)
// Ack deletes a processed message.
Ack(ctx context.Context, m *Message) error
// Release makes m available again at m.AvailableAt (a retry).
Release(ctx context.Context, m *Message) error
// Fail moves m to the failed jobs.
Fail(ctx context.Context, m *Message) error
}
Driver stores messages and hands them to workers.
type FailedStore ¶
type FailedStore interface {
// Failed lists failed jobs, newest first.
Failed(ctx context.Context) ([]FailedJob, error)
// Retry re-queues failed jobs (all when ids is empty).
Retry(ctx context.Context, ids ...string) (int, error)
// Forget deletes failed jobs (all when ids is empty).
Forget(ctx context.Context, ids ...string) (int, error)
}
FailedStore is implemented by drivers that keep failed jobs (Database).
type Fake ¶
type Fake struct {
*Memory
// contains filtered or unexported fields
}
Fake records dispatched jobs for tests; it stores them like Memory, so a test can also run them with Queue.Work.
fake := queue.NewFake()
app := vukemtest.New(t, bootstrap.New, vukemtest.Options{AppOptions: []vukem.Option{vukem.WithQueueDriver(fake)}})
... exercise the app ...
invoices := queue.Dispatched[jobs.SendInvoice](fake)
Example ¶
package main
import (
"context"
"fmt"
"log/slog"
"github.com/AlexGalhardo/vukem/queue"
)
type SendInvoice struct{ OrderID int64 }
func (SendInvoice) Kind() string { return "send_invoice" }
func main() {
ctx := context.Background()
fake := queue.NewFake()
q := queue.New(fake, slog.New(slog.DiscardHandler))
// Dispatch only accepts registered kinds; the Fake stores the job
// instead of running it.
_ = q.Register(queue.Handle(func(context.Context, SendInvoice) error { return nil }))
_ = q.Dispatch(ctx, SendInvoice{OrderID: 7})
for _, j := range queue.Dispatched[SendInvoice](fake) {
fmt.Println("dispatched order", j.OrderID)
}
}
Output: dispatched order 7
type Inline ¶
type Inline interface {
Inline() bool
}
Inline is implemented by drivers that run jobs during Dispatch (Sync).
type Job ¶
type Job interface {
Kind() string
}
Job is a unit of background work. Kind names its handler and must be stable across deploys (it is stored with the payload).
type JobEvent ¶ added in v0.19.0
type JobEvent struct {
ID string
Kind string
Queue string
Status JobStatus
Attempt int
Duration time.Duration
Err error
}
JobEvent describes one step of a job for an Observer. It carries no payload: payloads may hold user data.
type JobStatus ¶ added in v0.19.0
type JobStatus string
JobStatus is the step a JobEvent reports.
const ( // JobDispatched: the job was stored (or is about to run inline). JobDispatched JobStatus = "dispatched" // JobProcessed: an attempt succeeded. JobProcessed JobStatus = "processed" // JobFailed: an attempt failed (it may still be retried). JobFailed JobStatus = "failed" )
Job steps reported to an Observer.
type Memory ¶
type Memory struct {
// contains filtered or unexported fields
}
Memory keeps messages in process: jobs are lost on restart. Use it for tests and single-process apps that can afford that.
func (*Memory) Pop ¶
Pop implements Driver: the earliest available message of the first queue in queues that has one.
type Message ¶
type Message struct {
ID string `json:"id"`
Kind string `json:"kind"`
Queue string `json:"queue"`
Payload json.RawMessage `json:"payload"`
Attempts int `json:"attempts"`
MaxRetries int `json:"max_retries"`
Timeout time.Duration `json:"timeout"`
AvailableAt time.Time `json:"available_at"`
CreatedAt time.Time `json:"created_at"`
// LastError is set when the job failed.
LastError string `json:"last_error,omitempty"`
}
Message is a stored job.
type Observer ¶ added in v0.19.0
Observer is notified when jobs are dispatched and attempted. It runs on the caller's goroutine, so it must be fast and safe for concurrent use.
type Option ¶
type Option func(*options)
Option configures a handler (defaults) or a single dispatch.
type Queue ¶
type Queue struct {
// contains filtered or unexported fields
}
Queue dispatches jobs to a driver and runs them in workers.
func (*Queue) Dispatch ¶
Dispatch enqueues j. Options override the handler's defaults for this dispatch. With an Inline driver the job runs now and its error is returned.
func (*Queue) DispatchKind ¶
DispatchKind enqueues payload (encoded as JSON) for the handler of kind.
func (*Queue) DispatchTx ¶
DispatchTx enqueues j inside the transaction of ex: the job exists only if the transaction commits, so it never runs for rolled-back work and is never lost for committed work.
func (*Queue) Message ¶
Message builds the stored message for j (drivers with their own transactional push use it).
func (*Queue) Process ¶
Process runs one reserved message and acks, releases (retry with backoff) or fails it.
func (*Queue) Register ¶
func (q *Queue) Register(regs ...Registration) error
Register adds handlers; a kind may be registered once.
func (*Queue) SetObserver ¶ added in v0.19.0
SetObserver reports job steps to obs (development tooling such as devtools). Call it before the queue is used.
type Registration ¶
type Registration struct {
// contains filtered or unexported fields
}
Registration binds a handler to its job kind; create it with Handle.
func HandleKind ¶
func HandleKind[T any](kind string, fn func(ctx context.Context, v T) error, opts ...Option) Registration
HandleKind registers fn for jobs named kind whose payload decodes into T (framework jobs such as queued mail and listeners; DispatchKind sends them).
func (Registration) Kind ¶
func (r Registration) Kind() string
Kind returns the job kind of the registration.
type Sync ¶
type Sync struct{}
Sync runs jobs inline during Dispatch (development, tests, simple apps). Delays and retries do not apply: the handler's error goes to the caller.
type TxPusher ¶
TxPusher is implemented by drivers that can enqueue inside a database transaction (Database).
type WorkOptions ¶
type WorkOptions struct {
// Queues are polled in priority order (default ["default"]).
Queues []string
// Concurrency is the number of parallel workers (default 1).
Concurrency int
// Sleep is the wait when no job is available (default 1s).
Sleep time.Duration
// MaxJobs stops after processing this many jobs (0 = no limit), e.g.
// for "queue:work --once".
MaxJobs int
// StopWhenEmpty returns when no job is available.
StopWhenEmpty bool
}
WorkOptions configure Work.