Documentation
¶
Overview ¶
Package jq is a small job queue backed by a Redis list.
A Queue publishes jobs with Queue.Pub and consumes them with Queue.StartWorker. Payloads are encoded with msgpack, so anything msgpack can marshal can be a payload. Failed jobs are retried with exponential backoff and, after WorkerOptions.MaxRetry attempts, either discarded or moved to a "dropped" list when WorkerOptions.SafeDrop is set.
For a queue named "email", the following Redis keys are used:
email:queue list of pending jobs email:dropped list of jobs that exhausted their retries (SafeDrop only) email:count hash of process/success/failed/dropped counters email:active timestamp of the last counter update
It intentionally uses a plain list rather than Redis Streams: there are no consumer groups, no acknowledgements and no fan-out. A job taken by a worker that dies mid-handler is lost.
Index ¶
- Variables
- type HandlerFunc
- type Job
- type Option
- type Queue
- func (q *Queue) Drop(ctx context.Context, job *Job) error
- func (q *Queue) Get(ctx context.Context) (*Job, error)
- func (q *Queue) Len(ctx context.Context) (int64, error)
- func (q *Queue) Name() string
- func (q *Queue) Pub(ctx context.Context, payload any) (string, error)
- func (q *Queue) PubTo(ctx context.Context, name string, payload any) (string, error)
- func (q *Queue) Retry(ctx context.Context, job *Job) error
- func (q *Queue) StartWorker(ctx context.Context, handle HandlerFunc, opt *WorkerOptions)
- func (q *Queue) Status(ctx context.Context) (*Status, error)
- type RedisClient
- type ReportFunc
- type Status
- type WorkerOptions
Constants ¶
This section is empty.
Variables ¶
var ErrEmpty = fmt.Errorf("jq: queue is empty: %w", redis.Nil)
ErrEmpty is returned by Queue.Get when the queue has no job. It wraps redis.Nil, so callers checking that still work.
Functions ¶
This section is empty.
Types ¶
type HandlerFunc ¶
HandlerFunc processes one job. It is called concurrently from up to WorkerOptions.Parallel goroutines, so it must be safe for concurrent use. ctx is the worker context: it is cancelled when the worker is stopping.
type Option ¶ added in v0.7.0
type Option func(*Queue)
Option configures a Queue.
func WithLogger ¶ added in v0.7.0
WithLogger sets the logger. By default nothing is logged.
type Queue ¶
type Queue struct {
// contains filtered or unexported fields
}
Queue is a single named job queue.
func NewQueue ¶
func NewQueue(name string, rdb RedisClient, opts ...Option) *Queue
NewQueue creates a queue named name on rdb.
func (*Queue) Drop ¶ added in v0.2.0
Drop moves a job to the dropped list. The worker calls it when a job has exhausted its retries and WorkerOptions.SafeDrop is set.
func (*Queue) Get ¶
Get pops one job from the queue. It returns ErrEmpty when there is none. Most callers should use Queue.StartWorker instead.
func (*Queue) Pub ¶
Pub publishes a job to this queue and returns its ID. The payload must be marshallable by msgpack (https://github.com/vmihailenco/msgpack).
func (*Queue) Retry ¶
Retry republishes a failed job after an exponential backoff.
ctx controls only the waiting, not the Redis calls: cancelling it skips the backoff so a stopping worker hands the job back quickly. If Redis is unreachable, Retry keeps trying every minute until ctx is done and then returns the last error, so a transient outage does not lose the job. Most callers should use Queue.StartWorker instead.
func (*Queue) StartWorker ¶
func (q *Queue) StartWorker(ctx context.Context, handle HandlerFunc, opt *WorkerOptions)
StartWorker consumes the queue with handle until ctx is done, then waits for in-flight jobs and pending retries before returning. A nil opt uses the defaults. opt is copied, so it can be shared between workers.
type RedisClient ¶
type RedisClient interface {
Get(ctx context.Context, key string) *redis.StringCmd
Set(ctx context.Context, key string, value any, expiration time.Duration) *redis.StatusCmd
Del(ctx context.Context, keys ...string) *redis.IntCmd
HIncrBy(ctx context.Context, key, field string, incr int64) *redis.IntCmd
HGetAll(ctx context.Context, key string) *redis.MapStringStringCmd
LPush(ctx context.Context, key string, values ...any) *redis.IntCmd
RPop(ctx context.Context, key string) *redis.StringCmd
LLen(ctx context.Context, key string) *redis.IntCmd
Pipeline() redis.Pipeliner
}
RedisClient is the subset of go-redis used by this package, so any of the go-redis client types (Client, ClusterClient, Ring, UniversalClient) fits.
type ReportFunc ¶ added in v0.5.0
type ReportFunc func(status *Status)
ReportFunc receives the counters collected since the last report. See WorkerOptions.Idle and WorkerOptions.Reporter.
type Status ¶ added in v0.2.0
type Status struct {
// IsRunning is false when no counter exists, i.e. nothing happened since
// the last reset.
IsRunning bool
Process int
Success int
Failed int
Dropped int
// Total is Success + Dropped: jobs that reached a final state.
Total int
}
Status holds the counters collected since the last reset.
type WorkerOptions ¶
type WorkerOptions struct {
// MaxRetry is how many times a failed job is retried. Default: 3.
MaxRetry int
// Parallel is the number of jobs handled concurrently. Default: 2.
Parallel int
// Interval is how long to wait when the queue is empty. Default: 3s.
Interval time.Duration
// Idle is how long the queue must be inactive before Reporter is called
// and the counters are reset. Default: 3min.
Idle time.Duration
// Reporter, when set, receives the counters after Idle of inactivity.
Reporter ReportFunc
// Recover is how long to wait after a Redis error. Default: 1min.
Recover time.Duration
// SafeDrop keeps jobs that exhausted their retries in the dropped list
// instead of discarding them. Default: false, so the list can not grow forever.
SafeDrop bool
// WG, when set, is incremented when StartWorker starts and decremented
// when it returns. Since the Add happens inside StartWorker, do not Wait
// on it before StartWorker has begun; wg.Go(func() { q.StartWorker(...) })
// is the simpler alternative.
WG *sync.WaitGroup
}
WorkerOptions tunes a worker. The zero value uses the documented defaults.