jq

package module
v0.7.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 2, 2026 License: MIT Imports: 11 Imported by: 3

README

jq — Redis job queue

Go Reference CI

A small job queue that uses a Redis list as its backend. Not Redis Streams.

Use it for:

  • the simplest multi-publisher, multi-worker model;
  • workloads where a few seconds of scheduling delay is fine;
  • when you want to understand exactly what the queue does.

Not for:

  • blocking consumers or push delivery;
  • fan-out to several consumers;
  • strict ordering or exactly-once delivery.

Install

go get github.com/hack-fan/jq

Requires Go 1.26+ and go-redis v9.

Usage

package main

import (
	"context"
	"log/slog"
	"os"
	"os/signal"

	"github.com/hack-fan/jq"
	"github.com/redis/go-redis/v9"
)

type Email struct {
	To, Subject, Body string
}

func main() {
	rdb := redis.NewClient(&redis.Options{Addr: "localhost:6379"})
	q := jq.NewQueue("email", rdb, jq.WithLogger(slog.Default()))

	// Publish. The payload can be anything msgpack can marshal.
	ctx := context.Background()
	id, err := q.Pub(ctx, Email{To: "a@example.com", Subject: "hi"})
	if err != nil {
		panic(err)
	}
	slog.Info("published", "id", id)

	// Consume until interrupted. StartWorker blocks and drains in-flight
	// jobs before returning.
	ctx, stop := signal.NotifyContext(ctx, os.Interrupt)
	defer stop()
	q.StartWorker(ctx, func(ctx context.Context, job *jq.Job) error {
		var e Email
		if err := job.Bind(&e); err != nil {
			return err
		}
		return send(ctx, e) // returning an error schedules a retry
	}, &jq.WorkerOptions{
		Parallel: 4,
		MaxRetry: 5,
	})
}
Worker options
Option Default Meaning
MaxRetry 3 Retries before a job is dropped. Backoff is 1s, 2s, 4s, … (max 1024s).
Parallel 2 Jobs handled concurrently.
Interval 3s Wait when the queue is empty.
Recover 1m Wait after a Redis error.
Idle 3m Inactivity before Reporter is called and counters reset.
Reporter nil Receives a *Status with the counters collected since the last report.
SafeDrop false Keep jobs that exhausted their retries in <name>:dropped.
WG nil A sync.WaitGroup the worker joins for the duration of StartWorker.
Redis keys

For a queue named email:

Key Type Content
email:queue list pending jobs
email:dropped list jobs that exhausted their retries (SafeDrop)
email:count hash process / success / failed / dropped
email:active str time of the last counter update (binary time.Time)
Lower level API

Get, Retry, Drop, Len and Status are exported so you can write your own worker loop. Get returns jq.ErrEmpty (which also matches redis.Nil) when there is nothing to do.

Delivery semantics

A job is popped from the list before it is handled. If the process dies while a handler is running, that job is lost. Cancelling the worker context is safe: running handlers finish, pending retries are pushed back immediately, and the final counters are still recorded.

License

MIT

Documentation

Overview

Package jq is a small job queue backed by a Redis list.

A Queue publishes jobs with Queue.Pub and consumes them with Queue.StartWorker. Payloads are encoded with msgpack, so anything msgpack can marshal can be a payload. Failed jobs are retried with exponential backoff and, after WorkerOptions.MaxRetry attempts, either discarded or moved to a "dropped" list when WorkerOptions.SafeDrop is set.

For a queue named "email", the following Redis keys are used:

email:queue    list of pending jobs
email:dropped  list of jobs that exhausted their retries (SafeDrop only)
email:count    hash of process/success/failed/dropped counters
email:active   timestamp of the last counter update

It intentionally uses a plain list rather than Redis Streams: there are no consumer groups, no acknowledgements and no fan-out. A job taken by a worker that dies mid-handler is lost.

Index

Constants

This section is empty.

Variables

View Source
var ErrEmpty = fmt.Errorf("jq: queue is empty: %w", redis.Nil)

ErrEmpty is returned by Queue.Get when the queue has no job. It wraps redis.Nil, so callers checking that still work.

Functions

This section is empty.

Types

type HandlerFunc

type HandlerFunc func(ctx context.Context, job *Job) error

HandlerFunc processes one job. It is called concurrently from up to WorkerOptions.Parallel goroutines, so it must be safe for concurrent use. ctx is the worker context: it is cancelled when the worker is stopping.

type Job

type Job struct {
	ID      string
	PubAt   time.Time
	Retried int
	Payload []byte
}

Job is the envelope stored in the Redis list.

func (*Job) Bind

func (j *Job) Bind(v any) error

Bind decodes the job payload into v, which must be a pointer to the same type that was published.

type Option added in v0.7.0

type Option func(*Queue)

Option configures a Queue.

func WithLogger added in v0.7.0

func WithLogger(l *slog.Logger) Option

WithLogger sets the logger. By default nothing is logged.

type Queue

type Queue struct {
	// contains filtered or unexported fields
}

Queue is a single named job queue.

func NewQueue

func NewQueue(name string, rdb RedisClient, opts ...Option) *Queue

NewQueue creates a queue named name on rdb.

func (*Queue) Drop added in v0.2.0

func (q *Queue) Drop(ctx context.Context, job *Job) error

Drop moves a job to the dropped list. The worker calls it when a job has exhausted its retries and WorkerOptions.SafeDrop is set.

func (*Queue) Get

func (q *Queue) Get(ctx context.Context) (*Job, error)

Get pops one job from the queue. It returns ErrEmpty when there is none. Most callers should use Queue.StartWorker instead.

func (*Queue) Len added in v0.7.0

func (q *Queue) Len(ctx context.Context) (int64, error)

Len returns the number of pending jobs.

func (*Queue) Name added in v0.7.0

func (q *Queue) Name() string

Name returns the queue name.

func (*Queue) Pub

func (q *Queue) Pub(ctx context.Context, payload any) (string, error)

Pub publishes a job to this queue and returns its ID. The payload must be marshallable by msgpack (https://github.com/vmihailenco/msgpack).

func (*Queue) PubTo

func (q *Queue) PubTo(ctx context.Context, name string, payload any) (string, error)

PubTo publishes a job to another queue on the same Redis.

func (*Queue) Retry

func (q *Queue) Retry(ctx context.Context, job *Job) error

Retry republishes a failed job after an exponential backoff.

ctx controls only the waiting, not the Redis calls: cancelling it skips the backoff so a stopping worker hands the job back quickly. If Redis is unreachable, Retry keeps trying every minute until ctx is done and then returns the last error, so a transient outage does not lose the job. Most callers should use Queue.StartWorker instead.

func (*Queue) StartWorker

func (q *Queue) StartWorker(ctx context.Context, handle HandlerFunc, opt *WorkerOptions)

StartWorker consumes the queue with handle until ctx is done, then waits for in-flight jobs and pending retries before returning. A nil opt uses the defaults. opt is copied, so it can be shared between workers.

func (*Queue) Status added in v0.2.0

func (q *Queue) Status(ctx context.Context) (*Status, error)

Status returns the counters of the queue.

type RedisClient

type RedisClient interface {
	Get(ctx context.Context, key string) *redis.StringCmd
	Set(ctx context.Context, key string, value any, expiration time.Duration) *redis.StatusCmd
	Del(ctx context.Context, keys ...string) *redis.IntCmd
	HIncrBy(ctx context.Context, key, field string, incr int64) *redis.IntCmd
	HGetAll(ctx context.Context, key string) *redis.MapStringStringCmd
	LPush(ctx context.Context, key string, values ...any) *redis.IntCmd
	RPop(ctx context.Context, key string) *redis.StringCmd
	LLen(ctx context.Context, key string) *redis.IntCmd
	Pipeline() redis.Pipeliner
}

RedisClient is the subset of go-redis used by this package, so any of the go-redis client types (Client, ClusterClient, Ring, UniversalClient) fits.

type ReportFunc added in v0.5.0

type ReportFunc func(status *Status)

ReportFunc receives the counters collected since the last report. See WorkerOptions.Idle and WorkerOptions.Reporter.

type Status added in v0.2.0

type Status struct {
	// IsRunning is false when no counter exists, i.e. nothing happened since
	// the last reset.
	IsRunning bool
	Process   int
	Success   int
	Failed    int
	Dropped   int
	// Total is Success + Dropped: jobs that reached a final state.
	Total int
}

Status holds the counters collected since the last reset.

type WorkerOptions

type WorkerOptions struct {
	// MaxRetry is how many times a failed job is retried. Default: 3.
	MaxRetry int
	// Parallel is the number of jobs handled concurrently. Default: 2.
	Parallel int
	// Interval is how long to wait when the queue is empty. Default: 3s.
	Interval time.Duration
	// Idle is how long the queue must be inactive before Reporter is called
	// and the counters are reset. Default: 3min.
	Idle time.Duration
	// Reporter, when set, receives the counters after Idle of inactivity.
	Reporter ReportFunc
	// Recover is how long to wait after a Redis error. Default: 1min.
	Recover time.Duration
	// SafeDrop keeps jobs that exhausted their retries in the dropped list
	// instead of discarding them. Default: false, so the list can not grow forever.
	SafeDrop bool
	// WG, when set, is incremented when StartWorker starts and decremented
	// when it returns. Since the Add happens inside StartWorker, do not Wait
	// on it before StartWorker has begun; wg.Go(func() { q.StartWorker(...) })
	// is the simpler alternative.
	WG *sync.WaitGroup
}

WorkerOptions tunes a worker. The zero value uses the documented defaults.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL