jobqueue

package module
v1.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 1, 2026 License: BSD-3-Clause Imports: 11 Imported by: 0

README

job-queue

A small job queue for Go that keeps its jobs in your SQL database, through GORM. Runs on MySQL 8, MariaDB 10.5, Postgres and SQLite, and on several instances at once: each job runs on one worker, retries are counted, and jobs can be scheduled for later.

go get github.com/LibertusDio/job-queue

Create the tables with the migration for your database: MySQL/MariaDB, Postgres, SQLite.

Quick start

package main

import (
	"context"
	"os"
	"os/signal"

	jobqueue "github.com/LibertusDio/job-queue"
	"github.com/LibertusDio/job-queue/middlewares"
	gormstore "github.com/LibertusDio/job-queue/store/gorm"
	"github.com/LibertusDio/job-queue/utils"

	"gorm.io/driver/mysql"
	"gorm.io/gorm"
)

func main() {
	logger := utils.DumpLogger{}
	db, err := gorm.Open(mysql.Open("user:pass@tcp(127.0.0.1:3306)/app?parseTime=true"), &gorm.Config{})
	if err != nil {
		panic(err)
	}

	// "db" is the context key the store reads the *gorm.DB from.
	store, err := gormstore.NewGormStore("jobs", "schedule_jobs", "db", db, logger)
	if err != nil {
		panic(err)
	}

	queue := jobqueue.NewForeman(&jobqueue.QueueConfig{
		JobDescription: map[string]jobqueue.JobDescription{
			"send-email": {Title: "send-email", TTL: 30, Concurrent: 5, Priority: 1, MaxRetry: 3},
		},
		Concurrent: 10,   // workers in total
		BreakTime:  1000, // max ms between polls when idle
		RampTime:   100,  // ms between polls while busy
	}, store, logger)

	// Runs each handler in a transaction, passed to it under "db".
	queue.AddMiddleware(middlewares.GormTransactionMiddleware("db", db))

	queue.AddWorker("send-email", func(ctx context.Context) error {
		job := ctx.Value(jobqueue.ContextJobKey).(*jobqueue.Job)
		if err := sendEmail(job.Payload); err != nil {
			job.Status = jobqueue.JobStatus.RETRY
			job.Result = err.Error()
			return nil
		}
		job.Status = jobqueue.JobStatus.DONE
		return nil
	})

	go queue.Serve() // blocks until Strike

	queue.AddJob(&jobqueue.Job{Title: "send-email", JobID: "welcome-42", Payload: `{"to":"a@example.com"}`})

	stop := make(chan os.Signal, 1)
	signal.Notify(stop, os.Interrupt)
	<-stop
	queue.Strike(30) // stop taking jobs, wait 30 s, cancel the rest
}

func sendEmail(payload string) error { return nil }

Runnable examples: test/gorm (MySQL), test/gorm-postgres, test/gorm-sqlite.

Writing a worker

The handler gets the job from ctx.Value(jobqueue.ContextJobKey) and sets its outcome on it:

In the handler Result
job.Status = JobStatus.DONE Finished.
job.Status = JobStatus.RETRY Runs again, up to MaxRetry times, then max_retry.
job.Status = JobStatus.ERROR, return an error, or panic Failed, no retry. The error goes into job.Result.
Still running after TTL seconds Cancelled and treated as RETRY.

Always set a status. A job left processing is picked up again after its TTL. Each run increments job.Try and doubles job.Priority (1 runs first), so retries queue behind fresh jobs. Keep job.Message short (100 characters); put details in job.Result.

Adding jobs

A job is identified by Title + JobID. Adding one while the same pair is init, retry or processing returns JobError.DUPLICATE_JOB; once it has finished, the pair can be added again.

err := queue.AddJob(&jobqueue.Job{Title: "send-email", JobID: "welcome-42"})

// Run later: runat is unix seconds.
err = queue.AddJobWithSchedule(context.WithValue(ctx, "db", db),
	&jobqueue.Job{Title: "reminder", JobID: order.ID}, time.Now().Add(time.Hour).Unix())
In your own transaction

AddJobTransaction creates the job inside your transaction, so it exists only if your writes commit. If it fails, return the error and roll back. Never log it and carry on: on MySQL/MariaDB a deadlock has already rolled back your earlier writes, and committing would keep only the writes made after it. Deadlocks are normal under load, so retry the whole transaction when jobqueue.IsRetryable(err):

for attempt := 1; ; attempt++ {
	err := db.Transaction(func(tx *gorm.DB) error {
		if err := tx.Create(&order).Error; err != nil {
			return err
		}
		return queue.AddJobTransaction(context.WithValue(ctx, "db", tx),
			&jobqueue.Job{Title: "invoice", JobID: order.ID})
	})
	if err == nil || !jobqueue.IsRetryable(err) || attempt == 5 {
		return err
	}
	time.Sleep(time.Duration(attempt) * 20 * time.Millisecond)
}

DUPLICATE_JOB is the exception: it leaves your transaction intact.

Recurring jobs

A handler can schedule its own next run. Only one pending schedule per Title + JobID is allowed (DUPLICATE_JOB otherwise), and the run in progress doesn't count:

queue.AddWorker("nightly-sync", func(ctx context.Context) error {
	job := ctx.Value(jobqueue.ContextJobKey).(*jobqueue.Job)
	// ... do the work ...
	next := time.Now().Add(24 * time.Hour).Unix()
	err := queue.AddJobWithSchedule(ctx, &jobqueue.Job{Title: job.Title, JobID: job.JobID}, next)
	if err != nil && err != jobqueue.JobError.DUPLICATE_JOB { // already scheduled is fine
		return err
	}
	job.Status = jobqueue.JobStatus.DONE
	return nil
})

The scheduler checks for due schedules every ScheduleInterval ms (59 s by default). If a schedule comes due while the same job is still running, it waits and fires after that run ends.

Configuration

QueueConfig
JobDescription One entry per job title (below). Jobs with other titles are rejected.
Concurrent Workers in total.
BreakTime, RampTime ms between polls: RampTime while jobs are coming in, doubling up to BreakTime while idle.
ScheduleInterval ms between scheduler passes, default 59 s.
JobDescription
Title Same as the map key and AddWorker title.
TTL Seconds a run may take.
Concurrent Max running jobs with this title.
Priority Default priority, 1 is highest.
MaxRetry Retries allowed before max_retry.

CleanError and Secure exist but aren't used yet.

Errors and shutdown

Serve blocks until Strike, then returns JobError.TERMINATING. It never stops because of database errors: deadlocks are retried, and other errors are logged as fetch job error:... and retried with a backoff from 50 ms up to 5 s. Watch for that log line to spot a database outage.

Strike(ttl) stops taking new jobs, waits ttl seconds, then cancels the jobs still running.

Databases

  • MySQL/MariaDB: the job claim runs at READ COMMITTED, which avoids most deadlocks with concurrent inserts. With binlog_format=STATEMENT MySQL refuses that; the store logs a warning and uses the server default.
  • Postgres: no special setup.
  • SQLite: one writer at a time, so best for a single process. Use sqlDB.SetMaxOpenConns(1), as in test/gorm-sqlite.
Upgrading from v1.0

Add the new index:

CREATE INDEX schedule_jobs_title_job_id_IDX ON schedule_jobs (title, job_id);

Behavior changes: Serve no longer returns on database errors. AddJobWithSchedule now rejects only a second pending schedule, not a running job with the same id. A schedule due while its job runs now waits instead of being marked error.

Running the tests

SQLite always runs. MySQL, MariaDB and Postgres run when their DSN is set:

docker compose -f store/gorm/docker-compose.test.yml up -d
export JOBQUEUE_TEST_MYSQL_DSN="root:jobqueue@tcp(127.0.0.1:3307)/jobqueue?charset=utf8mb4&parseTime=True&loc=Local"
export JOBQUEUE_TEST_MARIADB_DSN="root:jobqueue@tcp(127.0.0.1:3308)/jobqueue?charset=utf8mb4&parseTime=True&loc=Local"
export JOBQUEUE_TEST_POSTGRES_DSN="host=127.0.0.1 port=5544 user=postgres password=jobqueue dbname=jobqueue sslmode=disable"
go test ./...
docker compose -f store/gorm/docker-compose.test.yml down -v

-short skips the stress test (750 jobs, about 10 s per database). The compose file turns off fsync for speed; don't copy its settings to real databases.

Documentation

Index

Constants

View Source
const ContextJobKey = "job-key"
View Source
const ContextQueueKey = "queue-key"
View Source
const MaxTime int64 = 9223372036854775807

Variables

View Source
var JobError = joberror{
	INVALID_JD:     errors.New("invalid job description"),
	INVALID_JOB:    errors.New("invalid job"),
	INVALID_WORKER: errors.New("invalid worker"),
	INVALID_PL:     errors.New("invalid production line"),
	NOT_FOUND:      errors.New("not found"),
	TERMINATING:    errors.New("terminating"),
	DUPLICATE_JOB:  errors.New("duplicate job"),
}
View Source
var JobStatus = jobstatus{
	INIT:       "init",
	PROCESSING: "processing",
	RETRY:      "retry",
	ERROR:      "error",
	MAX_RETRY:  "max_retry",
	DONE:       "done",
}
View Source
var StoreError = storeerror{
	INVALID_GORM_TX: errors.New("invalid gorm transaction"),
	INVALID_GORM_DB: errors.New("invalid gorm session"),
	LOCK_TIMEOUT:    errors.New("timed out waiting for job lock"),
}

Functions

func IsRetryable added in v1.1.0

func IsRetryable(err error) bool

IsRetryable reports whether err is a deadlock or lock wait timeout: MySQL/ MariaDB error 1213 or 1205, Postgres SQLSTATE 40P01 or 55P03, or the gorm store's StoreError.LOCK_TIMEOUT. These are an expected outcome of concurrent transactions touching the same rows and indexes (e.g. enqueueing while workers claim jobs), not a sign of a broken connection or a bad query, and running the transaction again normally succeeds.

It recognizes go-sql-driver/mysql's *MySQLError and any error with a SQLState() string method (pgx's *pgconn.PgError), wrapped or not, without importing either driver.

Retry the whole transaction, never just the failed statement: a MySQL deadlock has already rolled back everything the transaction did, and on Postgres any error aborts the transaction.

Types

type Foreman

type Foreman interface {
	//AddWorker add a worker function for handler
	AddWorker(title string, worker HandlerFunc, midl ...MiddlewareFunc) error

	//AddJobTransaction add a new job to the queue. ctx is handed to JobStorage.CreateJob; with
	//store/gorm it carries the caller's *gorm.DB, and when that is an open transaction the job
	//is created inside it, so it only exists if that transaction commits.
	//
	//If it fails inside the caller's transaction with anything but a JobError (such as
	//DUPLICATE_JOB, which leaves the transaction intact), treat the whole transaction as
	//failed: return the error, roll back, and retry the transaction from the start if
	//IsRetryable(err). Do not log it and carry on. On MySQL/MariaDB a deadlock has already
	//rolled back everything the transaction wrote before this call, yet the transaction stays
	//usable: later statements and the final Commit still succeed, persisting only the writes
	//made after the failure. On Postgres any error aborts the transaction.
	AddJobTransaction(ctx context.Context, job *Job) error

	//AddJob add a new job to the queue without context, dont recommend but handy for multiple context
	AddJob(job *Job) error

	//AddJobWithSchedule add a new job to the queue to run at a schedule (unix seconds).
	//It returns JobError.DUPLICATE_JOB if a schedule for the same title + job_id is still
	//pending. An active job with that job_id doesn't block it, so a recurring job can
	//schedule its own next run from its handler. If the schedule comes due while that
	//job_id is still active, it stays pending and fires on a later scheduler pass.
	AddJobWithSchedule(ctx context.Context, job *Job, runat int64) error

	//Serve start the worker service. It blocks until Strike is called and then returns
	//JobError.TERMINATING; store errors while fetching jobs are logged and retried with
	//a backoff, never returned.
	Serve() error

	//Strike graceful shutdown service. Reject new jobs and wait for running one to finish. Force kill at TTL.
	Strike(ttl int) error // ttl in seconds

	//AddMiddleware add global middleware, order of adding matters
	AddMiddleware(midl ...MiddlewareFunc)

	//GetCounter return worker counter for debug purpose
	GetCounter() int
}

func NewForeman

func NewForeman(config *QueueConfig, store JobStorage, logger Logger) Foreman

type Governor

type Governor interface {
	AddJob(string)
	DelJob(string)
	NoJob()
	Spawn() (bool, []string)
	GetCounter() int
}

func NewLocalOndemandGovernor

func NewLocalOndemandGovernor(max, min, maxworker int, jd map[string]JobDescription) Governor

OndemandGovernor ondemand governor

type HandlerFunc

type HandlerFunc func(ctx context.Context) error

type Job

type Job struct {
	ID        string `mapstructure:"id" yaml:"id" json:"id"`
	JobID     string `mapstructure:"job_id" yaml:"job_id" json:"job_id"`
	Title     string `mapstructure:"title" yaml:"title" json:"title"`
	Payload   string `mapstructure:"payload" yaml:"payload" json:"payload"`
	Try       int    `mapstructure:"try" yaml:"try" json:"try"`
	Priority  int    `mapstructure:"priority" yaml:"priority" json:"priority"`
	Status    string `mapstructure:"status" yaml:"status" json:"status"`
	Result    string `yaml:"result" mapstructure:"result" json:"result"`
	Message   string `yaml:"message" mapstructure:"message" json:"message"`
	UpdatedAt int64  `yaml:"updated_at" mapstructure:"updated_at" json:"updated_at"`
}

type JobDescription

type JobDescription struct {
	Title      string `mapstructure:"title" yaml:"title" json:"title"`
	TTL        int    `mapstructure:"ttl" yaml:"ttl" json:"ttl"`                      // seconds
	Concurrent int    `mapstructure:"concurrent" yaml:"concurrent" json:"concurrent"` // 0 is invalid
	Priority   int    `mapstructure:"priority" yaml:"priority" json:"priority"`       // 0 is invalid here, 1 is highest
	MaxRetry   int    `mapstructure:"max_retry" yaml:"max_retry" json:"max_retry"`    // number of retry
	Secure     bool   `mapstructure:"secure" yaml:"secure" json:"secure"`
}

Config models

type JobStorage

type JobStorage interface {
	CheckDuplicateJob(job Job) error
	CreateJob(ctx context.Context, job Job) error
	GetAndLockAvailableJob(jd map[string]JobDescription, ignorelist ...string) (*Job, error)
	UpdateJobResult(job Job) error

	InjectJob(Job) error
	// CreateScheduleJob stores a schedule, returning JobError.DUPLICATE_JOB if
	// one with the same title + job_id is already pending (status init).
	// Active jobs with that job_id must not block it.
	CreateScheduleJob(ctx context.Context, job ScheduleJob) error
	GetScheduledJob(from, to int64) ([]*ScheduleJob, error)
	UpdateScheduledJob(ScheduleJob) error
}

type LocalOndemandGovernor

type LocalOndemandGovernor struct {
	MaxSleep       int
	MinSleep       int
	CurSleep       *int
	WorkerCounter  *int
	Locker         *sync.Mutex
	JobCounter     map[string]*int
	MaxWorker      int
	JobDescription map[string]JobDescription
}

func (LocalOndemandGovernor) AddJob

func (g LocalOndemandGovernor) AddJob(title string)

func (LocalOndemandGovernor) DelJob

func (g LocalOndemandGovernor) DelJob(title string)

func (LocalOndemandGovernor) GetCounter

func (g LocalOndemandGovernor) GetCounter() int

func (LocalOndemandGovernor) NoJob

func (g LocalOndemandGovernor) NoJob()

func (LocalOndemandGovernor) Spawn

func (g LocalOndemandGovernor) Spawn() (bool, []string)

type Logger

type Logger interface {
	Debug(msg string)
	Info(msg string)
	Warn(msg string)
	Error(msg string)
	Fatal(msg string)
	Panic(msg string)
}

type MiddlewareFunc

type MiddlewareFunc func(HandlerFunc) HandlerFunc

type QueueConfig

type QueueConfig struct {
	JobDescription map[string]JobDescription `mapstructure:"job_description" yaml:"job_description" json:"job_description"`
	CleanError     bool                      `yaml:"clean_error" mapstructure:"clean_error" json:"clean_error"` // will the GC clear maxed try job
	Concurrent     int                       `yaml:"concurrent" mapstructure:"concurrent" json:"concurrent"`    // number of workers can be run, will ignore JD.Concurrent if it >
	BreakTime      int                       `mapstructure:"break_time" yaml:"break_time" json:"break_time"`    // miliseconds, break between loop
	RampTime       int                       `mapstructure:"ramp_time" yaml:"ramp_time" json:"ramp_time"`
	// ScheduleInterval is the time between scheduler passes that fire due
	// scheduled jobs, in milliseconds. 0 keeps the default of 59 seconds.
	ScheduleInterval int `mapstructure:"schedule_interval" yaml:"schedule_interval" json:"schedule_interval"`
}

type ScheduleJob

type ScheduleJob struct {
	ID        string `mapstructure:"id" yaml:"id" json:"id"`
	JobID     string `mapstructure:"job_id" yaml:"job_id" json:"job_id"`
	Title     string `mapstructure:"title" yaml:"title" json:"title"`
	Payload   string `mapstructure:"payload" yaml:"payload" json:"payload"`
	Priority  int    `mapstructure:"priority" yaml:"priority" json:"priority"`
	Status    string `mapstructure:"status" yaml:"status" json:"status"`
	UpdatedAt int64  `yaml:"updated_at" mapstructure:"updated_at" json:"updated_at"`
	Schedule  int64  `json:"schedule"`
	ExecuteID string `json:"execute_id"`
	Log       string `json:"log"`
}

Directories

Path Synopsis
internal
sqlerr
Package sqlerr reads server error codes out of database driver errors without importing the drivers, so depending on job-queue doesn't pull in (or register) a driver the application doesn't use.
Package sqlerr reads server error codes out of database driver errors without importing the drivers, so depending on job-queue doesn't pull in (or register) a driver the application doesn't use.
store
test
gorm command
gorm-postgres command
gorm-sqlite command
gorvernortest command
sleepstrike command

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL