queue

package
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 9, 2026 License: GPL-3.0 Imports: 17 Imported by: 0

Documentation

Overview

Package queue runs the server's tasks: one database row per piece of work (match a work, check it for updates, download a chapter). Tasks are claimed with SKIP LOCKED, hold a lease while they run, retry with back-off, and are capped per kind and per limit key (a source), across all users' work. Identical live tasks are stored once (dedupe keys).

Index

Constants

View Source
const (
	StateQueued    = "queued"
	StateRunning   = "running"
	StateSucceeded = "succeeded"
	StateFailed    = "failed"
	StateCancelled = "cancelled"
)

Task states.

Variables

View Source
var (
	ErrNotFound    = errors.New("queue: task not found")
	ErrNotRetrying = errors.New("queue: only failed or cancelled tasks can be retried")
	ErrFinished    = errors.New("queue: the task has finished")
	ErrUnknownKind = errors.New("queue: unknown task kind")
)

Errors callers map to responses.

Functions

func DefaultBackoff

func DefaultBackoff(attempts int32) time.Duration

DefaultBackoff waits 30s, 1m, 2m, … up to an hour, with jitter.

func Permanent

func Permanent(err error) error

Permanent marks err as not worth retrying (bad input, gone for good).

Types

type Event

type Event struct {
	ID        uuid.UUID `json:"id"`
	Kind      string    `json:"kind"`
	State     string    `json:"state"`
	Attempts  int32     `json:"attempts"`
	LastError string    `json:"last_error,omitempty"`
	Progress  *Progress `json:"progress,omitempty"`
}

Event is the data of task events.

type Handler

type Handler func(ctx context.Context, t *Task) (result any, err error)

Handler does a task's work and returns a result to keep (or nil).

type Kind

type Kind struct {
	Handler Handler
	// Concurrency caps how many of this kind run at once. Default 2.
	Concurrency int
	// MaxAttempts before a task fails. Default 5.
	MaxAttempts int
}

Kind configures one kind of task.

type KindState

type KindState struct {
	Kind        string
	Paused      bool
	Running     int
	Concurrency int
}

KindState is a kind's live state.

type NewTask

type NewTask struct {
	Kind    string
	Payload any
	// DedupeKey keeps one live (queued or running) task per key.
	DedupeKey string
	// LimitKey caps concurrent tasks per key, e.g. "source:<id>".
	LimitKey  string
	Priority  int32
	CreatedBy uuid.UUID
	// RunAfter delays the task.
	RunAfter time.Time
}

NewTask is a task to enqueue.

type Progress

type Progress struct {
	Phase string `json:"phase,omitempty"`
	Done  int64  `json:"done"`
	Total int64  `json:"total,omitempty"`
	Unit  string `json:"unit,omitempty"`
}

Progress is how far a running task is.

type Queue

type Queue struct {

	// Workers caps all running tasks. Default 8.
	Workers int
	// Lease is how long a claimed task stays claimed without a heartbeat.
	Lease time.Duration
	// Poll is how often the queue looks for due tasks without a wake-up.
	Poll time.Duration
	// Backoff gives the delay before attempt n+1 after n failed attempts.
	Backoff func(attempts int32) time.Duration
	// contains filtered or unexported fields
}

Queue runs tasks.

func New

func New(st *store.Store, hub *events.Hub, lim *limits.Registry, m *metrics.Metrics, log *slog.Logger) *Queue

New returns a queue. hub, lim and m may be nil.

func (*Queue) Cancel

func (q *Queue) Cancel(ctx context.Context, id uuid.UUID) (db.Task, error)

Cancel stops a queued or running task.

func (*Queue) Enqueue

func (q *Queue) Enqueue(ctx context.Context, n NewTask) (db.Task, bool, error)

Enqueue adds a task, or returns the live task with the same dedupe key (created is then false).

func (*Queue) EnqueueWith

func (q *Queue) EnqueueWith(ctx context.Context, dbq *db.Queries, n NewTask) (db.Task, bool, error)

EnqueueWith enqueues with dbq, so the task can join a transaction.

func (*Queue) Get

func (q *Queue) Get(ctx context.Context, id uuid.UUID) (db.Task, error)

Get returns a task.

func (*Queue) Kinds

func (q *Queue) Kinds() []string

Kinds lists the registered kinds.

func (*Queue) Register

func (q *Queue) Register(kind string, k Kind)

Register adds a task kind. Call before Run.

func (*Queue) Retry

func (q *Queue) Retry(ctx context.Context, id uuid.UUID) (db.Task, error)

Retry queues a failed or cancelled task again.

func (*Queue) Run

func (q *Queue) Run(ctx context.Context) error

Run claims and runs tasks until ctx ends, then waits for running tasks to stop (they see ctx cancelled and go back to the queue).

func (*Queue) SetPaused

func (q *Queue) SetPaused(kind string, paused bool) error

SetPaused pauses or resumes a kind: queued tasks of a paused kind wait; running ones finish.

func (*Queue) State

func (q *Queue) State() []KindState

State returns each kind's live state.

type RetryAfterError

type RetryAfterError struct {
	Err   error
	After time.Duration
}

RetryAfterError asks for the next attempt no sooner than After (e.g. a site rate-limited the plugin).

func (*RetryAfterError) Error

func (e *RetryAfterError) Error() string

func (*RetryAfterError) Unwrap

func (e *RetryAfterError) Unwrap() error

type Task

type Task struct {
	db.Task
	// contains filtered or unexported fields
}

Task is a running task, handed to its handler.

func (*Task) Decode

func (t *Task) Decode(v any) error

Decode reads the task's payload into v.

func (*Task) SetProgress

func (t *Task) SetProgress(p Progress)

SetProgress reports how far the task is. Writes and events are throttled.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL