queue

package
v0.11.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 13, 2026 License: MIT Imports: 7 Imported by: 0

Documentation

Overview

Package queue provides a durable, distributed work queue for Chronos graph runs. It decouples run intake from execution: producers Enqueue runs and a pool of Workers claims them with a time-bounded lease, executes the graph, heartbeats to hold the lease, and re-enqueues in-flight runs abandoned by a dead worker once the lease expires.

The queue is backed by a small Store interface (implemented by SQLStore for both Postgres and SQLite) so the durable state — the work queue, leases, durable timers, external signals, the idempotent outbox, and admission bookkeeping — survives process restarts and is shared across workers.

Capabilities:

  • Leased dequeue (Postgres FOR UPDATE SKIP LOCKED; SQLite atomic UPDATE).
  • Heartbeat, lease expiry, and orphan recovery.
  • Idempotency keys and a reliable outbox for external effects.
  • Durable timers/sleeps and external signals, including webhook-as-signal for human-in-the-loop approval.
  • Global admission control / back-pressure with reject or park policies.

Index

Constants

View Source
const (
	StatusPending   = "pending"   // available (subject to AvailableAt) to be claimed
	StatusLeased    = "leased"    // claimed by a worker under an active lease
	StatusParked    = "parked"    // waiting on an external signal (HITL/back-pressure)
	StatusCompleted = "completed" // finished successfully
	StatusFailed    = "failed"    // failed terminally (attempts exhausted)
)

Run states.

View Source
const (
	KindStart  = "start"  // begin a fresh graph execution from the payload
	KindResume = "resume" // resume an existing session from its latest checkpoint
)

Run kinds tell the executor how to start work for the run.

View Source
const (
	OutboxPending = "pending"
	OutboxSent    = "sent"
	OutboxFailed  = "failed"
)

Outbox entry states.

View Source
const AdmissionParkSignal = "__admission__"

AdmissionParkSignal is the reserved signal name used to park runs shed by admission control; ReleaseAdmissionParked promotes them back to pending.

View Source
const DefaultOutboxMaxAttempts = 10

DefaultOutboxMaxAttempts is the number of delivery attempts after which a permanently-failing outbox entry is dead-lettered (marked OutboxFailed) so it stops being claimed and can no longer starve newer effects.

Variables

View Source
var (
	// ErrEmpty is returned by Dequeue when no run is currently available.
	ErrEmpty = errors.New("queue: no available run")
	// ErrOverloaded is returned by Enqueue when admission control rejects the
	// run because the queue is at capacity.
	ErrOverloaded = errors.New("queue: overloaded, run rejected")
	// ErrLeaseLost indicates a heartbeat/complete failed because the caller no
	// longer holds the lease (it expired and was recovered by another worker).
	ErrLeaseLost = errors.New("queue: lease lost")
	// ErrNotFound is returned when a run does not exist.
	ErrNotFound = errors.New("queue: run not found")
)

Errors returned by the queue.

Functions

This section is empty.

Types

type AdmissionPolicy

type AdmissionPolicy string

AdmissionPolicy controls what happens when the queue is at capacity.

const (
	// PolicyReject fails Enqueue with ErrOverloaded past MaxDepth.
	PolicyReject AdmissionPolicy = "reject"
	// PolicyPark parks runs past MaxDepth so they can be released later.
	PolicyPark AdmissionPolicy = "park"
)

type Config

type Config struct {
	// MaxDepth bounds the number of pending runs. Zero disables admission
	// control (unbounded intake).
	MaxDepth int
	// Policy selects reject vs. park behavior at capacity.
	Policy AdmissionPolicy
	// DefaultMaxAttempts is applied to enqueued runs that do not set one.
	DefaultMaxAttempts int
}

Config configures a Queue.

type Dialect

type Dialect string

Dialect selects SQL syntax for the target database.

const (
	// DialectSQLite targets SQLite (dev/test). Dequeue uses an atomic UPDATE over
	// a LIMIT-1 subquery; SQLite serializes writers so no SKIP LOCKED is needed.
	DialectSQLite Dialect = "sqlite"
	// DialectPostgres targets PostgreSQL (production). Dequeue uses
	// FOR UPDATE SKIP LOCKED so many workers claim disjoint runs concurrently.
	DialectPostgres Dialect = "postgres"
)

type Dispatcher

type Dispatcher interface {
	Dispatch(ctx context.Context, e *OutboxEntry) error
}

Dispatcher delivers an outbox entry to its external destination. It must be safe to call more than once for the same entry (delivery may be retried after a crash between send and the sent-mark); use the entry's IdempotencyKey to deduplicate downstream if the destination is not naturally idempotent.

type DispatcherFunc

type DispatcherFunc func(ctx context.Context, e *OutboxEntry) error

DispatcherFunc adapts a function to the Dispatcher interface.

func (DispatcherFunc) Dispatch

func (f DispatcherFunc) Dispatch(ctx context.Context, e *OutboxEntry) error

Dispatch calls f.

type Executor

type Executor func(ctx context.Context, r *Run) Result

Executor performs the work for a claimed run. Implementations must respect ctx cancellation: the worker cancels ctx if the lease is lost mid-execution.

type Outbox

type Outbox struct {
	// contains filtered or unexported fields
}

Outbox records external effects transactionally with run progress and drains them reliably. Recording an effect twice with the same idempotency key is a no-op, so retries and resumes never double-emit.

func NewOutbox

func NewOutbox(store Store) *Outbox

NewOutbox constructs an Outbox over a Store with the default dead-letter cap.

func (*Outbox) Drain

func (o *Outbox) Drain(ctx context.Context, d Dispatcher, interval time.Duration, batch int) error

Drain runs DrainOnce on a ticker until ctx is canceled.

func (*Outbox) DrainOnce

func (o *Outbox) DrainOnce(ctx context.Context, d Dispatcher, limit int) (int, error)

DrainOnce dispatches up to limit pending entries. It returns the number of entries successfully delivered. Failed dispatches remain pending for retry.

func (*Outbox) Record

func (o *Outbox) Record(ctx context.Context, sessionID, idempotencyKey, topic string, payload []byte) error

Record persists an external effect for later delivery. The idempotencyKey must be stable for a given logical effect so retries collapse to one row.

func (*Outbox) WithMaxAttempts

func (o *Outbox) WithMaxAttempts(n int) *Outbox

WithMaxAttempts sets the delivery-attempt cap after which an entry is dead-lettered. A value <= 0 restores the default. It returns the Outbox for chaining.

type OutboxEntry

type OutboxEntry struct {
	ID             string     `json:"id"`
	SessionID      string     `json:"session_id"`
	IdempotencyKey string     `json:"idempotency_key"`
	Topic          string     `json:"topic"`
	Payload        []byte     `json:"payload,omitempty"`
	Status         string     `json:"status"` // pending, sent, failed
	Attempts       int        `json:"attempts"`
	LastError      string     `json:"last_error,omitempty"`
	CreatedAt      time.Time  `json:"created_at"`
	SentAt         *time.Time `json:"sent_at,omitempty"`
}

OutboxEntry is an external effect recorded transactionally so it can be delivered exactly once by the drainer, even across retries and resumes. The IdempotencyKey is unique: enqueuing the same effect twice is a no-op.

type Queue

type Queue struct {
	// contains filtered or unexported fields
}

Queue is the high-level producer/administration API over a Store. It layers admission control on top of the durable Store operations. Workers consume via the same Queue.

func New

func New(store Store, cfg Config) *Queue

New constructs a Queue over the given Store.

func (*Queue) Complete

func (q *Queue) Complete(ctx context.Context, runID, owner, status, lastErr string) error

Complete marks a leased run finished with the given terminal status.

func (*Queue) Dequeue

func (q *Queue) Dequeue(ctx context.Context, owner string, lease time.Duration) (*Run, error)

Dequeue claims the next available run under a lease held by owner. It returns ErrEmpty when nothing is available.

func (*Queue) Enqueue

func (q *Queue) Enqueue(ctx context.Context, r *Run) error

Enqueue admits and persists a run. Admission control may reject (ErrOverloaded) or park the run under overload depending on the configured policy. The run's ID, timestamps, and default MaxAttempts are filled in when unset.

func (*Queue) Get

func (q *Queue) Get(ctx context.Context, id string) (*Run, error)

Get returns a run by ID.

func (*Queue) Heartbeat

func (q *Queue) Heartbeat(ctx context.Context, runID, owner string, lease time.Duration) error

Heartbeat extends the lease on an in-flight run. Returns ErrLeaseLost if the caller no longer owns the run.

func (*Queue) Migrate

func (q *Queue) Migrate(ctx context.Context) error

Migrate creates the queue's schema.

func (*Queue) Park

func (q *Queue) Park(ctx context.Context, runID, owner, waitSignal string, patch []byte) error

Park suspends a leased run until an external signal named waitSignal is delivered for its session. If a matching signal was already delivered, the run is made immediately available instead (no lost-signal race).

func (*Queue) PendingDepth

func (q *Queue) PendingDepth(ctx context.Context) (int, error)

PendingDepth reports the number of pending runs (for observability/back-pressure).

func (*Queue) RecoverOrphans

func (q *Queue) RecoverOrphans(ctx context.Context) (int, error)

RecoverOrphans re-enqueues leased runs whose lease has expired (their worker died). Retry-exhausted runs are failed. It returns the number of runs recovered.

func (*Queue) ReleaseAdmissionParked

func (q *Queue) ReleaseAdmissionParked(ctx context.Context, limit int) (int, error)

ReleaseAdmissionParked promotes up to limit admission-parked runs back to pending, but only while there is spare capacity under MaxDepth. It returns the number of runs released.

func (*Queue) Retry

func (q *Queue) Retry(ctx context.Context, runID, owner string, delay time.Duration, patch []byte) error

Retry durably reschedules a leased run to become available after delay AND increments its attempts counter (the retry budget). It is the post-error backoff path; callers must decide separately when attempts are exhausted.

func (*Queue) Signal

func (q *Queue) Signal(ctx context.Context, sig *Signal) (int, error)

Signal delivers an external signal, waking any parked run of the session that is waiting on it. If no run is waiting yet, the signal is retained so a subsequent Park observes it. It returns the number of runs awakened. This is the entry point a webhook handler calls to resume a HITL run.

func (*Queue) Sleep

func (q *Queue) Sleep(ctx context.Context, runID, owner string, delay time.Duration, patch []byte) error

Sleep durably reschedules a leased run to become available after delay. It is the primitive behind "wait N, then continue" that survives process restarts. Sleep does NOT consume retry budget (attempts is unchanged): durable timers and HITL parks are healthy redeliveries, not failures.

func (*Queue) Store

func (q *Queue) Store() Store

Store exposes the underlying durable store (for advanced callers, e.g. the outbox drainer or orphan recovery loop).

type Reaper

type Reaper struct {
	// contains filtered or unexported fields
}

Reaper periodically recovers orphaned runs (whose worker died) by re-enqueuing expired leases. Run it on one or more nodes; recovery is idempotent.

func NewReaper

func NewReaper(q *Queue, interval time.Duration) *Reaper

NewReaper constructs a Reaper that scans every interval.

func (*Reaper) Run

func (rp *Reaper) Run(ctx context.Context) error

Run scans for orphaned runs until ctx is canceled.

type Result

type Result struct {
	// Err, if non-nil, marks the attempt failed; the worker retries (with
	// backoff) until MaxAttempts, then fails the run terminally.
	Err error
	// Sleep, if > 0, durably reschedules the run to run again after the delay.
	Sleep time.Duration
	// ParkSignal, if set, parks the run until that signal is delivered.
	ParkSignal string
	// Patch, if non-nil, replaces the run's persisted payload on sleep/park so
	// the resumed attempt observes updated state.
	Patch []byte
}

Result is what an Executor reports after handling a run. Exactly one outcome applies, checked in this order: Err/failed → retry or fail; Sleep>0 → durable timer; ParkSignal set → wait for external signal; otherwise → completed.

type Run

type Run struct {
	ID             string     `json:"id"`
	SessionID      string     `json:"session_id"`
	GraphID        string     `json:"graph_id"`
	Kind           string     `json:"kind"`
	Payload        []byte     `json:"payload,omitempty"`
	Status         string     `json:"status"`
	Priority       int        `json:"priority"`
	Attempts       int        `json:"attempts"`
	MaxAttempts    int        `json:"max_attempts"`
	LeaseOwner     string     `json:"lease_owner,omitempty"`
	LeaseExpiresAt *time.Time `json:"lease_expires_at,omitempty"`
	AvailableAt    time.Time  `json:"available_at"`
	WaitSignal     string     `json:"wait_signal,omitempty"`
	SignalPayload  []byte     `json:"signal_payload,omitempty"`
	IdempotencyKey string     `json:"idempotency_key,omitempty"`
	LastError      string     `json:"last_error,omitempty"`
	CreatedAt      time.Time  `json:"created_at"`
	UpdatedAt      time.Time  `json:"updated_at"`
}

Run is a unit of durable work: one graph execution or resume.

type SQLStore

type SQLStore struct {
	// contains filtered or unexported fields
}

SQLStore implements Store over any database/sql database. It supports SQLite and PostgreSQL via the Dialect. It defines and owns the queue's own schema and never touches the shared storage package's tables.

func NewSQLStore

func NewSQLStore(db *sql.DB, dialect Dialect) *SQLStore

NewSQLStore constructs a store over an already-open *sql.DB. The caller owns the DB lifecycle unless Close is used.

func (*SQLStore) ClaimOutbox

func (s *SQLStore) ClaimOutbox(ctx context.Context, limit int) ([]*OutboxEntry, error)

ClaimOutbox returns pending outbox entries.

func (*SQLStore) Close

func (s *SQLStore) Close() error

Close closes the underlying database.

func (*SQLStore) CompleteRun

func (s *SQLStore) CompleteRun(ctx context.Context, runID, owner, status, lastErr string, now time.Time) error

CompleteRun sets a terminal status.

func (*SQLStore) CountByStatus

func (s *SQLStore) CountByStatus(ctx context.Context, status string) (int, error)

CountByStatus counts runs in a status.

func (*SQLStore) DeliverSignal

func (s *SQLStore) DeliverSignal(ctx context.Context, sig *Signal, now time.Time) (int, error)

DeliverSignal wakes parked runs waiting on sig.Name; retains the signal if none.

func (*SQLStore) DequeueRun

func (s *SQLStore) DequeueRun(ctx context.Context, owner string, lease time.Duration, now time.Time) (*Run, error)

DequeueRun atomically claims the next available run.

func (*SQLStore) EnqueueOutbox

func (s *SQLStore) EnqueueOutbox(ctx context.Context, e *OutboxEntry) error

EnqueueOutbox records an external effect, ignoring duplicate idempotency keys.

func (*SQLStore) EnqueueRun

func (s *SQLStore) EnqueueRun(ctx context.Context, r *Run) error

EnqueueRun inserts a run.

func (*SQLStore) GetRun

func (s *SQLStore) GetRun(ctx context.Context, id string) (*Run, error)

GetRun returns a run by ID.

func (*SQLStore) Heartbeat

func (s *SQLStore) Heartbeat(ctx context.Context, runID, owner string, lease time.Duration, now time.Time) error

Heartbeat extends the lease.

func (*SQLStore) MarkIdempotent

func (s *SQLStore) MarkIdempotent(ctx context.Context, key string, now time.Time) (bool, error)

MarkIdempotent records key once.

func (*SQLStore) MarkOutboxFailed

func (s *SQLStore) MarkOutboxFailed(ctx context.Context, id, errMsg string, maxAttempts int, now time.Time) error

MarkOutboxFailed records a delivery failure and increments attempts. When the incremented attempts reach maxAttempts (> 0) the entry is dead-lettered (status OutboxFailed) so ClaimOutbox stops returning it and a poison entry can no longer starve newer effects; otherwise it stays pending for retry.

func (*SQLStore) MarkOutboxSent

func (s *SQLStore) MarkOutboxSent(ctx context.Context, id string, now time.Time) error

MarkOutboxSent marks an entry delivered.

func (*SQLStore) Migrate

func (s *SQLStore) Migrate(ctx context.Context) error

Migrate creates the queue schema.

func (*SQLStore) ParkRun

func (s *SQLStore) ParkRun(ctx context.Context, runID, owner, waitSignal string, patch []byte, now time.Time) error

ParkRun suspends a leased run pending a signal, honoring an already-delivered signal to avoid a lost-signal race.

func (*SQLStore) RecoverOrphans

func (s *SQLStore) RecoverOrphans(ctx context.Context, now time.Time) (int, error)

RecoverOrphans re-enqueues expired-lease runs and fails exhausted ones.

func (*SQLStore) ReleaseParked

func (s *SQLStore) ReleaseParked(ctx context.Context, waitSignal string, limit int, now time.Time) (int, error)

ReleaseParked promotes parked runs back to pending.

func (*SQLStore) RescheduleRun

func (s *SQLStore) RescheduleRun(ctx context.Context, runID, owner string, availableAt time.Time, patch []byte, now time.Time) error

RescheduleRun returns a run to pending, available at availableAt.

func (*SQLStore) RetryRun

func (s *SQLStore) RetryRun(ctx context.Context, runID, owner string, availableAt time.Time, patch []byte, now time.Time) error

RetryRun returns a run to pending like RescheduleRun but also increments the attempts counter, consuming retry budget. It is the post-error retry path.

type Signal

type Signal struct {
	SessionID string `json:"session_id"`
	Name      string `json:"name"`
	Payload   []byte `json:"payload,omitempty"`
}

Signal is an external event delivered to parked runs of a session.

type Store

type Store interface {
	// Migrate creates the queue schema (idempotent).
	Migrate(ctx context.Context) error

	// EnqueueRun inserts a run.
	EnqueueRun(ctx context.Context, r *Run) error

	// DequeueRun atomically claims the highest-priority available run for owner,
	// setting a lease that expires at now+lease. Returns ErrEmpty if none.
	DequeueRun(ctx context.Context, owner string, lease time.Duration, now time.Time) (*Run, error)

	// Heartbeat extends the lease on a run still owned by owner. Returns
	// ErrLeaseLost if the run is no longer leased to owner.
	Heartbeat(ctx context.Context, runID, owner string, lease time.Duration, now time.Time) error

	// CompleteRun sets a leased run to a terminal status. Returns ErrLeaseLost if
	// owner no longer holds the lease.
	CompleteRun(ctx context.Context, runID, owner, status, lastErr string, now time.Time) error

	// RescheduleRun returns a leased run to pending, available at availableAt,
	// optionally replacing its payload (durable sleep). It does NOT touch the
	// attempts counter: durable timers and parks are normal redeliveries, not
	// failed attempts, and must not burn the retry budget.
	RescheduleRun(ctx context.Context, runID, owner string, availableAt time.Time, patch []byte, now time.Time) error

	// RetryRun returns a leased run to pending, available at availableAt, AND
	// increments its attempts counter. Unlike RescheduleRun this consumes retry
	// budget: it is the path taken after an execution error.
	RetryRun(ctx context.Context, runID, owner string, availableAt time.Time, patch []byte, now time.Time) error

	// ParkRun suspends a leased run pending a signal named waitSignal. If a
	// matching signal was already delivered for the session, the run is made
	// available immediately instead of parked (no lost-signal race).
	ParkRun(ctx context.Context, runID, owner, waitSignal string, patch []byte, now time.Time) error

	// DeliverSignal wakes parked runs of the session waiting on sig.Name and
	// returns the count awakened. If none are waiting, the signal is retained.
	DeliverSignal(ctx context.Context, sig *Signal, now time.Time) (int, error)

	// ReleaseParked promotes up to limit runs parked on waitSignal to pending.
	ReleaseParked(ctx context.Context, waitSignal string, limit int, now time.Time) (int, error)

	// RecoverOrphans re-enqueues runs whose lease expired (worker died); runs
	// past MaxAttempts are failed. Returns the number of runs recovered.
	RecoverOrphans(ctx context.Context, now time.Time) (int, error)

	// CountByStatus returns the number of runs in the given status.
	CountByStatus(ctx context.Context, status string) (int, error)

	// GetRun returns a run by ID (ErrNotFound if absent).
	GetRun(ctx context.Context, id string) (*Run, error)

	// MarkIdempotent records key; fresh is true only for the first observation.
	MarkIdempotent(ctx context.Context, key string, now time.Time) (fresh bool, err error)

	// EnqueueOutbox records an external effect. Duplicate IdempotencyKeys are
	// ignored (returns nil without inserting).
	EnqueueOutbox(ctx context.Context, e *OutboxEntry) error

	// ClaimOutbox returns up to limit pending outbox entries.
	ClaimOutbox(ctx context.Context, limit int) ([]*OutboxEntry, error)

	// MarkOutboxSent marks an entry delivered.
	MarkOutboxSent(ctx context.Context, id string, now time.Time) error

	// MarkOutboxFailed records a delivery failure, incrementing attempts. When
	// the incremented attempts reach maxAttempts the entry is dead-lettered
	// (status OutboxFailed) so it is no longer claimed; otherwise it stays
	// pending for retry. A maxAttempts <= 0 disables dead-lettering.
	MarkOutboxFailed(ctx context.Context, id, errMsg string, maxAttempts int, now time.Time) error

	// Close releases resources owned by the store.
	Close() error
}

Store is the minimal durable persistence contract the queue depends on. It is defined here (rather than in the shared storage package) so the queue owns its own schema and can be backed by any *sql.DB via SQLStore. All timestamps are passed explicitly so behavior is deterministic and testable.

type Worker

type Worker struct {
	// contains filtered or unexported fields
}

Worker claims runs from a Queue and executes them, holding the lease with periodic heartbeats and durably rescheduling/parking/failing per the Result.

func NewWorker

func NewWorker(q *Queue, exec Executor, cfg WorkerConfig) (*Worker, error)

NewWorker constructs a Worker. It panics only on programmer error (nil queue or executor); all runtime failures are returned as errors.

func (*Worker) Run

func (w *Worker) Run(ctx context.Context) error

Run drives the claim/execute loop until ctx is canceled.

func (*Worker) RunOnce

func (w *Worker) RunOnce(ctx context.Context) (bool, error)

RunOnce claims and processes at most one run. It returns processed=false when the queue is empty.

type WorkerConfig

type WorkerConfig struct {
	// ID uniquely identifies this worker; it owns the leases it claims.
	ID string
	// Lease is the lease TTL granted on claim and extended by each heartbeat.
	Lease time.Duration
	// Heartbeat is how often the worker extends the lease while executing.
	Heartbeat time.Duration
	// PollInterval is how long to wait after an empty dequeue before retrying.
	PollInterval time.Duration
	// Backoff computes the retry delay from the attempt count. If nil, a bounded
	// exponential backoff is used.
	Backoff func(attempt int) time.Duration
}

WorkerConfig configures a Worker.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL