Documentation
¶
Overview ¶
Package queue provides a durable, distributed work queue for Chronos graph runs. It decouples run intake from execution: producers Enqueue runs and a pool of Workers claims them with a time-bounded lease, executes the graph, heartbeats to hold the lease, and re-enqueues in-flight runs abandoned by a dead worker once the lease expires.
The queue is backed by a small Store interface (implemented by SQLStore for both Postgres and SQLite) so the durable state — the work queue, leases, durable timers, external signals, the idempotent outbox, and admission bookkeeping — survives process restarts and is shared across workers.
Capabilities:
- Leased dequeue (Postgres FOR UPDATE SKIP LOCKED; SQLite atomic UPDATE).
- Heartbeat, lease expiry, and orphan recovery (P1-001, P1-002).
- Idempotency keys and a reliable outbox for external effects (P1-003).
- Durable timers/sleeps and external signals, including webhook-as-signal for human-in-the-loop approval (P1-004).
- Global admission control / back-pressure with reject or park policies (P1-005).
Index ¶
- Constants
- Variables
- type AdmissionPolicy
- type Config
- type Dialect
- type Dispatcher
- type DispatcherFunc
- type Executor
- type Outbox
- func (o *Outbox) Drain(ctx context.Context, d Dispatcher, interval time.Duration, batch int) error
- func (o *Outbox) DrainOnce(ctx context.Context, d Dispatcher, limit int) (int, error)
- func (o *Outbox) Record(ctx context.Context, sessionID, idempotencyKey, topic string, payload []byte) error
- func (o *Outbox) WithMaxAttempts(n int) *Outbox
- type OutboxEntry
- type Queue
- func (q *Queue) Complete(ctx context.Context, runID, owner, status, lastErr string) error
- func (q *Queue) Dequeue(ctx context.Context, owner string, lease time.Duration) (*Run, error)
- func (q *Queue) Enqueue(ctx context.Context, r *Run) error
- func (q *Queue) Get(ctx context.Context, id string) (*Run, error)
- func (q *Queue) Heartbeat(ctx context.Context, runID, owner string, lease time.Duration) error
- func (q *Queue) Migrate(ctx context.Context) error
- func (q *Queue) Park(ctx context.Context, runID, owner, waitSignal string, patch []byte) error
- func (q *Queue) PendingDepth(ctx context.Context) (int, error)
- func (q *Queue) RecoverOrphans(ctx context.Context) (int, error)
- func (q *Queue) ReleaseAdmissionParked(ctx context.Context, limit int) (int, error)
- func (q *Queue) Retry(ctx context.Context, runID, owner string, delay time.Duration, patch []byte) error
- func (q *Queue) Signal(ctx context.Context, sig *Signal) (int, error)
- func (q *Queue) Sleep(ctx context.Context, runID, owner string, delay time.Duration, patch []byte) error
- func (q *Queue) Store() Store
- type Reaper
- type Result
- type Run
- type SQLStore
- func (s *SQLStore) ClaimOutbox(ctx context.Context, limit int) ([]*OutboxEntry, error)
- func (s *SQLStore) Close() error
- func (s *SQLStore) CompleteRun(ctx context.Context, runID, owner, status, lastErr string, now time.Time) error
- func (s *SQLStore) CountByStatus(ctx context.Context, status string) (int, error)
- func (s *SQLStore) DeliverSignal(ctx context.Context, sig *Signal, now time.Time) (int, error)
- func (s *SQLStore) DequeueRun(ctx context.Context, owner string, lease time.Duration, now time.Time) (*Run, error)
- func (s *SQLStore) EnqueueOutbox(ctx context.Context, e *OutboxEntry) error
- func (s *SQLStore) EnqueueRun(ctx context.Context, r *Run) error
- func (s *SQLStore) GetRun(ctx context.Context, id string) (*Run, error)
- func (s *SQLStore) Heartbeat(ctx context.Context, runID, owner string, lease time.Duration, now time.Time) error
- func (s *SQLStore) MarkIdempotent(ctx context.Context, key string, now time.Time) (bool, error)
- func (s *SQLStore) MarkOutboxFailed(ctx context.Context, id, errMsg string, maxAttempts int, now time.Time) error
- func (s *SQLStore) MarkOutboxSent(ctx context.Context, id string, now time.Time) error
- func (s *SQLStore) Migrate(ctx context.Context) error
- func (s *SQLStore) ParkRun(ctx context.Context, runID, owner, waitSignal string, patch []byte, ...) error
- func (s *SQLStore) RecoverOrphans(ctx context.Context, now time.Time) (int, error)
- func (s *SQLStore) ReleaseParked(ctx context.Context, waitSignal string, limit int, now time.Time) (int, error)
- func (s *SQLStore) RescheduleRun(ctx context.Context, runID, owner string, availableAt time.Time, patch []byte, ...) error
- func (s *SQLStore) RetryRun(ctx context.Context, runID, owner string, availableAt time.Time, patch []byte, ...) error
- type Signal
- type Store
- type Worker
- type WorkerConfig
Constants ¶
const ( StatusPending = "pending" // available (subject to AvailableAt) to be claimed StatusLeased = "leased" // claimed by a worker under an active lease StatusParked = "parked" // waiting on an external signal (HITL/back-pressure) StatusCompleted = "completed" // finished successfully StatusFailed = "failed" // failed terminally (attempts exhausted) )
Run states.
const ( KindStart = "start" // begin a fresh graph execution from the payload KindResume = "resume" // resume an existing session from its latest checkpoint )
Run kinds tell the executor how to start work for the run.
const ( OutboxPending = "pending" OutboxSent = "sent" OutboxFailed = "failed" )
Outbox entry states.
const AdmissionParkSignal = "__admission__"
AdmissionParkSignal is the reserved signal name used to park runs shed by admission control; ReleaseAdmissionParked promotes them back to pending.
const DefaultOutboxMaxAttempts = 10
DefaultOutboxMaxAttempts is the number of delivery attempts after which a permanently-failing outbox entry is dead-lettered (marked OutboxFailed) so it stops being claimed and can no longer starve newer effects.
Variables ¶
var ( // ErrEmpty is returned by Dequeue when no run is currently available. ErrEmpty = errors.New("queue: no available run") // ErrOverloaded is returned by Enqueue when admission control rejects the // run because the queue is at capacity. ErrOverloaded = errors.New("queue: overloaded, run rejected") // ErrLeaseLost indicates a heartbeat/complete failed because the caller no // longer holds the lease (it expired and was recovered by another worker). ErrLeaseLost = errors.New("queue: lease lost") // ErrNotFound is returned when a run does not exist. ErrNotFound = errors.New("queue: run not found") )
Errors returned by the queue.
Functions ¶
This section is empty.
Types ¶
type AdmissionPolicy ¶
type AdmissionPolicy string
AdmissionPolicy controls what happens when the queue is at capacity.
const ( // PolicyReject fails Enqueue with ErrOverloaded past MaxDepth. PolicyReject AdmissionPolicy = "reject" // PolicyPark parks runs past MaxDepth so they can be released later. PolicyPark AdmissionPolicy = "park" )
type Config ¶
type Config struct {
// MaxDepth bounds the number of pending runs. Zero disables admission
// control (unbounded intake).
MaxDepth int
// Policy selects reject vs. park behavior at capacity.
Policy AdmissionPolicy
// DefaultMaxAttempts is applied to enqueued runs that do not set one.
DefaultMaxAttempts int
}
Config configures a Queue.
type Dialect ¶
type Dialect string
Dialect selects SQL syntax for the target database.
const ( // DialectSQLite targets SQLite (dev/test). Dequeue uses an atomic UPDATE over // a LIMIT-1 subquery; SQLite serializes writers so no SKIP LOCKED is needed. DialectSQLite Dialect = "sqlite" // DialectPostgres targets PostgreSQL (production). Dequeue uses // FOR UPDATE SKIP LOCKED so many workers claim disjoint runs concurrently. DialectPostgres Dialect = "postgres" )
type Dispatcher ¶
type Dispatcher interface {
Dispatch(ctx context.Context, e *OutboxEntry) error
}
Dispatcher delivers an outbox entry to its external destination. It must be safe to call more than once for the same entry (delivery may be retried after a crash between send and the sent-mark); use the entry's IdempotencyKey to deduplicate downstream if the destination is not naturally idempotent.
type DispatcherFunc ¶
type DispatcherFunc func(ctx context.Context, e *OutboxEntry) error
DispatcherFunc adapts a function to the Dispatcher interface.
func (DispatcherFunc) Dispatch ¶
func (f DispatcherFunc) Dispatch(ctx context.Context, e *OutboxEntry) error
Dispatch calls f.
type Executor ¶
Executor performs the work for a claimed run. Implementations must respect ctx cancellation: the worker cancels ctx if the lease is lost mid-execution.
type Outbox ¶
type Outbox struct {
// contains filtered or unexported fields
}
Outbox records external effects transactionally with run progress and drains them reliably. Recording an effect twice with the same idempotency key is a no-op, so retries and resumes never double-emit (P1-003).
func (*Outbox) DrainOnce ¶
DrainOnce dispatches up to limit pending entries. It returns the number of entries successfully delivered. Failed dispatches remain pending for retry.
func (*Outbox) Record ¶
func (o *Outbox) Record(ctx context.Context, sessionID, idempotencyKey, topic string, payload []byte) error
Record persists an external effect for later delivery. The idempotencyKey must be stable for a given logical effect so retries collapse to one row.
func (*Outbox) WithMaxAttempts ¶
WithMaxAttempts sets the delivery-attempt cap after which an entry is dead-lettered. A value <= 0 restores the default. It returns the Outbox for chaining.
type OutboxEntry ¶
type OutboxEntry struct {
ID string `json:"id"`
SessionID string `json:"session_id"`
IdempotencyKey string `json:"idempotency_key"`
Topic string `json:"topic"`
Payload []byte `json:"payload,omitempty"`
Status string `json:"status"` // pending, sent, failed
Attempts int `json:"attempts"`
LastError string `json:"last_error,omitempty"`
CreatedAt time.Time `json:"created_at"`
SentAt *time.Time `json:"sent_at,omitempty"`
}
OutboxEntry is an external effect recorded transactionally so it can be delivered exactly once by the drainer, even across retries and resumes. The IdempotencyKey is unique: enqueuing the same effect twice is a no-op (P1-003).
type Queue ¶
type Queue struct {
// contains filtered or unexported fields
}
Queue is the high-level producer/administration API over a Store. It layers admission control on top of the durable Store operations. Workers consume via the same Queue.
func (*Queue) Dequeue ¶
Dequeue claims the next available run under a lease held by owner. It returns ErrEmpty when nothing is available.
func (*Queue) Enqueue ¶
Enqueue admits and persists a run. Admission control may reject (ErrOverloaded) or park the run under overload depending on the configured policy. The run's ID, timestamps, and default MaxAttempts are filled in when unset.
func (*Queue) Heartbeat ¶
Heartbeat extends the lease on an in-flight run. Returns ErrLeaseLost if the caller no longer owns the run.
func (*Queue) Park ¶
Park suspends a leased run until an external signal named waitSignal is delivered for its session. If a matching signal was already delivered, the run is made immediately available instead (no lost-signal race).
func (*Queue) PendingDepth ¶
PendingDepth reports the number of pending runs (for observability/back-pressure).
func (*Queue) RecoverOrphans ¶
RecoverOrphans re-enqueues leased runs whose lease has expired (their worker died). Retry-exhausted runs are failed. It returns the number of runs recovered.
func (*Queue) ReleaseAdmissionParked ¶
ReleaseAdmissionParked promotes up to limit admission-parked runs back to pending, but only while there is spare capacity under MaxDepth. It returns the number of runs released.
func (*Queue) Retry ¶
func (q *Queue) Retry(ctx context.Context, runID, owner string, delay time.Duration, patch []byte) error
Retry durably reschedules a leased run to become available after delay AND increments its attempts counter (the retry budget). It is the post-error backoff path; callers must decide separately when attempts are exhausted.
func (*Queue) Signal ¶
Signal delivers an external signal, waking any parked run of the session that is waiting on it. If no run is waiting yet, the signal is retained so a subsequent Park observes it. It returns the number of runs awakened. This is the entry point a webhook handler calls to resume a HITL run.
func (*Queue) Sleep ¶
func (q *Queue) Sleep(ctx context.Context, runID, owner string, delay time.Duration, patch []byte) error
Sleep durably reschedules a leased run to become available after delay. It is the primitive behind "wait N, then continue" that survives process restarts. Sleep does NOT consume retry budget (attempts is unchanged): durable timers and HITL parks are healthy redeliveries, not failures.
type Reaper ¶
type Reaper struct {
// contains filtered or unexported fields
}
Reaper periodically recovers orphaned runs (whose worker died) by re-enqueuing expired leases. Run it on one or more nodes; recovery is idempotent.
type Result ¶
type Result struct {
// Err, if non-nil, marks the attempt failed; the worker retries (with
// backoff) until MaxAttempts, then fails the run terminally.
Err error
// Sleep, if > 0, durably reschedules the run to run again after the delay.
Sleep time.Duration
// ParkSignal, if set, parks the run until that signal is delivered.
ParkSignal string
// Patch, if non-nil, replaces the run's persisted payload on sleep/park so
// the resumed attempt observes updated state.
Patch []byte
}
Result is what an Executor reports after handling a run. Exactly one outcome applies, checked in this order: Err/failed → retry or fail; Sleep>0 → durable timer; ParkSignal set → wait for external signal; otherwise → completed.
type Run ¶
type Run struct {
ID string `json:"id"`
SessionID string `json:"session_id"`
GraphID string `json:"graph_id"`
Kind string `json:"kind"`
Payload []byte `json:"payload,omitempty"`
Status string `json:"status"`
Priority int `json:"priority"`
Attempts int `json:"attempts"`
MaxAttempts int `json:"max_attempts"`
LeaseOwner string `json:"lease_owner,omitempty"`
LeaseExpiresAt *time.Time `json:"lease_expires_at,omitempty"`
AvailableAt time.Time `json:"available_at"`
WaitSignal string `json:"wait_signal,omitempty"`
SignalPayload []byte `json:"signal_payload,omitempty"`
IdempotencyKey string `json:"idempotency_key,omitempty"`
LastError string `json:"last_error,omitempty"`
CreatedAt time.Time `json:"created_at"`
UpdatedAt time.Time `json:"updated_at"`
}
Run is a unit of durable work: one graph execution or resume.
type SQLStore ¶
type SQLStore struct {
// contains filtered or unexported fields
}
SQLStore implements Store over any database/sql database. It supports SQLite and PostgreSQL via the Dialect. It defines and owns the queue's own schema and never touches the shared storage package's tables.
func NewSQLStore ¶
NewSQLStore constructs a store over an already-open *sql.DB. The caller owns the DB lifecycle unless Close is used.
func (*SQLStore) ClaimOutbox ¶
ClaimOutbox returns pending outbox entries.
func (*SQLStore) CompleteRun ¶
func (s *SQLStore) CompleteRun(ctx context.Context, runID, owner, status, lastErr string, now time.Time) error
CompleteRun sets a terminal status.
func (*SQLStore) CountByStatus ¶
CountByStatus counts runs in a status.
func (*SQLStore) DeliverSignal ¶
DeliverSignal wakes parked runs waiting on sig.Name; retains the signal if none.
func (*SQLStore) DequeueRun ¶
func (s *SQLStore) DequeueRun(ctx context.Context, owner string, lease time.Duration, now time.Time) (*Run, error)
DequeueRun atomically claims the next available run.
func (*SQLStore) EnqueueOutbox ¶
func (s *SQLStore) EnqueueOutbox(ctx context.Context, e *OutboxEntry) error
EnqueueOutbox records an external effect, ignoring duplicate idempotency keys.
func (*SQLStore) EnqueueRun ¶
EnqueueRun inserts a run.
func (*SQLStore) Heartbeat ¶
func (s *SQLStore) Heartbeat(ctx context.Context, runID, owner string, lease time.Duration, now time.Time) error
Heartbeat extends the lease.
func (*SQLStore) MarkIdempotent ¶
MarkIdempotent records key once.
func (*SQLStore) MarkOutboxFailed ¶
func (s *SQLStore) MarkOutboxFailed(ctx context.Context, id, errMsg string, maxAttempts int, now time.Time) error
MarkOutboxFailed records a delivery failure and increments attempts. When the incremented attempts reach maxAttempts (> 0) the entry is dead-lettered (status OutboxFailed) so ClaimOutbox stops returning it and a poison entry can no longer starve newer effects; otherwise it stays pending for retry.
func (*SQLStore) MarkOutboxSent ¶
MarkOutboxSent marks an entry delivered.
func (*SQLStore) ParkRun ¶
func (s *SQLStore) ParkRun(ctx context.Context, runID, owner, waitSignal string, patch []byte, now time.Time) error
ParkRun suspends a leased run pending a signal, honoring an already-delivered signal to avoid a lost-signal race.
func (*SQLStore) RecoverOrphans ¶
RecoverOrphans re-enqueues expired-lease runs and fails exhausted ones.
func (*SQLStore) ReleaseParked ¶
func (s *SQLStore) ReleaseParked(ctx context.Context, waitSignal string, limit int, now time.Time) (int, error)
ReleaseParked promotes parked runs back to pending.
func (*SQLStore) RescheduleRun ¶
func (s *SQLStore) RescheduleRun(ctx context.Context, runID, owner string, availableAt time.Time, patch []byte, now time.Time) error
RescheduleRun returns a run to pending, available at availableAt.
func (*SQLStore) RetryRun ¶
func (s *SQLStore) RetryRun(ctx context.Context, runID, owner string, availableAt time.Time, patch []byte, now time.Time) error
RetryRun returns a run to pending like RescheduleRun but also increments the attempts counter, consuming retry budget. It is the post-error retry path.
type Signal ¶
type Signal struct {
SessionID string `json:"session_id"`
Name string `json:"name"`
Payload []byte `json:"payload,omitempty"`
}
Signal is an external event delivered to parked runs of a session.
type Store ¶
type Store interface {
// Migrate creates the queue schema (idempotent).
Migrate(ctx context.Context) error
// EnqueueRun inserts a run.
EnqueueRun(ctx context.Context, r *Run) error
// DequeueRun atomically claims the highest-priority available run for owner,
// setting a lease that expires at now+lease. Returns ErrEmpty if none.
DequeueRun(ctx context.Context, owner string, lease time.Duration, now time.Time) (*Run, error)
// Heartbeat extends the lease on a run still owned by owner. Returns
// ErrLeaseLost if the run is no longer leased to owner.
Heartbeat(ctx context.Context, runID, owner string, lease time.Duration, now time.Time) error
// CompleteRun sets a leased run to a terminal status. Returns ErrLeaseLost if
// owner no longer holds the lease.
CompleteRun(ctx context.Context, runID, owner, status, lastErr string, now time.Time) error
// RescheduleRun returns a leased run to pending, available at availableAt,
// optionally replacing its payload (durable sleep). It does NOT touch the
// attempts counter: durable timers and parks are normal redeliveries, not
// failed attempts, and must not burn the retry budget.
RescheduleRun(ctx context.Context, runID, owner string, availableAt time.Time, patch []byte, now time.Time) error
// RetryRun returns a leased run to pending, available at availableAt, AND
// increments its attempts counter. Unlike RescheduleRun this consumes retry
// budget: it is the path taken after an execution error.
RetryRun(ctx context.Context, runID, owner string, availableAt time.Time, patch []byte, now time.Time) error
// ParkRun suspends a leased run pending a signal named waitSignal. If a
// matching signal was already delivered for the session, the run is made
// available immediately instead of parked (no lost-signal race).
ParkRun(ctx context.Context, runID, owner, waitSignal string, patch []byte, now time.Time) error
// DeliverSignal wakes parked runs of the session waiting on sig.Name and
// returns the count awakened. If none are waiting, the signal is retained.
DeliverSignal(ctx context.Context, sig *Signal, now time.Time) (int, error)
// ReleaseParked promotes up to limit runs parked on waitSignal to pending.
ReleaseParked(ctx context.Context, waitSignal string, limit int, now time.Time) (int, error)
// RecoverOrphans re-enqueues runs whose lease expired (worker died); runs
// past MaxAttempts are failed. Returns the number of runs recovered.
RecoverOrphans(ctx context.Context, now time.Time) (int, error)
// CountByStatus returns the number of runs in the given status.
CountByStatus(ctx context.Context, status string) (int, error)
// GetRun returns a run by ID (ErrNotFound if absent).
GetRun(ctx context.Context, id string) (*Run, error)
// MarkIdempotent records key; fresh is true only for the first observation.
MarkIdempotent(ctx context.Context, key string, now time.Time) (fresh bool, err error)
// EnqueueOutbox records an external effect. Duplicate IdempotencyKeys are
// ignored (returns nil without inserting).
EnqueueOutbox(ctx context.Context, e *OutboxEntry) error
// ClaimOutbox returns up to limit pending outbox entries.
ClaimOutbox(ctx context.Context, limit int) ([]*OutboxEntry, error)
// MarkOutboxSent marks an entry delivered.
MarkOutboxSent(ctx context.Context, id string, now time.Time) error
// MarkOutboxFailed records a delivery failure, incrementing attempts. When
// the incremented attempts reach maxAttempts the entry is dead-lettered
// (status OutboxFailed) so it is no longer claimed; otherwise it stays
// pending for retry. A maxAttempts <= 0 disables dead-lettering.
MarkOutboxFailed(ctx context.Context, id, errMsg string, maxAttempts int, now time.Time) error
// Close releases resources owned by the store.
Close() error
}
Store is the minimal durable persistence contract the queue depends on. It is defined here (rather than in the shared storage package) so the queue owns its own schema and can be backed by any *sql.DB via SQLStore. All timestamps are passed explicitly so behavior is deterministic and testable.
type Worker ¶
type Worker struct {
// contains filtered or unexported fields
}
Worker claims runs from a Queue and executes them, holding the lease with periodic heartbeats and durably rescheduling/parking/failing per the Result.
func NewWorker ¶
func NewWorker(q *Queue, exec Executor, cfg WorkerConfig) (*Worker, error)
NewWorker constructs a Worker. It panics only on programmer error (nil queue or executor); all runtime failures are returned as errors.
type WorkerConfig ¶
type WorkerConfig struct {
// ID uniquely identifies this worker; it owns the leases it claims.
ID string
// Lease is the lease TTL granted on claim and extended by each heartbeat.
Lease time.Duration
// Heartbeat is how often the worker extends the lease while executing.
Heartbeat time.Duration
// PollInterval is how long to wait after an empty dequeue before retrying.
PollInterval time.Duration
// Backoff computes the retry delay from the attempt count. If nil, a bounded
// exponential backoff is used.
Backoff func(attempt int) time.Duration
}
WorkerConfig configures a Worker.