Documentation
¶
Overview ¶
Package worker turns the in-process workflow engine into a durable, queue-backed runner. It provides:
- A small QueueStore interface describing the persistence contract a backing store must satisfy (claim, heartbeat, complete, reap).
- A Worker that drives a claim loop, a heartbeat goroutine, and a reaper against any QueueStore implementation.
- A Handler interface that the consumer implements to turn a claimed Spec blob into an executed github.com/deepnoodle-ai/workflow.Execution.
The worker package is intentionally transport-agnostic: it knows nothing about SQL, HTTP, multitenancy, billing, or workflow types. All of those concerns live either in the QueueStore implementation (persistence) or in the Handler (domain logic).
Pair this package with github.com/deepnoodle-ai/workflow/experimental/store/postgres for a Postgres-backed store and checkpointer, or provide your own QueueStore.
Index ¶
- Constants
- Variables
- type Claim
- type Config
- type CreditStore
- type DeadLetteredRun
- type Event
- type EventStore
- type FailedRun
- type Handler
- type HandlerContext
- type HandlerFunc
- type HandlerStores
- type NewRun
- type Outcome
- type QueueStore
- type Status
- type Trigger
- type TriggerStatus
- type TriggerStore
- type WebhookDeliverer
- type WebhookDelivery
- type WebhookStore
- type Worker
Constants ¶
const ( DefaultConcurrency = 10 DefaultPollInterval = 5 * time.Second DefaultHeartbeatInterval = 30 * time.Second DefaultStaleAfter = 2 * time.Minute DefaultReaperInterval = 60 * time.Second DefaultMaxAttempts = 3 DefaultRunTimeout = 30 * time.Minute DefaultTriggerInterval = 5 * time.Second DefaultWebhookInterval = 10 * time.Second DefaultReconcileInterval = 5 * time.Minute DefaultTriggerMaxAttempts = 5 DefaultWebhookMaxAttempts = 5 )
Defaults used when Config fields are left zero.
Variables ¶
var ErrLeaseLost = errors.New("worker: lease lost")
ErrLeaseLost is returned by QueueStore operations that fence on (worker_id, attempt). It means another worker has since reclaimed the run, or the run has been dead-lettered by the reaper.
Callers should treat ErrLeaseLost as a normal, expected condition: stop writing for this run and move on.
var ErrTriggerAlreadyClaimed = errors.New("worker: trigger already claimed")
ErrTriggerAlreadyClaimed is returned by TriggerStore.MarkTriggerProcessing when a compare-and-swap finds no matching row, meaning another worker already claimed the trigger.
var ErrWebhookAlreadyClaimed = errors.New("worker: webhook already claimed")
ErrWebhookAlreadyClaimed is returned by WebhookStore.MarkWebhookProcessing when a compare-and-swap finds no matching row, meaning another worker already claimed the webhook delivery.
Functions ¶
This section is empty.
Types ¶
type Claim ¶
type Claim struct {
// ID is the run's stable identifier, also used as the workflow
// engine's ExecutionID.
ID string
// Spec is the opaque payload supplied when the run was enqueued.
Spec []byte
// Attempt is the 1-based attempt counter. First claim sets
// Attempt = 1; each subsequent reclaim increments it.
Attempt int
// WorkerID is the worker that holds the lease. Populated by
// ClaimQueued and used for fencing subsequent writes.
WorkerID string
// OrgID is the organization that owns this run.
OrgID string
// ProjectID is the project that owns this run.
ProjectID string
// ParentRunID is the parent run that enqueued this one, or
// empty for top-level runs.
ParentRunID string
// WorkflowType classifies the run.
WorkflowType string
// InitiatedBy identifies who or what triggered this run.
InitiatedBy string
// CreditCost is the credit cost for this run.
CreditCost int
// CallbackURL is the webhook URL to notify on terminal status.
CallbackURL string
// Metadata carries the arbitrary tag map supplied at enqueue.
Metadata map[string]string
}
Claim is a run that has been atomically claimed by a worker and transitioned from StatusQueued to StatusRunning. A Claim is also the unit of lease fencing: QueueStore writes fence on (WorkerID, Attempt), and passing a *Claim into those calls gives the store full access to the run's metadata without an out-of-band lookup.
type Config ¶
type Config struct {
// QueueStore is the backing persistence. Required.
QueueStore QueueStore
// Handler executes claimed runs. Required.
Handler Handler
// Stores, if set, builds the pre-fenced Checkpointer /
// StepProgressStore / ActivityLogger on HandlerContext before
// each call to Handle. When nil, HandlerContext's store fields
// are left nil and the Handler must do its own wiring (useful
// for in-memory tests or engine-less handlers).
Stores HandlerStores
// SignalStore, if set, is propagated into HandlerContext so
// workflows can send and receive signals. Shared across runs.
SignalStore workflow.SignalStore
// WorkerID identifies this worker process for lease fencing.
// Must be stable across the worker's lifetime. Defaults to
// "worker-<hostname>-<random>".
WorkerID string
// IDGenerator returns a unique identifier for outbox rows
// (triggers, webhook deliveries). Defaults to a 16-hex-char
// random string. Consumers with custom ID schemes can override.
IDGenerator func() string
// Concurrency caps the number of runs executed in parallel.
// Defaults to DefaultConcurrency.
Concurrency int
// PollInterval is how often the claim loop wakes up when idle.
// Defaults to DefaultPollInterval.
PollInterval time.Duration
// HeartbeatInterval is how often the worker refreshes its lease
// on an active run. Defaults to DefaultHeartbeatInterval.
HeartbeatInterval time.Duration
// StaleAfter is the threshold after which a run with no recent
// heartbeat is considered stale and eligible for reclaim or
// dead-lettering. Must be strictly greater than HeartbeatInterval
// to avoid spurious reclaims. Defaults to DefaultStaleAfter.
StaleAfter time.Duration
// ReaperInterval is how often the reaper scans for stale runs.
// Defaults to DefaultReaperInterval.
ReaperInterval time.Duration
// MaxAttempts caps retries. Runs that exceed it are dead-lettered
// to StatusFailed instead of being reclaimed. Defaults to
// DefaultMaxAttempts.
MaxAttempts int
// RunTimeout is the wall-clock timeout applied to each Handler
// invocation. Defaults to DefaultRunTimeout.
RunTimeout time.Duration
// EventStore, if set, receives lifecycle events for each run.
EventStore EventStore
// TriggerStore, if set, enables the outbox pattern for workflow
// chaining. Triggers returned in Outcome are persisted here and
// processed asynchronously.
TriggerStore TriggerStore
// CreditStore, if set, enables credit debit/refund tracking
// per run.
CreditStore CreditStore
// WebhookStore, if set, enables durable webhook delivery for
// runs with a CallbackURL.
WebhookStore WebhookStore
// WebhookDeliverer performs the actual HTTP delivery. Required
// when WebhookStore is set; ignored otherwise.
WebhookDeliverer WebhookDeliverer
// TriggerInterval is how often the trigger processor polls for
// pending triggers. Defaults to DefaultTriggerInterval.
TriggerInterval time.Duration
// WebhookInterval is how often the webhook processor polls for
// pending deliveries. Defaults to DefaultWebhookInterval.
WebhookInterval time.Duration
// ReconcileInterval is how often the credit reconciler runs.
// Defaults to DefaultReconcileInterval.
ReconcileInterval time.Duration
// TriggerMaxAttempts caps trigger processing retries.
// Defaults to DefaultTriggerMaxAttempts.
TriggerMaxAttempts int
// WebhookMaxAttempts caps webhook delivery retries.
// Defaults to DefaultWebhookMaxAttempts.
WebhookMaxAttempts int
// Logger is the structured logger. Defaults to a discard logger.
Logger *slog.Logger
// Clock returns the current time. Injected for tests; defaults
// to time.Now.
Clock func() time.Time
}
Config is the worker configuration. QueueStore and Handler are the only required fields; everything else falls back to sane defaults.
type CreditStore ¶
type CreditStore interface {
// Debit records a credit charge for a run. Idempotent: calling
// Debit twice for the same runID is a no-op.
Debit(ctx context.Context, orgID, runID, workflowType string, amount int) error
// Refund records a credit refund for a failed run. Idempotent:
// calling Refund twice for the same runID is a no-op.
Refund(ctx context.Context, orgID, runID, workflowType string, amount int) error
// HasRefund reports whether a refund exists for the given run.
HasRefund(ctx context.Context, orgID, runID string) (bool, error)
// Balance returns the net credit balance for an org. Positive
// means credits consumed; negative means net refunds.
Balance(ctx context.Context, orgID string) (int, error)
}
CreditStore tracks credit debits and refunds per workflow run. Implementations must make Debit and Refund idempotent per run ID so that retries and the reconciler cannot double-charge or double-refund.
CreditStore is pure ledger: listing which failed runs still need a refund is a QueueStore concern (ListRefundPending) because it joins the ledger against run status.
type DeadLetteredRun ¶
DeadLetteredRun is a run transitioned from running to failed by the reaper after exhausting its retry budget. Returned from DeadLetterStale so the worker can refund credits inline.
type Event ¶
type Event struct {
// Seq is a store-assigned sequence number for cursor-based
// pagination. Zero on input to AppendEvent; set by the store.
Seq int64
RunID string
EventType string // "running", "completed", "failed", "suspended", "review"
Attempt int
WorkerID string
StepName string
Payload map[string]any
CreatedAt time.Time
}
Event is a lifecycle event emitted by the worker during run execution. Events are append-only and intended for real-time streaming (SSE) and observability.
type EventStore ¶
type EventStore interface {
// AppendEvent records an event. The store assigns Seq.
AppendEvent(ctx context.Context, event *Event) error
// ListEvents returns events for a run with Seq > afterSeq,
// ordered by Seq ascending.
ListEvents(ctx context.Context, runID string, afterSeq int64) ([]*Event, error)
// CleanupEvents deletes events older than the given time.
// Returns the number of events deleted.
CleanupEvents(ctx context.Context, olderThan time.Time) (int, error)
}
EventStore persists and retrieves lifecycle events for runs.
type FailedRun ¶
FailedRun is a credit-tracking failed run returned by ListRefundPending. Used by the reconcile loop as a backstop for DeadLetterStale's inline refund.
type Handler ¶
type Handler interface {
Handle(ctx context.Context, hc *HandlerContext) Outcome
}
Handler executes a claimed run. Implementations are responsible for materializing the workflow engine from the opaque Spec bytes, choosing run vs. resume based on claim.Attempt, and reporting the final status back as an Outcome.
A typical implementation:
- Unmarshal hc.Claim.Spec into a workflow definition and inputs.
- Build a *workflow.Execution with hc.Checkpointer, hc.ProgressStore, hc.ActivityLogger, and any activities the consumer registers.
- Call exec.Run(ctx) on the first attempt or exec.Resume(ctx, id) on subsequent attempts (falling back to Run on ErrNoCheckpoint).
- Classify the returned result into an Outcome: - ExecutionStatusCompleted -> StatusCompleted - ExecutionStatusFailed -> StatusFailed (set ErrorMessage) - ExecutionStatusSuspended/Paused -> StatusSuspended
The ctx passed to Handle is scoped to the run. It is cancelled when:
- The worker's parent context is cancelled.
- The run timeout elapses.
- The heartbeat goroutine detects lease loss.
Handlers must respect ctx cancellation and return promptly. Handlers should not call QueueStore methods directly — the worker takes care of status persistence.
type HandlerContext ¶
type HandlerContext struct {
// Claim is the run being executed.
Claim *Claim
// Checkpointer is a lease-fenced workflow.Checkpointer scoped
// to this claim. Writes that fail the (WorkerID, Attempt) fence
// return worker.ErrLeaseLost. This is the only store in the
// bundle that enforces lease fencing.
Checkpointer workflow.Checkpointer
// ProgressStore is a workflow.StepProgressStore scoped to this
// claim. Step progress writes are derived observability data
// and are not fenced — the "latest update wins" semantics mean
// a stale writer cannot corrupt durable state.
ProgressStore workflow.StepProgressStore
// ActivityLogger is a workflow.ActivityLogger scoped to this
// claim. Activity log writes are append-only and not fenced.
ActivityLogger workflow.ActivityLogger
// SignalStore is a workflow.SignalStore for signal/wait
// coordination. Shared across runs and claims; not fenced.
SignalStore workflow.SignalStore
}
HandlerContext carries everything a Handler needs to execute a claimed run. The worker constructs it once per claim, wires in any pre-fenced stores from HandlerStores, and hands the whole bundle to Handle.
All store fields are optional. They are populated only when the worker's Config.Stores factory is set and returns a non-nil value for that concern. A Handler backed by an in-memory engine can ignore the store fields entirely.
Lease fencing is **only** applied to Checkpointer. The other store fields are either append-only (ProgressStore, ActivityLogger) or globally shared (SignalStore), so they do not need to fence on (WorkerID, Attempt). Concrete stores may accept a *Claim in their factory method for symmetry and ignore it — that is expected, not a bug.
type HandlerFunc ¶
type HandlerFunc func(ctx context.Context, hc *HandlerContext) Outcome
HandlerFunc adapts a plain function to the Handler interface.
func (HandlerFunc) Handle ¶
func (f HandlerFunc) Handle(ctx context.Context, hc *HandlerContext) Outcome
Handle implements Handler.
type HandlerStores ¶
type HandlerStores interface {
// NewCheckpointer returns a lease-fenced checkpointer whose
// writes must return worker.ErrLeaseLost when the claim's
// (WorkerID, Attempt) pair no longer owns the run.
NewCheckpointer(claim *Claim) workflow.Checkpointer
// NewStepProgressStore returns a step progress store for the
// claim. The claim is usually ignored; lease fencing is not
// required because writes are idempotent replacements.
NewStepProgressStore(claim *Claim) workflow.StepProgressStore
// NewActivityLogger returns an activity logger for the claim.
// The claim is usually ignored; activity logs are append-only.
NewActivityLogger(claim *Claim) workflow.ActivityLogger
}
HandlerStores is an optional factory that the worker uses to build a HandlerContext's store fields from a Claim. Backing stores that speak the workflow engine's persistence interfaces (postgres, sqlite) implement this — the memstore does not.
Each method returns nil when the factory does not support that concern; the worker propagates the nil into HandlerContext so the Handler can check for availability.
Of the three factory methods, **only NewCheckpointer is expected to return a lease-fenced store**. NewStepProgressStore and NewActivityLogger take a *Claim for API symmetry, but the returned implementations typically ignore the claim and return the shared store unchanged: step progress is write-through observability and activity logs are append-only, so neither needs fencing. SignalStore is not part of this factory at all — it is shared across claims and lives on Config directly.
type NewRun ¶
type NewRun struct {
// ID uniquely identifies the run. Required. Must be unique
// across the QueueStore.
ID string
// Spec is an opaque payload — typically JSON describing the
// workflow definition and inputs — that the Handler consumes at
// execution time.
Spec []byte
// OrgID identifies the organization owning this run. Empty
// means the run is not scoped to an org (single-tenant).
OrgID string
// ProjectID identifies the project (workspace, team, board,
// environment — whatever the consumer product calls it) that
// owns this run. Empty means the run is not scoped to a project.
ProjectID string
// ParentRunID is the run that enqueued this one via the
// trigger outbox. Empty for top-level runs.
ParentRunID string
// WorkflowType classifies the run (e.g., "research", "indexing").
WorkflowType string
// InitiatedBy identifies who or what triggered this run.
InitiatedBy string
// CreditCost is the credit cost for this run. Zero means no
// credit tracking. Consumers typically default this to 1.
CreditCost int
// CallbackURL is an optional webhook URL notified on completion
// or failure.
CallbackURL string
// Metadata is an arbitrary string map persisted alongside the
// run. Typical use: correlation IDs, feature flags, tenant tags,
// anything that does not earn a first-class column. Backed by
// JSONB in the postgres store.
Metadata map[string]string
}
NewRun is a run to enqueue. Spec is an opaque blob interpreted by the Handler, not by the worker.
OrgID, ProjectID, ParentRunID, and InitiatedBy are nullable in the database. Empty string in the Go API means NULL in the database — single-tenant deployments should not invent sentinel values.
type Outcome ¶
type Outcome struct {
// Status is the final status to persist. Must be one of
// StatusCompleted, StatusFailed, StatusSuspended, StatusReview.
Status Status
// Result is an optional opaque blob persisted alongside the
// status. Typical use: JSON outputs, SuspensionInfo, etc.
Result []byte
// ErrorMessage is the human-readable failure reason. Set when
// Status == StatusFailed; ignored otherwise.
ErrorMessage string
// Triggers lists child runs to enqueue via the outbox pattern
// after the run completes. Ignored if no TriggerStore is
// configured on the Worker.
Triggers []NewRun
}
Outcome is the terminal (or dormant) state a Handler reports back to the worker after executing a claim.
type QueueStore ¶
type QueueStore interface {
// Enqueue inserts a new run in StatusQueued with attempt = 0.
// Returns an error if a run with the same ID already exists.
Enqueue(ctx context.Context, run NewRun) error
// ClaimQueued atomically claims the oldest available StatusQueued
// run for the given worker, transitioning it to StatusRunning
// and incrementing its attempt counter.
//
// Returns (nil, nil) when no queued runs are available.
ClaimQueued(ctx context.Context, workerID string) (*Claim, error)
// Heartbeat refreshes the lease on a claimed run. Must fence on
// (claim.WorkerID, claim.Attempt) and status == StatusRunning.
Heartbeat(ctx context.Context, claim *Claim) error
// Complete writes the terminal or dormant status for a claimed
// run. Must fence on (claim.WorkerID, claim.Attempt).
Complete(ctx context.Context, claim *Claim, outcome Outcome) error
// ReclaimStale transitions StatusRunning runs whose heartbeats
// are older than staleBefore back to StatusQueued, for runs with
// attempt < maxAttempts. Runs whose IDs appear in excludeIDs are
// never transitioned.
//
// Returns the number of runs reclaimed.
ReclaimStale(ctx context.Context, staleBefore time.Time, maxAttempts int, excludeIDs []string) (int, error)
// DeadLetterStale transitions StatusRunning runs whose heartbeats
// are older than staleBefore to StatusFailed, for runs with
// attempt >= maxAttempts. Runs whose IDs appear in excludeIDs
// are never transitioned.
//
// Returns metadata for each dead-lettered run. The worker uses
// this to emit observability events and refund credits inline.
DeadLetterStale(ctx context.Context, staleBefore time.Time, maxAttempts int, excludeIDs []string) ([]DeadLetteredRun, error)
// ListRefundPending returns failed runs that were debited
// but have not yet been refunded. Used by the credit reconcile
// loop as a backstop for DeadLetterStale's inline refund.
//
// Implementations that do not track credits can return an empty
// slice — the reconcile loop will do nothing.
ListRefundPending(ctx context.Context, limit int) ([]FailedRun, error)
}
QueueStore is the persistence contract a backing store must satisfy to power a Worker. Implementations are free to use any database, message bus, or in-memory structure — the worker only depends on this interface.
Concurrency contract:
- ClaimQueued must be atomic: two workers calling it concurrently must never receive the same run.
- Heartbeat, SaveCheckpoint, and Complete must fence on (WorkerID, Attempt). Writes that fail the fencing check must return ErrLeaseLost.
- ReclaimStale and DeadLetterStale must honor excludeIDs: runs whose IDs appear in the list must not be transitioned, even if their heartbeats look stale. This protects against DB write contention where a heartbeat write is delayed but the run is in fact still healthy in-process.
type Status ¶
type Status string
Status is the lifecycle status of a run in the queue.
Terminal statuses (Completed, Failed) stop further processing. Suspended marks a run as dormant — waiting on a signal, sleep, or pause — and is not reclaimed by the reaper. Handlers re-enqueue suspended runs when external input arrives.
const ( // StatusQueued is a run waiting to be claimed. StatusQueued Status = "queued" // StatusRunning is a run actively executing under a worker lease. StatusRunning Status = "running" // StatusCompleted is a terminal success status. StatusCompleted Status = "completed" // StatusFailed is a terminal failure status. StatusFailed Status = "failed" // StatusSuspended is a non-terminal dormant status. The run is // waiting on external input (signal, sleep, pause). It is not // reclaimed by the reaper. StatusSuspended Status = "suspended" // StatusReview is a non-terminal dormant status. The run is // waiting for human review or approval. Like Suspended, it is // not reclaimed by the reaper. StatusReview Status = "review" )
type Trigger ¶
type Trigger struct {
ID string
ParentRunID string
ChildSpec NewRun
Status TriggerStatus
Attempts int
ErrorMessage string
ChildRunID string
CreatedAt time.Time
ProcessedAt time.Time
}
Trigger represents a pending child workflow to enqueue, written via the transactional outbox pattern. The worker persists triggers returned in Outcome.Triggers and processes them asynchronously.
type TriggerStatus ¶
type TriggerStatus string
TriggerStatus is the processing status of a workflow trigger.
const ( TriggerPending TriggerStatus = "pending" TriggerProcessing TriggerStatus = "processing" TriggerCompleted TriggerStatus = "completed" TriggerFailed TriggerStatus = "failed" )
type TriggerStore ¶
type TriggerStore interface {
InsertTriggers(ctx context.Context, triggers []Trigger) error
ListPendingTriggers(ctx context.Context, limit int) ([]Trigger, error)
MarkTriggerProcessing(ctx context.Context, id string) error
MarkTriggerCompleted(ctx context.Context, id string, childRunID string) error
IncrementTriggerAttempts(ctx context.Context, id string, errMsg string) error
MarkTriggerFailed(ctx context.Context, id string, errMsg string) error
}
TriggerStore persists and processes workflow triggers using the transactional outbox pattern. Method names are prefixed to avoid collisions when a single store struct implements multiple interfaces.
type WebhookDeliverer ¶
WebhookDeliverer performs the actual HTTP delivery of a webhook payload. The consumer provides an implementation backed by their HTTP client of choice.
type WebhookDelivery ¶
type WebhookDelivery struct {
ID string
RunID string
URL string
EventType string // "workflow.completed", "workflow.failed", etc.
Payload []byte
Status string // "pending", "delivered", "failed"
Attempts int
LastError string
CreatedAt time.Time
DeliveredAt time.Time
}
WebhookDelivery represents a pending or completed webhook notification for a workflow run.
type WebhookStore ¶
type WebhookStore interface {
EnqueueWebhook(ctx context.Context, delivery *WebhookDelivery) error
ListPendingWebhooks(ctx context.Context, limit int) ([]*WebhookDelivery, error)
// MarkWebhookProcessing claims a webhook for delivery using a
// compare-and-swap on status. Returns an error if the webhook was
// already claimed by another worker.
MarkWebhookProcessing(ctx context.Context, id string) error
MarkWebhookDelivered(ctx context.Context, id string) error
IncrementWebhookAttempts(ctx context.Context, id string, lastError string) error
MarkWebhookFailed(ctx context.Context, id string, errMsg string) error
}
WebhookStore persists and manages webhook delivery state. Method names are prefixed to avoid collisions when a single store struct implements multiple interfaces.
type Worker ¶
type Worker struct {
// contains filtered or unexported fields
}
Worker drives a QueueStore: it claims queued runs, executes them via the Handler under a heartbeat lease, and reaps stale runs in the background.
func New ¶
New constructs a Worker from cfg. Returns an error if required fields are missing or if time thresholds are inconsistent.
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
Package memstore provides an in-memory implementation of worker.QueueStore suitable for tests and local development.
|
Package memstore provides an in-memory implementation of worker.QueueStore suitable for tests and local development. |
|
Package runquery defines the backend-neutral read API for workflow runs.
|
Package runquery defines the backend-neutral read API for workflow runs. |