sqlstore

package module
v0.1.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 10, 2026 License: Apache-2.0 Imports: 32 Imported by: 0

Documentation

Overview

Package sqlstore is the durable SQL driver of workflow.Client: every run, step result, wait, sleep, schedule, signal and lease is a row, so a restart or a dead worker loses nothing. It runs on SQLite (durable on one machine), Postgres and MySQL (many workers on many machines over one database), through database/sql and github.com/ubgo/sqlb.

db, _ := driversqlite.Open("app.db")      // or driverpgx.Open(dsn)
c, _ := sqlstore.New(db, sqlb.SQLiteDialect{})
_ = c.Migrate(ctx)                         // explicit, versioned; Start never migrates
_ = c.Register(wf)
_ = c.Start(ctx)                           // nothing runs before this

Capabilities

Every capability, each proven by the workflowtest conformance suite on SQLite, Postgres and MySQL 8: CapDurable, CapDelayedStart, CapWaitForEvent, CapDurableSleep, CapConcurrencyKey, CapRateLimit, CapIdempotencyKey, CapCron, CapCancel, CapRunFinished, CapRetryJitter and CapRetryDelay (retry delays are computed in Go by workflow.RetryPolicy.NextDelay; WithJitterSource pins the draws in tests), CapCronZone (declared crons run in Workflow.CronLocation, schedules in Schedule.Location) and CapMisfireSkip, CapMisfireFireOnce and CapMisfireFireAll, and the queue, retry, admin, pause, schedule and drain capabilities (CapQueue, CapQueueOverride, CapJobRetry, CapRetryUntil, CapRetryMaxErrors, CapDiscard, CapDiscardExhausted, CapQueueLimitGlobal, CapDrainReport, CapRunQuery, CapRetryRun, CapDeleteRun, CapCancelAll, CapPauseWorkflow, CapPauseQueue, CapSchedulePause, CapScheduleWhen, CapScheduleWindow, CapTickEvents, CapScheduleList, CapDrain), and CapRunStates, CapChain and CapWorkerScale, and CapOverlapSkip, CapScheduleTrigger, CapScheduleHistory, CapEnqueueOnly, CapErrorHistory, CapShutdownRequeue and CapForceRun, and CapTenant, CapResult, CapDueQuery, CapNodeIdentity, CapSnooze, CapHeartbeat, CapRunTimeout, CapKeyFIFO, CapCronSeconds, CapScheduleActive, CapScheduleKeep and CapTickTime (see Run facts and schedules below). CapClosure is not declared: a Go function cannot be stored, so it could not survive the restart this driver exists for, and workflow.EnqueueClosure refuses the client with an UnsupportedError; register a workflow instead. Keys are Go functions (workflow.KeyFunc); an expression key (KeyExpr) is refused at Start with an UnsupportedError, because SQL has no engine expression language.

How work moves

  • Enqueue writes a run row. A run with RunAt in the future is "scheduled" until the dispatcher sees its time come.
  • Admission moves a pending run to active and writes a row per step. A workflow with a concurrency or rate policy admits under its lock row (a row-level lock on Postgres and MySQL, the single writer on SQLite), so the bucket counts a decision reads hold until it commits.
  • Workers claim ready steps: FOR UPDATE SKIP LOCKED on Postgres and MySQL, a single conditional UPDATE on SQLite. A claim is a lease renewed every third of WithLease; a step whose lease expires (its worker died) is claimed again by any client, so a step runs at least once. A client that stops hands its running steps back at once. An attempt taken over more than WithMaxReclaims times (default 5) is not run again: it fails with an error wrapping ErrAbandoned that names how many workers abandoned it, and the retry policy then applies as for any failure (a retry starts with a fresh count), so a step that kills every worker it lands on cannot loop through the cluster.
  • Every transaction that changes a run's steps first updates the run row, which serialises them per run on every engine.
  • Retries follow workflow.EffectiveRetries; a failed attempt's step becomes ready again after the backoff. A step that fails for good marks the run failing, OnFailure runs once with the failing step's input (under a lease, so a crash reruns it on another client: at least once), then the run is failed.
  • Sleeps and wait deadlines are due times in the steps table. Signals are rows; a run takes only signals stored after its enqueue (its signal floor), and the consumed table's primary key means one run never takes one signal twice. Signals no unfinished run can take are pruned.
  • Cron schedules (declared and Client.Schedule) are rows with their next fire time, parsed with cron.Parse or WithCronParser (every client of one database must use the same parser). A tick is fired by moving the schedule's next fire time on from the value read and inserting the (schedule, fire time) row, in one transaction: the client that wins enqueues the run, any other fails and enqueues nothing, so a tick runs once across any number of clients. Under the default misfire policy a tick missed while no client ran fires once on return and older missed ticks are not replayed; a workflow.Misfire on the schedule or workflow (stored in <prefix>schedule_misfire) fires what Misfire.Due picks instead: nothing late (MisfireSkip), the newest tick within Within (MisfireFireOnce), or up to Max missed ticks, oldest first, each with its own tick row (MisfireFireAll).
  • Client.Schedule with a Name is an upsert by (Workflow, Name): the same name gives the same ScheduleID and row on every node and boot, so an app can declare its schedules at start-up; an unchanged declaration keeps its next fire time. A schedule without a Name is new on every call.
  • Queues: every run's queue (workflow.EffectiveQueue) is stored in <prefix>run_ext with its Job.Retry override and idempotency key. A client built WithQueues claims only its queues, at most each queue's Workers at once, higher Priority tiers first; without it, it claims every queue. Runs of a queue no started client serves wait. QueueConfig.Limit caps a queue's in-flight step attempts across every client sharing the database (CapQueueLimitGlobal): the claim takes a per-queue lock row, then counts the queue's running steps with a live lease, so two nodes cannot both take the last slot and a dead node's slots free when its leases expire. Workers stays per client.
  • Pauses (Pauser, SchedulePauser) are rows in <prefix>pauses and <prefix>schedule_ext. The claim statement itself excludes held runs, so a pause is obeyed by every client from its commit on and survives restarts. A paused schedule's due ticks are reported TickSkipped (reason paused) once across the cluster and never fire, not even on resume: its next fire time moves past them.
  • A failed attempt is judged by workflow.RetryPolicy.Next with the run's Job.Retry (else the step's or workflow's policy) and the step's stored counts: steps.failures is Attempts.N, <prefix>step_ext holds Errors and First, so Until and MaxErrors hold across restarts and nodes. A retry waits as a ready step whose due time is the end of its backoff; while one does (and no attempt of the run executes) the run reads StatusRetrying on every client, and RunInfo.NextAttempt, NextAttemptAt and Error give that step's next attempt and its last attempt's error (kept in <prefix>step_err). A step returning workflow.Discard ends its run with the stored status discarded (StatusDiscarded) and an Err matching ErrDiscarded and ErrCancelled; OnFailure does not run and Retry refuses it. A run out of retries, MaxErrors or Until ends failed, unless the policy sets RetryPolicy.DiscardOnExhaust (CapDiscardExhausted), which ends it discarded, with no OnFailure. Retention keeps a discarded run as long as a cancelled one (Retention.Cancelled).
  • Chains: Job.Chain, and the edits a step makes with PrependToChain and AppendToChain, are a row of <prefix>run_chain. An attempt's edits are written in the transaction that records its success, so a failed attempt's edits are dropped and a crash loses none. The run's success inserts the next job as a new run carrying the rest of the chain, in the same transaction; when that job's workflow is not registered on the finishing client, the row waits until a client that registers it hands it on. A failed, discarded or cancelled run stops its chain; Retry of a failed run resumes it on success. Enqueue middleware does not run for the jobs a chain hands on.
  • Worker scaling (workflow.WorkerScaler): QueueLoads reads the depth and oldest wait of the ready steps of each queue given to WithQueues, and SetWorkers changes that queue's cap on this client while it runs (not stored; a restart starts from WithQueues). Drive it with workflow.BalanceLoop; nothing balances unless the app runs one.
  • Runs lists runs of every workflow in the database; Retry puts a failed run back to pending with its succeeded steps kept, and admission resumes it from the steps not yet done; Delete and retention remove a run with its steps and side rows.
  • Schedule windows and ScheduleWhen are decided on the firing node before the firing transaction; a skipped tick still gets its tick row, so one node reports it. OnTick observers hear every fired, skipped and misfired tick after its commit (best effort on a crash). Every run a schedule fires carries workflow.LabelSchedule.
  • EnqueueMany writes several runs in one transaction (all or none); BatchStore implements batch.Pruner.
  • Drain stops claiming, waits for every attempt from claim to written outcome, then stops; WithStopWhenIdle drains once no unfinished run of the client's workflows (on its queues) existed for the grace. DrainWithReport (workflow.ReportingDrainer, CapDrainReport) also returns the attempts still running at the deadline, each with the disposition workflow.DrainRequeued: it was handed back as ready without spending an attempt.
  • WithRecycle (workflow.Recycle: MaxJobs, MaxTime, MaxMemory) drains the client the same way once a limit is reached, so a supervisor restarts the process; StopCause returns the *workflow.RecycleError and workflow.Serve returns it. Runs it never claimed stay queued.
  • A label filter (RunQuery.Labels and, with CapLabelPrefix, RunQuery.LabelPrefixes, in Runs and CancelAll) reads the run_labels index: one row per label of a run, key and value cut to 191 characters, written with the run and removed with it. A prefix is a LIKE pattern with its wildcards escaped. The exact labels are then checked in Go, so a long value never matches wrongly.
  • Run ends are delivered to OnRunFinished observers from the table, claimed under a lease and marked after delivery: at least once, including ends that happened while no client was running.

Time

Due times (delays, sleeps, deadlines, backoff, timeouts, rate windows, cron) come from WithClock. Leases and the poll interval are wall time (WithLeaseClock), because they measure whether a worker is alive: a fake scheduling clock moved by an hour must not make every running step look abandoned. Times are stored as BIGINT Unix nanoseconds.

Nodes whose clocks disagree compare due times and lease ends against different "now"s. WithDatabaseClock makes the database server's clock the one source for both (DatabaseClock: read at Start and every resync, advanced in between on the local monotonic clock, so a wall-clock jump on a node moves nothing). A lease holder fences itself: when a renewal has not landed a whole lease after the last one began, measured on its own lease clock, the attempt is cancelled with ErrLeaseLost and its outcome is not written, because another node may already hold the step. WithLeaseSkew makes the other nodes wait that much longer past a lease's end before taking it over (steps, OnFailure, run-end delivery, the prune lock), so clocks that disagree by less than the margin never let two nodes hold one lease.

Clusters

Clients over one database and table prefix are workers of one queue. A client only claims, admits, fires and delivers for the workflows it has registered, so services with different workflows can share the tables. Cancel reaches a step running on another client at that client's next lease renewal.

A client finds work another client made due at its next poll (WithPollInterval, a second by default). WithNotifier adds wake-ups on top: with pgnotify.New(db) on Postgres (sqlstore/pgnotify, its own module so only an app that uses it downloads pgx), an enqueue, a finished step, a retry or a resume on one client sends a NOTIFY and every idle client listening on the tables' channel starts at once. Measured on the host Postgres with a one-second poll, 15 samples each, enqueue to step start on another client: polling p50 591ms, p95 897ms; with pgnotify p50 38ms, p95 99ms (shared development machine under load). Polling stays as the fallback, so a lost notification only delays work.

Every client fires cron ticks, and the unique tick row lets one fire per tick win: no election, and it survives a split brain. Where the fire must happen on one known node, WithLeaderElection (a LeaderElector over the tables' leaders row: a renewed lease, self-fencing, a Term that grows on each handover) or WithElector (any Elector: a Redis lock, a Kubernetes Lease) limits cron firing and housekeeping to the leader; the tick row stays as the second guard. Steps run on every client either way.

WithFetchBatch sets how many rows one dispatcher query reads (default 100). Workers claim one step per query, each into a free worker, because a claimed step is leased and a lease held for a busy worker would keep the step from every other node.

Schedule overlap, trigger and history

Each schedule has a schedule_last row with its overlap policy (Schedule.Overlap, Workflow.CronOverlap) and its last fire: the time and the run of the last tick or TriggerSchedule. Under OverlapSkip a tick is skipped (TickSkipped, TickReasonOverlap) while that run is unfinished, and TriggerSchedule refuses with an error matching workflow.ErrOverlap. The fire moves the last run with a compare-and-set on the run its decision read, so two fires deciding at once on two nodes start one run. A last run that was deleted or pruned does not block. Schedules reports Overlap and Last, with the last run's status read at list time.

Enqueue-only clients

WithEnqueueOnly builds a client that only writes work (web nodes), while worker clients over the same tables run it; see WithEnqueueOnly for what a job of a workflow it did not register still gets.

Run facts and schedules

Migration 008 adds run_info, one row per run: Job.Tenant (StepInfo, RunInfo, RunQuery.Tenant), the result stored when the run succeeds (workflow.RunOutput of its final steps; Result, RunInfo.Output), the latest workflow.ReportProgress (RunInfo.Progress) and the nodes that claimed its attempts (RunInfo.AttemptedBy; the node of each recorded attempt error is in attempt_err_node). Reschedule moves a scheduled run; RunQuery.DueFrom and DueTo select delayed runs by RunInfo.RunAt. A workflow.Snooze puts its step back as the same attempt (status snoozed, run Waiting); a step's HeartbeatTimeout fails an attempt that stops calling workflow.Heartbeat (a beat also renews the lease, at most every third of it); Workflow.RunTimeout fails the run from its admission, and OnTimeout replaces OnFailure for every timeout (the kind is stored, so the run's error matches its sentinel after a restart). CancelStep stops a running attempt with workflow.ErrStepCancelled, on another client at its next lease renewal. Concurrency.FIFO keeps runs of one key in enqueue order past a rate limit too. Schedule.Seconds, Active and KeepRuns (and the Workflow.Cron equivalents) are stored in schedule_spec; KeepRuns is applied at each fire and at housekeeping; every fired run carries workflow.LabelTickAt.

Every client reports itself in the nodes table every third of its lease (Nodes; WithNodeID) and leaves it on a clean stop; WithSchedulerOnly makes a client that fires schedules and runs no step, WithoutScheduler one that runs steps and fires nothing. Health answers liveness and readiness (started, database ping). OnLifecycle reports Inserted, Started, Retrying, Snoozed, Progress and Finished for the runs and attempts this client handles. QueueConfig.Weight draws among the queues of one priority by weight on the jitter source. WithPayloadLimit refuses big payloads. UniqueStore also records each key's holder run (middleware.UniqueHolders, migration 002_unique_holder of its list).

Operator actions and error history

ForceComplete and ForceFail end a run from outside (workflow.RunForcer); Retry re-runs a force-failed one. Every failed attempt is kept in attempt_errors (RunInfo.Errors), the newest workflow.DefaultErrorHistory per run unless WithErrorHistory says otherwise. A step cut short by a clean stop (Start ctx cancelled, Stop) is handed back with its attempt refunded: the next worker runs it as the same attempt, and it is neither counted against the retry policy nor recorded as an error. A crash is not a clean stop: the step's lease expires and the takeover counts.

Captured output

OutputStore (Client.OutputStore, or NewOutputStore with its own Migrate for an app without the queue) is a middleware.OutputSink that keeps each attempt's captured output in <prefix>outputs, bounded per run and per attempt (OutputLimits), and RunOutput reads a run's attempts back for an admin screen. A Client deletes a run's output with the run; OutputStore.PruneBefore bounds the table where no Client runs.

Batches

Client.BatchStore is a durable batch.Store in the same database: one row per batch, Update atomic per batch on every engine.

Retention

Finished runs are kept forever unless WithRetention says otherwise: deleting history is the app's decision. Retention sets a keep-for per terminal status (zero keeps forever) and a KeepLast count per workflow (only the newest N finished runs stay; either rule deletes), with per-workflow overrides. Prune deletes what the rules select in chunks of 100 (WithPruneChunk), each in a short transaction and each passed first to the WithArchive hook, whose error keeps the chunk; a run's steps and consumed-signal markers go with it. Prune never deletes a run that holds an idempotency key, whose end has not been delivered to an OnRunFinished observer (unless Retention.Undelivered, for clusters with no observer; without it such a cluster deletes nothing, which Prune reports in PruneResult.HeldUndelivered and the background prune logs as a warning), or that belongs to an open batch in Client.BatchStore. A batch kept in another store cannot be seen here, and its status poller reads each job's Status, so with retention on keep batches in Client.BatchStore or a job's run may be deleted before its batch finishes. A started client prunes every WithPruneInterval (default an hour) when it wins the prune lock's lease, so one client of a cluster prunes per interval; WithoutAutoPrune leaves Prune to the app's own scheduler. Old cron tick rows (older than WithTickRetention, default 24 hours), unreachable signals and expired uniqueness locks are pruned by the dispatcher every WithHousekeepingEvery passes (default 50) whatever the retention.

The queue's tables are hot: claims and admissions rewrite their rows, so on Postgres every finished run kept there adds index entries and every update a dead row for VACUUM. Retention keeps them small. To keep the history anyway, WithArchiveTable makes Prune move each run, with its steps' outputs, into the cold tables <prefix>runs_archive and <prefix>steps_archive in the transaction that deletes it, append-only and never read by the queue; Client.ArchivedRuns reads them back.

Transactional enqueue

Client.EnqueueTx writes a run inside the app's own *sql.Tx, so it commits or rolls back with the app's writes (the transactional outbox, with no relay because the queue is this database). The run is written pending and admitted after the commit by the dispatcher, within WithPollInterval, or at once on a client that is woken with Client.Wake after the commit. Idempotency keys hold as for Enqueue; the insert runs under a savepoint, so a duplicate key never aborts the caller's transaction. Enqueue middleware does not run, because its effects would not roll back with the transaction. To enqueue transactionally on another backend (Hatchet, memory) or through middleware, write to the sqlstore/outbox package and let its relay enqueue after the commit, at least once.

An app whose transactions are pgx transactions passes them through the sqlstore/pgxtx module: pgxtx.Run hands EnqueueTx (or outbox.Write) a *sql.Tx whose statements run on the app's pgx.Tx.

Uniqueness locks

UniqueStore is a durable middleware.UniqueStore (ADR-0005): locks are rows of <prefix>unique_locks, so middleware.Unique holds across every process sharing the database, with MemoryUnique's semantics exactly. Take it from a Client (Client.UniqueStore, created by Client.Migrate) or build it with NewUniqueStore over any *sql.DB, which needs no running Client: an app on Hatchet with a database gets durable uniqueness that way. Its table has its own migration list (owner ubgo/workflow/sqlstore/unique) in the same ledger, so UniqueStore.Migrate creates only that table. Expired locks are deleted by the dispatcher's prune, or by PruneExpired.

Debounce state

DebounceStore is a durable middleware.DebounceStore: each debounce key's pending burst is a row of <prefix>debounce, so middleware.Debounce with WithDebounceStore debounces a key enqueued from every process sharing the database. Swap is atomic across processes (it locks the key's row before reading the burst it replaces; a first-insert race is retried), so every replaced run is cancelled once. Take it from a Client (Client.DebounceStore, created by Client.Migrate) or build it with NewDebounceStore over any *sql.DB; its table has its own migration list (owner ubgo/workflow/sqlstore/debounce, migration 001_debounce) in the shared <prefix>migrations ledger. Bursts whose run has started are deleted by the dispatcher's housekeeping, or by PruneExpired.

Journal and bucket state

JournalStore is a durable workflow.JournalStore (the memo of workflow.Memo and the helpers built on it): a run's memoised results are rows of <prefix>journal, so a step retried after a crash, on any node, gets them back. BucketStore is a durable workflow.BucketStore, the leaky bucket behind workflow.UpstreamLimit: one row per bucket key in <prefix>buckets, and Take is atomic across processes (it locks the key's row before reading the level; a first-insert race is retried), so every worker calling one upstream draws from one bucket; time is the now the caller passes. Take each from a Client (Client.JournalStore, Client.BucketStore, created by Client.Migrate) or build it with NewJournalStore or NewBucketStore over any *sql.DB; each has its own migration list (owners ubgo/workflow/sqlstore/journal and ubgo/workflow/sqlstore/bucket, both 001) in the shared ledger. Migration 009 adds schedule_tenant: Schedule.Tenant, the Job.Tenant of every run the schedule fires or is triggered to run, reported in ScheduleInfo.Tenant.

Database policies

Three things about the database are pluggable, each with today's behaviour as the default:

  • WithErrorClassifier: how a driver error is sorted into a lost unique race, a retryable lock conflict or an outage (DefaultErrorClassifier knows the drivers sqlb's contrib modules open). A transaction that loses a lock race is tried WithTxAttempts times (default 5).
  • Errors the classifier calls unavailable (a refused or dropped connection, a server shutting down) are marked with workflow.Unavailable, so errors.Is(err, workflow.ErrUnavailable) holds and a failover client can tell an outage from a bad request.
  • WithIDGenerator: how RunIDs, ScheduledIDs, step IDs and unnamed ScheduleIDs are made (default "<kind>_<32 hex>"); an ID is 1 to 64 bytes, and a generator that errors fails the operation.

Placement

WithTablePrefix starts every table name ("wf_" by default). On Postgres, WithSchema puts the tables in a schema of their own; the migration ledger then stays in the search_path with its rows owned as "<owner>@<schema>". SQLite and MySQL refuse WithSchema.

Schema versions

The schema is a list of numbered migrations (migrations.go; 001_initial is the baseline; UniqueStore's table is a second list under its own owner) applied by Migrate through sqlb's contrib/migrate ledger, in a table named <prefix>migrations, under the engine's migration lock, so concurrent Migrate calls are safe. A released migration is never edited (its checksum would refuse every database that applied it); a schema change is a new migration appended to the list (002_schedule_misfire adds the misfire table, 003_queues_admin the queue, retry, pause and schedule-window tables and backfills a run_ext row for every existing run, 004_run_states_chain the run_chain and step_err tables and rewrites runs earlier releases stored as cancelled with run_ext.discarded = 1 to discarded, 005_run_labels the run_labels label index, whose backfill of existing runs is Client.IndexLabels, run by Migrate after the scripts and safe to run twice and from several processes, 006_schedule_history the schedule_last table (each schedule's overlap policy and last fire, backfilled for every existing schedule) and the attempt_errors table (each run's error history), 007_leader_archive the leaders table (LeaderElector) and the cold runs_archive and steps_archive tables (WithArchiveTable), 008_run_info_nodes the run_info, attempt_err_node, step_cancel, schedule_spec and nodes tables (backfilling a run_info row for every existing run); Start refuses a database without them until Migrate runs). Migrate and Start refuse a database a newer release migrated, with an error matching ErrSchemaTooNew that names the unknown migrations. There are no down migrations.

MigrationScripts returns the same migrations as SQL text, per dialect and table options, for teams that ship DDL through goose, Atlas, Flyway, Liquibase or a DBA review: apply the statements there (each is safe to run twice). Start checks the tables, not the ledger, so a schema applied that way starts without Migrate; call Client.IndexLabels once after applying 005_run_labels, since its backfill decodes JSON in Go.

Index

Constants

View Source
const (
	DefaultOutputPerRun = 20       // attempts kept per run, the newest
	DefaultOutputBytes  = 64 << 10 // bytes of text kept per attempt
)

Defaults of OutputLimits, the same bounds as admin's in-memory store, so moving from it to this one keeps what an operator sees.

View Source
const BucketMigrationOwner = "ubgo/workflow/sqlstore/bucket"

BucketMigrationOwner is the ledger owner of BucketStore's migrations: a list of its own, so NewBucketStore(...).Migrate creates only the buckets table, while Client.Migrate applies it too.

View Source
const DebounceMigrationOwner = "ubgo/workflow/sqlstore/debounce"

DebounceMigrationOwner is the ledger owner of DebounceStore's migrations: a list of its own, so NewDebounceStore(...).Migrate on a database with no sqlstore queue creates only the debounce table, while Client.Migrate applies it too.

View Source
const JournalMigrationOwner = "ubgo/workflow/sqlstore/journal"

JournalMigrationOwner is the ledger owner of JournalStore's migrations: a list of its own, so NewJournalStore(...).Migrate creates only the journal table, while Client.Migrate applies it too.

View Source
const MigrationOwner = "ubgo/workflow/sqlstore"

MigrationOwner is the ledger owner of every sqlstore migration, so an app that keeps its own migrations in a ledger table of the same name never collides with these. Under WithSchema the recorded owner is "<MigrationOwner>@<schema>".

View Source
const OutputMigrationOwner = "ubgo/workflow/sqlstore/output"

OutputMigrationOwner is the ledger owner of OutputStore's migrations, a list of its own like UniqueStore's: NewOutputStore(...).Migrate creates only the outputs table, Client.Migrate applies it too.

View Source
const UniqueMigrationOwner = "ubgo/workflow/sqlstore/unique"

UniqueMigrationOwner is the ledger owner of UniqueStore's migrations: a list of its own, so NewUniqueStore(...).Migrate on a database with no sqlstore queue creates only the lock table, while Client.Migrate applies both.

Variables

View Source
var ErrAbandoned = errors.New("sqlstore: step abandoned by its workers")

ErrAbandoned is wrapped by the error of an attempt that failed because its lease expired more than WithMaxReclaims times: every worker that took it died before finishing. The retry policy treats it like any failure.

View Source
var ErrLeaseLost = errors.New("sqlstore: step lease lost")

ErrLeaseLost is the context cause of an attempt whose lease went unrenewed for a whole lease (the database was unreachable or too slow): from then on another client may take the step over, so this one stops. Its outcome is not written; the step runs again once the lease expires for the other clients (WithLeaseSkew later), and the takeover counts toward WithMaxReclaims. A handler can tell it apart with context.Cause.

View Source
var ErrOutputNoRun = errors.New("sqlstore: output has no run (captured outside a driver)")

ErrOutputNoRun is returned by Deliver for output captured outside a driver (no StepInfo, or no RunID): there is no run to keep it under.

View Source
var ErrSchemaTooNew = errors.New("sqlstore: the database schema is newer than this code")

ErrSchemaTooNew means the database records a sqlstore migration this code does not know: a newer release migrated it. Running older code against it could write rows the new schema does not expect, so Migrate and Start refuse it. The error is a *SchemaTooNewError naming the migrations.

Functions

func DefaultIDGenerator

func DefaultIDGenerator(kind IDKind) (string, error)

DefaultIDGenerator returns the kind, an underscore and 16 random bytes in hex ("run_3f2a..."). It panics only if the operating system's random source fails, which crypto/rand documents as never happening on supported platforms.

Types

type ArchiveFunc

type ArchiveFunc func(ctx context.Context, runs []FinishedRun) error

ArchiveFunc receives each chunk of runs before it is deleted. Returning an error keeps the chunk (and stops the prune), so history can be copied to cold storage first and is never lost to a failed copy.

type ArchiveQuery

type ArchiveQuery struct {
	// Workflow keeps one workflow's runs.
	Workflow string
	// RunID keeps one run.
	RunID workflow.RunID
	// FinishedBefore keeps runs that finished before it, for paging: pass
	// the last FinishedAt of the previous page.
	FinishedBefore time.Time
	// Limit caps the result; zero or less means 100.
	Limit int
}

ArchiveQuery selects archived runs (ArchivedRuns). Zero fields select everything.

type BatchStore

type BatchStore struct {
	// contains filtered or unexported fields
}

BatchStore is a durable batch.Store over the client's database and table prefix: one row per batch, its State as a JSON document. Update locks the row before reading it (the same first-statement-is-a-write rule every run transaction follows), so the read, fn and the write are atomic per batch on every engine and the once-only latches in batch.State hold across processes.

func (*BatchStore) Create

func (b *BatchStore) Create(ctx context.Context, st batch.State) error

Create implements batch.Store. An existing ID is an error.

func (*BatchStore) Get

func (b *BatchStore) Get(ctx context.Context, id batch.ID) (batch.State, error)

Get implements batch.Store.

func (*BatchStore) Prune

func (b *BatchStore) Prune(ctx context.Context, f batch.PruneFilter) (int, error)

Prune implements batch.Pruner: it deletes every batch f.Matches accepts and returns how many. The times live in the JSON document, so it reads the table a page at a time, decides in Go with f.Matches (the one rule every store applies) and deletes each match only if its row did not change since it was read (its ver), so a batch an Update moved on in between is judged again on the next Prune rather than deleted on stale state. A filter with no rule deletes nothing.

func (*BatchStore) Unfinished

func (b *BatchStore) Unfinished(ctx context.Context) ([]batch.State, error)

Unfinished implements batch.Store: every open batch (State.Open: a job has not ended, or a decided callback is still pending), read through the finished column's index. The column holds !State.Open(), "closed", and keeps the name finished from the 001 schema.

func (*BatchStore) Update

func (b *BatchStore) Update(ctx context.Context, id batch.ID, fn func(*batch.State) error) (batch.State, error)

Update implements batch.Store: lock, read, apply fn, write, in one transaction. When fn fails nothing is written.

type BucketStore

type BucketStore struct {
	// contains filtered or unexported fields
}

BucketStore is a durable workflow.BucketStore: each bucket key's level is a row of <prefix>buckets, so every process calling one upstream draws from one bucket. Like UniqueStore it needs no running Client: build it with NewBucketStore over any *sql.DB, or take Client.BucketStore.

limit, err := workflow.NewUpstreamLimit(store, spec, nil)

Take is atomic across processes: it first bumps the key's row, which locks it (a row lock on Postgres and MySQL, the single writer on SQLite), then reads the level, drains it by the leak since the stored time and admits or refuses the cost, in the same transaction. When the key has no row yet, the insert's primary key decides: of two processes inserting at once, the loser's transaction is tried again and finds the winner's row, so the cost of both is counted. The time is the now the caller passes (never the database's clock), so an injected clock drives it.

func NewBucketStore

func NewBucketStore(db *sql.DB, d sqlb.Dialect, opts ...Option) (*BucketStore, error)

NewBucketStore returns the bucket store over db for dialect d. Of the Options, WithTablePrefix, WithSchema, WithTxAttempts and WithErrorClassifier apply; the others configure a Client and are ignored. Call Migrate before the first Take.

func (*BucketStore) Migrate

func (s *BucketStore) Migrate(ctx context.Context) error

Migrate creates the buckets table (its own migration list, under the same ledger and lock as Client.Migrate), and nothing else.

func (*BucketStore) Observe

func (s *BucketStore) Observe(ctx context.Context, key string, spec workflow.BucketSpec, level float64, now time.Time) error

Observe implements workflow.BucketStore: the level is set to what the upstream reported, clamped to [0, Capacity].

func (*BucketStore) Take

func (s *BucketStore) Take(ctx context.Context, key string, spec workflow.BucketSpec, cost float64, now time.Time) (time.Duration, error)

Take implements workflow.BucketStore, atomically across processes (see BucketStore), with MemoryBuckets' arithmetic: the level leaks since the stored time, then cost is added when it fits; otherwise nothing is added (the drained level and the time are still stored) and the wait until it would fit is returned.

type Client

type Client struct {
	// contains filtered or unexported fields
}

Client is the SQL workflow.Client. Build it with New, create the schema with Migrate, Register workflows, then Start. All state is in the database or hangs off the value: two clients over one database are two workers of one queue, and two clients over two prefixes share nothing.

func New

func New(db *sql.DB, d sqlb.Dialect, opts ...Option) (*Client, error)

New returns a client over db, whose SQL is written for d (sqlb's Postgres, SQLite or MySQL dialect). It opens nothing and starts nothing: db is the app's pool, used from Migrate and Start on.

func (*Client) ArchivedRuns

func (c *Client) ArchivedRuns(ctx context.Context, q ArchiveQuery) ([]FinishedRun, error)

ArchivedRuns reads runs WithArchiveTable moved to the cold tables, newest finish first, each with its succeeded steps' outputs. It reads the tables directly and needs Migrate, not Start.

func (*Client) BatchStore

func (c *Client) BatchStore() *BatchStore

BatchStore returns the batch.Store kept in this client's tables. It needs Migrate, nothing else: it works before Start and with no workflow registered.

func (*Client) BucketStore

func (c *Client) BucketStore() *BucketStore

BucketStore returns the bucket store in this client's database and prefix. Client.Migrate creates its table.

func (*Client) Cancel

func (c *Client) Cancel(ctx context.Context, ref workflow.RunRef) error

Cancel stops a run or a delayed start. Finished runs are left alone. Steps running on this client stop at once; on another client they stop at its next lease renewal (a third of the lease).

func (*Client) CancelAll

func (c *Client) CancelAll(ctx context.Context, q workflow.RunQuery) (int, error)

CancelAll cancels every run matching q whose status is among workflow.CancelStatuses(q) (workflow.BulkCanceller), each exactly as Cancel would, and returns how many it cancelled. It lists the matches first and cancels them one by one, so a run that moved on in between (finished, or started when only waiting runs were asked for) is left alone and not counted.

func (*Client) CancelStep

func (c *Client) CancelStep(ctx context.Context, ref workflow.RunRef, step string) error

CancelStep stops one step of a run (workflow.StepCanceller). A running attempt has its context cancelled with cause workflow.ErrStepCancelled (at once on this client, at the next lease renewal on another), and its handler's return decides: an output completes the step and the run goes on; an error fails the step for good, without retry, matching ErrStepCancelled, and the run fails. A step waiting on its retry backoff, a snooze, a signal or a sleep fails that way at once. A step not started yet or already ended, or a run that is not running, returns ErrWrongState; an unknown run or step ErrNotFound.

func (*Client) Capabilities

func (c *Client) Capabilities() workflow.Caps

Capabilities returns every capability the conformance suite proves on this driver.

func (*Client) DB

func (c *Client) DB() *sql.DB

DB returns the pool the client was built with: the one escape hatch, for reading the tables directly (a dashboard, a report) or running Migrate inside a wider schema step.

func (*Client) DatabaseClock

func (c *Client) DatabaseClock() *DatabaseClock

DatabaseClock returns the clock WithDatabaseClock installed, or nil, so an app can read its Offset or give the same clock to a UniqueStore or a DebounceStore over the same database.

func (*Client) DebounceStore

func (c *Client) DebounceStore() *DebounceStore

DebounceStore returns the debounce store in this client's database, prefix and scheduling clock. Client.Migrate creates its table.

func (*Client) Delete

func (c *Client) Delete(ctx context.Context, ref workflow.RunRef) error

Delete forgets a finished run with its steps (workflow.RunDeleter). A run that is not finished returns ErrWrongState.

func (*Client) Done

func (c *Client) Done() <-chan struct{}

Done is closed once the client has fully stopped: after Drain, after the Start ctx was cancelled or Stop was called, or after WithStopWhenIdle fired. Before Start it is open.

func (*Client) Drain

func (c *Client) Drain(ctx context.Context) error

Drain stops the client gracefully (workflow.Drainer), as before a deploy: workers stop claiming at once, Drain waits until every attempt this client claimed has run and its outcome is written, or until ctx ends, then shuts the client down as cancelling the Start ctx would. An attempt cut off by ctx is handed back as ready and not charged an attempt (CapShutdownRequeue), so another client runs it at once; runs stay in the database. It returns nil when every attempt finished and ctx's error otherwise. Calling it again, or after the Start ctx ended, waits for the same shutdown; before Start it returns ErrNotStarted. It is DrainWithReport without the report.

func (*Client) DrainWithReport

func (c *Client) DrainWithReport(ctx context.Context) (workflow.DrainReport, error)

DrainWithReport is Drain that also returns what the drain finished and cut off (workflow.ReportingDrainer). Every cut-off attempt has the disposition DrainRequeued. The report is fixed when the deadline ends, before the client's context is cancelled. A second call returns the first call's report.

func (*Client) Enqueue

func (c *Client) Enqueue(ctx context.Context, job workflow.Job) (workflow.RunRef, error)

Enqueue starts a run now, or at job.RunAt. A key held by an unfinished run returns that run's RunRef and starts nothing; the unique index on (workflow, idem_active) makes that atomic across nodes.

func (*Client) EnqueueMany

func (c *Client) EnqueueMany(ctx context.Context, jobs []workflow.Job) ([]workflow.RunRef, error)

EnqueueMany starts jobs in one transaction (batch.ManyEnqueuer, which batch.EnqueueMany prefers): every run is written or none is, so a crash or an error never leaves half a bulk enqueue behind. It returns one RunRef per job, in order. Each job is checked as Enqueue checks it before anything is written; an error returns nil refs and starts nothing. A job whose idempotency key an unfinished run holds (one already in the table, or an earlier job of this call) gets that run's RunRef and starts nothing, as Enqueue does. Enqueue middleware of a wrapping client does not run, since the call bypasses the wrapper.

func (*Client) EnqueueTx

func (c *Client) EnqueueTx(ctx context.Context, tx *sql.Tx, job workflow.Job) (workflow.RunRef, error)

EnqueueTx writes a run inside the caller's transaction tx, so the run commits or rolls back with the app's own writes: a rolled-back tx leaves no run, a committed one is picked up by the workers of every client registering the workflow. It is the transactional outbox without a relay, for apps whose queue is this database.

tx must be open on the *sql.DB this client was built with (the tables are read and written through it), and the workflow must be registered here and the client started, as for Enqueue.

The run is written pending (or scheduled, for a future RunAt) and is not admitted inside tx: admission may take a workflow's lock row, which would then be held for as long as the app's transaction runs. Nothing can wake a worker before the commit, so a committed run is found by the dispatcher within WithPollInterval (default a second); call Wake after the commit to have this client admit it at once.

Idempotency keys follow the workflow's policy, as for Enqueue: a key held by an unfinished run (committed, or written earlier in tx) returns that run's RunRef and writes nothing. A key written by another transaction that has not committed yet makes this insert wait for it. The insert runs under a savepoint, so a duplicate key never aborts the caller's transaction. On MySQL the holder is read with a locking read (FOR SHARE), which sees rows committed after tx began under REPEATABLE READ; on Postgres use READ COMMITTED (its default), because a REPEATABLE READ transaction cannot see a holder that committed after its snapshot and fails instead.

Enqueue middleware does not run: it wraps workflow.Client.Enqueue, and what it does (a lock or counter in another store, a debounce timer, a trace export) would not roll back with tx. An app that needs middleware on a transactional enqueue writes through the outbox package instead, whose relay enqueues on any client, middleware included, after the commit.

An error leaves tx open; the caller decides whether to roll it back.

func (*Client) ForceComplete

func (c *Client) ForceComplete(ctx context.Context, ref workflow.RunRef) error

ForceComplete ends a run as succeeded from outside (workflow.RunForcer): an unfinished run, or a failed one. Steps still executing are cancelled with cause workflow.ErrForced and their results are ignored. It counts as a success: the idempotency key is released, Job.Chain hands on to its next job, and run-finished observers hear StatusSucceeded, a second time for a run that had failed. Any other state returns ErrWrongState; an unknown run ErrNotFound.

func (*Client) ForceFail

func (c *Client) ForceFail(ctx context.Context, ref workflow.RunRef, reason string) error

ForceFail ends an unfinished run as failed from outside (workflow.RunForcer), with workflow.ForcedFailure(reason) as its error: no retry, and OnFailure does not run (a run already failing has its OnFailure claim dropped). RunRetrier can run it again. A finished run returns ErrWrongState; an unknown run ErrNotFound.

func (*Client) Health

func (c *Client) Health(ctx context.Context) workflow.HealthReport

Health reports this client's liveness and readiness (workflow.HealthChecker): live unless it has stopped, ready when it has started (an enqueue-only client: at once) and the database answers a ping within ctx.

func (*Client) IndexLabels

func (c *Client) IndexLabels(ctx context.Context) (int, error)

IndexLabels writes the label index rows (migration 005's run_labels) of every run past the backfill watermark (rewriting rows a run already has), in transactions of a few hundred runs, and returns how many runs it indexed. Migrate calls it after applying the migrations, so most apps never call it. Call it after applying MigrationScripts with another tool, and once more after a rolling deploy in which a client older than 005 kept enqueueing: until then a label filter in Runs or CancelAll does not see those runs.

It is safe to run twice and from several processes at once: each chunk first takes the watermark row's lock, re-reads the watermark, rewrites the rows of the runs past it and moves it on, so a second caller waits and then finds the work done.

func (*Client) IsLeader

func (c *Client) IsLeader() bool

IsLeader reports whether this client may fire cron ticks and run housekeeping: always true without an elector.

func (*Client) JournalStore

func (c *Client) JournalStore() *JournalStore

JournalStore returns the journal store in this client's database and prefix. Client.Migrate creates its table.

func (*Client) Migrate

func (c *Client) Migrate(ctx context.Context) error

Migrate brings the schema up to this code's version: it applies, once each and in order, every migration the ledger has not recorded, under the engine's migration lock, so concurrent Migrate calls from several processes are safe (on SQLite the ledger begins its transactions IMMEDIATE, so a second caller waits rather than failing). It refuses with ErrSchemaTooNew a database that a newer release already migrated, and with migrate.ErrChecksumMismatch one whose recorded migration differs from this code's. It never drops anything, and Start never migrates: running it is the deploy's decision. It applies UniqueStore's and DebounceStore's lists too, so Client.UniqueStore and Client.DebounceStore need no Migrate of their own. Last it runs IndexLabels, the label index backfill of migration 005, which finds nothing to do once the index is current.

func (*Client) Nodes

func (c *Client) Nodes(ctx context.Context) ([]workflow.NodeInfo, error)

Nodes lists the clients of these tables that reported in within the last hour (workflow.NodeLister), by ID. A client reports every third of its lease while it runs and leaves the list when it stops cleanly; a crashed one stays until it ages out, so read LastSeen for liveness.

func (*Client) OnLifecycle

func (c *Client) OnLifecycle(fn workflow.LifecycleFunc) error

OnLifecycle registers fn to observe the lifecycle of runs on this client (workflow.LifecycleNotifier): Inserted for runs this client enqueued (Enqueue and EnqueueMany; not EnqueueTx, whose run exists only once the caller commits), Started, Retrying and Snoozed for attempts it ran, Progress its steps reported, and Finished for runs whose end it wrote. Best-effort and local to the process: another client's events are seen there. Observers run on the goroutine that caused the event, in order per run; a panic is recovered and logged. It must be called before Start.

func (*Client) OnRunFinished

func (c *Client) OnRunFinished(fn workflow.RunFinishedFunc) error

OnRunFinished registers fn to observe run ends (workflow.RunNotifier). Delivery is durable: a run's end is marked delivered only after every observer returned, so a crash before that delivers it again (at least once). In a cluster each run's end is delivered on one client that registers its workflow and has observers, not on every client.

func (*Client) OnTick

func (c *Client) OnTick(fn workflow.TickObserver) error

OnTick registers fn to observe every schedule tick this client fires, skips or drops (workflow.TickNotifier): TickFired with the run, TickSkipped for a tick a window, ScheduleWhen or a schedule pause held back, and TickMisfired for the ticks a late wake dropped. Each tick is reported by the one node that fired or skipped it, after its transaction committed; a crash between the commit and the call loses the report (best effort, like RunNotifier's contract allows). Register before Start.

func (*Client) OutputStore

func (c *Client) OutputStore(lim OutputLimits) *OutputStore

OutputStore returns the output store in this client's database and prefix, bounded by lim. Client.Migrate creates its table, and the client deletes a run's output with the run.

func (*Client) Pause

func (c *Client) Pause(ctx context.Context, t workflow.PauseTarget) error

Pause holds the runs t targets (workflow.Pauser): from the next claim on, on every client of this database, no step attempt of a held run starts. The pause is a row, so it survives restarts; Pause on a paused target is a no-op. It works before Start too.

func (*Client) PauseSchedule

func (c *Client) PauseSchedule(ctx context.Context, t workflow.ScheduleTarget) error

PauseSchedule stops a schedule, or with All every schedule (including ones made later), from firing without removing it (workflow.SchedulePauser). Its due ticks are reported as TickSkipped with TickReasonPaused, once each, by the node that sees them. The pause is a row, so every node obeys it and it survives restarts. An unknown ID returns ErrNotFound.

func (*Client) Paused

func (c *Client) Paused(ctx context.Context) ([]workflow.PauseTarget, error)

Paused lists the Pauser targets in force: All first, then workflows, then queues, each by name.

func (*Client) PayloadLimit

func (c *Client) PayloadLimit() int

PayloadLimit is WithPayloadLimit's bound (workflow.PayloadLimiter); 0: none.

func (*Client) Prune

func (c *Client) Prune(ctx context.Context) (PruneResult, error)

Prune deletes the finished runs the retention allows, in chunks of WithPruneChunk, each through the archive hook first, and reports what it deleted and what each guard held back. Each run's steps and consumed-signal markers go with it. It never deletes a run that still holds an idempotency key, whose end is not delivered (unless Retention.Undelivered), or that belongs to a batch in this client's BatchStore that is still open (State.Open). Without WithRetention it deletes nothing. It needs Migrate, not Start, so an app can run it from its own scheduler (with WithoutAutoPrune).

func (*Client) QueueLoads

func (c *Client) QueueLoads(ctx context.Context) ([]workflow.QueueLoad, error)

QueueLoads reports the load of every queue given to WithQueues on this client, sorted by name (workflow.WorkerScaler). Depth and OldestWait count the steps of this client's workflows that are ready and due on the queue, whichever client enqueued them, since any of them is work this client's workers could take; a pause holding them still counts them, as they are waiting. OldestWait runs from when a step became claimable (its due time) on the scheduling clock. Workers and Running are this client's own: its cap (the pool size, WithWorkers, for a queue without one) and the attempts executing here. Without WithQueues the list is empty.

func (*Client) Register

func (c *Client) Register(wf workflow.Workflow) error

Register validates wf and stores it. Cron expressions are parsed here so a typo fails now, not at the first fire.

func (*Client) Reschedule

func (c *Client) Reschedule(ctx context.Context, ref workflow.RunRef, at time.Time) (workflow.RunRef, error)

Reschedule moves a scheduled run's start to at (workflow.RunRescheduler). The RunRef, payload, labels and key stay; RunInfo.RunAt becomes at; a zero or past at starts it now. A run that is not scheduled returns ErrWrongState, an unknown one ErrNotFound.

func (*Client) Result

func (c *Client) Result(ctx context.Context, ref workflow.RunRef) ([]byte, error)

Result returns the output of a succeeded run (workflow.ResultReader): workflow.RunOutput of its final steps, stored when it succeeded. A run that has not succeeded returns ErrWrongState; an unknown one ErrNotFound.

func (*Client) Resume

func (c *Client) Resume(ctx context.Context, t workflow.PauseTarget) error

Resume removes the pause of t; Resume(workflow.PauseAll()) removes every Pauser pause. Resuming a target that is not paused is a no-op. Held runs are claimed at once on this client and within the poll interval on the others.

func (*Client) ResumeSchedule

func (c *Client) ResumeSchedule(ctx context.Context, t workflow.ScheduleTarget) error

ResumeSchedule lets a paused schedule fire again; with All it clears every schedule pause. The ticks that fell due during the pause never fire: they are reported skipped (if no dispatcher pass did so already) and the schedule resumes at its next tick. Resuming a schedule that is not paused is a no-op; an unknown ID returns ErrNotFound.

func (*Client) Retry

func (c *Client) Retry(ctx context.Context, ref workflow.RunRef) (workflow.RunRef, error)

Retry re-runs a failed run from the step that failed (workflow.RunRetrier). The steps that succeeded keep their outputs; every other step goes back to pending with fresh attempt counts, and the run goes back to queued, keeping its RunID. It is admitted again under the workflow's concurrency and rate limits, by this client when it registers the workflow and is started, else by the next client that does. Its idempotency key is taken again (ErrDuplicate when another unfinished run holds it), and its end is delivered to run-finished observers again. A run that is not failed returns ErrWrongState.

func (*Client) Runs

Runs lists the runs of every workflow in this database (not only those registered here) that match q, newest enqueue first, ties by RunID descending (workflow.RunQuerier). Ref, workflow, queue, enqueue-time, the stored half of the status filter and the label index (run_labels, whose cut values only narrow) are SQL; the exact labels and the exact status of an admitted run (queued, running or waiting, from its steps) are checked in Go, reading on until the page is full. Payload is the stored payload. The cursor is opaque; one this driver did not issue wraps ErrInvalid.

func (*Client) Schedule

Schedule registers a recurring trigger, stored so every client running the workflow fires it and each tick fires once. Its Location must be loadable by name on every node (time.LoadLocation) or be a fixed offset; time.Local is refused because it differs from machine to machine. Its Misfire policy is stored with it (the zero policy is this driver's default: see the package doc).

A schedule with a Name is upserted by (Workflow, Name): calling Schedule again with the same name, on any node or after a reboot, returns the same ScheduleID and updates that one row instead of adding a second, so an app can declare its schedules at every boot. The update replaces the cron, zone, input, labels and misfire policy and makes the schedule active again (after an Unschedule too); when the cron and zone are unchanged and the schedule was active, its next fire time is kept, so a reboot neither skips nor repeats a tick. Two nodes declaring one name at once end with one row. A schedule without a Name is a new schedule on every call, with an ID from the ID generator. Rows made by a version before named upserts keep firing until they are Unscheduled.

func (*Client) Schedules

func (c *Client) Schedules(ctx context.Context) ([]workflow.ScheduleInfo, error)

Schedules lists the live schedules of every workflow in this database, declared and made with Schedule, sorted by Workflow, Name, then ID (workflow.ScheduleLister). An unscheduled schedule, or a declared one whose workflow dropped the expression, is not listed. Next is the stored next fire time; Overlap and Last (the last fire, tick or trigger, its run and that run's status now) come from schedule_last.

func (*Client) SetWorkers

func (c *Client) SetWorkers(ctx context.Context, queue string, n int) error

SetWorkers sets the worker cap of a queue given to WithQueues on this client (workflow.WorkerScaler). The cap is within the pool (WithWorkers): a cap above the pool size cannot run more than the pool at once, so give the pool at least the balancer's budget. Lowering it lets running attempts finish; raising it wakes idle workers at once. The cap is this client's alone and is not stored: another client keeps its own, and a restart starts from WithQueues again.

func (*Client) Signal

func (c *Client) Signal(ctx context.Context, ev workflow.Event) error

Signal stores ev and resumes every run of a workflow registered here that waits on its name and key. Runs of workflows registered on other clients take it on those clients' next poll; a run that reaches its wait later takes it then (the look-back Client.Signal documents).

func (*Client) Start

func (c *Client) Start(ctx context.Context) error

Start checks every workflow against Capabilities, refuses expression keys, checks that Migrate has run, records the declared cron schedules and then starts the dispatcher and the workers. It does not block. Cancelling ctx stops the client; runs stay in the database for the next client, and steps this client was executing are released so another client runs them at once.

func (*Client) Status

func (c *Client) Status(ctx context.Context, ref workflow.RunRef) (workflow.Status, error)

Status reports where a run is. An admitted run is Running while a step is claimed, Retrying while a step waits on its retry backoff, Queued while a step waits for a worker, and Waiting while only waits and sleeps are in progress (stepStatus).

func (*Client) Stop

func (c *Client) Stop()

Stop shuts the client down and waits until every goroutine it started has exited. Runs are left as they are in the database. It is safe to call more than once, and before Start (a no-op).

func (*Client) StopCause

func (c *Client) StopCause() error

StopCause returns the *workflow.RecycleError when a WithRecycle limit stopped the client, and nil otherwise; see workflow.StopCauser.

func (*Client) TriggerSchedule

func (c *Client) TriggerSchedule(ctx context.Context, id workflow.ScheduleID) (workflow.RunRef, error)

TriggerSchedule runs schedule id now (workflow.ScheduleTriggerer): one run with the schedule's input and labels plus workflow.TriggeredLabels. It obeys the schedule's overlap policy (an unfinished previous run refuses it with an error matching workflow.ErrOverlap, enqueueing nothing), ignores its windows and Workflow.ScheduleWhen, works while the schedule is paused, becomes the schedule's last fire and leaves its next fire time as it was. An unknown or unscheduled id returns ErrNotFound, and so does a schedule whose workflow is not registered on this client. Tick observers hear a TickFired with Manual set.

func (*Client) UniqueStore

func (c *Client) UniqueStore() *UniqueStore

UniqueStore returns the lock store in this client's database, prefix and scheduling clock. Client.Migrate creates its table.

func (*Client) Unschedule

func (c *Client) Unschedule(ctx context.Context, id workflow.ScheduleID) error

Unschedule stops a schedule. An empty id, or one already removed, is a no-op; an id never issued returns ErrNotFound. The row stays, inactive, so the second call can tell "removed" from "never existed".

func (*Client) Wake

func (c *Client) Wake()

Wake makes this client look for work at once instead of at its next poll: call it after committing a transaction that used EnqueueTx, so the run is admitted without waiting up to WithPollInterval. It never blocks and is a no-op on a client that is not running. Other clients of the cluster find the run at their next poll, or at once when the client has a Notifier (WithNotifier), which Wake signals too.

type DatabaseClock

type DatabaseClock struct {
	// contains filtered or unexported fields
}

DatabaseClock is a clock.Clock whose Now is the database server's clock: one time source for every client of the tables, so due times, leases and cron fires compare against the same "now" however far the nodes' own clocks have drifted apart.

It never queries on Now. Sync reads the database's time once and records it against the local clock; Now is that reading plus the local time elapsed since, which on clock.Real is the monotonic clock, so a jump of this machine's wall clock (an NTP step, a manual change) moves nothing. The reading is taken as the midpoint of the round trip, so it is off by at most half the round trip (Uncertainty). Before the first Sync, Now is the local clock.

Timers run on the local clock: a duration is a duration on any clock. Postgres reads clock_timestamp(), MySQL UTC_TIMESTAMP(6) and SQLite julianday('now'); a SQLite file lives on one machine, so there it only makes every process on that machine agree, which they already do.

func NewDatabaseClock

func NewDatabaseClock(db *sql.DB, d sqlb.Dialect) (*DatabaseClock, error)

NewDatabaseClock returns a clock over db, whose SQL dialect is d (Postgres, SQLite or MySQL). It reads nothing until Sync.

func (*DatabaseClock) NewTimer

func (k *DatabaseClock) NewTimer(d time.Duration) clock.Timer

NewTimer runs on the local clock.

func (*DatabaseClock) Now

func (k *DatabaseClock) Now() time.Time

Now is the database's time: the last reading plus the local time elapsed since. Before the first Sync it is the local clock's.

func (*DatabaseClock) Offset

func (k *DatabaseClock) Offset() time.Duration

Offset is how far the database's clock is ahead of the local wall clock as of the last Sync (negative: behind); zero before the first Sync.

func (*DatabaseClock) Sync

func (k *DatabaseClock) Sync(ctx context.Context) error

Sync reads the database's time and makes it the clock's reference. It takes a connection from the pool, so never call it while holding one of the pool's transactions on a pool that may have a single connection.

func (*DatabaseClock) Uncertainty

func (k *DatabaseClock) Uncertainty() time.Duration

Uncertainty bounds the error of the last reading: half its round trip.

type DebounceStore

type DebounceStore struct {
	// contains filtered or unexported fields
}

DebounceStore is a durable middleware.DebounceStore: each debounce key's pending burst is a row of <prefix>debounce, so middleware.Debounce debounces a key enqueued from every process sharing the database (a web tier with several replicas), not one burst per process. Like UniqueStore it needs no running Client: build it with NewDebounceStore over any *sql.DB (an app on Hatchet with a database), or take Client.DebounceStore.

mw, err := middleware.Debounce(c, nil, byProduct, 10*time.Second, time.Minute,
	middleware.WithDebounceStore(store))

Swap is atomic across processes: it first bumps the key's row, which locks it (a row lock on Postgres and MySQL, the single writer on SQLite), then reads the burst it replaces and writes the new one in the same transaction. When the key has no row, the insert's primary key decides: of two processes inserting at once, the loser's transaction is tried again and finds the winner's row, so every replaced run is returned to exactly one caller and is cancelled once.

Rows whose run has started (RunAt passed) are over; Get and Swap still return them (Debounce treats a past RunAt as no burst), and a running Client's dispatcher deletes them, or PruneExpired for an app with no Client.

This package imports middleware only for the DebounceBurst type the interface names; middleware imports nothing of sqlstore.

func NewDebounceStore

func NewDebounceStore(db *sql.DB, d sqlb.Dialect, opts ...Option) (*DebounceStore, error)

NewDebounceStore returns the debounce store over db for dialect d. Of the Options, WithTablePrefix, WithSchema, WithClock (pruning), WithTxAttempts and WithErrorClassifier apply; the others configure a Client and are ignored. Call Migrate before the first Swap.

func (*DebounceStore) Get

Get implements middleware.DebounceStore.

func (*DebounceStore) Migrate

func (s *DebounceStore) Migrate(ctx context.Context) error

Migrate creates the debounce table (its own migration list, under the same ledger and lock as Client.Migrate), and nothing else.

func (*DebounceStore) PruneExpired

func (s *DebounceStore) PruneExpired(ctx context.Context) (int64, error)

PruneExpired deletes every burst whose run has started and returns how many. Such rows never change what Debounce does, so this only bounds the table's size; a running Client does it on its dispatcher's housekeeping.

func (*DebounceStore) Swap

Swap implements middleware.DebounceStore, atomically across processes (see DebounceStore). A transaction that loses the first-insert race or a lock conflict is tried again, up to WithTxAttempts.

type Elector

type Elector interface {
	IsLeader() bool
}

Elector answers whether this process is the leader of a cluster role. A client given one (WithElector, WithLeaderElection) fires cron ticks and runs the dispatcher's housekeeping only while IsLeader is true; every other node still claims and runs steps. IsLeader is called on every dispatcher pass, so it must answer from memory (campaigning happens elsewhere) and be safe for concurrent use. LeaderElector is the SQL one; a Redis lock, a Kubernetes Lease or a Consul session fits the same way.

type ErrorClass

type ErrorClass struct {
	// UniqueViolation means a unique constraint refused the write. The
	// driver's correctness guards are unique indexes (an idempotency key
	// held twice, a cron tick fired twice, a lock row inserted twice), so
	// this is how a lost race is told from a real failure. Saying true for
	// an error that is not one makes the driver treat a failure as "someone
	// else won", so a classifier says it only when sure.
	UniqueViolation bool
	// Retryable means the transaction lost a lock race (a deadlock, a
	// serialization failure, SQLite busy) and is tried again from the
	// start, up to WithTxAttempts.
	Retryable bool
	// Unavailable means the database could not be reached or dropped the
	// connection (refused, reset, shut down, out of connections), as
	// opposed to refusing the statement. The driver marks such errors with
	// workflow.Unavailable, so errors.Is(err, workflow.ErrUnavailable)
	// holds and a failover client can tell an outage from a bad request.
	Unavailable bool
}

ErrorClass is what an ErrorClassifier says about one database error. The zero value, "neither", is the safe answer: the error reaches the caller as a failure.

func DefaultErrorClassifier

func DefaultErrorClassifier(err error) ErrorClass

DefaultErrorClassifier recognises the errors of the drivers sqlb's contrib modules open: Postgres by SQLSTATE (23505 unique; 40P01 and 40001 retryable; class 08 and 57P01-57P03 unavailable), MySQL by "Error 1062" (unique), "Error 1213"/"Error 1205" (retryable) and "invalid connection"/"Error 1040"/"Error 1053" (unavailable), SQLite by "UNIQUE constraint failed" (unique) and "SQLITE_BUSY"/"database is locked" (retryable). On any engine a *net.OpError, driver.ErrBadConn, sql.ErrConnDone or a pgx "failed to connect to" dial error is unavailable.

type ErrorClassifier

type ErrorClassifier func(err error) ErrorClass

ErrorClassifier sorts a database error the driver got back. It is called only with a non-nil error, which may be wrapped (use errors.As), and must be safe for concurrent use. DefaultErrorClassifier is the default; a classifier for a driver with other error text usually checks its own forms and falls back to DefaultErrorClassifier:

sqlstore.WithErrorClassifier(func(err error) sqlstore.ErrorClass {
	var e *mydriver.Error
	if errors.As(err, &e) {
		return sqlstore.ErrorClass{UniqueViolation: e.Code == mydriver.Dup, Retryable: e.Code == mydriver.Busy}
	}
	return sqlstore.DefaultErrorClassifier(err)
})

type FinishedRun

type FinishedRun struct {
	Ref        workflow.RunRef
	Workflow   string
	Status     workflow.Status
	Labels     map[string]string
	Payload    []byte
	Err        string // the stored error text; empty when it succeeded
	CreatedAt  time.Time
	FinishedAt time.Time
	// Outputs holds each step's stored output by step name; a step that
	// never succeeded is absent.
	Outputs map[string][]byte
}

FinishedRun is one run Prune is about to delete, as the archive hook sees it.

type IDGenerator

type IDGenerator func(kind IDKind) (string, error)

IDGenerator returns a new ID of kind. IDs must be unique per kind across every client of the database: a duplicate fails the write that uses it (each ID column is a primary or unique key). An ID is 1 to 64 bytes; the driver refuses anything else, and refuses the operation when the generator returns an error, so a broken generator fails closed rather than writing a bad row. Use it for time-sortable IDs (UUIDv7, ULID) or IDs that line up with the app's own:

sqlstore.WithIDGenerator(func(kind sqlstore.IDKind) (string, error) {
	id, err := uuid.NewV7()
	return string(kind) + "_" + id.String(), err
})

type IDKind

type IDKind string

IDKind names what a generated ID identifies, so one IDGenerator can shape each kind differently. The values are the prefixes the default uses.

const (
	// IDRun is a run's RunID.
	IDRun IDKind = "run"
	// IDScheduled is the ScheduledID of a run enqueued with a future RunAt.
	IDScheduled IDKind = "sched"
	// IDStep is a step row's ID (StepInfo carries it).
	IDStep IDKind = "step"
	// IDSchedule is the ScheduleID of a Client.Schedule without a Name. A
	// named schedule's ID is derived from its workflow and name, so the
	// same name is the same row on every node and every boot.
	IDSchedule IDKind = "sch"
)

The kinds of ID the driver asks an IDGenerator for. Lease and failure tokens are not among them: they are internal, never shown to a caller, and always 128 random bits.

type JournalStore

type JournalStore struct {
	// contains filtered or unexported fields
}

JournalStore is a durable workflow.JournalStore: a run's memoised results (workflow.Memo) are rows of <prefix>journal, so a step retried after the process died, on any node sharing the database, gets its stored results back. Like UniqueStore it needs no running Client: build it with NewJournalStore over any *sql.DB (an app on Hatchet with a database), or take Client.JournalStore.

plugin := workflow.Journal(store)

Save of an existing key replaces it, in one transaction (an update, then an insert when no row was there; a concurrent first Save of one key is decided by the primary key and the loser's transaction is tried again, up to WithTxAttempts).

func NewJournalStore

func NewJournalStore(db *sql.DB, d sqlb.Dialect, opts ...Option) (*JournalStore, error)

NewJournalStore returns the journal store over db for dialect d. Of the Options, WithTablePrefix, WithSchema, WithTxAttempts and WithErrorClassifier apply; the others configure a Client and are ignored. Call Migrate before the first Save.

func (*JournalStore) Forget

func (s *JournalStore) Forget(ctx context.Context, run workflow.RunID) error

Forget implements workflow.JournalStore.

func (*JournalStore) Load

func (s *JournalStore) Load(ctx context.Context, run workflow.RunID, key string) ([]byte, bool, error)

Load implements workflow.JournalStore.

func (*JournalStore) Migrate

func (s *JournalStore) Migrate(ctx context.Context) error

Migrate creates the journal table (its own migration list, under the same ledger and lock as Client.Migrate), and nothing else.

func (*JournalStore) Save

func (s *JournalStore) Save(ctx context.Context, run workflow.RunID, key string, val []byte) error

Save implements workflow.JournalStore.

type LeaderElector

type LeaderElector struct {
	// contains filtered or unexported fields
}

LeaderElector elects one leader per role among the processes sharing a set of tables, through a lease row in <prefix>leaders: the holder renews it every third of its lease; when it stops (a crash, a network split), another node takes the role once the lease and the skew margin have passed. Each change of holder increments Term, a fencing token.

A leader fences itself: IsLeader turns false once a whole lease has passed on its own (monotonic) lease clock since its last renewal began, before any other node can take over, so two nodes never both believe they lead for longer than the clocks' disagreement beyond the skew margin. Use it for work that must not run twice at once; where a duplicate is merely wasteful, a unique row (the cron tick table) is simpler and survives a split brain.

func NewLeaderElector

func NewLeaderElector(db *sql.DB, d sqlb.Dialect, role, node string, opts ...LeaderOption) (*LeaderElector, error)

NewLeaderElector returns an elector for role, campaigning as node (an id unique among the nodes, such as workflow.DefaultNodeID()), over the leader table a Client's Migrate created in db. It reads nothing until Campaign or Run.

func (*LeaderElector) Campaign

func (e *LeaderElector) Campaign(ctx context.Context) (bool, error)

Campaign makes one attempt to take or renew the role and reports whether this node leads after it. Run calls it every third of the lease.

func (*LeaderElector) IsLeader

func (e *LeaderElector) IsLeader() bool

IsLeader reports whether this node holds the role: its last campaign won and less than a whole lease has passed since that renewal began.

func (*LeaderElector) Resign

func (e *LeaderElector) Resign(ctx context.Context) error

Resign gives the role up at once, so another node takes it at its next campaign instead of after the lease.

func (*LeaderElector) Run

func (e *LeaderElector) Run(ctx context.Context)

Run campaigns every third of the lease until ctx ends, then resigns. Errors are logged through the Client's logger when the client runs it, and otherwise only change IsLeader.

func (*LeaderElector) Term

func (e *LeaderElector) Term() int64

Term is the fencing token of the leadership this node last won (0: never won). It grows each time the role changes hands.

type LeaderOption

type LeaderOption func(*LeaderElector)

LeaderOption configures a LeaderElector.

func LeaderClock

func LeaderClock(c clock.Clock) LeaderOption

LeaderClock sets the clock leases are measured on (default clock.Real; a DatabaseClock makes every node compare against the database's time).

func LeaderLease

func LeaderLease(d time.Duration) LeaderOption

LeaderLease sets how long a leader holds its role without renewing (default 15 seconds). A shorter lease hands the role on sooner after a crash and costs more writes. Values of zero or less are ignored.

func LeaderSchema

func LeaderSchema(name string) LeaderOption

LeaderSchema puts the leader table in a Postgres schema (WithSchema).

func LeaderSkew

func LeaderSkew(d time.Duration) LeaderOption

LeaderSkew sets the margin other nodes wait past a lease's end before taking the role (see WithLeaseSkew); default zero.

func LeaderTablePrefix

func LeaderTablePrefix(p string) LeaderOption

LeaderTablePrefix and LeaderSchema name the tables, as WithTablePrefix and WithSchema do for a Client; they must match the client that migrated them.

type MigrationScript

type MigrationScript struct {
	Owner, Name string
	Statements  []string
}

MigrationScript is one migration as SQL text: the exact statements Migrate runs for it, in order, with table names resolved for the options' prefix and schema. Owner and Name are what Migrate records in its ledger.

func MigrationScripts

func MigrationScripts(d sqlb.Dialect, opts ...Option) ([]MigrationScript, error)

MigrationScripts returns every migration Client.Migrate applies, in order, as SQL text for dialect d and the table options among opts (WithTablePrefix, WithSchema; the others are ignored): sqlstore's list (owner MigrationOwner), then UniqueStore's (UniqueMigrationOwner) and DebounceStore's (DebounceMigrationOwner); take only a store's own list for a database that holds just that store. It is for teams whose DDL goes through goose, Atlas, Flyway, Liquibase or a DBA review, or whose app user has no DDL rights: apply the statements in your own pipeline, recording each (Owner, Name) there.

The text is frozen per release exactly as Migrate's is, and every statement is safe to run twice (IF NOT EXISTS forms). Start does not need Migrate's ledger: it checks that every table is readable, so a schema applied this way starts as is. Migration 005's backfill is Go, not SQL: run Client.IndexLabels once after applying it. Migrate is still safe to run later on such a database: it re-runs each script (a no-op) and records it. A release that adds a migration adds it here too, and Start fails on its missing table until it is applied.

type Notifier

type Notifier interface {
	// Notify tells every client listening on channel that work may be due.
	// The client calls it from one background goroutine, coalescing
	// bursts, with a bounded context; an error is logged and dropped.
	Notify(ctx context.Context, channel string) error
	// Listen subscribes to channel and calls wake for every notification
	// until ctx ends, then returns. It may call wake spuriously (once after
	// subscribing is recommended, to cover the gap before it). Any other
	// return is a lost subscription: the client logs it and calls Listen
	// again after its poll interval (at most a second), polling meanwhile.
	Listen(ctx context.Context, channel string, wake func()) error
}

Notifier carries wake-ups between the clients of one database, so an idle worker on another machine starts a run as soon as it is enqueued instead of at its next poll. It is a latency tool only: every client still polls (WithPollInterval), so a lost, late or duplicated wake-up changes when work starts, never whether it runs.

The sqlstore/pgnotify package implements it on Postgres LISTEN/NOTIFY. Any other bus (Redis pub/sub, NATS, an in-process fan-out in tests) fits the same two methods.

type Option

type Option func(*Client)

Option configures a Client.

func WithArchive

func WithArchive(fn ArchiveFunc) Option

WithArchive sets the hook each chunk of runs passes through before it is deleted.

func WithArchiveTable

func WithArchiveTable() Option

WithArchiveTable makes Prune move finished runs into the cold tables (<prefix>runs_archive and <prefix>steps_archive) instead of dropping them: the queue's hot tables stay as small as retention keeps them while the history stays in the database, readable with ArchivedRuns. The move is one transaction per chunk, so a run is in exactly one of the two places. It works with WithArchive (whose hook still sees each chunk first) and needs WithRetention, which decides what is moved and when. The cold tables have no retention of their own: trim them with the app's own DELETE, or partition them, as the history's owner decides.

func WithClock

func WithClock(c clock.Clock) Option

WithClock sets the time source of every due time: delayed starts, sleeps, wait deadlines, retry backoff, step timeouts, rate-limit windows and cron fires. The default is clock.Real; tests pass clock.NewFake.

func WithCronParser

func WithCronParser(p cron.Parser) Option

WithCronParser replaces the cron parser behind Workflow.Cron and Schedule.Cron (cron.Parse by default), for a dialect the standard one lacks: the extended "L", "W" and "#" fields, a seconds field, hashed minutes. Stored expressions are parsed again when they fire, so every client of one database must use the same parser. It is opt-in and gives up "the same spec on every driver" for the expressions only this parser understands; see cron.Parser. Nil is ignored.

func WithDatabaseClock

func WithDatabaseClock(resync time.Duration) Option

WithDatabaseClock makes the database server's clock the time source of every due time and lease of this client (see DatabaseClock), in place of WithClock and WithLeaseClock, which it overrides whatever the order of options: the nodes of a cluster then agree on "now" even when their own clocks do not. Start reads the database's clock before anything else (and fails when it cannot), then again every resync on a running client; zero or less means one minute. An enqueue-only client reads it at Start.

func WithElector

func WithElector(e Elector) Option

WithElector makes this client fire cron ticks and run housekeeping only while e says it leads (see Elector). Without it (the default) every client fires and the unique tick row picks one fire per tick, which needs no election and survives a split brain; prefer that unless the fire itself must happen on one known node. The tick row stays as the second guard either way. Steps run on every client regardless. Nil is ignored.

func WithEnqueueOnly

func WithEnqueueOnly() Option

WithEnqueueOnly makes a client that writes work for other clients to run (workflow.CapEnqueueOnly): a web node enqueues, worker nodes over the same tables run. Enqueue, EnqueueMany, EnqueueTx, Status, Cancel and Signal work without Start, and Start returns nil and starts nothing: no worker, dispatcher, cron firing or run-end delivery runs on it. Stop makes later calls return ErrNotStarted and closes Done.

Register is optional. A registered workflow's jobs are checked and keyed as on any client (workflow idempotency, concurrency and rate keys are evaluated here) and, for a workflow with no concurrency or rate policy, admitted at once. A job of a workflow not registered here passes only the checks that need no definition (workflow.CheckJob); its workflow-level idempotency policy does not apply (Job.IdempotencyKey still does), and its concurrency key, rate keys and default queue are evaluated by the first worker that admits it.

With a Notifier (WithNotifier) each enqueue notifies the workers before it returns, since there is no background sender on this client.

func WithErrorClassifier

func WithErrorClassifier(f ErrorClassifier) Option

WithErrorClassifier sets how database errors are sorted into lost unique races and retryable lock conflicts (see ErrorClassifier). The default, DefaultErrorClassifier, knows the drivers sqlb's contrib modules open; a driver, proxy or wrapper whose errors read differently needs its own, or a lost idempotency race comes back as an error and a deadlock is not retried. A nil classifier is ignored.

func WithErrorHistory

func WithErrorHistory(n int) Option

WithErrorHistory bounds RunInfo.Errors, the history of failed attempts each run keeps (workflow.CapErrorHistory): the newest n, oldest first. Without it a run keeps workflow.DefaultErrorHistory; n <= 0 turns the history off (nothing is written, and rows written before stay until the run is deleted). Every failed attempt is recorded, retried, final or discarded; an attempt cut short by a clean stop is not a failure and is not recorded.

func WithFetchBatch

func WithFetchBatch(n int) Option

WithFetchBatch sets how many rows one dispatcher query reads (default 100): pending runs to admit, delayed runs to start, due sleeps, waits and signals, chains to hand on, schedules to fire, failing runs and run ends. A larger batch drains a backlog in fewer round trips; a smaller one keeps each pass and its transactions short. Workers claim one step per query by design: a claim leases its step, so claiming steps no idle worker can start would hold them from every other node. Values below 1 are ignored.

func WithHousekeepingEvery

func WithHousekeepingEvery(passes int) Option

WithHousekeepingEvery sets how many dispatcher passes run between two clean-ups of consumed signals, consumed-signal markers, old tick rows, expired uniqueness locks and finished debounce bursts (default 50). Lower it when signals or unique locks pile up faster than that; raise it to spend fewer queries on a quiet database. Values below 1 are ignored. Retention of finished runs has its own cadence (WithPruneInterval).

func WithIDGenerator

func WithIDGenerator(g IDGenerator) Option

WithIDGenerator sets how run, step, delayed-start and unnamed schedule IDs are made (see IDGenerator). The default is DefaultIDGenerator, "<kind>_<32 hex>". A nil generator is ignored.

func WithJitterSource

func WithJitterSource(rnd func() float64) Option

WithJitterSource sets where Backoff.Jitter draws its uniform values in [0, 1), so a test can pin them. The default is math/rand/v2. It must be safe for concurrent use. Nil is ignored.

func WithLeaderElection

func WithLeaderElection(node string, opts ...LeaderOption) Option

WithLeaderElection is WithElector with a LeaderElector over this client's own tables, for the role "scheduler", as node (empty: workflow.DefaultNodeID()), on the client's lease clock and WithLeaseSkew. The client campaigns from Start and resigns when it stops. opts tune the elector (LeaderLease); its tables follow the client's.

func WithLease

func WithLease(d time.Duration) Option

WithLease sets how long a claimed step stays claimed without renewal; a dead worker's step is run again by another client after at most this long. Values of zero or less are ignored.

func WithLeaseClock

func WithLeaseClock(c clock.Clock) Option

WithLeaseClock sets the clock leases are measured on. It is separate from WithClock because a lease is about a worker being alive, which is wall time: a test that moves a fake scheduling clock by an hour must not make every running step look abandoned. The default is clock.Real.

func WithLeaseSkew

func WithLeaseSkew(d time.Duration) Option

WithLeaseSkew sets the margin other clients wait past a lease's end before they take its step over, for clocks that disagree: a holder whose clock runs behind writes lease ends that look expired early to a node whose clock runs ahead. Set it to the largest clock difference expected between nodes (with WithDatabaseClock, a little above the database clock's Uncertainty). The holder stops its own attempt once a lease has gone unrenewed for the whole lease on its monotonic clock (ErrLeaseLost), so the margin is the gap between it stopping and another node starting. The same margin applies to the OnFailure, run-end delivery and prune leases. The default is zero; negative values are ignored.

func WithLogger

func WithLogger(l *slog.Logger) Option

WithLogger sets where the driver reports what it cannot return to a caller (a failed poll, an OnFailure error, a cron fire that could not enqueue). The default discards.

func WithMaxReclaims

func WithMaxReclaims(n int) Option

WithMaxReclaims sets how often one attempt of a step may be taken over after its worker died before it fails with ErrAbandoned (then the retry policy applies, and without retries left the run fails). Zero fails the attempt at its first takeover. Negative values are ignored.

func WithMiddleware

func WithMiddleware(mw ...workflow.Middleware) Option

WithMiddleware wraps every step handler and OnFailure handler. The first middleware is the outermost.

func WithNodeID

func WithNodeID(id string) Option

WithNodeID names this client in the cluster (StepInfo.Node, the error history, RunInfo.AttemptedBy, Nodes). The default is workflow.DefaultNodeID(), "host:pid". Empty is ignored.

func WithNotifier

func WithNotifier(n Notifier) Option

WithNotifier wakes idle clients of the same tables on other machines when this client makes work due (an enqueue, a finished step, a retry, a resume), and wakes this client when another one does. Without it, which is the default, a client finds another node's work at its next poll. Clients over different table prefixes or schemas use different channels.

func WithPayloadLimit

func WithPayloadLimit(n int) Option

WithPayloadLimit refuses, at Enqueue, EnqueueTx and EnqueueMany, a job whose payload (or that of any job in its Chain) is over n bytes, with a *workflow.PayloadTooLargeError; the database has no limit of its own worth enforcing. Zero or less (the default) sets none.

func WithPollInterval

func WithPollInterval(d time.Duration) Option

WithPollInterval sets how often the client looks for work that another node made due (a step another client queued, a signal for a run this client did not start). Values of zero or less are ignored.

func WithPruneChunk

func WithPruneChunk(n int) Option

WithPruneChunk sets how many finished runs one Prune transaction deletes (and the archive hook receives at once); default 100. Smaller chunks hold row locks for less time; larger ones finish a big backlog in fewer round trips. Values below 1 are ignored.

func WithPruneInterval

func WithPruneInterval(d time.Duration) Option

WithPruneInterval sets how often the background prune runs. Values of zero or less are ignored.

func WithQueues

func WithQueues(qs ...workflow.QueueConfig) Option

WithQueues makes this client serve only the named queues, each with its own worker cap and priority (workflow.QueueConfig). Runs of other queues stay in the database for a client that serves them, which is how separate worker processes per queue are deployed. Workers caps how many attempts of one queue run at once on this client, within WithWorkers; a free worker takes the ready attempts of a higher Priority queue before any lower one, then Job.Priority, then arrival. Without it the client serves every queue from its one pool. New refuses an invalid list (workflow.ValidateQueues).

func WithRecycle

func WithRecycle(r workflow.Recycle) Option

WithRecycle makes the client stop itself once a limit in r is reached (finished attempts, time since Start on the scheduling clock, memory), so a supervisor can replace the process; see workflow.Recycle. The client then drains as Drain does: it claims nothing more, lets the attempts it holds finish and write their outcomes, then stops, its Done channel closes and StopCause returns the *workflow.RecycleError (workflow.Serve returns it). Runs it never claimed stay queued in the database for the next client. Start refuses an invalid r (workflow.ValidateRecycle).

func WithRetention

func WithRetention(r Retention) Option

WithRetention turns on deletion of finished runs per r (see Retention).

func WithSchedulerOnly

func WithSchedulerOnly() Option

WithSchedulerOnly makes a client that fires schedules and moves time-driven work forward (admissions, delayed starts, sleeps, waits) but runs no step (workflow.NodeScheduler): a small scheduler process beside worker processes started WithoutScheduler. It overrides WithWorkers.

func WithSchema

func WithSchema(name string) Option

WithSchema puts every table in the named Postgres schema ("jobs" makes "jobs"."wf_runs"), for apps that keep queue tables apart for grants, backup or ownership, or one schema per tenant, without changing the pool's search_path. The schema must exist (Migrate does not create it) and match the table prefix's pattern. New refuses it on SQLite, which has no schemas, and on MySQL, whose schema is the database: point the pool at that database instead.

The migration ledger stays in the connection's search_path (sqlb's ledger table is not schema-qualified), named <prefix>migrations as always; its rows for this store are owned as "<owner>@<schema>", so several schemas share one ledger table and keep separate histories. MigrationScripts with the same options gives the schema-qualified DDL.

func WithStopWhenIdle

func WithStopWhenIdle(grace time.Duration) Option

WithStopWhenIdle drains the client once it has had no unfinished run (scheduled, queued, running or waiting) of its workflows, on the queues it serves, for grace on the scheduling clock: a worker that stops when its queues are empty. Schedules do not keep it alive. Done closes when it has stopped. A grace of zero or less is ignored.

func WithTablePrefix

func WithTablePrefix(p string) Option

WithTablePrefix sets the prefix of every table name (default "wf_"), so two independent queues can share one database. It must match ^[a-z][a-z0-9_]{0,30}$; New refuses anything else.

func WithTickRetention

func WithTickRetention(d time.Duration) Option

WithTickRetention sets how long a fired cron tick's row is kept (default 24 hours). A tick row is what stops a second node firing the same tick, which only matters while a node may still see that tick due, so keep it well above the longest pause a node can have between reading a schedule and firing it. Values of zero or less are ignored.

func WithTxAttempts

func WithTxAttempts(n int) Option

WithTxAttempts sets how many times in all a transaction that lost a lock race is tried before its error reaches the caller (default 5). Raise it for a SQLite file many processes write; one turns retrying off. Values below 1 are ignored.

func WithWorkers

func WithWorkers(n int) Option

WithWorkers sets how many step attempts may execute at once. Values below 1 are ignored.

func WithoutAutoPrune

func WithoutAutoPrune() Option

WithoutAutoPrune turns off the background prune; the app calls Prune from its own scheduler instead.

func WithoutScheduler

func WithoutScheduler() Option

WithoutScheduler makes a client that runs steps but never fires cron ticks (workflow.NodeWorker); some other client of the tables must fire them (WithSchedulerOnly, or a full client).

type OutputLimits

type OutputLimits struct {
	// PerRun is how many attempts are kept per run, the newest; older ones
	// are deleted as new ones arrive. Default DefaultOutputPerRun.
	PerRun int
	// Bytes caps one attempt's text; longer text is cut on a character
	// boundary and marked Truncated. Default DefaultOutputBytes.
	Bytes int
}

OutputLimits bounds what an OutputStore keeps. A zero or negative field takes its default.

type OutputStore

type OutputStore struct {
	// contains filtered or unexported fields
}

OutputStore keeps the output middleware.CaptureOutput captured from each step attempt in <prefix>outputs, so it lives with the run and every process reads the same history: it is a middleware.OutputSink for writes, and RunOutput reads a run's attempts back (an admin screen, through admin.OutputSourceFunc, which this package does not import).

store := c.OutputStore(sqlstore.OutputLimits{})
capture := middleware.CaptureOutput(store /* , options */)
// admin.WithOutputs(admin.OutputSourceFunc(func(ctx context.Context, id workflow.RunID) ([]admin.AttemptOutput, error) {
//	rows, err := store.RunOutput(ctx, id) // map each StoredOutput to admin.AttemptOutput
// }))

Like UniqueStore it needs no running Client: NewOutputStore builds it over any *sql.DB (an app on Hatchet with a database). A run's rows are deleted with the run by a Client over the same prefix (Delete, retention); a store without a Client bounds its table with PruneBefore. Output may hold whatever a step logged, so treat the table like the payloads: it is read only by what the app wires to it.

func NewOutputStore

func NewOutputStore(db *sql.DB, d sqlb.Dialect, lim OutputLimits, opts ...Option) (*OutputStore, error)

NewOutputStore returns the output store over db for dialect d, bounded by lim. Of the Options, WithTablePrefix, WithSchema, WithTxAttempts and WithErrorClassifier apply; the others configure a Client and are ignored. Call Migrate before the first Deliver.

func (*OutputStore) Deliver

func (s *OutputStore) Deliver(ctx context.Context, out middleware.Output) error

Deliver implements middleware.OutputSink: it stores one attempt's output under its run and deletes the run's attempts past OutputLimits.PerRun, in one transaction.

func (*OutputStore) Migrate

func (s *OutputStore) Migrate(ctx context.Context) error

Migrate creates the outputs table (its own migration list, under the same ledger and lock as Client.Migrate), and nothing else.

func (*OutputStore) PruneBefore

func (s *OutputStore) PruneBefore(ctx context.Context, t time.Time) (int64, error)

PruneBefore deletes the output of every attempt that finished before t and returns how many rows went: the retention of a store used without a Client, whose runs' deletes would otherwise take their output along.

func (*OutputStore) RunOutput

func (s *OutputStore) RunOutput(ctx context.Context, run workflow.RunID) ([]StoredOutput, error)

RunOutput returns the stored output of run's attempts, oldest first; an empty slice when it has none.

type PruneResult

type PruneResult struct {
	// Deleted is how many runs were deleted (with their steps).
	Deleted int
	// HeldUndelivered counts runs whose end was never delivered to an
	// OnRunFinished observer. When it keeps growing, no client of the
	// cluster registers an observer: register one, or set
	// Retention.Undelivered.
	HeldUndelivered int
	// HeldIdempotency counts runs still holding an idempotency key. A run
	// that both an age rule and KeepLast select is counted once per rule.
	HeldIdempotency int
	// HeldBatch counts runs of a batch in BatchStore that is still open
	// (State.Open).
	HeldBatch int
}

PruneResult is what one Prune did. The Held counts are runs old enough to delete that a guard kept, so a setup that can never delete anything shows up as a number rather than as silence.

type Retention

type Retention struct {
	// Succeeded, Failed and Cancelled are how long a run that ended with
	// that status is kept after it finished. Zero keeps it forever. A
	// discarded run is kept as long as a cancelled one (Cancelled), as it
	// was when discards were stored as cancelled.
	Succeeded, Failed, Cancelled time.Duration

	// KeepLast caps how many finished runs (succeeded, failed,
	// cancelled and discarded together) each workflow keeps: past the newest KeepLast by
	// finish time, older runs are deleted even when their status's
	// duration above would keep them. Both rules apply: a run goes when either says so. The guards
	// below still hold, so a run they keep stays and still counts toward
	// the N. Zero (the default) sets no count limit.
	KeepLast int

	// PerWorkflow replaces the durations and KeepLast above for the named
	// workflows, whole: a workflow listed here uses only its own entry.
	PerWorkflow map[string]Retention

	// Undelivered also deletes runs whose end was never delivered to an
	// OnRunFinished observer. Without it, a cluster where no client
	// registers an observer deletes nothing at all: Prune reports those runs
	// in PruneResult.HeldUndelivered and the background prune logs a
	// warning naming both fixes. Leave it false when any client of the cluster
	// registers an observer, or an end it has not seen yet is lost; set it
	// when none does, or no run would ever be deleted (an end nobody
	// observes is never marked delivered).
	Undelivered bool
}

Retention says how long finished runs are kept before Prune deletes them. It is opt-in: without WithRetention nothing is ever deleted, because deleting history is a decision only the app can make.

type SchemaTooNewError

type SchemaTooNewError struct {
	Unknown []string
}

SchemaTooNewError names the migrations the database has that this code does not. It matches ErrSchemaTooNew under errors.Is.

func (*SchemaTooNewError) Error

func (e *SchemaTooNewError) Error() string

func (*SchemaTooNewError) Is

func (e *SchemaTooNewError) Is(target error) bool

Is makes errors.Is(err, ErrSchemaTooNew) true.

type StoredOutput

type StoredOutput struct {
	Step    string
	Attempt int
	// Started and Finished bound the attempt, on the capturing clock.
	Started, Finished time.Time
	// Text is middleware.FormatOutput's rendering of the attempt.
	Text string
	// Truncated is true when Text was cut to OutputLimits.Bytes.
	Truncated bool
	// Error is the attempt's error message; empty when it succeeded.
	Error string
}

StoredOutput is one attempt's captured output as RunOutput returns it. Its fields match admin.AttemptOutput one for one, same names, types and order, so an app converts with admin.AttemptOutput(o) inside an admin.OutputSourceFunc. Keep them identical: a change here breaks that conversion at compile time.

type UniqueStore

type UniqueStore struct {
	// contains filtered or unexported fields
}

UniqueStore is a durable middleware.UniqueStore: uniqueness locks as rows of <prefix>unique_locks, so middleware.Unique holds across every process sharing the database (ADR-0005). It does not need a running Client: build it with NewUniqueStore over any *sql.DB, which is how an app that runs on Hatchet gets durable uniqueness, or take Client.UniqueStore.

The semantics are MemoryUnique's exactly: a key is held while now is before its expiry (now from the store's clock); Acquire on a held key reports false whoever the owner, including the holder itself; Acquire on a free or expired key takes it for owner until now+ttl (a ttl of zero or less takes it already expired, so the next Acquire wins); Release frees the key only for its owner, expired or not, and is a no-op otherwise.

Acquire is atomic across processes: on Postgres and SQLite it is one upsert that takes the key only when absent or expired, decided by rows affected; on MySQL (whose rows-affected count is changed by the clientFoundRows setting sqlb's driver turns on) it deletes the key's expired row and inserts, and the primary key makes exactly one insert win (a deadlock between the two statements of two acquirers is retried). Expired rows are dropped by a running Client's dispatcher, or by PruneExpired for an app with no Client.

The struct type is not named in middleware: the module's core cannot import sqlstore, so the interface is satisfied structurally.

func NewUniqueStore

func NewUniqueStore(db *sql.DB, d sqlb.Dialect, opts ...Option) (*UniqueStore, error)

NewUniqueStore returns the lock store over db for dialect d. Of the Options, WithTablePrefix (default "wf_", same rules as New) and WithClock (expiry; default clock.Real) apply; the others configure a Client and are ignored. Call Migrate before the first Acquire.

func (*UniqueStore) Acquire

func (u *UniqueStore) Acquire(ctx context.Context, key, owner string, ttl time.Duration) (bool, error)

Acquire implements middleware.UniqueStore. An attempt that loses a lock race (a MySQL deadlock between two acquirers taking over one expired row, SQLite busy) is tried again, up to WithTxAttempts: each attempt decides afresh from the rows, so a retry can only win a key that is free.

func (*UniqueStore) Holder

func (u *UniqueStore) Holder(ctx context.Context, key string) (string, workflow.RunRef, bool, error)

Holder reports the owner and run of a held, unexpired key whose run was recorded (middleware.UniqueHolders); ok is false otherwise.

func (*UniqueStore) Migrate

func (u *UniqueStore) Migrate(ctx context.Context) error

Migrate creates the lock table (its own migration list, under the same ledger and lock as Client.Migrate), and nothing else, so a database with no sqlstore queue gets only this table.

func (*UniqueStore) PruneExpired

func (u *UniqueStore) PruneExpired(ctx context.Context) (int64, error)

PruneExpired deletes every expired lock and returns how many. Expired rows never block an Acquire, so this only bounds the table's size; a running Client does it on its dispatcher's prune.

func (*UniqueStore) Release

func (u *UniqueStore) Release(ctx context.Context, key, owner string) error

Release implements middleware.UniqueStore.

func (*UniqueStore) SetHolder

func (u *UniqueStore) SetHolder(ctx context.Context, key, owner string, ref workflow.RunRef) error

SetHolder records ref as the run of key while owner holds it (middleware.UniqueHolders); a key that is free, expired or held by another owner is left alone.

Directories

Path Synopsis
internal
probe
Package probe carries the few views of sqlstore's unexported state that its test module (sqlstore/enginetest) needs: abandoning a client the way a crash would, counting transaction attempts, cancelling mid-transaction, reading a query plan and swapping a clock.
Package probe carries the few views of sqlstore's unexported state that its test module (sqlstore/enginetest) needs: abandoning a client the way a crash would, counting transaction attempts, cancelling mid-transaction, reading a query plan and swapping a clock.
Package outbox is the transactional outbox for any workflow.Client: an app writes a job inside its own database transaction, and a Relay enqueues it on the target client (Hatchet, memory, a middleware-wrapped client, a sqlstore client in another database) after the transaction commits.
Package outbox is the transactional outbox for any workflow.Client: an app writes a job inside its own database transaction, and a Relay enqueues it on the target client (Hatchet, memory, a middleware-wrapped client, a sqlstore client in another database) after the transaction commits.
pgnotify module
pgxtx module

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL