batch

package
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 10, 2026 License: Apache-2.0 Imports: 16 Imported by: 0

Documentation

Overview

Package batch runs a group of jobs as one unit with Then, Catch and Finally callbacks, on any workflow.Client.

It is built on the public Client only: each job is an ordinary Enqueue, tagged with two labels (LabelBatch and LabelIndex) so the batch can tell which of its jobs a finished run was.

How a batch learns that a job ended

Through Sources (see Source), connected by Manager.Attach. Unless WithCompletionSource names them, Attach picks them by what the client can do:

  • a client declaring workflow.CapRunFinished (the memory driver): the run-finished hook alone (HookSource), because every run end is guaranteed to arrive;
  • a workflow.RunNotifier without that capability (the Hatchet driver, whose delivery is best-effort): the hook for speed, plus a status poller (PollSource, every DefaultPollInterval or WithStatusPoll) as the safety net, so a lost event delays a batch by one poll instead of hanging it;
  • any other client: the poller alone, since Client.Status exists on every driver.

Attach therefore works on every driver; it refuses only a Source that cannot work, such as HookSource on a client that is not a RunNotifier or a poll interval that is not positive. Two Sources reporting the same job are safe: the first terminal status recorded wins.

Semantics

  • Then runs once when every job succeeded and the batch was not cancelled.
  • Catch runs once, on the first failed job. Without AllowFailures that failure also cancels the batch and the jobs still running; with it, the other jobs keep running (Then still does not run, because not every job succeeded).
  • Finally runs once when every job has finished, whatever the outcome, including after Cancel.
  • A job cancelled on its own is not a failure: it fires no Catch, but it stops Then.

Each callback is an Enqueue of the named workflow with a JSON Progress as its payload and Batch.Labels as its labels (without the reserved batch labels), so a callback passes a client that requires a tenant label. The once-only decisions are made inside Store.Update, so a hook that fires twice for one run, or two hooks racing, still fire each callback at most once, and so does a job reported by both the hook and the poller.

Callbacks that fail to enqueue

A decided callback is saved as pending (State.Pending) in the same Store.Update as the decision, then enqueued, then removed. One whose Enqueue fails stays pending and is retried once its claim ends (see WithCallbackRetry): on the status poller's next tick, on the next job end recorded for that batch, or when the app calls Manager.RetryCallbacks. When no Source is a poller (a client whose hook is guaranteed), Attach also starts a retry loop that calls RetryCallbacks every WithCallbackRetry interval, so a Finally that failed after the last job end is still retried; WithManualCallbackRetry turns the loop off for an app that calls RetryCallbacks itself. A claim stops two attempts from overlapping. When the client has workflow.CapIdempotencyKey every callback carries the key "batch:<id>:<kind>" (a Progress callback adds ":<job>", and one decided after RetryFailed adds ":r<round>"), so a retry after an enqueue that landed but was not recorded (a crash between the two) cannot double it; without that capability such a crash can enqueue a callback twice, so callbacks are delivered at least once.

Before and Progress (Batch.Before, Batch.Progress) are callbacks too: Before is enqueued once when the batch is created, before its jobs start, and Progress once per job end that a run reports.

Changing a batch after Dispatch

  • Manager.Add starts more jobs in a batch that is still running (a job of the batch fanning out further work); Then, Catch and Finally wait for them too.
  • Manager.RetryFailed starts a finished batch's failed jobs again and re-arms its callbacks; it needs WithRetryableJobs, which keeps every job's spec in the State. A retried job's run carries LabelAttempt, and the late end of an earlier attempt is ignored.
  • Manager.Prune deletes old batch state from a Store that implements Pruner, by age of finished, unfinished and cancelled batches (PruneFilter).

Waiting for a batch inside a workflow

Manager.AwaitStep is a step that dispatches a batch and completes when it finishes, so a workflow fans out and resumes after the batch (a batch inside a chain). EnqueueMany starts many jobs in one call without any batch tracking.

Native group runs

When the client implements GroupRunner and no job has RunAt or an IdempotencyKey, Dispatch starts every job in one RunGroup call instead of one Enqueue each (see GroupRunner). A client that is a GroupSpawner with SpawnsGroups on starts such a batch as one coordinator run instead, whose children have no refs; that needs a LabelSource (see GroupSpawner).

A dispatch that dies part-way

Dispatch holds a lease on its batch (WithDispatchLease) and renews it with every write. When the dispatching process dies, the jobs it had not started would otherwise never finish; ResumeDispatches, which the status poller, the callback retry loop and the Hatchet BatchSource call each tick, takes such a batch over once its lease has ended. It starts again only the jobs known never to have started (from a RunFinder on the client, or the per-job path's order) whose specs were kept, and records the others as cancelled; see ResumeDispatches.

Index

Constants

View Source
const (
	LabelBatch = "workflow.batch.id"
	LabelIndex = "workflow.batch.index"

	// LabelAttempt is JobState.Attempt, put only on the run of a job that
	// Manager.RetryFailed started again (attempt 1 and up); a run without
	// it is attempt 0. A run end whose attempt is not the job's current
	// one is ignored.
	LabelAttempt = "workflow.batch.attempt"
)

The labels a batch puts on each of its jobs. RunFinished reads them back; a run without both is not a batch job and is ignored. They live under workflow.LabelPrefixBatch, which this package owns: Dispatch refuses a job or a Batch.Labels carrying any label under that prefix with workflow.ErrInvalid, rather than overwriting it.

View Source
const DefaultAwaitPoll = 5 * time.Second

DefaultAwaitPoll is how long an await step waits between two looks at its batch: each look is one retry of the step, so it trades the engine's retry traffic against how soon the workflow resumes after the batch ends. Five seconds matches DefaultPollInterval, the pace at which the batch itself learns of job ends by polling. Change it with AwaitPoll.

View Source
const DefaultAwaitPolls = 720

DefaultAwaitPolls is how many looks an await step makes before it gives up: 720 at DefaultAwaitPoll is one hour. Change it with AwaitPolls; an engine may cap a step's retries lower than the value asked for, so check the engine's limit for a long batch.

View Source
const DefaultCallbackRetry = 30 * time.Second

DefaultCallbackRetry is how long a callback enqueue is claimed before another attempt may take it over. It must outlast one Enqueue, or two attempts could overlap; thirty seconds is far beyond any healthy enqueue and short enough that a failed Then or Finally runs within a minute of the engine recovering. Change it with WithCallbackRetry.

View Source
const DefaultDispatchLease = 5 * time.Minute

DefaultDispatchLease is how long a Dispatch may go without writing to the Store before ResumeDispatches treats its process as dead and takes the batch over. It must exceed the longest single Enqueue or RunGroup call, since the lease is renewed between calls, and the engine's run list lag, so a RunFinder sees every run the dead process started.

View Source
const DefaultPollInterval = 5 * time.Second

DefaultPollInterval is how often the status poller checks a batch's unfinished jobs. Each tick costs one Client.Status call per unfinished job, so it trades engine load against callback latency; five seconds keeps a 1,000-job batch at 200 calls a second at worst, while a Then or Finally that runs a few seconds late is acceptable for work batched in the background. Change it with WithStatusPoll.

Variables

View Source
var ErrBatchFailed = errors.New("batch: finished with failed or cancelled jobs")

ErrBatchFailed is the permanent error an await step ends with when its batch finished with a failed or cancelled job, or was cancelled, unless AwaitAllowFailures was given.

View Source
var ErrBatchPending = errors.New("batch: still running")

ErrBatchPending is the error an await step returns while its batch has not finished, so the step's retry policy looks again after the poll interval. It shows up as a failed attempt in an engine's run history and in step metrics; that is the polling, not a fault.

View Source
var ErrClosed = errors.New("batch: the batch has finished or was cancelled")

ErrClosed is returned by Manager.Add for a batch that has finished or was cancelled: nothing would track the new jobs' ends to any callback.

View Source
var ErrDispatching = errors.New("batch: a dispatch of this batch is in progress")

ErrDispatching is returned by Manager.Add and Manager.RetryFailed while another dispatch of the batch (Dispatch, Add, RetryFailed or a resumed dispatch) has not finished. It is transient: try again later, for example by returning it from the step that called Add so the step's retry policy repeats the call.

Functions

func EnqueueMany

func EnqueueMany(ctx context.Context, c workflow.Client, jobs []workflow.Job) ([]workflow.RunRef, error)

EnqueueMany starts jobs on c without batch tracking: no batch labels, no Store, no callbacks. It picks, in order:

  • c's own EnqueueMany when c is a ManyEnqueuer;
  • one RunGroup call when c is a GroupRunner and no job has RunAt or an IdempotencyKey (a GroupRunner that answers ErrUnsupported falls through to the loop);
  • one Enqueue per job, in order, stopping at the first error.

It returns one RunRef per job, in order. On an error the refs it returns has len(jobs) entries with a zero RunRef for each job that did not start, so the caller can cancel or retry exactly those; the error of the loop names the first job that failed. No jobs is a no-op.

A client wrapper (workflow.Use, middleware.WithCodec, a Recorder) implements neither interface, so EnqueueMany through it is the loop and every job passes the wrapper's middleware.

func JobLabels

func JobLabels(id ID, idx, attempt int) map[string]string

JobLabels returns the batch labels of job idx of batch id at attempt (JobState.Attempt): what a Source that learned a job's end by its ref, not from the run's own labels, passes to RunFinished. LabelAttempt is set only for attempt 1 and up, as on the runs.

Types

type AwaitOption

type AwaitOption func(*awaitConfig)

AwaitOption configures Manager.AwaitStep.

func AwaitAllowFailures

func AwaitAllowFailures() AwaitOption

AwaitAllowFailures makes the step succeed whatever the batch's outcome, so the next step reads the Progress and decides; by default a batch with a failed or cancelled job fails the step with ErrBatchFailed.

func AwaitPoll

func AwaitPoll(d time.Duration) AwaitOption

AwaitPoll sets the wait between two looks at the batch (see DefaultAwaitPoll). A value that is not positive keeps the default. An engine may round it (Hatchet uses whole seconds).

func AwaitPolls

func AwaitPolls(n int) AwaitOption

AwaitPolls sets how many looks the step makes before it fails with ErrBatchPending (see DefaultAwaitPolls). A value below 1 keeps the default.

type Batch

type Batch struct {
	// Name is a human-readable name, shown in Progress.
	Name string

	// Jobs are enqueued in order. At least one is required.
	Jobs []workflow.Job

	// Then, Catch and Finally name registered workflows to enqueue when the
	// batch succeeds, on its first failure, and when it finishes. Empty
	// means none.
	Then, Catch, Finally string

	// Before names a workflow enqueued once when the batch is created,
	// before its jobs start. Progress names a workflow
	// enqueued once per job end that a run reports, whatever the status,
	// after that end is recorded (not only successes; the payload's counts
	// tell them apart). A job recorded as
	// cancelled without a run (a dispatch that failed, a resumed dispatch)
	// fires no Progress. Both get the same payload and labels as the
	// other callbacks, and both are retried like them when their enqueue
	// fails. Empty means none.
	Before, Progress string

	// AllowFailures keeps the other jobs running after a failure instead of
	// cancelling them; Catch still runs on the first failure either way.
	AllowFailures bool

	// Labels are copied onto every Then, Catch and Finally job, so a
	// callback carries what the batch was for (a tenant, a user) and passes
	// enqueue middleware that requires a label or routes on one. They are
	// saved in State, so a callback enqueued by another process or after a
	// restart carries them too. A label under workflow.LabelPrefixBatch is
	// refused by Dispatch, as on the jobs. They are not put on the batch's
	// own jobs; set those on each Job.
	Labels map[string]string

	// DiscardedAs is how a job that ended workflow.StatusDiscarded (a step
	// returned workflow.Discard, on a driver with CapRunStates) counts:
	// workflow.StatusFailed (the zero value's meaning: it fires Catch and,
	// without AllowFailures, cancels the rest), workflow.StatusCancelled
	// (stops Then, fires no Catch) or workflow.StatusSucceeded (as if it
	// had done its work). Any other value is refused by Dispatch. It is
	// saved in State, so every process decides the same way.
	DiscardedAs workflow.Status
}

Batch is a group of jobs to run together.

type CallbackKind

type CallbackKind string

CallbackKind names one of a batch's callbacks.

const (
	CallbackBefore   CallbackKind = "before"
	CallbackProgress CallbackKind = "progress"
	CallbackCatch    CallbackKind = "catch"
	CallbackThen     CallbackKind = "then"
	CallbackFinally  CallbackKind = "finally"
)

The callbacks, in the order they are enqueued when several are decided at once. Before is decided only when the batch is created, Progress once per job end reported by a run.

type GroupRunner

type GroupRunner interface {
	// RunGroup starts jobs, which all have an empty RunAt and
	// IdempotencyKey, and returns one RunRef per job in the same order.
	// group is the batch ID, for the engine's grouping or metadata.
	//
	// An error wrapping workflow.ErrUnsupported, returned before anything
	// started (for example a group the engine cannot run as one, such as
	// jobs of several workflows), makes Dispatch fall back to one Enqueue
	// per job. Any other error means the group did not fully start: refs
	// is then nil, or has len(jobs) entries with a zero RunRef for each job
	// that did not start, and Dispatch cancels the batch (which cancels the
	// children that did start).
	RunGroup(ctx context.Context, group string, jobs []workflow.Job) ([]workflow.RunRef, error)
}

GroupRunner is an optional interface a workflow.Client implements to start every job of a batch in one call, as one group on its engine (a bulk run), instead of one Enqueue per job. Manager.Dispatch uses it when the client implements it and no job carries RunAt or IdempotencyKey; otherwise it enqueues the jobs one by one.

The batch learns of each child's end exactly as for an enqueued job, through its Sources, so an implementation must keep two promises:

  • each job's Labels (which carry LabelBatch and LabelIndex) reach that child's workflow.RunEvent.Labels when the client is a RunNotifier;
  • each returned RunRef answers Client.Status and Client.Cancel like a ref from Enqueue.

A client wrapper (middleware.WrapClient, middleware.WithCodec, workflow.Use) embeds workflow.Client and so hides GroupRunner: Dispatch then enqueues through the wrapper, and its middleware (a codec sealing payloads, say) is never bypassed.

type GroupSpawner

type GroupSpawner interface {
	// SpawnsGroups reports whether Dispatch should use SpawnGroup for a
	// new batch. A client that implements the interface but has it
	// switched off returns false, and Dispatch takes the other paths.
	SpawnsGroups() bool

	// SpawnGroup starts the coordinator run for group (the batch ID) over
	// jobs, which all have an empty RunAt and IdempotencyKey and carry
	// the batch labels, and returns its ref. It must be idempotent per
	// group: a second call for the same group, as a resumed dispatch
	// makes, returns the first coordinator rather than starting another.
	// An error wrapping workflow.ErrUnsupported, returned before anything
	// started (a group too large for one coordinator, say), makes Dispatch
	// start the jobs another way.
	SpawnGroup(ctx context.Context, group string, jobs []workflow.Job) (workflow.RunRef, error)

	// CancelGroup cancels the coordinator parent and every unfinished
	// child of group created since since (the batch's CreatedAt). It does
	// not rely on the engine cancelling children with their parent, and
	// works with a zero parent (a SpawnGroup call that failed but may have
	// landed) by finding the runs by label.
	CancelGroup(ctx context.Context, group string, parent workflow.RunRef, since time.Time) error
}

GroupSpawner is an optional interface a workflow.Client implements to start a batch as one coordinator run on its engine, which then starts every job as its own child run. Dispatch is one trigger whatever the batch's size, and the engine, not the dispatching process, sees the jobs through to started. The Hatchet driver implements it behind hatchet.Config.BatchCoordinator.

The children get no RunRef: the engine names a child only when the coordinator starts it, after Dispatch has returned. So the batch keeps Parent instead, learns each child's end by its LabelBatch label (which is why Dispatch refuses a coordinated batch unless a LabelSource is attached), and cancels with CancelGroup. The engine must not let a retried coordinator start a child twice.

type ID

type ID string

ID identifies one batch.

func AwaitID

func AwaitID(run workflow.RunID, step string) ID

AwaitID is the ID of the batch an AwaitStep named step dispatches in run: a hash of the two, so it is the same on every attempt, unique per run and step, and made of characters every engine accepts in a label. A test or an operator uses it to find that batch with Manager.Progress.

func (ID) String

func (id ID) String() string

String returns the ID as a string.

type JobState

type JobState struct {
	// Ref is the run, set once Enqueue returns; zero before that or when
	// the job was never enqueued.
	Ref workflow.RunRef

	// Status is the run's terminal status, or "" while it has not
	// finished. Only terminal statuses are recorded.
	Status workflow.Status

	// Attempt is 0 for the job's first run and grows by one each time
	// Manager.RetryFailed starts it again. A retried job's run carries it
	// in LabelAttempt, and a run end whose attempt differs is ignored, so
	// the late end of an earlier run never counts for the new one.
	Attempt int `json:",omitempty"`
}

JobState is one job of a batch.

type LabelSource

type LabelSource interface {
	Source
	FindsJobsByLabel()
}

LabelSource marks a Source that learns job ends by the batch labels of the engine's runs, without the jobs' refs, and records the jobs a coordinator never started once it has ended. A coordinated batch (GroupSpawner) needs one; Attach records whether one is attached.

type Manager

type Manager struct {
	// contains filtered or unexported fields
}

Manager dispatches batches and tracks them. It holds no package-level state; several Managers may share one Client and one Store.

func New

func New(client workflow.Client, store Store, opts ...Option) *Manager

New returns a Manager that enqueues on client and keeps state in store. Call Attach before client.Start so run ends reach the batch.

func (*Manager) Add

func (m *Manager) Add(ctx context.Context, id ID, jobs ...workflow.Job) error

Add adds jobs to a batch that is still running: a job of the batch can fan out further work, and Then, Catch and Finally wait for the new jobs too. The jobs are labelled and started as Dispatch starts its own (one group on a GroupRunner client when eligible), take the next indexes, and count in Progress from now on.

It refuses jobs with a label under workflow.LabelPrefixBatch (workflow.ErrInvalid), a batch that has finished or was cancelled (ErrClosed), one whose dispatch is still running (ErrDispatching, try again), and a coordinated batch (an UnsupportedError). An unknown ID returns an error wrapping workflow.ErrNotFound. Adding no jobs is a no-op.

When an Enqueue fails, the new jobs not yet started are recorded as cancelled and the batch is cancelled, exactly as for a Dispatch that fails part-way, and the error is returned.

func (*Manager) Attach

func (m *Manager) Attach(ctx context.Context) error

Attach connects the Manager's Sources (see the package doc for the defaults) and must be called once, before client.Start, because a hook cannot be registered after it. When no Source is a poller it also starts the callback retry loop (see WithManualCallbackRetry). Background sources such as the poller, and the retry loop, run until ctx is cancelled; pass the ctx given to client.Start. An error from any Source is returned and the Manager must not be used.

func (*Manager) AwaitStep

func (m *Manager) AwaitStep(name string, build func(ctx context.Context, in []byte) (Batch, error), opts ...AwaitOption) workflow.Step

AwaitStep returns a run step that dispatches a batch and completes when the batch has finished, so a workflow can fan out to many jobs and resume after them: a batch inside a chain of steps. Put it in a workflow like any step; its successors run after the batch.

On its first attempt the step calls build with its input and dispatches the batch build returns. Every later attempt looks at that batch: while it runs, the step returns ErrBatchPending and its retry policy (set by AwaitStep: AwaitPolls retries, AwaitPoll apart, constant) looks again; once it has finished, the step returns the batch's Progress as JSON, or fails permanently with ErrBatchFailed when a job failed or was cancelled (see AwaitAllowFailures). It is polling built on retries because every driver has retries; a driver without durable retries (memory) loses the wait with the process, as it loses any retry.

The batch ID is derived from the run and step (workflow.StepInfo), not from WithIDFunc, so a retried attempt finds the batch the first one dispatched instead of dispatching another. That needs a driver that sets StepInfo (every driver in this module does); a ctx without it fails the step permanently. build should be quick and deterministic; an error from it, or from Dispatch, is retried like ErrBatchPending.

The Manager must be attached and running in the worker process, since the step reads the batch from the Manager's Store; with a shared Store (sqlstore's BatchStore) any process may run the step.

func (*Manager) Cancel

func (m *Manager) Cancel(ctx context.Context, id ID) error

Cancel cancels the batch and every job still running. Then will not run; Finally runs once the cancelled jobs report back. Cancelling a finished or already cancelled batch is a no-op; an unknown ID returns an error wrapping workflow.ErrNotFound.

func (*Manager) Dispatch

func (m *Manager) Dispatch(ctx context.Context, b Batch) (ID, error)

Dispatch saves the batch and enqueues its jobs in order, returning its ID. On a GroupRunner client it starts them in one RunGroup call instead when every job is eligible (see GroupRunner). When an Enqueue fails, the jobs not yet enqueued are recorded as cancelled, the batch is cancelled (which cancels the jobs already enqueued and, once they finish, runs Finally) and the error is returned together with the ID, so the caller can still inspect the batch.

Dispatch holds a lease on the batch (WithDispatchLease) and renews it with every saved ref. If the process dies part-way, another process's Manager.ResumeDispatches finishes the dispatch once the lease has ended.

func (*Manager) Progress

func (m *Manager) Progress(ctx context.Context, id ID) (Progress, error)

Progress returns a snapshot of the batch; an unknown ID returns an error wrapping workflow.ErrNotFound.

func (*Manager) Prune

func (m *Manager) Prune(ctx context.Context, f PruneFilter) (int, error)

Prune deletes the batches f matches (see PruneFilter) from a Store that implements Pruner and returns how many it deleted. A Store without Pruner returns an UnsupportedError, and a filter with no rule an error wrapping workflow.ErrInvalid. Run it on a schedule the app owns; the Manager starts no pruning of its own.

func (*Manager) ResumeDispatches

func (m *Manager) ResumeDispatches(ctx context.Context) error

ResumeDispatches finishes the dispatch of every open batch whose dispatch did not finish and whose lease has ended, which means the process dispatching it died or stalled past the lease. The status poller, the callback retry loop and the Hatchet BatchSource call it on each tick; an app with neither calls it itself.

For each such batch it takes the lease and looks at the jobs that are not finished and have no ref:

  • one a RunFinder (on the client) lists as started gets its ref saved;
  • one known never to have started is started again, when its spec was kept (see WithResumableDispatch) and the batch is not cancelled;
  • every other one is recorded as cancelled, with the reason logged.

A job is known never to have started when the finder does not list it and it has no RunAt, or, without a finder, when it comes after the first job without a ref on the one-Enqueue-per-job path (which saves each ref before the next Enqueue). The first such job, and after a GroupRunner call every such job, may have started with its ref lost, and is cancelled rather than risk running it twice; if it did run, its end is then ignored, since a job's first recorded status wins. A cancelled batch starts nothing: its unstarted jobs are recorded as cancelled and the found runs are cancelled.

func (*Manager) RetryCallbacks

func (m *Manager) RetryCallbacks(ctx context.Context) error

RetryCallbacks enqueues every pending callback whose claim has ended, across every batch in the store, and returns the errors joined. The status poller calls it each tick, and every recorded job end retries its own batch's callbacks. Without a poller, Attach starts a loop that calls it (see WithManualCallbackRetry), because once a batch has finished no further job end arrives to retry a failed Finally.

func (*Manager) RetryFailed

func (m *Manager) RetryFailed(ctx context.Context, id ID) (int, error)

RetryFailed starts the failed jobs of a finished batch again under the same batch and re-arms its callbacks. It returns how many jobs it started.

The jobs it starts are those that failed and, when a failure cancelled the batch (no AllowFailures), also those that cancel stopped, since Then cannot run without them. It clears the batch's cancellation and its Catch, Then and Finally latches, so Catch fires again on a new failure and Then or Finally once the retried jobs end; those callbacks carry a new idempotency key (see State.Round). Each retried run carries LabelAttempt, and a late end of an earlier run is ignored.

It needs the jobs' specs, so the Manager must be built with WithRetryableJobs (an UnsupportedError otherwise). It refuses a batch that has not finished and one cancelled by Manager.Cancel or a failed dispatch (workflow.ErrInvalid), one whose dispatch is still running (ErrDispatching) and a coordinated batch (an UnsupportedError). A batch with nothing to retry returns 0 and changes nothing.

func (*Manager) RunFinished

func (m *Manager) RunFinished(ctx context.Context, labels map[string]string, status workflow.Status) error

RunFinished is the run-finished hook: the app calls it (directly or through the driver's hook) with a finished run's Job.Labels and terminal status. A run that is not a batch job, a non-terminal status, an unknown batch, or a repeat for a job already recorded is ignored and returns nil, so the hook can be wired to every run.

type ManyEnqueuer

type ManyEnqueuer interface {
	EnqueueMany(ctx context.Context, jobs []workflow.Job) ([]workflow.RunRef, error)
}

ManyEnqueuer is an optional interface a workflow.Client implements to start several jobs in one call of its own (one transaction on a SQL store, one bulk request on an engine), which EnqueueMany prefers over every other path. It keeps EnqueueMany's contract: one RunRef per job in order, and on an error either nil refs or len(jobs) refs with a zero RunRef for each job that did not start.

type MemoryStore

type MemoryStore struct {
	// contains filtered or unexported fields
}

MemoryStore is an in-process Store, for tests and the memory driver. The zero value is not usable; call NewMemoryStore.

func NewMemoryStore

func NewMemoryStore() *MemoryStore

NewMemoryStore returns an empty MemoryStore.

func (*MemoryStore) Create

func (m *MemoryStore) Create(_ context.Context, st State) error

Create implements Store.

func (*MemoryStore) Get

func (m *MemoryStore) Get(_ context.Context, id ID) (State, error)

Get implements Store.

func (*MemoryStore) Prune

func (m *MemoryStore) Prune(_ context.Context, f PruneFilter) (int, error)

Prune implements Pruner.

func (*MemoryStore) Unfinished

func (m *MemoryStore) Unfinished(_ context.Context) ([]State, error)

Unfinished implements Store.

func (*MemoryStore) Update

func (m *MemoryStore) Update(_ context.Context, id ID, fn func(*State) error) (State, error)

Update implements Store.

type Option

type Option func(*Manager)

Option configures a Manager.

func WithCallbackRetry

func WithCallbackRetry(d time.Duration) Option

WithCallbackRetry sets how long a callback enqueue is claimed before it may be retried (see DefaultCallbackRetry). A value that is not positive keeps the default.

func WithClock

func WithClock(c clock.Clock) Option

WithClock sets the clock used for the CreatedAt, CancelledAt and FinishedAt times. The default is clock.Real.

func WithCompletionSource

func WithCompletionSource(srcs ...Source) Option

WithCompletionSource replaces the default Sources with srcs, for example a native engine feed, or HookSource plus PollSource on a client whose hook is guaranteed but which the app wants double-checked.

func WithDispatchLease

func WithDispatchLease(d time.Duration) Option

WithDispatchLease sets the dispatch lease; see DefaultDispatchLease for what it must exceed. A value that is not positive keeps the default.

func WithIDFunc

func WithIDFunc(fn func() (ID, error)) Option

WithIDFunc makes batch IDs with fn instead of 128 random bits as hex, for example a UUIDv7 or ULID so IDs sort by time or match the app's own. An ID must be unique across every process sharing the Store (Create refuses a repeat) and non-empty; Dispatch fails with fn's error, or workflow.ErrInvalid for an empty ID, and saves nothing. The ID appears in the LabelBatch label and the callbacks' idempotency keys, so keep it to characters the engine accepts in a label value. A nil fn keeps the default.

func WithLogger

func WithLogger(l *slog.Logger) Option

WithLogger sets where the Manager reports what the run-finished hook cannot return to anyone: a callback that failed to enqueue, a cancel that failed. The default discards, as the drivers do.

func WithManualCallbackRetry

func WithManualCallbackRetry() Option

WithManualCallbackRetry stops Attach from starting the callback retry loop it adds when no Source is a poller. Use it only when the app calls Manager.RetryCallbacks on its own schedule; without either, a callback that fails after the batch's last job end is never retried.

func WithResumableDispatch

func WithResumableDispatch() Option

WithResumableDispatch keeps every batch's job specs in its State until its dispatch finishes on the one-Enqueue-per-job path too, so a resumer can start the jobs a dead process never started. The GroupRunner path keeps them anyway. The cost is that each saved ref rewrites the State with every job's payload, so a dispatch of n jobs writes O(n²) payload bytes to the Store; without it, those never-started jobs are recorded as cancelled instead.

func WithRetryableJobs

func WithRetryableJobs() Option

WithRetryableJobs keeps every job's spec (workflow, payload, labels) in the batch's State for the batch's whole life, which Manager.RetryFailed needs to start a failed job again. It implies what WithResumableDispatch does, and costs the same: each State write carries every job's payload, so keep payloads small (a claim check reference rather than the data) in a batch that uses it.

func WithStatusPoll

func WithStatusPoll(interval time.Duration) Option

WithStatusPoll sets the interval of the status poller Attach adds by default (see the package doc); the default is DefaultPollInterval. It does not add a poller where the defaults would not; name PollSource in WithCompletionSource for that.

type PendingCallback

type PendingCallback struct {
	Kind     CallbackKind
	Workflow string

	// Job is the index of the job whose end decided a Progress callback;
	// zero for the other kinds.
	Job int `json:",omitempty"`

	// Round is State.Round when the callback was decided; it keeps the
	// idempotency keys of callbacks from different RetryFailed rounds
	// apart.
	Round int `json:",omitempty"`

	// Attempts counts the enqueues started, including one in flight.
	Attempts int

	// RetryAt is when the current attempt's claim ends: until then no
	// other caller starts an attempt, so two never overlap. An attempt
	// that fails leaves it, so the next try comes after the claim.
	RetryAt time.Time

	// LastError is the latest failed attempt's error, for operators.
	LastError string `json:",omitempty"`
}

PendingCallback is a callback that was decided but whose Enqueue has not yet succeeded. The decision itself stays once-only (State's latches); a pending entry only carries that decision until the Enqueue lands, so a failed enqueue is retried instead of lost.

type Progress

type Progress struct {
	ID            ID     `json:"id"`
	Name          string `json:"name,omitempty"`
	Total         int    `json:"total"`
	Pending       int    `json:"pending"`
	Succeeded     int    `json:"succeeded"`
	Failed        int    `json:"failed"`
	CancelledJobs int    `json:"cancelledJobs"`
	// Discarded counts the jobs that ended StatusDiscarded; each is also
	// counted in the bucket Batch.DiscardedAs names.
	Discarded int  `json:"discarded,omitempty"`
	Cancelled bool `json:"cancelled"` // the batch itself was cancelled
	Finished  bool `json:"finished"`  // every job is terminal
}

Progress is a snapshot of a batch; it is also the payload of every callback, as JSON.

func (Progress) Percent

func (p Progress) Percent() int

Percent is Processed as a whole percentage of Total.

func (Progress) Processed

func (p Progress) Processed() int

Processed is the number of jobs that have finished, in any status.

type PruneFilter

type PruneFilter struct {
	// FinishedBefore deletes closed batches (every job finished and no
	// callback pending) that finished before it.
	FinishedBefore time.Time

	// UnfinishedBefore deletes open batches (State.Open: a job still
	// running or a callback still pending) created before it. Their jobs'
	// later ends are then ignored, since the batch is gone.
	UnfinishedBefore time.Time

	// CancelledBefore deletes cancelled batches cancelled before it,
	// whether or not their jobs have finished.
	CancelledBefore time.Time
}

PruneFilter says which batches Prune deletes. Each non-zero time adds one rule, and a batch matching any rule is deleted: finished, unfinished and cancelled batches each have their own age. A filter with every time zero matches nothing, and Manager.Prune refuses it as a mistake.

func (PruneFilter) IsZero

func (f PruneFilter) IsZero() bool

IsZero reports whether f has no rule.

func (PruneFilter) Matches

func (f PruneFilter) Matches(st State) bool

Matches reports whether Prune deletes st under f. It is the whole pruning decision, so every Store applies the same rule.

type Pruner

type Pruner interface {
	Prune(ctx context.Context, f PruneFilter) (int, error)
}

Pruner is an optional interface a Store implements to delete old batch state. Manager.Prune calls it and refuses with an UnsupportedError for a Store without it. An implementation deletes exactly the batches PruneFilter.Matches accepts (a SQL store may express the same rule in its WHERE clause) and returns how many it deleted.

type RunFinder

type RunFinder interface {
	FindBatchRuns(ctx context.Context, id ID, since time.Time) (map[int]workflow.RunRef, error)
}

RunFinder is an optional interface a workflow.Client implements to list the runs that exist for a batch's jobs, found by their LabelBatch label whatever their status, keyed by their LabelIndex. ResumeDispatches uses it to tell which jobs of a dispatch that died part-way had started, so it can save their refs and start only the others. since is the batch's CreatedAt; an implementation widens it for clock skew as it needs. A delayed start that has not fired need not be found, and a job with a RunAt is therefore never restarted on the finder's word alone.

type Source

type Source interface {
	// Attach connects the source to m. It is called once, by
	// Manager.Attach, before the client starts. A source that works in the
	// background starts its goroutine here and stops it when ctx is
	// cancelled. An error means the source cannot work with this client.
	Attach(ctx context.Context, m *Manager) error
}

Source tells a Manager that batch jobs ended. It is the seam between the batch and however a driver reports run ends: the run-finished hook, a status poller, or a native engine feed. A Source reports through Manager.RunFinished, which ignores repeats, so several Sources may report the same job: the first terminal status recorded wins.

func HookSource

func HookSource() Source

HookSource reports run ends from the client's run-finished hook (workflow.RunNotifier). Its Attach fails with an error wrapping workflow.ErrUnsupported when the client is not a RunNotifier.

func PollSource

func PollSource(interval time.Duration) Source

PollSource reports run ends by calling Client.Status for every unfinished job of every open batch each interval, on the Manager's clock, and retries the batch's callbacks that failed to enqueue. It works on every driver, so it is the safety net when hook delivery is best-effort. A job's RunRef is saved once Enqueue returns, so a job that ended before that is found on the next tick. A Status error for one job (an engine that pruned the run returns workflow.ErrNotFound) is logged and retried on the next tick, never guessed into a status. Its Attach fails with workflow.ErrInvalid for an interval that is not positive.

type State

type State struct {
	ID   ID
	Name string

	// Labels is Batch.Labels without the reserved batch labels, copied
	// onto every callback job.
	Labels map[string]string `json:",omitempty"`

	// Then, Catch and Finally are the callback workflow names; empty means
	// none.
	Then, Catch, Finally string

	// Before and Progress are Batch.Before and Batch.Progress: the
	// workflows enqueued once when the batch is created and once per job
	// end reported by a run. Empty means none.
	Before   string `json:",omitempty"`
	Progress string `json:",omitempty"`

	// AllowFailures is Batch.AllowFailures: it stops a failure from
	// cancelling the other jobs, and Catch fires on the first failure anyway.
	AllowFailures bool

	// DiscardedAs is Batch.DiscardedAs; empty means StatusFailed.
	DiscardedAs workflow.Status `json:",omitempty"`

	// Jobs holds one entry per job, in the order of Batch.Jobs; a job's
	// index is its identity, because its RunRef is not known until Enqueue
	// returns and a fast run can finish before that.
	Jobs []JobState

	// Cancelled is set by Manager.Cancel, or by the first failure when
	// AllowFailures is false. A cancelled batch never runs Then.
	Cancelled bool

	// FailCancelled is set together with Cancelled when a failed job
	// cancelled the batch, as opposed to Manager.Cancel or a dispatch that
	// failed. Manager.RetryFailed revives only such a batch, and then also
	// re-runs the jobs that cancel stopped.
	FailCancelled bool `json:",omitempty"`

	// Round counts Manager.RetryFailed calls that re-ran jobs. It goes
	// into the idempotency key of every callback decided after a retry,
	// so a re-armed Then, Catch or Finally is not deduplicated against the
	// one the previous round enqueued.
	Round int `json:",omitempty"`

	// CatchFired and Finished are the once-only latches: each flips from
	// false to true inside one Store.Update, so exactly one caller wins the
	// right to enqueue Catch, and exactly one wins Then and Finally.
	CatchFired bool
	Finished   bool

	// Pending holds the callbacks decided but not yet enqueued, in the
	// order they were decided; see PendingCallback.
	Pending []PendingCallback `json:",omitempty"`

	CreatedAt   time.Time
	CancelledAt time.Time // zero unless Cancelled
	FinishedAt  time.Time // zero unless Finished

	// Dispatch progress, which lets another process finish a Dispatch whose
	// process died part-way (see Manager.ResumeDispatches).
	//
	// Dispatched is set once every job was started or recorded as never
	// started. DispatchOwner names the Dispatch (or resume) that may start
	// jobs, and DispatchUntil is when its lease ends; every Store.Update of
	// a dispatch checks the owner and extends the lease, so a resumer takes
	// over only after the dispatcher stopped writing. Grouped is set before
	// a GroupRunner call, after which no job without a saved ref is known
	// to be unstarted. Spec holds the labelled jobs while the dispatch is
	// open, so a resumer can start the ones never started; it is kept on
	// the GroupRunner path, and on the one-Enqueue-per-job path only with
	// WithResumableDispatch, because there every saved ref rewrites the
	// whole State. A State written before these fields existed reads as
	// not dispatched with an expired lease; a resumer then finds every job
	// started and only sets Dispatched.
	//
	// Coordinated is set before a GroupSpawner call, and Parent holds the
	// coordinator run it returned: the batch's jobs are that run's
	// children, which never get a ref of their own (see GroupSpawner).
	Dispatched    bool            `json:",omitempty"`
	DispatchOwner string          `json:",omitempty"`
	DispatchUntil time.Time       `json:",omitzero"`
	Grouped       bool            `json:",omitempty"`
	Spec          []workflow.Job  `json:",omitempty"`
	Coordinated   bool            `json:",omitempty"`
	Parent        workflow.RunRef `json:",omitzero"`
}

State is everything a Store keeps for one batch. The Manager is the only writer; a Store persists it as one record and never interprets it.

func (State) Open

func (s State) Open() bool

Open reports whether the batch still needs the Manager: a job has not finished, or a callback is pending. Store.Unfinished returns exactly the open batches.

type Store

type Store interface {
	// Create saves a new batch. An existing ID is an error.
	Create(ctx context.Context, st State) error

	// Get returns a batch, or an error wrapping workflow.ErrNotFound.
	Get(ctx context.Context, id ID) (State, error)

	// Update applies fn to the current state and saves the result
	// atomically, returning the saved state. When fn returns an error
	// nothing is saved and Update returns that error. An unknown ID
	// returns an error wrapping workflow.ErrNotFound.
	Update(ctx context.Context, id ID, fn func(*State) error) (State, error)

	// Unfinished returns every open batch (State.Open: a job not finished
	// or a callback pending), for the status poller and RetryCallbacks. A
	// SQL store backs it with an indexed flag written from Open, so a long
	// history of closed batches costs nothing.
	Unfinished(ctx context.Context) ([]State, error)
}

Store persists batch state. Update must be atomic per batch: the read, the mutation and the write are one step with no other Update for the same ID in between (a SQL store uses a transaction with SELECT ... FOR UPDATE, or a version column), because the once-only latches in State rely on it. The guarantee a batch gives across processes is exactly the store's: MemoryStore holds only inside one process.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL