Documentation
¶
Overview ¶
Package workflow is one API for cron schedules, delayed jobs, durable background jobs and multi-step workflows, with drivers that range from an in-process memory driver to engines such as Hatchet.
The package holds the contract only: the Client interface every driver implements, the specs an app writes (Workflow, Step, Job, Schedule), the capability set a driver declares (Caps), the sentinel errors, and the typed helpers that sit on top of Handler. It needs nothing beyond the standard library, and deps_test.go fails the build if that changes.
Three rules hold everywhere:
- Honest capabilities. A driver declares what it can really do. Start refuses a registered workflow that needs a capability the driver lacks (see Requires and Caps.Missing), and a call the driver cannot honour returns an error wrapping ErrUnsupported that names the field. Nothing is silently dropped.
- Nothing runs until Start. Constructing a client opens no connections and starts no goroutines.
- No hidden state and no wall clock in logic. Everything hangs off the value the app constructs, and time comes from a clock.Clock.
Seams a driver applies the same way, so their meaning never drifts: the retry delay (RetryPolicy.NextDelay: Backoff.Jitter, DelayFor and RetryAfter), the cron misfire policy (Misfire.Due), the zone check (IsUTC, RequiresSchedule, CheckScheduleCaps), the backend-outage marker (ErrUnavailable, Unavailable, IsUnavailable) and the reserved label prefixes (ReservedLabelPrefixes, ValidateLabels), the retry decision (RetryPolicy.Next with Attempts: Max, MaxErrors, Until), job checks (ValidateJob, CheckJob, CheckJobQueue, EffectiveQueue, EffectiveRetriesFor), the misfire plan with its dropped count (Misfire.Plan), time-of-day windows (Windows.Allows), run query filters (RunQuery.Match, ValidateRunQuery, ValidateCancelQuery), pause targets (CheckPause, PauseTarget.Holds), the labels of a fired run (FiredLabels), the status a step error ends a run with (EndStatus), and a chain's next job and edits (NextInChain, ChainEdits).
Optional interfaces beside Client, each declared by a capability: RunNotifier, RunQuerier, RunRetrier, RunDeleter, BulkCanceller, Pauser, TickNotifier, ScheduleLister, SchedulePauser, Drainer, WorkerScaler and ClosureEnqueuer. A client wrapper embeds Forward so it never hides them. RunInline runs a workflow in the caller's goroutine with no client at all.
Driver-independent pieces built on Client: Deferred, DeferEnqueue and DeferHTTP hold enqueues until a request succeeds; Balancer, ByDepth, ByWait and BalanceLoop move workers between queues on a WorkerScaler; EnqueueClosure runs a Go function on an in-process driver.
Example ¶
Example is the README's example, kept here so it is compiled and run.
package main
import (
"context"
"fmt"
"github.com/ubgo/workflow"
"github.com/ubgo/workflow/memory"
)
func main() {
type greet struct{ Name string }
done := make(chan string, 1)
c := memory.New()
err := c.Register(workflow.Workflow{
Name: "greet",
Steps: []workflow.Step{{
Name: "say",
Run: workflow.Define(func(ctx context.Context, in greet) (string, error) {
done <- "hello " + in.Name
return "ok", nil
}),
}},
})
if err != nil {
panic(err)
}
ctx, cancel := context.WithCancel(context.Background())
defer cancel() // stops the workers
if err := c.Start(ctx); err != nil {
panic(err)
}
if _, err := c.Enqueue(ctx, workflow.Job{Workflow: "greet", Payload: []byte(`{"Name":"Ada"}`)}); err != nil {
panic(err)
}
fmt.Println(<-done)
}
Output: hello Ada
Index ¶
- Constants
- Variables
- func AllWhen(conds ...func(context.Context, Tick) bool) func(context.Context, Tick) bool
- func AllowReservedLabels(ctx context.Context, keys ...string) context.Context
- func AllowReservedPrefix(ctx context.Context, prefix string) context.Context
- func AppendToChain(ctx context.Context, jobs ...Job) error
- func AttachScheduleHooks(c Client, hooks ...ScheduleHooks) error
- func AwaitChild(ctx context.Context, client Client, name string, job Job, opts ChildOptions) ([]byte, error)
- func AwaitChildren(ctx context.Context, client Client, name string, jobs []Job, opts ChildOptions) ([][]byte, error)
- func AwaitResult(ctx context.Context, c Client, ref RunRef, opts AwaitOptions) ([]byte, error)
- func CanForceComplete(st Status) bool
- func CanForceFail(st Status) bool
- func CheckCaps(have Caps, wf Workflow) error
- func CheckJob(have Caps, job Job) error
- func CheckJobLabels(ctx context.Context, job Job) error
- func CheckJobQueue(have Caps, wf Workflow, job Job) error
- func CheckPause(have Caps, t PauseTarget) error
- func CheckPayload(job Job, limit int, source string) error
- func CheckQueueLimits(have Caps, qs []QueueConfig) error
- func CheckRunQuery(have Caps, q RunQuery) error
- func CheckScheduleCaps(have Caps, s Schedule) error
- func DecodePayload[T any](raw []byte) (T, error)
- func DefaultNodeID() string
- func DeferHTTP(next http.Handler, opts DeferOptions) http.Handler
- func DeferSucceeded(status int) bool
- func Discard(err error) error
- func DiscardedError(step string, err error) error
- func DrainContext(ctx context.Context, grace time.Duration) (context.Context, context.CancelFunc)
- func DrainOutcome(report DrainReport, ctxErr error) error
- func DrainTimeout(ctx context.Context, d Drainer, grace time.Duration) error
- func EffectiveQueue(wf Workflow, job Job) string
- func FiredLabels(labels map[string]string, id ScheduleID, name string) map[string]string
- func FiredLabelsAt(labels map[string]string, id ScheduleID, name string, at time.Time) map[string]string
- func ForcedFailure(reason string) error
- func HasStepEventSink(ctx context.Context) bool
- func HeapObjectBytes() uint64
- func Heartbeat(ctx context.Context, details []byte) error
- func InEnvironment(current string, allowed ...string) func(context.Context, Tick) bool
- func IsDiscarded(err error) bool
- func IsPermanent(err error) bool
- func IsRelease(err error) bool
- func IsTimeout(err error) bool
- func IsUTC(loc *time.Location) bool
- func IsUnavailable(err error) bool
- func LastTick(src TickSource, from, to time.Time, loc *time.Location) time.Time
- func Loop[S any](ctx context.Context, name string, init S, limit int, ...) (S, error)
- func MapEach[In, Out any](ctx context.Context, name string, items []In, ...) ([]Out, error)
- func MarkDuplicate(ctx context.Context)
- func MarkInserted(ctx context.Context)
- func Memo[T any](ctx context.Context, name string, fn func(ctx context.Context) (T, error)) (T, error)
- func NextActive(s cron.Schedule, after time.Time, loc *time.Location, a Active) time.Time
- func OverlapRefusal(id ScheduleID, prev RunRef, st Status) error
- func ParseCron(expr string, seconds bool, custom cron.Parser) (cron.Schedule, error)
- func PayloadVersionOf(labels map[string]string) (int, error)
- func PeriodKey(t time.Time, p Period, loc *time.Location) (string, error)
- func Permanent(err error) error
- func PickWeighted(weights []int, r float64) int
- func PrependToChain(ctx context.Context, jobs ...Job) error
- func ProcessChunks(ctx context.Context, name string, opts ChunkOptions, ...) (int64, error)
- func RecordEvent(ctx context.Context, msg string, attrs ...slog.Attr)
- func RecordEventLevel(ctx context.Context, level slog.Level, msg string, attrs ...slog.Attr)
- func RegisterFunc(client Client, f FuncDefinition) error
- func ReportProgress(ctx context.Context, p Progress) error
- func ReservedLabel(key string) (prefix string, ok bool)
- func ReservedLabelPrefixes() []string
- func ResolveSecret(ctx context.Context, r SecretResolver, ref SecretRef) ([]byte, error)
- func RetryAfter(err error, d time.Duration) error
- func RetryAfterDelay(err error, _ int) (time.Duration, bool)
- func RunData[T any](ctx context.Context, name string) (T, bool, error)
- func RunOutput(wf Workflow, outs map[string][]byte) []byte
- func Serve(ctx context.Context, client Client) error
- func SetOutcome(ctx context.Context, o Outcome, note string) error
- func SetRunData(ctx context.Context, name string, v any) error
- func ShutdownHook(d Drainer, grace time.Duration) func(context.Context) error
- func ShutdownHookReport(d Drainer, grace time.Duration, ...) func(context.Context) error
- func Sinks(wf Workflow) []string
- func Snooze(d time.Duration) error
- func SnoozeDelay(err error) (time.Duration, bool)
- func StackOf(err error) string
- func StepInput(payload []byte, parents []string, outs map[string][]byte) []byte
- func Supports(c Client, op Op) bool
- func TickTime(labels map[string]string) (time.Time, bool)
- func TimeoutError(kind error, detail string) error
- func TriggeredLabels(labels map[string]string, id ScheduleID, name string) map[string]string
- func TriggeredLabelsAt(labels map[string]string, id ScheduleID, name string, at time.Time) map[string]string
- func Unavailable(err error) error
- func Unsupported(field, reason string) error
- func Upcast(raw []byte, from, to int, ups map[int]Upcaster) ([]byte, error)
- func Validate(wf Workflow) error
- func ValidateCancelQuery(q RunQuery) error
- func ValidateClosure(c ClosureJob) error
- func ValidateJob(job Job) error
- func ValidateLabels(labels map[string]string) error
- func ValidatePauseTarget(t PauseTarget) error
- func ValidateProgress(p Progress) error
- func ValidateQueueName(name string) error
- func ValidateQueues(qs []QueueConfig) error
- func ValidateRecycle(r Recycle) error
- func ValidateRunQuery(q RunQuery) error
- func ValidateSchedule(s Schedule) error
- func ValidateScheduleTarget(t ScheduleTarget) error
- func ValidateSource(s Source) error
- func ValidateTenant(tenant string) error
- func WithChainEditor(ctx context.Context, ed ChainEditor) context.Context
- func WithDeferred(ctx context.Context, d *Deferred) context.Context
- func WithHeartbeater(ctx context.Context, h Heartbeater) context.Context
- func WithOutcomeSetter(ctx context.Context, s OutcomeSetter) context.Context
- func WithProgressReporter(ctx context.Context, r ProgressReporter) context.Context
- func WithStepEventSink(ctx context.Context, s StepEventSink) context.Context
- func WithStepInfo(ctx context.Context, info StepInfo) context.Context
- type Active
- type AttemptError
- type Attempts
- type AwaitOptions
- type Backoff
- type BalanceConfig
- type BalanceLoop
- type Balancer
- type BalancerFunc
- type Branch
- type BucketSpec
- type BucketStore
- type BulkCanceller
- type Caps
- type ChainEditor
- type ChainEdits
- type ChildOptions
- type ChoiceSpec
- type ChunkOptions
- type ChunkProgress
- type Client
- type ClosureEnqueuer
- type ClosureJob
- type Concurrency
- type ConcurrencyKey
- type ConcurrencyStrategy
- type DeferOptions
- type Deferred
- type DrainCut
- type DrainDisposition
- type DrainReport
- type DrainTracker
- type Drainer
- type EnqueueFunc
- type EnqueueMiddleware
- type EnqueueMiddlewareProvider
- type EnqueueOutcome
- type Event
- type EventAttr
- type Failure
- type Forward
- func (f Forward) CancelAll(ctx context.Context, q RunQuery) (int, error)
- func (f Forward) CancelStep(ctx context.Context, ref RunRef, step string) error
- func (f Forward) Delete(ctx context.Context, ref RunRef) error
- func (f Forward) Done() <-chan struct{}
- func (f Forward) Drain(ctx context.Context) error
- func (f Forward) DrainWithReport(ctx context.Context) (DrainReport, error)
- func (f Forward) EnqueueClosure(ctx context.Context, c ClosureJob) (RunRef, error)
- func (f Forward) ForceComplete(ctx context.Context, ref RunRef) error
- func (f Forward) ForceFail(ctx context.Context, ref RunRef, reason string) error
- func (f Forward) Health(ctx context.Context) HealthReport
- func (f Forward) Nodes(ctx context.Context) ([]NodeInfo, error)
- func (f Forward) OnLifecycle(fn LifecycleFunc) error
- func (f Forward) OnTick(fn TickObserver) error
- func (f Forward) Pause(ctx context.Context, t PauseTarget) error
- func (f Forward) PauseSchedule(ctx context.Context, t ScheduleTarget) error
- func (f Forward) Paused(ctx context.Context) ([]PauseTarget, error)
- func (f Forward) PayloadLimit() int
- func (f Forward) QueueLoads(ctx context.Context) ([]QueueLoad, error)
- func (f Forward) Reschedule(ctx context.Context, ref RunRef, at time.Time) (RunRef, error)
- func (f Forward) Result(ctx context.Context, ref RunRef) ([]byte, error)
- func (f Forward) Resume(ctx context.Context, t PauseTarget) error
- func (f Forward) ResumeSchedule(ctx context.Context, t ScheduleTarget) error
- func (f Forward) Retry(ctx context.Context, ref RunRef) (RunRef, error)
- func (f Forward) Runs(ctx context.Context, q RunQuery) (RunPage, error)
- func (f Forward) Schedules(ctx context.Context) ([]ScheduleInfo, error)
- func (f Forward) SetWorkers(ctx context.Context, queue string, n int) error
- func (f Forward) StopCause() error
- func (f Forward) Supports(op Op) bool
- func (f Forward) TriggerSchedule(ctx context.Context, id ScheduleID) (RunRef, error)
- type Func
- type FuncConfig
- type FuncDefinition
- type FuncOption
- type Handler
- type HealthCheck
- type HealthChecker
- type HealthReport
- type HeartbeatFunc
- type Heartbeater
- type IdempotencyPolicy
- type InlineOption
- type InlineResult
- type Job
- func CloneJob(j Job) Job
- func FuncJob(name string, arg any) (Job, error)
- func JobOf[T any](workflow string, v T) (Job, error)
- func NextInChain(chain []Job) (Job, bool)
- func OncePerPeriod(job Job, scope string, p Period, t time.Time, loc *time.Location) (Job, error)
- func WithSource(job Job, source Source, ref string) Job
- type JournalOption
- type JournalStore
- type LifecycleEvent
- type LifecycleFunc
- type LifecycleKind
- type LifecycleNotifier
- type LocalCron
- type LocalCronFunc
- type LocalCronOption
- type MapOption
- type MemoryBuckets
- type MemoryJournal
- type Middleware
- type Misfire
- type MisfirePolicy
- type NextRun
- type NextRunsQuery
- type NodeInfo
- type NodeLister
- type NodeMode
- type Op
- type OpProber
- type Outcome
- type OutcomeFunc
- type OutcomeSetter
- type OverlapPolicy
- type PauseTarget
- type Pauser
- type PayloadLimiter
- type PayloadTooLargeError
- type Period
- type Plugin
- type Progress
- type ProgressFunc
- type ProgressReporter
- type QueueBounds
- type QueueConfig
- type QueueLoad
- type RateLimit
- type Recycle
- type RecycleError
- type RecycleReason
- type RecycleWatch
- type Reporter
- type ReporterFunc
- type ReportingDrainer
- type ResultReader
- type RetryPolicy
- func (p RetryPolicy) ExhaustedError(attempts int, err error) error
- func (p RetryPolicy) Next(err error, a Attempts, now time.Time, rnd func() float64) (delay time.Duration, retry bool)
- func (p RetryPolicy) NextDelay(err error, retry int, rnd func() float64) time.Duration
- func (p RetryPolicy) Retryable(err error, attempts int) bool
- type RunDeleter
- type RunError
- type RunEvent
- type RunFailedError
- type RunFinishedFunc
- type RunFinishedProvider
- type RunForcer
- type RunID
- type RunInfo
- type RunNotifier
- type RunPage
- type RunQuerier
- type RunQuery
- type RunRef
- type RunRescheduler
- type RunRetrier
- type Schedule
- type ScheduleHooks
- type ScheduleID
- type ScheduleInfo
- type ScheduleLast
- type ScheduleLister
- type SchedulePauser
- type ScheduleTarget
- type ScheduleTriggerer
- type ScheduledID
- type SecretDetector
- type SecretFinding
- type SecretRef
- type SecretResolver
- type SecretRules
- type Source
- type SpawnOption
- type StackTracer
- type Status
- type Step
- type StepCanceller
- type StepEvent
- type StepEventFunc
- type StepEventSink
- type StepInfo
- type StepKind
- type StepMiddlewareProvider
- type StopCauser
- type Tick
- type TickNotifier
- type TickObserver
- type TickOutcome
- type TickSource
- type Transition
- type UnsupportedError
- type Upcaster
- type UpstreamLimit
- func (u *UpstreamLimit) Observe(ctx context.Context, key string, level float64) error
- func (u *UpstreamLimit) Take(ctx context.Context, key string, cost float64) error
- func (u *UpstreamLimit) TakeOrSnooze(ctx context.Context, key string, cost float64) error
- func (u *UpstreamLimit) Wait(ctx context.Context, key string, cost float64) (time.Duration, error)
- type WaitSpec
- type Window
- type Windows
- type WorkerScaler
- type Workflow
Examples ¶
Constants ¶
const ( // LabelParentRun is the RunID of the run whose step spawned the job. LabelParentRun = LabelPrefixInternal + "parent-run" // LabelParentStep is the name of that step. LabelParentStep = LabelPrefixInternal + "parent-step" )
Labels Spawn writes on every job it enqueues, so a run listing can find the children of a run. Spawn sets them itself, overwriting any value a job carried under the same key, and enqueues under AllowReservedLabels for these two keys only: they are refused on a public Enqueue unless the owner opts in (CheckJobLabels, on drivers that enforce it), so they cannot arrive by accident or from untrusted input. Like every reserved label they are a guard, not a sandbox (see AllowReservedLabels).
const ( // LabelPrefixInternal is owned by the middleware package (unique // locks, claim-check references). LabelPrefixInternal = "ubgo.workflow/" // LabelPrefixBatch is owned by the batch package (batch ID and job // index). LabelPrefixBatch = "workflow.batch." // LabelPrefixOTel is owned by the otel package (trace context carried // from Enqueue to the step). LabelPrefixOTel = "otel." )
Reserved label prefixes. The library's own packages write labels under these prefixes onto the jobs they pass down (unique keys, claim-check references, batch membership, trace context), so an app label under one of them would collide with, or be removed by, the package that owns it. The three spellings differ for historical reasons and are kept as they are: runs in flight carry them, and renaming one would break those runs.
const ( // DefaultRunsLimit is the page size of a RunQuery whose Limit is zero. DefaultRunsLimit = 100 // MaxRunsLimit is the largest Limit a RunQuery may ask for, so one // call cannot load an unbounded table into memory. MaxRunsLimit = 1000 )
Page sizes of a run query.
const ( // LabelSource holds the run's Source. LabelSource = LabelPrefixInternal + "source" // LabelSourceRef holds a free-form reference within the source: a // webhook delivery ID, a chat thread, the user who pressed a button. LabelSourceRef = LabelPrefixInternal + "source-ref" )
Trigger source: what started a run (a schedule, an operator, a parent run, an app's webhook handler). It is plain run metadata kept in two reserved labels, so every driver stores and returns it with no change, and a run recorder or an admin view can filter "fired at 2 am" apart from "an operator pressed Run now" or "a Slack message asked for it".
Why labels and not a ctx value: metadata a caller must remember to put on a ctx gets forgotten, and the run then silently lacks it. A label travels on the Job, which every driver stores, and RequireSource (in the middleware package) can refuse a job that has none.
Why reserved yet app-settable: the keys sit under LabelPrefixInternal so no app label collides with them, but unlike the other reserved keys an app is expected to set them (WithSource), so CheckJobLabels and ValidateLabels let these two through when their values are valid. A source is provenance for people and filters, never an authorization input: anything that can enqueue can claim any source.
const ( // MaxSourceLen is the longest Source, in bytes. MaxSourceLen = 64 // MaxSourceRefLen is the longest source reference, in bytes. MaxSourceRefLen = 256 )
Bounds of a source and its reference, so a label value stays an index key, not a payload.
const ( // MaxStepEventMessage is the longest StepEvent.Message, in bytes. MaxStepEventMessage = 1 << 10 // MaxStepEventAttrs is the most attributes one StepEvent keeps. MaxStepEventAttrs = 32 // MaxStepEventValue is the longest attribute value, in bytes. MaxStepEventValue = 1 << 10 )
Bounds of a step event, so a step that records events in a loop cannot grow what a sink stores without limit. RecordEvent cuts a longer message or value and keeps at most MaxStepEventAttrs attributes.
const ( TickReasonWhen = "when" // Workflow.ScheduleWhen returned false TickReasonWindow = "window" // the tick fell outside the schedule's Windows TickReasonPaused = "paused" // the schedule was paused (SchedulePauser) )
The reasons of a TickSkipped.
const ChoiceDecideSuffix = ".decide"
ChoiceDecideSuffix is appended to a Choice's name to name its decide step; each branch step is named "<choice>.<branch>", and the join step carries the Choice's own name, so later steps list the Choice's name in After as they would any step.
const ClosureStep = "closure"
ClosureStep is the name of a closure run's one step.
const ClosureWorkflow = "ubgo.workflow.closure"
ClosureWorkflow is the workflow name a closure run carries in RunInfo, RunEvent and StepInfo, so run listings, run-finished observers and middleware see closures as runs of one name. Validate refuses it as the name of a registered workflow. The closure's own name is in the LabelClosure label.
const DefaultAwaitPoll = 500 * time.Millisecond
DefaultAwaitPoll is how often Await asks Status when no AwaitOptions.Poll is given.
const DefaultBalanceEvery = 5 * time.Second
DefaultBalanceEvery is how often a BalanceLoop with no Every runs.
const DefaultChildPoll = 5 * time.Second
DefaultChildPoll is how long AwaitChild snoozes the parent step between looks at an unfinished child.
const DefaultErrorHistory = 20
DefaultErrorHistory is how many attempt errors a run keeps when the driver's error-history option is not set: enough to see a pattern in a run that retried a dozen times, small enough that a run retried for a week does not grow a row without bound.
const DefaultQueue = "default"
DefaultQueue is the queue of a run whose Job.Queue and Workflow.Queue are both empty. Naming it lets a QueueConfig give the default queue its own worker limit and priority.
const DefaultRecycleCheck = time.Second
DefaultRecycleCheck is Recycle.CheckEvery when it is zero.
const DefaultRetryMax = 0
DefaultRetryMax is the number of retries a step gets when no policy at any level (Job.Retry, Step.Retry, Workflow.Retry) says otherwise: none. It is the last link of the precedence chain EffectiveRetriesFor resolves, and the Max of the zero RetryPolicy.
const FuncStep = "func"
FuncStep is the name of the one step of a Func's workflow.
const LabelClosure = LabelPrefixInternal + "closure"
LabelClosure is the label holding a closure run's ClosureJob.Name. It sits under the reserved internal prefix, so app labels cannot forge it.
const LabelPayloadVersion = LabelPrefixInternal + "payload-version"
LabelPayloadVersion carries the version of a job's payload shape, set at enqueue by middleware.PayloadVersion, so a step can upcast a payload written by older code (Upcast).
const LabelSchedule = LabelPrefixInternal + "schedule"
LabelSchedule is the label drivers with CapTickEvents set on every run a schedule fires, to the schedule's ScheduleID, so a run-finished observer can tell which schedule a run came from (AttachScheduleHooks relies on it). It is under the reserved LabelPrefixInternal, so an app cannot set it by accident.
const LabelScheduleName = LabelPrefixInternal + "schedule-name"
LabelScheduleName is set beside LabelSchedule to the schedule's Name (for a declared cron, its expression), so a run-finished observer on any node can match a run to a named schedule without a lookup.
const LabelTickAt = LabelPrefixInternal + "tick-at"
LabelTickAt is the label a run fired by a schedule carries with the tick it represents (its logical time), RFC 3339 with nanoseconds in UTC. A job reads it through StepInfo.TickAt or TickTime: a run that starts late, or a misfire catch-up that fires three missed days in a row, still knows which day it is for, where the wall time it started would be wrong. For a manual trigger (ScheduleTriggerer) it is the driver's clock at the trigger. It is under the reserved LabelPrefixInternal, so an app cannot set it by accident. Drivers with CapTickTime set it on every fired run.
const LabelTrigger = LabelPrefixInternal + "trigger"
LabelTrigger marks a run a ScheduleTriggerer started by hand, with the value LabelTriggerManual, so run listings and observers can tell it from a cron fire.
const LabelTriggerManual = "manual"
LabelTriggerManual is the value of LabelTrigger.
const MaxChainJobs = 1000
MaxChainJobs bounds Job.Chain, counting every job after the first, so a chain stored with a run (one database row on a durable driver) stays a size a driver can load and copy on every step. A step that edits its chain past it gets an ErrInvalid.
const MaxErrorStack = 8 << 10
MaxErrorStack caps AttemptError.Stack in bytes, so one deep panic does not bloat a stored run. A longer stack keeps its first MaxErrorStack bytes, where the panicking frame is.
const (
MaxNextRuns = 100
)
MaxNextRuns caps NextRunsQuery.Count, and maxNextRunsScan the ticks examined: a window that rejects almost every tick must not turn a display call into a long loop.
const MaxOutcomeNote = 1 << 10
MaxOutcomeNote is the longest note SetOutcome accepts, in bytes.
const MaxProgressData = 64 << 10
MaxProgressData bounds Progress.Data, so a step reporting progress in a loop cannot grow a run's stored record without limit.
const MaxSnooze = 30 * 24 * time.Hour
MaxSnooze bounds one snooze, so a bug that computes a huge delay cannot park a run for years. A longer delay is cut to it.
const MisfireGrace = time.Minute
MisfireGrace is how late MisfireSkip still fires a tick: a scheduler wakes a little after a tick's time even when nothing was missed, and a tick reached within this window is on time, not missed.
const ScheduleJSONVersion = 1
ScheduleJSONVersion is the version of the JSON form of a Schedule. A reader refuses any other version, and any field it does not know, so a schedule stored by a newer release is never loaded with part of its meaning (a window, a misfire policy) silently dropped.
const TickReasonOverlap = "overlap"
TickReasonOverlap is the reason of a TickSkipped that OverlapSkip held back because the schedule's previous run was unfinished.
Variables ¶
var ( // ErrUnsupported means the driver cannot honour a field or a // workflow's needs. Drivers return it wrapped in an UnsupportedError // naming the field, never by silently dropping the field. ErrUnsupported = errors.New("workflow: unsupported by this driver") // ErrNotFound means the workflow, run or schedule does not exist. ErrNotFound = errors.New("workflow: not found") // ErrNotStarted means a call that needs workers came before Start. ErrNotStarted = errors.New("workflow: client not started") // ErrAlreadyStarted means Start was called twice, or Register after // Start. ErrAlreadyStarted = errors.New("workflow: client already started") // ErrDuplicate means an idempotency key collided in a way the driver // reports as an error rather than by returning the existing run. ErrDuplicate = errors.New("workflow: duplicate idempotency key") // ErrInvalid means a Workflow, Job or Schedule failed validation. ErrInvalid = errors.New("workflow: invalid spec") // ErrCancelled is the error a run's steps see when it is cancelled; // it is never retried. ErrCancelled = errors.New("workflow: run cancelled") // answer (a refused connection, a timeout talking to the engine or // database), as opposed to refusing the request. Drivers mark such // errors with Unavailable, so a caller such as a failover client can // tell an outage from an app bug without parsing driver messages. ErrUnavailable = errors.New("workflow: backend unavailable") // ErrDiscarded is the Err of a run a step ended with Discard. A // discarded run's Err matches ErrCancelled too, so code that treats // cancelled runs as "not failed" keeps working. ErrDiscarded = errors.New("workflow: run discarded") // ErrWrongState means the run exists but its status does not allow // the call: Retry of a run that did not fail, Delete of an unfinished // run. ErrWrongState = errors.New("workflow: run is in the wrong state for this call") )
Sentinel errors. Callers dispatch on them with errors.Is, never on the message text.
var ( // ErrTimeout matches every timeout below. ErrTimeout = errors.New("workflow: deadline missed") // ErrStepTimeout: a step attempt ran past Step.Timeout (or the // workflow's Timeout). The retry policy applies. ErrStepTimeout = errors.New("workflow: step attempt timed out") // ErrWaitTimeout: a wait step reached its Deadline or Timeout with no // matching event. ErrWaitTimeout = errors.New("workflow: wait expired") // ErrHeartbeatTimeout: an attempt went Step.HeartbeatTimeout without // calling Heartbeat. The retry policy applies. ErrHeartbeatTimeout = errors.New("workflow: heartbeat timed out") // ErrRunTimeout: the run went past Workflow.RunTimeout. It is never // retried. ErrRunTimeout = errors.New("workflow: run timed out") )
The timeout errors. Each of ErrStepTimeout, ErrWaitTimeout, ErrHeartbeatTimeout and ErrRunTimeout matches ErrTimeout too, so code reacting to "a deadline was missed" checks one sentinel (IsTimeout), and code that cares which deadline checks the specific one. Drivers with CapRunTimeout wrap every timeout they report with TimeoutError.
var ErrDeferredClosed = errors.New("workflow: deferred enqueues already flushed or discarded")
ErrDeferredClosed is returned by an enqueue made through a Deferred that was already flushed or discarded, such as a goroutine of a request handler enqueueing after the response was written. Nothing is sent.
var ErrDuplicatePlugin = errors.New("workflow: duplicate plugin name")
ErrDuplicatePlugin means two plugins handed to one Use call share a name.
var ErrForced = errors.New("workflow: run ended by an operator")
ErrForced is the context cause of a step attempt cut short because an operator forced its run to end (RunForcer), and the error a force-failed run reports (it matches ErrForced).
var ErrLocalCronStarted = errors.New("workflow: local cron already started")
ErrLocalCronStarted is returned by LocalCron.Add after Start and by a second Start: entries are fixed once the cron runs.
var ErrLoopLimit = errors.New("workflow: loop reached its iteration limit")
ErrLoopLimit means a Loop reached its iteration limit without its body saying it was done. It is returned marked Permanent: running the same loop again would reach the same limit.
var ErrNoJournal = errors.New("workflow: no journal on this context (use the Journal plugin)")
ErrNoJournal means Memo (or a helper built on it: MapEach, Loop, Spawn, RunData) was called from a ctx that carries no journal: the step's client was not built with the Journal plugin or the WithJournal middleware, or the call is not inside a step. It is returned marked Permanent, because retrying the step cannot wire a journal in.
var ErrNoSecretRef = errors.New("workflow: secret reference has no name")
ErrNoSecretRef means ResolveSecret got a SecretRef with no name.
var ErrOverlap = errors.New("workflow: the schedule's previous run is unfinished")
ErrOverlap is returned by ScheduleTriggerer.TriggerSchedule when the schedule's OverlapPolicy is OverlapSkip and its previous run is still unfinished: a manual trigger obeys the same rule as a tick.
var ErrPayloadTooLarge = errors.New("workflow: payload too large")
ErrPayloadTooLarge matches PayloadTooLargeError.
var ErrRecycle = errors.New("workflow: worker recycled")
ErrRecycle is matched by the error a client reports when it stopped itself because a Recycle limit was reached. A supervisor (systemd, a container restart policy, a process manager) then starts a fresh process, which frees whatever the old one leaked.
var ErrScheduleJSON = errors.New("workflow: invalid schedule JSON")
ErrScheduleJSON is wrapped by every error of the JSON form.
var ErrSecretInPayload = fmt.Errorf("%w: job carries a secret; pass a SecretRef instead", ErrInvalid)
ErrSecretInPayload means SecretGuard found what looks like a credential in a job's payload or labels. It wraps ErrInvalid, so errors.Is matches both. The error names where and which rule, never the value.
var ErrSnoozed = errors.New("workflow: step snoozed")
ErrSnoozed matches the error of a step that snoozed (Snooze), for code that inspects an attempt's outcome, such as a lifecycle observer.
var ErrStepCancelled = errors.New("workflow: step cancelled")
ErrStepCancelled is the cause of a step attempt's ctx when an operator cancelled that step (StepCanceller), and matches the step's error when it ends because of it.
var ErrThrottled = errors.New("workflow: upstream rate limit reached")
ErrThrottled means an upstream's rate limit has no room for the call now. UpstreamLimit.Take returns it marked with RetryAfter, so a step that returns it is retried once the bucket has room on a workflow whose RetryPolicy.DelayFor is RetryAfterDelay, and on its normal backoff otherwise.
Functions ¶
func AllWhen ¶
AllWhen returns a Workflow.ScheduleWhen that fires a tick only when every condition does, asking them in order and stopping at the first false. Nil conditions are ignored; with none it always fires.
func AllowReservedLabels ¶
AllowReservedLabels returns a ctx under which an Enqueue may carry the given reserved label keys, adding to what ctx already allows. The package that owns a key calls it around the Enqueue that carries the label it stamped (Spawn for its parent labels, middleware.Unique for its lock labels), so a label it wrote passes CheckJobLabels while the same key typed by an app, or copied in from a request, is refused.
It is a guard against labels arriving by accident or from untrusted input, not a sandbox: code in the same process that calls it can set any label it names, as it could call any other function. Only the owning package should.
func AllowReservedPrefix ¶
AllowReservedPrefix is AllowReservedLabels for every key under prefix, for the package that owns a whole prefix (batch for LabelPrefixBatch, otel for LabelPrefixOTel).
func AppendToChain ¶
AppendToChain puts jobs at the end of the chain of the run whose step ctx is, in the order given. It applies only if the step attempt succeeds, and is refused as PrependToChain is.
func AttachScheduleHooks ¶
func AttachScheduleHooks(c Client, hooks ...ScheduleHooks) error
AttachScheduleHooks registers hooks on c before Start: one TickObserver when any hook needs ticks and one RunFinishedFunc when any needs run ends. It refuses, before registering anything, a hook without a Workflow (ErrInvalid) and a client lacking the notifier a hook needs (an UnsupportedError naming CapTickEvents or CapRunFinished). After Start the registrations return ErrAlreadyStarted.
func AwaitChild ¶
func AwaitChild(ctx context.Context, client Client, name string, job Job, opts ChildOptions) ([]byte, error)
AwaitChild runs job as a child run of the calling step and returns the child's result: the parent waits for the child without holding a worker. The first call spawns the child (Spawn, so the Journal plugin is needed and a retried or snoozed step never spawns it twice); while the child is unfinished it returns Snooze(Poll), so the parent's step is put back without spending an attempt and asks again later; once the child succeeded it returns the child's result (client must implement ResultReader). A child that ended any other way fails the parent's step with a Permanent *RunFailedError, since asking again would see the same end.
The parent's driver needs CapSnooze: without it each look would spend an attempt of the parent step, which could fail before its child ends, so AwaitChild refuses with a Permanent UnsupportedError instead. client is the client the child runs on: the parent's own, or another backend (then set ChildOptions.Parent to the parent's client).
func AwaitChildren ¶
func AwaitChildren(ctx context.Context, client Client, name string, jobs []Job, opts ChildOptions) ([][]byte, error)
AwaitChildren is AwaitChild for several children at once (multiple instances whose number is known when the step runs, results joined back in job order): it spawns them all, snoozes until every one is terminal, and returns their results in the order of jobs. The first child that did not succeed fails the step, as AwaitChild does.
func AwaitResult ¶
AwaitResult waits like Await and returns the run's result: Result when the run succeeded (c must implement ResultReader, else an UnsupportedError), a *RunFailedError when it ended otherwise.
func CanForceComplete ¶
CanForceComplete reports whether ForceComplete accepts a run in status st. Every driver decides with it.
func CanForceFail ¶
CanForceFail reports whether ForceFail accepts a run in status st.
func CheckCaps ¶
CheckCaps returns an UnsupportedError naming wf when have lacks what wf needs, and nil otherwise. Drivers call it from Start.
func CheckJob ¶
CheckJob returns an UnsupportedError when have lacks what job needs (RequiresJob), naming the first field responsible, and nil otherwise. Drivers call it from Enqueue after ValidateJob.
func CheckJobLabels ¶
CheckJobLabels refuses, with an error wrapping ErrInvalid that names the key, a label of job (or of a job in its Chain) under a reserved prefix that ctx does not allow (AllowReservedLabels, AllowReservedPrefix). Every driver calls it first in its public Enqueue (and any other public entry that takes a Job from an app), never on its internal paths (a fired schedule, a chained job, a retry), which carry labels the library wrote. The source labels (LabelSource, LabelSourceRef) pass without an allowance when their values are valid (see WithSource). Keys are checked in sorted order, so the error is stable.
func CheckJobQueue ¶
CheckJobQueue refuses a Job.Queue that differs from wf's effective queue when have lacks CapQueueOverride: an engine that binds a workflow to the workers of its queue cannot send one run elsewhere. Drivers call it from Enqueue once they have looked the workflow up.
func CheckPause ¶
func CheckPause(have Caps, t PauseTarget) error
CheckPause validates t and refuses it with an UnsupportedError when have lacks what it needs. Drivers call it first in Pause and Resume.
func CheckPayload ¶
CheckPayload returns a *PayloadTooLargeError when job.Payload, or the payload of any job in its Chain, is over limit bytes, and nil otherwise or when limit <= 0. Drivers with a hard cap call it from Enqueue, before anything is stored.
func CheckQueueLimits ¶
func CheckQueueLimits(have Caps, qs []QueueConfig) error
CheckQueueLimits refuses a QueueConfig list that sets Limit when have lacks CapQueueLimitGlobal, so a driver that cannot count attempts across clients says so at Start instead of silently enforcing only a per-client cap. Drivers call it next to ValidateQueues.
func CheckRunQuery ¶
CheckRunQuery refuses what q asks for that have lacks: LabelPrefixes without CapLabelPrefix, Tenant without CapTenant, DueFrom or DueTo without CapDueQuery, as an *UnsupportedError (matches ErrUnsupported) naming the field. Every RunQuerier and BulkCanceller calls it after ValidateRunQuery, so no driver silently drops a filter.
func CheckScheduleCaps ¶
CheckScheduleCaps returns an UnsupportedError when have lacks what s needs, and nil otherwise. Its Field names the cause ("Schedule", "Schedule.Location", "Schedule.Windows", "Schedule.Overlap", "Schedule.Seconds", "Schedule.Active", "Schedule.KeepRuns" or "Schedule.Misfire") and its Reason the workflow. Drivers call it from Schedule, after ValidateSchedule.
func DecodePayload ¶
DecodePayload decodes a JSON payload (a step input, RunInfo.Payload, a result) into a T. Empty input decodes to the zero T. A decode failure is Permanent: retrying the same bytes cannot succeed.
func DefaultNodeID ¶
func DefaultNodeID() string
DefaultNodeID is the node identity a driver uses when the app gives none: the host name and the process ID ("web-1:4182"), so two processes on one host are told apart. An app on a platform whose host names are meaningless (containers) passes its own through the driver's node-ID option.
func DeferHTTP ¶
func DeferHTTP(next http.Handler, opts DeferOptions) http.Handler
DeferHTTP wraps next so every request gets its own Deferred on its context: jobs enqueued during the request through a DeferEnqueue client are sent after next returns, when opts.Succeeded accepts the response status, and dropped when it does not or when next panics (the panic goes on up after the drop). The flush runs on the request's context without its cancellation, so a client that hung up after a successful response does not cancel the jobs.
func DeferSucceeded ¶
DeferSucceeded is DeferOptions.Succeeded's default: a status below 400.
func Discard ¶
Discard marks err as "drop this run": the step is not retried, OnFailure does not run, and the run ends StatusDiscarded on a driver with CapRunStates (StatusCancelled on one with only CapDiscard; see EndStatus), with an Err matching ErrDiscarded and ErrCancelled. Use it when the work no longer makes sense (the order it was about is gone), as opposed to Permanent, which fails the run so an operator sees it. On a driver without CapDiscard it behaves as Permanent: the run fails, the direction that never hides a failure. Discard(nil) is nil.
func DiscardedError ¶
DiscardedError is the Err a driver records for a run a step discarded: it matches ErrDiscarded, ErrCancelled and the step's own error under errors.Is. step names the step; err is what the step returned.
func DrainContext ¶
DrainContext returns ctx limited to a grace period: the running attempts get that long to finish, then the drain cuts the rest off. A grace of zero or less means no waiting at all (the returned ctx is already ended), never "wait forever": a shutdown that can hang is the bug a grace period exists to prevent. A tighter deadline already on ctx wins. The caller must call the returned cancel.
func DrainOutcome ¶
func DrainOutcome(report DrainReport, ctxErr error) error
DrainOutcome is the one decision a drain makes about its result: given what was still running at the deadline and why the deadline ended, did the drain succeed? It is nil when nothing was cut off, even if ctx also ended (the work finished in the same instant), and ctxErr otherwise. Every driver's Drain returns this, so they cannot disagree.
func DrainTimeout ¶
DrainTimeout drains d, giving running attempts at most grace to finish (see DrainContext), so a caller does not hand-roll context.WithTimeout. It returns what d.Drain returns.
func EffectiveQueue ¶
EffectiveQueue is the queue a run of wf enqueued as job runs on: job.Queue, else wf.Queue, else DefaultQueue. Every driver calls it, so no two drivers resolve a queue differently.
func FiredLabels ¶
FiredLabels returns the labels of a run a schedule fires: a copy of the schedule's labels plus LabelSchedule and LabelScheduleName. Drivers with CapTickEvents build every fired job's labels with it.
func FiredLabelsAt ¶
func FiredLabelsAt(labels map[string]string, id ScheduleID, name string, at time.Time) map[string]string
FiredLabelsAt is FiredLabels plus LabelTickAt set to at. Drivers with CapTickTime build every fired job's labels with it.
func ForcedFailure ¶
ForcedFailure returns the error a force-failed run carries: it matches ErrForced and its message ends with reason. Drivers store its message as RunInfo.Error.
func HasStepEventSink ¶ added in v0.1.1
HasStepEventSink reports whether any sink is bound on ctx, so a step can skip building an expensive message nobody will read.
func HeapObjectBytes ¶
func HeapObjectBytes() uint64
HeapObjectBytes returns the bytes held by live and not yet swept heap objects (runtime/metrics), the default Recycle.Memory. It reads no process-wide state other than the runtime's own counters.
func Heartbeat ¶
Heartbeat reports from inside a step that the attempt is alive, so a long job is not mistaken for a stuck one (Step.HeartbeatTimeout) and, on a durable driver, keeps its lease. It is cheap enough to call in a loop. Outside a step, or on a driver without CapHeartbeat, it returns nil and does nothing: liveness is a hint, and a portable handler must not fail because one driver has no use for it.
func InEnvironment ¶
InEnvironment returns a Workflow.ScheduleWhen that fires ticks only when current is one of allowed ("production"), for schedules that must not run in staging or on a laptop. The environment names are the app's: the library has no notion of one and reads nothing from the process, so the app passes its own value (a config field, an env var it already reads). With no allowed names it never fires. Skipped ticks are reported as TickSkipped with TickReasonWhen.
The decision is made once, here: the returned function ignores the tick. Combine it with other conditions through AllWhen.
func IsDiscarded ¶
IsDiscarded reports whether err, or any error it wraps, was marked by Discard.
func IsPermanent ¶
IsPermanent reports whether err, or any error it wraps, was marked by Permanent.
func IsRelease ¶
IsRelease reports whether err carries a RetryAfter delay: a release, which counts against RetryPolicy.Max but not against MaxErrors.
func IsTimeout ¶
IsTimeout reports whether err, or any error it wraps, is a missed deadline (matches ErrTimeout).
func IsUTC ¶
IsUTC reports whether loc evaluates times as UTC: nil (which Workflow. CronLocation treats as UTC), time.UTC, or a zone named "UTC".
func IsUnavailable ¶
IsUnavailable reports whether err, or any error it wraps, is ErrUnavailable or was marked by Unavailable.
func LastTick ¶
LastTick returns the newest tick of src in (from, to], or zero when there is none. Drivers use it to fill Tick.At of a TickMisfired.
func Loop ¶
func Loop[S any](ctx context.Context, name string, init S, limit int, body func(ctx context.Context, i int, state S) (next S, done bool, err error)) (S, error)
Loop is a structured loop (WC-21): it calls body with the iteration number and the current state until body reports done, and returns the final state. Each iteration is memoised under name and its number, so a retried step replays the finished iterations from the journal (without calling body) and resumes at the first one that did not finish.
limit bounds the iterations and is required (at least 1), so a loop is never unbounded by accident; reaching it without done returns a Permanent ErrLoopLimit. A body error is returned as it is and the iteration runs again on the next attempt. The state is stored as JSON after every iteration, so keep it small (a cursor, a counter), not the data itself. Loop needs the Journal plugin, as Memo does.
For a loop over steps of the graph rather than code (repeat a block of steps), enqueue a new run of the workflow from its last step: a run's graph is fixed when it starts.
func MapEach ¶
func MapEach[In, Out any](ctx context.Context, name string, items []In, fn func(ctx context.Context, i int, item In) (Out, error), opts ...MapOption) ([]Out, error)
MapEach runs fn once per item and returns the results in the order of items: multiple instances whose number is known only when the step runs (WC-14), joined back into the step's result. Each item is memoised under name and its index, as Memo does, so a retried step runs again only the items that did not succeed; items must therefore keep their order across attempts.
Every item runs even when others fail, so successes are kept for the next attempt; the error is then the join of each failed item's error (labelled with its index), and the results are nil. A Permanent error from any item makes the joined error Permanent. When ctx ends, items not yet begun are skipped with ctx's error.
To run each instance as its own run instead (its own retries, its own worker, visible in run listings), enqueue them with Spawn. MapEach needs the Journal plugin, as Memo does.
func MarkDuplicate ¶
MarkDuplicate records that the Enqueue handling ctx returned an existing run. Drivers call it when an idempotency key returns the holder's run; middleware.Unique calls it when it returns the holder instead of ErrDuplicate. It does nothing when the caller did not ask.
func MarkInserted ¶
MarkInserted records that the Enqueue handling ctx started a new run. Drivers call it once the run is stored. The first mark wins, so a middleware that marked a duplicate is not overwritten.
func Memo ¶
func Memo[T any](ctx context.Context, name string, fn func(ctx context.Context) (T, error)) (T, error)
Memo runs fn once per run under name and returns its result; on any later attempt of the same step of the same run (a step retry, or a run retried with RunRetrier) it returns the stored result without calling fn. This is step memoisation as a journal: code re-runs from the top and each named piece of work returns what it returned the first time.
Rules a caller keeps:
- Names are stable across attempts and unique within one attempt of a step; a name used twice in one attempt is refused (Permanent, ErrInvalid), because the second call would silently get the first call's result.
- An error from fn is returned and nothing is stored, so the next attempt calls fn again.
- The result is stored as JSON. A stored value that no longer decodes into T (the type changed between deploys) is a Permanent error.
- The guarantee is at least once: if the process dies after fn returns and before Save, or Save fails, fn runs again on the next attempt. Work that must not repeat also carries an idempotency key the remote side honours.
Memo needs the Journal plugin (or WithJournal) on the client, and returns a Permanent ErrNoJournal without it. Memo is safe to call from several goroutines of one step with different names.
func NextActive ¶
NextActive is s.Next limited to a: the first fire time strictly after `after` that lies inside a, or the zero time when the period has ended (or s never fires again). Before From it starts the search just before From, so a tick exactly at From fires. Every driver computes ticks of a schedule with an Active period through it, so the bounds mean the same everywhere.
func OverlapRefusal ¶
func OverlapRefusal(id ScheduleID, prev RunRef, st Status) error
OverlapRefusal returns the error TriggerSchedule gives when the overlap policy holds a trigger back. It matches ErrOverlap and names the schedule and the unfinished run.
func ParseCron ¶
ParseCron parses a schedule's expression the way every driver must: seconds true (Schedule.Seconds, Workflow.CronSeconds) reads six fields with cron.ParseSeconds; otherwise custom parses it when set (a driver's WithCronParser dialect), else cron.Parse. The seconds flag wins over a custom dialect because it is part of the schedule's stored meaning, while a custom parser is one process's choice.
func PayloadVersionOf ¶
PayloadVersionOf returns the payload version a run's labels carry, and 0 when none is set (a job enqueued before versioning, which counts as the oldest version).
func PeriodKey ¶
PeriodKey names the period of p that t falls in, on loc's wall clock: "2026-10-14", "2026-W42", "2026-10", "2026-Q4", "2026". Every instant of one period gets the same key, so a key built from it is stable across retries, nodes and late fires. loc is required and time.Local is refused, as for Schedule.Location: a period boundary depends on the zone, and the host's zone differs from machine to machine. Errors wrap ErrInvalid.
func Permanent ¶
Permanent marks err as non-retryable: the step fails now and its remaining attempts are skipped. Permanent(nil) is nil.
func PickWeighted ¶
PickWeighted returns the index of the queue a weighted pick chooses from weights (each >= 1 here; a zero or negative weight counts as 1) given r, a uniform value in [0, 1). Every driver that weights queues calls it with its injectable random source, so all of them choose the same way and a test can pin the draw. It returns -1 for no weights.
func PrependToChain ¶
PrependToChain puts jobs in front of the rest of the chain of the run whose step ctx is, so jobs[0] runs next once the run succeeds. Several calls stack: the jobs of the latest call run first. It applies only if the step attempt succeeds. Outside a step, or on a driver without CapChain (an engine whose run shape is fixed at registration), it returns an UnsupportedError; start a child run instead there. A job that fails ValidateJob or the driver's CheckJob is refused at once, and so is one carrying a reserved label that ctx does not allow (CheckJobLabels): a chained job reaches the driver's internal enqueue, so this is where an app's labels are checked.
func ProcessChunks ¶
func ProcessChunks(ctx context.Context, name string, opts ChunkOptions, fn func(ctx context.Context, offset, size int64) (handled int64, err error)) (int64, error)
ProcessChunks runs an export or import as chunks of Size items with progress (SA-05): it calls fn with each chunk's offset and size until fn handles fewer items than size (the source ran dry), reporting progress after every chunk with ReportProgress, so a page can show how far the job has got. It returns the number of items handled.
Each chunk is a Loop iteration, memoised in the journal, so a retried step resumes at the first chunk that did not finish instead of starting over; it needs the Journal plugin. fn must therefore be idempotent per chunk (an upsert, or a file part named by its offset), since a crash after fn and before the journal's save runs that chunk again.
The caller asked for progress, so a driver that keeps none is not silently ignored: the first report's UnsupportedError fails the step (Permanent). Reaching MaxChunks is a Permanent ErrLoopLimit.
func RecordEvent ¶ added in v0.1.1
RecordEvent records msg with attrs on the current step's timeline, at slog.LevelInfo. With no sink bound it does nothing: an event is a log line, not data the caller asked to keep (contrast ReportProgress, which errors), so a step records events the same way with or without a recorder. Values are rendered with slog.Value.String after Resolve.
func RecordEventLevel ¶ added in v0.1.1
RecordEventLevel is RecordEvent at a chosen level (a warning in a timeline that a sink may colour or alert on).
func RegisterFunc ¶
func RegisterFunc(client Client, f FuncDefinition) error
RegisterFunc registers f on client. A Func whose function is nil fails Validate (no steps) with ErrInvalid.
func ReportProgress ¶
ReportProgress records p as the run's progress from inside a step, so an app (an export page) can show how far a long job has got: read it back with RunInfo.Progress (RunQuerier) or the LifecycleProgress event. It validates p first (ErrInvalid). Outside a step, or on a driver that keeps no progress, it returns an UnsupportedError: unlike Heartbeat, the caller asked for data to be kept, so silently dropping it would be a lie. A handler that does not care can ignore the error.
func ReservedLabel ¶
ReservedLabel reports whether key falls under a reserved prefix, and which.
func ReservedLabelPrefixes ¶
func ReservedLabelPrefixes() []string
ReservedLabelPrefixes returns every label prefix the library reserves for itself, as a fresh slice.
func ResolveSecret ¶
ResolveSecret returns the value ref names, from r. An empty ref is a Permanent ErrNoSecretRef; r's own errors are returned as they are, so a vault outage retries the step.
func RetryAfter ¶
RetryAfter marks err with the wait the next retry should observe, such as the Retry-After of a 429. It is honoured only under a policy whose DelayFor reads it (RetryAfterDelay); under any other policy the error retries on the normal Backoff, on every driver. Permanent still wins: RetryAfter(Permanent(err), d) is not retried. RetryAfter(nil, d) is nil and a negative d counts as zero.
func RetryAfterDelay ¶
RetryAfterDelay is the stock RetryPolicy.DelayFor: the delay of the outermost RetryAfter in err's chain, or false when there is none.
func RunData ¶
RunData returns the value SetRunData stored under name in this run, and false when none was stored. A stored value that does not decode into T is a Permanent error.
func RunOutput ¶
RunOutput is the result of a succeeded run of wf whose step outputs are outs: with one final step (Sinks), its output unchanged; with several, a JSON object keyed by step name, built by the StepInput rule. Every driver with CapResult computes the result with it, so a one-step job's result is simply its handler's return value everywhere.
func Serve ¶
Serve starts client and blocks until it stops, so a worker's main is one call whose error says whether to exit for a restart:
if err := workflow.Serve(ctx, client); errors.Is(err, workflow.ErrRecycle) {
os.Exit(0) // the supervisor starts a fresh process
}
It returns Start's error, the client's StopCause (a *RecycleError) when a recycle limit stopped it, and nil for every requested stop. It waits on the Drainer's Done channel; a client without CapDrain is waited on until ctx ends.
func SetOutcome ¶ added in v0.1.1
SetOutcome records the run's outcome from inside a step, with an optional note for a person ("3 of 100 rows failed"). The last call in a run wins. It validates o and the note's length first (ErrInvalid). Without a setter on ctx (no run recorder, or ctx is not a step's) it returns an UnsupportedError, as ReportProgress does: the caller asked for data to be kept, so dropping it silently would be a lie. A step that sets an outcome whether or not one is recorded can ignore that error.
func SetRunData ¶
SetRunData stores v under name for the whole run (WX-01): every later step of the run, and every later attempt, reads it with RunData. It is the run scope between a step's own input and output (block scope) and the app's database (global scope). It writes through the journal, so it needs the Journal plugin and is as durable as the JournalStore; values are forgotten with the run's journal when the run finishes.
A later SetRunData of the same name replaces the value. Two steps running in parallel that set one name leave whichever wrote last, so give parallel steps their own names. v is stored as JSON; keep it small.
func ShutdownHook ¶
ShutdownHook adapts a Drainer to the one shape every phased shutdown manager accepts: a func(context.Context) error to register in a phase. The hook drains d within grace, or sooner if the phase's own ctx ends first, and returns Drain's error (the ctx error when attempts were cut off). It is a plain func so any manager (ubgo/shutdown, errgroup-style stoppers, signal.NotifyContext loops) can call it; this package imports none of them. Register it in the phase that runs before the database and the HTTP listener close, since running attempts still use both.
func ShutdownHookReport ¶
func ShutdownHookReport(d Drainer, grace time.Duration, onReport func(report DrainReport, detailed bool)) func(context.Context) error
ShutdownHookReport is ShutdownHook that hands the report to onReport (when non-nil) before returning, so the phase can log what was cut off. onReport also runs when the report is not detailed (see DrainTimeoutReport), with a zero report and detailed false.
func Sinks ¶
Sinks returns the names of wf's final steps: the steps no other step lists in After, in declaration order.
func Snooze ¶
Snooze, returned by a step handler, puts the step back to run again after d without spending an attempt: the next try has the same StepInfo.Attempt, the retry policy is not consulted (a snooze is not a failure, so it spends none of Max or MaxErrors; Until keeps counting from the first attempt), it is not counted in RunInfo.Attempts nor recorded in RunInfo.Errors, and the run reports StatusWaiting meanwhile. Use it for "not yet" (a resource is not ready, an upstream limit is spent) as opposed to a failure. A negative d is zero (run again as soon as a worker is free); a d over MaxSnooze is MaxSnooze. A driver without CapSnooze treats it as an ordinary error, so it spends an attempt: check the capability when that matters.
func SnoozeDelay ¶
SnoozeDelay reports whether err, or any error it wraps, is a Snooze, and its delay.
func StackOf ¶
StackOf returns the stack trace of the first error in err's chain that implements StackTracer, cut to MaxErrorStack bytes; empty when there is none.
func StepInput ¶
StepInput builds the input bytes a step receives, so every driver follows one rule:
- no parents: the run's payload;
- one parent: that parent's output, unchanged;
- several parents: a JSON object keyed by parent name, in the order of parents, each output embedded as raw JSON when it is valid JSON and as a JSON string otherwise. A parent missing from outs is null.
A handler joining several parents decodes the object (Define does this for a struct whose fields are named after the parents).
The rule is part of the contract, not an implementation detail: middleware.DecodeStep relies on valid JSON being embedded unchanged (a sealed value is a JSON string) and on empty or missing outputs being null, so changing it means changing the codec with it.
func Supports ¶
Supports reports whether c really does op: its OpProber's answer when it has one (a wrapper), else whether c implements the operation's interface. An unknown op is false.
func TickTime ¶
TickTime returns the tick a run's labels name (LabelTickAt), in UTC; false when the run was not fired by a schedule, or by a driver without CapTickTime, or the label does not parse.
func TimeoutError ¶
TimeoutError returns an error matching kind (ErrStepTimeout, ErrWaitTimeout, ErrHeartbeatTimeout or ErrRunTimeout) and ErrTimeout, with detail in its message, such as `step "fetch" attempt 2 after 30s`. A kind that is not one of them is still matched by kind, and by ErrTimeout. Drivers build every timeout error with it, so IsTimeout is reliable on every driver with CapRunTimeout.
func TriggeredLabels ¶
TriggeredLabels is FiredLabels plus LabelTrigger, the labels of a run TriggerSchedule starts. Drivers build a triggered job's labels with it.
func TriggeredLabelsAt ¶
func TriggeredLabelsAt(labels map[string]string, id ScheduleID, name string, at time.Time) map[string]string
TriggeredLabelsAt is TriggeredLabels plus LabelTickAt set to at, the trigger time. Drivers with CapTickTime build a triggered job's labels with it.
func Unavailable ¶
Unavailable marks err as "the backend could not be reached": errors.Is matches both ErrUnavailable and err itself. Unavailable(nil) is nil.
func Unsupported ¶
Unsupported returns an UnsupportedError for field.
func Upcast ¶
Upcast brings raw, written at version from, up to version to by applying ups[from], ups[from+1], ... ups[to-1] in turn; ups[n] turns version n into n+1. A missing step in the chain, a from above to, or an upcaster's error is Permanent, since retrying the same bytes cannot help.
func Validate ¶
Validate checks the shape of wf, independent of any driver: a name, at least one step, unique non-empty step names, exactly one kind per step, After naming existing steps and forming no cycle, and policies within their ranges. Every error wraps ErrInvalid and names the step or field. Drivers call it from Register; it does not parse cron expressions, which the driver's cron parser checks.
func ValidateCancelQuery ¶
ValidateCancelQuery checks q for CancelAll: ValidateRunQuery, no terminal status, and no Limit or Cursor (a bulk cancel covers every match, so paging it would be ambiguous). Errors wrap ErrInvalid.
func ValidateClosure ¶
func ValidateClosure(c ClosureJob) error
ValidateClosure checks c: a Name, a Fn, a valid queue and retry policy, a Timeout that is not negative, and no label under a reserved prefix. Errors wrap ErrInvalid.
func ValidateJob ¶
ValidateJob checks the shape of a job, independent of any driver and of the workflow it names: a workflow name, a valid Queue when one is set, and a Retry override within range that carries no functions (a durable driver stores the job and cannot store a NoRetryOn or DelayFor). Errors wrap ErrInvalid. Drivers call it first in Enqueue. It does not check labels, which need the caller's ctx: the library's own packages set reserved labels on the jobs they pass down, so a driver's public Enqueue also calls CheckJobLabels(ctx, job).
func ValidateLabels ¶
ValidateLabels refuses app labels under a reserved prefix with an error wrapping ErrInvalid that names the key and the prefix, instead of letting the owning package drop or overwrite them silently. Keys are checked in sorted order, so the error is the same on every call.
The source labels (LabelSource, LabelSourceRef) are the exception: an app sets them (WithSource), so they pass when their values are valid.
It is for labels an app wrote. The library's own packages set reserved labels on the jobs they pass down, so a driver's Enqueue calls CheckJobLabels instead, which lets those through; ValidateSchedule does (schedules are written only by apps), and each package that owns a prefix refuses incoming labels under its own.
func ValidatePauseTarget ¶
func ValidatePauseTarget(t PauseTarget) error
ValidatePauseTarget refuses a target that sets none or more than one field, or names an invalid queue. Errors wrap ErrInvalid.
func ValidateProgress ¶
ValidateProgress checks p: Percent within [0, 100] and Data within MaxProgressData. Errors wrap ErrInvalid.
func ValidateQueueName ¶
ValidateQueueName checks a queue name: 1 to 64 characters from letters, digits and ".", "_", ":", "-". Errors wrap ErrInvalid.
func ValidateQueues ¶
func ValidateQueues(qs []QueueConfig) error
ValidateQueues checks a QueueConfig list: valid, unique names and Workers not negative. Errors wrap ErrInvalid. Drivers call it from their option or from Start.
func ValidateRecycle ¶
ValidateRecycle refuses negative limits and a negative CheckEvery. Errors wrap ErrInvalid.
func ValidateRunQuery ¶
ValidateRunQuery checks q: known statuses, a valid queue name when one is set, From before To when both are set, and 0 <= Limit <= MaxRunsLimit. Errors wrap ErrInvalid.
func ValidateSchedule ¶
ValidateSchedule checks the fields every driver needs: a workflow, an expression, an explicit time zone (time.Local is refused), well-formed windows, a well-formed misfire policy and no label under a reserved prefix (see ValidateLabels). Errors wrap ErrInvalid. The cron expression itself is parsed by the driver.
func ValidateScheduleTarget ¶
func ValidateScheduleTarget(t ScheduleTarget) error
ValidateScheduleTarget refuses a target with neither or both fields set. Errors wrap ErrInvalid.
func ValidateSource ¶ added in v0.1.1
ValidateSource checks s: non-empty, at most MaxSourceLen bytes, only lower-case letters, digits and . _ - (so it is safe as a filter value, a metric label and a URL segment). Errors wrap ErrInvalid.
func ValidateTenant ¶
ValidateTenant checks a Job.Tenant or RunQuery.Tenant: 1 to 128 characters from letters, digits and ".", "_", ":", "-", "@". The characters are the queue-name set plus "@", so a tenant ID can be put into a key, a label or a log line without escaping. Errors wrap ErrInvalid.
func WithChainEditor ¶
func WithChainEditor(ctx context.Context, ed ChainEditor) context.Context
WithChainEditor returns a ctx carrying ed. Drivers call it before invoking a step handler.
func WithDeferred ¶
WithDeferred returns a ctx carrying d, so enqueues made with it through a DeferEnqueue client are held by d.
func WithHeartbeater ¶
func WithHeartbeater(ctx context.Context, h Heartbeater) context.Context
WithHeartbeater returns a ctx carrying h. Drivers call it.
func WithOutcomeSetter ¶ added in v0.1.1
func WithOutcomeSetter(ctx context.Context, s OutcomeSetter) context.Context
WithOutcomeSetter returns a ctx carrying s. A run recorder's step middleware calls it.
func WithProgressReporter ¶
func WithProgressReporter(ctx context.Context, r ProgressReporter) context.Context
WithProgressReporter returns a ctx carrying r. Drivers call it.
func WithStepEventSink ¶ added in v0.1.1
func WithStepEventSink(ctx context.Context, s StepEventSink) context.Context
WithStepEventSink returns a ctx carrying s in addition to the sinks ctx already carries, so a recorder, a tracer and a log capture each bind their own and every one sees every event, in binding order. Plugins and middleware call it on the step ctx before the handler runs. A nil s returns ctx unchanged.
Types ¶
type Active ¶
Active is the period a schedule is live: ticks t with From <= t < Until fire, and ticks outside it are not ticks at all (no TickSkipped, no misfire catch-up across the time before From). A zero From means no start and a zero Until no end; an Until at or before From is refused, since it would mean a schedule that never fires.
type AttemptError ¶
type AttemptError struct {
// Step is the step whose attempt failed.
Step string
// Attempt is the attempt number within that step, from 1.
Attempt int
// Error is the attempt's error message. It is a string because an
// engine stores only the message.
Error string
// At is when the attempt ended, on the driver's clock.
At time.Time
// Node is the node that ran the attempt; empty on a driver without
// CapNodeIdentity.
Node string
// Stack is the error's stack trace when it carries one (StackOf);
// empty otherwise.
Stack string
}
AttemptError is one failed step attempt in a run's error history (RunInfo.Errors). Drivers with CapErrorHistory record one for every attempt that returned an error, oldest first.
func AppendAttemptError ¶
func AppendAttemptError(hist []AttemptError, e AttemptError, max int) []AttemptError
AppendAttemptError appends e to hist and keeps the newest max entries, dropping the oldest. max <= 0 keeps none (history turned off). Every driver bounds its history with it, so the bound means the same thing on each backend.
func NewAttemptError ¶
NewAttemptError builds the history entry of a failed attempt.
type Attempts ¶
type Attempts struct {
// N is the number of attempts made, including the one that just
// failed: 1 after the first.
N int
// Errors is how many of those attempts failed with an error that was
// not a release (IsRelease false), including the one that just failed
// when it is such an error.
Errors int
// First is when the step's first attempt started, on the driver's
// clock. Zero disables Until.
First time.Time
}
Attempts is what a driver knows about one step's attempts when one of them fails.
type AwaitOptions ¶
type AwaitOptions struct {
// Poll is the wait between Status calls; zero means DefaultAwaitPoll.
Poll time.Duration
// Sleep waits d or until ctx ends; nil means a real timer. A test
// passes one driven by a fake clock.
Sleep func(ctx context.Context, d time.Duration) error
}
AwaitOptions tunes Await.
type Backoff ¶
type Backoff struct {
Initial time.Duration
Factor float64 // below 1 is treated as 1 (constant backoff)
Max time.Duration
// Jitter spreads retries that failed together (an outage failing
// thousands of runs at once) so they do not all retry at the same
// instant: a delay d becomes a uniform value in [d*(1-Jitter), d].
// It is a fraction in [0, 1]; zero means no jitter (the exact delay,
// as before). Needs CapRetryJitter.
Jitter float64
}
Backoff is an exponential wait: Initial, then times Factor per attempt, capped at Max, optionally shortened by a random Jitter.
func (Backoff) JitteredDelay ¶
JitteredDelay is Delay(n) shortened by Jitter for the uniform draw u in [0, 1): u = 0 gives the full delay, u near 1 the shortest. The draw is an argument, not read here, so the result is a pure function a test can pin.
type BalanceConfig ¶
type BalanceConfig struct {
// Total is the worker budget shared by the queues. Zero means the
// sum of their current Workers, so balancing moves workers between
// queues without adding any. It is raised to the sum of the minimums
// when lower.
Total int
// Bounds holds per-queue bounds by queue name.
Bounds map[string]QueueBounds
// Default bounds a queue absent from Bounds.
Default QueueBounds
// MaxShift caps how far one Balance call moves a queue's count, up or
// down, so a burst does not swing every worker at once. Zero means no
// cap. With a cap the counts can briefly sum to more or less than
// Total while they converge.
MaxShift int
}
BalanceConfig is the shared configuration of the built-in strategies.
type BalanceLoop ¶
type BalanceLoop struct {
// Scaler is the client to balance (a client declaring
// CapWorkerScale).
Scaler WorkerScaler
// Balancer decides the counts.
Balancer Balancer
// Every is the interval; zero means DefaultBalanceEvery.
Every time.Duration
// Clock drives the interval; nil means clock.Real.
Clock clock.Clock
// OnError receives an error of one round; the loop goes on. Nil logs
// it with Logger (nil: slog.Default()).
OnError func(error)
Logger *slog.Logger
}
BalanceLoop applies a Balancer to a WorkerScaler on an interval. Balancing is off unless an app runs one: no driver balances on its own. Run it on the client's Start ctx so it stops with the client.
type Balancer ¶
Balancer decides how many workers each queue should have, from the queues' loads. It returns the desired count per queue name; a queue it leaves out keeps its count. A count below one is refused by the scaler, so a Balancer never stops a queue. It must be a pure function of loads (time arrives through QueueLoad.OldestWait), so a test can call it directly. ByDepth and ByWait are the built-in strategies; any other policy (a schedule, a cost model, an external autoscaler's answer) implements this interface or BalancerFunc.
func ByDepth ¶
func ByDepth(cfg BalanceConfig) Balancer
ByDepth balances by backlog: after each queue's minimum, the budget is shared in proportion to each queue's Depth plus Running (the work it has in hand), within each queue's maximum.
func ByWait ¶
func ByWait(cfg BalanceConfig) Balancer
ByWait balances by latency: after each queue's minimum, the budget is shared in proportion to how long each queue's oldest attempt has waited, within each queue's maximum, so the queue falling furthest behind gets the most help.
func Proportional ¶
func Proportional(cfg BalanceConfig, weight func(QueueLoad) float64) Balancer
Proportional is the strategy behind ByDepth and ByWait with any weight: every queue gets its minimum, then the rest of the budget is shared in proportion to weight (negative counts as zero), never past a queue's maximum; what a capped queue cannot take goes to the others. When no queue has weight the rest is shared evenly. Ties go to the queue whose name sorts first, so the result is deterministic.
type BalancerFunc ¶
BalancerFunc adapts a function to Balancer.
type Branch ¶
type Branch struct {
// Name identifies the branch; Decide returns it. Unique within the
// Choice.
Name string
// Run is the branch's work. It receives the input the decide step
// received, and its output becomes the Choice's output.
Run Handler
// Retry overrides the workflow default for this branch, as Step.Retry.
Retry *RetryPolicy
// Timeout caps one attempt of this branch, as Step.Timeout.
Timeout time.Duration
}
Branch is one path of a Choice: an ordinary step body with its own retry policy and timeout.
type BucketSpec ¶
BucketSpec is a leaky bucket: it holds at most Capacity units of cost and drains LeakPerSecond units every second. A call of cost c fits when the level plus c is at most Capacity. Shopify's REST limit is Capacity 40, LeakPerSecond 2 with cost 1 per call; its GraphQL limit is cost based (Capacity 1000, LeakPerSecond 50, cost per query).
type BucketStore ¶
type BucketStore interface {
// Take adds cost to key's level and returns zero when it fits;
// otherwise it adds nothing and returns how long until it would fit.
Take(ctx context.Context, key string, spec BucketSpec, cost float64, now time.Time) (wait time.Duration, err error)
// Observe sets key's level to what the upstream reported (Shopify's
// "32/40" call-limit header, a GraphQL throttle status), so the
// bucket follows the upstream's own count, including calls made by
// other apps on the same credential.
Observe(ctx context.Context, key string, spec BucketSpec, level float64, now time.Time) error
}
BucketStore holds bucket levels per key (one key per credential or shop). MemoryBuckets keeps them in this process; a shared store (Redis, SQL) makes several workers draw from one bucket, which is what an upstream limit per credential needs once more than one process calls it. Implementations apply the leak themselves, from the now they are given, and must be safe for concurrent use; Take must check and add in one atomic step.
type BulkCanceller ¶
BulkCanceller cancels every run matching a query, for example to clear a queue's backlog. A client implements it when it declares CapCancelAll.
The statuses cancelled are CancelStatuses(q): by default only runs not yet started; running and waiting runs only when listed. Each run is cancelled exactly as Cancel would cancel it (its key released, its run-finished event delivered). q must pass ValidateCancelQuery. It returns how many runs it cancelled.
type Caps ¶
type Caps uint64
Caps is a set of capabilities a driver really has. Drivers declare it through Client.Capabilities; Start compares it with Requires for every registered workflow and refuses what cannot be honoured, so a missing feature is an error at start-up rather than a silent no-op at run time.
Caps is 64 bits and 58 are used, so a new bit is kept for a spec field that Requires, RequiresJob or RequiresSchedule must check, or for data a driver would otherwise leave silently empty. An optional interface whose call can answer for itself (RunRescheduler, StepCanceller, HealthChecker, LifecycleNotifier, ReportProgress) gets no bit: it returns an UnsupportedError when the driver cannot do it.
const ( // CapDurable: runs, waits and sleeps survive a process restart. CapDurable Caps = 1 << iota // CapDelayedStart: Job.RunAt is honoured. CapDelayedStart // CapWaitForEvent: wait steps suspend without holding a worker until // a Signal or their deadline. Whether the wait survives a restart is // CapDurable's claim. CapWaitForEvent // CapDurableSleep: sleep steps suspend without holding a worker. // Whether the sleep survives a restart is CapDurable's claim. CapDurableSleep // CapConcurrencyKey: Workflow.Concurrency is enforced per key. CapConcurrencyKey // CapRateLimit: Workflow.RateLimits are enforced. CapRateLimit // CapIdempotencyKey: Job.IdempotencyKey dedups enqueues. CapIdempotencyKey // CapCron: Workflow.Cron and Client.Schedule fire. CapCron // CapCancel: Client.Cancel stops a run. CapCancel // CapRunFinished: the driver implements RunNotifier with the // guaranteed delivery described there (every terminal run, at least // once). A driver that implements RunNotifier without this capability // delivers best-effort only. CapRunFinished // CapRetryJitter: Backoff.Jitter spreads retry delays. CapRetryJitter // CapRetryDelay: RetryPolicy.DelayFor sets a retry's delay from the // error (RetryAfter). CapRetryDelay // CapMisfireSkip: Misfire{Policy: MisfireSkip} is honoured. CapMisfireSkip // CapMisfireFireOnce: Misfire{Policy: MisfireFireOnce}, with Within, // is honoured. CapMisfireFireOnce // CapMisfireFireAll: Misfire{Policy: MisfireFireAll}, with Within and // Max, is honoured. CapMisfireFireAll // CapCronZone: Workflow.CronLocation and Schedule.Location may be a // zone other than UTC, evaluated in that zone across DST. CapCronZone // CapQueue: Workflow.Queue and Job.Queue route runs to named queues, // and the driver's per-queue worker limits (QueueConfig) are honoured. CapQueue // CapQueueOverride: Job.Queue may differ from the workflow's queue. CapQueueOverride // CapJobRetry: Job.Retry replaces the step policies for one run. CapJobRetry // CapRetryUntil: RetryPolicy.Until is honoured. CapRetryUntil // CapRetryMaxErrors: RetryPolicy.MaxErrors is honoured. CapRetryMaxErrors // CapDiscard: a step returning Discard(err) ends its run cancelled, // not failed. Without it Discard behaves as Permanent. CapDiscard // CapRunQuery: the client implements RunQuerier. CapRunQuery // CapRetryRun: the client implements RunRetrier, re-running a failed // run from the step that failed. CapRetryRun // CapDeleteRun: the client implements RunDeleter. CapDeleteRun // CapCancelAll: the client implements BulkCanceller. CapCancelAll // CapPauseWorkflow: the client implements Pauser for PauseTarget.Workflow // and PauseTarget.All. CapPauseWorkflow // CapPauseQueue: the client implements Pauser for PauseTarget.Queue. CapPauseQueue // CapSchedulePause: the client implements SchedulePauser. CapSchedulePause // CapScheduleWhen: Workflow.ScheduleWhen is asked before every tick. CapScheduleWhen // CapScheduleWindow: Schedule.Windows and Workflow.CronWindows are // honoured. CapScheduleWindow // CapTickEvents: the client implements TickNotifier. CapTickEvents // CapScheduleList: the client implements ScheduleLister. CapScheduleList // CapDrain: the client implements Drainer. CapDrain // CapRunStates: the driver reports StatusRetrying while a step waits // on its retry backoff and StatusDiscarded for a run a step ended with // Discard, and fills RunInfo.NextAttempt and NextAttemptAt (with // CapRunQuery). Without it those moments read StatusQueued or // StatusRunning, and a discarded run StatusCancelled (CapDiscard) or // StatusFailed; see EndStatus. CapRunStates // CapChain: Job.Chain is honoured (each job enqueued when the one // before it succeeds) and a running step may edit its run's chain with // PrependToChain and AppendToChain. The rest of a chain is data stored // with the run, so on a durable driver it survives a crash. CapChain // CapWorkerScale: the client implements WorkerScaler, so a Balancer // can change its per-queue worker counts while it runs. CapWorkerScale // CapClosure: the client implements ClosureEnqueuer, running a Go // function as a job in this process. A closure cannot be stored or // sent, so only an in-process driver declares it. CapClosure // CapLabelPrefix: RunQuery.LabelPrefixes is honoured by Runs and // CancelAll (a label value matched by prefix). CapLabelPrefix // CapOverlapSkip: OverlapSkip on Workflow.CronOverlap and // Schedule.Overlap is honoured, decided from the previous run's // stored status (OverlapPolicy.Blocks). CapOverlapSkip // CapScheduleTrigger: the client implements ScheduleTriggerer. CapScheduleTrigger // CapScheduleHistory: ScheduleInfo.Last (last fire, last run and its // status) is filled, and stored with the schedule on a durable driver. CapScheduleHistory // CapEnqueueOnly: the driver has an enqueue-only mode (an option of // the driver's constructor) whose client enqueues, and answers // Status, Cancel and Signal, without Start and without running any // worker or scheduler, so web nodes enqueue and worker nodes run. On // an enqueue-only client Register is optional (a registered // workflow's checks apply; an unregistered one gets the job-level // checks only) and Start returns nil without starting anything. An // in-process driver cannot declare it: nothing else would ever run // what it enqueued. CapEnqueueOnly // CapErrorHistory: RunInfo.Errors holds the run's failed attempts // (AttemptError), oldest first, bounded by the driver's // error-history option (DefaultErrorHistory when unset). CapErrorHistory // CapShutdownRequeue: a step attempt cut short by a clean stop (the // Start context cancelled) is put back without spending an attempt, // so the next worker runs it with the same attempt number and it // does not count toward the retry limit or appear in the error // history. Only a durable driver can declare it. CapShutdownRequeue // CapForceRun: the client implements RunForcer. CapForceRun // CapSnooze: a step returning Snooze(d) is put back for d without // spending an attempt (see Snooze). CapSnooze // CapHeartbeat: Step.HeartbeatTimeout is honoured (an attempt that // goes that long without Heartbeat fails with ErrHeartbeatTimeout), // and Heartbeat from a step reaches the driver (a durable driver // extends the attempt's lease). CapHeartbeat // CapTenant: Job.Tenant is stored with the run and reported on // StepInfo.Tenant, RunInfo.Tenant and LifecycleEvent.Tenant, and // RunQuery.Tenant filters by it. CapTenant // CapRunTimeout: Workflow.RunTimeout ends a run that runs too long, // Workflow.OnTimeout runs when a run fails because of a timeout, and // every timeout error matches ErrTimeout. CapRunTimeout // CapNodeIdentity: StepInfo.Node, AttemptError.Node and // RunInfo.AttemptedBy name the node that ran each attempt, and the // client implements NodeLister. CapNodeIdentity // CapKeyFIFO: Concurrency.FIFO is honoured: the queued runs of one // concurrency key start in enqueue order (Job.Priority aside). CapKeyFIFO // CapDueQuery: RunQuery.DueFrom and DueTo are honoured and // RunInfo.RunAt is filled. CapDueQuery // CapResult: the client implements ResultReader and fills // RunInfo.Output for a succeeded run. CapResult // CapCronSeconds: Schedule.Seconds and Workflow.CronSeconds (a // six-field expression, seconds first) are honoured. CapCronSeconds // CapScheduleActive: Schedule.Active and Workflow.CronActive (a start // and end instant) are honoured. CapScheduleActive // CapScheduleKeep: Schedule.KeepRuns and Workflow.CronKeepRuns (keep // only the newest N finished runs of a schedule) are honoured. CapScheduleKeep // CapTickTime: every run a schedule fires carries the tick it stands // for in LabelTickAt (TickTimeOf reads it), not only the wall time it // started. CapTickTime // CapQueueLimitGlobal: QueueConfig.Limit caps the in-flight step // attempts of a queue across every client sharing the store, not only // this client's workers (QueueConfig.Workers stays per client). CapQueueLimitGlobal // CapDiscardExhausted: RetryPolicy.DiscardOnExhaust ends a run whose // retries ran out StatusDiscarded instead of StatusFailed. CapDiscardExhausted // CapDrainReport: Drain reports the runs it cut off (DrainReport). CapDrainReport )
The capabilities.
func Requires ¶
Requires computes the capabilities a driver needs to run wf as written. Retry needs come from each Run step's effective policy, so a workflow default every step overrides asks for nothing. Durability is not inferred: a workflow reads the same on any driver, and the app chooses a durable driver when it needs one.
func RequiresJob ¶
RequiresJob computes the capabilities a driver needs to honour job as written: a delayed start, an idempotency key, a queue, a tenant, a per-dispatch retry policy (with what that policy needs) and a chain (CapChain plus what every chained job needs, since the same driver enqueues them). A driver checks it in Enqueue and refuses with an UnsupportedError rather than ignoring the field.
A key counts even when the workflow's policy is IdempotencyOff. The policy is the workflow's to choose, but a key on a job is the caller's request, and a driver without the capability must say so instead of silently ignoring it: a key is never dropped without an error. It also keeps the check a pure function of the job, so a driver can refuse before looking the workflow up.
func RequiresPause ¶
func RequiresPause(t PauseTarget) Caps
RequiresPause is the capability a target needs: CapPauseQueue for a queue, CapPauseWorkflow otherwise.
func RequiresSchedule ¶
RequiresSchedule computes the capabilities a driver needs to honour s as written: CapCron, CapCronZone for a zone other than UTC, the capability of its misfire policy, CapScheduleWindow for windows, CapOverlapSkip for OverlapSkip, CapCronSeconds for Seconds, CapScheduleActive for an Active period, CapScheduleKeep for KeepRuns and CapTenant for Tenant.
type ChainEditor ¶
type ChainEditor interface {
// Prepend puts jobs in front of the rest of the chain, in the order
// given, so jobs[0] runs next once this run succeeds.
Prepend(jobs ...Job) error
// Append puts jobs at the end of the chain, in the order given.
Append(jobs ...Job) error
}
ChainEditor edits the rest of the chain of the run a step belongs to. A driver with CapChain sets one on every step attempt's ctx with WithChainEditor; steps reach it through PrependToChain and AppendToChain.
Edits belong to the attempt: a driver applies them only when the attempt succeeds, so a step that fails and is retried does not add its jobs twice. ChainEdits implements this for drivers.
func ChainEditorFrom ¶
func ChainEditorFrom(ctx context.Context) (ChainEditor, bool)
ChainEditorFrom returns the ChainEditor a driver set on ctx, and false when ctx is not a step's context or the driver lacks CapChain.
type ChainEdits ¶
type ChainEdits struct {
// contains filtered or unexported fields
}
ChainEdits is the ChainEditor drivers use: it checks and buffers the edits of one step attempt (the reserved-label check, which needs the step's ctx, happens in PrependToChain and AppendToChain before it), and Apply produces the run's new chain once the attempt succeeded. Make one per attempt with NewChainEdits. It is safe for concurrent use, since a step may fan out goroutines.
func NewChainEdits ¶
func NewChainEdits(have Caps, chain []Job) *ChainEdits
NewChainEdits returns the editor of one attempt of a run whose stored chain is chain, checking each edit against have with CheckJob and the edited chain against MaxChainJobs. chain is only counted, not kept.
func (*ChainEdits) Append ¶
func (e *ChainEdits) Append(jobs ...Job) error
Append implements ChainEditor.
func (*ChainEdits) Apply ¶
func (e *ChainEdits) Apply(chain []Job) []Job
Apply returns chain with the buffered edits applied: the prepended jobs, then chain, then the appended jobs. chain is not modified.
func (*ChainEdits) Changed ¶
func (e *ChainEdits) Changed() bool
Changed reports whether any edit was made.
func (*ChainEdits) Prepend ¶
func (e *ChainEdits) Prepend(jobs ...Job) error
Prepend implements ChainEditor.
type ChildOptions ¶
type ChildOptions struct {
// Poll is the snooze between looks at unfinished children; zero
// means DefaultChildPoll.
Poll time.Duration
// Spawn are passed to Spawn (SpawnAllowDuplicates).
Spawn []SpawnOption
// Parent is the client the calling step's workflow runs on, whose
// driver must honour Snooze. Nil means the client the children run
// on, the usual case of one backend.
Parent Client
}
ChildOptions tunes AwaitChild and AwaitChildren.
type ChoiceSpec ¶
type ChoiceSpec struct {
// Name is the name of the step that carries the chosen branch's
// output; later steps put it in After.
Name string
// After lists the steps the choice waits for, as Step.After. Decide
// and the chosen branch receive their input (StepInput).
After []string
// Decide returns the Name of the branch to run. A name that is not a
// branch fails the run (Permanent, ErrInvalid); an error retries the
// decide step under the workflow's policy.
Decide func(ctx context.Context, in []byte) (string, error)
// Branches are the paths; at least one.
Branches []Branch
}
ChoiceSpec is an exclusive choice (WC-04): exactly one of several branches runs, picked at run time by Decide.
type ChunkOptions ¶
type ChunkOptions struct {
// Total is the number of items in all, when known; it turns progress
// into a percentage. Zero means unknown: Percent stays 0 until the
// end and Message counts items only.
Total int64
// Size is the number of items per chunk; at least 1.
Size int64
// MaxChunks bounds the loop, so a source that never runs dry cannot
// keep the step busy for ever. Zero means Total/Size + 1 when Total is
// set, and is required when it is not.
MaxChunks int
}
ChunkOptions configures ProcessChunks.
type ChunkProgress ¶
ChunkProgress is the Data ProcessChunks puts in each Progress report, as JSON, for a page that wants the numbers rather than the message.
type Client ¶
type Client interface {
// Register adds a workflow definition. It validates wf (see Validate)
// and fails with ErrAlreadyStarted after Start. Registering the same
// name twice replaces the definition, and its declared Cron schedules
// are replaced, never duplicated.
Register(wf Workflow) error
// Start checks every registered workflow against Capabilities (an error
// wrapping ErrUnsupported names what is missing) and then starts the
// workers. It is the only call that starts goroutines or opens
// connections. A second Start returns ErrAlreadyStarted.
//
// Start returns once the workers are running; it does not block. The
// workers live until ctx is cancelled, which is how an app stops a
// client: no new step starts after that, and running steps see their
// own ctx cancelled. There is no separate Stop.
Start(ctx context.Context) error
// Enqueue starts a run now, or later when job.RunAt is set (which
// needs CapDelayedStart). A delayed run returns a RunRef with only
// ScheduledID set until it starts. With an idempotency key held by an
// unfinished run, it starts nothing and returns that run's RunRef.
Enqueue(ctx context.Context, job Job) (RunRef, error)
// Cancel stops a run or a delayed start. It is idempotent: cancelling
// a finished or already cancelled run returns nil. An unknown ref
// returns ErrNotFound.
Cancel(ctx context.Context, ref RunRef) error
// Status reports where a run is. An unknown ref returns ErrNotFound.
Status(ctx context.Context, ref RunRef) (Status, error)
// Signal delivers ev to runs whose wait step matches its name and key
// (the fixed WaitSpec.Key, or the run's resolved KeyFrom). The
// contract every driver keeps, and the conformance suite asserts:
//
// - An event is kept from the run's enqueue: a run whose wait step
// begins after the Signal still takes it, so a fast reply that
// arrives before the run reaches its wait is not lost.
// - An event sent before a run was enqueued never resumes that run.
// - One run never takes the same event twice, so two wait steps on
// the same name and key need two events.
// - One event resumes every run waiting with that name and key, not
// only the first.
//
// A signal nobody waits for is not an error.
Signal(ctx context.Context, ev Event) error
// Schedule registers a recurring trigger and returns its ID, which the
// app stores to Unschedule later. It needs CapCron, plus CapCronZone
// for a Location other than UTC and the capability of its Misfire
// policy (RequiresSchedule; drivers refuse with CheckScheduleCaps).
Schedule(ctx context.Context, s Schedule) (ScheduleID, error)
// Unschedule removes a schedule. An empty id is a no-op, so callers can
// pass a stored ID without checking it first.
Unschedule(ctx context.Context, id ScheduleID) error
// Capabilities is what this driver can really do. It never changes
// over the life of a client.
Capabilities() Caps
}
Client is the one API an app uses; every driver implements it. App code depends on this interface only, so moving from the memory driver to an engine changes one constructor call and nothing else.
Lifecycle: Register every workflow, then Start once. Register after Start returns ErrAlreadyStarted; Enqueue, Signal and Schedule before Start return ErrNotStarted, so a forgotten Start fails loudly instead of queueing work nobody will run.
func Use ¶
Use applies plugins to c and returns the client to use from then on. Call it before Register and Start: step middleware reaches only workflows registered through the returned client, and observers can only be registered before Start.
Order is the order given, across plugins and within each: the first plugin's middleware is outermost, and its observers are registered first. Use checks every plugin before registering any observer on c, and refuses:
- a nil plugin or an empty name (ErrInvalid);
- two plugins with one name (ErrDuplicatePlugin), within this call only, since Use cannot see plugins applied by an earlier call;
- a RunFinishedProvider when c does not implement RunNotifier (an UnsupportedError naming CapRunFinished and the plugin).
An observer registration that fails (ErrAlreadyStarted after Start) returns the error naming the plugin; observers registered before it stay registered, because RunNotifier has no way to remove one.
The returned client keeps c's Capabilities and forwards RunNotifier and every interface Forward covers to c, so wrappers stacked on it never hide the run-finished hook, run queries, pauses or drain. With no plugins Use returns c unchanged.
Example ¶
ExampleUse adds a plugin that contributes enqueue middleware, step middleware and a run-finished observer, with one call.
package main
import (
"context"
"fmt"
"sync"
"github.com/ubgo/workflow"
"github.com/ubgo/workflow/memory"
)
// auditLog is one ordered log several plugins can share, so a test reads
// the order of their contributions from one place.
type auditLog struct {
mu sync.Mutex
lines []string
done chan struct{}
}
func newAuditLog(runs int) *auditLog { return &auditLog{done: make(chan struct{}, runs)} }
func (l *auditLog) record(s string) {
l.mu.Lock()
defer l.mu.Unlock()
l.lines = append(l.lines, s)
}
func (l *auditLog) all() []string {
l.mu.Lock()
defer l.mu.Unlock()
return append([]string(nil), l.lines...)
}
// auditPlugin uses every contribution: it records each enqueue, each step
// attempt and each run end into its log.
type auditPlugin struct {
name string
log *auditLog
}
func (a *auditPlugin) Name() string { return a.name }
func (a *auditPlugin) EnqueueMiddleware() []workflow.EnqueueMiddleware {
return []workflow.EnqueueMiddleware{func(next workflow.EnqueueFunc) workflow.EnqueueFunc {
return func(ctx context.Context, job workflow.Job) (workflow.RunRef, error) {
a.log.record(a.name + " enqueue " + job.Workflow)
return next(ctx, job)
}
}}
}
func (a *auditPlugin) StepMiddleware() []workflow.Middleware {
return []workflow.Middleware{func(next workflow.Handler) workflow.Handler {
return func(ctx context.Context, in []byte) ([]byte, error) {
a.log.record(a.name + " step")
return next(ctx, in)
}
}}
}
func (a *auditPlugin) RunFinishedObservers() []workflow.RunFinishedFunc {
return []workflow.RunFinishedFunc{func(_ context.Context, ev workflow.RunEvent) {
a.log.record(a.name + " finished " + ev.Workflow + " " + string(ev.Status))
a.log.done <- struct{}{}
}}
}
func main() {
log := newAuditLog(1)
c, err := workflow.Use(memory.New(), &auditPlugin{name: "audit", log: log})
if err != nil {
panic(err)
}
err = c.Register(workflow.Workflow{
Name: "greet",
Steps: []workflow.Step{{Name: "say", Run: func(context.Context, []byte) ([]byte, error) { return nil, nil }}},
})
if err != nil {
panic(err)
}
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
if err := c.Start(ctx); err != nil {
panic(err)
}
if _, err := c.Enqueue(ctx, workflow.Job{Workflow: "greet"}); err != nil {
panic(err)
}
<-log.done
for _, line := range log.all() {
fmt.Println(line)
}
}
Output: audit enqueue greet audit step audit finished greet succeeded
type ClosureEnqueuer ¶
type ClosureEnqueuer interface {
EnqueueClosure(ctx context.Context, c ClosureJob) (RunRef, error)
}
ClosureEnqueuer runs closures. A client implements it when it declares CapClosure. EnqueueClosure validates c (ValidateClosure), then starts a run of ClosureWorkflow whose one step (ClosureStep) calls c.Fn; the RunRef works with Status, Cancel and the run query family as any other. Before Start it returns ErrNotStarted. A closure run is never matched by an idempotency key and is never chained.
type ClosureJob ¶
type ClosureJob struct {
// Name says what the closure is, for logs and run listings (the
// LabelClosure label). Required.
Name string
// Fn is the work. Its ctx carries StepInfo and is cancelled when the
// run is cancelled or the client stops. Returning an error retries it
// under Retry. Required.
Fn func(ctx context.Context) error
// Queue is the queue it runs on; empty means DefaultQueue.
Queue string
// RunAt delays the start; zero means now.
RunAt time.Time
// Labels are copied onto the run, beside LabelClosure.
Labels map[string]string
// Priority orders it among waiting runs, as Job.Priority.
Priority int
// Retry is the retry policy; nil means no retry. Since the closure
// never leaves the process, NoRetryOn and DelayFor are allowed.
Retry *RetryPolicy
// Timeout bounds each attempt; zero means none.
Timeout time.Duration
}
ClosureJob is a Go function run as a job in this process, for small fire-and-forget work that does not deserve a registered workflow (send a log line elsewhere, warm a cache).
A closure cannot be stored or sent: it lives only in the memory of the process that enqueued it, is lost on a restart or crash, and never runs on another machine. Anything that must survive a restart is a registered workflow with a payload. Only an in-process driver declares CapClosure; durable drivers and engines refuse closures.
type Concurrency ¶
type Concurrency struct {
// Key derives the bucket. A zero Key puts every run in one bucket.
Key ConcurrencyKey
// MaxRuns is the cap per bucket; it must be at least 1.
MaxRuns int
// Strategy applies when a bucket is full. Empty means
// ConcurrencyQueueNewest.
Strategy ConcurrencyStrategy
// FIFO promises that the runs queued on one key start in the order
// they were enqueued (higher Job.Priority still goes first). With
// MaxRuns 1 that is a per-key sequence: one at a time, in insert
// order. Only with ConcurrencyQueueNewest (the default). Needs
// CapKeyFIFO.
FIFO bool
}
Concurrency caps simultaneous runs per key: each distinct key value is its own bucket of MaxRuns.
type ConcurrencyKey ¶
ConcurrencyKey derives a key from a job: the concurrency bucket, the rate-limit bucket, or a per-run wait key (WaitSpec.KeyFrom). It has two forms: a Go function (in-process drivers) or an expression in the engine's own language (Hatchet's CEL). A driver that cannot evaluate the form it is given returns an UnsupportedError naming the field. Build it with KeyFunc or KeyExpr.
func KeyExpr ¶
func KeyExpr(expr string) ConcurrencyKey
KeyExpr returns a ConcurrencyKey written in the engine's expression language.
func KeyFunc ¶
func KeyFunc(f func(Job) string) ConcurrencyKey
KeyFunc returns a ConcurrencyKey computed by a Go function.
func (ConcurrencyKey) IsZero ¶
func (k ConcurrencyKey) IsZero() bool
IsZero reports whether the key is unset (one bucket for the workflow).
type ConcurrencyStrategy ¶
type ConcurrencyStrategy string
ConcurrencyStrategy is what happens when a run arrives for a key that is already at Concurrency.MaxRuns.
const ( ConcurrencyQueueNewest ConcurrencyStrategy = "queue_newest" // the new run waits ConcurrencyDropNewest ConcurrencyStrategy = "drop_newest" // the new run is dropped; see below ConcurrencyCancelNewest ConcurrencyStrategy = "cancel_newest" // the new run is cancelled; see below ConcurrencyCancelInProgress ConcurrencyStrategy = "cancel_in_progress" // the running one is cancelled ConcurrencyRoundRobin ConcurrencyStrategy = "round_robin" // keys take turns )
The concurrency strategies. They map onto Hatchet's limit strategies; a driver that cannot do one returns an UnsupportedError.
Under DropNewest and CancelNewest, a run that arrives for a full bucket is still created: Enqueue returns its RunRef and a nil error, its Status is StatusCancelled, and none of its steps runs. An enqueue is therefore never an error because of concurrency, on any driver.
type DeferOptions ¶
type DeferOptions struct {
// Succeeded decides from the response status whether the request
// succeeded, so its held jobs are flushed rather than discarded. Nil
// means DeferSucceeded (any status below 400). A handler that writes
// no status counts as 200.
Succeeded func(status int) bool
// OnError receives a flush that failed for some jobs. The response is
// already written, so it cannot reach the client. Nil logs it with
// Logger.
OnError func(r *http.Request, err error)
// Logger is used by the default OnError. Nil means slog.Default().
Logger *slog.Logger
}
DeferOptions tunes DeferHTTP. The zero value is the default.
type Deferred ¶
type Deferred struct {
// contains filtered or unexported fields
}
Deferred holds enqueues until the work that made them is known to have succeeded: Flush sends them in order, Discard drops them. It is how a request handler enqueues "send the welcome email" without sending it when the request then fails. Put one in a ctx with WithDeferred (DeferHTTP does this per request) and enqueue through a client wrapped with DeferEnqueue.
It is not a transaction: a Flush after a database commit can still fail or the process can die between the two. When the job must be enqueued if and only if a database transaction commits, use the SQL driver's EnqueueTx or its transactional outbox instead.
A Deferred is safe for concurrent use. Use NewDeferred; it is single use: after Flush or Discard every Add returns ErrDeferredClosed.
func DeferredFrom ¶
DeferredFrom returns the Deferred on ctx, and false when there is none.
func (*Deferred) Add ¶
func (d *Deferred) Add(enqueue EnqueueFunc, job Job) error
Add holds job, to be sent with enqueue by Flush. It returns ErrDeferredClosed once the Deferred was flushed or discarded. The job is copied, so the caller may reuse it.
func (*Deferred) Flush ¶
Flush sends every held job, in the order held, and closes d. It sends all of them even when one fails, and returns the refs of the ones sent (the zero RunRef for a failure, so refs[i] is the i-th held job) with every failure joined. A second Flush, or a Flush after Discard, sends nothing.
type DrainCut ¶
type DrainCut struct {
RunID RunID
Workflow string
Step string
Attempt int
Disposition DrainDisposition
}
DrainCut is one step attempt that was still running when a drain ran out of time. Attempt is the number the attempt had (StepInfo.Attempt); on a driver that requeues, the same number runs again, because the cut attempt was not charged.
type DrainDisposition ¶
type DrainDisposition string
DrainDisposition says what became of an attempt a drain cut off. It is a closed set, so a caller can switch on it.
const ( // DrainRequeued: the attempt was handed back to a durable store as // ready and was not charged an attempt (CapShutdownRequeue), so // another client runs it at once. DrainRequeued DrainDisposition = "requeued" // DrainCancelled: nothing outlives the client (an in-memory driver), // so the run became StatusCancelled. DrainCancelled DrainDisposition = "cancelled" // DrainEngineOwned: the engine, not this client, decides what happens // to the cut-off attempt (a Hatchet worker stopping mid-task: the // engine reclaims it after its own timeout). The driver cannot say // more without lying. DrainEngineOwned DrainDisposition = "engine" )
type DrainReport ¶
DrainReport is what a drain did, for the log line a deploy script writes. FinishedSteps counts attempts that returned after the drain began and before its deadline; FinishedRuns counts runs that reached a terminal status from the drain's start until the client stopped, except runs with a cut-off attempt (a driver that cannot see runs end leaves it 0, see its doc); Cut lists the attempts still running at the deadline. A drain that waited for everything has an empty Cut.
func DrainTimeoutReport ¶
func DrainTimeoutReport(ctx context.Context, d Drainer, grace time.Duration) (report DrainReport, detailed bool, err error)
DrainTimeoutReport is DrainTimeout that also returns the report. When d does not implement ReportingDrainer it still drains and returns an empty report with detailed false: the caller learns the report is unknown, not that nothing was cut off.
func (DrainReport) Clean ¶
func (r DrainReport) Clean() bool
Clean reports whether the drain cut nothing off.
func (DrainReport) CutRuns ¶
func (r DrainReport) CutRuns() int
CutRuns is the number of distinct runs with a cut-off attempt (one run can have several attempts running at once).
type DrainTracker ¶
type DrainTracker struct {
// contains filtered or unexported fields
}
DrainTracker is the bookkeeping every driver's drain report needs, as a mechanism a driver embeds: it knows which attempts are running, counts what finishes once a drain began, and fixes the report in two steps. The zero value is ready to use and safe for concurrent use.
Freeze runs when the drain's deadline ends, before the client's context is cancelled: the attempts still running are the cuts, and the steps that ended before are the finished steps. Seal runs once the client has stopped: it adds the runs that reached a terminal status meanwhile (a run's last result is often recorded just after its attempt returns, so counting at Freeze would miss it), minus any run with a cut-off attempt. Invariants: Begin's returned func may be called more than once and only the first call counts; Freeze and Seal are idempotent and their first results are the ones every later call returns; nothing is counted as a finished step after Freeze, so attempts the shutdown itself ends never look like finished work.
func (*DrainTracker) Begin ¶
func (t *DrainTracker) Begin(a DrainCut) (end func())
Begin registers a running attempt and returns the func that ends it.
func (*DrainTracker) Freeze ¶
func (t *DrainTracker) Freeze(disposition DrainDisposition) DrainReport
Freeze fixes the cuts and the finished steps: every attempt still running is a cut, with disposition. Call it when the drain's deadline ended and before the client's context is cancelled. The returned report has no FinishedRuns yet; Seal adds them.
func (*DrainTracker) RunFinished ¶
func (t *DrainTracker) RunFinished(id RunID)
RunFinished notes that run id reached a terminal status. Before StartDrain and after Seal it does nothing.
func (*DrainTracker) Seal ¶
func (t *DrainTracker) Seal() DrainReport
Seal completes the report once the client has stopped; call it after Freeze. A run with a cut-off attempt is not counted as finished, even if the shutdown ended it.
func (*DrainTracker) StartDrain ¶
func (t *DrainTracker) StartDrain()
StartDrain marks that a drain began, so finishes are counted from now.
type Drainer ¶
Drainer stops a client gracefully, as before a deploy. A client implements it when it declares CapDrain.
Drain stops claiming new step attempts at once, waits until the running attempts return or ctx ends, then cancels what is left and shuts the client down exactly as cancelling the Start ctx would. Unfinished runs stay queued on a durable driver, so another client resumes them, and become StatusCancelled on an in-memory one. It returns nil when every running attempt finished, and ctx's error when it had to cut some off. Calling it again, or after the Start ctx ended, waits for the same shutdown. Before Start it returns ErrNotStarted.
Done is closed once the client has fully stopped: after Drain, after the Start ctx was cancelled, or after a driver's stop-when-idle option fired. It is the one way a process waits for its client to finish.
type EnqueueFunc ¶
EnqueueFunc is the shape of Client.Enqueue, so middleware can wrap it.
type EnqueueMiddleware ¶
type EnqueueMiddleware func(next EnqueueFunc) EnqueueFunc
EnqueueMiddleware wraps every enqueue: uniqueness, debouncing, routing and required-label checks live here, above any driver.
func DeferEnqueue ¶
func DeferEnqueue() EnqueueMiddleware
DeferEnqueue is enqueue middleware that holds a job in the ctx's Deferred instead of sending it, returning the zero RunRef and nil: the run does not exist yet, so there is no ref to give. Without a Deferred on the ctx it sends the job at once, so the same client works inside and outside requests. Validation and capability checks happen when the job is flushed, by the client underneath; a caller that needs them up front, or needs the RunRef, enqueues without a Deferred on its ctx. Middleware listed after it in the chain runs at flush time.
func SecretGuard ¶
func SecretGuard(d SecretDetector) EnqueueMiddleware
SecretGuard returns enqueue middleware that refuses a job carrying credentials, with an error matching ErrSecretInPayload (and ErrInvalid) that lists where and which rule, never the value. A payload is stored with the run, shown in admin screens and logs, and kept by retention; a secret in it outlives its rotation. Pass a SecretRef naming the secret and resolve it inside the step instead. A nil detector means NewSecretDetector(SecretRules{}).
type EnqueueMiddlewareProvider ¶
type EnqueueMiddlewareProvider interface {
EnqueueMiddleware() []EnqueueMiddleware
}
EnqueueMiddlewareProvider contributes enqueue middleware, run on every Enqueue through the client Use returns. The first entry sees the job first.
type EnqueueOutcome ¶
type EnqueueOutcome struct {
// Reported is true when something below the caller reported the
// outcome. False means the client cannot tell (a driver without the
// report), so Duplicate false is "unknown", not "new".
Reported bool
// Duplicate is true when no new run was started: the RunRef is an
// existing run's (an idempotency key or a unique lock held by an
// unfinished run).
Duplicate bool
}
EnqueueOutcome is what Enqueue did beyond the RunRef it returned. A caller asks for it with WithEnqueueOutcome; drivers and enqueue middleware fill it with MarkDuplicate and MarkInserted, so it passes through any stack of wrappers without changing Enqueue's signature.
func WithEnqueueOutcome ¶
func WithEnqueueOutcome(ctx context.Context) (context.Context, *EnqueueOutcome)
WithEnqueueOutcome returns a ctx that collects the outcome of the Enqueue it is passed to, and the outcome to read after the call:
ctx, out := workflow.WithEnqueueOutcome(ctx)
ref, err := client.Enqueue(ctx, job)
if err == nil && out.Duplicate { ... }
Use a fresh one per Enqueue: an Enqueue of a chain or a batch made with the same ctx reports its first job only.
type Event ¶
Event resumes runs waiting on a matching WaitSpec. Its Payload becomes the wait step's output.
type Failure ¶
type Failure struct {
// Ref is the run.
Ref RunRef
// Workflow is the workflow name.
Workflow string
// Status is the run's end status for a final failure (StatusFailed,
// or StatusDiscarded/StatusCancelled when the caller asked for
// those); StatusRetrying for an attempt failure.
Status Status
// Step is the failed step when known; empty for a run-level report
// from a driver that does not say which step failed.
Step string
// Attempt is the attempt number for an attempt failure; zero for a
// final failure.
Attempt int
// Err is the error.
Err error
// Final is true when the run has ended, false for an attempt that
// will be retried.
Final bool
// Labels are the run's labels. Receivers must not modify them; an
// integration forwards them only on the caller's opt-in.
Labels map[string]string
// At is when it happened, on the observing process's clock.
At time.Time
}
Failure is one failure handed to a Reporter or an alert sink: a run that ended badly (Final), or, when the reporter opted in, one failed step attempt that will be retried. It never carries the payload, so a third-party error tracker never receives job data.
type Forward ¶
type Forward struct {
Inner Client
}
Forward implements every optional client interface of the run query, pause, schedule, drain, scaling and closure families (RunQuerier, RunRetrier, RunDeleter, BulkCanceller, Pauser, SchedulePauser, ScheduleLister, ScheduleTriggerer, TickNotifier, Drainer, StopCauser, WorkerScaler, ClosureEnqueuer, RunForcer, RunRescheduler, StepCanceller, ResultReader, NodeLister, HealthChecker, LifecycleNotifier and PayloadLimiter) by forwarding to Inner, and refuses with an UnsupportedError naming the capability when Inner does not implement one. A client wrapper embeds it beside the Client it wraps, so stacking a wrapper never hides what the inner client can do:
type wrapper struct {
workflow.Client
workflow.Forward
}
w := &wrapper{Client: inner, Forward: workflow.Forward{Inner: inner}}
A wrapper that rewrites RunRefs or queries (failover's prefixed refs) implements those methods itself instead. RunNotifier is not here: the client Use returns forwards it on its own.
func (Forward) CancelStep ¶
CancelStep forwards to Inner's StepCanceller.
func (Forward) Done ¶
func (f Forward) Done() <-chan struct{}
Done forwards to Inner's Drainer. An inner client without one has no way to say it stopped, so Done returns a nil channel, which never closes; check CapDrain before waiting on it.
func (Forward) DrainWithReport ¶
func (f Forward) DrainWithReport(ctx context.Context) (DrainReport, error)
DrainWithReport forwards to Inner's ReportingDrainer; an inner client without one returns an UnsupportedError naming CapDrainReport.
func (Forward) EnqueueClosure ¶
EnqueueClosure forwards to Inner's ClosureEnqueuer. Enqueue middleware of the wrapper does not run for a closure: it has no Job to rewrite.
func (Forward) ForceComplete ¶
ForceComplete forwards to Inner's RunForcer.
func (Forward) Health ¶
func (f Forward) Health(ctx context.Context) HealthReport
Health forwards to Inner (CheckHealth, so an inner client without a HealthChecker reads as healthy with a check saying it cannot tell).
func (Forward) OnLifecycle ¶
func (f Forward) OnLifecycle(fn LifecycleFunc) error
OnLifecycle forwards to Inner's LifecycleNotifier.
func (Forward) OnTick ¶
func (f Forward) OnTick(fn TickObserver) error
OnTick forwards to Inner's TickNotifier.
func (Forward) Pause ¶
func (f Forward) Pause(ctx context.Context, t PauseTarget) error
Pause forwards to Inner's Pauser.
func (Forward) PauseSchedule ¶
func (f Forward) PauseSchedule(ctx context.Context, t ScheduleTarget) error
PauseSchedule forwards to Inner's SchedulePauser.
func (Forward) Paused ¶
func (f Forward) Paused(ctx context.Context) ([]PauseTarget, error)
Paused forwards to Inner's Pauser.
func (Forward) PayloadLimit ¶
PayloadLimit forwards to Inner's PayloadLimiter; zero (no cap) when Inner has none.
func (Forward) QueueLoads ¶
QueueLoads forwards to Inner's WorkerScaler.
func (Forward) Reschedule ¶
Reschedule forwards to Inner's RunRescheduler.
func (Forward) Resume ¶
func (f Forward) Resume(ctx context.Context, t PauseTarget) error
Resume forwards to Inner's Pauser.
func (Forward) ResumeSchedule ¶
func (f Forward) ResumeSchedule(ctx context.Context, t ScheduleTarget) error
ResumeSchedule forwards to Inner's SchedulePauser.
func (Forward) Schedules ¶
func (f Forward) Schedules(ctx context.Context) ([]ScheduleInfo, error)
Schedules forwards to Inner's ScheduleLister.
func (Forward) SetWorkers ¶
SetWorkers forwards to Inner's WorkerScaler.
func (Forward) StopCause ¶
StopCause forwards to Inner's StopCauser; an inner client without one never stops itself, so it returns nil.
func (Forward) Supports ¶
Supports answers for Inner (see OpProber), so a wrapper embedding Forward reports only what the wrapped client really does.
func (Forward) TriggerSchedule ¶
TriggerSchedule forwards to Inner's ScheduleTriggerer.
type Func ¶
type Func[In any] struct { // contains filtered or unexported fields }
Func is a durable closure: a Go function registered under a name when the process starts and enqueued by that name with an encoded argument. A closure itself cannot be stored or sent to another machine, but a name and a JSON argument can, so a Func survives a restart and runs on any worker that registered the same name, on every driver (a Func is a one-step workflow underneath; no capability beyond the driver's own is needed). For a function that only ever runs in this process, the quick form is EnqueueClosure.
Build one with NewFunc, register it with RegisterFunc(client, f) (or client.Register(f.Workflow())) in every process that runs workers, and enqueue with f.Enqueue or f.Job. The argument is encoded as JSON; input that does not decode is a Permanent error.
func NewFunc ¶
func NewFunc[In any](name string, fn func(ctx context.Context, in In) error, opts ...FuncOption) Func[In]
NewFunc names fn. The name is the workflow name runs carry in RunInfo, RunEvent and StepInfo; Validate checks it at Register. Two Funcs with one name replace each other, as two workflows do.
func (Func[In]) Job ¶
Job returns the Job that runs the Func with in, for a caller that sets RunAt, Labels, Priority, IdempotencyKey or Queue before Enqueue.
type FuncConfig ¶
type FuncConfig struct {
// Queue is the default queue of its runs; empty means DefaultQueue.
Queue string
// Retry is the retry policy of its one step; nil means no retry.
Retry *RetryPolicy
// Timeout bounds each attempt; zero means none.
Timeout time.Duration
// Catch runs once when a run fails for good, with the decoded
// argument. Nil means nothing runs. Its error is reported the way the
// driver reports an OnFailure error.
Catch func(ctx context.Context, in []byte) error
}
FuncConfig holds what FuncOptions set: the same knobs a ClosureJob has that belong to the definition rather than to one enqueue. RunAt, Labels, Priority, IdempotencyKey and a per-run Queue are set on the Job (see Func.Job).
type FuncDefinition ¶
FuncDefinition is what RegisterFunc accepts: a Func of any argument type.
type FuncOption ¶
type FuncOption func(*FuncConfig)
FuncOption configures a Func.
func FuncCatch ¶
func FuncCatch[In any](fn func(ctx context.Context, in In) error) FuncOption
FuncCatch runs fn once when a run fails for good, with the argument the run was enqueued with, decoded. An argument that does not decode is passed as the zero In.
func FuncRetry ¶
func FuncRetry(p *RetryPolicy) FuncOption
FuncRetry sets the retry policy of the Func's step. Nil means no retry.
type Handler ¶
Handler runs one step. Its input follows StepInput: the run's payload for a first step, a single parent's output unchanged, or a JSON object keyed by parent name for several parents. It returns bytes the engine stores as the step's output. Returning Permanent(err) fails the step at once, skipping the remaining retries. Handlers read time from an injected clock, never from time.Now, so a replay or a test sees the same values.
func Define ¶
Define wraps a typed function into a Handler that decodes its input from JSON and encodes its output to JSON, so app code works with structs, not bytes. Empty input decodes to the zero In. Input that does not decode is a Permanent error: retrying the same bytes can never succeed.
func DefineVersioned ¶
func DefineVersioned[In, Out any](current int, ups map[int]Upcaster, fn func(ctx context.Context, in In) (Out, error)) Handler
DefineVersioned is Define for a first step whose payload shape has versions: it reads the run's payload version (PayloadVersionOf on the StepInfo labels), upcasts the input to current with ups, then decodes it into In. Pair it with middleware.PayloadVersion(current), which stamps new jobs. Use it on steps that receive the run's payload (no After); a later step's input is a parent's output, which has no version.
func FailureHandler ¶
FailureHandler is the handler a driver runs when a run fails for good with err: wf.OnTimeout when err is a timeout and OnTimeout is set, else wf.OnFailure (nil when neither applies). Every driver calls it, so none decides differently.
type HealthCheck ¶
type HealthCheck struct {
// Name says what was checked ("database", "engine", "workers").
Name string
// OK is the check's verdict.
OK bool
// Detail says why it failed, or a short fact when it passed.
Detail string
}
HealthCheck is one check of a HealthReport.
type HealthChecker ¶
type HealthChecker interface {
Health(ctx context.Context) HealthReport
}
HealthChecker is an optional interface a client implements to report its health, for a liveness or readiness probe. Health must answer within ctx's deadline and must not block on a broken backend longer than that: a probe that hangs is worse than one that fails.
type HealthReport ¶
type HealthReport struct {
Live, Ready bool
Checks []HealthCheck
}
HealthReport is a client's health: Live means the process should keep running (restart it when false); Ready means it can take work now (route traffic to it, or not, when false). A client is live but not ready before Start and while its backend is unreachable.
func CheckHealth ¶
func CheckHealth(ctx context.Context, c Client) HealthReport
CheckHealth asks c for its health. A client without HealthChecker is reported live and ready, with one check saying it cannot tell: an app on such a driver needs its own probe.
type HeartbeatFunc ¶
HeartbeatFunc adapts a function to Heartbeater.
type Heartbeater ¶
type Heartbeater interface {
// Heartbeat records liveness. details is an optional progress marker
// the driver may keep (nil is fine). An error means the beat was not
// recorded (the backend is unreachable, or the attempt has lost its
// lease and will be run elsewhere); a handler that gets
// ErrCancelled should stop.
Heartbeat(ctx context.Context, details []byte) error
}
Heartbeater receives a running attempt's heartbeats. A driver with CapHeartbeat sets one on the step's ctx (WithHeartbeater) before the Middleware chain runs: it resets the attempt's HeartbeatTimeout clock and, on a durable driver, extends the attempt's lease.
type IdempotencyPolicy ¶
type IdempotencyPolicy string
IdempotencyPolicy says how Job.IdempotencyKey is treated.
const ( IdempotencyOptional IdempotencyPolicy = "optional" // honoured when present (the default) IdempotencyRequired IdempotencyPolicy = "required" // an enqueue without a key fails IdempotencyOff IdempotencyPolicy = "off" // keys are ignored )
The idempotency policies.
type InlineOption ¶
type InlineOption func(*inlineConfig)
InlineOption configures RunInline.
func InlineEvents ¶
func InlineEvents(events ...Event) InlineOption
InlineEvents supplies the events wait steps take: each wait step takes the first event not yet taken whose name and resolved key match. A wait step with no matching event fails as if it timed out. Without this option a workflow with a wait step is refused.
func InlineMiddleware ¶
func InlineMiddleware(mw ...Middleware) InlineOption
InlineMiddleware wraps every step and OnFailure handler, first outermost, the way a driver's middleware option does. Pass the same middleware the app gives its driver to run steps exactly as they run there.
func InlineNow ¶
func InlineNow(now func() time.Time) InlineOption
InlineNow replaces RunInline's clock, read for RetryPolicy.Until and WaitSpec.Deadline. The default is time.Now. Nil is ignored.
func InlineSleep ¶
InlineSleep replaces how RunInline waits (retry backoffs and sleep steps). The default is a real timer that ends early with ctx. A test passes a function that advances a fake clock, or one that returns at once. Nil is ignored.
type InlineResult ¶
type InlineResult struct {
// Outputs holds the output of every step that succeeded (a wait
// step's event payload, a sleep step's input), by step name.
Outputs map[string][]byte
// Attempts holds how many attempts each Run step made.
Attempts map[string]int
}
InlineResult is what RunInline returns, success or failure.
func RunInline ¶
func RunInline(ctx context.Context, wf Workflow, job Job, opts ...InlineOption) (InlineResult, error)
RunInline runs wf once, as job, in the caller's goroutine: no client, no Start, no goroutine of its own, for tests, command-line tools and one-off scripts.
Steps run one at a time in a fixed order: of the steps whose parents have all succeeded, the first in declaration order goes next. Each gets its StepInput, a StepInfo (RunID InlineRunID), its timeout, its retry policy (EffectiveRetriesFor, decided by RetryPolicy.Next, waiting through the sleeper) and the InlineMiddleware. Sleep steps sleep. Wait steps take an InlineEvents event; a workflow with a wait step and no InlineEvents option is refused with an UnsupportedError before anything runs. When a step fails for good, OnFailure runs once with that step's input, and the error is returned wrapped with the step name. A step that returns Discard, or whose policy has DiscardOnExhaust and gives up, ends the call with an error matching ErrDiscarded and ErrCancelled, without OnFailure.
Concurrency, rate limits, idempotency keys, cron, queues, priority and RunAt shape how a driver schedules many runs; a single inline call has nothing to schedule, so it ignores them. It validates wf and job first (ErrInvalid).
type Job ¶
type Job struct {
// Workflow is the registered workflow name.
Workflow string
// Payload is the input of the first steps.
Payload []byte
// RunAt delays the start. Zero means now. Needs CapDelayedStart.
RunAt time.Time
// IdempotencyKey dedups enqueues while a run holding the key has not
// finished. Honoured per the workflow's Idempotency policy; needs
// CapIdempotencyKey.
IdempotencyKey string
// Labels carry app metadata (source, trace) explicitly, so a
// middleware can require one instead of trusting a context convention.
Labels map[string]string
// Tenant is the tenant the run belongs to, a first-class field rather
// than a label convention: drivers store it, report it on
// StepInfo.Tenant and RunInfo.Tenant, and filter by it
// (RunQuery.Tenant). Empty means none. It must pass ValidateTenant.
// Each chained job carries its own Tenant. Needs CapTenant.
Tenant string
// Priority orders runs waiting for a worker: higher goes first. Zero
// is the default priority.
Priority int
// Queue overrides the workflow's queue for this run. Empty means the workflow's (EffectiveQueue). Needs
// CapQueue, and CapQueueOverride when it differs from the workflow's
// queue (CheckJobQueue).
Queue string
// Retry replaces every step's effective policy for this run (a retry
// budget chosen per dispatch), so it wins over Step.Retry and Workflow.Retry;
// see EffectiveRetriesFor. It is data only: NoRetryOn and DelayFor
// must be nil (ValidateJob refuses them), because a durable driver
// stores the job and cannot store a function. Nil means no override.
// Needs CapJobRetry plus what the policy itself needs.
Retry *RetryPolicy
// Chain lists the jobs that run after this one, in order: when this
// run succeeds the driver enqueues Chain[0] carrying Chain[1:] as its
// own Chain (NextInChain), so each job starts only once the one before
// it succeeded. A run that fails, is discarded or is cancelled ends
// the chain. A running step may edit the rest of its run's chain with
// PrependToChain and AppendToChain. The chain is data stored with the
// run, so on a durable driver it survives a crash. Enqueue middleware
// runs for this job only, not for the jobs the driver enqueues later.
// At most MaxChainJobs jobs; each passes ValidateJob. Needs CapChain
// plus what every chained job needs.
Chain []Job
}
Job is one request to run a workflow.
func CloneJob ¶
CloneJob returns a deep copy of j (Payload, Labels, Retry and Chain), so a driver or a recorder can keep a job without sharing the caller's slices and maps.
func FuncJob ¶
FuncJob returns the Job that runs the Func registered under name with arg encoded as JSON, for a caller that knows only the name (another service, an admin action). The driver refuses a name no process registered the way it refuses any unknown workflow.
func JobOf ¶
JobOf builds a Job for workflow whose Payload is v encoded as JSON, the typed counterpart of filling Job.Payload by hand. Labels, RunAt and the rest are set on the returned Job as usual.
func NextInChain ¶
NextInChain is the job a driver enqueues when a run with this chain succeeds: chain[0], carrying its own Chain followed by chain[1:], so the rest of the chain travels with it. It returns false for an empty chain. Every driver calls it, so a chain flattens the same way everywhere. The result shares nothing with chain.
func OncePerPeriod ¶
OncePerPeriod returns job with an IdempotencyKey of "period/<scope>/<PeriodKey>", for work that must happen once per scope per period (SA-02: charge customer 42 once a month): every enqueue of the same scope in the same period, from any node and any number of schedule fires, carries the same key. It needs CapIdempotencyKey. A job that already has a key is refused (ErrInvalid) rather than overwritten.
What the key does and does not cover: a driver dedups on it while the run holding it is unfinished (Client.Enqueue). To keep a later enqueue in the same period from running again after that run succeeded, also pass the same key to the payment provider's own idempotency key (the period key alone, PeriodKey, is the stable value for that), and memoise the charge with Memo so a retried step does not charge twice.
func WithSource ¶ added in v0.1.1
WithSource returns a copy of job whose labels carry source and ref (ref may be empty), leaving the caller's Labels map unchanged. It panics on nothing: an invalid source is caught by CheckJobLabels when the job is enqueued, and ValidateSource tells a caller earlier.
job = workflow.WithSource(job, "webhook", deliveryID)
type JournalOption ¶
type JournalOption func(*journalPlugin)
JournalOption configures the Journal plugin.
func JournalKeepFailed ¶
func JournalKeepFailed() JournalOption
JournalKeepFailed keeps the journal of a run that failed, so a later RunRetrier retry of that run resumes from its memoised work instead of redoing it. The app then forgets those runs itself (JournalStore.Forget, or the store's own retention). By default every finished run is forgotten, failed ones included, so an in-process journal never grows without bound.
type JournalStore ¶
type JournalStore interface {
// Load returns the value saved under key for run, and false when
// there is none.
Load(ctx context.Context, run RunID, key string) (val []byte, ok bool, err error)
// Save stores val under key for run.
Save(ctx context.Context, run RunID, key string, val []byte) error
// Forget removes every entry of run. Forgetting a run with no
// entries is not an error.
Forget(ctx context.Context, run RunID) error
}
JournalStore keeps the results of memoised work inside a run (XM-02), so a step that is retried, or a run retried with RunRetrier (which keeps its RunRef), skips the work that already succeeded and gets its stored result back instead of doing it twice.
Entries are keyed by run and by a key Memo builds from the step name and the name the caller gave. A store must be safe for concurrent use, must return a copy (or a value it never changes again) from Load, and Save of an existing key replaces it. MemoryJournal is the in-process default; a durable store (SQL, Redis) makes the memo survive a crash of the process, which an in-process journal cannot.
type LifecycleEvent ¶
type LifecycleEvent struct {
Kind LifecycleKind
Ref RunRef
Workflow string
Queue string
Tenant string
// Step and Attempt name the attempt for Started, Retrying, Snoozed
// and Progress. For Finished, Step is the step whose failure ended the
// run (RunEvent.Step) and Attempt is zero; for Inserted both are
// empty.
Step string
Attempt int
// Status is the run's status after the event.
Status Status
// Err is the attempt's error (Retrying, Snoozed) or the run's
// (Finished, failed); nil otherwise.
Err error
// Progress is set for LifecycleProgress.
Progress *Progress
// Node is the node the event happened on (CapNodeIdentity); empty
// otherwise.
Node string
// Labels are the run's labels. Observers must not modify them.
Labels map[string]string
// At is when it happened, on the driver's clock.
At time.Time
}
LifecycleEvent is one moment in a run's life, for a live view (a dashboard, a progress bar, an audit trail) that wants more than the end RunNotifier reports.
type LifecycleFunc ¶
type LifecycleFunc func(ctx context.Context, ev LifecycleEvent)
LifecycleFunc observes lifecycle events.
type LifecycleKind ¶
type LifecycleKind string
LifecycleKind is what happened to a run in a LifecycleEvent.
Code that switches on it must keep a default branch: a later version may add kinds.
const ( // LifecycleInserted: Enqueue accepted a new run (not a duplicate // returned for an idempotency key). LifecycleInserted LifecycleKind = "inserted" // LifecycleStarted: a step attempt started (Step and Attempt set). LifecycleStarted LifecycleKind = "started" // LifecycleRetrying: a step attempt failed and the step will run // again after its backoff (Err set, Attempt is the failed one). LifecycleRetrying LifecycleKind = "retrying" // LifecycleSnoozed: a step snoozed (Snooze) and runs again later. LifecycleSnoozed LifecycleKind = "snoozed" // LifecycleProgress: a step reported progress (Progress set). LifecycleProgress LifecycleKind = "progress" // LifecycleFinished: the run reached a terminal status (Status set; // Err for a failure). The same moment RunNotifier reports. LifecycleFinished LifecycleKind = "finished" )
The lifecycle kinds.
type LifecycleNotifier ¶
type LifecycleNotifier interface {
OnLifecycle(fn LifecycleFunc) error
}
LifecycleNotifier is an optional interface a client implements to stream its runs' lifecycle. The contract:
- Register before Start: OnLifecycle after Start returns ErrAlreadyStarted. Each observer sees every event, in registration order.
- Delivery is best-effort and local: an observer sees the events that happen in this client's process (an Enqueue it accepted, an attempt its workers ran), so on a cluster each node streams its own share. It is a live feed, not a log; RunNotifier with CapRunFinished is the guaranteed channel for run ends.
- An observer runs on a driver goroutine, must return quickly, and a panic in it is recovered and never changes a run.
- For one run, the events of one process arrive in the order they happened.
type LocalCron ¶
type LocalCron struct {
// contains filtered or unexported fields
}
LocalCron runs functions on a cron schedule inside this process, on every node that starts one: cache warmers, local temp cleanup, a per-node metrics flush. It is the opposite of Workflow.Cron and Schedule on a shared driver, which fire once across the fleet and may run the job on any worker; a LocalCron tick runs here, in-process, with no queue, no retries and no persistence.
Each entry runs on its own goroutine, one tick at a time: ticks that fall due while the previous run is still going are skipped, and ticks missed while the process was down are never caught up. An error or a panic goes to the error handler (WithLocalCronErrors) and the entry keeps its schedule.
Time comes from the injected clock (WithLocalCronClock, clock.Real by default), so a test drives it with clock.Fake.
func NewLocalCron ¶
func NewLocalCron(opts ...LocalCronOption) *LocalCron
NewLocalCron returns a LocalCron with no entries.
func (*LocalCron) Add ¶
Add registers fn to run at expr in loc. loc is required (time.UTC for UTC) and time.Local is refused, for the same reason as on Schedule. Names are unique. Every validation error wraps ErrInvalid; Add after Start returns ErrLocalCronStarted.
type LocalCronFunc ¶
LocalCronFunc is the code a LocalCron entry runs. at is the tick it runs for (the logical time), not the wall time it started.
type LocalCronOption ¶
type LocalCronOption func(*LocalCron)
LocalCronOption configures a LocalCron.
func WithLocalCronClock ¶
func WithLocalCronClock(c clock.Clock) LocalCronOption
WithLocalCronClock sets the clock; nil keeps clock.Real.
func WithLocalCronErrors ¶
func WithLocalCronErrors(f func(name string, at time.Time, err error)) LocalCronOption
WithLocalCronErrors receives every error and recovered panic of an entry's run, with the entry's name and tick. Nil (the default) drops them; the handler runs on the entry's goroutine and must not block.
func WithLocalCronParser ¶
func WithLocalCronParser(p cron.Parser) LocalCronOption
WithLocalCronParser sets the cron dialect (cron.ParseSeconds for a seconds field); nil keeps cron.Parse.
type MapOption ¶
type MapOption func(*mapConfig)
MapOption configures MapEach.
func MapConcurrency ¶
MapConcurrency runs at most n items at once. The default is 1 (one after another); n below 1 counts as 1.
type MemoryBuckets ¶
type MemoryBuckets struct {
// contains filtered or unexported fields
}
MemoryBuckets is an in-process BucketStore. The zero value is ready to use.
type MemoryJournal ¶
type MemoryJournal struct {
// contains filtered or unexported fields
}
MemoryJournal is an in-process JournalStore: entries live in this process only and are lost when it exits, so the memo holds across step retries and RunRetrier retries in one process, not across a crash. The zero value is ready to use.
func NewMemoryJournal ¶
func NewMemoryJournal() *MemoryJournal
NewMemoryJournal returns an empty MemoryJournal.
func (*MemoryJournal) Forget ¶
func (m *MemoryJournal) Forget(_ context.Context, run RunID) error
Forget removes every entry of run.
func (*MemoryJournal) Runs ¶
func (m *MemoryJournal) Runs() []RunID
Runs returns the runs that hold at least one entry, sorted, for tests and for checking that finished runs were forgotten.
type Middleware ¶
Middleware wraps every step execution, the same way on every driver: logging, tracing, recovery, timeouts and codecs are middleware, never driver wrappers that would hide what the driver can do.
func WithJournal ¶
func WithJournal(store JournalStore) Middleware
WithJournal returns step middleware that puts store on every step's ctx, for Memo and the helpers built on it. A ctx without StepInfo (a handler called outside a driver) passes through unchanged, so Memo there returns ErrNoJournal. The Journal plugin bundles this with the cleanup of finished runs; use WithJournal alone only when something else forgets runs.
type Misfire ¶
type Misfire struct {
// Policy is what happens to missed ticks. Empty is the driver default.
Policy MisfirePolicy
// Within drops a missed tick reached more than this long after its
// time, for MisfireFireOnce and MisfireFireAll (a starting deadline).
// Zero means no limit. MisfireSkip uses MisfireGrace and refuses it.
Within time.Duration
// Max caps how many missed ticks MisfireFireAll fires per catch-up:
// the newest Max, oldest first. Required (at least 1) for
// MisfireFireAll and refused for the other policies, so a backfill is
// never unbounded by accident.
Max int
}
Misfire is a cron misfire policy. The zero value means the driver's default.
func (Misfire) Due ¶
Due returns the ticks of src in (last, now] that m fires, oldest first, in loc: the one rule every driver applies so the policies mean the same everywhere. last is when the scheduler last caught up (or started); now is the instant it woke.
- MisfireSkip: the newest tick, only when now is within MisfireGrace of it; every older tick is dropped.
- MisfireFireOnce: the newest tick, unless Within is set and it is more than Within old.
- MisfireFireAll: every tick no more than Within old (any age when Within is zero), the newest Max of them.
The zero policy is the driver's own default, so Due returns nil for it and the driver applies its documented behaviour instead.
func (Misfire) Plan ¶
func (m Misfire) Plan(src TickSource, last, now time.Time, loc *time.Location) (fire []time.Time, dropped int)
Plan is Due plus how many ticks the policy dropped, so a driver can report TickMisfired. The fire half is exactly Due's answer. The count covers ticks in (last, now] the policy did not fire, scanning in chunks so a long outage of a frequent cron costs bounded memory; it is capped at maxCountedTicks. The zero policy returns nil and 0, as Due does.
type MisfirePolicy ¶
type MisfirePolicy string
MisfirePolicy is what a scheduler does with cron ticks it missed: ticks that fell due while nothing was running, or that it reached late.
const ( // MisfireSkip drops every missed tick: a tick fires only when the // scheduler reaches it within MisfireGrace of its time (a 09:00 digest // must not go out at 14:00). MisfireSkip MisfirePolicy = "skip" // MisfireFireOnce fires the most recent missed tick once and drops the // older ones. MisfireFireOnce MisfirePolicy = "fire_once" // MisfireFireAll fires every missed tick, oldest first, at most // Misfire.Max of them per catch-up (a daily report per missed day). MisfireFireAll MisfirePolicy = "fire_all" )
The misfire policies. A driver declares the ones it honours through CapMisfireSkip, CapMisfireFireOnce and CapMisfireFireAll and refuses the rest; the empty policy is the driver's own default and needs nothing.
type NextRun ¶
type NextRun struct {
// Local is the fire time in the schedule's zone.
Local time.Time
// UTC is the same instant in UTC.
UTC time.Time
}
NextRun is one upcoming fire time shown two ways: on the schedule's wall clock and in UTC, so an operator in another zone reads both the time the schedule means and the instant it happens.
func NextRuns ¶
func NextRuns(q NextRunsQuery) []NextRun
NextRuns lists the next q.Count fire times of q.Schedule that pass its Windows and Active period, each in the schedule's zone and in UTC (a dashboard's "next run" column). It returns fewer when the schedule ends or the scan limit is reached. It is a pure function: the reference time comes from q.After.
type NextRunsQuery ¶
type NextRunsQuery struct {
// Schedule is the parsed expression (cron.Parse, or the driver's
// parser).
Schedule cron.Schedule
// Location is the schedule's zone; nil means UTC.
Location *time.Location
// After is the instant to list from (exclusive): the caller's clock,
// never read here.
After time.Time
// Count is how many fire times to list; at most MaxNextRuns.
Count int
// Windows and Active drop the fire times the schedule would skip, so
// the list shows runs that will really start.
Windows Windows
Active Active
}
NextRunsQuery describes the schedule whose upcoming fire times NextRuns lists.
type NodeInfo ¶
type NodeInfo struct {
// ID is the node's identity (StepInfo.Node).
ID string
// Mode is what the node does.
Mode NodeMode
// Queues are the queues its workers serve; empty means every queue
// (or none for NodeScheduler and NodeEnqueueOnly).
Queues []string
// Workers is its worker pool size.
Workers int
// Running is how many step attempts it was running at LastSeen.
Running int
// StartedAt is when its client started; LastSeen when it last
// reported in, on the driver's clock.
StartedAt, LastSeen time.Time
}
NodeInfo is one node as NodeLister reports it.
type NodeLister ¶
NodeLister lists the nodes that are online, for an operator's view. It answers for itself: a client may implement it without CapNodeIdentity (an engine that lists its workers but keeps no per-attempt node), and callers check the interface, not the capability. Every client that declares CapNodeIdentity implements it. A node is online while it keeps reporting in; one that stops (a crash) drops out after the driver's expiry, which the driver documents.
type NodeMode ¶
type NodeMode string
NodeMode is what a node does.
const ( // NodeFull runs workers and the scheduler (the default). NodeFull NodeMode = "full" // NodeWorker runs workers only; another node fires schedules. NodeWorker NodeMode = "worker" // NodeScheduler fires schedules only, enqueueing the runs for worker // nodes (a separate scheduler process). NodeScheduler NodeMode = "scheduler" // NodeEnqueueOnly enqueues and answers queries, running nothing // (CapEnqueueOnly). NodeEnqueueOnly NodeMode = "enqueue-only" )
The node modes.
type Op ¶
type Op string
Op names an optional operation that carries no capability bit, so a caller can ask whether a client really does it before offering it (an admin button), instead of finding out from the call's refusal.
type OpProber ¶
OpProber is implemented by a client wrapper whose optional methods exist whether or not what it wraps can do them (Forward, failover): it answers for what is underneath.
type Outcome ¶ added in v0.1.1
type Outcome string
Outcome says whether a run's work succeeded, separately from its Status, which says whether the worker ran it to the end. A sync that skipped because nothing changed, or that finished with 3 bad rows out of 100, is StatusSucceeded either way; its outcome tells them apart. A step sets it with SetOutcome; a run recorder stores the last one set and derives one from the status when none was (DefaultOutcome).
Code that switches on it must keep a default branch: a later version may add outcomes.
const ( // OutcomeSuccess: the work was done in full. OutcomeSuccess Outcome = "success" // OutcomePartial: some of the work was done, some was not (bad rows, // a page that failed); the run still succeeded. OutcomePartial Outcome = "partial" // OutcomeError: the work was not done. OutcomeError Outcome = "error" // OutcomeSkipped: there was nothing to do (no changes since the last // sync, a feature turned off). OutcomeSkipped Outcome = "skipped" )
The outcomes.
func DefaultOutcome ¶ added in v0.1.1
DefaultOutcome is the outcome of a run that ended with status and set none: OutcomeSuccess for StatusSucceeded, OutcomeError for StatusFailed and StatusDiscarded, and "" for every other status (a cancelled or unfinished run has no outcome yet).
type OutcomeFunc ¶ added in v0.1.1
OutcomeFunc adapts a function to OutcomeSetter.
func (OutcomeFunc) SetOutcome ¶ added in v0.1.1
SetOutcome calls f.
type OutcomeSetter ¶ added in v0.1.1
OutcomeSetter stores the outcome a step sets. A run recorder binds one on the step's ctx (WithOutcomeSetter); drivers keep none.
type OverlapPolicy ¶
type OverlapPolicy string
OverlapPolicy says what a schedule does with a tick that falls due while the run its previous fire started is still unfinished.
const ( // OverlapAllow fires every tick whatever the previous run is doing. // It is the default (the zero value means it too). OverlapAllow OverlapPolicy = "allow" // OverlapSkip skips a tick while the schedule's previous run is not // terminal, and reports it as TickSkipped with TickReasonOverlap. The // decision reads the previous run's status from the driver's run // state, not a lock held by one node, so it holds across a fleet and // across a restart on a durable driver. A previous run the driver no // longer knows (deleted, pruned) does not block. It needs // CapOverlapSkip. OverlapSkip OverlapPolicy = "skip" )
The overlap policies.
func (OverlapPolicy) Blocks ¶
func (p OverlapPolicy) Blocks(prev Status) bool
Blocks reports whether p holds back a fire when the schedule's previous run has status prev. prev is empty when the schedule never fired or the driver no longer knows its previous run; that never blocks. Every driver decides overlap with this one function, so the rule cannot drift between backends.
type PauseTarget ¶
type PauseTarget struct {
// Workflow holds the runs of one workflow. Needs CapPauseWorkflow.
Workflow string
// Queue holds the runs on one queue (EffectiveQueue). Needs
// CapPauseQueue.
Queue string
// All holds every run. Needs CapPauseWorkflow.
All bool
}
PauseTarget names what Pauser holds: one workflow, one queue, or everything. Exactly one field is set; build it with PauseWorkflow, PauseQueue or PauseAll.
func PauseAll ¶
func PauseAll() PauseTarget
PauseAll targets every run. Resume(PauseAll()) clears every pause.
func PauseQueue ¶
func PauseQueue(name string) PauseTarget
PauseQueue targets the runs on one queue.
func PauseWorkflow ¶
func PauseWorkflow(name string) PauseTarget
PauseWorkflow targets the runs of one workflow.
func (PauseTarget) Holds ¶
func (t PauseTarget) Holds(wf, q string) bool
Holds reports whether a paused target t holds a run of workflow wf on queue q.
type Pauser ¶
type Pauser interface {
Pause(ctx context.Context, t PauseTarget) error
Resume(ctx context.Context, t PauseTarget) error
Paused(ctx context.Context) ([]PauseTarget, error)
}
Pauser holds runs without cancelling them, and resumes them. A client implements it when it declares CapPauseWorkflow or CapPauseQueue, and refuses the targets of the one it lacks (CheckPause).
While a target is paused, no step attempt of a run it holds is claimed: new runs and the next steps of running runs alike stay StatusQueued. Attempts already executing finish. Enqueue is still accepted. Timers (RunAt, sleeps, wait deadlines, retry backoffs) keep running, and what they make ready waits. A run is held while any paused target holds it. On a durable driver a pause is stored, so every node obeys it and it survives a restart.
Pause is idempotent. Resume removes exactly that target, except Resume(PauseAll()), which removes every pause; resuming a target that is not paused is a no-op. Paused lists the targets in force, All first, then workflows, then queues, each by name.
type PayloadLimiter ¶
type PayloadLimiter interface {
PayloadLimit() int
}
PayloadLimiter is an optional interface a driver's client implements to declare its hard payload cap in bytes (an engine's gRPC message limit, a broker's message size), so code above it (the claim check, a batch splitter) can size against the real number. Zero means no cap.
type PayloadTooLargeError ¶
type PayloadTooLargeError struct {
Workflow string
Size int
Limit int
// Source says whose limit it is ("driver", "middleware", or a driver
// name).
Source string
}
PayloadTooLargeError says a payload is over a limit: the driver's hard cap (PayloadLimiter) or one the app set (middleware.MaxPayload). It matches ErrPayloadTooLarge and ErrInvalid, since the same bytes will never fit.
func (*PayloadTooLargeError) Error ¶
func (e *PayloadTooLargeError) Error() string
Error names the sizes, so the message alone says by how much it is over.
func (*PayloadTooLargeError) Is ¶
func (e *PayloadTooLargeError) Is(target error) bool
Is matches ErrPayloadTooLarge and ErrInvalid.
type Plugin ¶
type Plugin interface {
Name() string
}
Plugin is the only interface a plugin must satisfy: a feature (unique jobs, batches, tracing, an audit log) bundled so an app adds it with one Use call instead of wiring each piece by hand.
Every contribution is an optional interface that Use type-asserts for: StepMiddlewareProvider, EnqueueMiddlewareProvider and RunFinishedProvider. A plugin implements only what it uses, so an absent contribution is absent rather than an empty method that looks the same as a forgotten one.
Invariant: Name is non-empty and unique among the plugins of one Use call. It names the plugin in every refusal, so an operator knows which one to remove or reconfigure.
func Journal ¶
func Journal(store JournalStore, opts ...JournalOption) Plugin
Journal returns the plugin that turns on step memoisation: it puts store on every step's ctx (WithJournal) and forgets a run's entries when the run finishes. Pass it to Use before Register and Start; it needs a client with RunNotifier, as every run-finished plugin does.
type Progress ¶
type Progress struct {
// Percent is from 0 to 100.
Percent float64
// Message is a short human-readable note ("exported 4,000 of 10,000
// orders").
Message string
// Data is optional app data, at most MaxProgressData bytes.
Data []byte
// Step is the step that reported it; the driver fills it.
Step string
// At is when it was reported, on the driver's clock; the driver
// fills it.
At time.Time
}
Progress is how far a run has got, as a step reported it from inside (ReportProgress): a percentage, a message for a person, and optional app data (rows exported so far, a cursor). The latest report wins.
type ProgressFunc ¶
ProgressFunc adapts a function to ProgressReporter.
func (ProgressFunc) ReportProgress ¶
func (f ProgressFunc) ReportProgress(ctx context.Context, p Progress) error
ReportProgress calls f.
type ProgressReporter ¶
ProgressReporter stores a running step's progress with its run. A driver that keeps progress sets one on the step's ctx (WithProgressReporter); it fills Step and At, keeps the latest report as RunInfo.Progress, and (with a LifecycleNotifier) emits LifecycleProgress.
type QueueBounds ¶
type QueueBounds struct {
// Min is the fewest workers the queue keeps, however idle it is.
// Zero means one.
Min int
// Max is the most workers it gets, however busy. Zero means the
// whole budget (BalanceConfig.Total).
Max int
}
QueueBounds limits one queue's worker count under a Balancer.
type QueueConfig ¶
type QueueConfig struct {
// Name is the queue; it must pass ValidateQueueName.
Name string
// Workers caps how many step attempts of this queue execute at once
// on this client. Zero means no per-queue cap: only the driver's pool
// size limits it.
Workers int
// Limit caps the in-flight step attempts of this queue across EVERY
// client sharing the store, not only this one: Workers is per client
// (per node), Limit is the queue's whole-cluster ceiling. Zero means no
// cluster cap. It needs CapQueueLimitGlobal; a driver without it
// refuses a non-zero Limit at Start (CheckQueueLimits). An attempt
// holds its slot from claim to outcome, and a slot whose holder died is
// freed when the attempt's lease expires, never leaked. Every client
// that serves the queue should declare the same Limit: a client that
// declares none does not enforce it (it still counts toward it).
Limit int
// Priority orders queues when a worker is free: the ready attempts of
// a higher queue are always taken before a lower one's. Equal priorities fall back to Job.Priority,
// then arrival. A driver that cannot order queues refuses a non-zero
// Priority at Start.
Priority int
// Weight shares a worker between queues of equal Priority: when
// several of them have ready work, a free worker picks one at random
// in proportion to the weights (PickWeighted), so a queue of weight 3
// gets about three times the turns of a queue of weight 1 and none
// starves. Zero means strict order within the priority: Job.Priority,
// then arrival, as before. A queue with weight among queues without
// counts as weight 1. A driver that cannot weight queues refuses a
// non-zero Weight at Start.
Weight int
}
QueueConfig is how one client's workers serve one named queue. Drivers take a list of them through their own option (memory.WithQueues and its siblings); the type is shared so the meaning is the same everywhere.
A client given no QueueConfig serves every queue from one pool, as before queues existed. A client given some serves only those queues: on a shared store the runs of other queues wait for a client that serves them, which is how separate worker processes per queue are deployed.
type QueueLoad ¶
type QueueLoad struct {
// Queue is the queue name.
Queue string
// Depth is how many step attempts are ready and waiting for a worker
// of this client (attempts in a retry backoff or a delayed start are
// not counted: no worker could take them yet).
Depth int
// OldestWait is how long the oldest of those attempts has waited,
// on the driver's clock; zero when Depth is zero.
OldestWait time.Duration
// Workers is the queue's current worker count on this client.
Workers int
// Running is how many of its attempts execute now.
Running int
}
QueueLoad is one queue's load on one client, as a WorkerScaler reports it and a Balancer reads it.
type RateLimit ¶
type RateLimit struct {
// Key derives the limited bucket. Zero means one bucket for the
// workflow.
Key ConcurrencyKey
// Limit is the number of starts allowed per window; at least 1.
Limit int
// Per is the window length; positive.
Per time.Duration
}
RateLimit caps run starts per key value: at most Limit starts per window of length Per. The contract admits both fixed-window and sliding-window engines; for one key, every driver guarantees all three of:
- (a) no span of length Per contains more than 2*Limit starts (a fixed window allows a burst at the end of one window and the start of the next, never more);
- (b) N starts take at least (ceil(N/Limit) - 2) * Per from the first to the last (they fill at least ceil(N/Limit) windows, and a fixed window engine may start runs at the very end of the first and the very start of the last: Limit 2, six starts at 0:59.9, 1:00 and 2:00 span 60.1s, not 120s);
- (c) within one window, at most Limit runs start.
A driver may be stricter (a sliding window keeps any span of length Per to Limit starts) and documents which kind it is. A run held back by the limit stays queued; it is delayed, never dropped.
type Recycle ¶
type Recycle struct {
// MaxJobs is how many step attempts this client may finish, whatever
// their outcome. Attempts already running when it is reached still
// finish, so a client with several workers may finish a few more.
MaxJobs int
// MaxTime is how long the client may run, from Start, on the
// driver's clock.
MaxTime time.Duration
// MaxMemory is the memory reading, in bytes, above which the client
// recycles. Checked every CheckEvery and after every attempt.
MaxMemory uint64
// Memory reads the current memory use in bytes. Nil means
// HeapObjectBytes. A test passes a function returning what it needs;
// an app may read RSS or a cgroup counter instead.
Memory func() uint64
// CheckEvery is how often the driver checks MaxMemory between
// attempts. Zero means DefaultRecycleCheck.
CheckEvery time.Duration
}
Recycle sets when a client stops itself so its process can be replaced: after MaxJobs step attempts, after MaxTime since Start, or once the memory reading passes MaxMemory. A zero field is no limit; the zero Recycle never recycles. When a limit is reached the client drains as Drain does (claims nothing more, lets running attempts finish), then stops; its StopCause is a *RecycleError. A driver takes it through its own option (memory.WithRecycle, sqlstore.WithRecycle).
type RecycleError ¶
type RecycleError struct {
Reason RecycleReason
// Jobs is how many step attempts the client had finished.
Jobs int
// Elapsed is the time from Start to the decision.
Elapsed time.Duration
// Memory is the last memory reading (zero when MaxMemory is unset).
Memory uint64
}
RecycleError says which limit stopped a client. It matches ErrRecycle.
func (*RecycleError) Error ¶
func (e *RecycleError) Error() string
func (*RecycleError) Is ¶
func (e *RecycleError) Is(target error) bool
Is makes errors.Is(err, ErrRecycle) true.
type RecycleReason ¶
type RecycleReason string
RecycleReason names the limit that stopped a client.
const ( RecycleMaxJobs RecycleReason = "max_jobs" RecycleMaxTime RecycleReason = "max_time" RecycleMaxMemory RecycleReason = "max_memory" )
The recycle reasons.
type RecycleWatch ¶
type RecycleWatch struct {
// contains filtered or unexported fields
}
RecycleWatch is the decision every driver shares: it counts finished attempts and checks the limits. It latches: once a limit is reached every later call returns the same error. Safe for concurrent use.
func NewRecycleWatch ¶
func NewRecycleWatch(r Recycle, started time.Time) *RecycleWatch
NewRecycleWatch starts watching r from started (the driver's clock at Start).
func (*RecycleWatch) Check ¶
func (w *RecycleWatch) Check(now time.Time) *RecycleError
Check checks every limit at now without counting an attempt.
func (*RecycleWatch) CheckEvery ¶
func (w *RecycleWatch) CheckEvery() time.Duration
CheckEvery is the memory check interval: zero when MaxMemory is unset (no periodic check needed), else CheckEvery or DefaultRecycleCheck.
func (*RecycleWatch) Deadline ¶
func (w *RecycleWatch) Deadline() (time.Time, bool)
Deadline is when MaxTime ends, and false when it is unset.
func (*RecycleWatch) JobDone ¶
func (w *RecycleWatch) JobDone(now time.Time) *RecycleError
JobDone counts one finished attempt at now and checks every limit.
type Reporter ¶
Reporter receives failures for an error tracker (Sentry and the like). It must be safe for concurrent use and should not block for long: it is called on a driver goroutine. Errors are the reporter's to log.
type ReporterFunc ¶
ReporterFunc adapts a function to Reporter.
type ReportingDrainer ¶
type ReportingDrainer interface {
Drainer
DrainWithReport(ctx context.Context) (DrainReport, error)
}
ReportingDrainer is a Drainer that also says what it cut off. A client implements it when it declares CapDrainReport. DrainWithReport behaves exactly as Drain (same shutdown, same errors) and additionally returns the report. Called again, or after the client stopped, it returns the same report as the first call: the report describes one drain.
type ResultReader ¶
ResultReader returns a succeeded run's result. A client implements it when it declares CapResult.
The result is RunOutput of the run's steps: kept with the run until the run is deleted or pruned by retention. A run that is not StatusSucceeded returns ErrWrongState (ErrNotFound when unknown), so a caller can tell "not done" from "done with an empty result".
type RetryPolicy ¶
type RetryPolicy struct {
// Max is the number of retries after the first attempt, so a step runs
// at most Max+1 times. The value that applies is the one of the policy
// EffectiveRetriesFor resolves: job over step over workflow over
// DefaultRetryMax.
Max int
// Backoff is the wait between attempts. Zero means retry at once.
Backoff Backoff
// NoRetryOn marks errors that must not be retried. Returning
// Permanent(err) from the handler does the same without a policy.
// It is a function, so it is never encoded: JSON skips it.
NoRetryOn func(error) bool `json:"-"`
// DelayFor sets the wait before a retry from the error itself, such
// as a 429's Retry-After: it returns the delay and true, or false to
// fall back to Backoff. retry is 1 for the first retry. The delay is
// used as returned (no jitter; a negative one is zero). Nil means
// Backoff alone. RetryAfterDelay is the stock implementation, reading
// errors wrapped by RetryAfter. Needs CapRetryDelay.
// It is a function, so it is never encoded: JSON skips it.
DelayFor func(err error, retry int) (time.Duration, bool) `json:"-"`
// Until stops retrying once a retry would start more than Until after
// the step's first attempt started. It wins
// over Max: a step with Max 100 and Until one hour stops at the hour.
// Zero means no deadline. Needs CapRetryUntil.
Until time.Duration
// MaxErrors fails the step for good on its MaxErrors-th error, even
// when Max allows more attempts. A release
// (an error marked by RetryAfter, see IsRelease) counts against Max
// only; any other error counts against both. Zero means no cap. Needs
// CapRetryMaxErrors.
MaxErrors int
// DiscardOnExhaust ends the run StatusDiscarded (Err matching
// ErrDiscarded and ErrCancelled, no OnFailure, no chain) instead of
// StatusFailed when this policy stops retrying a failed attempt:
// attempts ran out (Max), MaxErrors or Until stopped it, NoRetryOn
// excluded the error, or the handler returned Permanent. Use it for
// work whose giving up is routine (a notification nobody needs after
// five tries) so failed stays a signal that needs a person. The default
// (false) keeps the run failed. An explicit Discard and ErrCancelled are
// not affected. Needs CapDiscardExhausted, which also needs
// CapRunStates to report the status.
DiscardOnExhaust bool
}
RetryPolicy says how often and how soon a failed step is tried again. Its zero value means no retries (Max is DefaultRetryMax).
A policy reaches a step from four places; the first that is set wins and the others are ignored whole (policies are never merged field by field):
- Job.Retry, chosen per dispatch, for every step of that run;
- Step.Retry (RetriesNone() means never retry);
- Workflow.Retry, the default for steps whose Retry is nil;
- the zero policy, DefaultRetryMax retries.
EffectiveRetriesFor is the one function that applies this order, and every driver calls it, so no two drivers can resolve a step's policy differently.
func EffectiveRetries ¶
func EffectiveRetries(step Step, wf Workflow) RetryPolicy
EffectiveRetries resolves the policy a driver must apply to step inside wf when no per-dispatch override exists: the step's own Retry when set, else the workflow default. It is EffectiveRetriesFor with an empty Job, for callers that decide at declaration time (a driver that registers retries with an engine); a driver that sees runs calls EffectiveRetriesFor.
func EffectiveRetriesFor ¶
func EffectiveRetriesFor(step Step, wf Workflow, job Job) RetryPolicy
EffectiveRetriesFor is the one place the retry precedence lives, job over step over workflow over default: job.Retry when set (a per-dispatch override wins over every declared policy), else step.Retry, else wf.Retry, whose zero value is the default of DefaultRetryMax retries. The winning policy is returned whole; nothing is merged. Every driver that retries (memory, sqlstore, redis, RunInline) calls it with the run's own Job, so a driver that ignored Job.Retry on a step would be a bug the conformance suite catches.
func RetriesNone ¶
func RetriesNone() *RetryPolicy
RetriesNone returns a policy that never retries, for a step that must not inherit the workflow default. It returns a fresh value each call so no caller can change it for another.
func (RetryPolicy) ExhaustedError ¶
func (p RetryPolicy) ExhaustedError(attempts int, err error) error
ExhaustedError is what a driver ends a run with when policy stopped retrying a step's failed attempt number attempts with err and policy.DiscardOnExhaust is set: the returned error is a Discard of err (IsDiscarded), so EndStatus maps it to StatusDiscarded and DiscardedError builds the run's Err from it. With the option off, or when err already is a discard or a cancellation (those end the run their own way), it returns nil and the driver fails the run as before. Every driver calls it at the point Next says stop, so none decides differently.
func (RetryPolicy) Next ¶
func (p RetryPolicy) Next(err error, a Attempts, now time.Time, rnd func() float64) (delay time.Duration, retry bool)
Next is the one retry decision every driver makes when a step attempt fails with err: whether to try again and, if so, after how long. It applies, in order:
- Retryable(err, a.N): nil, Permanent, Discard, ErrCancelled and NoRetryOn stop it, and so does a.N beyond Max;
- MaxErrors: a.Errors reaching it stops it;
- the delay, NextDelay(err, a.N, rnd);
- Until: a retry starting after a.First + Until (now + delay) stops it.
now is the driver's clock. It is a pure function of its arguments (with a pinned rnd), so a test can pin every branch.
func (RetryPolicy) NextDelay ¶
NextDelay is the wait before retry number retry (1 for the first) after err: DelayFor's answer when it gives one, else the Backoff delay with its jitter drawn from rnd, a source of uniform values in [0, 1) (nil means math/rand/v2). Every driver that retries calls it, so no two drivers can disagree on the rule.
func (RetryPolicy) Retryable ¶
func (p RetryPolicy) Retryable(err error, attempts int) bool
Retryable reports whether err, after the given number of attempts (1 for the first), may be tried again: not permanent, not discarded, not excluded by NoRetryOn, and within Max. It ignores Until and MaxErrors, which need more than an attempt count; drivers call Next.
type RunDeleter ¶
RunDeleter forgets a terminal run: afterwards Status returns ErrNotFound and Runs no longer lists it. A client implements it when it declares CapDeleteRun. A run that is not terminal returns ErrWrongState; an unknown ref returns ErrNotFound. Forgetting many runs is Runs plus Delete, or a driver's retention.
type RunError ¶
RunError is the error a driver records for a run that failed: it carries the run, the workflow, the step and the attempt as fields, so code holding the error (a RunEvent.Err, an error tracker's hook) finds which run to look up with RunErrorOf or errors.As. Its Error text is the step's own message unchanged, so RunInfo.Error, logs and dashboards keep showing what the app said; Describe gives the message with the context in front. It wraps the step's own error, so errors.Is and errors.As keep working through it.
func RunErrorOf ¶
RunErrorOf returns the RunError err carries, if any.
func (*RunError) Describe ¶
Describe reads `workflow "send-email" run r-12 step "deliver" attempt 3: <err>`, for a log line that has nothing else to say which run it was.
type RunEvent ¶
type RunEvent struct {
// Ref is the run. ScheduledID is set too when the run was enqueued
// with RunAt, so a caller that stored the RunRef Enqueue returned
// (which held only the ScheduledID) can match the event.
Ref RunRef
// Workflow is the workflow name.
Workflow string
// Status is terminal: StatusSucceeded, StatusFailed or
// StatusCancelled, or StatusDiscarded on a driver with CapRunStates.
Status Status
// Err is why the run failed; nil when it succeeded. In-process
// drivers pass the original error; an engine rebuilds it from the
// message it stored, so errors.Is against an app's own sentinel is
// reliable only on in-process drivers. A cancelled run's Err matches
// ErrCancelled on every driver.
Err error
// Labels are the run's Job.Labels. Observers must not modify them.
Labels map[string]string
// FinishedAt is when the run reached its terminal status, from the
// driver's clock: the engine's timestamp for an engine driver when the
// engine records one, else the driver's own clock at the moment it
// learned of the end (a delayed run cancelled before it fired).
FinishedAt time.Time
// Step is the step whose failure ended the run (RunInfo.FailedStep);
// empty for other ends and for drivers that do not know it.
Step string
// Tenant is the run's Job.Tenant (CapTenant).
Tenant string
}
RunEvent reports that a run reached a terminal status. It is what lets app-side code react to a run's end without polling Status: an enqueue middleware releasing a uniqueness key, a batch counting finished members, an audit log.
type RunFailedError ¶
type RunFailedError struct {
Ref RunRef
Status Status
// Err is RunInfo.Error when the client could say why; empty otherwise.
Err string
}
RunFailedError is what AwaitResult returns for a run that ended without succeeding. It matches ErrCancelled for a cancelled or discarded run.
func (*RunFailedError) Error ¶
func (e *RunFailedError) Error() string
Error names the run and why it ended.
func (*RunFailedError) Is ¶
func (e *RunFailedError) Is(target error) bool
Is makes a cancelled or discarded run match ErrCancelled.
type RunFinishedFunc ¶
RunFinishedFunc observes run ends. It must be idempotent, because delivery is at least once.
type RunFinishedProvider ¶
type RunFinishedProvider interface {
RunFinishedObservers() []RunFinishedFunc
}
RunFinishedProvider contributes run-finished observers. Use registers them on the client's RunNotifier, so it must run before Start, and it refuses a client without RunNotifier rather than dropping them. The delivery contract, best-effort included, is RunNotifier's.
type RunForcer ¶
type RunForcer interface {
ForceComplete(ctx context.Context, ref RunRef) error
ForceFail(ctx context.Context, ref RunRef, reason string) error
}
RunForcer ends a run from outside, for an operator handling a work item a step cannot finish on its own. A client implements it when it declares CapForceRun.
ForceComplete ends a run as StatusSucceeded without running its remaining steps. It accepts a run that is not terminal, or one that is StatusFailed (accepting a failure as done). Attempts in progress have their context cancelled with cause ErrForced and their results are ignored. The run is then treated exactly as a success: its idempotency key is released, its Job.Chain continues, and run-finished observers see StatusSucceeded (again, for a run that had failed: they already tolerate repeats). Other terminal statuses return ErrWrongState.
ForceFail ends a run that is not terminal as StatusFailed, with an error matching ErrForced that carries reason. No retry is made and OnFailure does not run (the operator has decided; it is not the workflow's failure path); RunRetrier can still re-run it. A terminal run returns ErrWrongState.
Both return ErrNotFound for an unknown ref. Neither moves a run to another worker or queue: re-running a failed run elsewhere is RunRetrier (any worker may pick it up).
type RunID ¶
type RunID string
RunID identifies one run. Drivers treat it as opaque: the memory driver uses a counter, an engine uses its own ID format.
const InlineRunID RunID = "inline"
InlineRunID is the RunID a step sees in its StepInfo under RunInline.
type RunInfo ¶
type RunInfo struct {
// Ref is the run.
Ref RunRef
// Workflow is the workflow name.
Workflow string
// Queue is the run's queue (EffectiveQueue).
Queue string
// Status is where the run is now.
Status Status
// Error is why the run failed, was discarded or was cancelled; while
// it is StatusRetrying (CapRunStates), the error of the failed attempt
// that waits on its backoff; empty otherwise. It is a string because
// an engine stores only the message.
Error string
// FailedStep is the step whose failure ended the run; empty otherwise.
FailedStep string
// Labels are the run's Job.Labels. Callers must not modify them.
Labels map[string]string
// Payload is the job's payload as the driver stored it (after any
// codec or claim check ran).
Payload []byte
// Priority is Job.Priority.
Priority int
// EnqueuedAt is when Enqueue accepted the run, on the driver's clock.
EnqueuedAt time.Time
// StartedAt is when the first step attempt started; zero before.
StartedAt time.Time
// FinishedAt is when the run became terminal; zero before.
FinishedAt time.Time
// Attempts counts the step attempts made across the run.
Attempts int
// NextAttempt is the attempt number a step waiting on its retry
// backoff will run as (2 for the first retry), and NextAttemptAt
// when it becomes ready; with several steps in backoff, the one
// ready first. Both are zero unless the run is StatusRetrying, and
// are filled only by drivers with CapRunStates.
NextAttempt int
NextAttemptAt time.Time
// Errors is the run's error history: one entry per failed step
// attempt, oldest first, bounded by the driver's error-history
// option. Filled only by drivers with CapErrorHistory.
Errors []AttemptError
// Tenant is Job.Tenant (CapTenant).
Tenant string
// RunAt is the time a delayed run is (or was) due: Job.RunAt, or the
// time Reschedule moved it to. Zero for a run enqueued to start now.
// Filled by drivers with CapDueQuery.
RunAt time.Time
// Output is the run's result (RunOutput) once it succeeded; nil
// before and for other ends. Filled by drivers with CapResult.
Output []byte
// Progress is the latest progress a step reported (ReportProgress);
// nil when none was reported or the driver keeps none.
Progress *Progress
// AttemptedBy lists the nodes that ran the run's step attempts, in
// the order each first ran one, without repeats. Filled by drivers
// with CapNodeIdentity.
AttemptedBy []string
}
RunInfo is one run as a RunQuerier returns it.
type RunNotifier ¶
type RunNotifier interface {
// OnRunFinished registers fn. It returns ErrAlreadyStarted after
// Start.
OnRunFinished(fn RunFinishedFunc) error
}
RunNotifier is an optional interface a driver's client implements to report run ends. Callers discover it with a type assertion on the Client, so a driver that cannot report run ends is never forced to pretend it can, and app-side code falls back (for example to a TTL) when the assertion fails.
The delivery contract every implementation keeps:
- Register before Start: OnRunFinished after Start returns ErrAlreadyStarted. Several observers may be registered; each sees every event, in registration order.
- An observer runs on a driver goroutine, never on the goroutine that enqueued or cancelled the run, and may run concurrently for different runs. It should return quickly; long work belongs in a new Enqueue.
- It is called after Status reports the terminal status, so an observer that calls Status sees the same status as ev.Status.
- Its ctx is cancelled when the client shuts down (the Start ctx).
- A panic in an observer is recovered by the driver and never changes the run's status.
- Delivery is at least once: an observer may see the same run's event more than once (an engine retrying a delivery) and must tolerate it.
- With CapRunFinished, every run that reaches a terminal status is delivered, including cancelled runs and runs that never started; a durable driver also delivers, after a restart, the ends that happened while no client was running. Without CapRunFinished, delivery is best-effort: an event may be lost (for example on a crash), so a consumer must keep a fallback.
type RunPage ¶
RunPage is one page of a run listing, newest enqueue first (ties broken by RunID, descending). Next is empty on the last page.
type RunQuerier ¶
RunQuerier lists runs, failed ones included, for an operator's view. A client implements it when it declares CapRunQuery. Errors: an invalid query wraps ErrInvalid; a cursor the driver did not issue wraps ErrInvalid too.
type RunQuery ¶
type RunQuery struct {
// Ref picks one run: every field of Ref that is set must equal the
// run's, so a RunRef that Enqueue returned for a delayed run (only its
// ScheduledID) still finds the run after it started. The zero RunRef
// matches any run. With Ref set, Runs returns zero or
// one run, which is how an admin surface shows a single run.
Ref RunRef
// Workflow is the workflow name; empty matches any.
Workflow string
// Queue is the run's queue (EffectiveQueue); empty matches any.
Queue string
// Statuses matches runs in any of these statuses; empty matches any
// (CancelAll gives it its own default).
Statuses []Status
// Labels matches runs whose Job.Labels hold every pair exactly.
Labels map[string]string
// LabelPrefixes matches runs whose Job.Labels hold every key with a
// value starting with the given prefix (an empty prefix matches any
// value under that key). It needs CapLabelPrefix; a driver without it
// refuses the query (CheckRunQuery) rather than ignoring the filter.
LabelPrefixes map[string]string
// From and To bound the enqueue time to [From, To); a zero bound is
// open.
From, To time.Time
// Tenant matches runs whose Job.Tenant equals it; empty matches any.
// It needs CapTenant; a driver without it refuses the query
// (CheckRunQuery).
Tenant string
// DueFrom and DueTo bound the time a delayed run is due (RunInfo.RunAt)
// to [DueFrom, DueTo), for "what fires in the next hour". With either
// set, only runs enqueued with a RunAt match. A zero bound is open.
// They need CapDueQuery (CheckRunQuery).
DueFrom, DueTo time.Time
// Limit is the page size: zero means DefaultRunsLimit, more than
// MaxRunsLimit is invalid. CancelAll refuses it.
Limit int
// Cursor continues a listing from RunPage.Next; empty starts at the
// newest run. It is opaque and driver-specific. CancelAll refuses it.
Cursor string
}
RunQuery filters runs for RunQuerier.Runs and BulkCanceller.CancelAll. Every field that is set must match; the zero value matches every run.
type RunRef ¶
type RunRef struct {
RunID RunID
ScheduledID ScheduledID
}
RunRef points at a run. ScheduledID is set while a delayed run has not started; RunID is set once it has. A driver that learns the RunID of a delayed run resolves it itself, so callers keep passing the RunRef they were given.
func EnqueueClosure ¶
EnqueueClosure runs c on client when it implements ClosureEnqueuer and declares CapClosure, and otherwise returns an UnsupportedError naming CapClosure, never silently dropping the work.
func Spawn ¶
func Spawn(ctx context.Context, client Client, name string, jobs []Job, opts ...SpawnOption) ([]RunRef, error)
Spawn enqueues jobs from inside a step as runs of their own and returns their RunRefs in order, without waiting for them: multiple instances without synchronization (WC-12). Each child is its own run, with its own retries and its own worker, and carries LabelParentRun and LabelParentStep.
A retried step does not spawn twice: each job's RunRef is memoised under name and the job's index (Spawn needs the Journal plugin, as Memo does), and a job without an IdempotencyKey gets one derived from the run, the step, name and the index, which covers a crash between the enqueue and the save. That key needs CapIdempotencyKey on client; a client without it is refused with an UnsupportedError unless SpawnAllowDuplicates is given. A job that already has a key keeps it.
client is the client the app enqueues with (a step has no client of its own); pass the same one the step's workflow is registered on, or another one to spawn onto another backend.
type RunRescheduler ¶
type RunRescheduler interface {
Reschedule(ctx context.Context, ref RunRef, at time.Time) (RunRef, error)
}
RunRescheduler moves a delayed run that has not started to a new time.
Reschedule needs a run in StatusScheduled (a Job.RunAt enqueue not yet due): it runs at `at` instead, keeps its RunRef, payload, labels and idempotency key, and RunInfo.RunAt reads `at`. An `at` in the past (or zero) means now. Any other status returns ErrWrongState, an unknown ref ErrNotFound. A client without it returns an UnsupportedError from the Forward wrapper; the fallback is Cancel plus Enqueue.
type RunRetrier ¶
RunRetrier re-runs a failed run from the step that failed, without re-running the steps that already succeeded. A client implements it when it declares CapRetryRun.
Steps that succeeded keep their outputs and do not run again; the step that failed, and every step the failure cancelled, start again with fresh attempt counts. The run keeps its RunRef, goes back to StatusQueued and, when it ends again, is reported again to run-finished observers (which already tolerate repeats). Its idempotency key is taken again; when another unfinished run holds it, Retry returns ErrDuplicate. A run that is not StatusFailed returns ErrWrongState; an unknown ref returns ErrNotFound.
type Schedule ¶
type Schedule struct {
// Name is a stable, human-readable name an engine may show. A
// schedule with a Name is upserted by (Workflow, Name) on every
// driver: calling Schedule again with the same name returns the same
// ScheduleID and replaces that schedule instead of adding a second, so
// an app can declare its schedules at every boot. Without a Name every
// call adds a schedule.
Name string
// Workflow is the registered workflow to run.
Workflow string
// Cron is a 5-field cron expression.
Cron string
// Location is the time zone Cron is evaluated in. It is required, so
// no schedule is ever evaluated in an implicit zone; use time.UTC for
// UTC. time.Local is refused for the same reason: it differs from
// machine to machine. A driver whose engine only runs UTC crons returns an
// UnsupportedError for any other zone rather than shifting it.
Location *time.Location
// Input is the payload of each fired run.
Input []byte
// Labels are copied onto each fired run's Job.Labels. Keys under a
// reserved prefix (ReservedLabelPrefixes) are refused.
Labels map[string]string
// Misfire says what happens to ticks the driver reached late or not
// at all. The zero value is the driver's default.
Misfire Misfire
// Windows limits ticks to time-of-day windows on Location's wall
// clock (only between, or never between, two times). A tick outside them is
// skipped, which tick observers see as TickSkipped with Reason
// TickReasonWindow. Zero means no limit. Needs CapScheduleWindow.
Windows Windows
// Overlap is the schedule's overlap policy: OverlapSkip skips a tick
// (and refuses a manual trigger with ErrOverlap) while the run of
// its previous fire is unfinished. The zero value is OverlapAllow.
Overlap OverlapPolicy
// Seconds says Cron has six fields, seconds first (cron.ParseSeconds).
// Needs CapCronSeconds.
Seconds bool
// Active limits the schedule to a period (see Active). Zero means
// always. Needs CapScheduleActive.
Active Active
// KeepRuns keeps only the newest N finished runs this schedule fired;
// older finished runs are forgotten as RunDeleter would. Zero keeps
// them all. Needs CapScheduleKeep.
KeepRuns int
// Tenant is the Job.Tenant of every run the schedule fires, by tick or
// by TriggerSchedule, so a per-tenant schedule's runs are found by
// RunQuery.Tenant like any other run of that tenant. Empty means none.
// It must pass ValidateTenant and is stored with the schedule. Needs
// CapTenant.
Tenant string
}
Schedule is a recurring trigger made at run time (per tenant, per user), as opposed to Workflow.Cron, which is declared with the workflow.
func UnmarshalSchedule ¶
func UnmarshalSchedule(data []byte, loadZone func(name string) (*time.Location, error)) (Schedule, error)
UnmarshalSchedule reads a Schedule's JSON form. loadZone resolves the zone name; nil means cron.LoadZone, which refuses "Local" and unknown zones, so a stored schedule with a mistyped zone fails at load, not as a schedule firing at the wrong hour. A version other than ScheduleJSONVersion and unknown fields are refused. Every error wraps ErrScheduleJSON.
func (Schedule) MarshalJSON ¶
MarshalJSON writes s in its versioned JSON form (ScheduleJSONVersion), for storing schedules as data: a settings table, a config file, an admin form. The zone is written by its IANA name, so a Location without one (time.Local, a time.FixedZone) is refused rather than written as a name no reader can load. It does not validate the rest; Validate does.
func (*Schedule) UnmarshalJSON ¶
UnmarshalJSON reads the form MarshalJSON writes, loading the zone with cron.LoadZone. Use UnmarshalSchedule to load zones another way.
type ScheduleHooks ¶
type ScheduleHooks struct {
// Workflow is the schedule's workflow; required.
Workflow string
// Name is the Schedule.Name (for a declared cron, its expression);
// empty matches every schedule of Workflow.
Name string
// Fired is called when a tick enqueued a run, or failed to (t.Err).
Fired func(ctx context.Context, t Tick)
// Skipped is called when a tick was skipped or misfired.
Skipped func(ctx context.Context, t Tick)
// OnSuccess is called when a run the schedule fired succeeded.
OnSuccess func(ctx context.Context, ev RunEvent)
// OnFailure is called when a run the schedule fired failed.
OnFailure func(ctx context.Context, ev RunEvent)
// After is called when a run the schedule fired ended, in any status.
After func(ctx context.Context, ev RunEvent)
}
ScheduleHooks are before, on-success, on-failure and after callbacks for one schedule, or for every schedule of a workflow, built only from TickNotifier and RunNotifier so they work on any driver that has both. Ping a heartbeat service from After or OnSuccess.
Every hook is optional. They run on the driver's observer goroutines with that contract: at least once, so a hook must be idempotent, and best-effort unless the driver declares CapRunFinished and CapTickEvents.
type ScheduleID ¶
type ScheduleID string
ScheduleID identifies a recurring trigger made by Client.Schedule. The app stores it and passes it back to Unschedule.
func (ScheduleID) IsZero ¶
func (id ScheduleID) IsZero() bool
IsZero reports whether the ID is unset.
type ScheduleInfo ¶
type ScheduleInfo struct {
// ID is the schedule. A declared cron has the driver's stable id for
// that declaration.
ID ScheduleID
// Name is the Schedule.Name; for a declared cron, the expression.
Name string
// Workflow is the workflow the schedule runs.
Workflow string
// Cron is the expression.
Cron string
// Location is the zone it is evaluated in.
Location *time.Location
// Declared is true for a Workflow.Cron entry.
Declared bool
// Paused is true while SchedulePauser holds it.
Paused bool
// Next is the next fire time in Location; zero when the expression
// never fires again. It ignores windows and ScheduleWhen, which are
// decided at tick time.
Next time.Time
// Misfire is the schedule's misfire policy.
Misfire Misfire
// Windows are the schedule's time-of-day windows.
Windows Windows
// Labels are the schedule's labels. Callers must not modify them.
Labels map[string]string
// Overlap is the schedule's overlap policy (for a declared cron,
// Workflow.CronOverlap); empty means OverlapAllow.
Overlap OverlapPolicy
// Last is the schedule's most recent fire, filled by drivers with
// CapScheduleHistory; zero otherwise.
Last ScheduleLast
// Seconds is Schedule.Seconds (Workflow.CronSeconds for a declared
// cron).
Seconds bool
// Active is the schedule's period (Workflow.CronActive for a declared
// cron). Next already respects it.
Active Active
// KeepRuns is Schedule.KeepRuns (Workflow.CronKeepRuns).
KeepRuns int
// Tenant is Schedule.Tenant; empty for a declared cron.
Tenant string
}
ScheduleInfo is one live schedule as ScheduleLister reports it.
type ScheduleLast ¶
type ScheduleLast struct {
// Fire is when the schedule last enqueued a run: the tick's time for
// a cron fire, the driver's clock for a trigger. Zero when it never
// fired.
Fire time.Time
// Run is the run that fire enqueued. Zero when it never fired.
Run RunRef
// Status is that run's status when the schedule was listed; empty
// when it never fired or the driver no longer knows the run (deleted,
// pruned). It is the "last outcome" once the run is terminal.
Status Status
}
ScheduleLast is the bookkeeping of a schedule's most recent fire, cron tick or manual trigger alike. Drivers with CapScheduleHistory store it with the schedule (durably on a durable driver) and report it in ScheduleInfo.
type ScheduleLister ¶
type ScheduleLister interface {
Schedules(ctx context.Context) ([]ScheduleInfo, error)
}
ScheduleLister lists the live schedules with their next fire time, declared and made at run time, sorted by Workflow, then Name, then ID. Unscheduled ones are not listed. A client implements it when it declares CapScheduleList.
type SchedulePauser ¶
type SchedulePauser interface {
PauseSchedule(ctx context.Context, t ScheduleTarget) error
ResumeSchedule(ctx context.Context, t ScheduleTarget) error
}
SchedulePauser stops schedules firing without removing them, and resumes them. A client implements it when it declares CapSchedulePause.
A paused schedule's ticks do not fire and are reported as TickSkipped with TickReasonPaused. A tick that fell due during the pause never fires, not even on resume and whatever the misfire policy (a pause means "not now", not "later"): the schedule resumes at its next tick, so each tick is reported once. The misfire policy still governs ticks missed while nothing was running. Pausing All pauses every schedule, including ones made after the call; ResumeSchedule with All clears every pause. Pause and resume are idempotent; an unknown ID returns ErrNotFound. On a durable driver the pause is stored.
type ScheduleTarget ¶
type ScheduleTarget struct {
ID ScheduleID
All bool
}
ScheduleTarget names a schedule for SchedulePauser, or all of them. Exactly one field is set.
type ScheduleTriggerer ¶
type ScheduleTriggerer interface {
TriggerSchedule(ctx context.Context, id ScheduleID) (RunRef, error)
}
ScheduleTriggerer runs a live schedule now, outside its cron, with the schedule's Input and labels (FiredLabels plus LabelTrigger), as if a tick had fired at the driver's current time. A client implements it when it declares CapScheduleTrigger.
A trigger obeys the schedule's OverlapPolicy: with OverlapSkip and the previous run unfinished it returns ErrOverlap and enqueues nothing. It does not consult Windows or ScheduleWhen (an operator asking for a run now has already decided), and it fires a paused schedule too (a pause stops the cron, not the operator). The triggered run becomes the schedule's last run (ScheduleInfo.Last: Fire and Run), so a later tick's overlap check sees it, and drivers with CapTickEvents report it as a TickFired with Tick.Manual set. The schedule's next cron fire is unchanged. An unknown or unscheduled ID returns ErrNotFound.
type ScheduledID ¶
type ScheduledID string
ScheduledID identifies a delayed start that has not begun yet. Engines such as Hatchet return it for a RunAt enqueue and hand out the RunID only once the run fires, so the two cannot share one field.
func (ScheduledID) IsZero ¶
func (id ScheduledID) IsZero() bool
IsZero reports whether the ID is unset.
func (ScheduledID) String ¶
func (id ScheduledID) String() string
String returns the ID as a string.
type SecretDetector ¶
type SecretDetector interface {
Detect(job Job) []SecretFinding
}
SecretDetector decides whether a job carries credentials (MS-05). The stock one is NewSecretDetector; an app with its own rules (a vault's token format, an entropy check) supplies its own.
func NewSecretDetector ¶
func NewSecretDetector(r SecretRules) SecretDetector
NewSecretDetector returns the stock detector. It reports a JSON object key or label key that names a credential ("password", "api_key", "client_secret", "token", ...) holding a non-empty string, and any string value matching a known credential format: a PEM private key, an AWS access key ID, a Stripe live key, a GitHub or Slack token, a JWT. A value that is an object (a SecretRef) is never reported. A payload that is not JSON is scanned whole by the value rules only.
type SecretFinding ¶
type SecretFinding struct {
// Where locates it: "payload" plus a JSON path ("payload.user.password"),
// "label <key>", or "chain[2].payload...".
Where string
// Rule names the rule that matched ("key", "pem-private-key",
// "aws-access-key", ...), so a false positive can be allowed by name.
Rule string
}
SecretFinding is one place a detector believes holds a credential.
type SecretRef ¶
type SecretRef struct {
Ref string `json:"secret_ref"`
}
SecretRef names a secret instead of carrying it: put it in a payload where the credential would go, and resolve it in the step with ResolveSecret. Its JSON form is {"secret_ref": "<name>"}, an object, so SecretGuard never reports it.
type SecretResolver ¶
SecretResolver turns a secret's name into its value at the moment a step needs it: a vault, a cloud secret manager, environment variables.
type SecretRules ¶
type SecretRules struct {
// AllowKeys are JSON object keys and label keys never reported by the
// key rule, compared after the same folding the rule applies
// (lower case, "_", "-" and "." removed), such as "pagetoken".
AllowKeys []string
// AllowRules are value rules turned off by name, such as "jwt" for
// an app that passes signed links.
AllowRules []string
}
SecretRules tunes the stock detector.
type Source ¶ added in v0.1.1
type Source string
Source names what started a run. The library derives the sources below from labels it already writes (SourceOf); an app names its own ("api", "webhook", "slack", "admin") with WithSource. A Source passes ValidateSource: lower-case letters, digits and . _ - only.
const ( // SourceSchedule: a schedule or a declared cron fired the run // (LabelSchedule). SourceSchedule Source = "schedule" // SourceScheduleManual: ScheduleTriggerer started a schedule's run by // hand (LabelTrigger = LabelTriggerManual). SourceScheduleManual Source = "schedule-manual" // SourceChild: a step spawned the run (LabelParentRun). SourceChild Source = "child" // SourceBatch: the run is a member of a batch (batch.LabelBatch). SourceBatch Source = "batch" // SourceEvent: an events.Bus subscription started the run // (events.LabelEvent). SourceEvent Source = "event" // SourceExternal: a run recorder found the run on the engine without // having seen it enqueued (an engine UI, its CLI, a crash window). SourceExternal Source = "external" )
The sources SourceOf derives from the library's own labels, and the one a run recorder gives a run it found on the engine but never saw enqueued.
func LibrarySources ¶ added in v0.1.1
func LibrarySources() []Source
LibrarySources returns every source the library itself names, as a fresh slice. App sources are not in it.
func SourceOf ¶ added in v0.1.1
SourceOf returns the source of a run from its labels, and the source reference when there is one. An explicit LabelSource wins. Otherwise it derives one from the labels the library stamps, most specific first: a manual trigger of a schedule, a schedule fire, a spawned child, a batch member, an event subscription. For a derived source the reference is the natural ID (the schedule, the parent run, the batch, the event name). It returns "" when nothing says where the run came from; a caller picks its own default (a run recorder's default source).
type SpawnOption ¶
type SpawnOption func(*spawnConfig)
SpawnOption configures Spawn.
func SpawnAllowDuplicates ¶
func SpawnAllowDuplicates() SpawnOption
SpawnAllowDuplicates lets Spawn run on a client without CapIdempotencyKey. The journal still keeps a retried step from enqueueing a job twice, except when the process dies between an enqueue and the journal's save of its RunRef; with this option that window can start a job twice.
type StackTracer ¶
StackTracer is an error that carries a stack trace, such as the panic error of middleware.Recover. StackOf finds it anywhere in an error's chain.
type Status ¶
type Status string
Status is where a run is.
The run state machine ¶
A run is created StatusScheduled (Job.RunAt in the future, CapDelayedStart) or StatusQueued, and ends in exactly one terminal status. The legal moves, what triggers each and the capabilities that gate it are the Transitions table below; this is the same thing in prose.
scheduled -> queued the start time arrives (CapDelayedStart)
queued -> running a worker claims a ready step
running -> queued the attempt is handed back (graceful stop, dead
worker, lease lost) or the next step is ready
running -> waiting a Wait or Sleep step parks (CapWaitForEvent,
CapDurableSleep) or a step Snoozes (CapSnooze)
waiting -> queued the event arrives or the sleep ends
waiting -> running the resumed step starts at once
running -> retrying an attempt failed and RetryPolicy.Next allows
another; RunInfo.NextAttemptAt says when
(reported only with CapRunStates; other drivers
show queued or running for the same moment)
retrying -> running the backoff elapsed and a worker claimed it
retrying -> queued the backoff elapsed, waiting for a worker
running -> succeeded every step succeeded
running -> failed an attempt failed and RetryPolicy.Next says stop
(Max, MaxErrors, Until, Permanent, NoRetryOn),
and RetryPolicy.DiscardOnExhaust is off
running -> discarded a step returned Discard (CapDiscard +
CapRunStates), or the same stop with
RetryPolicy.DiscardOnExhaust on
(CapDiscardExhausted + CapRunStates)
any non-terminal -> cancelled Cancel or CancelAll
(CapCancel); a Discard on a driver with only
CapDiscard also ends cancelled (see EndStatus)
any non-terminal -> failed the run went past Workflow.RunTimeout
(CapRunTimeout) or ForceFail (CapForceRun)
any non-terminal -> succeeded ForceComplete (CapForceRun)
failed -> queued Retry re-opens the run from its failed step
(CapRetryRun); the only move out of a terminal
status. A discarded or cancelled run is not
re-opened: enqueue a new job instead.
The active statuses (queued, running, waiting, retrying) describe the run as a whole, so with parallel steps a run can move between any two of them as its steps progress (one step retrying while another waits). The Transitions table therefore lists every active-to-active pair that a driver with all of the gating capabilities can report; a driver without CapRunStates never reports retrying or discarded.
Failed versus discarded: failed is a failure an operator must see, so OnFailure runs for it. Discarded is a run dropped on purpose or given up on by policy: OnFailure does not run, the chain does not continue, the idempotency key is released, and the Err matches ErrDiscarded and ErrCancelled. A run exhausts into discarded only when its effective policy (job, step or workflow, see EffectiveRetriesFor) opted in with DiscardOnExhaust; the default is failed.
Code that switches on a Status must keep a default branch: drivers that declare CapRunStates report StatusRetrying and StatusDiscarded, which older drivers never did, and a later version may add more.
const ( StatusScheduled Status = "scheduled" // a delayed start that has not begun StatusQueued Status = "queued" // waiting for a worker StatusRunning Status = "running" // a step is executing StatusWaiting Status = "waiting" // suspended on a wait or sleep step StatusSucceeded Status = "succeeded" StatusFailed Status = "failed" StatusCancelled Status = "cancelled" // StatusRetrying: no step is executing and at least one step attempt // failed and waits on its retry backoff. RunInfo.NextAttemptAt says // when it runs again. Only drivers with CapRunStates report it; the // others report StatusQueued or StatusRunning for the same moment. StatusRetrying Status = "retrying" // StatusDiscarded is terminal: the run was dropped on purpose rather // than failed. Two things give it: a step returned Discard(err), or a // step stopped retrying (attempts ran out, MaxErrors or Until stopped // it, or a Permanent error) under a RetryPolicy with DiscardOnExhaust // set. Without that opt-in a run that runs out of retries ends // StatusFailed, because that is a failure an operator must see. Its // Err matches ErrDiscarded and ErrCancelled. Only drivers with // CapRunStates report it; the others report StatusCancelled (with // CapDiscard) or StatusFailed for the same run. StatusDiscarded Status = "discarded" )
The run statuses.
func Await ¶
Await blocks until the run is terminal and returns its status, or ctx's error (give ctx a deadline for a timeout). It polls Client.Status, so it works on every driver; it holds no worker, but it does hold the caller, so call it from a request handler or a CLI, never from a step (use AwaitChild there).
func CancelStatuses ¶
CancelStatuses is the statuses CancelAll cancels: q.Statuses, or, when empty, the runs not yet started (StatusScheduled and StatusQueued): clearing a backlog must not cut short work that has already begun.
func EndStatus ¶
EndStatus is the terminal status a driver records for a run whose step ended it with err, given the driver's capabilities: a discarded run (IsDiscarded) is StatusDiscarded with CapRunStates, StatusCancelled with CapDiscard alone, and StatusFailed otherwise (Discard falls back to Permanent, the direction that never hides a failure); any other error is StatusFailed. Every driver calls it, so none decides the mapping differently.
func (Status) CanTransitionTo ¶
CanTransitionTo reports whether a run in s may be reported as to by a driver with every capability of some transition between them. It ignores capabilities (a driver without CapRunStates never reports retrying); use TransitionsFrom to see what gates each move.
func (Status) IsTerminal ¶
IsTerminal reports whether the run has finished and will not change again, with one exception: Retry re-opens a failed run (CapRetryRun). An idempotency key is released once its run is terminal.
func (Status) TransitionsFrom ¶
func (s Status) TransitionsFrom() []Transition
TransitionsFrom returns the moves that leave s: none for a terminal status except failed, which Retry re-opens.
type Step ¶
type Step struct {
// Name is unique within the workflow; After refers to it.
Name string
// After names the parent steps that must succeed before this one
// starts. Empty means the step starts with the run.
After []string
// Run is the code the step executes.
Run Handler
// Wait suspends the run until a matching Event or the deadline.
// Needs CapWaitForEvent.
Wait *WaitSpec
// Sleep suspends the run for this long. Needs CapDurableSleep.
Sleep time.Duration
// Retry overrides the workflow default. Nil means the workflow's;
// RetriesNone() means never retry. See EffectiveRetries.
Retry *RetryPolicy
// Timeout caps one attempt. Zero means the workflow default.
Timeout time.Duration
// HeartbeatTimeout fails an attempt that goes this long without
// calling Heartbeat (counted from the attempt's start, then from each
// Heartbeat), with an error matching ErrHeartbeatTimeout and
// ErrTimeout; the step's retry policy then applies. It catches a
// handler that hangs well before a long Timeout would. Zero means
// none. Run steps only. Needs CapHeartbeat.
HeartbeatTimeout time.Duration
}
Step is one node of a workflow. Exactly one of Run, Wait and Sleep is set; Validate rejects anything else.
func Choice ¶
func Choice(c ChoiceSpec) ([]Step, error)
Choice expands c into ordinary steps, so it runs on every driver with no capability beyond what the steps need: a decide step ("<Name>.decide"), one step per branch ("<Name>.<branch>") and a join step named Name. Every branch step starts once the decide step is done; the branches not chosen return at once without calling their Run, and the join step returns the chosen branch's output unchanged. A run listing therefore shows every branch step as succeeded; the one that did work is the one whose Run was called.
Append the result to Workflow.Steps. It returns an error wrapping ErrInvalid when the spec is incomplete or a branch name repeats.
type StepCanceller ¶
StepCanceller withdraws one step of a run, leaving the rest of the run to decide what happens next.
CancelStep acts on a step that is running, waiting on its retry backoff, snoozed, or a wait or sleep step that is suspended:
- a running attempt's ctx is cancelled with cause ErrStepCancelled (context.Cause). Whatever the handler then returns decides: an output means the step succeeded and the run continues (a handler can return a fallback, the "interrupting boundary" of a process model); an error fails the step for good, without retry, with an error matching ErrStepCancelled, and the run fails as any failed run does (OnFailure runs);
- a step with no attempt running (backoff, snooze, wait, sleep) fails for good at once with an error matching ErrStepCancelled.
A step that has not started, or has ended, returns ErrWrongState; an unknown run or step ErrNotFound. Cancelling the whole run is Cancel.
type StepEvent ¶ added in v0.1.1
type StepEvent struct {
// Message is a short note for a person.
Message string
// Attrs are the event's key/value pairs, values already rendered as
// strings so every sink stores the same text. Order is the caller's.
Attrs []EventAttr
// Level is the slog level the caller gave (slog.LevelInfo by default
// with RecordEvent, any level with RecordEventLevel).
Level slog.Level
// At is when it was recorded. RecordEvent leaves it zero, because the
// core reads no wall clock on a step's path (a fake clock in a test
// must see the same time a sink stores); each sink stamps it from its
// own clock when zero.
At time.Time
// Step is the StepInfo of the attempt that recorded it; zero outside
// a driver-run step.
Step StepInfo
}
StepEvent is one line of a step's timeline ("downloaded 12 MB, 4,000 rows"), recorded from inside a step with RecordEvent and handed to every StepEventSink bound on the step's ctx: a run recorder stores it with the step, a tracer adds it as a span event, a log capture writes it.
type StepEventFunc ¶ added in v0.1.1
StepEventFunc adapts a function to StepEventSink.
func (StepEventFunc) RecordStepEvent ¶ added in v0.1.1
func (f StepEventFunc) RecordStepEvent(ctx context.Context, e StepEvent)
RecordStepEvent calls f.
type StepEventSink ¶ added in v0.1.1
StepEventSink receives the events a step records. A sink must be safe for concurrent use (steps of one run may run in parallel) and must not block: RecordEvent calls it on the step's goroutine.
type StepInfo ¶
type StepInfo struct {
// Workflow is the workflow name.
Workflow string
// Step is the step name.
Step string
// RunID is the run the step belongs to.
RunID RunID
// Attempt is 1 for the first attempt and grows by one per retry.
Attempt int
// Labels are the run's Job.Labels. Drivers pass the run's own map;
// handlers and middleware must not modify it.
Labels map[string]string
// Queue is the run's queue (EffectiveQueue). Drivers without
// CapQueue set DefaultQueue or leave it empty.
Queue string
// ReadyAt is when this attempt became ready to claim, on the
// driver's clock: the run's admission for a first step, its parents'
// end for a later one, the backoff's end for a retry. Now minus
// ReadyAt is how long the attempt waited for a worker (the queue wait
// a metric reports). Zero when the driver does not know it.
ReadyAt time.Time
// Tenant is the run's Job.Tenant; empty without one or on a driver
// without CapTenant.
Tenant string
// Node is the identity of the node running this attempt (the
// driver's node-ID option, DefaultNodeID when unset); empty on a
// driver without CapNodeIdentity.
Node string
}
StepInfo describes the step attempt a Handler is running, so middleware (logging, tracing, metrics) and handlers can tell runs and attempts apart without a driver-specific context. Every driver sets it on the ctx it passes to a step handler, with WithStepInfo, before the Middleware chain runs.
func StepInfoFrom ¶
StepInfoFrom returns the StepInfo a driver set on ctx, and false when ctx is not a step's context (for example in an enqueue middleware).
type StepKind ¶
type StepKind string
StepKind is what a step does. Exactly one kind is set per step.
type StepMiddlewareProvider ¶
type StepMiddlewareProvider interface {
StepMiddleware() []Middleware
}
StepMiddlewareProvider contributes step middleware. It wraps every step's Run and the workflow's OnFailure for workflows registered through the client Use returns, inside any middleware the driver itself was built with (so a driver-level codec has already opened the input). The first entry is the outermost.
type StopCauser ¶
type StopCauser interface {
StopCause() error
}
StopCauser reports why a stopped client stopped. A driver with recycling implements it; Forward, failover and the middleware client wrappers pass it through.
StopCause returns nil while the client runs and after a stop that was asked for (the Start ctx ended, Drain, a stop-when-idle option), and a *RecycleError when a Recycle limit stopped it. It is final once the Drainer's Done channel is closed.
type Tick ¶
type Tick struct {
// ScheduleID is the schedule. For a declared Workflow.Cron entry it
// is the driver's stable id for that declaration, the same one
// ScheduleLister reports.
ScheduleID ScheduleID
// Name is the Schedule.Name; for a declared cron, the expression.
Name string
// Workflow is the workflow the schedule runs.
Workflow string
// Declared is true for a Workflow.Cron entry.
Declared bool
// At is the tick's time; for TickMisfired, the newest dropped tick.
At time.Time
// Location is the zone the schedule is evaluated in.
Location *time.Location
// Outcome is what happened.
Outcome TickOutcome
// Reason is set for TickSkipped: TickReasonWhen, TickReasonWindow or
// TickReasonPaused.
Reason string
// Missed is how many ticks a TickMisfired dropped.
Missed int
// Run is the run a TickFired enqueued.
Run RunRef
// Err is why a TickFired could not enqueue.
Err error
// Labels are the schedule's labels. Observers must not modify them.
Labels map[string]string
// Manual is true for a TickFired that ScheduleTriggerer started by
// hand; At is then the driver's clock at the trigger.
Manual bool
}
Tick reports one tick of a schedule, or, for TickMisfired, one batch of dropped ticks.
type TickNotifier ¶
type TickNotifier interface {
OnTick(fn TickObserver) error
}
TickNotifier is an optional interface a client implements to report every tick outcome: fired, skipped and misfired. A client implements it when it declares CapTickEvents. The contract is RunNotifier's: register before Start (ErrAlreadyStarted after), observers run on a driver goroutine in registration order, panics are recovered, delivery is at least once and may be lost on a crash. Ticks are reported only by the node that fired or skipped them.
type TickObserver ¶
TickObserver observes ticks. It must be idempotent, because delivery is at least once.
type TickOutcome ¶
type TickOutcome string
TickOutcome is what happened to one schedule tick.
const ( // TickFired: the tick enqueued a run (Tick.Run), or tried to and the // enqueue failed (Tick.Err). TickFired TickOutcome = "fired" // TickSkipped: the tick was due and on time but did not fire; // Tick.Reason says why. TickSkipped TickOutcome = "skipped" // TickMisfired: the scheduler woke late and its misfire policy // dropped Tick.Missed ticks, which a scheduler that looks only at the // current minute cannot even see. TickMisfired TickOutcome = "misfired" )
The tick outcomes.
type TickSource ¶
type TickSource interface {
// Between returns the fire times t with from < t <= to, ascending, at
// most limit of them.
Between(from, to time.Time, loc *time.Location, limit int) []time.Time
}
TickSource lists a schedule's fire times. cron.Schedule satisfies it; the core names only the one method it needs, so it does not import a parser.
type Transition ¶
type Transition struct {
From, To Status
// Trigger says what causes the move, in a few words.
Trigger string
// Needs is every capability the driver must declare for the move to be
// reported; zero means any driver.
Needs Caps
}
Transition is one legal move of the run state machine: a run in From may be reported as To when Trigger happens on a driver that has every capability in Needs.
func Transitions ¶
func Transitions() []Transition
Transitions returns every legal move of the run state machine. The slice is a copy. Drivers and tests use it to check that a status change they are about to store (or one they read) is a move the machine allows.
type UnsupportedError ¶
type UnsupportedError struct {
// Field is the spec field or feature, such as "Job.RunAt" or
// "Schedule.Location".
Field string
// Missing is the capabilities that would be needed, when the refusal
// comes from a capability check; zero otherwise.
Missing Caps
// Reason is an optional detail, such as the workflow name.
Reason string
}
UnsupportedError names what a driver cannot do. It matches ErrUnsupported under errors.Is.
func (*UnsupportedError) Error ¶
func (e *UnsupportedError) Error() string
Error names the field, the missing capabilities and the reason, so the message alone tells an operator what to change.
func (*UnsupportedError) Is ¶
func (e *UnsupportedError) Is(target error) bool
Is makes errors.Is(err, ErrUnsupported) true.
type Upcaster ¶
Upcaster turns a payload of one version into the next version's shape (version n to n+1): rename a field, fill a default, split a value.
type UpstreamLimit ¶
type UpstreamLimit struct {
// contains filtered or unexported fields
}
UpstreamLimit is a leaky-bucket, cost-based rate limit on calls a step makes to an outside API (PS-02), one bucket per key. Unlike Workflow.RateLimits, which counts run starts, it counts what the upstream counts: the cost of each call, against a bucket that drains over time. Build it with NewUpstreamLimit.
func NewUpstreamLimit ¶
func NewUpstreamLimit(store BucketStore, spec BucketSpec, clk clock.Clock) (*UpstreamLimit, error)
NewUpstreamLimit returns a limit over store with spec, reading time from clk. A nil store means a new MemoryBuckets; a nil clk means clock.Real. It returns an error wrapping ErrInvalid when spec is not positive and finite.
func (*UpstreamLimit) Observe ¶
Observe records the level the upstream reported for key, so the next Take decides from the upstream's count. Call it with each response's rate-limit header.
func (*UpstreamLimit) Take ¶
Take reserves cost units in key's bucket. It returns nil when they fit; otherwise RetryAfter(ErrThrottled, wait), which a step returns as it is so the attempt is retried once the bucket has room (see ErrThrottled). A cost above the bucket's capacity can never fit and is a Permanent error wrapping ErrInvalid; a negative cost is refused the same way.
func (*UpstreamLimit) TakeOrSnooze ¶
TakeOrSnooze is Take for a driver with CapSnooze: when the bucket has no room it returns Snooze(wait), which puts the step back without spending an attempt, since a spent upstream limit is "not yet", not a failure. On a driver without CapSnooze a snooze is an ordinary error and spends an attempt; use Take there.
type WaitSpec ¶
type WaitSpec struct {
// Event is the event name to wait for.
Event string
// Key is a fixed key, the same for every run of the workflow. Empty
// matches an Event with an empty key. Use KeyFrom for a key that
// differs per run.
Key string
// KeyFrom derives the key per run, so many runs of one workflow can
// wait on the same event name for different things ("payment.succeeded
// for THIS order"). The driver evaluates it once, when the wait step
// begins, against the run's Job (its original Payload and Labels, not
// the step's input), and Signal matches Event.Key against the result.
// Func is evaluated in Go by any driver (ResolveKey does it); Expr is
// an expression in the engine's language over the run input (Hatchet's
// CEL), and a driver without one refuses it with an UnsupportedError.
// At most one of Key and KeyFrom is set.
KeyFrom ConcurrencyKey
// Deadline is the absolute time the wait fails.
Deadline time.Time
// Timeout is how long the wait lasts from when it begins.
Timeout time.Duration
}
WaitSpec is a durable wait: the run releases its worker until Signal delivers an Event with the same name and key, or the wait expires. At most one of Deadline and Timeout is set; neither means wait forever.
func (WaitSpec) ResolveKey ¶
ResolveKey returns the key a run waits on: KeyFrom.Func applied to job when set, else the fixed Key. A KeyFrom.Expr is the engine's to evaluate, so here it returns an UnsupportedError naming the field; a driver that evaluates expressions handles that case before calling.
type Window ¶
type Window struct {
From, To string
}
Window is a time-of-day range on the wall clock of a schedule's zone, "HH:MM" each (24-hour). From is inclusive and To exclusive; a To before From wraps midnight ("22:00" to "06:00" is the night). From equal to To is refused, because it would mean either nothing or the whole day.
type Windows ¶
Windows limits a schedule's ticks by time of day: a tick fires only inside at least one Between window (no Between windows: at any time) and outside every Unless window.
type WorkerScaler ¶
type WorkerScaler interface {
// QueueLoads returns the load of every configured queue, sorted by
// name. Before Start it returns ErrNotStarted.
QueueLoads(ctx context.Context) ([]QueueLoad, error)
// SetWorkers sets queue's worker count to n, effective for the next
// attempt claimed: lowering it never interrupts running attempts.
// n below one wraps ErrInvalid; a queue the client was not configured
// with wraps ErrNotFound. Before Start it returns ErrNotStarted.
SetWorkers(ctx context.Context, queue string, n int) error
}
WorkerScaler is implemented by a client whose per-queue worker counts can change while it runs. A client implements it when it declares CapWorkerScale. It covers the queues the client was configured with (its QueueConfig list); the counts are this client's own, so on a shared store each process balances its own workers.
type Workflow ¶
type Workflow struct {
// Name is the key Job.Workflow and Schedule.Workflow refer to.
Name string
// Steps form a DAG through Step.After. A step with no After starts
// first; a plain list with no After runs its steps in parallel, so
// chain them with After for sequence.
Steps []Step
// Cron lists 5-field expressions declared with the workflow and
// evaluated in CronLocation. They are replaced, never duplicated, when
// the workflow registers again. Needs CapCron.
Cron []string
// CronLocation is the time zone Cron is evaluated in. Nil means UTC
// (the behaviour before the field existed); time.Local is refused, so
// no declared cron runs in the host's implicit zone. Any zone but UTC
// needs CapCronZone: a UTC-only engine refuses it rather than shifting
// the expression, which would break across DST.
CronLocation *time.Location
// Misfire says what happens to Cron ticks the driver reached late or
// not at all (nothing was running). The zero value is the driver's
// default, documented in its package. Setting it without Cron is
// invalid. See Misfire for the capability each policy needs.
Misfire Misfire
// Concurrency caps simultaneous runs per key. Nil means no cap.
// Needs CapConcurrencyKey.
Concurrency *Concurrency
// RateLimits cap how often runs may start. Needs CapRateLimit.
RateLimits []RateLimit
// Idempotency says how Job.IdempotencyKey is treated. Empty means
// IdempotencyOptional.
Idempotency IdempotencyPolicy
// Retry is the default for steps whose Retry is nil. Its zero value
// means no retries.
Retry RetryPolicy
// Timeout is the default per-step timeout. Zero means none.
Timeout time.Duration
// OnFailure runs once when a run fails for good, with the failing
// step's input. Nil means nothing runs.
OnFailure Handler
// Queue is the default queue of this workflow's runs; Job.Queue
// overrides it per run. Empty means DefaultQueue. A queue other than
// DefaultQueue needs CapQueue. See QueueConfig for how workers serve
// queues.
Queue string
// ScheduleWhen is asked before each tick of this workflow's schedules
// fires: the declared Cron and every Client.Schedule of the workflow
// alike (switch on tick.Name for a rule per schedule). Returning false
// skips the tick, which tick observers see as TickSkipped with Reason
// TickReasonWhen. Nil fires every tick. It lives on the workflow, not
// on Schedule, because a condition is code that every node registers,
// while a durable driver stores a Schedule as data and may fire it on
// any node. It is called on the firing node with the driver's context
// and must return quickly. Needs CapScheduleWhen.
ScheduleWhen func(ctx context.Context, tick Tick) bool
// CronWindows limits the declared Cron ticks to time-of-day windows on
// CronLocation's wall clock. Zero means no limit. Needs
// CapScheduleWindow.
CronWindows Windows
// CronOverlap is the overlap policy of every Cron entry: OverlapSkip
// skips a tick while that entry's previous run is unfinished. The
// zero value is OverlapAllow. Each entry is its own schedule, so one
// entry's run does not hold back another's tick. It needs Cron.
CronOverlap OverlapPolicy
// CronSeconds says every Cron entry has six fields, seconds first
// (cron.ParseSeconds). It is never guessed from the field count,
// because six fields could as well mean a trailing year. Needs Cron
// and CapCronSeconds.
CronSeconds bool
// CronActive limits the declared Cron to a period: ticks before
// Active.From or from Active.Until on do not fire (see Active). Zero
// means always. Needs Cron and CapScheduleActive.
CronActive Active
// CronKeepRuns keeps only the newest N finished runs fired by each
// Cron entry; older finished runs are forgotten as RunDeleter would.
// Zero keeps them all (the driver's retention still applies). Needs
// Cron and CapScheduleKeep.
CronKeepRuns int
// RunTimeout caps a run's whole life, from its admission to its end
// (waits and sleeps included). A run past it is ended: running
// attempts see their ctx cancelled with cause ErrRunTimeout, and the
// run fails with an error matching ErrRunTimeout and ErrTimeout, which
// is never retried. Zero means none. Needs CapRunTimeout.
RunTimeout time.Duration
// OnTimeout runs once, in place of OnFailure, when a run fails for
// good because of a timeout (IsTimeout: a step attempt past its
// Timeout on its last attempt, a wait past its deadline, a heartbeat
// timeout or RunTimeout), with the failing step's input (the run's
// payload for RunTimeout outside any step). Nil means OnFailure runs
// for timeouts too. Needs CapRunTimeout.
OnTimeout Handler
}
Workflow is a named graph of steps plus the policies that apply to every run of it. A one-step workflow is a plain background job.
Source Files
¶
- balance.go
- caps.go
- chain.go
- child.go
- client.go
- closure.go
- deferred.go
- doc.go
- drain.go
- errors.go
- flow_bucket.go
- flow_choice.go
- flow_chunks.go
- flow_data.go
- flow_iter.go
- flow_journal.go
- flow_period.go
- flow_secret.go
- flow_spawn.go
- force.go
- forward.go
- func.go
- heartbeat.go
- history.go
- hooks.go
- inline.go
- input.go
- labels.go
- lifecycle.go
- misfire.go
- node.go
- outcome.go
- overlap.go
- pause.go
- payload.go
- plugin.go
- progress.go
- query.go
- queue.go
- recycle.go
- report.go
- reschedule.go
- result.go
- retry.go
- runerror.go
- runevent.go
- runoutcome.go
- sched_active.go
- sched_display.go
- sched_env.go
- sched_json.go
- sched_local.go
- sched_parse.go
- sched_tick.go
- snooze.go
- source.go
- spec.go
- status.go
- stepevent.go
- stepinfo.go
- supports.go
- tenant.go
- tick.go
- timeout.go
- typed.go
- validate.go
Directories
¶
| Path | Synopsis |
|---|---|
|
admin
module
|
|
|
Package batch runs a group of jobs as one unit with Then, Catch and Finally callbacks, on any workflow.Client.
|
Package batch runs a group of jobs as one unit with Then, Catch and Finally callbacks, on any workflow.Client. |
|
cli
module
|
|
|
Package clock is the time source every scheduling decision reads.
|
Package clock is the time source every scheduling decision reads. |
|
Package cron parses cron expressions and computes their fire times.
|
Package cron parses cron expressions and computes their fire times. |
|
tzdata
Package tzdata embeds the IANA time zone database in the binary, for apps that run schedules in a container image without one (scratch, distroless and many slim images ship no /usr/share/zoneinfo).
|
Package tzdata embeds the IANA time zone database in the binary, for apps that run schedules in a container image without one (scratch, distroless and many slim images ship no /usr/share/zoneinfo). |
|
Package events starts runs from events: publish/subscribe inside the library (EV-01) and an inbound webhook handler that verifies a signature and enqueues (EV-03), with HMAC signing schemes for both directions (WH-03).
|
Package events starts runs from events: publish/subscribe inside the library (EV-01) and an inbound webhook handler that verifies a signature and enqueues (EV-03), with HMAC signing schemes for both directions (WH-03). |
|
Package failover joins two workflow clients into one: enqueues go to a primary backend, and when the primary is unavailable they go to a secondary, so a job is still accepted while (say) the Hatchet engine is down and a local SQL store takes the work.
|
Package failover joins two workflow clients into one: enqueues go to a primary backend, and when the primary is unavailable they go to a secondary, so a job is still accepted while (say) the Hatchet engine is down and a local SQL store takes the work. |
|
hatchet
module
|
|
|
Package hybrid splits one workflow.Client across two backends: every schedule fires on a scheduler client (usually the in-process memory driver) while every run lives on a jobs client (a durable driver), the common "cron in the process, jobs in the database" shape.
|
Package hybrid splits one workflow.Client across two backends: every schedule fires on a scheduler client (usually the in-process memory driver) while every run lives on a jobs client (a durable driver), the common "cron in the process, jobs in the database" shape. |
|
Package memory is the in-process driver of workflow.Client: runs, waits, sleeps, schedules and queues live in maps and goroutines inside the client value, with no database and no server.
|
Package memory is the in-process driver of workflow.Client: runs, waits, sleeps, schedules and queues live in maps and goroutines inside the client value, with no database and no server. |
|
Package middleware is the app-side layer that sits above every driver: the conveniences a job queue offers on top of its broker, written once for Go.
|
Package middleware is the app-side layer that sits above every driver: the conveniences a job queue offers on top of its broker, written once for Go. |
|
otel
module
|
|
|
recorder
module
|
|
|
sqlrecorder
module
|
|
|
redis
module
|
|
|
sentry
module
|
|
|
sqlstore
module
|
|
|
pgnotify
module
|
|
|
pgxtx
module
|
|
|
Package workflowtest holds the conformance suite every workflow driver runs, and two doubles for app tests: Fake, a recording client that runs nothing and refuses what a real driver refuses, with expressive assertions (AssertEnqueued and its siblings); and Recorder, which wraps a real client and records its enqueues.
|
Package workflowtest holds the conformance suite every workflow driver runs, and two doubles for app tests: Fake, a recording client that runs nothing and refuses what a real driver refuses, with expressive assertions (AssertEnqueued and its siblings); and Recorder, which wraps a real client and records its enqueues. |