Documentation
¶
Overview ¶
Package middleware is the app-side layer that sits above every driver: the conveniences a job queue offers on top of its broker, written once for Go. Step middleware (workflow.Middleware) wraps each step execution; enqueue middleware (workflow.EnqueueMiddleware) wraps each Client.Enqueue, and WrapClient applies it to any driver.
Step middleware: Logging, Recover, Timeout, FailOn, Skip, Release (puts the step back for a delay without running it, as workflow.RetryAfter wrapping ErrReleased), ThrottleErrors, ReleaseUniqueOnStart (the step half of UniqueUntilStarted), DecodeStep (the step half of a Codec) and ClaimCheck.Step (resolves claim check references before the handler and stores large outputs). Logging picks each event's level through WithLogLevel (by default Debug for start and finish, Error for every failed attempt). Encrypt(key, oldKeys...) seals with key and also opens values sealed with any old key, so a key can be rotated without breaking runs already in flight.
Enqueue middleware: Unique, UniqueUntilFinished (the lock is freed when the run ends) and UniqueUntilStarted (freed when a step starts, through its step half), Debounce, Route, EncodeEnqueue (the enqueue half of a Codec) and ClaimCheck.Enqueue (moves a large payload into a BlobStore). WithCodec wraps a whole Client so Signal payloads and Schedule inputs are sealed too, and ClaimCheck.CleanupOnFinish deletes a run's blobs when the run ends.
Chains: a driver enqueues the jobs of a Job.Chain itself, past enqueue middleware, so the codec and the claim check rewrite those jobs when the first job is enqueued (each chained job gets its own claim token) and wrap the step's chain editor so jobs a step adds with workflow.PrependToChain or AppendToChain are rewritten too. The other enqueue middleware (Unique, Debounce, Route, OpenTelemetry's) applies to the first job only. Deferred dispatch (workflow.DeferEnqueue, workflow.DeferHTTP) lives in the core: it is enqueue middleware too.
Schedule hooks: Pinger turns heartbeat URLs into the funcs workflow.ScheduleHooks takes (a ping when a tick fires, after a run, on success and on failure), sending through a PingFunc (HTTPPing by default) under a timeout, and logging a failure without the URL.
Task output: CaptureOutput collects what a step logged through OutputLogger, how the attempt ended and, with CaptureResult only, its result, and hands it after each attempt to an OutputSink: WriterSink, FileSink (FileReplace or FileAppend) or an OutputSinkFunc for mail, chat or any other destination. A sink error never fails the step.
Unique by arguments: UniqueByArgs is a key function for Unique and its siblings that makes a job unique by its workflow and canonical payload (ArgsFields picks the top-level fields that count; ArgsQueue and ArgsLabels add the queue and labels).
Failures: Alerts is a run-finished observer that calls an Alerter (the app's Slack, mail or pager sink) on every run that ends failed, with a pluggable AlertLimiter for dedup and rate limits (MemoryAlertLimiter, ten minutes per workflow and step by default); AlertPlugin bundles it for workflow.Use. Report hands failures to a workflow.Reporter (an error tracker): final failures always, each failed attempt with ReportConfig.Attempts. Neither sends the payload. Recover's error carries the panic's stack (workflow.StackOf), which drivers with workflow.CapErrorHistory keep in RunInfo.Errors.
Nothing here keeps hidden state: every piece of state (a unique store, a debounce table, a circuit breaker) hangs off the value its constructor returns, and time comes from an injected clock.Clock. State that lives in such a value is per process; a guarantee that must hold across processes needs a store that is shared, which each constructor takes: UniqueStore for Unique, DebounceStore (WithDebounceStore) for Debounce and CircuitStore (ThrottleConfig.Store) for ThrottleErrors and AlertLimiter (AlertConfig.Limiter) for Alerts, each with an in-process default (MemoryUnique, MemoryDebounce, MemoryCircuit, MemoryAlertLimiter).
UniqueWith adds two choices to Unique: which states of the holder's run count (UniqueConfig.States, looked up through Client.Status, so a key can open again once its run started) and answering a duplicate with the holder's RunRef (ReturnExisting, marking workflow.EnqueueOutcome) instead of ErrDuplicate; both need a store implementing UniqueHolders. ArgsTagged builds UniqueByArgs's field list from `unique:"true"` struct tags. MaxPayload refuses an oversized payload before the driver sees it, and PayloadVersion stamps workflow.LabelPayloadVersion so a step built with workflow.DefineVersioned can upcast older payloads. Stuck reports a step attempt that runs longer than expected without touching it.
Batches (Then, Catch, Finally) are not middleware: they live in package batch, on any Client.
Index ¶
- Constants
- Variables
- func AlertPlugin(a Alerter, cfg AlertConfig) workflow.Plugin
- func Alerts(a Alerter, cfg AlertConfig) workflow.RunFinishedFunc
- func CaptureOutput(sink OutputSink, opts ...CaptureOption) workflow.Middleware
- func Chain(mws ...workflow.Middleware) workflow.Middleware
- func ChainEnqueue(mws ...workflow.EnqueueMiddleware) workflow.EnqueueMiddleware
- func Debounce(c workflow.Client, clk clock.Clock, key func(workflow.Job) string, ...) (workflow.EnqueueMiddleware, error)
- func DecodeStep(c Codec) workflow.Middleware
- func EncodeEnqueue(c Codec) workflow.EnqueueMiddleware
- func FailOn(match func(error) bool) workflow.Middleware
- func FormatOutput(w io.Writer, out Output) error
- func Logging(l *slog.Logger, clk clock.Clock, opts ...LoggingOption) workflow.Middleware
- func MaxPayload(limit int) workflow.EnqueueMiddleware
- func OutputLogger(ctx context.Context, fallback *slog.Logger) *slog.Logger
- func PayloadVersion(current int) workflow.EnqueueMiddleware
- func Recover(l *slog.Logger) workflow.Middleware
- func Release(when func(ctx context.Context, in []byte) bool, delay time.Duration) workflow.Middleware
- func ReleaseUniqueOnStart(store UniqueStore) workflow.Middleware
- func Report(r workflow.Reporter, cfg ReportConfig) workflow.Plugin
- func ReportAttempts(r workflow.Reporter, cfg ReportConfig) workflow.Middleware
- func ReportFailures(r workflow.Reporter, cfg ReportConfig) workflow.RunFinishedFunc
- func RequireSource(allowed ...workflow.Source) workflow.EnqueueMiddleware
- func Route(rules ...RouteRule) workflow.EnqueueMiddleware
- func Skip(skip func(ctx context.Context, in []byte) bool) workflow.Middleware
- func Stuck(cfg StuckConfig) workflow.Middleware
- func ThrottleErrors(cfg ThrottleConfig) (workflow.Middleware, error)
- func Timeout(d time.Duration) workflow.Middleware
- func Unique(by func(workflow.Job) string, store UniqueStore, ttl time.Duration) workflow.EnqueueMiddleware
- func UniqueByArgs(opts ...ArgsOption) func(workflow.Job) string
- func UniqueUntilFinished(c workflow.Client, by func(workflow.Job) string, store UniqueStore, ...) (workflow.EnqueueMiddleware, error)
- func UniqueUntilStarted(by func(workflow.Job) string, store UniqueStore, ttl time.Duration) (enqueue workflow.EnqueueMiddleware, step workflow.Middleware)
- func UniqueWith(cfg UniqueConfig) (workflow.EnqueueMiddleware, error)
- func WithCodec(c workflow.Client, codec Codec, mws ...workflow.EnqueueMiddleware) workflow.Client
- func WrapClient(c workflow.Client, mws ...workflow.EnqueueMiddleware) workflow.Client
- type Alert
- type AlertConfig
- type AlertLimiter
- type AlertLimits
- type Alerter
- type AlerterFunc
- type ArgsOption
- type BlobPrefixDeleter
- type BlobStore
- type CaptureOption
- func CaptureClock(clk clock.Clock) CaptureOption
- func CaptureLevel(l slog.Leveler) CaptureOption
- func CaptureOnSinkError(fn func(ctx context.Context, out Output, err error)) CaptureOption
- func CaptureOnly(keep func(workflow.StepInfo) bool) CaptureOption
- func CaptureOnlyFailures() CaptureOption
- func CaptureResult() CaptureOption
- func CaptureTee(h slog.Handler) CaptureOption
- type CircuitPolicy
- type CircuitStore
- type ClaimCheck
- type ClaimCheckOption
- type Codec
- type DebounceBurst
- type DebounceOption
- type DebounceStore
- type FileMode
- type LoggingOption
- type MemoryAlertLimiter
- type MemoryBlobStore
- func (m *MemoryBlobStore) Delete(_ context.Context, key string) error
- func (m *MemoryBlobStore) DeletePrefix(_ context.Context, prefix string) error
- func (m *MemoryBlobStore) Get(_ context.Context, key string) ([]byte, error)
- func (m *MemoryBlobStore) Len() int
- func (m *MemoryBlobStore) Put(_ context.Context, key string, data []byte) error
- type MemoryCircuit
- type MemoryDebounce
- type MemoryUnique
- func (m *MemoryUnique) Acquire(_ context.Context, key, owner string, ttl time.Duration) (bool, error)
- func (m *MemoryUnique) Holder(_ context.Context, key string) (string, workflow.RunRef, bool, error)
- func (m *MemoryUnique) Release(_ context.Context, key, owner string) error
- func (m *MemoryUnique) SetHolder(_ context.Context, key, owner string, ref workflow.RunRef) error
- type Output
- type OutputRecord
- type OutputSink
- type OutputSinkFunc
- type PingFunc
- type Pinger
- type PingerOption
- type ReportConfig
- type RouteRule
- type StepLogEvent
- type StepPhase
- type StuckConfig
- type StuckEvent
- type ThrottleConfig
- type UniqueConfig
- type UniqueHolders
- type UniqueStore
Constants ¶
const ( UniqueKeyLabel = "ubgo.workflow/unique-key" UniqueOwnerLabel = "ubgo.workflow/unique-owner" )
The labels Unique stamps on every job it passes, so the run-finished observer of UniqueUntilFinished can find the lock from the RunEvent alone, in any process and after a restart, without an in-process table. Unique owns them: a job that arrives carrying either, or any other label under workflow.LabelPrefixInternal except ClaimCheckLabel (which ClaimCheck.Enqueue may have stamped before it), is refused with workflow.ErrInvalid, so a caller cannot release someone else's lock by setting them. A job enqueued straight on the driver, bypassing Unique, is not covered by that and must not carry them. One Unique per chain: a second would refuse the first one's labels.
const ClaimCheckLabel = "ubgo.workflow/claim-check"
ClaimCheckLabel carries the run's claim check token, a random value ClaimCheck.Enqueue draws per job. Every blob of the run is stored under "<token>/", and the step middleware resolves a reference only when its key is under the token of the run it runs in. ClaimCheck owns the label: Enqueue refuses, with workflow.ErrInvalid, a job that arrives carrying it or any other label under workflow.LabelPrefixInternal except Unique's two (Unique may run before it), so a caller can neither choose a token nor point a run at another run's blobs. A job enqueued straight on the driver, bypassing ClaimCheck.Enqueue, must not carry it.
const DefaultAlertDedup = 10 * time.Minute
DefaultAlertDedup is how long Alerts stays quiet about a key it has just alerted on, when AlertConfig.Limiter is nil: one page per broken workflow every ten minutes, not one per failed run.
const DefaultArgsKeyPrefix = "args:"
DefaultArgsKeyPrefix starts every key UniqueByArgs builds, so its locks cannot collide with keys an app's own key function builds in the same UniqueStore.
const DefaultPingTimeout = 10 * time.Second
DefaultPingTimeout bounds one ping when no WithPingTimeout is given: long enough for a heartbeat service across the internet, short enough that a dead one does not hold up the driver goroutine that delivers hooks.
const UniqueTag = "unique"
UniqueTag is the struct tag ArgsTagged reads: a field tagged `unique:"true"` counts toward the key.
Variables ¶
var ErrBlobNotFound = errors.New("workflow/middleware: blob not found")
ErrBlobNotFound is what a BlobStore's Get returns for a key it does not hold. The claim check step middleware turns it into a Permanent error: a blob that is gone (expired, deleted by hand) does not come back on retry.
var ErrDecode = errors.New("workflow/middleware: payload does not decode")
ErrDecode is wrapped by the error DecodeStep returns when the input does not decode (a wrong key, tampered or truncated bytes, or a value that was never encoded). The error never carries the bytes themselves.
var ErrPanic = errors.New("workflow/middleware: step panicked")
ErrPanic is wrapped by the error Recover returns for a panicking step.
var ErrReleased = errors.New("workflow/middleware: step released")
ErrReleased is wrapped by the error Release returns for a step it did not run, so a retry policy, a log filter or a test can tell a release from a real failure with errors.Is.
var ErrThrottled = errors.New("workflow/middleware: step throttled after repeated errors")
ErrThrottled is returned, without calling the step, while a key's circuit is open. It is retryable on purpose: the retry policy's backoff is what brings the step back after the cooldown, as a release back onto the queue would.
Functions ¶
func AlertPlugin ¶
func AlertPlugin(a Alerter, cfg AlertConfig) workflow.Plugin
AlertPlugin bundles Alerts as a plugin named "alerts" for workflow.Use.
func Alerts ¶
func Alerts(a Alerter, cfg AlertConfig) workflow.RunFinishedFunc
Alerts returns a run-finished observer that calls a on every run that ends failed (and discarded, when cfg.Discarded) and that the limiter lets through. A limiter error sends the alert anyway: a broken limit store must not hide a failure. Register it with OnRunFinished, or use AlertPlugin.
func CaptureOutput ¶
func CaptureOutput(sink OutputSink, opts ...CaptureOption) workflow.Middleware
CaptureOutput is step middleware that captures each attempt's log records (written through OutputLogger, and step events recorded with workflow.RecordEvent) and outcome and delivers them to sink after the attempt, the way a scheduler sends a task's output to a file or by mail. It never changes the step's result. A nil sink makes it a no-op.
func Chain ¶
func Chain(mws ...workflow.Middleware) workflow.Middleware
Chain composes step middleware into one. The first argument is the outermost: Chain(a, b)(h) runs a, then b, then h, the order the list is read in. Nil entries are skipped, so a caller can build the list with optional pieces. An empty chain returns the handler unchanged.
func ChainEnqueue ¶
func ChainEnqueue(mws ...workflow.EnqueueMiddleware) workflow.EnqueueMiddleware
ChainEnqueue composes enqueue middleware with the same ordering rule as Chain: the first argument sees the job first.
func Debounce ¶
func Debounce(c workflow.Client, clk clock.Clock, key func(workflow.Job) string, window, maxWait time.Duration, opts ...DebounceOption) (workflow.EnqueueMiddleware, error)
Debounce makes the last of a burst of jobs with the same key win: each enqueue delays its run to now+window and cancels the pending run of the previous job with that key, so only the last one runs, once the key has been quiet for window. maxWait caps the delay from the first job of the burst, so a key that never goes quiet still runs at least every maxWait; zero means no cap. A job for which key returns "" passes through.
It is built on Job.RunAt and Client.Cancel of c, the client the jobs finally reach, and returns an UnsupportedError (matching workflow.ErrUnsupported) when c lacks CapDelayedStart or CapCancel, rather than letting every job through undebounced.
Behaviour to know:
- The job's own RunAt is replaced.
- The new run is enqueued before the old one is cancelled, so a failed enqueue never loses the burst. If the cancel then fails, Debounce returns the new RunRef together with the error: both runs may fire.
- A previous run whose start time has passed is left alone and the job opens a new burst; cancelling it could stop a run already working.
- Do not set Job.IdempotencyKey on debounced jobs: the pending run holds the key, so the replacement would be folded into it.
- The bursts live in a DebounceStore, by default a MemoryDebounce of the returned middleware's own, so debouncing is per process; a key enqueued from two processes is debounced in each unless WithDebounceStore names a store they share. Calls in one process are serialised by one lock, held across the driver calls.
A store shared by several processes, for example:
mw, err := middleware.Debounce(c, nil, byUser, 10*time.Second, time.Minute, middleware.WithDebounceStore(myRedisDebounce))
func DecodeStep ¶
func DecodeStep(c Codec) workflow.Middleware
DecodeStep opens a step's input before the handler sees it and seals its output afterwards, and seals the payload of every job the step adds to its chain (workflow.PrependToChain, AppendToChain), so a child step's input (its parent's output) is sealed just like a job payload and every byte the engine stores stays sealed. It handles every input shape a driver builds with workflow.StepInput: the payload, one parent's output, and the JSON object of several parents, which it opens per parent. A wait step's output is the Signal payload, so Signal must go through WithCodec.
Input that does not open, or that was never sealed, is a Permanent error wrapping ErrDecode: retrying the same bytes with the same key cannot succeed, and accepting plaintext would make the codec optional without anyone noticing.
func EncodeEnqueue ¶
func EncodeEnqueue(c Codec) workflow.EnqueueMiddleware
EncodeEnqueue seals Job.Payload, and the payload of every job in Job.Chain, before the job reaches the driver, so the engine only ever stores sealed bytes. Pair it with DecodeStep using the same Codec on the worker side. WithCodec does this and also seals Signal payloads and Schedule inputs; prefer it unless only Enqueue carries data.
func FailOn ¶
func FailOn(match func(error) bool) workflow.Middleware
FailOn marks an error permanent when match reports true, so the step fails at once instead of spending its retries. An error already permanent stays so. A nil match is a no-op.
func FormatOutput ¶
FormatOutput writes out as text: one header line naming the run, step, attempt and outcome, one line per record, then the result when it was captured. It is the format WriterSink and FileSink use.
func Logging ¶
func Logging(l *slog.Logger, clk clock.Clock, opts ...LoggingOption) workflow.Middleware
Logging logs each step's start and end at Debug and a failure at Error (WithLogLevel changes those levels), with the input and output sizes and the duration read from clk (nil means clock.Real). When the driver set a workflow.StepInfo on ctx, each line also carries workflow, step, run_id and attempt. It never logs payload bytes or labels: either can hold personal data or secrets, and a codec may not have run yet. A nil logger makes the middleware a no-op.
func MaxPayload ¶
func MaxPayload(limit int) workflow.EnqueueMiddleware
MaxPayload refuses a job whose payload, or a chained job's, is over limit bytes, with a *workflow.PayloadTooLargeError (matches ErrPayloadTooLarge and ErrInvalid), before it reaches the driver: an app picks its own cap below the engine's hard one, so an oversized job fails at the caller rather than deep in a broker. Put it after a claim check, which shrinks big payloads first, so only what would really travel is measured. A limit of zero or less refuses every enqueue with ErrInvalid rather than meaning "no limit" by accident.
func OutputLogger ¶
OutputLogger returns a logger whose records CaptureOutput captures for the step running on ctx (and passes to the CaptureTee handler). Outside a capture it returns fallback, or slog.Default() when fallback is nil, so a step can always log through it.
func PayloadVersion ¶
func PayloadVersion(current int) workflow.EnqueueMiddleware
PayloadVersion stamps every job with the payload version this code writes (workflow.LabelPayloadVersion), so a step built with workflow.DefineVersioned can upcast a payload an older version of the app enqueued. A job that already carries a version (a retry, a job forwarded from elsewhere) keeps it; one carrying a version newer than current is refused with ErrInvalid, since this code could not read it. The labels map is copied, never changed in place.
func Recover ¶
func Recover(l *slog.Logger) workflow.Middleware
Recover turns a panic in a step into an error wrapping ErrPanic, so one bad step fails its attempt instead of killing the worker. The stack goes to l at Error (nil l drops it), never into the error string, which ends up in run status where it would leak file paths and be unreadable. The error is retryable: whether a panic is permanent is the retry policy's call (NoRetryOn: errors.Is(err, ErrPanic)), not this middleware's. The error implements workflow.StackTracer, so a driver with CapErrorHistory stores the stack in AttemptError.Stack, apart from the message.
func Release ¶
func Release(when func(ctx context.Context, in []byte) bool, delay time.Duration) workflow.Middleware
Release does not run the step when when reports true: the attempt ends with workflow.RetryAfter(ErrReleased, delay), so the driver retries the step after delay: a release decided before the step runs. To release unless a condition holds, negate the condition. A nil when is a no-op; a negative delay counts as zero.
Two rules come from the retry policy, not from this middleware, and hold on every driver:
- The delay is honoured only under a policy whose DelayFor reads it (workflow.RetryAfterDelay); under any other policy a released attempt waits the policy's normal Backoff.
- A release is an attempt: it counts against RetryPolicy.Max, and a step released on its last attempt fails its run with an error wrapping ErrReleased. A step that may be released often needs a Max to match.
To release from inside a handler, after it has started, return workflow.RetryAfter(err, d) there; this middleware is for a condition checked before the handler, across every workflow it wraps.
func ReleaseUniqueOnStart ¶
func ReleaseUniqueOnStart(store UniqueStore) workflow.Middleware
ReleaseUniqueOnStart is the step half of UniqueUntilStarted: before a step runs it frees the lock that Unique stamped on the run's labels (UniqueKeyLabel and UniqueOwnerLabel, read from the driver's workflow.StepInfo), then calls the step whatever the release returned. A run without those labels, or a ctx without a StepInfo, passes straight through.
It releases on every step attempt, not only the first: Release is a no-op once the lock is gone, and frees nothing a newer job holds because it names this run's owner token, so the repeats are safe and also retry a release that failed earlier. Each costs one store call. A failed release is not reported: the step has no reason to fail for it, and the ttl frees the key anyway.
func Report ¶
func Report(r workflow.Reporter, cfg ReportConfig) workflow.Plugin
Report bundles ReportFailures (and ReportAttempts when cfg.Attempts) as a plugin named "report" for workflow.Use.
func ReportAttempts ¶
func ReportAttempts(r workflow.Reporter, cfg ReportConfig) workflow.Middleware
ReportAttempts returns step middleware that hands every failed attempt to r as a Failure with Final false and Status StatusRetrying, then returns the step's result unchanged. A released step (ErrReleased) is not a failure and is not reported.
func ReportFailures ¶
func ReportFailures(r workflow.Reporter, cfg ReportConfig) workflow.RunFinishedFunc
ReportFailures returns a run-finished observer that hands every run that ends failed (and discarded, when cfg.Discarded) to r.
func RequireSource ¶ added in v0.1.1
func RequireSource(allowed ...workflow.Source) workflow.EnqueueMiddleware
RequireSource refuses, with an error wrapping workflow.ErrInvalid, a job whose trigger source cannot be told (workflow.SourceOf returns ""), so a run never lands in a runs view without saying who started it. Put it in the client's enqueue middleware; a job made with workflow.WithSource, or carrying a label the library stamps (a schedule fire, a spawned child, a batch member, an event), passes.
With allowed sources given, the job's source must also be one of them, so a typo ("webook") is refused at the enqueue site instead of creating a filter value nobody looks for. Each allowed source must pass workflow.ValidateSource, or every enqueue is refused with that error, so a bad allow-list fails on the first call rather than never matching.
It checks the job it is handed only: a chained job's source is its own (the driver enqueues it later, past the middleware).
func Route ¶
func Route(rules ...RouteRule) workflow.EnqueueMiddleware
Route applies the first rule whose Match accepts the job and passes every job on, matched or not: routing is an app-side table that keeps queue names and tenant labels out of the code that enqueues. The job's Labels map is copied before it is changed, so the caller's map is never mutated.
func Skip ¶
Skip does not run the step when skip reports true, and returns a nil output and a nil error: the step counts as done. Child steps then receive empty input, so a step whose children need its output should not be skipped. A nil skip is a no-op.
func Stuck ¶
func Stuck(cfg StuckConfig) workflow.Middleware
Stuck reports step attempts that run far longer than expected (a hung HTTP call, a lock never released), the way a dashboard flags a job stuck for an hour: detection only, the attempt is not touched (a Timeout or HeartbeatTimeout ends attempts). It works on every driver, since it wraps the step in the process running it. A config without OnStuck, or with neither After nor a positive Default, fails every step with ErrInvalid rather than watching nothing.
func ThrottleErrors ¶
func ThrottleErrors(cfg ThrottleConfig) (workflow.Middleware, error)
ThrottleErrors is a circuit breaker per key: after MaxErrors errors within Window, every call for that key returns ErrThrottled without running the step until Cooldown has passed. A success resets the count. The state lives in cfg.Store, by default a MemoryCircuit of the returned middleware's own, so it is per process; each worker process trips its own circuit, which is the point of a breaker that protects a downstream service from this process's calls. A shared Store makes it fleet-wide.
Errors marked Permanent and ErrThrottled itself are not counted: a permanent error is about the input, not the downstream service.
func Timeout ¶
func Timeout(d time.Duration) workflow.Middleware
Timeout gives each step attempt a context that ends after d. It is cooperative: the step must watch ctx, because abandoning a goroutine that keeps running would leak it and let two attempts of one step overlap. A d of zero or less is a no-op. Drivers already apply Step.Timeout; this is for a cap set by the app across every workflow.
func Unique ¶
func Unique(by func(workflow.Job) string, store UniqueStore, ttl time.Duration) workflow.EnqueueMiddleware
Unique refuses a job whose key, from by, is already held: the enqueue returns an error wrapping workflow.ErrDuplicate and nothing reaches the driver. A job for which by returns "" is not unique and passes through.
The lock lasts ttl from the enqueue. Use UniqueUntilFinished to also free it when the run ends, which is what an app usually wants; plain Unique suits a driver with no run-finished hook. If the driver refuses the enqueue, the lock is released at once so a retry can pass. A ttl of zero or less fails each enqueue with workflow.ErrInvalid rather than holding a lock forever.
func UniqueByArgs ¶
func UniqueByArgs(opts ...ArgsOption) func(workflow.Job) string
UniqueByArgs returns a key function for Unique, UniqueUntilFinished and UniqueUntilStarted that makes a job unique by its workflow and its arguments: prefix + workflow + ":" + the hex SHA-256 of the canonical payload (plus the queue and labels the options add).
A JSON payload is canonical after decoding and encoding again, so key order and whitespace do not matter; numbers keep their literal text (1 and 1.0 differ). A payload that is not JSON is hashed as its bytes. It hashes the payload the middleware sees, so place it before a codec or claim check whose output changes from call to call (encryption).
func UniqueUntilFinished ¶
func UniqueUntilFinished(c workflow.Client, by func(workflow.Job) string, store UniqueStore, ttl time.Duration) (workflow.EnqueueMiddleware, error)
UniqueUntilFinished is Unique whose lock is also freed when the run ends, whatever its status: once the work is done, failed or not, a new job for the key may be enqueued. c is the driver's client or any wrapper from this package (they forward workflow.RunNotifier), and this must be called before c.Start: the driver refuses a late observer with workflow.ErrAlreadyStarted, which is returned here.
ttl stays the safety net: without CapRunFinished the driver's delivery is best-effort, and a release can fail, so a lost event costs at most ttl of blocked enqueues, never a lock held forever. A client that does not implement workflow.RunNotifier gets an UnsupportedError naming CapRunFinished; use Unique there and accept the TTL alone.
func UniqueUntilStarted ¶
func UniqueUntilStarted(by func(workflow.Job) string, store UniqueStore, ttl time.Duration) (enqueue workflow.EnqueueMiddleware, step workflow.Middleware)
UniqueUntilStarted is Unique whose lock is also freed as soon as a step of the run starts, so a second job for the same key may be enqueued while the first one runs. It returns the two halves: enqueue goes in the enqueuing process's WrapClient chain, step in the worker's step middleware (memory.WithMiddleware, sqlstore.WithMiddleware, hatchet.Config.Middleware). A worker in another process from the enqueuer installs ReleaseUniqueOnStart(store) with the same store instead.
Without the step half on the worker, nothing frees the lock early and it lasts ttl, exactly like Unique: the ttl stays the safety net for a run that never starts, and for a release that fails. The store must be one every enqueuing and working process shares (a SQL store, not MemoryUnique) when they are separate processes.
func UniqueWith ¶
func UniqueWith(cfg UniqueConfig) (workflow.EnqueueMiddleware, error)
UniqueWith is Unique with a choice of which holder states count (States) and of answering a duplicate with the existing run (ReturnExisting). It refuses a config it cannot honour with ErrInvalid: a missing By, Store or TTL, States without a Client, or States or ReturnExisting on a store that does not implement UniqueHolders.
func WithCodec ¶
WithCodec returns a Client that seals every byte the app hands the driver: Job.Payload on Enqueue, Event.Payload on Signal (it becomes a wait step's output, which a child step decodes) and Schedule.Input on Schedule (each fired run's payload). The extra enqueue middleware, if any, runs after sealing, so it sees sealed payloads; put Unique or Route keyed on payload content in a WrapClient outside this one instead. Capabilities are c's own, and workflow.RunNotifier and the optional interfaces workflow.Forward covers are forwarded to c. Those pass bytes through as stored: a RunInfo.Payload from Runs is sealed, and opening it is the app's call (Codec.Decode).
func WrapClient ¶
WrapClient returns a Client whose Enqueue runs through mws before reaching c, so enqueue middleware works on any driver without the driver knowing. Every other method, Capabilities included, is c's own: the wrapper never claims or hides a capability. It also implements workflow.RunNotifier by delegating to c (see forwardRunFinished), so stacking wrappers never hides the run-finished hook from code such as UniqueUntilFinished. The optional interfaces of the run query, pause, schedule and drain families (workflow.RunQuerier, Pauser, ScheduleLister, TickNotifier, Drainer and the rest) are forwarded to c through workflow.Forward, and refused with an UnsupportedError when c lacks one, so a wrapper never hides them either.
Types ¶
type Alert ¶
type Alert struct {
workflow.Failure
// Key is the dedup key (AlertConfig.Key).
Key string
// Suppressed counts the alerts for Key the limiter held back since
// the previous one sent, so a sink can say "and 12 more".
Suppressed int
}
Alert is one alert: the failure, the dedup key it was counted under and how many alerts for that key the limiter held back since the last one sent.
type AlertConfig ¶
type AlertConfig struct {
// Discarded also alerts on runs a step discarded.
Discarded bool
// When, if set, filters failures: only those it returns true for
// alert.
When func(workflow.Failure) bool
// Key is the dedup key of a failure. Nil means workflow + "/" + step.
Key func(workflow.Failure) string
// Limiter decides dedup and rate limits. Nil means a
// MemoryAlertLimiter with Dedup DefaultAlertDedup on Clock; pass one
// built with AlertLimits{} to send every alert.
Limiter AlertLimiter
// Clock stamps Failure.At and drives the default limiter. Nil means
// clock.Real.
Clock clock.Clock
// Logger receives errors from the Alerter and the Limiter. Nil
// discards them.
Logger *slog.Logger
}
AlertConfig configures Alerts. The zero value alerts on every final failure, deduplicated per workflow and step for DefaultAlertDedup.
type AlertLimiter ¶
type AlertLimiter interface {
Allow(ctx context.Context, key string, at time.Time) (ok bool, suppressed int, err error)
}
AlertLimiter decides whether an alert for key goes out at at. It reports how many alerts for key it held back since the last one it let through (reset when it lets one through). A shared implementation (a database row per key) makes the limit fleet-wide; MemoryAlertLimiter holds only inside one process.
type AlertLimits ¶
type AlertLimits struct {
// Dedup holds back an alert for a key that alerted less than Dedup
// ago. Zero turns dedup off.
Dedup time.Duration
// Max and Per cap alerts across all keys at Max per sliding Per
// window. Either zero turns the cap off.
Max int
Per time.Duration
}
AlertLimits are MemoryAlertLimiter's rules. The zero value lets every alert through.
type Alerter ¶
Alerter sends an alert where a human sees it: Slack, email, a pager. The sink is the app's; this package only decides when to call it. It is called on a driver goroutine and should not block for long.
type AlerterFunc ¶
AlerterFunc adapts a function to Alerter.
type ArgsOption ¶
type ArgsOption func(*argsKey)
ArgsOption tunes UniqueByArgs.
func ArgsFields ¶
func ArgsFields(names ...string) ArgsOption
ArgsFields makes only these top-level fields of a JSON object payload count: two jobs whose payloads differ only in other fields share a key. A field missing from the payload counts as absent, which differs from one set to null. A payload that is not a JSON object cannot be cut to fields, so its whole bytes count instead: the key gets narrower, never wider, so no job is ever dropped as a duplicate of a different one.
func ArgsKeyPrefix ¶
func ArgsKeyPrefix(p string) ArgsOption
ArgsKeyPrefix replaces DefaultArgsKeyPrefix.
func ArgsLabels ¶
func ArgsLabels(keys ...string) ArgsOption
ArgsLabels makes the values of these Job.Labels keys count (a missing label counts as absent).
func ArgsQueue ¶
func ArgsQueue() ArgsOption
ArgsQueue makes Job.Queue count, so the same arguments on two queues are two unique jobs.
func ArgsTagged ¶
func ArgsTagged[T any]() ArgsOption
ArgsTagged makes the fields of T tagged `unique:"true"` the ones that count (ArgsFields with their JSON names), so the uniqueness rule lives on the payload type next to the fields it names:
type SyncArgs struct {
ShopID string `json:"shop_id" unique:"true"`
Cursor string `json:"cursor"`
}
by := middleware.UniqueByArgs(middleware.ArgsTagged[SyncArgs]())
The JSON name comes from the json tag (its first part), else the Go field name, as encoding/json does. A T that is not a struct, or has no tagged field, makes the whole payload count, the same as no option.
type BlobPrefixDeleter ¶
type BlobPrefixDeleter interface {
// DeletePrefix removes every blob whose key starts with prefix.
DeletePrefix(ctx context.Context, prefix string) error
}
BlobPrefixDeleter is an optional upgrade of BlobStore: a store that can delete every key under a prefix (an S3 list-and-delete, a directory removal). With it, CleanupOnFinish removes a run's step outputs too; without it, only the run's payload is removed and step outputs are left to the store's own expiry. A store that cannot do it is never forced to pretend it can.
type BlobStore ¶
type BlobStore interface {
// Put stores data under key, replacing any blob already there.
Put(ctx context.Context, key string, data []byte) error
// Get returns the blob under key, or an error wrapping ErrBlobNotFound.
Get(ctx context.Context, key string) ([]byte, error)
// Delete removes the blob under key. Deleting a missing key is a
// no-op, not an error, because run-finished events arrive at least
// once.
Delete(ctx context.Context, key string) error
}
BlobStore holds the payloads a ClaimCheck moves out of the queue. It is the seam for S3, a filesystem or a database; adapters for those live in their own modules, MemoryBlobStore is the in-process one for tests.
A store shared by every process that enqueues or runs steps is what makes the claim check work across processes: the step that resolves a reference may run on another machine than the enqueue that stored it.
type CaptureOption ¶
type CaptureOption func(*captureConfig)
CaptureOption configures CaptureOutput.
func CaptureClock ¶
func CaptureClock(clk clock.Clock) CaptureOption
CaptureClock sets the clock Started, Finished and record times are read from; the default is the real clock. Nil is ignored.
func CaptureLevel ¶
func CaptureLevel(l slog.Leveler) CaptureOption
CaptureLevel keeps records at or above l; the default is slog.LevelInfo. Nil is ignored.
func CaptureOnSinkError ¶
func CaptureOnSinkError(fn func(ctx context.Context, out Output, err error)) CaptureOption
CaptureOnSinkError receives the sink's errors; the default drops them, since a sink failure must never fail the step. Nil is ignored.
func CaptureOnly ¶
func CaptureOnly(keep func(workflow.StepInfo) bool) CaptureOption
CaptureOnly captures only the attempts keep returns true for, such as one workflow's steps; others run untouched. An attempt without a StepInfo is passed the zero StepInfo. Nil is ignored.
func CaptureOnlyFailures ¶
func CaptureOnlyFailures() CaptureOption
CaptureOnlyFailures delivers only attempts that returned an error.
func CaptureResult ¶
func CaptureResult() CaptureOption
CaptureResult includes the step's output bytes in Output.Result. Off by default: a result can hold payload data.
func CaptureTee ¶
func CaptureTee(h slog.Handler) CaptureOption
CaptureTee also passes every record OutputLogger receives to h (the app's normal log handler), whatever the capture level. Nil is ignored.
type CircuitPolicy ¶
CircuitPolicy is the part of ThrottleConfig a CircuitStore applies when it counts an error.
type CircuitStore ¶
type CircuitStore interface {
// IsOpen reports whether key's circuit is open now.
IsOpen(ctx context.Context, key string) (bool, error)
// RecordError counts one error for key and opens its circuit for
// p.Cooldown once p.MaxErrors errors fall inside p.Window, counted
// from the first of them.
RecordError(ctx context.Context, key string, p CircuitPolicy) error
// Reset closes key's circuit and clears its count, after a success.
Reset(ctx context.Context, key string) error
}
CircuitStore holds ThrottleErrors' circuits, one per key. The store owns the counting, so a shared store can count every worker's errors in one atomic step (an increment, not a read and a write) and open one circuit for the whole fleet.
An IsOpen error fails closed: the step is not run and the call returns an error wrapping ErrThrottled and the store's error, retryable like any throttled call. A RecordError or Reset error never turns a step's result around: it is joined onto a failed step's error and dropped after a successful one, where the worst it can do is let the circuit open early.
type ClaimCheck ¶
type ClaimCheck struct {
// contains filtered or unexported fields
}
ClaimCheck stores payloads larger than a threshold outside the queue and puts a small reference in their place (the claim check pattern), so a broker or engine with a payload cap (SQS, a Hatchet row) carries only the reference. Build it with NewClaimCheck, then wire all three halves:
- Enqueue, an enqueue middleware: stamps ClaimCheckLabel on every job and moves a Job.Payload above the threshold into the store;
- Step, a step middleware: resolves references in a step's input before the handler runs, and moves an output above the threshold into the store so the child step's input is a reference too;
- CleanupOnFinish: deletes the run's blobs when the run finishes.
Order with a Codec: claim check runs after encoding on the way in and before decoding on the way out, so it only ever sees sealed bytes and the store never holds plaintext. Concretely, pass Enqueue as an extra middleware of WithCodec (WithCodec(c, codec, cc.Enqueue()) seals first, then claims), and put Step outside DecodeStep (Chain(cc.Step(), DecodeStep(codec)): references are resolved before opening, and outputs are sealed before they are claimed).
Not covered: Signal payloads and Schedule inputs stay inline (they are made outside any run, so there is no token to scope them by), and runs fired by a Schedule or a declared Cron carry no token, so their payload and outputs stay inline whatever their size.
func NewClaimCheck ¶
func NewClaimCheck(store BlobStore, threshold int, opts ...ClaimCheckOption) (*ClaimCheck, error)
NewClaimCheck returns a ClaimCheck storing in store any payload or step output longer than threshold bytes; one of exactly threshold bytes stays inline. A nil store or a threshold of zero or less is workflow.ErrInvalid.
func (*ClaimCheck) CleanupOnFinish ¶
func (cc *ClaimCheck) CleanupOnFinish(c workflow.Client) error
CleanupOnFinish deletes a run's blobs when the run reaches a terminal status, never before, so every retry still finds its payload. With a BlobPrefixDeleter store it deletes everything under the run's token; otherwise it deletes the payload and leaves step outputs to the store's expiry. c is the driver's client or any wrapper from this package (they forward workflow.RunNotifier), and this must be called before c.Start.
A client that does not implement workflow.RunNotifier gets an UnsupportedError naming CapRunFinished: there, and for every event a best-effort driver loses, the store's own expiry is the only cleanup, so set one longer than the longest run. A run replayed by hand on an engine after it finished finds its blobs gone. Failed deletes are not reported (the observer has no caller); expiry catches them too.
func (*ClaimCheck) Enqueue ¶
func (cc *ClaimCheck) Enqueue() workflow.EnqueueMiddleware
Enqueue returns the enqueue middleware. It stamps a fresh token in ClaimCheckLabel on every job, large or not, so the run's step outputs can be claimed too; a Payload above the threshold is stored under "<token>/payload" and replaced by its reference.
When the driver refuses the enqueue, the blob is kept: an error does not prove the run was not created (a timeout after the engine stored it), and deleting the payload of a run that exists would fail it. A blob whose run never existed, or a job deduplicated by its IdempotencyKey onto an older run, is an orphan the store's own expiry must remove.
func (*ClaimCheck) Step ¶
func (cc *ClaimCheck) Step() workflow.Middleware
Step returns the step middleware. Before the handler runs it resolves every reference in the input that belongs to the run (see claimRefPrefix): the payload, a single parent's output, or each parent's value in the several-parents object, which it rebuilds with the workflow.StepInput embedding rule so the handler sees exactly the bytes it would see without a claim check. Anything else, a lookalike included, passes through unchanged. After the handler, an output above the threshold is stored and replaced by its reference. Jobs the step adds to its chain (workflow.PrependToChain, AppendToChain) are prepared as Enqueue prepares a job: their own token, and a claimed payload when big.
The blobs stay until the run finishes, so a retried attempt resolves the same references again. A missing blob is a Permanent error wrapping ErrBlobNotFound; any other store error, on Get or Put, fails the attempt and leaves the retry to the policy. A step outside a run stamped by Enqueue (no workflow.StepInfo, or no ClaimCheckLabel) runs untouched.
type ClaimCheckOption ¶
type ClaimCheckOption func(*ClaimCheck)
ClaimCheckOption configures a ClaimCheck.
func WithClaimKeys ¶
func WithClaimKeys(fn func() (string, error)) ClaimCheckOption
WithClaimKeys replaces the random key source. fn returns the run token and the per-output key segment, so it must return values that are unique and unguessable: a token another run can predict lets a forged reference in that run's payload reach its blobs. A value that is empty or holds "/" fails the enqueue or step with workflow.ErrInvalid, because "/" separates a token from its keys and would let one run's prefix cover another's. A nil fn keeps the default, 128 random bits in hex.
type Codec ¶
type Codec interface {
Encode(plain []byte) ([]byte, error)
Decode(encoded []byte) ([]byte, error)
}
Codec turns plain bytes into the bytes an engine stores, and back. It is how payload encryption and compression plug in without a driver knowing. Decode(Encode(b)) must equal b. A Codec works on raw bytes; the middleware in this file wraps its output in a sealed envelope (see envelopePrefix) so the result survives every place a driver moves it.
func Encrypt ¶
Encrypt returns an AES-GCM Codec keyed by key, which must be 16, 24 or 32 bytes (AES-128, -192, -256). Each Encode draws a fresh random nonce and writes nonce || ciphertext || tag, so equal payloads never encrypt to equal bytes. Decode authenticates before returning anything, so a wrong key or a changed byte is an error, never garbage. The key belongs to the app.
oldKeys rotate it: Encode always uses key, while Decode tries key first and then each old key in order, so payloads sealed before the rotation still open. The sealed format is the same for every key (no key ID is written), so rotating needs no change to stored runs; drop an old key once no run sealed with it can still be read. Every old key must be a valid AES key too.
codec, err := middleware.Encrypt(newKey, previousKey)
type DebounceBurst ¶
type DebounceBurst struct {
// First is when the burst's first job was enqueued; maxWait counts
// from it.
First time.Time
// RunAt is when the pending run starts.
RunAt time.Time
// Ref is the pending run, which the next job of the burst cancels.
Ref workflow.RunRef
}
DebounceBurst is one key's pending run, as a DebounceStore keeps it. Invariant: RunAt is the RunAt the pending run was enqueued with, and First <= RunAt <= First+maxWait. A burst whose RunAt has passed is over: Debounce treats it as absent, and a store may drop it at any time.
type DebounceOption ¶
type DebounceOption func(*debouncer)
DebounceOption configures Debounce.
func WithDebounceStore ¶
func WithDebounceStore(s DebounceStore) DebounceOption
WithDebounceStore keeps the bursts in s instead of a MemoryDebounce of the debouncer's own, for example a store shared by every process that enqueues the key. A nil s keeps the default.
type DebounceStore ¶
type DebounceStore interface {
// Get returns key's burst, or found false when there is none.
Get(ctx context.Context, key string) (b DebounceBurst, found bool, err error)
// Swap stores b as key's burst and returns the burst it replaced, or
// found false when there was none.
Swap(ctx context.Context, key string, b DebounceBurst) (prev DebounceBurst, found bool, err error)
}
DebounceStore holds the pending burst of each debounce key. It is where the debounce state lives, so its reach is Debounce's: MemoryDebounce debounces inside one process; a store shared by several processes (a SQL table, Redis) debounces a key enqueued from all of them.
Swap must be atomic: replacing a key's burst and returning the one it replaced is one step, so when two processes debounce the same key at once each gets back a different previous run and every replaced run is cancelled by exactly one of them. Get is only a hint for First and may race; Swap is what decides which run is cancelled.
An error from either method fails the enqueue closed: on Get the job is not enqueued; on Swap the job's new run is cancelled and the error returned, so the store and the engine never disagree about which run is pending.
type LoggingOption ¶
type LoggingOption func(*loggingConfig)
LoggingOption configures Logging.
func WithLogLevel ¶
func WithLogLevel(fn func(ctx context.Context, ev StepLogEvent) slog.Level) LoggingOption
WithLogLevel picks the level of each line Logging writes. The driver's retry policy is not visible to middleware, so a function that logs retried attempts at Warn and only the last one at Error compares ev.Info.Attempt with the attempts the app configured:
middleware.Logging(l, nil, middleware.WithLogLevel(func(_ context.Context, ev middleware.StepLogEvent) slog.Level {
switch {
case ev.Phase != middleware.StepFailed:
return slog.LevelDebug
case workflow.IsPermanent(ev.Err) || ev.Info.Attempt >= maxAttempts:
return slog.LevelError
default:
return slog.LevelWarn
}
}))
A nil fn keeps the default.
type MemoryAlertLimiter ¶
type MemoryAlertLimiter struct {
// contains filtered or unexported fields
}
MemoryAlertLimiter is the in-process AlertLimiter. Its state lives in one process: each replica of a fleet dedups on its own.
func NewMemoryAlertLimiter ¶
func NewMemoryAlertLimiter(clk clock.Clock, l AlertLimits) *MemoryAlertLimiter
NewMemoryAlertLimiter returns a limiter with rules l. clk prunes state; nil means clock.Real.
type MemoryBlobStore ¶
type MemoryBlobStore struct {
// contains filtered or unexported fields
}
MemoryBlobStore is a BlobStore inside one process, for tests and a single server where every step runs in the enqueuing process. It has no expiry. Use NewMemoryBlobStore.
func NewMemoryBlobStore ¶
func NewMemoryBlobStore() *MemoryBlobStore
NewMemoryBlobStore returns an empty store.
func (*MemoryBlobStore) Delete ¶
func (m *MemoryBlobStore) Delete(_ context.Context, key string) error
Delete implements BlobStore.
func (*MemoryBlobStore) DeletePrefix ¶
func (m *MemoryBlobStore) DeletePrefix(_ context.Context, prefix string) error
DeletePrefix implements BlobPrefixDeleter.
func (*MemoryBlobStore) Len ¶
func (m *MemoryBlobStore) Len() int
Len reports how many blobs the store holds, so a test can prove that a payload was moved out and that cleanup removed it.
type MemoryCircuit ¶
type MemoryCircuit struct {
// contains filtered or unexported fields
}
MemoryCircuit is a CircuitStore inside one process, the default of ThrottleErrors. Use NewMemoryCircuit.
func NewMemoryCircuit ¶
func NewMemoryCircuit(clk clock.Clock) *MemoryCircuit
NewMemoryCircuit returns an empty store reading time from clk (nil means clock.Real).
func (*MemoryCircuit) RecordError ¶
func (m *MemoryCircuit) RecordError(_ context.Context, key string, p CircuitPolicy) error
RecordError implements CircuitStore.
type MemoryDebounce ¶
type MemoryDebounce struct {
// contains filtered or unexported fields
}
MemoryDebounce is a DebounceStore inside one process, the default of Debounce. Bursts whose run has started are dropped on the next Swap. Use NewMemoryDebounce.
func NewMemoryDebounce ¶
func NewMemoryDebounce(clk clock.Clock) *MemoryDebounce
NewMemoryDebounce returns an empty store reading time from clk (nil means clock.Real) to drop finished bursts.
func (*MemoryDebounce) Get ¶
func (m *MemoryDebounce) Get(_ context.Context, key string) (DebounceBurst, bool, error)
Get implements DebounceStore.
func (*MemoryDebounce) Swap ¶
func (m *MemoryDebounce) Swap(_ context.Context, key string, b DebounceBurst) (DebounceBurst, bool, error)
Swap implements DebounceStore.
type MemoryUnique ¶
type MemoryUnique struct {
// contains filtered or unexported fields
}
MemoryUnique is a UniqueStore inside one process, for tests, a single server, or a cap that only needs to hold per process. Expired keys are dropped when they are next touched. Use NewMemoryUnique.
func NewMemoryUnique ¶
func NewMemoryUnique(clk clock.Clock) *MemoryUnique
NewMemoryUnique returns an empty store reading time from clk (nil means clock.Real).
func (*MemoryUnique) Acquire ¶
func (m *MemoryUnique) Acquire(_ context.Context, key, owner string, ttl time.Duration) (bool, error)
Acquire implements UniqueStore.
type Output ¶
type Output struct {
// Info is the attempt's StepInfo; HasInfo is false when the ctx
// carried none (a handler called outside a driver).
Info workflow.StepInfo
HasInfo bool
// Started and Finished bound the attempt, on the middleware's clock.
Started, Finished time.Time
// Records are the log records the step wrote through OutputLogger, in
// order, at or above the capture level.
Records []OutputRecord
// Result is the step's output bytes. It is nil unless CaptureResult
// was given, because a result may hold payload data an app wants kept
// out of logs, files and mail.
Result []byte
// Err is the attempt's error; nil when it succeeded.
Err error
}
Output is what CaptureOutput delivers for one step attempt: the log records the step wrote through OutputLogger, how it ended, and its result bytes when CaptureResult was given. For a scheduled task or a Func (one step) one Output is one run attempt.
type OutputRecord ¶
OutputRecord is one captured log record.
type OutputSink ¶
OutputSink receives every captured Output. Deliver runs on the step's goroutine after the attempt returned and before the driver records the result, so a slow sink holds the worker; a sink that sends mail or calls a network service should hand off to its own goroutine or queue. Its error never changes the step's result; it goes to CaptureOnSinkError.
func FileSink ¶
func FileSink(path string, mode FileMode) OutputSink
FileSink writes each Output to the file at path with FormatOutput, creating it when missing (mode 0600). Deliveries from one sink are serialised; two sinks on one path are not.
func WriterSink ¶
func WriterSink(w io.Writer) OutputSink
WriterSink writes each Output to w with FormatOutput, one delivery at a time.
type OutputSinkFunc ¶
OutputSinkFunc adapts a function to OutputSink: the way to mail output, post it to a chat or store it anywhere the library does not know.
type PingFunc ¶
PingFunc sends one ping to target, a URL. It is the seam for any heartbeat service: HTTPPing is the stock one, and a PingFunc can POST a body, add an auth header or call a client library instead.
func HTTPPing ¶
HTTPPing returns a PingFunc that sends GET target with client (nil means http.DefaultClient; the Pinger's timeout applies either way) and fails on a status outside 2xx. Its errors never contain the URL, because a heartbeat URL usually carries the check's secret token and errors end up in logs.
type Pinger ¶
type Pinger struct {
// contains filtered or unexported fields
}
Pinger turns heartbeat URLs into schedule hooks: a ping when a tick fires, after a run, on success or on failure. Its methods return funcs with the shapes of workflow.ScheduleHooks' fields, so a hook is one assignment:
p := middleware.NewPinger(nil)
workflow.ScheduleHooks{
Workflow: "backup",
Fired: p.Tick("https://hc.example/abc/start"), // a run started
OnSuccess: p.Run("https://hc.example/abc"), // it succeeded
OnFailure: p.Run("https://hc.example/abc/fail"), // it failed
}
After takes p.Run(url) for a ping whatever the outcome. A ping is best effort: a failure is logged at Warn (without the URL) and never fails the run or the tick. A Pinger holds no state beyond its options and is safe for concurrent use.
func NewPinger ¶
func NewPinger(send PingFunc, opts ...PingerOption) *Pinger
NewPinger returns a Pinger that sends through send (nil means HTTPPing(nil)) and logs failures to slog.Default() unless WithPingLogger says otherwise.
func (*Pinger) Run ¶
Run returns a run hook that pings target for every run event it is given: for ScheduleHooks.After, OnSuccess or OnFailure, which already choose the outcome.
type PingerOption ¶
type PingerOption func(*Pinger)
PingerOption configures a Pinger.
func WithPingLogger ¶
func WithPingLogger(l *slog.Logger) PingerOption
WithPingLogger sets where failed pings are logged; nil drops them.
func WithPingTimeout ¶
func WithPingTimeout(d time.Duration) PingerOption
WithPingTimeout bounds each ping (DefaultPingTimeout when not given). A d of zero or less keeps the default, so a ping can never block the hook goroutine forever.
type ReportConfig ¶
type ReportConfig struct {
// Attempts also reports every failed step attempt (Failure.Final
// false), from step middleware on the worker that ran it. The last
// attempt of a failing run is then reported twice: once as an
// attempt, once as the run's end.
Attempts bool
// Discarded also reports runs a step discarded.
Discarded bool
// Clock stamps Failure.At. Nil means clock.Real.
Clock clock.Clock
}
ReportConfig configures Report. The zero value reports final failures only.
type RouteRule ¶
type RouteRule struct {
// Match selects the jobs this rule routes. Nil matches every job.
Match func(workflow.Job) bool
// Workflow, when set, replaces Job.Workflow.
Workflow string
// Labels are added to Job.Labels. A label the job already carries
// wins: a choice made at the enqueue site beats the routing table.
Labels map[string]string
}
RouteRule maps the jobs Match accepts to a workflow name and labels.
type StepLogEvent ¶
StepLogEvent is what a LogLevel function sees for each line Logging writes. Err is set only for StepFailed. Info is the driver's StepInfo and HasInfo is false when ctx carried none.
type StuckConfig ¶
type StuckConfig struct {
// After is how long an attempt of a step may run before it counts as
// stuck; zero or less means never. Nil means Default for every step.
After func(workflow.StepInfo) time.Duration
// Default is the threshold when After is nil.
Default time.Duration
// OnStuck is told once per attempt, on its own goroutine, while the
// attempt keeps running. Required.
OnStuck func(ctx context.Context, ev StuckEvent)
// Clock times the attempts; nil means clock.Real.
Clock clock.Clock
}
StuckConfig configures Stuck.
type StuckEvent ¶
type StuckEvent struct {
// Info is the attempt (workflow, step, run, attempt, node).
Info workflow.StepInfo
// Started is when the attempt started, on the middleware's clock.
Started time.Time
// Running is how long it had run when reported.
Running time.Duration
}
StuckEvent reports a step attempt that has run longer than expected.
type ThrottleConfig ¶
type ThrottleConfig struct {
// Clock reads time for the window and the cooldown of the default
// store. Nil means clock.Real. A custom Store keeps its own time.
Clock clock.Clock
// Store holds each key's circuit. Nil means a MemoryCircuit of the
// middleware's own, so each process trips its own circuit; a store
// shared by the fleet (a SQL table, Redis) gives one circuit per key
// across every worker, for a downstream with a fleet-wide error
// budget.
Store CircuitStore
// Key picks the circuit a call belongs to, such as the downstream API
// the step calls. Nil keys by the workflow name from the
// workflow.StepInfo on ctx, so one failing workflow does not throttle
// the others; a ctx without StepInfo falls into the "" circuit.
Key func(ctx context.Context, in []byte) string
// MaxErrors is how many errors inside Window open the circuit; at
// least 1.
MaxErrors int
// Window is how long errors are counted from the first one; positive.
Window time.Duration
// Cooldown is how long an open circuit fails fast; positive.
Cooldown time.Duration
}
ThrottleConfig configures ThrottleErrors.
type UniqueConfig ¶
type UniqueConfig struct {
// By derives the key; "" passes the job through. Required.
By func(workflow.Job) string
// Store holds the locks. Required. States and ReturnExisting need it
// to implement UniqueHolders.
Store UniqueStore
// TTL bounds every lock, whatever the holder's state. Required.
TTL time.Duration
// States are the holder statuses that make a new job a duplicate. A
// held key whose run is in none of them (or is gone) is taken over by
// the new job: States{StatusScheduled, StatusQueued} lets a job in
// once the holder started; adding StatusSucceeded keeps a key closed
// after success until TTL. Empty means every status counts (plain
// Unique). Needs Client to look the holder up.
States []workflow.Status
// Client answers Status for the holder's run; required with States.
Client workflow.Client
// ReturnExisting answers a duplicate with the holder's RunRef and a
// nil error, marking the EnqueueOutcome a duplicate
// (workflow.MarkDuplicate), instead of an error wrapping ErrDuplicate.
// When the holder's run is not known yet (a race with its enqueue),
// it still returns ErrDuplicate.
ReturnExisting bool
}
UniqueConfig configures UniqueWith.
type UniqueHolders ¶
type UniqueHolders interface {
// SetHolder records ref as the run of key while owner holds it; a key
// held by another owner, or free, is left alone.
SetHolder(ctx context.Context, key, owner string, ref workflow.RunRef) error
// Holder reports the owner and run of a held, unexpired key whose run
// was recorded; ok is false otherwise.
Holder(ctx context.Context, key string) (owner string, ref workflow.RunRef, ok bool, err error)
}
UniqueHolders is an optional interface of a UniqueStore that also remembers which run holds each key, so UniqueWith can look the holder up: to decide by the holder's status (UniqueConfig.States) or to return it (UniqueConfig.ReturnExisting). MemoryUnique implements it, and so should a shared store (sqlstore.UniqueStore).
type UniqueStore ¶
type UniqueStore interface {
// Acquire takes key for owner for ttl and reports true, or reports
// false when another holder has it and its ttl has not run out.
Acquire(ctx context.Context, key, owner string, ttl time.Duration) (bool, error)
// Release frees key if owner holds it. Releasing a free key, or one
// held by another owner, is a no-op, not an error.
Release(ctx context.Context, key, owner string) error
}
UniqueStore holds uniqueness locks. Acquire must be atomic: checking that key is free and taking it is one step, or two enqueues at once both pass (the first of the two caveats in the spec). The guarantee Unique gives is exactly the store's: a SQL store with a unique index on the key makes it hold across every process sharing the database; MemoryUnique holds only inside one process.
Every lock has an owner, a token Unique draws per enqueue, and Release frees the key only while that owner still holds it. Run-finished events arrive at least once, so a late duplicate for an old run must not free the lock a newer job has taken since.