Documentation
¶
Overview ¶
Package failover joins two workflow clients into one: enqueues go to a primary backend, and when the primary is unavailable they go to a secondary, so a job is still accepted while (say) the Hatchet engine is down and a local SQL store takes the work.
c := failover.New(hatchetClient, sqlClient)
c.Register(wf) // registers on both
c.Start(ctx) // starts both
ref, err := c.Enqueue(ctx, workflow.Job{Workflow: "email", IdempotencyKey: "order-42"})
st, err := c.Status(ctx, ref) // asks whichever backend holds the run
What fails over ¶
Only Enqueue fails over, and only when the Classifier says the primary's error means "backend unavailable". DefaultClassifier never fails over on a cancelled or expired caller context, nor on an error the secondary would return just the same (ErrInvalid, ErrUnsupported, ErrNotFound, ErrNotStarted, ErrAlreadyStarted, ErrDuplicate, ErrCancelled, a Permanent error); an error a driver marked with workflow.Unavailable, and everything else, counts as unavailable. The Unavailable mark wins over those sentinels: a driver's own timeout talking to its engine wraps context.DeadlineExceeded, and marked Unavailable it fails over, while the caller's own cancelled or expired context is checked before any Classifier runs and never fails over. StrictClassifier fails over only on errors marked with workflow.Unavailable, for deployments whose drivers all mark their outages; a deployment that can tell another way (a gRPC Unavailable code, a refused connection) passes its own Classifier through WithClassifier.
Running a job twice ¶
A failure can be ambiguous: the primary may have accepted the job and then failed to answer (a timeout after the write). Failing over then runs the job on both backends. So a Job without an IdempotencyKey is refused with ErrKeyRequired (which also matches workflow.ErrInvalid), unless the client was built with AllowWithoutKey. A key dedups only within one backend: the primary and the secondary do not see each other's keys, so the key protects a caller's retries against the same backend, and the handlers must still be idempotent across backends. A workflow with IdempotencyOff ignores keys altogether.
Run references ¶
Every RunRef this client returns, and every RunEvent and StepInfo it passes on, names its backend: RunID and ScheduledID carry the prefix "primary:" or "secondary:" in front of the driver's own ID, which is kept unchanged after it, so Status, Cancel and the run-finished hook reach the backend holding the run. A ref without a prefix (one a driver handed out directly) is tried on the primary, then on the secondary when the primary does not know it or is unavailable. A driver's middleware sees the driver's own RunID in StepInfo; the handler sees the prefixed one.
Capabilities ¶
Capabilities is the intersection of both backends', so Start refuses a workflow the secondary could not run rather than discovering it during an outage. The exceptions: the schedule family follows where schedules live (see below); CapRunFinished needs both backends to implement workflow.RunNotifier; and CapClosure and CapWorkerScale are the primary's alone. EnqueueClosure goes to the primary with no failover, because a closure has no idempotency key and an ambiguous failure retried on the secondary could run it twice; QueueLoads and SetWorkers scale the primary's workers (balance the secondary with a loop of its own). CapRunStates and CapChain are the intersection like the rest. A chained job (Job.Chain) is enqueued by the backend whose run finished it, never through this client, so it does not fail over: a chain stays on the backend that took its first job.
Managing runs ¶
The client implements every optional run interface, each needing its capability on both backends and refused with an UnsupportedError otherwise, never answered from one backend alone. Runs merges both backends' feeds newest first with prefixed refs and a cursor that records each feed's position; Retry and Delete go to the backend the ref names; CancelAll, Pause and Resume go to both (Paused lists the union); Drain drains both.
Schedules ¶
Workflow.Cron and Schedule live on the primary only, because a schedule on both backends fires on both: every tick would run twice. The cost is that no schedule fires while the primary is down. WithSchedulesOnBoth puts them on both for apps whose scheduled handlers are idempotent per tick. The schedule capabilities (CapCron, CapScheduleWhen, CapScheduleWindow, CapSchedulePause, CapScheduleList, CapTickEvents) are the primary's alone by default and need both under WithSchedulesOnBoth; the secondary's copy of a workflow then carries no Cron, CronWindows or ScheduleWhen. Ticks, listed schedules and the LabelSchedule of fired runs carry the backend prefix; PauseSchedule takes those IDs or the one Schedule returned.
Signals and run ends ¶
Signal goes to both backends, since a waiting run may sit on either; when one of them fails the error says which, and the other has already taken the event, so a retried Signal may reach that one twice. OnRunFinished registers the observer on both backends and needs both to implement workflow.RunNotifier; CapRunFinished holds only when both guarantee delivery.
Cool-off ¶
WithCoolOff stops trying the primary for a while after N consecutive failovers, so an outage does not cost every enqueue a timeout first. It reads time from WithClock (clock.Real by default).
Observing ¶
WithObserver reports each failover, each enqueue sent straight to the secondary during a cool-off, and the start and end of each cool-off, so an app can log, count or alert on them; by default nothing is reported.
Index ¶
- Variables
- func DefaultClassifier(err error) bool
- func StrictClassifier(err error) bool
- type Classifier
- type Client
- func (f *Client) Cancel(ctx context.Context, ref workflow.RunRef) error
- func (f *Client) CancelAll(ctx context.Context, q workflow.RunQuery) (int, error)
- func (f *Client) CancelStep(ctx context.Context, ref workflow.RunRef, step string) error
- func (f *Client) Capabilities() workflow.Caps
- func (f *Client) Delete(ctx context.Context, ref workflow.RunRef) error
- func (f *Client) Done() <-chan struct{}
- func (f *Client) Drain(ctx context.Context) error
- func (f *Client) DrainWithReport(ctx context.Context) (workflow.DrainReport, error)
- func (f *Client) Enqueue(ctx context.Context, job workflow.Job) (workflow.RunRef, error)
- func (f *Client) EnqueueClosure(ctx context.Context, c workflow.ClosureJob) (workflow.RunRef, error)
- func (f *Client) ForceComplete(ctx context.Context, ref workflow.RunRef) error
- func (f *Client) ForceFail(ctx context.Context, ref workflow.RunRef, reason string) error
- func (f *Client) Health(ctx context.Context) workflow.HealthReport
- func (f *Client) Nodes(ctx context.Context) ([]workflow.NodeInfo, error)
- func (f *Client) OnLifecycle(fn workflow.LifecycleFunc) error
- func (f *Client) OnRunFinished(fn workflow.RunFinishedFunc) error
- func (f *Client) OnTick(fn workflow.TickObserver) error
- func (f *Client) Pause(ctx context.Context, t workflow.PauseTarget) error
- func (f *Client) PauseSchedule(ctx context.Context, t workflow.ScheduleTarget) error
- func (f *Client) Paused(ctx context.Context) ([]workflow.PauseTarget, error)
- func (f *Client) PayloadLimit() int
- func (f *Client) QueueLoads(ctx context.Context) ([]workflow.QueueLoad, error)
- func (f *Client) Register(wf workflow.Workflow) error
- func (f *Client) Reschedule(ctx context.Context, ref workflow.RunRef, at time.Time) (workflow.RunRef, error)
- func (f *Client) Result(ctx context.Context, ref workflow.RunRef) ([]byte, error)
- func (f *Client) Resume(ctx context.Context, t workflow.PauseTarget) error
- func (f *Client) ResumeSchedule(ctx context.Context, t workflow.ScheduleTarget) error
- func (f *Client) Retry(ctx context.Context, ref workflow.RunRef) (workflow.RunRef, error)
- func (f *Client) Runs(ctx context.Context, q workflow.RunQuery) (workflow.RunPage, error)
- func (f *Client) Schedule(ctx context.Context, s workflow.Schedule) (workflow.ScheduleID, error)
- func (f *Client) Schedules(ctx context.Context) ([]workflow.ScheduleInfo, error)
- func (f *Client) SetWorkers(ctx context.Context, queue string, n int) error
- func (f *Client) Signal(ctx context.Context, ev workflow.Event) error
- func (f *Client) Start(ctx context.Context) error
- func (f *Client) Status(ctx context.Context, ref workflow.RunRef) (workflow.Status, error)
- func (f *Client) StopCause() error
- func (f *Client) Supports(op workflow.Op) bool
- func (f *Client) TriggerSchedule(ctx context.Context, id workflow.ScheduleID) (workflow.RunRef, error)
- func (f *Client) Unschedule(ctx context.Context, id workflow.ScheduleID) error
- type Event
- type EventKind
- type Observer
- type Option
Constants ¶
This section is empty.
Variables ¶
var ErrKeyRequired = errors.New("failover: Job.IdempotencyKey is required, because a primary that accepted the job and then failed to answer would make failover run it twice (use failover.AllowWithoutKey to accept that)")
ErrKeyRequired refuses a Job without an IdempotencyKey: an ambiguous primary failure followed by a failover could otherwise run it twice. It is returned wrapped together with workflow.ErrInvalid, so errors.Is matches both. AllowWithoutKey lifts the requirement.
Functions ¶
func DefaultClassifier ¶
DefaultClassifier fails over on an error a driver marked with workflow.Unavailable, and on every other error except the core sentinels that describe the request or the client's state rather than the backend's health (see callerFault). It never fails over on a Permanent error. Not every driver marks its transport errors yet, so the default errs towards failing over on unknown errors: an unknown error is most often a transport one. StrictClassifier is the opposite trade.
func StrictClassifier ¶
StrictClassifier fails over only on an error a driver marked with workflow.Unavailable (errors.Is(err, workflow.ErrUnavailable)), never on an unknown one, so an app bug a driver surfaces cannot send jobs to the secondary. Use it when every backend marks its outages:
failover.New(primary, secondary, failover.WithClassifier(failover.StrictClassifier))
A Permanent error never fails over.
Types ¶
type Classifier ¶
Classifier reports whether an error from the primary's Enqueue means the primary is unavailable, so the job should go to the secondary. It is only consulted when the caller's context is still live.
type Client ¶
type Client struct {
// contains filtered or unexported fields
}
Client is a workflow.Client over a primary and a secondary backend. Build it with New.
func New ¶
New returns a client that enqueues on primary and fails over to secondary. Both must be fresh, unstarted clients owned by this one: it registers and starts them.
func (*Client) CancelAll ¶
CancelAll cancels the matching runs on both backends, since runs of one workflow may sit on either, and returns the total. When one backend fails, the other's cancellations stand and the error names the failing one. A LabelSchedule filter is translated as in Runs. Needs CapCancelAll on both.
func (*Client) CancelStep ¶
CancelStep withdraws a step of the run on the backend its ref names.
func (*Client) Capabilities ¶
Capabilities is what both backends can do, except the schedule family (CapCron, CapScheduleWhen, CapScheduleWindow, CapSchedulePause, CapScheduleList, CapTickEvents), which is the primary's alone unless WithSchedulesOnBoth puts schedules on both: only the backends holding schedules fire, condition, report, list or pause them. Every other capability, the run query and pause families included, is the intersection, because a run may sit on either backend. CapClosure and CapWorkerScale are the primary's alone: a closure cannot be retried on another backend without risking a second run (it has no idempotency key), and the worker counts scaled are the primary's (balance the secondary with a BalanceLoop of its own). CapRunFinished also needs both backends to implement workflow.RunNotifier, which OnRunFinished requires.
func (*Client) Delete ¶
Delete forgets a terminal run on the backend its ref names. Needs CapDeleteRun on both.
func (*Client) Done ¶
func (f *Client) Done() <-chan struct{}
Done is closed once both backends have stopped. Without CapDrain on both it returns a nil channel, which never closes, as workflow.Forward does.
func (*Client) Drain ¶
Drain drains both backends at once and returns when both have stopped or ctx ends; the error joins each backend's. Needs CapDrain on both.
func (*Client) DrainWithReport ¶
DrainWithReport drains both backends at once and merges their reports; each cut run's ID carries its backend's prefix, as every other ID the failover client returns. Needs CapDrainReport on both.
func (*Client) Enqueue ¶
Enqueue sends job to the primary, and to the secondary when the primary is cooling off or fails with an error the Classifier calls unavailable. The RunRef names the backend that took the job.
func (*Client) EnqueueClosure ¶
func (f *Client) EnqueueClosure(ctx context.Context, c workflow.ClosureJob) (workflow.RunRef, error)
EnqueueClosure runs c on the primary only, with no failover: a closure has no idempotency key, so an ambiguous primary failure retried on the secondary could run it twice. The RunRef carries the primary prefix.
func (*Client) ForceComplete ¶
ForceComplete force-completes the run on the backend its ref names. Needs CapForceRun on both.
func (*Client) ForceFail ¶
ForceFail force-fails the run on the backend its ref names. Needs CapForceRun on both.
func (*Client) Health ¶
func (f *Client) Health(ctx context.Context) workflow.HealthReport
Health is live while both backends are live and ready while either is ready, since a failover client can take work as long as one backend can. Each check is named after its backend.
func (*Client) Nodes ¶
Nodes lists the nodes of both backends, primary first. A node ID is the backend's own; the same process may appear twice when it runs workers for both. Both must implement workflow.NodeLister.
func (*Client) OnLifecycle ¶
func (f *Client) OnLifecycle(fn workflow.LifecycleFunc) error
OnLifecycle registers fn on both backends, with each event's Ref and schedule label prefixed like the refs this client hands out. Both must implement workflow.LifecycleNotifier.
func (*Client) OnRunFinished ¶
func (f *Client) OnRunFinished(fn workflow.RunFinishedFunc) error
OnRunFinished registers fn on both backends, with each event's Ref prefixed by the backend it came from. Both backends must implement workflow.RunNotifier; otherwise it returns an UnsupportedError, since runs on the other backend would end unseen.
func (*Client) OnTick ¶
func (f *Client) OnTick(fn workflow.TickObserver) error
OnTick registers fn on the primary, and on the secondary under WithSchedulesOnBoth, with each tick's ScheduleID and Run prefixed by its backend. Needs CapTickEvents. Register before Start.
func (*Client) Pause ¶
Pause holds matching runs on both backends, since the runs a target names may sit on either. It is applied to the primary, then the secondary; when the secondary refuses, the primary's pause stands and the error says so. Needs the capability t needs on both.
func (*Client) PauseSchedule ¶
PauseSchedule pauses the schedule t names on every backend that holds it (decoded from the ScheduleID, as Unschedule does), or every schedule with t.All. Needs CapSchedulePause, which is the primary's alone unless WithSchedulesOnBoth.
func (*Client) Paused ¶
Paused returns the targets paused on either backend, without repeats, in the order Pauser documents. A target paused on one backend only (a half-applied Pause) is listed, so it can be seen and resumed.
func (*Client) PayloadLimit ¶
PayloadLimit is the smaller of the backends' hard caps (zero for one without a cap), since a job may land on either.
func (*Client) QueueLoads ¶
QueueLoads reports the primary's queue loads.
func (*Client) Register ¶
Register validates wf and registers it on both backends, with handlers that report the prefixed RunID in StepInfo. Without WithSchedulesOnBoth the secondary's copy has no Cron, CronWindows, ScheduleWhen, Misfire or CronOverlap.
func (*Client) Reschedule ¶
func (f *Client) Reschedule(ctx context.Context, ref workflow.RunRef, at time.Time) (workflow.RunRef, error)
Reschedule moves a delayed run on the backend its ref names, and prefixes the ref it returns.
func (*Client) Result ¶
Result reads the result from the backend the ref names. Needs CapResult on both.
func (*Client) ResumeSchedule ¶
ResumeSchedule resumes what PauseSchedule paused.
func (*Client) Retry ¶
Retry re-runs a failed run on the backend its ref names, and returns the ref prefixed the same way. Needs CapRetryRun on both.
func (*Client) Runs ¶
Runs lists the runs of both backends as one feed, newest enqueue first (ties broken by RunID descending), with every Ref prefixed by its backend like the refs Enqueue returns. Each page holds up to the query's limit; the cursor records where each backend's feed stands, so paging merges the two without gaps or repeats as long as each backend's own paging is stable. A fired run's LabelSchedule is prefixed like the schedule IDs this client hands out, and a LabelSchedule filter in q is turned back into each backend's own ID (a backend not holding that schedule is not asked). Needs CapRunQuery on both.
func (*Client) Schedule ¶
Schedule registers s on the primary, or on both under WithSchedulesOnBoth. It never fails over: a schedule made on the secondary during an outage would fire beside the primary's copies once the app retries it there.
func (*Client) Schedules ¶
Schedules lists the schedules of the primary, and of the secondary under WithSchedulesOnBoth, with each ID prefixed by its backend. Under WithSchedulesOnBoth a schedule therefore appears twice, once per copy, and pausing one ID pauses that copy only; the ScheduleID Schedule returned covers both. Sorted by Workflow, then Name, primary first. Needs CapScheduleList.
func (*Client) SetWorkers ¶
SetWorkers sets a queue's worker count on the primary.
func (*Client) Signal ¶
Signal delivers ev to both backends, since a waiting run may sit on either. When one fails, the other has already taken the event; the error names the failing backend.
func (*Client) Start ¶
Start checks every registered workflow against Capabilities, so a workflow the secondary cannot run is refused before anything starts, then starts the primary and the secondary. When the secondary fails to start the primary is stopped again.
func (*Client) StopCause ¶
StopCause joins each backend's StopCause, each named by its side, so a recycle limit reached on either backend reads as workflow.ErrRecycle. It is nil while neither stopped itself. Done still closes only once both backends stopped: give both the same recycle limits, or drain the other side when one recycles.
func (*Client) Supports ¶
Supports reports an operation as done only when both backends do it, since a run, or a node, may sit on either.
func (*Client) TriggerSchedule ¶
func (f *Client) TriggerSchedule(ctx context.Context, id workflow.ScheduleID) (workflow.RunRef, error)
TriggerSchedule runs a schedule now on the backend holding it and returns the run's ref prefixed by that backend. A schedule made under WithSchedulesOnBoth is triggered on the primary's copy only, so one trigger makes one run. It never fails over: a run triggered on the secondary would not be the schedule's last run there for the overlap check. Needs CapScheduleTrigger.
func (*Client) Unschedule ¶
Unschedule removes the schedule from every backend that holds it. An empty id is a no-op.
type Event ¶
type Event struct {
Kind EventKind
// Workflow is the job's workflow name.
Workflow string
// Err is the primary's error, for EventFailedOver.
Err error
// Until is the end of the cool-off, for EventCoolOffStarted.
Until time.Time
// At is when the event happened, read from the Client's clock.
At time.Time
}
Event is one failover event. It never carries the job's payload or labels, which can hold personal data.
type EventKind ¶
type EventKind string
EventKind names what happened to the Client, as an Observer sees it.
const ( // EventFailedOver: the primary's Enqueue failed with an error the // Classifier called unavailable, and the job went to the secondary // (Err is the primary's error). EventFailedOver EventKind = "failed_over" // EventPrimarySkipped: the primary was cooling off, so the job went // straight to the secondary. EventPrimarySkipped EventKind = "primary_skipped" // EventCoolOffStarted: the failover streak reached WithCoolOff's n; // Until is when the primary is tried again. EventCoolOffStarted EventKind = "cool_off_started" // EventCoolOffEnded: the primary took a job again after a cool-off. EventCoolOffEnded EventKind = "cool_off_ended" )
The events a Client reports to its Observer.
type Observer ¶
Observer receives the Client's events, synchronously, on the Enqueue that caused them; it must return quickly and must not call Enqueue on the same Client. A secondary that fails too is reported to the caller by Enqueue's error, not here.
type Option ¶
type Option func(*Client)
Option configures a Client.
func AllowWithoutKey ¶
func AllowWithoutKey() Option
AllowWithoutKey accepts jobs without an IdempotencyKey, accepting that an ambiguous primary failure may run such a job on both backends.
func WithClassifier ¶
func WithClassifier(c Classifier) Option
WithClassifier replaces DefaultClassifier. A nil c is ignored.
func WithClock ¶
WithClock sets the time source of the cool-off. The default is clock.Real. A nil c is ignored.
func WithCoolOff ¶
WithCoolOff skips the primary for d after n consecutive enqueues failed over, sending jobs straight to the secondary. After d the next enqueue tries the primary again; one more failure starts a new cool-off, and a success ends the streak. n below 1 or d of zero or less disables it (the default).
func WithObserver ¶
WithObserver reports failovers and cool-offs to fn, for a log line, a metric or an alert. The default reports nothing. A nil fn is ignored.
failover.WithObserver(func(ctx context.Context, ev failover.Event) {
logger.WarnContext(ctx, "workflow failover", "kind", ev.Kind, "workflow", ev.Workflow, "error", ev.Err)
})
func WithSchedulesOnBoth ¶
func WithSchedulesOnBoth() Option
WithSchedulesOnBoth registers Workflow.Cron and Client.Schedule on both backends, so schedules keep firing while the primary is down, at the price of every tick firing on both while both are up. Only for scheduled workflows that tolerate running twice per tick.