failover

package
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 10, 2026 License: Apache-2.0 Imports: 14 Imported by: 0

Documentation

Overview

Package failover joins two workflow clients into one: enqueues go to a primary backend, and when the primary is unavailable they go to a secondary, so a job is still accepted while (say) the Hatchet engine is down and a local SQL store takes the work.

c := failover.New(hatchetClient, sqlClient)
c.Register(wf) // registers on both
c.Start(ctx)   // starts both
ref, err := c.Enqueue(ctx, workflow.Job{Workflow: "email", IdempotencyKey: "order-42"})
st, err := c.Status(ctx, ref) // asks whichever backend holds the run

What fails over

Only Enqueue fails over, and only when the Classifier says the primary's error means "backend unavailable". DefaultClassifier never fails over on a cancelled or expired caller context, nor on an error the secondary would return just the same (ErrInvalid, ErrUnsupported, ErrNotFound, ErrNotStarted, ErrAlreadyStarted, ErrDuplicate, ErrCancelled, a Permanent error); an error a driver marked with workflow.Unavailable, and everything else, counts as unavailable. The Unavailable mark wins over those sentinels: a driver's own timeout talking to its engine wraps context.DeadlineExceeded, and marked Unavailable it fails over, while the caller's own cancelled or expired context is checked before any Classifier runs and never fails over. StrictClassifier fails over only on errors marked with workflow.Unavailable, for deployments whose drivers all mark their outages; a deployment that can tell another way (a gRPC Unavailable code, a refused connection) passes its own Classifier through WithClassifier.

Running a job twice

A failure can be ambiguous: the primary may have accepted the job and then failed to answer (a timeout after the write). Failing over then runs the job on both backends. So a Job without an IdempotencyKey is refused with ErrKeyRequired (which also matches workflow.ErrInvalid), unless the client was built with AllowWithoutKey. A key dedups only within one backend: the primary and the secondary do not see each other's keys, so the key protects a caller's retries against the same backend, and the handlers must still be idempotent across backends. A workflow with IdempotencyOff ignores keys altogether.

Run references

Every RunRef this client returns, and every RunEvent and StepInfo it passes on, names its backend: RunID and ScheduledID carry the prefix "primary:" or "secondary:" in front of the driver's own ID, which is kept unchanged after it, so Status, Cancel and the run-finished hook reach the backend holding the run. A ref without a prefix (one a driver handed out directly) is tried on the primary, then on the secondary when the primary does not know it or is unavailable. A driver's middleware sees the driver's own RunID in StepInfo; the handler sees the prefixed one.

Capabilities

Capabilities is the intersection of both backends', so Start refuses a workflow the secondary could not run rather than discovering it during an outage. The exceptions: the schedule family follows where schedules live (see below); CapRunFinished needs both backends to implement workflow.RunNotifier; and CapClosure and CapWorkerScale are the primary's alone. EnqueueClosure goes to the primary with no failover, because a closure has no idempotency key and an ambiguous failure retried on the secondary could run it twice; QueueLoads and SetWorkers scale the primary's workers (balance the secondary with a loop of its own). CapRunStates and CapChain are the intersection like the rest. A chained job (Job.Chain) is enqueued by the backend whose run finished it, never through this client, so it does not fail over: a chain stays on the backend that took its first job.

Managing runs

The client implements every optional run interface, each needing its capability on both backends and refused with an UnsupportedError otherwise, never answered from one backend alone. Runs merges both backends' feeds newest first with prefixed refs and a cursor that records each feed's position; Retry and Delete go to the backend the ref names; CancelAll, Pause and Resume go to both (Paused lists the union); Drain drains both.

Schedules

Workflow.Cron and Schedule live on the primary only, because a schedule on both backends fires on both: every tick would run twice. The cost is that no schedule fires while the primary is down. WithSchedulesOnBoth puts them on both for apps whose scheduled handlers are idempotent per tick. The schedule capabilities (CapCron, CapScheduleWhen, CapScheduleWindow, CapSchedulePause, CapScheduleList, CapTickEvents) are the primary's alone by default and need both under WithSchedulesOnBoth; the secondary's copy of a workflow then carries no Cron, CronWindows or ScheduleWhen. Ticks, listed schedules and the LabelSchedule of fired runs carry the backend prefix; PauseSchedule takes those IDs or the one Schedule returned.

Signals and run ends

Signal goes to both backends, since a waiting run may sit on either; when one of them fails the error says which, and the other has already taken the event, so a retried Signal may reach that one twice. OnRunFinished registers the observer on both backends and needs both to implement workflow.RunNotifier; CapRunFinished holds only when both guarantee delivery.

Cool-off

WithCoolOff stops trying the primary for a while after N consecutive failovers, so an outage does not cost every enqueue a timeout first. It reads time from WithClock (clock.Real by default).

Observing

WithObserver reports each failover, each enqueue sent straight to the secondary during a cool-off, and the start and end of each cool-off, so an app can log, count or alert on them; by default nothing is reported.

Index

Constants

This section is empty.

Variables

View Source
var ErrKeyRequired = errors.New("failover: Job.IdempotencyKey is required, because a primary that accepted the job and then failed to answer would make failover run it twice (use failover.AllowWithoutKey to accept that)")

ErrKeyRequired refuses a Job without an IdempotencyKey: an ambiguous primary failure followed by a failover could otherwise run it twice. It is returned wrapped together with workflow.ErrInvalid, so errors.Is matches both. AllowWithoutKey lifts the requirement.

Functions

func DefaultClassifier

func DefaultClassifier(err error) bool

DefaultClassifier fails over on an error a driver marked with workflow.Unavailable, and on every other error except the core sentinels that describe the request or the client's state rather than the backend's health (see callerFault). It never fails over on a Permanent error. Not every driver marks its transport errors yet, so the default errs towards failing over on unknown errors: an unknown error is most often a transport one. StrictClassifier is the opposite trade.

func StrictClassifier

func StrictClassifier(err error) bool

StrictClassifier fails over only on an error a driver marked with workflow.Unavailable (errors.Is(err, workflow.ErrUnavailable)), never on an unknown one, so an app bug a driver surfaces cannot send jobs to the secondary. Use it when every backend marks its outages:

failover.New(primary, secondary, failover.WithClassifier(failover.StrictClassifier))

A Permanent error never fails over.

Types

type Classifier

type Classifier func(err error) bool

Classifier reports whether an error from the primary's Enqueue means the primary is unavailable, so the job should go to the secondary. It is only consulted when the caller's context is still live.

type Client

type Client struct {
	// contains filtered or unexported fields
}

Client is a workflow.Client over a primary and a secondary backend. Build it with New.

func New

func New(primary, secondary workflow.Client, opts ...Option) *Client

New returns a client that enqueues on primary and fails over to secondary. Both must be fresh, unstarted clients owned by this one: it registers and starts them.

func (*Client) Cancel

func (f *Client) Cancel(ctx context.Context, ref workflow.RunRef) error

Cancel cancels the run on the backend the ref names.

func (*Client) CancelAll

func (f *Client) CancelAll(ctx context.Context, q workflow.RunQuery) (int, error)

CancelAll cancels the matching runs on both backends, since runs of one workflow may sit on either, and returns the total. When one backend fails, the other's cancellations stand and the error names the failing one. A LabelSchedule filter is translated as in Runs. Needs CapCancelAll on both.

func (*Client) CancelStep

func (f *Client) CancelStep(ctx context.Context, ref workflow.RunRef, step string) error

CancelStep withdraws a step of the run on the backend its ref names.

func (*Client) Capabilities

func (f *Client) Capabilities() workflow.Caps

Capabilities is what both backends can do, except the schedule family (CapCron, CapScheduleWhen, CapScheduleWindow, CapSchedulePause, CapScheduleList, CapTickEvents), which is the primary's alone unless WithSchedulesOnBoth puts schedules on both: only the backends holding schedules fire, condition, report, list or pause them. Every other capability, the run query and pause families included, is the intersection, because a run may sit on either backend. CapClosure and CapWorkerScale are the primary's alone: a closure cannot be retried on another backend without risking a second run (it has no idempotency key), and the worker counts scaled are the primary's (balance the secondary with a BalanceLoop of its own). CapRunFinished also needs both backends to implement workflow.RunNotifier, which OnRunFinished requires.

func (*Client) Delete

func (f *Client) Delete(ctx context.Context, ref workflow.RunRef) error

Delete forgets a terminal run on the backend its ref names. Needs CapDeleteRun on both.

func (*Client) Done

func (f *Client) Done() <-chan struct{}

Done is closed once both backends have stopped. Without CapDrain on both it returns a nil channel, which never closes, as workflow.Forward does.

func (*Client) Drain

func (f *Client) Drain(ctx context.Context) error

Drain drains both backends at once and returns when both have stopped or ctx ends; the error joins each backend's. Needs CapDrain on both.

func (*Client) DrainWithReport

func (f *Client) DrainWithReport(ctx context.Context) (workflow.DrainReport, error)

DrainWithReport drains both backends at once and merges their reports; each cut run's ID carries its backend's prefix, as every other ID the failover client returns. Needs CapDrainReport on both.

func (*Client) Enqueue

func (f *Client) Enqueue(ctx context.Context, job workflow.Job) (workflow.RunRef, error)

Enqueue sends job to the primary, and to the secondary when the primary is cooling off or fails with an error the Classifier calls unavailable. The RunRef names the backend that took the job.

func (*Client) EnqueueClosure

func (f *Client) EnqueueClosure(ctx context.Context, c workflow.ClosureJob) (workflow.RunRef, error)

EnqueueClosure runs c on the primary only, with no failover: a closure has no idempotency key, so an ambiguous primary failure retried on the secondary could run it twice. The RunRef carries the primary prefix.

func (*Client) ForceComplete

func (f *Client) ForceComplete(ctx context.Context, ref workflow.RunRef) error

ForceComplete force-completes the run on the backend its ref names. Needs CapForceRun on both.

func (*Client) ForceFail

func (f *Client) ForceFail(ctx context.Context, ref workflow.RunRef, reason string) error

ForceFail force-fails the run on the backend its ref names. Needs CapForceRun on both.

func (*Client) Health

func (f *Client) Health(ctx context.Context) workflow.HealthReport

Health is live while both backends are live and ready while either is ready, since a failover client can take work as long as one backend can. Each check is named after its backend.

func (*Client) Nodes

func (f *Client) Nodes(ctx context.Context) ([]workflow.NodeInfo, error)

Nodes lists the nodes of both backends, primary first. A node ID is the backend's own; the same process may appear twice when it runs workers for both. Both must implement workflow.NodeLister.

func (*Client) OnLifecycle

func (f *Client) OnLifecycle(fn workflow.LifecycleFunc) error

OnLifecycle registers fn on both backends, with each event's Ref and schedule label prefixed like the refs this client hands out. Both must implement workflow.LifecycleNotifier.

func (*Client) OnRunFinished

func (f *Client) OnRunFinished(fn workflow.RunFinishedFunc) error

OnRunFinished registers fn on both backends, with each event's Ref prefixed by the backend it came from. Both backends must implement workflow.RunNotifier; otherwise it returns an UnsupportedError, since runs on the other backend would end unseen.

func (*Client) OnTick

func (f *Client) OnTick(fn workflow.TickObserver) error

OnTick registers fn on the primary, and on the secondary under WithSchedulesOnBoth, with each tick's ScheduleID and Run prefixed by its backend. Needs CapTickEvents. Register before Start.

func (*Client) Pause

func (f *Client) Pause(ctx context.Context, t workflow.PauseTarget) error

Pause holds matching runs on both backends, since the runs a target names may sit on either. It is applied to the primary, then the secondary; when the secondary refuses, the primary's pause stands and the error says so. Needs the capability t needs on both.

func (*Client) PauseSchedule

func (f *Client) PauseSchedule(ctx context.Context, t workflow.ScheduleTarget) error

PauseSchedule pauses the schedule t names on every backend that holds it (decoded from the ScheduleID, as Unschedule does), or every schedule with t.All. Needs CapSchedulePause, which is the primary's alone unless WithSchedulesOnBoth.

func (*Client) Paused

func (f *Client) Paused(ctx context.Context) ([]workflow.PauseTarget, error)

Paused returns the targets paused on either backend, without repeats, in the order Pauser documents. A target paused on one backend only (a half-applied Pause) is listed, so it can be seen and resumed.

func (*Client) PayloadLimit

func (f *Client) PayloadLimit() int

PayloadLimit is the smaller of the backends' hard caps (zero for one without a cap), since a job may land on either.

func (*Client) QueueLoads

func (f *Client) QueueLoads(ctx context.Context) ([]workflow.QueueLoad, error)

QueueLoads reports the primary's queue loads.

func (*Client) Register

func (f *Client) Register(wf workflow.Workflow) error

Register validates wf and registers it on both backends, with handlers that report the prefixed RunID in StepInfo. Without WithSchedulesOnBoth the secondary's copy has no Cron, CronWindows, ScheduleWhen, Misfire or CronOverlap.

func (*Client) Reschedule

func (f *Client) Reschedule(ctx context.Context, ref workflow.RunRef, at time.Time) (workflow.RunRef, error)

Reschedule moves a delayed run on the backend its ref names, and prefixes the ref it returns.

func (*Client) Result

func (f *Client) Result(ctx context.Context, ref workflow.RunRef) ([]byte, error)

Result reads the result from the backend the ref names. Needs CapResult on both.

func (*Client) Resume

func (f *Client) Resume(ctx context.Context, t workflow.PauseTarget) error

Resume removes t on both backends.

func (*Client) ResumeSchedule

func (f *Client) ResumeSchedule(ctx context.Context, t workflow.ScheduleTarget) error

ResumeSchedule resumes what PauseSchedule paused.

func (*Client) Retry

func (f *Client) Retry(ctx context.Context, ref workflow.RunRef) (workflow.RunRef, error)

Retry re-runs a failed run on the backend its ref names, and returns the ref prefixed the same way. Needs CapRetryRun on both.

func (*Client) Runs

Runs lists the runs of both backends as one feed, newest enqueue first (ties broken by RunID descending), with every Ref prefixed by its backend like the refs Enqueue returns. Each page holds up to the query's limit; the cursor records where each backend's feed stands, so paging merges the two without gaps or repeats as long as each backend's own paging is stable. A fired run's LabelSchedule is prefixed like the schedule IDs this client hands out, and a LabelSchedule filter in q is turned back into each backend's own ID (a backend not holding that schedule is not asked). Needs CapRunQuery on both.

func (*Client) Schedule

Schedule registers s on the primary, or on both under WithSchedulesOnBoth. It never fails over: a schedule made on the secondary during an outage would fire beside the primary's copies once the app retries it there.

func (*Client) Schedules

func (f *Client) Schedules(ctx context.Context) ([]workflow.ScheduleInfo, error)

Schedules lists the schedules of the primary, and of the secondary under WithSchedulesOnBoth, with each ID prefixed by its backend. Under WithSchedulesOnBoth a schedule therefore appears twice, once per copy, and pausing one ID pauses that copy only; the ScheduleID Schedule returned covers both. Sorted by Workflow, then Name, primary first. Needs CapScheduleList.

func (*Client) SetWorkers

func (f *Client) SetWorkers(ctx context.Context, queue string, n int) error

SetWorkers sets a queue's worker count on the primary.

func (*Client) Signal

func (f *Client) Signal(ctx context.Context, ev workflow.Event) error

Signal delivers ev to both backends, since a waiting run may sit on either. When one fails, the other has already taken the event; the error names the failing backend.

func (*Client) Start

func (f *Client) Start(ctx context.Context) error

Start checks every registered workflow against Capabilities, so a workflow the secondary cannot run is refused before anything starts, then starts the primary and the secondary. When the secondary fails to start the primary is stopped again.

func (*Client) Status

func (f *Client) Status(ctx context.Context, ref workflow.RunRef) (workflow.Status, error)

Status asks the backend the ref names. See onRef for unprefixed refs.

func (*Client) StopCause

func (f *Client) StopCause() error

StopCause joins each backend's StopCause, each named by its side, so a recycle limit reached on either backend reads as workflow.ErrRecycle. It is nil while neither stopped itself. Done still closes only once both backends stopped: give both the same recycle limits, or drain the other side when one recycles.

func (*Client) Supports

func (f *Client) Supports(op workflow.Op) bool

Supports reports an operation as done only when both backends do it, since a run, or a node, may sit on either.

func (*Client) TriggerSchedule

func (f *Client) TriggerSchedule(ctx context.Context, id workflow.ScheduleID) (workflow.RunRef, error)

TriggerSchedule runs a schedule now on the backend holding it and returns the run's ref prefixed by that backend. A schedule made under WithSchedulesOnBoth is triggered on the primary's copy only, so one trigger makes one run. It never fails over: a run triggered on the secondary would not be the schedule's last run there for the overlap check. Needs CapScheduleTrigger.

func (*Client) Unschedule

func (f *Client) Unschedule(ctx context.Context, id workflow.ScheduleID) error

Unschedule removes the schedule from every backend that holds it. An empty id is a no-op.

type Event

type Event struct {
	Kind EventKind
	// Workflow is the job's workflow name.
	Workflow string
	// Err is the primary's error, for EventFailedOver.
	Err error
	// Until is the end of the cool-off, for EventCoolOffStarted.
	Until time.Time
	// At is when the event happened, read from the Client's clock.
	At time.Time
}

Event is one failover event. It never carries the job's payload or labels, which can hold personal data.

type EventKind

type EventKind string

EventKind names what happened to the Client, as an Observer sees it.

const (
	// EventFailedOver: the primary's Enqueue failed with an error the
	// Classifier called unavailable, and the job went to the secondary
	// (Err is the primary's error).
	EventFailedOver EventKind = "failed_over"
	// EventPrimarySkipped: the primary was cooling off, so the job went
	// straight to the secondary.
	EventPrimarySkipped EventKind = "primary_skipped"
	// EventCoolOffStarted: the failover streak reached WithCoolOff's n;
	// Until is when the primary is tried again.
	EventCoolOffStarted EventKind = "cool_off_started"
	// EventCoolOffEnded: the primary took a job again after a cool-off.
	EventCoolOffEnded EventKind = "cool_off_ended"
)

The events a Client reports to its Observer.

type Observer

type Observer func(ctx context.Context, ev Event)

Observer receives the Client's events, synchronously, on the Enqueue that caused them; it must return quickly and must not call Enqueue on the same Client. A secondary that fails too is reported to the caller by Enqueue's error, not here.

type Option

type Option func(*Client)

Option configures a Client.

func AllowWithoutKey

func AllowWithoutKey() Option

AllowWithoutKey accepts jobs without an IdempotencyKey, accepting that an ambiguous primary failure may run such a job on both backends.

func WithClassifier

func WithClassifier(c Classifier) Option

WithClassifier replaces DefaultClassifier. A nil c is ignored.

func WithClock

func WithClock(c clock.Clock) Option

WithClock sets the time source of the cool-off. The default is clock.Real. A nil c is ignored.

func WithCoolOff

func WithCoolOff(n int, d time.Duration) Option

WithCoolOff skips the primary for d after n consecutive enqueues failed over, sending jobs straight to the secondary. After d the next enqueue tries the primary again; one more failure starts a new cool-off, and a success ends the streak. n below 1 or d of zero or less disables it (the default).

func WithObserver

func WithObserver(fn Observer) Option

WithObserver reports failovers and cool-offs to fn, for a log line, a metric or an alert. The default reports nothing. A nil fn is ignored.

failover.WithObserver(func(ctx context.Context, ev failover.Event) {
	logger.WarnContext(ctx, "workflow failover", "kind", ev.Kind, "workflow", ev.Workflow, "error", ev.Err)
})

func WithSchedulesOnBoth

func WithSchedulesOnBoth() Option

WithSchedulesOnBoth registers Workflow.Cron and Client.Schedule on both backends, so schedules keep firing while the primary is down, at the price of every tick firing on both while both are up. Only for scheduled workflows that tolerate running twice per tick.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL