Documentation
¶
Overview ¶
Package hatchet is the Hatchet driver of github.com/ubgo/workflow: a workflow.Client whose runs, DAG steps, durable waits and sleeps, retries, timeouts, crons, concurrency keys and idempotency keys are Hatchet's own (github.com/hatchet-dev/hatchet sdks/go).
It is a separate module so an app that never uses Hatchet never downloads the SDK. Build a client with New, Register workflows, then Start; nothing connects before Start. Native returns the SDK client as the one escape hatch.
What Hatchet cannot express is refused with an error wrapping workflow.ErrUnsupported that names the field (and, where one applies, the missing capability), never dropped. Workflows are refused at Register, schedules at Schedule, jobs at Enqueue: Concurrency.Key.Func (keys are CEL, use workflow.KeyExpr), WaitSpec.KeyFrom.Expr (use workflow.KeyFunc), a Workflow.CronLocation or Schedule.Location other than UTC (Hatchet evaluates crons in UTC only; no CapCronZone), any Workflow.Misfire or Schedule.Misfire policy (Hatchet never catches up a missed tick and has no lateness window, so only the driver default holds: a tick missed while the engine was down is dropped; no CapMisfire*), cron macros and @every, a retry Backoff whose Initial is not Factor seconds (Hatchet waits Factor^n seconds), a Backoff.Jitter (no CapRetryJitter) and a RetryPolicy.DelayFor such as workflow.RetryAfterDelay (the engine never takes a retry's delay from the error; no CapRetryDelay), a negative Job.Priority, Job.IdempotencyKey together with Job.RunAt, a RateLimit.Per other than a second, minute, hour, day or week, and RateLimits on a workflow with more than one first step (Hatchet limits tasks, not runs), a RetryPolicy.Until or MaxErrors (Hatchet fixes a task's retries when it is declared; no CapRetryUntil, CapRetryMaxErrors), a Job.Retry (no CapJobRetry, same reason), a Job.Queue other than the workflow's (a run goes to whichever worker registered its workflow; no CapQueueOverride), a QueueConfig.Priority or a non-zero QueueConfig.Limit (refused at Start: no CapQueueLimitGlobal, since the engine does not count a whole queue's in-flight steps across workers), RetryPolicy.DiscardOnExhaust (the engine has no discarded status; no CapDiscardExhausted), Workflow.ScheduleWhen and Workflow.CronWindows or Schedule.Windows (the engine fires crons itself; no CapScheduleWhen, CapScheduleWindow), OverlapSkip on Workflow.CronOverlap or Schedule.Overlap (the engine fires a tick without asking a worker and has no setting that skips one while the previous run is unfinished; no CapOverlapSkip), Workflow.CronActive and Schedule.Active (a Hatchet cron has no start or end; no CapScheduleActive), Workflow.CronKeepRuns and Schedule.KeepRuns (the engine keeps runs for its retention and has no run delete; no CapScheduleKeep), Workflow.RunTimeout and OnTimeout (the engine times out tasks, not runs, with its own error text; no CapRunTimeout), Step.HeartbeatTimeout (see below; no CapHeartbeat) and a QueueConfig.Weight (refused at Start: each queue is its own worker, so no worker picks between queues). The core's cron parser is the only one: the driver checks expressions with cron.Parse and has no option for another dialect, since the engine parses them itself. Start refuses Config tunings past the SDK's limits (GroupChunkRuns over 1000, GroupChunkBytes over 2 MiB) and a BatchCoordinatorRetry Hatchet cannot express (as above, or with NoRetryOn).
An engine call that fails because the engine could not be reached (a gRPC Unavailable status, a network error from the REST client, an HTTP 502, 503 or 504) returns an error matching workflow.ErrUnavailable, so a failover client can tell an outage from a refusal; the caller's own cancellation or deadline never does.
Tunings: Config.GroupChunkRuns and GroupChunkBytes size bulk triggers, BatchCoordinatorMaxInput caps a coordinator trigger, BatchCoordinatorRetry is the coordinator's retry policy, RegisterTimeout bounds Start's wait for the worker, and VisibilityLag is how long the engine's read side may trail a write. Config.Logger receives the background errors (run-finished polls, observer panics, worker stop) and Config.Clock drives what is local to the process (poll tickers, the delivered-run window, FinishedAt of a delayed start cancelled before it fired); times compared with the engine's timestamps stay on the wall clock.
Rate limits are Hatchet's own: a KeyExpr limit becomes a dynamic limit (KeyExpr plus LimitValueExpr, so the engine creates each bucket on first use). The engine keys a dynamic bucket by the key's value alone, per tenant, so the driver prefixes the value with the limit's declaration (namespace, workflow, index): two workflows whose keys evaluate to the same value have separate buckets, as on the other drivers (TestRateLimitScopeLive; before, a run of one waited for the other's bucket). A keyless RateLimit is refused (field RateLimits[i].Key): its static-limit mapping missed the long-run bound by 1.2s in its live test, so it is not claimed; workflow.KeyExpr(`"all"`) gives one bucket through the proven dynamic path. Hatchet's limiter is a fixed window that is not aligned to the clock: a bucket is refilled to Limit once Per has passed since its last refill (pkg/repository/sqlcv1/rate_limits.sql, lastRefill), so up to 2×Limit runs can start in one Per-long span across a refill, which the core RateLimit contract allows. A run held back by the limit stays queued and starts after a refill; it is never dropped or failed. Refills happen when the engine next checks the bucket, so the next run of a full bucket starts up to about 30s after the window ends.
OnRunFinished (workflow.RunNotifier) is implemented by polling the engine's run list, and CapRunFinished is not claimed: Hatchet has no run-end callback, worker-side hooks cannot see cancelled or dropped runs, and a restarted client replays only the ends of runs created within Config.RunFinishedCatchUp. A delayed start cancelled before it fires is reported only by the process that cancelled it. See notify.go.
Batches (package batch) run natively: the Client implements batch.GroupRunner as Hatchet bulk triggers, one RunMany call per chunk of up to 1000 jobs of one workflow and about 2 MiB of input, so a batch costs about N/1000 calls instead of N Enqueues; each run carries what Enqueue stamps. BatchSource is the batch completion source to use with it: one run-list call per open batch each tick, filtered by the batch's label, instead of one Status call per job; it also resumes a dispatch whose process died (batch.Manager.ResumeDispatches), using Client.FindBatchRuns (batch.RunFinder) to list which jobs the dead process had started, by label, whatever their status.
With Config.BatchCoordinator the Client is also a batch.GroupSpawner: Dispatch starts an eligible batch (no RunAt or IdempotencyKey, at most about 2 MiB of input; larger ones fall back to bulk triggers) as one coordinator run, whose task starts every job as a child run in bulk calls, each child keyed by its job index. A retried coordinator gets the existing child back for a key the engine has seen, so a worker that dies mid-spawn does not start a job twice (proven live by TestBatchCoordinatorCrashLive). The children have no refs at Dispatch, since the engine names a child only when it is started, so the batch keeps the coordinator's ref, BatchSource finds the children by label (Dispatch refuses a coordinated batch without it), and a job the coordinator never started (it was cancelled, or ran out of retries) is recorded as cancelled once the coordinator has ended. The engine does not cancel children with their parent (checked live, for a running and a completed parent), so CancelGroup cancels the parent and then every run carrying the batch's label in one bulk cancel; while a cancelled batch is open, BatchSource repeats that each tick, so a child started late is cancelled too. The coordinator trigger is idempotent per batch for a day (TTL), so a resumed dispatch gets the same coordinator. Config.BatchCoordinatorTimeout bounds a coordinator attempt, and is how long a coordinator on a dead worker waits for the engine's retry.
Run states, chains, worker scaling and closures are not claimed. The engine's run statuses are queued, running, completed, failed and cancelled, so a run in retry backoff reads StatusQueued and a discarded run StatusFailed (no CapRunStates); a RunQuery or CancelAll filter naming StatusRetrying or StatusDiscarded is refused (field RunQuery.Statuses) rather than answered with a page that says "none". A RunQuery.LabelPrefixes filter is refused the same way by Runs and CancelAll (field RunQuery.LabelPrefixes, no CapLabelPrefix): the engine filters metadata by exact value only. A Job.Chain is refused at Enqueue, RunGroup and SpawnGroup (field Job.Chain, no CapChain), and no chain editor is put on a step's ctx, so PrependToChain and AppendToChain return an UnsupportedError: a Hatchet run's shape is fixed when its workflow is declared; a step that needs follow-up work enqueues it as a new run. The client does not implement workflow.WorkerScaler (no CapWorkerScale): the SDK takes a worker's slots once, when the worker is created, and has no call to change them. It does not implement workflow.ClosureEnqueuer (no CapClosure), so workflow.EnqueueClosure refuses: a Go closure cannot be sent to the engine. A durable closure by name (workflow.NewFunc, RegisterFunc, FuncJob) needs nothing of the driver: it is a one-step workflow, registered and declared like any other, so a process that registered the name runs it and an unregistered name is refused at Enqueue with ErrNotFound. Its FuncRetry follows the Retry.Backoff rule below.
Worker recycling (Config.Recycle, workflow.Recycle): after MaxJobs finished handler attempts, MaxTime on Config.Clock since Start, or a heap above MaxMemory, the client drains itself as Drain does with no deadline: its workers are paused on the engine, the handlers running here finish, the workers stop, Done closes and StopCause returns the *workflow.RecycleError, so workflow.Serve returns it and a supervisor starts a fresh process. A requested stop (the Start ctx ended, Drain) leaves StopCause nil. The engine may have handed a worker its next attempt just before the pause landed; that attempt runs to the end here, so up to Slots attempts can start past MaxJobs (tested with one slot by TestRecycleLive). Durable wait and sleep steps do not count, since they hold no worker.
Queues, run management, pause and drain: Workflow.Queue and Config.Queues give one Hatchet worker per served queue, with that queue's Workers as slots (CapQueue); Enqueue stamps the queue and the priority on the run. Client.Runs (workflow.RunQuerier) merges the run list with the unfired delayed starts and reads each run's details for its failed step, attempts and waiting status; CancelAll (workflow.BulkCanceller) lists then cancels; Retry (workflow.RunRetrier) is Hatchet's replay of the failed and cancelled step tasks, which keeps the steps that succeeded; Schedules (workflow.ScheduleLister) lists the engine's cron triggers; Drain (workflow.Drainer) pauses the client's workers on the engine, waits for the handlers running here, then stops them; DrainWithReport (workflow.ReportingDrainer, CapDrainReport) also lists the handlers cut off at the deadline with the disposition workflow.DrainEngineOwned, because whether the engine requeues them is the engine's decision, and reads FinishedRuns back from the engine (a worker is not told when a run ends): the runs of the handlers that returned are looked up for up to two seconds and the terminal ones counted. A named Schedule is one trigger per (workflow, name) with one ID ("named:" plus the pair), the same across a changed definition, which replaces the engine trigger (keeping its pause when Config.API is set). Not claimed: CapDiscard (a Discard fails the run, as Permanent does), CapDeleteRun (the engine has no run delete), CapPauseQueue (a worker pause is not durable across nodes) and CapTickEvents (the engine fires crons without the driver). CapPauseWorkflow and CapSchedulePause are declared only with Config.API (NewAPI): SDK v0.107.1 has no call for the engine's workflow pause or cron enable and exposes no REST client, so the driver calls the REST API itself. PauseAll and PauseSchedule(All) are refused (the engine stores a pause per workflow and per trigger, with nothing that would also hold ones made later); Resume(PauseAll()) and ResumeSchedule(All) clear every pause of the namespace. A cron tick that falls due while its trigger is disabled is dropped. Retry does not take the idempotency key again on the engine: it refuses with ErrDuplicate when another unfinished run holds the key, but an Enqueue with that key while the retried run is unfinished starts a second run.
Schedule trigger, history and enqueue-only clients: TriggerSchedule (workflow.ScheduleTriggerer, CapScheduleTrigger) reads a cron trigger, declared or made by Schedule, and enqueues its workflow now with the trigger's input and labels plus workflow.TriggeredLabelsAt, on a disabled trigger too, leaving its cron untouched. ScheduleInfo.Last (CapScheduleHistory) is not stored by the driver: Schedules reads it from the engine's run list, as the newest run the engine fired for the trigger (by the hatchet__cron_name or, for a declared trigger, hatchet__cron_expression metadata the engine stamps on every cron run, with Fire from hatchet__cron_scheduled_at) or TriggerSchedule started for it (by workflow.LabelSchedule), within Config.ScheduleHistoryWindow (a week by default), so it survives restarts as long as the engine keeps the runs, costs two run-list calls and one run read per trigger, and reads as never fired past the window. Config.EnqueueOnly (CapEnqueueOnly) makes a client for web nodes: Enqueue, Status, Cancel, Signal, Schedule and the listings work with no Start and no worker, connecting on first use; Start starts nothing; a registered workflow keeps its checks, an unregistered one gets workflow.CheckJob only; OnRunFinished is refused. RunEvent.Step names the failed step of a failed run, as RunInfo.FailedStep does. Not claimed: CapErrorHistory (the run details carry task events with an attempt and an error message, but the driver has not proven that they list every failed attempt, and the separate task event list returned no rows on hatchet-lite v0.110.5), CapShutdownRequeue (a worker stopped by its Start ctx cancels the steps running on it and the engine counts that attempt; Drain lets them finish instead) and CapForceRun (the engine has no call that completes or fails a run from outside, only cancel, replay and restore of an evicted durable task), so the client does not implement workflow.RunForcer. The SDK's panic on a client config that does not load (a missing token) is turned into an error by Start and by the enqueue-only connect.
Tenants, results, rescheduling and the rest of batch 5: Job.Tenant (CapTenant) travels as run metadata, reaches StepInfo.Tenant, RunInfo.Tenant and RunEvent.Tenant, and RunQuery.Tenant filters on the engine. Schedule.Tenant is stored on the cron trigger's metadata, which the engine copies onto every run it fires; TriggerSchedule sets it as the run's Job.Tenant, Schedules lists it as ScheduleInfo.Tenant, and a named schedule whose tenant changed is replaced. Every run the engine's cron fires carries workflow.LabelTickAt (CapTickTime), read from the tick time the engine stamps on it (hatchet__cron_scheduled_at, the fire time its scheduler computed, whole seconds), and TriggerSchedule sets it to the trigger time. A six-field expression with seconds first (Schedule.Seconds, Workflow.CronSeconds; CapCronSeconds) is sent as is: the engine's scheduler reads six fields as seconds first, proven by TestCronSecondsTickLive firing every two seconds. Result (workflow.ResultReader, CapResult) and RunInfo.Output are workflow.RunOutput of the final steps' outputs, which the engine keeps with the run's tasks; a client that never registered the workflow rebuilds the steps from the run's shape, so with several final steps the result's keys come in name order there, and a token the engine withholds payloads from cannot read a result. RunQuery.DueFrom and DueTo (CapDueQuery) filter on RunInfo.RunAt, the RunAt of a delayed start (or the time Reschedule moved it to), kept after it fires; the filter is applied to each listed page, so a page can be short. Concurrency.FIFO (CapKeyFIFO) needs nothing of the driver: the engine queues a key's waiting runs by insert order (the concurrency slot's sort_id), proven for 40 runs over two keys by TestKeyFIFOLive and by the conformance case, and a run's Priority still goes first.
Reschedule (workflow.RunRescheduler) creates the delayed start again at the new time with the same token, payload, labels and priority, then deletes the old one, so the RunRef is unchanged; if the delete fails the copy is removed again and the error says whether that worked. A start due within 15 seconds is refused with ErrWrongState: the engine loads a scheduled run into its ticker about five seconds before it is due and then fires it at the loaded time whatever a later change says. The ScheduledID still names the first RunAt, and new tokens are UUIDv7, so Status, Cancel and Runs find a run moved earlier by searching from when it was scheduled; for about a second after such a run fires, while the engine's read side has not listed it, Status may report it cancelled.
Config.NodeID (workflow.DefaultNodeID by default) is StepInfo.Node and is set as a label on every worker the client registers. Nodes (workflow.NodeLister) lists the nodes whose labelled workers the engine shows ACTIVE, with their queues, slots and heartbeats; enqueue-only clients register no worker and are not listed. CapNodeIdentity is not claimed: RunInfo.AttemptedBy and AttemptError.Node need the worker of each attempt, and the run details' task events, which would hold it, come back empty on hatchet-lite v0.110.5. Health (workflow.HealthChecker) is live until a recycle stops the client, and ready while it is started, the engine answers its version call and every worker it registered is ACTIVE. Enqueue refuses a payload over Config.PayloadLimit (DefaultPayloadLimit, 1.5 MiB, sized for the engine's default 4 MiB gRPC message, since a JSON object payload is stored about 2.34 times over) with a *workflow.PayloadTooLargeError (workflow.PayloadLimiter), reports a new run or a duplicate key through workflow.WithEnqueueOutcome, and a failed run's RunEvent.Err is a *workflow.RunError naming the run, workflow, failed step and its attempts, whose text (and RunInfo.Error) is the step's own message.
Not claimed in batch 5: CapSnooze (the engine counts every failed attempt; workflow.Snooze is an ordinary error here), CapHeartbeat (the engine's RefreshTimeout adds to a task's one execution deadline, so it cannot fail an attempt for a missed beat while keeping Step.Timeout; workflow.Heartbeat is a no-op), CapRunTimeout, CapScheduleActive and CapScheduleKeep (above). The client does not implement workflow.StepCanceller (the engine's cancel ends a task as cancelled, while the contract has the handler's return decide the step and a failure end the run without a retry), workflow.LifecycleNotifier (the engine has no insert or start callback) or a progress store (workflow.ReportProgress returns an UnsupportedError).
Hatchet engine quirks (hatchet-lite v0.110.5, SDK v0.107.1) and the workaround for each:
- Scheduled runs store triggerAt without a zone, so a local time fires hours off: Enqueue sends RunAt in UTC.
- A scheduled-run record is deleted when it fires: the run carries a token in its metadata and the ScheduledID encodes it, so Status and Cancel find the fired run by that token.
- A run is missing from the REST read side for about a second after RunNoWait: Status retries a 404 for up to Config.VisibilityLag (5s by default) before ErrNotFound.
- Between a failed attempt and its retry the engine shows the task, and often the run, as FAILED for tens of milliseconds, then QUEUED again; during a retry backoff the task is QUEUED. Taken at face value a run that later succeeds would read failed (Status, Runs, Retry, the run-finished hook, BatchSource, a batch's Catch). One function judges every FAILED read (failed.go): it is final when the failure carries the driver's marker (every failure the driver makes non-retryable: Permanent, Discard, NoRetryOn, a wait timeout, an unreadable input), when the task used the Retries the engine registered for it, or when the task failed more than Config.FailedSettle ago (5s by default); otherwise the run reads Running or Queued, as between attempts. The marker is stripped from RunInfo.Error and RunEvent.Err, so users read the step's own text, but Hatchet's dashboard shows it. The REST task event list, which would name a retry, returned no rows on hatchet-lite v0.110.5 and is not used.
- Runs whose failures were stored before the marker existed have no marker: a Permanent failure of such a run with retries left reads Queued for Config.FailedSettle after it failed, then Failed, as before. It matters only for runs failing while the new driver rolls out, and ends when those runs end.
- A task held by a rate limit waits outside the scheduler until its bucket's next window, then is requeued, and the scheduler fails it, with an empty error and no attempt, once its schedule timeout (counted from creation) has passed. The engine's default is five minutes, so a run a busy limit held past that failed (the ratelimit conformance group passed 1 of 5 times before the fix, 5 of 5 after). Every step and the batch coordinator declare Config.ScheduleTimeout (a day by default) instead (TestScheduleTimeoutLive). The on-failure task takes no options in the SDK and keeps the five minutes.
- The SDK panics when a worker is stopped before it registers: Start returns only once the engine lists the worker.
- Delete endpoints answer 204, which the SDK reports as an error: a 204 is treated as success.
- A DAG wait condition exposes neither the event payload nor which condition fired: wait steps run as durable tasks (WaitFor).
- A durable wait filtered by a CEL expression never resumed once a non-matching event of the same name arrived in its look-back window: the correlation key is part of the event key and scope instead, and events count from the run's enqueue time.
- A step whose worker process dies is reclaimed when the step's execution timeout expires and then retried (it needs a retry left): the live crash case abandons a client mid-step with a 15s timeout and the step reruns about 28s later. In production the delay is the step's timeout (Config.StepTimeout, 24h by default), so set Step.Timeout near the step's real duration. Reclaim through missed worker heartbeats is not tested.
- The engine rejects a rate limit without Key even when KeyExpr is set ("Validation failed" at worker registration): every limit carries a Key naming it.
- Runs are listed by creation time only, with no "finished since" filter: the run-end notifier tracks unfinished runs between polls.
- A run's stored input nests the trigger input under "input": Runs reads the payload from there.
- Workflows reach the engine only through a worker's registration: Enqueue of a workflow of a queue that no process has ever served fails, because the engine does not know the workflow yet.
- The workflow list's name filter matches by substring, and the SDK's Workflows().Get returns the first row: pausing "x" paused "x_ctl" in a live test. The driver compares every row's full name.
- Workflow pause takes a mandatory queue TTL, after which the engine CANCELS the held runs (checked live with a 10s TTL in TestPauseLive), and an empty or unparsable TTL silently becomes five minutes: Pause sends pauseQueueTTL, a hundred years, pinned by TestPauseRequestTTL; TestPauseHoldLive (WFTEST_PAUSE_TTL=1) holds a run for six minutes and then runs it.
Every claim in Capabilities is proven by the shared workflowtest suite against a real engine (live_test.go, `task test:live`).
Index ¶
- Constants
- func DefaultBatchCoordinatorRetry() *workflow.RetryPolicy
- type API
- type Client
- func (c *Client) BatchSource(store batch.Store, interval time.Duration, log *slog.Logger) batch.Source
- func (c *Client) Cancel(ctx context.Context, ref workflow.RunRef) error
- func (c *Client) CancelAll(ctx context.Context, q workflow.RunQuery) (int, error)
- func (c *Client) CancelGroup(ctx context.Context, group string, parent workflow.RunRef, since time.Time) error
- func (c *Client) Capabilities() workflow.Caps
- func (c *Client) Done() <-chan struct{}
- func (c *Client) Drain(ctx context.Context) error
- func (c *Client) DrainWithReport(ctx context.Context) (workflow.DrainReport, error)
- func (c *Client) Enqueue(ctx context.Context, job workflow.Job) (workflow.RunRef, error)
- func (c *Client) FindBatchRuns(ctx context.Context, id batch.ID, since time.Time) (map[int]workflow.RunRef, error)
- func (c *Client) Health(ctx context.Context) workflow.HealthReport
- func (c *Client) Native() *hatchetsdk.Client
- func (c *Client) Nodes(ctx context.Context) ([]workflow.NodeInfo, error)
- func (c *Client) OnRunFinished(fn workflow.RunFinishedFunc) error
- func (c *Client) Pause(ctx context.Context, t workflow.PauseTarget) error
- func (c *Client) PauseSchedule(ctx context.Context, t workflow.ScheduleTarget) error
- func (c *Client) Paused(ctx context.Context) ([]workflow.PauseTarget, error)
- func (c *Client) PayloadLimit() int
- func (c *Client) Register(wf workflow.Workflow) error
- func (c *Client) Reschedule(ctx context.Context, ref workflow.RunRef, at time.Time) (workflow.RunRef, error)
- func (c *Client) Result(ctx context.Context, ref workflow.RunRef) ([]byte, error)
- func (c *Client) Resume(ctx context.Context, t workflow.PauseTarget) error
- func (c *Client) ResumeSchedule(ctx context.Context, t workflow.ScheduleTarget) error
- func (c *Client) Retry(ctx context.Context, ref workflow.RunRef) (workflow.RunRef, error)
- func (c *Client) RunGroup(ctx context.Context, group string, jobs []workflow.Job) ([]workflow.RunRef, error)
- func (c *Client) Runs(ctx context.Context, q workflow.RunQuery) (workflow.RunPage, error)
- func (c *Client) Schedule(ctx context.Context, s workflow.Schedule) (workflow.ScheduleID, error)
- func (c *Client) Schedules(ctx context.Context) ([]workflow.ScheduleInfo, error)
- func (c *Client) Signal(ctx context.Context, ev workflow.Event) error
- func (c *Client) SpawnGroup(ctx context.Context, group string, jobs []workflow.Job) (workflow.RunRef, error)
- func (c *Client) SpawnsGroups() bool
- func (c *Client) Start(ctx context.Context) error
- func (c *Client) Status(ctx context.Context, ref workflow.RunRef) (workflow.Status, error)
- func (c *Client) StopCause() error
- func (c *Client) TriggerSchedule(ctx context.Context, id workflow.ScheduleID) (workflow.RunRef, error)
- func (c *Client) Unschedule(ctx context.Context, id workflow.ScheduleID) error
- type Config
Constants ¶
const ( // DefaultWorkerName is the worker name shown in Hatchet's dashboard. DefaultWorkerName = "ubgo-workflow" // DefaultStepTimeout stands in for "no timeout": Hatchet always // applies an execution timeout, and its server default is short // enough to kill a long job minutes in. DefaultStepTimeout = 24 * time.Hour // DefaultScheduleTimeout is how long a step may wait for a worker // when Config.ScheduleTimeout is zero: a day, like DefaultStepTimeout, // in place of the engine's five minutes (see Config.ScheduleTimeout). DefaultScheduleTimeout = 24 * time.Hour // DefaultIdempotencyMaxHold caps how long a key stays held when its // run never finishes; normally the key is released when the run is // terminal (Hatchet's STATUS method). DefaultIdempotencyMaxHold = 24 * time.Hour )
Defaults for Config fields left zero.
const ( // DefaultRegisterTimeout is how long Start waits for the engine to // list the new worker. An engine that starts slowly (a cold pod on // Kubernetes) may need longer. DefaultRegisterTimeout = 30 * time.Second // DefaultVisibilityLag is how long a run may be missing from the // engine's read side right after it was created (see // runDetailsLagged). It also sizes the run-finished notifier's list // overlap and how long a batch waits after its coordinator ended. DefaultVisibilityLag = 5 * time.Second )
Engine-facing tunings: defaults for the Config fields that override them.
const ( // DefaultGroupChunkRuns is the default and the largest // Config.GroupChunkRuns: the SDK's own bulk-trigger run limit. DefaultGroupChunkRuns = 1000 // DefaultGroupChunkBytes is the default and the largest // Config.GroupChunkBytes: 2 MiB of JSON stays under the SDK's 3 MiB // protobuf split and the engine's default 4 MiB gRPC message. DefaultGroupChunkBytes = 2 << 20 )
Chunk limits of RunGroup. The SDK splits a bulk trigger itself (1000 runs, 3 MiB under the engine's default 4 MiB gRPC message), but when one of its chunks fails it returns the IDs of the others without their positions, so a failed trigger could not be mapped back to its jobs. RunGroup therefore makes chunks the SDK never splits again: at most Config.GroupChunkRuns runs, and at most Config.GroupChunkBytes of JSON input and metadata, which by default stays under the SDK's proto-size limit with room for framing. A single larger job goes alone, as in the SDK.
const ( // DefaultRunFinishedPoll is how often the notifier asks the engine for // run ends. DefaultRunFinishedPoll = time.Second // DefaultRunFinishedWindow is how old an unfinished run may be and // still be watched: runs created earlier are not tracked. DefaultRunFinishedWindow = 24 * time.Hour )
Defaults for the notifier's Config fields.
const DefaultBatchCoordinatorMaxInput = 2 << 20
DefaultBatchCoordinatorMaxInput is the default Config.BatchCoordinatorMaxInput: one trigger's JSON under the engine's default 4 MiB gRPC message, with room for framing.
const DefaultBatchCoordinatorTimeout = 5 * time.Minute
DefaultBatchCoordinatorTimeout is the default Config.BatchCoordinatorTimeout: starting a thousand children is one bulk call, so a batch of tens of thousands of jobs starts well within it.
const DefaultFailedSettle = 5 * time.Second
DefaultFailedSettle is how long after a failed task finished a FAILED read with retries left and no marker is still taken as the moment before a retry; see Config.FailedSettle. The moment seen live is tens of milliseconds; the margin covers clock skew between the engine and this process, since the task's finish time is the engine's.
const DefaultPayloadLimit = 3 << 19
DefaultPayloadLimit is Config.PayloadLimit's default: 1.5 MiB. The engine and the SDK carry a run's input in one gRPC message of at most 4 MiB by default (SERVER_GRPC_MAX_MSG_SIZE), and the driver stores a JSON object payload twice, as fields and as base64 (encodeInput), about 2.34 times its size; 1.5 MiB leaves room for that and the run metadata.
const DefaultScheduleHistoryWindow = 7 * 24 * time.Hour
DefaultScheduleHistoryWindow is how far back Schedules looks for a trigger's last fire when Config.ScheduleHistoryWindow is zero. The engine lists runs only from a start time, so a trigger whose last fire is older than the window reads as never fired; a week covers every cron that fires at least weekly.
Variables ¶
This section is empty.
Functions ¶
func DefaultBatchCoordinatorRetry ¶
func DefaultBatchCoordinatorRetry() *workflow.RetryPolicy
DefaultBatchCoordinatorRetry returns the default Config.BatchCoordinatorRetry, a fresh value each call: 10 retries at 2^n seconds capped at two minutes, which gives an engine outage about ten minutes before a coordinator fails. A retry starts the children again with the same child keys, and the engine returns the existing child for a key it has seen (checked live: same run IDs), so no job runs twice however many retries are allowed.
Types ¶
type API ¶
type API struct {
// Client is the engine's REST client, authenticated as the SDK
// client is (a bearer token for the same tenant).
Client *rest.ClientWithResponses
// TenantID is the tenant the token belongs to.
TenantID string
}
API is the engine's REST API for the calls SDK v0.107.1 has no method for and whose REST client it does not expose: workflow pause and resume, and enabling or disabling a cron trigger. Set Config.API to turn on CapPauseWorkflow and CapSchedulePause; without it both are refused. Build one with NewAPI, or set Client and TenantID yourself (for a custom HTTP client or proxy).
func NewAPI ¶
NewAPI builds an API from an engine API token, the value of HATCHET_CLIENT_TOKEN. The tenant comes from the token's sub claim and, when serverURL is empty, the REST address from its server_url claim, as the SDK itself does; pass serverURL when the engine is reached at another address (HATCHET_CLIENT_SERVER_URL). The token is only decoded, not verified: the engine verifies it on every call.
type Client ¶
type Client struct {
// contains filtered or unexported fields
}
Client is the Hatchet driver: a workflow.Client whose runs, waits, sleeps, retries, crons and concurrency are Hatchet's own.
Nothing connects until Start: Register only records definitions, and Start builds the SDK client (unless Config.Client was given), declares every workflow, registers the worker and returns once it is running. The worker stops when Start's ctx is cancelled.
func (*Client) BatchSource ¶
func (c *Client) BatchSource(store batch.Store, interval time.Duration, log *slog.Logger) batch.Source
BatchSource returns a batch.Source that learns a batch's job ends from the engine's run list: one Runs().List call per unfinished batch each interval, filtered by the batch's batch.LabelBatch label and the terminal statuses, instead of batch.PollSource's one Client.Status call per unfinished job. Pass it to batch.WithCompletionSource, with batch.HookSource() for speed when wanted.
store must be the Store the Manager uses: the source reads the unfinished batches from it. A job whose ref holds only a ScheduledID (a delayed start that has not fired) has no engine run to list, so the source asks Client.Status for that one job; that is how a delayed start cancelled before it fired is found. A List error is logged and retried on the next tick, never guessed into a status. Each tick also calls Manager.RetryCallbacks, as batch.PollSource does.
A nil log means Config.Logger. Ticks come from Config.Clock, so a test on a fake clock drives them with Advance. Its Attach fails with workflow.ErrInvalid for an interval that is not positive.
func (*Client) Cancel ¶
Cancel stops a run, or deletes a scheduled run that has not fired. A finished or already cancelled run is left alone and nil returned.
func (*Client) CancelAll ¶
CancelAll cancels every run matching q (workflow.BulkCanceller): by default the delayed starts that have not fired and the queued runs. It lists every match first and then cancels, so cancelling cannot shift the listing under it. Delayed starts are cancelled one by one as Cancel does (each reported to run-finished observers by this process); runs in bulk cancel calls of up to 500. It returns how many it asked the engine to cancel; a run that finished between the listing and the cancel is counted but left as it ended.
func (*Client) CancelGroup ¶
func (c *Client) CancelGroup(ctx context.Context, group string, parent workflow.RunRef, since time.Time) error
CancelGroup implements batch.GroupSpawner: it cancels the parent, when known, then every run labelled with group created since since, in one bulk cancel by metadata. The engine does not cancel children with their parent (checked live on hatchet-lite v0.110.5, for a running and for a completed parent), so the label is what reaches them.
func (*Client) Capabilities ¶
Capabilities reports what the driver proves on a real engine.
func (*Client) Done ¶
func (c *Client) Done() <-chan struct{}
Done is closed once every worker of this client has stopped: after Drain, or after the Start ctx was cancelled. Before Start it is open.
func (*Client) Drain ¶
Drain stops this client gracefully (workflow.Drainer): it pauses every worker Start registered, so the engine assigns them no new step attempt, waits until the step and on-failure handlers running here return or ctx ends, then stops the workers as cancelling the Start ctx would. Afterwards every call that needs the engine (Enqueue, Status, Runs, ...) returns ErrNotStarted; query runs from another client. Runs it did not finish stay on the engine (queued, or running until the engine reclaims an attempt that was cut off, see doc.go), and any other worker serving the workflow takes them. Wait and sleep steps do not hold it up: they are durable and resume on any worker. It returns nil when every handler finished and ctx's error otherwise.
func (*Client) DrainWithReport ¶
DrainWithReport is Drain that also returns what the drain finished and cut off (workflow.ReportingDrainer). What the engine allows: this client sees only the handlers it runs, so FinishedSteps counts handlers that returned during the drain and each cut-off handler is listed with DrainEngineOwned, because whether the engine requeues it (and when) is the engine's decision, not something this driver can promise. A worker is not told when a run ends, so FinishedRuns is read back from the engine: the runs of the handlers that returned are looked up (for up to drainRunSettle) and those the engine reports terminal are counted; a run that only reached terminal through another worker's step is not, because this client never saw it. A second call, concurrent or after Done, waits for the first drain and returns its report and its error.
func (*Client) Enqueue ¶
Enqueue starts a run now (RunNoWait) or at job.RunAt (a Hatchet scheduled run, whose RunRef holds only the ScheduledID until it fires). A key held by an unfinished run returns that run instead of starting another.
A label under a reserved prefix is refused unless ctx allows it (workflow.CheckJobLabels): only the package owning a prefix may set it.
func (*Client) FindBatchRuns ¶
func (c *Client) FindBatchRuns(ctx context.Context, id batch.ID, since time.Time) (map[int]workflow.RunRef, error)
FindBatchRuns implements batch.RunFinder: one paged run-list call, filtered by the batch's batch.LabelBatch label and no status, so a resumed dispatch learns which jobs the dead process started (the runs of a bulk trigger whose refs were never saved). A delayed start that has not fired is a schedule, not a run, and is not listed; the batch package never restarts such a job on this answer alone.
func (*Client) Health ¶
func (c *Client) Health(ctx context.Context) workflow.HealthReport
Health reports the client's health (workflow.HealthChecker). Live is false only once a recycle (Config.Recycle) has stopped the client, since the process should then be replaced; a requested stop leaves it live. Ready needs the client started and not stopped, the engine answering (its version call, within ctx), and, for a worker client, every worker it registered listed ACTIVE by the engine; an enqueue-only client checks the engine alone.
func (*Client) Native ¶
func (c *Client) Native() *hatchetsdk.Client
Native returns the Hatchet SDK client, the one escape hatch for anything the workflow API does not cover. It is nil before Start when Config.Client was not given. Do not start a second worker on it.
func (*Client) Nodes ¶
Nodes lists the nodes whose workers the engine reports ACTIVE (workflow.NodeLister): one NodeInfo per Config.NodeID, merging the node's workers (one per served queue). A worker of another app, or of a driver version that set no node label, is left out, as are enqueue-only clients, which register no worker. A node drops out once the engine marks its workers inactive, which it does when their heartbeats stop (a crash) or they stop. Workers is the sum of the workers' slots and Running the attempts the engine shows on them; StartedAt is when the first worker registered and LastSeen its newest heartbeat, both on the engine's clock. Mode is NodeFull: a worker client also enqueues and schedules, and the engine fires crons itself.
func (*Client) OnRunFinished ¶
func (c *Client) OnRunFinished(fn workflow.RunFinishedFunc) error
OnRunFinished registers fn to observe the end of every run of a registered workflow, best-effort (see notify.go). It returns ErrAlreadyStarted after Start.
func (*Client) Pause ¶
Pause holds the runs of one workflow (workflow.Pauser) through the engine's workflow pause: queued steps move aside and are not assigned, running steps finish, Enqueue still works, and crons and delayed starts that come due are queued and held too. The pause is stored on the engine, so every worker obeys it. Needs Config.API. PauseQueue and PauseAll are refused: a queue is not a thing the engine pauses, and the engine has no pause of everything (see errPauseAll).
func (*Client) PauseSchedule ¶
PauseSchedule disables one cron trigger (workflow.SchedulePauser); declared crons included. Ticks that fall due while it is disabled are never fired, whatever the misfire policy: the engine does not catch up (Hatchet accepts no misfire policy). PauseSchedule(All) is refused (see errPauseAll). An unknown ID returns ErrNotFound. Needs Config.API.
func (*Client) Paused ¶
Paused lists the paused workflows of this client's namespace, by name (workflow.Pauser). It reads the engine, so it sees pauses made by any process or in the dashboard.
func (*Client) PayloadLimit ¶
PayloadLimit is the largest Job.Payload Enqueue accepts, in bytes (workflow.PayloadLimiter): Config.PayloadLimit.
func (*Client) Register ¶
Register checks wf against what Hatchet can honour (the capabilities and the exact fields, see doc.go) and records it; it returns an error wrapping workflow.ErrUnsupported for anything the driver would otherwise have to ignore or change. Its Hatchet declaration is built in Start.
func (*Client) Reschedule ¶
func (c *Client) Reschedule(ctx context.Context, ref workflow.RunRef, at time.Time) (workflow.RunRef, error)
Reschedule moves a delayed start that has not fired to at (workflow.RunRescheduler); a zero or past at means now. The RunRef, payload, labels, priority and key are kept: the driver creates the scheduled run again at the new time with the same token and metadata, plus the new due time (metaRunAt), then deletes the old one, so the run's ScheduledID is unchanged. A run that has fired, or is due within rescheduleGuard (15s), returns ErrWrongState; an unknown or cancelled one ErrNotFound.
func (*Client) Result ¶
Result returns the result of a succeeded run (workflow.ResultReader): workflow.RunOutput of its final steps' outputs, which the engine keeps with the run's tasks for as long as it keeps the run. A run that has not succeeded returns ErrWrongState, an unknown one ErrNotFound.
func (*Client) Resume ¶
Resume releases a paused workflow (workflow.Pauser); the engine queues its held steps again. Resume(PauseAll()) resumes every paused workflow of this client's namespace. Resuming a workflow that is not paused is a no-op.
func (*Client) ResumeSchedule ¶
ResumeSchedule enables a cron trigger again; ResumeSchedule(All) enables every disabled trigger of this client's namespace. It fires from the next tick on.
func (*Client) Retry ¶
Retry re-runs a failed run from the steps that failed (workflow.RunRetrier), through Hatchet's replay of those tasks: the engine replays a task and every task after it, and keeps the output of the steps that succeeded, so they do not run again. Every failed or cancelled step task of the run is replayed, which covers a branch the failure cancelled. The run keeps its RunID.
The idempotency key: Retry refuses with ErrDuplicate when another unfinished run carries the same caller key. The engine does not take the key again for a replayed run, so an Enqueue with that key while the retried run is unfinished starts a second run; this is a known gap of the Hatchet driver.
func (*Client) RunGroup ¶
func (c *Client) RunGroup(ctx context.Context, group string, jobs []workflow.Job) ([]workflow.RunRef, error)
RunGroup implements batch.GroupRunner as Hatchet bulk triggers: one RunMany call per chunk of jobs of the same workflow, in the order each workflow first appears, so a batch of N jobs costs about N/1000 calls instead of N. Each run carries exactly what Enqueue stamps (input envelope, labels, idempotency metadata, priority), so Status, Cancel and the run-finished hook treat it like an enqueued run; group itself is not stamped, because the jobs' labels already name their batch.
Every job is checked before anything starts, and a refusal returns nil refs with Enqueue's error for that job. When a chunk fails, the chunks already started keep their refs, the rest are zero, and the error is returned, so the caller can cancel what started.
func (*Client) Runs ¶
Runs lists runs (workflow.RunQuerier), newest first. It merges two engine listings: runs (immediate ones, and delayed starts that fired) and scheduled runs that have not fired yet (StatusScheduled). Each run is read in full (one detail read per run listed), so the page carries the failed step, the attempt count and the waiting status exactly.
On Hatchet, EnqueuedAt is when the engine created the run: for a delayed start that fired, its fire time; for one that has not, when it was scheduled. From and To filter on that time. Priority is Job.Priority as Enqueue stamped it (zero for runs enqueued before the driver stamped it). The cursor is an offset into each listing, so runs created while a caller pages can make a later page repeat a run of an earlier one. Runs of workflows that are not registered on this client are listed too when Enqueue of this driver created them (the workflow name is stamped on the run), so any process can query.
func (*Client) Schedule ¶
Schedule creates a Hatchet cron trigger. Hatchet evaluates crons in UTC only, so any other Location is refused rather than shifted.
A named schedule is one schedule per (Workflow, Name), as on the other drivers, so an app can call Schedule on every boot: when the workflow already has a cron trigger of that name with the same expression, input and labels, nothing is created. When the definition changed, the old trigger is deleted and a new one created (Hatchet cannot edit a trigger's expression, input or labels in place), keeping the old one's pause when Config.API is set. Either way the returned ID is the same: a named schedule's ID names the (Workflow, Name) pair ("named:" plus an encoding of both), not the engine trigger, and Unschedule, PauseSchedule, ResumeSchedule and TriggerSchedule find the trigger by it. A trigger of that name made before this ID form is replaced on the next Schedule call. An unnamed schedule always creates a new trigger, whose ID is the engine's.
func (*Client) Schedules ¶
Schedules lists the engine's cron triggers (workflow.ScheduleLister): the declared ones (Workflow.Cron, Declared true, named by their expression) and the ones Schedule made, of every workflow in this client's namespace, whichever process declared or made them. A trigger made in Hatchet's dashboard or by another client of the namespace is listed too: the engine does not record who made a trigger. ID is the named schedule's ID for a trigger made by a named Schedule (stable across replacements), else the engine's trigger ID; a declared trigger gets a new one when its workflow is declared again with a changed definition. Location is always UTC, the only zone Hatchet runs crons in; Next is computed with the core cron parser on Config.Clock. A trigger disabled in the dashboard is listed with Paused true and no Next. Last (workflow.CapScheduleHistory) is read from the engine's run list, not stored by the driver: the newest run the engine fired for the trigger or TriggerSchedule started for it within Config.ScheduleHistoryWindow, with its status as Status reads it; this costs two run-list calls and one run read per trigger.
func (*Client) Signal ¶
Signal pushes ev as a Hatchet event. Wait steps on ev.Name whose Key equals ev.Key resume with ev.Payload.
func (*Client) SpawnGroup ¶
func (c *Client) SpawnGroup(ctx context.Context, group string, jobs []workflow.Job) (workflow.RunRef, error)
SpawnGroup implements batch.GroupSpawner: it triggers the coordinator for group with every job prepared exactly as Enqueue would, so each child carries the same input envelope, labels, idempotency and workflow stamps as an enqueued run, and Status, the run-finished hook and BatchSource treat it as one. The trigger is idempotent per group for coordinatorKeyTTL: a second call returns the first coordinator.
It returns an error wrapping workflow.ErrUnsupported, before anything starts, when the jobs' input and metadata exceed Config.BatchCoordinatorMaxInput (the coordinator's own input would near the engine's message limit), so Dispatch starts them as bulk triggers instead; and workflow.ErrInvalid when Config.BatchCoordinator is off, because no worker of this client runs the coordinator then.
func (*Client) SpawnsGroups ¶
SpawnsGroups implements batch.GroupSpawner: true when Config.BatchCoordinator is set.
func (*Client) Start ¶
Start checks the Config tunings, connects, declares the workflows and starts the worker. The worker runs until ctx is cancelled.
func (*Client) Status ¶
Status reports where the run is. A run whose only unstarted work is a wait or sleep step, with nothing queued or running, is StatusWaiting.
func (*Client) StopCause ¶
StopCause reports why this client's workers stopped (workflow.StopCauser): the *workflow.RecycleError when a Config.Recycle limit stopped them, nil while they run and after every requested stop (the Start ctx ended, Drain). It is final once Done is closed.
func (*Client) TriggerSchedule ¶
func (c *Client) TriggerSchedule(ctx context.Context, id workflow.ScheduleID) (workflow.RunRef, error)
TriggerSchedule starts a run of the cron trigger id now (workflow.ScheduleTriggerer): the trigger's workflow, with the trigger's input and labels plus workflow.TriggeredLabelsAt, through the same path as Enqueue. A declared trigger (Workflow.Cron) has an engine ID too, as Schedules lists it, and is triggered the same way. It works on a disabled trigger and leaves the next cron fire unchanged. The overlap policy needs no check: OverlapSkip is refused at Register and Schedule. An unknown ID, or one of another namespace, returns workflow.ErrNotFound.
func (*Client) Unschedule ¶
Unschedule deletes a cron trigger (every trigger of the name, for the ID of a named schedule). An empty or unknown ID is a no-op.
type Config ¶
type Config struct {
// Client is an SDK client the app already built. When nil, Start
// builds one with ClientOptions.
Client *hatchetsdk.Client
// ClientOptions are passed to hatchet.NewClient when Client is nil
// (token, host and port, namespace, TLS).
ClientOptions []hatchetsdk.ClientOpt
// Namespace must equal the namespace the SDK client uses (option or
// HATCHET_CLIENT_NAMESPACE), so idempotency keys are scoped by the
// name the engine knows each workflow by.
Namespace string
// WorkerName is the worker's name in Hatchet's dashboard.
WorkerName string
// Slots caps how many steps this worker runs at once. Zero keeps the
// SDK default. With Queues it is the slots of a queue whose Workers is
// zero.
Slots int
// Queues are the named queues this client's workers serve
// (workflow.QueueConfig). Empty means one worker serving every
// registered workflow, whatever its queue. Otherwise Start starts one
// Hatchet worker per listed queue that has registered workflows, each
// with Workers slots, and a workflow whose queue is not listed gets no
// worker here: its runs wait for a process that serves that queue. A
// queue is bound to a workflow on Hatchet (Workflow.Queue), because a
// run goes to whichever worker registered its workflow, so a Job.Queue
// other than the workflow's is refused (no CapQueueOverride), and a
// non-zero Priority is refused at Start (Hatchet cannot order one
// worker's queues against another's).
Queues []workflow.QueueConfig
// API is the engine's REST API (NewAPI), for workflow pause and cron
// enable, which SDK v0.107.1 cannot reach. Set, the client also
// declares CapPauseWorkflow and CapSchedulePause and implements
// workflow.Pauser and workflow.SchedulePauser; nil refuses both.
API *API
// StepTimeout is the execution timeout of a step whose Workflow and
// Step leave Timeout zero.
StepTimeout time.Duration
// ScheduleTimeout is how long a step may wait to be scheduled,
// counted from its creation. The engine fails a waiting step past it,
// with no error message and no attempt made; a step a rate limit
// holds is checked each time its bucket's window lets it back into
// the queue. The engine's own default is five minutes, which failed
// runs a rate limit held for about six (TestScheduleTimeoutLive, the
// ratelimit conformance group), against the core contract that a held
// run is delayed, never dropped; so every step declares this instead.
// Zero means DefaultScheduleTimeout. A run held longer than it still
// fails, so set it above the longest wait you expect.
ScheduleTimeout time.Duration
// IdempotencyMaxHold caps how long an idempotency key stays held.
IdempotencyMaxHold time.Duration
// Middleware wraps every step handler, outermost first.
Middleware []workflow.Middleware
// RunFinishedPoll is how often OnRunFinished observers' notifier polls
// the engine. Zero means DefaultRunFinishedPoll.
RunFinishedPoll time.Duration
// RunFinishedWindow is the oldest an unfinished run may be and still
// be watched for its end. Zero means DefaultRunFinishedWindow.
RunFinishedWindow time.Duration
// RunFinishedCatchUp makes Start replay the ends of runs created this
// long before it, so a restarted process delivers ends it missed while
// down (again, if an earlier process delivered them). Zero replays
// nothing older than Start.
RunFinishedCatchUp time.Duration
// BatchCoordinator makes batch.Manager.Dispatch start an eligible
// batch (no RunAt, no IdempotencyKey, at most about 2 MiB of input) as
// one coordinator run whose task starts every job as a child run,
// instead of bulk triggers from the dispatching process (see
// SpawnGroup). Start then registers the coordinator workflow on this
// worker. The batch's Manager needs this client's BatchSource, which
// finds the children by label; every process sharing the batch store
// should set it alike, so a resumed dispatch finds the coordinator.
BatchCoordinator bool
// BatchCoordinatorTimeout bounds one attempt of the coordinator's
// task. It is also how long a coordinator whose worker died waits
// before the engine retries it elsewhere, so keep it near the time
// starting the largest batch takes. Zero means
// DefaultBatchCoordinatorTimeout.
BatchCoordinatorTimeout time.Duration
// BatchCoordinatorRetry is the coordinator task's retry policy. Nil
// means DefaultBatchCoordinatorRetry (10 retries, 2^n seconds capped
// at two minutes: about ten minutes of engine outage). Its Backoff
// must fit Hatchet's factor^n-seconds formula (see Retry.Backoff in
// doc.go) and NoRetryOn must be nil, or Start refuses it.
BatchCoordinatorRetry *workflow.RetryPolicy
// BatchCoordinatorMaxInput caps the JSON input and metadata one
// coordinator trigger carries; a larger batch is started as bulk
// triggers instead. Zero means DefaultBatchCoordinatorMaxInput, which
// fits the engine's default 4 MiB gRPC message; raise it only when
// the engine's message limit was raised.
BatchCoordinatorMaxInput int
// GroupChunkRuns caps the runs of one bulk trigger (RunGroup and the
// coordinator). Zero means DefaultGroupChunkRuns; Start refuses more
// than that, because the SDK splits a larger call itself and then
// cannot say which runs of it failed.
GroupChunkRuns int
// GroupChunkBytes caps the JSON input and metadata of one bulk
// trigger. Zero means DefaultGroupChunkBytes; lower it for an engine
// whose gRPC message limit is below the default 4 MiB. Start refuses
// more than the default, for the same reason as GroupChunkRuns (the
// SDK splits at 3 MiB of protobuf whatever the engine accepts).
GroupChunkBytes int
// RegisterTimeout is how long Start waits for the engine to list the
// worker. Zero means DefaultRegisterTimeout.
RegisterTimeout time.Duration
// VisibilityLag is how long the engine's read side may trail a write
// (see DefaultVisibilityLag). Zero means DefaultVisibilityLag.
VisibilityLag time.Duration
// FailedSettle is how long after a task failed a FAILED read is still
// taken as the moment before its retry when the failure carries no
// driver marker and the task has retries left (see failed.go: the
// engine shows FAILED for a moment between a failed attempt and its
// retry). The run reads Queued or Running meanwhile. Zero means
// DefaultFailedSettle.
FailedSettle time.Duration
// Recycle stops this client's workers once a limit is reached (a
// number of finished handler attempts, a time since Start, a heap
// size), so a supervisor can start a fresh process: the workers are
// drained as Drain does with no deadline (paused, the handlers running
// here finish, then stopped), Done closes and StopCause returns the
// *workflow.RecycleError; workflow.Serve wraps that into one call. On-
// failure handlers count as attempts; durable wait and sleep steps do
// not, since they hold no worker. Runs left on the engine go to any
// other worker serving their workflow. The zero value never recycles.
Recycle workflow.Recycle
// Logger receives the errors of the driver's background work (the
// run-finished notifier's polls, observer panics, worker stop) and of
// BatchSource when it is given no logger. Nil discards them.
Logger *slog.Logger
// Clock is the time source of everything the driver decides locally:
// the run-finished notifier's poll ticker and its delivered-run
// window, BatchSource's ticker, and FinishedAt of a delayed start
// cancelled before it fired. Nil means clock.Real. Times compared
// with the engine's own timestamps (run creation and finish times,
// event look-back, the read-side lag deadlines, the worker
// registration wait) stay on the wall clock the engine shares,
// because a fake clock there would make the driver ask the engine
// about a time the engine has never seen.
Clock clock.Clock
// EnqueueOnly makes a client that starts runs but never runs them
// (workflow.CapEnqueueOnly), for web nodes beside separate worker
// processes: Enqueue, Status, Cancel, Signal, Schedule and the run
// and schedule listings work without Start, connecting on first use;
// Start starts no worker and no run-finished poller and returns nil,
// and when its ctx ends calls return ErrNotStarted as after any stop.
// Register is optional: a registered workflow's job gets that
// workflow's checks (idempotency policy, queue), an unregistered one
// only workflow.CheckJob, and the engine refuses a workflow that no
// worker has ever registered. OnRunFinished is refused, since the
// poller does not run. The engine version check of Start is skipped:
// the worker processes make it.
EnqueueOnly bool
// ScheduleHistoryWindow is how far back Schedules looks for each
// trigger's last fire (ScheduleInfo.Last). Zero means
// DefaultScheduleHistoryWindow.
ScheduleHistoryWindow time.Duration
// NodeID names this process in StepInfo.Node and Nodes. It travels
// to the engine as a label on every worker this client registers, so
// the engine's worker list maps back to it. RunInfo.AttemptedBy stays
// empty and workflow.CapNodeIdentity is not claimed: the engine keeps
// no per-attempt worker history the driver can read (see doc.go).
// Empty means workflow.DefaultNodeID() (hostname:pid).
NodeID string
// PayloadLimit is the largest Job.Payload, in bytes, Enqueue and
// RunGroup accept; a larger one is refused with a
// *workflow.PayloadTooLargeError before anything reaches the engine
// (workflow.PayloadLimiter). Zero means DefaultPayloadLimit, which
// fits the engine's default gRPC message size; raise it only for an
// engine and SDK configured with a larger one.
PayloadLimit int
}
Config configures a Client. The zero value builds an SDK client from the HATCHET_CLIENT_* environment variables at Start.