README
¶
Durable Queue — distributed execution plane
Demonstrates engine/queue, the durable, distributed execution plane added in
Phase P1. It decouples run intake from execution so that work survives
process restarts and scales horizontally across workers.
go run ./examples/durable_queue
No API keys or network needed — the "work" is simulated so the example is self-contained and deterministic to read.
What it shows
| Capability | In the example |
|---|---|
| Leased dequeue | Three workers claim runs under a time-bounded lease. On Postgres this uses FOR UPDATE SKIP LOCKED so workers claim disjoint runs concurrently; on SQLite an atomic UPDATE … RETURNING. |
| Durable sleep | The report run yields with Result{Sleep: 1s} and is re-delivered later. Sleeps are intentional yields — they do not consume the run's error-retry budget. |
| Park + signal (HITL) | The deploy run parks with Result{ParkSignal: "approve"} and only resumes when q.Signal(...) delivers the matching signal — the webhook-as-signal pattern for human approval. Signals are durable: one delivered before a run parks is retained and consumed on park (no lost-wakeup race). |
| Orphan recovery | A Reaper re-enqueues runs whose worker died mid-flight once their lease expires (a lost lease counts as one failed attempt). |
| Admission control | Config{MaxDepth, Policy} bounds intake; past capacity runs are parked (or rejected) instead of unbounded growth. |
How an Executor drives the queue
A worker calls your Executor for each claimed run; the returned Result
tells the queue what happens next:
type Result struct {
Err error // failed → retry with backoff until MaxAttempts, then fail
Sleep time.Duration // durably reschedule after the delay
ParkSignal string // park until this signal is delivered
Patch []byte // replace the run's persisted payload (state across yields)
}
The Patch field is how state survives a yield: the example stores a stage
in the run payload so the re-delivered run knows whether it has already slept or
been approved.
Production notes
- Swap
queue.DialectSQLiteforqueue.DialectPostgresand hand it a Postgres*sql.DBto run many workers across processes/pods. - Pair with
graph.NewQueuedExecutorto run full ChronosStateGraphs (agents, HITL approval nodes) on the queue instead of the simulated work here. - The idempotent outbox (
EnqueueOutbox/ClaimOutbox) delivers external side effects exactly once across retries and resumes; poison entries are dead-lettered after a configurable attempt cap.
See ../durable_hitl for the graph-level human-in-the-loop
flow and PLAN.md Phase P1-A for the full design.
Documentation
¶
Overview ¶
Command durable_queue demonstrates the Chronos durable, distributed execution plane (engine/queue) delivered in Phase P1.
It shows the four capabilities that make execution survive process restarts and scale across workers:
- Leased dequeue: multiple workers claim disjoint runs under a lease.
- Durable sleep: a run yields ("wait N, then continue") and is re-delivered later without burning its retry budget.
- Park + signal (human-in-the-loop): a run parks until an external signal (e.g. an approval webhook) wakes it.
- Orphan recovery: a reaper re-enqueues runs whose worker died mid-flight.
Run it:
go run ./examples/durable_queue
No API keys or network are required — the "work" is simulated so the example is fully self-contained and deterministic to read.
Source Files