dwarf

Dwarf is a standalone, embeddable workflow-orchestration engine for Go.

You describe a workflow as a graph of tasks; dwarf runs it — dispatching one task at a time per step, persisting state between steps in a SQL database, and handling the hard parts of durable orchestration: parallel fan-out/fan-in, conditional routing, retries with backoff, timed sleeps, subgraphs, and human-in-the-loop pauses.

Dwarf has no built-in transport. It doesn't know how your tasks are reached (a local function call, an RPC, a message bus) or where your graphs live. You wire it to your world through a few small dependency interfaces, and it handles scheduling, state, durability, and recovery.

g := workflow.NewGraph("Greet")
g.SetEndpoint("Hello", "http://example/hello") // node "Hello" dispatches to this endpoint URL
g.AddTransition("Hello", workflow.END)

proxy := engine.NewTestProxy()
proxy.HandleGraph("http://example/greet", g)
proxy.HandleTask("http://example/hello", func(ctx context.Context, f *workflow.Flow) error {
    f.SetString("greeting", "hello "+f.GetString("name"))
    return nil
})

eng := dwarf.NewEngineUnderTest(t.Name()) // SQLite in-memory, auto-dropped
defer eng.Shutdown(ctx)
eng.SetHost(proxy) // TestProxy implements the Host interface
eng.Startup(ctx)

_, out, _ := eng.Run(ctx, "http://example/greet", map[string]any{"name": "ada"}, nil) // Run returns (flowKey, outcome, err)
fmt.Println(out.State.GetString("greeting")) // hello ada

Why dwarf

• Durable by construction. Every step is checkpointed to SQL. A crashed worker's in-flight step is recovered by lease expiry; a flow can be inspected, resumed, forked, or continued days later.
• Parallelism that merges cleanly. Static and dynamic (forEach) fan-out run branches concurrently; fan-in merges their state with per-field reducers (append, add, union, merge, …).
• Human-in-the-loop. A task can Interrupt to park the flow for external input and Resume later — approvals, manual review, async callbacks.
• Fair and prioritized. Two-level scheduling: strict priority bands across the cluster, weighted fairness within a band so one tenant can't starve another.
• Scales horizontally. Run many replicas against sharded databases; they coordinate entirely through the databases they already share, with no message bus to run between them.
• OTEL-native observability. Structured logs (slog), 33 dwarf_* instruments, and distributed tracing, all through standard providers you inject.
• Four SQL dialects. PostgreSQL, MySQL/MariaDB, SQL Server, and SQLite (testing / single-instance).
• Free, and cheap at scale. Apache 2.0 — no license tier, no vendor, no per-operation pricing. A single 4-vCPU engine host sustains ~9,400 steps/s — billions of steps a month — and the whole deployment is your own replicas plus a few ordinary SQL instances. No broker, no control plane, nothing else to run or pay for, so the bill is a handful of cloud instances rather than a metered service.

Dwarf depends only on sequel (SQL), plus the OpenTelemetry API.

Install

go get github.com/microbus-io/dwarf

Requires Go 1.26+.

Packages

The split matters: dwarf/workflow is a lightweight type package. Code that defines tasks and graphs imports only dwarf/workflow, never the engine, so the engine's heavy dependencies (SQL drivers, the scheduler) stay out of those builds.

The host model

The engine reaches the outside world through a single Host interface, registered once with SetHost. Two methods, both required — and that is the whole contract, single-replica or not: the engine sends nothing between replicas, so there is no transport for you to wire.

type Host interface {
    // Required. Fetch a workflow graph by name (called at Create; the graph is then frozen on the flow,
    // and on subgraph spawn).
    LoadGraph(ctx context.Context, workflowURL string) (*workflow.Graph, error)

    // Required. Execute one task. The Flow carrier arrives with its input state populated; write outputs.
    ExecuteTask(ctx context.Context, taskURL string, flow *workflow.Flow) error
}

A standalone host backs LoadGraph with an in-memory registry / file / database, and ExecuteTask with a local function table or an RPC client. A bus-based host (for example a microservice mesh) bridges them to its transport. The engine never learns how tasks are reached.

Production wiring

Each Set* returns an error, so misconfiguration fails loudly at wiring time:

eng := dwarf.NewEngine()
eng.SetShard(engine.ShardSpec{Index: 1, DSN: "postgres://user:pass@db-a.internal:5432/dwarf", VirtualCPUs: 8})
eng.SetShard(engine.ShardSpec{Index: 2, DSN: "postgres://user:pass@db-b.internal:5432/dwarf", VirtualCPUs: 8})
eng.SetWorkers(64)
eng.SetHost(host)
eng.SetLogger(slog.Default())
eng.SetMeterProvider(otel.GetMeterProvider())
eng.SetTracerProvider(otel.GetTracerProvider())

err := eng.Startup(ctx)
if err != nil {
    log.Fatal(err)
}
defer eng.Shutdown(ctx)

flowKey, err := eng.Create(ctx, "checkout", initialState, &workflow.FlowOptions{
    Priority:    10,
    FairnessKey: tenantID,
    Baggage:     actorClaims,
})
outcome, err := eng.Await(ctx, flowKey) // Create returns a running flow; there is no separate start call

The live Set* methods (SetMaxOpenConns, SetTimeBudget, SetDefaultPriority, SetRefillInterval) may be called after Startup for hot reconfiguration; the rest are construction-time only.

Database support

See docs/deployment.md for tuning, sharding, and connection-pool guidance.

Documentation

Full guides live in docs/, split by who they are written for — start at the documentation index for the recommended reading order.

Building workflows:

• Getting started — install, wiring, your first flow
• Concepts — graph, task, flow, step, thread, reducer, lifecycle
• Building graphs — transitions, conditions, fan-out, error handling, reducers
• Writing tasks — the Flow carrier, state, control signals, baggage, error handling
• Driving flows — create, run, inspect, resume, cancel, fork, continue, retain
• Detecting completion — Await vs. orchestration, and how to deliver reliably
• Fan-out & subgraphs — parallelism, dynamic forEach, calling sub-workflows
• Scheduling & reliability — priority, fairness, retries, recovery
• Testing — NewEngineUnderTest and TestProxy

Running it in production:

• Operating dwarf — what the engine repairs itself, what needs you, what to watch
• Production checklist — the pre-flight list before you go live
• Runbook — symptom-indexed incident response
• Deployment — database choice, sharding, config, multi-replica
• Observability — logs, metrics, tracing
• Upgrading — redeploying your app, upgrading dwarf itself, graph evolution
• Resharding — adding, retiring and moving shards
• Backup and restore — what to back up, and what a restore does to flows
• Data handling — what is stored, what is searchable, retention
• Benchmarks — throughput & latency per dialect and shard count, and how to run them
• Cloud benchmarks — the sizing formula, measured against managed PostgreSQL

API reference: pkg.go.dev/github.com/microbus-io/dwarf.

License

Apache License 2.0. See LICENSE.