dwarf

Dwarf is a standalone, embeddable workflow-orchestration engine for Go.

You describe a workflow as a graph of tasks; dwarf runs it — dispatching one task at a time per step, persisting state between steps in a SQL database, and handling the hard parts of durable orchestration: parallel fan-out/fan-in, conditional routing, retries with backoff, timed sleeps, subgraphs, and human-in-the-loop pauses.

Dwarf has no built-in transport. It doesn't know how your tasks are reached (a local function call, an RPC, a message bus) or where your graphs live. You wire it to your world through a few small dependency interfaces, and it handles scheduling, state, durability, and recovery.

g := workflow.NewGraph("Greet")
g.SetEndpoint("Hello", "http://example/hello") // node "Hello" dispatches to this endpoint URL
g.AddTransition("Hello", workflow.END)

proxy := engine.NewTestProxy()
proxy.HandleGraph("http://example/greet", g)
proxy.HandleTask("http://example/hello", func(ctx context.Context, f *workflow.Flow) error {
    f.SetString("greeting", "hello "+f.GetString("name"))
    return nil
})

eng := dwarf.NewEngine()
eng.SetHost(proxy) // TestProxy implements the Host interface
eng.RunInTest(t)   // SQLite in-memory, auto-cleanup

_, out, _ := eng.Run(ctx, "http://example/greet", map[string]any{"name": "ada"}, nil) // Run returns (flowKey, outcome, err)
fmt.Println(out.State["greeting"]) // hello ada

Why dwarf

• Durable by construction. Every step is checkpointed to SQL. A crashed worker's in-flight step is recovered by lease expiry; a flow can be inspected, resumed, forked, or continued days later.
• Parallelism that merges cleanly. Static and dynamic (forEach) fan-out run branches concurrently; fan-in merges their state with per-field reducers (append, add, union, merge, …).
• Human-in-the-loop. A task can Interrupt to park the flow for external input and Resume later — approvals, manual review, async callbacks.
• Fair and prioritized. Two-level scheduling: strict priority bands across the cluster, weighted fairness within a band so one tenant can't starve another.
• Scales horizontally. Run many replicas against sharded databases; replicas coordinate through fire-and-forget peer signals you publish however you like.
• OTEL-native observability. Structured logs (slog), 10 dwarf_* metrics, and distributed tracing, all through standard providers you inject.
• Four SQL dialects. PostgreSQL, MySQL/MariaDB, SQL Server, and SQLite (testing / single-instance).

Dwarf depends only on sequel (SQL), plus the OpenTelemetry API.

Install

go get github.com/microbus-io/dwarf

Requires Go 1.26+.

Packages

The split matters: dwarf/workflow is a lightweight type package. Code that defines tasks and graphs imports only dwarf/workflow, never the engine, so the engine's heavy dependencies (SQL drivers, the scheduler) stay out of those builds.

The host model

The engine reaches the outside world through a single Host interface, registered once with SetHost. The first two methods are required; SignalPeers does nothing for a single-replica host.

type Host interface {
    // Required. Fetch a workflow graph by name (called at Create; the graph is then frozen on the flow,
    // and on subgraph spawn).
    LoadGraph(ctx context.Context, workflowURL string) (*workflow.Graph, error)

    // Required. Execute one task. The Flow carrier arrives with its input state populated; write outputs.
    ExecuteTask(ctx context.Context, taskName string, flow *workflow.Flow) error

    // Optional. Ship one cross-replica coordination signal to the other replicas (no-op for
    // single-replica). op is a routing key; payload is opaque bytes. Peers hand it back via
    // eng.DeliverSignal(ctx, op, payload).
    SignalPeers(ctx context.Context, op string, payload []byte)
}

A standalone host backs LoadGraph with an in-memory registry / file / database, and ExecuteTask with a local function table or an RPC client. A bus-based host (for example a microservice mesh) bridges them to its transport. The engine never learns how tasks are reached.

Production wiring

Each Set* returns an error, so misconfiguration fails loudly at wiring time:

eng := dwarf.NewEngine()
eng.SetDSN("postgres://user:pass@db:5432/dwarf")
eng.SetNumShards(2)
eng.SetWorkers(64)
eng.SetHost(host)
eng.SetLogger(slog.Default())
eng.SetMeterProvider(otel.GetMeterProvider())
eng.SetTracerProvider(otel.GetTracerProvider())

err := eng.Startup(ctx)
if err != nil {
    log.Fatal(err)
}
defer eng.Shutdown(ctx)

flowKey, err := eng.Create(ctx, "checkout", initialState, &workflow.FlowOptions{
    Priority:    10,
    FairnessKey: tenantID,
    Baggage:     actorClaims,
})
outcome, err := eng.Await(ctx, flowKey) // Create returns a running flow; there is no separate start call

The live Set* methods (SetMaxOpenConns, SetWorkersPerConn, SetTimeBudget, SetDefaultPriority) may be called after Startup for hot reconfiguration; the rest are construction-time only.

Database support

See docs/deployment.md for tuning, sharding, and connection-pool guidance.

Documentation

Full guides live in docs/:

• Getting started — install, wiring, your first flow
• Concepts — graph, task, flow, step, thread, reducer, lifecycle
• Building graphs — transitions, conditions, fan-out, error handling, reducers
• Writing tasks — the Flow carrier, state, control signals, baggage, error handling
• Engine operations — create, run, inspect, resume, cancel, fork, continue, retain
• Detecting completion — Await vs. orchestration, and how to deliver reliably
• Fan-out & subgraphs — parallelism, dynamic forEach, calling sub-workflows
• Scheduling & reliability — priority, fairness, retries, recovery
• Observability — logs, metrics, tracing
• Deployment — database choice, sharding, config, multi-replica
• Benchmarks — throughput & latency per dialect and shard count, and how to run them
• Testing — RunInTest and TestProxy

API reference: pkg.go.dev/github.com/microbus-io/dwarf.

License

Apache License 2.0. See LICENSE.