README
¶
Flo
Build durable, event‑driven systems by importing a package—not by adopting a platform.
Flo is a library‑first, single‑binary runtime you embed in your service. Your code gets hard guarantees—replayable logs, leased queues, retries/DLQ—without spinning up a new cluster. When you need multi‑process or non‑Go clients, flip on the optional thin gRPC/REST server. Same semantics, no rewrite.
Flo provides two core building blocks with durable semantics:
- EventLog: append‑only, replayable logs with cursors, blocking reads, and retention trims
- WorkQueue: leased task queues with priority, delay/retry, DLQ, and backpressure
Designed to embed in Go processes by default with local durability, plus an optional thin gRPC/REST server for multi‑process access and SDK parity.
Why Flo?
- Built‑in, not bolted‑on: add durable logs/queues to your app with a function call
- No ceremony: a single binary you can run locally and ship anywhere
- Strong guarantees, simple API: at‑least‑once, strict ordering, blocking reads, retries/DLQ
- Evolve gradually: start embedded; enable a thin server for SDKs when you need it
Project status
MVP (Milestone M1) is in progress. Highlights implemented so far:
- EventLog: keys/encoding, Append, Read/Seek, blocking reads, cursors, trims, archiver hook
- WorkQueue: keys/encoding, Enqueue (priority/delay), Dequeue with leases, ExtendLease, Complete, Fail (retry/DLQ), basic backpressure, expired‑lease reclaim + background sweeper
- Tooling/CI and protobuf toolchain are set up; public protos limited to Channels/Admin/Health (facades); EventLog/WorkQueue are internal‑only
Quickstart
Prereqs: Go 1.23+
git clone https://github.com/rzbill/flo
cd flo
make test # run unit tests
make build # build all packages
Minimal embed example (pseudo‑usage):
// Open an embedded queue and enqueue then dequeue
q, _ := workqueue.OpenQueue(db, "ns", "payments", 1)
seq, _ := q.Enqueue(ctx, nil, []byte("hello"), 5 /*priority*/, 0 /*delayMs*/, 0)
msgs, _ := q.Dequeue(ctx, "workers", 1, 30_000 /*leaseMs*/, 0)
_ = q.Complete(ctx, "workers", []uint64{seq})
Repository layout (selected)
internal/eventlog/— EventLog core (keys/encoding, append/read/seek, cursors, trims)internal/workqueue/— WorkQueue core (keys/encoding, enqueue/dequeue, leases, retry/DLQ)internal/storage/pebble/— Pebble wrapper (options, batches, snapshots)proto/flo/v1/— Public service surfaces (Channels/Admin/Health)
Contributing
Issues and PRs are welcome. Please run tests locally:
make test
Follow conventional commits and include context in PR descriptions.
License
This project is licensed under the MIT License - see the LICENSE file for details.
Directories
¶
| Path | Synopsis |
|---|---|
|
api
module
|
|
|
cmd
|
|
|
flo
command
|
|
|
internal
|
|
|
cmd/client
Package client provides the `flo` command-line client.
|
Package client provides the `flo` command-line client. |
|
cmd/client/transports
Package transports provides pluggable transport implementations (gRPC/HTTP) used by the Flo CLI so commands can be decoupled from the underlying wire.
|
Package transports provides pluggable transport implementations (gRPC/HTTP) used by the Flo CLI so commands can be decoupled from the underlying wire. |
|
cmd/server
Package serverrun exposes a shared Run entrypoint used by the CLI to start the Flo runtime with gRPC and HTTP servers, handling lifecycle and shutdown.
|
Package serverrun exposes a shared Run entrypoint used by the CLI to start the Flo runtime with gRPC and HTTP servers, handling lifecycle and shutdown. |
|
config
Package config provides loading and environment overlay for Flo runtime configuration.
|
Package config provides loading and environment overlay for Flo runtime configuration. |
|
eventlog
Package eventlog implements Flo's internal append-only EventLog.
|
Package eventlog implements Flo's internal append-only EventLog. |
|
runtime
Package runtime wires storage, config, and facades into a single-node Flo instance.
|
Package runtime wires storage, config, and facades into a single-node Flo instance. |
|
server/grpc
Package grpcserver hosts the gRPC server for Flo, registering Health and Streams services and delegating to the shared services layer.
|
Package grpcserver hosts the gRPC server for Flo, registering Health and Streams services and delegating to the shared services layer. |
|
server/http
Package httpserver provides a minimal REST gateway for Flo with SSE subscribe support and JSON endpoints mirroring the Streams surface.
|
Package httpserver provides a minimal REST gateway for Flo with SSE subscribe support and JSON endpoints mirroring the Streams surface. |
|
services/streams
Package streamsvc implements the Streams facade on top of the internal EventLog.
|
Package streamsvc implements the Streams facade on top of the internal EventLog. |
|
services/workqueues
Package workqueues provides the gRPC service layer for WorkQueue operations.
|
Package workqueues provides the gRPC service layer for WorkQueue operations. |
|
storage/pebble
Package pebblestore provides a thin wrapper around Pebble with fsync policy, snapshots, batches, and minimal metrics hooks.
|
Package pebblestore provides a thin wrapper around Pebble with fsync policy, snapshots, batches, and minimal metrics hooks. |
|
ui
Package ui contains embedded static assets for the dashboard.
|
Package ui contains embedded static assets for the dashboard. |
|
workqueue
Package workqueue implements a one-of-N task queue with lease-based delivery.
|
Package workqueue implements a one-of-N task queue with lease-based delivery. |
|
pkg
|
|
|
id
Package id provides a 128-bit, lexicographically sortable identifier.
|
Package id provides a 128-bit, lexicographically sortable identifier. |
|
log
Package log provides Flo's structured logging facade and utilities.
|
Package log provides Flo's structured logging facade and utilities. |
Click to show internal directories.
Click to hide internal directories.