README
¶
ClickHouse® is a great database with a subpar API. WaveHouse fixes that.
The open-source real-time API gateway for ClickHouse: schema-aware ingest, async batching, real-time SSE streaming, and tiered query caching. All in a single binary.
Docs · Quick start · Why WaveHouse · Discussions
Quick Start Guide
git clone https://github.com/Wave-RF/WaveHouse.git && cd WaveHouse
docker compose -f deployments/compose/standalone.yaml up -d
The in-repo compose spins up WaveHouse and ClickHouse. Then, create a table (or import your existing schema):
docker compose -f deployments/compose/standalone.yaml exec clickhouse \
clickhouse-client --query "CREATE TABLE IF NOT EXISTS events (kind String, user String, received_timestamp DateTime64(3,'UTC') DEFAULT now64(3,'UTC')) ENGINE=MergeTree ORDER BY kind"
Then, stream live events in one terminal, ingest one in another, and watch the ingested events arrive from the stream:
curl -N "http://localhost:8080/v1/stream?table=events" & sleep 1
[!IMPORTANT] No auth is needed to ingest with the trial policy the stack mounts from
deployments/compose/settings/. The policy is VERY permissive, and not intended for production deployments.
curl -sX POST "http://localhost:8080/v1/ingest?table=events" \
-H 'content-type: application/json' -d '{"kind":"click","user":"u_42"}'
[!TIP] A 404 "unknown table" error here just means schema discovery hasn't seen the new table yet. Just retry; worst case 60s before discovery.
Full walkthrough at wavehouse.dev/getting-started.
Why WaveHouse?
ClickHouse is a phenomenal OLAP database, but pointing a frontend right at it leaves a lot to be desired: one-row inserts trigger Too many parts, there's no backpressure or edge validation, no real-time push, and no row/column security. You end up building custom APIs, a Kafka queue, a batch consumer, a cache tier, and an auth service. WaveHouse is that whole stack as one binary — the only external dependency is ClickHouse.
If you're building user-facing analytics, WaveHouse is like Supabase for ClickHouse. Or an open-source Tinybird that pushes data to the frontend in real time over SSE, not just pull-based REST.
- Ingest — async durable WAL (embedded NATS JetStream),
200 OKinstantly, background batch-flush; schema-validated againstsystem.columns; optional ID-based dedup (idempotent ingest); dead-letter queue for rows ClickHouse rejects (an unavailable ClickHouse is retried with backoff, not dead-lettered). - Query — result cache (in-process Ristretto, or a Redis shared by every instance) +
singleflightcoalescing; type-safe structured query AST; Tinybird-style named pipes (parameterized SQL endpoints). - Real-time — native SSE push, broadcast before the ClickHouse flush, with JetStream gap-fill for late/reconnecting clients.
- Security — Hasura-style per-table, per-role column + row policies with JWT claim templating, defined in the hot-reloadable settings directory.
- Client —
@wavehouse/sdk: TypeScript client with query builder, live queries, streaming, and schema codegen; one runtime dependency (an SSE frame parser, ~1.4 KB gzipped).
How it compares
| Direct ClickHouse | Kafka + CH (DIY) | Tinybird | WaveHouse | |
|---|---|---|---|---|
| Self-hosted, single binary | — | — | ✗ (SaaS) | ✓ |
| Safe high-rate inserts | ✗ | ✓ (via Kafka) | ✓ | ✓ |
| Schema validation at the edge | ✗ | custom | ✓ | ✓ |
| Real-time push (SSE) | ✗ | custom service | ✗ | ✓ native |
| Thundering-herd coalescing | ✗ | custom | ✓ | ✓ |
| Row/column policies (JWT) | ✗ | custom | tokens only | ✓ Hasura-style |
| Cost model | infra | infra + eng time | per-vCPU SaaS | infra only |
Full breakdown, failure modes, and our engineering rationale at wavehouse.dev/why-wavehouse.
Getting Started
Pick between the in-repo Docker Compose, pulling a container image, or go install. The Compose option starts WaveHouse; the other options just download/install a container or binary. Run the selected artifact before using http://localhost:8080.
A. Docker Compose (recommended)
git clone https://github.com/Wave-RF/WaveHouse.git && cd WaveHouse
docker compose -f deployments/compose/standalone.yaml up -d
The stack bind-mounts deployments/compose/settings/ as its settings directory whose policies.json / roles.json carry a permissive trial policy, so you can ingest without a token; edit the files on the host and the running container adopts them. Create a table in ClickHouse (or import existing schema), then ingest — see the getting-started walkthrough for the full ingest → query → stream tour.
B. Prebuilt container image
docker pull ghcr.io/wave-rf/wavehouse:latest # latest stable release
docker pull ghcr.io/wave-rf/wavehouse:dev # rolling main-branch build
Tags carry a signed Sigstore build-provenance attestation — verify before you deploy:
gh attestation verify oci://ghcr.io/wave-rf/wavehouse:dev \
--repo Wave-RF/WaveHouse \
--signer-workflow Wave-RF/WaveHouse/.github/workflows/publish-dev.yml
Swap in :vX.Y.Z and release.yml for a release image. Pin the signer either way. --repo alone accepts an attestation from any workflow in the repo.
C. go install (binary, no Docker)
go install github.com/Wave-RF/WaveHouse/cmd/wavehouse@latest
wavehouse bootstrap ./settings # starter settings directory, every key at its default
WH_SETTINGS_DIR=./settings wavehouse
The settings directory is required (no default; the container images ship none — mount one at /app/settings). The seed ships no policy, so every request is denied until you write one to ./settings/policies.json (and declare its roles in roles.json) — copy deployments/compose/settings/policies.json for the permissive trial policy. You'll still need ClickHouse reachable: the seed points at localhost:9000; for anything else, edit clickhouse.addr in ./settings/config.json. Everything in the directory hot-reloads. See Settings Directory and Configuration for more information.
Project status
WaveHouse is in alpha. See SUPPORT.md for where to ask what, the alpha-stage response cadence (best-effort, 1–2 business days), and what's in vs. out of scope right now.
Track what's shipped, in progress, and planned on the project board.
Alpha = expect change. WaveHouse is pre-1.0: APIs, configuration, wire formats, and on-disk state can change between releases without a migration path, and some capabilities are still hardening. Pin a version and don't rely on stability guarantees until a tagged GA release.
Local Development
You'll need Go 1.26+, GNU Make 4+, Docker (Compose v2), Node.js 22 LTS, and pnpm 11.21+. See development docs for the authoritative source of truth with the full list, version requirements, and gotchas.
make tools # one-time bootstrap
docker compose -f deployments/compose/dependencies.yaml up -d clickhouse
make dev # hot-reload on .go save
🤝 Contributing
Issues, pull requests, and feedback welcome! See our CONTRIBUTING.md guidelines on how to structure your code and run the integration test suites.
Security
Found a vulnerability? Do NOT open a public issue. Email security@wave-rf.com per SECURITY.md — we acknowledge within 48 hours and aim for an initial assessment in 5 business days.
AI-Assisted Development
The repo contains a minimal Claude Code configuration: safety guardrails, a couple of slash commands and subagents, auto-format hooks, and worktrunk project hooks for parallel agent workflows. Personal preferences (status line, model, allow lists) stay user-level. See Claude Code & AI agents for setup + reference. AGENTS.md at the repo root is the canonical source of truth for project conventions.
Responsible GenAI Usage Disclosure
AI-assisted, human-reviewed. WaveHouse has been developed with AI assistance (Claude Code). Every change, whether AI- or human-authored, goes through the same review gates, tests, and CI before it gets merged. The docs are the source of truth, and please open an issue if anything is out of date. (or, obviously, if you notice an issue that isn't security-related)
License
WaveHouse is open source under the Apache License 2.0.
Directories
¶
| Path | Synopsis |
|---|---|
|
clients
|
|
|
go
module
|
|
|
cmd
|
|
|
wavehouse
command
|
|
|
internal
|
|
|
app
Package app wires the WaveHouse process.
|
Package app wires the WaveHouse process. |
|
auth
Package auth handles authentication: it validates JWT Bearer tokens and records the caller's role, claims, and any token-validation error in the request context.
|
Package auth handles authentication: it validates JWT Bearer tokens and records the caller's role, claims, and any token-validation error in the request context. |
|
chconn
Package chconn owns the process's ClickHouse connections: one pool per distinct address, database, user, password and tls tuple (ClickHouse authenticates per connection, so different credentials never share one), shared by the tenants whose settings name it and reconciled after every settings reload under the boot config's connection ceiling.
|
Package chconn owns the process's ClickHouse connections: one pool per distinct address, database, user, password and tls tuple (ClickHouse authenticates per connection, so different credentials never share one), shared by the tenants whose settings name it and reconciled after every settings reload under the boot config's connection ceiling. |
|
chsql
Package chsql holds ClickHouse SQL helpers shared across packages that build SQL — primarily safe identifier quoting.
|
Package chsql holds ClickHouse SQL helpers shared across packages that build SQL — primarily safe identifier quoting. |
|
coord
Package coord holds leases for work that must run in one process at a time — the sweeper, and the ingest processes' shard claim membership.
|
Package coord holds leases for work that must run in one process at a time — the sweeper, and the ingest processes' shard claim membership. |
|
coord/coordtest
Package coordtest is the behavior every coord.Coordinator must share, as one suite each implementation runs against its own backend.
|
Package coordtest is the behavior every coord.Coordinator must share, as one suite each implementation runs against its own backend. |
|
dedupe/dedupetest
Package dedupetest is the conformance suite every dedupe backend runs: the Deduplicator contract (dedupe.go) as tests, driven through the production path — a backend's Factory and the Managed switch it returns — so a backend that passes here behaves the same under ingest as every other.
|
Package dedupetest is the conformance suite every dedupe backend runs: the Deduplicator contract (dedupe.go) as tests, driven through the production path — a backend's Factory and the Managed switch it returns — so a backend that passes here behaves the same under ingest as every other. |
|
keyenc
Package keyenc is the one escaping composite WaveHouse keys are built from: NATS subject tokens, cache keys and dedupe keys.
|
Package keyenc is the one escaping composite WaveHouse keys are built from: NATS subject tokens, cache keys and dedupe keys. |
|
mq
Package mq is the message-queue boundary: the only package that imports NATS/JetStream (enforced by the depguard rule in .golangci.yml, Key Design Decision #20), and the only one that knows how the broker works.
|
Package mq is the message-queue boundary: the only package that imports NATS/JetStream (enforced by the depguard rule in .golangci.yml, Key Design Decision #20), and the only one that knows how the broker works. |
|
mq/mqtest
Package mqtest is the conformance suite for mq.Broker: the behavior the rest of the process relies on, stated once and run by every implementation from its own tests.
|
Package mqtest is the conformance suite for mq.Broker: the behavior the rest of the process relies on, stated once and run by every implementation from its own tests. |
|
mq/natstest
Package natstest stands up NATS the way an operator deploys it for mq.backend: nats: the shipped Helm values' accounts, users and permissions (deployments/nats/values.yaml) and the shipped nack manifests (deployments/nats/jetstream.yaml).
|
Package natstest stands up NATS the way an operator deploys it for mq.backend: nats: the shipped Helm values' accounts, users and permissions (deployments/nats/values.yaml) and the shipped nack manifests (deployments/nats/jetstream.yaml). |
|
settings
Package settings defines the on-disk settings directory — the JSON documents that hold WaveHouse's hot-reloadable configuration — and Validate, the single gate every consumer of that directory runs before adopting its contents.
|
Package settings defines the on-disk settings directory — the JSON documents that hold WaveHouse's hot-reloadable configuration — and Validate, the single gate every consumer of that directory runs before adopting its contents. |
|
stream
Package stream holds the SSE fan-out primitives shared by the streaming API: the Subscriber connection handle (subscriber.go), the Bucket fan-out primitive (bucket.go), the Heartbeater keepalive wheel (heartbeat.go) that nudges idle GET /v1/stream connections, and the event Hub (hub.go).
|
Package stream holds the SSE fan-out primitives shared by the streaming API: the Subscriber connection handle (subscriber.go), the Bucket fan-out primitive (bucket.go), the Heartbeater keepalive wheel (heartbeat.go) that nudges idle GET /v1/stream connections, and the event Hub (hub.go). |
|
tenant
Package tenant defines the tenant identifier: the token that selects a settings folder, and with it the tables, policies, and pipes a request or an event belongs to (#583).
|
Package tenant defines the tenant identifier: the token that selects a settings folder, and with it the tables, policies, and pipes a request or an event belongs to (#583). |
|
testutil
Package testutil — OTLP gRPC test receiver.
|
Package testutil — OTLP gRPC test receiver. |
|
testutil/cachetest
Package cachetest is the conformance suite every cache.Cache backend runs: what a hit, a miss and an invalidation mean, independent of where the entries and versions live.
|
Package cachetest is the conformance suite every cache.Cache backend runs: what a hit, a miss and an invalidation mean, independent of where the entries and versions live. |
|
testutil/logtest
Package logtest points slog's default logger somewhere a test can use.
|
Package logtest points slog's default logger somewhere a test can use. |
|
testutil/mutationtest
Package mutationtest holds the statements api.IsMutation is tested on, shared by its unit test and by the integration test that checks each one against ClickHouse's own parser.
|
Package mutationtest holds the statements api.IsMutation is tested on, shared by its unit test and by the integration test that checks each one against ClickHouse's own parser. |
|
testutil/storedir
Package storedir gives a test a directory for the embedded message broker's store.
|
Package storedir gives a test a directory for the embedded message broker's store. |
|
scripts
|
|
|
cov
command
Coverage rendering + threshold gating for WaveHouse.
|
Coverage rendering + threshold gating for WaveHouse. |
|
orchestrator
command
E2E orchestrator — drives a clean, isolated E2E test session against one ClickHouse + one Redis + one WaveHouse, then runs the vitest suite once.
|
E2E orchestrator — drives a clean, isolated E2E test session against one ClickHouse + one Redis + one WaveHouse, then runs the vitest suite once. |
|
tests
|
|
|
integration
Package tests holds WaveHouse's Go integration tests.
|
Package tests holds WaveHouse's Go integration tests. |