relay

Wyolet Relay

A high-throughput LLM router in Go. Put one endpoint in front of every provider, pool your own API keys for failover and higher effective rate limits, and self-host the whole thing. Built for teams running millions of LLM requests a day.

docker run -p 8080:8080 -p 8081:8081 wyolet/relay:standalone

A full relay with Postgres bundled and the catalog pre-seeded — admin UI on :8081, OpenAI/Anthropic-compatible API on :8080. The master key, admin password, and admin token are auto-generated on first boot and printed to the logs (and persisted on the data volume). See Quickstart for your first request.

Why Relay

• One API, every upstream. OpenAI- and Anthropic-shape endpoints (OpenAI, Anthropic, Bedrock, Vertex, Azure OpenAI, Ollama, Together, Groq, Fireworks — anything speaking either wire protocol). Drop-in for your existing OpenAI/Anthropic SDK code.
• Key pooling that multiplies your rate limits. Load-balanced, failover across many keys, with per-key circuit breakers (auth / quota / rate-limit / server-error aware). On an upstream auth failure a key is re-resolved out-of-band — the request fails over immediately and the rotated key heals in the background.
• BYO keys, your secret store. Resolve keys from env, an AES-GCM Postgres store, or fetch-only external backends (AWS / Azure / GCP Secret Manager, Vaultwarden/Bitwarden, 1Password). External secrets stay in memory, never persisted.
• Infra-grade performance. < 2 ms p50 added latency, 5–10k RPS/pod, k8s-native. Postgres-backed catalog fans out to every pod via NOTIFY/LISTEN in ~1s with zero PG round-trips on the hot path; responses stream byte-for-byte from upstream.
• Self-hostable, AGPL-3.0. Your keys, your data, your infra. Generated typed OpenAPI for both planes, so clients get full type information free.

Quickstart — first request in ~2 minutes

1. Start the relay (the docker run above). Also on GitHub Container Registry as ghcr.io/wyolet/relay:standalone.

2. Run the setup wizard. Open http://localhost:8081, log in with the admin credentials you set (RELAY_ADMIN_TOKEN / RELAY_ADMIN_PASSWORD). The wizard walks you through adding a provider key and minting a relay key — copy the relay-key plaintext when it's shown (it's shown exactly once).

3. Call it like the OpenAI API:

curl http://localhost:8080/openai/v1/chat/completions \
  -H "Authorization: Bearer <your-relay-key>" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4o","messages":[{"role":"user","content":"hello"}]}'

The bundled image is single-node with embedded Postgres — perfect for a try-out, not for production. For that, run the lean image (wyolet/relay:latest) against a managed Postgres; see other ways to run below.

Status

See docs/roadmap.md for the full picture and docs/canonical-protocol.md for the relay-canonical protocol design.

Other ways to run

Official images (:latest lean, :standalone all-in-one, and :<version> tags) are published to Docker Hub (wyolet/relay) and GHCR (ghcr.io/wyolet/relay) by the maintainers; this repository does not build or publish them in CI. To build your own from source, use the Dockerfile / docker-bake.hcl, or docker compose up --build below.

Docker Compose (standalone — bundled services)

Requires Docker. Bundles its own Postgres (plus optional ClickHouse, Valkey, Jaeger), and builds the relay image from source. Run it from the repo root — the root docker-compose.yml wires the whole stack:

cp .env.example .env
# REQUIRED: set a master key in .env — generate one with:
#   openssl rand -base64 32
docker compose up --build

Data plane: http://localhost:5100 · control plane + admin UI: http://localhost:5103. The image bakes in the catalog and seeds it into the bundled Postgres on first boot — no extra steps to get a populated catalog. (You still add your own host keys + a relay key to serve traffic, via the UI or the control API.)

Run from source (dev)

Requires Go 1.25+, Docker, and make. Ephemeral Postgres via compose. Seeding from source needs a local clone of the catalog data — the wyolet/relay-catalog repo's data/ tree (cloned alongside this repo below):

# Catalog data, cloned next to this repo.
git clone https://github.com/wyolet/relay-catalog ../relay-catalog

# Bring the test PG up.
docker compose -f deploy/compose/docker-compose.test.yml up -d --wait

# Run the binary.
RELAY_PG_DSN='postgres://relay:relay@127.0.0.1:5499/relay_test?sslmode=disable' \
  RELAY_AUTO_SEED_IF_EMPTY=1 \
  RELAY_CATALOG_DIR=../relay-catalog/data \
  RELAY_CONFIG_DIR=./config \
  RELAY_PORT=5199 \
  RELAY_CONTROL_PORT=5198 \
  RELAY_STATE_BACKEND=memory \
  RELAY_ADMIN_TOKEN=devtoken \
  RELAY_ADMIN_PASSWORD=devpass-min-8-chars \
  RELAY_COOKIE_SECURE=false \
  go run ./cmd/relay

On first boot the relay auto-seeds the catalog from RELAY_CATALOG_DIR into the empty Postgres (providers, hosts, models, bindings, pricing, policies). Omit RELAY_CATALOG_DIR to start with an empty catalog and populate it via the control API. (The Docker image above bakes the catalog in, so it needs no clone.)

Verify:

# Inference healthcheck (public)
curl localhost:5199/healthz

# Control version (public)
curl localhost:5198/version

# Admin CRUD with break-glass bearer
curl -H 'Authorization: Bearer devtoken' localhost:5198/providers

Tear down:

docker compose -f deploy/compose/docker-compose.test.yml down -v

Architecture

Two listeners on separate ports, sharing one in-memory catalog snapshot:

                ┌───────────────────────────┐
                │   Postgres (truth)        │
                │   + Redis (hot state)     │
                └───────────────────────────┘
                            ▲
                            │ NOTIFY catalog_events
                            │
                   ┌────────┴────────┐
                   │  app/catalog    │
                   │  Snapshot       │  COW reconciler, dependency-ordered
                   │  + Listener     │  apply, debounced ~1s
                   └────────┬────────┘
                            │
        ┌───────────────────┴───────────────────┐
        │                                       │
┌───────▼────────┐                     ┌────────▼────────┐
│ Inference plane│                     │ Control plane   │
│ RELAY_PORT     │                     │ RELAY_CONTROL_  │
│                │                     │ PORT            │
│ /v1/*          │                     │ /auth/*         │
│ /healthz       │                     │ /{kind}/...     │
│                │                     │ /master-key/... │
│ relay-key auth │                     │ session cookie  │
│                │                     │  + admin token  │
└────────┬───────┘                     └─────────────────┘
         │
         ▼
   app/pipeline ────► app/api/<adapter> ────► upstream
   (orchestration)    (wire format)

Component responsibilities (full breakdown in CLAUDE.md):

• app/catalog — immutable snapshot, COW reconciler, NOTIFY listener. Reads are O(1) by id or by name. Writes go through app/X.Store → PG → NOTIFY → listener → reconciler → atomic snapshot swap.
• app/pipeline — pure orchestration. Reserve rate-limit budget → pick a host key → call upstream via Adapter.Call → stream response → detached post-flight (commit + RecordSuccess + OnSuccess callback). Knows nothing about catalog snapshots, JSON shapes, or HTTP routing.
• app/api/{openai,anthropic} — adapter implementations. Build the upstream HTTP request, classify response status for retry, extract usage tokens. Pure shape parsers live separately under pkg/api/<shape> (vendorable).
• app/httpapi/{inference,control} — huma + chi HTTP layer. Each plane has its own generated OpenAPI spec.

API surface

Inference plane (data plane)

POST   /v1/generate                  relay canonical shape (provider-neutral)
POST   /openai/v1/chat/completions   OpenAI Chat Completions shape
POST   /openai/v1/responses          OpenAI Responses shape
POST   /openai/v1/embeddings         OpenAI Embeddings (byte-pass)
POST   /anthropic/v1/messages        Anthropic Messages shape
GET    /v1/models                    list models accessible to the relay key
GET    /healthz                      liveness + PG ping (public)
GET    /openapi.json                 generated typed spec

Auth: Authorization: Bearer <relay-key>. Relay keys are managed via the control plane.

Namespace convention: each vendor shape is served under its own prefix (/openai/..., /anthropic/...); the bare /v1 namespace belongs to relay's own canonical shape. POST /v1/generate accepts a canonical v1.Request and returns canonical — the provider-neutral surface that carries cross-cutting features like cache_config to any upstream regardless of the model's HostBinding adapter.

Control plane (admin)

Uniform CRUD across eight kinds:

GET    /{plural}                 list
GET    /{plural}/{ref}           read by slug or id
POST   /{plural}                 create (server stamps id + slug)
PUT    /{plural}/by-id/{id}      update
DELETE /{plural}/by-id/{id}      delete

Plurals: providers, hosts, models, host-keys, rate-limits, policies, pricings, relay-keys.

Plus:

POST   /auth/login               username + password → session cookie
POST   /auth/logout              destroy session
GET    /auth/whoami              return the authenticated caller
POST   /master-key/generate      mint a candidate RELAY_MASTER_KEY
POST   /reload                   force full snapshot rebuild from PG
GET    /version                  build version
GET    /openapi.json             generated typed spec

Auth: either a valid relay_session cookie (set by /auth/login) OR Authorization: Bearer ${RELAY_ADMIN_TOKEN} (break-glass).

Configuration

Auth model

Passwords are bcrypt-aware: hashes starting with $2a$ / $2b$ / $2y$ go through bcrypt.CompareHashAndPassword; plain-text passwords work for legacy YAML but log a deprecation warning.

Future-proof seams (already wired):

• app/actor.Actor in context.Context — handlers read it via actor.From(ctx), never the cookie or token directly.
• app/authz.Authorizer interface — every CRUD mutation routes through Authorize(ctx, action, resource). v1 implementation grants any authenticated caller; swap to Casbin / OpenFGA later without touching handlers.

JWT for programmatic API access is a planned third caller type (docs/roadmap.md B4).

Performance

The 18× gap is unavoidable I/O — Redis Lua RTT, LB hop, container network. Bench numbers are the architectural lower bound (regression gate); live numbers are what you can promise customers.

Hot-path rules (CLAUDE.md):

• Zero PG calls on the request path.
• One Redis Lua call (rate-limit reserve) is the goal; not three round-trips.
• Post-flight (Limiter.Commit, Selector.RecordSuccess, OnSuccess) runs in a detached goroutine — never blocks the response.
• All emits (usage, spans, payload capture) are async via bounded channels with drop-on-full.

Note: the in-process bench harness was retired with the legacy pipeline; a new harness against app/pipeline.Pipeline.Run is docs/roadmap.md A3.

Repository layout

Full breakdown in CLAUDE.md. Short version:

app/                  application: catalog, pipeline, http handlers, 8 entities
pkg/                  shared, vendorable: shapes, ratelimit, kv, ids, slug, ...
internal/             composition root: config, identity, storage, usage
cmd/relay/            binary entrypoint
cmd/litellm-import/   regenerate config YAML from LiteLLM JSON
config/               operator-authored YAML catalog
migrations/postgres/  versioned SQL up + down
deploy/compose/       dev pg, test pg, smoke stack
docs/                 design notes + roadmap + runbook

Development

# Unit tests
go test ./...

# Integration suite (spins up ephemeral pg, tears down)
make test-integration

# Regenerate config YAMLs from LiteLLM
go run ./cmd/litellm-import --out config

# Rebuild the dev stack
make dev

See CONTRIBUTING.md for the build/test workflow, the load-bearing codebase rules, and PR conventions. make lint-rules enforces the canonical-protocol import boundaries (rules 1/2/4/10).

Roadmap

The roadmap is split by product phase — docs/roadmap.md is the index:

• roadmap-oss.md — the open-source core: batch orchestration, webhooks, new adapters, observability, and launch readiness.
• roadmap-enterprise.md — on-prem: authN/authz, SSO, audit, HA, air-gap, security-review readiness.
• roadmap-saas.md — hosted multi-tenant: billing, quotas, tenant isolation, compliance.
• roadmap-v2.md — beyond v1: the Tool Gateway + icebox.

License

AGPL-3.0. Network use triggers the copyleft — run a modified relay as a service and you must publish your changes. Commercial / enterprise licensing is a separate arrangement.