README
¶
ai
LLM control plane, RAG, and model hub for the Hanzo platform. Native Go model routing with no Python middlemen.
Quick start
docker pull ghcr.io/hanzoai/ai:latest
curl -H "Authorization: Bearer hk-YOUR-API-KEY" \
https://api.hanzo.ai/v1/chat/completions \
-d '{"model":"zen4-pro","messages":[{"role":"user","content":"Hello"}]}'
What this is
ai is the canonical LLM control plane for the Hanzo platform — model routing, RAG, model hub, MCP management. OpenAI-compatible JSON over HTTP (/v1/chat/completions, /v1/models). Three auth modes (IAM API key hk-*, JWT, provider key sk-*). Static model routing — 66+ models mapped to upstream providers in pure Go. Per-request usage tracking fire-and-forget to IAM. KMS-resolved provider secrets, org-scoped. Renamed from hanzoai/cloud on 2026-05-19 when the unified binary took the cloud name per HIP-0106; ai mounts as the ai subsystem inside hanzoai/cloud.
Specs
Implements:
- HIP-0037 AI Cloud Platform
- HIP-0106 Unified Cloud Binary (ai subsystem)
Architecture
user / api key -> hanzoai/gateway -> hanzoai/ai (zip.App, Go)
|
auth: hk-* | JWT | sk-* (provider passthrough)
|
model routing: 66+ models -> upstream providers
|
+--------+--------+--------+--------+
| | | | |
DO-AI Fireworks OpenAI Anthropic ...
| | | | |
KMS-resolved provider secrets
|
IAM usage tracking (async)
Hanzo Cloud
AI Cloud OS — Native Go model routing, IAM-integrated auth, usage tracking, and KMS secrets management. Zero middlemen, pure performance.
Architecture
Hanzo Cloud implements the ZAP (Zero-overhead API Protocol) — a native Go model routing layer that connects users directly to upstream AI providers with no intermediaries.
User → cloud-api (Go, ZAP gateway) → upstream providers (DO-AI, Fireworks, OpenAI)
↕ ↕
Hanzo IAM Hanzo KMS
(auth, billing, usage) (multi-tenant secrets)
| Component | Description | Technology |
|---|---|---|
| Gateway | ZAP-native model routing, auth, billing | Go + Beego |
| Frontend | Admin UI, chat, knowledge base | React + Next.js |
| IAM | Identity, API keys (hk-), usage tracking |
hanzoai/iam |
| KMS | Multi-tenant secrets, org-scoped projects | hanzoai/kms |
| Engine | Local inference (mistral.rs fork) | hanzoai/engine |
ZAP Protocol
ZAP defines the fast native path from API gateway to AI inference:
- OpenAI-compatible JSON over HTTP (
/v1/chat/completions,/v1/models) - Three auth modes: IAM API key (
hk-*), JWT (hanzo.id OAuth), Provider key (sk-*) - Static model routing — 66+ models mapped to 3 upstream providers in pure Go
- Per-request usage tracking — async fire-and-forget to IAM
- KMS-resolved secrets — provider API keys from Hanzo KMS with org-scoped projects
- Zero Python — no legacy proxy middleware, no extra hops
Supported Models
Free Tier (DO-AI) — 28 models
gpt-4o, gpt-5, gpt-5-mini, claude-opus-4-6, claude-sonnet-4-5, claude-haiku-4-5, o3, o3-mini, qwen3-32b, deepseek-r1-distill-70b, llama-3.3-70b, and more.
Premium (Fireworks) — 17 models
fireworks/deepseek-r1, fireworks/deepseek-v3, fireworks/kimi-k2, fireworks/qwen3-235b-a22b, fireworks/qwen3-coder-480b, fireworks/cogito-671b, and more.
Premium (OpenAI Direct) — 5 models
openai-direct/gpt-4o, openai-direct/gpt-5, openai-direct/o3, openai-direct/o3-mini, openai-direct/gpt-4o-mini
Zen Models (Premium, 3X) — 8 models
zen4-mini, zen4-pro, zen4-max, zen4-ultra, zen4-coder-flash, zen4-coder-pro, zen-vl, zen3-omni
Full model list: GET /api/models
Auto-routing (auto / zen-router)
Opt-in virtual model that picks the best concrete model per request. Send
"model": "auto" (alias "zen-router") and the request is classified into a
coarse task (code, reasoning, math, creative, vision, long-context, cheap-chat,
general) and mapped to the first servable model in the matching preference
list — before provider, pricing, and billing resolution. So an auto request
is billed and reported as exactly the model that served it.
Enable it in conf/models.yaml (the cloud-api-models ConfigMap in prod), or
via env ROUTER_ENABLED=true / ROUTER_ENDPOINT=<url>:
router:
enabled: true
endpoint: "" # optional zen-router URL (see below); "" = built-in heuristic
cost_ceiling: 0.0
prefer:
code: [zen4-coder, zen5-coder, qwen3-coder, gpt-5.3-codex]
reasoning: [zen4-thinking, deepseek-reasoner, zen4-ultra]
cheap_chat: [zen4-mini, zen5-mini, gpt-4o-mini]
default: [zen4, zen5, gpt-4o]
curl -H "Authorization: Bearer hk-YOUR-API-KEY" \
-H "X-Max-Cost: 2.0" -H "X-Max-Latency-Ms: 800" \
https://api.hanzo.ai/v1/chat/completions \
-d '{"model":"auto","messages":[{"role":"user","content":"refactor this function"}]}'
# → response header: X-Routed-Model: zen4-coder
# → response body: {"model":"zen4-coder", ...}
Per-org enable/disable
Routing can be overridden per org via the admin OrgSettings surface
(/v1/*-org-settings, global-admin-gated like /v1/*-model-route). An org row
carries autoRouting: "" (unset), "enabled", or "disabled". It blends with
the global router.enabled flag as follows (AutoRoutingActive):
global router.enabled |
org "" (unset) |
org "enabled" |
org "disabled" |
|---|---|---|---|
| true | route | route | off (no rewrite, no header) |
| false | off | route¹ | off |
¹ Per-org opt-in routes even when the global flag is off, but ONLY when router
config is present (a prefer table or an endpoint) — otherwise there is nothing
to route with and auto is left unchanged.
When routing is not active for an org, an auto/zen-router request is left
untouched — no X-Routed-Model header, exactly its pre-routing behavior. Set an
org's preference with, e.g.:
curl -X POST "$CLOUD_ADMIN/v1/update-org-settings?owner=<org>" \
-H "Content-Type: application/json" -b "$ADMIN_SESSION" \
-d '{"owner":"<org>","autoRouting":"disabled"}'
- Transparency: the
X-Routed-Modelresponse header (and themodelfield in the body) report the model that served the request. - SLO (optional):
X-Max-Cost(per-1k, float) andX-Max-Latency-Ms(int) headers express a per-request budget, forwarded to the engine strategy. - Two strategies: with
endpointset, ai POSTs{prompt, tasks?, slo}to{endpoint}/routeand uses its{model, task, confidence}reply — a learned zen-router (huggingface.co/zenlm/zen-router) served OpenAI-compatibly by hanzo-engine. On any error (or with no endpoint) it falls back to a built-in pure-Go heuristic (a faithful port of hanzo-router's classifier), soautoworks with zero extra infrastructure. The learned zen-router is the quality upgrade over the heuristic — same interface, better routing. - Per-org: routing enable is deployment-global (there is no per-org settings
object yet); the resolved model id still goes through per-org
ModelRouteresolution and pricing, so orgs keep their overrides for whateverautopicks.
Quick Start
# Build
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -ldflags="-w -s" -o cloud-api-server .
# Run
./cloud-api-server
# Test
curl -H "Authorization: Bearer hk-YOUR-API-KEY" \
https://api.hanzo.ai/v1/chat/completions \
-d '{"model":"gpt-4o","messages":[{"role":"user","content":"Hello"}]}'
Configuration
Set via conf/app.conf or environment variables:
| Variable | Description |
|---|---|
HANZO_API_KEY |
Unified service token for internal IAM + KMS operations (billing/usage/auth support) |
iamEndpoint |
IAM service URL (production: http://iam.hanzo.svc.cluster.local:8000) |
clientId |
IAM OAuth client ID for cloud |
clientSecret |
IAM OAuth client secret for cloud |
dataSourceName |
Database DSN (do not commit; inject via KMS-managed secret) |
KMS_CLIENT_ID |
KMS Universal Auth client ID |
KMS_CLIENT_SECRET |
KMS Universal Auth client secret |
KMS_PROJECT_ID |
Default KMS project ID |
KMS_ENVIRONMENT |
KMS environment (default: production) |
Deploy
# Docker
docker pull ghcr.io/hanzoai/cloud:latest
# Kubernetes (production)
kubectl apply -f k8s/kms-secrets.yaml
kubectl apply -f k8s/configmap.yaml
kubectl apply -f k8s/deployment.yaml
kubectl apply -f k8s/service.yaml
GitHub Actions production deploy (.github/workflows/deploy-production.yml) resolves deployment credentials from Hanzo KMS. Preferred setup is a single GitHub secret: HANZO_API_KEY. Universal Auth (KMS_CLIENT_ID + KMS_CLIENT_SECRET) remains as a fallback.
License
Documentation
¶
Overview ¶
Package ai mounts the Hanzo AI subsystem (LLM control plane, RAG, model hub, MCP management) into the unified cloud binary per HIP-0106.
The legacy entry point at ~/work/hanzo/ai/main.go registers the existing beego ControllerRegister tree. Mount adapts that same ControllerRegister onto a zip.App via zip.AdaptNetHTTP so the routes continue to operate unchanged while running under the canonical zip-driven cloud entry.
All ~309 X-Org-Id call-sites inside controllers/* continue to read gateway-minted identity headers (X-Org-Id, X-User-Id, X-User-Email) per HIP-0026 — the adapter does not strip headers; zip middleware in the cloud binary already mints them from the JWT before forwarding.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func BillingQueue ¶ added in v1.785.4
func BillingQueue() *util.BillingQueue
BillingQueue returns the Commerce billing usage queue created by Bootstrap, or nil if none was configured / Bootstrap did not reach it.
func Bootstrap ¶ added in v1.785.4
func Bootstrap() error
Bootstrap performs the AI runtime initialization shared by BOTH entrypoints — the standalone server (cmd/aid) and the embedded unified cloud binary (Mount, per HIP-0106). It is the SINGLE source of the runtime boot sequence: DB + adapter + tables, model/pricing config, the HTTP client, GeoIP/parser, the background maintenance tasks, the Commerce-backed balance gate + tier cache + per-key rate limiter, the full beego BeforeRouter/AfterExec filter chain, the session config, and the billing usage queue. After wiring those it publishes the fully-configured beego ControllerRegister via SetHandler so the unified binary's /v1/ai/* adapter stops returning 503 "ai runtime not initialized".
What Bootstrap deliberately does NOT do (those are standalone-only concerns that the embedded binary owns differently, and several would break or collide when co-resident in the unified process):
- It binds NO listeners. The native ZAP inference node (port 9999), the inter-service ZAP transport (CLOUD_ZAP_PORT, default 9320) and beego.Run(:httpport) stay in cmd/aid. The unified binary serves the beego handler through zip on its own :8080 and runs its own ZAP at :9653; starting the legacy nodes here would collide.
- It calls NO util.StopOldInstance — that races on the legacy beego port and is meaningless when beego never listens.
- It installs NO signal handler and never calls os.Exit. cloud.Serve owns graceful shutdown for the unified binary; cmd/aid keeps its own drain goroutine, wired to the handles returned here.
Bootstrap is guarded by sync.Once: it is safe to call more than once, so Mount can call it at mount time and the standalone can call it explicitly without double-initializing global runtime state. The cached result (including the error) is returned on every call.
It returns an error rather than panicking: several init steps (notably the DB open) panic deep inside on a missing/unreachable backend, which would take down the whole multi-subsystem cloud binary. Bootstrap recovers those into a precise error so Mount can surface "mount ai: bootstrap: ..." and the operator sees exactly what failed.
The rate limiter and billing queue created here are exposed via RateLimiter() and BillingQueue() for the standalone's graceful-drain wiring. Either may be nil (e.g. the billing queue is nil when no Commerce endpoint is configured).
func Mount ¶
Mount registers AI's HTTP surface per HIP-0106 AND initializes the AI runtime so those routes actually serve.
Routes under /v1/ai/* are forwarded to the registered handler (the beego ControllerRegister built by routers/router.go). The MountSpec contract (cloud.MountAll) gives each subsystem exactly one hook — Mount — and it owns BOTH route wiring and runtime initialization. So Mount calls Bootstrap(), the single shared boot sequence (DB, model config, balance/tier/rate-limit, beego filters, billing queue) that ends by publishing the handler via SetHandler. Without that call the adapter's getHandler() stays nil and every /v1/ai/* request 503s with "ai runtime not initialized" — the exact defect this fixes.
The same Bootstrap() runs in the standalone cmd/aid entrypoint, so the runtime is defined ONCE and behaves identically embedded or standalone. Bootstrap is sync.Once-guarded, so calling it here is safe even if the process also calls it elsewhere.
func RateLimiter ¶ added in v1.785.4
func RateLimiter() *routers.RateLimiter
RateLimiter returns the per-key rate limiter created by Bootstrap, or nil if Bootstrap has not run (or failed before creating it).
func SetHandler ¶
SetHandler registers the ai runtime's public HTTP handler (typically beego.BeeApp.Handlers after routers/router.go init). Safe for concurrent use; pass nil to deactivate.
Types ¶
This section is empty.
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
cmd
|
|
|
aid
command
|
|
|
check_ddl
command
|
|
|
pg2sqlite
command
Command pg2sqlite migrates cloud-api's dbx-managed Postgres database to a SQLite file.
|
Command pg2sqlite migrates cloud-api's dbx-managed Postgres database to a SQLite file. |
|
Package router turns a chat request into a concrete model id.
|
Package router turns a chat request into a concrete model id. |