README
¶
ai
ai is the AI subsystem of the Hanzo cloud: it serves the /v1 inference surface, routes
each request to a model, meters it, and carries the RAG and model-hub surfaces alongside.
Pure Go.
It used to be the whole hanzoai/cloud repository. When the unified binary took the
cloud name (HIP-0106), this became ai and now mounts as the ai subsystem inside it.
That is why the root of this repository is package ai, a library, and not a main.
Call it
The hosted instance is api.hanzo.ai. Get a key at hanzo.ai:
curl -H "Authorization: Bearer sk-YOUR-API-KEY" \
https://api.hanzo.ai/v1/chat/completions \
-d '{"model":"zen5","messages":[{"role":"user","content":"Hello"}]}'
Three auth modes are accepted: a Hanzo API key (sk-*), a JWT from hanzo.id OAuth, or a
provider key you already hold, in which case the request is routed with your account
rather than metered on ours.
Run it yourself
docker run -p 8000:8000 ghcr.io/hanzoai/ai:vX.Y.Z # pin a release, never :latest
Or build the server binary. It is cmd/aid — building the repository root gives you a
library archive, not something you can run:
CGO_ENABLED=0 go build -o aid ./cmd/aid
./aid -h
compose.yml brings up the server with its Postgres.
Models
The gateway owns the catalog; this service does not keep a second list. Ask it:
curl -H "Authorization: Bearer sk-YOUR-API-KEY" https://api.hanzo.ai/v1/models
https://catalog.hanzo.ai/v1/models answers the same question without a key. Zen is our
own family — enso, enso-ultra and the zen5 line, owned_by: hanzo. Models you
connect with your own provider keys are routed as well, resolved from Hanzo KMS and scoped
to your org.
Auto-routing
Send "model": "auto" (alias "zen-router") and the request is classified into a coarse
task — code, reasoning, math, creative, vision, long-context, cheap-chat, general — and
mapped to the first servable model in the matching preference list, before pricing and
billing resolve. So an auto request is billed and reported as exactly the model that
served it.
Turn it on in conf/models.yaml (the cloud-api-models ConfigMap in production), or with
ROUTER_ENABLED=true / ROUTER_ENDPOINT=<url>:
router:
enabled: true
endpoint: "" # "" = built-in heuristic; a URL = the learned router
cost_ceiling: 0.0
prefer:
code: [zen5-coder]
reasoning: [enso-ultra, enso]
cheap_chat: [zen5-mini, zen5-flash]
default: [zen5]
curl -H "Authorization: Bearer sk-YOUR-API-KEY" \
-H "X-Max-Cost: 2.0" -H "X-Max-Latency-Ms: 800" \
https://api.hanzo.ai/v1/chat/completions \
-d '{"model":"auto","messages":[{"role":"user","content":"refactor this function"}]}'
# response header: X-Routed-Model: <the model that served it>
X-Routed-Model and the model field in the body always report what actually ran.
X-Max-Cost (per 1k, float) and X-Max-Latency-Ms (int) express a per-request budget.
Two strategies sit behind one interface. With endpoint set, ai POSTs
{prompt, tasks?, slo} to {endpoint}/route and takes its {model, task, confidence}
reply — that is the learned router,
zenlm/zen-router. On any error, or with no
endpoint configured, it falls back to a built-in Go heuristic, so auto works with no
extra infrastructure.
Per-org override
Orgs can opt in or out through the admin OrgSettings surface (/v1/*-org-settings,
global-admin-gated like /v1/*-model-route). The org row carries autoRouting: ""
(unset), "enabled" or "disabled", and blends with the global flag:
global router.enabled |
org "" |
org "enabled" |
org "disabled" |
|---|---|---|---|
| true | route | route | off — no rewrite, no header |
| false | off | route¹ | off |
¹ Per-org opt-in routes even when the global flag is off, but only when router config is
present (a prefer table or an endpoint) — otherwise there is nothing to route with and
auto is left unchanged.
Configuration
conf/app.conf, or environment variables:
| Variable | Description |
|---|---|
HANZO_API_KEY |
Service token for internal IAM and KMS calls (billing, usage, auth) |
iamEndpoint |
IAM service URL |
clientId / clientSecret |
IAM OAuth client credentials for ai |
dataSourceName |
Database DSN — inject from KMS, never commit it |
KMS_CLIENT_ID / KMS_CLIENT_SECRET |
KMS Universal Auth credentials |
KMS_PROJECT_ID |
Default KMS project |
KMS_ENVIRONMENT |
KMS environment (default production) |
Secrets come from Hanzo KMS; the deploy workflow resolves
them at run time from a single HANZO_API_KEY GitHub secret, with Universal Auth as the
fallback.
airuns single-replica: its balance ledger is an in-pod invariant. Deploy withstrategy: Recreateandmin=max=1— a boot assertion panics ifCLOUD_API_REPLICAS > 1.LLM.mdhas the scale-out path.
Development
go build -race ./... # build
go test $(go list ./...) -tags skipCi # test (needs MySQL)
golangci-lint run # lint
cd web && yarn install && yarn start # the admin UI
LLM.md is the deep reference — architecture, the mount seam into cloud, and the
conventions that apply here. ZAP.md covers the transport.
Lineage
NOTICE. The routing, auth, billing and model surfaces are ours; the admin UI
and knowledge-base scaffolding started there.
License
Apache-2.0 — see LICENSE.
Hanzo — the open AI cloud. hanzo.ai · docs.hanzo.ai
SDKs: Python · TypeScript · Go · Rust · C++ · Swift · Kotlin · umbrella
Documentation
¶
Overview ¶
Package ai mounts the Hanzo AI subsystem (LLM control plane, RAG, model hub, MCP management) into the unified cloud binary per HIP-0106.
App builds the zip.App and routers.Register binds every route onto it natively; there is no net/http router underneath and nothing is adapted onto zip. The adaptation runs the other way, once, at Handler: a host that wants an http.Handler gets the whole App exposed as one. The only net/http left inside is the /v1/voice trio, and routers/router.go says why.
All ~309 X-Org-Id call-sites inside controllers/* continue to read gateway-minted identity headers (X-Org-Id, X-User-Id, X-User-Email) per HIP-0026 — the adapter does not strip headers; zip middleware in the cloud binary already mints them from the JWT before forwarding.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func App ¶ added in v1.833.14
func App(secrets object.SecretStore) (*zip.App, error)
Mount registers AI's HTTP surface per HIP-0106 AND initializes the AI runtime so those routes actually serve.
Routes under /v1/ai/* are forwarded to the registered handler (the the router ControllerRegister built by routers/router.go). The MountSpec contract (cloud.MountAll) gives each subsystem exactly one hook — Mount — and it owns BOTH route wiring and runtime initialization. So Mount calls Bootstrap(), the single shared boot sequence (DB, model config, balance/tier/rate-limit, filters, billing queue) that ends by publishing the handler via SetHandler. Without that call the adapter's getHandler() stays nil and every /v1/ai/* request 503s with "ai runtime not initialized" — the exact defect this fixes.
The same Bootstrap() runs in the standalone cmd/aid entrypoint, so the runtime is defined ONCE and behaves identically embedded or standalone. Bootstrap is sync.Once-guarded, so calling it here is safe even if the process also calls it elsewhere. SECRETS ARRIVE AS AN INTERFACE, NOT AS THE HOST'S Deps. ai is a SUBSYSTEM: it is mounted by a host, and a subsystem that imports its host cannot be mounted by a second one — which is exactly what happened. hanzoai/cloud has two editions, and because this package took cloud.Deps, the private edition's cloud.Deps and the OSS edition's were different types with the same name and neither could mount ai.
object.SetSecretStore already took ai's OWN interface (object.SecretStore: GetSecret/PutSecret), so cloud.Deps was carried across the boundary to reach one field and then discarded. Taking the interface directly costs the host one argument and removes the whole dependency: ai now imports zip and nothing of its host's.
func BillingQueue ¶ added in v1.785.4
func BillingQueue() *util.BillingQueue
BillingQueue returns the Commerce billing usage queue created by Bootstrap, or nil if none was configured / Bootstrap did not reach it.
func Bootstrap ¶ added in v1.785.4
func Bootstrap() error
Bootstrap performs the AI runtime initialization shared by BOTH entrypoints — the standalone server (cmd/aid) and the embedded unified cloud binary (Mount, per HIP-0106). It is the SINGLE source of the runtime boot sequence: DB + adapter + tables, model/pricing config, the HTTP client, GeoIP/parser, the background maintenance tasks, the Commerce-backed balance gate + tier cache + per-key rate limiter, the full BeforeRouter/AfterExec filter chain, the session config, and the billing usage queue. After wiring those it publishes the fully-configured native router via SetHandler so the unified binary's /v1/ai/* adapter stops returning 503 "ai runtime not initialized".
What Bootstrap deliberately does NOT do (those are standalone-only concerns that the embedded binary owns differently, and several would break or collide when co-resident in the unified process):
- It binds NO listeners. The native ZAP inference node (port 9999), the inter-service ZAP transport (CLOUD_ZAP_PORT, default 9320) and the standalone HTTP listener (:httpport) stay in cmd/aid. The unified binary serves routers.App through zip on its own :8080 and runs its own ZAP at :9653; starting the legacy nodes here would collide.
- It calls NO util.StopOldInstance — that races on the legacy standalone port and is meaningless when the embedded binary never listens there.
- It installs NO signal handler and never calls os.Exit. cloud.Serve owns graceful shutdown for the unified binary; cmd/aid keeps its own drain goroutine, wired to the handles returned here.
Bootstrap is guarded by sync.Once: it is safe to call more than once, so Mount can call it at mount time and the standalone can call it explicitly without double-initializing global runtime state. The cached result (including the error) is returned on every call.
It returns an error rather than panicking: several init steps (notably the DB open) panic deep inside on a missing/unreachable backend, which would take down the whole multi-subsystem cloud binary. Bootstrap recovers those into a precise error so Mount can surface "mount ai: bootstrap: ..." and the operator sees exactly what failed.
The rate limiter and billing queue created here are exposed via RateLimiter() and BillingQueue() for the standalone's graceful-drain wiring. Either may be nil (e.g. the billing queue is nil when no Commerce endpoint is configured).
func Document ¶ added in v1.833.90
Document is the OpenAPI projection of the app this process built, or nil before there is one.
It reads the SAME live app Handler serves, for the same reason Handler resolves its router per request: the routes are a projection of the built app, so a document taken from anything else can describe a surface this process is not serving. A host that mounts this module (hanzoai/cloud publishes it into the fleet document) has no app of its own to pass, and should not need one to ask what we serve.
func Handler ¶ added in v1.833.89
Handler is the HTTP surface as an http.Handler, for the ZAP transports that carry an HTTP request over the binary protocol (controllers.InitZapHandlers and InitForwardBridge). It is the same app the socket serves, so a bridged request meets every filter a socketed one does.
The router is resolved PER REQUEST, never captured. App.Fiber() hands out the current generation's router and a generation is a projection — the next build materialises a fresh one, and a captured pointer would keep serving the old table. zip panics on the mirror-image mistake for the same reason.
func RateLimiter ¶ added in v1.785.4
func RateLimiter() *routers.RateLimiter
RateLimiter returns the per-key rate limiter created by Bootstrap, or nil if Bootstrap has not run (or failed before creating it).
Types ¶
This section is empty.
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
Package address names the caller's address on the wire between a host in front and this module.
|
Package address names the caller's address on the wire between a host in front and this module. |
|
builtin_tool/web
Package webtools gives every Responses-API agent the three capabilities the platform already serves and no agent could reach: web search, page fetch, and deep research.
|
Package webtools gives every Responses-API agent the three capabilities the platform already serves and no agent could reach: web search, page fetch, and deep research. |
|
cmd
|
|
|
aid
command
|
|
|
check_ddl
command
|
|
|
pg2sqlite
command
Command pg2sqlite migrates cloud-api's dbx-managed Postgres database to a SQLite file.
|
Command pg2sqlite migrates cloud-api's dbx-managed Postgres database to a SQLite file. |
|
routerdoc
command
Command routerdoc lifts each hand-written route's sentence out of the Go doc comment on the handler it names, into routers/wired_gen.go.
|
Command routerdoc lifts each hand-written route's sentence out of the Go doc comment on the handler it names, into routers/wired_gen.go. |
|
Package funding is the cash circuit-breaker.
|
Package funding is the cash circuit-breaker. |
|
Hedging: ask several providers at once and keep the first answer.
|
Hedging: ask several providers at once and keep the first answer. |
|
internal
|
|
|
authtest
Package authtest mints a credential a test can present.
|
Package authtest mints a credential a test can present. |
|
gemini
Package gemini is the ONE client for Google's Gemini API, covering exactly the four calls this module makes: countTokens, generateContent, batchEmbedContents and models.list.
|
Package gemini is the ONE client for Google's Gemini API, covering exactly the four calls this module makes: countTokens, generateContent, batchEmbedContents and models.list. |
|
iam
Package iam is ai's INTERNAL IAM client: the small, clean OIDC+REST surface ai needs to talk to Hanzo IAM (hanzo.id), decoupled from the retired SDK module github.com/hanzoai/iam-v1.
|
Package iam is ai's INTERNAL IAM client: the small, clean OIDC+REST surface ai needs to talk to Hanzo IAM (hanzo.id), decoupled from the retired SDK module github.com/hanzoai/iam-v1. |
|
Package log is the leveled logging surface for the ai runtime.
|
Package log is the leveled logging surface for the ai runtime. |
|
Package router turns a chat request into a concrete model id.
|
Package router turns a chat request into a concrete model id. |
|
Package upstream is the one place that knows where a provider lives and how a call proves it may reach one.
|
Package upstream is the one place that knows where a provider lives and how a call proves it may reach one. |