gateway

gateway

HTTP gateway: routing, JWT validation, identity strip + mint, rate limit, circuit break, telemetry.

Quick start

docker run -p 8080:8080 ghcr.io/hanzoai/gateway:latest

Build dependency: GOEXPERIMENT=jsonv2

The HIP-0106 in-process gateway mount runs on hanzoai/zip, which routes every JSON path through stdlib encoding/json/v2 when the binary is compiled with GOEXPERIMENT=jsonv2. Shipped Dockerfile + CI workflow set the flag; manual builds should do the same:

GOEXPERIMENT=jsonv2 make build

Without the experiment the binary still builds and runs - zip falls back to encoding/json v1. v2 is preferred for production: ~10% faster on the edge, ~25% fewer allocations per request. The mount startup log line json_variant=encoding/json/v2 confirms it is active.

No third-party JSON library is allowed in the Hanzo Go stack - stdlib only (HIP-0106 canonical Hanzo Go stack).

What this is

gateway is the unified API entry point for the Hanzo platform. Behind hanzoai/ingress (L7 TLS), in front of every Hanzo backend service. Validates JWTs against Hanzo IAM JWKS, strips every client-supplied identity header (X-User-*, X-Org-Id, ...), mints the three canonical identity headers from the JWT, then forwards in-process (when co-resident under hanzoai/cloud) or over the wire to downstream services.

Specs

Implements:

• HIP-0026 IAM (identity-header contract)
• HIP-0106 Unified Cloud Binary (gateway subsystem — already exposes Mount())

Architecture

   Internet  ->  hanzoai/ingress (TLS, L7)  ->  hanzoai/gateway
                                                    |
                                          JWT validation (IAM JWKS)
                                                    |
                          strip client X-* identity headers (unconditional)
                                                    |
                          mint X-User-Id / X-Org-Id / X-Roles from JWT
                                                    |
                          rate-limit, circuit-break, telemetry
                                                    |
                          in-process route to cloud subsystems
                                          OR
                          over-the-wire to standalone services

Hanzo Gateway

High-performance API gateway for Hanzo AI services. Routes 147+ API endpoints across production clusters with rate limiting, authentication forwarding, CORS, circuit breakers, and telemetry -- all driven by declarative JSON configuration.

Overview

Hanzo Gateway is the unified API entry point for all Hanzo platform traffic. It sits behind Hanzo Ingress (L7 reverse proxy) and routes requests to internal services with per-endpoint rate limiting, header forwarding, and circuit breaker protection.

The gateway can also be deployed independently by other organizations with their own configuration.

For full documentation, see docs.hanzo.ai/docs/services/gateway.

Architecture

                    Internet
                       |
              +--------+--------+
              |  Cloudflare CDN |
              +--------+--------+
                       |
              +--------+---------+
              | Hanzo Ingress    |
              | (L7 TLS/routing) |
              +--------+---------+
                       |
              +--------+---------+
              | Hanzo Gateway    |
              | 133 endpoints    |
              +---+----+----+----+
                  |    |    |
               Cloud  IAM  Commerce
               API         API

API Endpoints

OpenAI-Compatible LLM Routes (api.hanzo.ai)

These endpoints are fully compatible with the OpenAI API format. Point any OpenAI SDK client at https://api.hanzo.ai and it works out of the box.

Platform Service Routes (api.hanzo.ai)

All platform routes are available at both /{service}/* and /v1/{service}/*.

Monitoring Endpoints

Model Routing

Hanzo Gateway proxies all LLM requests through the Hanzo Cloud API (cloud-api), which handles model routing, load balancing, and provider selection. The gateway itself is provider-agnostic -- it forwards authenticated requests and streams responses back to the client.

How It Works

Client                Gateway              Cloud API            Provider
  |                      |                     |                    |
  |-- POST /v1/chat ---->|                     |                    |
  |   model: "zen4"      |-- forward --------->|                    |
  |                      |                     |-- route to tier -->|
  |                      |                     |   (Fireworks)      |
  |<---- streaming ------|----- streaming -----|<--- streaming -----|

1. The client sends a request to api.hanzo.ai/v1/chat/completions with a model field.
2. The gateway forwards the request (with all auth headers) to the Cloud API backend.
3. The Cloud API resolves the model name to a provider and endpoint based on the model's tier and availability.
4. Responses stream back through the gateway to the client with no buffering.

Model Tiers

The gateway does not need to know about model tiers -- it passes all requests to the Cloud API, which handles routing logic, fallback, and retries. Model availability is returned by GET /v1/models.

Authentication

All requests to api.hanzo.ai require a valid API key. Keys are issued through the Hanzo Console and scoped to a project.

API Key Authentication

Pass your API key in the Authorization header using the Bearer scheme:

curl https://api.hanzo.ai/v1/chat/completions \
  -H "Authorization: Bearer hk_live_..." \
  -H "Content-Type: application/json" \
  -d '{
    "model": "zen4-pro",
    "messages": [{"role": "user", "content": "Hello"}]
  }'

Auth Flow

Client --> Gateway --> Cloud API --> IAM (hanzo.id)
                                        |
                                    Validate key
                                    Resolve org/project
                                    Check rate limits
                                    Return user context

The gateway validates bearer JWTs against IAM (hanzo.id) using JWKS and re-emits the canonical 3 identity headers. Opaque API keys (hk-, sk-, fw_, hz_, pk-) pass through to the backend services for validation.

Header Forwarding

After JWT validation, the gateway emits exactly three canonical identity headers to downstream services and strips every other vendor/legacy variant on ingress:

• X-User-Id -- user ID from JWT sub claim
• X-Org-Id -- org slug from JWT owner claim
• X-Roles -- comma-joined role names from JWT roles claim

Auxiliary headers emitted by the gateway (derivatives of the JWT):

• X-User-Email -- email from JWT email claim
• X-Phone-Number -- phone from JWT phone_number/phone claim
• X-User-IsAdmin -- "true" when the JWT asserts isAdmin

Standard passthrough headers:

• Authorization -- Bearer token or API key
• Content-Type -- Request body encoding
• Accept -- Response format preference
• X-Request-ID -- Client-provided request tracing ID

Headers stripped unconditionally on ingress (never trusted from clients): X-User-Id, X-Org-Id, X-Roles, X-User-Email, X-Phone-Number, X-User-IsAdmin, X-User-Role, X-User-Roles, X-User-Name, X-Tenant-Id, X-Org, and every X-IAM-* / X-HANZO-* variant.

Rate Limiting

The gateway enforces rate limits at two levels: global (across all clients) and per-client (by IP address).

Global Configuration

{
  "extra_config": {
    "qos/ratelimit/router": {
      "max_rate": 5000,
      "client_max_rate": 100,
      "strategy": "ip"
    }
  }
}

Per-Endpoint Overrides

Individual endpoints can override the global limits. This is useful for high-traffic inference routes or sensitive administrative endpoints:

{
  "endpoint": "/v1/chat/completions",
  "method": "POST",
  "extra_config": {
    "qos/ratelimit/router": {
      "max_rate": 10000,
      "client_max_rate": 50,
      "strategy": "ip",
      "every": "1s"
    }
  }
}

The every field sets the time window for the rate counter. Default is "1s" (per second). Set to "1m" for per-minute limits.

Rate Limit Responses

When a client exceeds their rate limit, the gateway returns:

HTTP/1.1 429 Too Many Requests
Content-Type: application/json

{"message": "rate limit exceeded"}

Observability

Logging

Structured logging is enabled by default with the [GATEWAY] prefix:

{
  "extra_config": {
    "telemetry/logging": {
      "level": "INFO",
      "prefix": "[GATEWAY]",
      "syslog": false,
      "stdout": true
    }
  }
}

Log levels: DEBUG, INFO, WARNING, ERROR, CRITICAL.

Health Check

# Gateway health (always returns 200 when the process is up)
curl http://localhost:8080/__health

# Application health (checks backend connectivity)
curl https://api.hanzo.ai/health

Metrics

The gateway exposes Prometheus-compatible metrics for scraping. Key metrics include:

• Request count by endpoint and status code
• Response latency histograms
• Backend connection pool utilization
• Circuit breaker state transitions
• Rate limiter rejection counts

Circuit Breakers

Backend failures are automatically isolated. When a backend exceeds the error threshold, the circuit opens and requests are rejected immediately until the backend recovers. This prevents cascade failures across services.

Quick Start

Build from Source

# Build gateway binary
make build

# Build ingress sidecar binary
make build-ingress

# Run tests
make test

# Validate all configs
make validate

Run Locally

# Run with default config
./gateway run -c configs/hanzo/gateway.json

Docker

# Pull and run the latest image
docker run -p 8080:8080 ghcr.io/hanzoai/gateway:latest

# Build from source
make docker

Docker Compose

services:
  gateway:
    image: ghcr.io/hanzoai/gateway:latest
    ports:
      - "8080:8080"
    volumes:
      - ./configs/hanzo/gateway.json:/etc/gateway/gateway.json:ro
    healthcheck:
      test: ["CMD", "wget", "-qO-", "http://localhost:8080/__health"]
      interval: 10s
      timeout: 3s
      retries: 3
    restart: unless-stopped

Save as compose.yml and run:

docker compose up -d

Production Deployment

Hanzo Gateway runs on the hanzo-k8s DOKS cluster (do-sfo3-hanzo-k8s) in the hanzo namespace. Continuous deployment is handled by GitHub Actions -- every push to main builds a new image, applies the ConfigMap, and performs a rolling restart.

Deploy

# Apply config and restart pods
make deploy-hanzo

# Check status
make status

# Tail logs
make logs-hanzo

Infrastructure Details

K8s Manifests

k8s/
  hanzo/
    deployment.yaml     # Gateway deployment (2 replicas)
    service.yaml        # ClusterIP service
    ingress.yaml        # Ingress resource for api.hanzo.ai

Configuration

All routing is defined in JSON configuration files. Each cluster has its own config.

Editing Routes

1. Edit the config file:

   $EDITOR configs/hanzo/gateway.json
   

2. Validate the config:

   make validate
   

3. Deploy:

   make deploy-hanzo
   

The Makefile creates a ConfigMap from the JSON file and triggers a rolling restart.

Config Structure

{
  "version": 3,
  "name": "Hanzo API Gateway",
  "port": 8080,
  "timeout": "120s",
  "extra_config": {
    "router": {
      "return_error_msg": true
    },
    "qos/ratelimit/router": {
      "max_rate": 5000,
      "client_max_rate": 100,
      "strategy": "ip"
    },
    "telemetry/logging": {
      "level": "INFO",
      "prefix": "[GATEWAY]",
      "stdout": true
    }
  },
  "endpoints": [
    {
      "endpoint": "/v1/chat/completions",
      "method": "POST",
      "input_headers": ["*"],
      "output_encoding": "no-op",
      "backend": [{
        "url_pattern": "/api/chat/completions",
        "host": ["http://cloud-api.hanzo.svc.cluster.local:8000"],
        "encoding": "no-op"
      }]
    }
  ]
}

Repository Structure

configs/
  hanzo/
    gateway.json        # Hanzo API Gateway config (133 endpoints)
    ingress.json        # Hanzo Ingress sidecar config
k8s/
  hanzo/                # K8s manifests for hanzo-k8s cluster
cmd/
  gateway/              # Gateway binary entry point
  ingress/              # Ingress sidecar binary entry point
tests/                  # Integration tests
Dockerfile              # Multi-stage build (Go 1.25 + Alpine 3.23)
Makefile                # Build, test, validate, deploy commands

DNS

Related Projects

Hanzo Gateway is one of four products in the Hanzo AI infrastructure stack:

Internet -> Ingress (TLS/L7) -> Gateway (API routing) -> Engine (inference) / Cloud API / Services
                                                          Edge (on-device, client-side)

See also:

• Hanzo Cloud API -- Backend that handles model routing and provider selection
• Hanzo LLM Gateway -- Unified proxy for 100+ LLM providers (used by Cloud API)
• Hanzo MCP -- Model Context Protocol tools (260+ tools)
• Hanzo SDK (Python) -- Python client library
• Hanzo SDK (JS) -- TypeScript client library

License

MIT -- see LICENSE.