Documentation
¶
Index ¶
- Constants
- Variables
- func DefaultProxyTransport() *http.Transport
- func NewAnthropicProxyHandler(upstream *url.URL, transport http.RoundTripper) http.Handler
- func NewAuthSwapTransport(next http.RoundTripper, token string) http.RoundTripper
- func NewHealthzHandler() http.Handler
- func NewLoggingRoundTripper(inner http.RoundTripper, bodySampler liblog.Sampler, ...) http.RoundTripper
- func NewModelRouter(routes []ModelRoute, defaultProviderName string, defaultHandler http.Handler, ...) http.Handler
- func NewNotFoundHandler() http.Handler
- func NewRootLivenessHandler() http.Handler
- func NewSetLoglevelHandler() http.Handler
- func NewSetLoglevelHandlerWithRevert(autoRevert time.Duration) http.Handler
- func NewTraceMiddleware(next http.Handler, traceDir string) http.Handler
- func RedactBearerTokensInBody(b []byte) []byte
- func RedactHeadersForLog(h http.Header) map[string]string
- type Metrics
- type ModelRoute
Constants ¶
const ( SetLoglevelDefault = 1 SetLoglevelAutoRevert = 5 * time.Minute )
SetLoglevelDefault is the verbosity we revert to after autoRevertAfter. V(1) keeps the structured per-request line; the operator bumps to 2/3 for alias/route detail or to higher tiers for deep debug.
const BodySampleMaxBytes = 4 << 10 // 4 KB
BodySampleMaxBytes is the maximum number of bytes captured from a request or response body for V(4) body-sample logging. 4 KB is large enough to see the full JSON envelope of any ordinary Anthropic /messages request while staying small enough to avoid log-line bloat for large streaming bodies.
const MaxRequestBodyBytes = 32 << 20 // 32 MB
MaxRequestBodyBytes caps inbound /v1/* request bodies at 32 MB to match the Anthropic API ceiling. Long Claude Code sessions (full conversation history + tool definitions + sub-agent results) routinely exceed 1 MB, so a tighter cap surfaces as confusing 413s that read as upstream errors. 32 MB still bounds memory exhaustion from an accidental multi-GB upload via io.ReadAll. On overflow, the wrapped body returns *http.MaxBytesError; the router responds with HTTP 413 Request Entity Too Large + a generic body (no internal state leaked).
Variables ¶
var LatencyBucketsSeconds = []float64{0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30, 60}
LatencyBucketsSeconds covers expected LLM end-to-end latency distribution: sub-second for cached / token-counter calls, multi- second for streaming completions, up to ~60s for long contexts. Bucket boundaries match what a Grafana p95/p99 panel on `histogram_quantile(0.95, ...ccrouter_request_duration_seconds...)` can meaningfully resolve.
Functions ¶
func DefaultProxyTransport ¶ added in v0.3.0
DefaultProxyTransport returns an http.Transport with explicit timeouts suitable for upstream LLM API calls. Generous ResponseHeaderTimeout because long-generation requests (`/compact` on a large session, big code-gen prompts) can delay 60-300s before Anthropic sends the first byte of headers; short Dial because connections are quick HTTPS to api.anthropic.com.
Observed 2026-06-28: previous 60s ResponseHeaderTimeout produced `net/http: timeout awaiting response headers` 502s on `/compact` that took several minutes total. 300s (5 min) is generous enough for the worst observed case while still bounding a genuinely-wedged connection.
func NewAnthropicProxyHandler ¶ added in v0.3.0
NewAnthropicProxyHandler returns an HTTP handler that reverse-proxies every incoming request to upstream (typically https://api.anthropic.com).
The Authorization header passes through unchanged — this is what lets Claude Code's subscription OAuth bearer travel through the router to Anthropic without the router ever holding it. No body parsing, no model-based routing: that's task 3. v1 of task 2 = single upstream, verbatim forward.
Upstream errors (connection refused, 5xx before body, etc.) are logged server-side with the full error string for debugging, but the client sees only a generic "502 Bad Gateway / upstream unavailable" — the internal error details (IPs, TLS handshake failures, connection strings) are not leaked.
If transport is nil, DefaultProxyTransport is used.
func NewAuthSwapTransport ¶ added in v0.4.0
func NewAuthSwapTransport(next http.RoundTripper, token string) http.RoundTripper
NewAuthSwapTransport wraps next so each outbound request has its Authorization header replaced with `Bearer <token>`. Used by the model router to swap the client's subscription OAuth bearer for a per-provider API token (MiniMax, Ollama, vLLM) before forwarding.
If token is empty, the wrapper is a no-op and returns next.
func NewHealthzHandler ¶
NewHealthzHandler returns a handler that responds 200 with body "OK".
func NewLoggingRoundTripper ¶ added in v0.10.0
func NewLoggingRoundTripper( inner http.RoundTripper, bodySampler liblog.Sampler, currentDateTime libtime.CurrentDateTimeGetter, ) http.RoundTripper
NewLoggingRoundTripper wraps inner with upstream-call logging at three verbosity tiers:
V(3) [upstream.headers]: emitted before the inner RoundTrip call; dumps the outbound request headers (after the auth-swap transport has applied its Authorization rewrite) as a JSON object with credential-shaped values redacted via RedactHeadersForLog. Useful for confirming exactly what token / headers reached the provider. Enable via `curl http://127.0.0.1:8788/setloglevel/3`.
V(4) [upstream.start] / [upstream.end]: method+path on start; on end, adds TTFB (time-to-first-byte from when inner.RoundTrip was invoked until it returned with response headers) + status code (or error). Useful for debugging slow upstream behavior — distinguishes "Anthropic took 90s to send first byte" (high TTFB) from "body streaming was slow" (low TTFB, high total latency in the surrounding [req] line). Enable via `curl http://127.0.0.1:8788/setloglevel/4`.
V(4) [upstream.req.body] / [upstream.resp.body]: body-sample lines gated by bodySampler (typically SamplerList{NewSampleTime(1s), NewSamplerGlogLevel(5)}). Captures up to BodySampleMaxBytes (4 KB) of the request body before forwarding it, and wraps the response body so the same size prefix is logged on Close. Bearer tokens are redacted via RedactBearerTokensInBody before the line is emitted.
If inner is nil, http.DefaultTransport is used (matches the nil-default pattern in NewAnthropicProxyHandler).
Silent at default V(1)-V(2).
func NewModelRouter ¶ added in v0.4.0
func NewModelRouter( routes []ModelRoute, defaultProviderName string, defaultHandler http.Handler, aliases map[string]string, sampler liblog.Sampler, metrics *Metrics, currentDateTime libtime.CurrentDateTimeGetter, ) http.Handler
NewModelRouter returns an HTTP handler that body-parses each request's JSON `model` field, resolves it through the aliases map (single-hop, case-sensitive exact match), then dispatches to the first matching ModelRoute. Unmatched models (and non-JSON / no-model requests) fall through to defaultHandler (logged as provider=defaultProviderName). The body is fully read and replayed for the downstream handler — fine for /v1/messages JSON payloads (typically <100 KB); not suitable for unbounded upload bodies.
aliases may be nil or empty — both mean "no alias rewriting". On a hit, the body's top-level .model field is re-marshaled to the resolved value before route dispatch, so the upstream sees the full model name.
One structured `[req]` log line per request at V(1):
[req] POST /v1/messages model=m3 alias=MiniMax-M3-highspeed provider=minimax status=200 latency=842ms
Non-200 responses are ALWAYS logged; 200 responses are gated by the sampler. `log.DefaultSamplerFactory` gives the canonical OR-combo: at most once per 10s, OR unconditionally when glog `-v` ≥ 4. This keeps the steady-state log readable while preserving every error event and giving full visibility once the operator bumps verbosity via `/setloglevel/4`.
At V(2), alias resolution and route match get their own `[alias]` / `[route]` detail lines (independent of the sampler — V(2) detail is already operator-opt-in, additional gating buys nothing).
match route → dispatch → observe metrics → emit logs. Each step's branching is local and reads sequentially; extracting any of it into a helper buys nothing (the prior `logReq` extraction was a naive line-count fix per architecture audit 2026-06-28 — inlined back). If a second log-event shape ever needs the same data, introduce a `requestLogger` struct holding `sampler` + `metrics` then.
func NewNotFoundHandler ¶ added in v0.7.0
NewNotFoundHandler returns a 404 handler that logs the unknown path before responding. Registered at `/` in the factory's mux so it catches everything not matched by a more specific route (`/v1/`, `/healthz`, `/readiness`, `/metrics`, `/setloglevel/`, `/gc`).
Logged at glog V(1) — same level as `[req]` — so unknown-path probes surface in the operator's default log alongside real traffic. Useful for catching misconfigured clients (wrong base URL, typo in `/v1/messages`) and any probing of the listener.
func NewRootLivenessHandler ¶ added in v0.13.0
NewRootLivenessHandler returns a 200 OK handler for `HEAD /`.
Claude Code's HTTP client probes the base URL with `HEAD /` before dispatching its first /v1/messages request on a fresh connection. Without this handler the probe falls through to NewNotFoundHandler and logs a `[404] HEAD /` line ahead of every real request, which is noisy and obscures the actual `[req]` traffic.
func NewSetLoglevelHandler ¶
NewSetLoglevelHandler returns a handler that flips glog's -v verbosity at runtime. Convenience wrapper for the production default (SetLoglevelAutoRevert = 5 min); tests use NewSetLoglevelHandlerWithRevert with a short window.
func NewSetLoglevelHandlerWithRevert ¶ added in v0.6.0
NewSetLoglevelHandlerWithRevert returns a handler that flips glog's -v verbosity at runtime. URL shape is `/setloglevel/<level>`; the integer suffix is parsed and passed to `log.LogLevelSetter.Set`, which auto- reverts to SetLoglevelDefault after autoRevert so a forgotten bump can't leave the router in verbose mode indefinitely.
The LogLevelSetter is created once at handler construction (single instance, shared across requests); `Set()` itself spawns the auto- revert goroutine per call — that's upstream bborbe/log behavior, and `resetLogLevel` is idempotent so overlapping timers are harmless.
Level validation: negative values are rejected with 400; glog itself accepts any int32, but a negative verbosity is meaningless and almost always a typo (`-1` instead of `1`).
Example:
$ curl http://127.0.0.1:8788/setloglevel/3 set loglevel to 3 completed
Stdlib-mux compatible: parses the level from URL.Path directly rather than relying on gorilla/mux path vars.
func NewTraceMiddleware ¶ added in v0.14.0
NewTraceMiddleware wraps next in a handler that, for each request, captures the full request (method, path, headers, body) and full response (status, headers, body) and writes one JSON file to traceDir. Authorization and x-api-key request headers are redacted to "***" (case-insensitive header lookup); all other headers and the entire request/response bodies are logged verbatim. Trace file writes are best-effort: a write failure is logged at glog.Warningf and the request still succeeds. The trace directory is created on demand on the first write (MkdirAll, 0o700).
func RedactBearerTokensInBody ¶ added in v0.11.0
RedactBearerTokensInBody returns a copy of b with every "Bearer <token>" substring replaced by "Bearer <redacted>". The replacement is case-insensitive on the "Bearer" keyword. Returns b unchanged (same slice, no allocation) when no Bearer token is found.
func RedactHeadersForLog ¶ added in v0.10.0
RedactHeadersForLog returns a flat map of header name → value suitable for JSON marshaling. Multi-value headers are joined with ", ". Credential-shaped headers (see isCredentialHeader) have their joined value replaced with "<redacted len=N>" where N is the byte-length of the original joined string.
This helper is designed for V(3) upstream-header logging so operators can see exactly what went on the wire without leaking tokens into log files.
Types ¶
type Metrics ¶ added in v0.9.0
type Metrics struct {
RequestsTotal *prometheus.CounterVec
RequestDuration *prometheus.HistogramVec
AliasResolutions *prometheus.CounterVec
}
Metrics groups the Prometheus collectors emitted from the model router: one CounterVec for request totals labeled by provider + model + status_class (`2xx`/`4xx`/`5xx`), one HistogramVec for request latency labeled by provider + model with LLM-shaped buckets, and one CounterVec for alias resolutions (operator-side observability for `/model qwen`-style short names).
Cardinality budget: 5 providers × ~15 active models × 3 status_classes = 225 series for the requests counter; histogram adds 5×15×len(buckets) = 750 series; aliases counter bounded by the YAML config (≤10). Total ~1k series — fine for a local Prometheus scrape.
func NewMetrics ¶ added in v0.9.0
NewMetrics constructs the three collectors but does NOT register them. Call Register on a *prometheus.Registry to expose them; that split lets tests verify behavior against a fresh registry per spec without colliding on the global default registry.
aliases is used to pre-initialize the alias_resolutions counter so that `rate(...) > X` alerts evaluate to 0 (not no-data) for aliases that haven't been hit yet. A nil aliases map is safe — ranging a nil map yields zero iterations.
func (*Metrics) ObserveAliasResolution ¶ added in v0.9.0
ObserveAliasResolution increments the alias counter on each hit; labels are bounded by the YAML config's `aliases:` map size.
func (*Metrics) ObserveRequest ¶ added in v0.9.0
ObserveRequest is the call-site shorthand used by NewModelRouter after each /v1/* dispatch: increments the request counter (bucketing status into 2xx/4xx/5xx to keep cardinality bounded) and observes the latency on the histogram.
func (*Metrics) Register ¶ added in v0.9.0
func (m *Metrics) Register(reg prometheus.Registerer) error
Register registers all collectors against reg. Pass a fresh registry in tests; pass the registry the /metrics endpoint scrapes in production. Returns the first registration error (if any) so caller can decide whether to abort startup.
type ModelRoute ¶ added in v0.4.0
ModelRoute pairs a glob pattern (filepath.Match syntax) with the provider name + handler to invoke when an incoming request's `model` field matches. ProviderName is what appears in the structured log (`provider=minimax`) and is the same key as in the YAML config's `providers:` map.
Source Files