Documentation
¶
Index ¶
- Constants
- Variables
- func DefaultProxyTransport() *http.Transport
- func NewAnthropicProxyHandler(upstream *url.URL, transport http.RoundTripper) http.Handler
- func NewAuthSwapTransport(next http.RoundTripper, token string) http.RoundTripper
- func NewHealthzHandler() http.Handler
- func NewLoggingRoundTripper(inner http.RoundTripper) http.RoundTripper
- func NewModelRouter(routes []ModelRoute, defaultProviderName string, defaultHandler http.Handler, ...) http.Handler
- func NewNotFoundHandler() http.Handler
- func NewSetLoglevelHandler() http.Handler
- func NewSetLoglevelHandlerWithRevert(autoRevert time.Duration) http.Handler
- func RedactHeadersForLog(h http.Header) map[string]string
- type Metrics
- type ModelRoute
Constants ¶
const ( SetLoglevelDefault = 1 SetLoglevelAutoRevert = 5 * time.Minute )
SetLoglevelDefault is the verbosity we revert to after autoRevertAfter. V(1) keeps the structured per-request line; the operator bumps to 2/3 for alias/route detail or to higher tiers for deep debug.
Variables ¶
var LatencyBucketsSeconds = []float64{0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30, 60}
LatencyBucketsSeconds covers expected LLM end-to-end latency distribution: sub-second for cached / token-counter calls, multi- second for streaming completions, up to ~60s for long contexts. Bucket boundaries match what a Grafana p95/p99 panel on `histogram_quantile(0.95, ...ccrouter_request_duration_seconds...)` can meaningfully resolve.
Functions ¶
func DefaultProxyTransport ¶ added in v0.3.0
DefaultProxyTransport returns an http.Transport with explicit timeouts suitable for upstream LLM API calls. Generous ResponseHeaderTimeout because long-generation requests (`/compact` on a large session, big code-gen prompts) can delay 60-300s before Anthropic sends the first byte of headers; short Dial because connections are quick HTTPS to api.anthropic.com.
Observed 2026-06-28: previous 60s ResponseHeaderTimeout produced `net/http: timeout awaiting response headers` 502s on `/compact` that took several minutes total. 300s (5 min) is generous enough for the worst observed case while still bounding a genuinely-wedged connection.
func NewAnthropicProxyHandler ¶ added in v0.3.0
NewAnthropicProxyHandler returns an HTTP handler that reverse-proxies every incoming request to upstream (typically https://api.anthropic.com).
The Authorization header passes through unchanged — this is what lets Claude Code's subscription OAuth bearer travel through the router to Anthropic without the router ever holding it. No body parsing, no model-based routing: that's task 3. v1 of task 2 = single upstream, verbatim forward.
Upstream errors (connection refused, 5xx before body, etc.) are logged server-side with the full error string for debugging, but the client sees only a generic "502 Bad Gateway / upstream unavailable" — the internal error details (IPs, TLS handshake failures, connection strings) are not leaked.
If transport is nil, DefaultProxyTransport is used.
func NewAuthSwapTransport ¶ added in v0.4.0
func NewAuthSwapTransport(next http.RoundTripper, token string) http.RoundTripper
NewAuthSwapTransport wraps next so each outbound request has its Authorization header replaced with `Bearer <token>`. Used by the model router to swap the client's subscription OAuth bearer for a per-provider API token (MiniMax, Ollama, vLLM) before forwarding.
If token is empty, the wrapper is a no-op and returns next.
func NewHealthzHandler ¶
NewHealthzHandler returns a handler that responds 200 with body "OK".
func NewLoggingRoundTripper ¶ added in v0.10.0
func NewLoggingRoundTripper(inner http.RoundTripper) http.RoundTripper
NewLoggingRoundTripper wraps inner with upstream-call logging at two verbosity tiers:
V(3) [upstream.headers]: emitted before the inner RoundTrip call; dumps the outbound request headers (after the auth-swap transport has applied its Authorization rewrite) as a JSON object with credential-shaped values redacted via RedactHeadersForLog. Useful for confirming exactly what token / headers reached the provider. Enable via `curl http://127.0.0.1:8788/setloglevel/3`.
V(4) [upstream.start] / [upstream.end]: method+path on start; on end, adds TTFB (time-to-first-byte from when inner.RoundTrip was invoked until it returned with response headers) + status code (or error). Useful for debugging slow upstream behavior — distinguishes "Anthropic took 90s to send first byte" (high TTFB) from "body streaming was slow" (low TTFB, high total latency in the surrounding [req] line). Enable via `curl http://127.0.0.1:8788/setloglevel/4`.
If inner is nil, http.DefaultTransport is used (matches the nil-default pattern in NewAnthropicProxyHandler).
Silent at default V(1)-V(2).
func NewModelRouter ¶ added in v0.4.0
func NewModelRouter( routes []ModelRoute, defaultProviderName string, defaultHandler http.Handler, aliases map[string]string, sampler liblog.Sampler, metrics *Metrics, ) http.Handler
NewModelRouter returns an HTTP handler that body-parses each request's JSON `model` field, resolves it through the aliases map (single-hop, case-sensitive exact match), then dispatches to the first matching ModelRoute. Unmatched models (and non-JSON / no-model requests) fall through to defaultHandler (logged as provider=defaultProviderName). The body is fully read and replayed for the downstream handler — fine for /v1/messages JSON payloads (typically <100 KB); not suitable for unbounded upload bodies.
aliases may be nil or empty — both mean "no alias rewriting". On a hit, the body's top-level .model field is re-marshaled to the resolved value before route dispatch, so the upstream sees the full model name.
One structured `[req]` log line per request at V(1):
[req] POST /v1/messages model=m3 alias=MiniMax-M3-highspeed provider=minimax status=200 latency=842ms
Non-200 responses are ALWAYS logged; 200 responses are gated by the sampler. `log.DefaultSamplerFactory` gives the canonical OR-combo: at most once per 10s, OR unconditionally when glog `-v` ≥ 4. This keeps the steady-state log readable while preserving every error event and giving full visibility once the operator bumps verbosity via `/setloglevel/4`.
At V(2), alias resolution and route match get their own `[alias]` / `[route]` detail lines (independent of the sampler — V(2) detail is already operator-opt-in, additional gating buys nothing).
func NewNotFoundHandler ¶ added in v0.7.0
NewNotFoundHandler returns a 404 handler that logs the unknown path before responding. Registered at `/` in the factory's mux so it catches everything not matched by a more specific route (`/v1/`, `/healthz`, `/readiness`, `/metrics`, `/setloglevel/`, `/gc`).
Logged at glog V(1) — same level as `[req]` — so unknown-path probes surface in the operator's default log alongside real traffic. Useful for catching misconfigured clients (wrong base URL, typo in `/v1/messages`) and any probing of the listener.
func NewSetLoglevelHandler ¶
NewSetLoglevelHandler returns a handler that flips glog's -v verbosity at runtime. Convenience wrapper for the production default (SetLoglevelAutoRevert = 5 min); tests use NewSetLoglevelHandlerWithRevert with a short window.
func NewSetLoglevelHandlerWithRevert ¶ added in v0.6.0
NewSetLoglevelHandlerWithRevert returns a handler that flips glog's -v verbosity at runtime. URL shape is `/setloglevel/<level>`; the integer suffix is parsed and passed to `log.LogLevelSetter.Set`, which auto- reverts to SetLoglevelDefault after autoRevert so a forgotten bump can't leave the router in verbose mode indefinitely.
The LogLevelSetter is created once at handler construction (single instance, shared across requests); `Set()` itself spawns the auto- revert goroutine per call — that's upstream bborbe/log behavior, and `resetLogLevel` is idempotent so overlapping timers are harmless.
Level validation: negative values are rejected with 400; glog itself accepts any int32, but a negative verbosity is meaningless and almost always a typo (`-1` instead of `1`).
Example:
$ curl http://127.0.0.1:8788/setloglevel/3 set loglevel to 3 completed
Stdlib-mux compatible: parses the level from URL.Path directly rather than relying on gorilla/mux path vars.
func RedactHeadersForLog ¶ added in v0.10.0
RedactHeadersForLog returns a flat map of header name → value suitable for JSON marshaling. Multi-value headers are joined with ", ". Credential-shaped headers (see isCredentialHeader) have their joined value replaced with "<redacted len=N>" where N is the byte-length of the original joined string.
This helper is designed for V(3) upstream-header logging so operators can see exactly what went on the wire without leaking tokens into log files.
Types ¶
type Metrics ¶ added in v0.9.0
type Metrics struct {
RequestsTotal *prometheus.CounterVec
RequestDuration *prometheus.HistogramVec
AliasResolutions *prometheus.CounterVec
}
Metrics groups the Prometheus collectors emitted from the model router: one CounterVec for request totals labeled by provider + model + status_class (`2xx`/`4xx`/`5xx`), one HistogramVec for request latency labeled by provider + model with LLM-shaped buckets, and one CounterVec for alias resolutions (operator-side observability for `/model qwen`-style short names).
Cardinality budget: 5 providers × ~15 active models × 3 status_classes = 225 series for the requests counter; histogram adds 5×15×len(buckets) = 750 series; aliases counter bounded by the YAML config (≤10). Total ~1k series — fine for a local Prometheus scrape.
func NewMetrics ¶ added in v0.9.0
func NewMetrics() *Metrics
NewMetrics constructs the three collectors but does NOT register them. Call Register on a *prometheus.Registry to expose them; that split lets tests verify behavior against a fresh registry per spec without colliding on the global default registry.
func (*Metrics) ObserveAliasResolution ¶ added in v0.9.0
ObserveAliasResolution increments the alias counter on each hit; labels are bounded by the YAML config's `aliases:` map size.
func (*Metrics) ObserveRequest ¶ added in v0.9.0
ObserveRequest is the call-site shorthand used by NewModelRouter after each /v1/* dispatch: increments the request counter (bucketing status into 2xx/4xx/5xx to keep cardinality bounded) and observes the latency on the histogram.
func (*Metrics) Register ¶ added in v0.9.0
func (m *Metrics) Register(reg prometheus.Registerer) error
Register registers all collectors against reg. Pass a fresh registry in tests; pass the registry the /metrics endpoint scrapes in production. Returns the first registration error (if any) so caller can decide whether to abort startup.
type ModelRoute ¶ added in v0.4.0
ModelRoute pairs a glob pattern (filepath.Match syntax) with the provider name + handler to invoke when an incoming request's `model` field matches. ProviderName is what appears in the structured log (`provider=minimax`) and is the same key as in the YAML config's `providers:` map.
Source Files