Documentation
¶
Overview ¶
Package inference's Dispatch is the shape-agnostic per-request flow. It owns: classification branching (proxy vs normal), routing resolution, translator chaining (inbound ↔ canonical ↔ upstream), pipeline invocation, and response wrapping.
Per-shape routes (registered via app/adapter.MountRoutes) own only:
- Minimal parse to extract the model name + stream flag.
- The Dispatch call with the inbound shape Name.
This keeps shape-specific files out of app/httpapi/inference/.
Package inference is the data-plane HTTP API: /v1/* and /healthz.
Mount(r, deps) wires huma+chi onto an existing chi router and returns the huma.API. The package owns its huma.Config; main.go constructs Deps and calls Mount.
Index ¶
- Constants
- Variables
- func ClassifyMiddleware() func(http.Handler) http.Handler
- func Dispatch(d Deps, w http.ResponseWriter, r *http.Request, in DispatchInput)
- func ExtractModelStream(body []byte) (model string, stream bool, err error)
- func ForwardUpstreamHeaders(dst, src http.Header)
- func MapPipelineErr(w http.ResponseWriter, err error)
- func Mount(r chi.Router, d Deps) huma.API
- func ReadinessMiddleware(cat *appcatalog.Catalog) func(http.Handler) http.Handler
- func RelayKeyAuthMiddleware(cat *appcatalog.Catalog) func(http.Handler) http.Handler
- func RelayKeyFromContext(ctx context.Context) *relaykey.RelayKey
- func RewriteModelField(body []byte, upstreamModel string) []byte
- func SplitSSEChunks(data []byte, atEOF bool) (advance int, token []byte, err error)
- func WithClassification(ctx context.Context, c Classification) context.Context
- func WriteAPIError(w http.ResponseWriter, status int, errType, code, msg string, attrs ...any)
- type Classification
- type Deps
- type DispatchInput
- type Mode
- type Pinger
- type RouteMounter
Constants ¶
const ( // HeaderOrigin is "relay" when relay minted the response (auth, // routing, rate limit, admission — or a relay verdict about an upstream // failure), "upstream" when the provider's bytes passed through. HeaderOrigin = "X-WR-Origin" // HeaderUpstreamStatus carries the provider's HTTP status on // relay-minted errors that report an upstream failure, so the two // layers' statuses are never conflated. HeaderUpstreamStatus = "X-WR-Upstream-Status" )
Error-attribution headers. Relay's error envelope and an OpenAI-shaped upstream's envelope are indistinguishable by body alone, so every response declares which side produced it: a 401 with origin "relay" means your relay key; origin "upstream" means the provider rejected the upstream key.
Variables ¶
var ( ErrInvalidProxyMode = errors.New("inference: invalid X-WR-Proxy-Mode value") ErrMissingUpstreamKey = errors.New("inference: proxy mode requires Authorization: Bearer <upstream-key>") )
Sentinel errors returned by Classify. The middleware maps these to 400.
Functions ¶
func ClassifyMiddleware ¶
ClassifyMiddleware runs Classify once per request and stashes the result on ctx. Shape-invalid requests get a 400 envelope; everything else falls through to the next handler. Gating (Settings.ProxyMode flags, unknown-slug rejection) lives downstream so this middleware stays snapshot-free.
func Dispatch ¶
func Dispatch(d Deps, w http.ResponseWriter, r *http.Request, in DispatchInput)
Dispatch runs the shape-agnostic flow. Called from route handlers after a minimal parse to extract ModelName + Stream.
func ExtractModelStream ¶
ExtractModelStream parses the caller-supplied model name and stream flag from a request body. Reused by the batch subsystem's per-item runner.
func ForwardUpstreamHeaders ¶
ForwardUpstreamHeaders copies src → dst, dropping hop-by-hop, and stamps the response origin as "upstream" (Set, not Add — an upstream must not be able to spoof a relay-origin claim). The caller is responsible for any further adjustments (e.g. clearing Content-Length when the body size will change between upstream and client). Exported so adapter packages can use it from their own cross-shape handlers.
func MapPipelineErr ¶
func MapPipelineErr(w http.ResponseWriter, err error)
MapPipelineErr is the exported form for adapter-side cross-shape handlers that drive pipeline.Pipeline.Run directly.
func Mount ¶
Mount installs the data-plane huma API on r and registers all operations. Returns the huma.API so the caller can attach test-only operations.
func ReadinessMiddleware ¶
ReadinessMiddleware short-circuits inbound /v1/* requests with 503 until the catalog has built its first snapshot. Until then, lookups against the empty zero-value snapshot would look identical to a legitimately empty catalog — better to refuse traffic explicitly so the caller retries instead of seeing a misleading 404. /healthz is mounted outside this middleware so liveness/readiness probes can still report.
func RelayKeyAuthMiddleware ¶
RelayKeyAuthMiddleware authenticates the inbound relay key according to the request's Mode classification (set by ClassifyMiddleware upstream):
- ModeNormal — relay key is required; lookup must succeed.
- ModeProxyAuthed — relay key is required; lookup must succeed.
- ModeProxyAnonymous — no relay key; this middleware is a no-op and no *RelayKey is stashed on ctx.
Snapshot is read on every request so admins toggling Enabled / RevokedAt take effect within the NOTIFY debounce window.
func RelayKeyFromContext ¶
RelayKeyFromContext returns the authenticated relay key from ctx, or nil if no relay-key middleware fired.
func RewriteModelField ¶
RewriteModelField returns body with its "model" field rewritten to upstreamModel (the resolved upstream model name). Reused by the batch subsystem's per-item runner.
func SplitSSEChunks ¶
SplitSSEChunks is exported so adapter packages can use the same SSE chunking logic in their cross-shape stream handlers.
func WithClassification ¶
func WithClassification(ctx context.Context, c Classification) context.Context
WithClassification returns a child ctx carrying c.
func WriteAPIError ¶
func WriteAPIError(w http.ResponseWriter, status int, errType, code, msg string, attrs ...any)
WriteAPIError emits an OpenAI-shape error envelope. Exported so per-shape route packages (app/adapters/<name>/routes.go) can use it without depending on shape-specific helpers. Extra slog attrs are attached to the warning log only — never to the client envelope.
Types ¶
type Classification ¶
type Classification struct {
Mode Mode
// RelayKey is the raw inbound relay-key token. Empty in
// ModeProxyAnonymous. In ModeNormal pulled from X-WR-API-Key first,
// then Authorization (Bearer). In ModeProxyAuthed pulled from
// X-WR-API-Key only.
RelayKey string
// UpstreamAuth is the verbatim Authorization header value (including
// the "Bearer " prefix when present) supplied by the caller for proxy
// mode. The proxy forwarder re-attaches it on the outbound request.
// Empty in ModeNormal.
UpstreamAuth string
// UpstreamHost is the X-WR-Upstream-Host slug naming a configured
// Host row. Empty in ModeNormal. Required in proxy mode; the proxy
// dispatcher rejects missing/unknown values.
UpstreamHost string
// ClientIP is the resolved client address — XFF-walked when the peer
// matches RELAY_TRUSTED_PROXIES, RemoteAddr otherwise.
ClientIP string
}
Classification carries the result of Classify. Stashed on ctx by the classifier middleware and read by downstream handlers / pipeline / proxy. The classifier is pure — it does not touch the catalog snapshot or any settings; gating decisions happen later.
func ClassificationFrom ¶
func ClassificationFrom(ctx context.Context) Classification
ClassificationFrom returns the mode classification from ctx. The zero Classification (Mode == ModeUnknown) is returned if no classifier ran.
func Classify ¶
func Classify(r *http.Request) (Classification, error)
Classify extracts the mode, credentials, and client IP from r. Pure; no ctx writes, no snapshot reads. Returns an error only for shape- invalid inputs (bad header value, missing upstream key in proxy mode); the caller maps errors to HTTP responses.
type Deps ¶
type Deps struct {
// Pinger reports backend health for /healthz. Storage satisfies
// this; tests can pass a stub.
Pinger Pinger
// Catalog is the in-memory snapshot used for relay-key auth lookup
// and the /v1/models listing.
Catalog *appcatalog.Catalog
// Resolver translates inbound model+policy refs into a pipeline-
// ready Plan against the snapshot.
Resolver *routing.Resolver
// Pipeline orchestrates one normal-mode inference request.
Pipeline *pipeline.Pipeline
// Proxy orchestrates a proxy-mode (BYO upstream key) request.
Proxy *proxy.Pipeline
// Lifecycle is the observer registry. Dispatch uses it to fire a
// failure post-flight event for requests rejected before a runner is
// reached (routing / proxy gating / translate errors) — runner-stage
// failures are fired by the runner. Same registry the runners hold.
Lifecycle *lifecycle.Registry
// Adapters keys the wire-protocol implementation by adapters.Name.
// Handlers look up the binding's Adapter Name here at request time;
// proxy mode looks up the extractor by inbound endpoint shape.
Adapters map[adapters.Name]pipeline.Adapter
// Specs is the generic adapter registry. Dispatch uses it to look up
// inbound and upstream translators and to determine routing strategy
// (byte-pass vs canonical cross-shape). Built once at boot and
// populated from cmd/relay/main.go.
Specs *adapter.Registry
// RouteMounters are per-adapter route registration functions. Each
// adapter (or the generic framework) exposes a MountRoutes function
// that satisfies RouteMounter; cmd/relay/main.go wires them in.
RouteMounters []RouteMounter
// TrustEventTime makes Dispatch honor the X-WR-Event-Time header as
// the usage Event timestamp (RELAY_DEV_TRUST_EVENT_TIME). Dev/replay
// tooling only; off by default.
TrustEventTime bool
}
Deps is the typed dependency bundle for the data plane.
type DispatchInput ¶
type DispatchInput struct {
// Inbound is the wire shape the caller spoke (the route's Name).
Inbound adapters.Name
// Body is the raw inbound request body, already consumed from r.Body.
Body []byte
// ModelName is the caller-supplied model identifier extracted by the
// per-shape minimal parse. Routing resolution uses this.
ModelName string
// Stream is the caller-supplied stream flag from the minimal parse.
// Determines whether the response leg streams chunks or buffers + emits.
Stream bool
}
DispatchInput is what a per-shape route passes to Dispatch after its own minimal parse. The route knows the inbound shape Name (because it owns the URL); Dispatch handles everything from routing onward.
type Mode ¶
type Mode int
Mode classifies the proxy mode of an inbound inference request.
const ( ModeUnknown Mode = iota // ModeNormal — default; no X-WR-Proxy-Mode; relay key authenticates // against a Policy and the upstream key comes from the keypool. ModeNormal // ModeProxyAuthed — X-WR-Proxy-Mode: Proxy + relay key in X-WR-API-Key // + caller-supplied upstream key in Authorization. Logged against the // relay key; subject to the inference-api-proxy system rate limit. ModeProxyAuthed // ModeProxyAnonymous — X-WR-Proxy-Mode: Proxy + caller-supplied // upstream key in Authorization, no relay key. Subject to the // inference-api-proxy-anonymous per-IP rate limit. Gated by // Settings.ProxyMode.AllowUnauthenticated. ModeProxyAnonymous )
type Pinger ¶
Pinger reports backend health for /healthz. Storage satisfies this via its own Ping method; tests can supply a stub.
type RouteMounter ¶
type RouteMounter = func(api huma.API, d Deps, mw huma.Middlewares)
RouteMounter is what an adapter package exposes to mount its own inbound HTTP surface. Each adapter (openai, anthropic, ...) provides a MountRoutes(api, deps, mw) function matching this signature.
func MountRegistry ¶
func MountRegistry(reg *adapter.Registry) RouteMounter
MountRegistry returns a RouteMounter that registers one POST route per InboundPath across all specs in reg. Each route performs a minimal JSON parse (model + stream), then calls Dispatch with the spec's Name as the inbound shape.
This is the generic route mounter that replaces per-shape MountRoutes functions. The composition root calls it once with the populated registry.
Source Files