ext

type Endpoint ¶

type Endpoint string

Endpoint identifies an inference endpoint whose raw JSON body can be rewritten before core parses it.

const (
	EndpointChatCompletions Endpoint = "/v1/chat/completions"
	EndpointMessages        Endpoint = "/v1/messages"
	EndpointResponses       Endpoint = "/v1/responses"
)

type Input ¶

type Input struct {
	Endpoint Endpoint
	// Body is the raw JSON request body, already bounded by the server's
	// body-size limit.
	Body []byte
	// Header is a clone of the inbound request headers with credential
	// values (Authorization, cookies, API keys, ...) redacted. Rewriters
	// run post-auth; use UserPath for identity.
	Header http.Header
	// UserPath is the canonical authenticated user path, when present.
	UserPath string
	// RequestID is the request correlation ID (X-Request-ID).
	RequestID string
	// SessionID is the detected client session, when present, already scoped
	// by the effective user path. Session detection runs before rewriters, so
	// a rewriter can keep its decisions stable across a conversation.
	SessionID string
}

Input is the raw inbound request handed to a rewriter before core parses it. Body and Header are snapshots owned by the middleware; rewriters must treat them as read-only and return new values in Result when changing anything.

type Registry ¶

type Registry struct {
	// contains filtered or unexported fields
}

Registry collects extensions to be consumed by the gateway at startup. Register everything before the server is constructed (before run.Run or app.New); core snapshots the registry once and never consults it again.

func (r *Registry) AddPublicPaths(paths ...string)

func (r *Registry) Middleware() []echo.MiddlewareFunc

func (r *Registry) PublicPaths() []string

func (r *Registry) RegisterRewriter(rw RequestRewriter)

func (r *Registry) RegisterRouteSelector(sel RouteSelector)

func (r *Registry) RegisterRoutes(fn func(e *echo.Echo))

func (r *Registry) Rewriters() []RequestRewriter

func (r *Registry) RouteSelector() RouteSelector

func (r *Registry) Routes() []func(*echo.Echo)

func (r *Registry) UseMiddleware(m echo.MiddlewareFunc)

type RejectionError ¶

type RejectionError struct {
	Status  int
	Code    string
	Message string
}

RejectionError rejects the request with a client-visible status code and machine-readable error code, rendered in the endpoint's native error dialect (OpenAI or Anthropic envelope).

func (e *RejectionError) Error() string

type RequestRewriter ¶

type RequestRewriter interface {
	Name() string
	Rewrite(ctx context.Context, in Input) (*Result, error)
}

RequestRewriter rewrites raw JSON request bodies at ingress, after authentication and before model resolution, so body changes (including the "model" field) affect routing, failover, guardrails, budgets, and caching.

Rewriters run once per request in registration order; each receives the previous rewriter's output. Implementations must be safe for concurrent use. Errors fail the request (fail-closed): return a *RejectionError for a client-visible status, any other error maps to HTTP 500.

type Result ¶

type Result struct {
	Body []byte
	// ResponseHeader entries are merged into the HTTP response so rewriters
	// can annotate what they did (for example X-GoModel-Pro-Tokens-Saved).
	ResponseHeader http.Header
	// Detail optionally carries a JSON-serializable summary of what the
	// rewriter changed. It is recorded in the audit trail's request-revision
	// chain and never sent upstream; it must never contain secrets or
	// request credentials.
	Detail any
	// TokensSaved is the rewriter's estimate of prompt tokens its body
	// change removed from the request. When positive and the rewritten body
	// is applied, core adds it to the request's usage record together with
	// the input cost those tokens would have incurred, and the dashboard
	// aggregates both as rewrite savings. Leave zero when the rewrite does
	// not shrink the prompt.
	TokensSaved int
}

Result carries a rewritten body and response-header annotations. A nil Result (or nil Body) means the request is unchanged.

type RouteCandidate ¶ added in v0.1.65

type RouteCandidate struct {
	// Provider is the configured provider name (e.g. "openai", "azure-eu").
	Provider string
	// Model is the provider-native model ID (e.g. "gpt-4o").
	Model string
	// Qualified is "provider/model", the stable key selection answers with.
	Qualified string
	// Weight is the operator-configured target weight; 0 means unset (treat
	// as 1).
	Weight        float64
	InputPerMtok  *float64
	OutputPerMtok *float64
}

RouteCandidate is one currently viable target of a load-balanced virtual model, offered to a RouteSelector. Pricing comes from the model registry and is per million tokens; nil means the registry has no price for the target.

type RouteOutcome ¶ added in v0.1.65

type RouteOutcome struct {
	RouteTarget
	// Endpoint is the upstream API endpoint (e.g. "/chat/completions").
	Endpoint string
	// StatusCode is the final upstream HTTP status; 0 on a network error.
	StatusCode int
	// Duration is the call duration, including any transport-level retries.
	// For streaming requests it measures time to stream establishment, not
	// the full stream lifetime.
	Duration time.Duration
	Stream   bool
	// Err is the client-layer error, nil on success.
	Err error
}

RouteOutcome describes one completed upstream call. Every call is reported — primaries and failover attempts alike — so selectors learn from traffic they did not steer. Transport-level retries inside the provider client are aggregated into their call's single outcome: StatusCode and Err reflect the final result, and Duration spans the whole call including retry backoff, so a target that only succeeds after internal retries still scores slower than a target that succeeds at once.

type RouteRequest ¶ added in v0.1.65

type RouteRequest struct {
	// Source is the virtual model name the request addressed.
	Source string
	// SessionID is the detected client session, when present. Session
	// affinity is enforced by core before the selector runs; the ID is
	// provided for observability only.
	SessionID  string
	Candidates []RouteCandidate
}

RouteRequest asks a RouteSelector to pick one target for a request routed through a load-balanced virtual model. Candidates are the targets that are catalog-supported and have rate-limit capacity right now, in declared order; there are always at least two (single-candidate picks bypass the selector so an alias behaves identically with and without one).

type RouteSelector ¶ added in v0.1.65

type RouteSelector interface {
	Name() string
	Select(req RouteRequest) (qualified string, ok bool)
	OnAttemptStart(target RouteTarget)
	OnAttemptEnd(outcome RouteOutcome)
}

RouteSelector steers load balancing for virtual models using the "adaptive" strategy. Core consults the selector only to pick among currently viable targets; session affinity, rate-limit capacity, failover chains, and retries all remain core's responsibility.

Select must be fast and must not block: it runs on the request path before the upstream call. Implementations must be safe for concurrent use. A (_, false) answer — and any answer naming a model outside Candidates — falls back to weighted round robin, so selectors fail open by declining.

OnAttemptStart and OnAttemptEnd observe the upstream client lifecycle, once per upstream call (transport-level retries within a call are aggregated — see RouteOutcome). For streaming requests OnAttemptEnd fires when the stream is established, not when it closes.

type RouteTarget ¶ added in v0.1.65

type RouteTarget struct {
	Provider string
	Model    string
}

RouteTarget identifies a provider/model pair as seen by the upstream client layer.

func (t RouteTarget) Qualified() string