Documentation
¶
Overview ¶
Package gateway carries model calls over HTTP, so that an application can reach models through a gateway that holds the vendor keys and meters the usage: an agent in a sandbox, say, that must never see a key. Provider is the client, an ordinary litellm.Provider whose calls run on a Server; the Server makes each call with the Client it routes the call to and streams the call back.
A call POSTs its litellm.Request as JSON, and the reply streams the call's events as JSON lines up to done or error, with a heartbeat line whenever the upstream is silent for a while, so that proxies keep the connection open, even before it answers. A call refused before the first line gets an error body instead, with the HTTP status of its error and a Retry-After when it suggests a wait. Blocks keep their provider state, and errors keep their type, retry facts and upstream provider, so a call through a gateway behaves as one made to the vendor directly; only the Server's own vendor key being rejected is a provider error rather than an auth error, which would blame the caller's key. Both ends must run the same version of this package.
A Server neither authenticates nor meters. Serve it behind your authentication, which puts the caller on the request context, and meter with a litellm.Observer on the routed Clients, which see that context.
Index ¶
Constants ¶
const MaxRequestBytes = 64 << 20
MaxRequestBytes caps the body of a call; a larger one is refused with 413.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Config ¶
type Config struct {
// Name names the Provider in its own errors, "gateway" by default.
// Responses name the provider that made the call.
Name string
// BaseURL is the URL the Server is served at.
BaseURL string
// APIKey, if set, is sent as a bearer token; APIKeyFunc, when set,
// resolves it per call instead.
APIKey string
APIKeyFunc func(context.Context) (string, error)
// HTTPClient sends calls; nil uses http.DefaultClient.
HTTPClient litellm.HTTPClient
UserAgent string
// Headers are sent with every call, after the defaults.
Headers map[string]string
}
Config configures a Provider.
type Provider ¶
type Provider struct {
// contains filtered or unexported fields
}
Provider is a litellm.Provider whose calls run on a Server.
type Server ¶
type Server struct {
// Route returns the Client to make the call req for r with. It may
// rewrite req, such as to map the model the caller names to the vendor's,
// or to cap its tokens. An error refuses the call: a litellm.Error with
// its type, any other as a model not found.
Route func(r *http.Request, req *litellm.Request) (*litellm.Client, error)
}
Server serves model calls. It is an http.Handler.