gateway

package
v1.9.3 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 2, 2026 License: Apache-2.0 Imports: 13 Imported by: 0

Documentation

Overview

Package gateway carries model calls over HTTP, so that an application can reach models through a gateway that holds the vendor keys and meters the usage: an agent in a sandbox, say, that must never see a key. Provider is the client, an ordinary litellm.Provider whose calls run on a Server; the Server makes each call with the Client it routes the call to and streams the call back.

A call POSTs its litellm.Request as JSON, and the reply streams the call's events as JSON lines up to done or error, with a heartbeat line whenever the upstream is silent for a while, so that proxies keep the connection open. Blocks keep their provider state, and errors keep their type, retry facts and upstream provider, so a call through a gateway behaves as one made to the vendor directly; only the Server's own vendor key being rejected is a provider error rather than an auth error, which would blame the caller's key. Both ends must run the same version of this package.

A Server neither authenticates nor meters. Serve it behind your authentication, which puts the caller on the request context, and meter with a litellm.Observer on the routed Clients, which see that context.

Index

Constants

View Source
const MaxRequestBytes = 64 << 20

MaxRequestBytes caps the body of a call; a larger one is refused with 413.

Variables

This section is empty.

Functions

This section is empty.

Types

type Config

type Config struct {
	// Name names the Provider in its own errors, "gateway" by default.
	// Responses name the provider that made the call.
	Name string
	// BaseURL is the URL the Server is served at.
	BaseURL string
	// APIKey, if set, is sent as a bearer token; APIKeyFunc, when set,
	// resolves it per call instead.
	APIKey     string
	APIKeyFunc func(context.Context) (string, error)
	// HTTPClient sends calls; nil uses http.DefaultClient.
	HTTPClient litellm.HTTPClient
	UserAgent  string
	// Headers are sent with every call, after the defaults.
	Headers map[string]string
}

Config configures a Provider.

type Provider

type Provider struct {
	// contains filtered or unexported fields
}

Provider is a litellm.Provider whose calls run on a Server.

func New

func New(cfg Config) (*Provider, error)

New returns a Provider for the Server at cfg.BaseURL.

func (*Provider) Chat

func (p *Provider) Chat(ctx context.Context, req *litellm.Request) (*litellm.Response, error)

Chat makes the call and collects its reply.

func (*Provider) Name

func (p *Provider) Name() string

Name returns the configured name. Responses name the provider that made the call.

func (*Provider) Stream

func (p *Provider) Stream(ctx context.Context, req *litellm.Request) (litellm.Stream, error)

Stream makes the call and streams its reply.

type Server

type Server struct {
	// Route returns the Client to make the call req for r with. It may
	// rewrite req, such as to map the model the caller names to the vendor's,
	// or to cap its tokens. An error refuses the call, with the type and
	// status a litellm.Error carries or else as a model not found (404).
	Route func(r *http.Request, req *litellm.Request) (*litellm.Client, error)
}

Server serves model calls. It is an http.Handler.

func (*Server) ServeHTTP

func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request)

ServeHTTP makes the call r carries, once: retrying is the caller's.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL