Documentation
¶
Overview ¶
Package proxy provides the transparent proxy functionality for the LLM API.
Index ¶
- func CacheKeyFromRequest(r *http.Request) string
- func CacheKeyFromRequestWithVary(r *http.Request, vary string) string
- func Chain(h http.Handler, middleware ...Middleware) http.Handler
- type APIConfig
- type APIProviderConfig
- type AuditLogger
- type CacheMetricType
- type ConnectionConfig
- type ErrorResponse
- type Middleware
- type Project
- type ProjectActiveChecker
- type ProjectStore
- type Proxy
- type ProxyConfig
- type ProxyMetrics
- type TimeoutConfig
- type TokenValidator
- type TransparentProxy
- func NewTransparentProxy(config ProxyConfig, validator TokenValidator, store ProjectStore) (*TransparentProxy, error)
- func NewTransparentProxyWithAudit(config ProxyConfig, validator TokenValidator, store ProjectStore, ...) (*TransparentProxy, error)
- func NewTransparentProxyWithLogger(config ProxyConfig, validator TokenValidator, store ProjectStore, ...) (*TransparentProxy, error)
- func NewTransparentProxyWithLoggerAndObservability(config ProxyConfig, validator TokenValidator, store ProjectStore, ...) (*TransparentProxy, error)
- func NewTransparentProxyWithObservability(config ProxyConfig, validator TokenValidator, store ProjectStore, ...) (*TransparentProxy, error)
- func (p *TransparentProxy) Cache() httpCache
- func (p *TransparentProxy) Handler() http.Handler
- func (p *TransparentProxy) LoggingMiddleware() Middleware
- func (p *TransparentProxy) Metrics() *ProxyMetrics
- func (p *TransparentProxy) MetricsMiddleware() Middleware
- func (p *TransparentProxy) SetMetrics(m *ProxyMetrics)
- func (p *TransparentProxy) Shutdown(ctx context.Context) error
- func (p *TransparentProxy) TimeoutMiddleware(timeout time.Duration) Middleware
- func (p *TransparentProxy) ValidateRequestMiddleware() Middleware
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func CacheKeyFromRequest ¶
func CacheKeyFromRequestWithVary ¶
CacheKeyFromRequestWithVary generates a cache key using the specified Vary header from the upstream response. This enables per-response driven cache key generation.
Behavior:
- When vary is empty or "*", this function returns a key that ignores header values (treated as "no vary" for key generation purposes).
- Header names parsed from vary are normalized to canonical MIME header names.
Types ¶
type APIConfig ¶
type APIConfig struct { // APIs contains configurations for different API providers APIs map[string]*APIProviderConfig `yaml:"apis"` // DefaultAPI is the default API provider to use if not specified DefaultAPI string `yaml:"default_api"` }
APIConfig represents the top-level configuration for API proxying
func LoadAPIConfigFromFile ¶
LoadAPIConfigFromFile loads API configuration from a YAML file
func (*APIConfig) GetProxyConfigForAPI ¶
func (c *APIConfig) GetProxyConfigForAPI(apiName string) (*ProxyConfig, error)
GetProxyConfigForAPI returns a ProxyConfig for the specified API provider
type APIProviderConfig ¶
type APIProviderConfig struct { // BaseURL is the target API base URL BaseURL string `yaml:"base_url"` // AllowedEndpoints is a list of endpoint prefixes that are allowed to be accessed AllowedEndpoints []string `yaml:"allowed_endpoints"` // AllowedMethods is a list of HTTP methods that are allowed AllowedMethods []string `yaml:"allowed_methods"` // Timeouts for various operations Timeouts TimeoutConfig `yaml:"timeouts"` // Connection settings Connection ConnectionConfig `yaml:"connection"` // ParamWhitelist is a map of parameter names to allowed values ParamWhitelist map[string][]string `yaml:"param_whitelist"` AllowedOrigins []string `yaml:"allowed_origins"` RequiredHeaders []string `yaml:"required_headers"` }
APIProviderConfig represents the configuration for a specific API provider
type AuditLogger ¶
AuditLogger defines the interface for audit event logging
type CacheMetricType ¶
type CacheMetricType int
CacheMetricType represents the kind of cache metric to increment.
const ( CacheMetricHit CacheMetricType = iota CacheMetricMiss CacheMetricBypass CacheMetricStore )
type ConnectionConfig ¶
type ConnectionConfig struct { // MaxIdleConns is the maximum number of idle connections MaxIdleConns int `yaml:"max_idle_conns"` // MaxIdleConnsPerHost is the maximum number of idle connections per host MaxIdleConnsPerHost int `yaml:"max_idle_conns_per_host"` }
ConnectionConfig contains connection settings for the proxy
type ErrorResponse ¶
type ErrorResponse struct { Error string `json:"error"` Description string `json:"description,omitempty"` Code string `json:"code,omitempty"` }
ErrorResponse is the standard format for error responses
type Middleware ¶
Middleware defines a function that wraps an http.Handler
func CircuitBreakerMiddleware ¶
func CircuitBreakerMiddleware(failureThreshold int, cooldown time.Duration, isTransient func(status int) bool) Middleware
CircuitBreakerMiddleware returns a middleware that opens the circuit after N consecutive failures. While open, it returns 503 immediately. After a cooldown, it closes and allows requests again.
func ProjectActiveGuardMiddleware ¶
func ProjectActiveGuardMiddleware(enforceActive bool, checker ProjectActiveChecker, auditLogger AuditLogger) Middleware
ProjectActiveGuardMiddleware creates middleware that enforces project active status If enforceActive is false, the middleware passes through all requests without checking If enforceActive is true, inactive projects receive a 403 Forbidden response
type Project ¶
type Project struct { ID string `json:"id"` Name string `json:"name"` OpenAIAPIKey string `json:"openai_api_key"` IsActive bool `json:"is_active"` DeactivatedAt *time.Time `json:"deactivated_at,omitempty"` CreatedAt time.Time `json:"created_at"` UpdatedAt time.Time `json:"updated_at"` }
Project represents a project for the management API and proxy (copied from database/models.go)
type ProjectActiveChecker ¶
type ProjectActiveChecker interface { // GetProjectActive checks if a project is active GetProjectActive(ctx context.Context, projectID string) (bool, error) }
ProjectActiveChecker defines the interface for checking project active status
type ProjectStore ¶
type ProjectStore interface { // GetAPIKeyForProject retrieves the API key for a project GetAPIKeyForProject(ctx context.Context, projectID string) (string, error) // GetProjectActive checks if a project is active GetProjectActive(ctx context.Context, projectID string) (bool, error) // Management API CRUD ListProjects(ctx context.Context) ([]Project, error) CreateProject(ctx context.Context, project Project) error GetProjectByID(ctx context.Context, projectID string) (Project, error) UpdateProject(ctx context.Context, project Project) error DeleteProject(ctx context.Context, projectID string) error }
ProjectStore defines the interface for retrieving and managing project information (extended for management API)
type Proxy ¶
type Proxy interface { // Handler returns an http.Handler for the proxy Handler() http.Handler // Shutdown gracefully shuts down the proxy Shutdown(ctx context.Context) error }
Proxy defines the interface for a transparent HTTP proxy
type ProxyConfig ¶
type ProxyConfig struct { // TargetBaseURL is the base URL of the API to proxy to TargetBaseURL string // AllowedEndpoints is a whitelist of endpoints that can be accessed AllowedEndpoints []string // AllowedMethods is a whitelist of HTTP methods that can be used AllowedMethods []string // RequestTimeout is the maximum duration for a complete request RequestTimeout time.Duration // ResponseHeaderTimeout is the time to wait for response headers ResponseHeaderTimeout time.Duration // FlushInterval is how often to flush streaming responses FlushInterval time.Duration // MaxIdleConns is the maximum number of idle connections MaxIdleConns int // MaxIdleConnsPerHost is the maximum number of idle connections per host MaxIdleConnsPerHost int // IdleConnTimeout is how long to keep idle connections alive IdleConnTimeout time.Duration // LogLevel controls the verbosity of logging LogLevel string // LogFormat controls the log output format (json or console) LogFormat string // LogFile specifies a file path for logs (stdout if empty) LogFile string // SetXForwardedFor determines whether to set the X-Forwarded-For header SetXForwardedFor bool // ParamWhitelist is a map of parameter names to allowed values ParamWhitelist map[string][]string // AllowedOrigins is a list of allowed CORS origins for this provider AllowedOrigins []string // RequiredHeaders is a list of required request headers (case-insensitive) RequiredHeaders []string // Project active guard configuration EnforceProjectActive bool // Whether to enforce project active status // --- HTTP cache (global, opt-in; set programmatically, not via YAML) --- // HTTPCacheEnabled toggles the proxy cache for GET/HEAD based on HTTP semantics HTTPCacheEnabled bool // HTTPCacheDefaultTTL is used only when upstream allows caching but omits explicit TTL HTTPCacheDefaultTTL time.Duration // HTTPCacheMaxObjectBytes is a guardrail for maximum cacheable response size HTTPCacheMaxObjectBytes int64 // RedisCacheURL enables Redis-backed cache when non-empty (e.g., redis://localhost:6379/0) RedisCacheURL string // RedisCacheKeyPrefix allows namespacing cache keys (default: llmproxy:cache:) RedisCacheKeyPrefix string }
ProxyConfig contains configuration for the proxy
func (*ProxyConfig) Validate ¶
func (c *ProxyConfig) Validate() error
Validate checks that the ProxyConfig is valid and returns an error if not.
type ProxyMetrics ¶
type ProxyMetrics struct { RequestCount int64 ErrorCount int64 TotalResponseTime time.Duration // Cache metrics (provider-agnostic counters) CacheHits int64 // Cache hits (responses served from cache) CacheMisses int64 // Cache misses (responses fetched from upstream) CacheBypass int64 // Cache bypassed (e.g., due to authorization) CacheStores int64 // Cache stores (responses stored in cache) // contains filtered or unexported fields }
ProxyMetrics tracks proxy usage statistics
type TimeoutConfig ¶
type TimeoutConfig struct { // Request is the overall request timeout Request time.Duration `yaml:"request"` // ResponseHeader is the timeout for receiving response headers ResponseHeader time.Duration `yaml:"response_header"` // IdleConnection is the timeout for idle connections IdleConnection time.Duration `yaml:"idle_connection"` // FlushInterval controls how often to flush streaming responses FlushInterval time.Duration `yaml:"flush_interval"` }
TimeoutConfig contains timeout settings for the proxy
type TokenValidator ¶
type TokenValidator interface { // ValidateToken validates a token and returns the associated project ID ValidateToken(ctx context.Context, token string) (string, error) // ValidateTokenWithTracking validates a token, increments its usage, and returns the project ID ValidateTokenWithTracking(ctx context.Context, token string) (string, error) }
TokenValidator defines the interface for token validation
type TransparentProxy ¶
type TransparentProxy struct {
// contains filtered or unexported fields
}
TransparentProxy implements the Proxy interface for transparent proxying
func NewTransparentProxy ¶
func NewTransparentProxy(config ProxyConfig, validator TokenValidator, store ProjectStore) (*TransparentProxy, error)
NewTransparentProxy creates a new proxy instance with an internally configured logger based on the provided ProxyConfig.
func NewTransparentProxyWithAudit ¶
func NewTransparentProxyWithAudit(config ProxyConfig, validator TokenValidator, store ProjectStore, logger *zap.Logger, auditLogger AuditLogger, obsCfg middleware.ObservabilityConfig) (*TransparentProxy, error)
NewTransparentProxyWithAudit creates a proxy with audit logging capabilities.
func NewTransparentProxyWithLogger ¶
func NewTransparentProxyWithLogger(config ProxyConfig, validator TokenValidator, store ProjectStore, logger *zap.Logger) (*TransparentProxy, error)
NewTransparentProxyWithLogger allows providing a custom logger. If logger is nil a new one is created based on the ProxyConfig.
func NewTransparentProxyWithLoggerAndObservability ¶
func NewTransparentProxyWithLoggerAndObservability(config ProxyConfig, validator TokenValidator, store ProjectStore, logger *zap.Logger, obsCfg middleware.ObservabilityConfig) (*TransparentProxy, error)
NewTransparentProxyWithLoggerAndObservability creates a proxy with observability middleware using an existing logger.
func NewTransparentProxyWithObservability ¶
func NewTransparentProxyWithObservability(config ProxyConfig, validator TokenValidator, store ProjectStore, obsCfg middleware.ObservabilityConfig) (*TransparentProxy, error)
NewTransparentProxyWithObservability creates a new proxy with observability middleware.
func (*TransparentProxy) Cache ¶
func (p *TransparentProxy) Cache() httpCache
Cache returns the HTTP cache instance for management operations. Returns nil if caching is disabled.
func (*TransparentProxy) Handler ¶
func (p *TransparentProxy) Handler() http.Handler
Handler returns the HTTP handler for the proxy
func (*TransparentProxy) LoggingMiddleware ¶
func (p *TransparentProxy) LoggingMiddleware() Middleware
LoggingMiddleware logs request details
func (*TransparentProxy) Metrics ¶
func (p *TransparentProxy) Metrics() *ProxyMetrics
Metrics returns a pointer to the current proxy metrics.
func (*TransparentProxy) MetricsMiddleware ¶
func (p *TransparentProxy) MetricsMiddleware() Middleware
MetricsMiddleware collects metrics about requests
func (*TransparentProxy) SetMetrics ¶
func (p *TransparentProxy) SetMetrics(m *ProxyMetrics)
SetMetrics overwrites the current metrics (primarily for testing).
func (*TransparentProxy) Shutdown ¶
func (p *TransparentProxy) Shutdown(ctx context.Context) error
Shutdown gracefully shuts down the proxy
func (*TransparentProxy) TimeoutMiddleware ¶
func (p *TransparentProxy) TimeoutMiddleware(timeout time.Duration) Middleware
TimeoutMiddleware adds a timeout to requests
func (*TransparentProxy) ValidateRequestMiddleware ¶
func (p *TransparentProxy) ValidateRequestMiddleware() Middleware
ValidateRequestMiddleware validates the incoming request against allowed endpoints and methods