Documentation
¶
Overview ¶
Package server implements the LLM proxy HTTP server
Index ¶
- type CostTracker
- type RateLimiter
- type RequestLogger
- func (l *RequestLogger) Close() error
- func (l *RequestLogger) LogError(requestID string, req *llmproxy.ChatRequest, err error)
- func (l *RequestLogger) LogRequest(requestID string, req *llmproxy.ChatRequest, r *http.Request)
- func (l *RequestLogger) LogResponse(requestID string, req *llmproxy.ChatRequest, resp *llmproxy.ChatResponse, ...)
- func (l *RequestLogger) LogStreamingComplete(requestID string, req *llmproxy.ChatRequest, totalTokens int, ...)
- type Server
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type CostTracker ¶
type CostTracker struct {
// contains filtered or unexported fields
}
CostTracker tracks API usage costs and enforces budget limits
func NewCostTracker ¶
func NewCostTracker(config llmproxy.CostBudgetConfig) *CostTracker
NewCostTracker creates a new cost tracker
func (*CostTracker) CheckBudget ¶
func (c *CostTracker) CheckBudget(ctx context.Context) bool
CheckBudget checks if the budget allows for more requests
func (*CostTracker) GetStats ¶
func (c *CostTracker) GetStats() map[string]interface{}
GetStats returns current cost statistics
func (*CostTracker) RecordCost ¶
func (c *CostTracker) RecordCost(cost *llmproxy.Cost)
RecordCost records a cost and updates running totals
type RateLimiter ¶
type RateLimiter struct {
// contains filtered or unexported fields
}
RateLimiter implements token bucket rate limiting
func NewRateLimiter ¶
func NewRateLimiter(config llmproxy.RateLimitConfig) *RateLimiter
NewRateLimiter creates a new rate limiter
func (*RateLimiter) AllowTokens ¶
func (r *RateLimiter) AllowTokens(key string, tokens int) bool
AllowTokens checks if token consumption is allowed
func (*RateLimiter) Middleware ¶
func (r *RateLimiter) Middleware(next http.Handler) http.Handler
Middleware returns an HTTP middleware for rate limiting
type RequestLogger ¶
type RequestLogger struct {
// contains filtered or unexported fields
}
RequestLogger logs requests and responses
func NewRequestLogger ¶
func NewRequestLogger(logRequests, logResponses, redactPII bool) *RequestLogger
NewRequestLogger creates a new request logger
func (*RequestLogger) Close ¶
func (l *RequestLogger) Close() error
Close closes the log file if open
func (*RequestLogger) LogError ¶
func (l *RequestLogger) LogError(requestID string, req *llmproxy.ChatRequest, err error)
LogError logs an error
func (*RequestLogger) LogRequest ¶
func (l *RequestLogger) LogRequest(requestID string, req *llmproxy.ChatRequest, r *http.Request)
LogRequest logs an incoming request
func (*RequestLogger) LogResponse ¶
func (l *RequestLogger) LogResponse(requestID string, req *llmproxy.ChatRequest, resp *llmproxy.ChatResponse, duration time.Duration)
LogResponse logs a response
func (*RequestLogger) LogStreamingComplete ¶
func (l *RequestLogger) LogStreamingComplete(requestID string, req *llmproxy.ChatRequest, totalTokens int, duration time.Duration)
LogStreamingComplete logs a completed streaming request
type Server ¶
type Server struct {
// contains filtered or unexported fields
}
Server is the LLM proxy HTTP server
func (*Server) GetProvider ¶
GetProvider returns a provider by type, or the default if not specified
func (*Server) GetProviderByModel ¶
func (s *Server) GetProviderByModel(model string) (llmproxy.ProviderType, error)
GetProviderByModel returns the appropriate provider for a given model
Source Files