Documentation
¶
Overview ¶
Package llmclient provides a base HTTP client for LLM providers with: - Request marshaling/unmarshaling - Retries with exponential backoff and jitter - Standardized error parsing (429, 502, 503, 504) - Circuit breaking with half-open state protection
Index ¶
- Constants
- type Client
- func (c *Client) BaseURL() string
- func (c *Client) Do(ctx context.Context, req Request, result any) error
- func (c *Client) DoPassthrough(ctx context.Context, req Request) (*http.Response, error)
- func (c *Client) DoRaw(ctx context.Context, req Request) (*Response, error)
- func (c *Client) DoStream(ctx context.Context, req Request) (io.ReadCloser, error)
- func (c *Client) SetBaseURL(url string)
- type Config
- type HeaderSetter
- type Hooks
- type Request
- type RequestInfo
- type Response
- type ResponseInfo
Constants ¶
const UnknownModel = "unknown"
extractModel attempts to extract the model name from a request body UnknownModel is the model label reported for requests whose model cannot be recovered from the request body (body-less discovery GETs, availability probes, multipart uploads). Hook consumers that attribute traffic per model should treat it as "not model-attributed".
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Client ¶
type Client struct {
// contains filtered or unexported fields
}
Client is a base HTTP client for LLM providers
func New ¶
func New(cfg Config, headerSetter HeaderSetter) *Client
New creates a new LLM client with the given configuration
func NewWithHTTPClient ¶
func NewWithHTTPClient(httpClient *http.Client, cfg Config, headerSetter HeaderSetter) *Client
NewWithHTTPClient creates a new LLM client with a custom HTTP client
func (*Client) Do ¶
Do executes a request with retries and circuit breaking, then unmarshals the response
func (*Client) DoPassthrough ¶
DoPassthrough executes a request and returns the raw upstream HTTP response. Unlike DoRaw, it preserves non-200 responses for the caller to proxy unchanged.
func (*Client) DoRaw ¶
DoRaw executes a request with retries and circuit breaking, returning the raw response.
Metrics Behavior ¶
Metrics hooks (OnRequestStart/OnRequestEnd) are called at this level to track logical requests from the caller's perspective, not individual retry attempts. This ensures:
- Request counts reflect user-facing requests, not internal HTTP calls
- Duration metrics include total time across all retries (useful for SLOs)
- In-flight gauge accurately reflects concurrent logical requests
Behavior comparison (hooks at DoRaw vs per-attempt):
| Scenario | Per-attempt (old) | DoRaw level (current) | |--------------------------------------|-----------------------------|----------------------------------| | 1 request, succeeds first try | 1 observation | 1 observation | | 1 request, fails twice then succeeds | 3 observations | 1 observation (success) | | 1 request, fails all 3 retries | 3 observations | 1 observation (error) | | Duration metric | Each attempt's duration | Total duration including retries | | In-flight gauge | Bounces up/down per attempt | Accurate concurrent count |
The final status code and error in metrics reflect the outcome after all retry attempts.
func (*Client) DoStream ¶
DoStream executes a streaming request, returning a ReadCloser Note: Streaming requests do NOT retry (as partial data may have been sent) Metrics note: Duration is measured from start to stream establishment, not stream close
func (*Client) SetBaseURL ¶
SetBaseURL updates the base URL (thread-safe)
type Config ¶
type Config struct {
// ProviderName is the identifier used in logs and metrics (e.g., "openai", "anthropic").
ProviderName string
// BaseURL is the base URL for the provider's API (e.g., "https://api.openai.com/v1").
BaseURL string
// Retry specifies retry behaviour for failed requests, including backoff and jitter settings.
Retry config.RetryConfig
// CircuitBreaker configures the circuit breaker that prevents cascading failures by
// stopping requests to an unhealthy provider until it recovers.
CircuitBreaker config.CircuitBreakerConfig
// Hooks provides optional observability callbacks invoked on request start and end.
Hooks Hooks
}
Config holds configuration for the LLM client
func DefaultConfig ¶
DefaultConfig returns default client configuration
type HeaderSetter ¶
HeaderSetter is a function that sets headers on an HTTP request
type Hooks ¶
type Hooks struct {
// OnRequestStart is called before a request is sent.
// The returned context can be used to propagate trace spans or request IDs.
OnRequestStart func(ctx context.Context, info RequestInfo) context.Context
// OnRequestEnd is called after a request completes (success or failure).
// For streaming requests, this is called when the stream starts, not when it closes.
OnRequestEnd func(ctx context.Context, info ResponseInfo)
}
Hooks defines observability callbacks for request lifecycle events. These hooks enable instrumentation without polluting business logic.
type Request ¶
type Request struct {
Method string
Endpoint string
Body any // Will be JSON marshaled if not nil
RawBody []byte // Used as-is (e.g., multipart form bodies). Mutually exclusive with Body and RawBodyReader.
// RawBodyReader streams the request body without buffering it in memory.
// It is intended for one-shot passthrough requests and is not replayable for retries.
RawBodyReader io.Reader
Headers http.Header
}
Request represents an HTTP request to be made
type RequestInfo ¶
type RequestInfo struct {
Provider string // Provider name (e.g., "openai", "anthropic")
Model string // Model name (e.g., "gpt-4", "claude-3-opus")
Endpoint string // API endpoint (e.g., "/chat/completions", "/models")
Method string // HTTP method (e.g., "POST", "GET")
Stream bool // Whether this is a streaming request
}
RequestInfo contains metadata about a request for observability hooks
type Response ¶
type Response struct {
StatusCode int
// ContentType is the upstream response Content-Type header, preserved so
// callers can describe the bytes actually returned (e.g. audio formats).
ContentType string
// Header carries the upstream response headers. It is used to audit failed
// provider attempts; it is not relayed to API clients.
Header http.Header
Body []byte
}
Response represents an HTTP response
type ResponseInfo ¶
type ResponseInfo struct {
Provider string // Provider name
Model string // Model name
Endpoint string // API endpoint
StatusCode int // HTTP status code (0 if network error)
Duration time.Duration // Request duration
Stream bool // Whether this was a streaming request
Error error // Error if request failed (nil on success)
// CircuitState is the provider's circuit breaker state after this request
// completed ("closed", "half-open", "open"); empty when the breaker is
// disabled. It reflects the moment of completion, so metrics built from it
// update as traffic flows.
CircuitState string
}
ResponseInfo contains metadata about a response for observability hooks
Source Files