Documentation
¶
Overview ¶
Package llmx provides reusable decorators around github.com/bornholm/genai llm.Client. The RetryClient adds bounded, context-aware retries with exponential backoff and optional client-side rate limiting, so a transient LLM failure (network blip, provider 429/5xx) no longer fails the whole operation that relies on it (HyDE, Judge, grounding evaluation, embeddings).
Index ¶
- Constants
- func DefaultRetryable(err error) bool
- type CachingClient
- func (c *CachingClient) ChatCompletion(ctx context.Context, funcs ...llm.ChatCompletionOptionFunc) (llm.ChatCompletionResponse, error)
- func (c *CachingClient) ChatCompletionStream(ctx context.Context, funcs ...llm.ChatCompletionOptionFunc) (<-chan llm.StreamChunk, error)
- func (c *CachingClient) ChatStats() (hits, misses int64)
- func (c *CachingClient) Embeddings(ctx context.Context, inputs []string, funcs ...llm.EmbeddingsOptionFunc) (llm.EmbeddingsResponse, error)
- func (c *CachingClient) Stats() (hits, misses int64)
- type CachingOption
- type ObservableClient
- func (c *ObservableClient) ChatCompletion(ctx context.Context, funcs ...llm.ChatCompletionOptionFunc) (llm.ChatCompletionResponse, error)
- func (c *ObservableClient) ChatCompletionStream(ctx context.Context, funcs ...llm.ChatCompletionOptionFunc) (<-chan llm.StreamChunk, error)
- func (c *ObservableClient) Embeddings(ctx context.Context, inputs []string, funcs ...llm.EmbeddingsOptionFunc) (llm.EmbeddingsResponse, error)
- type OptionFunc
- type Options
- type RetryClient
- func (c *RetryClient) ChatCompletion(ctx context.Context, funcs ...llm.ChatCompletionOptionFunc) (llm.ChatCompletionResponse, error)
- func (c *RetryClient) ChatCompletionStream(ctx context.Context, funcs ...llm.ChatCompletionOptionFunc) (<-chan llm.StreamChunk, error)
- func (c *RetryClient) Embeddings(ctx context.Context, inputs []string, funcs ...llm.EmbeddingsOptionFunc) (llm.EmbeddingsResponse, error)
Constants ¶
const ( DefaultMaxRetries = 3 DefaultBaseBackoff = 500 * time.Millisecond DefaultMaxBackoff = 30 * time.Second )
Default retry parameters.
Variables ¶
This section is empty.
Functions ¶
func DefaultRetryable ¶
DefaultRetryable retries every error except context cancellation and deadline expiry, which signal the caller gave up and must not be retried.
Types ¶
type CachingClient ¶
type CachingClient struct {
// contains filtered or unexported fields
}
CachingClient decorates an llm.Client with a persistent on-disk cache for Embeddings calls and, optionally (WithChatCache), for deterministic ChatCompletion calls. Embedding a text is deterministic for a given model, so the vector can be reused across runs — turning repeated corpus ingestions (evaluation re-runs, benchmarks) into free cache hits instead of thousands of billable, rate-limited calls. A chat completion is only cacheable when the caller pins a seed (e.g. the HyDE transformer, seeded per query for reproducibility): unseeded, tool-using or multimodal calls always pass through.
Each embeddings input is cached individually under sha256(namespace, dimensions, text) below dir/embeddings, so hits survive re-batching. On a batch call only the misses are forwarded to the wrapped client (in one batch), and the reported usage covers those misses only. Chat completions are cached below dir/chat under a key covering the messages, seed, temperature and response schema. Files are written atomically (temp file + rename) making the cache safe for concurrent use; an unreadable or corrupted entry is treated as a miss and rewritten.
The namespace must identify the embedding space — typically the model name. Reusing a directory with a different model but the same namespace would serve vectors from the wrong space. The chat namespace (WithChatCache) must likewise identify the chat model.
func NewCachingClient ¶
func NewCachingClient(client llm.Client, dir, namespace string, funcs ...CachingOption) (*CachingClient, error)
NewCachingClient wraps client with an embeddings cache rooted at dir (created if missing), keyed under namespace (typically the embedding model name).
func (*CachingClient) ChatCompletion ¶
func (c *CachingClient) ChatCompletion(ctx context.Context, funcs ...llm.ChatCompletionOptionFunc) (llm.ChatCompletionResponse, error)
ChatCompletion implements llm.ChatCompletionClient. Deterministic calls — seeded, tool-free, text-only — are served from the chat cache when enabled; everything else is delegated untouched.
func (*CachingClient) ChatCompletionStream ¶
func (c *CachingClient) ChatCompletionStream(ctx context.Context, funcs ...llm.ChatCompletionOptionFunc) (<-chan llm.StreamChunk, error)
ChatCompletionStream implements llm.ChatCompletionStreamingClient by delegation.
func (*CachingClient) ChatStats ¶ added in v0.2.0
func (c *CachingClient) ChatStats() (hits, misses int64)
ChatStats returns the number of chat-completion cache hits and misses served so far (both stay 0 unless WithChatCache is enabled).
func (*CachingClient) Embeddings ¶
func (c *CachingClient) Embeddings(ctx context.Context, inputs []string, funcs ...llm.EmbeddingsOptionFunc) (llm.EmbeddingsResponse, error)
Embeddings implements llm.EmbeddingsClient: served from the cache when every input is known, otherwise the misses are fetched from the wrapped client in a single batch and stored before assembling the response in input order.
func (*CachingClient) Stats ¶
func (c *CachingClient) Stats() (hits, misses int64)
Stats returns the number of embeddings cache hits and misses served so far.
type CachingOption ¶ added in v0.2.0
type CachingOption func(*CachingClient)
CachingOption configures a CachingClient.
func WithChatCache ¶ added in v0.2.0
func WithChatCache(namespace string) CachingOption
WithChatCache enables caching of deterministic (seeded) chat completions below dir/chat, keyed under namespace — which must identify the chat model, exactly like the embeddings namespace identifies the embedding space.
type ObservableClient ¶
type ObservableClient struct {
// contains filtered or unexported fields
}
ObservableClient decorates an llm.Client with OpenTelemetry spans and metrics (call count, latency, token usage). It is a thin wrapper: when no OTel provider is installed the instrumentation is a no-op. Compose it with RetryClient as needed (e.g. observe the retried client to count every attempt's latency, or wrap the outside to measure the logical call).
func NewObservableClient ¶
func NewObservableClient(client llm.Client) *ObservableClient
NewObservableClient wraps client so its calls emit spans and metrics under the amoxtli instrumentation scope.
func (*ObservableClient) ChatCompletion ¶
func (c *ObservableClient) ChatCompletion(ctx context.Context, funcs ...llm.ChatCompletionOptionFunc) (llm.ChatCompletionResponse, error)
ChatCompletion implements llm.ChatCompletionClient with instrumentation.
func (*ObservableClient) ChatCompletionStream ¶
func (c *ObservableClient) ChatCompletionStream(ctx context.Context, funcs ...llm.ChatCompletionOptionFunc) (<-chan llm.StreamChunk, error)
ChatCompletionStream implements llm.ChatCompletionStreamingClient. Only the call opening the stream is instrumented (span + latency); token usage is not available until the stream completes and is therefore not recorded here.
func (*ObservableClient) Embeddings ¶
func (c *ObservableClient) Embeddings(ctx context.Context, inputs []string, funcs ...llm.EmbeddingsOptionFunc) (llm.EmbeddingsResponse, error)
Embeddings implements llm.EmbeddingsClient with instrumentation.
type OptionFunc ¶
type OptionFunc func(*Options)
func WithBackoff ¶
func WithBackoff(base, max time.Duration) OptionFunc
WithBackoff sets the base and maximum backoff delays.
func WithLimiter ¶
func WithLimiter(limiter *rate.Limiter) OptionFunc
WithLimiter installs a pre-built rate limiter (e.g. shared across clients).
func WithMaxRetries ¶
func WithMaxRetries(n int) OptionFunc
WithMaxRetries sets the number of retries attempted after the first failure.
func WithRateLimit ¶
func WithRateLimit(r rate.Limit, burst int) OptionFunc
WithRateLimit throttles calls to at most r events per second with the given burst. It applies to every attempt, retries included.
func WithRetryable ¶
func WithRetryable(fn func(error) bool) OptionFunc
WithRetryable overrides the predicate deciding whether an error is retryable.
type Options ¶
type Options struct {
// MaxRetries is the number of retries attempted after the first failure
// (so a total of MaxRetries+1 attempts). A negative value disables retries.
MaxRetries int
// BaseBackoff is the delay before the first retry; it doubles on each
// subsequent retry, capped at MaxBackoff.
BaseBackoff time.Duration
// MaxBackoff caps the backoff delay.
MaxBackoff time.Duration
// Retryable decides whether an error is worth retrying. Defaults to
// DefaultRetryable (everything except context cancellation/deadline).
Retryable func(error) bool
// Limiter, when set, throttles every call (Wait before each attempt,
// including retries).
Limiter *rate.Limiter
}
Options configures a RetryClient.
type RetryClient ¶
type RetryClient struct {
// contains filtered or unexported fields
}
RetryClient decorates an llm.Client with retries (exponential backoff) and an optional rate limiter. It is safe for concurrent use as long as the wrapped client is.
func NewRetryClient ¶
func NewRetryClient(client llm.Client, funcs ...OptionFunc) *RetryClient
NewRetryClient wraps client with retry (and optional rate-limit) behaviour.
func (*RetryClient) ChatCompletion ¶
func (c *RetryClient) ChatCompletion(ctx context.Context, funcs ...llm.ChatCompletionOptionFunc) (llm.ChatCompletionResponse, error)
ChatCompletion implements llm.ChatCompletionClient with retries.
func (*RetryClient) ChatCompletionStream ¶
func (c *RetryClient) ChatCompletionStream(ctx context.Context, funcs ...llm.ChatCompletionOptionFunc) (<-chan llm.StreamChunk, error)
ChatCompletionStream implements llm.ChatCompletionStreamingClient. Only the call opening the stream is retried; once the channel is returned, chunks flow through unchanged (a mid-stream failure cannot be safely retried).
func (*RetryClient) Embeddings ¶
func (c *RetryClient) Embeddings(ctx context.Context, inputs []string, funcs ...llm.EmbeddingsOptionFunc) (llm.EmbeddingsResponse, error)
Embeddings implements llm.EmbeddingsClient with retries.
Source Files