Documentation
¶
Overview ¶
Package middleware provides provider wrappers: Cache, Retry, Timeout, and Tracing.
Package middleware provides provider wrappers for retry, timeout, response cache, and tracing.
Wrap a niro.Provider to add production behavior:
p := openai.New(apiKey)
p = middleware.NewRetryProvider(p, middleware.RetryConfig{...})
p = middleware.NewTimeoutProvider(p, 5*time.Minute)
All wrappers implement niro.Provider. Compose in any order; typically retry innermost, then timeout, then cache or tracing.
Index ¶
- func Composed(p niro.Provider, timeout time.Duration, retryConfig *RetryConfig) niro.Provider
- func NewTimeoutProvider(p niro.Provider, timeout time.Duration) niro.Provider
- func WithGenerationTimeout(ctx context.Context, timeout time.Duration) (context.Context, context.CancelFunc)
- func WithTraceContext(ctx context.Context, trace TraceContext) context.Context
- func WrapWithSmartRetry(p niro.Provider, cfg RetryConfig) niro.Provider
- type BackoffStrategy
- type Cache
- type CacheOptions
- type ConstantBackoff
- type ExponentialBackoff
- type RequestID
- type RetryConfig
- type RetryHintProvider
- type RetryProvider
- type TraceContext
- type TracingProvider
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func Composed ¶
Composed combines multiple provider wrappers (timeout, tracing, retry). Useful for building a production-ready provider from building blocks. Apply order: retry (outermost) → timeout → tracing → base.
func NewTimeoutProvider ¶
NewTimeoutProvider returns a Provider that enforces generation timeouts. It delegates to niro.NewTimeoutProvider; use niro.TimeoutConfig and niro.DefaultTimeoutConfig / niro.TelephonyTimeoutConfig for presets.
func WithGenerationTimeout ¶
func WithGenerationTimeout(ctx context.Context, timeout time.Duration) (context.Context, context.CancelFunc)
WithGenerationTimeout returns a context with a generation timeout applied. Deprecated: use niro.WithGenerationTimeout instead.
func WithTraceContext ¶
func WithTraceContext(ctx context.Context, trace TraceContext) context.Context
WithTraceContext injects trace context into a request context.
func WrapWithSmartRetry ¶
func WrapWithSmartRetry(p niro.Provider, cfg RetryConfig) niro.Provider
WrapWithSmartRetry adds retries only when they are useful.
If cfg.DisableWhenProviderRetries is true and p indicates built-in retries, p is returned unchanged.
Types ¶
type BackoffStrategy ¶
type BackoffStrategy interface {
// Delay returns the duration to wait before the nth attempt (0-indexed).
Delay(attempt int) time.Duration
}
BackoffStrategy defines how to calculate delays between retries.
type Cache ¶
type Cache struct {
// contains filtered or unexported fields
}
Cache is a sharded LRU response cache with TTL for Provider responses. Safe for concurrent use by multiple goroutines.
func (*Cache) Clear ¶
func (c *Cache) Clear()
Clear removes all entries from the cache and resets stats.
type CacheOptions ¶
type CacheOptions struct {
// MaxEntries is the maximum total number of entries across all shards.
// Defaults to 1024 if zero.
MaxEntries int
// TTL is how long a cached entry is considered valid.
// Zero means no expiry.
TTL time.Duration
// KeyFn allows custom cache key generation.
// Defaults to sha256 of (model + messages + tools + response_format).
KeyFn func(*niro.Request) [32]byte
}
CacheOptions controls Cache behavior.
type ConstantBackoff ¶
ConstantBackoff uses a fixed delay between retries.
type ExponentialBackoff ¶
type ExponentialBackoff struct {
InitialDelay time.Duration
Multiplier float64
MaxDelay time.Duration
Jitter bool // Add random jitter to avoid thundering herd
}
ExponentialBackoff uses exponential backoff with optional jitter. Formula: min(initialDelay * (multiplier ^ attempt) + jitter, maxDelay)
type RequestID ¶
type RequestID string
RequestID is a unique identifier for a request used for tracing.
func GenerateRequestID ¶
func GenerateRequestID() RequestID
GenerateRequestID creates a new unique request ID using crypto/rand.
type RetryConfig ¶
type RetryConfig struct {
// Maximum number of attempts (including initial attempt)
MaxAttempts int
// Backoff strategy
Backoff BackoffStrategy
// Should retry based on error
ShouldRetry func(err error) bool
// Optional: callback on retry
OnRetry func(attempt int, err error)
// DisableWhenProviderRetries avoids double-retry when the underlying provider
// or SDK already performs retries internally.
DisableWhenProviderRetries bool
}
RetryConfig configures retry behavior.
func DefaultRetryConfig ¶
func DefaultRetryConfig() RetryConfig
DefaultRetryConfig returns sensible defaults: 3 attempts, exponential backoff with jitter.
type RetryHintProvider ¶
type RetryHintProvider interface {
ProviderHandlesRetries() bool
}
RetryHintProvider is an optional provider extension that reports whether the underlying SDK/client already performs retries.
type RetryProvider ¶
type RetryProvider struct {
// contains filtered or unexported fields
}
RetryProvider wraps a Provider with automatic retry logic.
func NewRetryProvider ¶
func NewRetryProvider(p niro.Provider, config RetryConfig) *RetryProvider
NewRetryProvider creates a Provider that retries on transient failures.
type TraceContext ¶
type TraceContext struct {
RequestID RequestID // Unique request ID
ParentID string // Parent span ID (optional)
UserID string // User identifier (optional)
SessionID string // Session identifier (optional)
Attributes map[string]string // Additional attributes
}
TraceContext holds tracing information for a generation.
func GetTraceContext ¶
func GetTraceContext(ctx context.Context) TraceContext
GetTraceContext retrieves trace context from a request context.
type TracingProvider ¶
type TracingProvider struct {
// contains filtered or unexported fields
}
TracingProvider wraps a Provider and injects tracing information.
func NewTracingProvider ¶
func NewTracingProvider(p niro.Provider) *TracingProvider
NewTracingProvider creates a Provider that automatically generates and propagates trace IDs.
Source Files