Documentation
¶
Overview ¶
Package middleware provides reusable model.Client middlewares such as adaptive rate limiting.
Index ¶
Constants ¶
This section is empty.
Variables ¶
var ErrRequestTooLarge = errors.New("model middleware: request exceeds rate limiter capacity")
ErrRequestTooLarge is returned when a single request's estimated input-token cost exceeds the limiter's maximum tokens-per-minute capacity. Such a request can never be admitted no matter how long it waits, so the limiter fails fast with this error instead of blocking. Callers should raise the limiter's max TPM or reduce the request size.
Functions ¶
This section is empty.
Types ¶
type AdaptiveRateLimiter ¶
type AdaptiveRateLimiter struct {
// contains filtered or unexported fields
}
AdaptiveRateLimiter applies an AIMD-style adaptive token bucket on top of a model.Client. It estimates the token cost of each request, blocks callers until capacity is available, and adjusts its effective tokens-per-minute budget in response to rate limiting signals from the provider.
The limiter is process-local and designed to sit at the provider client boundary. Callers construct a single instance per process and wrap the underlying model.Client with Middleware before passing it to planners or runtimes.
func NewAdaptiveRateLimiter ¶
func NewAdaptiveRateLimiter(ctx context.Context, m *rmap.Map, key string, initialTPM, maxTPM float64) *AdaptiveRateLimiter
NewAdaptiveRateLimiter constructs an AdaptiveRateLimiter with a tokens-per-minute budget. When m and key are set, it coordinates capacity across processes using a Pulse replicated map; otherwise it operates as a process-local limiter.
func (*AdaptiveRateLimiter) Middleware ¶
func (l *AdaptiveRateLimiter) Middleware() func(model.Client) model.Client
Middleware returns a model.Client middleware that enforces the adaptive tokens-per-minute limit for both Complete and Stream calls.
Source Files