Documentation
¶
Overview ¶
Package ratelimit provides the shared provider rate budget: a reusable token-bucket limiter generalized from the rerank pacer (PR #138) so every subsystem that dials the same provider base URL (rerank, embedding, agent chat) draws from one bucket instead of competing for the same key without coordination.
The limiter is a classic token bucket: tokens accrue at the provider rate (rpm/60 per second) up to the per-minute budget as burst capacity, and every Wait consumes one token. The lock only guards bucket bookkeeping — callers NEVER hold it while sleeping:
- callers within budget (a token is available) proceed immediately, so independent concurrent requests are never serialized;
- excess callers reserve a future refill slot by letting the balance go negative and then concurrently sleep only their own computed delay; no caller waits behind another caller's sleep;
- the provider rate is respected globally: emissions are spaced by the refill rate and instantaneous burst is bounded by the accrued budget.
now and sleep are injectable so tests never wait on wall time.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func NormalizeProviderKey ¶
NormalizeProviderKey collapses the base-URL spellings of one provider to a single registry key: scheme and host are case-insensitive, a trailing slash is insignificant, and an unparseable value degrades to the trimmed lowercase literal so budgeting never panics on operator input.
Types ¶
type Limiter ¶
type Limiter struct {
// contains filtered or unexported fields
}
Limiter paces one shared provider budget in requests per minute. A Limiter with rpm <= 0 is an unlimited passthrough: Wait returns immediately and no bookkeeping happens (the documented default when CORTEX_PROVIDER_RPM is unset, preserving current behavior).
func New ¶
New builds a limiter for the shared provider budget expressed in requests per minute. rpm <= 0 yields an unlimited limiter.
type Option ¶
type Option func(*Limiter)
Option customizes a Limiter. The clock and sleep options exist so tests never wait on wall time; production composition uses the defaults.
type Registry ¶
type Registry struct {
// contains filtered or unexported fields
}
Registry holds one shared budget per normalized provider base URL, so all server subsystems dialing the same provider draw from the same bucket. A nil Registry, or a Registry built with rpm <= 0, hands out nil budgets and every consumer falls back to its local default behavior.
func NewRegistry ¶
NewRegistry builds the server-wide provider budget registry. rpm <= 0 means unlimited: For always returns nil and consumers keep current behavior.
Source Files