Documentation
¶
Overview ¶
Package ratelimit decorates an inference.Client with client-side rate limiting: a requests-per-minute pacing limiter, an in-flight concurrency cap, and automatic retry with exponential backoff on rate-limit (HTTP 429) and transient server (5xx) / network failures.
It is llm-free by design — it depends only on the inference interfaces the root module already carries (via eval), so it sits in pkg/ and is wrapped around whatever concrete client cmd/pluto constructs, without pulling github.com/looprig/llm into the root module graph. Both paid commands (`pluto run`, `pluto gen`) route their target and judge clients through it.
Limitation: providers signal a precise wait via the HTTP Retry-After response header, but the inference transport does not surface response headers on an error (failure.APIError carries only Status/Message/Body), so this package cannot honor Retry-After. It falls back to exponential backoff with full jitter, which is the standard behavior when Retry-After is unavailable. Threading Retry-After through would require an inference-module change to capture the header on APIError.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
Types ¶
type Config ¶
type Config struct {
// MaxRPM caps request starts to this many per minute, spacing them evenly
// (a request may wait up to 60s/MaxRPM before it is issued). 0 = unlimited.
MaxRPM int
// MaxConcurrent caps simultaneous in-flight requests. 0 = unlimited.
MaxConcurrent int
// MaxRetries is how many times a rate-limited / transient failure is
// retried after the first attempt (so total attempts = MaxRetries+1).
// 0 = no retries.
MaxRetries int
// BaseBackoff is the first retry's backoff base; each subsequent retry
// doubles it, capped by MaxBackoff, with full jitter applied. Zero uses
// defaultBaseBackoff when retries are enabled.
BaseBackoff time.Duration
// MaxBackoff caps any single backoff wait. Zero uses defaultMaxBackoff.
MaxBackoff time.Duration
}
Config configures the decorator. The zero value enables nothing, and New returns the wrapped client unchanged — no pacing, no cap, no retries.
Source Files