ratelimit

package
v0.0.92 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 3, 2026 License: MIT Imports: 12 Imported by: 0

Documentation

Overview

Package ratelimit paces outbound provider requests so pi-go stays inside a published quota instead of discovering it by being rejected.

It exists because retrying is not pacing. pi-go's stream retry (see internal/provider/retry.go) re-sends a rejected request after 3s, 5s, 7s — pauses shorter than the per-minute window that rejected it, so the retries land inside the same exhausted window and burn the budget before the turn gives up. The failure this was written against: Gemini returns

Quota exceeded for metric:
generativelanguage.googleapis.com/generate_content_paid_tier_input_token_count,
limit: 2000000, model: gemini-3.8-flash

after pi-go pushed 2,005,778 input tokens through 23 requests in 60 seconds. No request was individually unreasonable; their sum was.

Two properties follow from that failure and shape the design:

  • The binding limit is *input tokens* per minute, not requests per minute. 23 requests a minute is nothing; 2M tokens a minute is the wall. A requests-per-minute cap tuned to stop it would throttle small requests for no reason, so tokens are the primary budget and requests are an optional secondary one.

  • The budget is shared, not per-client. The three rejections above arrived 0.7s apart from concurrent callers — a subagent and the main turn, each holding its own model.LLM. A limiter owned by one client would not have seen the others' spend, so limiters are looked up from a process-wide registry keyed by endpoint (see Shared).

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func ScopeFor

func ScopeFor(provider, model, baseURL string) string

ScopeFor names the budget a request spends from.

It is keyed by model as well as provider because that is how the quota is actually enforced: Gemini's rejection carries quotaId=GenerateContentPaidTierInputTokensPerModelPerMinute with quotaDimensions{model: gemini-3.8-flash}. Keying on the provider alone would pool two models that have separate budgets, throttling both to one's limit.

The base URL is included so the same model reached through a local gateway and reached directly are not treated as one budget — they may well be different accounts.

Types

type Limiter

type Limiter struct {
	// contains filtered or unexported fields
}

Limiter paces requests against a token-per-minute and/or request-per-minute budget, and absorbs server-ordered cooldowns.

The zero value is not usable; a nil *Limiter is, and does nothing — that is how "no limits configured" is represented at call sites.

func New

func New(l Limits) *Limiter

New builds a Limiter for the given budget, or nil when nothing is limited.

Each bucket's burst is a full window's worth rather than 1. Smoothing to a single unit would serialize an agent turn that legitimately fans out several small requests at once, and it would make the token bucket reject any request larger than one token. A full-window burst matches the shape of the server's own limit: spend it all at once if you like, then wait.

func Shared

func Shared(scope string, l Limits) *Limiter

Shared returns the process-wide Limiter for scope, creating it on first use.

Every model.LLM aimed at the same endpoint must share one limiter, because the quota is enforced on the account, not on the client: the main turn, a subagent, and a title generation all spend from the same budget. scope is the identity of that budget — see ScopeFor.

The first caller's limits win. A later caller asking for the same scope with different numbers gets the existing limiter rather than a second bucket, since two buckets over one quota would each permit the full rate.

func (*Limiter) Backoff

func (l *Limiter) Backoff(d time.Duration)

Backoff records a pause the server asked for, holding back every caller sharing this limiter until it lapses.

This is what makes a Retry-After actually change pi-go's behavior. Applying the hint only to the request that was rejected would leave the other concurrent callers on the same quota sending into the closed window — which is how three rejections arrived 0.7s apart in the failure this was written against. Extending an existing cooldown rather than replacing it keeps the longest hint any caller was given.

func (*Limiter) Wait

func (l *Limiter) Wait(ctx context.Context, inputTokens int) error

Wait blocks until sending a request costing inputTokens is within budget, or until ctx is canceled.

A nil Limiter returns immediately, so callers never have to branch on whether pacing is configured.

type Limits

type Limits struct {
	// RequestsPerMinute caps request count. Secondary to the token budget:
	// useful for providers that publish an RPM ceiling, and for free tiers
	// where it is the limit that actually binds.
	RequestsPerMinute int

	// InputTokensPerMinute caps the input tokens sent per minute. This is the
	// limit that produces the 429 in practice — see the package comment.
	InputTokensPerMinute int
}

Limits describes how fast pi-go may send to one provider endpoint. A zero field means "no limit of this kind"; a zero Limits disables pacing entirely.

func DefaultsFor

func DefaultsFor(provider, model string) Limits

DefaultsFor returns the built-in budget for a provider/model pair, or a zero Limits when pi-go has no evidence for one.

Defaults are deliberately sparse. A limit invented for a provider whose quota has not been observed would either throttle a user who was fine or sit too high to help, and it would be indistinguishable in the config from a number that means something. Gemini is here because its rejection states the quota outright; everything else is unlimited until it earns an entry.

func (Limits) Enabled

func (l Limits) Enabled() bool

Enabled reports whether these limits ask for any pacing at all.

type Transport

type Transport struct {
	Base    http.RoundTripper
	Limiter *Limiter
}

Transport paces every request through a Limiter, and feeds the server's own retry hints back into it.

It is an http.RoundTripper rather than a wrapper around each provider's Generate because that is the one place every provider agrees on: pi-go reaches Gemini through four different code paths (the genai client, the OpenAI Chat Completions client, the Responses client, and an OpenAI- compatible local gateway), and all four end up here.

func (*Transport) RoundTrip

func (t *Transport) RoundTrip(req *http.Request) (*http.Response, error)

RoundTrip waits for budget, sends, and records any cooldown the server asked for.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL