ratelimit

package
v0.4.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 11, 2026 License: MIT Imports: 13 Imported by: 0

Documentation

Overview

Package ratelimit paces outbound provider requests so pi-go stays inside a published quota instead of discovering it by being rejected.

It exists because retrying is not pacing. pi-go's stream retry (see internal/provider/retry.go) re-sends a rejected request after 3s, 5s, 7s — pauses shorter than the per-minute window that rejected it, so the retries land inside the same exhausted window and burn the budget before the turn gives up. The failure this was written against: Gemini returns

Quota exceeded for metric:
generativelanguage.googleapis.com/generate_content_paid_tier_input_token_count,
limit: 2000000, model: gemini-3.8-flash

after pi-go pushed 2,005,778 input tokens through 23 requests in 60 seconds. No request was individually unreasonable; their sum was.

Two properties follow from that failure and shape the design:

  • The binding limit is *input tokens* per minute, not requests per minute. 23 requests a minute is nothing; 2M tokens a minute is the wall. A requests-per-minute cap tuned to stop it would throttle small requests for no reason, so tokens are the primary budget and requests are an optional secondary one.

  • The budget is shared, not per-client. The three rejections above arrived 0.7s apart from concurrent callers — a subagent and the main turn, each holding its own model.LLM. A limiter owned by one client would not have seen the others' spend, so limiters are looked up from a process-wide registry keyed by endpoint (see Shared).

The token budget is a *rolling window*, not a token bucket. The server's quota is a rolling 60-second window: a burst at t=0 still counts against the window at t=59 and only falls out at t=60. A token bucket with a full-window burst lets a client dump a whole window's worth instantly and again a minute later, which the server counts as two windows' worth in one — the bursty failure mode that produced mid-stream 429s after the original fix. The rolling window here delays a request until its input tokens fit in the last window, so a second full burst is held rather than sent and rejected.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func MetricsHandler added in v0.1.0

func MetricsHandler() http.Handler

MetricsHandler serves every provider+model scope's rate-limit metrics in Prometheus text exposition format. Mount it at whatever path the caller chooses (e.g. "/metrics"); it does not register itself anywhere.

func ScopeFor

func ScopeFor(provider, model, baseURL string) string

ScopeFor names the budget a request spends from.

It is keyed by model as well as provider because that is how the quota is actually enforced: Gemini's rejection carries quotaId=GenerateContentPaidTierInputTokensPerModelPerMinute with quotaDimensions{model: gemini-3.8-flash}. Keying on the provider alone would pool two models that have separate budgets, throttling both to one's limit.

The base URL is included so the same model reached through a local gateway and reached directly are not treated as one budget — they may well be different accounts.

func WriteMetrics added in v0.1.0

func WriteMetrics(w io.Writer, scopes []ScopeMetrics) error

WriteMetrics renders every scope in scopes as Prometheus text exposition format (https://prometheus.io/docs/instrumenting/exposition_formats/): six gauge families, each with one HELP/TYPE header followed by one sample line per scope.

Reporting both the current rolling value and its peak is the point: the dashboard being mirrored is a peak view precisely because a point-in-time gauge alone hides the bursts that trip a 429 — see the package-level comment in ratelimit.go. The limit gauges let a scraper compute headroom the same way the dashboard's red line does.

Types

type Limiter

type Limiter struct {
	// contains filtered or unexported fields
}

Limiter paces requests against a rolling input-token window and/or a request-per-minute budget, and absorbs server-ordered cooldowns.

The zero value is not usable; a nil *Limiter is, and does nothing — that is how "no limits configured" is represented at call sites.

func New

func New(l Limits) *Limiter

New builds a Limiter for the given budget, or nil when nothing is limited.

The RPM budget is a token bucket with a full-window burst, so a turn that legitimately fans out several small requests at once is not serialized. The token budget is a rolling window instead — see the package comment for why a bucket is the wrong shape for a per-minute quota.

func Shared

func Shared(scope string, l Limits) *Limiter

Shared returns the process-wide Limiter for scope, creating it on first use.

Every model.LLM aimed at the same endpoint must share one limiter, because the quota is enforced on the account, not on the client: the main turn, a subagent, and a title generation all spend from the same budget. scope is the identity of that budget — see ScopeFor.

The first caller's limits win. A later caller asking for the same scope with different numbers gets the existing limiter rather than a second bucket, since two buckets over one quota would each permit the full rate.

func (*Limiter) Backoff

func (l *Limiter) Backoff(d time.Duration)

Backoff records a pause the server asked for, holding back every caller sharing this limiter until it lapses.

This is what makes a Retry-After actually change pi-go's behavior. Applying the hint only to the request that was rejected would leave the other concurrent callers on the same quota sending into the closed window — which is how three rejections arrived 0.7s apart in the failure this was written against. Extending an existing cooldown rather than replacing it keeps the longest hint any caller was given.

func (*Limiter) Wait

func (l *Limiter) Wait(ctx context.Context, inputTokens int) error

Wait blocks until sending a request costing inputTokens is within budget, or until ctx is canceled.

A nil Limiter returns immediately, so callers never have to branch on whether pacing is configured.

type Limits

type Limits struct {
	// RequestsPerMinute caps request count. Secondary to the token budget:
	// useful for providers that publish an RPM ceiling, and for free tiers
	// where it is the limit that actually binds.
	RequestsPerMinute int

	// InputTokensPerMinute caps the input tokens sent per minute. This is the
	// limit that produces the 429 in practice — see the package comment.
	InputTokensPerMinute int
}

Limits describes how fast pi-go may send to one provider endpoint. A zero field means "no limit of this kind"; a zero Limits disables pacing entirely.

func DefaultsFor

func DefaultsFor(provider, model string) Limits

DefaultsFor returns the built-in budget for a provider/model pair, or a zero Limits when pi-go has no evidence for one.

Defaults are deliberately sparse. A limit invented for a provider whose quota has not been observed would either throttle a user who was fine or sit too high to help, and it would be indistinguishable in the config from a number that means something. Gemini is here because its rejection states the quota outright; everything else is unlimited until it earns an entry.

func (Limits) Enabled

func (l Limits) Enabled() bool

Enabled reports whether these limits ask for any pacing at all.

type ScopeMetrics added in v0.1.0

type ScopeMetrics struct {
	Provider string
	Model    string

	// RequestsPerMinute is the request count admitted in the trailing 60s.
	RequestsPerMinute int
	// PeakRequestsPerMinute is the highest RequestsPerMinute has been since
	// the process started.
	PeakRequestsPerMinute int
	// RequestsLimit is the configured requests-per-minute budget. Zero means
	// unlimited.
	RequestsLimit int

	// InputTokensPerMinute is the input tokens admitted in the trailing 60s.
	InputTokensPerMinute int
	// PeakInputTokensPerMinute is the highest InputTokensPerMinute has been
	// since the process started.
	PeakInputTokensPerMinute int
	// InputTokensLimit is the configured input-tokens-per-minute budget. Zero
	// means unlimited.
	InputTokensLimit int
}

ScopeMetrics is a rendered snapshot of one provider+model scope's rate-limit metrics: the current rolling-60s value and the peak observed since the process started, for both requests and input tokens, alongside the configured budget each is paced against.

func MetricsSnapshot added in v0.1.0

func MetricsSnapshot() []ScopeMetrics

MetricsSnapshot returns the current rolling-window and peak values for every provider+model scope that has admitted at least one request, sorted by provider then model for stable output.

type Transport

type Transport struct {
	Base    http.RoundTripper
	Limiter *Limiter

	// Scope names the budget this transport paces against (see ScopeFor). A
	// non-empty Scope also turns on metrics: each admitted request is counted
	// against it for the /metrics endpoint (see MetricsHandler). Left empty,
	// a caller that does not want metrics is unaffected.
	Scope string
	// Limits is the configured budget for Scope, recorded as-is so a scraper
	// can read it back as the pi_ratelimit_*_limit gauges. Ignored when Scope
	// is empty.
	Limits Limits
}

Transport paces every request through a Limiter, and feeds the server's own retry hints back into it.

It is an http.RoundTripper rather than a wrapper around each provider's Generate because that is the one place every provider agrees on: pi-go reaches Gemini through four different code paths (the genai client, the OpenAI Chat Completions client, the Responses client, and an OpenAI- compatible local gateway), and all four end up here.

func (*Transport) RoundTrip

func (t *Transport) RoundTrip(req *http.Request) (*http.Response, error)

RoundTrip waits for budget, sends, and records any cooldown the server asked for.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL