limiter

package
v0.73.0-rc.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 7, 2026 License: MIT Imports: 8 Imported by: 0

Documentation

Overview

Package limiter implements the bounded-concurrency admission control used by the upstream tool-call choke point (spec 093, GH #955).

A Limiter caps the number of concurrently *running* calls in one scope (a single upstream server, or the proxy-wide aggregate) and parks excess calls in a bounded FIFO wait queue. Callers acquire with an absolute queue deadline that is shared across tiers (FR-004): waiting for a per-server slot and then for a global slot must never grant two full queue timeouts.

A Limiter owns OCCUPANCY (how many calls are running, who is queued, whether the scope is retired) and nothing else. It never stores the limits themselves: those live in the registry's immutable published generation, and every admission decision is made against the values handed to it, so one admission can never combine one scope's new cap with another scope's old one (FR-021).

Implementation note: the queue is a hand-rolled FIFO waiter list guarded by one mutex rather than golang.org/x/sync/semaphore. A semaphore's capacity is fixed at construction, but FR-021 requires occupancy to be *shared across generations* on hot reload — lowering a cap must not grant new capacity until the grandfathered running calls drain, and raising it must admit eligible waiters immediately. Both need a resizable cap over a live occupancy counter, which the waiter list gives us directly.

Index

Constants

View Source
const RetryAdvice = "Retry in a few seconds."

RetryAdvice is the closing sentence of every shed message. Agents read tool-call error text; telling them the failure is transient is what turns a rejection into a retry instead of an abandoned task (FR-010).

Variables

View Source
var (
	// ErrQueueFull matches a rejection caused by a full (or zero-sized) queue.
	ErrQueueFull = errors.New("concurrency limit: queue full")
	// ErrQueueTimeout matches a rejection caused by the queue deadline expiring.
	ErrQueueTimeout = errors.New("concurrency limit: queue timeout")
	// ErrServerUnavailable matches a rejection caused by the target server being
	// disabled/quarantined/removed (FR-009).
	ErrServerUnavailable = errors.New("concurrency limit: server unavailable")
	// ErrRateLimited matches a call refused because its agent token is over its
	// call budget.
	ErrRateLimited = errors.New("rate limit: token over its call budget")
)

Sentinel errors for errors.Is matching at the shed-semantics seams (MCP isError results, REST 429 mapping, activity "rejected" status). The concrete error returned by Acquire is always a *LimitError, which reports Is() true for the sentinel matching its Reason.

Functions

This section is empty.

Types

type LimitError

type LimitError struct {
	Scope      Scope
	Reason     Reason
	Server     string
	Limit      int
	RetryAfter time.Duration
	// Burst is the bucket size of a ScopeToken rejection (0 for the other
	// scopes, and for a token whose burst equals its per-minute rate).
	Burst int
}

LimitError is the typed rejection identity that must survive end-to-end through every dispatch path (FR-011). It carries everything the shed seams need: which tier shed the call, why, which server (empty for the global scope — the message must never blame a server for a proxy-wide limit), the limit that was in force, and the Retry-After hint derived from the shedding scope's effective queue_timeout.

func (*LimitError) Error

func (e *LimitError) Error() string

func (*LimitError) Is

func (e *LimitError) Is(target error) bool

Is makes errors.Is(err, ErrQueueFull|ErrQueueTimeout|ErrServerUnavailable|ErrRateLimited) work against the typed error.

func (*LimitError) Shed added in v0.72.0

func (e *LimitError) Shed() bool

Shed reports whether the call was refused by a limit (queue full, queue timeout or the token's call budget) rather than because the server went away. Every dispatch path answers a shed the same way — a retry-friendly error result, or 429 with Retry-After — and the admission observer has already written the call's "rejected" record.

func (*LimitError) UserMessage

func (e *LimitError) UserMessage() string

UserMessage renders the caller-facing explanation of a shed (FR-010). It is the single rendering of that text: the MCP isError result, the REST 429 body and the activity record all use it, so an operator never sees two different descriptions of the same rejection.

The global branch NEVER names a server: the proxy-wide limiter shed the call because the whole instance is saturated, and blaming the upstream the call happened to target would send an operator debugging the wrong thing.

type Limiter

type Limiter struct {
	// contains filtered or unexported fields
}

Limiter guards one scope. The zero value is not usable; use New. A nil *Limiter is a valid no-op passthrough so callers can skip nil checks.

func New

func New(scope Scope, server string, limits Limits) *Limiter

New builds a STANDALONE limiter with fixed limits. Registry-owned scopes use newPublished instead, so their limits always come from the current generation. server is the upstream name for ScopeServer and must be empty for ScopeGlobal (a global rejection must never blame a server, FR-010).

func (*Limiter) Acquire

func (l *Limiter) Acquire(ctx context.Context, queueDeadline time.Time) (func(), error)

Acquire admits one call into the scope under this limiter's currently published limits, waiting in the bounded FIFO queue if the cap is reached. queueDeadline is the ABSOLUTE deadline for the whole admission (shared across tiers, FR-004); the zero time means "wait until the caller's context ends".

The registry does NOT use this entry point: a two-tier admission must resolve both scopes from one generation, which is what acquire takes as an argument.

It returns an idempotent release closure bound to this instance, or:

  • *LimitError{Reason: queue_full} when there is no pending capacity,
  • *LimitError{Reason: queue_timeout} when the deadline expired while queued,
  • *LimitError{Reason: server_unavailable} when the scope was retired,
  • the caller's context error (context.Canceled / DeadlineExceeded) when the CALLER's context ended while queued — never reported as shedding (FR-005).

func (*Limiter) Limits

func (l *Limiter) Limits() Limits

Limits returns the limits currently published for this scope.

func (*Limiter) Retire

func (l *Limiter) Retire()

Retire marks the scope dead (server disabled, quarantined or removed) and fails every queued call promptly with ErrServerUnavailable instead of letting them sit until the queue deadline (FR-009). Outstanding holds keep draining into this instance; a re-added server gets a fresh instance so its capacity is never double-counted against the retired one.

func (*Limiter) Retired

func (l *Limiter) Retired() bool

Retired reports whether this instance has been retired.

func (*Limiter) Stats

func (l *Limiter) Stats() Stats

Stats returns the current occupancy and queue depth.

type Limits

type Limits struct {
	// Max is the maximum number of concurrently running calls. <= 0 means the
	// scope does not limit anything (occupancy is still tracked so a later
	// hot-reload that enables the limit sees the in-flight calls).
	Max int
	// QueueSize is the number of calls allowed to wait for a slot. 0 means no
	// pending capacity: calls arriving at the cap are shed immediately.
	QueueSize int
	// QueueTimeout is the scope's configured wait budget. The limiter itself
	// takes an absolute deadline from the caller; this value is reported as the
	// Retry-After hint on rejections produced by this scope.
	QueueTimeout time.Duration
}

Limits is one scope's configured concurrency settings, already resolved from the config tri-states. A Limits value is IMMUTABLE once published: a reload publishes a new value, it never edits one in place.

func (Limits) Enabled

func (l Limits) Enabled() bool

Enabled reports whether this scope actually caps concurrency.

type Observer

type Observer func(ctx context.Context, rej Rejection)

Observer receives every shed call. Implementations must not block: they run on the caller's goroutine at the moment of rejection.

type RateDecision added in v0.72.0

type RateDecision struct {
	// Allowed is true when the call may proceed (it has been charged).
	Allowed bool
	// Spec is the budget that applied. Zero when the caller has none.
	Spec RateSpec
	// RetryAfter is how long until one call is available again. Zero when the
	// call was allowed.
	RetryAfter time.Duration
}

RateDecision is the outcome of one TokenRates.Allow.

type RateSpec added in v0.72.0

type RateSpec struct {
	PerMinute int
	Burst     int
}

RateSpec is one call budget: PerMinute calls a minute on average, with capacity for Burst calls at once. PerMinute <= 0 means no limit. Burst <= 0 means "as many as PerMinute".

func (RateSpec) Enabled added in v0.72.0

func (s RateSpec) Enabled() bool

Enabled reports whether the spec limits anything.

type Reason

type Reason string

Reason is the stable machine-readable cause of a rejection (FR-012).

const (
	// ReasonQueueFull means the wait queue had no pending capacity, so the call
	// was shed immediately without waiting.
	ReasonQueueFull Reason = "queue_full"
	// ReasonQueueTimeout means the call waited past its absolute queue deadline.
	ReasonQueueTimeout Reason = "queue_timeout"
	// ReasonServerUnavailable means the target server was disabled, quarantined
	// or removed while the call was queued (or just before it enqueued).
	ReasonServerUnavailable Reason = "server_unavailable"
	// ReasonRateLimited means the calling agent token used up its call budget
	// (rate_limit / agent_token_rate_limit). The call was refused before it
	// queued for a concurrency slot and never reached the upstream.
	ReasonRateLimited Reason = "rate_limited"
)

type Registry

type Registry struct {
	// contains filtered or unexported fields
}

Registry owns the live limiter instances: one per configured upstream server plus the proxy-wide aggregate. Readers take a whole GENERATION through one atomic pointer (no lock on the hot path); writers (config apply / hot reload) serialize on the registry mutex and republish the generation as a unit.

Three invariants drive the shape of this type:

  • FR-021, atomic publication: an admission must never combine one scope's new settings with another scope's old ones, nor a new cap with a queue deadline derived from an older generation. Everything an admission needs — both limiter instances, both scopes' limits, and the resolved wait budget — is resolved from ONE generation load, and the limits themselves live only in generations. Publishing therefore cannot disturb an admission already in flight: the new generation is stored first and the wait queues are re-evaluated against it afterwards.

  • FR-021, shared occupancy: limiter instances SURVIVE a reload (only the limits are republished), and an instance exists for every eligible scope even when that scope currently caps nothing. An unlimited scope still counts its running calls, so enabling a cap later sees the calls already in flight instead of starting from zero and admitting a second cap's worth on top of them.

  • FR-009, no admit-after-disable: retiring a server TOMBSTONES its entry and the tombstone is never removed. Absence of a scope means "unlimited", so absence must never be able to follow retirement — a caller that resolved its client before the server was disabled can reach admission at any later time, and it has to be refused, not waved through. Outstanding holds drain into the retired instance; a server re-added under the same name replaces the tombstone with a fresh instance, so the map holds at most one entry per distinct server name the process has ever configured.

func NewRegistry

func NewRegistry() *Registry

NewRegistry returns an empty registry: nothing is published, so every Acquire is a no-op passthrough (FR-006).

func (*Registry) Acquire

func (r *Registry) Acquire(ctx context.Context, server string) (func(), error)

Acquire admits a call through both tiers of ONE published generation: the same generation supplies both limiters, both scopes' limits — which are what the admission decision itself is made against — and the single absolute queue deadline (FR-004, FR-021). The per-server slot is taken first so a slow upstream's queue does not pin global capacity while waiting (FR-002); on a global rejection the per-server slot is released again.

The returned release closure releases both tiers and is safe to call more than once.

func (*Registry) Apply

func (r *Registry) Apply(global Limits, servers map[string]Limits)

Apply publishes one atomic generation of limits for every scope (FR-021): the global aggregate plus the resolved per-server limits. Servers missing from the map are retired (their tombstone stays, see the type comment).

func (*Registry) Global

func (r *Registry) Global() *Limiter

Global returns the proxy-wide limiter, or nil when none was published.

func (*Registry) QueueBudget

func (r *Registry) QueueBudget(server string) time.Duration

QueueBudget reports the wait budget an admission for this server would get from the currently published generation. It is the single owner of the FR-004 rule (the config layer resolves per-scope settings; combining them into one deadline happens here, where the generation guarantees both scopes come from the same publish).

func (*Registry) RetireServer

func (r *Registry) RetireServer(name string)

RetireServer tombstones a server's limiter: queued calls fail immediately with ErrServerUnavailable and later admissions are refused, even if the caller snapshotted the client before the state change (FR-009, admit-after-disable race). Outstanding holds drain into the retired instance.

func (*Registry) Server

func (r *Registry) Server(name string) *Limiter

Server returns the LIVE limiter for an upstream, or nil when that server has no limiter — unconfigured, or retired. A retired instance is deliberately invisible here (it is a tombstone, not a working scope); admission still finds it internally and refuses the call.

func (*Registry) ServerStats

func (r *Registry) ServerStats() map[string]Stats

ServerStats returns the occupancy of every live per-server limiter, keyed by server name (FR-013: per-server queue depth for the metrics surface). Retired tombstones are omitted: they belong to servers that are no longer configured.

func (*Registry) SetGlobal

func (r *Registry) SetGlobal(limits Limits)

SetGlobal publishes the global aggregate limiter's settings.

func (*Registry) SetServer

func (r *Registry) SetServer(name string, limits Limits) *Limiter

SetServer publishes one server's limits and returns the live instance.

func (*Registry) TokenRates added in v0.72.0

func (r *Registry) TokenRates() *TokenRates

TokenRates returns the per-agent-token call budget. Nil-safe.

type Rejection

type Rejection struct {
	// Server is the upstream the call targeted. Always set, even for a global
	// rejection — the caller-facing MESSAGE must not blame it (FR-010), but the
	// activity record and the metric label still need to know where the call was
	// headed.
	Server string
	// Tool is the upstream tool name (without the server prefix).
	Tool string
	// Scope is the tier that shed the call: server, global or token.
	Scope Scope
	// Reason is queue_full, queue_timeout or rate_limited.
	Reason Reason
	// Limit is the cap that was in force in the shedding scope.
	Limit int
	// RetryAfter is the shedding scope's effective queue_timeout, used as the
	// REST Retry-After hint (FR-011).
	RetryAfter time.Duration
	// Burst is the bucket size of a ScopeToken rejection (0 otherwise).
	Burst int
	// Token names the agent token that was refused. Set for ScopeToken only.
	Token TokenIdentity
	// Waited is how long the call spent in the queue before being shed (0 for a
	// queue_full shed, which never waits).
	Waited time.Duration
	// Message is the caller-facing explanation (LimitError.Error()).
	Message string
}

Rejection describes one shed call. It is the payload of the ORIGIN-INDEPENDENT rejection seam required by FR-012/FR-013: the observer is invoked at the admission point inside the managed client, which every in-process dispatch path funnels through (MCP tool-call variants, the REST tool-call endpoint, sandboxed code execution, activity replay). Paths that never reach the MCP dispatch layer therefore still produce a "rejected" activity record and a rejection metric.

type Scope

type Scope string

Scope identifies which limiter tier produced a decision (spec 093, FR-010).

const (
	// ScopeServer is a per-upstream-server limiter.
	ScopeServer Scope = "server"
	// ScopeGlobal is the proxy-wide aggregate limiter.
	ScopeGlobal Scope = "global"
	// ScopeToken is a per-agent-token call budget (issue #1699).
	ScopeToken Scope = "token"
)

type Stats

type Stats struct {
	Running int
	Queued  int
}

Stats is a point-in-time view of one scope's occupancy, used by the metrics surface (FR-013).

type TokenIdentity added in v0.72.0

type TokenIdentity struct {
	Name string
	// Prefix is the 12-character display prefix of the token.
	Prefix string
	// ClientID is set for a client credential.
	ClientID string
}

TokenIdentity is the agent token behind a rate-limited call, as much of it as the activity record needs. Never the secret.

type TokenRates added in v0.72.0

type TokenRates struct {
	// contains filtered or unexported fields
}

TokenRates is the per-caller call budget (issue #1699): one token bucket per key, all sharing a default spec that a caller may override. It is in memory only, so a restart gives every caller a full bucket again.

The zero value is ready to use and limits nothing until SetDefault or an override says otherwise. It is safe for concurrent use.

func NewTokenRates added in v0.72.0

func NewTokenRates(now func() time.Time) *TokenRates

NewTokenRates returns a TokenRates reading the time from now (time.Now when nil). A fake clock is how the tests make a minute pass.

func (*TokenRates) Allow added in v0.72.0

func (t *TokenRates) Allow(key string, override *RateSpec) RateDecision

Allow charges one call to key. override, when non-nil, replaces the default budget for this key (a PerMinute of 0 exempts it). A key with no budget is always allowed and holds no state.

A budget that changes between calls (a config reload, a different override) is picked up on the next call: the bucket keeps the calls it has left, capped to the new size, so editing a limit neither refunds a caller that has used it up nor locks one out.

func (*TokenRates) Default added in v0.72.0

func (t *TokenRates) Default() RateSpec

Default returns the current default budget.

func (*TokenRates) Len added in v0.72.0

func (t *TokenRates) Len() int

Len is the number of live buckets (tests, diagnostics).

func (*TokenRates) SetClock added in v0.72.0

func (t *TokenRates) SetClock(now func() time.Time)

SetClock replaces the time source. For tests; call it before any Allow.

func (*TokenRates) SetDefault added in v0.72.0

func (t *TokenRates) SetDefault(spec RateSpec)

SetDefault publishes the budget applied to every key that has no override. A spec with PerMinute <= 0 turns the default off.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL