Documentation
¶
Overview ¶
Package limiter implements the bounded-concurrency admission control used by the upstream tool-call choke point (spec 093, GH #955).
A Limiter caps the number of concurrently *running* calls in one scope (a single upstream server, or the proxy-wide aggregate) and parks excess calls in a bounded FIFO wait queue. Callers acquire with an absolute queue deadline that is shared across tiers (FR-004): waiting for a per-server slot and then for a global slot must never grant two full queue timeouts.
A Limiter owns OCCUPANCY (how many calls are running, who is queued, whether the scope is retired) and nothing else. It never stores the limits themselves: those live in the registry's immutable published generation, and every admission decision is made against the values handed to it, so one admission can never combine one scope's new cap with another scope's old one (FR-021).
Implementation note: the queue is a hand-rolled FIFO waiter list guarded by one mutex rather than golang.org/x/sync/semaphore. A semaphore's capacity is fixed at construction, but FR-021 requires occupancy to be *shared across generations* on hot reload — lowering a cap must not grant new capacity until the grandfathered running calls drain, and raising it must admit eligible waiters immediately. Both need a resizable cap over a live occupancy counter, which the waiter list gives us directly.
Index ¶
- Constants
- Variables
- type LimitError
- type Limiter
- type Limits
- type Observer
- type Reason
- type Registry
- func (r *Registry) Acquire(ctx context.Context, server string) (func(), error)
- func (r *Registry) Apply(global Limits, servers map[string]Limits)
- func (r *Registry) Global() *Limiter
- func (r *Registry) QueueBudget(server string) time.Duration
- func (r *Registry) RetireServer(name string)
- func (r *Registry) Server(name string) *Limiter
- func (r *Registry) ServerStats() map[string]Stats
- func (r *Registry) SetGlobal(limits Limits)
- func (r *Registry) SetServer(name string, limits Limits) *Limiter
- type Rejection
- type Scope
- type Stats
Constants ¶
const RetryAdvice = "Retry in a few seconds."
RetryAdvice is the closing sentence of every shed message. Agents read tool-call error text; telling them the failure is transient is what turns a rejection into a retry instead of an abandoned task (FR-010).
Variables ¶
var ( // ErrQueueFull matches a rejection caused by a full (or zero-sized) queue. ErrQueueFull = errors.New("concurrency limit: queue full") // ErrQueueTimeout matches a rejection caused by the queue deadline expiring. ErrQueueTimeout = errors.New("concurrency limit: queue timeout") // disabled/quarantined/removed (FR-009). ErrServerUnavailable = errors.New("concurrency limit: server unavailable") )
Sentinel errors for errors.Is matching at the shed-semantics seams (MCP isError results, REST 429 mapping, activity "rejected" status). The concrete error returned by Acquire is always a *LimitError, which reports Is() true for the sentinel matching its Reason.
Functions ¶
This section is empty.
Types ¶
type LimitError ¶
type LimitError struct {
Scope Scope
Reason Reason
Server string
Limit int
RetryAfter time.Duration
}
LimitError is the typed rejection identity that must survive end-to-end through every dispatch path (FR-011). It carries everything the shed seams need: which tier shed the call, why, which server (empty for the global scope — the message must never blame a server for a proxy-wide limit), the limit that was in force, and the Retry-After hint derived from the shedding scope's effective queue_timeout.
func (*LimitError) Error ¶
func (e *LimitError) Error() string
func (*LimitError) Is ¶
func (e *LimitError) Is(target error) bool
Is makes errors.Is(err, ErrQueueFull|ErrQueueTimeout|ErrServerUnavailable) work against the typed error.
func (*LimitError) UserMessage ¶
func (e *LimitError) UserMessage() string
UserMessage renders the caller-facing explanation of a shed (FR-010). It is the single rendering of that text: the MCP isError result, the REST 429 body and the activity record all use it, so an operator never sees two different descriptions of the same rejection.
The global branch NEVER names a server: the proxy-wide limiter shed the call because the whole instance is saturated, and blaming the upstream the call happened to target would send an operator debugging the wrong thing.
type Limiter ¶
type Limiter struct {
// contains filtered or unexported fields
}
Limiter guards one scope. The zero value is not usable; use New. A nil *Limiter is a valid no-op passthrough so callers can skip nil checks.
func New ¶
New builds a STANDALONE limiter with fixed limits. Registry-owned scopes use newPublished instead, so their limits always come from the current generation. server is the upstream name for ScopeServer and must be empty for ScopeGlobal (a global rejection must never blame a server, FR-010).
func (*Limiter) Acquire ¶
Acquire admits one call into the scope under this limiter's currently published limits, waiting in the bounded FIFO queue if the cap is reached. queueDeadline is the ABSOLUTE deadline for the whole admission (shared across tiers, FR-004); the zero time means "wait until the caller's context ends".
The registry does NOT use this entry point: a two-tier admission must resolve both scopes from one generation, which is what acquire takes as an argument.
It returns an idempotent release closure bound to this instance, or:
- *LimitError{Reason: queue_full} when there is no pending capacity,
- *LimitError{Reason: queue_timeout} when the deadline expired while queued,
- *LimitError{Reason: server_unavailable} when the scope was retired,
- the caller's context error (context.Canceled / DeadlineExceeded) when the CALLER's context ended while queued — never reported as shedding (FR-005).
func (*Limiter) Retire ¶
func (l *Limiter) Retire()
Retire marks the scope dead (server disabled, quarantined or removed) and fails every queued call promptly with ErrServerUnavailable instead of letting them sit until the queue deadline (FR-009). Outstanding holds keep draining into this instance; a re-added server gets a fresh instance so its capacity is never double-counted against the retired one.
type Limits ¶
type Limits struct {
// Max is the maximum number of concurrently running calls. <= 0 means the
// scope does not limit anything (occupancy is still tracked so a later
// hot-reload that enables the limit sees the in-flight calls).
Max int
// QueueSize is the number of calls allowed to wait for a slot. 0 means no
// pending capacity: calls arriving at the cap are shed immediately.
QueueSize int
// QueueTimeout is the scope's configured wait budget. The limiter itself
// takes an absolute deadline from the caller; this value is reported as the
// Retry-After hint on rejections produced by this scope.
QueueTimeout time.Duration
}
Limits is one scope's configured concurrency settings, already resolved from the config tri-states. A Limits value is IMMUTABLE once published: a reload publishes a new value, it never edits one in place.
type Observer ¶
Observer receives every shed call. Implementations must not block: they run on the caller's goroutine at the moment of rejection.
type Reason ¶
type Reason string
Reason is the stable machine-readable cause of a rejection (FR-012).
const ( // ReasonQueueFull means the wait queue had no pending capacity, so the call // was shed immediately without waiting. ReasonQueueFull Reason = "queue_full" // ReasonQueueTimeout means the call waited past its absolute queue deadline. ReasonQueueTimeout Reason = "queue_timeout" // or removed while the call was queued (or just before it enqueued). ReasonServerUnavailable Reason = "server_unavailable" )
type Registry ¶
type Registry struct {
// contains filtered or unexported fields
}
Registry owns the live limiter instances: one per configured upstream server plus the proxy-wide aggregate. Readers take a whole GENERATION through one atomic pointer (no lock on the hot path); writers (config apply / hot reload) serialize on the registry mutex and republish the generation as a unit.
Three invariants drive the shape of this type:
FR-021, atomic publication: an admission must never combine one scope's new settings with another scope's old ones, nor a new cap with a queue deadline derived from an older generation. Everything an admission needs — both limiter instances, both scopes' limits, and the resolved wait budget — is resolved from ONE generation load, and the limits themselves live only in generations. Publishing therefore cannot disturb an admission already in flight: the new generation is stored first and the wait queues are re-evaluated against it afterwards.
FR-021, shared occupancy: limiter instances SURVIVE a reload (only the limits are republished), and an instance exists for every eligible scope even when that scope currently caps nothing. An unlimited scope still counts its running calls, so enabling a cap later sees the calls already in flight instead of starting from zero and admitting a second cap's worth on top of them.
FR-009, no admit-after-disable: retiring a server TOMBSTONES its entry and the tombstone is never removed. Absence of a scope means "unlimited", so absence must never be able to follow retirement — a caller that resolved its client before the server was disabled can reach admission at any later time, and it has to be refused, not waved through. Outstanding holds drain into the retired instance; a server re-added under the same name replaces the tombstone with a fresh instance, so the map holds at most one entry per distinct server name the process has ever configured.
func NewRegistry ¶
func NewRegistry() *Registry
NewRegistry returns an empty registry: nothing is published, so every Acquire is a no-op passthrough (FR-006).
func (*Registry) Acquire ¶
Acquire admits a call through both tiers of ONE published generation: the same generation supplies both limiters, both scopes' limits — which are what the admission decision itself is made against — and the single absolute queue deadline (FR-004, FR-021). The per-server slot is taken first so a slow upstream's queue does not pin global capacity while waiting (FR-002); on a global rejection the per-server slot is released again.
The returned release closure releases both tiers and is safe to call more than once.
func (*Registry) Apply ¶
Apply publishes one atomic generation of limits for every scope (FR-021): the global aggregate plus the resolved per-server limits. Servers missing from the map are retired (their tombstone stays, see the type comment).
func (*Registry) QueueBudget ¶
QueueBudget reports the wait budget an admission for this server would get from the currently published generation. It is the single owner of the FR-004 rule (the config layer resolves per-scope settings; combining them into one deadline happens here, where the generation guarantees both scopes come from the same publish).
func (*Registry) RetireServer ¶
RetireServer tombstones a server's limiter: queued calls fail immediately with ErrServerUnavailable and later admissions are refused, even if the caller snapshotted the client before the state change (FR-009, admit-after-disable race). Outstanding holds drain into the retired instance.
func (*Registry) Server ¶
Server returns the LIVE limiter for an upstream, or nil when that server has no limiter — unconfigured, or retired. A retired instance is deliberately invisible here (it is a tombstone, not a working scope); admission still finds it internally and refuses the call.
func (*Registry) ServerStats ¶
ServerStats returns the occupancy of every live per-server limiter, keyed by server name (FR-013: per-server queue depth for the metrics surface). Retired tombstones are omitted: they belong to servers that are no longer configured.
type Rejection ¶
type Rejection struct {
// Server is the upstream the call targeted. Always set, even for a global
// rejection — the caller-facing MESSAGE must not blame it (FR-010), but the
// activity record and the metric label still need to know where the call was
// headed.
Server string
// Tool is the upstream tool name (without the server prefix).
Tool string
// Scope is the tier that shed the call.
Scope Scope
// Reason is queue_full or queue_timeout.
Reason Reason
// Limit is the cap that was in force in the shedding scope.
Limit int
// RetryAfter is the shedding scope's effective queue_timeout, used as the
// REST Retry-After hint (FR-011).
RetryAfter time.Duration
// Waited is how long the call spent in the queue before being shed (0 for a
// queue_full shed, which never waits).
Waited time.Duration
// Message is the caller-facing explanation (LimitError.Error()).
Message string
}
Rejection describes one shed call. It is the payload of the ORIGIN-INDEPENDENT rejection seam required by FR-012/FR-013: the observer is invoked at the admission point inside the managed client, which every in-process dispatch path funnels through (MCP tool-call variants, the REST tool-call endpoint, sandboxed code execution, activity replay). Paths that never reach the MCP dispatch layer therefore still produce a "rejected" activity record and a rejection metric.