limiter

package
v0.54.0-rc.4 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 8, 2026 License: MIT Imports: 7 Imported by: 0

Documentation

Overview

Package limiter implements the bounded-concurrency admission control used by the upstream tool-call choke point (spec 093, GH #955).

A Limiter caps the number of concurrently *running* calls in one scope (a single upstream server, or the proxy-wide aggregate) and parks excess calls in a bounded FIFO wait queue. Callers acquire with an absolute queue deadline that is shared across tiers (FR-004): waiting for a per-server slot and then for a global slot must never grant two full queue timeouts.

A Limiter owns OCCUPANCY (how many calls are running, who is queued, whether the scope is retired) and nothing else. It never stores the limits themselves: those live in the registry's immutable published generation, and every admission decision is made against the values handed to it, so one admission can never combine one scope's new cap with another scope's old one (FR-021).

Implementation note: the queue is a hand-rolled FIFO waiter list guarded by one mutex rather than golang.org/x/sync/semaphore. A semaphore's capacity is fixed at construction, but FR-021 requires occupancy to be *shared across generations* on hot reload — lowering a cap must not grant new capacity until the grandfathered running calls drain, and raising it must admit eligible waiters immediately. Both need a resizable cap over a live occupancy counter, which the waiter list gives us directly.

Index

Constants

View Source
const RetryAdvice = "Retry in a few seconds."

RetryAdvice is the closing sentence of every shed message. Agents read tool-call error text; telling them the failure is transient is what turns a rejection into a retry instead of an abandoned task (FR-010).

Variables

View Source
var (
	// ErrQueueFull matches a rejection caused by a full (or zero-sized) queue.
	ErrQueueFull = errors.New("concurrency limit: queue full")
	// ErrQueueTimeout matches a rejection caused by the queue deadline expiring.
	ErrQueueTimeout = errors.New("concurrency limit: queue timeout")
	// ErrServerUnavailable matches a rejection caused by the target server being
	// disabled/quarantined/removed (FR-009).
	ErrServerUnavailable = errors.New("concurrency limit: server unavailable")
)

Sentinel errors for errors.Is matching at the shed-semantics seams (MCP isError results, REST 429 mapping, activity "rejected" status). The concrete error returned by Acquire is always a *LimitError, which reports Is() true for the sentinel matching its Reason.

Functions

This section is empty.

Types

type LimitError

type LimitError struct {
	Scope      Scope
	Reason     Reason
	Server     string
	Limit      int
	RetryAfter time.Duration
}

LimitError is the typed rejection identity that must survive end-to-end through every dispatch path (FR-011). It carries everything the shed seams need: which tier shed the call, why, which server (empty for the global scope — the message must never blame a server for a proxy-wide limit), the limit that was in force, and the Retry-After hint derived from the shedding scope's effective queue_timeout.

func (*LimitError) Error

func (e *LimitError) Error() string

func (*LimitError) Is

func (e *LimitError) Is(target error) bool

Is makes errors.Is(err, ErrQueueFull|ErrQueueTimeout|ErrServerUnavailable) work against the typed error.

func (*LimitError) UserMessage

func (e *LimitError) UserMessage() string

UserMessage renders the caller-facing explanation of a shed (FR-010). It is the single rendering of that text: the MCP isError result, the REST 429 body and the activity record all use it, so an operator never sees two different descriptions of the same rejection.

The global branch NEVER names a server: the proxy-wide limiter shed the call because the whole instance is saturated, and blaming the upstream the call happened to target would send an operator debugging the wrong thing.

type Limiter

type Limiter struct {
	// contains filtered or unexported fields
}

Limiter guards one scope. The zero value is not usable; use New. A nil *Limiter is a valid no-op passthrough so callers can skip nil checks.

func New

func New(scope Scope, server string, limits Limits) *Limiter

New builds a STANDALONE limiter with fixed limits. Registry-owned scopes use newPublished instead, so their limits always come from the current generation. server is the upstream name for ScopeServer and must be empty for ScopeGlobal (a global rejection must never blame a server, FR-010).

func (*Limiter) Acquire

func (l *Limiter) Acquire(ctx context.Context, queueDeadline time.Time) (func(), error)

Acquire admits one call into the scope under this limiter's currently published limits, waiting in the bounded FIFO queue if the cap is reached. queueDeadline is the ABSOLUTE deadline for the whole admission (shared across tiers, FR-004); the zero time means "wait until the caller's context ends".

The registry does NOT use this entry point: a two-tier admission must resolve both scopes from one generation, which is what acquire takes as an argument.

It returns an idempotent release closure bound to this instance, or:

  • *LimitError{Reason: queue_full} when there is no pending capacity,
  • *LimitError{Reason: queue_timeout} when the deadline expired while queued,
  • *LimitError{Reason: server_unavailable} when the scope was retired,
  • the caller's context error (context.Canceled / DeadlineExceeded) when the CALLER's context ended while queued — never reported as shedding (FR-005).

func (*Limiter) Limits

func (l *Limiter) Limits() Limits

Limits returns the limits currently published for this scope.

func (*Limiter) Retire

func (l *Limiter) Retire()

Retire marks the scope dead (server disabled, quarantined or removed) and fails every queued call promptly with ErrServerUnavailable instead of letting them sit until the queue deadline (FR-009). Outstanding holds keep draining into this instance; a re-added server gets a fresh instance so its capacity is never double-counted against the retired one.

func (*Limiter) Retired

func (l *Limiter) Retired() bool

Retired reports whether this instance has been retired.

func (*Limiter) Stats

func (l *Limiter) Stats() Stats

Stats returns the current occupancy and queue depth.

type Limits

type Limits struct {
	// Max is the maximum number of concurrently running calls. <= 0 means the
	// scope does not limit anything (occupancy is still tracked so a later
	// hot-reload that enables the limit sees the in-flight calls).
	Max int
	// QueueSize is the number of calls allowed to wait for a slot. 0 means no
	// pending capacity: calls arriving at the cap are shed immediately.
	QueueSize int
	// QueueTimeout is the scope's configured wait budget. The limiter itself
	// takes an absolute deadline from the caller; this value is reported as the
	// Retry-After hint on rejections produced by this scope.
	QueueTimeout time.Duration
}

Limits is one scope's configured concurrency settings, already resolved from the config tri-states. A Limits value is IMMUTABLE once published: a reload publishes a new value, it never edits one in place.

func (Limits) Enabled

func (l Limits) Enabled() bool

Enabled reports whether this scope actually caps concurrency.

type Observer

type Observer func(ctx context.Context, rej Rejection)

Observer receives every shed call. Implementations must not block: they run on the caller's goroutine at the moment of rejection.

type Reason

type Reason string

Reason is the stable machine-readable cause of a rejection (FR-012).

const (
	// ReasonQueueFull means the wait queue had no pending capacity, so the call
	// was shed immediately without waiting.
	ReasonQueueFull Reason = "queue_full"
	// ReasonQueueTimeout means the call waited past its absolute queue deadline.
	ReasonQueueTimeout Reason = "queue_timeout"
	// ReasonServerUnavailable means the target server was disabled, quarantined
	// or removed while the call was queued (or just before it enqueued).
	ReasonServerUnavailable Reason = "server_unavailable"
)

type Registry

type Registry struct {
	// contains filtered or unexported fields
}

Registry owns the live limiter instances: one per configured upstream server plus the proxy-wide aggregate. Readers take a whole GENERATION through one atomic pointer (no lock on the hot path); writers (config apply / hot reload) serialize on the registry mutex and republish the generation as a unit.

Three invariants drive the shape of this type:

  • FR-021, atomic publication: an admission must never combine one scope's new settings with another scope's old ones, nor a new cap with a queue deadline derived from an older generation. Everything an admission needs — both limiter instances, both scopes' limits, and the resolved wait budget — is resolved from ONE generation load, and the limits themselves live only in generations. Publishing therefore cannot disturb an admission already in flight: the new generation is stored first and the wait queues are re-evaluated against it afterwards.

  • FR-021, shared occupancy: limiter instances SURVIVE a reload (only the limits are republished), and an instance exists for every eligible scope even when that scope currently caps nothing. An unlimited scope still counts its running calls, so enabling a cap later sees the calls already in flight instead of starting from zero and admitting a second cap's worth on top of them.

  • FR-009, no admit-after-disable: retiring a server TOMBSTONES its entry and the tombstone is never removed. Absence of a scope means "unlimited", so absence must never be able to follow retirement — a caller that resolved its client before the server was disabled can reach admission at any later time, and it has to be refused, not waved through. Outstanding holds drain into the retired instance; a server re-added under the same name replaces the tombstone with a fresh instance, so the map holds at most one entry per distinct server name the process has ever configured.

func NewRegistry

func NewRegistry() *Registry

NewRegistry returns an empty registry: nothing is published, so every Acquire is a no-op passthrough (FR-006).

func (*Registry) Acquire

func (r *Registry) Acquire(ctx context.Context, server string) (func(), error)

Acquire admits a call through both tiers of ONE published generation: the same generation supplies both limiters, both scopes' limits — which are what the admission decision itself is made against — and the single absolute queue deadline (FR-004, FR-021). The per-server slot is taken first so a slow upstream's queue does not pin global capacity while waiting (FR-002); on a global rejection the per-server slot is released again.

The returned release closure releases both tiers and is safe to call more than once.

func (*Registry) Apply

func (r *Registry) Apply(global Limits, servers map[string]Limits)

Apply publishes one atomic generation of limits for every scope (FR-021): the global aggregate plus the resolved per-server limits. Servers missing from the map are retired (their tombstone stays, see the type comment).

func (*Registry) Global

func (r *Registry) Global() *Limiter

Global returns the proxy-wide limiter, or nil when none was published.

func (*Registry) QueueBudget

func (r *Registry) QueueBudget(server string) time.Duration

QueueBudget reports the wait budget an admission for this server would get from the currently published generation. It is the single owner of the FR-004 rule (the config layer resolves per-scope settings; combining them into one deadline happens here, where the generation guarantees both scopes come from the same publish).

func (*Registry) RetireServer

func (r *Registry) RetireServer(name string)

RetireServer tombstones a server's limiter: queued calls fail immediately with ErrServerUnavailable and later admissions are refused, even if the caller snapshotted the client before the state change (FR-009, admit-after-disable race). Outstanding holds drain into the retired instance.

func (*Registry) Server

func (r *Registry) Server(name string) *Limiter

Server returns the LIVE limiter for an upstream, or nil when that server has no limiter — unconfigured, or retired. A retired instance is deliberately invisible here (it is a tombstone, not a working scope); admission still finds it internally and refuses the call.

func (*Registry) ServerStats

func (r *Registry) ServerStats() map[string]Stats

ServerStats returns the occupancy of every live per-server limiter, keyed by server name (FR-013: per-server queue depth for the metrics surface). Retired tombstones are omitted: they belong to servers that are no longer configured.

func (*Registry) SetGlobal

func (r *Registry) SetGlobal(limits Limits)

SetGlobal publishes the global aggregate limiter's settings.

func (*Registry) SetServer

func (r *Registry) SetServer(name string, limits Limits) *Limiter

SetServer publishes one server's limits and returns the live instance.

type Rejection

type Rejection struct {
	// Server is the upstream the call targeted. Always set, even for a global
	// rejection — the caller-facing MESSAGE must not blame it (FR-010), but the
	// activity record and the metric label still need to know where the call was
	// headed.
	Server string
	// Tool is the upstream tool name (without the server prefix).
	Tool string
	// Scope is the tier that shed the call.
	Scope Scope
	// Reason is queue_full or queue_timeout.
	Reason Reason
	// Limit is the cap that was in force in the shedding scope.
	Limit int
	// RetryAfter is the shedding scope's effective queue_timeout, used as the
	// REST Retry-After hint (FR-011).
	RetryAfter time.Duration
	// Waited is how long the call spent in the queue before being shed (0 for a
	// queue_full shed, which never waits).
	Waited time.Duration
	// Message is the caller-facing explanation (LimitError.Error()).
	Message string
}

Rejection describes one shed call. It is the payload of the ORIGIN-INDEPENDENT rejection seam required by FR-012/FR-013: the observer is invoked at the admission point inside the managed client, which every in-process dispatch path funnels through (MCP tool-call variants, the REST tool-call endpoint, sandboxed code execution, activity replay). Paths that never reach the MCP dispatch layer therefore still produce a "rejected" activity record and a rejection metric.

type Scope

type Scope string

Scope identifies which limiter tier produced a decision (spec 093, FR-010).

const (
	// ScopeServer is a per-upstream-server limiter.
	ScopeServer Scope = "server"
	// ScopeGlobal is the proxy-wide aggregate limiter.
	ScopeGlobal Scope = "global"
)

type Stats

type Stats struct {
	Running int
	Queued  int
}

Stats is a point-in-time view of one scope's occupancy, used by the metrics surface (FR-013).

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL