Documentation
¶
Overview ¶
Package middleware provides composable func(Handler) Handler batteries for the model call — the "LiteLLM for Go" layer at the semantic level (handlers see messages, tool calls, and token usage, not raw bytes). Attach with agent.Agent.Use.
a := agent.New(model, store, tools...).Use( middleware.Retry(3, middleware.WithRetryIf(middleware.Retryable)), middleware.RateLimit(middleware.NewRateLimiter(time.Second, 5)), )
Per-run limits that must hold across a resume live on the agent, where the journal is: Agent.WithTokenBudget and Agent.WithMaxTurns.
Index ¶
- func Cost(m *CostMeter, r Rates) agent.Middleware
- func ErrorSummary(err error) string
- func Hedge(delay time.Duration, backups ...agent.Model) agent.Middleware
- func RateLimit(r *RateLimiter) agent.Middleware
- func Retry(n int, opts ...RetryOption) agent.Middleware
- func Retryable(err error) bool
- func ToolCache() agent.ToolMiddleware
- func ToolLog(logf func(format string, args ...any), opts ...ToolLogOption) agent.ToolMiddleware
- func ToolRateLimit(r *RateLimiter) agent.ToolMiddleware
- func ToolRetry(n int, opts ...RetryOption) agent.ToolMiddleware
- type CostMeter
- type RateLimiter
- type Rates
- type RetryOption
- type ToolLogOption
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func Cost ¶
func Cost(m *CostMeter, r Rates) agent.Middleware
Cost is a model middleware that records token usage and computes running cost after each successful model call. The caller reads accumulated values via m.
func ErrorSummary ¶ added in v0.7.0
ErrorSummary describes err by what failed, never by what its text says: the control-flow signal it is (pending approval, resume halted, saga aborted), the provider's HTTP status and error type or code (from *agent.APIError) or a rate limit, and the condition or category sentinel it wraps. An error's text often carries content: a provider error body that echoes the prompt, a tool error that embeds the call's arguments, a URL with a credential in its query. The summary carries none of it, so it is safe to put in a log line, a span, or a metric label when content capture is off. An error that wraps none of these is summarized by its Go type, as "unclassified error (*url.Error)". A nil error summarizes to "".
api error: status 400 (invalid_request_error): model rate limited (retry after 2s): model unknown tool: tool
func Hedge ¶
Hedge races the model call against one or more backup models and returns the FIRST successful response, cancelling the rest. It cuts tail latency (a slow primary no longer blocks the turn, you take whichever target answers first) and adds provider-outage failover (a down or throttled primary does not stall the run). It is safe here in a way it is not in a bare service: the agent loop journals only the winning response, at most once, and the losing calls are cancelled before they can touch state, so the durable record stays exactly-once and replays deterministically regardless of which target won the race.
Scheduling:
- The primary (the wrapped model, i.e. the one passed to agent.New) fires immediately.
- If delay > 0, the backups fire only after the primary has been outstanding for delay, so you pay for a backup generation only when the primary is actually slow. A delay of 0 fires every target at once (pure redundancy, always pays for all of them).
- If a target returns an ERROR while backups are still pending, the backups fire immediately (fast failover), rather than waiting out the remaining delay on a fast failure.
The first target to return a nil-error result wins; every other in-flight call is cancelled via a derived context. If every target fails, Hedge returns the joined error. With no backups it is a pass-through, so it is safe to wire unconditionally and add backups later.
Streaming: with backups, a streaming caller (Agent.Stream) receives the winning response's events once the winner is chosen, not token by token. The targets race without access to the caller's stream, so a target that loses can never show the caller text the run does not record, or send anything after the run has ended. With no backups, streaming is live as usual.
Hedge only races the model generation. It does not duplicate tool calls or any other side effect: those run in the agent loop, above this middleware, under the at-most-once journal and the Safety layer. Note it hedges latency and availability, not correctness: taking the first answer is not the same as requiring agreement across models (that is a governance-quorum concern, not this).
// Primary is Anthropic; if it is quiet for 800ms, also try OpenAI and take the first.
a := agent.New(anthropicModel, store, tools...).
Use(middleware.Hedge(800*time.Millisecond, openaiModel))
func RateLimit ¶
func RateLimit(r *RateLimiter) agent.Middleware
RateLimit throttles model calls through the shared limiter, blocking (respecting context cancellation) until a token is available. Proactive throttling complements Retry's reactive Retry-After backoff: it keeps you under the provider's ceiling instead of bouncing off it.
func Retry ¶
func Retry(n int, opts ...RetryOption) agent.Middleware
Retry retries the model call up to n additional times on error, respecting context cancellation. Between attempts it sleeps with exponential backoff (full jitter, base×2^attempt, capped at max). When the error is *agent.RateLimited it sleeps for min(RetryAfter, max) instead.
Default backoff: base=200ms, max=10s. Override with WithBackoff.
func Retryable ¶
Retryable is a ready-made classifier for WithRetryIf: it retries transient failures and fails fast on terminal ones. It retries *agent.RateLimited, a per-attempt timeout (context.DeadlineExceeded), and *agent.APIError with a 5xx or 408 status. It does not retry parent cancellation (context.Canceled), 4xx API errors (auth, validation), or failures the same request would repeat: agent.ErrConfig (a request the adapter refused to build), agent.ErrQuotaExhausted (used-up quota or credit, which no wait lifts), and agent.ErrResponseTooLarge. Errors it cannot classify (e.g. raw network errors, a provider's mid-stream server error) are retried, since those are usually transient.
agent.New(model, store, tools...).Use(middleware.Retry(3, middleware.WithRetryIf(middleware.Retryable)))
func ToolCache ¶
func ToolCache() agent.ToolMiddleware
ToolCache memoizes successful tool results by (name, args) in a process-local map and short-circuits a repeat call — the tool never runs on a hit. It demonstrates the short-circuit power of tool middleware.
It caches only tools marked ReadOnly (agent.Safety); every other tool call runs, since two calls with the same arguments to a side effect are two effects, such as two separate charges. Even for a ReadOnly tool a hit returns the earlier result, so use it only where the output depends solely on the arguments. Errors are never cached. This is an in-memory, unbounded, per-instance cache — it is NOT the durable journal (which already dedupes each tool-use ID at-most-once); it dedupes DISTINCT calls with identical arguments within one process.
func ToolLog ¶
func ToolLog(logf func(format string, args ...any), opts ...ToolLogOption) agent.ToolMiddleware
ToolLog logs one line per tool call — name, duration, and ok/error — via logf (pass log.Printf, t.Logf, or a structured logger's Printf-shaped method). Attach with agent.Agent.UseTool.
a := agent.New(model, store, tools...).UseTool(middleware.ToolLog(log.Printf))
A failed call is logged by its ErrorSummary (category, condition, provider status), not its text: a tool's error text commonly embeds the call's arguments or a URL with a credential in it, and a log line is not the place for them. Pass LogErrorText to log the full text.
func ToolRateLimit ¶
func ToolRateLimit(r *RateLimiter) agent.ToolMiddleware
ToolRateLimit throttles tool calls through the shared limiter.
func ToolRetry ¶
func ToolRetry(n int, opts ...RetryOption) agent.ToolMiddleware
ToolRetry retries a failing tool call up to n additional times with exponential backoff + jitter (honoring an *agent.RateLimited RetryAfter), respecting context cancellation. Reuses RetryOption/WithBackoff from the model-side Retry.
It retries only a tool that is retry-safe (agent.Safety.RetrySafe: ReadOnly, Idempotent, or keyed). A tool that is not runs once, and its error is returned as is: a failed call to a side effect may still have taken effect, as when a payment gateway times out after charging, and running it again could repeat it.
Default backoff: base=200ms, max=10s. Override with WithBackoff.
a := agent.New(model, store, tools...).UseTool(middleware.ToolRetry(3))
Types ¶
type CostMeter ¶
type CostMeter struct {
// contains filtered or unexported fields
}
CostMeter accumulates token usage and computed cost across model calls. It is safe for concurrent use.
type RateLimiter ¶
type RateLimiter struct {
// contains filtered or unexported fields
}
RateLimiter is a token-bucket limiter shared across calls: it allows one token every `interval` with a burst up to `burst`. It is dependency-free (no background goroutine): tokens accrue lazily from elapsed time. Safe for concurrent use. Share one limiter across model and tool middleware to cap a whole agent, or use separate limiters per surface.
func NewRateLimiter ¶
func NewRateLimiter(interval time.Duration, burst int) *RateLimiter
NewRateLimiter allows one call every `interval` (so rate = 1/interval), bursting up to `burst` calls. A burst < 1 is treated as 1. Starts full so the first `burst` calls proceed immediately.
type Rates ¶
type Rates struct {
InputPer1M float64
OutputPer1M float64
CacheReadPer1M float64
CacheWritePer1M float64
}
Rates holds the billing rates for a model in USD per 1,000,000 tokens.
type RetryOption ¶
type RetryOption func(*retryConfig)
RetryOption configures optional behavior of Retry.
func WithBackoff ¶
func WithBackoff(base, max time.Duration) RetryOption
WithBackoff sets the initial backoff duration and the maximum cap. Between retries the sleep doubles each attempt (full jitter applied) and is capped at max. If the error is *agent.RateLimited the Retry-After hint overrides the computed backoff (capped at max).
func WithRetryIf ¶
func WithRetryIf(pred func(error) bool) RetryOption
WithRetryIf classifies which errors are worth retrying: pred returns true to retry, false to fail fast. Use it to stop burning attempts (and tokens) on terminal errors such as HTTP 4xx auth/validation failures, while still retrying transient ones. The default (nil) retries every error. Context cancellation always stops the loop regardless. See Retryable for a ready-made classifier.
func WithTimeout ¶
func WithTimeout(d time.Duration) RetryOption
WithTimeout bounds each individual attempt with its own deadline: the call gets a context that is cancelled after d, so a hung model or tool call fails that attempt (and is retried) instead of blocking the run forever. The parent context still governs overall cancellation. 0 (default) means no per-attempt timeout.
type ToolLogOption ¶ added in v0.7.0
type ToolLogOption func(*toolLogConfig)
ToolLogOption configures ToolLog.
func LogErrorText ¶ added in v0.7.0
func LogErrorText() ToolLogOption
LogErrorText makes ToolLog log a failed call's full error text instead of its ErrorSummary. The text can carry content (the call's arguments, a provider's echo of them, a credential in a URL), so enable it only where the log may hold that content.
Source Files