middleware

func Cost ¶

func Cost(m *CostMeter, r Rates) agent.Middleware

Cost is a model middleware that records token usage and computes running cost after each successful model call. The caller reads accumulated values via m.

func Hedge ¶

func Hedge(delay time.Duration, backups ...agent.Model) agent.Middleware

Hedge races the model call against one or more backup models and returns the FIRST successful response, cancelling the rest. It cuts tail latency (a slow primary no longer blocks the turn, you take whichever target answers first) and adds provider-outage failover (a down or throttled primary does not stall the run). It is safe here in a way it is not in a bare service: the agent loop journals only the winning response, at most once, and the losing calls are cancelled before they can touch state, so the durable record stays exactly-once and replays deterministically regardless of which target won the race.

Scheduling:

• The primary (the wrapped model, i.e. the one passed to agent.New) fires immediately.
• If delay > 0, the backups fire only after the primary has been outstanding for delay, so you pay for a backup generation only when the primary is actually slow. A delay of 0 fires every target at once (pure redundancy, always pays for all of them).
• If a target returns an ERROR while backups are still pending, the backups fire immediately (fast failover), rather than waiting out the remaining delay on a fast failure.

The first target to return a nil-error result wins; every other in-flight call is cancelled via a derived context. If every target fails, Hedge returns the joined error. With no backups it is a pass-through, so it is safe to wire unconditionally and add backups later.

Streaming: with backups, a streaming caller (Agent.Stream) receives the winning response's events once the winner is chosen, not token by token. The targets race without access to the caller's stream, so a target that loses can never show the caller text the run does not record, or send anything after the run has ended. With no backups, streaming is live as usual.

Hedge only races the model generation. It does not duplicate tool calls or any other side effect: those run in the agent loop, above this middleware, under the at-most-once journal and the Safety layer. Note it hedges latency and availability, not correctness: taking the first answer is not the same as requiring agreement across models (that is a governance-quorum concern, not this).

// Primary is Anthropic; if it is quiet for 800ms, also try OpenAI and take the first.
a := agent.New(anthropicModel, store, tools...).
    Use(middleware.Hedge(800*time.Millisecond, openaiModel))

func RateLimit ¶

func RateLimit(r *RateLimiter) agent.Middleware

RateLimit throttles model calls through the shared limiter, blocking (respecting context cancellation) until a token is available. Proactive throttling complements Retry's reactive Retry-After backoff: it keeps you under the provider's ceiling instead of bouncing off it.

func Retry ¶

func Retry(n int, opts ...RetryOption) agent.Middleware

Retry retries the model call up to n additional times on error, respecting context cancellation. Between attempts it sleeps with exponential backoff (full jitter, base×2^attempt, capped at max). When the error is *agent.RateLimited it sleeps for min(RetryAfter, max) instead.

Default backoff: base=200ms, max=10s. Override with WithBackoff.

func Retryable ¶

func Retryable(err error) bool

Retryable is a ready-made classifier for WithRetryIf: it retries transient failures and fails fast on terminal ones. It retries *agent.RateLimited, a per-attempt timeout (context.DeadlineExceeded), and *agent.APIError with a 5xx or 408 status; it does not retry parent cancellation (context.Canceled) or 4xx API errors (auth, validation). Errors it cannot classify (e.g. raw network errors) are retried, since those are usually transient.

agent.New(model, store, tools...).Use(middleware.Retry(3, middleware.WithRetryIf(middleware.Retryable)))

func ToolCache ¶

func ToolCache() agent.ToolMiddleware

ToolCache memoizes successful tool results by (name, args) in a process-local map and short-circuits a repeat call — the tool never runs on a hit. It demonstrates the short-circuit power of tool middleware.

It caches only tools marked ReadOnly (agent.Safety); every other tool call runs, since two calls with the same arguments to a side effect are two effects, such as two separate charges. Even for a ReadOnly tool a hit returns the earlier result, so use it only where the output depends solely on the arguments. Errors are never cached. This is an in-memory, unbounded, per-instance cache — it is NOT the durable journal (which already dedupes each tool-use ID at-most-once); it dedupes DISTINCT calls with identical arguments within one process.

func ToolLog ¶

func ToolLog(logf func(format string, args ...any)) agent.ToolMiddleware

ToolLog logs one line per tool call — name, duration, and ok/error — via logf (pass log.Printf, t.Logf, or a structured logger's Printf-shaped method). Attach with agent.Agent.UseTool.

a := agent.New(model, store, tools...).UseTool(middleware.ToolLog(log.Printf))

func ToolRateLimit ¶

func ToolRateLimit(r *RateLimiter) agent.ToolMiddleware

ToolRateLimit throttles tool calls through the shared limiter.

func ToolRetry ¶

func ToolRetry(n int, opts ...RetryOption) agent.ToolMiddleware

ToolRetry retries a failing tool call up to n additional times with exponential backoff + jitter (honoring an *agent.RateLimited RetryAfter), respecting context cancellation. Reuses RetryOption/WithBackoff from the model-side Retry.

It retries only a tool that is retry-safe (agent.Safety.RetrySafe: ReadOnly, Idempotent, or keyed). A tool that is not runs once, and its error is returned as is: a failed call to a side effect may still have taken effect, as when a payment gateway times out after charging, and running it again could repeat it.

Default backoff: base=200ms, max=10s. Override with WithBackoff.

a := agent.New(model, store, tools...).UseTool(middleware.ToolRetry(3))