retry

package
v0.15.3 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 14, 2026 License: Apache-2.0 Imports: 15 Imported by: 0

Documentation

Overview

Package retry retries model calls at the HTTP layer, where a provider's rate-limit headers are still visible.

Why the HTTP layer

mast's modelretry and core-agent's RetryPolicy both retry at model.LLM, the one interface every provider arrives through. That level cannot see Retry-After: genai's APIError carries no response, so on the Gemini path the header is gone before a model.LLM decorator runs (mast's modelretry documents this). Each vendor SDK also has its own opinion — anthropic-sdk-go retries twice, genai never — so the same workload got different resilience depending on which model it named.

core-models' adapters therefore turn their SDK's own retries off and install Transport on the HTTP client instead. One policy, one header rule, every provider. A product's model.LLM-level policy stays where it is and now sees only what this layer gave up on (docs/design.md Q7).

What is retried

A response whose status is 408, 429, 500, 502, 503, 504 or 529 (Anthropic's "overloaded"), and a request that failed before any response arrived (a refused or dropped connection, a timeout, a DNS failure — anything but a certificate error), unless the server sent x-should-retry: false (OpenAI and Anthropic send it). x-should-retry: true makes any status retryable. A canceled context is never retried.

Only what happens before the response body is handed back can be retried: a stream that breaks halfway through has already delivered tokens to the caller, and replaying the request would deliver them twice. That failure surfaces to the adapter as an error, as it should.

How long it waits

retry-after-ms if present (OpenAI), else retry-after as seconds or an HTTP date, else exponential backoff with jitter. A server that asks for longer than Policy.MaxHeaderDelay gets its response handed back rather than a shorter wait: retrying sooner than asked earns another 429 and spends budget, and a wait that long is the product's call, not a transport's.

The approach — prefer retry-after-ms, accept an HTTP date, bound the delay — follows charmbracelet/fantasy's retry.go (mast model-support §10.2); no code is taken from it.

Index

Constants

View Source
const MetadataKey = "core_models.retry"

MetadataKey is the llm.Response.CustomMetadata key an adapter stamps a non-empty Snapshot under.

Variables

View Source
var ErrNoResponse = errors.New("retry: no response headers within the header timeout")

ErrNoResponse is the error of an attempt that received no response headers within its header timeout (Policy.HeaderTimeout, UnaryHeaderTimeout, or a WithHeaderTimeout override). It does not wrap context.Canceled: the attempt was abandoned, not the call.

Functions

func MarkUnary added in v0.6.4

func MarkUnary(ctx context.Context) context.Context

MarkUnary marks ctx as carrying a non-streaming request, so the transport bounds its wait for headers with UnaryHeaderTimeout. An adapter calls it on the context it sends a non-streaming request with.

func WithHeaderTimeout added in v0.6.4

func WithHeaderTimeout(ctx context.Context, d time.Duration) context.Context

WithHeaderTimeout overrides the header timeout for requests sent under ctx, streaming or not; d <= 0 removes the limit. For a host call it knows to be long, such as a compaction summary over a large context.

Types

type GaveUp

type GaveUp string

GaveUp says why a retried request stopped retrying with a failure still in hand.

const (
	// Budget means MaxRetries was spent.
	Budget GaveUp = "budget"
	// HeaderDelay means the server asked to wait longer than
	// MaxHeaderDelay.
	HeaderDelay GaveUp = "header_delay"
	// NotReplayable means the request body could not be sent again.
	NotReplayable GaveUp = "not_replayable"
	// Canceled means the context ended during a wait.
	Canceled GaveUp = "canceled"
)

type Policy

type Policy struct {
	// MaxRetries is how many times one request may be re-sent. Zero
	// disables retrying.
	MaxRetries int
	// InitialBackoff is the first wait when the server gives no header;
	// each later wait multiplies it by BackoffFactor.
	InitialBackoff time.Duration
	BackoffFactor  float64
	// MaxBackoff caps a computed backoff.
	MaxBackoff time.Duration
	// MaxHeaderDelay is the longest server-requested wait honored. A
	// longer request ends retrying and returns the response.
	MaxHeaderDelay time.Duration
	// HeaderTimeout bounds how long one attempt may wait for response
	// headers. Expiry abandons the attempt and counts as a transport
	// failure, so it is retried like a dropped connection. It never
	// limits the body: a stream that has sent its headers may run as long
	// as it generates. A non-streaming request receives headers only once
	// the whole answer exists, so set it above the slowest complete
	// answer you expect. Zero means no limit.
	//
	// It applies to streaming requests and to requests not marked
	// either way; a request marked non-streaming (MarkUnary) uses
	// UnaryHeaderTimeout instead, and WithHeaderTimeout overrides both
	// for one call.
	HeaderTimeout time.Duration
	// UnaryHeaderTimeout is HeaderTimeout for a request an adapter
	// marked non-streaming (MarkUnary). Such a request receives headers
	// only once the whole answer exists, so this bounds total generation
	// time and is longer than HeaderTimeout. Zero means no limit.
	UnaryHeaderTimeout time.Duration
	// Jitter is the fraction (0–1) of a computed backoff that is
	// randomized, so callers that failed together do not retry
	// together. Header-requested delays are never jittered.
	Jitter float64

	// AfterSuccess retries a response the rules above would hand back,
	// once per request, when the call's session has already been served
	// (callctx.PriorCallSucceeded) and AfterSuccess says so. It is for
	// a rejection that is ambiguous on its own: Vertex's bare 400
	// INVALID_ARGUMENT names nothing, so on a first call it is as likely
	// a malformed request as a transient fault, but on a session whose
	// previous call under the same config succeeded it has been
	// transient every time it was recorded (core-agent #898, #1247).
	// It is consulted only for a 4xx or 5xx; body is that response's
	// body, at most 64 KiB, and stays readable by the caller when the
	// response is handed back. Nil disables it.
	AfterSuccess func(status int, body []byte) bool

	// Sleep waits for d or until ctx is done. Nil means a real timer;
	// tests substitute it.
	Sleep func(ctx context.Context, d time.Duration) error
	// Now is the clock used to resolve an HTTP-date Retry-After. Nil
	// means time.Now.
	Now func() time.Time
	// Rand returns a float in [0, 1) for jitter. Nil means math/rand.
	Rand func() float64
}

Policy configures Transport. The zero value is not useful; start from Default.

func Default

func Default() Policy

Default is the policy adapters install unless configured otherwise: two retries, starting at one second.

func (Policy) Transport

func (p Policy) Transport(base http.RoundTripper) http.RoundTripper

Transport returns an http.RoundTripper that retries base's retryable failures under p. A nil base means http.DefaultTransport.

type Record

type Record struct {
	// contains filtered or unexported fields
}

Record is what Transport did for the requests made under one call's context. An adapter puts one on the context with WithRecord and, when the call ends, stamps a non-empty Snapshot onto the response, so a retry is visible in the transcript and not only in a log.

One model call can make several HTTP requests (a pause_turn continuation, for instance); a Record accumulates across them. Safe for concurrent use.

func WithRecord

func WithRecord(ctx context.Context) (context.Context, *Record)

WithRecord returns a context carrying a fresh Record, and the Record.

func (*Record) Snapshot

func (r *Record) Snapshot() Snapshot

Snapshot returns the record's current contents. Nil-safe.

type Snapshot

type Snapshot struct {
	Retries    int    `json:"retries,omitempty"`
	WaitedMS   int64  `json:"waited_ms,omitempty"`
	LastStatus int    `json:"last_status,omitempty"`
	GaveUp     GaveUp `json:"gave_up,omitempty"`
}

Snapshot is a Record's contents at one moment, in the shape it is persisted. Zero means no request was retried.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL