modelretry

package
v3.1.0-dev.3 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 3, 2026 License: Apache-2.0 Imports: 8 Imported by: 0

Documentation

Overview

Package modelretry waits out a provider that says "not now".

Why this exists (#239, then #452)

It was written for the judge eval tier. That tier's expensive failure is not a low score — a low score reports and does not gate — it is a row that never ran, which makes the board incomplete and the night red. On 2026-08-21 three of thirty-one rows were lost to `Error 429 ... RESOURCE_EXHAUSTED` from Vertex, and the tier made no attempt to get them back, so a provider-side quota blip presented as a broken measurement of mast.

The asymmetry that hid this is worth stating, because it is invisible from mast's side of the interface: anthropic-sdk-go retries 429s and 5xxs twice of its own accord, and google.golang.org/genai returns APIError{Code: 429} straight out of api_client.go with no retry at all. So the same corpus, on the same night, was measuring two different amounts of resilience depending on which model it named. Retrying at model.LLM — the one interface both providers arrive through — is what makes the two boards comparable.

The promotion (#452)

For eight releases this lived in internal/evals/judge and wrapped nothing but the eval harness's own models. That is the inversion #452 named: mast's *measurement* of mast was resilient to a 429 and the product was not. The measurement is mast's own and it is dated — three rows in thirty-one, through compose.BuildModel, against real Vertex — so the issue's "only if the evals say yes" gate was already answered by mast's own artifact and did not need a second study.

What moved is the mechanism. What did not move is the judge tier's reasoning about its own numbers: a nightly with a 90-minute budget and thirty-one sequential metered rows wants four attempts over half a minute, and an unattended turn does not. The two policies are ProductionConfig and JudgeConfig, side by side, so the divergence is a diff rather than a rediscovery.

This does not soften anything. An incomplete board still fails and a dispatch that ends short still reports short; the point is to stop paying for a provider blip with a result.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Config

type Config struct {
	// Backoff is the wait before each retry; its length is the retry
	// budget. Required — a nil Backoff is a policy that retries nothing,
	// which is a thing a caller may legitimately want and must therefore
	// not be silently replaced with a default.
	Backoff []time.Duration

	// Cooldown is the minimum gap between two retries served by this
	// policy, across every call it wraps. Zero disables it.
	Cooldown time.Duration

	// OnWait announces a wait before it is served. Optional, and the
	// judge tier passes one: a 27-second pause nobody narrated is
	// indistinguishable from a hung run in a nightly's progress log.
	OnWait func(model string, attempt int, wait time.Duration, err error)

	// Observer receives one [Event] per affected call. Optional; a host
	// with a metric registry sets it, a library embed leaves it nil and
	// reads [Policy.Stats] instead.
	Observer func(Event)

	// Sleep is time.Sleep in production and a recorder in tests. Nil
	// means the real one. It returns the context's error if the wait was
	// interrupted.
	Sleep func(ctx context.Context, d time.Duration) error

	// Now is time.Now in production and a clock in tests. Nil means the
	// real one.
	Now func() time.Time
}

Config builds a Policy.

func JudgeConfig

func JudgeConfig() Config

JudgeConfig is the eval tier's policy: four attempts total, spread over about half a minute, no cooldown.

Sized against what a Vertex per-region quota window actually is: quota refills on the order of a minute, so a sub-second retry storm would burn the budget inside one exhausted window and report the same failure faster. It is bounded rather than open-ended because a nightly with a 90-minute timeout and thirty-one sequential metered rows cannot afford to wait out a real outage — it should fail and say the board is short.

No cooldown, because the tier's rows are sequential and each one is a separate measurement: suppressing the second row's retry because the first row used one would lose a row to bookkeeping.

func ProductionConfig

func ProductionConfig() Config

ProductionConfig is what compose.BuildModel wraps a real provider model in: one retry, two seconds later, at most once a minute process-wide.

Every number differs from JudgeConfig, and each difference is the same fact read from the other side.

**One attempt, not four.** The evidence for a retry at all is that the next call a few seconds later succeeds; four attempts over half a minute is not more of that argument, it is a different bet. The judge tier takes it because a lost row costs a whole night and the tier is already measured in tens of minutes. An unattended turn is not: a workload on a fifteen-minute cadence can absorb two seconds and should not absorb thirty-nine, because a turn that waits that long has stopped being late and started being wedged.

**A cooldown, which the judge has none of.** The judge runs its rows one at a time, so "retry under pressure adds load" bounded itself. mast dispatches in parallel: a fan-out of eight specialists that all meet the same shed would, with no cooldown, answer it with eight retries two seconds later — precisely the objection. One retry per minute per process is the answer, and it is process-wide rather than per-model for the same reason: a specialist's `model:` override builds its own model.LLM, so per-instance state would give a roster of eight one cooldown each and no cooldown at all.

**Still no jitter.** The judge declined it because there was one serial caller and nothing to desynchronize. Under the cooldown there is at most one retry in flight per window, which is the same conclusion by the stronger route: jitter spreads a thundering herd, and the cooldown has already deleted it.

type Event

type Event struct {
	// Model is the wrapped model's name, so an operator can tell which
	// provider is shedding without reading the error.
	Model string

	// Outcome is one of the constants above.
	Outcome Outcome

	// Attempts is how many retries were served (0 when Declined).
	Attempts int

	// Waited is the total time spent sleeping before those retries.
	Waited time.Duration

	// Err is the provider's rejection — the last one, if there were
	// several. Non-nil even on Recovered, where it is the rejection that
	// was waited out.
	Err error
}

Event is one call's terminal retry accounting, emitted once, and only for a call that met a retryable rejection. A call that never saw one emits nothing — the denominator lives in mast_model_calls_total, and an event per successful call would bury the three that matter.

type Outcome

type Outcome string

Outcome is how one call's brush with a transient rejection ended. It is a metric label value; see docs/site/.../reference/metrics.md.

const (
	// Recovered: a retry was served and the call then produced content.
	// The provider's "not now" cost a wait instead of a result.
	Recovered Outcome = "recovered"

	// Exhausted: a retry was served and the call failed anyway, or the
	// schedule ran out. The caller got the provider's error.
	Exhausted Outcome = "exhausted"

	// Declined: the error qualified for a retry and the cooldown refused
	// one, because this process already spent a retry recently. The
	// caller got the provider's error without a wait.
	Declined Outcome = "declined"
)

type Policy

type Policy struct {
	// contains filtered or unexported fields
}

Policy is one retry policy and the counters it has accumulated. Safe for concurrent use; a shared policy is the point.

func New

func New(cfg Config) *Policy

New builds a policy from cfg.

func PolicyOf

func PolicyOf(m adkmodel.LLM) *Policy

PolicyOf reports the policy driving m, or nil if m is not wrapped.

Exists so a caller can assert that a model it did not construct is resilient. "Is this path retrying?" is otherwise only answerable by pointing a real provider at a real quota, which is the check nobody runs — and an unwired retry is indistinguishable from a wired one until the night it was needed.

func Shared

func Shared() *Policy

Shared is the process-wide production policy, the one compose.BuildModel wraps every real provider model in.

A package-level singleton, which is not a shape this codebase reaches for lightly. It is the right one here because the cooldown is the part that has to be shared: a cooldown scoped to anything smaller than the process does not bound what the process sends, which is the only thing a provider shedding load can observe.

func (*Policy) SetObserver

func (p *Policy) SetObserver(fn func(Event))

SetObserver installs the terminal-event hook, replacing any previous one. Last writer wins, and the expected caller is the host at startup — cmd/mast, once, when it knows its workload name and has a registry to count into.

func (*Policy) Stats

func (p *Policy) Stats() (retries, declined int, waited time.Duration)

Stats reports what this policy has spent: retries served, retries the cooldown declined, and total time waited.

Reported, not swallowed. A retry nobody can see turns a provider under sustained pressure into a green board, which is the failure where a measurement quietly stops measuring.

func (*Policy) Wrap

func (p *Policy) Wrap(m adkmodel.LLM) adkmodel.LLM

Wrap returns m driven under this policy.

It returns model.LLM rather than *retryingLLM on purpose. A nil model has to come back as a nil *interface*, so that a constructor's `m == nil` guard still fires — returning a typed nil pointer would move the failure to a dereference with no context attached.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL