adaptive

package
v0.7.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 5, 2026 License: Apache-2.0 Imports: 10 Imported by: 0

Documentation

Overview

Package adaptive is the adaptive model router: it picks a model for each call from the configured pool for the caller's tier (high, low, frontier) by sampling a score from each candidate's recorded reliability, speed, price and current load, and it learns from every registered outcome (latency, throughput, rate limits and other failures, each with its own cooldown).

A tier whose pool is empty resolves to the high pool, so a run configured with nothing but a high pool routes every tier on it.

A single router-wide mutex serialises every public method; Pick takes and releases it around each TryPick and never holds it across a sleep.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func IsLikelyProviderIncompatible 

func IsLikelyProviderIncompatible(err error) bool

func IsLikelyRateLimit 

func IsLikelyRateLimit(err error) bool

func IsLikelyStructuredFailure 

func IsLikelyStructuredFailure(err error) bool

func IsLikelyTimeout 

func IsLikelyTimeout(err error) bool

func IsLikelyTransientProviderError 

func IsLikelyTransientProviderError(err error) bool

IsLikelyTransientProviderError covers HTTP 5xx and generic upstream provider errors that are neither rate limits nor schema issues.

func IsRetryableRouteError 

func IsRetryableRouteError(err error) bool

func NormalizeCandidateModel 

func NormalizeCandidateModel(model string) string

NormalizeCandidateModel trims a model id; the "openrouter/" prefix stays.

func SetClockForTesting 

func SetClockForTesting(f func() float64) func()

SetClockForTesting pins the clock. Returns a restore func.

func SetRandomForTesting 

func SetRandomForTesting(f func() float64) func()

SetRandomForTesting pins the unseeded random source. Returns a restore func.

func SetSleeperForTesting 

func SetSleeperForTesting(f func(ms float64, signal *AbortSignal)) func()

SetSleeperForTesting pins the pick() backoff sleep. Returns a restore func.

Types

type AbortSignal 

type AbortSignal struct {
	// contains filtered or unexported fields
}

AbortSignal lets a caller interrupt a blocking Pick.

func NewAbortSignal 

func NewAbortSignal() *AbortSignal

func (*AbortSignal) Abort 

func (s *AbortSignal) Abort()

func (*AbortSignal) Aborted 

func (s *AbortSignal) Aborted() bool

func (*AbortSignal) Done 

func (s *AbortSignal) Done() <-chan struct{}

type AdaptiveModelRouter 

type AdaptiveModelRouter struct {
	// contains filtered or unexported fields
}

func NewAdaptiveModelRouter 

func NewAdaptiveModelRouter(cfg AdaptiveRouterConfig) *AdaptiveModelRouter

func (*AdaptiveModelRouter) CandidatesForTier 

func (r *AdaptiveModelRouter) CandidatesForTier(tier ModelTier) []ModelCandidate

CandidatesForTier is the pool the tier routes on, after degradation.

func (*AdaptiveModelRouter) EffectiveTier 

func (r *AdaptiveModelRouter) EffectiveTier(tier ModelTier) ModelTier

EffectiveTier resolves the tier a caller should actually route on: a tier whose pool is empty degrades to HIGH, so no run depends on LOW or FRONTIER being configured, and an unrecognised tier routes on HIGH.

func (*AdaptiveModelRouter) MaxAttempts 

func (r *AdaptiveModelRouter) MaxAttempts() float64

MaxAttempts is the configured attempt budget per call.

func (*AdaptiveModelRouter) Pick 

func (r *AdaptiveModelRouter) Pick(
	slot string, tier ModelTier, opts ...PickOptions,
) (RouteChoice, error)

Pick blocks (polling TryPick with bounded exponential backoff) until a candidate is available or `timeoutMs` (default 5 minutes) elapses.

The caller must settle the returned lease exactly once, with Register or RegisterCanceled.

func (*AdaptiveModelRouter) PickContext 

func (r *AdaptiveModelRouter) PickContext(
	ctx context.Context, slot string, tier ModelTier,
) (RouteChoice, error)

PickContext binds waiting for a route to the caller's actual lifetime. A cancellation before a lease is returned must not leak an in-flight slot.

func (*AdaptiveModelRouter) PickSync 

func (r *AdaptiveModelRouter) PickSync(slot string, tier ModelTier) RouteChoice

PickSync always returns a choice: when every candidate is cooling it takes the least bad one from the pool and says so.

func (*AdaptiveModelRouter) Register 

func (r *AdaptiveModelRouter) Register(choice RouteChoice, elapsedSeconds float64, completionTokens float64, err error) AdaptiveRouteEvent

Register records the outcome of a call started by Pick. Pass err == nil on success.

func (*AdaptiveModelRouter) RegisterCanceled 

func (r *AdaptiveModelRouter) RegisterCanceled(choice RouteChoice)

RegisterCanceled releases a caller-canceled lease without attributing a success, failure, latency sample or cooldown to the provider. The owner must serialize this with Register: exactly one terminal accounting action per pick.

func (*AdaptiveModelRouter) TryPick 

func (r *AdaptiveModelRouter) TryPick(slot string, tier ModelTier, nowMs ...float64) TryPickResult

TryPick is the non-blocking pick. Pass at most one nowMs to override the clock.

type AdaptiveRouteEvent 

type AdaptiveRouteEvent struct {
	Slot          string    `json:"slot"`
	Tier          ModelTier `json:"tier"`
	Model         string    `json:"model"`
	PreviousModel string    `json:"previous_model"`
	Switched      bool      `json:"switched"`
	Reason        string    `json:"reason"`
	Score         float64   `json:"score"`
	ElapsedS      float64   `json:"elapsed_s"`
	Attempts      float64   `json:"attempts"`
	Successes     float64   `json:"successes"`
	Failures      float64   `json:"failures"`
	RateLimits    float64   `json:"rate_limits"`
	LatencyEwma   float64   `json:"latency_ewma"`
	ToksecEwma    float64   `json:"toksec_ewma"`
	Error         string    `json:"error"`
}

AdaptiveRouteEvent describes one registered outcome or cancellation.

type AdaptiveRouterConfig 

type AdaptiveRouterConfig struct {
	HighModels []ModelCandidate `json:"high_models"`
	// LowModels and FrontierModels have no default: an unset pool stays empty
	// and every caller asking for it routes on HIGH instead.
	LowModels      []ModelCandidate `json:"low_models"`
	FrontierModels []ModelCandidate `json:"frontier_models"`
	MaxAttempts    *float64         `json:"max_attempts"`
	// RandomSeed makes the router's sampling reproducible. Nil uses the
	// process-wide random source.
	RandomSeed *float64                 `json:"random_seed"`
	OnEvent    func(AdaptiveRouteEvent) `json:"-"`
}

AdaptiveRouterConfig configures a router. A nil pointer field takes its default, and so does an empty high pool; an empty low or frontier pool stays empty and degrades to high instead.

type Detailer 

type Detailer interface {
	ErrorDetail() string
}

Detailer is implemented by errors that carry extra text the classifiers should search, such as a provider's raw error body.

type ModelCandidate 

type ModelCandidate struct {
	// ID is the full model id, e.g. "openrouter/qwen/qwen3.6-plus".
	ID string `json:"id"`
	// Tier is the pool the candidate was configured into.
	Tier                 ModelTier `json:"tier"`
	PromptUSDPerMtok     float64   `json:"prompt_usd_per_mtok"`
	CompletionUSDPerMtok float64   `json:"completion_usd_per_mtok"`
	// Priority is the config index: lower = preferred when stats are absent.
	Priority float64 `json:"priority"`
}

ModelCandidate is one routable model.

func DefaultHighModels 

func DefaultHighModels() []ModelCandidate

func ParseModelList 

func ParseModelList(raw *string, tier ModelTier) []ModelCandidate

ParseModelList parses a comma-separated `--high`, `--low` or `--frontier` flag value into the named tier's pool. Each entry is either "openrouter/qwen/qwen3.6-plus" or "openrouter/qwen/qwen3.6-plus@0.325/1.95" (id + prompt$/completion$ per Mtok). Cost defaults to zero, which degenerates the cost term to a neutral 0.5.

type ModelTier 

type ModelTier string

ModelTier names one of the three configurable model pools. A caller asks for a tier; EffectiveTier turns that request into the tier actually routed on, which is the high tier whenever the requested pool is empty. 

const (
	ModelTierHigh     ModelTier = "high"
	ModelTierLow      ModelTier = "low"
	ModelTierFrontier ModelTier = "frontier"
)

type Namer 

type Namer interface {
	ErrorName() string
}

Namer is implemented by errors with a symbolic name; "AbortError" marks a timeout regardless of message text.

type PickOptions 

type PickOptions struct {
	TimeoutMs *float64
	Signal    *AbortSignal
}

PickOptions bounds a blocking Pick.

type RouteChoice 

type RouteChoice struct {
	Slot string `json:"slot"`
	// Tier is the tier actually routed on, after degradation.
	Tier          ModelTier      `json:"tier"`
	Candidate     ModelCandidate `json:"candidate"`
	Score         float64        `json:"score"`
	PreviousModel string         `json:"previous_model"`
	Switched      bool           `json:"switched"`
	Reason        string         `json:"reason"`
}

RouteChoice is a picked candidate together with why it was picked.

type StatusCoder 

type StatusCoder interface {
	ErrorStatusCode() (float64, bool)
}

StatusCoder is implemented by errors that carry an HTTP status.

type TryPickResult 

type TryPickResult struct {
	Ok           bool        `json:"ok"`
	Choice       RouteChoice `json:"choice,omitzero"`
	Reason       string      `json:"reason,omitempty"`
	RetryAfterMs float64     `json:"retryAfterMs,omitempty"`
}

TryPickResult is the non-blocking pick outcome. Ok==false means every candidate in the pool is cooling.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.