Documentation
¶
Overview ¶
Package llmpool implements LLM failover ("轮询"): a Provider decorator that walks an ordered chain of LLM profiles and moves to the next one when the current one can't serve the request — out of credit, revoked key, rate-limited past the SDK's own retries, or down.
The chain is built by the server from llm_profiles (active profile first, then by priority), so the whole process shares ONE chain and ONE circuit-breaker registry: when a task discovers a profile is out of credit, every other task skips it immediately.
Index ¶
- Constants
- Variables
- type Member
- type Pool
- type Registry
- func (r *Registry) Get(id int64) State
- func (r *Registry) IsOpen(id int64) bool
- func (r *Registry) Pass(id int64)
- func (r *Registry) Reset(id int64)
- func (r *Registry) Restore(id int64, st State)
- func (r *Registry) SetPolicy(softTrip int, cooldown time.Duration)
- func (r *Registry) Snapshot() map[int64]State
- func (r *Registry) Trip(id int64, errMsg string, hard bool) (tripped bool)
- type State
Constants ¶
const RankActive = int(^uint(0)>>1) - 1
RankActive is the Rank the caller gives the chain head (the active profile, or an explicitly bound one) so it always outranks any configured priority.
Variables ¶
var ErrExhausted = errors.New("LLM 轮询：所有配置均不可用")
ErrExhausted is returned when every member of the chain failed.
Functions ¶
This section is empty.
Types ¶
type Member ¶
type Member struct {
ID int64 // llm_profiles.id
Name string // profile name, for logs / UI
Model string
Format string // "anthropic" | "openai"
Priority int // the profile's configured priority (display only)
Active bool // is_default (display only)
// Rank is the ordering key the caller assigned: higher goes first, and members
// sharing a Rank take turns leading (load-spreading across duplicate keys).
// The caller encodes "active profile heads the chain" as a Rank above every
// user-settable priority, so this type needs no policy of its own.
Rank int
// WindowTokens is the profile's context window in tokens. A member whose
// window can't hold the request is skipped rather than made to fail on it.
WindowTokens int
Prov llm.Provider
}
Member is one LLM profile in the chain, already built into a provider (wrapped with the recorder by the caller, so a failed attempt is still recorded under its own profile name).
type Pool ¶
type Pool struct {
// contains filtered or unexported fields
}
Pool is an llm.Provider that fails over across an ordered chain of members. It is safe for concurrent use: members are immutable after construction and all mutable state lives in the shared Registry.
func New ¶
New builds a Pool over members (already in chain order). Returns nil when the chain is empty. A single-member chain is still a valid Pool — it just behaves exactly like the bare provider.
func (*Pool) Complete ¶
func (p *Pool) Complete(ctx context.Context, req llm.CompletionRequest) (llm.Message, string, llm.Usage, error)
Complete implements llm.Provider with failover for non-streaming calls. A non-streaming request is atomic — it never delivers partial output — so every failover-eligible failure lands in the safe window and the next member can be tried without risk of duplicated output. Mirrors Stream's health-tripping and chain-exhaustion behavior.
func (*Pool) Stream ¶
func (p *Pool) Stream(ctx context.Context, req llm.CompletionRequest) iter.Seq2[llm.StreamEvent, error]
Stream implements llm.Provider with failover.
The one hard rule: a member may only be abandoned BEFORE it has yielded any event. Once text or a tool_use has reached the caller, re-sending the same request to another model would duplicate output and corrupt the conversation history — so a mid-stream failure is surfaced as-is and left to the agent harness's resume logic. Fortunately the failures this exists for (402 no credit, 401 bad key, 429, 5xx) all surface during request establishment, before the body is read, so they always land in the safe window.
type Registry ¶
type Registry struct {
// contains filtered or unexported fields
}
Registry holds the circuit-breaker state of every profile, keyed by profile id. It is process-wide and outlives any single Pool instance, so rebuilding the pool (saving an unrelated profile, toggling a setting) never clears what we've learned about which backends are broken.
func NewRegistry ¶
NewRegistry builds an empty registry. persist/forget may be nil.
func (*Registry) Pass ¶
Pass records a successful call: clears the failure counters so a profile that recovers is fully trusted again (and drops the persisted row).
func (*Registry) Restore ¶
Restore seeds state loaded from the DB at startup (bypasses persistence).
func (*Registry) SetPolicy ¶
SetPolicy overrides the breaker's two knobs. softTrip: how many consecutive transient failures trip it (0 = default softTripAfter; negative = transient failures never trip it, leaving only the deterministic ones). cooldown: a fixed cooling-off window (0 = the 1min/5min/30min ladder). Safe to call at any time.
func (*Registry) Trip ¶
Trip records a failed call. hard=true marks a deterministic failure (no credit, invalid key, missing model) which opens the breaker immediately; hard=false is a transient one (429 / 5xx / network) that needs softTripAfter in a row. Returns true when this call is what opened the breaker, so the caller can log it once.
Source Files