llm

package
v0.51.7 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 14, 2026 License: Apache-2.0 Imports: 18 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var ErrDecisionTooLarge = errors.New("decision input exceeds max_input_tokens")

ErrDecisionTooLarge is returned when the estimated input exceeds the decider's max_input_tokens. The input is never truncated: a truncated view can hide the payload that matters.

View Source 
var ErrHardLimitExceeded = errors.New("session hard cost limit exceeded")

ErrHardLimitExceeded is returned when the session's hard cost limit is reached.

View Source 
var ErrNoDecisionProvider = errors.New("provider does not serve decisions")

ErrNoDecisionProvider is returned when a live decider's provider instance is not registered or does not serve decisions. The call is never made.

View Source 
var ErrSoftLimitExceeded = errors.New("session soft cost limit exceeded")

ErrSoftLimitExceeded is returned when the session's soft cost limit is reached.

View Source 
var ErrStreamIdleTimeout = errors.New("LLM stream idle timeout: no data received")

ErrStreamIdleTimeout is returned when the LLM SSE stream has not produced any data for longer than the configured idle timeout.

View Source 
var ErrStreamTruncated = errors.New("stream ended before completion (no finish_reason)")

ErrStreamTruncated indicates an SSE stream ended (EOF) before delivering a terminal chunk (finish_reason). The response is partial and must not be treated as a completed turn.

Functions

func DecisionErrorCause added in v0.51.0 

func DecisionErrorCause(err error) string

DecisionErrorCause classifies a failed decision call for audit trails. A call refused for budget, one that timed out, one whose input was too large and one whose provider is down all fall through to the next stage, so the distinction has to survive somewhere filterable.

func EstimateDecisionTokens added in v0.48.0 

func EstimateDecisionTokens(state any, qs map[string]Question) (int, error)

EstimateDecisionTokens estimates the input size of a decision call at one token per 3 bytes of JSON: conservative for punctuation-heavy JSON, so an estimate under the limit is unlikely to be rejected upstream.

func HTTPStatusForError added in v0.33.0 

func HTTPStatusForError(err error) int

HTTPStatusForError returns the HTTP status code that best represents the error to an API client. Provider-side failures (auth, credits, model not found, server errors) map to 502 Bad Gateway; rate limits pass through as 429; cancellations and timeouts get their own codes.

func IsRateLimit added in v0.38.0 

func IsRateLimit(err error) bool

IsRateLimit returns true if err is specifically a 429 rate-limit error.

func IsRetryable added in v0.38.0 

func IsRetryable(err error) bool

IsRetryable returns true if err is worth retrying. Stream idle timeouts are retryable (the provider stalled; a fallback can succeed). Context cancellation/deadline errors are not — the caller's context is dead, so any retry against it fails instantly. LLMErrors are retryable based on their status code; other network/unknown errors are assumed retryable.

For the distinct question of whether an error implicates the upstream provider's health, use IsUpstreamError.

func IsUpstreamError added in v0.42.2 

func IsUpstreamError(err error) bool

IsUpstreamError reports whether err implicates the upstream provider's health — 429/5xx server errors, network faults, and stream stalls or truncation — as opposed to the caller or the request itself (context cancellation, 4xx client errors). Provider-internal state such as sticky-routing preferences should only be discarded when this is true.

It currently shares IsRetryable's truth table by delegation: every error worth retrying today is also one that implicates the upstream. The two questions are distinct (e.g. a 408 could be retryable without indicting upstream health); when they diverge, inline the classification here and update the truth-table tests — see issue #200.

func LeakedToolCallText added in v0.47.2 

func LeakedToolCallText(content string) bool

LeakedToolCallText reports whether content carries a tool call rendered as plain text instead of a structured tool_calls entry.

func LooksLikeLeakedToolCall added in v0.47.2 

func LooksLikeLeakedToolCall(resp *ChatResponse) bool

LooksLikeLeakedToolCall reports whether resp is a "final" answer that is really an unparsed tool call: an upstream fault worth one retry, not model output.

func TokenCost added in v0.15.0 

func TokenCost(resp *ChatResponse, reg *pricing.Registry, providerName string) (float64, string)

TokenCost returns the USD cost for a chat response using the pricing registry. providerName selects the per-provider override layer (empty string skips it). Priority: provider-reported cost > registry lookup > fallback rate > $0. Also returns a pricing source string ("provider"/"registry"/"fallback"/"unknown").

func UserFacingError added in v0.33.0 

func UserFacingError(err error) string

UserFacingError translates an LLM pipeline error into a short, actionable message for end users. Unknown errors get a generic fallback.

func ValidateQuestions added in v0.48.0 

func ValidateQuestions(qs map[string]Question) error

ValidateQuestions checks question shapes against the decisions API limits so a malformed question fails before any call is made.

Types

type AgentStats added in v0.12.0 

type AgentStats struct {
	Agent        string  `json:"agent"`
	Cost         float64 `json:"cost"`
	InputTokens  int     `json:"input_tokens"`
	OutputTokens int     `json:"output_tokens"`
	CachedTokens int     `json:"cached_tokens"`
	Messages     int     `json:"messages"`
	Sessions     int     `json:"sessions"`
}

AgentStats holds aggregated per-agent cost and token data.

type Answer added in v0.48.0 

type Answer struct {
	Type          QuestionType       `json:"type"`
	Choice        string             `json:"choice,omitempty"`
	Probabilities map[string]float64 `json:"probabilities,omitempty"`
	Confidence    float64            `json:"confidence,omitempty"`
	Noul          float64            `json:"noul,omitempty"`
	Score         float64            `json:"score,omitempty"`
}

Answer is a decision model's answer to one Question. Which fields are set depends on Type: Choice/Probabilities/Confidence for choice, Noul for noul, Score/Probabilities/Confidence for score (probabilities keyed by level index).

type ChatRequest 

type ChatRequest struct {
	Model             string
	Messages          []Message
	MaxTokens         int
	Temperature       *float64
	Tools             []ToolDef
	OnStream          StreamCallback // if non-nil, provider streams chunks via this callback
	StreamIdleTimeout time.Duration  // if > 0, cancel the SSE stream if no data arrives within this duration
}

type ChatResponse 

type ChatResponse struct {
	Content         string
	ThinkingContent string // accumulated thinking/reasoning content (if model supports it)
	ToolCalls       []ToolCall
	TokensUsed      TokenUsage
	Model           string
	FinishReason    string
	CostUSD         float64 // provider-reported or estimated cost in USD
	// BilledUSD is what the router recorded for this response after pricing
	// lookup. CostUSD stays provider-reported (0 for providers that report
	// none). A leaked attempt the router retried away is not included.
	BilledUSD float64
	// Upstream is the provider-reported serving upstream (OpenRouter's routed
	// provider). Empty when the provider has no such concept.
	Upstream string
	// LeakRetry is set by the router when this completion was re-issued
	// because the first attempt returned a tool call as plain text (see
	// LooksLikeLeakedToolCall). It is set whether or not the retry recovered,
	// so the audit trail records that the upstream misbehaved.
	LeakRetry bool
}

type CostTracker 

type CostTracker struct {
	// contains filtered or unexported fields
}

CostTracker tracks token usage and estimated costs per session and globally.

func NewCostTracker 

func NewCostTracker(defaults SessionLimits, agentOverrides map[string]SessionLimits) *CostTracker

NewCostTracker creates a CostTracker with default limits and optional per-agent overrides.

func (*CostTracker) AgentCosts added in v0.12.0 

func (ct *CostTracker) AgentCosts() []AgentStats

AgentCosts returns per-agent aggregated stats from the in-memory tracker.

func (*CostTracker) AllSessionCosts 

func (ct *CostTracker) AllSessionCosts() map[string]float64

AllSessionCosts returns a copy of all session costs.

func (*CostTracker) AllSessionStats added in v0.12.0 

func (ct *CostTracker) AllSessionStats() map[string]SessionStats

AllSessionStats returns a copy of all session stats.

func (*CostTracker) DefaultLimits added in v0.16.5 

func (ct *CostTracker) DefaultLimits() SessionLimits

DefaultLimits returns the default cost limits.

func (*CostTracker) ExceedsBudget 

func (ct *CostTracker) ExceedsBudget(sessionID string) bool

ExceedsBudget checks if a session has exceeded its hard budget. Deprecated: use ExceedsHardLimit.

func (*CostTracker) ExceedsHardLimit added in v0.16.5 

func (ct *CostTracker) ExceedsHardLimit(sessionID string) bool

ExceedsHardLimit checks if a session has exceeded its hard cost limit. Returns false if the hard limit is disabled (zero).

func (*CostTracker) ExceedsSoftLimit added in v0.16.5 

func (ct *CostTracker) ExceedsSoftLimit(sessionID string) bool

ExceedsSoftLimit checks if a session has exceeded its soft cost limit. Returns false if the soft limit is disabled (zero).

func (*CostTracker) GlobalCost 

func (ct *CostTracker) GlobalCost() float64

GlobalCost returns the total cost across all sessions.

func (*CostTracker) MaxBudgetPerSession 

func (ct *CostTracker) MaxBudgetPerSession() float64

MaxBudgetPerSession returns the default hard cost cap. Deprecated: use DefaultLimits().Hard.

func (*CostTracker) ProviderLimits added in v0.34.0 

func (ct *CostTracker) ProviderLimits(provider string) SessionLimits

ProviderLimits returns the cost limits for a specific provider.

func (*CostTracker) Record 

func (ct *CostTracker) Record(sessionID string, cost float64) bool

Record adds cost for a session. Returns true if within hard budget.

func (*CostTracker) RecordWithProvider added in v0.32.0 

func (ct *CostTracker) RecordWithProvider(sessionID, provider string, cost float64, usage TokenUsage, pricingSource string) bool

RecordWithProvider adds cost, token usage, and provider attribution for a session. Returns true if within hard budget. usage carries the full token breakdown; its CachedPrompt is the cache-read portion of the prompt, reported separately by providers (never included in Prompt). Taking TokenUsage whole (rather than positional counts) means new token fields are threaded through without a signature change on every call site.

func (*CostTracker) RecordWithTokens added in v0.12.0 

func (ct *CostTracker) RecordWithTokens(sessionID string, cost float64, inputTokens, outputTokens int, pricingSource ...string) bool

RecordWithTokens adds cost and token usage for a session. Returns true if within budget. The optional pricingSource parameter records which pricing method was used. Deprecated: use RecordWithProvider for correct per-provider limit enforcement.

func (*CostTracker) RegisterSessionAgent added in v0.30.1 

func (ct *CostTracker) RegisterSessionAgent(sessionID, agent string)

RegisterSessionAgent associates a session ID with an agent name for accurate per-agent cost attribution and limit enforcement. This is needed for channel-based session IDs (e.g. "chan:channelname") where the agent name cannot be parsed from the ID format.

func (*CostTracker) SessionCost 

func (ct *CostTracker) SessionCost(sessionID string) float64

SessionCost returns the total cost for a session.

func (*CostTracker) SetAgentLimits added in v0.16.5 

func (ct *CostTracker) SetAgentLimits(agent string, limits SessionLimits)

SetAgentLimits sets per-agent cost limit overrides.

func (*CostTracker) SetDefaultLimits added in v0.29.10 

func (ct *CostTracker) SetDefaultLimits(limits SessionLimits)

SetDefaultLimits updates the default cost limits applied to sessions without agent-specific overrides.

func (*CostTracker) SetProviderLimits added in v0.32.0 

func (ct *CostTracker) SetProviderLimits(provider string, limits SessionLimits)

SetProviderLimits sets per-provider cost limit overrides.

type CredentialChecker added in v0.50.0 

type CredentialChecker interface {
	CheckCredentials(ctx context.Context) error
}

CredentialChecker is implemented by providers whose model listing does not prove the key is valid (OpenRouter's /models is public). CheckCredentials returns an *LLMError carrying the upstream status when the key is refused.

type Decider added in v0.48.0 

type Decider struct {
	// contains filtered or unexported fields
}

Decider is the decision-side counterpart of Router: it binds a DecisionProvider, a model and the cost tracker, and enforces the size, budget and timeout guards around each call.

func NewDecider added in v0.48.0 

func NewDecider(cfg DeciderConfig, provider DecisionProvider, costs *CostTracker) *Decider

NewDecider creates a Decider over a fixed provider client. costs may be nil (no budget guard or recording).

func NewLiveDecider added in v0.51.0 

func NewLiveDecider(cfg DeciderConfig, providers *ProviderSet, costs *CostTracker) *Decider

NewLiveDecider creates a Decider that looks cfg.Provider up in providers on every call, so a provider replaced at runtime (a rotated API key) is used from the next call on. costs may be nil.

func (*Decider) CostTracker added in v0.49.0 

func (d *Decider) CostTracker() *CostTracker

CostTracker returns the tracker calls are billed to (may be nil).

func (*Decider) Decide added in v0.48.0 

func (d *Decider) Decide(ctx context.Context, sessionID string, state any, qs map[string]Question) (*DecisionResponse, error)

Decide asks the configured model the questions about state, billing the call to sessionID. It returns ErrDecisionTooLarge or ErrHardLimitExceeded without calling the provider, and an error if any asked question is left unanswered.

func (*Decider) Model added in v0.48.0 

func (d *Decider) Model() string

Model returns the decision model the decider calls.

func (*Decider) Name added in v0.48.0 

func (d *Decider) Name() string

Name returns the decider's configured name.

func (*Decider) Provider added in v0.49.0 

func (d *Decider) Provider() string

Provider returns the provider instance name the decider calls through.

func (*Decider) WithTimeout added in v0.51.0 

func (d *Decider) WithTimeout(timeout time.Duration) *Decider

WithTimeout returns a shallow clone with a different per-call timeout, so a consumer whose inputs are far larger than the decider's usual ones (the eval judge versus a supervisor review) can give the same model more time without a second [[llm.deciders]] entry.

type DeciderConfig added in v0.48.0 

type DeciderConfig struct {
	Name           string
	Provider       string // provider instance name, for cost attribution
	Model          string
	Timeout        time.Duration // 0 = caller's context only
	MaxInputTokens int           // 0 = no pre-flight size check
}

DeciderConfig binds a named decider to a provider instance and model.

type DeciderSet added in v0.51.0 

type DeciderSet struct {
	// contains filtered or unexported fields
}

DeciderSet is a concurrency-safe registry of live deciders keyed by name. Consumers look a decider up when they bind it, and Sync rebuilds the set from config, so a decider added or changed at runtime needs no restart.

func NewDeciderSet added in v0.51.0 

func NewDeciderSet(providers *ProviderSet, costs *CostTracker) *DeciderSet

NewDeciderSet returns an empty set whose deciders resolve their provider in providers and bill to costs (either may be nil).

func (*DeciderSet) Get added in v0.51.0 

func (s *DeciderSet) Get(name string) *Decider

Get returns the decider registered under name, or nil.

func (*DeciderSet) Names added in v0.51.0 

func (s *DeciderSet) Names() []string

Names returns the registered names, sorted.

func (*DeciderSet) Put added in v0.51.0 

func (s *DeciderSet) Put(d *Decider)

Put registers d under d.Name(), replacing any decider of that name.

func (*DeciderSet) Sync added in v0.51.0 

func (s *DeciderSet) Sync(cfgs []DeciderConfig)

Sync makes the set match cfgs. An entry whose config is unchanged keeps its decider, so consumers can tell "nothing changed" by pointer equality; any other entry gets a new live decider, and names not in cfgs are dropped.

type DecisionProvider added in v0.48.0 

type DecisionProvider interface {
	Decide(ctx context.Context, req DecisionRequest) (*DecisionResponse, error)
}

DecisionProvider is implemented by provider clients that expose a decisions API. Deliberately not part of Provider: a decider cannot chat, so it must never be selectable as an agent model, and it has no tool channel.

type DecisionRequest added in v0.48.0 

type DecisionRequest struct {
	Model     string
	State     any
	Questions map[string]Question
}

DecisionRequest asks a decision model a set of questions about State, which may be a string or any JSON-marshalable value.

type DecisionResponse added in v0.48.0 

type DecisionResponse struct {
	ID, Model string
	Answers   map[string]Answer
	Usage     TokenUsage
	CostUSD   float64
}

DecisionResponse carries the answers keyed by question id.

type FallbackRule 

type FallbackRule struct {
	Trigger    string // "error" | "rate_limit" | "cost_limit"
	Action     string // "switch_provider" | "switch_model" | "wait_and_retry"
	Provider   string // provider name — for switch_provider
	Model      string // model name — for switch_model; optional for switch_provider
	Scope      string // "soft" | "hard" — for cost_limit (which agent limit triggers)
	MaxRetries int    // number of retry attempts — for wait_and_retry
	Backoff    string // "exponential" | "constant" — for wait_and_retry
}

FallbackRule describes a single fallback step the router will attempt.

type FunctionCall 

type FunctionCall struct {
	Name      string `json:"name"`
	Arguments string `json:"arguments"`
}

FunctionCall is the function name and JSON-encoded arguments within a ToolCall.

type FunctionDef 

type FunctionDef struct {
	Name        string         `json:"name"`
	Description string         `json:"description"`
	Parameters  map[string]any `json:"parameters"` // JSON Schema object
}

FunctionDef describes the function signature within a ToolDef.

type IdleTimeoutReader added in v0.21.4 

type IdleTimeoutReader struct {
	// contains filtered or unexported fields
}

IdleTimeoutReader wraps an io.Reader and cancels the associated context if no data is read within the configured timeout. Each successful Read that returns n > 0 resets the timer.

This is designed for LLM SSE streaming responses where we want to detect stalled connections without setting a fixed deadline on the entire request (which would kill legitimately long-running streams that are actively producing data). It is NOT used for MCP tool SSE connections — those use per-request context deadlines and TCP keepalive instead (see 087357b).

func NewIdleTimeoutReader added in v0.21.4 

func NewIdleTimeoutReader(r io.Reader, timeout time.Duration, cancel context.CancelCauseFunc) *IdleTimeoutReader

NewIdleTimeoutReader creates a reader that cancels the given context if no data arrives within timeout. The caller must call Stop() when done (typically via defer) to release the timer.

func (*IdleTimeoutReader) Read added in v0.21.4 

func (r *IdleTimeoutReader) Read(p []byte) (int, error)

Read delegates to the underlying reader and resets the idle timer whenever data is successfully received.

func (*IdleTimeoutReader) Stop added in v0.21.4 

func (r *IdleTimeoutReader) Stop()

Stop releases the idle timer. Safe to call multiple times.

type LLMError 

type LLMError struct {
	StatusCode int
	Message    string
}

LLMError is returned by providers for API-level failures. Use errors.As to unwrap from wrapped errors.

func (*LLMError) Error 

func (e *LLMError) Error() string

func (*LLMError) Retryable 

func (e *LLMError) Retryable() bool

Retryable reports whether the error is worth retrying. Non-retryable: 400 (bad request), 401 (auth), 402 (payment), 404 (not found), 422 (unprocessable). Retryable: 429 (rate limit), 5xx (server errors), and any unrecognised status.

type Message 

type Message struct {
	Role             string     `json:"role"`
	Content          string     `json:"content"`
	ReasoningContent string     `json:"reasoning_content,omitempty"` // reasoning/thinking from previous turn
	ToolCalls        []ToolCall `json:"tool_calls,omitempty"`
	ToolCallID       string     `json:"tool_call_id,omitempty"`
}

type ModelDetailLister added in v0.16.11 

type ModelDetailLister interface {
	ListModelDetails(ctx context.Context) ([]ModelInfo, error)
}

ModelDetailLister is an optional interface for providers that can return enriched model metadata (pricing, capabilities). Providers that implement this are preferred over the static heuristic in Router.ListModelDetails.

type ModelInfo added in v0.16.11 

type ModelInfo struct {
	ID            string   `json:"id"`
	Name          string   `json:"name"`
	Provider      string   `json:"provider"`
	InputPerMTok  *float64 `json:"input_per_mtok"`  // nil = pricing unknown
	OutputPerMTok *float64 `json:"output_per_mtok"` // nil = pricing unknown
	SupportsTools bool     `json:"supports_tools"`
	WeeklyTokens  int64    `json:"weekly_tokens"` // 0 = unknown; used for popularity sort
}

ModelInfo holds enriched metadata about an available LLM model.

type ModelLister added in v0.15.1 

type ModelLister interface {
	ListModels(ctx context.Context) ([]string, error)
}

ModelLister is an optional interface implemented by providers that can enumerate their available models.

type OAIPromptTokenDetail added in v0.16.9 

type OAIPromptTokenDetail struct {
	CachedTokens int `json:"cached_tokens"`
}

OAIPromptTokenDetail holds cached token info from the usage block.

type OAIStreamResult added in v0.16.9 

type OAIStreamResult struct {
	Content      string
	ToolCalls    []ToolCall
	Model        string
	FinishReason string
	Usage        *OAIStreamUsage
	// ReasoningContent captures reasoning_content deltas (OpenRouter).
	ReasoningContent string
	// Provider is the upstream provider that served the response, when the
	// gateway reports it (OpenRouter sets a top-level "provider" field).
	Provider string
}

OAIStreamResult holds the accumulated data from an OpenAI-compatible streaming response. Providers call ReadOAIStream to parse the SSE body and then convert this into an llm.ChatResponse.

func ReadOAIStream added in v0.16.9 

func ReadOAIStream(body io.Reader, onStream StreamCallback, idle *StreamIdleConfig) (*OAIStreamResult, error)

ReadOAIStream reads an OpenAI-compatible SSE stream body and calls onStream for each content/reasoning delta. It accumulates tool calls and usage and returns the full result. onStream may be nil.

If idle is non-nil, the body is wrapped with an IdleTimeoutReader that cancels the stream if no data arrives within idle.Timeout. When the idle timeout fires, ErrStreamIdleTimeout is returned directly so callers can match it with errors.Is.

type OAIStreamUsage added in v0.16.9 

type OAIStreamUsage struct {
	PromptTokens        int                   `json:"prompt_tokens"`
	CompletionTokens    int                   `json:"completion_tokens"`
	TotalTokens         int                   `json:"total_tokens"`
	PromptTokensDetails *OAIPromptTokenDetail `json:"prompt_tokens_details,omitempty"`
	Cost                float64               `json:"cost"` // OpenRouter reports cost here
}

OAIStreamUsage mirrors the usage block in the final SSE chunk.

type Provider 

type Provider interface {
	ChatCompletion(ctx context.Context, req ChatRequest) (*ChatResponse, error)
	Name() string
	HealthCheck(ctx context.Context) error
}

Provider defines the interface for LLM backends.

type ProviderMetadata 

type ProviderMetadata struct {
	Name    string
	BaseURL string
	Models  []string
}

type ProviderSet added in v0.50.0 

type ProviderSet struct {
	// contains filtered or unexported fields
}

ProviderSet is a concurrency-safe registry of providers keyed by name. Routers built with NewRouterWithProviders share one set, so a provider added or replaced at runtime (the providers API, a config reload) is visible to every agent on its next request.

func NewProviderSet added in v0.50.0 

func NewProviderSet() *ProviderSet

NewProviderSet returns an empty set.

func (*ProviderSet) Get added in v0.50.0 

func (s *ProviderSet) Get(name string) (Provider, bool)

Get returns the provider registered under name.

func (*ProviderSet) Names added in v0.50.0 

func (s *ProviderSet) Names() []string

Names returns the registered names, sorted.

func (*ProviderSet) Put added in v0.50.0 

func (s *ProviderSet) Put(p Provider)

Put registers p under p.Name(), replacing any provider of that name. A request already holding the old provider finishes on it.

func (*ProviderSet) Remove added in v0.50.0 

func (s *ProviderSet) Remove(name string)

Remove unregisters name. Removing an absent name is a no-op.

func (*ProviderSet) Snapshot added in v0.50.0 

func (s *ProviderSet) Snapshot() map[string]Provider

Snapshot returns a copy of the registry, safe to range over while the set changes.

type Question added in v0.48.0 

type Question struct {
	Type         QuestionType
	Instructions string
	Choices      map[string]string // choice options; for noul, optional "true"/"false" descriptions
	Levels       []string          // score levels, worst -> best
}

Question is one typed question put to a decision model.

type QuestionType added in v0.48.0 

type QuestionType string

QuestionType is the answer shape a decision model is asked for. 

const (
	QuestionChoice QuestionType = "choice" // one of 2-255 named options
	QuestionNoul   QuestionType = "noul"   // probability the statement is true
	QuestionScore  QuestionType = "score"  // 2-10 ordered levels
)

type Router 

type Router struct {
	// contains filtered or unexported fields
}

Router selects the appropriate LLM provider for a request.

func NewRouter 

func NewRouter(defaultProvider, defaultModel string, costTracker *CostTracker) *Router

NewRouter returns a router with its own private provider set.

func NewRouterWithProviders added in v0.50.0 

func NewRouterWithProviders(defaultProvider, defaultModel string, costTracker *CostTracker, set *ProviderSet) *Router

NewRouterWithProviders returns a router that resolves providers from set, which may be shared with other routers.

func (*Router) Complete 

func (r *Router) Complete(ctx context.Context, sessionID string, messages []Message) (*ChatResponse, error)

func (*Router) CompleteFiltered added in v0.48.0 

func (r *Router) CompleteFiltered(ctx context.Context, sessionID string, messages []Message, filter ToolFilter) (*ChatResponse, error)

CompleteFiltered is Complete with the advertised tool set narrowed by filter. A nil filter is exactly Complete.

func (*Router) CompleteFinal added in v0.40.8 

func (r *Router) CompleteFinal(ctx context.Context, sessionID string, messages []Message) (*ChatResponse, error)

CompleteFinal is like Complete but omits all tool definitions from the request, so the provider cannot return further tool calls. Used for wrap-up completions that must terminate a turn (the "no more tools" contract is enforced by request shape, not prompt instructions).

func (*Router) CompleteStream added in v0.16.9 

func (r *Router) CompleteStream(ctx context.Context, sessionID string, messages []Message, onStream StreamCallback) (*ChatResponse, error)

CompleteStream is like Complete but enables real-time streaming of content chunks via the onStream callback. If the active provider does not support streaming, it falls back to the non-streaming path transparently.

func (*Router) CompleteStreamFiltered added in v0.48.0 

func (r *Router) CompleteStreamFiltered(ctx context.Context, sessionID string, messages []Message, onStream StreamCallback, filter ToolFilter) (*ChatResponse, error)

CompleteStreamFiltered is CompleteStream with the advertised tool set narrowed by filter. A nil filter is exactly CompleteStream.

func (*Router) CostTracker added in v0.16.5 

func (r *Router) CostTracker() *CostTracker

CostTracker returns the router's cost tracker for soft-limit checks.

func (*Router) DefaultModel 

func (r *Router) DefaultModel() string

DefaultModel returns the router's default model name.

func (*Router) DefaultProvider added in v0.17.2 

func (r *Router) DefaultProvider() string

DefaultProvider returns the router's default provider name.

func (*Router) HasProvider added in v0.43.5 

func (r *Router) HasProvider(provider string) bool

HasProvider reports whether the named provider is registered on this router, so a caller accepting a provider override can reject an unknown name up front rather than failing every request that follows.

func (*Router) HealthCheck 

func (r *Router) HealthCheck(ctx context.Context) error

func (*Router) ListModelDetails added in v0.16.11 

func (r *Router) ListModelDetails(ctx context.Context, providerFilter string) []ModelInfo

ListModelDetails queries all providers and returns enriched model metadata including pricing and tool support. Providers implementing ModelDetailLister supply authoritative data; others fall back to the pricing registry and a static tool-support heuristic. When providerFilter is non-empty only the matching provider is queried, which avoids expensive remote calls.

func (*Router) ListModels added in v0.15.1 

func (r *Router) ListModels(ctx context.Context) []string

ListModels queries all registered providers that implement ModelLister and returns a de-duplicated sorted list of available model names.

func (*Router) RegisterProvider 

func (r *Router) RegisterProvider(p Provider)

RegisterProvider adds p to the router's provider set. On a shared set, every router built from it sees p.

func (*Router) SetDefaultModel added in v0.11.0 

func (r *Router) SetDefaultModel(model string)

SetDefaultModel changes the router's default model for subsequent requests.

func (*Router) SetDefaultProvider added in v0.20.0 

func (r *Router) SetDefaultProvider(provider string) error

SetDefaultProvider changes the router's default provider for subsequent requests. Returns an error if the provider is not registered.

func (*Router) SetFallbacks 

func (r *Router) SetFallbacks(rules []FallbackRule)

SetFallbacks configures the ordered list of fallback rules.

func (*Router) SetPricing added in v0.15.1 

func (r *Router) SetPricing(reg *pricing.Registry)

SetPricing configures the model pricing registry used by TokenCost.

func (*Router) SetStreamIdleTimeout added in v0.21.4 

func (r *Router) SetStreamIdleTimeout(d time.Duration)

SetStreamIdleTimeout configures the idle timeout applied to LLM SSE streams. If no data arrives within this duration, the stream is cancelled. Zero disables.

func (*Router) SetTools 

func (r *Router) SetTools(source func() []ToolDef)

SetTools configures a dynamic tool definition source. The function is called on every LLM request so that tools added at runtime are visible immediately.

func (*Router) WithModel added in v0.43.0 

func (r *Router) WithModel(model string) *Router

WithModel returns a shallow copy of r whose default model is model, for callers that need a one-off model override — a dry run previewing a candidate model, or an eval sample running a variant.

It is a copy rather than a SetDefaultModel call because the router is shared by every turn on an agent: mutating it would silently retarget the model of any live turn already in flight. The copy deliberately shares the provider map, cost tracker, pricing registry and fallback rules, so an override still bills to the same budget and honours the same routing rules — only the model differs. Returns r unchanged when the override is empty or already current.

func (*Router) WithProvider added in v0.43.5 

func (r *Router) WithProvider(provider string) *Router

WithProvider is WithModel's counterpart for the provider half of an override, with the same shallow-copy contract: the clone shares the provider map, cost tracker, pricing registry and fallback rules, so only the routing target differs. Composes with WithModel — an eval variant that overrides both chains the two calls and pays for one extra struct copy.

An unregistered name is *not* rejected here: the copy is cheap and side-effect-free, and failing at request time would be an unhelpful place to learn about it. Callers that can validate earlier should ask HasProvider first. Returns r unchanged when the override is empty or already current.

type SSEEvent added in v0.16.9 

type SSEEvent struct {
	Type string // the "event:" field (empty if not present)
	Data string // the "data:" field payload
}

SSEEvent represents a single Server-Sent Event with an optional event type.

type SSEScanner added in v0.16.9 

type SSEScanner struct {
	// contains filtered or unexported fields
}

SSEScanner reads Server-Sent Events from an io.Reader. It yields one SSEEvent per event block, stopping when the reader is exhausted or a "data: [DONE]" sentinel is encountered.

func NewSSEScanner added in v0.16.9 

func NewSSEScanner(r io.Reader) *SSEScanner

NewSSEScanner creates a scanner over the given reader.

func (*SSEScanner) Err added in v0.16.9 

func (s *SSEScanner) Err() error

Err returns the first non-EOF error encountered by the scanner.

func (*SSEScanner) Event added in v0.16.9 

func (s *SSEScanner) Event() SSEEvent

Event returns the most recently scanned SSE event.

func (*SSEScanner) Next added in v0.16.9 

func (s *SSEScanner) Next() bool

Next advances to the next SSE event. Returns false when no more events are available (either EOF or [DONE] sentinel).

type SessionLimits added in v0.16.5 

type SessionLimits struct {
	Soft float64 `json:"soft"`
	Hard float64 `json:"hard"`
}

SessionLimits holds cost limit thresholds for a session. A zero value means the corresponding limit is disabled.

type SessionStats added in v0.12.0 

type SessionStats struct {
	Cost           float64        `json:"cost"`
	InputTokens    int            `json:"input_tokens"`
	OutputTokens   int            `json:"output_tokens"`
	CachedTokens   int            `json:"cached_tokens"`
	Messages       int            `json:"messages"`
	PricingSources map[string]int `json:"pricing_sources,omitempty"`
}

SessionStats holds per-session cost and token tracking.

type StreamCallback added in v0.16.9 

type StreamCallback func(chunk StreamChunk)

StreamCallback is invoked for each chunk during a streaming LLM call. It is called synchronously from the provider's HTTP response reader.

type StreamChunk added in v0.16.9 

type StreamChunk struct {
	ContentDelta  string // incremental text content
	ThinkingDelta string // incremental thinking/reasoning content
	// Reset signals that a failed attempt is being retried and any deltas
	// already delivered for this completion must be discarded — the retry
	// replays the turn from the start. No other field is set alongside it.
	Reset bool
}

StreamChunk carries a single incremental piece of a streaming response.

type StreamIdleConfig added in v0.21.4 

type StreamIdleConfig struct {
	Ctx     context.Context         // the context created with WithCancelCause
	Timeout time.Duration           // idle timeout duration
	Cancel  context.CancelCauseFunc // cancels Ctx when idle timeout fires
}

StreamIdleConfig bundles the parameters needed to guard an SSE stream against stalled providers. Pass nil to ReadOAIStream to disable.

func StreamIdleConfigFor added in v0.21.4 

func StreamIdleConfigFor(ctx context.Context, timeout time.Duration, cancel context.CancelCauseFunc) *StreamIdleConfig

StreamIdleConfigFor returns a *StreamIdleConfig if timeout > 0, or nil otherwise. This is a convenience for providers that conditionally enable idle timeout based on the ChatRequest.

type StreamingProvider added in v0.16.9 

type StreamingProvider interface {
	Provider
	SupportsStreaming() bool
}

StreamingProvider is an optional interface. Providers that implement it honour the OnStream callback field on ChatRequest.

type TokenUsage 

type TokenUsage struct {
	Prompt       int
	Completion   int
	CachedPrompt int // tokens served from cache (Anthropic cache_read, OpenAI cached_tokens)
	Total        int
}

func (*TokenUsage) Add added in v0.40.6 

func (t *TokenUsage) Add(u TokenUsage)

Add accumulates u into t field-by-field. Keeping accumulation in one place means new TokenUsage fields get summed by updating this method rather than every hand-rolled call site (the omission of CachedPrompt from the tool-round accumulator is exactly the bug this guards against).

type ToolCall 

type ToolCall struct {
	ID       string       `json:"id"`
	Type     string       `json:"type"` // "function"
	Function FunctionCall `json:"function"`
}

ToolCall represents a tool invocation requested by the LLM (OpenAI format).

type ToolDef 

type ToolDef struct {
	Type     string      `json:"type"` // "function"
	Function FunctionDef `json:"function"`
}

ToolDef describes a tool available for the LLM to call (OpenAI format).

type ToolFilter added in v0.48.0 

type ToolFilter func([]ToolDef) []ToolDef

ToolFilter narrows the tool definitions advertised on one request. It is applied to the live tool source, so a runtime registration change is still picked up; returning the input unchanged advertises everything.

type UpstreamFaultReporter added in v0.47.2 

type UpstreamFaultReporter interface {
	ResetUpstreamPreference()
}

UpstreamFaultReporter is implemented by providers that keep per-upstream state (sticky routing) which should be discarded when a response was syntactically successful but semantically broken — a 200 the provider's own error classification cannot see.

Source Files

View all Source files

Directories

Path Synopsis
Package anthropic implements the llm.Provider interface against the Anthropic Messages API (https://docs.anthropic.com/en/api/messages).
 Package anthropic implements the llm.Provider interface against the Anthropic Messages API (https://docs.anthropic.com/en/api/messages).
Package llmfactory builds llm.Provider clients from config.
 Package llmfactory builds llm.Provider clients from config.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.