Documentation
¶
Overview ¶
Package provider implements LLM chat adapters for mivia.
Package provider implements LLM chat adapters for mivia.
Package provider implements LLM chat adapters for mivia.
Index ¶
- Constants
- Variables
- func EstimateMessageTokens(msg Message) int
- func EstimateMessageTokensAt(msgs []Message, index int, profile ContextAccountingProfile) int
- func EstimateMessagesPromptCost(messages []Message, schemaCost int, profile ContextAccountingProfile) int
- func EstimatePromptCost(messages []Message, tools []ToolSpec, profile ContextAccountingProfile) (int, error)
- func EstimateRequestCost(messages []Message, tools []ToolSpec, outputReserve int, ...) (int, error)
- func EstimateToolSchemaCost(tools []ToolSpec) (int, error)
- func IsConnectionRefused(err error) bool
- func IsTransient(err error) bool
- func MessageTokens(m Message) int
- func MessageTokensAt(msgs []Message, index int, profile ContextAccountingProfile) int
- func MessagesTokens(msgs []Message, profile ContextAccountingProfile) int
- func RequestTokens(request Request, profile ContextAccountingProfile) (int, error)
- func SetStreamWatchdogTimeouts(idle, firstByte time.Duration)
- func ValidateToolPairing(messages []Message) error
- type AnthropicCompleter
- func (c *AnthropicCompleter) Chat(ctx context.Context, req Request) (string, error)
- func (c *AnthropicCompleter) ChatStream(ctx context.Context, req Request, w io.Writer) (string, error)
- func (c *AnthropicCompleter) ChatTurn(ctx context.Context, req Request) (*Response, error)
- func (c *AnthropicCompleter) Name() string
- type CacheStyle
- type CacheUsage
- type CompatOptions
- type Completer
- func New(res *config.Resolved) (Completer, error)
- func NewAnthropic(opts Options) (Completer, error)
- func NewDeepSeek(opts Options) (Completer, error)
- func NewForProvider(res *config.Resolved, providerName string) (Completer, error)
- func NewLLMGateway(opts Options) (Completer, error)
- func NewLLMProxyCLI(opts Options) (Completer, error)
- func NewMiniMax(opts Options) (Completer, error)
- func NewOllama(opts Options) (Completer, error)
- func NewOpenRouter(opts Options) (Completer, error)
- func NewZAI(opts Options) (Completer, error)
- type ContextAccountingAware
- type ContextAccountingProfile
- type Message
- type OpenAICompat
- func NewOpenAICompat(name, baseURL, apiKey, httpReferer, xTitle string) *OpenAICompat
- func NewOpenAICompatWithOptions(opts CompatOptions) *OpenAICompat
- func NewOpenAICompatWithOptionsAndRetry(options CompatOptions, opts *retryOptions) *OpenAICompat
- func NewOpenAICompatWithRetry(name, baseURL, apiKey, httpReferer, xTitle string, opts *retryOptions) *OpenAICompat
- func (c *OpenAICompat) Chat(ctx context.Context, req Request) (string, error)
- func (c *OpenAICompat) ChatStream(ctx context.Context, req Request, w io.Writer) (string, error)
- func (c *OpenAICompat) ChatTurn(ctx context.Context, req Request) (*Response, error)
- func (c *OpenAICompat) ContextAccounting() ContextAccountingProfile
- func (c *OpenAICompat) Name() string
- func (c *OpenAICompat) ReasoningFields(req *Request) map[string]any
- func (c *OpenAICompat) ReasoningPolicy() ReasoningPolicy
- type Options
- type ReasoningBilling
- type ReasoningPolicy
- type ReasoningPolicyAware
- type Request
- type Response
- type TokenUsage
- type ToolCall
- type ToolSpec
- type TransientError
- type WebSearchResult
Constants ¶
const ( DefaultStreamIdleTimeout = 100 * time.Second DefaultStreamFirstByteTimeout = 240 * time.Second )
DefaultStreamIdleTimeout bounds the gap between successive bytes on a provider read, once the first byte has arrived. DefaultStreamFirstByteTimeout bounds the longer wait for that first byte (headers already arrived, but the provider has not started answering yet - reasoning models can sit silent for a while before the first token).
Both exist because, before this file, the ONLY bound on a provider read (streaming or not) was http.Client.Timeout (DefaultHTTPTimeout, 15 minutes), covering connection + headers + the entire body with no per-chunk reset. A dead-but-open connection sat silent for up to that full window with no observable signal.
const ( RoleSystem = "system" RoleUser = "user" RoleAssistant = "assistant" RoleTool = "tool" )
Role message roles.
const DefaultHTTPTimeout = 15 * time.Minute
DefaultHTTPTimeout is the transport backstop for one provider request. The agent loop's request context remains the tighter per-call policy when one is supplied.
const FinishReasonRefusal = "refusal"
FinishReasonRefusal marks a turn a provider's safety classifier declined rather than completed (Anthropic's stop_reason: "refusal" - an HTTP 200, not an error; Response.Content may be empty or a partial, already-billed prefix). It is not a finish reason any OpenAI-compatible provider in this tree can produce today. internal/agent's empty-response retry (retryOnEmptyResponse in agentloop_run.go) checks for this exact value before retrying a StopEmptyResponse turn: a policy refusal will refuse an identical retry the same way, so retrying only wastes latency and cost against a request that cannot succeed. Keep both sides of that comparison pointed at this constant rather than a duplicated string literal.
Variables ¶
var ErrMaxTokensExceeded = errors.New("max tokens cap exceeded")
ErrMaxTokensExceeded is the sentinel wrapped into a provider error when max_tokens exceeds the limit the serving route/upstream actually allows. llmgateway exposes one model id across upstream routes with different real max_output_tokens caps, so a request at the declared cap can be rejected by a tighter route even though it is legal for the model id. The caller matches this sentinel with errors.Is to re-ask the turn with a halved cap instead of failing the step on a rejection it can recover from. The provider-layer clamp is the recovery boundary; a binding-time pre-clamp of the declared cap is a possible future optimization, not this change.
var ErrPromptTooLong = errors.New("prompt too long")
ErrPromptTooLong is the sentinel wrapped into a provider error when the provider rejects a request because the prompt exceeds its context window. The agent loop matches it with errors.Is to compact the history to a small fixed target and retry exactly once instead of failing the whole run.
var ErrStreamIdle = errors.New("provider: stream idle timeout")
ErrStreamIdle marks a provider read that received no bytes within the configured bound. It is distinct from context.DeadlineExceeded on purpose: a deadline reports an exhausted budget the caller chose, while ErrStreamIdle reports a connection that went silent - callers (transient.go, downstream retry logic) tell the two apart via errors.Is/errors.As.
Functions ¶
func EstimateMessageTokens ¶
EstimateMessageTokens estimates the token cost of a single message using the len(s)/4 heuristic with per-role and per-call frame constants. This avoids re-marshaling tool schemas when only per-message costs are needed (e.g., the planner's incremental tail-fill loop).
It cannot fail - the cost is pure arithmetic over fields already in memory, with no marshaling. Returning no error keeps callers from writing an error branch that can never be taken (and never be tested).
This variant always charges ReasoningContent - it has no list context to resolve a ContextAccountingProfile's ReasoningBillingTerminalExchange against. Use it only for a host-synthesized message that never carries ReasoningContent (a pruning/compaction notice); a message read from real conversation history must go through EstimateMessageTokensAt.
func EstimateMessageTokensAt ¶
func EstimateMessageTokensAt(msgs []Message, index int, profile ContextAccountingProfile) int
EstimateMessageTokensAt is EstimateMessageTokens for msgs[index], charging ReasoningContent per profile (see billsReasoningAt).
func EstimateMessagesPromptCost ¶
func EstimateMessagesPromptCost(messages []Message, schemaCost int, profile ContextAccountingProfile) int
EstimateMessagesPromptCost is EstimatePromptCost with the tool-schema charge supplied by the caller instead of recomputed. Callers that price several candidate message selections against one fixed tool list hoist EstimateToolSchemaCost out of the loop and pass its result here, which is exactly the same number without re-marshaling every schema per candidate.
It cannot fail: with no tools to marshal, the remaining cost is arithmetic over fields already in memory.
func EstimatePromptCost ¶
func EstimatePromptCost(messages []Message, tools []ToolSpec, profile ContextAccountingProfile) (int, error)
EstimatePromptCost returns the input-side request cost. Callers whose budget already excludes the reserved completion allowance must use this rather than charging that allowance a second time.
func EstimateRequestCost ¶
func EstimateRequestCost(messages []Message, tools []ToolSpec, outputReserve int, profile ContextAccountingProfile) (int, error)
EstimateRequestCost returns a conservative, provider-neutral request cost. The estimate intentionally charges for fields that a compact content-only estimator misses: message framing, roles, names, tool IDs, function calls, registered tool schemas, and the reserved completion allowance.
This is an accounting helper, not a provider tokenizer. Callers use the same function before pruning, planning, and local hard-budget rejection so boundary decisions do not depend on the surface that made the request.
func EstimateToolSchemaCost ¶
EstimateToolSchemaCost computes the tool-schema portion of prompt cost once, so callers can hoist it out of hot loops. Returns 0 for an empty or nil list.
func IsConnectionRefused ¶
IsConnectionRefused reports whether err is a connection-refused dial failure on this platform: errors.Is(err, syscall.ECONNREFUSED) covers Unix, and on Windows the unwrapped errno is compared against WSAECONNREFUSED. Match on errno values rather than message text: Winsock wording ("connectex: ... actively refused it") differs from the Unix phrase and is locale-dependent.
func IsTransient ¶
IsTransient reports whether err says the call never delivered an answer.
It reports true for a failure already marked TransientError, and for the transport faults every HTTP client can raise: a timeout, a reset or refused connection, a body that ends early, and a stream the peer tore down.
A cancelled context is NOT transient. The caller stopped the call on purpose, so repeating it would work against that decision.
func MessageTokens ¶
MessageTokens returns the estimated token count for a single message outside any list context - it always charges ReasoningContent. Use this only for a host-synthesized message that never carries ReasoningContent (a pruning/compaction notice); anything read from real conversation history must go through MessageTokensAt so ReasoningContent is charged per the provider's ContextAccountingProfile.
func MessageTokensAt ¶
func MessageTokensAt(msgs []Message, index int, profile ContextAccountingProfile) int
MessageTokensAt is MessageTokens for msgs[index], charging ReasoningContent only when profile bills it for that position (see billsReasoningAt).
func MessagesTokens ¶
func MessagesTokens(msgs []Message, profile ContextAccountingProfile) int
MessagesTokens returns the estimated total token count for a slice of messages, charging ReasoningContent per profile.
func RequestTokens ¶
func RequestTokens(request Request, profile ContextAccountingProfile) (int, error)
RequestTokens returns the request cost using MaxTokens as the output reserve. It is the convenient form for a fully assembled provider request.
func SetStreamWatchdogTimeouts ¶
SetStreamWatchdogTimeouts configures the process-wide idle and first-byte bounds every OpenAICompat client's stream and non-stream body reads honor. A non-positive value leaves the corresponding bound unchanged, so a caller that only knows one of the two never resets the other to zero.
func ValidateToolPairing ¶
ValidateToolPairing rejects provider message histories that cannot be sent as a complete sequence. It deliberately repairs nothing; transport-level compatibility code may still use RepairToolPairing for legacy histories, while context planning must fail closed instead of silently losing data.
Types ¶
type AnthropicCompleter ¶
type AnthropicCompleter struct {
// contains filtered or unexported fields
}
AnthropicCompleter speaks Anthropic's native Messages API (POST /v1/messages) directly, translating this package's OpenAI-shaped Request/Message/ToolCall/Response types to and from Anthropic's own wire format. Unlike every other builtin provider it does not wrap OpenAICompat: Anthropic's request/response shape (system/messages/content-blocks, thinking blocks, output_config.effort) is structurally different from the OpenAI-compatible chat/completions shape the other clients share, so it has nothing to inherit from that type.
anthropicMaxTokensFloor encodes a residual open question this implementation defends against rather than resolves (no live API access in this environment to resolve it empirically): the right max_tokens floor for adaptive thinking at each effort level. This is a conservative, tunable heuristic, not a documented Anthropic invariant.
Thinking blocks are NOT replayed byte-for-byte across turns - only their plain display text survives into Response.ReasoningContent, since the rest of the codebase (the reasoning panel, session persistence) treats that field as plain text to render, not a structured payload. See anthropicThinkingDisplayText and anthropicSystemAndMessages' RoleAssistant case for the full rationale and an UNVERIFIED open question this leaves: whether omitting the thinking block is safe on a turn that also carries ToolCalls followed by a RoleTool message, since that shape has not been checked against a live Anthropic endpoint from this codebase.
func (*AnthropicCompleter) Chat ¶
Chat implements Completer: a plain-text turn discarding tool calls.
func (*AnthropicCompleter) ChatStream ¶
func (c *AnthropicCompleter) ChatStream(ctx context.Context, req Request, w io.Writer) (string, error)
ChatStream implements Completer. It streams the same way ChatTurn does when given a StreamWriter (see ChatTurn's doc comment) and discards everything but the final text - the shape every other provider's ChatStream already has for the no-tools case.
func (*AnthropicCompleter) ChatTurn ¶
ChatTurn implements Completer: a non-stream turn that may return tool calls, and surfaces a safety-classifier refusal as FinishReasonRefusal rather than as an error (Anthropic returns HTTP 200 with an empty or partial content array on a refusal, not a failure status). When req.Stream is set with a non-nil req.StreamWriter, the turn streams - this is how the SDK-backed agent loop gets both live text output and a full tool-call-carrying Response from one call (see internal/agent/agentloop_completer.go's applyStreaming).
func (*AnthropicCompleter) Name ¶
func (c *AnthropicCompleter) Name() string
Name implements Completer.
type CacheStyle ¶
type CacheStyle string
CacheStyle describes how a provider's wire format expresses prompt-cache reuse. Implicit means the provider caches automatically based on request-prefix bytes with no request-side marker (deepseek, zai, ollama). Explicit (marker-style, Anthropic cache_control content blocks) is spoken by the openrouter path, which marks the stable prefix by default and turns off with provider prompt_cache = "off".
const ( CacheStyleNone CacheStyle = "none" CacheStyleImplicit CacheStyle = "implicit" CacheStyleExplicit CacheStyle = "explicit" )
type CacheUsage ¶
type CacheUsage struct {
Reported bool
Style CacheStyle
InputTokens int
CachedInputTokens int
CacheWriteTokens int
}
CacheUsage is provider-reported prompt-cache accounting for one turn. Reported is false when the response carried none of the recognized cache usage fields - that means "not reported", not "zero tokens cached", and every other field is meaningless when it is false. Token counts decoded from an untrusted upstream response are clamped to zero rather than propagated as negative accounting.
type CompatOptions ¶
type CompatOptions struct {
Name string
BaseURL string
APIKey string
HTTPReferer string
XTitle string
ExtraHeaders map[string]string
ExtraBody map[string]any
ErrorParser func(statusCode int, body []byte) error
// NonRetryable classifies an error response as permanent so the transport
// stops retrying it. It is consulted only for statuses the shared policy
// already considers retryable, and nil keeps that policy unchanged.
NonRetryable func(statusCode int, body []byte) bool
// CacheUsageEnabled gates capture of provider-reported prompt-cache usage
// accounting into Response.CacheUsage. It never changes the outgoing
// request.
CacheUsageEnabled bool
// CacheMarkersEnabled requests explicit cache_control markers on the
// stable prefix. Default false. The openrouter factory enables it when
// [provider] prompt_cache != "off"; implicit-cache providers leave it
// off so their bodies stay byte-identical.
CacheMarkersEnabled bool
// Reasoning is this provider's default reasoning wire dialect, used when a
// request carries a level but names no dialect of its own. Empty means the
// provider has no vetted default and an unqualified level sends nothing.
Reasoning reasoning.Dialect
// RequiresReasoningReplay reports whether this provider's wire dialect
// requires the assistant reasoning_content to be echoed back verbatim on
// subsequent tool-call turns (DeepSeek thinking mode, z.ai preserved
// thinking). Default false: the field is never emitted, so existing request
// bodies are byte-identical.
RequiresReasoningReplay bool
// ReplayReasoningField is the replay wire field; empty = "reasoning_content".
ReplayReasoningField string
// RejectReasoningLessToolTurns is the documented-400 DROP gate (DeepSeek
// ONLY). When true, assistant tool-call turns with empty ReasoningContent
// are dropped with their tool results at emit. Independent of
// RequiresReasoningReplay: z.ai sets replay without this bit so a
// reasoning=off multi-step tool run still ships those turns.
RejectReasoningLessToolTurns bool
// ContextAccounting declares how this provider's server bills prompt
// context (see ContextAccountingProfile). The zero value is the
// conservative "bill everything" default, so a factory that leaves this
// unset behaves exactly as before the field existed.
ContextAccounting ContextAccountingProfile
// SendSessionUserKey opts into emitting a hashed session-stickiness key
// as the wire "user" field (see OpenAICompat.sendSessionUserKey). Only
// the openrouter and llmgateway factories set this true.
SendSessionUserKey bool
// DialContext pins every dial for keyless loopback clients; nil keeps http.DefaultTransport.
DialContext func(ctx context.Context, network, addr string) (net.Conn, error)
}
CompatOptions configures an OpenAI-compatible client.
ExtraHeaders and ExtraBody are copied by the options constructors. Reserved request fields are validated when a request is built.
type Completer ¶
type Completer interface {
Name() string
ChatStream(ctx context.Context, req Request, w io.Writer) (string, error)
Chat(ctx context.Context, req Request) (string, error)
// ChatTurn is a non-stream turn that may return tool_calls.
ChatTurn(ctx context.Context, req Request) (*Response, error)
}
Completer talks to an LLM provider.
func NewAnthropic ¶
NewAnthropic returns a native Anthropic Messages API completer.
func NewDeepSeek ¶
NewDeepSeek returns a DeepSeek OpenAI-compatible completer.
DeepSeek thinking mode requires reasoning_content to be replayed on subsequent tool-call turns (RequiresReasoningReplay) and 400s on a tools request that includes a reasoning-less tool-call turn (RejectReasoningLessToolTurns). The default dialect is read from the vetted table (thinking_effort) so config validation and the client agree.
func NewForProvider ¶
NewForProvider builds the configured backend for one provider without mutating the active session. Runtime records are resolved by config.Load; the compatibility projection supports hand-built Resolved values in tests.
func NewLLMGateway ¶
NewLLMGateway returns an LLM Gateway OpenAI-compatible completer. One code path serves DevPass and pay-as-you-go keys: same endpoint, same Bearer auth, same error envelope; the model-ID difference (DevPass rejects provider-prefixed IDs with 403) is gateway-side enforcement.
func NewLLMProxyCLI ¶
NewLLMProxyCLI returns a completer for a local LLM proxy (such as llmproxycli / LiteLLM / CLI proxy) running on a local loopback or custom endpoint. Every model on this provider speaks OpenAI-compatible chat/completions by default, EXCEPT any model entry that explicitly sets reasoning_dialect = "anthropic_adaptive" (opts.AnthropicNativeModels, computed by NewForProvider from the resolved catalog - see anthropicNativeModelsFor): that model's requests instead go out through a native Anthropic Messages API completer built against THIS provider's own BaseURL/APIKey, not a separate anthropic provider config. This exists because a proxy that translates OpenAI-compat requests to Anthropic's real API cannot always deliver Anthropic's own request-shape constraints (e.g. a non-default temperature rejected outright once reasoning is active) - speaking Anthropic's wire format directly to a proxy that also exposes it natively sidesteps the translation entirely.
func NewMiniMax ¶
NewMiniMax returns an OpenAI-compatible completer for the MiniMax API.
func NewOpenRouter ¶
NewOpenRouter returns an OpenRouter OpenAI-compatible completer.
func NewZAI ¶
NewZAI returns a ZAI GLM OpenAI-compatible completer for the standard PaaS endpoint.
GLM Coding Plan keys are not served here: they need base_url set to https://api.z.ai/api/coding/paas/v4. Against this endpoint such a key has no pay-as-you-go balance and every request fails with code 1113.
type ContextAccountingAware ¶
type ContextAccountingAware interface {
ContextAccounting() ContextAccountingProfile
}
ContextAccountingAware is an optional Completer capability: a client that knows how its provider bills context (see ContextAccountingProfile). It is a separate interface, not a Completer method, so the many test fakes that implement Completer without it keep compiling; ContextAccountingFor below treats a Completer that does not implement it exactly like the conservative zero-value profile.
type ContextAccountingProfile ¶
type ContextAccountingProfile struct {
// ReasoningBilling selects how historical assistant ReasoningContent is
// charged. See ReasoningBilling's constants.
ReasoningBilling ReasoningBilling
}
ContextAccountingProfile describes how a provider's server bills prompt context beyond raw text length. It is declared per provider at construction time (CompatOptions.ContextAccounting), the same way RequiresReasoningReplay, ReasoningDialect, and CacheUsageEnabled/ CacheMarkersEnabled are: a small, explicit trait set on the client at construction and read without synchronization afterward.
The zero value is the conservative default (bill everything), so a provider that declares nothing behaves exactly as before this type existed.
internal/contextmgr and the agent loop carry this value opaquely from Completer.ContextAccounting() to the estimators in context.go, which are the only code that interprets its fields. Add a field here (not a second, parallel plumbing path) when a provider needs another context-billing distinction, such as a tool-schema billing quirk or a different cache granularity.
func ContextAccountingFor ¶
func ContextAccountingFor(c Completer) ContextAccountingProfile
ContextAccountingFor returns c's declared context-billing profile, or the conservative zero-value profile (bill everything) when c is nil or does not implement ContextAccountingAware.
type Message ¶
type Message struct {
Role string `json:"role"`
Content string `json:"content,omitempty"`
ToolCalls []ToolCall `json:"tool_calls,omitempty"`
ToolCallID string `json:"tool_call_id,omitempty"`
Name string `json:"name,omitempty"`
// ReasoningContent is the model's chain-of-thought for this turn, preserved
// verbatim on the assistant message so providers whose thinking mode requires
// replay (DeepSeek v4, z.ai preserved thinking) can get it back on subsequent
// tool-call turns. Empty for non-reasoning models and for non-assistant roles.
// Persisted in session history; only ever re-emitted on the wire by providers
// that declare the replay capability (CompatOptions.RequiresReasoningReplay).
// Counted by the token estimators so prompt budgets see it.
ReasoningContent string `json:"reasoning_content,omitempty"`
// CreatedAt is local wall time when the message entered session history.
// Persisted in session JSONL; stripped before provider API requests.
// Zero means unknown (legacy sessions).
CreatedAt time.Time `json:"created_at,omitempty"`
}
Message is a chat turn (supports tool calls and tool results).
func DropEmptyAssistantTurns ¶
DropEmptyAssistantTurns removes assistant messages that carry neither content nor tool calls - the shape a provider's genuinely empty response leaves behind. toAPIMessages already drops this shape at the wire layer (its own doc comment: such a message "makes a session permanently unusable" once persisted, because it is replayed on every later turn and OpenAI-compatible APIs 400 on it) - but that repair only protects the provider request, not the message list itself. ValidateToolPairing (used by contextmgr's planner to gate every Prepare()/commit) hard-rejects the identical shape instead of silently dropping it, so a persisted session carrying this message fails EVERY subsequent turn's context preparation, not just the wire request. This is the same repair, applied to the message list itself so the shape is gone before it is ever validated or persisted, not just before it is serialized.
func PruneMessagesKeepTurns ¶
func PruneMessagesKeepTurns(msgs []Message, maxTokens int, profile ContextAccountingProfile) []Message
PruneMessagesKeepTurns is a smarter pruner that removes entire "turns" (user → assistant/tool exchanges) to preserve conversational coherence. It always keeps the system prompt and the most recent turns within budget.
func RepairReasoningLessToolExchanges ¶
RepairReasoningLessToolExchanges removes assistant tool-call turns that lack reasoning_content, together with their tool results. Used only when RejectReasoningLessToolTurns is set (DeepSeek): those turns 400 on a tools-carrying request. Non-tool assistant turns without reasoning are kept.
This is the documented repair for DeepSeek's 400 gate, and it costs context: an older reasoning-less exchange is dropped WITH its results, so only the terminal exchange (the current loop's pending call plus its results) survives. The tradeoff is accepted because the alternative is a session the API rejects on every later turn.
The gate is unconditional for every DeepSeek client regardless of configured reasoning effort - it is NOT limited to any particular thinking_effort value. Do not assume a shipped config avoids this path. Because dropping here is per-serialization, a caller that re-derives the wire body on every request (toAPIMessages) would otherwise silently re-rewrite history every time it is asked, breaking the provider's prompt-cache prefix and hiding context with no persisted trace. The primary call site is chat.finishAgentTurn, which runs this once at turn-adoption time so persisted history is already valid; toAPIMessages's own call is a defensive no-op backstop, not the source of truth.
func RepairToolPairing ¶
RepairToolPairing drops the message shapes an API rejects outright: an assistant tool_call with no matching tool result, and a tool result naming a call nobody announced.
Either shape poisons a session permanently - history is replayed on every turn, so the request keeps being rejected and no UI action recovers it. They arise from a torn session write (chunks are rewritten in place, and a reader stops at the last complete line without error) and from any producer that records a partial turn.
Repairing here rather than only at the source heals histories already on disk, whatever wrote them.
type OpenAICompat ¶
type OpenAICompat struct {
// contains filtered or unexported fields
}
OpenAICompat is a shared OpenAI-compatible chat client.
func NewOpenAICompat ¶
func NewOpenAICompat(name, baseURL, apiKey, httpReferer, xTitle string) *OpenAICompat
NewOpenAICompat constructs a client with sensible retry defaults. Deprecated: use NewOpenAICompatWithOptions.
func NewOpenAICompatWithOptions ¶
func NewOpenAICompatWithOptions(opts CompatOptions) *OpenAICompat
NewOpenAICompatWithOptions constructs an OpenAI-compatible client from extensible options. Maps are copied before the client accepts requests.
func NewOpenAICompatWithOptionsAndRetry ¶
func NewOpenAICompatWithOptionsAndRetry(options CompatOptions, opts *retryOptions) *OpenAICompat
NewOpenAICompatWithOptionsAndRetry constructs a client with custom retry options.
func NewOpenAICompatWithRetry ¶
func NewOpenAICompatWithRetry(name, baseURL, apiKey, httpReferer, xTitle string, opts *retryOptions) *OpenAICompat
NewOpenAICompatWithRetry constructs a client with custom retry options. Deprecated: use NewOpenAICompatWithOptionsAndRetry.
func (*OpenAICompat) ChatStream ¶
ChatStream streams SSE text deltas to w. With tools present, uses ChatTurn streaming so tool_calls still assemble correctly while content is live.
func (*OpenAICompat) ChatTurn ¶
ChatTurn completion supporting tool_calls. When req.Stream is true, uses SSE and writes content deltas to req.StreamWriter (if set) as they arrive.
func (*OpenAICompat) ContextAccounting ¶
func (c *OpenAICompat) ContextAccounting() ContextAccountingProfile
ContextAccounting returns this client's declared context-billing profile (see ContextAccountingProfile), set once at construction from CompatOptions.ContextAccounting.
func (*OpenAICompat) Name ¶
func (c *OpenAICompat) Name() string
func (*OpenAICompat) ReasoningFields ¶
func (c *OpenAICompat) ReasoningFields(req *Request) map[string]any
ReasoningFields is the exported wrapper around reasoningFields. It runs the same encoding path marshalBody uses, and is exposed for tests and callers that need the resolved wire shape without performing an HTTP round-trip. The side effect of populating req.SDKReasoningEffort is identical: a call here is one more place the SDK-shaped effort is set on the request.
func (*OpenAICompat) ReasoningPolicy ¶
func (c *OpenAICompat) ReasoningPolicy() ReasoningPolicy
ReasoningPolicy reports c's construction-time reasoning-replay wire contract, implementing ReasoningPolicyAware.
type Options ¶
type Options struct {
Name string
BaseURL string
APIKey string
Model string
HTTPReferer string
XTitle string
// CacheUsageEnabled gates capture of provider-reported prompt-cache usage
// accounting. It never changes what is sent to the provider.
CacheUsageEnabled bool
// CacheMarkersEnabled requests explicit cache_control markers on the
// stable prefix for providers whose upstream honors them (OpenRouter
// forwards them to Anthropic-family models; models without explicit
// caching ignore the marker). Factories for providers that only cache
// implicitly (deepseek, zai, ollama) ignore this option so their request
// bodies stay byte-identical.
CacheMarkersEnabled bool
// ContextWindowTokens is the configured model's declared context capacity
// (config.ModelSpec.ContextWindowTokens for the resolved model name), or 0
// if the model is unrecognized. Only consumed by providers whose server
// does not infer context length from the model name on its own (ollama's
// num_ctx); other factories ignore it.
ContextWindowTokens int
// ReasoningDialect is the resolved model's effective wire dialect
// (config.ModelSpec.ReasoningDialect for the resolved model name, falling
// back to the provider's own vetted default when the model entry sets
// none - see reasoningDialectFor). Only a factory whose provider serves a
// caller-chosen, heterogeneous model set needs this: a single-vendor
// factory (deepseek, zai) already knows its own dialect and ignores this
// field. llmgateway reads it to decide, per model, whether the upstream
// speaks a DeepSeek-style thinking dialect that requires the matching
// reasoning-replay and reasoning-less-tool-turn-reject wire contract.
ReasoningDialect reasoning.Dialect
// AnthropicNativeModels lists the names, among this provider's declared
// model catalog, whose resolved reasoning dialect is
// reasoning.DialectAnthropicAdaptive - i.e. every model entry that
// explicitly sets reasoning_dialect = "anthropic_adaptive" (this is never
// a provider default outside "anthropic" itself; see
// reasoning.CanCarryDialect, which config validation already uses to
// reject the dialect on any provider not in its allow-list). Only
// llmproxycli's factory reads this: it builds a per-model dispatcher
// (llmProxyDispatchCompleter) that routes these specific models through a
// native Anthropic Messages API completer, reusing this provider's own
// BaseURL/APIKey, while every other model keeps going through
// OpenAICompat unchanged. Every other factory ignores this field, the
// same way llmgateway is the only reader of ReasoningDialect today.
AnthropicNativeModels []string
// DialContext, when set, replaces the transport's default dial. Set only
// by NewForProvider, only when BaseURL resolved as a verified loopback
// address (config.IsOllamaLoopback) - see its own call site for why. A
// factory that builds its own pinned DialContext internally (ollama.go)
// ignores this field; every other builtin factory forwards it verbatim
// into CompatOptions.DialContext.
DialContext func(ctx context.Context, network, addr string) (net.Conn, error)
}
Options for constructing a completer from resolved config.
type ReasoningBilling ¶
type ReasoningBilling int
ReasoningBilling describes when a provider's server-side context accounting counts a historical assistant message's ReasoningContent toward billed prompt tokens.
const ( // ReasoningBillingAllTurns charges ReasoningContent on every assistant // message the client holds, regardless of turn age. This is the // conservative default (the zero value): overestimating never causes a // request a provider would have accepted to be rejected locally, so every // provider that declares nothing gets this value. ReasoningBillingAllTurns ReasoningBilling = iota // ReasoningBillingTerminalExchange charges ReasoningContent only on the // terminal (still-open) tool exchange; ReasoningContent on an // already-resolved, earlier tool round is free. Some reasoning-replay // providers document that a previous round's reasoning_content is not // itself replayed on the wire on later requests, so it is never billed // (see api-docs.deepseek.com/guides/reasoning_model). ReasoningBillingTerminalExchange // ReasoningBillingNever charges no ReasoningContent, ever. ReasoningBillingNever )
type ReasoningPolicy ¶
type ReasoningPolicy struct {
// RequiresReplay reports whether this client's dialect requires assistant
// reasoning_content to be echoed back verbatim on later tool-call turns.
RequiresReplay bool
// RejectReasoningLess reports whether this client's provider 400s on a
// tools-carrying request that includes a reasoning-less tool-call turn
// (see RepairReasoningLessToolExchanges).
RejectReasoningLess bool
}
ReasoningPolicy is a client's reasoning-replay wire contract, mirroring the CompatOptions bits an OpenAI-compatible client was constructed with.
func ReasoningPolicyFor ¶
func ReasoningPolicyFor(c Completer) ReasoningPolicy
ReasoningPolicyFor returns c's declared reasoning policy, or the zero value (no replay, no reject) when c is nil or does not implement ReasoningPolicyAware.
type ReasoningPolicyAware ¶
type ReasoningPolicyAware interface {
ReasoningPolicy() ReasoningPolicy
}
ReasoningPolicyAware is an optional Completer capability: a client that knows its own reasoning-replay wire contract. Separate from Completer for the same reason as ContextAccountingAware - test fakes implementing Completer alone keep compiling.
type Request ¶
type Request struct {
Model string
Messages []Message
Temperature *float64
MaxTokens *int
Stream bool
// StreamWriter receives content deltas when ChatTurn streams (Stream=true).
// Tool-call argument fragments are not written here - only assistant text.
StreamWriter io.Writer
Tools []ToolSpec
ToolChoice string // "auto", "none", or empty
Timeout time.Duration
// DisableProviderReplay prevents transport and protocol fallbacks from
// issuing a second provider request for this logical attempt.
DisableProviderReplay bool
// ReasoningLevel is the selected model's reasoning dial. Empty sends no
// reasoning field at all, which is the required shape for a non-reasoning
// model, and leaves the request body byte-identical to a pre-reasoning one.
ReasoningLevel reasoning.Level
// ReasoningDialect overrides the client's default wire dialect for this
// request. Empty falls back to the client default; when neither resolves,
// nothing is sent rather than a guessed wire shape.
ReasoningDialect reasoning.Dialect
// SDKReasoningEffort is the SDK-shaped reasoning effort for one request,
// produced by sdkadapter.LevelToReasoningEffort(ReasoningLevel). Empty
// string means "no SDK surface" (the user picked a level the SDK cannot
// carry on the wire, or the request is unset). Currently unused by any
// consumer in the tree; reserved for B.2 #8's SDK-backed inner loop.
SDKReasoningEffort sdkshape.ReasoningEffort `json:"sdk_reasoning_effort,omitempty"`
// SessionID is the caller's session/run identifier, threaded through
// unchanged from whatever principal issued this turn (chat session,
// delegated subagent, workflow step). Empty means unknown - only a
// client that opts into session-keyed routing (CompatOptions.
// SendSessionUserKey) ever reads it, and only then does it reach the
// wire, hashed rather than sent verbatim.
SessionID string
}
Request is a chat completion request.
type Response ¶
type Response struct {
Content string
ReasoningContent string
ToolCalls []ToolCall
FinishReason string
WebSearch []WebSearchResult
// CacheUsage is provider-reported prompt-cache accounting for this turn.
// Its zero value (Reported=false) means the provider reported nothing
// recognized, not that the cache was missed.
CacheUsage CacheUsage
// TokenUsage is provider-reported input/output token counts for this turn.
TokenUsage TokenUsage
}
Response is a non-stream completion result.
type TokenUsage ¶
TokenUsage holds provider-reported input and output token counts for one completion turn. Reported is false when the response carried no recognized usage fields — that means "not reported", not "zero tokens", and every other field is meaningless when it is false.
type ToolCall ¶
type ToolCall struct {
ID string `json:"id"`
Type string `json:"type"`
Function struct {
Name string `json:"name"`
Arguments string `json:"arguments"`
} `json:"function"`
}
ToolCall is an OpenAI-compatible function call from the model.
type TransientError ¶
type TransientError struct{ Err error }
TransientError marks a failure where the call never delivered an answer.
The distinction matters to every caller that decides what a failure means. A model that answers badly is a result: the caller must judge it. A call that is cut by the network is not a result at all, and the caller has nothing to judge. Before this type, both looked the same to a workflow step, so one network fault ended a run that had hours of finished work.
The provider layer owns this judgement because only it knows what its transport does. A caller asks IsTransient and stays free of vendor detail.
func (*TransientError) Error ¶
func (e *TransientError) Error() string
func (*TransientError) Unwrap ¶
func (e *TransientError) Unwrap() error
type WebSearchResult ¶
type WebSearchResult struct {
Title string `json:"title"`
Content string `json:"content"`
Link string `json:"link"`
Media string `json:"media"`
Icon string `json:"icon"`
Refer string `json:"refer"`
PublishDate string `json:"publish_date"`
}
WebSearchResult is provider-supplied search context attached to a completion. Fields are intentionally transport-level so adapters can preserve provider responses without interpreting or rendering them.
Source Files
¶
- anthropic.go
- anthropic_stream.go
- api_message.go
- cache.go
- context.go
- context_profile.go
- deepseek.go
- idle_watchdog.go
- incomplete_body.go
- llmgateway.go
- llmproxycli.go
- max_tokens.go
- minimax.go
- no_replay_redirect.go
- ollama.go
- ollama_loopback.go
- openai_compat.go
- openai_compat_helpers.go
- openai_compat_max_tokens.go
- openai_compat_nonstream.go
- openai_compat_options.go
- openai_compat_request.go
- openai_compat_retry.go
- openai_compat_stream.go
- openai_errors.go
- openrouter.go
- prompt_too_long.go
- provider.go
- reasoning.go
- retry.go
- transient.go
- usage.go
- zai.go
- zai_errors.go