provider

package
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 27, 2026 License: AGPL-3.0 Imports: 27 Imported by: 0

Documentation

Overview

Package provider implements LLM chat adapters for mivia.

Package provider implements LLM chat adapters for mivia.

Package provider implements LLM chat adapters for mivia.

Index

Constants

View Source
const (
	DefaultStreamIdleTimeout      = 100 * time.Second
	DefaultStreamFirstByteTimeout = 240 * time.Second
)

DefaultStreamIdleTimeout bounds the gap between successive bytes on a provider read, once the first byte has arrived. DefaultStreamFirstByteTimeout bounds the longer wait for that first byte (headers already arrived, but the provider has not started answering yet - reasoning models can sit silent for a while before the first token).

Both exist because, before this file, the ONLY bound on a provider read (streaming or not) was http.Client.Timeout (DefaultHTTPTimeout, 15 minutes), covering connection + headers + the entire body with no per-chunk reset. A dead-but-open connection sat silent for up to that full window with no observable signal.

View Source
const (
	RoleSystem    = "system"
	RoleUser      = "user"
	RoleAssistant = "assistant"
	RoleTool      = "tool"
)

Role message roles.

View Source
const DefaultHTTPTimeout = 15 * time.Minute

DefaultHTTPTimeout is the transport backstop for one provider request. The agent loop's request context remains the tighter per-call policy when one is supplied.

View Source
const FinishReasonRefusal = "refusal"

FinishReasonRefusal marks a turn a provider's safety classifier declined rather than completed (Anthropic's stop_reason: "refusal" - an HTTP 200, not an error; Response.Content may be empty or a partial, already-billed prefix). It is not a finish reason any OpenAI-compatible provider in this tree can produce today. internal/agent's empty-response retry (retryOnEmptyResponse in agentloop_run.go) checks for this exact value before retrying a StopEmptyResponse turn: a policy refusal will refuse an identical retry the same way, so retrying only wastes latency and cost against a request that cannot succeed. Keep both sides of that comparison pointed at this constant rather than a duplicated string literal.

Variables

View Source
var ErrMaxTokensExceeded = errors.New("max tokens cap exceeded")

ErrMaxTokensExceeded is the sentinel wrapped into a provider error when max_tokens exceeds the limit the serving route/upstream actually allows. llmgateway exposes one model id across upstream routes with different real max_output_tokens caps, so a request at the declared cap can be rejected by a tighter route even though it is legal for the model id. The caller matches this sentinel with errors.Is to re-ask the turn with a halved cap instead of failing the step on a rejection it can recover from. The provider-layer clamp is the recovery boundary; a binding-time pre-clamp of the declared cap is a possible future optimization, not this change.

View Source
var ErrPromptTooLong = errors.New("prompt too long")

ErrPromptTooLong is the sentinel wrapped into a provider error when the provider rejects a request because the prompt exceeds its context window. The agent loop matches it with errors.Is to compact the history to a small fixed target and retry exactly once instead of failing the whole run.

View Source
var ErrStreamIdle = errors.New("provider: stream idle timeout")

ErrStreamIdle marks a provider read that received no bytes within the configured bound. It is distinct from context.DeadlineExceeded on purpose: a deadline reports an exhausted budget the caller chose, while ErrStreamIdle reports a connection that went silent - callers (transient.go, downstream retry logic) tell the two apart via errors.Is/errors.As.

Functions

func EstimateMessageTokens

func EstimateMessageTokens(msg Message) int

EstimateMessageTokens estimates the token cost of a single message using the len(s)/4 heuristic with per-role and per-call frame constants. This avoids re-marshaling tool schemas when only per-message costs are needed (e.g., the planner's incremental tail-fill loop).

It cannot fail - the cost is pure arithmetic over fields already in memory, with no marshaling. Returning no error keeps callers from writing an error branch that can never be taken (and never be tested).

This variant always charges ReasoningContent - it has no list context to resolve a ContextAccountingProfile's ReasoningBillingTerminalExchange against. Use it only for a host-synthesized message that never carries ReasoningContent (a pruning/compaction notice); a message read from real conversation history must go through EstimateMessageTokensAt.

func EstimateMessageTokensAt

func EstimateMessageTokensAt(msgs []Message, index int, profile ContextAccountingProfile) int

EstimateMessageTokensAt is EstimateMessageTokens for msgs[index], charging ReasoningContent per profile (see billsReasoningAt).

func EstimateMessagesPromptCost

func EstimateMessagesPromptCost(messages []Message, schemaCost int, profile ContextAccountingProfile) int

EstimateMessagesPromptCost is EstimatePromptCost with the tool-schema charge supplied by the caller instead of recomputed. Callers that price several candidate message selections against one fixed tool list hoist EstimateToolSchemaCost out of the loop and pass its result here, which is exactly the same number without re-marshaling every schema per candidate.

It cannot fail: with no tools to marshal, the remaining cost is arithmetic over fields already in memory.

func EstimatePromptCost

func EstimatePromptCost(messages []Message, tools []ToolSpec, profile ContextAccountingProfile) (int, error)

EstimatePromptCost returns the input-side request cost. Callers whose budget already excludes the reserved completion allowance must use this rather than charging that allowance a second time.

func EstimateRequestCost

func EstimateRequestCost(messages []Message, tools []ToolSpec, outputReserve int, profile ContextAccountingProfile) (int, error)

EstimateRequestCost returns a conservative, provider-neutral request cost. The estimate intentionally charges for fields that a compact content-only estimator misses: message framing, roles, names, tool IDs, function calls, registered tool schemas, and the reserved completion allowance.

This is an accounting helper, not a provider tokenizer. Callers use the same function before pruning, planning, and local hard-budget rejection so boundary decisions do not depend on the surface that made the request.

func EstimateToolSchemaCost

func EstimateToolSchemaCost(tools []ToolSpec) (int, error)

EstimateToolSchemaCost computes the tool-schema portion of prompt cost once, so callers can hoist it out of hot loops. Returns 0 for an empty or nil list.

func IsConnectionRefused

func IsConnectionRefused(err error) bool

IsConnectionRefused reports whether err is a connection-refused dial failure on this platform: errors.Is(err, syscall.ECONNREFUSED) covers Unix, and on Windows the unwrapped errno is compared against WSAECONNREFUSED. Match on errno values rather than message text: Winsock wording ("connectex: ... actively refused it") differs from the Unix phrase and is locale-dependent.

func IsTransient

func IsTransient(err error) bool

IsTransient reports whether err says the call never delivered an answer.

It reports true for a failure already marked TransientError, and for the transport faults every HTTP client can raise: a timeout, a reset or refused connection, a body that ends early, and a stream the peer tore down.

A cancelled context is NOT transient. The caller stopped the call on purpose, so repeating it would work against that decision.

func MessageTokens

func MessageTokens(m Message) int

MessageTokens returns the estimated token count for a single message outside any list context - it always charges ReasoningContent. Use this only for a host-synthesized message that never carries ReasoningContent (a pruning/compaction notice); anything read from real conversation history must go through MessageTokensAt so ReasoningContent is charged per the provider's ContextAccountingProfile.

func MessageTokensAt

func MessageTokensAt(msgs []Message, index int, profile ContextAccountingProfile) int

MessageTokensAt is MessageTokens for msgs[index], charging ReasoningContent only when profile bills it for that position (see billsReasoningAt).

func MessagesTokens

func MessagesTokens(msgs []Message, profile ContextAccountingProfile) int

MessagesTokens returns the estimated total token count for a slice of messages, charging ReasoningContent per profile.

func RequestTokens

func RequestTokens(request Request, profile ContextAccountingProfile) (int, error)

RequestTokens returns the request cost using MaxTokens as the output reserve. It is the convenient form for a fully assembled provider request.

func SetStreamWatchdogTimeouts

func SetStreamWatchdogTimeouts(idle, firstByte time.Duration)

SetStreamWatchdogTimeouts configures the process-wide idle and first-byte bounds every OpenAICompat client's stream and non-stream body reads honor. A non-positive value leaves the corresponding bound unchanged, so a caller that only knows one of the two never resets the other to zero.

func ValidateToolPairing

func ValidateToolPairing(messages []Message) error

ValidateToolPairing rejects provider message histories that cannot be sent as a complete sequence. It deliberately repairs nothing; transport-level compatibility code may still use RepairToolPairing for legacy histories, while context planning must fail closed instead of silently losing data.

Types

type AnthropicCompleter

type AnthropicCompleter struct {
	// contains filtered or unexported fields
}

AnthropicCompleter speaks Anthropic's native Messages API (POST /v1/messages) directly, translating this package's OpenAI-shaped Request/Message/ToolCall/Response types to and from Anthropic's own wire format. Unlike every other builtin provider it does not wrap OpenAICompat: Anthropic's request/response shape (system/messages/content-blocks, thinking blocks, output_config.effort) is structurally different from the OpenAI-compatible chat/completions shape the other clients share, so it has nothing to inherit from that type.

anthropicMaxTokensFloor encodes a residual open question this implementation defends against rather than resolves (no live API access in this environment to resolve it empirically): the right max_tokens floor for adaptive thinking at each effort level. This is a conservative, tunable heuristic, not a documented Anthropic invariant.

Thinking blocks are NOT replayed byte-for-byte across turns - only their plain display text survives into Response.ReasoningContent, since the rest of the codebase (the reasoning panel, session persistence) treats that field as plain text to render, not a structured payload. See anthropicThinkingDisplayText and anthropicSystemAndMessages' RoleAssistant case for the full rationale and an UNVERIFIED open question this leaves: whether omitting the thinking block is safe on a turn that also carries ToolCalls followed by a RoleTool message, since that shape has not been checked against a live Anthropic endpoint from this codebase.

func (*AnthropicCompleter) Chat

func (c *AnthropicCompleter) Chat(ctx context.Context, req Request) (string, error)

Chat implements Completer: a plain-text turn discarding tool calls.

func (*AnthropicCompleter) ChatStream

func (c *AnthropicCompleter) ChatStream(ctx context.Context, req Request, w io.Writer) (string, error)

ChatStream implements Completer. It streams the same way ChatTurn does when given a StreamWriter (see ChatTurn's doc comment) and discards everything but the final text - the shape every other provider's ChatStream already has for the no-tools case.

func (*AnthropicCompleter) ChatTurn

func (c *AnthropicCompleter) ChatTurn(ctx context.Context, req Request) (*Response, error)

ChatTurn implements Completer: a non-stream turn that may return tool calls, and surfaces a safety-classifier refusal as FinishReasonRefusal rather than as an error (Anthropic returns HTTP 200 with an empty or partial content array on a refusal, not a failure status). When req.Stream is set with a non-nil req.StreamWriter, the turn streams - this is how the SDK-backed agent loop gets both live text output and a full tool-call-carrying Response from one call (see internal/agent/agentloop_completer.go's applyStreaming).

func (*AnthropicCompleter) Name

func (c *AnthropicCompleter) Name() string

Name implements Completer.

type CacheStyle

type CacheStyle string

CacheStyle describes how a provider's wire format expresses prompt-cache reuse. Implicit means the provider caches automatically based on request-prefix bytes with no request-side marker (deepseek, zai, ollama). Explicit (marker-style, Anthropic cache_control content blocks) is spoken by the openrouter path, which marks the stable prefix by default and turns off with provider prompt_cache = "off".

const (
	CacheStyleNone     CacheStyle = "none"
	CacheStyleImplicit CacheStyle = "implicit"
	CacheStyleExplicit CacheStyle = "explicit"
)

type CacheUsage

type CacheUsage struct {
	Reported          bool
	Style             CacheStyle
	InputTokens       int
	CachedInputTokens int
	CacheWriteTokens  int
}

CacheUsage is provider-reported prompt-cache accounting for one turn. Reported is false when the response carried none of the recognized cache usage fields - that means "not reported", not "zero tokens cached", and every other field is meaningless when it is false. Token counts decoded from an untrusted upstream response are clamped to zero rather than propagated as negative accounting.

type CompatOptions

type CompatOptions struct {
	Name         string
	BaseURL      string
	APIKey       string
	HTTPReferer  string
	XTitle       string
	ExtraHeaders map[string]string
	ExtraBody    map[string]any
	ErrorParser  func(statusCode int, body []byte) error
	// NonRetryable classifies an error response as permanent so the transport
	// stops retrying it. It is consulted only for statuses the shared policy
	// already considers retryable, and nil keeps that policy unchanged.
	NonRetryable func(statusCode int, body []byte) bool
	// CacheUsageEnabled gates capture of provider-reported prompt-cache usage
	// accounting into Response.CacheUsage. It never changes the outgoing
	// request.
	CacheUsageEnabled bool
	// CacheMarkersEnabled requests explicit cache_control markers on the
	// stable prefix. Default false. The openrouter factory enables it when
	// [provider] prompt_cache != "off"; implicit-cache providers leave it
	// off so their bodies stay byte-identical.
	CacheMarkersEnabled bool
	// Reasoning is this provider's default reasoning wire dialect, used when a
	// request carries a level but names no dialect of its own. Empty means the
	// provider has no vetted default and an unqualified level sends nothing.
	Reasoning reasoning.Dialect
	// RequiresReasoningReplay reports whether this provider's wire dialect
	// requires the assistant reasoning_content to be echoed back verbatim on
	// subsequent tool-call turns (DeepSeek thinking mode, z.ai preserved
	// thinking). Default false: the field is never emitted, so existing request
	// bodies are byte-identical.
	RequiresReasoningReplay bool
	// ReplayReasoningField is the replay wire field; empty = "reasoning_content".
	ReplayReasoningField string
	// RejectReasoningLessToolTurns is the documented-400 DROP gate (DeepSeek
	// ONLY). When true, assistant tool-call turns with empty ReasoningContent
	// are dropped with their tool results at emit. Independent of
	// RequiresReasoningReplay: z.ai sets replay without this bit so a
	// reasoning=off multi-step tool run still ships those turns.
	RejectReasoningLessToolTurns bool
	// ContextAccounting declares how this provider's server bills prompt
	// context (see ContextAccountingProfile). The zero value is the
	// conservative "bill everything" default, so a factory that leaves this
	// unset behaves exactly as before the field existed.
	ContextAccounting ContextAccountingProfile
	// SendSessionUserKey opts into emitting a hashed session-stickiness key
	// as the wire "user" field (see OpenAICompat.sendSessionUserKey). Only
	// the openrouter and llmgateway factories set this true.
	SendSessionUserKey bool
	// DialContext pins every dial for keyless loopback clients; nil keeps http.DefaultTransport.
	DialContext func(ctx context.Context, network, addr string) (net.Conn, error)
}

CompatOptions configures an OpenAI-compatible client.

ExtraHeaders and ExtraBody are copied by the options constructors. Reserved request fields are validated when a request is built.

type Completer

type Completer interface {
	Name() string
	ChatStream(ctx context.Context, req Request, w io.Writer) (string, error)
	Chat(ctx context.Context, req Request) (string, error)
	// ChatTurn is a non-stream turn that may return tool_calls.
	ChatTurn(ctx context.Context, req Request) (*Response, error)
}

Completer talks to an LLM provider.

func New

func New(res *config.Resolved) (Completer, error)

New builds a Completer from resolved config.

func NewAnthropic

func NewAnthropic(opts Options) (Completer, error)

NewAnthropic returns a native Anthropic Messages API completer.

func NewDeepSeek

func NewDeepSeek(opts Options) (Completer, error)

NewDeepSeek returns a DeepSeek OpenAI-compatible completer.

DeepSeek thinking mode requires reasoning_content to be replayed on subsequent tool-call turns (RequiresReasoningReplay) and 400s on a tools request that includes a reasoning-less tool-call turn (RejectReasoningLessToolTurns). The default dialect is read from the vetted table (thinking_effort) so config validation and the client agree.

func NewForProvider

func NewForProvider(res *config.Resolved, providerName string) (Completer, error)

NewForProvider builds the configured backend for one provider without mutating the active session. Runtime records are resolved by config.Load; the compatibility projection supports hand-built Resolved values in tests.

func NewLLMGateway

func NewLLMGateway(opts Options) (Completer, error)

NewLLMGateway returns an LLM Gateway OpenAI-compatible completer. One code path serves DevPass and pay-as-you-go keys: same endpoint, same Bearer auth, same error envelope; the model-ID difference (DevPass rejects provider-prefixed IDs with 403) is gateway-side enforcement.

func NewLLMProxyCLI

func NewLLMProxyCLI(opts Options) (Completer, error)

NewLLMProxyCLI returns a completer for a local LLM proxy (such as llmproxycli / LiteLLM / CLI proxy) running on a local loopback or custom endpoint. Every model on this provider speaks OpenAI-compatible chat/completions by default, EXCEPT any model entry that explicitly sets reasoning_dialect = "anthropic_adaptive" (opts.AnthropicNativeModels, computed by NewForProvider from the resolved catalog - see anthropicNativeModelsFor): that model's requests instead go out through a native Anthropic Messages API completer built against THIS provider's own BaseURL/APIKey, not a separate anthropic provider config. This exists because a proxy that translates OpenAI-compat requests to Anthropic's real API cannot always deliver Anthropic's own request-shape constraints (e.g. a non-default temperature rejected outright once reasoning is active) - speaking Anthropic's wire format directly to a proxy that also exposes it natively sidesteps the translation entirely.

func NewMiniMax

func NewMiniMax(opts Options) (Completer, error)

NewMiniMax returns an OpenAI-compatible completer for the MiniMax API.

func NewOllama

func NewOllama(opts Options) (Completer, error)

NewOllama returns an Ollama OpenAI-compatible completer.

func NewOpenRouter

func NewOpenRouter(opts Options) (Completer, error)

NewOpenRouter returns an OpenRouter OpenAI-compatible completer.

func NewZAI

func NewZAI(opts Options) (Completer, error)

NewZAI returns a ZAI GLM OpenAI-compatible completer for the standard PaaS endpoint.

GLM Coding Plan keys are not served here: they need base_url set to https://api.z.ai/api/coding/paas/v4. Against this endpoint such a key has no pay-as-you-go balance and every request fails with code 1113.

type ContextAccountingAware

type ContextAccountingAware interface {
	ContextAccounting() ContextAccountingProfile
}

ContextAccountingAware is an optional Completer capability: a client that knows how its provider bills context (see ContextAccountingProfile). It is a separate interface, not a Completer method, so the many test fakes that implement Completer without it keep compiling; ContextAccountingFor below treats a Completer that does not implement it exactly like the conservative zero-value profile.

type ContextAccountingProfile

type ContextAccountingProfile struct {
	// ReasoningBilling selects how historical assistant ReasoningContent is
	// charged. See ReasoningBilling's constants.
	ReasoningBilling ReasoningBilling
}

ContextAccountingProfile describes how a provider's server bills prompt context beyond raw text length. It is declared per provider at construction time (CompatOptions.ContextAccounting), the same way RequiresReasoningReplay, ReasoningDialect, and CacheUsageEnabled/ CacheMarkersEnabled are: a small, explicit trait set on the client at construction and read without synchronization afterward.

The zero value is the conservative default (bill everything), so a provider that declares nothing behaves exactly as before this type existed.

internal/contextmgr and the agent loop carry this value opaquely from Completer.ContextAccounting() to the estimators in context.go, which are the only code that interprets its fields. Add a field here (not a second, parallel plumbing path) when a provider needs another context-billing distinction, such as a tool-schema billing quirk or a different cache granularity.

func ContextAccountingFor

func ContextAccountingFor(c Completer) ContextAccountingProfile

ContextAccountingFor returns c's declared context-billing profile, or the conservative zero-value profile (bill everything) when c is nil or does not implement ContextAccountingAware.

type Message

type Message struct {
	Role       string     `json:"role"`
	Content    string     `json:"content,omitempty"`
	ToolCalls  []ToolCall `json:"tool_calls,omitempty"`
	ToolCallID string     `json:"tool_call_id,omitempty"`
	Name       string     `json:"name,omitempty"`
	// ReasoningContent is the model's chain-of-thought for this turn, preserved
	// verbatim on the assistant message so providers whose thinking mode requires
	// replay (DeepSeek v4, z.ai preserved thinking) can get it back on subsequent
	// tool-call turns. Empty for non-reasoning models and for non-assistant roles.
	// Persisted in session history; only ever re-emitted on the wire by providers
	// that declare the replay capability (CompatOptions.RequiresReasoningReplay).
	// Counted by the token estimators so prompt budgets see it.
	ReasoningContent string `json:"reasoning_content,omitempty"`
	// CreatedAt is local wall time when the message entered session history.
	// Persisted in session JSONL; stripped before provider API requests.
	// Zero means unknown (legacy sessions).
	CreatedAt time.Time `json:"created_at,omitempty"`
}

Message is a chat turn (supports tool calls and tool results).

func DropEmptyAssistantTurns

func DropEmptyAssistantTurns(msgs []Message) []Message

DropEmptyAssistantTurns removes assistant messages that carry neither content nor tool calls - the shape a provider's genuinely empty response leaves behind. toAPIMessages already drops this shape at the wire layer (its own doc comment: such a message "makes a session permanently unusable" once persisted, because it is replayed on every later turn and OpenAI-compatible APIs 400 on it) - but that repair only protects the provider request, not the message list itself. ValidateToolPairing (used by contextmgr's planner to gate every Prepare()/commit) hard-rejects the identical shape instead of silently dropping it, so a persisted session carrying this message fails EVERY subsequent turn's context preparation, not just the wire request. This is the same repair, applied to the message list itself so the shape is gone before it is ever validated or persisted, not just before it is serialized.

func PruneMessagesKeepTurns

func PruneMessagesKeepTurns(msgs []Message, maxTokens int, profile ContextAccountingProfile) []Message

PruneMessagesKeepTurns is a smarter pruner that removes entire "turns" (user → assistant/tool exchanges) to preserve conversational coherence. It always keeps the system prompt and the most recent turns within budget.

func RepairReasoningLessToolExchanges

func RepairReasoningLessToolExchanges(msgs []Message) []Message

RepairReasoningLessToolExchanges removes assistant tool-call turns that lack reasoning_content, together with their tool results. Used only when RejectReasoningLessToolTurns is set (DeepSeek): those turns 400 on a tools-carrying request. Non-tool assistant turns without reasoning are kept.

This is the documented repair for DeepSeek's 400 gate, and it costs context: an older reasoning-less exchange is dropped WITH its results, so only the terminal exchange (the current loop's pending call plus its results) survives. The tradeoff is accepted because the alternative is a session the API rejects on every later turn.

The gate is unconditional for every DeepSeek client regardless of configured reasoning effort - it is NOT limited to any particular thinking_effort value. Do not assume a shipped config avoids this path. Because dropping here is per-serialization, a caller that re-derives the wire body on every request (toAPIMessages) would otherwise silently re-rewrite history every time it is asked, breaking the provider's prompt-cache prefix and hiding context with no persisted trace. The primary call site is chat.finishAgentTurn, which runs this once at turn-adoption time so persisted history is already valid; toAPIMessages's own call is a defensive no-op backstop, not the source of truth.

func RepairToolPairing

func RepairToolPairing(msgs []Message) []Message

RepairToolPairing drops the message shapes an API rejects outright: an assistant tool_call with no matching tool result, and a tool result naming a call nobody announced.

Either shape poisons a session permanently - history is replayed on every turn, so the request keeps being rejected and no UI action recovers it. They arise from a torn session write (chunks are rewritten in place, and a reader stops at the last complete line without error) and from any producer that records a partial turn.

Repairing here rather than only at the source heals histories already on disk, whatever wrote them.

type OpenAICompat

type OpenAICompat struct {
	// contains filtered or unexported fields
}

OpenAICompat is a shared OpenAI-compatible chat client.

func NewOpenAICompat

func NewOpenAICompat(name, baseURL, apiKey, httpReferer, xTitle string) *OpenAICompat

NewOpenAICompat constructs a client with sensible retry defaults. Deprecated: use NewOpenAICompatWithOptions.

func NewOpenAICompatWithOptions

func NewOpenAICompatWithOptions(opts CompatOptions) *OpenAICompat

NewOpenAICompatWithOptions constructs an OpenAI-compatible client from extensible options. Maps are copied before the client accepts requests.

func NewOpenAICompatWithOptionsAndRetry

func NewOpenAICompatWithOptionsAndRetry(options CompatOptions, opts *retryOptions) *OpenAICompat

NewOpenAICompatWithOptionsAndRetry constructs a client with custom retry options.

func NewOpenAICompatWithRetry

func NewOpenAICompatWithRetry(name, baseURL, apiKey, httpReferer, xTitle string, opts *retryOptions) *OpenAICompat

NewOpenAICompatWithRetry constructs a client with custom retry options. Deprecated: use NewOpenAICompatWithOptionsAndRetry.

func (*OpenAICompat) Chat

func (c *OpenAICompat) Chat(ctx context.Context, req Request) (string, error)

Chat non-streaming text-only convenience.

func (*OpenAICompat) ChatStream

func (c *OpenAICompat) ChatStream(ctx context.Context, req Request, w io.Writer) (string, error)

ChatStream streams SSE text deltas to w. With tools present, uses ChatTurn streaming so tool_calls still assemble correctly while content is live.

func (*OpenAICompat) ChatTurn

func (c *OpenAICompat) ChatTurn(ctx context.Context, req Request) (*Response, error)

ChatTurn completion supporting tool_calls. When req.Stream is true, uses SSE and writes content deltas to req.StreamWriter (if set) as they arrive.

func (*OpenAICompat) ContextAccounting

func (c *OpenAICompat) ContextAccounting() ContextAccountingProfile

ContextAccounting returns this client's declared context-billing profile (see ContextAccountingProfile), set once at construction from CompatOptions.ContextAccounting.

func (*OpenAICompat) Name

func (c *OpenAICompat) Name() string

func (*OpenAICompat) ReasoningFields

func (c *OpenAICompat) ReasoningFields(req *Request) map[string]any

ReasoningFields is the exported wrapper around reasoningFields. It runs the same encoding path marshalBody uses, and is exposed for tests and callers that need the resolved wire shape without performing an HTTP round-trip. The side effect of populating req.SDKReasoningEffort is identical: a call here is one more place the SDK-shaped effort is set on the request.

func (*OpenAICompat) ReasoningPolicy

func (c *OpenAICompat) ReasoningPolicy() ReasoningPolicy

ReasoningPolicy reports c's construction-time reasoning-replay wire contract, implementing ReasoningPolicyAware.

type Options

type Options struct {
	Name        string
	BaseURL     string
	APIKey      string
	Model       string
	HTTPReferer string
	XTitle      string
	// CacheUsageEnabled gates capture of provider-reported prompt-cache usage
	// accounting. It never changes what is sent to the provider.
	CacheUsageEnabled bool
	// CacheMarkersEnabled requests explicit cache_control markers on the
	// stable prefix for providers whose upstream honors them (OpenRouter
	// forwards them to Anthropic-family models; models without explicit
	// caching ignore the marker). Factories for providers that only cache
	// implicitly (deepseek, zai, ollama) ignore this option so their request
	// bodies stay byte-identical.
	CacheMarkersEnabled bool
	// ContextWindowTokens is the configured model's declared context capacity
	// (config.ModelSpec.ContextWindowTokens for the resolved model name), or 0
	// if the model is unrecognized. Only consumed by providers whose server
	// does not infer context length from the model name on its own (ollama's
	// num_ctx); other factories ignore it.
	ContextWindowTokens int
	// ReasoningDialect is the resolved model's effective wire dialect
	// (config.ModelSpec.ReasoningDialect for the resolved model name, falling
	// back to the provider's own vetted default when the model entry sets
	// none - see reasoningDialectFor). Only a factory whose provider serves a
	// caller-chosen, heterogeneous model set needs this: a single-vendor
	// factory (deepseek, zai) already knows its own dialect and ignores this
	// field. llmgateway reads it to decide, per model, whether the upstream
	// speaks a DeepSeek-style thinking dialect that requires the matching
	// reasoning-replay and reasoning-less-tool-turn-reject wire contract.
	ReasoningDialect reasoning.Dialect
	// AnthropicNativeModels lists the names, among this provider's declared
	// model catalog, whose resolved reasoning dialect is
	// reasoning.DialectAnthropicAdaptive - i.e. every model entry that
	// explicitly sets reasoning_dialect = "anthropic_adaptive" (this is never
	// a provider default outside "anthropic" itself; see
	// reasoning.CanCarryDialect, which config validation already uses to
	// reject the dialect on any provider not in its allow-list). Only
	// llmproxycli's factory reads this: it builds a per-model dispatcher
	// (llmProxyDispatchCompleter) that routes these specific models through a
	// native Anthropic Messages API completer, reusing this provider's own
	// BaseURL/APIKey, while every other model keeps going through
	// OpenAICompat unchanged. Every other factory ignores this field, the
	// same way llmgateway is the only reader of ReasoningDialect today.
	AnthropicNativeModels []string
	// DialContext, when set, replaces the transport's default dial. Set only
	// by NewForProvider, only when BaseURL resolved as a verified loopback
	// address (config.IsOllamaLoopback) - see its own call site for why. A
	// factory that builds its own pinned DialContext internally (ollama.go)
	// ignores this field; every other builtin factory forwards it verbatim
	// into CompatOptions.DialContext.
	DialContext func(ctx context.Context, network, addr string) (net.Conn, error)
}

Options for constructing a completer from resolved config.

type ReasoningBilling

type ReasoningBilling int

ReasoningBilling describes when a provider's server-side context accounting counts a historical assistant message's ReasoningContent toward billed prompt tokens.

const (
	// ReasoningBillingAllTurns charges ReasoningContent on every assistant
	// message the client holds, regardless of turn age. This is the
	// conservative default (the zero value): overestimating never causes a
	// request a provider would have accepted to be rejected locally, so every
	// provider that declares nothing gets this value.
	ReasoningBillingAllTurns ReasoningBilling = iota
	// ReasoningBillingTerminalExchange charges ReasoningContent only on the
	// terminal (still-open) tool exchange; ReasoningContent on an
	// already-resolved, earlier tool round is free. Some reasoning-replay
	// providers document that a previous round's reasoning_content is not
	// itself replayed on the wire on later requests, so it is never billed
	// (see api-docs.deepseek.com/guides/reasoning_model).
	ReasoningBillingTerminalExchange
	// ReasoningBillingNever charges no ReasoningContent, ever.
	ReasoningBillingNever
)

type ReasoningPolicy

type ReasoningPolicy struct {
	// RequiresReplay reports whether this client's dialect requires assistant
	// reasoning_content to be echoed back verbatim on later tool-call turns.
	RequiresReplay bool
	// RejectReasoningLess reports whether this client's provider 400s on a
	// tools-carrying request that includes a reasoning-less tool-call turn
	// (see RepairReasoningLessToolExchanges).
	RejectReasoningLess bool
}

ReasoningPolicy is a client's reasoning-replay wire contract, mirroring the CompatOptions bits an OpenAI-compatible client was constructed with.

func ReasoningPolicyFor

func ReasoningPolicyFor(c Completer) ReasoningPolicy

ReasoningPolicyFor returns c's declared reasoning policy, or the zero value (no replay, no reject) when c is nil or does not implement ReasoningPolicyAware.

type ReasoningPolicyAware

type ReasoningPolicyAware interface {
	ReasoningPolicy() ReasoningPolicy
}

ReasoningPolicyAware is an optional Completer capability: a client that knows its own reasoning-replay wire contract. Separate from Completer for the same reason as ContextAccountingAware - test fakes implementing Completer alone keep compiling.

type Request

type Request struct {
	Model       string
	Messages    []Message
	Temperature *float64
	MaxTokens   *int
	Stream      bool
	// StreamWriter receives content deltas when ChatTurn streams (Stream=true).
	// Tool-call argument fragments are not written here - only assistant text.
	StreamWriter io.Writer
	Tools        []ToolSpec
	ToolChoice   string // "auto", "none", or empty
	Timeout      time.Duration
	// DisableProviderReplay prevents transport and protocol fallbacks from
	// issuing a second provider request for this logical attempt.
	DisableProviderReplay bool
	// ReasoningLevel is the selected model's reasoning dial. Empty sends no
	// reasoning field at all, which is the required shape for a non-reasoning
	// model, and leaves the request body byte-identical to a pre-reasoning one.
	ReasoningLevel reasoning.Level
	// ReasoningDialect overrides the client's default wire dialect for this
	// request. Empty falls back to the client default; when neither resolves,
	// nothing is sent rather than a guessed wire shape.
	ReasoningDialect reasoning.Dialect
	// SDKReasoningEffort is the SDK-shaped reasoning effort for one request,
	// produced by sdkadapter.LevelToReasoningEffort(ReasoningLevel). Empty
	// string means "no SDK surface" (the user picked a level the SDK cannot
	// carry on the wire, or the request is unset). Currently unused by any
	// consumer in the tree; reserved for B.2 #8's SDK-backed inner loop.
	SDKReasoningEffort sdkshape.ReasoningEffort `json:"sdk_reasoning_effort,omitempty"`
	// SessionID is the caller's session/run identifier, threaded through
	// unchanged from whatever principal issued this turn (chat session,
	// delegated subagent, workflow step). Empty means unknown - only a
	// client that opts into session-keyed routing (CompatOptions.
	// SendSessionUserKey) ever reads it, and only then does it reach the
	// wire, hashed rather than sent verbatim.
	SessionID string
}

Request is a chat completion request.

type Response

type Response struct {
	Content          string
	ReasoningContent string
	ToolCalls        []ToolCall
	FinishReason     string
	WebSearch        []WebSearchResult
	// CacheUsage is provider-reported prompt-cache accounting for this turn.
	// Its zero value (Reported=false) means the provider reported nothing
	// recognized, not that the cache was missed.
	CacheUsage CacheUsage
	// TokenUsage is provider-reported input/output token counts for this turn.
	TokenUsage TokenUsage
}

Response is a non-stream completion result.

type TokenUsage

type TokenUsage struct {
	Reported     bool
	InputTokens  int
	OutputTokens int
}

TokenUsage holds provider-reported input and output token counts for one completion turn. Reported is false when the response carried no recognized usage fields — that means "not reported", not "zero tokens", and every other field is meaningless when it is false.

type ToolCall

type ToolCall struct {
	ID       string `json:"id"`
	Type     string `json:"type"`
	Function struct {
		Name      string `json:"name"`
		Arguments string `json:"arguments"`
	} `json:"function"`
}

ToolCall is an OpenAI-compatible function call from the model.

type ToolSpec

type ToolSpec = map[string]any

ToolSpec is an OpenAI tools[] entry (already shaped as map from tools.Registry).

type TransientError

type TransientError struct{ Err error }

TransientError marks a failure where the call never delivered an answer.

The distinction matters to every caller that decides what a failure means. A model that answers badly is a result: the caller must judge it. A call that is cut by the network is not a result at all, and the caller has nothing to judge. Before this type, both looked the same to a workflow step, so one network fault ended a run that had hours of finished work.

The provider layer owns this judgement because only it knows what its transport does. A caller asks IsTransient and stays free of vendor detail.

func (*TransientError) Error

func (e *TransientError) Error() string

func (*TransientError) Unwrap

func (e *TransientError) Unwrap() error

type WebSearchResult

type WebSearchResult struct {
	Title       string `json:"title"`
	Content     string `json:"content"`
	Link        string `json:"link"`
	Media       string `json:"media"`
	Icon        string `json:"icon"`
	Refer       string `json:"refer"`
	PublishDate string `json:"publish_date"`
}

WebSearchResult is provider-supplied search context attached to a completion. Fields are intentionally transport-level so adapters can preserve provider responses without interpreting or rendering them.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL