provider

package
v0.1.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 6, 2026 License: MIT Imports: 16 Imported by: 0

Documentation

Overview

Package provider defines the model-backend abstraction and a registry mapping a provider "kind" to a factory. Concrete implementations live in subpackages (e.g. provider/openai) and self-register via init(). The core resolves providers by kind from config and never hardcodes a specific model.

Index

Constants

View Source
const MaxRetries = 10

MaxRetries is the number of times SendWithRetry re-attempts the connection + header phase after the initial try (so up to MaxRetries+1 total attempts).

Variables

This section is empty.

Functions

func BudgetKeyForConfig

func BudgetKeyForConfig(name, baseURL, apiKey string) string

BudgetKeyForConfig returns the budget bucket key for a provider Config — baseURL + the resolved API key. Two providers hitting the same endpoint with the same key share one RPM quota (matching how providers actually meter: the platform counts requests per key, regardless of which model/feature/client issued them). The name parameter is accepted for call-site symmetry with provider.Config but intentionally NOT included, so the main conversation, subagents, multimodal tools, RAG extraction/embedding, and RagAsk all draw from one bucket when they share an endpoint+key.

func CanonicalizeSchema

func CanonicalizeSchema(raw json.RawMessage) json.RawMessage

CanonicalizeSchema recursively stabilizes a JSON Schema so the same logical schema always produces the same byte representation.

func ContentLen

func ContentLen(content any) int

ContentLen estimates the byte size of Content for token budgeting. For multimodal content, text bytes + base64 image bytes (approximate).

func ContentString

func ContentString(content any) string

ContentString extracts the text portion of a Message.Content field. When Content is nil or a plain string, returns it directly. When Content is a structured multimodal block, extracts and concatenates text parts.

func ImageContent

func ImageContent(text string, imageURLs ...string) any

ImageContent creates a multimodal Content value with text and image parts.

func IsConnReset

func IsConnReset(err error) bool

IsConnReset reports whether err is a connection-level drop (peer reset, truncated body, closed socket) as opposed to a protocol or caller error. A stream cut this way mid-body can be replayed from scratch, unlike a decode or 4xx error. The common trigger is a local proxy (v2rayN/sing-box) idle-closing the long-lived SSE connection during a reasoner's first-token gap.

func IsStreamInterrupted

func IsStreamInterrupted(err error) bool

func IsTextOnly

func IsTextOnly(content any) bool

IsTextOnly reports whether Content contains no image parts.

func Kinds

func Kinds() []string

Kinds returns the registered kinds, sorted.

func ParseImageDataURL

func ParseImageDataURL(dataURL string) (mediaType, base64Data string, ok bool)

ParseImageDataURL splits a "data:image/png;base64,AAAA..." data URL into its MIME type and raw base64 payload. Returns ("", "", false) when the prefix is not a valid data URL.

func Register

func Register(kind string, f Factory)

Register adds a factory under a kind (e.g. "openai"). Intended for init(). It panics on a duplicate kind, since that is a compile-time wiring mistake.

func RetryableStatus

func RetryableStatus(s int) bool

RetryableStatus reports whether a backoff can plausibly recover from status s: 408 (request timeout), 429 (rate limit) and 5xx (incl. Anthropic's 529). Other 4xx (400/401/402/422, …) are caller/config problems retrying can't fix.

func SendWithRetry

func SendWithRetry(ctx context.Context, httpClient *http.Client, opts SendOptions, newReq func(context.Context) (*http.Request, error)) (*http.Response, error)

SendWithRetry POSTs a streaming request built by newReq and returns the OK response. It retries the connection+header phase up to MaxRetries times on transient network errors and retryable statuses with capped exponential backoff + jitter, honoring Retry-After. 401/403 become *AuthError; other non-OK statuses become *APIError. A RetryNotify in ctx fires before each sleep. Retries cover only the header phase — once the body streams, mid-stream failures are not retried (the model has already emitted tokens).

func TextContent

func TextContent(text string) any

TextContent creates a simple text-only Content value.

func WithRetryNotify

func WithRetryNotify(ctx context.Context, fn RetryNotify) context.Context

WithRetryNotify attaches a callback that SendWithRetry invokes before each backoff sleep, so the agent can surface a transient "retrying (n/m)" status.

Types

type APIError

type APIError struct {
	Provider string
	Status   int
	Body     string
}

APIError reports a non-OK HTTP status that isn't an auth failure. Status carries the code so the display layer can map it to an actionable, localized message; Body is a trimmed snippet of the response.

func (*APIError) Error

func (e *APIError) Error() string

type AuthError

type AuthError struct {
	Provider string // the provider instance name, e.g. "openai"
	KeyEnv   string // the api_key_env the key is read from, when known
	Status   int    // the HTTP status (401 or 403)
	HasKey   bool   // a non-empty key was sent vs. no key configured
}

AuthError reports that a provider rejected the API key (HTTP 401/403). Its message is already user-facing and actionable — it names the provider and, when known, the environment variable the key comes from — so the CLI can surface it verbatim instead of dumping a raw status body. Providers should return this (rather than a generic status error) for auth failures.

func (*AuthError) Error

func (e *AuthError) Error() string

type BudgetStatus

type BudgetStatus struct {
	RPM         int `json:"rpm"`
	Used        int `json:"used"`
	Remaining   int `json:"remaining"`
	ReserveMain int `json:"reserveMain"`
	WindowSecs  int `json:"windowSecs"` // seconds until the window resets
}

Status reports the current-window usage for a key, for the UI's "RPM: 3/5 remaining" indicator and cost estimates.

type Chunk

type Chunk struct {
	Type      ChunkType
	Text      string    // ChunkText, ChunkReasoning
	Signature string    // ChunkReasoning: opaque proof for the reasoning (Anthropic thinking signature), when issued
	ToolCall  *ToolCall // ChunkToolCallStart (ID+Name only), ChunkToolCall (complete)
	Usage     *Usage    // ChunkUsage
	Err       error     // ChunkError
}

Chunk is a single streamed event. Read the field matching Type.

type ChunkType

type ChunkType int

ChunkType identifies the kind of a streamed increment.

const (
	ChunkText          ChunkType = iota // text delta
	ChunkReasoning                      // thinking-mode reasoning delta (before the visible answer)
	ChunkToolCallStart                  // a tool call has begun (ToolCall: ID+Name; args still streaming)
	ChunkToolCall                       // one complete tool call
	ChunkUsage                          // token usage for the completion
	ChunkDone                           // completion finished normally
	ChunkError                          // an error occurred
)

type Config

type Config struct {
	Name    string         // instance name, e.g. "openai"
	BaseURL string         // OpenAI-compatible endpoint
	Model   string         // model id
	APIKey  string         // resolved from api_key_env
	Extra   map[string]any // kind-specific options
}

Config is a resolved provider instance configuration.

type ContentPart

type ContentPart struct {
	Type     string    `json:"type"`                // "text" or "image_url"
	Text     string    `json:"text,omitempty"`      // for Type == "text"
	ImageURL *ImageURL `json:"image_url,omitempty"` // for Type == "image_url"
}

ContentPart is one block in a multimodal message (OpenAI content parts format).

func ImageParts

func ImageParts(content any) []ContentPart

ImageParts extracts image ContentParts from a multimodal Content value. Returns nil when Content is a plain string or contains no images.

type Factory

type Factory func(cfg Config) (Provider, error)

Factory builds a Provider from a resolved Config.

type ImageURL

type ImageURL struct {
	URL    string `json:"url"`              // "data:image/png;base64,..."
	Detail string `json:"detail,omitempty"` // "low", "high", "auto"
}

ImageURL holds a base64 data URL for inline image content.

type Message

type Message struct {
	Role             Role   `json:"role"`
	Content          any    `json:"content,omitempty"`           // string or []ContentPart (multimodal)
	ReasoningContent string `json:"reasoning_content,omitempty"` // assistant: thinking-mode chain-of-thought, round-tripped on multi-turn
	// ReasoningSignature is an opaque, provider-issued proof that ReasoningContent
	// is genuine model output. Anthropic requires the signed thinking block be
	// replayed on the next turn when a tool call followed thinking; providers
	// without signed reasoning (e.g. the openai-compatible ones) leave it empty.
	// Round-tripped alongside ReasoningContent.
	ReasoningSignature string     `json:"reasoning_signature,omitempty"`
	ToolCalls          []ToolCall `json:"tool_calls,omitempty"`   // set by assistant
	ToolCallID         string     `json:"tool_call_id,omitempty"` // links a tool result to its call
	Name               string     `json:"name,omitempty"`         // tool message: tool name
}

Message is a single conversation message.

func NormalizeMessages

func NormalizeMessages(msgs []Message) []Message

NormalizeMessages repairs a conversation history so it satisfies the tool-call contract the OpenAI-compatible and Anthropic APIs enforce: every assistant tool_calls entry must be answered by a following tool message for its id, and a tool message must follow such a call. It backfills a placeholder result for any unanswered call (so the turn stays intact), drops orphan tool messages, backfills empty tool-call names from their results (old sessions saved before name-tracking landed can carry an empty name), and closes truncated call-argument JSON (some gateways 400 on replayed half-streamed args).

This is the wire-safe entry point for provider requests. Stored session loads use NormalizeSessionMessages so they can share the assistant-turn repairs without deleting standalone tool messages that must round-trip through resume.

A well-formed history — no unanswered calls, no orphan results, no empty tool-call names, no truncated args — returns the input slice unchanged (same backing array, zero allocation). This keeps the prefix-cache key stable for healthy sessions and makes repeated normalization cheap.

Ported from DeepSeek-Reasonix (PR #4811 unifying history normalization).

func NormalizeSessionMessages

func NormalizeSessionMessages(msgs []Message) []Message

NormalizeSessionMessages applies only repairs that are safe to persist in a saved session. It shares assistant-turn repairs with NormalizeMessages, but preserves existing tool messages instead of dropping or reordering them so Save/LoadSession remains a byte-for-byte conversation round trip for histories that were already on disk.

func SanitizeToolPairing

func SanitizeToolPairing(msgs []Message) []Message

SanitizeToolPairing repairs a history for a provider request (every assistant tool_calls answered by a following tool message, orphan tool messages dropped, empty tool-call names backfilled from results, truncated args closed) right before sending it to the wire — without touching the stored session. Kept as a distinct name so call sites read as "defensive wire prep" rather than "session mutation". Now a thin alias over NormalizeMessages so the wire path and the session-load path share one repair implementation.

func (*Message) UnmarshalJSON

func (m *Message) UnmarshalJSON(data []byte) error

UnmarshalJSON restores Content as its concrete type. The field is `any` (string for plain text, []ContentPart for multimodal). encoding/json has no way to recover that from a generic []interface{} on reload, so without this a saved multimodal message comes back as []interface{} — and every downstream switch on `content.(type)` (ContentString, ContentLen, buildRequest) misses its []ContentPart case, dumping the image data URL as plain text. We decode content separately: a JSON string stays a string; a JSON array becomes []ContentPart, fixing the type for the whole pipeline.

type Pricing

type Pricing struct {
	CacheHit float64 `toml:"cache_hit"` // per 1M cached prompt tokens
	Input    float64 `toml:"input"`     // per 1M uncached prompt tokens
	Output   float64 `toml:"output"`    // per 1M completion tokens
	Currency string  `toml:"currency"`
}

Pricing is a provider's per-1M-token rates, used to estimate spend. Currency is just a display symbol (default "¥"). toml tags let config decode it.

func (*Pricing) Cost

func (p *Pricing) Cost(u *Usage) float64

Cost estimates the spend for a usage record.

func (*Pricing) Symbol

func (p *Pricing) Symbol() string

Symbol returns the currency display symbol, defaulting to "¥". Normalizes common ISO codes to their symbol equivalents.

type Provider

type Provider interface {
	// Name returns the provider instance name, e.g. "openai" / "anthropic".
	Name() string
	// Stream starts a streaming completion, pushing increments on the channel.
	// Cancelling ctx must abort the underlying request; a closed channel marks
	// the end of the completion.
	Stream(ctx context.Context, req Request) (<-chan Chunk, error)
}

Provider is a chat-capable model backend.

func New

func New(kind string, cfg Config) (Provider, error)

New instantiates the provider of the given kind.

func NewRateLimitedProvider

func NewRateLimitedProvider(inner Provider, budget *RequestBudget, key string, priority bool) Provider

NewRateLimitedProvider wraps inner. key identifies the budget bucket (baseURL+apiKey). priority=true marks this as a main-agent provider (always granted, may use reserve); false = background (waits on reserve).

func UnwrapProvider

func UnwrapProvider(p Provider) Provider

UnwrapProvider peels decorator layers (anything implementing Unwrap() Provider) off p until it reaches a provider that has no inner, returning that base. Use this before a type assertion on a provider that may have been wrapped by boot.NewProviderWithProxy (which adds RateLimitedProvider when a global RPM budget is configured). nil-safe: a nil p returns nil.

type RateLimitedProvider

type RateLimitedProvider struct {
	// contains filtered or unexported fields
}

RateLimitedProvider wraps an inner Provider with request-budget enforcement.

func (*RateLimitedProvider) Name

func (p *RateLimitedProvider) Name() string

func (*RateLimitedProvider) Stream

func (p *RateLimitedProvider) Stream(ctx context.Context, req Request) (<-chan Chunk, error)

func (*RateLimitedProvider) Unwrap

func (p *RateLimitedProvider) Unwrap() Provider

Unwrap returns the inner Provider this decorator wraps, so callers (and tests) can reach the concrete provider underneath any number of decorators. Pairs with UnwrapProvider, the standard Go unwrap idiom: NewProviderWithProxy may wrap a provider with this rate limiter depending on the global budget, so a direct type assertion on its return value is unsafe — unwrap first.

type Request

type Request struct {
	Messages    []Message
	Tools       []ToolSchema
	Temperature float64
	MaxTokens   int
}

Request is a single completion request.

type RequestBudget

type RequestBudget struct {
	// contains filtered or unexported fields
}

RequestBudget meters RPM across one or more API keys. Each key (identified by baseURL+apiKey) gets its own rolling window, because different providers have independent quotas. nil / RPM=0 means no limiting.

func NewRequestBudget

func NewRequestBudget(rpm, reserveMain int) *RequestBudget

NewRequestBudget builds a budget. rpm<=0 returns a disabled budget (Acquire is a no-op) so callers don't need to branch on "is limiting configured".

func (*RequestBudget) Acquire

func (b *RequestBudget) Acquire(ctx context.Context, key string, priority bool) error

Acquire blocks until a request slot is available for the given key, or ctx is cancelled. priority=true grants immediately (main-agent); priority=false (background) waits when remaining quota <= reserve.

Returns nil on success, ctx.Err() if the context is cancelled while waiting.

func (*RequestBudget) Status

func (b *RequestBudget) Status(key string) BudgetStatus

Status returns the current budget status for a key. When limiting is disabled, returns a zeroed status (RPM=0).

type RetryInfo

type RetryInfo struct {
	Attempt int
	Max     int
	Delay   time.Duration
	Err     error
}

RetryInfo describes a backoff about to happen: Attempt is the 1-based retry number (of Max) and Delay is how long SendWithRetry will wait before it.

type RetryNotify

type RetryNotify func(RetryInfo)

type Role

type Role string

Role is the role of a message.

const (
	RoleSystem    Role = "system"
	RoleUser      Role = "user"
	RoleAssistant Role = "assistant"
	RoleTool      Role = "tool"
)

type SendOptions

type SendOptions struct {
	ProvName   string // provider instance name for error messages
	KeyEnv     string // api_key_env for AuthError
	KeyPresent bool   // a non-empty key was configured
	RetryAuth  bool   // retry 401/403 up to maxAuthRetries (key previously worked)
}

SendOptions configures SendWithRetry's behaviour.

type StreamInterruptedError

type StreamInterruptedError struct {
	Err error
}

StreamInterruptedError marks a recoverable transport cut that happened after the caller had already received model output. Providers must not replay these requests themselves because doing so could duplicate visible text or tool calls; the agent can append a tail recovery prompt instead.

func (*StreamInterruptedError) Error

func (e *StreamInterruptedError) Error() string

func (*StreamInterruptedError) Unwrap

func (e *StreamInterruptedError) Unwrap() error

type ToolCall

type ToolCall struct {
	ID        string `json:"id"`
	Name      string `json:"name"`
	Arguments string `json:"arguments"`
}

ToolCall is a tool invocation requested by the model. Arguments is raw JSON.

type ToolSchema

type ToolSchema struct {
	Name        string          `json:"name"`
	Description string          `json:"description"`
	Parameters  json.RawMessage `json:"parameters"`
}

ToolSchema is a tool definition exposed to the model. Parameters is JSON Schema.

type Usage

type Usage struct {
	PromptTokens     int
	CompletionTokens int
	TotalTokens      int
	CacheHitTokens   int    // prompt tokens served from cache
	CacheMissTokens  int    // prompt tokens not cached
	ReasoningTokens  int    // subset of CompletionTokens spent on chain-of-thought
	FinishReason     string // "stop", "tool_calls", "length", "content_filter", "repetition_truncation", …
}

Usage reports token accounting for a completion. Cache hit/miss come from either top-level prompt_cache_{hit,miss}_tokens or the OpenAI standard prompt_tokens_details.cached_tokens — the openai provider normalises both shapes into these fields. Note: some providers do not report cache tokens (both fields stay 0); the normalisation is kept for future support. ReasoningTokens is the thinking-mode subset of CompletionTokens reported by thinking-capable models. FinishReason carries the model's last reported choices[0].finish_reason so the agent can surface abnormal terminations ("length", "content_filter", "repetition_truncation").

Directories

Path Synopsis
Package anthropic implements the Anthropic Messages API provider (POST /v1/messages, SSE streaming) with a hand-written net/http client — no SDK.
Package anthropic implements the Anthropic Messages API provider (POST /v1/messages, SSE streaming) with a hand-written net/http client — no SDK.
Package openai implements the OpenAI-compatible /chat/completions provider.
Package openai implements the OpenAI-compatible /chat/completions provider.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL