model

package
v0.12.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 19, 2026 License: MIT Imports: 19 Imported by: 0

Documentation

Overview

Package model defines pluggable LLM provider interfaces.

Index

Constants

View Source
const (
	RoleSystem    = "system"
	RoleUser      = "user"
	RoleAssistant = "assistant"
	RoleTool      = "tool"
)

Role constants for chat messages.

Variables

View Source
var ErrCircuitOpen = errors.New("circuit breaker open")

ErrCircuitOpen is returned when a provider's circuit breaker is open and requests are short-circuited to give the upstream time to recover.

Functions

func ContextLimit

func ContextLimit(modelName string, fallback int) int

ContextLimit returns the maximum context window (in tokens) for a model. If the model is unknown, it returns the provided fallback value.

func EstimateTokens added in v0.3.0

func EstimateTokens(messages []Message) int

EstimateTokens provides a rough token estimate for a list of messages.

func IsRetryable added in v0.4.0

func IsRetryable(err error) bool

IsRetryable reports whether err represents a transient failure worth retrying (network/transport failure, HTTP 408, 429 or 5xx). Context cancellation, deadline expiry and an open circuit are not retryable.

func IsTerminal added in v0.4.0

func IsTerminal(err error) bool

IsTerminal reports whether err is a permanent failure — a 4xx client error other than 408/429 — that will not succeed on retry or on a different provider (e.g. malformed request, authentication failure, not found).

func RegisterProviderFactory added in v0.3.0

func RegisterProviderFactory(name string, factory ProviderFactory)

RegisterProviderFactory registers a factory for a provider name.

Types

type APIError added in v0.4.0

type APIError struct {
	StatusCode int           `json:"status_code"`
	Status     string        `json:"status"`
	Body       string        `json:"body"`
	RetryAfter time.Duration `json:"retry_after"`
}

APIError represents a non-2xx HTTP response from a model provider API. It carries the status code, body snippet and any server-requested retry delay so callers (retry logic, fallback routing, circuit breakers) can classify the failure as retryable or terminal.

func (*APIError) Error added in v0.4.0

func (e *APIError) Error() string

Error implements the error interface.

type Anthropic

type Anthropic struct {
	// contains filtered or unexported fields
}

Anthropic implements Provider for Claude models via the Anthropic Messages API.

func NewAnthropic

func NewAnthropic(apiKey string) *Anthropic

NewAnthropic creates a new Anthropic provider with the given API key.

func NewAnthropicWithConfig

func NewAnthropicWithConfig(cfg ProviderConfig) *Anthropic

NewAnthropicWithConfig creates an Anthropic provider with full configuration.

func (*Anthropic) Chat

func (a *Anthropic) Chat(ctx context.Context, req *ChatRequest) (*ChatResponse, error)

func (*Anthropic) Model

func (a *Anthropic) Model() string

func (*Anthropic) Name

func (a *Anthropic) Name() string

func (*Anthropic) StreamChat

func (a *Anthropic) StreamChat(ctx context.Context, req *ChatRequest) (<-chan *ChatResponse, error)

type AudioContent added in v0.3.0

type AudioContent struct {
	Data       []byte `json:"-"`
	Format     string `json:"format"`               // wav, mp3, ogg, etc.
	Transcript string `json:"transcript,omitempty"` // transcribed text (for input) or source text (for output)
}

AudioContent holds audio data for input transcription or output TTS.

type AudioProvider added in v0.3.0

type AudioProvider interface {
	// Transcribe converts audio data to text using a speech-to-text model (e.g., Whisper).
	Transcribe(ctx context.Context, audio AudioContent) (string, error)
	// Synthesize converts text to speech using a TTS model.
	// voice is the voice ID (e.g., "alloy", "echo", "fable", "onyx", "nova", "shimmer").
	Synthesize(ctx context.Context, text, voice string) (*AudioContent, error)
}

AudioProvider is the interface for audio transcription and synthesis.

type AzureConfig

type AzureConfig struct {
	ProviderConfig
	Deployment string `json:"deployment"`
	APIVersion string `json:"api_version"`
}

AzureConfig extends ProviderConfig with Azure-specific fields.

type AzureOpenAI

type AzureOpenAI struct {
	// contains filtered or unexported fields
}

AzureOpenAI implements Provider for Azure-hosted OpenAI models. Standard requests use the deployment-scoped Chat Completions endpoint; native reasoning requests use /openai/v1/responses so reasoning and tools can be combined without the Chat Completions API restriction.

func NewAzureOpenAI

func NewAzureOpenAI(endpoint, apiKey, deployment string) *AzureOpenAI

NewAzureOpenAI creates a new Azure OpenAI provider. endpoint is the Azure resource endpoint (e.g., "https://myresource.openai.azure.com"). apiKey is the Azure API key. deployment is the model deployment name.

func NewAzureOpenAIWithConfig

func NewAzureOpenAIWithConfig(cfg AzureConfig) *AzureOpenAI

NewAzureOpenAIWithConfig creates an Azure OpenAI provider with full configuration.

func (*AzureOpenAI) Chat

func (a *AzureOpenAI) Chat(ctx context.Context, req *ChatRequest) (*ChatResponse, error)

func (*AzureOpenAI) Model

func (a *AzureOpenAI) Model() string

func (*AzureOpenAI) Name

func (a *AzureOpenAI) Name() string

func (*AzureOpenAI) StreamChat

func (a *AzureOpenAI) StreamChat(ctx context.Context, req *ChatRequest) (<-chan *ChatResponse, error)

type AzureOpenAIEmbeddings added in v0.3.0

type AzureOpenAIEmbeddings struct {
	// contains filtered or unexported fields
}

AzureOpenAIEmbeddings implements EmbeddingsProvider using Azure OpenAI's embeddings API.

func NewAzureOpenAIEmbeddings added in v0.3.0

func NewAzureOpenAIEmbeddings(endpoint, apiKey, deployment string) *AzureOpenAIEmbeddings

NewAzureOpenAIEmbeddings creates an Azure OpenAI embeddings provider. endpoint is the Azure OpenAI resource endpoint (e.g., "https://myresource.openai.azure.com"). apiKey is the Azure API key. deployment is the deployment name (e.g., "text-embedding-3-large").

func NewAzureOpenAIEmbeddingsWithConfig added in v0.3.0

func NewAzureOpenAIEmbeddingsWithConfig(cfg ProviderConfig, deployment, apiVersion string) *AzureOpenAIEmbeddings

NewAzureOpenAIEmbeddingsWithConfig creates an Azure OpenAI embeddings provider with full config.

func (*AzureOpenAIEmbeddings) Embed added in v0.3.0

type BPECounter added in v0.4.0

type BPECounter struct {
	// contains filtered or unexported fields
}

BPECounter counts tokens using a real byte-pair-encoding (BPE) tokenizer (tiktoken encodings), giving exact counts for OpenAI models and a close approximation for other providers. It replaces the character-ratio heuristic for accuracy in budgeting and context-window management.

func NewBPECounter added in v0.4.0

func NewBPECounter(modelName string) (*BPECounter, error)

NewBPECounter builds a BPE token counter for the given model. Known OpenAI models map to their exact encoding; unknown models (including Anthropic, Gemini, etc.) fall back to the newest general-purpose encoding (o200k_base), which yields a close approximation.

func (*BPECounter) CountString added in v0.4.0

func (c *BPECounter) CountString(s string) int

CountString returns the exact BPE token count for s.

func (*BPECounter) CountTokens added in v0.4.0

func (c *BPECounter) CountTokens(messages []Message) int

CountTokens returns the token count for a full set of chat messages, including per-message and conversation framing overhead.

type Bedrock added in v0.3.0

type Bedrock struct {
	// contains filtered or unexported fields
}

Bedrock implements Provider for AWS Bedrock's InvokeModel API. Supports Claude, Titan, Llama, and other models hosted on Bedrock.

func NewBedrock added in v0.3.0

func NewBedrock(region, accessKey, secretKey, modelID string) *Bedrock

NewBedrock creates a Bedrock provider. region is the AWS region (e.g., "us-east-1"). accessKey and secretKey are AWS credentials. modelID is the Bedrock model identifier (e.g., "anthropic.claude-3-sonnet-20240229-v1:0").

func NewBedrockWithConfig added in v0.3.0

func NewBedrockWithConfig(region string, cfg ProviderConfig, secretKey string) *Bedrock

NewBedrockWithConfig creates a Bedrock provider with full configuration.

func (*Bedrock) Chat added in v0.3.0

func (b *Bedrock) Chat(ctx context.Context, req *ChatRequest) (*ChatResponse, error)

func (*Bedrock) Model added in v0.3.0

func (b *Bedrock) Model() string

func (*Bedrock) Name added in v0.3.0

func (b *Bedrock) Name() string

func (*Bedrock) StreamChat added in v0.3.0

func (b *Bedrock) StreamChat(ctx context.Context, req *ChatRequest) (<-chan *ChatResponse, error)

type CachedEmbeddings

type CachedEmbeddings struct {
	// contains filtered or unexported fields
}

CachedEmbeddings wraps an EmbeddingsProvider with an in-memory cache.

func NewCachedEmbeddings

func NewCachedEmbeddings(inner EmbeddingsProvider) *CachedEmbeddings

func (*CachedEmbeddings) Embed

type ChatRequest

type ChatRequest struct {
	Model       string           `json:"model"`
	Messages    []Message        `json:"messages"`
	MaxTokens   int              `json:"max_tokens,omitempty"`
	Temperature float64          `json:"temperature,omitempty"`
	TopP        float64          `json:"top_p,omitempty"`
	Stream      bool             `json:"stream,omitempty"`
	Tools       []ToolDefinition `json:"tools,omitempty"`
	Stop        []string         `json:"stop,omitempty"`
	// ResponseFormat controls output format. Values:
	//   "json_object"  — model returns valid JSON (no schema enforcement)
	//   "json_schema"  — model returns JSON conforming to Metadata["json_schema"]
	ResponseFormat string           `json:"response_format,omitempty"`
	Metadata       map[string]any   `json:"metadata,omitempty"`
	Reasoning      *ReasoningConfig `json:"reasoning,omitempty"`
}

ChatRequest is the input to a chat completion.

type ChatResponse

type ChatResponse struct {
	ID         string     `json:"id"`
	Content    string     `json:"content"`
	Role       string     `json:"role"`
	Usage      Usage      `json:"usage"`
	ToolCalls  []ToolCall `json:"tool_calls,omitempty"`
	StopReason StopReason `json:"stop_reason,omitempty"`
	// Reasoning carries provider-approved thinking/reasoning output separately
	// from the final answer. Callers should not display it unless explicitly
	// requested because providers may treat internal reasoning as sensitive.
	Reasoning string `json:"reasoning,omitempty"`
	// ProviderState carries opaque provider output required to continue a
	// stateless tool-calling conversation. It is never exposed in JSON.
	ProviderState any `json:"-"`
	// Delta is true when this is a partial streaming response.
	Delta bool `json:"delta,omitempty"`
	// Err carries a streaming transport/parse error surfaced mid-stream. It is
	// non-nil only on the final delta emitted when a stream fails to read
	// cleanly (e.g. bufio.Scanner error). Not serialized.
	Err error `json:"-"`
}

ChatResponse is the output of a chat completion.

func AggregateStream added in v0.4.0

func AggregateStream(ctx context.Context, ch <-chan *ChatResponse) (*ChatResponse, error)

AggregateStream consumes a StreamChat delta channel and merges the partial responses into a single complete ChatResponse: it concatenates text content, reassembles streamed tool calls, accumulates token usage and captures the final stop reason. It is the recommended way for non-streaming callers (and SDK-level orchestration) to obtain a full response from Provider.StreamChat.

If ctx is canceled it returns the context error. If any delta carried a streaming error (ChatResponse.Err) that error is returned once the stream drains.

type Cohere added in v0.3.0

type Cohere struct {
	// contains filtered or unexported fields
}

Cohere implements Provider for the Cohere Chat API. Supports Command, Command-R, and Command-R+ models.

func NewCohere added in v0.3.0

func NewCohere(apiKey, modelID string) *Cohere

NewCohere creates a Cohere provider. apiKey is the Cohere API key. modelID is the model identifier (e.g., "command-r-plus", "command-r", "command").

func NewCohereWithConfig added in v0.3.0

func NewCohereWithConfig(cfg ProviderConfig) *Cohere

NewCohereWithConfig creates a Cohere provider with full configuration.

func (*Cohere) Chat added in v0.3.0

func (c *Cohere) Chat(ctx context.Context, req *ChatRequest) (*ChatResponse, error)

func (*Cohere) Model added in v0.3.0

func (c *Cohere) Model() string

func (*Cohere) Name added in v0.3.0

func (c *Cohere) Name() string

func (*Cohere) StreamChat added in v0.3.0

func (c *Cohere) StreamChat(ctx context.Context, req *ChatRequest) (<-chan *ChatResponse, error)

type CohereEmbeddings added in v0.3.0

type CohereEmbeddings struct {
	// contains filtered or unexported fields
}

CohereEmbeddings implements EmbeddingsProvider using the Cohere Embed API.

func NewCohereEmbeddings added in v0.3.0

func NewCohereEmbeddings(apiKey, modelID string) *CohereEmbeddings

NewCohereEmbeddings creates a Cohere embeddings provider. apiKey is the Cohere API key. modelID is the model identifier (e.g., "embed-english-v3.0", "embed-multilingual-v3.0").

func NewCohereEmbeddingsWithConfig added in v0.3.0

func NewCohereEmbeddingsWithConfig(cfg ProviderConfig) *CohereEmbeddings

NewCohereEmbeddingsWithConfig creates a Cohere embeddings provider with full config.

func (*CohereEmbeddings) Embed added in v0.3.0

type ContentPart added in v0.3.0

type ContentPart struct {
	Type     string `json:"type"`                // "text", "image_url", "file", "audio"
	Text     string `json:"text,omitempty"`      // for type "text"
	ImageURL string `json:"image_url,omitempty"` // for type "image_url" — URL or base64 data URI
	MimeType string `json:"mime_type,omitempty"` // MIME type for image or file
	FileName string `json:"file_name,omitempty"` // original filename for attachments
	Data     []byte `json:"-"`                   // raw binary data (not serialized to JSON)
}

ContentPart represents a multi-modal content part within a message.

type EmbeddingRequest

type EmbeddingRequest struct {
	Model string   `json:"model"`
	Input []string `json:"input"`
}

EmbeddingRequest is input for embedding generation.

type EmbeddingResponse

type EmbeddingResponse struct {
	Embeddings [][]float32 `json:"embeddings"`
	Usage      Usage       `json:"usage"`
}

EmbeddingResponse contains the generated embeddings.

type EmbeddingsProvider

type EmbeddingsProvider interface {
	Embed(ctx context.Context, req *EmbeddingRequest) (*EmbeddingResponse, error)
}

EmbeddingsProvider generates vector embeddings from text.

type EstimatingCounter

type EstimatingCounter struct {
	CharsPerToken float64
}

EstimatingCounter uses a character-ratio heuristic (1 token ~ 4 chars) to estimate token counts without external dependencies.

func NewEstimatingCounter

func NewEstimatingCounter() *EstimatingCounter

NewEstimatingCounter returns a counter using the default 4-chars-per-token ratio.

func (*EstimatingCounter) CountString

func (c *EstimatingCounter) CountString(s string) int

func (*EstimatingCounter) CountTokens

func (c *EstimatingCounter) CountTokens(messages []Message) int

type FallbackProvider

type FallbackProvider struct {

	// OnFallback is called when a provider fails and the next one is tried.
	// It receives the failed provider index, its name, and the error.
	OnFallback func(index int, name string, err error)
	// contains filtered or unexported fields
}

FallbackProvider wraps multiple providers and tries them in order. If the primary provider fails, it automatically falls back to the next one. This is useful for scenarios like primary-cloud -> cheaper-model or cloud-provider -> local-ollama.

func NewFallbackProvider

func NewFallbackProvider(providers ...Provider) (*FallbackProvider, error)

NewFallbackProvider creates a fallback provider from the given providers. At least one provider is required.

func (*FallbackProvider) Chat

Chat tries each provider in order until one succeeds. It does not fall through on terminal (4xx client) errors, which will not succeed on a different provider — those are returned immediately.

func (*FallbackProvider) Model

func (f *FallbackProvider) Model() string

func (*FallbackProvider) Name

func (f *FallbackProvider) Name() string

func (*FallbackProvider) StreamChat

func (f *FallbackProvider) StreamChat(ctx context.Context, req *ChatRequest) (<-chan *ChatResponse, error)

StreamChat tries each provider in order until one succeeds. Like Chat, it aborts immediately on terminal (4xx) errors rather than trying every provider.

type FunctionDef

type FunctionDef struct {
	Name        string         `json:"name"`
	Description string         `json:"description"`
	Parameters  map[string]any `json:"parameters"` // JSON Schema
}

FunctionDef describes a callable function for the model.

type Gemini

type Gemini struct {
	// contains filtered or unexported fields
}

Gemini implements Provider for Google's Gemini API.

func NewGemini

func NewGemini(apiKey string) *Gemini

NewGemini creates a new Gemini provider with the given API key.

func NewGeminiWithConfig

func NewGeminiWithConfig(cfg ProviderConfig) *Gemini

NewGeminiWithConfig creates a Gemini provider with full configuration.

func (*Gemini) Chat

func (g *Gemini) Chat(ctx context.Context, req *ChatRequest) (*ChatResponse, error)

func (*Gemini) Model

func (g *Gemini) Model() string

func (*Gemini) Name

func (g *Gemini) Name() string

func (*Gemini) StreamChat

func (g *Gemini) StreamChat(ctx context.Context, req *ChatRequest) (<-chan *ChatResponse, error)

type GoogleEmbeddings added in v0.3.0

type GoogleEmbeddings struct {
	// contains filtered or unexported fields
}

GoogleEmbeddings implements EmbeddingsProvider using the Google AI Gemini embeddings API.

func NewGoogleEmbeddings added in v0.3.0

func NewGoogleEmbeddings(apiKey, modelID string) *GoogleEmbeddings

NewGoogleEmbeddings creates a Google AI embeddings provider. apiKey is the Google AI API key. modelID is the model identifier (e.g., "text-embedding-004", "embedding-001").

func NewGoogleEmbeddingsWithConfig added in v0.3.0

func NewGoogleEmbeddingsWithConfig(cfg ProviderConfig) *GoogleEmbeddings

NewGoogleEmbeddingsWithConfig creates a Google AI embeddings provider with full config.

func (*GoogleEmbeddings) Embed added in v0.3.0

type Message

type Message struct {
	Role       string         `json:"role"` // system, user, assistant, tool
	Content    string         `json:"content"`
	Parts      []ContentPart  `json:"parts,omitempty"` // multi-modal content parts
	Audio      []AudioContent `json:"audio,omitempty"` // audio input/output attachments
	Name       string         `json:"name,omitempty"`
	ToolCallID string         `json:"tool_call_id,omitempty"`
	ToolCalls  []ToolCall     `json:"tool_calls,omitempty"`
	// ProviderState carries opaque, provider-owned conversation items needed on
	// a later tool round (for example encrypted Responses API reasoning items).
	// The agent passes it back unchanged and never serializes or displays it.
	ProviderState any `json:"-"`
}

Message represents a chat message.

func (*Message) AddAudio added in v0.3.0

func (m *Message) AddAudio(data []byte, format string)

AddAudio adds an audio content part to the message. format is the audio format (e.g., "wav", "mp3", "ogg").

func (*Message) AddFile added in v0.3.0

func (m *Message) AddFile(filename, mimeType string, data []byte)

AddFile adds a file attachment content part to the message.

func (*Message) AddImageURL added in v0.3.0

func (m *Message) AddImageURL(url, mimeType string)

AddImageURL adds an image URL content part to the message.

type Mistral

type Mistral struct {
	// contains filtered or unexported fields
}

Mistral implements Provider for the Mistral AI API. Mistral uses an OpenAI-compatible chat completions format.

func NewMistral

func NewMistral(apiKey string) *Mistral

NewMistral creates a new Mistral provider with the given API key.

func NewMistralWithConfig

func NewMistralWithConfig(cfg ProviderConfig) *Mistral

NewMistralWithConfig creates a Mistral provider with full configuration.

func (*Mistral) Chat

func (m *Mistral) Chat(ctx context.Context, req *ChatRequest) (*ChatResponse, error)

func (*Mistral) Model

func (m *Mistral) Model() string

func (*Mistral) Name

func (m *Mistral) Name() string

func (*Mistral) StreamChat

func (m *Mistral) StreamChat(ctx context.Context, req *ChatRequest) (<-chan *ChatResponse, error)

type ModelRef added in v0.3.0

type ModelRef struct {
	Provider string
	ModelID  string
}

ModelRef represents a parsed "provider:model_id" string.

func ParseModelString added in v0.3.0

func ParseModelString(s string) (ModelRef, error)

ParseModelString parses "provider:model_id" into its components. Examples: "openai:gpt-4o", "anthropic:claude-3-5-sonnet", "azure:gpt-4o-mini"

func (ModelRef) String added in v0.3.0

func (r ModelRef) String() string

String returns the "provider:model_id" format.

type Ollama

type Ollama struct {
	// contains filtered or unexported fields
}

Ollama implements Provider for locally running Ollama models. Ollama exposes an OpenAI-compatible API at /v1/chat/completions.

func NewOllama

func NewOllama(host, modelName string) *Ollama

NewOllama creates a new Ollama provider pointing at a local instance. host is the Ollama server address (e.g., "http://localhost:11434"). modelName is the model tag (e.g., "llama3.2", "mistral", "codellama").

func NewOllamaWithConfig

func NewOllamaWithConfig(cfg ProviderConfig) *Ollama

NewOllamaWithConfig creates an Ollama provider with full configuration.

func (*Ollama) Chat

func (o *Ollama) Chat(ctx context.Context, req *ChatRequest) (*ChatResponse, error)

func (*Ollama) Model

func (o *Ollama) Model() string

func (*Ollama) Name

func (o *Ollama) Name() string

func (*Ollama) StreamChat

func (o *Ollama) StreamChat(ctx context.Context, req *ChatRequest) (<-chan *ChatResponse, error)

type OllamaEmbeddings

type OllamaEmbeddings struct {
	// contains filtered or unexported fields
}

OllamaEmbeddings implements EmbeddingsProvider using a local Ollama server.

func NewOllamaEmbeddings

func NewOllamaEmbeddings(baseURL, modelID string) *OllamaEmbeddings

NewOllamaEmbeddings creates an Ollama embeddings provider.

func (*OllamaEmbeddings) Embed

type OpenAI

type OpenAI struct {
	// contains filtered or unexported fields
}

OpenAI implements Provider for OpenAI chat completion endpoints. Works with GPT-4, GPT-4o, GPT-3.5-turbo, o1, o3, and any OpenAI-compatible API.

func NewOpenAI

func NewOpenAI(apiKey string) *OpenAI

NewOpenAI creates a new OpenAI provider with the given API key.

func NewOpenAIWithConfig

func NewOpenAIWithConfig(cfg ProviderConfig) *OpenAI

NewOpenAIWithConfig creates an OpenAI provider with full configuration.

func (*OpenAI) Chat

func (o *OpenAI) Chat(ctx context.Context, req *ChatRequest) (*ChatResponse, error)

func (*OpenAI) Model

func (o *OpenAI) Model() string

func (*OpenAI) Name

func (o *OpenAI) Name() string

func (*OpenAI) StreamChat

func (o *OpenAI) StreamChat(ctx context.Context, req *ChatRequest) (<-chan *ChatResponse, error)

type OpenAIAudio added in v0.3.0

type OpenAIAudio struct {
	// contains filtered or unexported fields
}

OpenAIAudio implements AudioProvider for OpenAI's Whisper transcription and TTS endpoints.

func NewOpenAIAudio added in v0.3.0

func NewOpenAIAudio(apiKey string) *OpenAIAudio

NewOpenAIAudio creates an OpenAIAudio provider with the given API key and default models.

func NewOpenAIAudioWithConfig added in v0.3.0

func NewOpenAIAudioWithConfig(cfg OpenAIAudioConfig) *OpenAIAudio

NewOpenAIAudioWithConfig creates an OpenAIAudio provider with full configuration.

func (*OpenAIAudio) Synthesize added in v0.3.0

func (o *OpenAIAudio) Synthesize(ctx context.Context, text, voice string) (*AudioContent, error)

Synthesize sends text to the OpenAI TTS endpoint and returns an AudioContent with the synthesized audio bytes. voice should be one of: "alloy", "echo", "fable", "onyx", "nova", "shimmer". An empty voice defaults to "alloy".

func (*OpenAIAudio) Transcribe added in v0.3.0

func (o *OpenAIAudio) Transcribe(ctx context.Context, audio AudioContent) (string, error)

Transcribe sends audio data to the OpenAI Whisper transcription endpoint and returns the transcribed text. audio.Format is used as the filename extension (e.g., "wav", "mp3"). If audio.Transcript is already set it is returned immediately without an API call.

type OpenAIAudioConfig added in v0.3.0

type OpenAIAudioConfig struct {
	// APIKey is the OpenAI API key.
	APIKey string `json:"api_key"`
	// BaseURL overrides the default OpenAI base URL.
	BaseURL string `json:"base_url,omitempty"`
	// OrgID is the optional OpenAI organization ID.
	OrgID string `json:"org_id,omitempty"`
	// TranscribeModel is the Whisper model to use for transcription (default: "whisper-1").
	TranscribeModel string `json:"transcribe_model,omitempty"`
	// TTSModel is the TTS model to use for speech synthesis (default: "tts-1").
	TTSModel string `json:"tts_model,omitempty"`
	// TTSFormat is the output audio format for TTS (default: "mp3").
	// Supported: "mp3", "opus", "aac", "flac", "wav", "pcm".
	TTSFormat string `json:"tts_format,omitempty"`
	// TimeoutSec is the HTTP request timeout in seconds (default: 120).
	TimeoutSec int `json:"timeout_sec,omitempty"`
}

OpenAIAudioConfig holds configuration for the OpenAI audio provider.

type OpenAICompatible

type OpenAICompatible struct {
	// contains filtered or unexported fields
}

OpenAICompatible implements Provider for any OpenAI-compatible API endpoint. Use this for self-hosted models (vLLM, TGI, LiteLLM), cloud providers that expose an OpenAI-compatible interface (Together, Anyscale, Fireworks, Groq, DeepSeek, OpenRouter, Perplexity), or any custom LLM server.

func NewAnyscale

func NewAnyscale(apiKey, modelID string) *OpenAICompatible

NewAnyscale creates a provider for Anyscale Endpoints.

func NewDeepSeek

func NewDeepSeek(apiKey, modelID string) *OpenAICompatible

NewDeepSeek creates a provider for DeepSeek.

func NewFireworks

func NewFireworks(apiKey, modelID string) *OpenAICompatible

NewFireworks creates a provider for Fireworks AI.

func NewGroq

func NewGroq(apiKey, modelID string) *OpenAICompatible

NewGroq creates a provider for Groq.

func NewOpenAICompatible

func NewOpenAICompatible(name, baseURL, apiKey, modelID string) *OpenAICompatible

NewOpenAICompatible creates a provider for any OpenAI-compatible endpoint. name is a human-readable identifier (e.g., "vllm", "together", "deepseek"). baseURL is the API base (e.g., "https://api.together.xyz/v1"). apiKey is the authentication key. modelID is the model identifier (e.g., "meta-llama/Llama-3.1-70B-Instruct").

func NewOpenAICompatibleWithConfig

func NewOpenAICompatibleWithConfig(name string, cfg ProviderConfig) *OpenAICompatible

NewOpenAICompatibleWithConfig creates an OpenAI-compatible provider with full config.

func NewOpenRouter

func NewOpenRouter(apiKey, modelID string) *OpenAICompatible

NewOpenRouter creates a provider for OpenRouter.

func NewPerplexity

func NewPerplexity(apiKey, modelID string) *OpenAICompatible

NewPerplexity creates a provider for Perplexity.

func NewTogether

func NewTogether(apiKey, modelID string) *OpenAICompatible

NewTogether creates a provider for Together AI.

func (*OpenAICompatible) Chat

func (*OpenAICompatible) Model

func (c *OpenAICompatible) Model() string

func (*OpenAICompatible) Name

func (c *OpenAICompatible) Name() string

func (*OpenAICompatible) StreamChat

func (c *OpenAICompatible) StreamChat(ctx context.Context, req *ChatRequest) (<-chan *ChatResponse, error)

type OpenAIEmbeddings

type OpenAIEmbeddings struct {
	// contains filtered or unexported fields
}

OpenAIEmbeddings implements EmbeddingsProvider using the OpenAI embeddings API.

func NewOpenAIEmbeddings

func NewOpenAIEmbeddings(apiKey string) *OpenAIEmbeddings

NewOpenAIEmbeddings creates an OpenAI embeddings provider with the given API key.

func NewOpenAIEmbeddingsWithConfig

func NewOpenAIEmbeddingsWithConfig(cfg ProviderConfig) *OpenAIEmbeddings

NewOpenAIEmbeddingsWithConfig creates an OpenAI embeddings provider with full config.

func (*OpenAIEmbeddings) Embed

type Provider

type Provider interface {
	// Chat sends a request and returns a complete response.
	Chat(ctx context.Context, req *ChatRequest) (*ChatResponse, error)
	// StreamChat returns a channel of partial responses for streaming.
	// The channel is closed when the response is complete.
	StreamChat(ctx context.Context, req *ChatRequest) (<-chan *ChatResponse, error)
	// Name returns a human-readable name for this provider.
	Name() string
	// Model returns the default model ID for this provider.
	Model() string
}

Provider is the interface all LLM backends must implement.

func ProviderFromString added in v0.3.0

func ProviderFromString(s string) (Provider, error)

ProviderFromString creates a Provider from a "provider:model_id" string.

type ProviderConfig

type ProviderConfig struct {
	APIKey        string `json:"api_key"`
	BaseURL       string `json:"base_url,omitempty"`
	Model         string `json:"model"`
	MaxRetries    int    `json:"max_retries,omitempty"`
	TimeoutSec    int    `json:"timeout_sec,omitempty"`
	OrgID         string `json:"org_id,omitempty"`
	ContextWindow int    `json:"context_window,omitempty"` // override default context window size for the model
}

ProviderConfig holds common configuration shared by all providers.

type ProviderFactory added in v0.3.0

type ProviderFactory func(ref ModelRef) (Provider, error)

ProviderFactory creates a Provider from a model reference. Register provider constructors here for model-as-string support.

type ReasoningConfig added in v0.11.0

type ReasoningConfig struct {
	Enabled      bool   `json:"enabled" yaml:"enabled"`
	Effort       string `json:"effort,omitempty" yaml:"effort,omitempty"` // low, medium, high
	BudgetTokens int    `json:"budget_tokens,omitempty" yaml:"budget_tokens,omitempty"`
	Summary      bool   `json:"summary,omitempty" yaml:"summary,omitempty"`
}

ReasoningConfig controls provider-native reasoning/thinking when supported. Providers ignore fields they cannot represent rather than leaking a provider-specific request shape into the agent SDK.

type StopReason

type StopReason string

StopReason indicates why the model stopped generating.

const (
	StopReasonEnd       StopReason = "end"
	StopReasonMaxTokens StopReason = "max_tokens"
	StopReasonToolCall  StopReason = "tool_call"
	StopReasonFilter    StopReason = "content_filter"
)

type SummarizationConfig

type SummarizationConfig struct {
	Threshold           float64 // fraction of context window that triggers summarization (default 0.8)
	PreserveRecentTurns int     // number of recent user/assistant pairs to keep verbatim
}

SummarizationConfig controls when and how auto-summarization triggers.

type SummarizationResult added in v0.3.0

type SummarizationResult struct {
	Summary           string
	PreservedMessages []Message
	SummarizedCount   int
}

SummarizationResult holds the output of a summarization pass.

type Summarizer

type Summarizer struct {
	// contains filtered or unexported fields
}

Summarizer compresses conversation history to fit within context windows.

func NewSummarizer

func NewSummarizer(p Provider, counter TokenCounter, cfg SummarizationConfig) *Summarizer

NewSummarizer creates a summarizer with the given provider, counter, and config.

func (*Summarizer) NeedsSummarization

func (s *Summarizer) NeedsSummarization(systemTokens int, history []Message, contextLimit int) bool

NeedsSummarization returns true if the total estimated tokens (system + history) exceed the threshold fraction of the context limit.

func (*Summarizer) Summarize

func (s *Summarizer) Summarize(ctx context.Context, existingSummary string, messages []Message) (SummarizationResult, error)

Summarize compresses older messages into a running summary. If an existing summary is provided, it is incorporated into the new summary.

type TokenCounter

type TokenCounter interface {
	CountTokens(messages []Message) int
	CountString(s string) int
}

TokenCounter estimates the token count for a set of messages.

func NewTokenCounter added in v0.4.0

func NewTokenCounter(modelName string) TokenCounter

NewTokenCounter returns the best available token counter for a model: a real BPE counter when its encoding can be loaded, otherwise the character-ratio estimator. It never returns an error, so callers can use it inline.

type ToolCall

type ToolCall struct {
	ID        string `json:"id"`
	Name      string `json:"name"`
	Arguments string `json:"arguments"` // JSON string
}

ToolCall represents a model-requested tool invocation.

type ToolDefinition

type ToolDefinition struct {
	Type     string      `json:"type"` // "function"
	Function FunctionDef `json:"function"`
}

ToolDefinition describes a tool the model can call (OpenAI function-calling format).

type Usage

type Usage struct {
	PromptTokens     int `json:"prompt_tokens"`
	CompletionTokens int `json:"completion_tokens"`
}

Usage tracks token consumption.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL