llm

package
v2.11.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 31, 2026 License: GPL-3.0 Imports: 13 Imported by: 0

Documentation

Overview

Package llm provides a generic abstraction layer for interacting with Large Language Model providers.

Index

Constants

View Source
const WebSearchToolName = "web_search"

WebSearchToolName is the canonical explicit tool name for web search.

Variables

View Source
var (
	// ErrNoAPIKey indicates that no API key was provided.
	ErrNoAPIKey = errors.New("no API key provided")
	// ErrInvalidProvider indicates an unknown or invalid provider type.
	ErrInvalidProvider = errors.New("invalid provider")
	// ErrRateLimited indicates the API rate limit has been exceeded.
	ErrRateLimited = errors.New("rate limited")
	// ErrContextTooLong indicates the input exceeds the model's context limit.
	ErrContextTooLong = errors.New("context length exceeded")
	// ErrModelNotFound indicates the requested model does not exist.
	ErrModelNotFound = errors.New("model not found")
	// ErrInvalidRequest indicates a malformed request.
	ErrInvalidRequest = errors.New("invalid request")
	// ErrUnauthorized indicates invalid or missing authentication.
	ErrUnauthorized = errors.New("unauthorized")
	// ErrServerError indicates a server-side error.
	ErrServerError = errors.New("server error")
	// ErrTimeout indicates the request timed out.
	ErrTimeout = errors.New("request timeout")
	// ErrStreamClosed indicates the stream was unexpectedly closed.
	ErrStreamClosed = errors.New("stream closed unexpectedly")
)

Sentinel errors for common LLM error conditions.

Functions

func ApplyOptions

func ApplyOptions(cfg *Config, opts ...Option)

ApplyOptions applies the given options to a Config.

func ApplyRequestOptions

func ApplyRequestOptions(req *ChatRequest, opts ...RequestOption)

ApplyRequestOptions applies the given options to a ChatRequest.

func DefaultAPIKeyEnvVar

func DefaultAPIKeyEnvVar(providerType ProviderType) string

DefaultAPIKeyEnvVar returns the standard environment variable name for the API key of a given provider.

func DefaultBaseURL

func DefaultBaseURL(providerType ProviderType) string

DefaultBaseURL returns the default API endpoint URL for a given provider.

func GetAPIKeyFromEnv

func GetAPIKeyFromEnv(providerType ProviderType) string

GetAPIKeyFromEnv attempts to retrieve the API key for a provider from environment variables.

func IsAuthError

func IsAuthError(err error) bool

IsAuthError returns true if the error is an authentication error.

func IsRateLimitError

func IsRateLimitError(err error) bool

IsRateLimitError returns true if the error is a rate limit error.

func IsRetryable

func IsRetryable(err error) bool

IsRetryable returns true if the error is retryable.

func RegisterProvider

func RegisterProvider(providerType ProviderType, factory ProviderFactory)

RegisterProvider registers a provider factory for a given type. This is typically called in the init() function of each provider package.

func ShouldRetryRequest

func ShouldRetryRequest(ctx context.Context, err error) bool

ShouldRetryRequest returns true when an LLM request failure is transient and the caller context is still active, making another logical attempt safe.

func WrapError

func WrapError(provider string, err error) error

WrapError wraps an error with provider context.

Types

type APIError

type APIError struct {
	// Provider is the name of the provider that returned the error.
	Provider string
	// StatusCode is the HTTP status code returned.
	StatusCode int
	// Message is the error message from the API.
	Message string
	// Retryable indicates whether the request can be retried.
	Retryable bool
	// Err is the underlying error, if any.
	Err error
}

APIError represents an error response from an LLM API.

func NewAPIError

func NewAPIError(provider string, statusCode int, message string) *APIError

NewAPIError creates a new APIError with the given parameters.

func (*APIError) Error

func (e *APIError) Error() string

Error returns the error message.

func (*APIError) Unwrap

func (e *APIError) Unwrap() error

Unwrap returns the underlying error.

type ChatRequest

type ChatRequest struct {
	// Model is the identifier of the model to use.
	Model string
	// Messages is the session history to send to the model.
	Messages []Message
	// Temperature controls randomness in the response (0.0 to 2.0).
	// Lower values make output more deterministic.
	Temperature *float64
	// MaxTokens is the maximum number of tokens to generate.
	MaxTokens *int
	// TopP is the nucleus sampling parameter (0.0 to 1.0).
	TopP *float64
	// Stop is a list of sequences where the model will stop generating.
	Stop []string
	// Thinking enables extended thinking/reasoning mode.
	// Provider-specific handling in each provider implementation.
	Thinking *ThinkingRequest
	// Tools is a list of tools available for the model to call.
	Tools []Tool
	// ToolChoice controls how the model uses tools.
	// Values: "auto" (default), "required", "none", or a specific tool name.
	ToolChoice string
	// WebSearch enables provider-native web search when non-nil and Enabled.
	// Provider-specific handling in each provider implementation.
	WebSearch *WebSearchRequest
}

ChatRequest contains the input for a chat completion request.

func NewChatRequest

func NewChatRequest(model string, messages []Message, opts ...RequestOption) *ChatRequest

NewChatRequest creates a new ChatRequest with the given model, messages, and options.

func NormalizeChatRequest

func NormalizeChatRequest(req *ChatRequest) *ChatRequest

NormalizeChatRequest applies provider-independent request invariants before a request reaches a concrete provider.

type ChatResponse

type ChatResponse struct {
	// Content is the generated text content.
	Content string
	// ReasoningContent holds the model's internal reasoning/thinking text.
	// Must be preserved and echoed back in the next assistant message when
	// reasoning is enabled, as required by some providers (e.g. Moonshot/KIMI).
	ReasoningContent string
	// FinishReason indicates why the model stopped generating.
	// Common values: "stop", "length", "content_filter", "tool_calls".
	FinishReason string
	// Usage contains token usage statistics.
	Usage Usage
	// ToolCalls contains tool calls requested by the model.
	// Only populated when FinishReason is "tool_calls".
	ToolCalls []ToolCall
}

ChatResponse contains the output from a chat completion request.

func ChatWithRetry

func ChatWithRetry(ctx context.Context, provider Provider, req *ChatRequest, cfg LogicalRetryConfig) (*ChatResponse, error)

ChatWithRetry executes provider.Chat with bounded logical retries for transient request failures.

type Config

type Config struct {
	// APIKey is the authentication key for the provider.
	APIKey string
	// AccountID carries provider-specific account metadata when required.
	AccountID string
	// BaseURL is the API endpoint URL. If empty, the provider's default is used.
	BaseURL string
	// OAuthCredentialProvider resolves dynamic bearer credentials on demand.
	OAuthCredentialProvider OAuthCredentialProvider
	// Timeout is the maximum time to wait for a response.
	// Default is 60 seconds if not specified unless DisableRequestTimeout is true.
	Timeout time.Duration
	// DisableRequestTimeout leaves the underlying request client without a
	// response timeout. Callers must cancel through the request context.
	DisableRequestTimeout bool

	// Retry configuration
	// MaxRetries is the maximum number of retry attempts for transient errors.
	// Default is 3 if not specified.
	MaxRetries int
	// InitialInterval is the initial backoff interval before the first retry.
	// Default is 1 second if not specified.
	InitialInterval time.Duration
	// MaxInterval is the maximum backoff interval between retries.
	// Default is 30 seconds if not specified.
	MaxInterval time.Duration
	// Multiplier is the factor by which the interval increases after each retry.
	// Default is 2.0 if not specified.
	Multiplier float64
}

Config contains configuration for an LLM provider.

func DefaultConfig

func DefaultConfig() Config

DefaultConfig returns a Config with sensible defaults.

func NewConfig

func NewConfig(opts ...Option) Config

NewConfig creates a new Config with the given options applied.

type HTTPClient

type HTTPClient struct {
	// contains filtered or unexported fields
}

HTTPClient performs HTTP requests with retry logic. Uses plain net/http instead of resty to ensure response bodies are properly closed on retries (resty + SetDoNotParseResponse leaks FDs).

func NewHTTPClient

func NewHTTPClient(cfg Config) *HTTPClient

NewHTTPClient creates a new HTTP client with the given configuration. Each client gets its own http.Transport to avoid sharing connection state across unrelated providers.

func (*HTTPClient) Do

func (c *HTTPClient) Do(ctx context.Context, url string, body []byte, headers map[string]string) (io.ReadCloser, error)

Do performs an HTTP POST request with retry logic. Returns the response body as an io.ReadCloser for streaming support. Retries on network errors, 429 (rate limit), and 5xx (server errors).

type LogicalRetryConfig

type LogicalRetryConfig struct {
	MaxAttempts     int
	InitialInterval time.Duration
	MaxInterval     time.Duration
	Multiplier      float64
}

LogicalRetryConfig configures retries for a single logical LLM request. This sits above the provider's own HTTP retries and is intended for transient failures that happen around or after a successful HTTP exchange, such as interrupted body reads or response decode failures.

func DefaultLogicalRetryConfig

func DefaultLogicalRetryConfig() LogicalRetryConfig

DefaultLogicalRetryConfig returns the standard logical request retry budget.

type Message

type Message struct {
	// Role identifies who sent the message (system, user, assistant, or tool).
	Role Role `json:"role"`
	// Content is the text content of the message.
	Content string `json:"content"`
	// ReasoningContent holds the model's internal reasoning/thinking text from a
	// previous assistant turn. Some providers (e.g. Moonshot/KIMI via OpenCode)
	// require this to be echoed back in subsequent turns when reasoning is enabled.
	ReasoningContent string `json:"reasoning_content,omitempty"`
	// Name is an optional identifier for the message sender.
	// Useful in multi-agent scenarios or for tool messages.
	Name string `json:"name,omitempty"`
	// ToolCallID is the ID of the tool call this message is responding to.
	// Required when Role is "tool".
	ToolCallID string `json:"tool_call_id,omitempty"`
	// ToolCalls contains tool calls made by the assistant.
	// Only set when Role is "assistant" and the model requests tool calls.
	ToolCalls []ToolCall `json:"tool_calls,omitempty"`
}

Message represents a single message in a session.

type OAuthCredential

type OAuthCredential struct {
	AccessToken string
	AccountID   string
}

OAuthCredential contains dynamic bearer-auth details resolved at request time.

type OAuthCredentialProvider

type OAuthCredentialProvider interface {
	ResolveCredential(ctx context.Context) (OAuthCredential, error)
}

OAuthCredentialProvider resolves dynamic bearer credentials for providers that cannot safely persist a static API key inside model configuration.

type Option

type Option func(*Config)

Option is a functional option for configuring an LLM provider.

func WithAPIKey

func WithAPIKey(apiKey string) Option

WithAPIKey sets the API key for the provider.

func WithBackoff

func WithBackoff(initial, max time.Duration, multiplier float64) Option

WithBackoff sets the backoff configuration for retries.

func WithBaseURL

func WithBaseURL(baseURL string) Option

WithBaseURL sets the base URL for the provider.

func WithMaxRetries

func WithMaxRetries(maxRetries int) Option

WithMaxRetries sets the maximum number of retry attempts.

func WithTimeout

func WithTimeout(timeout time.Duration) Option

WithTimeout sets the request timeout.

type Provider

type Provider interface {
	// Chat sends messages and returns the complete response.
	Chat(ctx context.Context, req *ChatRequest) (*ChatResponse, error)

	// ChatStream sends messages and streams the response.
	// Returns a channel that receives StreamEvents until Done is true or an error occurs.
	ChatStream(ctx context.Context, req *ChatRequest) (<-chan StreamEvent, error)

	// Name returns the provider name for logging and error messages.
	Name() string
}

Provider is the interface that all LLM providers must implement.

func NewProvider

func NewProvider(providerType ProviderType, cfg Config) (Provider, error)

NewProvider creates a new Provider instance based on the provider type and configuration.

func NewProviderFromEnv

func NewProviderFromEnv(providerType ProviderType) (Provider, error)

NewProviderFromEnv creates a new Provider using API key from environment. This is a convenience function for simple use cases.

func NewProviderWithAPIKey

func NewProviderWithAPIKey(providerType ProviderType, apiKey string) (Provider, error)

NewProviderWithAPIKey creates a new Provider with the given API key. This is a convenience function for simple use cases.

type ProviderFactory

type ProviderFactory func(cfg Config) (Provider, error)

ProviderFactory is a function type that creates a new Provider instance.

type ProviderType

type ProviderType string

ProviderType identifies the LLM provider.

const (
	// ProviderOpenAI is the OpenAI provider (GPT models).
	ProviderOpenAI ProviderType = "openai"
	// ProviderOpenAICodex is the ChatGPT/Codex subscription provider.
	ProviderOpenAICodex ProviderType = "openai-codex"
	// ProviderAnthropic is the Anthropic provider (Claude models).
	ProviderAnthropic ProviderType = "anthropic"
	// ProviderGemini is the Google Gemini provider.
	ProviderGemini ProviderType = "gemini"
	// ProviderOpenRouter is the OpenRouter provider (multi-model gateway).
	ProviderOpenRouter ProviderType = "openrouter"
	// ProviderLocal is for local OpenAI-compatible servers (Ollama, vLLM, etc).
	ProviderLocal ProviderType = "local"
	// ProviderZAI is the Z.AI provider (GLM models).
	ProviderZAI ProviderType = "zai"
	// ProviderOpenCode is the OpenCode provider (Kimi, DeepSeek, GLM, etc. via opencode.ai).
	ProviderOpenCode ProviderType = "opencode"
)

func ParseProviderType

func ParseProviderType(s string) (ProviderType, error)

ParseProviderType converts a string to a ProviderType.

type RequestOption

type RequestOption func(*ChatRequest)

RequestOption is a functional option for configuring a ChatRequest.

func WithMaxTokens

func WithMaxTokens(tokens int) RequestOption

WithMaxTokens sets the maximum tokens for the request.

func WithStop

func WithStop(stop ...string) RequestOption

WithStop sets the stop sequences for the request.

func WithTemperature

func WithTemperature(temp float64) RequestOption

WithTemperature sets the temperature for the request.

func WithTopP

func WithTopP(topP float64) RequestOption

WithTopP sets the top_p (nucleus sampling) parameter.

type Role

type Role string

Role represents the role of a message sender in a session.

const (
	// RoleSystem represents a system message that sets behavior/context.
	RoleSystem Role = "system"
	// RoleUser represents a message from the user.
	RoleUser Role = "user"
	// RoleAssistant represents a message from the AI assistant.
	RoleAssistant Role = "assistant"
	// RoleTool represents a tool/function call result.
	RoleTool Role = "tool"
)

func ParseRole

func ParseRole(s string) Role

ParseRole converts a string to a Role, with support for common aliases.

type StreamEvent

type StreamEvent struct {
	// Delta is the incremental content received in this event.
	Delta string
	// Done indicates whether the stream has completed.
	Done bool
	// Error contains any error that occurred during streaming.
	Error error
	// Usage contains token usage statistics (only set when Done is true).
	Usage *Usage
	// ToolCalls contains tool calls accumulated during streaming.
	// Only populated when Done is true and the model requested tool calls.
	ToolCalls []ToolCall
	// FinishReason indicates why the model stopped (only set when Done is true).
	FinishReason string
}

StreamEvent represents a single event in a streaming response.

type ThinkingEffort

type ThinkingEffort string

ThinkingEffort represents the reasoning depth level for thinking mode.

const (
	// ThinkingEffortLow provides quick reasoning with minimal token budget.
	ThinkingEffortLow ThinkingEffort = "low"
	// ThinkingEffortMedium provides balanced reasoning (default).
	ThinkingEffortMedium ThinkingEffort = "medium"
	// ThinkingEffortHigh provides thorough analysis with larger token budget.
	ThinkingEffortHigh ThinkingEffort = "high"
	// ThinkingEffortXHigh provides maximum reasoning depth.
	// Note: Not all providers support this level.
	ThinkingEffortXHigh ThinkingEffort = "xhigh"
)

func ParseThinkingEffort

func ParseThinkingEffort(s string) (ThinkingEffort, error)

ParseThinkingEffort validates and returns a ThinkingEffort from a string. Returns empty string for empty input (no effort specified).

type ThinkingRequest

type ThinkingRequest struct {
	// Enabled activates thinking mode.
	Enabled bool
	// Effort controls reasoning depth: low, medium, high, xhigh.
	Effort ThinkingEffort
	// BudgetTokens sets explicit token budget (provider-specific).
	BudgetTokens *int
	// IncludeInOutput includes thinking blocks in the response content.
	IncludeInOutput bool
}

ThinkingRequest contains configuration for extended thinking/reasoning mode. Each provider maps this to their native format.

type Tool

type Tool struct {
	// Type is always "function" for function tools.
	Type string `json:"type"`
	// Function contains the function definition.
	Function ToolFunction `json:"function"`
}

Tool represents a function/tool available to the LLM.

type ToolCall

type ToolCall struct {
	// ID is a unique identifier for this tool call.
	ID string `json:"id"`
	// Type is always "function" for function calls.
	Type string `json:"type"`
	// Function contains the function call details.
	Function ToolCallFunction `json:"function"`
}

ToolCall represents an LLM's request to call a tool.

type ToolCallFunction

type ToolCallFunction struct {
	// Name is the name of the function to call.
	Name string `json:"name"`
	// Arguments is a JSON string containing the function arguments.
	Arguments string `json:"arguments"`
}

ToolCallFunction contains the details of a function call.

type ToolFunction

type ToolFunction struct {
	// Name is the function name (must match [a-zA-Z0-9_-]+).
	Name string `json:"name"`
	// Description explains what the function does.
	Description string `json:"description"`
	// Parameters is a JSON Schema describing the function parameters.
	Parameters map[string]any `json:"parameters"`
}

ToolFunction describes a callable function.

type Usage

type Usage struct {
	// PromptTokens is the number of tokens in the prompt.
	PromptTokens int `json:"prompt_tokens"`
	// CompletionTokens is the number of tokens in the completion.
	CompletionTokens int `json:"completion_tokens"`
	// TotalTokens is the sum of prompt and completion tokens.
	TotalTokens int `json:"total_tokens"`
}

Usage contains token usage information from an LLM API call.

type UserLocation

type UserLocation struct {
	// City is the city name.
	City string
	// Region is the region or state.
	Region string
	// Country is the ISO country code (e.g., "US").
	Country string
	// Timezone is the IANA timezone (e.g., "America/New_York").
	Timezone string
}

UserLocation provides approximate location for search result localization.

type WebSearchRequest

type WebSearchRequest struct {
	// Enabled activates provider-native web search.
	Enabled bool
	// MaxUses limits search invocations per request.
	// Anthropic: maps to max_uses. OpenRouter: maps to max_results.
	MaxUses *int
	// AllowedDomains restricts results to these domains (Anthropic only).
	AllowedDomains []string
	// BlockedDomains excludes results from these domains (Anthropic only).
	BlockedDomains []string
	// UserLocation localizes search results.
	UserLocation *UserLocation
}

WebSearchRequest contains configuration for provider-native web search. Each provider maps this to its native format during request building.

Directories

Path Synopsis
Package allproviders imports all LLM providers to register them.
Package allproviders imports all LLM providers to register them.
providers
anthropic
Package anthropic provides an LLM provider implementation for Anthropic's Claude API.
Package anthropic provides an LLM provider implementation for Anthropic's Claude API.
gemini
Package gemini provides an LLM provider implementation for Google's Gemini API.
Package gemini provides an LLM provider implementation for Google's Gemini API.
local
Package local provides an LLM provider implementation for local OpenAI-compatible servers.
Package local provides an LLM provider implementation for local OpenAI-compatible servers.
openai
Package openai provides an LLM provider implementation for OpenAI's API.
Package openai provides an LLM provider implementation for OpenAI's API.
openrouter
Package openrouter provides an LLM provider implementation for OpenRouter's API.
Package openrouter provides an LLM provider implementation for OpenRouter's API.
zai
Package zai provides an LLM provider implementation for Z.AI's API.
Package zai provides an LLM provider implementation for Z.AI's API.
Package toolschema derives LLM function-calling parameter schemas from DAG parameter definitions.
Package toolschema derives LLM function-calling parameter schemas from DAG parameter definitions.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL