llm

package
v1.30.3 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jan 4, 2026 License: GPL-3.0 Imports: 7 Imported by: 0

Documentation

Overview

Package llm provides a generic abstraction layer for interacting with Large Language Model providers.

Index

Constants

This section is empty.

Variables

View Source
var (
	// ErrNoAPIKey indicates that no API key was provided.
	ErrNoAPIKey = errors.New("no API key provided")
	// ErrInvalidProvider indicates an unknown or invalid provider type.
	ErrInvalidProvider = errors.New("invalid provider")
	// ErrRateLimited indicates the API rate limit has been exceeded.
	ErrRateLimited = errors.New("rate limited")
	// ErrContextTooLong indicates the input exceeds the model's context limit.
	ErrContextTooLong = errors.New("context length exceeded")
	// ErrModelNotFound indicates the requested model does not exist.
	ErrModelNotFound = errors.New("model not found")
	// ErrInvalidRequest indicates a malformed request.
	ErrInvalidRequest = errors.New("invalid request")
	// ErrUnauthorized indicates invalid or missing authentication.
	ErrUnauthorized = errors.New("unauthorized")
	// ErrServerError indicates a server-side error.
	ErrServerError = errors.New("server error")
	// ErrTimeout indicates the request timed out.
	ErrTimeout = errors.New("request timeout")
	// ErrStreamClosed indicates the stream was unexpectedly closed.
	ErrStreamClosed = errors.New("stream closed unexpectedly")
)

Sentinel errors for common LLM error conditions.

Functions

func ApplyOptions

func ApplyOptions(cfg *Config, opts ...Option)

ApplyOptions applies the given options to a Config.

func ApplyRequestOptions

func ApplyRequestOptions(req *ChatRequest, opts ...RequestOption)

ApplyRequestOptions applies the given options to a ChatRequest.

func DefaultAPIKeyEnvVar

func DefaultAPIKeyEnvVar(providerType ProviderType) string

DefaultAPIKeyEnvVar returns the standard environment variable name for the API key of a given provider.

func DefaultBaseURL

func DefaultBaseURL(providerType ProviderType) string

DefaultBaseURL returns the default API endpoint URL for a given provider.

func GetAPIKeyFromEnv

func GetAPIKeyFromEnv(providerType ProviderType) string

GetAPIKeyFromEnv attempts to retrieve the API key for a provider from environment variables.

func IsAuthError

func IsAuthError(err error) bool

IsAuthError returns true if the error is an authentication error.

func IsRateLimitError

func IsRateLimitError(err error) bool

IsRateLimitError returns true if the error is a rate limit error.

func IsRetryable

func IsRetryable(err error) bool

IsRetryable returns true if the error is retryable.

func RegisterProvider

func RegisterProvider(providerType ProviderType, factory ProviderFactory)

RegisterProvider registers a provider factory for a given type. This is typically called in the init() function of each provider package.

func WrapError

func WrapError(provider string, err error) error

WrapError wraps an error with provider context.

Types

type APIError

type APIError struct {
	// Provider is the name of the provider that returned the error.
	Provider string
	// StatusCode is the HTTP status code returned.
	StatusCode int
	// Message is the error message from the API.
	Message string
	// Retryable indicates whether the request can be retried.
	Retryable bool
	// Err is the underlying error, if any.
	Err error
}

APIError represents an error response from an LLM API.

func NewAPIError

func NewAPIError(provider string, statusCode int, message string) *APIError

NewAPIError creates a new APIError with the given parameters.

func (*APIError) Error

func (e *APIError) Error() string

Error returns the error message.

func (*APIError) Unwrap

func (e *APIError) Unwrap() error

Unwrap returns the underlying error.

type ChatRequest

type ChatRequest struct {
	// Model is the identifier of the model to use.
	Model string
	// Messages is the conversation history to send to the model.
	Messages []Message
	// Temperature controls randomness in the response (0.0 to 2.0).
	// Lower values make output more deterministic.
	Temperature *float64
	// MaxTokens is the maximum number of tokens to generate.
	MaxTokens *int
	// TopP is the nucleus sampling parameter (0.0 to 1.0).
	TopP *float64
	// Stop is a list of sequences where the model will stop generating.
	Stop []string
	// Thinking enables extended thinking/reasoning mode.
	// Provider-specific handling in each provider implementation.
	Thinking *ThinkingRequest
}

ChatRequest contains the input for a chat completion request.

func NewChatRequest

func NewChatRequest(model string, messages []Message, opts ...RequestOption) *ChatRequest

NewChatRequest creates a new ChatRequest with the given model, messages, and options.

type ChatResponse

type ChatResponse struct {
	// Content is the generated text content.
	Content string
	// FinishReason indicates why the model stopped generating.
	// Common values: "stop", "length", "content_filter".
	FinishReason string
	// Usage contains token usage statistics.
	Usage Usage
}

ChatResponse contains the output from a chat completion request.

type Config

type Config struct {
	// APIKey is the authentication key for the provider.
	APIKey string
	// BaseURL is the API endpoint URL. If empty, the provider's default is used.
	BaseURL string
	// Timeout is the maximum time to wait for a response.
	// Default is 60 seconds if not specified.
	Timeout time.Duration

	// Retry configuration
	// MaxRetries is the maximum number of retry attempts for transient errors.
	// Default is 3 if not specified.
	MaxRetries int
	// InitialInterval is the initial backoff interval before the first retry.
	// Default is 1 second if not specified.
	InitialInterval time.Duration
	// MaxInterval is the maximum backoff interval between retries.
	// Default is 30 seconds if not specified.
	MaxInterval time.Duration
	// Multiplier is the factor by which the interval increases after each retry.
	// Default is 2.0 if not specified.
	Multiplier float64
}

Config contains configuration for an LLM provider.

func DefaultConfig

func DefaultConfig() Config

DefaultConfig returns a Config with sensible defaults.

func NewConfig

func NewConfig(opts ...Option) Config

NewConfig creates a new Config with the given options applied.

type HTTPClient

type HTTPClient struct {
	// contains filtered or unexported fields
}

HTTPClient performs HTTP requests with retry logic using resty.

func NewHTTPClient

func NewHTTPClient(cfg Config) *HTTPClient

NewHTTPClient creates a new HTTP client with the given configuration.

func (*HTTPClient) Do

func (c *HTTPClient) Do(ctx context.Context, url string, body []byte, headers map[string]string) (io.ReadCloser, error)

Do performs an HTTP POST request with the configured retry policy. Returns the response body as an io.ReadCloser for streaming support.

type Message

type Message struct {
	// Role identifies who sent the message (system, user, assistant, or tool).
	Role Role `json:"role"`
	// Content is the text content of the message.
	Content string `json:"content"`
	// Name is an optional identifier for the message sender.
	// Useful in multi-agent scenarios or for tool messages.
	Name string `json:"name,omitempty"`
	// ToolCallID is the ID of the tool call this message is responding to.
	// Required when Role is "tool".
	ToolCallID string `json:"tool_call_id,omitempty"`
}

Message represents a single message in a conversation.

type Option

type Option func(*Config)

Option is a functional option for configuring an LLM provider.

func WithAPIKey

func WithAPIKey(apiKey string) Option

WithAPIKey sets the API key for the provider.

func WithBackoff

func WithBackoff(initial, max time.Duration, multiplier float64) Option

WithBackoff sets the backoff configuration for retries.

func WithBaseURL

func WithBaseURL(baseURL string) Option

WithBaseURL sets the base URL for the provider.

func WithMaxRetries

func WithMaxRetries(maxRetries int) Option

WithMaxRetries sets the maximum number of retry attempts.

func WithTimeout

func WithTimeout(timeout time.Duration) Option

WithTimeout sets the request timeout.

type Provider

type Provider interface {
	// Chat sends messages and returns the complete response.
	Chat(ctx context.Context, req *ChatRequest) (*ChatResponse, error)

	// ChatStream sends messages and streams the response.
	// Returns a channel that receives StreamEvents until Done is true or an error occurs.
	ChatStream(ctx context.Context, req *ChatRequest) (<-chan StreamEvent, error)

	// Name returns the provider name for logging and error messages.
	Name() string
}

Provider is the interface that all LLM providers must implement.

func NewProvider

func NewProvider(providerType ProviderType, cfg Config) (Provider, error)

NewProvider creates a new Provider instance based on the provider type and configuration.

func NewProviderFromEnv

func NewProviderFromEnv(providerType ProviderType) (Provider, error)

NewProviderFromEnv creates a new Provider using API key from environment. This is a convenience function for simple use cases.

func NewProviderWithAPIKey

func NewProviderWithAPIKey(providerType ProviderType, apiKey string) (Provider, error)

NewProviderWithAPIKey creates a new Provider with the given API key. This is a convenience function for simple use cases.

type ProviderFactory

type ProviderFactory func(cfg Config) (Provider, error)

ProviderFactory is a function type that creates a new Provider instance.

type ProviderType

type ProviderType string

ProviderType identifies the LLM provider.

const (
	// ProviderOpenAI is the OpenAI provider (GPT models).
	ProviderOpenAI ProviderType = "openai"
	// ProviderAnthropic is the Anthropic provider (Claude models).
	ProviderAnthropic ProviderType = "anthropic"
	// ProviderGemini is the Google Gemini provider.
	ProviderGemini ProviderType = "gemini"
	// ProviderOpenRouter is the OpenRouter provider (multi-model gateway).
	ProviderOpenRouter ProviderType = "openrouter"
	// ProviderLocal is for local OpenAI-compatible servers (Ollama, vLLM, etc).
	ProviderLocal ProviderType = "local"
)

func ParseProviderType

func ParseProviderType(s string) (ProviderType, error)

ParseProviderType converts a string to a ProviderType.

type RequestOption

type RequestOption func(*ChatRequest)

RequestOption is a functional option for configuring a ChatRequest.

func WithMaxTokens

func WithMaxTokens(tokens int) RequestOption

WithMaxTokens sets the maximum tokens for the request.

func WithStop

func WithStop(stop ...string) RequestOption

WithStop sets the stop sequences for the request.

func WithTemperature

func WithTemperature(temp float64) RequestOption

WithTemperature sets the temperature for the request.

func WithTopP

func WithTopP(topP float64) RequestOption

WithTopP sets the top_p (nucleus sampling) parameter.

type Role

type Role string

Role represents the role of a message sender in a conversation.

const (
	// RoleSystem represents a system message that sets behavior/context.
	RoleSystem Role = "system"
	// RoleUser represents a message from the user.
	RoleUser Role = "user"
	// RoleAssistant represents a message from the AI assistant.
	RoleAssistant Role = "assistant"
	// RoleTool represents a tool/function call result.
	RoleTool Role = "tool"
)

func ParseRole

func ParseRole(s string) Role

ParseRole converts a string to a Role, with support for common aliases.

type StreamEvent

type StreamEvent struct {
	// Delta is the incremental content received in this event.
	Delta string
	// Done indicates whether the stream has completed.
	Done bool
	// Error contains any error that occurred during streaming.
	Error error
	// Usage contains token usage statistics (only set when Done is true).
	Usage *Usage
}

StreamEvent represents a single event in a streaming response.

type ThinkingEffort

type ThinkingEffort string

ThinkingEffort represents the reasoning depth level for thinking mode.

const (
	// ThinkingEffortLow provides quick reasoning with minimal token budget.
	ThinkingEffortLow ThinkingEffort = "low"
	// ThinkingEffortMedium provides balanced reasoning (default).
	ThinkingEffortMedium ThinkingEffort = "medium"
	// ThinkingEffortHigh provides thorough analysis with larger token budget.
	ThinkingEffortHigh ThinkingEffort = "high"
	// ThinkingEffortXHigh provides maximum reasoning depth.
	// Note: Not all providers support this level.
	ThinkingEffortXHigh ThinkingEffort = "xhigh"
)

type ThinkingRequest

type ThinkingRequest struct {
	// Enabled activates thinking mode.
	Enabled bool
	// Effort controls reasoning depth: low, medium, high, xhigh.
	Effort ThinkingEffort
	// BudgetTokens sets explicit token budget (provider-specific).
	BudgetTokens *int
	// IncludeInOutput includes thinking blocks in the response content.
	IncludeInOutput bool
}

ThinkingRequest contains configuration for extended thinking/reasoning mode. Each provider maps this to their native format.

type Usage

type Usage struct {
	// PromptTokens is the number of tokens in the prompt.
	PromptTokens int `json:"prompt_tokens"`
	// CompletionTokens is the number of tokens in the completion.
	CompletionTokens int `json:"completion_tokens"`
	// TotalTokens is the sum of prompt and completion tokens.
	TotalTokens int `json:"total_tokens"`
}

Usage contains token usage information from an LLM API call.

Directories

Path Synopsis
Package allproviders imports all LLM providers to register them.
Package allproviders imports all LLM providers to register them.
providers
anthropic
Package anthropic provides an LLM provider implementation for Anthropic's Claude API.
Package anthropic provides an LLM provider implementation for Anthropic's Claude API.
gemini
Package gemini provides an LLM provider implementation for Google's Gemini API.
Package gemini provides an LLM provider implementation for Google's Gemini API.
local
Package local provides an LLM provider implementation for local OpenAI-compatible servers.
Package local provides an LLM provider implementation for local OpenAI-compatible servers.
openai
Package openai provides an LLM provider implementation for OpenAI's API.
Package openai provides an LLM provider implementation for OpenAI's API.
openrouter
Package openrouter provides an LLM provider implementation for OpenRouter's API.
Package openrouter provides an LLM provider implementation for OpenRouter's API.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL