litellm

package module
v1.5.3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 17, 2025 License: Apache-2.0 Imports: 17 Imported by: 2

README

LiteLLM (Go) — Multi‑Provider LLM Client

中文 | English

LiteLLM is a small, typed Go client that lets you call multiple LLM providers through one API.

Get Started

Install
go get github.com/voocel/litellm
1) Set one API key
export OPENAI_API_KEY="your-key"
2) Call a model in one line
package main

import (
	"fmt"
	"log"

	"github.com/voocel/litellm"
)

func main() {
	resp, err := litellm.Quick("gpt-4o-mini", "Hello, LiteLLM!")
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(resp.Content)
}

package main

import (
	"context"
	"log"
	"os"

	"github.com/voocel/litellm"
)

func main() {
	client, err := litellm.New(
		litellm.WithOpenAI(os.Getenv("OPENAI_API_KEY")),
		litellm.WithDefaults(1024, 0.7),
	)
	if err != nil {
		log.Fatal(err)
	}

	_, _ = client.Chat(context.Background(), &litellm.Request{
		Model: "gpt-4o-mini",
		Messages: []litellm.Message{
			{Role: "user", Content: "Explain AI in one sentence."},
		},
	})
}

Notes

  • The providers subpackage is an internal implementation detail. End users should only import github.com/voocel/litellm.
  • Model strings are passed to the upstream API unchanged. Auto‑resolution only selects a provider, so prefer official model IDs.

Core API

  • New(opts...) builds a client. If you call New() with no options, it auto‑discovers providers from environment variables.
  • Request is provider‑agnostic: set Model and Messages, then optional controls like MaxTokens, Temperature, TopP, Stop, etc.
  • Chat(ctx, req) returns a unified Response.
  • Stream(ctx, req) returns a StreamReader (not goroutine‑safe). Always defer stream.Close().
Streaming (minimal)
stream, err := client.Stream(ctx, &litellm.Request{
	Model: "gpt-4o-mini",
	Messages: []litellm.Message{
		{Role: "user", Content: "Tell me a joke."},
	},
})
if err != nil {
	log.Fatal(err)
}
defer stream.Close()

for {
	chunk, err := stream.Next()
	if err != nil || chunk.Done {
		break
	}
	fmt.Print(chunk.Content)
}

Advanced Features (optional)

Each feature below works across providers. Longer runnable examples live in examples/.

Structured outputs
schema := map[string]any{
	"type": "object",
	"properties": map[string]any{
		"name": map[string]any{"type": "string"},
		"age":  map[string]any{"type": "integer"},
	},
	"required": []string{"name", "age"},
}

resp, err := client.Chat(ctx, &litellm.Request{
	Model: "gpt-4o-mini",
	Messages: []litellm.Message{{Role: "user", Content: "Generate a person."}},
	ResponseFormat: litellm.NewResponseFormatJSONSchema("person", "", schema, true),
})
_ = resp
Function calling
tools := []litellm.Tool{
	{
		Type: "function",
		Function: litellm.FunctionDef{
			Name: "get_weather",
			Parameters: map[string]any{
				"type": "object",
				"properties": map[string]any{
					"city": map[string]any{"type": "string"},
				},
				"required": []string{"city"},
			},
		},
	},
}

resp, err := client.Chat(ctx, &litellm.Request{
	Model: "gpt-4o-mini",
	Messages: []litellm.Message{{Role: "user", Content: "Weather in Tokyo?"}},
	Tools: tools,
	ToolChoice: "auto",
})
_ = resp
Reasoning models / Responses API (OpenAI)
resp, err := client.Chat(ctx, &litellm.Request{
	Model: "o3-mini",
	Messages: []litellm.Message{{Role: "user", Content: "Solve 15*8 step by step."}},
	ReasoningEffort:  "medium",
	ReasoningSummary: "auto",
	UseResponsesAPI:  true,
})
_ = resp
Retries & timeouts
client, _ := litellm.New(
	litellm.WithOpenAI(os.Getenv("OPENAI_API_KEY")),
	litellm.WithRetries(3, 1*time.Second),
	litellm.WithTimeout(60*time.Second),
)
_ = client
Provider‑specific knobs

Use Request.Extra for vendor‑specific parameters (e.g., Qwen/GLM thinking). See examples/qwen, examples/glm, and examples/bedrock.

Custom Providers

Implement litellm.Provider and register it:

type MyProvider struct {
	name   string
	config litellm.ProviderConfig
}

func (p *MyProvider) Name() string                     { return p.name }
func (p *MyProvider) Validate() error                 { return nil }
func (p *MyProvider) SupportsModel(model string) bool { return true }
func (p *MyProvider) Models() []litellm.ModelInfo {
	return []litellm.ModelInfo{
		{
			ID:              "my-model",
			Provider:        "myprovider",
			Name:            "My Model",
			MaxOutputTokens: 4096,
			Capabilities:    []litellm.ModelCapability{litellm.CapabilityChat},
		},
	}
}

func (p *MyProvider) Chat(ctx context.Context, req *litellm.Request) (*litellm.Response, error) {
	return &litellm.Response{Content: "hello", Model: req.Model, Provider: p.name}, nil
}
func (p *MyProvider) Stream(ctx context.Context, req *litellm.Request) (litellm.StreamReader, error) {
	return nil, fmt.Errorf("streaming not implemented")
}

func init() {
	litellm.RegisterProvider("myprovider", func(cfg litellm.ProviderConfig) litellm.Provider {
		return &MyProvider{name: "myprovider", config: cfg}
	})
}

Supported Providers

Builtin providers: OpenAI, Anthropic, Google Gemini, DeepSeek, Qwen (DashScope), GLM, AWS Bedrock, OpenRouter.

LiteLLM does not rewrite model IDs; it only selects a provider. Always use official model IDs.

Configuration

Environment variables for auto‑discovery:

export OPENAI_API_KEY="sk-proj-..."
export ANTHROPIC_API_KEY="sk-ant-..."
export GEMINI_API_KEY="AIza..."
export DEEPSEEK_API_KEY="sk-..."
export QWEN_API_KEY="sk-..."
export GLM_API_KEY="your-glm-key"
export OPENROUTER_API_KEY="sk-or-v1-..."
export AWS_ACCESS_KEY_ID="..."
export AWS_SECRET_ACCESS_KEY="..."
export AWS_REGION="us-east-1"

License

Apache License

Documentation

Overview

Package litellm provides a unified interface for accessing multiple Large Language Model (LLM) platforms.

Design Philosophy

LiteLLM is designed with simplicity and elegance in mind:

  • Single Entry Point: Only litellm.New() - no confusing choice between multiple APIs
  • Auto-Resolution: Models automatically resolve to correct providers (gpt-4o → OpenAI, claude → Anthropic)
  • Type-Safe Configuration: WithOpenAI(), WithAnthropic() instead of error-prone string-based config
  • Zero Configuration: Works immediately with environment variables
  • Provider-Agnostic: Same code works across all AI providers

Quick Start

The simplest way to get started is using Quick(): 

response, err := litellm.Quick("gemini-3.0-pro", "Hello, LiteLLM!")
if err != nil {
    log.Fatal(err)
}
fmt.Println(response.Content)

Full Example

For production use, create a client with explicit configuration: 

client, err := litellm.New(
    litellm.WithOpenAI(os.Getenv("OPENAI_API_KEY")),
    litellm.WithAnthropic(os.Getenv("ANTHROPIC_API_KEY")),
    litellm.WithDefaults(2048, 0.7),
    litellm.WithRetries(3, 1*time.Second),
)
if err != nil {
    log.Fatal(err)
}

response, err := client.Chat(context.Background(), &litellm.Request{
    Model: "gpt-5.1-mini",
    Messages: []litellm.Message{
        {Role: "user", Content: "Explain quantum computing"},
    },
    MaxTokens:   litellm.IntPtr(500),
    Temperature: litellm.Float64Ptr(0.8),
})

Streaming

Real-time streaming responses: 

stream, err := client.Stream(ctx, &litellm.Request{
    Model: "claude-4.5-sonnet",
    Messages: []litellm.Message{
        {Role: "user", Content: "Write a story"},
    },
})
if err != nil {
    log.Fatal(err)
}
defer stream.Close()

for {
    chunk, err := stream.Next()
    if err != nil || chunk.Done {
        break
    }
    fmt.Print(chunk.Content)
}

Function Calling

Unified tool calling across providers: 

tools := []litellm.Tool{
    {
        Type: "function",
        Function: litellm.FunctionDef{
            Name:        "get_weather",
            Description: "Get weather information",
            Parameters:  schema,
        },
    },
}

response, err := client.Chat(ctx, &litellm.Request{
    Model:      "gpt-5.1",
    Messages:   messages,
    Tools:      tools,
    ToolChoice: "auto",
})

Structured Outputs

JSON Schema validation for reliable responses: 

response, err := client.Chat(ctx, &litellm.Request{
    Model:    "gpt-4o",
    Messages: messages,
    ResponseFormat: litellm.NewResponseFormatJSONSchema(
        "person",
        "A person's profile",
        schema,
        true, // strict mode
    ),
})

Reasoning Models

Support for OpenAI o-series and other reasoning models: 

response, err := client.Chat(ctx, &litellm.Request{
    Model:            "gpt-5.1",
    Messages:         messages,
    ReasoningEffort:  "medium",
    ReasoningSummary: "detailed",
    MaxTokens:        litellm.IntPtr(1000),
})

if response.Reasoning != nil {
    fmt.Printf("Reasoning: %s\n", response.Reasoning.Summary)
}

Supported Providers

  • OpenAI: GPT-5, GPT-4o, o-series reasoning models
  • Anthropic: Claude 4/4.5 family
  • Google Gemini: Gemini 2.5/3.0 Pro/Flash
  • DeepSeek: Chat and Reasoner models
  • Qwen: Alibaba's Qwen3-Coder family
  • GLM: ZhiPu AI's GLM-4.6 family
  • OpenRouter: 200+ models from multiple providers

Package Layout

The `providers` subpackage contains builtin provider implementations and is considered an internal detail. End users should only import `litellm`. If you need a custom provider, implement `litellm.Provider` and register it with `litellm.RegisterProvider`.

Custom Providers

Extend with your own providers: 

type MyProvider struct {
    // implement litellm.Provider interface
}

litellm.RegisterProvider("myprovider", NewMyProvider)

client, err := litellm.New(
    litellm.WithProviderConfig("myprovider", config),
)

Error Handling

Structured error types with retry information: 

response, err := client.Chat(ctx, req)
if err != nil {
    if litellm.IsRateLimitError(err) {
        retryAfter := litellm.GetRetryAfter(err)
        log.Printf("Rate limited, retry after %d seconds", retryAfter)
    } else if litellm.IsRetryableError(err) {
        log.Printf("Retryable error: %v", err)
    } else {
        log.Printf("Permanent error: %v", err)
    }
}

Environment Variables

Auto-discovery uses these environment variables: 

OPENAI_API_KEY       - OpenAI API key
ANTHROPIC_API_KEY    - Anthropic API key
GEMINI_API_KEY       - Google Gemini API key
DEEPSEEK_API_KEY     - DeepSeek API key
QWEN_API_KEY         - Alibaba Qwen API key
GLM_API_KEY          - ZhiPu GLM API key
OPENROUTER_API_KEY   - OpenRouter API key

Resilience Configuration

Optional retry mechanism with exponential backoff: 

client, err := litellm.New(
    litellm.WithOpenAI(apiKey),
    litellm.WithRetries(3, 1*time.Second),
    litellm.WithTimeout(60*time.Second),
)

Thread Safety

The Client is safe for concurrent use. However, StreamReader instances are NOT thread-safe and should be used by a single goroutine at a time. Always call defer stream.Close() to prevent resource leaks.

Best Practices

1. Reuse Client instances - they maintain connection pools 2. Always defer stream.Close() when using streaming 3. Use context for cancellation and timeouts 4. Handle errors with type-specific checks 5. Use WithRetries() for production resilience

For more examples, see https://github.com/voocel/litellm/tree/main/examples

Index

Constants

View Source 
const (
	CacheTypeEphemeral  = "ephemeral"
	CacheTypePersistent = "persistent"
)

CacheControl type constants.

View Source 
const (
	ChunkTypeContent       = "content"
	ChunkTypeToolCallDelta = "tool_call_delta"
	ChunkTypeReasoning     = "reasoning"
)

Stream chunk type constants.

View Source 
const (
	ResponseFormatText       = "text"
	ResponseFormatJSONObject = "json_object"
	ResponseFormatJSONSchema = "json_schema"
)

ResponseFormat type constants.

Variables

View Source 
var DefaultRouter = NewAutoRouter().WithFallback(FallbackNone)

Default router instance (By default, do not downgrade to avoid misrouting)

Functions

func BoolPtr added in v1.2.1 

func BoolPtr(v bool) *bool

BoolPtr returns a pointer to a bool value Example: req.Store = litellm.BoolPtr(true)

func Float64Ptr 

func Float64Ptr(v float64) *float64

Float64Ptr returns a pointer to a float64 value Example: req.Temperature = litellm.Float64Ptr(0.7)

func GetRetryAfter added in v1.5.0 

func GetRetryAfter(err error) int

func IntPtr 

func IntPtr(v int) *int

IntPtr returns a pointer to an int value Example: req.MaxTokens = litellm.IntPtr(2048)

func IsAuthError added in v1.5.0 

func IsAuthError(err error) bool

func IsModelError added in v1.5.0 

func IsModelError(err error) bool

func IsNetworkError added in v1.5.0 

func IsNetworkError(err error) bool

func IsProviderRegistered added in v1.5.0 

func IsProviderRegistered(name string) bool

IsProviderRegistered checks if a provider is registered (built-in or custom)

func IsRateLimitError added in v1.5.0 

func IsRateLimitError(err error) bool

func IsRetryableError added in v1.5.0 

func IsRetryableError(err error) bool

func IsValidationError added in v1.5.0 

func IsValidationError(err error) bool

func ListRegisteredProviders added in v1.5.0 

func ListRegisteredProviders() []string

ListRegisteredProviders returns all registered provider names

func RegisterProvider 

func RegisterProvider(name string, factory ProviderFactory) error

RegisterProvider registers a custom provider factory Returns an error if the name is empty or factory is nil

func ResetDefaultClient added in v1.5.2 

func ResetDefaultClient()

ResetDefaultClient resets the default client singleton This is useful for testing or when environment variables change

func StringPtr added in v1.5.2 

func StringPtr(v string) *string

StringPtr returns a pointer to a string value Example: req.User = litellm.StringPtr("user-123")

func WrapError added in v1.5.0 

func WrapError(err error, provider string) error

Types

type CacheControl added in v1.5.0 

type CacheControl = providers.CacheControl

Core types are sourced from providers; litellm re-exports them.

func NewCacheControl added in v1.5.0 

func NewCacheControl(cacheType string, ttlSeconds ...int) *CacheControl

NewCacheControl creates a cache control with optional TTL.

func NewEphemeralCache added in v1.5.0 

func NewEphemeralCache() *CacheControl

NewEphemeralCache creates an ephemeral cache control (TTL is provider-defined, typically ~5 minutes).

func NewPersistentCache added in v1.5.0 

func NewPersistentCache(ttlSeconds int) *CacheControl

NewPersistentCache creates a persistent cache control with a custom TTL (seconds).

type Client 

type Client struct {
	// contains filtered or unexported fields
}

Client is the main LLM client

func New 

func New(opts ...ClientOption) (*Client, error)

New creates a new LiteLLM client with optional configuration

func (*Client) AddProvider 

func (c *Client) AddProvider(name string, provider Provider) error

AddProvider adds a provider to the client

func (*Client) Chat added in v1.5.0 

func (c *Client) Chat(ctx context.Context, req *Request) (*Response, error)

Chat performs a completion request

func (*Client) Models 

func (c *Client) Models() []ModelInfo

Models returns all available models

func (*Client) Providers 

func (c *Client) Providers() []string

Providers returns the names of all configured providers

func (*Client) Stream 

func (c *Client) Stream(ctx context.Context, req *Request) (StreamReader, error)

Stream performs a streaming completion request

type ClientOption 

type ClientOption func(*Client) error

ClientOption defines options for configuring the client

func WithAnthropic 

func WithAnthropic(apiKey string, baseURL ...string) ClientOption

WithAnthropic adds Anthropic provider with custom configuration

func WithBedrock added in v1.5.2 

func WithBedrock(accessKeyID, secretAccessKey string, region ...string) ClientOption

WithBedrock adds AWS Bedrock provider with custom configuration accessKeyID and secretAccessKey are AWS credentials regions is optional (defaults to "us-east-1")

func WithDeepSeek added in v1.1.0 

func WithDeepSeek(apiKey string, baseURL ...string) ClientOption

WithDeepSeek adds DeepSeek provider with custom configuration

func WithDefaults 

func WithDefaults(maxTokens int, temperature float64) ClientOption

WithDefaults sets default configuration values

func WithGLM added in v1.2.0 

func WithGLM(apiKey string, baseURL ...string) ClientOption

WithGLM adds GLM provider with custom configuration

func WithGemini 

func WithGemini(apiKey string, baseURL ...string) ClientOption

WithGemini adds Gemini provider with custom configuration

func WithOpenAI 

func WithOpenAI(apiKey string, baseURL ...string) ClientOption

WithOpenAI adds OpenAI provider with custom configuration

func WithOpenRouter added in v1.1.0 

func WithOpenRouter(apiKey string, baseURL ...string) ClientOption

WithOpenRouter adds OpenRouter provider with custom configuration

func WithProvider 

func WithProvider(name string, provider Provider) ClientOption

WithProvider adds a custom provider

func WithProviderConfig 

func WithProviderConfig(name string, config ProviderConfig) ClientOption

WithProviderConfig adds a provider using ProviderConfig

func WithQwen added in v1.1.0 

func WithQwen(apiKey string, baseURL ...string) ClientOption

WithQwen adds Qwen provider with custom configuration

func WithResilience added in v1.2.2 

func WithResilience(config ResilienceConfig) ClientOption

WithResilience sets default resilience configuration

func WithRetries added in v1.2.2 

func WithRetries(maxRetries int, initialDelay time.Duration) ClientOption

WithRetries sets retry configuration for all providers

func WithRouter added in v1.5.0 

func WithRouter(router Router) ClientOption

WithRouter sets a custom router for provider selection

func WithTimeout added in v1.2.2 

func WithTimeout(timeout time.Duration) ClientOption

WithTimeout sets request timeout for all providers

type CustomRouterFunc added in v1.5.0 

type CustomRouterFunc func(model string, providers []Provider) (Provider, error)

CustomRouterFunc allows users to provide custom routing logic

func (CustomRouterFunc) Route added in v1.5.0 

func (f CustomRouterFunc) Route(model string, providers []Provider) (Provider, error)

Route implements Router interface for CustomRouterFunc

type DefaultConfig 

type DefaultConfig struct {
	MaxTokens   int              `json:"max_tokens"`
	Temperature float64          `json:"temperature"`
	Resilience  ResilienceConfig `json:"resilience"`
}

DefaultConfig holds default configuration values

type ErrorType added in v1.5.0 

type ErrorType = providers.ErrorType

Error types and constructors are sourced from providers; this file is a thin re-export. 

const (
	ErrorTypeAuth       ErrorType = providers.ErrorTypeAuth
	ErrorTypeRateLimit  ErrorType = providers.ErrorTypeRateLimit
	ErrorTypeNetwork    ErrorType = providers.ErrorTypeNetwork
	ErrorTypeValidation ErrorType = providers.ErrorTypeValidation
	ErrorTypeProvider   ErrorType = providers.ErrorTypeProvider
	ErrorTypeTimeout    ErrorType = providers.ErrorTypeTimeout
	ErrorTypeQuota      ErrorType = providers.ErrorTypeQuota
	ErrorTypeModel      ErrorType = providers.ErrorTypeModel
	ErrorTypeInternal   ErrorType = providers.ErrorTypeInternal
)

type FallbackStrategy added in v1.5.0 

type FallbackStrategy string

FallbackStrategy defines what to do when primary routing fails 

const (
	FallbackNone  FallbackStrategy = "none"  // Fail immediately
	FallbackFirst FallbackStrategy = "first" // Use first available provider
	FallbackAny   FallbackStrategy = "any"   // Try any provider that supports the capability
	FallbackBest  FallbackStrategy = "best"  // Use provider with best capability match
)

type FunctionCall 

type FunctionCall = providers.FunctionCall

Core types are sourced from providers; litellm re-exports them.

type FunctionDef added in v1.5.0 

type FunctionDef = providers.FunctionDef

Core types are sourced from providers; litellm re-exports them.

type JSONSchema added in v1.2.1 

type JSONSchema = providers.JSONSchema

Core types are sourced from providers; litellm re-exports them.

type LiteLLMError added in v1.5.0 

type LiteLLMError = providers.LiteLLMError

func NewAuthError added in v1.5.0 

func NewAuthError(provider, message string) *LiteLLMError

func NewError added in v1.5.0 

func NewError(errorType ErrorType, message string) *LiteLLMError

func NewErrorWithCause added in v1.5.0 

func NewErrorWithCause(errorType ErrorType, message string, cause error) *LiteLLMError

func NewHTTPError added in v1.5.0 

func NewHTTPError(provider string, statusCode int, message string) *LiteLLMError

func NewModelError added in v1.5.0 

func NewModelError(provider, model, message string) *LiteLLMError

func NewNetworkError added in v1.5.0 

func NewNetworkError(provider, message string, cause error) *LiteLLMError

func NewProviderError added in v1.5.0 

func NewProviderError(provider string, errorType ErrorType, message string) *LiteLLMError

func NewRateLimitError added in v1.5.0 

func NewRateLimitError(provider, message string, retryAfter int) *LiteLLMError

func NewTimeoutError added in v1.5.0 

func NewTimeoutError(provider, message string) *LiteLLMError

func NewValidationError added in v1.5.0 

func NewValidationError(provider, message string) *LiteLLMError

type Message 

type Message = providers.Message

Core types are sourced from providers; litellm re-exports them.

func AssistantMessage added in v1.5.2 

func AssistantMessage(content string) Message

AssistantMessage creates an assistant message Example: litellm.AssistantMessage("Hello! How can I help you?")

func SystemMessage added in v1.5.2 

func SystemMessage(content string) Message

SystemMessage creates a system message Example: litellm.SystemMessage("You are a helpful assistant.")

func ToolMessage added in v1.5.2 

func ToolMessage(toolCallID, content string) Message

ToolMessage creates a tool response message Example: litellm.ToolMessage("call_abc123", `{"result": "success"}`)

func UserMessage added in v1.5.2 

func UserMessage(content string) Message

UserMessage creates a user message Example: litellm.UserMessage("Hello, AI!")

type MessageContent added in v1.5.3 

type MessageContent = providers.MessageContent

Core types are sourced from providers; litellm re-exports them.

type MessageImageURL added in v1.5.3 

type MessageImageURL = providers.MessageImageURL

Core types are sourced from providers; litellm re-exports them.

type ModelCapability 

type ModelCapability = providers.ModelCapability

Core types are sourced from providers; litellm re-exports them. 

const (
	CapabilityChat         ModelCapability = providers.CapabilityChat
	CapabilityFunctionCall ModelCapability = providers.CapabilityFunctionCall
	CapabilityVision       ModelCapability = providers.CapabilityVision
	CapabilityReasoning    ModelCapability = providers.CapabilityReasoning
	CapabilityCode         ModelCapability = providers.CapabilityCode
)

Model capability constants.

type ModelInfo 

type ModelInfo = providers.ModelInfo

Core types are sourced from providers; litellm re-exports them.

type Provider 

type Provider = providers.Provider

Provider and ProviderConfig are sourced from providers; re-exported here.

type ProviderConfig 

type ProviderConfig = providers.ProviderConfig

type ProviderFactory 

type ProviderFactory func(config ProviderConfig) Provider

ProviderFactory is used to register custom providers.

type ReasoningChunk 

type ReasoningChunk = providers.ReasoningChunk

Core types are sourced from providers; litellm re-exports them.

type ReasoningData 

type ReasoningData = providers.ReasoningData

Core types are sourced from providers; litellm re-exports them.

type Request 

type Request = providers.Request

Core types are sourced from providers; litellm re-exports them.

type ResilienceConfig added in v1.2.2 

type ResilienceConfig = providers.ResilienceConfig

ResilienceConfig and defaults are sourced from providers; re-exported here to keep the public API small.

func DefaultResilienceConfig added in v1.2.2 

func DefaultResilienceConfig() ResilienceConfig

type ResilientHTTPClient added in v1.2.2 

type ResilientHTTPClient struct {
	// contains filtered or unexported fields
}

ResilientHTTPClient wraps http.Client with retry logic

func NewResilientHTTPClient added in v1.2.2 

func NewResilientHTTPClient(config ResilienceConfig) *ResilientHTTPClient

NewResilientHTTPClient creates a new resilient HTTP client

func (*ResilientHTTPClient) Do added in v1.2.2 

func (c *ResilientHTTPClient) Do(req *http.Request) (*http.Response, error)

Do executes HTTP request with retry logic

type Response 

type Response = providers.Response

Core types are sourced from providers; litellm re-exports them.

func Quick 

func Quick(model, message string) (*Response, error)

Quick performs a quick completion with minimal configuration It uses a singleton client with auto-discovery and makes a simple completion request with a default timeout of 30 seconds.

The client is created once on first call and reused for subsequent calls, providing better performance through connection pooling.

func QuickWithTimeout added in v1.5.0 

func QuickWithTimeout(model, message string, timeout time.Duration) (*Response, error)

QuickWithTimeout performs a quick completion with a custom timeout It uses a singleton client with auto-discovery and makes a simple completion request.

The client is created once on first call and reused for subsequent calls, providing better performance through connection pooling.

type ResponseFormat added in v1.2.1 

type ResponseFormat = providers.ResponseFormat

Core types are sourced from providers; litellm re-exports them.

func NewResponseFormatJSONObject added in v1.2.1 

func NewResponseFormatJSONObject() *ResponseFormat

NewResponseFormatJSONObject creates a JSON object response format This ensures the model returns valid JSON without enforcing a specific schema

func NewResponseFormatJSONSchema added in v1.2.1 

func NewResponseFormatJSONSchema(name, description string, schema any, strict bool) *ResponseFormat

NewResponseFormatJSONSchema creates a JSON schema response format with strict validation enabled/disabled

Parameters:

  • name: Schema name (required)
  • description: Schema description (optional, can be empty)
  • schema: JSON Schema definition as a map[string]interface{}
  • strict: Enable strict schema validation (OpenAI only)

Example: 

schema := map[string]interface{}{
    "type": "object",
    "properties": map[string]interface{}{
        "name": map[string]interface{}{"type": "string"},
        "age": map[string]interface{}{"type": "integer"},
    },
    "required": []string{"name", "age"},
}
format := litellm.NewResponseFormatJSONSchema("person", "A person object", schema, true)

func NewResponseFormatText added in v1.2.1 

func NewResponseFormatText() *ResponseFormat

NewResponseFormatText creates a text response format

type ResponsesParams added in v1.5.3 

type ResponsesParams = providers.ResponsesParams

Core types are sourced from providers; litellm re-exports them.

type RouteStrategy added in v1.5.0 

type RouteStrategy string

RouteStrategy defines different routing strategies 

const (
	StrategyAuto       RouteStrategy = "auto"        // Intelligent routing based on context
	StrategyExact      RouteStrategy = "exact"       // Exact model match required
	StrategyFirst      RouteStrategy = "first"       // Use first available provider
	StrategyRoundRobin RouteStrategy = "round_robin" // Round-robin selection
)

type Router added in v1.5.0 

type Router interface {
	Route(model string, availableProviders []Provider) (Provider, error)
}

Router interface defines how to select a provider for a given model

func RouteByProviderName added in v1.5.0 

func RouteByProviderName(providerName string) Router

RouteByProviderName creates a router that selects provider by name

func RouteToProvider added in v1.5.0 

func RouteToProvider(provider Provider) Router

RouteToProvider creates a simple router that always returns a specific provider

type SmartRouter added in v1.5.0 

type SmartRouter struct {
	// contains filtered or unexported fields
}

SmartRouter implements intelligent routing logic

func NewAutoRouter added in v1.5.0 

func NewAutoRouter() *SmartRouter

NewAutoRouter creates a router with intelligent automatic routing

func NewExactRouter added in v1.5.0 

func NewExactRouter() *SmartRouter

NewExactRouter creates a router that requires exact model matches

func NewFirstRouter added in v1.5.0 

func NewFirstRouter() *SmartRouter

NewFirstRouter creates a router that uses the first matching provider

func NewRoundRobinRouter added in v1.5.0 

func NewRoundRobinRouter() *SmartRouter

NewRoundRobinRouter creates a router that distributes requests round-robin

func NewSmartRouter added in v1.5.0 

func NewSmartRouter(strategy RouteStrategy) *SmartRouter

NewSmartRouter creates a new smart router

func (*SmartRouter) Route added in v1.5.0 

func (r *SmartRouter) Route(model string, availableProviders []Provider) (Provider, error)

Route implements the Router interface

func (*SmartRouter) WithFallback added in v1.5.0 

func (r *SmartRouter) WithFallback(fallback FallbackStrategy) *SmartRouter

WithFallback sets the fallback strategy

type StreamChunk 

type StreamChunk = providers.StreamChunk

Core types are sourced from providers; litellm re-exports them.

type StreamReader 

type StreamReader = providers.StreamReader

Core types are sourced from providers; litellm re-exports them.

type Tool 

type Tool = providers.Tool

Core types are sourced from providers; litellm re-exports them.

type ToolCall 

type ToolCall = providers.ToolCall

Core types are sourced from providers; litellm re-exports them.

type ToolCallDelta 

type ToolCallDelta = providers.ToolCallDelta

Core types are sourced from providers; litellm re-exports them.

type Usage 

type Usage = providers.Usage

Core types are sourced from providers; litellm re-exports them.

Source Files

View all Source files

Directories

Path Synopsis
examples
anthropic command
bedrock command
deepseek command
gemini command
glm command
openai command
openrouter command
qwen command
routing command
otel module

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.