litellm

package module
v1.9.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 29, 2026 License: Apache-2.0 Imports: 14 Imported by: 2

README

LiteLLM Go

中文 | English

LiteLLM is a small, explicit Go SDK for calling LLM providers through one typed core model. The root package owns the provider-agnostic API; concrete providers live in provider/<name> subpackages.

The SDK maps structure only. It does not infer what a model supports, does not validate vendor values locally, and does not rewrite your input: the vendor API decides, and its error is returned as a typed *litellm.Error.

Install

go get github.com/voocel/litellm

Requires Go 1.26 or newer.

Quick Start

package main

import (
	"context"
	"fmt"
	"log"
	"os"

	"github.com/voocel/litellm"
	"github.com/voocel/litellm/provider/openai"
)

func main() {
	provider, err := openai.New(openai.Config{APIKey: os.Getenv("OPENAI_API_KEY")})
	if err != nil {
		log.Fatal(err)
	}
	client, err := litellm.New(provider)
	if err != nil {
		log.Fatal(err)
	}

	resp, err := client.Chat(context.Background(), litellm.Request{
		Model: "gpt-5.6",
		Messages: []litellm.Message{
			litellm.System("You are concise."),
			litellm.UserText("Explain Go interfaces in one sentence."),
		},
		MaxTokens: new(120),
	})
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(resp.Text())
}

A provider is safe to share across clients. litellm.New accepts ClientOptions such as WithObservers, WithStreamIdleTimeout and WithCaptureRawResponse.

Core Model

Messages and responses use ordered Block values: TextBlock, ImageBlock, ReasoningBlock, ToolUseBlock, ToolResultBlock and ToolReferenceBlock. Response.Blocks is the canonical content; Text(), Reasoning() and ToolCalls() are views.

msgs := []litellm.Message{
	litellm.User(litellm.Text("What is in this image?"), litellm.ImageURL("https://example.com/cat.png")),
}

For multi-turn tool workflows, append the previous response blocks as they are, and keep them whole when you store history:

msgs = append(msgs,
	litellm.Assistant(resp.Blocks...),
	litellm.ToolResultText("call_1", `{"ok":true}`),
)

Data a vendor needs back, such as a reasoning signature or an item id, travels in a block's State and is sent only to the provider that produced it, so history can move between providers (details). The Client checks the structure of messages and never rewrites history. Tool-call pairing and repair are conversation policy, owned by the layer that manages the session.

Raw provider response bodies are kept only with litellm.WithCaptureRawResponse(true).

Streaming

A stream is a sequence of blocks. BlockStart opens the block at Index, the position it takes in Response.Blocks; TextDelta, ReasoningDelta and ToolUseDelta grow it; BlockEnd closes it with the completed block, including metadata that arrives late, such as State. Blocks may interleave, and all end before DoneEvent. UsageEvent, WarningEvent and ProviderEvent (native events without a typed equivalent) carry the rest.

Handle aggregates a stream and passes each event to a callback; Collect only aggregates. On failure both return the partial response with the error.

stream, err := client.Stream(ctx, litellm.Request{
	Model:    "gpt-5.6",
	Messages: []litellm.Message{litellm.UserText("Tell me a short joke.")},
})
if err != nil {
	log.Fatal(err)
}
defer stream.Close()

resp, err := litellm.Handle(stream, func(event litellm.Event) error {
	switch e := event.(type) {
	case litellm.ReasoningDelta:
		fmt.Print(e.Text)
	case litellm.TextDelta:
		fmt.Print(e.Text)
	}
	return nil
})

Client.Stream aggregates as events are read, so Handle and Collect return the complete response even after some events were read with Next. A stream is consumed by one goroutine. WithStreamIdleTimeout sets an optional per-event idle timeout.

Tools

tool, err := litellm.NewTool("get_weather", "Get weather for a city.", map[string]any{
	"type": "object",
	"properties": map[string]any{
		"city": map[string]any{"type": "string"},
	},
	"required": []string{"city"},
})
if err != nil {
	log.Fatal(err)
}
tool.Strict = litellm.StrictEnabled

resp, err := client.Chat(ctx, litellm.Request{
	Model:      "gpt-5.6",
	Messages:   []litellm.Message{litellm.UserText("Weather in Paris?")},
	Tools:      []litellm.Tool{tool},
	ToolChoice: &litellm.ToolChoice{Mode: litellm.ToolChoiceAuto},
})

ToolChoice takes a Mode (Auto, None, Required) or a tool Name; nil leaves the vendor default.

Structured Output

format, err := litellm.NewResponseFormatJSONSchema("person", "", map[string]any{
	"type": "object",
	"properties": map[string]any{
		"name": map[string]any{"type": "string"},
	},
	"required": []string{"name"},
}, litellm.StrictEnabled)
if err != nil {
	log.Fatal(err)
}

resp, err := client.Chat(ctx, litellm.Request{
	Model:          "gpt-5.6",
	Messages:       []litellm.Message{litellm.UserText("Generate a person.")},
	ResponseFormat: format,
})

Thinking

Thinking == nil sends no thinking fields and keeps the vendor default. Otherwise the zero Mode enables thinking and ThinkingDisabled turns it off; Effort and BudgetTokens are sent as given, and IncludeOutput asks for reasoning text where the vendor makes it optional.

resp, err := client.Chat(ctx, litellm.Request{
	Model:     "claude-sonnet-5",
	Messages:  []litellm.Message{litellm.UserText("Explain the tradeoffs.")},
	MaxTokens: new(2048),
	Thinking:  &litellm.Thinking{Effort: "low"},
})

Which values a model accepts is the vendor's decision. providers.md lists the exact wire mapping per provider.

Prompt Caching

Mark a cache breakpoint on a block; the prompt prefix up to and including it may be cached. TTL is passed as is (litellm.CacheTTL5m, litellm.CacheTTL1h, or empty for the vendor default). Breakpoints are hints: providers without a slot drop them.

litellm.User(litellm.TextBlock{Text: longDocument, Cache: &litellm.CacheControl{TTL: litellm.CacheTTL1h}})

Provider Options

Request.ProviderOptions carries native wire fields: each key is a top-level field of the vendor's request body, and its value is JSON. Keys are checked against the provider's list and rejected when unknown. When a key names a field the adapter also generates, an object is merged into it and an array is appended to it; any other collision is an error.

options, err := litellm.NewProviderOptions(map[string]any{
	openai.ProviderOptionPromptCacheKey: "session-42",
	openai.ProviderOptionServiceTier:    "flex",
})
if err != nil {
	log.Fatal(err)
}
resp, err := client.Chat(ctx, litellm.Request{
	Model:           "gpt-5.6",
	Messages:        []litellm.Message{litellm.UserText("Hello")},
	ProviderOptions: options,
})

Providers

Package API
provider/openai OpenAI Chat Completions (default) or Responses
provider/anthropic Anthropic Messages
provider/gemini Gemini generateContent
provider/bedrock Amazon Bedrock Converse (SigV4)
provider/deepseek, glm, grok, mimo, minimax, ollama, openrouter, qwen each vendor's Chat Completions dialect
provider/compat any other OpenAI-compatible endpoint (vLLM, LM Studio, gateways)
anthropic.New(anthropic.Config{APIKey: os.Getenv("ANTHROPIC_API_KEY")})
gemini.New(gemini.Config{APIKey: os.Getenv("GEMINI_API_KEY")})
ollama.New(ollama.Config{})
compat.New(compat.Config{BaseURL: "http://localhost:8000/v1"})

bedrock.New(bedrock.Config{
	Region: "us-east-1",
	Credentials: bedrock.StaticCredentials(
		os.Getenv("AWS_ACCESS_KEY_ID"),
		os.Getenv("AWS_SECRET_ACCESS_KEY"),
		os.Getenv("AWS_SESSION_TOKEN"),
	),
})

Applications that choose a provider from configuration build it by name; provider.Config holds the shared settings:

import "github.com/voocel/litellm/provider"

provider, err := provider.New("anthropic", provider.Config{APIKey: os.Getenv("ANTHROPIC_API_KEY")})
names := provider.Names() // "anthropic", "bedrock", "compat", ...

openai follows the official protocol only. compat is for any other OpenAI-compatible server and passes provider options through unchecked, since it cannot know the server's fields.

client.Capabilities() reports what the adapter can express: whether Thinking, ThinkingDisabled, Effort and BudgetTokens are sent, whether MaxTokens is required, and the accepted option keys. It is static per provider (ok is false for a custom provider that declares nothing); whether a model honors a request is still the vendor's call.

OpenAI Responses

Set openai.Config.API = openai.APIResponses to route Chat and Stream through the Responses API with the same request and response types. Native Responses fields are provider options; using an option of the other API is an error.

provider, err := openai.New(openai.Config{APIKey: os.Getenv("OPENAI_API_KEY"), API: openai.APIResponses})

options, err := litellm.NewProviderOptions(map[string]any{
	openai.ProviderOptionPreviousResponseID: "resp_123",
	openai.ProviderOptionTools:              []any{map[string]any{"type": "web_search"}},
})

Model Listing

models, err := client.ListModels(ctx)

Available when the provider implements ModelLister. Returned fields are best-effort.

Errors

HTTP failures and errors inside a stream are classified the same way; check with the Is* helpers instead of matching messages:

switch {
case litellm.IsContextOverflowError(err):
	// compact history and resend
case litellm.IsRateLimitError(err), litellm.IsOverloadedError(err):
	time.Sleep(litellm.RetryAfter(err)) // 0 when the provider sent no Retry-After
case litellm.IsContentFilterError(err):
	// do not retry
}

Messages render as provider: code: message. Context overflow and content filtering are detected from vendor codes and messages even when a proxy rewrites the status; they are never marked temporary.

Retry

Providers never retry. Error.Temporary describes a possibly transient failure, not permission to replay: the vendor may already have processed and billed the request. To opt in, wrap the HTTP client you pass to the provider:

import "github.com/voocel/litellm/retry"

provider, err := openai.New(openai.Config{
	APIKey:     os.Getenv("OPENAI_API_KEY"),
	HTTPClient: retry.NewHTTPClient(nil, retry.DefaultPolicy()),
})
  • Retried: complete 408, 429, 500, 502, 503, 504 and 529 responses, unless the body shows exhausted quota, an auth failure, content filtering or context overflow.
  • Not retried, returning the original response or error: network failures, interrupted streams, requests whose body cannot be resent, and, with RespectRetryAfter (on in DefaultPolicy), a Retry-After beyond MaxRetryAfter (60s by default).
  • Bedrock: the retried request reuses its SigV4 signature, which stays valid for five minutes.

Observers And OTel

An Observer starts a CallObserver for every Chat/Stream invocation, including local validation failures. CallInfo.Request is a snapshot of the caller's request, shared by all observers. OnEvent receives stream events (and Chat warnings) as they arrive; End runs once with the status (completed, failed, canceled, closed), duration, error and the final or partial response. Consume streams to termination or Close them.

type logCall struct{ info litellm.CallInfo }

func (c logCall) OnEvent(litellm.Event) {}
func (c logCall) End(r litellm.CallResult) {
	log.Printf("%s/%s: %s in %s, err=%v", c.info.Provider, c.info.Request.Model, r.Status, r.Duration, r.Err)
}

observer := litellm.ObserverFunc(func(ctx context.Context, info litellm.CallInfo) (context.Context, litellm.CallObserver) {
	return ctx, logCall{info}
})
client, err := litellm.New(provider, litellm.WithObservers(observer))

The optional github.com/voocel/litellm/otel module creates a GenAI semantic-convention span per call and propagates its context to the transport. Content capture is off by default; enable it explicitly, since messages may contain user data:

import litellmotel "github.com/voocel/litellm/otel"

observer := litellmotel.New(tracer, litellmotel.WithCaptureContent(true))

Usage And Model Catalog

Token counts are *int: nil is unknown, new(0) a known zero. Input(), Output(), Total(), Reasoning(), CacheRead() and CacheWrite() return (count, known). Input includes cache reads and writes; output includes reasoning; detail counts are subsets.

The catalog holds model facts (context window, output limit, reasoning support and prices) from LiteLLM's model list, and never loads remote data implicitly:

import (
	"fmt"

	"github.com/voocel/litellm/catalog"
)

var models catalog.Catalog
if err := models.LoadFromURL(ctx, catalog.DefaultURL); err != nil {
	return err
}

model, ok := models.Get("anthropic/claude-sonnet-4-5") // exact model-list key
if !ok {
	return fmt.Errorf("model not found in catalog")
}
if model.Pricing == nil {
	return fmt.Errorf("model pricing is unavailable")
}

cost, err := model.Pricing.Cost(resp.Usage)
if err != nil {
	return err
}

Names are the list's keys, whose vendor prefixes follow LiteLLM's provider names, such as xai/, zai/ and dashscope/, rather than provider.Names(). The catalog does not translate them, since one model may be listed under several sites at different prices; keep the catalog name of each model you price. provider.CatalogName(name, model) gives the name for a built-in provider, such as xai/grok-4 for grok.

Get matches the complete key exactly. If vendor/model is absent, it returns ok == false even when model exists.

ok == false means the model is absent; Pricing == nil means its price is unknown. A non-nil Pricing with zero rates means usage is free.

Model.Reasoning is *bool: nil means unknown, false means unsupported, and true means supported. Token limits of zero are unknown. Set and model-list loading validate names, limits and rates; a failed load leaves the catalog unchanged. Providers never consult the catalog to change requests.

Custom Providers

Implement the provider interface; CapabilityProvider and ModelLister are optional:

type Provider interface {
	Name() string
	Chat(context.Context, *litellm.Request) (*litellm.Response, error)
	Stream(context.Context, *litellm.Request) (litellm.Stream, error)
}

License

Apache License

Documentation

Overview

Package litellm provides a small, explicit multi-provider LLM SDK core.

The root package owns the provider-agnostic domain model: Request, Response, Message, Block, Stream, Event, structured errors, warnings, and observers. Concrete providers live in the subpackages of package provider, which builds them by name; provider/compat connects any OpenAI-compatible endpoint. catalog and retry are optional utilities.

Quick Start

Create a provider with its package-specific config, then bind a Client:

import (
    "context"
    "fmt"
    "os"

    "github.com/voocel/litellm"
    "github.com/voocel/litellm/provider/anthropic"
)

provider, err := anthropic.New(anthropic.Config{
    APIKey: os.Getenv("ANTHROPIC_API_KEY"),
})
if err != nil {
    panic(err)
}
client, err := litellm.New(provider)
if err != nil {
    panic(err)
}

maxTokens := 1024
resp, err := client.Chat(context.Background(), litellm.Request{
    Model:     "claude-sonnet-4-5",
    MaxTokens: &maxTokens,
    Messages:  []litellm.Message{litellm.UserText("Explain AI in one sentence.")},
})
if err != nil {
    panic(err)
}
fmt.Println(resp.Text())

Blocks

Message and Response content is represented as ordered Blocks. This preserves the order of text, reasoning, tool use, and tool results across multi-turn agent workflows. Data a vendor needs back, such as a reasoning signature, travels in a block's ProviderState and is sent only to the provider that produced it.

Streaming

A stream is a sequence of blocks. BlockStart opens the block at Index, the position it takes in Response.Blocks; deltas of the same kind grow it; BlockEnd closes it with the completed block. Blocks may interleave, and all end before DoneEvent. Handle aggregates the stream and passes each event to a callback; Collect only aggregates. Stream is for single-goroutine consumption.

stream, err := client.Stream(ctx, req)
if err != nil {
    panic(err)
}
defer stream.Close()

resp, err := litellm.Handle(stream, func(event litellm.Event) error {
    if e, ok := event.(litellm.TextDelta); ok {
        fmt.Print(e.Text)
    }
    return nil
})

Design

The SDK is intentionally not a gateway, router, agent runtime, account system, or request scheduler. It binds one Client to one Provider and exposes explicit configuration and structural validation. It maps structure only: it does not infer model features, validate vendor values locally, or rewrite user input; the vendor API is the authority. ProviderOptions carry native wire fields, and optional usage counters distinguish unknown from zero.

Index

Constants

View Source
const (
	CacheTTL5m = "5m"
	CacheTTL1h = "1h"
)

Common CacheControl TTL values.

Variables

This section is empty.

Functions

func IsAuthError added in v1.5.0

func IsAuthError(err error) bool

IsAuthError reports whether err wraps an *Error of type ErrorTypeAuth.

func IsContentFilterError added in v1.8.6

func IsContentFilterError(err error) bool

IsContentFilterError reports whether err wraps an *Error of type ErrorTypeContentFilter.

func IsContextOverflowError added in v1.6.0

func IsContextOverflowError(err error) bool

IsContextOverflowError reports whether err wraps an *Error of type ErrorTypeContextOverflow.

func IsInternalError added in v1.9.0

func IsInternalError(err error) bool

IsInternalError reports whether err wraps an *Error of type ErrorTypeInternal.

func IsModelError added in v1.5.0

func IsModelError(err error) bool

IsModelError reports whether err wraps an *Error of type ErrorTypeModel.

func IsNetworkError added in v1.5.0

func IsNetworkError(err error) bool

IsNetworkError reports whether err wraps an *Error of type ErrorTypeNetwork.

func IsOverloadedError added in v1.6.3

func IsOverloadedError(err error) bool

IsOverloadedError reports whether err wraps an *Error of type ErrorTypeOverloaded.

func IsProviderError added in v1.8.0

func IsProviderError(err error) bool

IsProviderError reports whether err wraps an *Error of type ErrorTypeProvider.

func IsQuotaError added in v1.9.0

func IsQuotaError(err error) bool

IsQuotaError reports whether err wraps an *Error of type ErrorTypeQuota.

func IsRateLimitError added in v1.5.0

func IsRateLimitError(err error) bool

IsRateLimitError reports whether err wraps an *Error of type ErrorTypeRateLimit.

func IsStreamIdleError added in v1.6.8

func IsStreamIdleError(err error) bool

IsStreamIdleError reports whether err is the WithStreamIdleTimeout failure.

func IsTemporaryError added in v1.9.0

func IsTemporaryError(err error) bool

IsTemporaryError reports whether the failure may resolve over time. It does not establish that repeating the operation is safe, even before any output.

func IsTimeoutError added in v1.8.0

func IsTimeoutError(err error) bool

IsTimeoutError reports whether err wraps an *Error of type ErrorTypeTimeout.

func IsValidationError added in v1.5.0

func IsValidationError(err error) bool

IsValidationError reports whether err wraps an *Error of type ErrorTypeValidation.

func RetryAfter added in v1.9.0

func RetryAfter(err error) time.Duration

RetryAfter returns the server-suggested delay carried by err, or zero.

func WrapError added in v1.5.0

func WrapError(provider string, fallback ErrorType, err error) error

WrapError attributes err to provider. Existing *Error values keep their type and gain the provider when unset; context cancellation and deadlines become network and timeout errors; anything else becomes a fallback-typed error.

Types

type Annotation added in v1.8.0

type Annotation struct {
	Type  string
	Text  string
	URL   string
	Extra json.RawMessage
}

Annotation is a citation attached to text. Extra keeps the vendor entry verbatim.

type Block added in v1.8.0

type Block interface {
	// contains filtered or unexported methods
}

Block is one piece of message content. The implementations are TextBlock, ImageBlock, ReasoningBlock, ToolUseBlock, ToolResultBlock and ToolReferenceBlock.

type BlockEnd added in v1.9.0

type BlockEnd struct {
	Index int
	Block Block
}

BlockEnd closes a block. Providers set Block only to deliver metadata that arrives late (State, annotations, logprobs, a tool's id and name): its non-empty metadata fields replace the started ones, while its text and arguments are ignored. Streams read through Handle, Collect or a Client deliver the completed block instead.

type BlockStart added in v1.9.0

type BlockStart struct {
	Index int
	Block Block
}

BlockStart opens a TextBlock, ReasoningBlock or ToolUseBlock. Block carries the metadata known up front, such as a tool's ID and Name, and may carry initial text.

type CacheControl added in v1.5.0

type CacheControl struct {
	TTL string
}

CacheControl marks a cache breakpoint: the prompt prefix up to and including this block may be cached. TTL is passed to the vendor as is; empty selects the vendor default.

type CallInfo added in v1.9.0

type CallInfo struct {
	Provider  string
	Streaming bool
	StartedAt time.Time
	Request   *Request
}

CallInfo describes the caller's request before validation. Request is a snapshot shared by all observers: read it, never modify it.

type CallObserver added in v1.9.0

type CallObserver interface {
	OnEvent(Event)
	End(CallResult)
}

CallObserver belongs to one invocation. Events and the Response are snapshots shared by all observers and isolated from the caller: read, never modify. OnEvent receives stream events as they arrive, and Chat warnings as WarningEvent. End runs exactly once; observers end in reverse registration order so nested observation scopes unwind correctly. Streams must be consumed to termination or explicitly closed; cancellation alone does not run callbacks in a background goroutine.

type CallResult added in v1.9.0

type CallResult struct {
	Status   CallStatus
	Response *Response
	Err      error
	Duration time.Duration
}

CallResult describes SDK execution, not errors in application consumer callbacks. Response may be partial on failure or early Close. A later resource cleanup error does not revise an already completed invocation; Close returns that error.

type CallStatus added in v1.9.0

type CallStatus string

CallStatus is how an invocation ended. CallClosed means the caller closed a stream before it finished.

const (
	CallCompleted CallStatus = "completed"
	CallFailed    CallStatus = "failed"
	CallCanceled  CallStatus = "canceled"
	CallClosed    CallStatus = "closed"
)

type Capabilities added in v1.8.1

type Capabilities struct {
	// Thinking reports that Request.Thinking is sent to the vendor.
	Thinking bool
	// DisableThinking, ThinkingEffort and ThinkingBudget report that
	// ThinkingDisabled, Thinking.Effort and Thinking.BudgetTokens are sent;
	// otherwise the adapter rejects them before sending.
	DisableThinking bool
	ThinkingEffort  bool
	ThinkingBudget  bool
	// MaxTokensRequired reports that the vendor rejects requests without
	// Request.MaxTokens.
	MaxTokensRequired bool
	// ProviderOptions lists the accepted Request.ProviderOptions keys, sorted.
	// A provider that passes every key through, such as compat, lists none.
	ProviderOptions []string
}

Capabilities states what a provider adapter can express on the wire. It is static per provider: whether a model honors a request is the vendor's call, and the vendor's error is the source of truth.

type CapabilityProvider added in v1.8.1

type CapabilityProvider interface {
	Capabilities() Capabilities
}

CapabilityProvider is implemented by providers that declare Capabilities.

type Client

type Client struct {
	// contains filtered or unexported fields
}

Client runs requests through a Provider. It copies and validates each request, attributes errors to the provider, and notifies Observers. A Client is safe for concurrent use.

func New

func New(provider Provider, opts ...ClientOption) (*Client, error)

New returns a Client for provider.

func (*Client) Capabilities added in v1.8.1

func (c *Client) Capabilities() (caps Capabilities, ok bool)

Capabilities reports what the provider adapter can express on the wire; ok is false when the provider does not implement CapabilityProvider, so the facts are unknown.

func (*Client) Chat added in v1.5.0

func (c *Client) Chat(ctx context.Context, req Request) (*Response, error)

Chat sends req and returns the complete response. On error the response may still be returned when the provider produced one.

func (*Client) ListModels added in v1.5.5

func (c *Client) ListModels(ctx context.Context) ([]ModelInfo, error)

ListModels lists the provider's models; it fails with a validation error when the provider is not a ModelLister.

func (*Client) ProviderName added in v1.5.7

func (c *Client) ProviderName() string

ProviderName returns the provider's Name.

func (*Client) Stream

func (c *Client) Stream(ctx context.Context, req Request) (Stream, error)

Stream sends req and returns its event stream, which the caller must Close. Consume it with Next, Handle or Collect.

type ClientOption

type ClientOption func(*Client) error

ClientOption configures a Client.

func WithCaptureRawResponse added in v1.8.0

func WithCaptureRawResponse(enabled bool) ClientOption

WithCaptureRawResponse keeps the raw vendor body in Response.Raw.

func WithObservers added in v1.9.0

func WithObservers(observers ...Observer) ClientOption

WithObservers adds observers, skipping nil ones.

func WithStreamIdleTimeout added in v1.8.0

func WithStreamIdleTimeout(timeout time.Duration) ClientOption

WithStreamIdleTimeout fails a stream, with an error IsStreamIdleError reports, when no event arrives for timeout. Zero disables the check.

type DoneEvent added in v1.8.0

type DoneEvent struct {
	FinishReason    FinishReason
	FinishReasonRaw string
	Provider        string
	Model           string
}

DoneEvent ends a successful stream.

type Error added in v1.9.0

type Error struct {
	Type       ErrorType
	Code       string
	Message    string
	Provider   string
	StatusCode int
	// Temporary describes a potentially transient failure, not permission to replay
	// a request. The provider may already have processed or billed the operation.
	Temporary bool
	// RetryAfter is the server-suggested delay; zero means unspecified.
	RetryAfter time.Duration
	Cause      error
}

Error is the error type returned by the client and all providers.

func NewError added in v1.5.0

func NewError(provider string, errorType ErrorType, message string, cause error) *Error

NewError builds an error of the given type. provider may be empty and cause nil; Temporary follows the type.

func NewNetworkError added in v1.5.0

func NewNetworkError(provider, message string, cause error) *Error

NewNetworkError builds a transport error. Context cancellation is a non-temporary network error and a deadline a timeout; other causes are temporary.

func (*Error) Error added in v1.9.0

func (e *Error) Error() string

Error renders "provider: code: message". Messages carry no provider prefix.

func (*Error) Unwrap added in v1.9.0

func (e *Error) Unwrap() error

Unwrap returns Cause.

type ErrorType added in v1.5.0

type ErrorType string

ErrorType classifies an Error.

const (
	ErrorTypeAuth            ErrorType = "auth"
	ErrorTypeRateLimit       ErrorType = "rate_limit"
	ErrorTypeNetwork         ErrorType = "network"
	ErrorTypeValidation      ErrorType = "validation"
	ErrorTypeProvider        ErrorType = "provider"
	ErrorTypeTimeout         ErrorType = "timeout"
	ErrorTypeQuota           ErrorType = "quota"
	ErrorTypeModel           ErrorType = "model"
	ErrorTypeInternal        ErrorType = "internal"
	ErrorTypeContextOverflow ErrorType = "context_overflow"
	ErrorTypeOverloaded      ErrorType = "overloaded"
	ErrorTypeContentFilter   ErrorType = "content_filter"
)

type Event added in v1.8.0

type Event interface {
	// contains filtered or unexported methods
}

Event is a stream event. See the types below for the block lifecycle.

type FinishReason added in v1.8.0

type FinishReason string

FinishReason is a normalized stop reason; FinishReasonRaw keeps the vendor value. Empty means the vendor reported none.

const (
	FinishReasonStop     FinishReason = "stop"
	FinishReasonLength   FinishReason = "length"
	FinishReasonToolCall FinishReason = "tool_calls"
	FinishReasonError    FinishReason = "error"
	// FinishReasonSafety also covers explicit refusals; the refusal text, if
	// any, is an ordinary TextBlock.
	FinishReasonSafety FinishReason = "safety"
	// FinishReasonOther is a reported reason without a normalized equivalent.
	FinishReasonOther FinishReason = "other"
)

type HTTPClient added in v1.9.0

type HTTPClient interface {
	Do(*http.Request) (*http.Response, error)
}

HTTPClient sends provider requests; *http.Client satisfies it.

type ImageBlock added in v1.8.0

type ImageBlock struct {
	URL     string
	Data    []byte
	MIME    string
	FileURI string
	Detail  string
	Cache   *CacheControl
}

ImageBlock is an image. Set one source: URL, Data with MIME, or FileURI, a vendor file reference. Detail is sent where the wire format has it.

func ImageURL added in v1.8.0

func ImageURL(url string) ImageBlock

ImageURL returns an ImageBlock for a URL, which may be a data URL.

type JSONSchema added in v1.2.1

type JSONSchema struct {
	Name        string
	Description string
	Schema      Schema
	Strict      StrictMode
}

JSONSchema is a named schema for structured output.

type Message

type Message struct {
	Role   Role
	Blocks []Block
}

Message is one conversation turn.

func Assistant added in v1.8.0

func Assistant(blocks ...Block) Message

Assistant returns an assistant message, such as a prior reply to replay.

func AssistantText added in v1.8.0

func AssistantText(text string) Message

AssistantText returns an assistant message with one text block.

func System added in v1.8.0

func System(text string) Message

System returns a system message.

func ToolResult added in v1.8.0

func ToolResult(toolUseID string, blocks ...Block) Message

ToolResult returns a tool message answering the tool call toolUseID.

func ToolResultText added in v1.8.0

func ToolResultText(toolUseID, text string) Message

ToolResultText returns a tool message with a text result.

func User added in v1.8.0

func User(blocks ...Block) Message

User returns a user message.

func UserText added in v1.8.0

func UserText(text string) Message

UserText returns a user message with one text block.

type ModelInfo

type ModelInfo struct {
	ID               string
	Name             string
	Provider         string
	Description      string
	ContextLength    int
	InputTokenLimit  int
	OutputTokenLimit int
	Created          int64
}

ModelInfo describes a model; fields the vendor does not report are zero.

type ModelLister added in v1.5.5

type ModelLister interface {
	ListModels(context.Context) ([]ModelInfo, error)
}

ModelLister is implemented by providers that can list models.

type Observer added in v1.9.0

type Observer interface {
	Start(context.Context, CallInfo) (context.Context, CallObserver)
}

Observer starts one observation per Chat/Stream invocation, including local validation failures. Its returned non-nil context reaches subsequent observers and the provider; derive it from the input to preserve cancellation and values. Returning a nil CallObserver opts out of observing this call. Callbacks are synchronous; the SDK does not recover observer panics.

type ObserverFunc added in v1.9.0

type ObserverFunc func(context.Context, CallInfo) (context.Context, CallObserver)

ObserverFunc adapts a function to Observer.

func (ObserverFunc) Start added in v1.9.0

Start calls f.

type Provider

type Provider interface {
	Name() string
	Chat(context.Context, *Request) (*Response, error)
	Stream(context.Context, *Request) (Stream, error)
}

Provider adapts one vendor protocol. Chat and Stream receive a validated copy of the request that the provider owns. Implementations may also implement ModelLister and CapabilityProvider.

type ProviderEvent added in v1.8.0

type ProviderEvent struct {
	Name string
	Raw  json.RawMessage
}

ProviderEvent exposes a provider-native event that has no typed equivalent. It does not affect the aggregated Response.

type ProviderOptions added in v1.8.0

type ProviderOptions map[string]json.RawMessage

ProviderOptions contains JSON values owned by the request. Use NewProviderOptions or Set to encode Go values; the client copies each value before observation or execution.

func NewProviderOptions added in v1.9.0

func NewProviderOptions(values map[string]any) (ProviderOptions, error)

NewProviderOptions serializes values immediately, so subsequent mutations of the supplied Go objects cannot change the request. Encoding errors are returned.

func (ProviderOptions) Decode added in v1.9.0

func (o ProviderOptions) Decode() (map[string]any, error)

Decode gives a provider an independent JSON tree. Numbers remain json.Number to preserve integer precision until the provider validates its wire type.

func (ProviderOptions) Set added in v1.9.0

func (o ProviderOptions) Set(key string, value any) error

Set encodes a value into an initialized options map.

type ProviderState added in v1.9.0

type ProviderState struct {
	// Provider is the Name of the provider that produced the block.
	Provider string
	// Model is the requested model.
	Model string
	// Data holds the vendor's native fields as JSON.
	Data json.RawMessage
}

ProviderState is data a provider attaches to a block it produced so the block can be sent back to it: a reasoning signature, encrypted reasoning or an item id. Keep it with the block when storing history; only the provider named by Provider reads it.

Replay rule: portable content (text, reasoning text, tool calls) maps to wherever the target wire format can carry it, while State is sent only to the provider that produced it. A block the wire cannot carry without its State, such as Anthropic thinking, is dropped; where the vendor documents a placeholder for foreign content, the provider supplies it. Blocks built by the caller (nil State) count as foreign.

type ReasoningBlock added in v1.8.0

type ReasoningBlock struct {
	Text    string
	Summary bool
	State   *ProviderState
	Cache   *CacheControl
}

ReasoningBlock is model reasoning. Summary marks Text as a summary rather than the full reasoning. Text is empty when the vendor returns the reasoning encrypted or redacted; State then carries it.

type ReasoningDelta added in v1.8.0

type ReasoningDelta struct {
	Index int
	Text  string
}

ReasoningDelta appends text to a ReasoningBlock.

type Request

type Request struct {
	Model    string
	Messages []Message

	MaxTokens   *int
	Temperature *float64
	TopP        *float64
	Stop        []string

	Tools      []Tool
	ToolChoice *ToolChoice

	ResponseFormat *ResponseFormat
	Thinking       *Thinking

	ProviderOptions ProviderOptions
}

Request is a provider-neutral chat request. Nil pointers and empty fields are omitted from the wire, leaving the vendor default.

type Response

type Response struct {
	Blocks []Block
	Usage  Usage

	Model    string
	Provider string

	FinishReason    FinishReason
	FinishReasonRaw string
	Warnings        []Warning
	Raw             json.RawMessage
}

Response is a complete model reply. Providers set Raw to the vendor body of a non-streaming reply; the Client keeps it only with WithCaptureRawResponse.

func Collect added in v1.8.0

func Collect(stream Stream) (*Response, error)

Collect consumes the stream and returns the aggregated Response. On failure it returns the partial response together with the error. EOF before DoneEvent is io.ErrUnexpectedEOF; partial responses must not be treated as completed output.

func Handle added in v1.8.0

func Handle(stream Stream, fn func(Event) error) (*Response, error)

Handle consumes the stream, invoking fn for each event as it arrives, and returns the aggregated Response. Events are aggregated before fn runs, so a BlockEnd passed to fn carries the completed block. A nil fn behaves exactly like Collect. On any failure Handle returns the partial response and the error. The caller still owns Close.

func (*Response) Reasoning

func (r *Response) Reasoning() string

Reasoning concatenates the reasoning block texts.

func (*Response) Text added in v1.8.0

func (r *Response) Text() string

Text concatenates the text blocks.

func (*Response) ToolCalls

func (r *Response) ToolCalls() []ToolUseBlock

ToolCalls returns the tool use blocks in order.

type ResponseFormat added in v1.2.1

type ResponseFormat struct {
	Type       ResponseFormatType
	JSONSchema *JSONSchema
}

ResponseFormat constrains the output format. JSONSchema is used with ResponseFormatJSONSchema. Providers without native schema support may use a prompt instead and report a Warning; this does not enforce schema adherence, including when StrictEnabled is set. See providers.md for the mapping.

func NewResponseFormatJSONObject added in v1.2.1

func NewResponseFormatJSONObject() *ResponseFormat

NewResponseFormatJSONObject requests a JSON object.

func NewResponseFormatJSONSchema added in v1.2.1

func NewResponseFormatJSONSchema(name, description string, schema any, strict StrictMode) (*ResponseFormat, error)

NewResponseFormatJSONSchema requests output matching schema, converted with SchemaFrom.

func NewResponseFormatText added in v1.2.1

func NewResponseFormatText() *ResponseFormat

NewResponseFormatText requests plain text output.

type ResponseFormatType added in v1.8.0

type ResponseFormatType string

ResponseFormatType selects the output format.

const (
	ResponseFormatText       ResponseFormatType = "text"
	ResponseFormatJSONObject ResponseFormatType = "json_object"
	ResponseFormatJSONSchema ResponseFormatType = "json_schema"
)

type Role added in v1.8.0

type Role string

Role identifies the author of a Message.

const (
	RoleSystem    Role = "system"
	RoleUser      Role = "user"
	RoleAssistant Role = "assistant"
	RoleTool      Role = "tool"
)

type Schema added in v1.8.0

type Schema json.RawMessage

Schema is a JSON Schema document.

func SchemaFrom added in v1.8.0

func SchemaFrom(v any) (Schema, error)

SchemaFrom returns v as a Schema. JSON text (Schema, json.RawMessage, []byte or string) is validated and copied; any other value is marshaled. A nil v returns a nil Schema.

type Stream added in v1.8.0

type Stream interface {
	Next() (Event, error)
	Close() error
}

Stream is consumed by one goroutine. A successful stream emits exactly one DoneEvent. Failures are returned by Next, never encoded as events. Callers must Close the stream, including after completion or failure.

type StrictMode added in v1.8.0

type StrictMode int

StrictMode controls strict schema adherence. StrictDefault leaves the wire flag unset.

const (
	StrictDefault StrictMode = iota
	StrictEnabled
	StrictDisabled
)

func (StrictMode) Value added in v1.9.0

func (m StrictMode) Value() (strict, set bool)

Value returns the wire strict flag and whether the mode sets one.

type TextBlock added in v1.8.0

type TextBlock struct {
	Text        string
	Annotations []Annotation
	Logprobs    json.RawMessage
	State       *ProviderState
	Cache       *CacheControl
}

TextBlock is plain text. Annotations and Logprobs are response metadata.

func Text added in v1.8.0

func Text(text string) TextBlock

Text returns a TextBlock.

type TextDelta added in v1.9.0

type TextDelta struct {
	Index int
	Text  string
}

TextDelta appends text to a TextBlock.

type Thinking added in v1.8.0

type Thinking struct {
	Mode          ThinkingMode
	Effort        string
	BudgetTokens  *int
	IncludeOutput bool
}

Thinking configures model reasoning. Effort and BudgetTokens are sent as given; which values a model accepts is the vendor's decision. IncludeOutput is a hint to return reasoning where the vendor makes it optional; providers without such a switch ignore it.

type ThinkingMode added in v1.8.0

type ThinkingMode int

ThinkingMode is ThinkingEnabled by default; a nil *Thinking leaves thinking to the vendor default.

const (
	ThinkingEnabled ThinkingMode = iota
	ThinkingDisabled
)

type Tool

type Tool struct {
	Name        string
	Description string
	Parameters  Schema
	Strict      StrictMode
}

Tool declares a function the model may call.

func NewTool added in v1.5.4

func NewTool(name, description string, parameters any) (Tool, error)

NewTool builds a Tool, converting parameters with SchemaFrom.

type ToolChoice added in v1.8.0

type ToolChoice struct {
	Mode ToolChoiceMode
	Name string
}

ToolChoice selects a policy, or a named tool when Name is set. A nil choice leaves selection to the provider. Mode and Name are mutually exclusive.

type ToolChoiceMode added in v1.9.0

type ToolChoiceMode string

ToolChoiceMode is a tool selection policy.

const (
	ToolChoiceAuto     ToolChoiceMode = "auto"
	ToolChoiceNone     ToolChoiceMode = "none"
	ToolChoiceRequired ToolChoiceMode = "required"
)

type ToolReferenceBlock added in v1.8.0

type ToolReferenceBlock struct {
	ToolName string
	Cache    *CacheControl
}

ToolReferenceBlock names a tool inside tool result content, as returned by a tool search tool.

type ToolResultBlock added in v1.8.0

type ToolResultBlock struct {
	ToolUseID string
	Content   []Block
	IsError   bool
	Cache     *CacheControl
}

ToolResultBlock answers the ToolUseBlock with ID ToolUseID. Content holds TextBlock, ImageBlock or ToolReferenceBlock values.

type ToolUseBlock added in v1.8.0

type ToolUseBlock struct {
	ID        string
	Name      string
	Arguments json.RawMessage
	State     *ProviderState
	Cache     *CacheControl
}

ToolUseBlock is a tool call from the assistant. Arguments is the JSON the model produced.

type ToolUseDelta added in v1.8.0

type ToolUseDelta struct {
	Index     int
	Arguments string
}

ToolUseDelta appends a fragment of a ToolUseBlock's JSON arguments.

type Usage

type Usage struct {
	InputTokens      *int
	OutputTokens     *int
	TotalTokens      *int
	ReasoningTokens  *int
	CacheReadTokens  *int
	CacheWriteTokens *int
}

Usage contains reported token counts. A nil count is unknown; a pointer to zero is a known zero. InputTokens includes cache reads and writes; OutputTokens includes reasoning. Detail counts are subsets, not additions.

func (Usage) CacheRead added in v1.9.0

func (u Usage) CacheRead() (int, bool)

CacheRead returns CacheReadTokens and whether it is known.

func (Usage) CacheWrite added in v1.9.0

func (u Usage) CacheWrite() (int, bool)

CacheWrite returns CacheWriteTokens and whether it is known.

func (Usage) Clone added in v1.9.0

func (u Usage) Clone() Usage

Clone returns an independent copy of the reported counts.

func (Usage) HasTokens added in v1.8.0

func (u Usage) HasTokens() bool

HasTokens reports whether any token count is known, including a known zero.

func (Usage) Input added in v1.9.0

func (u Usage) Input() (int, bool)

Input returns InputTokens and whether it is known.

func (Usage) Output added in v1.9.0

func (u Usage) Output() (int, bool)

Output returns OutputTokens and whether it is known.

func (Usage) Reasoning added in v1.9.0

func (u Usage) Reasoning() (int, bool)

Reasoning returns ReasoningTokens and whether it is known.

func (Usage) Total added in v1.9.0

func (u Usage) Total() (int, bool)

Total returns TotalTokens and whether it is known.

type UsageEvent added in v1.8.0

type UsageEvent struct {
	Usage Usage
}

UsageEvent carries cumulative usage; a later event replaces an earlier one.

type Warning added in v1.8.0

type Warning struct {
	Code     string
	Provider string
	Message  string
}

Warning reports a non-fatal issue, such as dropped content or a generated ID. Code has the form "<source>.<snake_case>", such as "litellm.tool_arguments_invalid".

type WarningEvent added in v1.8.0

type WarningEvent struct {
	Warning Warning
}

WarningEvent carries a Warning; it is also collected into Response.Warnings.

Directories

Path Synopsis
Package catalog holds model facts (context window, output limit, reasoning support and per-token prices) loaded from LiteLLM's model list or set explicitly.
Package catalog holds model facts (context window, output limit, reasoning support and per-token prices) loaded from LiteLLM's model list or set explicitly.
examples
anthropic command
bedrock command
deepseek command
gemini command
glm command
grok command
minimax command
ollama command
openai command
openrouter command
qwen command
internal
claude
Package claude holds the Claude request mapping shared by the Anthropic and Bedrock adapters.
Package claude holds the Claude request mapping shared by the Anthropic and Bedrock adapters.
wire
Package wire holds what provider adapters share to speak vendor wire protocols: HTTP transport and Server-Sent Events, error and stop-reason classification, stream block indexing, ProviderOptions merging, replay state, and token and data URL helpers.
Package wire holds what provider adapters share to speak vendor wire protocols: HTTP transport and Server-Sent Events, error and stop-reason classification, stream block indexing, ProviderOptions merging, replay state, and token and data URL helpers.
otel module
Package provider builds the built-in providers by name, for applications that choose a provider from configuration.
Package provider builds the built-in providers by name, for applications that choose a provider from configuration.
anthropic
Package anthropic connects to the Anthropic Messages API.
Package anthropic connects to the Anthropic Messages API.
bedrock
Package bedrock connects to the Amazon Bedrock Converse API.
Package bedrock connects to the Amazon Bedrock Converse API.
compat
Package compat connects to any OpenAI-compatible Chat Completions endpoint, such as vLLM, LM Studio or a gateway, and holds the Config shared by the vendor wrappers (deepseek, glm, grok, mimo, minimax, ollama, openrouter, qwen).
Package compat connects to any OpenAI-compatible Chat Completions endpoint, such as vLLM, LM Studio or a gateway, and holds the Config shared by the vendor wrappers (deepseek, glm, grok, mimo, minimax, ollama, openrouter, qwen).
deepseek
Package deepseek connects to the DeepSeek API.
Package deepseek connects to the DeepSeek API.
gemini
Package gemini connects to the Gemini API.
Package gemini connects to the Gemini API.
glm
Package glm connects to the Zhipu GLM API.
Package glm connects to the Zhipu GLM API.
grok
Package grok connects to the xAI Grok API.
Package grok connects to the xAI Grok API.
internal/openaicompat
Package openaicompat implements the Chat Completions protocol shared by OpenAI and the OpenAI-compatible providers.
Package openaicompat implements the Chat Completions protocol shared by OpenAI and the OpenAI-compatible providers.
internal/openaicompat/compattest
Package compattest runs openaicompat providers against canned HTTP responses in tests.
Package compattest runs openaicompat providers against canned HTTP responses in tests.
mimo
Package mimo connects to the Xiaomi MiMo API.
Package mimo connects to the Xiaomi MiMo API.
minimax
Package minimax connects to the MiniMax API.
Package minimax connects to the MiniMax API.
ollama
Package ollama connects to Ollama's OpenAI-compatible endpoint.
Package ollama connects to Ollama's OpenAI-compatible endpoint.
openai
Package openai connects to the OpenAI API, or any endpoint that implements it exactly (such as Azure OpenAI), through Chat Completions or Responses.
Package openai connects to the OpenAI API, or any endpoint that implements it exactly (such as Azure OpenAI), through Chat Completions or Responses.
openrouter
Package openrouter connects to the OpenRouter API.
Package openrouter connects to the OpenRouter API.
qwen
Package qwen connects to the Alibaba Cloud DashScope compatible-mode API.
Package qwen connects to the Alibaba Cloud DashScope compatible-mode API.
Package retry provides opt-in HTTP retries for providers.
Package retry provides opt-in HTTP retries for providers.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL