Documentation
¶
Overview ¶
Package litellm provides a small, explicit multi-provider LLM SDK core.
The root package owns the provider-agnostic domain model: Request, Response, Message, Block, Stream, Event, structured errors, warnings, and observers. Concrete providers live in the subpackages of package provider, which builds them by name; provider/compat connects any OpenAI-compatible endpoint. catalog, retry and gateway are optional utilities; gateway carries calls over HTTP to a server that holds the vendor keys.
Quick Start ¶
Create a provider with its package-specific config, then bind a Client:
import (
"context"
"fmt"
"os"
"github.com/voocel/litellm"
"github.com/voocel/litellm/provider/anthropic"
)
provider, err := anthropic.New(anthropic.Config{
APIKey: os.Getenv("ANTHROPIC_API_KEY"),
})
if err != nil {
panic(err)
}
client, err := litellm.New(provider)
if err != nil {
panic(err)
}
maxTokens := 1024
resp, err := client.Chat(context.Background(), litellm.Request{
Model: "claude-sonnet-4-5",
MaxTokens: &maxTokens,
Messages: []litellm.Message{litellm.UserText("Explain AI in one sentence.")},
})
if err != nil {
panic(err)
}
fmt.Println(resp.Text())
Blocks ¶
Message and Response content is represented as ordered Blocks. This preserves the order of text, reasoning, tool use, and tool results across multi-turn agent workflows. Data a vendor needs back, such as a reasoning signature, travels in a block's ProviderState and is sent only to the provider that produced it.
Streaming ¶
A stream is a sequence of blocks. BlockStart opens the block at Index, the position it takes in Response.Blocks; deltas of the same kind grow it; BlockEnd closes it with the completed block. Blocks may interleave, and all end before DoneEvent. Handle aggregates the stream and passes each event to a callback; Collect only aggregates. Stream is for single-goroutine consumption.
stream, err := client.Stream(ctx, req)
if err != nil {
panic(err)
}
defer stream.Close()
resp, err := litellm.Handle(stream, func(event litellm.Event) error {
if e, ok := event.(litellm.TextDelta); ok {
fmt.Print(e.Text)
}
return nil
})
Design ¶
The SDK is intentionally not a router, agent runtime, account system, or request scheduler, and its gateway leaves authentication and metering to the application. It binds one Client to one Provider and exposes explicit configuration and structural validation. It maps structure only: it does not infer model features, validate vendor values locally, or rewrite user input; the vendor API is the authority. ProviderOptions carry native wire fields. Usage counts are plain ints; a count the vendor does not report is zero.
Errors ¶
Errors are *Error values. Switch on ErrorTypeOf to act on a failure, and use IsTemporaryError to tell whether a fresh request may succeed: a stream cut off before its end is a temporary network error, and a call the caller's context cancelled is ErrorTypeCanceled.
Index ¶
- func IsTemporaryError(err error) bool
- func RetryAfter(err error) time.Duration
- func WrapError(provider string, fallback ErrorType, err error) error
- type Annotation
- type Block
- type BlockEnd
- type BlockStart
- type CacheControl
- type CallInfo
- type CallObserver
- type CallResult
- type CallStatus
- type Capabilities
- type CapabilityProvider
- type Client
- type ClientOption
- type DoneEvent
- type Error
- type ErrorType
- type Event
- type FinishReason
- type HTTPClient
- type ImageBlock
- type JSONSchema
- type Message
- type ModelInfo
- type ModelLister
- type Observer
- type ObserverFunc
- type Provider
- type ProviderEvent
- type ProviderOptions
- type ProviderState
- type ReasoningBlock
- type ReasoningDelta
- type Request
- type Response
- type ResponseFormat
- type ResponseFormatType
- type Role
- type Schema
- type Stream
- type TextBlock
- type TextDelta
- type Thinking
- type Tool
- type ToolChoice
- type ToolChoiceMode
- type ToolReferenceBlock
- type ToolResultBlock
- type ToolUseBlock
- type ToolUseDelta
- type Usage
- type UsageEvent
- type Warning
- type WarningEvent
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func IsTemporaryError ¶ added in v1.9.0
IsTemporaryError reports whether the failure may resolve over time. It does not establish that repeating the operation is safe, even before any output.
func RetryAfter ¶ added in v1.9.0
RetryAfter returns the server-suggested delay carried by err, or zero.
Types ¶
type Annotation ¶ added in v1.8.0
type Annotation struct {
Type string `json:"type"`
Text string `json:"text,omitempty"`
URL string `json:"url,omitempty"`
Extra json.RawMessage `json:"extra,omitempty"`
}
Annotation is a citation attached to text. Extra keeps the vendor entry verbatim.
type Block ¶ added in v1.8.0
type Block interface {
// contains filtered or unexported methods
}
Block is one piece of message content. The implementations are TextBlock, ImageBlock, ReasoningBlock, ToolUseBlock, ToolResultBlock and ToolReferenceBlock.
func UnmarshalBlock ¶ added in v1.9.3
func UnmarshalBlock(raw json.RawMessage) (Block, error)
UnmarshalBlock decodes a block encoded as JSON by its "type".
func UnmarshalBlocks ¶ added in v1.9.3
func UnmarshalBlocks(raws []json.RawMessage) ([]Block, error)
UnmarshalBlocks decodes blocks encoded as JSON, each by its "type".
type BlockEnd ¶ added in v1.9.0
BlockEnd closes a block. Providers set Block only to deliver metadata that arrives late (State, annotations, logprobs, a tool's id and name): its non-empty metadata fields replace the started ones, while its text and arguments are ignored. Streams read through Handle, Collect or a Client deliver the completed block instead.
type BlockStart ¶ added in v1.9.0
BlockStart opens a TextBlock, ReasoningBlock or ToolUseBlock. Block carries the metadata known up front, such as a tool's ID and Name, and may carry initial text.
type CacheControl ¶ added in v1.5.0
type CacheControl struct {
TTL string `json:"ttl,omitempty"`
}
CacheControl marks a cache breakpoint: the prompt prefix up to and including this block may be cached. TTL is how long, passed as is, such as "1h"; empty is the vendor's default, five minutes on Anthropic and Bedrock. Like the breakpoint, a TTL is a hint: adapters that cannot send it use the default.
type CallInfo ¶ added in v1.9.0
CallInfo describes the caller's request before validation. Request is a snapshot shared by all observers: read it, never modify it.
type CallObserver ¶ added in v1.9.0
type CallObserver interface {
OnEvent(Event)
End(CallResult)
}
CallObserver belongs to one invocation. Events and the Response are snapshots shared by all observers and isolated from the caller: read, never modify. OnEvent receives stream events as they arrive, and Chat warnings as WarningEvent. End runs exactly once; observers end in reverse registration order so nested observation scopes unwind correctly. Streams must be consumed to termination or explicitly closed; cancellation alone does not run callbacks in a background goroutine.
type CallResult ¶ added in v1.9.0
type CallResult struct {
Status CallStatus
Response *Response
Err error
Duration time.Duration
}
CallResult describes SDK execution, not errors in application consumer callbacks. Response may be partial on failure or early Close. A later resource cleanup error does not revise an already completed invocation; Close returns that error.
type CallStatus ¶ added in v1.9.0
type CallStatus string
CallStatus is how an invocation ended. CallClosed means the caller closed a stream before it finished.
const ( CallCompleted CallStatus = "completed" CallFailed CallStatus = "failed" CallCanceled CallStatus = "canceled" CallClosed CallStatus = "closed" )
type Capabilities ¶ added in v1.8.1
type Capabilities struct {
// MaxTokensRequired reports that the vendor rejects requests without
// Request.MaxTokens.
MaxTokensRequired bool
// ThinkingEffort and DisableThinking report that the adapter sends
// Thinking.Effort and Thinking.Disabled; it rejects them otherwise, before
// the request is sent. Which efforts a model takes is the vendor's call.
ThinkingEffort bool
DisableThinking bool
// DeferredTools reports that the vendor loads the tools marked Deferred
// once a tool reference names them, the adapter sending every tool from
// the start; other adapters send what the request offers (see Tool).
DeferredTools bool
// ProviderOptions lists, sorted, the Request.ProviderOptions keys the
// adapter knows: native fields of the vendor's request. One that also
// passes other keys through, as compat does and the Chat Completions
// vendors do with AllowUnknownProviderOptions, lists only those it knows.
ProviderOptions []string
}
Capabilities states protocol facts of a provider adapter. It is static per provider: whether a model honors a request is the vendor's call, and the vendor's error is the source of truth.
type CapabilityProvider ¶ added in v1.8.1
type CapabilityProvider interface {
Capabilities() Capabilities
}
CapabilityProvider is implemented by providers that declare Capabilities.
type Client ¶
type Client struct {
// contains filtered or unexported fields
}
Client runs requests through a Provider. It copies and validates each request, attributes errors to the provider, and notifies Observers. A Client is safe for concurrent use.
func New ¶
func New(provider Provider, opts ...ClientOption) (*Client, error)
New returns a Client for provider.
func (*Client) Capabilities ¶ added in v1.8.1
func (c *Client) Capabilities() (caps Capabilities, ok bool)
Capabilities reports what the provider adapter can express on the wire; ok is false when the provider does not implement CapabilityProvider, so the facts are unknown.
func (*Client) Chat ¶ added in v1.5.0
Chat sends req and returns the complete response. On error the response may still be returned when the provider produced one.
func (*Client) ProviderName ¶ added in v1.5.7
ProviderName returns the provider's Name.
type ClientOption ¶
ClientOption configures a Client.
func WithCaptureRawResponse ¶ added in v1.8.0
func WithCaptureRawResponse(enabled bool) ClientOption
WithCaptureRawResponse keeps the raw vendor body in Response.Raw.
func WithObservers ¶ added in v1.9.0
func WithObservers(observers ...Observer) ClientOption
WithObservers adds observers, skipping nil ones.
func WithStreamIdleTimeout ¶ added in v1.8.0
func WithStreamIdleTimeout(timeout time.Duration) ClientOption
WithStreamIdleTimeout fails a stream, with a temporary network error, once it waits timeout for data from its connection, as when the connection hung: for the response, the waits of any retries its HTTP client makes included, or then for its body. Any data counts, vendor pings and gateway heartbeats included, so a healthy stream fails only when silent for longer, as a vendor may be while the model reads a long prompt or thinks. Zero disables the check.
type DoneEvent ¶ added in v1.8.0
type DoneEvent struct {
FinishReason FinishReason
FinishReasonRaw string
Provider string
Model string
}
DoneEvent ends a successful stream.
type Error ¶ added in v1.9.0
type Error struct {
Type ErrorType
Code string
Message string
Provider string
StatusCode int
// Temporary describes a potentially transient failure, not permission to replay
// a request. The provider may already have processed or billed the operation.
Temporary bool
// RetryAfter is the server-suggested delay; zero means unspecified.
RetryAfter time.Duration
Cause error
}
Error is the error type returned by the client and all providers.
func NewError ¶ added in v1.5.0
NewError builds an error of the given type. provider may be empty and cause nil; Temporary follows the type.
func NewNetworkError ¶ added in v1.5.0
NewNetworkError builds a transport error. Context cancellation is ErrorTypeCanceled and a deadline a timeout, neither temporary; other causes are temporary.
type ErrorType ¶ added in v1.5.0
type ErrorType string
ErrorType classifies an Error.
const ( ErrorTypeAuth ErrorType = "auth" ErrorTypeRateLimit ErrorType = "rate_limit" ErrorTypeNetwork ErrorType = "network" ErrorTypeValidation ErrorType = "validation" ErrorTypeProvider ErrorType = "provider" // ErrorTypeTimeout is a call that ran out the time its caller gave it: // the context's deadline or the HTTP client's timeout. A server that // timed out is ErrorTypeProvider. ErrorTypeTimeout ErrorType = "timeout" ErrorTypeQuota ErrorType = "quota" ErrorTypeModel ErrorType = "model" ErrorTypeInternal ErrorType = "internal" ErrorTypeContextOverflow ErrorType = "context_overflow" ErrorTypeOverloaded ErrorType = "overloaded" ErrorTypeContentFilter ErrorType = "content_filter" // ErrorTypeCanceled is a call the caller's context cancelled. ErrorTypeCanceled ErrorType = "canceled" )
func ErrorTypeOf ¶ added in v1.9.3
ErrorTypeOf returns the type of the *Error err wraps, or "" when it wraps none.
type Event ¶ added in v1.8.0
type Event interface {
// contains filtered or unexported methods
}
Event is a stream event. See the types below for the block lifecycle.
type FinishReason ¶ added in v1.8.0
type FinishReason string
FinishReason is a normalized stop reason; FinishReasonRaw keeps the vendor value. Empty means the vendor reported none.
const ( FinishReasonStop FinishReason = "stop" FinishReasonLength FinishReason = "length" // FinishReasonToolCall ends a turn of tool calls. Providers report the // vendor's reason; a Client also reports this one for a reply that stops // with tool calls, as Gemini and OpenAI Responses end such turns. FinishReasonToolCall FinishReason = "tool_calls" FinishReasonError FinishReason = "error" // FinishReasonSafety also covers explicit refusals; the refusal text, if // any, is an ordinary TextBlock. FinishReasonSafety FinishReason = "safety" // FinishReasonOther is a reported reason without a normalized equivalent. FinishReasonOther FinishReason = "other" )
type HTTPClient ¶ added in v1.9.0
HTTPClient sends provider requests; *http.Client satisfies it.
type ImageBlock ¶ added in v1.8.0
type ImageBlock struct {
URL string `json:"url,omitempty"`
Data []byte `json:"data,omitempty"`
MIME string `json:"mime,omitempty"`
FileURI string `json:"file_uri,omitempty"`
Detail string `json:"detail,omitempty"`
Cache *CacheControl `json:"cache,omitempty"`
}
ImageBlock is an image. Set one source: URL, Data with MIME, or FileURI, a vendor file reference. Detail is sent where the wire format has it.
func ImageURL ¶ added in v1.8.0
func ImageURL(url string) ImageBlock
ImageURL returns an ImageBlock for a URL, which may be a data URL.
func (ImageBlock) MarshalJSON ¶ added in v1.9.3
func (b ImageBlock) MarshalJSON() ([]byte, error)
type JSONSchema ¶ added in v1.2.1
type JSONSchema struct {
Name string `json:"name"`
Description string `json:"description,omitempty"`
Schema Schema `json:"schema,omitempty"`
Strict *bool `json:"strict,omitempty"`
}
JSONSchema is a named schema for structured output. Strict is as in Tool.
type Message ¶
Message is one conversation turn.
func Assistant ¶ added in v1.8.0
Assistant returns an assistant message, such as a prior reply to replay.
func AssistantText ¶ added in v1.8.0
AssistantText returns an assistant message with one text block.
func ToolResult ¶ added in v1.8.0
ToolResult returns a tool message answering the tool call toolUseID.
func ToolResultText ¶ added in v1.8.0
ToolResultText returns a tool message with a text result.
func (*Message) UnmarshalJSON ¶ added in v1.9.3
type ModelInfo ¶
type ModelInfo struct {
ID string
// Name is the vendor's display name; empty when it gives none.
Name string
// Created is when the vendor released or listed the model; zero when it
// does not say.
Created time.Time
}
ModelInfo is a model as the vendor lists it, in the vendor's order.
type ModelLister ¶ added in v1.5.5
ModelLister is implemented by providers that list the models their credentials reach, which also checks the credentials: a rejected key fails with ErrorTypeAuth.
type Observer ¶ added in v1.9.0
Observer starts one observation per Chat/Stream invocation, including local validation failures. Its returned non-nil context reaches subsequent observers and the provider; derive it from the input to preserve cancellation and values. Returning a nil CallObserver opts out of observing this call. Callbacks are synchronous; the SDK does not recover observer panics.
type ObserverFunc ¶ added in v1.9.0
ObserverFunc adapts a function to Observer.
func (ObserverFunc) Start ¶ added in v1.9.0
func (f ObserverFunc) Start(ctx context.Context, info CallInfo) (context.Context, CallObserver)
Start calls f.
type Provider ¶
type Provider interface {
Name() string
Chat(context.Context, *Request) (*Response, error)
Stream(context.Context, *Request) (Stream, error)
}
Provider adapts one vendor protocol. Name identifies it in responses, errors and ProviderState, and must not be empty. Chat and Stream receive a validated copy of the request that the provider owns. Implementations may also implement CapabilityProvider.
type ProviderEvent ¶ added in v1.8.0
type ProviderEvent struct {
Name string
Raw json.RawMessage
}
ProviderEvent exposes a provider-native event that has no typed equivalent. It does not affect the aggregated Response.
type ProviderOptions ¶ added in v1.8.0
type ProviderOptions map[string]json.RawMessage
ProviderOptions contains JSON values owned by the request. Use NewProviderOptions or Set to encode Go values; the client copies each value before observation or execution.
func NewProviderOptions ¶ added in v1.9.0
func NewProviderOptions(values map[string]any) (ProviderOptions, error)
NewProviderOptions serializes values immediately, so subsequent mutations of the supplied Go objects cannot change the request. Encoding errors are returned.
type ProviderState ¶ added in v1.9.0
type ProviderState struct {
// Provider is the Name of the provider that produced the block.
Provider string `json:"provider"`
// Model is the requested model.
Model string `json:"model,omitempty"`
// Data holds the vendor's native fields as JSON.
Data json.RawMessage `json:"data,omitempty"`
}
ProviderState is data a provider attaches to a block it produced so the block can be sent back to it: a reasoning signature, encrypted reasoning or an item id. Keep it with the block when storing history; only the provider named by Provider reads it.
Replay rule: portable content (text, reasoning text, tool calls) maps to wherever the target wire format can carry it, while State is sent only to the provider that produced it. A block the wire cannot carry without its State, such as Anthropic thinking, is dropped; where the vendor documents a placeholder for foreign content, the provider supplies it. Blocks built by the caller (nil State) count as foreign.
type ReasoningBlock ¶ added in v1.8.0
type ReasoningBlock struct {
Text string `json:"text"`
Summary bool `json:"summary,omitempty"`
State *ProviderState `json:"state,omitempty"`
}
ReasoningBlock is model reasoning. Summary marks Text as a summary rather than the full reasoning. Text is empty when the vendor returns the reasoning encrypted or redacted; State then carries it.
func (ReasoningBlock) MarshalJSON ¶ added in v1.9.3
func (b ReasoningBlock) MarshalJSON() ([]byte, error)
type ReasoningDelta ¶ added in v1.8.0
ReasoningDelta appends text to a ReasoningBlock.
type Request ¶
type Request struct {
Model string `json:"model"`
Messages []Message `json:"messages"`
MaxTokens *int `json:"max_tokens,omitempty"`
Temperature *float64 `json:"temperature,omitempty"`
TopP *float64 `json:"top_p,omitempty"`
Stop []string `json:"stop,omitempty"`
Tools []Tool `json:"tools,omitempty"`
ToolChoice *ToolChoice `json:"tool_choice,omitempty"`
ResponseFormat *ResponseFormat `json:"response_format,omitempty"`
Thinking *Thinking `json:"thinking,omitempty"`
ProviderOptions ProviderOptions `json:"provider_options,omitempty"`
}
Request is a provider-neutral chat request. Nil pointers and empty fields are omitted from the wire, leaving the vendor default.
func (*Request) OfferedTools ¶ added in v1.9.4
OfferedTools returns the tools r offers the model: all but the deferred ones that no tool result in its messages references yet.
type Response ¶
type Response struct {
Blocks []Block
Usage Usage
Model string
Provider string
FinishReason FinishReason
FinishReasonRaw string
Warnings []Warning
Raw json.RawMessage
}
Response is a complete model reply. Providers set Raw to the vendor body of a non-streaming reply; the Client keeps it only with WithCaptureRawResponse.
func Collect ¶ added in v1.8.0
Collect consumes the stream and returns the aggregated Response. On failure it returns the partial response together with the error. EOF before DoneEvent is a network error wrapping io.ErrUnexpectedEOF; partial responses must not be treated as completed output.
func Handle ¶ added in v1.8.0
Handle consumes the stream, invoking fn for each event as it arrives, and returns the aggregated Response. Events are aggregated before fn runs, so a BlockEnd passed to fn carries the completed block. A nil fn behaves exactly like Collect. On any failure Handle returns the partial response and the error. The caller still owns Close.
func (*Response) ToolCalls ¶
func (r *Response) ToolCalls() []ToolUseBlock
ToolCalls returns the tool use blocks in order.
type ResponseFormat ¶ added in v1.2.1
type ResponseFormat struct {
Type ResponseFormatType `json:"type"`
JSONSchema *JSONSchema `json:"json_schema,omitempty"`
}
ResponseFormat constrains the output format. JSONSchema is used with ResponseFormatJSONSchema. Providers without native schema support may use a prompt instead and report a Warning; this does not enforce schema adherence, including when Strict is set. See providers.md for the mapping.
func NewResponseFormatJSONObject ¶ added in v1.2.1
func NewResponseFormatJSONObject() *ResponseFormat
NewResponseFormatJSONObject requests a JSON object.
func NewResponseFormatJSONSchema ¶ added in v1.2.1
func NewResponseFormatJSONSchema(name, description string, schema any) (*ResponseFormat, error)
NewResponseFormatJSONSchema requests output matching schema, converted with SchemaFrom.
func NewResponseFormatText ¶ added in v1.2.1
func NewResponseFormatText() *ResponseFormat
NewResponseFormatText requests plain text output.
type ResponseFormatType ¶ added in v1.8.0
type ResponseFormatType string
ResponseFormatType selects the output format.
const ( ResponseFormatText ResponseFormatType = "text" ResponseFormatJSONObject ResponseFormatType = "json_object" ResponseFormatJSONSchema ResponseFormatType = "json_schema" )
type Schema ¶ added in v1.8.0
type Schema json.RawMessage
Schema is a JSON Schema document.
func SchemaFrom ¶ added in v1.8.0
SchemaFrom returns v as a Schema. JSON text (Schema, json.RawMessage, []byte or string) is validated and copied; any other value is marshaled. A v that is nil, or JSON null, such as a nil map, returns a nil Schema.
func (Schema) MarshalJSON ¶ added in v1.9.3
MarshalJSON writes the schema as the JSON document it is.
func (*Schema) UnmarshalJSON ¶ added in v1.9.3
UnmarshalJSON keeps a copy of the JSON document.
type Stream ¶ added in v1.8.0
Stream is consumed by one goroutine. A successful stream emits exactly one DoneEvent. Failures are returned by Next, never encoded as events. Callers must Close the stream, including after completion or failure.
type TextBlock ¶ added in v1.8.0
type TextBlock struct {
Text string `json:"text"`
Annotations []Annotation `json:"annotations,omitempty"`
Logprobs json.RawMessage `json:"logprobs,omitempty"`
State *ProviderState `json:"state,omitempty"`
Cache *CacheControl `json:"cache,omitempty"`
}
TextBlock is plain text. Annotations and Logprobs are response metadata.
func (TextBlock) MarshalJSON ¶ added in v1.9.3
type Thinking ¶ added in v1.8.0
type Thinking struct {
Disabled bool `json:"disabled,omitempty"`
Effort string `json:"effort,omitempty"`
BudgetTokens *int `json:"budget_tokens,omitempty"`
IncludeOutput bool `json:"include_output,omitempty"`
}
Thinking configures model reasoning; a nil *Thinking leaves it to the vendor default. Disabled turns reasoning off and allows no other field. Effort and BudgetTokens are sent as given; which values a model accepts is the vendor's decision. IncludeOutput is a hint to return reasoning where the vendor makes it optional; providers without such a switch ignore it.
type Tool ¶
type Tool struct {
Name string `json:"name"`
Description string `json:"description,omitempty"`
Parameters Schema `json:"parameters,omitempty"`
Strict *bool `json:"strict,omitempty"`
Deferred bool `json:"deferred,omitempty"`
}
Tool declares a function the model may call. Strict, when set, asks the vendor to enforce, or not, that calls fit Parameters; nil leaves its default.
A Deferred tool is offered once a ToolReferenceBlock in a tool result names it, as a tool search returns. Anthropic receives every tool from the first request on, deferred ones marked defer_loading, so the tools of a conversation never change, which keeps its prompt cache and thinking valid; Bedrock, which cannot defer a tool, receives them all as offered; other vendors receive the OfferedTools of each request.
type ToolChoice ¶ added in v1.8.0
type ToolChoice struct {
Mode ToolChoiceMode `json:"mode,omitempty"`
Name string `json:"name,omitempty"`
}
ToolChoice selects a policy, or a named tool when Name is set. A nil choice leaves selection to the provider. Mode and Name are mutually exclusive.
type ToolChoiceMode ¶ added in v1.9.0
type ToolChoiceMode string
ToolChoiceMode is a tool selection policy.
const ( ToolChoiceAuto ToolChoiceMode = "auto" ToolChoiceNone ToolChoiceMode = "none" ToolChoiceRequired ToolChoiceMode = "required" )
type ToolReferenceBlock ¶ added in v1.8.0
type ToolReferenceBlock struct {
ToolName string `json:"tool_name"`
Cache *CacheControl `json:"cache,omitempty"`
}
ToolReferenceBlock names a tool inside tool result content, as returned by a tool search tool.
func (ToolReferenceBlock) MarshalJSON ¶ added in v1.9.3
func (b ToolReferenceBlock) MarshalJSON() ([]byte, error)
type ToolResultBlock ¶ added in v1.8.0
type ToolResultBlock struct {
ToolUseID string `json:"tool_use_id"`
Content []Block `json:"content,omitempty"`
IsError bool `json:"is_error,omitempty"`
Cache *CacheControl `json:"cache,omitempty"`
}
ToolResultBlock answers the ToolUseBlock with ID ToolUseID. Content holds TextBlock, ImageBlock or ToolReferenceBlock values.
func (ToolResultBlock) MarshalJSON ¶ added in v1.9.3
func (b ToolResultBlock) MarshalJSON() ([]byte, error)
func (*ToolResultBlock) UnmarshalJSON ¶ added in v1.9.3
func (b *ToolResultBlock) UnmarshalJSON(data []byte) error
type ToolUseBlock ¶ added in v1.8.0
type ToolUseBlock struct {
ID string `json:"id"`
Name string `json:"name"`
Arguments string `json:"arguments,omitempty"`
State *ProviderState `json:"state,omitempty"`
Cache *CacheControl `json:"cache,omitempty"`
}
ToolUseBlock is a tool call from the assistant. Arguments is the text the model produced, {} in a Client's reply for a call without arguments. It is meant to be a JSON object but may not be one, as when the response was cut off at the output limit.
func (ToolUseBlock) MarshalJSON ¶ added in v1.9.3
func (b ToolUseBlock) MarshalJSON() ([]byte, error)
type ToolUseDelta ¶ added in v1.8.0
ToolUseDelta appends a fragment of a ToolUseBlock's JSON arguments.
type Usage ¶
type Usage struct {
InputTokens int `json:"input_tokens,omitempty"`
OutputTokens int `json:"output_tokens,omitempty"`
ReasoningTokens int `json:"reasoning_tokens,omitempty"`
CacheReadTokens int `json:"cache_read_tokens,omitempty"`
CacheWriteTokens int `json:"cache_write_tokens,omitempty"`
CacheWrite1hTokens int `json:"cache_write_1h_tokens,omitempty"`
}
Usage holds the token counts a response reported; a count the vendor did not report is zero. InputTokens includes cache reads and writes; OutputTokens includes reasoning; CacheWriteTokens includes the writes cached for an hour, CacheWrite1hTokens.
type UsageEvent ¶ added in v1.8.0
type UsageEvent struct {
Usage Usage
}
UsageEvent carries cumulative usage; a later event replaces an earlier one.
type Warning ¶ added in v1.8.0
type Warning struct {
Code string `json:"code"`
Provider string `json:"provider,omitempty"`
Message string `json:"message"`
}
Warning reports a non-fatal issue, such as dropped content or a generated ID. Code has the form "<source>.<snake_case>", such as "litellm.tool_arguments_invalid".
type WarningEvent ¶ added in v1.8.0
type WarningEvent struct {
Warning Warning
}
WarningEvent carries a Warning; it is also collected into Response.Warnings.
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
Package catalog holds model facts (context window, output limit, reasoning support and per-token prices) loaded from LiteLLM's model list or set explicitly.
|
Package catalog holds model facts (context window, output limit, reasoning support and per-token prices) loaded from LiteLLM's model list or set explicitly. |
|
examples
|
|
|
anthropic
command
|
|
|
bedrock
command
|
|
|
deepseek
command
|
|
|
gemini
command
|
|
|
glm
command
|
|
|
grok
command
|
|
|
minimax
command
|
|
|
ollama
command
|
|
|
openai
command
|
|
|
openrouter
command
|
|
|
qwen
command
|
|
|
Package gateway carries model calls over HTTP, so that an application can reach models through a gateway that holds the vendor keys and meters the usage: an agent in a sandbox, say, that must never see a key.
|
Package gateway carries model calls over HTTP, so that an application can reach models through a gateway that holds the vendor keys and meters the usage: an agent in a sandbox, say, that must never see a key. |
|
internal
|
|
|
claude
Package claude holds the Claude request mapping shared by the Anthropic and Bedrock adapters.
|
Package claude holds the Claude request mapping shared by the Anthropic and Bedrock adapters. |
|
idle
Package idle fails a call over a connection that hung: one that waits too long for its response or, once that arrives, for data from its body.
|
Package idle fails a call over a connection that hung: one that waits too long for its response or, once that arrives, for data from its body. |
|
wire
Package wire holds what provider adapters share to speak vendor wire protocols: HTTP transport, Server-Sent Events and the stream event queue, error and stop-reason classification, stream block indexing, ProviderOptions merging, replay state, tool schemas and arguments, and token and data URL helpers.
|
Package wire holds what provider adapters share to speak vendor wire protocols: HTTP transport, Server-Sent Events and the stream event queue, error and stop-reason classification, stream block indexing, ProviderOptions merging, replay state, tool schemas and arguments, and token and data URL helpers. |
|
Package litellmtest provides a scripted Provider for testing code that calls models, without a network or a vendor.
|
Package litellmtest provides a scripted Provider for testing code that calls models, without a network or a vendor. |
|
otel
module
|
|
|
Package provider builds the built-in providers by name, for applications that choose a provider from configuration.
|
Package provider builds the built-in providers by name, for applications that choose a provider from configuration. |
|
anthropic
Package anthropic connects to the Anthropic Messages API.
|
Package anthropic connects to the Anthropic Messages API. |
|
bedrock
Package bedrock connects to the Amazon Bedrock Converse API.
|
Package bedrock connects to the Amazon Bedrock Converse API. |
|
compat
Package compat connects to any OpenAI-compatible Chat Completions endpoint, such as vLLM, LM Studio or a gateway, and holds the Config shared by the vendor wrappers (deepseek, glm, grok, mimo, minimax, ollama, openrouter, qwen).
|
Package compat connects to any OpenAI-compatible Chat Completions endpoint, such as vLLM, LM Studio or a gateway, and holds the Config shared by the vendor wrappers (deepseek, glm, grok, mimo, minimax, ollama, openrouter, qwen). |
|
deepseek
Package deepseek connects to the DeepSeek API.
|
Package deepseek connects to the DeepSeek API. |
|
gemini
Package gemini connects to the Gemini API.
|
Package gemini connects to the Gemini API. |
|
glm
Package glm connects to the Zhipu GLM API.
|
Package glm connects to the Zhipu GLM API. |
|
grok
Package grok connects to the xAI Grok API.
|
Package grok connects to the xAI Grok API. |
|
internal/openaicompat
Package openaicompat implements the Chat Completions protocol shared by OpenAI and the OpenAI-compatible providers.
|
Package openaicompat implements the Chat Completions protocol shared by OpenAI and the OpenAI-compatible providers. |
|
internal/openaicompat/compattest
Package compattest runs openaicompat providers against canned HTTP responses in tests.
|
Package compattest runs openaicompat providers against canned HTTP responses in tests. |
|
mimo
Package mimo connects to the Xiaomi MiMo API.
|
Package mimo connects to the Xiaomi MiMo API. |
|
minimax
Package minimax connects to the MiniMax API.
|
Package minimax connects to the MiniMax API. |
|
ollama
Package ollama connects to Ollama's OpenAI-compatible endpoint.
|
Package ollama connects to Ollama's OpenAI-compatible endpoint. |
|
openai
Package openai connects to the OpenAI API, or any endpoint that implements it exactly (such as Azure OpenAI), through Chat Completions or Responses.
|
Package openai connects to the OpenAI API, or any endpoint that implements it exactly (such as Azure OpenAI), through Chat Completions or Responses. |
|
openrouter
Package openrouter connects to the OpenRouter API.
|
Package openrouter connects to the OpenRouter API. |
|
qwen
Package qwen connects to the Alibaba Cloud DashScope compatible-mode API.
|
Package qwen connects to the Alibaba Cloud DashScope compatible-mode API. |
|
Package retry provides opt-in HTTP retries for providers, paced by a Policy that callers retrying at another layer, such as a whole stream, can pace their retries with too.
|
Package retry provides opt-in HTTP retries for providers, paced by a Policy that callers retrying at another layer, such as a whole stream, can pace their retries with too. |