llm

package
v1.6.13 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 13, 2026 License: Apache-2.0 Imports: 10 Imported by: 0

Documentation

Overview

Package llm adapts LLM providers to the agentcore.ChatModel interface. It wraps litellm to reach OpenAI, Anthropic, Gemini, and other backends, and classifies provider errors onto agentcore's retry and overflow contracts. Construct a model with NewModel.

Index

Constants

This section is empty.

Variables

View Source 
var DefaultGenerationConfig = &GenerationConfig{
	Temperature:      0.7,
	TopP:             0.9,
	TopK:             0,
	MaxTokens:        65536,
	StopSequences:    []string{},
	PresencePenalty:  0.0,
	FrequencyPenalty: 0.0,
	Seed:             nil,
}

Functions

func CalculateCost added in v1.5.1 

func CalculateCost(pricing *ModelPricing, usage *agentcore.Usage) *agentcore.Cost

CalculateCost computes the monetary cost from pricing rates and token usage. Returns nil if pricing or usage is nil.

Pricing semantic: usage.Input already includes usage.CacheRead per the underlying litellm convention. The cached portion must only be billed at the cache-read rate; charging full Input at input rate AND CacheRead at cache-read rate would double-bill.

func IsProviderRegistered added in v1.6.9 

func IsProviderRegistered(name string) bool

IsProviderRegistered reports whether the provider name is known to litellm.

func RegisteredProviders added in v1.6.9 

func RegisteredProviders() []string

RegisteredProviders returns all provider names known to litellm (builtin + custom).

func TransformMessages added in v1.5.1 

func TransformMessages(messages []agentcore.Message, targetProvider string) []agentcore.Message

TransformMessages normalizes a message sequence for a target provider. Use this when switching models mid-conversation to avoid provider rejections.

Two-pass algorithm:

  1. Normalize tool call IDs (truncate >64 chars, sanitize to [a-zA-Z0-9_-]), handle thinking blocks based on target provider.
  2. Apply ID mapping to tool results, insert synthetic results for orphaned tool calls.

Types

type BaseModel 

type BaseModel struct {
	// contains filtered or unexported fields
}

BaseModel provides common model metadata and capability checks.

func NewBaseModel 

func NewBaseModel(info ModelInfo, config *GenerationConfig) *BaseModel

func (*BaseModel) GetConfig 

func (m *BaseModel) GetConfig() *GenerationConfig

func (*BaseModel) Info 

func (m *BaseModel) Info() ModelInfo

func (*BaseModel) ModelName added in v1.6.13 

func (m *BaseModel) ModelName() string

ModelName implements agentcore.ModelNamer.

func (*BaseModel) SupportsCapability 

func (m *BaseModel) SupportsCapability(capability ModelCapability) bool

func (*BaseModel) SupportsStreaming 

func (m *BaseModel) SupportsStreaming() bool

func (*BaseModel) SupportsTools 

func (m *BaseModel) SupportsTools() bool

type GenerationConfig 

type GenerationConfig struct {
	Temperature      float64  `json:"temperature"`
	TopP             float64  `json:"top_p"`
	TopK             int      `json:"top_k"`
	MaxTokens        int      `json:"max_tokens"`
	StopSequences    []string `json:"stop_sequences"`
	PresencePenalty  float64  `json:"presence_penalty"`
	FrequencyPenalty float64  `json:"frequency_penalty"`
	Seed             *int64   `json:"seed"`
}

GenerationConfig defines sampling and length control parameters.

type LiteLLMAdapter 

type LiteLLMAdapter struct {
	*BaseModel
	// contains filtered or unexported fields
}

LiteLLMAdapter adapts litellm to the agentcore.ChatModel interface.

func NewLiteLLMAdapter 

func NewLiteLLMAdapter(model string, client *litellm.Client) *LiteLLMAdapter

NewLiteLLMAdapter wraps an existing litellm.Client as a ChatModel. Use this when you need to reuse a Client or inject a custom Provider instance; for the common case prefer NewModel.

func NewModel added in v1.6.9 

func NewModel(provider, model string, opts ...ModelOption) (*LiteLLMAdapter, error)

NewModel constructs a ChatModel by provider name. The provider must be registered in litellm (builtin or via litellm.RegisterProvider).

func (*LiteLLMAdapter) Generate 

func (l *LiteLLMAdapter) Generate(ctx context.Context, messages []agentcore.Message, tools []agentcore.ToolSpec, opts ...agentcore.CallOption) (*agentcore.LLMResponse, error)

Generate produces a synchronous response.

func (*LiteLLMAdapter) GenerateStream 

func (l *LiteLLMAdapter) GenerateStream(ctx context.Context, messages []agentcore.Message, tools []agentcore.ToolSpec, opts ...agentcore.CallOption) (<-chan agentcore.StreamEvent, error)

GenerateStream produces a streaming response with fine-grained events. Delegates lifecycle detection to litellm's CollectStreamWithCallbacks.

func (*LiteLLMAdapter) ProviderName 

func (l *LiteLLMAdapter) ProviderName() string

ProviderName returns the provider name (e.g. "openai", "anthropic"). Implements agentcore.ProviderNamer for per-provider API key resolution.

type ModelCapability 

type ModelCapability string

ModelCapability defines capability identifiers. 

const (
	CapabilityChat         ModelCapability = "chat"
	CapabilityCompletion   ModelCapability = "completion"
	CapabilityToolCalling  ModelCapability = "tool_calling"
	CapabilityStreaming    ModelCapability = "streaming"
	CapabilityMultimodal   ModelCapability = "multimodal"
	CapabilityFunctionCall ModelCapability = "function_call"
)

type ModelInfo 

type ModelInfo struct {
	Name         string        `json:"name"`
	Provider     string        `json:"provider"`
	Version      string        `json:"version"`
	MaxTokens    int           `json:"max_tokens"`
	ContextSize  int           `json:"context_size"`
	Capabilities []string      `json:"capabilities"`
	Pricing      *ModelPricing `json:"pricing,omitempty"`
}

ModelInfo contains basic model metadata.

type ModelOption added in v1.6.9 

type ModelOption func(*modelConfig)

ModelOption configures NewModel.

func WithAPIKey added in v1.6.9 

func WithAPIKey(key string) ModelOption

func WithBaseURL added in v1.6.9 

func WithBaseURL(url string) ModelOption

func WithClientOptions added in v1.6.11 

func WithClientOptions(opts ...litellm.ClientOption) ModelOption

WithClientOptions forwards litellm ClientOptions (e.g. litellm.WithHook) to the underlying client, letting callers attach observability or other cross-cutting behaviour without this package importing those concerns.

func WithExtra added in v1.6.12 

func WithExtra(extra map[string]any) ModelOption

WithExtra sets model-level, provider-specific request parameters merged into every request's Extra map (e.g. min_p, presence_penalty, or provider keys like chat_template_kwargs). OpenAI-compatible providers pass these through verbatim into the request body — the extra_body convention. Per-call Extra entries (e.g. session_id) are added alongside, not overwritten.

func WithRequestTimeout added in v1.6.9 

func WithRequestTimeout(d time.Duration) ModelOption

WithRequestTimeout sets the per-request timeout (default 10m).

func WithResilience added in v1.6.9 

func WithResilience(rc ResilienceConfig) ModelOption

WithResilience replaces the entire resilience config; later With* options may still override specific fields.

func WithStreamIdleTimeout added in v1.6.9 

func WithStreamIdleTimeout(d time.Duration) ModelOption

WithStreamIdleTimeout aborts a streaming response if no chunk arrives within the window (default 120s). Pass 0 to disable the watchdog explicitly.

type ModelPricing added in v1.5.1 

type ModelPricing struct {
	InputPerToken      float64 `json:"input_per_token"`
	OutputPerToken     float64 `json:"output_per_token"`
	CacheReadPerToken  float64 `json:"cache_read_per_token"`
	CacheWritePerToken float64 `json:"cache_write_per_token"`
}

ModelPricing defines per-token cost rates in USD. Set rates to 0 for categories that don't apply.

type ProviderConfig added in v1.6.9 

type ProviderConfig = litellm.ProviderConfig

Re-exports so callers do not need to import litellm directly.

type ResilienceConfig added in v1.6.9 

type ResilienceConfig = litellm.ResilienceConfig

Re-exports so callers do not need to import litellm directly.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.