llm

package
v1.7.5 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 26, 2026 License: Apache-2.0 Imports: 24 Imported by: 0

Documentation

Overview

Package llm adapts LLM providers to the agentcore.ChatModel interface. It wraps litellm to reach OpenAI, Anthropic, Gemini, and other backends, and classifies provider errors onto agentcore's retry and overflow contracts. Construct a model with NewModel.

Index

Constants

View Source 
const ThinkingAuto agentcore.ThinkingLevel = ""

Variables

View Source 
var DefaultGenerationConfig = &GenerationConfig{
	Temperature:      0.7,
	TopP:             0.9,
	TopK:             0,
	MaxTokens:        65536,
	StopSequences:    []string{},
	PresencePenalty:  0.0,
	FrequencyPenalty: 0.0,
	Seed:             nil,
}
View Source 
var ThinkingLevelOrder = []agentcore.ThinkingLevel{
	agentcore.ThinkingOff,
	agentcore.ThinkingMinimal,
	agentcore.ThinkingLow,
	agentcore.ThinkingMedium,
	agentcore.ThinkingHigh,
	agentcore.ThinkingXHigh,
	agentcore.ThinkingMax,
}

Functions

func CalculateCost added in v1.5.1 

func CalculateCost(pricing *ModelPricing, usage *agentcore.Usage) *agentcore.Cost

CalculateCost computes the monetary cost from pricing rates and token usage. Returns nil if pricing or usage is nil.

Pricing semantic: usage.Input already includes usage.CacheRead per the underlying litellm convention. The cached portion must only be billed at the cache-read rate; charging full Input at input rate AND CacheRead at cache-read rate would double-bill.

func IsProviderRegistered added in v1.6.9 

func IsProviderRegistered(name string) bool

IsProviderRegistered reports whether the provider name is known to this adapter.

func RegisteredProviders added in v1.6.9 

func RegisteredProviders() []string

RegisteredProviders returns all provider names known to this adapter.

func TransformMessages added in v1.5.1 

func TransformMessages(messages []agentcore.Message, targetProvider string) []agentcore.Message

TransformMessages normalizes a message sequence for a target provider. Use this when switching models mid-conversation to avoid provider rejections.

Two-pass algorithm:

  1. Normalize tool call IDs (truncate >64 chars, sanitize to [a-zA-Z0-9_-]), handle thinking blocks based on target provider.
  2. Apply ID mapping to tool results, insert synthetic results for orphaned tool calls.

Types

type BaseModel 

type BaseModel struct {
	// contains filtered or unexported fields
}

BaseModel provides common model metadata and capability checks.

func NewBaseModel 

func NewBaseModel(info ModelInfo, config *GenerationConfig) *BaseModel

func (*BaseModel) GetConfig 

func (m *BaseModel) GetConfig() *GenerationConfig

func (*BaseModel) Info 

func (m *BaseModel) Info() ModelInfo

func (*BaseModel) ModelName added in v1.6.13 

func (m *BaseModel) ModelName() string

ModelName implements agentcore.ModelNamer.

func (*BaseModel) SupportsCapability 

func (m *BaseModel) SupportsCapability(capability ModelCapability) bool

func (*BaseModel) SupportsStreaming 

func (m *BaseModel) SupportsStreaming() bool

func (*BaseModel) SupportsTools 

func (m *BaseModel) SupportsTools() bool

type Capabilities added in v1.7.3 

type Capabilities struct {
	Provider string
	Model    string

	Thinking   ThinkingCapabilities
	Tools      ToolCapabilities
	Structured StructuredCapabilities
	Streaming  StreamingCapabilities
	Usage      UsageCapabilities
}

Capabilities is agentcore's provider-neutral view of model capabilities.

func (Capabilities) ThinkingPolicy added in v1.7.4 

func (c Capabilities) ThinkingPolicy() ThinkingPolicy

type CapabilityProvider added in v1.7.3 

type CapabilityProvider interface {
	Capabilities() Capabilities
}

CapabilityProvider is implemented by models that can expose provider/model capabilities for UI preflight and configuration validation. It is advisory: request execution remains the source of truth and should still fail loudly.

type GenerationConfig 

type GenerationConfig struct {
	Temperature      float64  `json:"temperature"`
	TopP             float64  `json:"top_p"`
	TopK             int      `json:"top_k"`
	MaxTokens        int      `json:"max_tokens"`
	StopSequences    []string `json:"stop_sequences"`
	PresencePenalty  float64  `json:"presence_penalty"`
	FrequencyPenalty float64  `json:"frequency_penalty"`
	Seed             *int64   `json:"seed"`
}

GenerationConfig defines sampling and length control parameters.

type LiteLLMAdapter 

type LiteLLMAdapter struct {
	*BaseModel
	// contains filtered or unexported fields
}

LiteLLMAdapter adapts litellm to the agentcore.ChatModel interface.

func NewLiteLLMAdapter 

func NewLiteLLMAdapter(model string, client *litellm.Client) *LiteLLMAdapter

NewLiteLLMAdapter wraps an existing litellm.Client as a ChatModel. Use this when you need to reuse a Client or inject a custom Provider instance; for the common case prefer NewModel.

func NewModel added in v1.6.9 

func NewModel(provider, model string, opts ...ModelOption) (*LiteLLMAdapter, error)

NewModel constructs a ChatModel by provider name. The provider must be registered in litellm (builtin or via litellm.RegisterProvider).

func (*LiteLLMAdapter) Capabilities added in v1.7.3 

func (l *LiteLLMAdapter) Capabilities() Capabilities

Capabilities returns the provider/model capability view exposed by litellm.

func (*LiteLLMAdapter) Generate 

func (l *LiteLLMAdapter) Generate(ctx context.Context, messages []agentcore.Message, tools []agentcore.ToolSpec, opts ...agentcore.CallOption) (*agentcore.LLMResponse, error)

Generate produces a synchronous response.

func (*LiteLLMAdapter) GenerateStream 

func (l *LiteLLMAdapter) GenerateStream(ctx context.Context, messages []agentcore.Message, tools []agentcore.ToolSpec, opts ...agentcore.CallOption) (<-chan agentcore.StreamEvent, error)

GenerateStream produces a streaming response with fine-grained events.

func (*LiteLLMAdapter) ProviderName 

func (l *LiteLLMAdapter) ProviderName() string

ProviderName returns the provider name (e.g. "openai", "anthropic"). Implements agentcore.ProviderNamer for per-provider API key resolution.

type ModelCapability 

type ModelCapability string

ModelCapability defines capability identifiers. 

const (
	CapabilityChat         ModelCapability = "chat"
	CapabilityCompletion   ModelCapability = "completion"
	CapabilityToolCalling  ModelCapability = "tool_calling"
	CapabilityStreaming    ModelCapability = "streaming"
	CapabilityMultimodal   ModelCapability = "multimodal"
	CapabilityFunctionCall ModelCapability = "function_call"
)

type ModelInfo 

type ModelInfo struct {
	Name         string        `json:"name"`
	Provider     string        `json:"provider"`
	Version      string        `json:"version"`
	MaxTokens    int           `json:"max_tokens"`
	ContextSize  int           `json:"context_size"`
	Capabilities []string      `json:"capabilities"`
	Pricing      *ModelPricing `json:"pricing,omitempty"`
}

ModelInfo contains basic model metadata.

type ModelOption added in v1.6.9 

type ModelOption func(*modelConfig)

ModelOption configures NewModel.

func WithAPIKey added in v1.6.9 

func WithAPIKey(key string) ModelOption

func WithBaseURL added in v1.6.9 

func WithBaseURL(url string) ModelOption

func WithClientOptions added in v1.6.11 

func WithClientOptions(opts ...litellm.ClientOption) ModelOption

WithClientOptions forwards litellm ClientOptions (e.g. litellm.WithHook) to the underlying client, letting callers attach observability or other cross-cutting behaviour without this package importing those concerns.

func WithExtra added in v1.6.12 

func WithExtra(extra map[string]any) ModelOption

WithExtra sets model-level, provider-specific request parameters merged into every request's Extra map (e.g. min_p, presence_penalty, or provider keys like chat_template_kwargs). OpenAI-compatible providers pass these through verbatim into the request body — the extra_body convention. Per-call Extra entries (e.g. session_id) are added alongside, not overwritten.

func WithProviderExtra added in v1.7.2 

func WithProviderExtra(extra map[string]any) ModelOption

WithProviderExtra sets provider-level configuration passed to litellm.ProviderConfig.Extra. Use it for HTTP headers or other provider client options, while WithExtra remains request-body Extra.

func WithRequestTimeout added in v1.6.9 

func WithRequestTimeout(d time.Duration) ModelOption

WithRequestTimeout sets an optional per-request timeout. Zero leaves timeout control to the caller context.

func WithResilience added in v1.6.9 

func WithResilience(rc ResilienceConfig) ModelOption

WithResilience replaces the entire resilience config; later With* options may still override specific fields.

func WithStreamIdleTimeout added in v1.6.9 

func WithStreamIdleTimeout(d time.Duration) ModelOption

WithStreamIdleTimeout aborts a streaming response if no chunk arrives within the window (default 120s). Pass 0 to disable the watchdog explicitly.

type ModelPricing added in v1.5.1 

type ModelPricing struct {
	InputPerToken      float64 `json:"input_per_token"`
	OutputPerToken     float64 `json:"output_per_token"`
	CacheReadPerToken  float64 `json:"cache_read_per_token"`
	CacheWritePerToken float64 `json:"cache_write_per_token"`
}

ModelPricing defines per-token cost rates in USD. Set rates to 0 for categories that don't apply.

type ProviderConfig added in v1.6.9 

type ProviderConfig struct {
	APIKey  string
	BaseURL string
	Timeout time.Duration
	Extra   map[string]any
	Retry   *retry.Policy
}

ProviderConfig is kept for source compatibility with older agentcore callers. New litellm providers expose explicit Config structs; this package maps the common subset that agentcore needs.

type ResilienceConfig added in v1.6.9 

type ResilienceConfig struct {
	StreamIdleTimeout time.Duration
	Retry             *retry.Policy
}

ResilienceConfig is the compatibility shape for retry and stream idle knobs.

type StreamingCapabilities added in v1.7.3 

type StreamingCapabilities struct {
	Supported       Support
	Usage           Support
	ReasoningDeltas Support
	ToolCallDeltas  Support
	NativeResponses Support
	IdleTimeout     Support
}

type StructuredCapabilities added in v1.7.3 

type StructuredCapabilities struct {
	JSONObject Support
	JSONSchema Support
	Strict     Support
	PromptOnly bool
}

type Support added in v1.7.3 

type Support int

Support describes whether a model/provider supports a capability. 

const (
	SupportUnknown Support = iota
	SupportNo
	SupportYes
	SupportPartial
)

type ThinkingCapabilities added in v1.7.3 

type ThinkingCapabilities struct {
	Supported     Support
	Disable       Support
	Efforts       []agentcore.ThinkingLevel
	BudgetTokens  Support
	IncludeOutput Support
	Notes         []string
}

func (ThinkingCapabilities) SupportsEffort added in v1.7.3 

func (c ThinkingCapabilities) SupportsEffort(level agentcore.ThinkingLevel) bool

type ThinkingPolicy added in v1.7.4 

type ThinkingPolicy struct {
	Available []agentcore.ThinkingLevel
}

func ThinkingPolicyFor added in v1.7.4 

func ThinkingPolicyFor(model any) ThinkingPolicy

func ThinkingPolicyFromCapabilities added in v1.7.4 

func ThinkingPolicyFromCapabilities(caps Capabilities) ThinkingPolicy

func (ThinkingPolicy) Allows added in v1.7.4 

func (p ThinkingPolicy) Allows(level agentcore.ThinkingLevel) bool

func (ThinkingPolicy) Resolve added in v1.7.4 

func (p ThinkingPolicy) Resolve(level agentcore.ThinkingLevel) (agentcore.ThinkingLevel, bool)

type ToolCapabilities added in v1.7.3 

type ToolCapabilities struct {
	Calls               Support
	ParallelCalls       Support
	StrictSchema        Support
	Choice              Support
	MultimodalResults   Support
	RequiresAdjacency   bool
	RoundTripSignatures Support
	HostedProviderTools Support
}

type UsageCapabilities added in v1.7.3 

type UsageCapabilities struct {
	InputTokens      Support
	OutputTokens     Support
	TotalTokens      Support
	ReasoningTokens  Support
	CacheReadTokens  Support
	CacheWriteTokens Support
}

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.