Documentation
¶
Overview ¶
Package llm adapts LLM providers to the agentcore.ChatModel interface. It wraps litellm to reach OpenAI, Anthropic, Gemini, and other backends, and classifies provider errors onto agentcore's retry and overflow contracts. Construct a model with NewModel.
Index ¶
- Constants
- Variables
- type Capabilities
- type CapabilityProvider
- type LiteLLMAdapter
- func (l *LiteLLMAdapter) Capabilities() (Capabilities, bool)
- func (l *LiteLLMAdapter) Generate(ctx context.Context, messages []agentcore.Message, tools []agentcore.ToolSpec, ...) (*agentcore.LLMResponse, error)
- func (l *LiteLLMAdapter) GenerateStream(ctx context.Context, messages []agentcore.Message, tools []agentcore.ToolSpec, ...) (<-chan agentcore.StreamEvent, error)
- func (l *LiteLLMAdapter) ModelName() string
- func (l *LiteLLMAdapter) ProviderName() string
- func (l *LiteLLMAdapter) SupportsTools() bool
- type ModelOption
- func WithClientOptions(opts ...litellm.ClientOption) ModelOption
- func WithExtra(extra map[string]any) ModelOption
- func WithMaxTokens(n int) ModelOption
- func WithMaxTokensIfRequired(n int) ModelOption
- func WithPricing(p catalog.Pricing) ModelOption
- func WithRequestTimeout(d time.Duration) ModelOption
- func WithStop(stop ...string) ModelOption
- func WithTemperature(t float64) ModelOption
- func WithTopP(p float64) ModelOption
- type ThinkingPolicy
Constants ¶
const ThinkingAuto agentcore.ThinkingLevel = ""
Variables ¶
var ThinkingLevelOrder = []agentcore.ThinkingLevel{ agentcore.ThinkingOff, agentcore.ThinkingMinimal, agentcore.ThinkingLow, agentcore.ThinkingMedium, agentcore.ThinkingHigh, agentcore.ThinkingXHigh, agentcore.ThinkingMax, }
Functions ¶
This section is empty.
Types ¶
type Capabilities ¶ added in v1.7.3
type Capabilities struct {
Thinking bool
DisableThinking bool
ThinkingEffort bool
ThinkingBudget bool
ProviderOptions []string
}
Capabilities states what the provider adapter can express on the wire, independent of the model; whether a model honors a request is the vendor's call.
func (Capabilities) ThinkingPolicy ¶ added in v1.7.4
func (c Capabilities) ThinkingPolicy() ThinkingPolicy
type CapabilityProvider ¶ added in v1.7.3
type CapabilityProvider interface {
Capabilities() (caps Capabilities, ok bool)
}
CapabilityProvider is implemented by models that expose their provider adapter's static facts for UI preflight and configuration validation; ok is false when the facts are unknown. It is advisory: request execution remains the source of truth.
type LiteLLMAdapter ¶
type LiteLLMAdapter struct {
// contains filtered or unexported fields
}
LiteLLMAdapter adapts a litellm.Client to agentcore.ChatModel. Unset request defaults are omitted from the wire, leaving the vendor default.
func NewLiteLLMAdapter ¶
func NewLiteLLMAdapter(model string, client *litellm.Client, opts ...ModelOption) (*LiteLLMAdapter, error)
NewLiteLLMAdapter wraps an existing litellm.Client, to reuse a Client or inject a custom Provider. The client is already built, so WithClientOptions does not apply.
func NewModel ¶ added in v1.6.9
func NewModel(name, model string, conn provider.Config, opts ...ModelOption) (*LiteLLMAdapter, error)
NewModel builds the provider called name, one of provider.Names, and adapts it as a ChatModel. agentcore.WithAPIKey overrides conn's key for one call. Anthropic models need an output cap; see WithMaxTokensIfRequired.
func (*LiteLLMAdapter) Capabilities ¶ added in v1.7.3
func (l *LiteLLMAdapter) Capabilities() (Capabilities, bool)
Capabilities returns the provider adapter's static facts exposed by litellm.
func (*LiteLLMAdapter) Generate ¶
func (l *LiteLLMAdapter) Generate(ctx context.Context, messages []agentcore.Message, tools []agentcore.ToolSpec, opts ...agentcore.CallOption) (*agentcore.LLMResponse, error)
Generate produces a synchronous response.
func (*LiteLLMAdapter) GenerateStream ¶
func (l *LiteLLMAdapter) GenerateStream(ctx context.Context, messages []agentcore.Message, tools []agentcore.ToolSpec, opts ...agentcore.CallOption) (<-chan agentcore.StreamEvent, error)
GenerateStream produces a streaming response with fine-grained events.
func (*LiteLLMAdapter) ModelName ¶ added in v1.8.4
func (l *LiteLLMAdapter) ModelName() string
ModelName implements agentcore.ModelNamer.
func (*LiteLLMAdapter) ProviderName ¶
func (l *LiteLLMAdapter) ProviderName() string
ProviderName implements agentcore.ProviderNamer.
func (*LiteLLMAdapter) SupportsTools ¶ added in v1.8.4
func (l *LiteLLMAdapter) SupportsTools() bool
SupportsTools reports true: every litellm provider carries tool calls.
type ModelOption ¶ added in v1.6.9
type ModelOption func(*modelConfig)
ModelOption configures NewModel and NewLiteLLMAdapter.
func WithClientOptions ¶ added in v1.6.11
func WithClientOptions(opts ...litellm.ClientOption) ModelOption
WithClientOptions forwards litellm ClientOptions, such as litellm.WithObservers or litellm.WithStreamIdleTimeout, to the client NewModel builds.
func WithExtra ¶ added in v1.6.12
func WithExtra(extra map[string]any) ModelOption
WithExtra sets provider options sent with every request: top-level fields of the vendor's request body, such as min_p or chat_template_kwargs. An object merges into a field the adapter also generates. OpenAI-compatible providers pass every key through; the others reject keys they do not list.
func WithMaxTokens ¶ added in v1.8.4
func WithMaxTokens(n int) ModelOption
WithMaxTokens caps output tokens; agentcore.WithMaxTokens overrides it per call. Unset, no cap is sent and the vendor default applies, except as WithMaxTokensIfRequired sets.
func WithMaxTokensIfRequired ¶ added in v1.8.4
func WithMaxTokensIfRequired(n int) ModelOption
WithMaxTokensIfRequired caps output tokens for providers that reject requests without a cap, such as Anthropic, when neither WithMaxTokens nor the call sets one; the others keep the vendor default. The catalog's Model.MaxOutputTokens holds the limit of listed models.
func WithPricing ¶ added in v1.8.4
func WithPricing(p catalog.Pricing) ModelOption
WithPricing prices each call's usage into Usage.Cost; catalog.Catalog holds the rates of listed models.
func WithRequestTimeout ¶ added in v1.6.9
func WithRequestTimeout(d time.Duration) ModelOption
WithRequestTimeout bounds each call, streaming included. Zero leaves the deadline to the caller's context.
func WithStop ¶ added in v1.8.4
func WithStop(stop ...string) ModelOption
func WithTemperature ¶ added in v1.8.4
func WithTemperature(t float64) ModelOption
func WithTopP ¶ added in v1.8.4
func WithTopP(p float64) ModelOption
type ThinkingPolicy ¶ added in v1.7.4
type ThinkingPolicy struct {
Available []agentcore.ThinkingLevel
}
func ThinkingPolicyFor ¶ added in v1.7.4
func ThinkingPolicyFor(model any) ThinkingPolicy
func ThinkingPolicyFromCapabilities ¶ added in v1.7.4
func ThinkingPolicyFromCapabilities(caps Capabilities) ThinkingPolicy
ThinkingPolicyFromCapabilities offers auto always, off when thinking can be disabled and the effort levels when effort can be sent.
func (ThinkingPolicy) Allows ¶ added in v1.7.4
func (p ThinkingPolicy) Allows(level agentcore.ThinkingLevel) bool
func (ThinkingPolicy) Resolve ¶ added in v1.7.4
func (p ThinkingPolicy) Resolve(level agentcore.ThinkingLevel) (agentcore.ThinkingLevel, bool)