llm

package
v1.8.4 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 29, 2026 License: Apache-2.0 Imports: 12 Imported by: 0

Documentation

Overview

Package llm adapts LLM providers to the agentcore.ChatModel interface. It wraps litellm to reach OpenAI, Anthropic, Gemini, and other backends, and classifies provider errors onto agentcore's retry and overflow contracts. Construct a model with NewModel.

Index

Constants

View Source
const ThinkingAuto agentcore.ThinkingLevel = ""

Variables

Functions

This section is empty.

Types

type Capabilities added in v1.7.3

type Capabilities struct {
	Thinking        bool
	DisableThinking bool
	ThinkingEffort  bool
	ThinkingBudget  bool
	ProviderOptions []string
}

Capabilities states what the provider adapter can express on the wire, independent of the model; whether a model honors a request is the vendor's call.

func (Capabilities) ThinkingPolicy added in v1.7.4

func (c Capabilities) ThinkingPolicy() ThinkingPolicy

type CapabilityProvider added in v1.7.3

type CapabilityProvider interface {
	Capabilities() (caps Capabilities, ok bool)
}

CapabilityProvider is implemented by models that expose their provider adapter's static facts for UI preflight and configuration validation; ok is false when the facts are unknown. It is advisory: request execution remains the source of truth.

type LiteLLMAdapter

type LiteLLMAdapter struct {
	// contains filtered or unexported fields
}

LiteLLMAdapter adapts a litellm.Client to agentcore.ChatModel. Unset request defaults are omitted from the wire, leaving the vendor default.

func NewLiteLLMAdapter

func NewLiteLLMAdapter(model string, client *litellm.Client, opts ...ModelOption) (*LiteLLMAdapter, error)

NewLiteLLMAdapter wraps an existing litellm.Client, to reuse a Client or inject a custom Provider. The client is already built, so WithClientOptions does not apply.

func NewModel added in v1.6.9

func NewModel(name, model string, conn provider.Config, opts ...ModelOption) (*LiteLLMAdapter, error)

NewModel builds the provider called name, one of provider.Names, and adapts it as a ChatModel. agentcore.WithAPIKey overrides conn's key for one call. Anthropic models need an output cap; see WithMaxTokensIfRequired.

func (*LiteLLMAdapter) Capabilities added in v1.7.3

func (l *LiteLLMAdapter) Capabilities() (Capabilities, bool)

Capabilities returns the provider adapter's static facts exposed by litellm.

func (*LiteLLMAdapter) Generate

func (l *LiteLLMAdapter) Generate(ctx context.Context, messages []agentcore.Message, tools []agentcore.ToolSpec, opts ...agentcore.CallOption) (*agentcore.LLMResponse, error)

Generate produces a synchronous response.

func (*LiteLLMAdapter) GenerateStream

func (l *LiteLLMAdapter) GenerateStream(ctx context.Context, messages []agentcore.Message, tools []agentcore.ToolSpec, opts ...agentcore.CallOption) (<-chan agentcore.StreamEvent, error)

GenerateStream produces a streaming response with fine-grained events.

func (*LiteLLMAdapter) ModelName added in v1.8.4

func (l *LiteLLMAdapter) ModelName() string

ModelName implements agentcore.ModelNamer.

func (*LiteLLMAdapter) ProviderName

func (l *LiteLLMAdapter) ProviderName() string

ProviderName implements agentcore.ProviderNamer.

func (*LiteLLMAdapter) SupportsTools added in v1.8.4

func (l *LiteLLMAdapter) SupportsTools() bool

SupportsTools reports true: every litellm provider carries tool calls.

type ModelOption added in v1.6.9

type ModelOption func(*modelConfig)

ModelOption configures NewModel and NewLiteLLMAdapter.

func WithClientOptions added in v1.6.11

func WithClientOptions(opts ...litellm.ClientOption) ModelOption

WithClientOptions forwards litellm ClientOptions, such as litellm.WithObservers or litellm.WithStreamIdleTimeout, to the client NewModel builds.

func WithExtra added in v1.6.12

func WithExtra(extra map[string]any) ModelOption

WithExtra sets provider options sent with every request: top-level fields of the vendor's request body, such as min_p or chat_template_kwargs. An object merges into a field the adapter also generates. OpenAI-compatible providers pass every key through; the others reject keys they do not list.

func WithMaxTokens added in v1.8.4

func WithMaxTokens(n int) ModelOption

WithMaxTokens caps output tokens; agentcore.WithMaxTokens overrides it per call. Unset, no cap is sent and the vendor default applies, except as WithMaxTokensIfRequired sets.

func WithMaxTokensIfRequired added in v1.8.4

func WithMaxTokensIfRequired(n int) ModelOption

WithMaxTokensIfRequired caps output tokens for providers that reject requests without a cap, such as Anthropic, when neither WithMaxTokens nor the call sets one; the others keep the vendor default. The catalog's Model.MaxOutputTokens holds the limit of listed models.

func WithPricing added in v1.8.4

func WithPricing(p catalog.Pricing) ModelOption

WithPricing prices each call's usage into Usage.Cost; catalog.Catalog holds the rates of listed models.

func WithRequestTimeout added in v1.6.9

func WithRequestTimeout(d time.Duration) ModelOption

WithRequestTimeout bounds each call, streaming included. Zero leaves the deadline to the caller's context.

func WithStop added in v1.8.4

func WithStop(stop ...string) ModelOption

func WithTemperature added in v1.8.4

func WithTemperature(t float64) ModelOption

func WithTopP added in v1.8.4

func WithTopP(p float64) ModelOption

type ThinkingPolicy added in v1.7.4

type ThinkingPolicy struct {
	Available []agentcore.ThinkingLevel
}

func ThinkingPolicyFor added in v1.7.4

func ThinkingPolicyFor(model any) ThinkingPolicy

func ThinkingPolicyFromCapabilities added in v1.7.4

func ThinkingPolicyFromCapabilities(caps Capabilities) ThinkingPolicy

ThinkingPolicyFromCapabilities offers auto always, off when thinking can be disabled and the effort levels when effort can be sent.

func (ThinkingPolicy) Allows added in v1.7.4

func (p ThinkingPolicy) Allows(level agentcore.ThinkingLevel) bool

func (ThinkingPolicy) Resolve added in v1.7.4

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL