llm

package
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jun 19, 2026 License: MIT Imports: 17 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func APIKeyEnvVars

func APIKeyEnvVars(provider string) []string

APIKeyEnvVars returns the environment variables checked for provider in precedence order. The returned slice is safe for the caller to modify.

func FindEnvAPIKeys

func FindEnvAPIKeys(provider string) []string

FindEnvAPIKeys returns the names of configured API key environment variables for provider, in lookup order.

func FindEnvAPIKeysWithEnv

func FindEnvAPIKeysWithEnv(provider string, env ProviderEnv) []string

FindEnvAPIKeysWithEnv is FindEnvAPIKeys with request-scoped overrides.

func GetEnvAPIKey

func GetEnvAPIKey(provider string) string

GetEnvAPIKey returns the first configured API key for provider.

func GetEnvAPIKeyWithEnv

func GetEnvAPIKeyWithEnv(provider string, env ProviderEnv) string

GetEnvAPIKeyWithEnv returns the first configured API key for provider, preferring non-empty request-scoped values over process environment values.

func GetProviders

func GetProviders() []string

GetProviders returns all providers in the built-in model registry.

func IsContextOverflow

func IsContextOverflow(message AssistantMessage, contextWindow int64) bool

IsContextOverflow reports whether an assistant message indicates that the input exceeded the model's context window. It covers three cases:

  1. Error-based overflow: most providers return StopReasonError with a recognizable error message (matched against overflowPatterns, excluding nonOverflowPatterns such as rate limits).
  2. Silent overflow (e.g. z.ai): the request succeeds but usage.Input exceeds the context window. Pass a non-zero contextWindow to detect this.
  3. Length-stop overflow (e.g. Xiaomi MiMo): the server truncates oversized input to fill the window, leaving no room to generate, so it returns StopReasonLength with zero output and input filling the window.

Pass contextWindow as the model's window size to enable cases 2 and 3; pass 0 to check error messages only.

func OverflowPatterns

func OverflowPatterns() []*regexp.Regexp

OverflowPatterns returns a copy of the overflow detection patterns, primarily for tests.

func ParseToolArguments

func ParseToolArguments(raw string) map[string]any

ParseToolArguments decodes a tool call's accumulated JSON arguments into an object. Models occasionally emit JSON with unescaped control characters or bad escape sequences; a strict decode of that input loses every argument. So this first tries a strict decode, then retries on a repaired copy, and only then gives up. It always returns a non-nil map so callers have a usable value.

func RepairJSON

func RepairJSON(source string) string

RepairJSON fixes malformed JSON string literals by escaping raw control characters inside strings and doubling backslashes before invalid escape characters, while preserving valid escapes and \uXXXX sequences. Only ASCII characters carry special meaning here, so iterating over bytes leaves multi-byte UTF-8 sequences untouched.

func ValidateToolArguments

func ValidateToolArguments(tool ToolDefinition, toolCall ToolCall) (map[string]any, error)

ValidateToolArguments coerces a tool call's arguments toward the tool's JSON Schema (forgiving common model mistakes such as "3" for a number), then validates them. It returns the coerced arguments, or a detailed error naming the failing fields. The original toolCall.Arguments are left unchanged.

The coercion mirrors pi's coerceWithJsonSchema. Validation covers the JSON Schema features normally emitted for tool definitions, including composition keywords and object, array, string, and numeric constraints.

func ValidateToolCall

func ValidateToolCall(tools []ToolDefinition, toolCall ToolCall) (map[string]any, error)

ValidateToolCall finds the tool named by the call and validates its arguments. It mirrors pi's helper: a utility callers may invoke before dispatching a tool, not something the library calls itself. It returns the coerced arguments.

Types

type AnthropicMessagesCompatibility

type AnthropicMessagesCompatibility struct {
	SupportsTemperature       *bool `json:"supportsTemperature,omitempty"`
	SupportsCacheControl      *bool `json:"supportsCacheControl,omitempty"`
	SupportsCacheControlTools *bool `json:"supportsCacheControlOnTools,omitempty"`
	ForceAdaptiveThinking     *bool `json:"forceAdaptiveThinking,omitempty"`
	AllowEmptySignature       *bool `json:"allowEmptySignature,omitempty"`
}

AnthropicMessagesCompatibility describes differences between providers that implement an Anthropic Messages-compatible endpoint. Pointer booleans distinguish an explicit false value from an unspecified provider default. Anthropic-compatible vendors (e.g. MiniMax) are served by pointing the base URL at their endpoint; most need no overrides at all.

func (*AnthropicMessagesCompatibility) Protocol

Protocol identifies the API protocol whose request and message dialect this compatibility configuration describes.

type AssistantContent

type AssistantContent interface {
	// contains filtered or unexported methods
}

AssistantContent is content that can appear in an assistant message.

type AssistantMessage

type AssistantMessage struct {
	Content       []AssistantContent `json:"content"`
	Protocol      Protocol           `json:"protocol"`
	Provider      string             `json:"provider"`
	Model         string             `json:"model"`
	ResponseModel string             `json:"responseModel,omitempty"`
	ResponseID    string             `json:"responseId,omitempty"`
	Usage         Usage              `json:"usage"`
	StopReason    StopReason         `json:"stopReason"`
	ErrorMessage  string             `json:"errorMessage,omitempty"`
	Timestamp     int64              `json:"timestamp"`
}

AssistantMessage is the final or partial response returned by a provider.

func NewAssistantMessage

func NewAssistantMessage(model Model) AssistantMessage

NewAssistantMessage initializes provider-independent response metadata.

func (AssistantMessage) MarshalJSON

func (message AssistantMessage) MarshalJSON() ([]byte, error)

func (*AssistantMessage) UnmarshalJSON

func (message *AssistantMessage) UnmarshalJSON(data []byte) error

type Client

type Client struct {
	// contains filtered or unexported fields
}

Client routes LLM requests to the adapter registered for a model protocol.

func NewClient

func NewClient(registry *Registry) *Client

NewClient creates a client backed by the given provider registry.

func (*Client) Complete

func (c *Client) Complete(
	ctx context.Context,
	model Model,
	input Context,
	options StreamOptions,
) (AssistantMessage, error)

Complete consumes a provider stream and returns the final assistant message.

func (*Client) Stream

func (c *Client) Stream(ctx context.Context, model Model, input Context, options StreamOptions) (<-chan Event, error)

Stream starts a streaming completion request for the given model and input.

type Context

type Context struct {
	SystemPrompt string           `json:"systemPrompt,omitempty"`
	Messages     []Message        `json:"messages"`
	Tools        []ToolDefinition `json:"tools,omitempty"`
}

Context contains the prompt, conversation history, and available tools.

func (Context) MarshalJSON

func (input Context) MarshalJSON() ([]byte, error)

func (*Context) UnmarshalJSON

func (input *Context) UnmarshalJSON(data []byte) error

type Event

type Event struct {
	Type EventType

	ContentIndex int

	Delta string

	Content string

	ToolCall *ToolCall

	Partial *AssistantMessage

	Message *AssistantMessage

	Err error
}

Event is a single update emitted while streaming a provider response.

type EventType

type EventType string

EventType identifies the kind of update emitted by a provider stream.

const (
	// EventStart marks the beginning of a provider stream.
	EventStart EventType = "start"
	// EventTextStart marks the creation of a text content block.
	EventTextStart EventType = "text_start"
	// EventTextDelta carries newly generated text.
	EventTextDelta EventType = "text_delta"
	// EventTextEnd carries the completed text content block.
	EventTextEnd EventType = "text_end"
	// EventThinkingStart marks the creation of a reasoning content block.
	EventThinkingStart EventType = "thinking_start"
	// EventThinkingDelta carries newly generated reasoning content.
	EventThinkingDelta EventType = "thinking_delta"
	// EventThinkingEnd carries the completed reasoning content block.
	EventThinkingEnd EventType = "thinking_end"
	// EventToolCallStart marks the creation of a tool call content block.
	EventToolCallStart EventType = "toolcall_start"
	// EventToolCallDelta carries a fragment of a tool call's arguments as it streams.
	EventToolCallDelta EventType = "toolcall_delta"
	// EventToolCallEnd carries a completed tool call request.
	EventToolCallEnd EventType = "toolcall_end"
	// EventDone carries the final assistant message.
	EventDone EventType = "done"
	// EventError carries a stream failure.
	EventError EventType = "error"
)

type ImageContent

type ImageContent struct {
	Data     string `json:"data"`
	MIMEType string `json:"mimeType"`
}

ImageContent represents a base64-encoded image.

func (ImageContent) MarshalJSON

func (content ImageContent) MarshalJSON() ([]byte, error)

type Message

type Message interface {
	// contains filtered or unexported methods
}

Message is one item in the conversation context.

func TransformMessages

func TransformMessages(messages []Message, model Model, normalizeToolCallID func(string) string) []Message

TransformMessages prepares the library's provider-independent conversation history for replay against model. Provider adapters should call it before translating Message values into their own wire-format message types.

Transformation happens per request instead of modifying the Agent's canonical history. The same history may later be sent to a model with different image, reasoning, or tool capabilities. Mutating stored history would make that model switch lose information permanently.

Shared transformations are applied in this order:

  1. Replace images with descriptive text when the target model is text-only.
  2. Reconcile assistant turns produced by a different model: keep reasoning for the same model, downgrade it to text otherwise, and normalize tool-call identifiers via normalizeToolCallID when crossing providers.
  3. Drop assistant turns terminated by an error or cancellation because they may contain partial reasoning or half-streamed tool calls.
  4. Insert synthetic error results for tool calls with no matching result before the conversation continues or ends.

normalizeToolCallID rewrites a tool-call ID for the target provider; pass nil to leave identifiers unchanged.

The returned slice is new and this function does not mutate messages. Message objects requiring no changes may still be shared with the input, so callers should treat the input and result as immutable.

type Model

type Model struct {
	ID               string                         `json:"id"`
	Name             string                         `json:"name"`
	Protocol         Protocol                       `json:"protocol"`
	Provider         string                         `json:"provider"`
	BaseURL          string                         `json:"baseUrl"`
	Reasoning        bool                           `json:"reasoning"`
	ThinkingLevelMap map[ModelThinkingLevel]*string `json:"thinkingLevelMap,omitempty"`
	Input            []ModelInput                   `json:"input"`
	Cost             ModelCost                      `json:"cost"`
	ContextWindow    int64                          `json:"contextWindow"`
	MaxTokens        int64                          `json:"maxTokens"`
	Headers          map[string]string              `json:"headers,omitempty"`
	Compatibility    ModelCompatibility             `json:"compat,omitempty"`
}

Model identifies a model, its provider endpoint, capabilities, limits, and pricing. ThinkingLevelMap values are provider-specific; nil marks a level as unsupported while a missing key uses the provider default.

func GetModel

func GetModel(provider, modelID string) Model

GetModel returns a model from the package's built-in model registry. It panics when the provider/model pair is unknown. Use LookupModel when the identifiers come from dynamic or untrusted input.

func GetModels

func GetModels(provider string) []Model

GetModels returns all built-in models for a provider.

func LookupModel

func LookupModel(provider, modelID string) (Model, bool)

LookupModel returns a model from the package's built-in model registry.

func (*Model) UnmarshalJSON

func (model *Model) UnmarshalJSON(data []byte) error

UnmarshalJSON restores the concrete compatibility type selected by Protocol. The protocol acts as the discriminator, mirroring pi's Model<TApi> conditional compatibility type at runtime.

type ModelCompatibility

type ModelCompatibility interface {
	Protocol() Protocol
}

ModelCompatibility is implemented by protocol-specific compatibility configurations. It keeps Model independent from any one provider protocol while allowing registration and adapters to verify type/protocol agreement.

type ModelCost

type ModelCost struct {
	Input      float64 `json:"input"`
	Output     float64 `json:"output"`
	CacheRead  float64 `json:"cacheRead"`
	CacheWrite float64 `json:"cacheWrite"`
}

ModelCost stores prices in US dollars per million tokens.

type ModelInput

type ModelInput string

ModelInput identifies an input modality accepted by a model.

const (
	Text  ModelInput = "text"
	Image ModelInput = "image"
)

type ModelRegistry

type ModelRegistry struct {
	// contains filtered or unexported fields
}

ModelRegistry stores models by provider and model ID. It is safe for concurrent access and returns defensive copies of registered models.

func NewModelRegistry

func NewModelRegistry() *ModelRegistry

NewModelRegistry creates an empty model registry.

func (*ModelRegistry) Get

func (registry *ModelRegistry) Get(provider, modelID string) (Model, bool)

Get returns a model registered for provider and modelID.

func (*ModelRegistry) Models

func (registry *ModelRegistry) Models(provider string) []Model

Models returns a provider's models ordered by model ID.

func (*ModelRegistry) Providers

func (registry *ModelRegistry) Providers() []string

Providers returns registered provider IDs in lexical order.

func (*ModelRegistry) Register

func (registry *ModelRegistry) Register(model Model) error

Register adds or replaces a model with the same provider and ID.

type ModelThinkingLevel

type ModelThinkingLevel string

ModelThinkingLevel is a provider-independent reasoning effort level.

const (
	ModelThinkingOff     ModelThinkingLevel = "off"
	ModelThinkingMinimal ModelThinkingLevel = "minimal"
	ModelThinkingLow     ModelThinkingLevel = "low"
	ModelThinkingMedium  ModelThinkingLevel = "medium"
	ModelThinkingHigh    ModelThinkingLevel = "high"
	ModelThinkingXHigh   ModelThinkingLevel = "xhigh"
)

func ClampThinkingLevel

func ClampThinkingLevel(model Model, level ModelThinkingLevel) ModelThinkingLevel

ClampThinkingLevel adjusts a requested level to the nearest one the model supports: it prefers the requested level, then steps up, then down, and falls back to the lowest supported level (or "off").

func SupportedThinkingLevels

func SupportedThinkingLevels(model Model) []ModelThinkingLevel

SupportedThinkingLevels returns the thinking levels a model accepts. A non-reasoning model supports only "off". For reasoning models, a level mapped to nil is unsupported, and "xhigh" is supported only when explicitly mapped.

type OpenAICompletionsCompatibility

type OpenAICompletionsCompatibility struct {
	SupportsStore                               *bool  `json:"supportsStore,omitempty"`
	SupportsDeveloperRole                       *bool  `json:"supportsDeveloperRole,omitempty"`
	SupportsReasoningEffort                     *bool  `json:"supportsReasoningEffort,omitempty"`
	MaxTokensField                              string `json:"maxTokensField,omitempty"`
	SupportsStrictMode                          *bool  `json:"supportsStrictMode,omitempty"`
	RequiresReasoningContentOnAssistantMessages *bool  `json:"requiresReasoningContentOnAssistantMessages,omitempty"`
	// RequiresThinkingAsText makes replayed assistant turns carry thinking as a
	// leading text content block instead of a provider reasoning field, for
	// endpoints that reject reasoning fields on input.
	RequiresThinkingAsText *bool  `json:"requiresThinkingAsText,omitempty"`
	ThinkingFormat         string `json:"thinkingFormat,omitempty"`
	ZAIToolStream          *bool  `json:"zaiToolStream,omitempty"`
}

OpenAICompletionsCompatibility describes differences between providers that implement an OpenAI-compatible Chat Completions endpoint. Pointer booleans distinguish an explicit false value from an unspecified provider default.

func (*OpenAICompletionsCompatibility) Protocol

Protocol identifies the API protocol whose request and message dialect this compatibility configuration describes.

type Protocol

type Protocol string

Protocol identifies the API protocol used to communicate with a model.

const (
	ProtocolOpenAICompletions Protocol = "openai-completions"
	ProtocolAnthropicMessages Protocol = "anthropic-messages"
)

type ProtocolAdapter

type ProtocolAdapter interface {
	// Protocol returns the registry key used to select this adapter.
	Protocol() Protocol

	// Stream emits response events for the given model and conversation context.
	Stream(ctx context.Context, model Model, input Context, options StreamOptions) (<-chan Event, error)
}

ProtocolAdapter translates between a concrete LLM protocol and the package streaming interface.

type ProviderEnv

type ProviderEnv map[string]string

ProviderEnv contains request-scoped environment overrides. Non-empty values take precedence over process environment variables during credential lookup.

type Registry

type Registry struct {
	// contains filtered or unexported fields
}

Registry stores protocol adapters and is safe for concurrent access.

func NewRegistry

func NewRegistry() *Registry

NewRegistry creates an empty provider registry.

func (*Registry) Get

func (registry *Registry) Get(protocol Protocol) (ProtocolAdapter, bool)

Get returns the adapter registered for the protocol.

func (*Registry) Register

func (registry *Registry) Register(adapter ProtocolAdapter) error

Register adds or replaces an adapter for its protocol.

type StopReason

type StopReason string

StopReason explains why the model stopped generating a response.

const (
	// StopReasonStop marks a normal completion.
	StopReasonStop StopReason = "stop"
	// StopReasonLength marks truncation by the max output token limit.
	StopReasonLength StopReason = "length"
	// StopReasonToolUse marks a stop to let the caller execute tool calls.
	StopReasonToolUse StopReason = "toolUse"
	// StopReasonError marks a provider or runtime failure.
	StopReasonError StopReason = "error"
	// StopReasonAborted marks a cancelled request.
	StopReasonAborted StopReason = "aborted"
)

type StreamOptions

type StreamOptions struct {
	APIKey string
	Env    ProviderEnv
	// Temperature overrides the model's default sampling temperature when set.
	Temperature *float64
	// MaxTokens caps the output tokens for this request. Zero leaves it unset.
	MaxTokens int64
	// Headers are merged into the request, overriding model default headers.
	Headers map[string]string
	// Reasoning requests a thinking level. The provider clamps it to what the
	// model supports. Empty leaves the model's default; "off" disables thinking.
	Reasoning ModelThinkingLevel
}

StreamOptions contains provider-specific settings for a stream request.

type TextContent

type TextContent struct {
	Text          string `json:"text"`
	TextSignature string `json:"textSignature,omitempty"`
}

TextContent represents plain text.

func (TextContent) MarshalJSON

func (content TextContent) MarshalJSON() ([]byte, error)

type ThinkingContent

type ThinkingContent struct {
	Thinking          string `json:"thinking"`
	ThinkingSignature string `json:"thinkingSignature,omitempty"`
	Redacted          bool   `json:"redacted,omitempty"`
}

ThinkingContent represents model reasoning content.

func (ThinkingContent) MarshalJSON

func (content ThinkingContent) MarshalJSON() ([]byte, error)

type ToolCall

type ToolCall struct {
	ID               string         `json:"id"`
	Name             string         `json:"name"`
	Arguments        map[string]any `json:"arguments"`
	ThoughtSignature string         `json:"thoughtSignature,omitempty"`
}

ToolCall describes a request to invoke a named tool with JSON arguments.

func (ToolCall) MarshalJSON

func (content ToolCall) MarshalJSON() ([]byte, error)

type ToolDefinition

type ToolDefinition struct {
	Name        string          `json:"name"`
	Description string          `json:"description"`
	Parameters  json.RawMessage `json:"parameters"`
}

ToolDefinition describes a tool that the model may call.

type ToolResultContent

type ToolResultContent interface {
	// contains filtered or unexported methods
}

ToolResultContent is content that can appear in a tool result message.

type ToolResultMessage

type ToolResultMessage struct {
	ToolCallID string              `json:"toolCallId"`
	ToolName   string              `json:"toolName"`
	Content    []ToolResultContent `json:"content"`
	IsError    bool                `json:"isError"`
}

ToolResultMessage contains the result of an assistant tool call.

func (ToolResultMessage) MarshalJSON

func (message ToolResultMessage) MarshalJSON() ([]byte, error)

func (*ToolResultMessage) UnmarshalJSON

func (message *ToolResultMessage) UnmarshalJSON(data []byte) error

type Usage

type Usage struct {
	Input       int64     `json:"input"`
	Output      int64     `json:"output"`
	CacheRead   int64     `json:"cacheRead"`
	CacheWrite  int64     `json:"cacheWrite"`
	TotalTokens int64     `json:"totalTokens"`
	Cost        UsageCost `json:"cost"`
}

Usage records token consumption for one assistant response.

type UsageCost

type UsageCost struct {
	Input      float64 `json:"input"`
	Output     float64 `json:"output"`
	CacheRead  float64 `json:"cacheRead"`
	CacheWrite float64 `json:"cacheWrite"`
	Total      float64 `json:"total"`
}

UsageCost breaks down the US dollar cost of one response by token category.

func CalculateCost

func CalculateCost(model Model, usage Usage) UsageCost

CalculateCost returns the US dollar cost of usage at the model's prices. Model costs are quoted per million tokens.

type UserContent

type UserContent interface {
	// contains filtered or unexported methods
}

UserContent is content that can appear in a user message.

type UserMessage

type UserMessage struct {
	Content []UserContent `json:"content"`
}

UserMessage contains content supplied by the user.

func (UserMessage) MarshalJSON

func (message UserMessage) MarshalJSON() ([]byte, error)

func (*UserMessage) UnmarshalJSON

func (message *UserMessage) UnmarshalJSON(data []byte) error

Directories

Path Synopsis
cmd
genmodels command
Command genmodels builds llm's checked-in model catalog from the same public catalogs used by pi-ai.
Command genmodels builds llm's checked-in model catalog from the same public catalogs used by pi-ai.
providers
anthropic
Package anthropic implements the Anthropic Messages protocol on top of the official anthropic-sdk-go.
Package anthropic implements the Anthropic Messages protocol on top of the official anthropic-sdk-go.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL