Documentation
¶
Overview ¶
Package llm provides a unified interface for interacting with various Large Language Model providers.
Overview ¶
The llm package abstracts away the specific API details of each LLM provider, making it easy to switch between different models and providers. It serves as the foundation for the agent system that communicates with UIs via JSONRPC over Unix sockets.
Features ¶
- Provider-agnostic interface for LLMs
- Support for both regular and streaming completions
- Function/tool calling support with a registry system
- Multi-modal (text + image) content support
- Configurable with functional options pattern
- Easy to extend with new providers
Usage ¶
Basic usage:
import ( "context" "log/slog" "codeberg.org/MadsRC/aigent/pkg/llm" "codeberg.org/MadsRC/aigent/pkg/llm/providers/openai" ) // Create an LLM client client := llm.New() // Create and register a provider (OpenAI in this example) provider := openai.New( openai.WithAPIKey("your-api-key-here"), ) client.AddProvider(provider) // Create a chat request req := llm.ChatRequest{ ModelID: "gpt-4", Messages: []llm.Message{ { Role: llm.RoleSystem, Content: "You are a helpful assistant.", }, { Role: llm.RoleUser, Content: "What's the weather like today?", }, }, Temperature: 0.7, } // Send the chat request resp, err := client.Chat(context.Background(), "openai", req) if err != nil { // Handle error } // Use the response content, _ := resp.Message.Content.(string) fmt.Println(content)
Streaming API ¶
// Create a streaming request req := llm.ChatRequest{ ModelID: "gpt-4", Messages: []llm.Message{ { Role: llm.RoleUser, Content: "Write a poem about clouds.", }, }, Stream: true, } // Send the streaming request stream, err := client.ChatStream(context.Background(), "openai", req) if err != nil { // Handle error } defer stream.Close() // Process the stream for { chunk, err := stream.Next() if err == io.EOF { break } if err != nil { // Handle error break } // Process the chunk fmt.Print(chunk.Delta.Content) }
Tool Calling ¶
// Define a tool tools := []llm.Tool{ { Type: "function", Function: llm.Function{ Name: "get_weather", Description: "Get the current weather for a location", Parameters: map[string]interface{}{ "type": "object", "properties": map[string]interface{}{ "location": map[string]interface{}{ "type": "string", "description": "City name", }, }, "required": []string{"location"}, }, }, }, } // Create a request with tools req := llm.ChatRequest{ ModelID: "gpt-4", Messages: []llm.Message{ { Role: llm.RoleUser, Content: "What's the weather like in Paris?", }, }, Tools: tools, }
Tool Registry ¶
The package includes a ToolRegistry for managing tools and their executors:
// Create a tool registry registry := llm.NewToolRegistry( llm.WithToolRegistryLogger(logger), llm.WithToolRegistryTool(timeTool, timeExecutor), ) // Execute a tool call result, err := registry.ExecuteTool(toolCall)
Multi-modal Input ¶
// Create a multi-modal message req := llm.ChatRequest{ ModelID: "gpt-4-vision-preview", // Must be a vision model Messages: []llm.Message{ { Role: llm.RoleUser, Content: []llm.MessageContent{ llm.TextContent{ Text: "What's in this image?", }, llm.ImageContent{ URL: "https://example.com/image.jpg", }, }, }, }, }
Available Providers ¶
Currently implemented providers:
- openai: OpenAI's API (GPT-3.5, GPT-4, etc.) using the official openai-go SDK
Adding New Providers ¶
To add a new provider, implement the llm.ProviderClient interface:
type ProviderClient interface { ID() string Name() string ListModels(ctx context.Context) ([]Model, error) GetModel(ctx context.Context, modelID string) (*Model, error) Chat(ctx context.Context, req ChatRequest) (*ChatResponse, error) ChatStream(ctx context.Context, req ChatRequest) (ChatResponseStream, error) }
Integration with Persona System ¶
The llm package works closely with the persona system, which defines different roles for the LLM agent. Each persona can specify its own set of tools and system prompts, and the llm package provides the underlying communication with the LLM providers.
Package llm provides an abstraction layer for interacting with various LLM providers.
Package llm provides an abstraction layer for interacting with various LLM providers.
Index ¶
- Variables
- type ChatRequest
- type ChatResponse
- type ChatResponseChunk
- type ChatResponseStream
- type ChunkDelta
- type Client
- func (c *Client) AddProvider(provider ProviderClient)
- func (c *Client) Chat(ctx context.Context, providerID string, req ChatRequest) (*ChatResponse, error)
- func (c *Client) ChatStream(ctx context.Context, providerID string, req ChatRequest) (ChatResponseStream, error)
- func (c *Client) ContinueToolCall(ctx context.Context, providerID string, modelID string, messages []Message, ...) (*ChatResponse, error)
- func (c *Client) GetProvider(providerID string) (ProviderClient, error)
- func (c *Client) ListModels(ctx context.Context) ([]Model, error)
- func (c *Client) ListProviders() []ProviderClient
- type ClientOption
- type FinishReason
- type Function
- type ImageContent
- type MCPToolExecutor
- type Message
- type MessageContent
- type Model
- type ModelCapabilities
- type ProviderClient
- type ResponseFormat
- type Role
- type StandardToolExecutor
- type TextContent
- type TokenUsage
- type Tool
- type ToolCall
- type ToolCallFunction
- type ToolChoice
- type ToolChoiceFunction
- type ToolExecutor
- type ToolRegistry
- type ToolRegistryOption
- type ToolResult
Constants ¶
This section is empty.
Variables ¶
var ErrProviderNotFound = errors.New("provider not found")
ErrProviderNotFound is returned when a requested provider is not registered.
var GlobalClientOptions []ClientOption
GlobalClientOptions is a list of options that are applied to all clients created.
Functions ¶
This section is empty.
Types ¶
type ChatRequest ¶
type ChatRequest struct { // ModelID is the specific model to use. ModelID string // Messages is the conversation history. Messages []Message // MaxTokens is the maximum number of tokens to generate. MaxTokens int // Temperature controls randomness in generation (0-2). Temperature float32 // TopP controls diversity of generation (0-1). TopP float32 // ResponseFormat specifies the format of the response. ResponseFormat *ResponseFormat // Tools is a list of tools the model can call. Tools []Tool // ToolChoice controls how the model uses tools. ToolChoice *ToolChoice // Stream indicates if responses should be streamed. Stream bool }
ChatRequest represents a request to a chat model.
type ChatResponse ¶
type ChatResponse struct { // ID uniquely identifies this response. ID string // Message contains the assistant's response message. Message Message // Usage contains token usage information. Usage *TokenUsage // FinishReason indicates why the generation stopped. FinishReason FinishReason `json:"finish_reason,omitempty"` }
ChatResponse represents a response from a chat model.
type ChatResponseChunk ¶
type ChatResponseChunk struct { // ID uniquely identifies the response this chunk belongs to. ID string // Delta contains the new content in this chunk. Delta ChunkDelta // Finished indicates if this is the final chunk in the stream. Finished bool // FinishReason indicates why the generation stopped, if this chunk is the last one // and the reason is available. FinishReason FinishReason `json:"finish_reason,omitempty"` }
ChatResponseChunk represents a single chunk of a streaming chat response.
type ChatResponseStream ¶
type ChatResponseStream interface { // Next returns the next chunk in the stream. Next() (*ChatResponseChunk, error) // Close closes the stream. Close() error // Err returns any error that occurred during streaming, other than io.EOF. // This should be checked after Next() returns io.EOF. Err() error }
ChatResponseStream represents a streaming response from a chat model.
type ChunkDelta ¶
type ChunkDelta struct { // Content is the new content in this delta, if any. Content string // Role is the role of the message, usually only in the first chunk. Role Role // ToolCalls contains any tool calls in this delta. ToolCalls []ToolCall // ToolCallID is the ID of the tool call this message is responding to. ToolCallID string // Usage contains token usage information for this chunk. // Note: This may only be populated in the final chunk of a stream. Usage *TokenUsage }
ChunkDelta represents the content delta in a streaming response chunk.
type Client ¶
type Client struct {
// contains filtered or unexported fields
}
Client provides a unified way to interact with various LLM providers.
func New ¶
func New(options ...ClientOption) *Client
New creates a new LLM client with the given options.
func (*Client) AddProvider ¶
func (c *Client) AddProvider(provider ProviderClient)
AddProvider registers a new provider with this client.
func (*Client) Chat ¶
func (c *Client) Chat(ctx context.Context, providerID string, req ChatRequest) (*ChatResponse, error)
Chat sends a chat completion request to the specified provider and model.
func (*Client) ChatStream ¶
func (c *Client) ChatStream(ctx context.Context, providerID string, req ChatRequest) (ChatResponseStream, error)
ChatStream sends a streaming chat completion request to the specified provider and model.
func (*Client) ContinueToolCall ¶ added in v0.1.0
func (c *Client) ContinueToolCall(ctx context.Context, providerID string, modelID string, messages []Message, toolResults []ToolResult, availableTools []Tool) (*ChatResponse, error)
ContinueToolCall sends tool results back to the LLM to continue a conversation. It takes the existing messages and the results of tool executions, then calls the provider's Chat method.
func (*Client) GetProvider ¶
func (c *Client) GetProvider(providerID string) (ProviderClient, error)
GetProvider returns a provider by its ID.
func (*Client) ListModels ¶
ListModels returns all available models across all providers.
func (*Client) ListProviders ¶
func (c *Client) ListProviders() []ProviderClient
ListProviders returns all registered providers.
type ClientOption ¶
type ClientOption interface {
// contains filtered or unexported methods
}
ClientOption configures a Client.
func WithClientLogger ¶
func WithClientLogger(logger *slog.Logger) ClientOption
WithClientLogger sets the logger for the client.
type FinishReason ¶ added in v0.1.0
type FinishReason string
FinishReason defines why a language model's generation turn ended.
const ( // FinishReasonUnspecified indicates an unspecified reason. FinishReasonUnspecified FinishReason = "" // FinishReasonStop indicates generation stopped due to a stop sequence. FinishReasonStop FinishReason = "stop" // FinishReasonLength indicates generation stopped due to reaching max length. FinishReasonLength FinishReason = "length" // FinishReasonToolCalls indicates generation stopped to make tool calls. FinishReasonToolCalls FinishReason = "tool_calls" // FinishReasonError indicates generation stopped due to an error. FinishReasonError FinishReason = "error" // FinishReasonContentFilter indicates generation stopped due to content filtering. FinishReasonContentFilter FinishReason = "content_filter" )
type Function ¶
type Function struct { // Name is the name of the function. Name string `json:"name"` // Description is a human-readable description of the function. Description string `json:"description,omitempty"` // Parameters represents the function's parameters in JSON schema format. Parameters map[string]any `json:"parameters,omitempty"` }
Function represents a callable function definition.
type ImageContent ¶
type ImageContent struct {
URL string `json:"url"`
}
ImageContent represents image content in a message.
func (ImageContent) MessageContentType ¶
func (i ImageContent) MessageContentType() string
MessageContentType returns the type of message content.
type MCPToolExecutor ¶
type MCPToolExecutor struct {
// contains filtered or unexported fields
}
MCPToolExecutor executes tools via MCP protocol
func NewMCPToolExecutor ¶
func NewMCPToolExecutor(client *mcp.Client, logger *slog.Logger) *MCPToolExecutor
NewMCPToolExecutor creates a new MCP tool executor
type Message ¶
type Message struct { // Role is who sent this message (user, assistant, system, etc). Role Role `json:"role"` // Content is the content of the message, which can be either a string // for simple text messages or a slice of MessageContent for multi-modal content. Content any `json:"content"` // Name is an optional identifier for the sender, used in some providers. Name string `json:"name,omitempty"` // ToolCalls contains any tool calls made by the assistant in this message. ToolCalls []ToolCall `json:"tool_calls,omitempty"` // ToolCallID is the ID of the tool call this message is responding to. ToolCallID string `json:"tool_call_id,omitempty"` }
Message represents a single message in a chat conversation.
type MessageContent ¶
type MessageContent interface {
MessageContentType() string
}
MessageContent represents a part of a message's content.
type Model ¶
type Model struct { // ID is the unique identifier for this model within its provider. ID string // Name is a human-readable name for the model. Name string // Provider is the provider ID this model belongs to. Provider string // Capabilities describe what the model can do. Capabilities ModelCapabilities }
Model represents a specific model from a provider.
type ModelCapabilities ¶
type ModelCapabilities struct { // SupportsStreaming indicates if the model can stream responses. SupportsStreaming bool // SupportsJSON indicates if the model supports JSON mode responses. SupportsJSON bool // SupportsFunctions indicates if the model supports function calling. SupportsFunctions bool // SupportsVision indicates if the model supports analyzing images. SupportsVision bool }
ModelCapabilities describes the capabilities of a model.
type ProviderClient ¶
type ProviderClient interface { // ID returns the unique identifier for this provider. ID() string // Name returns a human-readable name for this provider. Name() string // ListModels returns all available models from this provider. ListModels(ctx context.Context) ([]Model, error) // GetModel returns information for a specific model. GetModel(ctx context.Context, modelID string) (*Model, error) // Chat sends a chat completion request to the provider. Chat(ctx context.Context, req ChatRequest) (*ChatResponse, error) // ChatStream sends a streaming chat completion request. ChatStream(ctx context.Context, req ChatRequest) (ChatResponseStream, error) }
ProviderClient is an interface that must be implemented by all LLM provider clients.
type ResponseFormat ¶
type ResponseFormat struct { // Type is the response format type, e.g., "json_object". Type string `json:"type"` }
ResponseFormat specifies the format of the response.
type StandardToolExecutor ¶
type StandardToolExecutor struct {
// contains filtered or unexported fields
}
StandardToolExecutor is a simple implementation of ToolExecutor for standard tools
func GetTimeExecutor ¶
func GetTimeExecutor() *StandardToolExecutor
GetTimeExecutor returns the current time
type TextContent ¶
type TextContent struct {
Text string `json:"text"`
}
TextContent represents plain text content in a message.
func (TextContent) MessageContentType ¶
func (t TextContent) MessageContentType() string
MessageContentType returns the type of message content.
type TokenUsage ¶
type TokenUsage struct { // PromptTokens is the number of tokens in the prompt. PromptTokens int // CompletionTokens is the number of tokens in the completion. CompletionTokens int // TotalTokens is the total number of tokens used. TotalTokens int }
TokenUsage provides token count information.
type Tool ¶
type Tool struct { // Type is the type of the tool. Type string `json:"type"` // Function is the function definition if Type is "function". Function Function `json:"function"` }
Tool represents a tool that can be called by the model.
func GetTimeTool ¶
func GetTimeTool() Tool
GetTimeTool returns a tool definition for the get_time tool
type ToolCall ¶
type ToolCall struct { // ID uniquely identifies this tool call. ID string `json:"id"` // Type is the type of tool being called. Type string `json:"type"` // Function contains details about the function call if Type is "function". Function ToolCallFunction `json:"function"` }
ToolCall represents a call to a tool by the model.
type ToolCallFunction ¶
type ToolCallFunction struct { // Name is the name of the function to call. Name string `json:"name"` // Arguments is a JSON string containing the function arguments. Arguments string `json:"arguments"` // IsComplete indicates if the arguments field contains complete valid JSON. // This is particularly important during streaming where arguments may come in fragments. IsComplete bool `json:"-"` }
ToolCallFunction represents a function call made by the model.
type ToolChoice ¶
type ToolChoice struct { // Type is the tool choice type ("none", "auto", or "function"). Type string `json:"type"` // Function is required if Type is "function". Function *ToolChoiceFunction `json:"function,omitempty"` }
ToolChoice controls how the model chooses to call tools.
type ToolChoiceFunction ¶
type ToolChoiceFunction struct { // Name is the name of the function. Name string `json:"name"` }
ToolChoiceFunction specifies a function the model should use.
type ToolExecutor ¶
type ToolExecutor interface { // Execute executes a tool call and returns the result Execute(toolCall ToolCall) (string, error) }
ToolExecutor is responsible for executing tools and returning their results
type ToolRegistry ¶
type ToolRegistry struct {
// contains filtered or unexported fields
}
ToolRegistry maintains a registry of available tools and their implementations
func NewToolRegistry ¶
func NewToolRegistry(opts ...ToolRegistryOption) *ToolRegistry
NewToolRegistry creates a new tool registry with optional configuration
func (*ToolRegistry) ExecuteTool ¶
func (tr *ToolRegistry) ExecuteTool(toolCall ToolCall) (string, error)
ExecuteTool executes a tool call and returns the result
func (*ToolRegistry) GetTools ¶
func (tr *ToolRegistry) GetTools() []Tool
GetTools returns all registered tools
func (*ToolRegistry) RegisterTool ¶
func (tr *ToolRegistry) RegisterTool(tool Tool, executor ToolExecutor)
RegisterTool adds a tool to the registry
type ToolRegistryOption ¶
type ToolRegistryOption func(*ToolRegistry)
ToolRegistryOption defines functional options for configuring ToolRegistry
func WithToolRegistryLogger ¶
func WithToolRegistryLogger(logger *slog.Logger) ToolRegistryOption
WithToolRegistryLogger sets the logger for the ToolRegistry
func WithToolRegistryTool ¶
func WithToolRegistryTool(tool Tool, executor ToolExecutor) ToolRegistryOption
WithToolRegistryTool registers a tool during initialization
type ToolResult ¶ added in v0.1.0
type ToolResult struct { // ID is the unique identifier of the tool call this result corresponds to. ID string `json:"id"` // ToolName is the name of the tool that was called. // While the LLM knows this from the original ToolCall, including it can be useful for logging and consistency. ToolName string `json:"tool_name"` // Content is the result of the tool's execution, typically a string (e.g., JSON output). Content string `json:"content"` // IsError indicates whether the tool execution resulted in an error. // If true, Content might contain an error message or a serialized error object. IsError bool `json:"is_error,omitempty"` }
ToolResult represents the result of a tool's execution. This is sent back to the LLM to inform it of the outcome of a tool call.