llm

package
v0.1.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jun 18, 2025 License: AGPL-3.0 Imports: 9 Imported by: 0

Documentation

Overview

Package llm provides a unified interface for interacting with various Large Language Model providers.

Overview

The llm package abstracts away the specific API details of each LLM provider, making it easy to switch between different models and providers. It serves as the foundation for the agent system that communicates with UIs via JSONRPC over Unix sockets.

Features

  • Provider-agnostic interface for LLMs
  • Support for both regular and streaming completions
  • Function/tool calling support with a registry system
  • Multi-modal (text + image) content support
  • Configurable with functional options pattern
  • Easy to extend with new providers

Usage

Basic usage:

import (
    "context"
    "log/slog"

    "codeberg.org/MadsRC/aigent/pkg/llm"
    "codeberg.org/MadsRC/aigent/pkg/llm/providers/openai"
)

// Create an LLM client
client := llm.New()

// Create and register a provider (OpenAI in this example)
provider := openai.New(
    openai.WithAPIKey("your-api-key-here"),
)
client.AddProvider(provider)

// Create a chat request
req := llm.ChatRequest{
    ModelID: "gpt-4",
    Messages: []llm.Message{
        {
            Role:    llm.RoleSystem,
            Content: "You are a helpful assistant.",
        },
        {
            Role:    llm.RoleUser,
            Content: "What's the weather like today?",
        },
    },
    Temperature: 0.7,
}

// Send the chat request
resp, err := client.Chat(context.Background(), "openai", req)
if err != nil {
    // Handle error
}

// Use the response
content, _ := resp.Message.Content.(string)
fmt.Println(content)

Streaming API

// Create a streaming request
req := llm.ChatRequest{
    ModelID: "gpt-4",
    Messages: []llm.Message{
        {
            Role:    llm.RoleUser,
            Content: "Write a poem about clouds.",
        },
    },
    Stream: true,
}

// Send the streaming request
stream, err := client.ChatStream(context.Background(), "openai", req)
if err != nil {
    // Handle error
}
defer stream.Close()

// Process the stream
for {
    chunk, err := stream.Next()
    if err == io.EOF {
        break
    }
    if err != nil {
        // Handle error
        break
    }

    // Process the chunk
    fmt.Print(chunk.Delta.Content)
}

Tool Calling

// Define a tool
tools := []llm.Tool{
    {
        Type: "function",
        Function: llm.Function{
            Name: "get_weather",
            Description: "Get the current weather for a location",
            Parameters: map[string]interface{}{
                "type": "object",
                "properties": map[string]interface{}{
                    "location": map[string]interface{}{
                        "type": "string",
                        "description": "City name",
                    },
                },
                "required": []string{"location"},
            },
        },
    },
}

// Create a request with tools
req := llm.ChatRequest{
    ModelID: "gpt-4",
    Messages: []llm.Message{
        {
            Role:    llm.RoleUser,
            Content: "What's the weather like in Paris?",
        },
    },
    Tools: tools,
}

Tool Registry

The package includes a ToolRegistry for managing tools and their executors:

// Create a tool registry
registry := llm.NewToolRegistry(
    llm.WithToolRegistryLogger(logger),
    llm.WithToolRegistryTool(timeTool, timeExecutor),
)

// Execute a tool call
result, err := registry.ExecuteTool(toolCall)

Multi-modal Input

// Create a multi-modal message
req := llm.ChatRequest{
    ModelID: "gpt-4-vision-preview", // Must be a vision model
    Messages: []llm.Message{
        {
            Role: llm.RoleUser,
            Content: []llm.MessageContent{
                llm.TextContent{
                    Text: "What's in this image?",
                },
                llm.ImageContent{
                    URL: "https://example.com/image.jpg",
                },
            },
        },
    },
}

Available Providers

Currently implemented providers:

  • openai: OpenAI's API (GPT-3.5, GPT-4, etc.) using the official openai-go SDK

Adding New Providers

To add a new provider, implement the llm.ProviderClient interface:

type ProviderClient interface {
    ID() string
    Name() string
    ListModels(ctx context.Context) ([]Model, error)
    GetModel(ctx context.Context, modelID string) (*Model, error)
    Chat(ctx context.Context, req ChatRequest) (*ChatResponse, error)
    ChatStream(ctx context.Context, req ChatRequest) (ChatResponseStream, error)
}

Integration with Persona System

The llm package works closely with the persona system, which defines different roles for the LLM agent. Each persona can specify its own set of tools and system prompts, and the llm package provides the underlying communication with the LLM providers.

Package llm provides an abstraction layer for interacting with various LLM providers.

Package llm provides an abstraction layer for interacting with various LLM providers.

Index

Constants

This section is empty.

Variables

View Source
var ErrProviderNotFound = errors.New("provider not found")

ErrProviderNotFound is returned when a requested provider is not registered.

View Source
var GlobalClientOptions []ClientOption

GlobalClientOptions is a list of options that are applied to all clients created.

Functions

This section is empty.

Types

type ChatRequest

type ChatRequest struct {
	// ModelID is the specific model to use.
	ModelID string
	// Messages is the conversation history.
	Messages []Message
	// MaxTokens is the maximum number of tokens to generate.
	MaxTokens int
	// Temperature controls randomness in generation (0-2).
	Temperature float32
	// TopP controls diversity of generation (0-1).
	TopP float32
	// ResponseFormat specifies the format of the response.
	ResponseFormat *ResponseFormat
	// Tools is a list of tools the model can call.
	Tools []Tool
	// ToolChoice controls how the model uses tools.
	ToolChoice *ToolChoice
	// Stream indicates if responses should be streamed.
	Stream bool
}

ChatRequest represents a request to a chat model.

type ChatResponse

type ChatResponse struct {
	// ID uniquely identifies this response.
	ID string
	// Message contains the assistant's response message.
	Message Message
	// Usage contains token usage information.
	Usage *TokenUsage
	// FinishReason indicates why the generation stopped.
	FinishReason FinishReason `json:"finish_reason,omitempty"`
}

ChatResponse represents a response from a chat model.

type ChatResponseChunk

type ChatResponseChunk struct {
	// ID uniquely identifies the response this chunk belongs to.
	ID string
	// Delta contains the new content in this chunk.
	Delta ChunkDelta
	// Finished indicates if this is the final chunk in the stream.
	Finished bool
	// FinishReason indicates why the generation stopped, if this chunk is the last one
	// and the reason is available.
	FinishReason FinishReason `json:"finish_reason,omitempty"`
}

ChatResponseChunk represents a single chunk of a streaming chat response.

type ChatResponseStream

type ChatResponseStream interface {
	// Next returns the next chunk in the stream.
	Next() (*ChatResponseChunk, error)
	// Close closes the stream.
	Close() error
	// Err returns any error that occurred during streaming, other than io.EOF.
	// This should be checked after Next() returns io.EOF.
	Err() error
}

ChatResponseStream represents a streaming response from a chat model.

type ChunkDelta

type ChunkDelta struct {
	// Content is the new content in this delta, if any.
	Content string
	// Role is the role of the message, usually only in the first chunk.
	Role Role
	// ToolCalls contains any tool calls in this delta.
	ToolCalls []ToolCall
	// ToolCallID is the ID of the tool call this message is responding to.
	ToolCallID string
	// Usage contains token usage information for this chunk.
	// Note: This may only be populated in the final chunk of a stream.
	Usage *TokenUsage
}

ChunkDelta represents the content delta in a streaming response chunk.

type Client

type Client struct {
	// contains filtered or unexported fields
}

Client provides a unified way to interact with various LLM providers.

func New

func New(options ...ClientOption) *Client

New creates a new LLM client with the given options.

func (*Client) AddProvider

func (c *Client) AddProvider(provider ProviderClient)

AddProvider registers a new provider with this client.

func (*Client) Chat

func (c *Client) Chat(ctx context.Context, providerID string, req ChatRequest) (*ChatResponse, error)

Chat sends a chat completion request to the specified provider and model.

func (*Client) ChatStream

func (c *Client) ChatStream(ctx context.Context, providerID string, req ChatRequest) (ChatResponseStream, error)

ChatStream sends a streaming chat completion request to the specified provider and model.

func (*Client) ContinueToolCall added in v0.1.0

func (c *Client) ContinueToolCall(ctx context.Context, providerID string, modelID string, messages []Message, toolResults []ToolResult, availableTools []Tool) (*ChatResponse, error)

ContinueToolCall sends tool results back to the LLM to continue a conversation. It takes the existing messages and the results of tool executions, then calls the provider's Chat method.

func (*Client) GetProvider

func (c *Client) GetProvider(providerID string) (ProviderClient, error)

GetProvider returns a provider by its ID.

func (*Client) ListModels

func (c *Client) ListModels(ctx context.Context) ([]Model, error)

ListModels returns all available models across all providers.

func (*Client) ListProviders

func (c *Client) ListProviders() []ProviderClient

ListProviders returns all registered providers.

type ClientOption

type ClientOption interface {
	// contains filtered or unexported methods
}

ClientOption configures a Client.

func WithClientLogger

func WithClientLogger(logger *slog.Logger) ClientOption

WithClientLogger sets the logger for the client.

type FinishReason added in v0.1.0

type FinishReason string

FinishReason defines why a language model's generation turn ended.

const (
	// FinishReasonUnspecified indicates an unspecified reason.
	FinishReasonUnspecified FinishReason = ""
	// FinishReasonStop indicates generation stopped due to a stop sequence.
	FinishReasonStop FinishReason = "stop"
	// FinishReasonLength indicates generation stopped due to reaching max length.
	FinishReasonLength FinishReason = "length"
	// FinishReasonToolCalls indicates generation stopped to make tool calls.
	FinishReasonToolCalls FinishReason = "tool_calls"
	// FinishReasonError indicates generation stopped due to an error.
	FinishReasonError FinishReason = "error"
	// FinishReasonContentFilter indicates generation stopped due to content filtering.
	FinishReasonContentFilter FinishReason = "content_filter"
)

type Function

type Function struct {
	// Name is the name of the function.
	Name string `json:"name"`
	// Description is a human-readable description of the function.
	Description string `json:"description,omitempty"`
	// Parameters represents the function's parameters in JSON schema format.
	Parameters map[string]any `json:"parameters,omitempty"`
}

Function represents a callable function definition.

type ImageContent

type ImageContent struct {
	URL string `json:"url"`
}

ImageContent represents image content in a message.

func (ImageContent) MessageContentType

func (i ImageContent) MessageContentType() string

MessageContentType returns the type of message content.

type MCPToolExecutor

type MCPToolExecutor struct {
	// contains filtered or unexported fields
}

MCPToolExecutor executes tools via MCP protocol

func NewMCPToolExecutor

func NewMCPToolExecutor(client *mcp.Client, logger *slog.Logger) *MCPToolExecutor

NewMCPToolExecutor creates a new MCP tool executor

func (*MCPToolExecutor) Execute

func (e *MCPToolExecutor) Execute(toolCall ToolCall) (string, error)

Execute executes a tool call via MCP

type Message

type Message struct {
	// Role is who sent this message (user, assistant, system, etc).
	Role Role `json:"role"`
	// Content is the content of the message, which can be either a string
	// for simple text messages or a slice of MessageContent for multi-modal content.
	Content any `json:"content"`
	// Name is an optional identifier for the sender, used in some providers.
	Name string `json:"name,omitempty"`
	// ToolCalls contains any tool calls made by the assistant in this message.
	ToolCalls []ToolCall `json:"tool_calls,omitempty"`
	// ToolCallID is the ID of the tool call this message is responding to.
	ToolCallID string `json:"tool_call_id,omitempty"`
}

Message represents a single message in a chat conversation.

type MessageContent

type MessageContent interface {
	MessageContentType() string
}

MessageContent represents a part of a message's content.

type Model

type Model struct {
	// ID is the unique identifier for this model within its provider.
	ID string
	// Name is a human-readable name for the model.
	Name string
	// Provider is the provider ID this model belongs to.
	Provider string
	// Capabilities describe what the model can do.
	Capabilities ModelCapabilities
}

Model represents a specific model from a provider.

type ModelCapabilities

type ModelCapabilities struct {
	// SupportsStreaming indicates if the model can stream responses.
	SupportsStreaming bool
	// SupportsJSON indicates if the model supports JSON mode responses.
	SupportsJSON bool
	// SupportsFunctions indicates if the model supports function calling.
	SupportsFunctions bool
	// SupportsVision indicates if the model supports analyzing images.
	SupportsVision bool
}

ModelCapabilities describes the capabilities of a model.

type ProviderClient

type ProviderClient interface {
	// ID returns the unique identifier for this provider.
	ID() string
	// Name returns a human-readable name for this provider.
	Name() string
	// ListModels returns all available models from this provider.
	ListModels(ctx context.Context) ([]Model, error)
	// GetModel returns information for a specific model.
	GetModel(ctx context.Context, modelID string) (*Model, error)
	// Chat sends a chat completion request to the provider.
	Chat(ctx context.Context, req ChatRequest) (*ChatResponse, error)
	// ChatStream sends a streaming chat completion request.
	ChatStream(ctx context.Context, req ChatRequest) (ChatResponseStream, error)
}

ProviderClient is an interface that must be implemented by all LLM provider clients.

type ResponseFormat

type ResponseFormat struct {
	// Type is the response format type, e.g., "json_object".
	Type string `json:"type"`
}

ResponseFormat specifies the format of the response.

type Role

type Role string

Role defines the role of a chat message sender.

const (
	RoleUser      Role = "user"
	RoleSystem    Role = "system"
	RoleAssistant Role = "assistant"
	RoleTool      Role = "tool"
)

Predefined message roles.

type StandardToolExecutor

type StandardToolExecutor struct {
	// contains filtered or unexported fields
}

StandardToolExecutor is a simple implementation of ToolExecutor for standard tools

func GetTimeExecutor

func GetTimeExecutor() *StandardToolExecutor

GetTimeExecutor returns the current time

func (*StandardToolExecutor) Execute

func (e *StandardToolExecutor) Execute(toolCall ToolCall) (string, error)

Execute executes the tool call

type TextContent

type TextContent struct {
	Text string `json:"text"`
}

TextContent represents plain text content in a message.

func (TextContent) MessageContentType

func (t TextContent) MessageContentType() string

MessageContentType returns the type of message content.

type TokenUsage

type TokenUsage struct {
	// PromptTokens is the number of tokens in the prompt.
	PromptTokens int
	// CompletionTokens is the number of tokens in the completion.
	CompletionTokens int
	// TotalTokens is the total number of tokens used.
	TotalTokens int
}

TokenUsage provides token count information.

type Tool

type Tool struct {
	// Type is the type of the tool.
	Type string `json:"type"`
	// Function is the function definition if Type is "function".
	Function Function `json:"function"`
}

Tool represents a tool that can be called by the model.

func GetTimeTool

func GetTimeTool() Tool

GetTimeTool returns a tool definition for the get_time tool

type ToolCall

type ToolCall struct {
	// ID uniquely identifies this tool call.
	ID string `json:"id"`
	// Type is the type of tool being called.
	Type string `json:"type"`
	// Function contains details about the function call if Type is "function".
	Function ToolCallFunction `json:"function"`
}

ToolCall represents a call to a tool by the model.

type ToolCallFunction

type ToolCallFunction struct {
	// Name is the name of the function to call.
	Name string `json:"name"`
	// Arguments is a JSON string containing the function arguments.
	Arguments string `json:"arguments"`
	// IsComplete indicates if the arguments field contains complete valid JSON.
	// This is particularly important during streaming where arguments may come in fragments.
	IsComplete bool `json:"-"`
}

ToolCallFunction represents a function call made by the model.

type ToolChoice

type ToolChoice struct {
	// Type is the tool choice type ("none", "auto", or "function").
	Type string `json:"type"`
	// Function is required if Type is "function".
	Function *ToolChoiceFunction `json:"function,omitempty"`
}

ToolChoice controls how the model chooses to call tools.

type ToolChoiceFunction

type ToolChoiceFunction struct {
	// Name is the name of the function.
	Name string `json:"name"`
}

ToolChoiceFunction specifies a function the model should use.

type ToolExecutor

type ToolExecutor interface {
	// Execute executes a tool call and returns the result
	Execute(toolCall ToolCall) (string, error)
}

ToolExecutor is responsible for executing tools and returning their results

type ToolRegistry

type ToolRegistry struct {
	// contains filtered or unexported fields
}

ToolRegistry maintains a registry of available tools and their implementations

func NewToolRegistry

func NewToolRegistry(opts ...ToolRegistryOption) *ToolRegistry

NewToolRegistry creates a new tool registry with optional configuration

func (*ToolRegistry) ExecuteTool

func (tr *ToolRegistry) ExecuteTool(toolCall ToolCall) (string, error)

ExecuteTool executes a tool call and returns the result

func (*ToolRegistry) GetTools

func (tr *ToolRegistry) GetTools() []Tool

GetTools returns all registered tools

func (*ToolRegistry) RegisterTool

func (tr *ToolRegistry) RegisterTool(tool Tool, executor ToolExecutor)

RegisterTool adds a tool to the registry

type ToolRegistryOption

type ToolRegistryOption func(*ToolRegistry)

ToolRegistryOption defines functional options for configuring ToolRegistry

func WithToolRegistryLogger

func WithToolRegistryLogger(logger *slog.Logger) ToolRegistryOption

WithToolRegistryLogger sets the logger for the ToolRegistry

func WithToolRegistryTool

func WithToolRegistryTool(tool Tool, executor ToolExecutor) ToolRegistryOption

WithToolRegistryTool registers a tool during initialization

type ToolResult added in v0.1.0

type ToolResult struct {
	// ID is the unique identifier of the tool call this result corresponds to.
	ID string `json:"id"`

	// ToolName is the name of the tool that was called.
	// While the LLM knows this from the original ToolCall, including it can be useful for logging and consistency.
	ToolName string `json:"tool_name"`

	// Content is the result of the tool's execution, typically a string (e.g., JSON output).
	Content string `json:"content"`

	// IsError indicates whether the tool execution resulted in an error.
	// If true, Content might contain an error message or a serialized error object.
	IsError bool `json:"is_error,omitempty"`
}

ToolResult represents the result of a tool's execution. This is sent back to the LLM to inform it of the outcome of a tool call.

Directories

Path Synopsis
providers
openai
Package openai provides an OpenAI provider implementation for the LLM client interface.
Package openai provides an OpenAI provider implementation for the LLM client interface.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL