llm

package
v1.0.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 11, 2026 License: AGPL-3.0 Imports: 20 Imported by: 0

Documentation

Overview

Package llm provides interfaces and implementations for LLM backends.

This package defines the LLMClient interface for interacting with language models (Anthropic Claude, OpenAI, local models, etc.) and provides streaming support for real-time token generation.

Architecture

The package follows the interface-first pattern:

  • LLMClient interface defines the contract
  • AnthropicClient implements for Claude models
  • Additional implementations can be added for other backends

Streaming

Streaming is implemented via callback pattern. The ChatStream method calls a callback for each token as it's generated, enabling real-time display in CLI and SSE endpoints.

Thread Safety

All implementations must be safe for concurrent use.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func SafeLogString added in v1.0.0 

func SafeLogString(s string) string

SafeLogString redacts known secret patterns from a string before logging.

Description: 

Iterates through a predefined set of regex patterns that match common
API key formats, bearer tokens, passwords, and connection strings.
Each match is replaced with a labeled placeholder (e.g., [REDACTED:openai_key])
so the log reader knows what class of secret was present without seeing
the actual value.

Inputs:

  • s: The string to redact. May contain zero or more secrets. Empty string is valid and returns empty string.

Outputs:

  • string: The input with all matched secret patterns replaced. If no patterns match, returns the original string unchanged.

Examples: 

SafeLogString("error: sk-ant-api03-abc123def456ghi789jkl012 returned 401")
// Returns: "error: [REDACTED:anthropic_key] returned 401"

SafeLogString("normal log message with no secrets")
// Returns: "normal log message with no secrets"

SafeLogString("key=AIzaSyAbcDefGhiJklMnoPqrStUvWxYz01234567 in URL")
// Returns: "key=[REDACTED] in URL"

Limitations:

  • Pattern-based detection only. Cannot detect secrets that do not match known formats (e.g., custom API keys with non-standard prefixes).
  • This is NOT cryptographically secure redaction. It catches common patterns per IT-00b lesson C-11 (regex can't do semantic work).
  • A secret that spans multiple lines will not be matched (single-line regex).

Assumptions:

  • The input string is a single log line or error message.
  • Patterns are ordered most-specific-first (see redactionPatterns comment).
  • The caller does not rely on the exact redacted output format for parsing.

Thread Safety: This function is safe for concurrent use.

Types

type AnthropicClient 

type AnthropicClient struct {
	// contains filtered or unexported fields
}

func NewAnthropicClient 

func NewAnthropicClient() (*AnthropicClient, error)

func NewAnthropicClientWithConfig added in v1.0.0 

func NewAnthropicClientWithConfig(apiKey, model, baseURL string) *AnthropicClient

NewAnthropicClientWithConfig creates an AnthropicClient with explicit configuration.

Description: 

Creates an AnthropicClient without reading environment variables. Useful
for testing with mock servers or when configuration comes from a source
other than environment variables.

Inputs:

  • apiKey: The Anthropic API key.
  • model: The model name (e.g., "claude-sonnet-4-20250514").
  • baseURL: The base URL for API requests.

Outputs:

  • *AnthropicClient: The configured client.

func (*AnthropicClient) Chat 

func (a *AnthropicClient) Chat(ctx context.Context, messages []datatypes.Message, params GenerationParams) (string, error)

Chat implements the LLMClient interface

func (*AnthropicClient) ChatStream 

func (a *AnthropicClient) ChatStream(
	ctx context.Context,
	messages []datatypes.Message,
	params GenerationParams,
	callback StreamCallback,
) error

ChatStream implements streaming chat for the LLMClient interface.

Description

Sends a chat request to Anthropic with streaming enabled, then reads the SSE response line-by-line and calls the callback for each token. Handles both regular text tokens and thinking tokens.

Inputs

  • ctx: Context for cancellation and timeout.
  • messages: Conversation history.
  • params: Generation parameters.
  • callback: Called for each streaming event.

Outputs

  • error: Non-nil on network failure, API error, or callback abort.

Examples 

err := client.ChatStream(ctx, messages, params, func(e StreamEvent) error {
    if e.Type == StreamEventToken {
        fmt.Print(e.Content)
    }
    return nil
})

Limitations

  • Requires valid Anthropic API key
  • Timeout applies to entire stream duration

Assumptions

  • Anthropic API is available
  • Network is stable for stream duration

func (*AnthropicClient) ChatWithTools added in v1.0.0 

func (a *AnthropicClient) ChatWithTools(ctx context.Context, messages []ChatMessage,
	params GenerationParams, tools []ToolDef) (*ChatWithToolsResult, error)

ChatWithTools sends a chat request with tool definitions and returns tool calls.

Description: 

Extends Chat to support Anthropic's native function calling API. Converts
generic ToolDef and ChatMessage types to Anthropic wire format, including
structured content blocks for tool_use and tool_result messages.

Inputs:

  • ctx: Context for cancellation and timeout.
  • messages: Conversation history with tool metadata.
  • params: Generation parameters.
  • tools: Tool definitions for function calling.

Outputs:

  • *ChatWithToolsResult: Content and/or tool calls.
  • error: Non-nil on failure.

Thread Safety: This method is safe for concurrent use.

func (*AnthropicClient) Generate 

func (a *AnthropicClient) Generate(ctx context.Context, prompt string, params GenerationParams) (string, error)

Generate implements the LLMClient interface

type ChatMessage added in v1.0.0 

type ChatMessage struct {
	// Role is the message role: "system", "user", "assistant", or "tool".
	Role string `json:"role"`

	// Content is the text content of the message.
	Content string `json:"content,omitempty"`

	// ToolCalls contains tool invocations (for assistant messages).
	ToolCalls []ToolCallResponse `json:"tool_calls,omitempty"`

	// ToolCallID links this message back to a specific tool call (for tool result messages).
	ToolCallID string `json:"tool_call_id,omitempty"`

	// ToolName is the tool name for tool result messages. Required by Gemini's functionResponse.
	ToolName string `json:"tool_name,omitempty"`
}

ChatMessage is a richer message type that carries tool call metadata.

Description: 

Regular messages use Role + Content. Tool results include ToolCallID.
Assistant messages with tool calls include ToolCalls.
This bridges the gap between datatypes.Message (which lacks tool call IDs)
and the wire formats required by each provider.

Thread Safety: ChatMessage is safe for concurrent read access.

type ChatWithToolsResult added in v1.0.0 

type ChatWithToolsResult struct {
	// Content is the text response (may be empty if only tool calls).
	Content string

	// ToolCalls contains tool calls from the model.
	ToolCalls []ToolCallResponse

	// StopReason indicates why generation stopped.
	// Values: "end" (normal completion) or "tool_use" (tool calls present).
	StopReason string
}

ChatWithToolsResult is the provider-agnostic result from ChatWithTools.

Description: 

Contains the LLM response including any tool calls. All provider
clients return this from their ChatWithTools method.

Thread Safety: ChatWithToolsResult is safe for concurrent read access.

type DefaultStreamProcessor 

type DefaultStreamProcessor struct {
	// contains filtered or unexported fields
}

DefaultStreamProcessor implements StreamProcessor with configurable behavior.

Description

DefaultStreamProcessor handles chunk processing with support for: - Thinking token redaction (privacy) - Response length limits (safety) - Thinking length limits (safety) - Rate limiting (backpressure)

Fields

  • cfg: Stream configuration.
  • rateLimiter: Optional rate limiter for callback invocations.
  • tokenCount: Running count of content tokens.
  • responseLen: Running total of response characters.
  • thinkingLen: Running total of thinking characters.

Thread Safety

Not thread-safe. Use one instance per stream.

Examples 

processor := NewDefaultStreamProcessor(StreamConfig{RedactThinking: true}, nil)
done, err := processor.ProcessChunk(ctx, chunk, callback)

Limitations

  • Single use per stream

Assumptions

  • Chunks arrive in order

func NewDefaultStreamProcessor 

func NewDefaultStreamProcessor(cfg StreamConfig, rateLimiter *rate.Limiter) *DefaultStreamProcessor

NewDefaultStreamProcessor creates a new DefaultStreamProcessor.

Description

Creates a processor with the given configuration and optional rate limiter. If cfg.RateLimitPerSecond > 0 and rateLimiter is nil, creates one automatically.

Inputs

  • cfg: Stream configuration.
  • rateLimiter: Optional pre-configured rate limiter. If nil and rate limiting is configured, one will be created.

Outputs

  • *DefaultStreamProcessor: Configured processor ready for use.

Examples 

// Auto-create rate limiter from config
p := NewDefaultStreamProcessor(StreamConfig{RateLimitPerSecond: 50}, nil)

// Use custom rate limiter
limiter := rate.NewLimiter(100, 10)
p := NewDefaultStreamProcessor(StreamConfig{}, limiter)

Limitations

  • Rate limiter is shared if passed in; be careful with concurrent streams

Assumptions

  • cfg has reasonable values (non-negative limits)

func (*DefaultStreamProcessor) GetResponseLength 

func (p *DefaultStreamProcessor) GetResponseLength() int

GetResponseLength returns the total response characters processed.

Description

Returns the total number of characters from content tokens, after any truncation from length limits.

Outputs

  • int: Total response characters.

Examples 

length := processor.GetResponseLength()

Limitations

  • Does not include thinking content

Assumptions

  • Called after processing is complete or for progress reporting

func (*DefaultStreamProcessor) GetTokenCount 

func (p *DefaultStreamProcessor) GetTokenCount() int

GetTokenCount returns the number of content tokens processed.

Description

Returns the running count of content tokens (chunks with non-empty Message.Content) processed so far.

Outputs

  • int: Number of content tokens.

Examples 

count := processor.GetTokenCount()

Limitations

  • Does not count thinking tokens

Assumptions

  • Called after processing is complete or for progress reporting

func (*DefaultStreamProcessor) ProcessChunk 

func (p *DefaultStreamProcessor) ProcessChunk(ctx context.Context, chunk *ollamaStreamChunk, callback StreamCallback) (bool, error)

ProcessChunk processes a single chunk and emits appropriate events.

Description

Handles a single NDJSON chunk from Ollama streaming response: 1. Checks for error in chunk and emits error event 2. Processes thinking content (if present and not redacted) 3. Processes content tokens 4. Applies length limits and rate limiting 5. Returns done status from chunk

Inputs

  • ctx: Context for cancellation and rate limiter waiting.
  • chunk: Parsed Ollama stream chunk.
  • callback: Callback to invoke for each event.

Outputs

  • bool: True if chunk.Done is true (stream complete).
  • error: Non-nil on callback error, rate limiter error, or chunk error.

Examples 

chunk := &ollamaStreamChunk{Message: datatypes.Message{Content: "Hello"}}
done, err := processor.ProcessChunk(ctx, chunk, callback)

Limitations

  • Truncates content silently when limits exceeded
  • Logging should be added for truncation events

Assumptions

  • chunk is non-nil and valid
  • callback handles events quickly

type GeminiClient added in v1.0.0 

type GeminiClient struct {
	// contains filtered or unexported fields
}

GeminiClient implements LLMClient for Google Gemini models.

Description: 

Uses the Gemini REST API (generateContent) for chat and generation.
Supports text generation and multi-turn conversations.

Thread Safety: GeminiClient is safe for concurrent use.

func NewGeminiClient added in v1.0.0 

func NewGeminiClient() (*GeminiClient, error)

func NewGeminiClientWithConfig added in v1.0.0 

func NewGeminiClientWithConfig(apiKey, model, baseURL string) *GeminiClient

NewGeminiClient creates a new GeminiClient from environment variables.

Description: 

Reads GEMINI_API_KEY and GEMINI_MODEL from the environment.
Defaults to "gemini-1.5-flash" if GEMINI_MODEL is not set.

Outputs:

  • *GeminiClient: The configured client.
  • error: Non-nil if GEMINI_API_KEY is missing.

NewGeminiClientWithConfig creates a GeminiClient with explicit configuration.

Description: 

Creates a GeminiClient without reading environment variables. Useful
for testing with mock servers or when configuration comes from a source
other than environment variables.

Inputs:

Outputs:

  • *GeminiClient: The configured client.

func (*GeminiClient) Chat added in v1.0.0 

func (g *GeminiClient) Chat(ctx context.Context, messages []datatypes.Message, params GenerationParams) (string, error)

Chat implements LLMClient.Chat using the Gemini generateContent API.

func (*GeminiClient) ChatStream added in v1.0.0 

func (g *GeminiClient) ChatStream(ctx context.Context, messages []datatypes.Message,
	params GenerationParams, callback StreamCallback) error

ChatStream implements LLMClient.ChatStream. Currently not implemented for Gemini.

func (*GeminiClient) ChatWithTools added in v1.0.0 

func (g *GeminiClient) ChatWithTools(ctx context.Context, messages []ChatMessage,
	params GenerationParams, tools []ToolDef) (*ChatWithToolsResult, error)

ChatWithTools sends a chat request with tool definitions and returns tool calls.

Description: 

Extends Chat to support Gemini's function calling API. Converts generic
ToolDef and ChatMessage types to Gemini wire format, including
functionCall and functionResponse parts.

Inputs:

  • ctx: Context for cancellation and timeout.
  • messages: Conversation history with tool metadata.
  • params: Generation parameters.
  • tools: Tool definitions for function calling.

Outputs:

  • *ChatWithToolsResult: Content and/or tool calls.
  • error: Non-nil on failure.

Thread Safety: This method is safe for concurrent use.

func (*GeminiClient) Generate added in v1.0.0 

func (g *GeminiClient) Generate(ctx context.Context, prompt string, params GenerationParams) (string, error)

Generate implements LLMClient.Generate using the Gemini API.

type GenerationParams 

type GenerationParams struct {
	Temperature     *float32      `json:"temperature"`
	TopK            *int          `json:"top_k"`
	TopP            *float32      `json:"top_p"`
	MaxTokens       *int          `json:"max_tokens"`
	Stop            []string      `json:"stop"`
	ToolDefinitions []interface{} `json:"tools,omitempty"`
	EnableThinking  bool          `json:"thinking,omitempty"`
	BudgetTokens    int           `json:"budget_tokens,omitempty"`

	// ModelOverride allows overriding the client's default model for this request.
	// Used for multi-model scenarios (e.g., tool routing with a fast model).
	// Empty string means use the client's default model.
	ModelOverride string `json:"model_override,omitempty"`

	// KeepAlive controls how long the model stays loaded in VRAM after the request.
	// Values: "-1" = infinite, "5m" = 5 minutes (default), "0" = unload immediately.
	// Used to prevent model thrashing when alternating between models.
	KeepAlive string `json:"keep_alive,omitempty"`

	// NumCtx sets the context window size (number of tokens).
	// For models like Granite4 with Mamba-2 architecture, this can be set high (e.g., 131072).
	// If nil or 0, uses Ollama's default (typically 2048-4096).
	NumCtx *int `json:"num_ctx,omitempty"`

	// ToolChoice controls tool selection behavior when tools are provided.
	// If nil, defaults to "auto" (model decides).
	// Used to force tool usage at the API level for analytical queries.
	// Fixed in cb_30a to pass through to Ollama adapter.
	ToolChoice *ToolChoice `json:"tool_choice,omitempty"`
}

GenerationParams holds parameters for LLM generation.

Description

Contains all configurable parameters for text generation including temperature, sampling parameters, and tool definitions. These parameters control the LLM's output behavior.

Fields

  • Temperature: Sampling temperature (0.0-1.0). Lower = more deterministic. nil uses the model's default.
  • TopK: Sample from top K tokens. nil uses model default.
  • TopP: Nucleus sampling threshold. nil uses model default.
  • MaxTokens: Maximum tokens to generate. nil uses model default.
  • Stop: Stop sequences to halt generation. Empty means no custom stops.
  • ToolDefinitions: Tool schemas for function calling (Claude tools format).
  • EnableThinking: Enable Claude extended thinking mode. Only works with Claude models that support extended thinking.
  • BudgetTokens: Token budget for thinking (max 65536). Only used when EnableThinking is true.
  • ModelOverride: Override the client's default model for this request. Used for multi-model scenarios like tool routing.
  • KeepAlive: Controls model VRAM lifetime. "-1" = infinite, "5m" = 5 min, "0" = unload immediately. Prevents model thrashing.

Examples 

// Default parameters
params := GenerationParams{}

// Custom temperature
temp := float32(0.7)
params := GenerationParams{Temperature: &temp}

// Extended thinking
params := GenerationParams{EnableThinking: true, BudgetTokens: 4096}

Limitations

  • Not all parameters are supported by all backends
  • EnableThinking only works with Claude models

Assumptions

  • nil values mean "use model default"

type HFTransformersClient 

type HFTransformersClient struct {
	// contains filtered or unexported fields
}

func (*HFTransformersClient) Chat 

func (h *HFTransformersClient) Chat(ctx context.Context, messages []datatypes.Message,
	params GenerationParams) (string, error)

Chat conducts a conversation with message history.

Description

Currently not implemented for HFTransformersClient. Returns an error indicating that this backend is not implemented.

Inputs

  • ctx: Context for cancellation and timeout.
  • messages: Conversation history.
  • params: Generation parameters.

Outputs

  • string: Empty string.
  • error: Always returns not implemented error.

Limitations

  • Not implemented.

Assumptions

  • None.

func (*HFTransformersClient) ChatStream 

func (h *HFTransformersClient) ChatStream(ctx context.Context, messages []datatypes.Message,
	params GenerationParams, callback StreamCallback) error

ChatStream streams a conversation response token-by-token.

Description

Currently not implemented for HFTransformersClient. Returns an error indicating that streaming is not supported for this backend.

Inputs

  • ctx: Context for cancellation and timeout.
  • messages: Conversation history.
  • params: Generation parameters.
  • callback: Callback for streaming events.

Outputs

  • error: Always returns ErrStreamingNotSupported.

Limitations

  • Streaming is not implemented for HuggingFace Transformers backend.

Assumptions

  • None.

func (*HFTransformersClient) Generate 

func (h *HFTransformersClient) Generate(ctx context.Context, prompt string,
	params GenerationParams) (string, error)

Generate produces text from a single prompt.

Description

Currently not implemented for HFTransformersClient. Returns an error indicating that this backend is not implemented.

Inputs

  • ctx: Context for cancellation and timeout.
  • prompt: Text prompt to complete.
  • params: Generation parameters.

Outputs

  • string: Empty string.
  • error: Always returns not implemented error.

Limitations

  • Not implemented.

Assumptions

  • None.

type LLMClient 

type LLMClient interface {
	// Generate produces text from a single prompt.
	//
	// # Description
	//
	// Sends a prompt to the LLM and returns the generated text.
	// This is a simple completion API without conversation context.
	// Prefer Chat for conversational interactions.
	//
	// # Inputs
	//
	//   - ctx: Context for cancellation and timeout. When cancelled,
	//     the method returns with context.Canceled error.
	//   - prompt: Text prompt to complete. Must not be empty.
	//   - params: Generation parameters. Use empty struct for defaults.
	//
	// # Outputs
	//
	//   - string: Generated text response.
	//   - error: Non-nil on network failure, API error, or cancellation.
	//
	// # Examples
	//
	//	response, err := client.Generate(ctx, "Explain OAuth in one sentence", GenerationParams{})
	//
	// # Limitations
	//
	//   - No conversation context (stateless)
	//   - Some backends may not support this method
	//
	// # Assumptions
	//
	//   - Prompt is within model's context window
	Generate(ctx context.Context, prompt string, params GenerationParams) (string, error)

	// Chat conducts a conversation with message history.
	//
	// # Description
	//
	// Sends a conversation (system, user, assistant messages) to the LLM
	// and returns the assistant's response. This is a blocking call that
	// waits for the complete response.
	//
	// # Inputs
	//
	//   - ctx: Context for cancellation and timeout.
	//   - messages: Conversation history. Must have at least one message.
	//     Messages should alternate user/assistant with optional system.
	//   - params: Generation parameters.
	//
	// # Outputs
	//
	//   - string: Assistant's complete response.
	//   - error: Non-nil on failure.
	//
	// # Examples
	//
	//	messages := []datatypes.Message{
	//	    {Role: "system", Content: "You are helpful."},
	//	    {Role: "user", Content: "What is 2+2?"},
	//	}
	//	response, err := client.Chat(ctx, messages, GenerationParams{})
	//
	// # Limitations
	//
	//   - Blocks until complete response received
	//   - No partial results on timeout
	//
	// # Assumptions
	//
	//   - Messages are well-formed with valid roles
	//   - Total tokens (input + output) within context window
	Chat(ctx context.Context, messages []datatypes.Message, params GenerationParams) (string, error)

	// ChatStream conducts a conversation with streaming response.
	//
	// # Description
	//
	// Like Chat, but streams the response token-by-token via callback.
	// Enables real-time display of generation progress. The callback
	// is called for each token as it's generated.
	//
	// If an error occurs during streaming, the callback is called with
	// a StreamEventError event before the method returns. This allows
	// callers to see partial results before the error.
	//
	// # Inputs
	//
	//   - ctx: Context for cancellation and timeout. Cancellation stops
	//     streaming and returns context.Canceled.
	//   - messages: Conversation history.
	//   - params: Generation parameters.
	//   - callback: Called for each streaming event. Return error to abort.
	//
	// # Outputs
	//
	//   - error: Non-nil on failure or if callback returns error.
	//     If streaming error occurs, callback receives error event first.
	//
	// # Examples
	//
	//	var fullResponse strings.Builder
	//	err := client.ChatStream(ctx, messages, params, func(e StreamEvent) error {
	//	    switch e.Type {
	//	    case StreamEventToken:
	//	        fullResponse.WriteString(e.Content)
	//	        fmt.Print(e.Content)  // Real-time display
	//	    case StreamEventThinking:
	//	        fmt.Printf("[thinking] %s", e.Content)
	//	    case StreamEventError:
	//	        fmt.Printf("[error] %s", e.Error)
	//	    }
	//	    return nil
	//	})
	//
	// # Limitations
	//
	//   - Callback errors abort the stream
	//   - No automatic retry on transient failures
	//
	// # Assumptions
	//
	//   - Backend supports streaming API
	//   - Callback handles events quickly
	ChatStream(ctx context.Context, messages []datatypes.Message, params GenerationParams, callback StreamCallback) error
}

LLMClient defines the standard interface for any LLM backend.

Description

LLMClient abstracts LLM interactions, enabling different backends (Anthropic Claude, OpenAI, local models) to be used interchangeably. The interface provides both blocking and streaming methods.

Methods

  • Generate: Single-prompt completion (legacy, prefer Chat)
  • Chat: Blocking conversation with full response
  • ChatStream: Streaming conversation with token-by-token callbacks

Thread Safety

Implementations must be safe for concurrent use. Multiple goroutines may call methods simultaneously.

Limitations

  • Not all backends support all features (e.g., extended thinking)
  • Streaming requires backend support

Assumptions

  • Backend is configured and authenticated before use
  • Context cancellation is respected

type LocalLlamaCppClient 

type LocalLlamaCppClient struct {
	// contains filtered or unexported fields
}

func NewLocalLlamaCppClient 

func NewLocalLlamaCppClient() (*LocalLlamaCppClient, error)

func (*LocalLlamaCppClient) Chat 

func (l *LocalLlamaCppClient) Chat(ctx context.Context, messages []datatypes.Message,
	params GenerationParams) (string, error)

Chat TODO: Implement

func (*LocalLlamaCppClient) ChatStream 

func (l *LocalLlamaCppClient) ChatStream(ctx context.Context, messages []datatypes.Message,
	params GenerationParams, callback StreamCallback) error

ChatStream streams a conversation response token-by-token.

Description

Currently not implemented for LocalLlamaCppClient. Returns an error indicating that streaming is not supported for this backend.

Inputs

  • ctx: Context for cancellation and timeout.
  • messages: Conversation history.
  • params: Generation parameters.
  • callback: Callback for streaming events.

Outputs

  • error: Always returns ErrStreamingNotSupported.

Limitations

  • Streaming is not implemented for llama.cpp backend.

Assumptions

  • None.

func (*LocalLlamaCppClient) Generate 

func (l *LocalLlamaCppClient) Generate(ctx context.Context, prompt string,
	params GenerationParams) (string, error)

Generate implements the LLMClient interface

type LocalLlamaCppClientPayload 

type LocalLlamaCppClientPayload struct {
	Prompt      string   `json:"prompt"`
	NPredict    int      `json:"n_predict"`
	Temperature *float32 `json:"temperature,omitempty"`
	TopK        *int     `json:"top_k,omitempty"`
	TopP        *float32 `json:"top_p,omitempty"`
	MaxTokens   *int     `json:"max_tokens,omitempty"`
	Stop        []string `json:"stop,omitempty"`
}

type ManagedModel added in v1.0.0 

type ManagedModel struct {
	// Name is the model identifier (e.g., "granite4:micro-h").
	Name string `json:"name"`

	// KeepAlive is the keep_alive setting for this model.
	// "-1" = infinite, "5m" = 5 minutes, "0" = unload immediately.
	KeepAlive string `json:"keep_alive"`

	// IsLoaded indicates whether the model is currently loaded in VRAM.
	IsLoaded bool `json:"is_loaded"`

	// LoadedAt is when the model was loaded into VRAM.
	LoadedAt time.Time `json:"loaded_at"`

	// LastUsed is when the model was last used for inference.
	LastUsed time.Time `json:"last_used"`

	// LoadDuration is how long it took to load the model.
	LoadDuration time.Duration `json:"load_duration"`

	// WarmupError contains any error from the warmup attempt.
	WarmupError error `json:"-"`
}

ManagedModel tracks a model's lifecycle state.

Description

Tracks whether a model is loaded, when it was loaded, and its keep_alive setting. Used by MultiModelManager to manage model lifecycle and detect warming issues.

type ModelWarmupConfig added in v1.0.0 

type ModelWarmupConfig struct {
	// Model is the model name (e.g., "granite4:micro-h").
	Model string

	// KeepAlive controls how long the model stays loaded.
	// "-1" = infinite (recommended for multi-model), "5m" = 5 minutes.
	KeepAlive string

	// Priority determines loading order. Higher = load first.
	Priority int

	// MaxWaitMs is the timeout for warmup in milliseconds.
	// 0 means use default (60 seconds).
	MaxWaitMs int

	// NumCtx is the context window size for this model.
	// MUST be set to prevent Ollama from using default 4096.
	// Recommended: 16384 for router, 65536 for main agent.
	NumCtx int
}

ModelWarmupConfig specifies how to warm a model.

Description

Configuration for warming a model including the keep_alive setting and priority for loading order.

type MultiModelManager added in v1.0.0 

type MultiModelManager struct {
	// contains filtered or unexported fields
}

MultiModelManager coordinates multiple Ollama models to prevent thrashing.

Description

Ollama by default unloads models when a different model is requested, which causes "thrashing" when alternating between models (e.g., tool router + main LLM). MultiModelManager uses keep_alive to keep multiple models loaded in VRAM.

Thread Safety

MultiModelManager is safe for concurrent use.

Example 

mgr := NewMultiModelManager("http://localhost:11434")
err := mgr.WarmModels(ctx, []ModelWarmupConfig{
    {Model: "granite4:micro-h", KeepAlive: "-1", Priority: 1},
    {Model: "glm-4.7-flash", KeepAlive: "-1", Priority: 2},
})
if err != nil {
    log.Fatal(err)
}
// Now both models are loaded and will stay in VRAM
resp, err := mgr.Chat(ctx, "granite4:micro-h", messages, params)

func NewMultiModelManager added in v1.0.0 

func NewMultiModelManager(baseURL string) *MultiModelManager

NewMultiModelManager creates a new MultiModelManager.

Description

Creates a manager for coordinating multiple Ollama models. The manager tracks model state and uses keep_alive to prevent thrashing.

Inputs

Outputs

  • *MultiModelManager: Configured manager ready for use.

Example 

mgr := NewMultiModelManager("http://localhost:11434")

func (*MultiModelManager) Chat added in v1.0.0 

func (m *MultiModelManager) Chat(ctx context.Context, model string,
	messages []datatypes.Message, params GenerationParams) (string, error)

Chat sends a chat request to a specific model.

Description

Routes the request to the specified model with keep_alive preservation. Updates last-used timestamp for the model.

Inputs

  • ctx: Context for cancellation/timeout.
  • model: Which model to use (e.g., "granite4:micro-h").
  • messages: Conversation history.
  • params: Generation parameters.

Outputs

  • string: Response content.
  • error: Non-nil on failure.

Thread Safety

This method is safe for concurrent use.

func (*MultiModelManager) ChatWithTools added in v1.0.0 

func (m *MultiModelManager) ChatWithTools(ctx context.Context, model string,
	messages []datatypes.Message, params GenerationParams,
	tools []OllamaTool) (*ChatWithToolsResult, error)

ChatWithTools sends a chat request with tools to a specific model.

Description

Like Chat but with tool definitions for function calling.

Inputs

  • ctx: Context for cancellation/timeout.
  • model: Which model to use.
  • messages: Conversation history.
  • params: Generation parameters.
  • tools: Tool definitions for function calling.

Outputs

  • *ChatWithToolsResult: Content and/or tool calls.
  • error: Non-nil on failure.

func (*MultiModelManager) GetLoadedModels added in v1.0.0 

func (m *MultiModelManager) GetLoadedModels() []ManagedModel

GetLoadedModels returns currently tracked models.

Description

Returns a snapshot of all models that have been warmed or used. Note: This doesn't query Ollama directly; it returns tracked state.

Outputs

  • []ManagedModel: Copy of tracked model states.

func (*MultiModelManager) UnloadModel added in v1.0.0 

func (m *MultiModelManager) UnloadModel(ctx context.Context, model string) error

UnloadModel explicitly unloads a model from VRAM.

Description

Sends a request with keep_alive="0" to immediately unload the model. Use this for cleanup when a session ends.

Inputs

  • ctx: Context for cancellation.
  • model: Model to unload.

Outputs

  • error: Non-nil if unload fails.

func (*MultiModelManager) WarmModel added in v1.0.0 

func (m *MultiModelManager) WarmModel(ctx context.Context, model string, keepAlive string, numCtx int) error

WarmModel loads a single model into VRAM with keep_alive.

Description

Sends a minimal chat request to load the model and set keep_alive. Uses a simple ping message to minimize token usage.

Inputs

  • ctx: Context for cancellation.
  • model: Model name (e.g., "granite4:micro-h").
  • keepAlive: Keep alive setting ("-1" for infinite).

Outputs

  • error: Non-nil if the model fails to load.

func (*MultiModelManager) WarmModels added in v1.0.0 

func (m *MultiModelManager) WarmModels(ctx context.Context, configs []ModelWarmupConfig) error

WarmModels pre-loads multiple models into VRAM.

Description

Loads models into VRAM by sending minimal requests with keep_alive set. Models are loaded in priority order (highest first). This prevents cold-start latency on first real request.

Inputs

  • ctx: Context for cancellation.
  • configs: Models to warm with their configurations.

Outputs

  • error: Non-nil if any model fails to load.

Example 

err := mgr.WarmModels(ctx, []ModelWarmupConfig{
    {Model: "glm-4.7-flash", KeepAlive: "-1", Priority: 1},
    {Model: "granite4:micro-h", KeepAlive: "-1", Priority: 2},
})

Limitations

  • Models are loaded sequentially by priority to avoid VRAM contention.
  • If VRAM is insufficient, later models may evict earlier ones.

type OllamaClient 

type OllamaClient struct {
	// contains filtered or unexported fields
}

func NewOllamaClient 

func NewOllamaClient() (*OllamaClient, error)

func (*OllamaClient) Chat 

func (o *OllamaClient) Chat(ctx context.Context, messages []datatypes.Message,
	params GenerationParams) (string, error)

func (*OllamaClient) ChatStream 

func (o *OllamaClient) ChatStream(ctx context.Context, messages []datatypes.Message,
	params GenerationParams, callback StreamCallback) error

ChatStream streams a conversation response token-by-token.

Description

Streams responses from Ollama's /api/chat endpoint with stream=true. Each token is delivered via the callback as a StreamEvent. Supports thinking models (gpt-oss, DeepSeek-R1) which emit thinking tokens. Uses default StreamConfig.

Inputs

  • ctx: Context for cancellation and timeout. Cancellation stops streaming.
  • messages: Conversation history with system, user, assistant messages.
  • params: Generation parameters (temperature, max_tokens, etc).
  • callback: Called for each token. Return error to abort streaming.

Outputs

  • error: Non-nil on network failure, API error, or callback error.

Examples 

var response strings.Builder
err := client.ChatStream(ctx, messages, params, func(e StreamEvent) error {
    switch e.Type {
    case StreamEventToken:
        response.WriteString(e.Content)
        fmt.Print(e.Content)
    case StreamEventThinking:
        fmt.Printf("[thinking] %s", e.Content)
    case StreamEventError:
        return fmt.Errorf("stream error: %s", e.Error)
    }
    return nil
})

Limitations

  • Requires Ollama server to support streaming
  • Thinking field only populated by thinking-capable models

Assumptions

  • Ollama server is running and accessible
  • Model supports chat API

func (*OllamaClient) ChatStreamWithConfig 

func (o *OllamaClient) ChatStreamWithConfig(ctx context.Context, messages []datatypes.Message,
	params GenerationParams, callback StreamCallback, cfg StreamConfig) error

ChatStreamWithConfig streams with explicit configuration.

Description

Like ChatStream but accepts a StreamConfig for fine-grained control over privacy (thinking redaction), rate limiting, and length limits. This is the primary implementation; ChatStream delegates to this.

Inputs

  • ctx: Context for cancellation and timeout.
  • messages: Conversation history.
  • params: Generation parameters.
  • callback: Streaming event callback.
  • cfg: Streaming configuration.

Outputs

  • error: Non-nil on failure.

Examples 

cfg := StreamConfig{RedactThinking: true, RateLimitPerSecond: 50}
err := client.ChatStreamWithConfig(ctx, messages, params, callback, cfg)

Limitations

  • Rate limiting adds latency
  • Truncation loses content

Assumptions

  • Config values are reasonable (not negative, etc)

func (*OllamaClient) ChatWithTools added in v1.0.0 

func (o *OllamaClient) ChatWithTools(ctx context.Context, messages []datatypes.Message,
	params GenerationParams, tools []OllamaTool) (*ChatWithToolsResult, error)

ChatWithTools sends a chat request with tools and returns both content and tool calls.

Description: 

Extends Chat to support Ollama's function calling API. When tools are provided,
the model may respond with tool_calls instead of (or in addition to) content.

Inputs: 

ctx - Context for cancellation and timeout.
messages - Conversation history.
params - Generation parameters.
tools - Tool definitions for function calling.

Outputs: 

*ChatWithToolsResult - Content and/or tool calls.
error - Non-nil on failure.

Thread Safety: This method is safe for concurrent use.

func (*OllamaClient) Generate 

func (o *OllamaClient) Generate(ctx context.Context, prompt string,
	params GenerationParams) (string, error)

Generate implements the LLMClient interface

type OllamaFunctionCall added in v1.0.0 

type OllamaFunctionCall struct {
	Name      string          `json:"name"`
	Arguments json.RawMessage `json:"arguments"`
}

OllamaFunctionCall contains the function name and arguments.

Note: Arguments can be either a JSON string or a JSON object depending on the model. Some models (like glm-4.7-flash) return arguments as an object, while others (OpenAI-compatible) return it as a JSON-encoded string.

func (*OllamaFunctionCall) ArgumentsString added in v1.0.0 

func (f *OllamaFunctionCall) ArgumentsString() string

ArgumentsString returns the arguments as a JSON string. If arguments is already a string, it returns the unquoted string. If arguments is an object, it returns the JSON-encoded string.

type OllamaParamDef added in v1.0.0 

type OllamaParamDef struct {
	Type        string `json:"type"`
	Description string `json:"description,omitempty"`
	Enum        []any  `json:"enum,omitempty"`
	Default     any    `json:"default,omitempty"`
}

OllamaParamDef defines a single parameter in JSON Schema format.

type OllamaTool added in v1.0.0 

type OllamaTool struct {
	Type     string             `json:"type"`
	Function OllamaToolFunction `json:"function"`
}

OllamaTool represents a tool definition for Ollama's function calling API.

Description: 

Ollama uses an OpenAI-compatible format for tool definitions.
This structure is serialized in the "tools" array of chat requests.

Example: 

{
  "type": "function",
  "function": {
    "name": "read_file",
    "description": "Read contents of a file",
    "parameters": {...}
  }
}

type OllamaToolCall added in v1.0.0 

type OllamaToolCall struct {
	ID       string             `json:"id,omitempty"`
	Type     string             `json:"type,omitempty"`
	Function OllamaFunctionCall `json:"function"`
}

OllamaToolCall represents a tool call from Ollama's response.

type OllamaToolFunction added in v1.0.0 

type OllamaToolFunction struct {
	Name        string               `json:"name"`
	Description string               `json:"description"`
	Parameters  OllamaToolParameters `json:"parameters"`
}

OllamaToolFunction contains the function definition within an OllamaTool.

type OllamaToolParameters added in v1.0.0 

type OllamaToolParameters struct {
	Type       string                    `json:"type"`
	Properties map[string]OllamaParamDef `json:"properties,omitempty"`
	Required   []string                  `json:"required,omitempty"`
}

OllamaToolParameters defines the JSON Schema for tool parameters.

type OpenAIClient 

type OpenAIClient struct {
	// contains filtered or unexported fields
}

OpenAIClient implements LLMClient for OpenAI models using raw net/http.

Description: 

Uses the OpenAI Chat Completions REST API directly without third-party SDKs.
Supports text generation, multi-turn conversations, and function calling.

Thread Safety: OpenAIClient is safe for concurrent use.

func NewOpenAIClient 

func NewOpenAIClient() (*OpenAIClient, error)

func NewOpenAIClientWithConfig added in v1.0.0 

func NewOpenAIClientWithConfig(apiKey, model, baseURL string) *OpenAIClient

NewOpenAIClient creates a new OpenAIClient from environment variables.

Description: 

Reads OPENAI_API_KEY and OPENAI_MODEL from the environment.
Defaults to "gpt-4o-mini" if OPENAI_MODEL is not set.

Outputs:

  • *OpenAIClient: The configured client.
  • error: Non-nil if OPENAI_API_KEY is missing.

NewOpenAIClientWithConfig creates an OpenAIClient with explicit configuration.

Description: 

Creates an OpenAIClient without reading environment variables. Useful
for testing with mock servers or when configuration comes from a source
other than environment variables.

Inputs:

  • apiKey: The OpenAI API key.
  • model: The model name (e.g., "gpt-4o").
  • baseURL: The base URL for API requests.

Outputs:

  • *OpenAIClient: The configured client.

func (*OpenAIClient) Chat 

func (o *OpenAIClient) Chat(ctx context.Context, messages []datatypes.Message, params GenerationParams) (string, error)

Chat implements LLMClient.Chat using the OpenAI chat completions API.

Description: 

Converts datatypes.Message to OpenAI format and sends a chat completion
request via raw HTTP. Handles system, user, and assistant roles.

Inputs:

  • ctx: Context for cancellation and timeout.
  • messages: Conversation history.
  • params: Generation parameters.

Outputs:

  • string: The assistant's response text.
  • error: Non-nil if the request fails.

Thread Safety: This method is safe for concurrent use.

func (*OpenAIClient) ChatStream 

func (o *OpenAIClient) ChatStream(ctx context.Context, messages []datatypes.Message,
	params GenerationParams, callback StreamCallback) error

ChatStream streams a conversation response token-by-token.

Description: 

Currently not implemented for OpenAIClient. Returns an error
indicating that streaming is not supported for this backend.

Inputs:

  • ctx: Context for cancellation and timeout.
  • messages: Conversation history.
  • params: Generation parameters.
  • callback: Callback for streaming events.

Outputs:

  • error: Always returns an error.

Limitations:

  • Streaming is not implemented for OpenAI backend.

func (*OpenAIClient) ChatWithTools added in v1.0.0 

func (o *OpenAIClient) ChatWithTools(ctx context.Context, messages []ChatMessage,
	params GenerationParams, tools []ToolDef) (*ChatWithToolsResult, error)

ChatWithTools sends a chat request with tool definitions and returns tool calls.

Description: 

Extends Chat to support OpenAI's function calling API. Converts generic
ToolDef and ChatMessage types to OpenAI wire format, sends the request,
and parses tool_calls from the response.

Inputs:

  • ctx: Context for cancellation and timeout.
  • messages: Conversation history with tool metadata.
  • params: Generation parameters.
  • tools: Tool definitions for function calling.

Outputs:

  • *ChatWithToolsResult: Content and/or tool calls.
  • error: Non-nil on failure.

Thread Safety: This method is safe for concurrent use.

func (*OpenAIClient) Generate 

func (o *OpenAIClient) Generate(ctx context.Context, prompt string, params GenerationParams) (string, error)

Generate implements the LLMClient interface.

type StreamCallback 

type StreamCallback func(event StreamEvent) error

StreamCallback is called for each event during streaming.

Description

StreamCallback receives events as they are generated by the LLM. Return an error to abort streaming (e.g., on client disconnect). The callback should process events quickly to avoid backpressure.

Inputs

  • event: The streaming event (token, thinking, or error).

Outputs

  • error: Non-nil to abort streaming. The ChatStream method will return this error after cleanup.

Examples 

callback := func(event StreamEvent) error {
    switch event.Type {
    case StreamEventToken:
        fmt.Print(event.Content)
    case StreamEventThinking:
        fmt.Printf("[thinking] %s", event.Content)
    case StreamEventError:
        return fmt.Errorf("stream error: %s", event.Error)
    }
    return nil
}

Limitations

  • Must handle events quickly to avoid backpressure
  • Errors abort the stream immediately

Assumptions

  • Called in event order (tokens arrive in generation order)
  • Called from a single goroutine (no concurrent calls)

type StreamConfig 

type StreamConfig struct {
	RedactThinking     bool `json:"redact_thinking"`
	MaxThinkingLength  int  `json:"max_thinking_length"`
	RateLimitPerSecond int  `json:"rate_limit_per_second"`
	MaxResponseLength  int  `json:"max_response_length"`
}

StreamConfig configures streaming behavior for ChatStream.

Description

StreamConfig provides fine-grained control over streaming behavior including privacy controls for thinking tokens, rate limiting to prevent overwhelming clients, and maximum length limits for safety.

Fields

  • RedactThinking: If true, thinking tokens from models like gpt-oss are not emitted to the callback. The thinking still occurs server-side.
  • MaxThinkingLength: Maximum characters for thinking content per stream. 0 means unlimited. Truncates if exceeded.
  • RateLimitPerSecond: Maximum callback invocations per second. 0 disables.
  • MaxResponseLength: Maximum total response characters. 0 means unlimited.

Examples 

// Privacy-focused config
cfg := StreamConfig{RedactThinking: true, MaxThinkingLength: 1000}

// Rate-limited streaming
cfg := StreamConfig{RateLimitPerSecond: 100}

Limitations

  • Rate limiting adds latency to token delivery
  • Truncation loses data (use logging to preserve if needed)

Assumptions

  • Reasonable limits prevent DoS from runaway generation

func DefaultStreamConfig 

func DefaultStreamConfig() StreamConfig

DefaultStreamConfig returns a StreamConfig with safe defaults.

Description

Returns a configuration suitable for most use cases: - Thinking tokens are passed through (not redacted) - No rate limiting (full speed) - 100KB max response length for safety

Outputs

  • StreamConfig: Default configuration.

type StreamEvent 

type StreamEvent struct {
	Type    StreamEventType
	Content string
	Error   string
}

StreamEvent represents a single event during LLM streaming.

Description

StreamEvent is emitted by ChatStream for each token or event during generation. The Type field indicates what kind of event this is and which fields are populated.

Fields

  • Type: Event type (token, thinking, error).
  • Content: Token content. Populated for token and thinking events.
  • Error: Error message. Populated for error events.

Examples 

// Token event
StreamEvent{Type: StreamEventToken, Content: "Hello"}

// Thinking event
StreamEvent{Type: StreamEventThinking, Content: "Let me analyze..."}

// Error event
StreamEvent{Type: StreamEventError, Error: "Connection reset"}

Limitations

  • Only one of Content or Error is populated per event

Assumptions

  • Events are delivered in generation order

type StreamEventType 

type StreamEventType string

StreamEventType represents the type of streaming event.

Description

StreamEventType categorizes streaming events to allow handlers to process different event types appropriately. Token events contain generated text, thinking events contain reasoning, and error events signal problems during generation. 

const (
	// StreamEventToken indicates a content token event.
	// The Content field contains the generated text fragment.
	StreamEventToken StreamEventType = "token"

	// StreamEventThinking indicates a thinking/reasoning token event.
	// The Content field contains Claude's reasoning text.
	// Only emitted when EnableThinking is true.
	StreamEventThinking StreamEventType = "thinking"

	// StreamEventError indicates an error occurred during streaming.
	// The Error field contains the error message.
	// Streaming typically stops after an error event.
	StreamEventError StreamEventType = "error"
)

type StreamProcessor 

type StreamProcessor interface {
	// ProcessChunk processes a single parsed chunk and invokes callback.
	//
	// # Inputs
	//   - ctx: Context for cancellation.
	//   - chunk: Parsed stream chunk.
	//   - callback: Event callback.
	//
	// # Outputs
	//   - bool: True if stream is done.
	//   - error: Non-nil on callback error or processing failure.
	ProcessChunk(ctx context.Context, chunk *ollamaStreamChunk, callback StreamCallback) (bool, error)

	// GetTokenCount returns total tokens processed so far.
	GetTokenCount() int

	// GetResponseLength returns total response characters processed.
	GetResponseLength() int
}

StreamProcessor defines the contract for processing streaming responses.

Description

StreamProcessor abstracts the streaming response processing logic, allowing for different implementations (e.g., testing, rate-limited). Implementations must be safe for single-threaded use within a stream.

Methods

  • ProcessChunk: Process a single NDJSON chunk and emit events.
  • GetTokenCount: Return total tokens processed.
  • GetResponseLength: Return total response characters.

Thread Safety

Not required to be thread-safe; called from single goroutine per stream.

Limitations

  • Single stream per processor instance

Assumptions

  • Called sequentially for each chunk in order

type ToolCallResponse added in v1.0.0 

type ToolCallResponse struct {
	// ID is the unique identifier for this tool call.
	// Anthropic provides this in tool_use blocks.
	// OpenAI provides this in tool_calls.
	// Gemini does not provide IDs; synthetic ones are generated.
	ID string `json:"id"`

	// Name is the function name to call.
	Name string `json:"name"`

	// Arguments is the raw JSON arguments for the function.
	Arguments json.RawMessage `json:"arguments"`
}

ToolCallResponse represents a tool call from any LLM provider.

Description: 

Provider-agnostic representation of a tool call. Each provider's
ChatWithTools method populates this from its native response format:
- Anthropic: tool_use content blocks
- OpenAI: tool_calls array
- Gemini: functionCall parts (with synthetic IDs)
- Ollama: OllamaToolCall (existing)

Thread Safety: ToolCallResponse is safe for concurrent read access.

func (*ToolCallResponse) ArgumentsString added in v1.0.0 

func (t *ToolCallResponse) ArgumentsString() string

ArgumentsString returns the arguments as a JSON string.

Description: 

If arguments is already a JSON string value (starts with quote),
it returns the unquoted string. If arguments is an object or other
JSON value, it returns the raw JSON as-is. Returns "{}" for nil/empty.

Outputs:

  • string: The arguments as a JSON string suitable for tool execution.

Thread Safety: This method is safe for concurrent use.

type ToolChoice added in v1.0.0 

type ToolChoice struct {
	Type string `json:"type"`
	Name string `json:"name,omitempty"`
}

ToolChoice specifies how the model should select tools.

Description: 

The tool_choice parameter controls whether and which tools the model calls.
This enables forcing tool usage at the API level rather than relying on prompts.

Fields:

  • Type: Controls tool selection behavior: "auto": Model decides whether to call tools (default) "any": Model MUST call at least one tool "tool": Model MUST call the specific named tool "none": Model cannot call tools (text response only)
  • Name: Required when Type is "tool". Specifies which tool to force.

Thread Safety: This type is immutable and safe for concurrent read access.

func ToolChoiceAny added in v1.0.0 

func ToolChoiceAny() *ToolChoice

ToolChoiceAny forces the model to call at least one tool.

func ToolChoiceAuto added in v1.0.0 

func ToolChoiceAuto() *ToolChoice

ToolChoiceAuto allows the model to decide whether to call tools.

func ToolChoiceNone added in v1.0.0 

func ToolChoiceNone() *ToolChoice

ToolChoiceNone prevents the model from calling any tools.

func ToolChoiceRequired added in v1.0.0 

func ToolChoiceRequired(toolName string) *ToolChoice

ToolChoiceRequired forces the model to call a specific tool by name.

type ToolDef added in v1.0.0 

type ToolDef struct {
	// Type is the tool type. Always "function" for function calling.
	Type string `json:"type"`

	// Function contains the function definition.
	Function ToolFunction `json:"function"`
}

ToolDef is the generic tool definition used as input to ChatWithTools for all providers. Follows the OpenAI function calling schema.

Description: 

Provides a provider-agnostic way to define tools. Each provider's
ChatWithTools method converts ToolDef into its wire format
(Anthropic input_schema, OpenAI function, Gemini functionDeclarations).

Thread Safety: ToolDef is immutable and safe for concurrent read access.

type ToolFunction added in v1.0.0 

type ToolFunction struct {
	// Name is the function name the model will call.
	Name string `json:"name"`

	// Description explains what the function does.
	Description string `json:"description"`

	// Parameters defines the JSON Schema for function parameters.
	Parameters ToolParameters `json:"parameters"`
}

ToolFunction contains the function name, description, and parameter schema.

type ToolParamDef added in v1.0.0 

type ToolParamDef struct {
	// Type is the JSON Schema type (string, integer, boolean, number).
	Type string `json:"type"`

	// Description explains what the parameter is for.
	Description string `json:"description,omitempty"`

	// Enum restricts values to a set of options.
	Enum []any `json:"enum,omitempty"`

	// Default is the default value if not provided.
	Default any `json:"default,omitempty"`
}

ToolParamDef defines a single parameter in JSON Schema format.

type ToolParameters added in v1.0.0 

type ToolParameters struct {
	// Type is the JSON Schema type. Always "object" for tool parameters.
	Type string `json:"type"`

	// Properties maps parameter names to their definitions.
	Properties map[string]ToolParamDef `json:"properties,omitempty"`

	// Required lists parameter names that must be provided.
	Required []string `json:"required,omitempty"`
}

ToolParameters defines the JSON Schema for tool parameters.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.