llm

package
v1.0.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Mar 11, 2026 License: AGPL-3.0 Imports: 21 Imported by: 0

Documentation

Overview

Package llm provides the LLM client interface for the agent loop.

This package defines the interface that LLM providers must implement to work with the agent. Actual implementations are injected at runtime.

Thread Safety:

All types in this package are designed for concurrent use.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func FormatAsObservation

func FormatAsObservation(toolName string, success bool, output string) string

FormatAsObservation formats a tool result as a ReAct observation.

Description:

Converts a tool result into the "Observation:" format expected
by the ReAct pattern. This is injected into the conversation
after tool execution.

Inputs:

toolName - The name of the tool that was executed.
success - Whether the tool execution succeeded.
output - The tool output text (or error message if failed).

Outputs:

string - The formatted observation.

Example:

obs := FormatAsObservation("find_entry_points", true, "Found 5 entry points...")
// obs = "Observation: Found 5 entry points..."

Thread Safety: This function is safe for concurrent use.

func ParseToolCalls

func ParseToolCalls(response *Response) []agent.ToolInvocation

ParseToolCalls extracts tool invocations from an LLM response.

Description:

Parses native tool_calls from the LLM response structure.
For models that don't support native function calling, use
ParseToolCallsWithReAct instead.

Inputs:

response - The LLM response

Outputs:

[]agent.ToolInvocation - Parsed tool invocations

Thread Safety: This function is safe for concurrent use.

func ParseToolCallsWithReAct

func ParseToolCallsWithReAct(response *Response) ([]agent.ToolInvocation, bool)

ParseToolCallsWithReAct extracts tool invocations with ReAct fallback.

Description:

First attempts to parse native tool_calls from the response.
If no native tool calls are found, falls back to parsing
ReAct-style text format (Thought/Action/Action Input).

This enables tool usage with models that don't support native
function calling.

Inputs:

response - The LLM response

Outputs:

[]agent.ToolInvocation - Parsed tool invocations (from either source)
bool - True if invocations came from ReAct parsing (not native)

Example:

invocations, usedReAct := ParseToolCallsWithReAct(response)
if usedReAct {
    // Format next tool result as Observation
}

Thread Safety: This function is safe for concurrent use.

func ReActInstructions

func ReActInstructions() string

ReActInstructions returns the system prompt addition for ReAct mode.

Description:

Returns the instruction text to append to the system prompt
when ReAct mode is enabled. This teaches the LLM the ReAct format.

Outputs:

string - The ReAct instructions to append.

Thread Safety: This function is safe for concurrent use.

Types

type AnthropicAgentAdapter

type AnthropicAgentAdapter struct {
	// contains filtered or unexported fields
}

AnthropicAgentAdapter adapts AnthropicClient to the agent's Client interface.

Description:

Wraps the existing AnthropicClient to provide LLM capabilities for the
agent loop using Anthropic Claude models. Converts between agent message
format and Anthropic's datatypes.Message format.

Thread Safety: AnthropicAgentAdapter is safe for concurrent use.

func NewAnthropicAgentAdapter

func NewAnthropicAgentAdapter(client *llm.AnthropicClient, model string) *AnthropicAgentAdapter

NewAnthropicAgentAdapter creates a new AnthropicAgentAdapter.

Inputs:

  • client: The AnthropicClient to wrap. Must not be nil.
  • model: The model name for identification.

Outputs:

  • *AnthropicAgentAdapter: The configured adapter.

func (*AnthropicAgentAdapter) Complete

func (a *AnthropicAgentAdapter) Complete(ctx context.Context, request *Request) (*Response, error)

Complete implements Client.Complete for Anthropic.

func (*AnthropicAgentAdapter) Model

func (a *AnthropicAgentAdapter) Model() string

Model implements Client.Model.

func (*AnthropicAgentAdapter) Name

func (a *AnthropicAgentAdapter) Name() string

Name implements Client.Name.

type Client

type Client interface {
	// Complete sends a prompt to the LLM and returns a response.
	//
	// Inputs:
	//   ctx - Context for cancellation and timeout
	//   request - The completion request
	//
	// Outputs:
	//   *Response - The LLM response
	//   error - Non-nil if the request failed
	Complete(ctx context.Context, request *Request) (*Response, error)

	// Name returns the provider name (e.g., "anthropic", "openai").
	Name() string

	// Model returns the model being used.
	Model() string
}

Client defines the interface for LLM interactions.

Implementations must be safe for concurrent use.

type CompletionCall

type CompletionCall struct {
	Request   *Request
	Timestamp time.Time
}

CompletionCall records a call to Complete.

type EmptyResponseError

type EmptyResponseError struct {
	// Duration is how long the request took before returning empty.
	Duration time.Duration

	// MessageCount is the number of input messages sent.
	MessageCount int

	// Model is the model that returned the empty response.
	Model string
}

EmptyResponseError indicates the LLM returned an empty response.

This typically indicates a problem with the model, context, or request. Empty responses should not silently pass through as they cause downstream issues in the agent loop.

Fixed in cb_30a.

func (*EmptyResponseError) Error

func (e *EmptyResponseError) Error() string

Error implements the error interface.

type GeminiAgentAdapter

type GeminiAgentAdapter struct {
	// contains filtered or unexported fields
}

GeminiAgentAdapter adapts GeminiClient to the agent's Client interface.

Description:

Wraps the existing GeminiClient to provide LLM capabilities for the
agent loop using Google Gemini models. Converts between agent message
format and Gemini's datatypes.Message format.

Thread Safety: GeminiAgentAdapter is safe for concurrent use.

func NewGeminiAgentAdapter

func NewGeminiAgentAdapter(client *llm.GeminiClient, model string) *GeminiAgentAdapter

NewGeminiAgentAdapter creates a new GeminiAgentAdapter.

Inputs:

  • client: The GeminiClient to wrap. Must not be nil.
  • model: The model name for identification.

Outputs:

  • *GeminiAgentAdapter: The configured adapter.

func (*GeminiAgentAdapter) Complete

func (a *GeminiAgentAdapter) Complete(ctx context.Context, request *Request) (*Response, error)

Complete implements Client.Complete for Gemini.

func (*GeminiAgentAdapter) Model

func (a *GeminiAgentAdapter) Model() string

Model implements Client.Model.

func (*GeminiAgentAdapter) Name

func (a *GeminiAgentAdapter) Name() string

Name implements Client.Name.

type Message

type Message struct {
	// Role is "user", "assistant", or "system".
	Role string `json:"role"`

	// Content is the text content.
	Content string `json:"content"`

	// ToolCalls contains tool invocations (for assistant messages).
	ToolCalls []ToolCall `json:"tool_calls,omitempty"`

	// ToolResults contains tool results (for tool messages).
	ToolResults []ToolCallResult `json:"tool_results,omitempty"`
}

Message represents a conversation message.

type MockClient

type MockClient struct {
	// contains filtered or unexported fields
}

MockClient is a mock LLM client for testing.

Thread Safety:

MockClient is safe for concurrent use.

func NewMockClient

func NewMockClient() *MockClient

NewMockClient creates a new mock LLM client.

func (*MockClient) CallCount

func (c *MockClient) CallCount() int

CallCount returns the number of calls made.

func (*MockClient) Complete

func (c *MockClient) Complete(ctx context.Context, request *Request) (*Response, error)

Complete implements the Client interface.

func (*MockClient) ExpectCall

func (c *MockClient) ExpectCall(count int) error

ExpectCall returns an error if the expected number of calls wasn't made.

func (*MockClient) GetCalls

func (c *MockClient) GetCalls() []CompletionCall

GetCalls returns all recorded calls.

func (*MockClient) LastRequest

func (c *MockClient) LastRequest() *Request

LastRequest returns the most recent request.

func (*MockClient) Model

func (c *MockClient) Model() string

Model implements the Client interface.

func (*MockClient) Name

func (c *MockClient) Name() string

Name implements the Client interface.

func (*MockClient) QueueFinalResponse

func (c *MockClient) QueueFinalResponse(content string) *MockClient

QueueFinalResponse queues a final response with no tool calls.

func (*MockClient) QueueResponse

func (c *MockClient) QueueResponse(response *Response) *MockClient

QueueResponse adds a response to the queue.

func (*MockClient) QueueToolCall

func (c *MockClient) QueueToolCall(toolName string, arguments map[string]any) *MockClient

QueueToolCall queues a response that invokes a tool.

func (*MockClient) Reset

func (c *MockClient) Reset()

Reset clears all state.

func (*MockClient) SetDefaultResponse

func (c *MockClient) SetDefaultResponse(response *Response) *MockClient

SetDefaultResponse sets the response to return when queue is empty.

func (*MockClient) Verify

func (c *MockClient) Verify() error

Verify ensures all queued responses were consumed.

func (*MockClient) WithDelay

func (c *MockClient) WithDelay(d time.Duration) *MockClient

WithDelay adds artificial latency.

func (*MockClient) WithError

func (c *MockClient) WithError(err error) *MockClient

WithError configures the client to return an error.

func (*MockClient) WithModel

func (c *MockClient) WithModel(model string) *MockClient

WithModel sets the model name.

func (*MockClient) WithName

func (c *MockClient) WithName(name string) *MockClient

WithName sets the provider name.

func (*MockClient) WithResponseFunc

func (c *MockClient) WithResponseFunc(f func(*Request) (*Response, error)) *MockClient

WithResponseFunc sets a dynamic response function.

type OllamaAdapter

type OllamaAdapter struct {
	// contains filtered or unexported fields
}

OllamaAdapter adapts services/llm.OllamaClient to the agent's Client interface.

Description:

OllamaAdapter wraps the existing OllamaClient to provide LLM capabilities
for the agent loop. This allows Trace to use local Ollama models
(like gpt-oss:20b) for code assistance.

Thread Safety:

OllamaAdapter is safe for concurrent use.

func NewOllamaAdapter

func NewOllamaAdapter(client *llm.OllamaClient, model string) *OllamaAdapter

NewOllamaAdapter creates a new OllamaAdapter.

Description:

Creates an adapter wrapping the provided OllamaClient.

Inputs:

client - The OllamaClient to wrap. Must not be nil.
model - The model name for identification.

Outputs:

*OllamaAdapter - The configured adapter.

Example:

ollamaClient, _ := llm.NewOllamaClient()
adapter := NewOllamaAdapter(ollamaClient, "gpt-oss:20b")

func (*OllamaAdapter) Complete

func (a *OllamaAdapter) Complete(ctx context.Context, request *Request) (*Response, error)

Complete implements Client.

Description:

Sends a completion request to Ollama and returns the response.
Converts between agent message format and Ollama's datatypes.Message format.
If tools are provided in the request, uses ChatWithTools to enable function calling.

Inputs:

ctx - Context for cancellation and timeout.
request - The completion request.

Outputs:

*Response - The LLM response.
error - Non-nil if the request failed.

Thread Safety: This method is safe for concurrent use.

func (*OllamaAdapter) Model

func (a *OllamaAdapter) Model() string

Model implements Client.

func (*OllamaAdapter) Name

func (a *OllamaAdapter) Name() string

Name implements Client.

type OpenAIAgentAdapter

type OpenAIAgentAdapter struct {
	// contains filtered or unexported fields
}

OpenAIAgentAdapter adapts OpenAIClient to the agent's Client interface.

Description:

Wraps the existing OpenAIClient to provide LLM capabilities for the
agent loop using OpenAI models. Converts between agent message format
and OpenAI's datatypes.Message format.

Thread Safety: OpenAIAgentAdapter is safe for concurrent use.

func NewOpenAIAgentAdapter

func NewOpenAIAgentAdapter(client *llm.OpenAIClient, model string) *OpenAIAgentAdapter

NewOpenAIAgentAdapter creates a new OpenAIAgentAdapter.

Inputs:

  • client: The OpenAIClient to wrap. Must not be nil.
  • model: The model name for identification.

Outputs:

  • *OpenAIAgentAdapter: The configured adapter.

func (*OpenAIAgentAdapter) Complete

func (a *OpenAIAgentAdapter) Complete(ctx context.Context, request *Request) (*Response, error)

Complete implements Client.Complete for OpenAI.

func (*OpenAIAgentAdapter) Model

func (a *OpenAIAgentAdapter) Model() string

Model implements Client.Model.

func (*OpenAIAgentAdapter) Name

func (a *OpenAIAgentAdapter) Name() string

Name implements Client.Name.

type ReActParseResult

type ReActParseResult struct {
	// Thought is the LLM's reasoning (may be empty).
	Thought string

	// Action is the tool name to invoke (empty if no action).
	Action string

	// ActionInput is the JSON arguments (empty if no action).
	ActionInput string

	// FinalAnswer is the final response (empty if not done).
	FinalAnswer string

	// RemainingText is any unparsed text.
	RemainingText string
}

ReActParseResult contains parsed ReAct components from LLM response text.

Thread Safety: This type is immutable and safe for concurrent read access.

func ParseReAct

func ParseReAct(text string) *ReActParseResult

ParseReAct extracts ReAct components from LLM response text.

Description:

Parses LLM output for ReAct-style structured text. The ReAct format is:
  Thought: [reasoning]
  Action: [tool_name]
  Action Input: {"param": "value"}

Or for final responses:
  Thought: [reasoning]
  Final Answer: [response]

Inputs:

text - The LLM response text to parse.

Outputs:

*ReActParseResult - Parsed components. Never nil.

Example:

result := ParseReAct("Thought: I need to find tests.\nAction: find_entry_points\nAction Input: {\"type\": \"test\"}")
// result.Action = "find_entry_points"
// result.ActionInput = "{\"type\": \"test\"}"

Thread Safety: This function is safe for concurrent use.

func (*ReActParseResult) HasAction

func (r *ReActParseResult) HasAction() bool

HasAction returns true if an action was parsed.

Description:

Indicates whether the LLM output contains a tool invocation.

Outputs:

bool - True if Action field is non-empty.

Thread Safety: This method is safe for concurrent use.

func (*ReActParseResult) HasFinalAnswer

func (r *ReActParseResult) HasFinalAnswer() bool

HasFinalAnswer returns true if a final answer was parsed.

Description:

Indicates whether the LLM has provided a final response
(meaning no more tool calls are expected).

Outputs:

bool - True if FinalAnswer field is non-empty.

Thread Safety: This method is safe for concurrent use.

func (*ReActParseResult) ToToolCall

func (r *ReActParseResult) ToToolCall() *ToolCall

ToToolCall converts a ReAct action to a ToolCall.

Description:

Creates a ToolCall struct from the parsed ReAct components.
Returns nil if no action was parsed.

Outputs:

*ToolCall - The tool call, or nil if no action.

Example:

result := ParseReAct("Action: find_entry_points\nAction Input: {\"type\": \"test\"}")
call := result.ToToolCall()
// call.Name = "find_entry_points"
// call.Arguments = "{\"type\": \"test\"}"

Thread Safety: This method is safe for concurrent use.

type Request

type Request struct {
	// SystemPrompt is the system message.
	SystemPrompt string `json:"system_prompt,omitempty"`

	// Messages is the conversation history.
	Messages []Message `json:"messages"`

	// Tools defines available tools for the LLM.
	Tools []tools.ToolDefinition `json:"tools,omitempty"`

	// ToolChoice controls tool selection behavior.
	// If nil, defaults to "auto" (model decides).
	ToolChoice *ToolChoice `json:"tool_choice,omitempty"`

	// MaxTokens limits the response length.
	MaxTokens int `json:"max_tokens,omitempty"`

	// Temperature controls randomness (0.0-1.0). Set to 0.0 for most
	// deterministic output. Set to a negative value (e.g., -1) to omit
	// from the request and use the provider's default. The Go zero value
	// (0.0) is treated as an explicit "most deterministic" setting.
	Temperature float64 `json:"temperature,omitempty"`

	// StopSequences defines sequences that stop generation.
	StopSequences []string `json:"stop_sequences,omitempty"`

	// ModelOverride allows using a different model for this request.
	// Used for multi-model scenarios (e.g., tool routing with a fast model).
	// Empty string means use the client's default model.
	ModelOverride string `json:"model_override,omitempty"`

	// KeepAlive controls how long the model stays loaded in VRAM.
	// Values: "-1" = infinite, "5m" = 5 minutes (default), "0" = unload immediately.
	// Used to prevent model thrashing when alternating between models.
	KeepAlive string `json:"keep_alive,omitempty"`
}

Request represents a completion request to the LLM.

func BuildRequest

func BuildRequest(
	ctx *agent.AssembledContext,
	availableTools []tools.ToolDefinition,
	maxTokens int,
) *Request

BuildRequest creates a request from agent context.

Description:

Converts the agent's assembled context into an LLM request
suitable for the Complete method.

Inputs:

ctx - Assembled context from the agent
availableTools - Tools available for this request
maxTokens - Maximum response tokens

Outputs:

*Request - The built request

type Response

type Response struct {
	// Content is the text response.
	Content string `json:"content"`

	// ToolCalls contains any tool calls the LLM wants to make.
	ToolCalls []ToolCall `json:"tool_calls,omitempty"`

	// StopReason indicates why generation stopped.
	// Values: "end", "max_tokens", "tool_use", "stop_sequence"
	StopReason string `json:"stop_reason"`

	// TokensUsed is the total tokens consumed (input + output).
	TokensUsed int `json:"tokens_used"`

	// InputTokens is the input token count.
	InputTokens int `json:"input_tokens"`

	// OutputTokens is the output token count.
	OutputTokens int `json:"output_tokens"`

	// Duration is how long the request took.
	Duration time.Duration `json:"duration"`

	// Model is the model that generated this response.
	Model string `json:"model,omitempty"`

	// TraceStep is the CRS trace step recorded for this LLM call.
	// Callers can use this to record the step in the CRS trace recorder.
	// May be nil if CRS recording was not applicable.
	TraceStep *crs.TraceStep `json:"trace_step,omitempty"`
}

Response represents an LLM response.

func (*Response) HasToolCalls

func (r *Response) HasToolCalls() bool

HasToolCalls returns true if the response contains tool calls.

type ToolCall

type ToolCall struct {
	// ID is a unique identifier for this call.
	ID string `json:"id"`

	// Name is the tool name.
	Name string `json:"name"`

	// Arguments are the tool arguments as JSON.
	Arguments string `json:"arguments"`
}

ToolCall represents a tool invocation by the LLM.

type ToolCallResult

type ToolCallResult struct {
	// ToolCallID links back to the tool call.
	ToolCallID string `json:"tool_call_id"`

	// Content is the result content.
	Content string `json:"content"`

	// IsError indicates if this is an error result.
	IsError bool `json:"is_error,omitempty"`
}

ToolCallResult contains the result of a tool call.

type ToolChoice

type ToolChoice struct {
	// Type controls tool selection behavior:
	// - "auto": Model decides whether to call tools (default)
	// - "any": Model MUST call at least one tool
	// - "tool": Model MUST call the specific named tool
	// - "none": Model cannot call tools (text response only)
	Type string `json:"type"`

	// Name is required when Type is "tool". Specifies which tool to force.
	Name string `json:"name,omitempty"`
}

ToolChoice specifies how the model should select tools.

The tool_choice parameter controls whether and which tools the model calls. This enables forcing tool usage at the API level rather than relying on prompts.

func ToolChoiceAny

func ToolChoiceAny() *ToolChoice

ToolChoiceAny forces the model to call at least one tool.

func ToolChoiceAuto

func ToolChoiceAuto() *ToolChoice

ToolChoiceAuto allows the model to decide whether to call tools.

func ToolChoiceNone

func ToolChoiceNone() *ToolChoice

ToolChoiceNone prevents the model from calling any tools.

func ToolChoiceRequired

func ToolChoiceRequired(toolName string) *ToolChoice

ToolChoiceRequired forces the model to call a specific tool by name.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL