Documentation
¶
Overview ¶
Package llm defines a provider-agnostic LLM client interface for the Cortex engine.
This abstraction allows the engine to call any LLM backend (Nexus, direct providers, mock) without importing provider-specific packages. Adapters convert between these types and the concrete provider types.
Index ¶
Constants ¶
const ( // MetadataRun is the ID of the run that made the request. MetadataRun = "run" // MetadataAgent is the ID of the agent the run belongs to. MetadataAgent = "agent" )
Keys the engine sets in Request.Metadata.
Variables ¶
var ErrContentBlocked = errors.New("llm: provider guard blocked the response")
ErrContentBlocked reports that a guard in front of the provider refused the request or the response. A client wraps the guard's own error with it, so errors.As still finds that error. The engine ends the run with stop reason safety_block and dispatches no tool call from the response.
Functions ¶
This section is empty.
Types ¶
type Chunk ¶
type Chunk struct {
// Content is the incremental text delta.
Content string
// ToolCalls contains incremental tool call deltas.
ToolCalls []ToolCall
// FinishReason is set on the final chunk.
FinishReason string
// Native is the whole reply in the provider's own format. A client sets
// it on the final chunk only.
Native *NativeContent `json:"Native,omitempty"`
}
Chunk is a single incremental piece of a streaming response.
type Client ¶
type Client interface {
// Complete sends a synchronous completion request and returns the full response.
Complete(ctx context.Context, req *Request) (*Response, error)
// CompleteStream sends a streaming completion request and returns a stream
// that yields incremental chunks.
CompleteStream(ctx context.Context, req *Request) (Stream, error)
}
Client is the interface for sending completion requests to an LLM.
type Message ¶
type Message struct {
// Role is the message author: "user", "assistant", or "tool".
Role string
// Content is the text content of the message.
Content string
// ToolCalls contains tool invocations requested by the assistant.
ToolCalls []ToolCall
// ToolCallID identifies which tool call this message is responding to (role=tool).
ToolCallID string
// Native is the provider's own form of an assistant message, carried
// back to the model that produced it. Nil for every other message.
Native *NativeContent `json:"Native,omitempty"`
}
Message is a single message in a conversation.
type MockClient ¶
type MockClient struct {
// TokenDelay is the delay between each streamed character. Default: 20ms.
TokenDelay time.Duration
}
MockClient is an echo-based LLM client for testing and fallback. It echoes the user input back as the assistant response with simulated delays.
func NewMockClient ¶
func NewMockClient() *MockClient
NewMockClient creates a new echo-based mock LLM client.
func (*MockClient) CompleteStream ¶
CompleteStream returns a stream that yields the echo response token by token.
type NativeContent ¶ added in v1.10.0
type NativeContent struct {
// Provider names the provider whose format Content is in.
Provider string `json:"provider"`
// Model is the model that produced the reply.
Model string `json:"model"`
// Content is the reply's raw content blocks, thinking signatures
// included.
Content json.RawMessage `json:"content"`
}
NativeContent is an assistant reply in the provider's own format, such as Anthropic content blocks with their signed thinking. A client sets it on a response, and the engine keeps it on the assistant message it appends and sends it back with the next request, so a thinking model sees its own earlier turns. A client may re-encode the JSON on the way out, so the model gets equivalent content, not necessarily the same bytes. It never goes into conversation memory and is never published. It does hold the model's own, unredacted content in progress snapshots and continuations.
func (*NativeContent) Clone ¶ added in v1.10.0
func (n *NativeContent) Clone() *NativeContent
Clone returns a copy of n that shares no memory with it, or nil for nil.
type Request ¶
type Request struct {
// Model is the model identifier (e.g. "gpt-4o", "claude-sonnet-4-20250514", "smart").
Model string
// System is the system prompt injected before the conversation.
System string
// Messages is the conversation history.
Messages []Message
// Tools is the set of tools available for the LLM to call.
Tools []Tool
// MaxTokens limits the maximum output tokens.
MaxTokens int
// Temperature controls sampling randomness. Nil means provider default.
Temperature *float64
// ToolChoice constrains whether and which tool the model calls. Nil
// leaves it to the model. The engine never sets it.
ToolChoice *ToolChoice `json:"ToolChoice,omitempty"`
// Metadata labels the request for the client's usage records, for
// example with MetadataRun and MetadataAgent. The engine sets it. It is
// kept out of the request's JSON, so request digests do not depend on
// it.
Metadata map[string]string `json:"-"`
}
Request is a completion request sent to an LLM.
type Response ¶
type Response struct {
// Content is the assistant's text response.
Content string
// ToolCalls contains any tool invocations requested by the assistant.
ToolCalls []ToolCall
// Usage tracks token consumption.
Usage Usage
// Model is the actual model used (may differ from request if aliased).
Model string
// FinishReason indicates why generation stopped: "stop", "tool_calls", "length", etc.
FinishReason string
// Native is the reply in the provider's own format, when the client
// has it.
Native *NativeContent `json:"Native,omitempty"`
}
Response is the result of a synchronous completion request.
type Stream ¶
type Stream interface {
// Next returns the next chunk. Returns io.EOF when the stream is complete.
Next(ctx context.Context) (*Chunk, error)
// Close releases stream resources.
Close() error
// Usage returns the final token usage after the stream completes.
// May return nil if usage is not yet available.
Usage() *Usage
}
Stream yields incremental chunks from a streaming completion.
type Tool ¶
type Tool struct {
// Name is the tool's unique identifier.
Name string
// Description explains what the tool does.
Description string
// Parameters is a JSON Schema describing the tool's input parameters.
Parameters any
}
Tool describes a function tool available for the LLM to call.
type ToolCall ¶
type ToolCall struct {
// ID uniquely identifies this tool call (used to match tool results).
ID string
// Name is the name of the tool to invoke.
Name string
// Arguments is the JSON-encoded arguments for the tool.
Arguments string
}
ToolCall represents a tool invocation from the assistant.
type ToolChoice ¶ added in v1.10.0
type ToolChoice struct {
// Mode is "auto", "none", "required" or "function".
Mode string
// Function names the tool to call when Mode is "function".
Function string
}
ToolChoice constrains the model's use of tools.