Documentation
¶
Overview ¶
Package llm provides LLM backend implementations.
Package llm provides LLM backend implementations for the vega package.
Anthropic Backend ¶
The primary backend is Anthropic's Claude API:
llm := llm.NewAnthropic() // Uses ANTHROPIC_API_KEY env var
// Or with custom API key
llm := llm.NewAnthropic(llm.WithAPIKey("sk-..."))
// Or with custom model
llm := llm.NewAnthropic(llm.WithModel("claude-opus-4-7"))
Using with Orchestrator ¶
Configure the orchestrator to use the LLM:
llm := llm.NewAnthropic() orch := vega.NewOrchestrator(vega.WithLLM(llm))
Streaming ¶
The Anthropic backend supports streaming responses:
stream, err := proc.SendStream(ctx, "Tell me a story")
for chunk := range stream.Chunks() {
fmt.Print(chunk)
}
Tool Support ¶
Tools are automatically converted to the Anthropic tool format:
tools := tools.NewTools()
tools.Register("search", searchFunc)
agent := vega.Agent{
Tools: tools,
// ...
}
When the model decides to use a tool, the tool is executed and the result is sent back automatically in a multi-turn conversation.
Rate Limiting ¶
The Anthropic API has rate limits. Configure rate limiting on the orchestrator:
orch := vega.NewOrchestrator(
vega.WithLLM(llm),
vega.WithRateLimits(map[string]vega.RateLimitConfig{
"claude-sonnet-4-6": {
RequestsPerMinute: 60,
TokensPerMinute: 100000,
},
}),
)
Implementing Custom Backends ¶
To implement a custom LLM backend, implement the llm.LLM interface:
type LLM interface {
Generate(ctx context.Context, messages []Message, tools []ToolSchema) (*LLMResponse, error)
GenerateStream(ctx context.Context, messages []Message, tools []ToolSchema) (<-chan StreamEvent, error)
}
Index ¶
- Constants
- func APIKeyFromContext(ctx context.Context) string
- func CalculateCost(model string, ...) float64
- func ContextWithOptions(ctx context.Context, opts Options) context.Context
- func WithAPIKeyContext(ctx context.Context, key string) context.Context
- type AnthropicLLM
- func (a *AnthropicLLM) Generate(ctx context.Context, messages []Message, tools []ToolSchema) (*LLMResponse, error)
- func (a *AnthropicLLM) GenerateStream(ctx context.Context, messages []Message, tools []ToolSchema) (<-chan StreamEvent, error)
- func (a *AnthropicLLM) ValidateKey(ctx context.Context) error
- type AnthropicOption
- func WithAPIKey(key string) AnthropicOption
- func WithBaseURL(url string) AnthropicOption
- func WithEffort(effort string) AnthropicOption
- func WithHTTPClient(client *http.Client) AnthropicOption
- func WithMaxConcurrent(n int) AnthropicOption
- func WithModel(model string) AnthropicOption
- func WithWebSearch() AnthropicOption
- type ContentBlock
- type LLM
- type LLMResponse
- type Message
- type ModelCapabilities
- type OpenAILLM
- type OpenAIOption
- type Options
- type Role
- type StopReason
- type StreamEvent
- type StreamEventType
- type ToolCall
- type ToolSchema
Constants ¶
const ( DefaultAnthropicTimeout = 5 * time.Minute DefaultAnthropicModel = "claude-sonnet-4-6" DefaultAnthropicBaseURL = "https://api.anthropic.com" )
Default Anthropic configuration values
const ( DefaultOpenAIModel = "qwen-coder" DefaultOpenAIBaseURL = "http://localhost:4000" )
const ( BlockText = "text" BlockThinking = "thinking" BlockToolUse = "tool_use" BlockToolResult = "tool_result" BlockImage = "image" // BlockOpaque carries a content block govega does not model — server-side // tool blocks like server_tool_use and web_search_tool_result — verbatim // in Raw, so replaying the turn stays lossless. BlockOpaque = "opaque" )
Content block types.
const DefaultMaxConcurrent = 5
DefaultMaxConcurrent is the default maximum number of in-flight API requests.
Variables ¶
This section is empty.
Functions ¶
func APIKeyFromContext ¶ added in v0.6.0
APIKeyFromContext returns the per-request key set by WithAPIKeyContext, or "" when none is attached.
func CalculateCost ¶
func CalculateCost(model string, inputTokens, outputTokens, cacheCreationTokens, cacheReadTokens int) float64
CalculateCost calculates the cost of a request including prompt cache tokens. Cache writes cost 125% of base input price; cache reads cost 10%.
func ContextWithOptions ¶ added in v0.5.1
ContextWithOptions returns a derived context carrying opts.
func WithAPIKeyContext ¶ added in v0.6.0
WithAPIKeyContext returns ctx carrying a per-request Anthropic API key. When set, outbound calls use this key instead of the one passed to NewAnthropic. Empty values are stored verbatim — callers should pass a real key.
Types ¶
type AnthropicLLM ¶
type AnthropicLLM struct {
// contains filtered or unexported fields
}
AnthropicLLM is an LLM implementation using the Anthropic API.
func NewAnthropic ¶
func NewAnthropic(opts ...AnthropicOption) *AnthropicLLM
NewAnthropic creates a new Anthropic LLM client.
func (*AnthropicLLM) Generate ¶
func (a *AnthropicLLM) Generate(ctx context.Context, messages []Message, tools []ToolSchema) (*LLMResponse, error)
Generate sends a request and returns the complete response.
func (*AnthropicLLM) GenerateStream ¶
func (a *AnthropicLLM) GenerateStream(ctx context.Context, messages []Message, tools []ToolSchema) (<-chan StreamEvent, error)
GenerateStream sends a request and returns a channel of streaming events.
func (*AnthropicLLM) ValidateKey ¶
func (a *AnthropicLLM) ValidateKey(ctx context.Context) error
ValidateKey makes a minimal API call to verify the API key is valid. Returns nil on success, or an error describing the failure (empty key, authentication failure, or network/other error).
type AnthropicOption ¶
type AnthropicOption func(*AnthropicLLM)
AnthropicOption configures the Anthropic client.
func WithEffort ¶ added in v0.5.1
func WithEffort(effort string) AnthropicOption
WithEffort sets output_config.effort for requests on capable models. Valid values: "low", "medium", "high", "xhigh", "max". Empty means "high" by default. "xhigh" is the recommended setting for agentic and coding workloads on Opus 4.7. "max" is Opus-tier only.
func WithHTTPClient ¶
func WithHTTPClient(client *http.Client) AnthropicOption
WithHTTPClient sets a custom HTTP client. It is used for both the non-streaming and streaming paths — callers overriding it own the timeout trade-off (a non-zero Timeout will cut streams that outlive it).
func WithMaxConcurrent ¶ added in v0.3.0
func WithMaxConcurrent(n int) AnthropicOption
WithMaxConcurrent sets the maximum number of concurrent API requests.
func WithWebSearch ¶ added in v0.8.18
func WithWebSearch() AnthropicOption
WithWebSearch enables Anthropic's server-side web search and web fetch tools (web_search_20260209 / web_fetch_20260209). Requires a model that supports them (Opus 4.6+ / Sonnet 4.6+ generations); searches bill separately from tokens. A paused server-side loop surfaces as StopReasonPause, which the process loop already resumes.
type ContentBlock ¶ added in v0.8.0
type ContentBlock struct {
Type string `json:"type"`
// Text carries text and thinking content.
Text string `json:"text,omitempty"`
// Signature authenticates a thinking block so it can be replayed to
// the API on the next request of the same turn.
Signature string `json:"signature,omitempty"`
// Tool use fields.
ID string `json:"id,omitempty"`
Name string `json:"name,omitempty"`
Arguments map[string]any `json:"arguments,omitempty"`
// Tool result fields.
ToolUseID string `json:"tool_use_id,omitempty"`
Content string `json:"content,omitempty"`
IsError bool `json:"is_error,omitempty"`
// Image fields (BlockImage): base64-encoded image Data plus its MediaType
// (e.g. "image/png"). Vision-capable models read these on user turns.
MediaType string `json:"media_type,omitempty"`
Data string `json:"data,omitempty"`
// Raw is the verbatim API block for BlockOpaque — block types govega
// does not model, preserved so turn replay is lossless.
Raw json.RawMessage `json:"raw,omitempty"`
}
ContentBlock is one typed unit of message content. Exactly one group of fields is meaningful per Type:
- BlockText: Text
- BlockThinking: Text (the reasoning), Signature (required for replay)
- BlockToolUse: ID, Name, Arguments
- BlockToolResult: ToolUseID, Content, IsError
type LLM ¶
type LLM interface {
// Generate sends a request and returns the complete response.
Generate(ctx context.Context, messages []Message, tools []ToolSchema) (*LLMResponse, error)
// GenerateStream sends a request and returns a channel of streaming events.
GenerateStream(ctx context.Context, messages []Message, tools []ToolSchema) (<-chan StreamEvent, error)
}
LLM is the interface for language model backends.
type LLMResponse ¶
type LLMResponse struct {
// Content is the text response
Content string
// Blocks is the ordered, typed content of the response (thinking,
// text, tool_use). Callers replaying the assistant turn should attach
// these to the next Message so thinking blocks and tool invocations
// survive the round trip. Empty on backends without block support.
Blocks []ContentBlock
// ToolCalls are any tool calls the model wants to make
ToolCalls []ToolCall
// Token counts
InputTokens int
OutputTokens int
// Cache token counts (Anthropic prompt caching)
CacheCreationInputTokens int
CacheReadInputTokens int
// Cost in USD
CostUSD float64
// Latency in milliseconds
LatencyMs int64
// StopReason indicates why generation stopped
StopReason StopReason
}
LLMResponse is the response from an LLM call.
type Message ¶
type Message struct {
Role Role
Content string
Blocks []ContentBlock `json:"Blocks,omitempty"`
// Volatile is per-conversation system content — who the agent is talking
// to, what it remembers about them — that belongs in the system prompt but
// is byte-unique per conversation. Meaningful only on a RoleSystem
// message. Providers that support prompt caching render it as a second,
// uncached system block after Content, so every conversation shares one
// cached prefix instead of writing its own. Empty ⇒ one block, as before.
Volatile string `json:"Volatile,omitempty"`
}
Message represents a conversation message.
Content carries plain text. Blocks, when non-empty, carries the full typed structure of the turn (text, thinking, tool_use, tool_result) and takes precedence over Content when building API requests — backends convert Blocks directly instead of round-tripping tool activity through XML markup in Content.
func (Message) SystemText ¶ added in v0.9.3
SystemText returns the whole system prompt as the model sees it: the cacheable Content followed by the per-conversation Volatile tail. Use it anywhere the split is an implementation detail — assertions, logging, estimates — rather than reading Content and silently missing half of it.
type ModelCapabilities ¶ added in v0.5.1
type ModelCapabilities struct {
// AdaptiveThinking — model supports {type: "adaptive"} thinking.
// Older models use the legacy budget_tokens form, which we don't emit.
AdaptiveThinking bool
// SupportsEffort — model accepts output_config.effort.
// Sonnet 4.5 and Haiku 4.5 return 400 if it's sent.
SupportsEffort bool
// SupportsTemperature — model accepts the temperature sampling
// parameter. Opus 4.7 removed sampling params and returns 400 if
// temperature/top_p/top_k are sent.
SupportsTemperature bool
// SupportsStructuredOutputs — model accepts output_config.format
// for JSON schema enforcement. Supported on Claude 4.5+ Opus,
// Sonnet 4.6, Haiku 4.5.
SupportsStructuredOutputs bool
// MaxOutputTokens is the streaming output ceiling for this model.
MaxOutputTokens int
}
ModelCapabilities describes which API features a model supports. Used to gate per-request fields so we don't send shapes that 400 on the model.
func CapabilitiesFor ¶ added in v0.5.1
func CapabilitiesFor(model string) ModelCapabilities
CapabilitiesFor returns the capabilities for the given model. Unknown models return the zero value (everything disabled: no adaptive thinking, no effort, conservative output cap) and log a warning so a stale table or a typo'd model ID surfaces instead of silently running degraded.
type OpenAILLM ¶ added in v0.3.0
type OpenAILLM struct {
// contains filtered or unexported fields
}
OpenAILLM is an LLM implementation using the OpenAI-compatible chat completions API. Works with LiteLLM, Ollama, vLLM, and any OpenAI-compatible endpoint.
func NewOpenAI ¶ added in v0.3.0
func NewOpenAI(opts ...OpenAIOption) *OpenAILLM
NewOpenAI creates a new OpenAI-compatible LLM client.
func (*OpenAILLM) Generate ¶ added in v0.3.0
func (o *OpenAILLM) Generate(ctx context.Context, messages []Message, tools []ToolSchema) (*LLMResponse, error)
Generate sends a request and returns the complete response.
func (*OpenAILLM) GenerateStream ¶ added in v0.3.0
func (o *OpenAILLM) GenerateStream(ctx context.Context, messages []Message, tools []ToolSchema) (<-chan StreamEvent, error)
GenerateStream sends a request and returns a channel of streaming events.
type OpenAIOption ¶ added in v0.3.0
type OpenAIOption func(*OpenAILLM)
OpenAIOption configures the OpenAI-compatible client.
func WithOpenAIAPIKey ¶ added in v0.3.0
func WithOpenAIAPIKey(key string) OpenAIOption
WithOpenAIAPIKey sets the API key.
func WithOpenAIBaseURL ¶ added in v0.3.0
func WithOpenAIBaseURL(url string) OpenAIOption
WithOpenAIBaseURL sets the API base URL.
func WithOpenAIModel ¶ added in v0.3.0
func WithOpenAIModel(model string) OpenAIOption
WithOpenAIModel sets the default model.
func WithOpenAIPricing ¶ added in v0.8.0
func WithOpenAIPricing(inputPer1M, outputPer1M float64) OpenAIOption
WithOpenAIPricing sets per-1M-token USD rates used for cost accounting. Unset (zero) rates report $0 — the right answer for local models.
type Options ¶ added in v0.5.1
type Options struct {
// Model overrides the client's default model for this call.
Model string
// Temperature is a float pointer because zero is a valid value;
// nil means "do not send" (use model default). Backends that
// don't support sampling parameters drop this silently.
Temperature *float64
// MaxTokens caps response output. 0 means use the capability default.
MaxTokens int
// Effort sets output_config.effort on supported models.
// Empty defaults to "high" if the model supports effort.
Effort string
// OutputSchema is a JSON Schema enforced on the response via
// output_config.format. When set on a model that supports
// structured outputs, the API returns valid JSON conforming to
// the schema (no markdown fences, no invented fields). Silently
// dropped on unsupported models — caller falls back to ad-hoc
// JSON parsing in that case.
OutputSchema map[string]any
}
Options carries per-call overrides for an LLM request. They flow through context.Context so callers can inject per-Agent values without changing the LLM interface.
Empty / zero fields fall back to the LLM client's defaults.
func OptionsFromContext ¶ added in v0.5.1
OptionsFromContext returns the Options previously installed on ctx, or the zero value if none.
type StopReason ¶
type StopReason string
StopReason indicates why the LLM stopped generating.
const ( StopReasonEnd StopReason = "end_turn" StopReasonToolUse StopReason = "tool_use" StopReasonLength StopReason = "max_tokens" StopReasonStop StopReason = "stop_sequence" StopReasonFiltered StopReason = "content_filter" // StopReasonPause is set when a server-side tool sampling loop hit // its iteration limit. Caller should re-send the assistant turn // unchanged; the API resumes automatically. StopReasonPause StopReason = "pause_turn" // StopReasonRefusal is set when Claude declined to respond for // safety reasons. Output may not match an expected schema. Do // NOT retry the same prompt — surface to the user. StopReasonRefusal StopReason = "refusal" // StopReasonContextExceeded is set when the model's context // window was exhausted (distinct from max_tokens, which is the // per-response output cap). Caller should compact or split the // conversation. StopReasonContextExceeded StopReason = "model_context_window_exceeded" )
type StreamEvent ¶
type StreamEvent struct {
// Type of event
Type StreamEventType
// Delta is new content for ContentDelta events
Delta string
// ToolCall for ToolCallStart events
ToolCall *ToolCall
// Error if something went wrong
Error error
// InputTokens after message start
InputTokens int
// OutputTokens after message end
OutputTokens int
// CostUSD is the request cost, set on MessageEnd by backends that
// know their pricing. Callers should prefer this over recomputing
// from the model name.
CostUSD float64
// StopReason, set on MessageEnd, indicates why generation stopped.
// Callers must handle refusal and pause_turn instead of treating
// every stream end as a completed answer.
StopReason StopReason
// Cache token counts (Anthropic prompt caching)
CacheCreationInputTokens int
CacheReadInputTokens int
}
StreamEvent is an event from streaming generation.
type StreamEventType ¶
type StreamEventType string
StreamEventType categorizes stream events.
const ( StreamEventMessageStart StreamEventType = "message_start" StreamEventContentStart StreamEventType = "content_start" StreamEventContentDelta StreamEventType = "content_delta" StreamEventContentEnd StreamEventType = "content_end" StreamEventToolStart StreamEventType = "tool_start" StreamEventToolDelta StreamEventType = "tool_delta" StreamEventToolEnd StreamEventType = "tool_end" StreamEventMessageEnd StreamEventType = "message_end" StreamEventError StreamEventType = "error" // Thinking block lifecycle. Delta carries the thinking text for // ThinkingDelta and the signature chunk for ThinkingSignature; the // block closes with the generic ContentEnd. StreamEventThinkingStart StreamEventType = "thinking_start" StreamEventThinkingDelta StreamEventType = "thinking_delta" StreamEventThinkingSignature StreamEventType = "thinking_signature" )
type ToolCall ¶
type ToolCall struct {
// ID is the unique identifier for this tool call
ID string
// Name is the tool being called
Name string
// Arguments are the parameters passed to the tool
Arguments map[string]any
}
ToolCall represents a tool call from the LLM.
type ToolSchema ¶
type ToolSchema struct {
// Name of the tool
Name string `json:"name"`
// Description of what the tool does
Description string `json:"description"`
// InputSchema is the JSON Schema for parameters
InputSchema map[string]any `json:"input_schema"`
}
ToolSchema describes a tool for the LLM.