llm

package
v0.9.3 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 20, 2026 License: MIT Imports: 14 Imported by: 0

Documentation

Overview

Package llm provides LLM backend implementations.

Package llm provides LLM backend implementations for the vega package.

Anthropic Backend

The primary backend is Anthropic's Claude API:

llm := llm.NewAnthropic()  // Uses ANTHROPIC_API_KEY env var

// Or with custom API key
llm := llm.NewAnthropic(llm.WithAPIKey("sk-..."))

// Or with custom model
llm := llm.NewAnthropic(llm.WithModel("claude-opus-4-7"))

Using with Orchestrator

Configure the orchestrator to use the LLM:

llm := llm.NewAnthropic()
orch := vega.NewOrchestrator(vega.WithLLM(llm))

Streaming

The Anthropic backend supports streaming responses:

stream, err := proc.SendStream(ctx, "Tell me a story")
for chunk := range stream.Chunks() {
    fmt.Print(chunk)
}

Tool Support

Tools are automatically converted to the Anthropic tool format:

tools := tools.NewTools()
tools.Register("search", searchFunc)

agent := vega.Agent{
    Tools: tools,
    // ...
}

When the model decides to use a tool, the tool is executed and the result is sent back automatically in a multi-turn conversation.

Rate Limiting

The Anthropic API has rate limits. Configure rate limiting on the orchestrator:

orch := vega.NewOrchestrator(
    vega.WithLLM(llm),
    vega.WithRateLimits(map[string]vega.RateLimitConfig{
        "claude-sonnet-4-6": {
            RequestsPerMinute: 60,
            TokensPerMinute:   100000,
        },
    }),
)

Implementing Custom Backends

To implement a custom LLM backend, implement the llm.LLM interface:

type LLM interface {
    Generate(ctx context.Context, messages []Message, tools []ToolSchema) (*LLMResponse, error)
    GenerateStream(ctx context.Context, messages []Message, tools []ToolSchema) (<-chan StreamEvent, error)
}

Index

Constants

View Source
const (
	DefaultAnthropicTimeout = 5 * time.Minute
	DefaultAnthropicModel   = "claude-sonnet-4-6"
	DefaultAnthropicBaseURL = "https://api.anthropic.com"
)

Default Anthropic configuration values

View Source
const (
	DefaultOpenAIModel   = "qwen-coder"
	DefaultOpenAIBaseURL = "http://localhost:4000"
)
View Source
const (
	BlockText       = "text"
	BlockThinking   = "thinking"
	BlockToolUse    = "tool_use"
	BlockToolResult = "tool_result"
	BlockImage      = "image"
	// BlockOpaque carries a content block govega does not model — server-side
	// tool blocks like server_tool_use and web_search_tool_result — verbatim
	// in Raw, so replaying the turn stays lossless.
	BlockOpaque = "opaque"
)

Content block types.

View Source
const DefaultMaxConcurrent = 5

DefaultMaxConcurrent is the default maximum number of in-flight API requests.

Variables

This section is empty.

Functions

func APIKeyFromContext added in v0.6.0

func APIKeyFromContext(ctx context.Context) string

APIKeyFromContext returns the per-request key set by WithAPIKeyContext, or "" when none is attached.

func CalculateCost

func CalculateCost(model string, inputTokens, outputTokens, cacheCreationTokens, cacheReadTokens int) float64

CalculateCost calculates the cost of a request including prompt cache tokens. Cache writes cost 125% of base input price; cache reads cost 10%.

func ContextWithOptions added in v0.5.1

func ContextWithOptions(ctx context.Context, opts Options) context.Context

ContextWithOptions returns a derived context carrying opts.

func WithAPIKeyContext added in v0.6.0

func WithAPIKeyContext(ctx context.Context, key string) context.Context

WithAPIKeyContext returns ctx carrying a per-request Anthropic API key. When set, outbound calls use this key instead of the one passed to NewAnthropic. Empty values are stored verbatim — callers should pass a real key.

Types

type AnthropicLLM

type AnthropicLLM struct {
	// contains filtered or unexported fields
}

AnthropicLLM is an LLM implementation using the Anthropic API.

func NewAnthropic

func NewAnthropic(opts ...AnthropicOption) *AnthropicLLM

NewAnthropic creates a new Anthropic LLM client.

func (*AnthropicLLM) Generate

func (a *AnthropicLLM) Generate(ctx context.Context, messages []Message, tools []ToolSchema) (*LLMResponse, error)

Generate sends a request and returns the complete response.

func (*AnthropicLLM) GenerateStream

func (a *AnthropicLLM) GenerateStream(ctx context.Context, messages []Message, tools []ToolSchema) (<-chan StreamEvent, error)

GenerateStream sends a request and returns a channel of streaming events.

func (*AnthropicLLM) ValidateKey

func (a *AnthropicLLM) ValidateKey(ctx context.Context) error

ValidateKey makes a minimal API call to verify the API key is valid. Returns nil on success, or an error describing the failure (empty key, authentication failure, or network/other error).

type AnthropicOption

type AnthropicOption func(*AnthropicLLM)

AnthropicOption configures the Anthropic client.

func WithAPIKey

func WithAPIKey(key string) AnthropicOption

WithAPIKey sets the API key.

func WithBaseURL

func WithBaseURL(url string) AnthropicOption

WithBaseURL sets the API base URL.

func WithEffort added in v0.5.1

func WithEffort(effort string) AnthropicOption

WithEffort sets output_config.effort for requests on capable models. Valid values: "low", "medium", "high", "xhigh", "max". Empty means "high" by default. "xhigh" is the recommended setting for agentic and coding workloads on Opus 4.7. "max" is Opus-tier only.

func WithHTTPClient

func WithHTTPClient(client *http.Client) AnthropicOption

WithHTTPClient sets a custom HTTP client. It is used for both the non-streaming and streaming paths — callers overriding it own the timeout trade-off (a non-zero Timeout will cut streams that outlive it).

func WithMaxConcurrent added in v0.3.0

func WithMaxConcurrent(n int) AnthropicOption

WithMaxConcurrent sets the maximum number of concurrent API requests.

func WithModel

func WithModel(model string) AnthropicOption

WithModel sets the default model.

func WithWebSearch added in v0.8.18

func WithWebSearch() AnthropicOption

WithWebSearch enables Anthropic's server-side web search and web fetch tools (web_search_20260209 / web_fetch_20260209). Requires a model that supports them (Opus 4.6+ / Sonnet 4.6+ generations); searches bill separately from tokens. A paused server-side loop surfaces as StopReasonPause, which the process loop already resumes.

type ContentBlock added in v0.8.0

type ContentBlock struct {
	Type string `json:"type"`

	// Text carries text and thinking content.
	Text string `json:"text,omitempty"`
	// Signature authenticates a thinking block so it can be replayed to
	// the API on the next request of the same turn.
	Signature string `json:"signature,omitempty"`

	// Tool use fields.
	ID        string         `json:"id,omitempty"`
	Name      string         `json:"name,omitempty"`
	Arguments map[string]any `json:"arguments,omitempty"`

	// Tool result fields.
	ToolUseID string `json:"tool_use_id,omitempty"`
	Content   string `json:"content,omitempty"`
	IsError   bool   `json:"is_error,omitempty"`

	// Image fields (BlockImage): base64-encoded image Data plus its MediaType
	// (e.g. "image/png"). Vision-capable models read these on user turns.
	MediaType string `json:"media_type,omitempty"`
	Data      string `json:"data,omitempty"`

	// Raw is the verbatim API block for BlockOpaque — block types govega
	// does not model, preserved so turn replay is lossless.
	Raw json.RawMessage `json:"raw,omitempty"`
}

ContentBlock is one typed unit of message content. Exactly one group of fields is meaningful per Type:

  • BlockText: Text
  • BlockThinking: Text (the reasoning), Signature (required for replay)
  • BlockToolUse: ID, Name, Arguments
  • BlockToolResult: ToolUseID, Content, IsError

type LLM

type LLM interface {
	// Generate sends a request and returns the complete response.
	Generate(ctx context.Context, messages []Message, tools []ToolSchema) (*LLMResponse, error)

	// GenerateStream sends a request and returns a channel of streaming events.
	GenerateStream(ctx context.Context, messages []Message, tools []ToolSchema) (<-chan StreamEvent, error)
}

LLM is the interface for language model backends.

func New added in v0.3.0

func New() LLM

New creates an LLM backend based on environment configuration. If OPENAI_BASE_URL is set, it returns an OpenAI-compatible client (works with LiteLLM, Ollama, vLLM, etc). Otherwise it returns an Anthropic client.

type LLMResponse

type LLMResponse struct {
	// Content is the text response
	Content string

	// Blocks is the ordered, typed content of the response (thinking,
	// text, tool_use). Callers replaying the assistant turn should attach
	// these to the next Message so thinking blocks and tool invocations
	// survive the round trip. Empty on backends without block support.
	Blocks []ContentBlock

	// ToolCalls are any tool calls the model wants to make
	ToolCalls []ToolCall

	// Token counts
	InputTokens  int
	OutputTokens int

	// Cache token counts (Anthropic prompt caching)
	CacheCreationInputTokens int
	CacheReadInputTokens     int

	// Cost in USD
	CostUSD float64

	// Latency in milliseconds
	LatencyMs int64

	// StopReason indicates why generation stopped
	StopReason StopReason
}

LLMResponse is the response from an LLM call.

type Message

type Message struct {
	Role    Role
	Content string
	Blocks  []ContentBlock `json:"Blocks,omitempty"`

	// Volatile is per-conversation system content — who the agent is talking
	// to, what it remembers about them — that belongs in the system prompt but
	// is byte-unique per conversation. Meaningful only on a RoleSystem
	// message. Providers that support prompt caching render it as a second,
	// uncached system block after Content, so every conversation shares one
	// cached prefix instead of writing its own. Empty ⇒ one block, as before.
	Volatile string `json:"Volatile,omitempty"`
}

Message represents a conversation message.

Content carries plain text. Blocks, when non-empty, carries the full typed structure of the turn (text, thinking, tool_use, tool_result) and takes precedence over Content when building API requests — backends convert Blocks directly instead of round-tripping tool activity through XML markup in Content.

func (Message) SystemText added in v0.9.3

func (m Message) SystemText() string

SystemText returns the whole system prompt as the model sees it: the cacheable Content followed by the per-conversation Volatile tail. Use it anywhere the split is an implementation detail — assertions, logging, estimates — rather than reading Content and silently missing half of it.

type ModelCapabilities added in v0.5.1

type ModelCapabilities struct {
	// AdaptiveThinking — model supports {type: "adaptive"} thinking.
	// Older models use the legacy budget_tokens form, which we don't emit.
	AdaptiveThinking bool

	// SupportsEffort — model accepts output_config.effort.
	// Sonnet 4.5 and Haiku 4.5 return 400 if it's sent.
	SupportsEffort bool

	// SupportsTemperature — model accepts the temperature sampling
	// parameter. Opus 4.7 removed sampling params and returns 400 if
	// temperature/top_p/top_k are sent.
	SupportsTemperature bool

	// SupportsStructuredOutputs — model accepts output_config.format
	// for JSON schema enforcement. Supported on Claude 4.5+ Opus,
	// Sonnet 4.6, Haiku 4.5.
	SupportsStructuredOutputs bool

	// MaxOutputTokens is the streaming output ceiling for this model.
	MaxOutputTokens int
}

ModelCapabilities describes which API features a model supports. Used to gate per-request fields so we don't send shapes that 400 on the model.

func CapabilitiesFor added in v0.5.1

func CapabilitiesFor(model string) ModelCapabilities

CapabilitiesFor returns the capabilities for the given model. Unknown models return the zero value (everything disabled: no adaptive thinking, no effort, conservative output cap) and log a warning so a stale table or a typo'd model ID surfaces instead of silently running degraded.

type OpenAILLM added in v0.3.0

type OpenAILLM struct {
	// contains filtered or unexported fields
}

OpenAILLM is an LLM implementation using the OpenAI-compatible chat completions API. Works with LiteLLM, Ollama, vLLM, and any OpenAI-compatible endpoint.

func NewOpenAI added in v0.3.0

func NewOpenAI(opts ...OpenAIOption) *OpenAILLM

NewOpenAI creates a new OpenAI-compatible LLM client.

func (*OpenAILLM) Generate added in v0.3.0

func (o *OpenAILLM) Generate(ctx context.Context, messages []Message, tools []ToolSchema) (*LLMResponse, error)

Generate sends a request and returns the complete response.

func (*OpenAILLM) GenerateStream added in v0.3.0

func (o *OpenAILLM) GenerateStream(ctx context.Context, messages []Message, tools []ToolSchema) (<-chan StreamEvent, error)

GenerateStream sends a request and returns a channel of streaming events.

type OpenAIOption added in v0.3.0

type OpenAIOption func(*OpenAILLM)

OpenAIOption configures the OpenAI-compatible client.

func WithOpenAIAPIKey added in v0.3.0

func WithOpenAIAPIKey(key string) OpenAIOption

WithOpenAIAPIKey sets the API key.

func WithOpenAIBaseURL added in v0.3.0

func WithOpenAIBaseURL(url string) OpenAIOption

WithOpenAIBaseURL sets the API base URL.

func WithOpenAIModel added in v0.3.0

func WithOpenAIModel(model string) OpenAIOption

WithOpenAIModel sets the default model.

func WithOpenAIPricing added in v0.8.0

func WithOpenAIPricing(inputPer1M, outputPer1M float64) OpenAIOption

WithOpenAIPricing sets per-1M-token USD rates used for cost accounting. Unset (zero) rates report $0 — the right answer for local models.

type Options added in v0.5.1

type Options struct {
	// Model overrides the client's default model for this call.
	Model string

	// Temperature is a float pointer because zero is a valid value;
	// nil means "do not send" (use model default). Backends that
	// don't support sampling parameters drop this silently.
	Temperature *float64

	// MaxTokens caps response output. 0 means use the capability default.
	MaxTokens int

	// Effort sets output_config.effort on supported models.
	// Empty defaults to "high" if the model supports effort.
	Effort string

	// OutputSchema is a JSON Schema enforced on the response via
	// output_config.format. When set on a model that supports
	// structured outputs, the API returns valid JSON conforming to
	// the schema (no markdown fences, no invented fields). Silently
	// dropped on unsupported models — caller falls back to ad-hoc
	// JSON parsing in that case.
	OutputSchema map[string]any
}

Options carries per-call overrides for an LLM request. They flow through context.Context so callers can inject per-Agent values without changing the LLM interface.

Empty / zero fields fall back to the LLM client's defaults.

func OptionsFromContext added in v0.5.1

func OptionsFromContext(ctx context.Context) Options

OptionsFromContext returns the Options previously installed on ctx, or the zero value if none.

type Role

type Role string

Role identifies the message sender.

const (
	RoleUser      Role = "user"
	RoleAssistant Role = "assistant"
	RoleSystem    Role = "system"
)

type StopReason

type StopReason string

StopReason indicates why the LLM stopped generating.

const (
	StopReasonEnd      StopReason = "end_turn"
	StopReasonToolUse  StopReason = "tool_use"
	StopReasonLength   StopReason = "max_tokens"
	StopReasonStop     StopReason = "stop_sequence"
	StopReasonFiltered StopReason = "content_filter"

	// StopReasonPause is set when a server-side tool sampling loop hit
	// its iteration limit. Caller should re-send the assistant turn
	// unchanged; the API resumes automatically.
	StopReasonPause StopReason = "pause_turn"

	// StopReasonRefusal is set when Claude declined to respond for
	// safety reasons. Output may not match an expected schema. Do
	// NOT retry the same prompt — surface to the user.
	StopReasonRefusal StopReason = "refusal"

	// StopReasonContextExceeded is set when the model's context
	// window was exhausted (distinct from max_tokens, which is the
	// per-response output cap). Caller should compact or split the
	// conversation.
	StopReasonContextExceeded StopReason = "model_context_window_exceeded"
)

type StreamEvent

type StreamEvent struct {
	// Type of event
	Type StreamEventType

	// Delta is new content for ContentDelta events
	Delta string

	// ToolCall for ToolCallStart events
	ToolCall *ToolCall

	// Error if something went wrong
	Error error

	// InputTokens after message start
	InputTokens int

	// OutputTokens after message end
	OutputTokens int

	// CostUSD is the request cost, set on MessageEnd by backends that
	// know their pricing. Callers should prefer this over recomputing
	// from the model name.
	CostUSD float64

	// StopReason, set on MessageEnd, indicates why generation stopped.
	// Callers must handle refusal and pause_turn instead of treating
	// every stream end as a completed answer.
	StopReason StopReason

	// Cache token counts (Anthropic prompt caching)
	CacheCreationInputTokens int
	CacheReadInputTokens     int
}

StreamEvent is an event from streaming generation.

type StreamEventType

type StreamEventType string

StreamEventType categorizes stream events.

const (
	StreamEventMessageStart StreamEventType = "message_start"
	StreamEventContentStart StreamEventType = "content_start"
	StreamEventContentDelta StreamEventType = "content_delta"
	StreamEventContentEnd   StreamEventType = "content_end"
	StreamEventToolStart    StreamEventType = "tool_start"
	StreamEventToolDelta    StreamEventType = "tool_delta"
	StreamEventToolEnd      StreamEventType = "tool_end"
	StreamEventMessageEnd   StreamEventType = "message_end"
	StreamEventError        StreamEventType = "error"

	// Thinking block lifecycle. Delta carries the thinking text for
	// ThinkingDelta and the signature chunk for ThinkingSignature; the
	// block closes with the generic ContentEnd.
	StreamEventThinkingStart     StreamEventType = "thinking_start"
	StreamEventThinkingDelta     StreamEventType = "thinking_delta"
	StreamEventThinkingSignature StreamEventType = "thinking_signature"
)

type ToolCall

type ToolCall struct {
	// ID is the unique identifier for this tool call
	ID string

	// Name is the tool being called
	Name string

	// Arguments are the parameters passed to the tool
	Arguments map[string]any
}

ToolCall represents a tool call from the LLM.

type ToolSchema

type ToolSchema struct {
	// Name of the tool
	Name string `json:"name"`

	// Description of what the tool does
	Description string `json:"description"`

	// InputSchema is the JSON Schema for parameters
	InputSchema map[string]any `json:"input_schema"`
}

ToolSchema describes a tool for the LLM.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL