llm

package
v0.1.3 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 15, 2026 License: Apache-2.0 Imports: 15 Imported by: 0

Documentation

Overview

Package llm defines the llm.Provider interface and implements the OpenAI, openai_compatible, and Anthropic adapters with streaming, tool-calling, structured-output support, failover, circuit-breaking, and token tracking. The openai_compatible adapter speaks the OpenAI chat-completions wire format to any server exposing it (Ollama, vLLM, LM Studio, Together, Groq).

See TAD §2.6, §2.7 and PRD §26 for the full specification. Implemented in Phase 7.

Package llm defines the llm.Provider abstraction and implements OpenAI and Anthropic adapters with streaming, tool-calling, structured-output support, failover, circuit-breaking, and token tracking (TAD §2.6–§2.7, PRD §26).

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type AnthropicProvider

type AnthropicProvider struct {
	// contains filtered or unexported fields
}

AnthropicProvider implements Provider against the Anthropic Messages API. Structured output (ResponseFormat) is not natively supported by the Messages API, so SupportsStructuredOutput returns false and ResponseFormat is ignored (TAD §11.3 plans only use ResponseFormat when the selected provider supports it). See TAD §2.7 and PRD §26.

func NewAnthropicProvider

func NewAnthropicProvider(opts ProviderOptions) *AnthropicProvider

NewAnthropicProvider constructs an Anthropic adapter. BaseURL and HTTPClient in opts default to the official endpoint and http.DefaultClient respectively.

func (*AnthropicProvider) ChatCompletion

func (p *AnthropicProvider) ChatCompletion(ctx context.Context, req ChatRequest) (*ChatResponse, error)

func (*AnthropicProvider) ModelInfo

func (p *AnthropicProvider) ModelInfo() ModelInfo

func (*AnthropicProvider) StreamChatCompletion

func (p *AnthropicProvider) StreamChatCompletion(ctx context.Context, req ChatRequest) (<-chan ChatChunk, error)

func (*AnthropicProvider) SupportsStructuredOutput

func (p *AnthropicProvider) SupportsStructuredOutput() bool

func (*AnthropicProvider) SupportsToolCalling

func (p *AnthropicProvider) SupportsToolCalling() bool

type AuthMode

type AuthMode string

AuthMode selects how a provider authenticates to its endpoint.

const (
	// AuthBearer sends an Authorization: Bearer <api_key> header on every
	// request. This is the OpenAI adapter's default and preserves its historical
	// behavior (an empty key still emits the header).
	AuthBearer AuthMode = "bearer"
	// AuthBearerIfKey sends the Authorization: Bearer header only when an API
	// key is configured. This is the openai_compatible adapter's default: local
	// endpoints (Ollama, LM Studio, vLLM) typically need no key, while hosted
	// compatible endpoints authenticate once api_key is set.
	AuthBearerIfKey AuthMode = "bearer_if_key"
	// AuthNone omits the Authorization header entirely.
	AuthNone AuthMode = "none"
)

type ChatChunk

type ChatChunk struct {
	// Content is a text delta.
	Content string
	// ToolCalls carries incremental tool-call fragments (argument text arrives
	// across multiple chunks and must be accumulated by the caller).
	ToolCalls []ToolCall
	// FinishReason is set on the final chunk when the model signals a stop.
	FinishReason string
}

ChatChunk is a partial streaming result.

type ChatRequest

type ChatRequest struct {
	// Model overrides the provider's configured default model. Empty = default.
	Model string
	// Messages is the full conversation history in order.
	Messages []Message
	// Tools is the set of tool definitions to expose to the model.
	Tools []ToolDefinition
	// Temperature controls sampling. 0 = deterministic (default).
	Temperature float64
	// MaxTokens caps the completion length. 0 = provider default.
	MaxTokens int
	// ResponseFormat constrains output to a JSON Schema. Non-nil only in
	// Plan-and-Execute mode (TAD §11.3). Providers that do not support it
	// (SupportsStructuredOutput() == false) ignore it.
	ResponseFormat *JSONSchemaFormat
}

ChatRequest is the provider-agnostic chat completion request. See TAD §2.7 and PRD §26.1.

type ChatResponse

type ChatResponse struct {
	// Content is the model's text output (empty when ToolCalls is non-empty).
	Content string
	// ToolCalls is the set of function invocations the model requested.
	ToolCalls []ToolCall
	// Usage reports token consumption for this completion.
	Usage TokenUsage
	// FinishReason is a provider-agnostic stop cause: "stop", "length",
	// "tool_calls", "content_filter", ...
	FinishReason string
}

ChatResponse is the provider-agnostic completion result. See PRD §26.1.

type Gateway

type Gateway struct {
	// contains filtered or unexported fields
}

Gateway implements PRD §26.4 Provider Resilience on top of one or more Provider implementations:

  • Automatic failover: a 5xx (or 5xx-class, any HTTPStatusError with StatusCode >= 500) from a provider triggers a retry on the next provider in the ordered list.
  • Circuit breaker: after FailoverThreshold consecutive failures a provider is skipped for a Cooldown period (matching PRD §26.4 "After N consecutive failures, stop sending to a provider for a cooldown period").
  • Token budget tracking: usage is accumulated globally and can be attributed per key (session/user id) via TrackUsage/UsageFor.

A non-5xx error (e.g. 401 auth, 400 validation) is returned immediately without trying fallbacks — a fallback provider would fail the same way.

Gateway satisfies the Provider interface (TAD §2.7), so it composes as a normal provider wherever the runtime expects one.

func NewGateway

func NewGateway(primary Provider, fallbacks ...Provider) *Gateway

NewGateway builds a Gateway with primary first, followed by fallbacks in priority order. Failover proceeds down the list.

func NewGatewayWithOptions

func NewGatewayWithOptions(primary Provider, opts GatewayOptions, fallbacks ...Provider) *Gateway

NewGatewayWithOptions is NewGateway with explicit resilience tuning.

func (*Gateway) ChatCompletion

func (g *Gateway) ChatCompletion(ctx context.Context, req ChatRequest) (*ChatResponse, error)

func (*Gateway) ModelInfo

func (g *Gateway) ModelInfo() ModelInfo

func (*Gateway) StreamChatCompletion

func (g *Gateway) StreamChatCompletion(ctx context.Context, req ChatRequest) (<-chan ChatChunk, error)

func (*Gateway) SupportsStructuredOutput

func (g *Gateway) SupportsStructuredOutput() bool

func (*Gateway) SupportsToolCalling

func (g *Gateway) SupportsToolCalling() bool

func (*Gateway) TrackUsage

func (g *Gateway) TrackUsage(key string, usage TokenUsage)

TrackUsage attributes token usage to a key (e.g. a session or user id) so the Safety Layer can enforce per-session/per-user budgets (PRD §26.4).

func (*Gateway) Usage

func (g *Gateway) Usage() TokenUsage

Usage returns the total token usage across all calls and keys.

func (*Gateway) UsageFor

func (g *Gateway) UsageFor(key string) TokenUsage

UsageFor returns the accumulated token usage for a key.

type GatewayOptions

type GatewayOptions struct {
	// FailoverThreshold is the number of consecutive failures that opens a
	// provider's circuit. Default: 3.
	FailoverThreshold int
	// Cooldown is how long an open circuit stays open before a probe request
	// is allowed through. Default: 30s.
	Cooldown time.Duration
}

GatewayOptions tunes failover behavior. Zero values select the defaults.

type HTTPStatusError

type HTTPStatusError struct {
	StatusCode int
	Message    string
}

HTTPStatusError identifies an LLM provider HTTP failure carrying its status code. The Gateway's failover logic walks the error chain (via Unwrap) looking for this type to decide whether to retry on a fallback provider: any status >= 500 triggers failover (PRD §26.4). It is wrapped inside an orjanda errors.Error by the concrete providers.

func (*HTTPStatusError) Error

func (e *HTTPStatusError) Error() string

type JSONSchemaFormat

type JSONSchemaFormat struct {
	Name   string
	Schema map[string]any
}

JSONSchemaFormat constrains model output to a named JSON Schema (TAD §11.3).

type Message

type Message struct {
	// Role identifies the speaker: "system" | "user" | "assistant" | "tool".
	Role string
	// Content is the plain-text body of the message.
	Content string
	// Name is the tool name for "tool" role messages.
	Name string
	// ToolCallID ties a "tool" role message to the assistant's tool call.
	ToolCallID string
	// ToolCalls appears on "assistant" messages that invoke tools.
	ToolCalls []ToolCall
}

Message is a single conversation turn in the provider-agnostic format. Roles are "system", "user", "assistant", and "tool".

type ModelInfo

type ModelInfo struct {
	// Name is the provider model identifier.
	Name string
	// SupportsTools mirrors Provider.SupportsToolCalling.
	SupportsTools bool
	// SupportsStructuredOutput mirrors Provider.SupportsStructuredOutput.
	SupportsStructuredOutput bool
}

ModelInfo describes the static capabilities of a configured model.

type OpenAICompatibleProvider

type OpenAICompatibleProvider struct {
	// contains filtered or unexported fields
}

OpenAICompatibleProvider implements Provider against any server exposing the OpenAI chat-completions API — e.g. Ollama, vLLM, LM Studio, Together, Groq. It shares the OpenAI adapter's wire format but, unlike the certified openai adapter, it is keyless by default (requests authenticate only when api_key is set) and reports capabilities per instance instead of hardcoding them, since compatible servers vary in tool and structured-output support.

func NewOpenAICompatibleProvider

func NewOpenAICompatibleProvider(opts ProviderOptions) (*OpenAICompatibleProvider, error)

NewOpenAICompatibleProvider constructs an OpenAI-compatible adapter. BaseURL is required — there is no official endpoint to fall back to. The adapter defaults to AuthBearerIfKey: requests carry an Authorization: Bearer header only when opts.APIKey is non-empty. Per-instance capability overrides can be supplied via opts.ToolCalling and opts.StructuredOutput.

func (*OpenAICompatibleProvider) ChatCompletion

func (p *OpenAICompatibleProvider) ChatCompletion(ctx context.Context, req ChatRequest) (*ChatResponse, error)

func (*OpenAICompatibleProvider) ModelInfo

func (p *OpenAICompatibleProvider) ModelInfo() ModelInfo

func (*OpenAICompatibleProvider) StreamChatCompletion

func (p *OpenAICompatibleProvider) StreamChatCompletion(ctx context.Context, req ChatRequest) (<-chan ChatChunk, error)

func (*OpenAICompatibleProvider) SupportsStructuredOutput

func (p *OpenAICompatibleProvider) SupportsStructuredOutput() bool

func (*OpenAICompatibleProvider) SupportsToolCalling

func (p *OpenAICompatibleProvider) SupportsToolCalling() bool

type OpenAIProvider

type OpenAIProvider struct {
	// contains filtered or unexported fields
}

OpenAIProvider implements Provider against the OpenAI chat-completions API. It also backs the openai_compatible adapter (see openai_compatible.go), which shares this wire format. See TAD §2.7 and PRD §26.

func NewOpenAIProvider

func NewOpenAIProvider(opts ProviderOptions) *OpenAIProvider

NewOpenAIProvider constructs an OpenAI adapter. BaseURL and HTTPClient in opts default to the official endpoint and http.DefaultClient respectively. Capabilities default to both tool calling and structured output supported; opts.ToolCalling/StructuredOutput override per instance.

func (*OpenAIProvider) ChatCompletion

func (p *OpenAIProvider) ChatCompletion(ctx context.Context, req ChatRequest) (*ChatResponse, error)

func (*OpenAIProvider) ModelInfo

func (p *OpenAIProvider) ModelInfo() ModelInfo

func (*OpenAIProvider) StreamChatCompletion

func (p *OpenAIProvider) StreamChatCompletion(ctx context.Context, req ChatRequest) (<-chan ChatChunk, error)

func (*OpenAIProvider) SupportsStructuredOutput

func (p *OpenAIProvider) SupportsStructuredOutput() bool

func (*OpenAIProvider) SupportsToolCalling

func (p *OpenAIProvider) SupportsToolCalling() bool

type Provider

type Provider interface {
	// ChatCompletion performs a complete, non-streaming chat round trip.
	ChatCompletion(ctx context.Context, req ChatRequest) (*ChatResponse, error)

	// StreamChatCompletion performs a streaming chat round trip. The initial
	// request is made synchronously so connection-level failures are returned
	// as the error; once established, partial results arrive on the channel,
	// which the provider closes when the stream completes. A stream that fails
	// mid-flight is closed early (the TAD §2.7 contract carries no per-chunk
	// error field, so callers must treat a short stream as an error).
	StreamChatCompletion(ctx context.Context, req ChatRequest) (<-chan ChatChunk, error)

	// SupportsToolCalling reports whether the provider can emit and consume
	// function-call tool definitions.
	SupportsToolCalling() bool

	// SupportsStructuredOutput reports whether the provider can constrain
	// output to a JSON Schema via ChatRequest.ResponseFormat.
	SupportsStructuredOutput() bool

	// ModelInfo returns static capabilities for the provider's configured model.
	ModelInfo() ModelInfo
}

Provider is the interface every LLM backend must implement. It is the framework's single abstraction over chat-completion providers; the Agent Runtime (Phase 8) and the Gateway (failover/circuit-breaker, this package) consume only this interface. See TAD §2.7 and PRD §26.1.

func ProviderFromConfig

func ProviderFromConfig(cfg *config.Config, model string) (Provider, error)

ProviderFromConfig builds the default configured provider from a config. It is the single place a site/CLI resolves site.Config.LLM into a Provider (TAD §9.3: llm.Provider | site.Config.LLM.Providers[name]). model overrides the provider's configured model when non-empty.

type ProviderOptions

type ProviderOptions struct {
	// APIKey is the provider secret.
	APIKey string
	// Model is the default model identifier.
	Model string
	// BaseURL overrides the provider endpoint (defaults to the official
	// endpoint; required for openai_compatible). Tests point this at an
	// httptest.Server.
	BaseURL string
	// MaxTokens is the default completion token cap (0 = provider default).
	MaxTokens int
	// HTTPClient overrides the shared HTTP client (tests inject a short one).
	HTTPClient *http.Client
	// Auth selects the authentication mode (see AuthMode). Zero value means
	// the provider's default: AuthBearer for openai, AuthBearerIfKey for
	// openai_compatible.
	Auth AuthMode
	// ToolCalling and StructuredOutput override the adapter's capability
	// report. OpenAI-compatible servers vary in support, so per-instance
	// overrides let a self-hosted endpoint disable features it lacks.
	// nil = adapter default.
	ToolCalling      *bool
	StructuredOutput *bool
}

ProviderOptions configures the built-in OpenAI, openai_compatible, and Anthropic adapters.

type TokenUsage

type TokenUsage struct {
	PromptTokens     int
	CompletionTokens int
	TotalTokens      int
}

TokenUsage tracks token consumption for a single completion.

type ToolCall

type ToolCall struct {
	// ID is the provider-assigned invocation id, echoed back on the tool result.
	ID string
	// Name is the tool name the model chose to invoke.
	Name string
	// Arguments is the JSON-encoded argument map the model produced.
	Arguments string
}

ToolCall is a function invocation requested by the model.

type ToolDefinition

type ToolDefinition struct {
	Name        string
	Description string
	// Parameters is a JSON Schema object ("type": "object", "properties",
	// "required", ...). See TAD §10.2 for the field-mapping rules.
	Parameters map[string]any
}

ToolDefinition is the provider-agnostic JSON Schema description of a tool. It is the exact shape ToolRegistry.ForIdentity returns (TAD §10).

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL