llm

package
v0.19.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 21, 2026 License: Apache-2.0 Imports: 8 Imported by: 0

Documentation

Overview

Package llm provides canonical types for LLM chat interactions. These types are provider-agnostic; each provider translates to/from its native API format.

Index

Constants

View Source
const (
	// AuthSchemeAWSSigV4 signs every outbound request with AWS SigV4
	// and skips the provider-native API-key header (issue #202 Phase 2).
	AuthSchemeAWSSigV4 = "aws_sigv4"

	// AuthSchemeAPIKeyHeader additionally sends the API key in a gateway
	// header (default `apikey`, overridable via AuthHeaderName) ON TOP OF
	// the provider-native header. For API gateways whose auth plugin reads
	// a fixed header name — e.g. Kong AI Gateway's key-auth, which reads
	// `apikey` and ignores `Authorization` / `x-api-key`. Additive, so it
	// is safe to enable against non-gateway endpoints too (issue #302).
	//
	// Use this when the gateway either replaces the native header upstream
	// (Kong request-transformer `replace`, or ai-proxy with
	// allow_override=true) or the key is itself a valid provider key that
	// passes through.
	AuthSchemeAPIKeyHeader = "apikey_header"

	// AuthSchemeAPIKeyHeaderOnly sends the API key ONLY in the gateway
	// header and SUPPRESSES the provider-native header (`x-api-key` /
	// `Authorization`), mirroring how aws_sigv4 skips the native header.
	// The gateway owns provider auth: it must inject the real upstream
	// credential itself. Use this when the gateway ADDS the native header
	// (Kong request-transformer `add`, which won't overwrite an existing
	// one) — sending the native header from Forge would block that
	// injection and the provider would 401 on the gateway key (issue #349
	// follow-up: Kong `add`-injection). The gateway key never reaches the
	// provider's native auth header.
	AuthSchemeAPIKeyHeaderOnly = "apikey_header_only"

	// DefaultAPIKeyHeaderName is the header the apikey_header schemes use
	// when AuthHeaderName is unset — Kong key-auth's default key_names.
	DefaultAPIKeyHeaderName = "apikey"
)

Outbound LLM auth schemes (ClientConfig.AuthScheme / ModelRef.auth_scheme).

View Source
const (
	RoleSystem    = "system"
	RoleUser      = "user"
	RoleAssistant = "assistant"
	RoleTool      = "tool"
)

Role constants for chat messages.

View Source
const ProviderOpenAIResponses = "openai-responses"

ProviderOpenAIResponses is the provider string that selects the OpenAI Responses API client (POST <baseURL>/responses) via config + API-key/gateway auth, distinct from the Chat Completions "openai" provider (#383). It shares the ResponsesClient with the ChatGPT OAuth path but does not force store=false. Kept here so both the providers factory and the runtime config resolver reference one spelling.

Variables

This section is empty.

Functions

This section is empty.

Types

type ChatMessage

type ChatMessage struct {
	Role       string     `json:"role"`
	Content    string     `json:"content,omitempty"`
	ToolCalls  []ToolCall `json:"tool_calls,omitempty"`
	ToolCallID string     `json:"tool_call_id,omitempty"`
	Name       string     `json:"name,omitempty"`
}

ChatMessage represents a single message in a chat conversation.

type ChatRequest

type ChatRequest struct {
	Model       string           `json:"model"`
	Messages    []ChatMessage    `json:"messages"`
	Tools       []ToolDefinition `json:"tools,omitempty"`
	Temperature *float64         `json:"temperature,omitempty"`
	MaxTokens   int              `json:"max_tokens,omitempty"`
	Stream      bool             `json:"stream,omitempty"`
}

ChatRequest is a provider-agnostic chat completion request.

type ChatResponse

type ChatResponse struct {
	ID           string      `json:"id"`
	Message      ChatMessage `json:"message"`
	Usage        UsageInfo   `json:"usage"`
	FinishReason string      `json:"finish_reason"`
	// Endpoint is the URL the client POSTed to (base URL + provider path).
	// Set by the provider client so the llm_call audit event can record the
	// invoked path even when payload capture is off. Internal only (json:"-").
	Endpoint string `json:"-"`
}

ChatResponse is a provider-agnostic chat completion response.

type Client

type Client interface {
	// Chat sends a chat completion request and returns the response.
	Chat(ctx context.Context, req *ChatRequest) (*ChatResponse, error)
	// ChatStream sends a streaming chat request and returns a channel of deltas.
	ChatStream(ctx context.Context, req *ChatRequest) (<-chan StreamDelta, error)
	// ModelID returns the model identifier this client is configured for.
	ModelID() string
}

Client is the interface for interacting with an LLM provider.

type ClientConfig

type ClientConfig struct {
	APIKey      string
	BaseURL     string
	Model       string
	OrgID       string
	MaxRetries  int
	TimeoutSecs int

	// AuthScheme + AWSRegion control outbound authentication when
	// the operator points the client at AWS Bedrock (Anthropic
	// passthrough or OpenAI compatibility endpoint) or any other
	// SigV4-fronted gateway. Issue #202 Phase 2. Mirrors the
	// matching forge.yaml ModelRef fields.
	//
	// AuthScheme == "" preserves the pre-#202 behavior — the
	// Anthropic client sets `x-api-key: <APIKey>`, the OpenAI
	// client sets `Authorization: Bearer <APIKey>`. AuthScheme ==
	// "aws_sigv4" wraps the client's transport with the SigV4
	// signer and skips the native header logic; APIKey is ignored.
	// AuthScheme == "apikey_header" ADDITIONALLY sends APIKey in the
	// AuthHeaderName header (default "apikey") alongside the native
	// header, for gateways like Kong that read a fixed key header
	// (issue #302). AuthScheme == "apikey_header_only" sends APIKey in the
	// gateway header but SUPPRESSES the native header, for gateways that
	// inject the real upstream credential themselves.
	AuthScheme string
	AWSRegion  string

	// AuthHeaderName overrides the header used by the "apikey_header"
	// scheme. Empty → DefaultAPIKeyHeaderName ("apikey"). Ignored for
	// every other scheme. Set it for a gateway with custom key_names,
	// e.g. "x-gateway-key". Issue #302.
	AuthHeaderName string

	// PromptCaching opts the provider client into injecting the
	// provider's prompt-cache primitives on every request:
	//
	//   - anthropic: a cache_control ephemeral breakpoint on the last
	//     tool definition and on the system prompt block, caching the
	//     stable tools+system prefix across turns. Also honored by
	//     Anthropic-on-Bedrock gateways (aws_sigv4), which speak the
	//     same wire format.
	//   - openai: a stable prompt_cache_key derived from
	//     (model, system, tool names), pinning cache routing for the
	//     session. OpenAI prefix caching itself is automatic ≥1024
	//     tokens; the key improves hit locality.
	//
	// Off by default — wire formats stay byte-identical to the
	// pre-compression contract unless the operator opts in
	// (compression.cache_hints / compression.enabled in forge.yaml).
	PromptCaching bool

	// DisableStore sends `store: false` on OpenAI Responses API requests
	// (the "openai-responses" provider), telling OpenAI not to retain the
	// response server-side (~30-day default retention). Only the Responses
	// client honors it; other clients ignore it. Empty/false leaves `store`
	// unset so the API applies its own default (#383). The ChatGPT OAuth
	// path forces this true regardless (Codex backend requires it).
	DisableStore bool
}

ClientConfig holds configuration for creating an LLM client.

type CooldownTracker

type CooldownTracker struct {
	// contains filtered or unexported fields
}

CooldownTracker manages per-provider cooldown state with exponential backoff.

func NewCooldownTracker

func NewCooldownTracker() *CooldownTracker

NewCooldownTracker creates a new cooldown tracker.

func (*CooldownTracker) IsAvailable

func (ct *CooldownTracker) IsAvailable(provider string) bool

IsAvailable returns true if the provider is not currently in cooldown.

func (*CooldownTracker) MarkFailure

func (ct *CooldownTracker) MarkFailure(provider string, reason FailoverReason)

MarkFailure records a failure for the given provider.

func (*CooldownTracker) MarkSuccess

func (ct *CooldownTracker) MarkSuccess(provider string)

MarkSuccess resets all cooldown state for a provider.

type Embedder

type Embedder interface {
	// Embed produces embeddings for the given texts.
	Embed(ctx context.Context, req *EmbeddingRequest) (*EmbeddingResponse, error)
	// Dimensions returns the dimensionality of the embedding vectors.
	Dimensions() int
}

Embedder generates vector embeddings from text.

type EmbeddingRequest

type EmbeddingRequest struct {
	Texts []string // texts to embed
	Model string   // optional model override
}

EmbeddingRequest is a provider-agnostic request to generate embeddings.

type EmbeddingResponse

type EmbeddingResponse struct {
	Embeddings [][]float32
	Model      string
	Usage      UsageInfo
}

EmbeddingResponse is a provider-agnostic embedding response.

type FailoverError

type FailoverError struct {
	Reason   FailoverReason
	Provider string
	Model    string
	Status   int
	Wrapped  error
}

FailoverError wraps an LLM provider error with classification metadata.

func ClassifyError

func ClassifyError(err error, provider, model string) *FailoverError

ClassifyError wraps a raw provider error into a FailoverError with the appropriate reason. It extracts HTTP status codes from known provider error formats and falls back to message pattern matching.

func (*FailoverError) Error

func (e *FailoverError) Error() string

func (*FailoverError) IsRetriable

func (e *FailoverError) IsRetriable() bool

IsRetriable returns true if this error should trigger a fallback attempt. Auth and format errors are never retriable.

func (*FailoverError) Unwrap

func (e *FailoverError) Unwrap() error

type FailoverReason

type FailoverReason string

FailoverReason describes why a provider failed.

const (
	FailoverAuth       FailoverReason = "auth"       // 401/403
	FailoverRateLimit  FailoverReason = "rate_limit" // 429
	FailoverBilling    FailoverReason = "billing"    // 402
	FailoverTimeout    FailoverReason = "timeout"    // 408/504/deadline
	FailoverOverloaded FailoverReason = "overloaded" // 500/502/503/529
	FailoverFormat     FailoverReason = "format"     // 400
	FailoverUnknown    FailoverReason = "unknown"    // unknown error (treated as retriable)
)

type FallbackCandidate

type FallbackCandidate struct {
	Provider string
	Model    string
	Client   Client
}

FallbackCandidate pairs a provider/model label with its LLM client.

type FallbackChain

type FallbackChain struct {
	// contains filtered or unexported fields
}

FallbackChain implements the Client interface by trying multiple LLM providers in order. When the primary provider fails with a retriable error (429, 503, timeouts), the chain moves to the next candidate. Non-retriable errors (400 bad request, 401 auth) abort immediately.

When there is only one candidate, FallbackChain delegates directly without error classification to preserve exact current behavior.

func NewFallbackChain

func NewFallbackChain(candidates []FallbackCandidate) *FallbackChain

NewFallbackChain creates a new fallback chain from the given candidates. At least one candidate is required.

func (*FallbackChain) Chat

func (fc *FallbackChain) Chat(ctx context.Context, req *ChatRequest) (*ChatResponse, error)

Chat tries each candidate in order until one succeeds or all are exhausted.

func (*FallbackChain) ChatStream

func (fc *FallbackChain) ChatStream(ctx context.Context, req *ChatRequest) (<-chan StreamDelta, error)

ChatStream tries each candidate in order for streaming requests.

func (*FallbackChain) ModelID

func (fc *FallbackChain) ModelID() string

ModelID returns the primary candidate's model identifier.

type FallbackExhaustedError

type FallbackExhaustedError struct {
	Errors []*FailoverError
}

FallbackExhaustedError is returned when all candidates have been tried and failed.

func (*FallbackExhaustedError) Error

func (e *FallbackExhaustedError) Error() string

type FunctionCall

type FunctionCall struct {
	Name      string `json:"name"`
	Arguments string `json:"arguments"` // JSON string
}

FunctionCall contains the function name and arguments for a tool call.

type FunctionSchema

type FunctionSchema struct {
	Name        string          `json:"name"`
	Description string          `json:"description"`
	Parameters  json.RawMessage `json:"parameters,omitempty"`
}

FunctionSchema describes a function's name, description, and parameters.

type StreamDelta

type StreamDelta struct {
	Content      string     `json:"content,omitempty"`
	ToolCalls    []ToolCall `json:"tool_calls,omitempty"`
	FinishReason string     `json:"finish_reason,omitempty"`
	Done         bool       `json:"done,omitempty"`
	Usage        *UsageInfo `json:"usage,omitempty"`
}

StreamDelta represents a single chunk in a streaming response.

type ToolCall

type ToolCall struct {
	ID       string       `json:"id"`
	Type     string       `json:"type"` // always "function"
	Function FunctionCall `json:"function"`
}

ToolCall represents an LLM request to invoke a tool.

type ToolDefinition

type ToolDefinition struct {
	Type     string         `json:"type"` // always "function"
	Function FunctionSchema `json:"function"`
}

ToolDefinition describes a tool available to the LLM.

type UsageInfo

type UsageInfo struct {
	InputTokens  int `json:"input_tokens"`
	OutputTokens int `json:"output_tokens"`
	TotalTokens  int `json:"total_tokens"`
}

UsageInfo contains token usage information.

Field naming aligns with OTel GenAI semantic conventions (gen_ai.usage.input_tokens / gen_ai.usage.output_tokens) so audit consumers can correlate Forge audit events with OTel traces without a translation table. See issue #87 / FWS-3.

Directories

Path Synopsis
Package providers implements LLM client providers for various APIs.
Package providers implements LLM client providers for various APIs.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL