Documentation
¶
Overview ¶
Package llm provides canonical types for LLM chat interactions. These types are provider-agnostic; each provider translates to/from its native API format.
Index ¶
- Constants
- type ChatMessage
- type ChatRequest
- type ChatResponse
- type Client
- type ClientConfig
- type CooldownTracker
- type Embedder
- type EmbeddingRequest
- type EmbeddingResponse
- type FailoverError
- type FailoverReason
- type FallbackCandidate
- type FallbackChain
- type FallbackExhaustedError
- type FunctionCall
- type FunctionSchema
- type StreamDelta
- type ToolCall
- type ToolDefinition
- type UsageInfo
Constants ¶
const ( // AuthSchemeAWSSigV4 signs every outbound request with AWS SigV4 // and skips the provider-native API-key header (issue #202 Phase 2). AuthSchemeAWSSigV4 = "aws_sigv4" // AuthSchemeAPIKeyHeader additionally sends the API key in a gateway // header (default `apikey`, overridable via AuthHeaderName) ON TOP OF // the provider-native header. For API gateways whose auth plugin reads // a fixed header name — e.g. Kong AI Gateway's key-auth, which reads // `apikey` and ignores `Authorization` / `x-api-key`. Additive, so it // is safe to enable against non-gateway endpoints too (issue #302). // // Use this when the gateway either replaces the native header upstream // (Kong request-transformer `replace`, or ai-proxy with // allow_override=true) or the key is itself a valid provider key that // passes through. AuthSchemeAPIKeyHeader = "apikey_header" // AuthSchemeAPIKeyHeaderOnly sends the API key ONLY in the gateway // header and SUPPRESSES the provider-native header (`x-api-key` / // `Authorization`), mirroring how aws_sigv4 skips the native header. // The gateway owns provider auth: it must inject the real upstream // credential itself. Use this when the gateway ADDS the native header // (Kong request-transformer `add`, which won't overwrite an existing // one) — sending the native header from Forge would block that // injection and the provider would 401 on the gateway key (issue #349 // follow-up: Kong `add`-injection). The gateway key never reaches the // provider's native auth header. AuthSchemeAPIKeyHeaderOnly = "apikey_header_only" // DefaultAPIKeyHeaderName is the header the apikey_header schemes use // when AuthHeaderName is unset — Kong key-auth's default key_names. DefaultAPIKeyHeaderName = "apikey" )
Outbound LLM auth schemes (ClientConfig.AuthScheme / ModelRef.auth_scheme).
const ( RoleSystem = "system" RoleUser = "user" RoleAssistant = "assistant" RoleTool = "tool" )
Role constants for chat messages.
const ProviderOpenAIResponses = "openai-responses"
ProviderOpenAIResponses is the provider string that selects the OpenAI Responses API client (POST <baseURL>/responses) via config + API-key/gateway auth, distinct from the Chat Completions "openai" provider (#383). It shares the ResponsesClient with the ChatGPT OAuth path but does not force store=false. Kept here so both the providers factory and the runtime config resolver reference one spelling.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type ChatMessage ¶
type ChatMessage struct {
Role string `json:"role"`
Content string `json:"content,omitempty"`
ToolCalls []ToolCall `json:"tool_calls,omitempty"`
ToolCallID string `json:"tool_call_id,omitempty"`
Name string `json:"name,omitempty"`
}
ChatMessage represents a single message in a chat conversation.
type ChatRequest ¶
type ChatRequest struct {
Model string `json:"model"`
Messages []ChatMessage `json:"messages"`
Tools []ToolDefinition `json:"tools,omitempty"`
Temperature *float64 `json:"temperature,omitempty"`
MaxTokens int `json:"max_tokens,omitempty"`
Stream bool `json:"stream,omitempty"`
}
ChatRequest is a provider-agnostic chat completion request.
type ChatResponse ¶
type ChatResponse struct {
ID string `json:"id"`
Message ChatMessage `json:"message"`
Usage UsageInfo `json:"usage"`
FinishReason string `json:"finish_reason"`
// Endpoint is the URL the client POSTed to (base URL + provider path).
// Set by the provider client so the llm_call audit event can record the
// invoked path even when payload capture is off. Internal only (json:"-").
Endpoint string `json:"-"`
}
ChatResponse is a provider-agnostic chat completion response.
type Client ¶
type Client interface {
// Chat sends a chat completion request and returns the response.
Chat(ctx context.Context, req *ChatRequest) (*ChatResponse, error)
// ChatStream sends a streaming chat request and returns a channel of deltas.
ChatStream(ctx context.Context, req *ChatRequest) (<-chan StreamDelta, error)
// ModelID returns the model identifier this client is configured for.
ModelID() string
}
Client is the interface for interacting with an LLM provider.
type ClientConfig ¶
type ClientConfig struct {
APIKey string
BaseURL string
Model string
OrgID string
MaxRetries int
TimeoutSecs int
// AuthScheme + AWSRegion control outbound authentication when
// the operator points the client at AWS Bedrock (Anthropic
// passthrough or OpenAI compatibility endpoint) or any other
// SigV4-fronted gateway. Issue #202 Phase 2. Mirrors the
// matching forge.yaml ModelRef fields.
//
// AuthScheme == "" preserves the pre-#202 behavior — the
// Anthropic client sets `x-api-key: <APIKey>`, the OpenAI
// client sets `Authorization: Bearer <APIKey>`. AuthScheme ==
// "aws_sigv4" wraps the client's transport with the SigV4
// signer and skips the native header logic; APIKey is ignored.
// AuthScheme == "apikey_header" ADDITIONALLY sends APIKey in the
// AuthHeaderName header (default "apikey") alongside the native
// header, for gateways like Kong that read a fixed key header
// (issue #302). AuthScheme == "apikey_header_only" sends APIKey in the
// gateway header but SUPPRESSES the native header, for gateways that
// inject the real upstream credential themselves.
AuthScheme string
AWSRegion string
// AuthHeaderName overrides the header used by the "apikey_header"
// scheme. Empty → DefaultAPIKeyHeaderName ("apikey"). Ignored for
// every other scheme. Set it for a gateway with custom key_names,
// e.g. "x-gateway-key". Issue #302.
AuthHeaderName string
// PromptCaching opts the provider client into injecting the
// provider's prompt-cache primitives on every request:
//
// - anthropic: a cache_control ephemeral breakpoint on the last
// tool definition and on the system prompt block, caching the
// stable tools+system prefix across turns. Also honored by
// Anthropic-on-Bedrock gateways (aws_sigv4), which speak the
// same wire format.
// - openai: a stable prompt_cache_key derived from
// (model, system, tool names), pinning cache routing for the
// session. OpenAI prefix caching itself is automatic ≥1024
// tokens; the key improves hit locality.
//
// Off by default — wire formats stay byte-identical to the
// pre-compression contract unless the operator opts in
// (compression.cache_hints / compression.enabled in forge.yaml).
PromptCaching bool
// DisableStore sends `store: false` on OpenAI Responses API requests
// (the "openai-responses" provider), telling OpenAI not to retain the
// response server-side (~30-day default retention). Only the Responses
// client honors it; other clients ignore it. Empty/false leaves `store`
// unset so the API applies its own default (#383). The ChatGPT OAuth
// path forces this true regardless (Codex backend requires it).
DisableStore bool
}
ClientConfig holds configuration for creating an LLM client.
type CooldownTracker ¶
type CooldownTracker struct {
// contains filtered or unexported fields
}
CooldownTracker manages per-provider cooldown state with exponential backoff.
func NewCooldownTracker ¶
func NewCooldownTracker() *CooldownTracker
NewCooldownTracker creates a new cooldown tracker.
func (*CooldownTracker) IsAvailable ¶
func (ct *CooldownTracker) IsAvailable(provider string) bool
IsAvailable returns true if the provider is not currently in cooldown.
func (*CooldownTracker) MarkFailure ¶
func (ct *CooldownTracker) MarkFailure(provider string, reason FailoverReason)
MarkFailure records a failure for the given provider.
func (*CooldownTracker) MarkSuccess ¶
func (ct *CooldownTracker) MarkSuccess(provider string)
MarkSuccess resets all cooldown state for a provider.
type Embedder ¶
type Embedder interface {
// Embed produces embeddings for the given texts.
Embed(ctx context.Context, req *EmbeddingRequest) (*EmbeddingResponse, error)
// Dimensions returns the dimensionality of the embedding vectors.
Dimensions() int
}
Embedder generates vector embeddings from text.
type EmbeddingRequest ¶
type EmbeddingRequest struct {
Texts []string // texts to embed
Model string // optional model override
}
EmbeddingRequest is a provider-agnostic request to generate embeddings.
type EmbeddingResponse ¶
EmbeddingResponse is a provider-agnostic embedding response.
type FailoverError ¶
type FailoverError struct {
Reason FailoverReason
Provider string
Model string
Status int
Wrapped error
}
FailoverError wraps an LLM provider error with classification metadata.
func ClassifyError ¶
func ClassifyError(err error, provider, model string) *FailoverError
ClassifyError wraps a raw provider error into a FailoverError with the appropriate reason. It extracts HTTP status codes from known provider error formats and falls back to message pattern matching.
func (*FailoverError) Error ¶
func (e *FailoverError) Error() string
func (*FailoverError) IsRetriable ¶
func (e *FailoverError) IsRetriable() bool
IsRetriable returns true if this error should trigger a fallback attempt. Auth and format errors are never retriable.
func (*FailoverError) Unwrap ¶
func (e *FailoverError) Unwrap() error
type FailoverReason ¶
type FailoverReason string
FailoverReason describes why a provider failed.
const ( FailoverAuth FailoverReason = "auth" // 401/403 FailoverRateLimit FailoverReason = "rate_limit" // 429 FailoverBilling FailoverReason = "billing" // 402 FailoverTimeout FailoverReason = "timeout" // 408/504/deadline FailoverOverloaded FailoverReason = "overloaded" // 500/502/503/529 FailoverFormat FailoverReason = "format" // 400 FailoverUnknown FailoverReason = "unknown" // unknown error (treated as retriable) )
type FallbackCandidate ¶
FallbackCandidate pairs a provider/model label with its LLM client.
type FallbackChain ¶
type FallbackChain struct {
// contains filtered or unexported fields
}
FallbackChain implements the Client interface by trying multiple LLM providers in order. When the primary provider fails with a retriable error (429, 503, timeouts), the chain moves to the next candidate. Non-retriable errors (400 bad request, 401 auth) abort immediately.
When there is only one candidate, FallbackChain delegates directly without error classification to preserve exact current behavior.
func NewFallbackChain ¶
func NewFallbackChain(candidates []FallbackCandidate) *FallbackChain
NewFallbackChain creates a new fallback chain from the given candidates. At least one candidate is required.
func (*FallbackChain) Chat ¶
func (fc *FallbackChain) Chat(ctx context.Context, req *ChatRequest) (*ChatResponse, error)
Chat tries each candidate in order until one succeeds or all are exhausted.
func (*FallbackChain) ChatStream ¶
func (fc *FallbackChain) ChatStream(ctx context.Context, req *ChatRequest) (<-chan StreamDelta, error)
ChatStream tries each candidate in order for streaming requests.
func (*FallbackChain) ModelID ¶
func (fc *FallbackChain) ModelID() string
ModelID returns the primary candidate's model identifier.
type FallbackExhaustedError ¶
type FallbackExhaustedError struct {
Errors []*FailoverError
}
FallbackExhaustedError is returned when all candidates have been tried and failed.
func (*FallbackExhaustedError) Error ¶
func (e *FallbackExhaustedError) Error() string
type FunctionCall ¶
type FunctionCall struct {
Name string `json:"name"`
Arguments string `json:"arguments"` // JSON string
}
FunctionCall contains the function name and arguments for a tool call.
type FunctionSchema ¶
type FunctionSchema struct {
Name string `json:"name"`
Description string `json:"description"`
Parameters json.RawMessage `json:"parameters,omitempty"`
}
FunctionSchema describes a function's name, description, and parameters.
type StreamDelta ¶
type StreamDelta struct {
Content string `json:"content,omitempty"`
ToolCalls []ToolCall `json:"tool_calls,omitempty"`
FinishReason string `json:"finish_reason,omitempty"`
Done bool `json:"done,omitempty"`
Usage *UsageInfo `json:"usage,omitempty"`
}
StreamDelta represents a single chunk in a streaming response.
type ToolCall ¶
type ToolCall struct {
ID string `json:"id"`
Type string `json:"type"` // always "function"
Function FunctionCall `json:"function"`
}
ToolCall represents an LLM request to invoke a tool.
type ToolDefinition ¶
type ToolDefinition struct {
Type string `json:"type"` // always "function"
Function FunctionSchema `json:"function"`
}
ToolDefinition describes a tool available to the LLM.
type UsageInfo ¶
type UsageInfo struct {
InputTokens int `json:"input_tokens"`
OutputTokens int `json:"output_tokens"`
TotalTokens int `json:"total_tokens"`
}
UsageInfo contains token usage information.
Field naming aligns with OTel GenAI semantic conventions (gen_ai.usage.input_tokens / gen_ai.usage.output_tokens) so audit consumers can correlate Forge audit events with OTel traces without a translation table. See issue #87 / FWS-3.