Documentation
¶
Overview ¶
Package llm provides canonical types for LLM chat interactions. These types are provider-agnostic; each provider translates to/from its native API format.
Index ¶
- Constants
- type ChatMessage
- type ChatRequest
- type ChatResponse
- type Client
- type ClientConfig
- type CooldownTracker
- type Embedder
- type EmbeddingRequest
- type EmbeddingResponse
- type FailoverError
- type FailoverReason
- type FallbackCandidate
- type FallbackChain
- type FallbackExhaustedError
- type FunctionCall
- type FunctionSchema
- type StreamDelta
- type ToolCall
- type ToolDefinition
- type UsageInfo
Constants ¶
const ( RoleSystem = "system" RoleUser = "user" RoleAssistant = "assistant" RoleTool = "tool" )
Role constants for chat messages.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type ChatMessage ¶
type ChatMessage struct {
Role string `json:"role"`
Content string `json:"content,omitempty"`
ToolCalls []ToolCall `json:"tool_calls,omitempty"`
ToolCallID string `json:"tool_call_id,omitempty"`
Name string `json:"name,omitempty"`
}
ChatMessage represents a single message in a chat conversation.
type ChatRequest ¶
type ChatRequest struct {
Model string `json:"model"`
Messages []ChatMessage `json:"messages"`
Tools []ToolDefinition `json:"tools,omitempty"`
Temperature *float64 `json:"temperature,omitempty"`
MaxTokens int `json:"max_tokens,omitempty"`
Stream bool `json:"stream,omitempty"`
}
ChatRequest is a provider-agnostic chat completion request.
type ChatResponse ¶
type ChatResponse struct {
ID string `json:"id"`
Message ChatMessage `json:"message"`
Usage UsageInfo `json:"usage"`
FinishReason string `json:"finish_reason"`
}
ChatResponse is a provider-agnostic chat completion response.
type Client ¶
type Client interface {
// Chat sends a chat completion request and returns the response.
Chat(ctx context.Context, req *ChatRequest) (*ChatResponse, error)
// ChatStream sends a streaming chat request and returns a channel of deltas.
ChatStream(ctx context.Context, req *ChatRequest) (<-chan StreamDelta, error)
// ModelID returns the model identifier this client is configured for.
ModelID() string
}
Client is the interface for interacting with an LLM provider.
type ClientConfig ¶
type ClientConfig struct {
APIKey string
BaseURL string
Model string
OrgID string
MaxRetries int
TimeoutSecs int
// AuthScheme + AWSRegion control outbound authentication when
// the operator points the client at AWS Bedrock (Anthropic
// passthrough or OpenAI compatibility endpoint) or any other
// SigV4-fronted gateway. Issue #202 Phase 2. Mirrors the
// matching forge.yaml ModelRef fields.
//
// AuthScheme == "" preserves the pre-#202 behavior — the
// Anthropic client sets `x-api-key: <APIKey>`, the OpenAI
// client sets `Authorization: Bearer <APIKey>`. AuthScheme ==
// "aws_sigv4" wraps the client's transport with the SigV4
// signer and skips the native header logic; APIKey is ignored.
AuthScheme string
AWSRegion string
// PromptCaching opts the provider client into injecting the
// provider's prompt-cache primitives on every request:
//
// - anthropic: a cache_control ephemeral breakpoint on the last
// tool definition and on the system prompt block, caching the
// stable tools+system prefix across turns. Also honored by
// Anthropic-on-Bedrock gateways (aws_sigv4), which speak the
// same wire format.
// - openai: a stable prompt_cache_key derived from
// (model, system, tool names), pinning cache routing for the
// session. OpenAI prefix caching itself is automatic ≥1024
// tokens; the key improves hit locality.
//
// Off by default — wire formats stay byte-identical to the
// pre-compression contract unless the operator opts in
// (compression.cache_hints / compression.enabled in forge.yaml).
PromptCaching bool
}
ClientConfig holds configuration for creating an LLM client.
type CooldownTracker ¶
type CooldownTracker struct {
// contains filtered or unexported fields
}
CooldownTracker manages per-provider cooldown state with exponential backoff.
func NewCooldownTracker ¶
func NewCooldownTracker() *CooldownTracker
NewCooldownTracker creates a new cooldown tracker.
func (*CooldownTracker) IsAvailable ¶
func (ct *CooldownTracker) IsAvailable(provider string) bool
IsAvailable returns true if the provider is not currently in cooldown.
func (*CooldownTracker) MarkFailure ¶
func (ct *CooldownTracker) MarkFailure(provider string, reason FailoverReason)
MarkFailure records a failure for the given provider.
func (*CooldownTracker) MarkSuccess ¶
func (ct *CooldownTracker) MarkSuccess(provider string)
MarkSuccess resets all cooldown state for a provider.
type Embedder ¶
type Embedder interface {
// Embed produces embeddings for the given texts.
Embed(ctx context.Context, req *EmbeddingRequest) (*EmbeddingResponse, error)
// Dimensions returns the dimensionality of the embedding vectors.
Dimensions() int
}
Embedder generates vector embeddings from text.
type EmbeddingRequest ¶
type EmbeddingRequest struct {
Texts []string // texts to embed
Model string // optional model override
}
EmbeddingRequest is a provider-agnostic request to generate embeddings.
type EmbeddingResponse ¶
EmbeddingResponse is a provider-agnostic embedding response.
type FailoverError ¶
type FailoverError struct {
Reason FailoverReason
Provider string
Model string
Status int
Wrapped error
}
FailoverError wraps an LLM provider error with classification metadata.
func ClassifyError ¶
func ClassifyError(err error, provider, model string) *FailoverError
ClassifyError wraps a raw provider error into a FailoverError with the appropriate reason. It extracts HTTP status codes from known provider error formats and falls back to message pattern matching.
func (*FailoverError) Error ¶
func (e *FailoverError) Error() string
func (*FailoverError) IsRetriable ¶
func (e *FailoverError) IsRetriable() bool
IsRetriable returns true if this error should trigger a fallback attempt. Auth and format errors are never retriable.
func (*FailoverError) Unwrap ¶
func (e *FailoverError) Unwrap() error
type FailoverReason ¶
type FailoverReason string
FailoverReason describes why a provider failed.
const ( FailoverAuth FailoverReason = "auth" // 401/403 FailoverRateLimit FailoverReason = "rate_limit" // 429 FailoverBilling FailoverReason = "billing" // 402 FailoverTimeout FailoverReason = "timeout" // 408/504/deadline FailoverOverloaded FailoverReason = "overloaded" // 500/502/503/529 FailoverFormat FailoverReason = "format" // 400 FailoverUnknown FailoverReason = "unknown" // unknown error (treated as retriable) )
type FallbackCandidate ¶
FallbackCandidate pairs a provider/model label with its LLM client.
type FallbackChain ¶
type FallbackChain struct {
// contains filtered or unexported fields
}
FallbackChain implements the Client interface by trying multiple LLM providers in order. When the primary provider fails with a retriable error (429, 503, timeouts), the chain moves to the next candidate. Non-retriable errors (400 bad request, 401 auth) abort immediately.
When there is only one candidate, FallbackChain delegates directly without error classification to preserve exact current behavior.
func NewFallbackChain ¶
func NewFallbackChain(candidates []FallbackCandidate) *FallbackChain
NewFallbackChain creates a new fallback chain from the given candidates. At least one candidate is required.
func (*FallbackChain) Chat ¶
func (fc *FallbackChain) Chat(ctx context.Context, req *ChatRequest) (*ChatResponse, error)
Chat tries each candidate in order until one succeeds or all are exhausted.
func (*FallbackChain) ChatStream ¶
func (fc *FallbackChain) ChatStream(ctx context.Context, req *ChatRequest) (<-chan StreamDelta, error)
ChatStream tries each candidate in order for streaming requests.
func (*FallbackChain) ModelID ¶
func (fc *FallbackChain) ModelID() string
ModelID returns the primary candidate's model identifier.
type FallbackExhaustedError ¶
type FallbackExhaustedError struct {
Errors []*FailoverError
}
FallbackExhaustedError is returned when all candidates have been tried and failed.
func (*FallbackExhaustedError) Error ¶
func (e *FallbackExhaustedError) Error() string
type FunctionCall ¶
type FunctionCall struct {
Name string `json:"name"`
Arguments string `json:"arguments"` // JSON string
}
FunctionCall contains the function name and arguments for a tool call.
type FunctionSchema ¶
type FunctionSchema struct {
Name string `json:"name"`
Description string `json:"description"`
Parameters json.RawMessage `json:"parameters,omitempty"`
}
FunctionSchema describes a function's name, description, and parameters.
type StreamDelta ¶
type StreamDelta struct {
Content string `json:"content,omitempty"`
ToolCalls []ToolCall `json:"tool_calls,omitempty"`
FinishReason string `json:"finish_reason,omitempty"`
Done bool `json:"done,omitempty"`
Usage *UsageInfo `json:"usage,omitempty"`
}
StreamDelta represents a single chunk in a streaming response.
type ToolCall ¶
type ToolCall struct {
ID string `json:"id"`
Type string `json:"type"` // always "function"
Function FunctionCall `json:"function"`
}
ToolCall represents an LLM request to invoke a tool.
type ToolDefinition ¶
type ToolDefinition struct {
Type string `json:"type"` // always "function"
Function FunctionSchema `json:"function"`
}
ToolDefinition describes a tool available to the LLM.
type UsageInfo ¶
type UsageInfo struct {
InputTokens int `json:"input_tokens"`
OutputTokens int `json:"output_tokens"`
TotalTokens int `json:"total_tokens"`
}
UsageInfo contains token usage information.
Field naming aligns with OTel GenAI semantic conventions (gen_ai.usage.input_tokens / gen_ai.usage.output_tokens) so audit consumers can correlate Forge audit events with OTel traces without a translation table. See issue #87 / FWS-3.