llm

package
v0.16.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 1, 2026 License: Apache-2.0 Imports: 8 Imported by: 0

Documentation

Overview

Package llm provides canonical types for LLM chat interactions. These types are provider-agnostic; each provider translates to/from its native API format.

Index

Constants

View Source
const (
	RoleSystem    = "system"
	RoleUser      = "user"
	RoleAssistant = "assistant"
	RoleTool      = "tool"
)

Role constants for chat messages.

Variables

This section is empty.

Functions

This section is empty.

Types

type ChatMessage

type ChatMessage struct {
	Role       string     `json:"role"`
	Content    string     `json:"content,omitempty"`
	ToolCalls  []ToolCall `json:"tool_calls,omitempty"`
	ToolCallID string     `json:"tool_call_id,omitempty"`
	Name       string     `json:"name,omitempty"`
}

ChatMessage represents a single message in a chat conversation.

type ChatRequest

type ChatRequest struct {
	Model       string           `json:"model"`
	Messages    []ChatMessage    `json:"messages"`
	Tools       []ToolDefinition `json:"tools,omitempty"`
	Temperature *float64         `json:"temperature,omitempty"`
	MaxTokens   int              `json:"max_tokens,omitempty"`
	Stream      bool             `json:"stream,omitempty"`
}

ChatRequest is a provider-agnostic chat completion request.

type ChatResponse

type ChatResponse struct {
	ID           string      `json:"id"`
	Message      ChatMessage `json:"message"`
	Usage        UsageInfo   `json:"usage"`
	FinishReason string      `json:"finish_reason"`
}

ChatResponse is a provider-agnostic chat completion response.

type Client

type Client interface {
	// Chat sends a chat completion request and returns the response.
	Chat(ctx context.Context, req *ChatRequest) (*ChatResponse, error)
	// ChatStream sends a streaming chat request and returns a channel of deltas.
	ChatStream(ctx context.Context, req *ChatRequest) (<-chan StreamDelta, error)
	// ModelID returns the model identifier this client is configured for.
	ModelID() string
}

Client is the interface for interacting with an LLM provider.

type ClientConfig

type ClientConfig struct {
	APIKey      string
	BaseURL     string
	Model       string
	OrgID       string
	MaxRetries  int
	TimeoutSecs int

	// AuthScheme + AWSRegion control outbound authentication when
	// the operator points the client at AWS Bedrock (Anthropic
	// passthrough or OpenAI compatibility endpoint) or any other
	// SigV4-fronted gateway. Issue #202 Phase 2. Mirrors the
	// matching forge.yaml ModelRef fields.
	//
	// AuthScheme == "" preserves the pre-#202 behavior — the
	// Anthropic client sets `x-api-key: <APIKey>`, the OpenAI
	// client sets `Authorization: Bearer <APIKey>`. AuthScheme ==
	// "aws_sigv4" wraps the client's transport with the SigV4
	// signer and skips the native header logic; APIKey is ignored.
	AuthScheme string
	AWSRegion  string
}

ClientConfig holds configuration for creating an LLM client.

type CooldownTracker

type CooldownTracker struct {
	// contains filtered or unexported fields
}

CooldownTracker manages per-provider cooldown state with exponential backoff.

func NewCooldownTracker

func NewCooldownTracker() *CooldownTracker

NewCooldownTracker creates a new cooldown tracker.

func (*CooldownTracker) IsAvailable

func (ct *CooldownTracker) IsAvailable(provider string) bool

IsAvailable returns true if the provider is not currently in cooldown.

func (*CooldownTracker) MarkFailure

func (ct *CooldownTracker) MarkFailure(provider string, reason FailoverReason)

MarkFailure records a failure for the given provider.

func (*CooldownTracker) MarkSuccess

func (ct *CooldownTracker) MarkSuccess(provider string)

MarkSuccess resets all cooldown state for a provider.

type Embedder

type Embedder interface {
	// Embed produces embeddings for the given texts.
	Embed(ctx context.Context, req *EmbeddingRequest) (*EmbeddingResponse, error)
	// Dimensions returns the dimensionality of the embedding vectors.
	Dimensions() int
}

Embedder generates vector embeddings from text.

type EmbeddingRequest

type EmbeddingRequest struct {
	Texts []string // texts to embed
	Model string   // optional model override
}

EmbeddingRequest is a provider-agnostic request to generate embeddings.

type EmbeddingResponse

type EmbeddingResponse struct {
	Embeddings [][]float32
	Model      string
	Usage      UsageInfo
}

EmbeddingResponse is a provider-agnostic embedding response.

type FailoverError

type FailoverError struct {
	Reason   FailoverReason
	Provider string
	Model    string
	Status   int
	Wrapped  error
}

FailoverError wraps an LLM provider error with classification metadata.

func ClassifyError

func ClassifyError(err error, provider, model string) *FailoverError

ClassifyError wraps a raw provider error into a FailoverError with the appropriate reason. It extracts HTTP status codes from known provider error formats and falls back to message pattern matching.

func (*FailoverError) Error

func (e *FailoverError) Error() string

func (*FailoverError) IsRetriable

func (e *FailoverError) IsRetriable() bool

IsRetriable returns true if this error should trigger a fallback attempt. Auth and format errors are never retriable.

func (*FailoverError) Unwrap

func (e *FailoverError) Unwrap() error

type FailoverReason

type FailoverReason string

FailoverReason describes why a provider failed.

const (
	FailoverAuth       FailoverReason = "auth"       // 401/403
	FailoverRateLimit  FailoverReason = "rate_limit" // 429
	FailoverBilling    FailoverReason = "billing"    // 402
	FailoverTimeout    FailoverReason = "timeout"    // 408/504/deadline
	FailoverOverloaded FailoverReason = "overloaded" // 500/502/503/529
	FailoverFormat     FailoverReason = "format"     // 400
	FailoverUnknown    FailoverReason = "unknown"    // unknown error (treated as retriable)
)

type FallbackCandidate

type FallbackCandidate struct {
	Provider string
	Model    string
	Client   Client
}

FallbackCandidate pairs a provider/model label with its LLM client.

type FallbackChain

type FallbackChain struct {
	// contains filtered or unexported fields
}

FallbackChain implements the Client interface by trying multiple LLM providers in order. When the primary provider fails with a retriable error (429, 503, timeouts), the chain moves to the next candidate. Non-retriable errors (400 bad request, 401 auth) abort immediately.

When there is only one candidate, FallbackChain delegates directly without error classification to preserve exact current behavior.

func NewFallbackChain

func NewFallbackChain(candidates []FallbackCandidate) *FallbackChain

NewFallbackChain creates a new fallback chain from the given candidates. At least one candidate is required.

func (*FallbackChain) Chat

func (fc *FallbackChain) Chat(ctx context.Context, req *ChatRequest) (*ChatResponse, error)

Chat tries each candidate in order until one succeeds or all are exhausted.

func (*FallbackChain) ChatStream

func (fc *FallbackChain) ChatStream(ctx context.Context, req *ChatRequest) (<-chan StreamDelta, error)

ChatStream tries each candidate in order for streaming requests.

func (*FallbackChain) ModelID

func (fc *FallbackChain) ModelID() string

ModelID returns the primary candidate's model identifier.

type FallbackExhaustedError

type FallbackExhaustedError struct {
	Errors []*FailoverError
}

FallbackExhaustedError is returned when all candidates have been tried and failed.

func (*FallbackExhaustedError) Error

func (e *FallbackExhaustedError) Error() string

type FunctionCall

type FunctionCall struct {
	Name      string `json:"name"`
	Arguments string `json:"arguments"` // JSON string
}

FunctionCall contains the function name and arguments for a tool call.

type FunctionSchema

type FunctionSchema struct {
	Name        string          `json:"name"`
	Description string          `json:"description"`
	Parameters  json.RawMessage `json:"parameters,omitempty"`
}

FunctionSchema describes a function's name, description, and parameters.

type StreamDelta

type StreamDelta struct {
	Content      string     `json:"content,omitempty"`
	ToolCalls    []ToolCall `json:"tool_calls,omitempty"`
	FinishReason string     `json:"finish_reason,omitempty"`
	Done         bool       `json:"done,omitempty"`
	Usage        *UsageInfo `json:"usage,omitempty"`
}

StreamDelta represents a single chunk in a streaming response.

type ToolCall

type ToolCall struct {
	ID       string       `json:"id"`
	Type     string       `json:"type"` // always "function"
	Function FunctionCall `json:"function"`
}

ToolCall represents an LLM request to invoke a tool.

type ToolDefinition

type ToolDefinition struct {
	Type     string         `json:"type"` // always "function"
	Function FunctionSchema `json:"function"`
}

ToolDefinition describes a tool available to the LLM.

type UsageInfo

type UsageInfo struct {
	InputTokens  int `json:"input_tokens"`
	OutputTokens int `json:"output_tokens"`
	TotalTokens  int `json:"total_tokens"`
}

UsageInfo contains token usage information.

Field naming aligns with OTel GenAI semantic conventions (gen_ai.usage.input_tokens / gen_ai.usage.output_tokens) so audit consumers can correlate Forge audit events with OTel traces without a translation table. See issue #87 / FWS-3.

Directories

Path Synopsis
Package providers implements LLM client providers for various APIs.
Package providers implements LLM client providers for various APIs.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL