Documentation
¶
Overview ¶
Package llm is Hermod's one interface to large language model providers.
Every AI feature (the AI nodes, the agent loop, error analysis) talks to a Provider and never to a vendor API. Adapters live in sub-packages (anthropic, openai, gemini) and the decorators in this package add the behaviour every call needs whatever the vendor: retries, concurrency limits, fallback and usage accounting.
Index ¶
- Variables
- func ErrRequiresModel(provider string) error
- func IsRetryable(err error) bool
- func PostJSON(ctx context.Context, client *http.Client, provider, url string, ...) (http.Header, error)
- func RetryAfter(h http.Header) time.Duration
- func RetryableStatus(status int) bool
- func WithScope(ctx context.Context, s Scope) context.Context
- type APIError
- type Budget
- type BudgetError
- type CallRecord
- type ChatRequest
- type ChatResponse
- type EmbedRequest
- type EmbedResponse
- type Embedder
- type Limit
- type Message
- type Middleware
- type Observer
- type Provider
- type RetryPolicy
- type Role
- type Scope
- type StopReason
- type ToolCall
- type ToolResult
- type ToolSpec
- type Usage
Constants ¶
This section is empty.
Variables ¶
var DefaultHTTPClient = &http.Client{Timeout: 5 * time.Minute}
DefaultHTTPClient is used by the HTTP adapters when given no client. The timeout is a backstop; callers bound each call with their context.
var DefaultRetryPolicy = RetryPolicy{MaxAttempts: 3, BaseDelay: 500 * time.Millisecond, MaxDelay: 20 * time.Second}
DefaultRetryPolicy is three attempts with jittered exponential backoff.
var ErrBudgetExceeded = errors.New("llm: token budget exceeded")
ErrBudgetExceeded is returned when a Budget refuses a call.
var ErrEmptyRequest = errors.New("llm: request has no messages")
ErrEmptyRequest is returned for a chat request with no messages.
var ErrUnsupported = errors.New("llm: not supported by this provider")
ErrUnsupported is returned for a capability a provider does not have.
Functions ¶
func ErrRequiresModel ¶
ErrRequiresModel is returned by adapters that have no default model.
func IsRetryable ¶
IsRetryable reports whether err is an APIError marked retryable.
func PostJSON ¶
func PostJSON(ctx context.Context, client *http.Client, provider, url string, headers map[string]string, body, out any) (http.Header, error)
PostJSON sends body as JSON to url and decodes the response into out. A non-2xx answer becomes an *APIError carrying a bounded excerpt of the body; network failures are retryable APIErrors. Context cancellation is returned as the context's error, not as an APIError.
func RetryAfter ¶
RetryAfter reads a Retry-After header given in seconds. HTTP dates and absent or malformed headers give zero.
func RetryableStatus ¶
RetryableStatus is the HTTP status rule every adapter shares.
Types ¶
type APIError ¶
type APIError struct {
Provider string
Status int
Message string
Retryable bool
RetryAfter time.Duration
}
APIError is a failed provider call. Retryable says whether the same request may succeed later (rate limits, overload, server errors, network errors).
type Budget ¶
type Budget interface {
// Allow returns an error (normally wrapping ErrBudgetExceeded) when no
// more calls may be made.
Allow(ctx context.Context) error
// Record adds a finished call's usage. The record names the provider and
// the model that answered, so a budget can price what it is told.
Record(ctx context.Context, rec CallRecord)
}
Budget gates calls on spend so far.
type BudgetError ¶
type BudgetError struct {
Limit Limit
VHost string
WorkflowID string
// Used and Max are tokens for the token limits and micro-units of the
// vhost's currency for the cost limits. Both are zero for the kill switch
// and for an unreadable budget.
Used int64
Max int64
// Cause is why the budget could not be read, for LimitUnavailable.
Cause error
}
BudgetError is a call refused by a spending limit. It matches ErrBudgetExceeded under errors.Is, so everything that already handles a spent budget (an AI node's error branch, the agent, the metrics) handles this one the same way.
func (*BudgetError) Error ¶
func (e *BudgetError) Error() string
func (*BudgetError) Unwrap ¶
func (e *BudgetError) Unwrap() error
Unwrap makes a BudgetError match ErrBudgetExceeded.
type CallRecord ¶
type CallRecord struct {
Provider string
Model string
Usage Usage
Latency time.Duration
StopReason StopReason
Err error
}
CallRecord describes one provider call, for metrics, audit and cost.
type ChatRequest ¶
type ChatRequest struct {
// Model is the provider's model id. Adapters that have a sensible
// default use it when this is empty; the others reject the request.
Model string
System string
Messages []Message
Tools []ToolSpec
// ResponseSchema, when set, asks for a JSON object matching this JSON
// Schema, using the provider's native structured output where it has one.
ResponseSchema map[string]any
// MaxTokens caps the response. Zero means the adapter's default.
MaxTokens int
// Temperature is sent only when set, and only to providers whose current
// models accept it.
Temperature *float64
}
ChatRequest is a provider-neutral chat completion request.
type ChatResponse ¶
type ChatResponse struct {
Text string
ToolCalls []ToolCall
StopReason StopReason
Usage Usage
// Model is the model that actually answered (it can differ from the one
// asked for when a fallback served the call).
Model string
Provider string
RequestID string
Latency time.Duration
}
ChatResponse is a provider-neutral chat completion response.
type EmbedRequest ¶
EmbedRequest asks for one vector per input.
type EmbedResponse ¶
EmbedResponse holds vectors in the order of EmbedRequest.Inputs.
type Embedder ¶
type Embedder interface {
Embed(ctx context.Context, req EmbedRequest) (EmbedResponse, error)
}
Embedder is a provider that can also produce embeddings.
type Limit ¶
type Limit string
Limit names what refused a call.
const ( // LimitDisabled is the vhost's AI kill switch. LimitDisabled Limit = "ai_disabled" // LimitVHostTokens and LimitVHostCost are the vhost's monthly budget. LimitVHostTokens Limit = "vhost_tokens" LimitVHostCost Limit = "vhost_cost" // LimitWorkflowTokens and LimitWorkflowCost are one workflow's monthly cap. LimitWorkflowTokens Limit = "workflow_tokens" LimitWorkflowCost Limit = "workflow_cost" // closed: a call that cannot be checked is not made. LimitUnavailable Limit = "budget_unavailable" )
type Message ¶
type Message struct {
Role Role
Text string
ToolCalls []ToolCall
ToolResults []ToolResult
}
Message is one turn. A user turn carries Text and/or ToolResults; an assistant turn carries Text and/or ToolCalls.
type Middleware ¶
Middleware decorates a Provider.
func WithBudget ¶
func WithBudget(b Budget) Middleware
WithBudget refuses calls once b says the budget is spent, and records the usage of every call that ran. Embeddings are gated and recorded the same way: they are paid for too.
The check comes before the call and the usage is only known after it, so a call that starts under the limit can finish over it: a budget is overshot by at most the calls already in flight when it ran out.
func WithConcurrencyLimit ¶
func WithConcurrencyLimit(n int) Middleware
WithConcurrencyLimit allows at most n calls in flight; others wait until a slot frees or their context ends.
func WithObserver ¶
func WithObserver(obs Observer) Middleware
WithObserver reports every call to obs. Placed inside WithRetry it sees each attempt; outside, each logical call.
func WithRetry ¶
func WithRetry(policy RetryPolicy) Middleware
WithRetry retries retryable failures with jittered exponential backoff, waiting at least as long as a provider's Retry-After asks.
type Observer ¶
type Observer func(context.Context, CallRecord)
Observer receives a CallRecord after every call. It must not block.
type Provider ¶
type Provider interface {
// Name is the provider kind, e.g. "anthropic", "openai", "gemini".
Name() string
Chat(ctx context.Context, req ChatRequest) (ChatResponse, error)
}
Provider is a chat model endpoint.
func Chain ¶
func Chain(p Provider, mws ...Middleware) Provider
Chain applies mws to p so that the first middleware is the outermost.
func Fallback ¶
Fallback tries providers in order. It moves to the next one when a provider fails in a way that is not the request's fault (a retryable error) or declines to answer (a refusal). A non-retryable error, such as a malformed request, is returned at once: the next provider would reject it too.
type RetryPolicy ¶
type RetryPolicy struct {
// MaxAttempts counts the first try. Values below 1 mean 1.
MaxAttempts int
BaseDelay time.Duration
MaxDelay time.Duration
}
RetryPolicy bounds WithRetry.
type Scope ¶
Scope says on whose behalf a call is made: the vhost whose workflow runs it and, when a workflow node makes it, that workflow. A Budget reads it from the call's context to decide whose spend the call counts against.
The engine sets it, never a message's data or metadata, so a payload cannot move its cost onto another vhost or workflow.
type StopReason ¶
type StopReason string
StopReason says why the model stopped.
const ( StopEnd StopReason = "end" StopToolUse StopReason = "tool_use" StopMaxTokens StopReason = "max_tokens" StopRefusal StopReason = "refusal" StopOther StopReason = "other" )
type ToolCall ¶
type ToolCall struct {
ID string
Name string
Input json.RawMessage
}
ToolCall is the model asking for a tool to be run.
type ToolResult ¶
ToolResult answers a ToolCall. Name is repeated because some providers (Gemini) match results to calls by name rather than by id.
Directories
¶
| Path | Synopsis |
|---|---|
|
Package anthropic is the llm adapter for Claude, through the official Anthropic Go SDK.
|
Package anthropic is the llm adapter for Claude, through the official Anthropic Go SDK. |
|
Package connect builds a ready-to-use llm.Provider from a connection description: the adapter for its kind, plus retries, a concurrency limit and an optional fallback chain.
|
Package connect builds a ready-to-use llm.Provider from a connection description: the adapter for its kind, plus retries, a concurrency limit and an optional fallback chain. |
|
Package gemini is the llm adapter for Google's Gemini API (generativelanguage.googleapis.com, v1beta generateContent).
|
Package gemini is the llm adapter for Google's Gemini API (generativelanguage.googleapis.com, v1beta generateContent). |
|
Package openai is the llm adapter for the OpenAI Chat Completions wire format.
|
Package openai is the llm adapter for the OpenAI Chat Completions wire format. |