llm

package
v1.18.3 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 13, 2026 License: MIT Imports: 11 Imported by: 0

Documentation

Overview

Package llm is Hermod's one interface to large language model providers.

Every AI feature (the AI nodes, the agent loop, error analysis) talks to a Provider and never to a vendor API. Adapters live in sub-packages (anthropic, openai, gemini) and the decorators in this package add the behaviour every call needs whatever the vendor: retries, concurrency limits, fallback and usage accounting.

Index

Constants

This section is empty.

Variables

View Source
var DefaultHTTPClient = &http.Client{Timeout: 5 * time.Minute}

DefaultHTTPClient is used by the HTTP adapters when given no client. The timeout is a backstop; callers bound each call with their context.

View Source
var DefaultRetryPolicy = RetryPolicy{MaxAttempts: 3, BaseDelay: 500 * time.Millisecond, MaxDelay: 20 * time.Second}

DefaultRetryPolicy is three attempts with jittered exponential backoff.

View Source
var ErrBudgetExceeded = errors.New("llm: token budget exceeded")

ErrBudgetExceeded is returned when a Budget refuses a call.

View Source
var ErrEmptyRequest = errors.New("llm: request has no messages")

ErrEmptyRequest is returned for a chat request with no messages.

View Source
var ErrUnsupported = errors.New("llm: not supported by this provider")

ErrUnsupported is returned for a capability a provider does not have.

Functions

func ErrRequiresModel

func ErrRequiresModel(provider string) error

ErrRequiresModel is returned by adapters that have no default model.

func IsRetryable

func IsRetryable(err error) bool

IsRetryable reports whether err is an APIError marked retryable.

func PostJSON

func PostJSON(ctx context.Context, client *http.Client, provider, url string, headers map[string]string, body, out any) (http.Header, error)

PostJSON sends body as JSON to url and decodes the response into out. A non-2xx answer becomes an *APIError carrying a bounded excerpt of the body; network failures are retryable APIErrors. Context cancellation is returned as the context's error, not as an APIError.

func RetryAfter

func RetryAfter(h http.Header) time.Duration

RetryAfter reads a Retry-After header given in seconds. HTTP dates and absent or malformed headers give zero.

func RetryableStatus

func RetryableStatus(status int) bool

RetryableStatus is the HTTP status rule every adapter shares.

func WithScope

func WithScope(ctx context.Context, s Scope) context.Context

WithScope returns ctx carrying s.

Types

type APIError

type APIError struct {
	Provider   string
	Status     int
	Message    string
	Retryable  bool
	RetryAfter time.Duration
}

APIError is a failed provider call. Retryable says whether the same request may succeed later (rate limits, overload, server errors, network errors).

func (*APIError) Error

func (e *APIError) Error() string

type Budget

type Budget interface {
	// Allow returns an error (normally wrapping ErrBudgetExceeded) when no
	// more calls may be made.
	Allow(ctx context.Context) error
	// Record adds a finished call's usage. The record names the provider and
	// the model that answered, so a budget can price what it is told.
	Record(ctx context.Context, rec CallRecord)
}

Budget gates calls on spend so far.

type BudgetError

type BudgetError struct {
	Limit      Limit
	VHost      string
	WorkflowID string
	// Used and Max are tokens for the token limits and micro-units of the
	// vhost's currency for the cost limits. Both are zero for the kill switch
	// and for an unreadable budget.
	Used int64
	Max  int64
	// Cause is why the budget could not be read, for LimitUnavailable.
	Cause error
}

BudgetError is a call refused by a spending limit. It matches ErrBudgetExceeded under errors.Is, so everything that already handles a spent budget (an AI node's error branch, the agent, the metrics) handles this one the same way.

func (*BudgetError) Error

func (e *BudgetError) Error() string

func (*BudgetError) Unwrap

func (e *BudgetError) Unwrap() error

Unwrap makes a BudgetError match ErrBudgetExceeded.

type CallRecord

type CallRecord struct {
	Provider   string
	Model      string
	Usage      Usage
	Latency    time.Duration
	StopReason StopReason
	Err        error
}

CallRecord describes one provider call, for metrics, audit and cost.

type ChatRequest

type ChatRequest struct {
	// Model is the provider's model id. Adapters that have a sensible
	// default use it when this is empty; the others reject the request.
	Model    string
	System   string
	Messages []Message
	Tools    []ToolSpec
	// ResponseSchema, when set, asks for a JSON object matching this JSON
	// Schema, using the provider's native structured output where it has one.
	ResponseSchema map[string]any
	// MaxTokens caps the response. Zero means the adapter's default.
	MaxTokens int
	// Temperature is sent only when set, and only to providers whose current
	// models accept it.
	Temperature *float64
}

ChatRequest is a provider-neutral chat completion request.

type ChatResponse

type ChatResponse struct {
	Text       string
	ToolCalls  []ToolCall
	StopReason StopReason
	Usage      Usage
	// Model is the model that actually answered (it can differ from the one
	// asked for when a fallback served the call).
	Model     string
	Provider  string
	RequestID string
	Latency   time.Duration
}

ChatResponse is a provider-neutral chat completion response.

type EmbedRequest

type EmbedRequest struct {
	Model  string
	Inputs []string
}

EmbedRequest asks for one vector per input.

type EmbedResponse

type EmbedResponse struct {
	Vectors [][]float32
	Usage   Usage
	Model   string
}

EmbedResponse holds vectors in the order of EmbedRequest.Inputs.

type Embedder

type Embedder interface {
	Embed(ctx context.Context, req EmbedRequest) (EmbedResponse, error)
}

Embedder is a provider that can also produce embeddings.

type Limit

type Limit string

Limit names what refused a call.

const (
	// LimitDisabled is the vhost's AI kill switch.
	LimitDisabled Limit = "ai_disabled"
	// LimitVHostTokens and LimitVHostCost are the vhost's monthly budget.
	LimitVHostTokens Limit = "vhost_tokens"
	LimitVHostCost   Limit = "vhost_cost"
	// LimitWorkflowTokens and LimitWorkflowCost are one workflow's monthly cap.
	LimitWorkflowTokens Limit = "workflow_tokens"
	LimitWorkflowCost   Limit = "workflow_cost"
	// LimitUnavailable means the budget could not be read. Budgets fail
	// closed: a call that cannot be checked is not made.
	LimitUnavailable Limit = "budget_unavailable"
)

type Message

type Message struct {
	Role        Role
	Text        string
	ToolCalls   []ToolCall
	ToolResults []ToolResult
}

Message is one turn. A user turn carries Text and/or ToolResults; an assistant turn carries Text and/or ToolCalls.

type Middleware

type Middleware func(Provider) Provider

Middleware decorates a Provider.

func WithBudget

func WithBudget(b Budget) Middleware

WithBudget refuses calls once b says the budget is spent, and records the usage of every call that ran. Embeddings are gated and recorded the same way: they are paid for too.

The check comes before the call and the usage is only known after it, so a call that starts under the limit can finish over it: a budget is overshot by at most the calls already in flight when it ran out.

func WithConcurrencyLimit

func WithConcurrencyLimit(n int) Middleware

WithConcurrencyLimit allows at most n calls in flight; others wait until a slot frees or their context ends.

func WithObserver

func WithObserver(obs Observer) Middleware

WithObserver reports every call to obs. Placed inside WithRetry it sees each attempt; outside, each logical call.

func WithRetry

func WithRetry(policy RetryPolicy) Middleware

WithRetry retries retryable failures with jittered exponential backoff, waiting at least as long as a provider's Retry-After asks.

type Observer

type Observer func(context.Context, CallRecord)

Observer receives a CallRecord after every call. It must not block.

type Provider

type Provider interface {
	// Name is the provider kind, e.g. "anthropic", "openai", "gemini".
	Name() string
	Chat(ctx context.Context, req ChatRequest) (ChatResponse, error)
}

Provider is a chat model endpoint.

func Chain

func Chain(p Provider, mws ...Middleware) Provider

Chain applies mws to p so that the first middleware is the outermost.

func Fallback

func Fallback(primary Provider, others ...Provider) Provider

Fallback tries providers in order. It moves to the next one when a provider fails in a way that is not the request's fault (a retryable error) or declines to answer (a refusal). A non-retryable error, such as a malformed request, is returned at once: the next provider would reject it too.

type RetryPolicy

type RetryPolicy struct {
	// MaxAttempts counts the first try. Values below 1 mean 1.
	MaxAttempts int
	BaseDelay   time.Duration
	MaxDelay    time.Duration
}

RetryPolicy bounds WithRetry.

type Role

type Role string

Role is who wrote a message in a conversation.

const (
	RoleUser      Role = "user"
	RoleAssistant Role = "assistant"
)

type Scope

type Scope struct {
	VHost      string
	WorkflowID string
}

Scope says on whose behalf a call is made: the vhost whose workflow runs it and, when a workflow node makes it, that workflow. A Budget reads it from the call's context to decide whose spend the call counts against.

The engine sets it, never a message's data or metadata, so a payload cannot move its cost onto another vhost or workflow.

func ScopeFrom

func ScopeFrom(ctx context.Context) Scope

ScopeFrom returns the scope ctx carries, or the zero Scope.

type StopReason

type StopReason string

StopReason says why the model stopped.

const (
	StopEnd       StopReason = "end"
	StopToolUse   StopReason = "tool_use"
	StopMaxTokens StopReason = "max_tokens"
	StopRefusal   StopReason = "refusal"
	StopOther     StopReason = "other"
)

type ToolCall

type ToolCall struct {
	ID    string
	Name  string
	Input json.RawMessage
}

ToolCall is the model asking for a tool to be run.

type ToolResult

type ToolResult struct {
	CallID  string
	Name    string
	Content string
	IsError bool
}

ToolResult answers a ToolCall. Name is repeated because some providers (Gemini) match results to calls by name rather than by id.

type ToolSpec

type ToolSpec struct {
	Name        string
	Description string
	Schema      map[string]any
}

ToolSpec describes a tool the model may call. Schema is a JSON Schema object for the tool's input.

type Usage

type Usage struct {
	InputTokens  int64
	OutputTokens int64
}

Usage is what a call consumed.

func (Usage) Add

func (u Usage) Add(o Usage) Usage

Add returns the sum of two usages.

Directories

Path Synopsis
Package anthropic is the llm adapter for Claude, through the official Anthropic Go SDK.
Package anthropic is the llm adapter for Claude, through the official Anthropic Go SDK.
Package connect builds a ready-to-use llm.Provider from a connection description: the adapter for its kind, plus retries, a concurrency limit and an optional fallback chain.
Package connect builds a ready-to-use llm.Provider from a connection description: the adapter for its kind, plus retries, a concurrency limit and an optional fallback chain.
Package gemini is the llm adapter for Google's Gemini API (generativelanguage.googleapis.com, v1beta generateContent).
Package gemini is the llm adapter for Google's Gemini API (generativelanguage.googleapis.com, v1beta generateContent).
Package openai is the llm adapter for the OpenAI Chat Completions wire format.
Package openai is the llm adapter for the OpenAI Chat Completions wire format.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL