llm

package
v0.16.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 5, 2026 License: MIT Imports: 9 Imported by: 0

Documentation

Overview

Package llm defines backend-agnostic abstractions for single-shot, structured-output language models.

The interaction model is deliberately narrow: there is no conversation state. A caller describes the desired output as a Go type T, fixes the model's system instruction at construction, supplies an Input (the payload to act on), and receives a Response[T] whose Output is a decoded value of T. This single turn maps cleanly onto the structured-output / JSON-mode features of the major providers (OpenAI, Anthropic Claude, Google Gemini, ...), each of which is expected to land later as a concrete Model implementation behind the interface defined here.

The central abstraction is the generic Model[T] interface, paired with Response[T]. The JSON Schema describing T is derived from the Go type by SchemaFor, so callers rarely write schema documents by hand. The system instruction is fixed when the model is built, so callers supply only an input. Alongside the decoded Output, Response[T] carries the metadata that backends report in common — token Usage, a normalized FinishReason, and the resolved Model identifier.

This package is a leaf: it depends on nothing beyond the standard library, so concrete backends (which need provider SDKs or HTTP clients) belong in their own subpackages or modules that import this one, keeping the abstraction dependency-free.

Error handling

Provider HTTP failures carry a shared, errors.Is-able taxonomy (#852) so callers can build retry loops and circuit breakers without string-matching provider text: 

ErrRateLimited   429; retry after backing off (APIError.RetryAfter when sent)
ErrUnavailable   5xx (incl. Anthropic 529), 408, 409; transient, retry
ErrUnauthorized  401/403; credential problem, do not retry
ErrBadRequest    other 4xx; deterministic request problem, do not retry

errors.As(&APIError{}) recovers the provider name, status code, Retry-After hint, and a body excerpt bounded by ErrBodyLimit. Separately, ErrTruncated marks a Generate whose structured output was cut by the max-token cap — raise the cap rather than retrying. Refusals and safety blocks remain provider-typed (anthropic.ErrRefusal, openai.ErrRefusal, gemini.ErrBlocked) because their semantics differ per provider.

Auth matrix (#854)

The supported provider x endpoint x credential combinations, each pinned by a request-shape test in the provider package (URL path + auth header), so a broken combination fails CI instead of a user: 

Provider   Endpoint          Credential            Recipe
---------  ----------------  --------------------  ------------------------------------------
openai     native            API key (Bearer)      NewClient(key, model)
openai     Azure OpenAI      Entra (MI/secret)     NewClient("", model, WithBaseURL(resource),
                                                     WithHTTPClient(llmauth.NewAzure*HTTPClient(...)))
openai     Azure OpenAI      static api-key        NewClient(key, model, WithBaseURL(resource+"/openai"),
                                                     WithAPIKeyHeader("api-key"))
anthropic  native            API key (x-api-key)   NewClient(key, model)
anthropic  Vertex AI         ADC / service acct    NewClient("", model, WithVertexAI(project, location),
                                                     WithHTTPClient(llmauth.NewGoogle*HTTPClient(...)))
gemini     native            API key (x-goog-...)  NewClient(key, model)
gemini     Vertex AI         ADC / service acct    NewClient("", model, WithVertexAI(project, location),
                                                     WithHTTPClient(llmauth.NewGoogle*HTTPClient(...)))

llmauth (server/llmauth) builds the credentialed http.Client for the token rows: Azure credentials sit behind an explicit expiry-aware singleflight cache, and Google token sources behind oauth2.ReuseTokenSource, so no combination pays a token round-trip per request.

Index

Constants

View Source 
const ErrBodyLimit = 2048

ErrBodyLimit bounds how many bytes of a provider error body are retained on an APIError (and therefore how much can ever reach a log line). Provider backends read error bodies through a reader capped at this size.

Variables

View Source 
var (
	// ErrRateLimited marks HTTP 429. Retryable after a backoff; when the
	// provider sent Retry-After it is parsed into APIError.RetryAfter.
	ErrRateLimited = errors.New("llm: rate limited")
	// ErrUnavailable marks transient provider-side failures — HTTP 5xx
	// (including Anthropic's 529 overloaded), plus 408/409. Retryable.
	ErrUnavailable = errors.New("llm: provider unavailable")
	// ErrUnauthorized marks HTTP 401/403 — a credential or permission
	// problem. Not retryable.
	ErrUnauthorized = errors.New("llm: unauthorized")
	// ErrBadRequest marks the remaining 4xx — a deterministic problem with
	// the request (schema, model id, payload shape). Not retryable.
	ErrBadRequest = errors.New("llm: bad request")
	// ErrTruncated marks a Generate whose output was cut by the max-token
	// cap before the structured JSON completed (FinishLength + decode
	// failure). Raise the cap (WithMaxTokens) instead of retrying.
	ErrTruncated = errors.New("llm: output truncated by max tokens")
)

Typed failure taxonomy (#852). Every provider backend maps a non-200 HTTP response through ClassifyHTTP so callers can branch on class instead of string-matching provider error text:

  • errors.Is(err, ErrRateLimited) → back off (honour APIError.RetryAfter)
  • errors.Is(err, ErrUnavailable) → transient; retry with backoff
  • errors.Is(err, ErrUnauthorized) → credential/config problem; do NOT retry
  • errors.Is(err, ErrBadRequest) → deterministic request problem; do NOT retry

errors.As(&APIError{}) recovers the provider, status code, Retry-After hint, and a bounded excerpt of the response body.

ErrTruncated is a separate, response-level condition: the model's output was cut by the max-token cap and the structured JSON could not be decoded. Retrying without raising the cap will fail the same way.

Functions

func ClassifyHTTP 

func ClassifyHTTP(provider string, status int, header http.Header, body []byte) error

ClassifyHTTP builds the APIError for a non-200 provider response. body may be arbitrarily large; it is truncated to ErrBodyLimit. The Retry-After header is honoured in both delta-seconds and HTTP-date forms (a past date yields zero).

func DecodeFailure 

func DecodeFailure(provider string, finish FinishReason, decodeErr error) error

DecodeFailure is the shared Generate-side diagnosis for a JSON decode failure (#852): when generation stopped at the token cap the decode failure is a symptom, not the cause, so it is wrapped as ErrTruncated with the remedy in the message. Any other decode failure stays a plain decode error. Provider backends call this from their Generate implementations.

Types

type APIError 

type APIError struct {
	// Provider is the backend that produced the failure: "anthropic",
	// "gemini", or "openai".
	Provider string
	// StatusCode is the HTTP status of the response.
	StatusCode int
	// RetryAfter is the parsed Retry-After hint for rate limits; zero when
	// the header was absent or unparseable.
	RetryAfter time.Duration
	// Body is the response body, truncated to ErrBodyLimit bytes.
	Body string
	// contains filtered or unexported fields
}

APIError is the structured form of a non-200 provider response. Unwrap returns the classification sentinel, so errors.Is picks the class and errors.As recovers the detail.

func (*APIError) Error 

func (e *APIError) Error() string

func (*APIError) Unwrap 

func (e *APIError) Unwrap() error

type FinishReason 

type FinishReason string

FinishReason is the normalized reason a model stopped generating, mapped from each provider's own vocabulary onto a small backend-agnostic set.

The mapping from the major providers is: 

FinishStop          OpenAI "stop"            Gemini "STOP"
FinishLength        OpenAI "length"          Gemini "MAX_TOKENS"
FinishContentFilter OpenAI "content_filter"  Gemini "SAFETY", "RECITATION"
FinishOther         anything else (e.g. tool/function calls) or unset

const (
	// FinishStop means the model emitted a natural, complete stop.
	FinishStop FinishReason = "stop"
	// FinishLength means generation was truncated by a token limit; the Output
	// may be incomplete and fail to satisfy the schema.
	FinishLength FinishReason = "length"
	// FinishContentFilter means generation was cut short by a safety or content
	// policy filter.
	FinishContentFilter FinishReason = "content_filter"
	// FinishOther covers any other or unspecified reason.
	FinishOther FinishReason = "other"
)

type Model 

type Model[T any] interface {
	Generate(ctx context.Context, input string) (Response[T], error)
}

Model is a single-shot, structured-output language model whose output conforms to the Go type T. Given an input, Generate returns a Response whose Output is a fully decoded value of T. A Model keeps no state between calls: every Generate is independent, with no conversation history.

T is the structured-output schema: the JSON Schema sent to the provider is derived from T (see SchemaFor), and the provider's JSON answer is decoded back into a T. The system instruction is fixed when the Model is built, so Generate takes only the input to act on. Instruction and schema are the two halves of one task definition and are confirmed together, at construction: binding T to the interface forces the schema there, and the instruction lives with the model rather than the input (Go methods cannot take type parameters).

Concrete implementations wrap a specific backend (OpenAI, Anthropic Claude, Google Gemini, a local model, ...) — typically a thin generic adapter over a shared, non-generic client that captures the instruction. Implementations should be safe for concurrent use by multiple goroutines and must honor ctx cancellation.

type ModelFunc 

type ModelFunc[T any] func(ctx context.Context, input string) (Response[T], error)

ModelFunc adapts an ordinary function to the Model interface; capture the instruction in the closure. Handy for tests and lightweight, stateless backends.

func (ModelFunc[T]) Generate 

func (f ModelFunc[T]) Generate(ctx context.Context, input string) (Response[T], error)

Generate calls f(ctx, input).

type Response 

type Response[T any] struct {
	Output       T
	Usage        Usage
	FinishReason FinishReason
	Model        string
}

Response is a Model's answer to an input. Output is the decoded result; the remaining fields carry the metadata that both OpenAI and Gemini report alongside the content.

Output is the structured result, already decoded into T. Usage reports token consumption. FinishReason is the normalized reason generation stopped. Model is the provider's resolved model/version identifier (OpenAI "model", Gemini "modelVersion").

type Schema 

type Schema struct {
	Name        string
	Description string
	Definition  json.RawMessage
	Strict      bool
}

Schema is a backend-agnostic description of the structure a Model's output must conform to. It is expressed as a JSON Schema document — the interchange format understood by every supported backend (OpenAI structured outputs, Gemini response schemas, Claude tool input schemas, ...). The document is a canonical, strict-leaning form (objects pin additionalProperties to false and list every non-optional field in required); each backend translates it into the subset its provider accepts.

Name identifies the schema; some backends require a non-empty name for a structured-output request. Description is an optional human-readable summary. Definition is the JSON Schema document itself. When Strict is true the output must validate against Definition exactly; backends that always enforce their response schema may treat the flag as implied.

func SchemaFor 

func SchemaFor[T any]() (Schema, error)

SchemaFor derives a strict Schema from the Go type T by reflection, so a caller describes the desired output as an ordinary struct rather than by hand-writing JSON Schema. Name defaults to T's type name ("output" when T is unnamed); set Name or Description on the result to override.

The supported subset covers what structured-output backends accept: structs (honoring json tags, with embedded structs flattened and json:"-" fields skipped), bool, the integer and float kinds, string, []byte (as a base64 string), slices and arrays, string-keyed maps, pointers (which mark a field optional), and time.Time (as a date-time string). A non-optional field — one without omitempty and not a pointer — is added to the object's required list. SchemaFor returns an error for types it cannot represent (channels, funcs, non-string map keys) and for recursive types.

type Usage 

type Usage struct {
	InputTokens  int
	OutputTokens int
	TotalTokens  int
}

Usage reports the token accounting a model returns with a Response. The three counts are the common ground between providers: InputTokens is OpenAI's prompt_tokens (Responses API input_tokens) and Gemini's promptTokenCount; OutputTokens is OpenAI's completion_tokens (output_tokens) and Gemini's candidatesTokenCount; TotalTokens is OpenAI's total_tokens and Gemini's totalTokenCount. A count is zero when the backend does not report it.

Source Files

View all Source files

Directories

Path Synopsis
Package anthropic implements the llm.Model interface against Anthropic's Messages API (POST /v1/messages) using structured JSON-schema output.
 Package anthropic implements the llm.Model interface against Anthropic's Messages API (POST /v1/messages) using structured JSON-schema output.
example command
Command example shows the public API of the core/llm/anthropic package: build a reusable Client, bind a structured-output type with New, and call Generate to get a fully decoded value plus token usage and finish reason.
 Command example shows the public API of the core/llm/anthropic package: build a reusable Client, bind a structured-output type with New, and call Generate to get a fully decoded value plus token usage and finish reason.
Package gemini implements the llm.Model interface against Google's Gemini generateContent API using a response schema (responseMimeType=application/json + responseSchema).
 Package gemini implements the llm.Model interface against Google's Gemini generateContent API using a response schema (responseMimeType=application/json + responseSchema).
example command
Command example shows the public API of the core/llm/gemini package: build a reusable Client, bind a structured-output type with New, and call Generate to get a fully decoded value plus token usage and finish reason.
 Command example shows the public API of the core/llm/gemini package: build a reusable Client, bind a structured-output type with New, and call Generate to get a fully decoded value plus token usage and finish reason.
Package openai implements the llm.Model interface against OpenAI's Responses API (POST /v1/responses) using strict structured output.
 Package openai implements the llm.Model interface against OpenAI's Responses API (POST /v1/responses) using strict structured output.
example command
Command example shows the public API of the core/llm/openai package: build a reusable Client, bind a structured-output type with New, and call Generate to get a fully decoded value plus token usage and finish reason.
 Command example shows the public API of the core/llm/openai package: build a reusable Client, bind a structured-output type with New, and call Generate to get a fully decoded value plus token usage and finish reason.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.