llm

package
v0.20.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 10, 2026 License: Apache-2.0 Imports: 7 Imported by: 0

Documentation

Overview

Package llm is the single seam through which every model call passes. Determinism mitigations (schema-constrained prompts, tolerant extraction with one retry, content-hash caching) live here rather than in each caller.

Index

Constants

This section is empty.

Variables

View Source
var ErrTimeout = errors.New("llm: gateway timed out")

ErrTimeout reports that a gateway call exceeded its configured timeout, distinct from context cancellation.

Functions

func CacheKey

func CacheKey(r Req) string

CacheKey derives a content-addressed key. Every field that changes the model's answer must contribute, or a cache hit serves a wrong response.

func ExtractJSON

func ExtractJSON(raw string) (json.RawMessage, error)

ExtractJSON pulls a JSON value out of a model response, tolerating code fences and surrounding prose. Fails loudly with the raw text quoted so a refusal, rate-limit notice, and truncated response stay distinguishable.

Types

type Cache

type Cache interface {
	Get(key string) (string, bool, error)
	Put(key, value string) error
}

Cache stores completions by content hash.

type Gateway

type Gateway interface {
	Complete(ctx context.Context, r Req) (Res, error)
	// ModelFor names the concrete model for a class without making a call.
	// Returns "" when unknown.
	ModelFor(class ModelClass) string
}

Gateway is a model backend.

type ModelClass

type ModelClass string

ModelClass routes a call to a capability tier rather than a model name.

const (
	ClassStrong ModelClass = "strong"
	ClassFast   ModelClass = "fast"
)

type Req

type Req struct {
	Role       string
	Prompt     string
	Schema     json.RawMessage
	ModelClass ModelClass
	CacheKey   string
}

Req is one model call.

type Res

type Res struct {
	Text   string
	Model  string
	Cached bool
}

Res is one model response.

func CompleteJSON

func CompleteJSON[T any](ctx context.Context, g Gateway, r Req) (T, Res, error)

CompleteJSON calls the gateway and decodes a JSON response into T, retrying once on a parse failure. Returns the Res alongside the decoded value so a caller can read the model that answered (for provenance).

Directories

Path Synopsis
Package api implements the LLM gateway against an HTTP endpoint that speaks either Anthropic's Messages API or OpenAI's Chat Completions API.
Package api implements the LLM gateway against an HTTP endpoint that speaks either Anthropic's Messages API or OpenAI's Chat Completions API.
Package fake provides an in-memory Gateway for tests.
Package fake provides an in-memory Gateway for tests.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL