llm

package
v0.14.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 16, 2026 License: Apache-2.0 Imports: 7 Imported by: 0

Documentation

Overview

Package llm is the single seam through which every model call passes. Determinism mitigations (schema-constrained prompts, tolerant extraction with one retry, content-hash caching) live here rather than in each caller.

Index

Constants

This section is empty.

Variables

View Source
var ErrTimeout = errors.New("llm: gateway timed out")

ErrTimeout reports that a gateway call exceeded its configured timeout, distinct from context cancellation.

Functions

func CacheKey

func CacheKey(r Req) string

CacheKey derives a content-addressed key. Every field that changes the model's answer must contribute, or a cache hit serves a wrong response.

func ExtractJSON

func ExtractJSON(raw string) (json.RawMessage, error)

ExtractJSON pulls a JSON value out of a model response, tolerating code fences and surrounding prose. Fails loudly with the raw text quoted so a refusal, rate-limit notice, and truncated response stay distinguishable.

Types

type Cache

type Cache interface {
	Get(key string) (string, bool, error)
	Put(key, value string) error
}

Cache stores completions by content hash.

type Gateway

type Gateway interface {
	Complete(ctx context.Context, r Req) (Res, error)
	// ModelFor names the concrete model for a class without making a call.
	// Returns "" when unknown (e.g. a subscription CLI).
	ModelFor(class ModelClass) string
}

Gateway is a model backend.

type ModelClass

type ModelClass string

ModelClass routes a call to a capability tier rather than a model name.

const (
	ClassStrong ModelClass = "strong"
	ClassFast   ModelClass = "fast"
)

type Req

type Req struct {
	Role       string
	Prompt     string
	Schema     json.RawMessage
	ModelClass ModelClass
	CacheKey   string
}

Req is one model call.

type Res

type Res struct {
	Text   string
	Model  string
	Cached bool
}

Res is one model response.

func CompleteJSON

func CompleteJSON[T any](ctx context.Context, g Gateway, r Req) (T, Res, error)

CompleteJSON calls the gateway and decodes a JSON response into T, retrying once on a parse failure. Returns the Res alongside the decoded value so a caller can read the model that answered (for provenance).

Directories

Path Synopsis
Package agentcli implements the LLM gateway by shelling out to a coding-agent CLI in print mode, billing to the operator's existing subscription.
Package agentcli implements the LLM gateway by shelling out to a coding-agent CLI in print mode, billing to the operator's existing subscription.
Package api implements the LLM gateway against a direct Anthropic-compatible HTTP endpoint.
Package api implements the LLM gateway against a direct Anthropic-compatible HTTP endpoint.
Package fake provides an in-memory Gateway for tests.
Package fake provides an in-memory Gateway for tests.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL