llm

package
v0.3.11 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 27, 2026 License: MIT Imports: 10 Imported by: 0

Documentation

Overview

Package llm is a thin, CGO-free client surface for LLM chat and embedding calls that go through the provider broker. It expresses those calls as a provider manifest with request descriptors, origin rules, and response schemas, so LLM egress inherits the broker's admission, SSRF hardening, credential vault, response-policy secret filtering, and secret-safe deterministic cassettes — chat streams over Server-Sent Events, and both the streaming and non-streaming paths record and replay through provider/cassette.

The package owns request shaping and typed decoding only; the host owns the broker session, the admitted origin, credentials, operation identity, and budget.

The provider protocol forbids secret-shaped values in a request by design, so prompt or embedding-input content that trips the secret heuristic cannot be sent through the broker. PlannedChat and PlannedEmbed screen content up front and return ErrSecretShapedContent, making that inherent constraint a clear, screenable error rather than a confusing failure deep in digest binding.

Index

Constants

View Source 
const (
	// ChatDescriptor is the streaming/non-streaming chat request descriptor.
	ChatDescriptor = "chat"
	// EmbedDescriptor is the embedding request descriptor.
	EmbedDescriptor = "embed"
	// CredentialPurpose is the runtime API-key purpose both descriptors use.
	CredentialPurpose = "runtime"
)

Request descriptor and credential identifiers packaged by the manifest.

Variables

View Source 
var ErrSecretShapedContent = errors.New("content is secret-shaped and cannot be sent through the provider broker")

ErrSecretShapedContent reports prompt or embedding-input content the provider broker refuses to send. LLM egress goes through the secret-safe provider broker, whose protocol forbids secret-shaped values in a request by design, so content that trips the secret heuristic (a literal API key, a "password=…" fragment, a PEM private-key header) cannot be sent through this path. This is an inherent property of routing model calls over the broker, surfaced here as a clear, screenable error rather than a confusing failure deep in digest binding. Callers that must send such content need a channel other than the provider broker.

Functions

func Manifest 

func Manifest(origin Origin) (*manifest.Manifest, error)

Manifest builds the Anthropic LLM provider manifest bound to origin. It covers chat over /v1/messages (streaming and non-streaming) and embeddings over /v1/embeddings.

func PlannedChat 

func PlannedChat(m *manifest.Manifest, origin *providerv0.AdmittedOrigin, request ChatRequest, idempotencyKey, responsePolicyDigest string) (*providerv0.PlannedRequest, error)

PlannedChat binds a chat request to a digest-bound PlannedRequest for the chat descriptor. The host wraps it in an ExecuteRequestRequest with the operation, budget, and credential handles before executing.

func PlannedEmbed 

func PlannedEmbed(m *manifest.Manifest, origin *providerv0.AdmittedOrigin, request EmbedRequest, idempotencyKey, responsePolicyDigest string) (*providerv0.PlannedRequest, error)

PlannedEmbed binds an embedding request to a digest-bound PlannedRequest for the embed descriptor.

func ScreenContent 

func ScreenContent(texts []string) error

ScreenContent rejects any free-form text the provider secret heuristic would flag, using the exact heuristic the broker enforces, so a caller fails fast and legibly before a request is built.

Types

type ChatDelta 

type ChatDelta struct {
	EventType  string
	Text       string
	StopReason string
	Usage      Usage
	Terminal   bool
}

ChatDelta is one incremental streamed event. Text is the fragment this event contributed, if any; StopReason is set on the event that reports it; Terminal marks the final event of the delivered stream.

func DecodeEvent 

func DecodeEvent(event *providerv0.FilteredEvent) ChatDelta

DecodeEvent types a single filtered stream event, for a caller wiring the broker's live OnStreamEvent callback to typed deltas as they arrive.

type ChatRequest 

type ChatRequest struct {
	Model       string
	Messages    []Message
	MaxTokens   int64
	System      string
	Temperature string // canonical decimal string; omitted when empty
	Stream      bool
}

ChatRequest is a typed chat completion request. It shapes the descriptor's allowed body fields; the host binds it into a PlannedRequest and runs it through the broker.

func (ChatRequest) Body 

func (r ChatRequest) Body() map[string]*providerv0.PublicValue

Body renders the request as descriptor-allowed body fields.

Prompt content is user text that may legitimately look secret-shaped; the provider protocol forbids secret-shaped structured values by design (see canonical validation), so callers must screen content with ScreenContent before building a request — Body itself does no screening.

type ChatResponse 

type ChatResponse struct {
	Text       string
	StopReason string
	Usage      Usage
}

ChatResponse is a whole non-streaming chat completion.

func DecodeChat 

func DecodeChat(response *providerv0.ExecuteRequestResponse) (*ChatResponse, error)

DecodeChat decodes a whole non-streaming chat response from the filtered forwarded fields.

type ChatStream 

type ChatStream struct {
	Deltas     []ChatDelta
	Text       string
	StopReason string
	Usage      Usage
	Complete   bool
}

ChatStream is the reconstructed typed stream of chat deltas plus the assembled text and terminal metadata. Complete distinguishes a stream the model finished (a stop reason was reported) from one that ended early — a truncated stream closed cleanly mid-generation decodes with Complete false, so a consumer never mistakes truncation for a finished answer.

func DecodeChatStream 

func DecodeChatStream(response *providerv0.ExecuteRequestResponse) (*ChatStream, error)

DecodeChatStream reconstructs the ordered typed stream from the filtered events. The assembled Text concatenates every text delta in event order.

type Client 

type Client struct {
	// contains filtered or unexported fields
}

Client issues typed LLM and embedding calls through an admitted broker executor and decodes the filtered result. It owns no admission, credentials, or network — the host builds the request envelope and supplies the executor.

func NewClient 

func NewClient(exec Executor) *Client

NewClient wraps a broker executor.

func (*Client) Chat 

func (c *Client) Chat(ctx context.Context, request *providerv0.ExecuteRequestRequest) (*ChatResponse, error)

Chat runs a non-streaming chat request and decodes the whole response.

func (*Client) ChatStream 

func (c *Client) ChatStream(ctx context.Context, request *providerv0.ExecuteRequestRequest) (*ChatStream, error)

ChatStream runs a streaming chat request and decodes the ordered deltas.

func (*Client) Embed 

func (c *Client) Embed(ctx context.Context, request *providerv0.ExecuteRequestRequest) (*EmbeddingResponse, error)

Embed runs an embedding request and decodes the ordered vectors.

type EmbedRequest 

type EmbedRequest struct {
	Model string
	Input []string
}

EmbedRequest is a typed embedding request.

func (EmbedRequest) Body 

func (r EmbedRequest) Body() map[string]*providerv0.PublicValue

Body renders the request as descriptor-allowed body fields. It does no secret screening; callers screen r.content() with ScreenContent first.

type Embedding 

type Embedding []float64

Embedding is one embedding vector.

type EmbeddingResponse 

type EmbeddingResponse struct {
	Embeddings []Embedding
	Usage      Usage
}

EmbeddingResponse is the decoded embedding result: one vector per input, in input order, plus token accounting.

func DecodeEmbedding 

func DecodeEmbedding(response *providerv0.ExecuteRequestResponse) (*EmbeddingResponse, error)

DecodeEmbedding decodes the filtered forwarded fields into ordered embedding vectors.

type Executor 

type Executor interface {
	Execute(ctx context.Context, request *providerv0.ExecuteRequestRequest) (*providerv0.ExecuteRequestResponse, error)
}

Executor runs one admitted broker request. *broker.Session satisfies it.

type Message 

type Message struct {
	Role    string
	Content string
}

Message is one chat turn.

type Origin 

type Origin struct {
	Scheme string
	Host   string
	Port   uint32
	// Class is the manifest private-network class token: loopback, link-local,
	// private, or public.
	Class string
}

Origin is the network origin the manifest targets. It is configurable so the same manifest can front the real vendor host, a local gateway or proxy, or — for offline record/replay tests — a loopback server, without a network call.

func AnthropicOrigin 

func AnthropicOrigin() Origin

AnthropicOrigin is the production Anthropic API origin.

type Usage 

type Usage struct {
	InputTokens  int64
	OutputTokens int64
}

Usage is the token accounting a chat call reports.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.