llm

package
v0.7.59 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 13, 2026 License: MIT Imports: 15 Imported by: 0

Documentation

Overview

Package llm is the harness's view of a model: one request in, a stream of pieces and one whole reply out, whatever wire API carries them.

Index

Constants

View Source
const AnthropicBaseURL = "https://api.anthropic.com"

AnthropicBaseURL is the Anthropic API's own address.

View Source
const GeminiBaseURL = "https://generativelanguage.googleapis.com"

GeminiBaseURL is the Gemini API's own address.

View Source
const OpenAIBaseURL = "https://api.openai.com/v1"

OpenAIBaseURL is the OpenAI API's own address.

Variables

This section is empty.

Functions

func IsError

func IsError(err error, c Class) bool

IsError reports whether err is a model failure of class c.

func StatedWindow

func StatedWindow(message string) int64

StatedWindow is the window, in tokens, a refusal's words state, or zero.

Types

type Anthropic

type Anthropic struct {
	// BaseURL is the server's address with no trailing slash.
	BaseURL string
	// Model is the model's id.
	Model string
	// Key returns the API key. It is called for every request, so the key
	// is held nowhere else.
	Key func() string
	// HTTP sends the requests. It must not set a response timeout: a
	// reply streams for as long as the model writes.
	HTTP *http.Client
}

Anthropic reaches a model over the Anthropic Messages API.

func (*Anthropic) Stream

func (a *Anthropic) Stream(ctx context.Context, req Request, onDelta func(Delta)) (*Response, error)

Stream sends req and reads the reply's event stream.

type Chat

type Chat struct {
	// BaseURL is the server's address up to and without /chat/completions.
	BaseURL string
	// Model is the model's id.
	Model string
	// Key returns the bearer. An empty key sends no Authorization header,
	// for a server that takes none.
	Key func() string
	// HTTP sends the requests. It must not set a response timeout.
	HTTP *http.Client

	// Sampling is sent with every request; a nil field is left out.
	Temperature, TopP, MinP, PresencePenalty, RepetitionPenalty *float64
	TopK                                                        *int64
	// ExtraBody's members are added to every request's body as they are.
	ExtraBody json.RawMessage
	// ReplayReasoning sends earlier reasoning back with each assistant
	// message, as reasoning_content.
	ReplayReasoning bool
}

Chat reaches a model over OpenAI-style Chat Completions, as OpenAI- compatible servers speak it: a model server on the person's own machine, or a hosted router. The servers agree on the request and differ in the details of the reply, so the reader accepts every shape it knows.

func (*Chat) Stream

func (c *Chat) Stream(ctx context.Context, req Request, onDelta func(Delta)) (*Response, error)

Stream sends req and reads the reply's chunks.

type Class

type Class string

Class is the kind of failure a model request met.

const (
	Overflow   Class = "overflow"
	Capacity   Class = "capacity"
	RateLimit  Class = "rate_limit"
	Credits    Class = "credits"
	Auth       Class = "auth"
	BadRequest Class = "request"
	Transport  Class = "transport"
	Server     Class = "server"
)

Classes of failure. Their names are the harness protocol's error classes.

type Client

type Client interface {
	// Stream sends a request and calls onDelta for each piece of text or
	// reasoning as it arrives. It returns the whole reply, or an *Error.
	// A reply cut off by a failure is not returned.
	Stream(ctx context.Context, req Request, onDelta func(Delta)) (*Response, error)
}

Client reaches one model.

type Delta

type Delta struct {
	Kind Kind
	Text string
}

Delta is a piece of a reply as it streams: Text or Reasoning.

type Error

type Error struct {
	Class Class
	// Status is the HTTP status, zero when the failure had none.
	Status int
	// Message is the server's own words, or the transport's.
	Message string
	// RetryAfter is the wait the server asked for, zero when it gave none.
	RetryAfter time.Duration
	// Window is the model's window in tokens, when the server stated it in
	// refusing a conversation as too long; zero when it stated none.
	Window int64
}

Error is a failed model request.

func (*Error) Error

func (e *Error) Error() string

type Finish

type Finish string

Finish is why a reply ended.

const (
	// FinishStop: the model finished its answer.
	FinishStop Finish = "stop"
	// FinishToolCalls: the model asked for its calls to be run.
	FinishToolCalls Finish = "tool_calls"
	// FinishLength: the reply stopped at the output limit.
	FinishLength Finish = "length"
	// FinishOther: the server ended the reply for a reason of its own.
	FinishOther Finish = "other"
)

Why a reply ended.

type Gemini

type Gemini struct {
	// BaseURL is the server's address with no trailing slash and no
	// version.
	BaseURL string
	// Model is the model's id.
	Model string
	// Key returns the API key.
	Key func() string
	// HTTP sends the requests. It must not set a response timeout.
	HTTP *http.Client
}

Gemini reaches a model over the Gemini API's generateContent.

A part of the model's reply may carry a thought signature, which the API expects back with that part, so each part the model sent is kept as it came and sent back unchanged.

func (*Gemini) Stream

func (g *Gemini) Stream(ctx context.Context, req Request, onDelta func(Delta)) (*Response, error)

Stream sends req and reads the reply's chunks.

type Image

type Image struct {
	MediaType string `json:"mediaType"`
	Data      string `json:"data"`
}

Image is an image in a tool result, base64-encoded.

type Kind

type Kind string

Kind is what a part of a message is.

const (
	// Text is text: the person's, or the model's answer.
	Text Kind = "text"
	// Reasoning is the model's reasoning. Meta holds what its server needs
	// to take it back, and it is sent back unchanged.
	Reasoning Kind = "reasoning"
	// Call is a tool call the model made.
	Call Kind = "call"
	// Result is a tool call's result.
	Result Kind = "result"
)

Kinds of part.

type Message

type Message struct {
	Role  Role   `json:"role"`
	Parts []Part `json:"parts"`
}

Message is one message of a conversation.

type Part

type Part struct {
	Kind Kind   `json:"kind"`
	Text string `json:"text,omitempty"`

	// CallID names a call, and the call a result answers. Name is the
	// tool's name and Arguments its arguments as the model wrote them.
	// Arguments is text, not parsed JSON: a reply cut at the output limit
	// leaves a call's arguments unfinished.
	CallID    string `json:"callId,omitempty"`
	Name      string `json:"name,omitempty"`
	Arguments string `json:"arguments,omitempty"`

	// Failed marks a result that is an error.
	Failed bool `json:"failed,omitempty"`
	// Images are the images a result carries.
	Images []Image `json:"images,omitempty"`

	// Meta is the server's own data for this part (a reasoning block's
	// signature, for one). Only the client that produced it reads it.
	Meta json.RawMessage `json:"meta,omitempty"`
}

Part is one piece of a message.

type Request

type Request struct {
	System    string
	Messages  []Message
	Tools     []Tool
	MaxTokens int
}

Request is one model request.

type Response

type Response struct {
	// Parts is the reply in order: reasoning, text and calls.
	Parts  []Part
	Finish Finish
	Usage  Usage
}

Response is a whole reply.

type Responses

type Responses struct {
	// BaseURL is the server's address up to and without /responses.
	BaseURL string
	// Model is the model's id.
	Model string
	// Key returns the bearer.
	Key func() string
	// HTTP sends the requests. It must not set a response timeout.
	HTTP *http.Client
}

Responses reaches a model over the OpenAI Responses API.

The client keeps no state on the server: every request sets store to false, carries the whole conversation, and asks for each reasoning item's encrypted content, which it sends back unchanged so the model has its earlier reasoning without the server having kept it.

func (*Responses) Stream

func (c *Responses) Stream(ctx context.Context, req Request, onDelta func(Delta)) (*Response, error)

Stream sends req and reads the reply's events.

type Role

type Role string

Role is who a message is from.

const (
	User      Role = "user"
	Assistant Role = "assistant"
)

Roles.

type Tool

type Tool struct {
	Name        string
	Description string
	// Schema is the JSON Schema of the tool's arguments.
	Schema json.RawMessage
}

Tool is a tool the model may call.

type Usage

type Usage struct {
	Input      int64
	CacheRead  int64
	CacheWrite int64
	Output     int64
}

Usage is the token count of one request. Input, CacheRead and CacheWrite do not overlap.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL