llm

package
v0.11.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 17, 2026 License: MIT Imports: 25 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	// ImageInputFormatFile is OpenRouter's universal `file` part (works for
	// Gemini; the default). Kept for backward compatibility.
	ImageInputFormatFile = "file"
	// ImageInputFormatOpenAI is the OpenAI-standard image_url/video_url parts,
	// required by OpenAI-compatible backends (litellm/vLLM) which reject `file`.
	ImageInputFormatOpenAI = "openai"
)

Image input formats — how a media file is encoded as an LLM content part.

Variables

This section is empty.

Functions

func EstimateMessageTokens added in v0.11.0 

func EstimateMessageTokens(message Message) int

EstimateMessageTokens estimates textual content in one chat message.

func EstimateMessagesTokens added in v0.11.0 

func EstimateMessagesTokens(messages []Message) int

EstimateMessagesTokens estimates textual content across chat messages.

func EstimateTextTokens added in v0.11.0 

func EstimateTextTokens(text string) int

EstimateTextTokens returns a model-agnostic text-only estimate using four Unicode code points per token. It intentionally excludes media tokenization.

func FilterReasoningForLog 

func FilterReasoningForLog(details interface{}) interface{}

FilterReasoningForLog filters out encrypted reasoning entries from reasoning_details. Keeps only "reasoning.text" entries which are human-readable and useful for debugging. This prevents massive base64 blobs from polluting logs.

func IsSafetyBlock added in v0.10.2 

func IsSafetyBlock(err error) bool

IsSafetyBlock reports whether err (or anything it wraps) is a provider content-policy rejection — Gemini PROHIBITED_CONTENT / RECITATION, an OpenAI content_filter, etc. — i.e. the permanent KindSafety class from classifyUpstreamError. Callers outside this package can't inspect the unexported providerBodyError directly, so this is the public seam. The bot uses it to show a "blocked by the safety filter" message instead of the generic API-error fallback, and to tag the turn's span.

func MediaPart 

func MediaPart(format, mimeType, fileName, dataURL string) interface{}

MediaPart builds the LLM input content part for a media file in the shape the configured backend accepts. With ImageInputFormatOpenAI, images become image_url and videos become video_url (litellm/vLLM); every other format, and every non-image/-video MIME (pdf, audio, …), uses OpenRouter's `file` part.

dataURL is the full "data:<mime>;base64,<...>" string.

🔴 This is the single chokepoint for encoding visual media for an LLM call. All call sites must route image/video parts through here — see the guard test in media_part_guard_test.go. A site that hand-builds FilePart for an image silently 400s on the OpenAI-compatible contour (class of bug: response/request shape, see CLAUDE.md).

func NewSafetyBlockErrorForTest added in v0.10.2 

func NewSafetyBlockErrorForTest(message string, code int) error

NewSafetyBlockErrorForTest builds a synthetic provider body error so tests in other packages (e.g. the bot's errorReplyText) can drive the IsSafetyBlock branch without standing up an HTTP server. providerBodyError is intentionally unexported; this is the sanctioned construction seam. Not used in production.

func RecordLLMRequest 

func RecordLLMRequest(userID string, model string, durationSeconds float64, success bool, promptTokens, completionTokens int, cost *float64, jobType string)

RecordLLMRequest records LLM request metrics. jobType should be "interactive" or "background" (use jobtype.JobType constants).

func RecordLLMRetry 

func RecordLLMRetry(model string)

RecordLLMRetry records a retry attempt.

Types

type Annotation added in v0.10.2 

type Annotation struct {
	Type        string      `json:"type"` // "url_citation"
	URLCitation URLCitation `json:"url_citation"`
}

Annotation is a single source citation returned by web-search models (perplexity/sonar-*) on the assistant message. The content carries 1-based [N] markers that index into this slice; url_citation holds the real source URL.

type ChatCompletionChunk 

type ChatCompletionChunk struct {
	ID          string        `json:"id"`
	Object      string        `json:"object"` // "chat.completion.chunk"
	Created     int64         `json:"created"`
	Model       string        `json:"model"`
	Provider    string        `json:"provider,omitempty"`
	ServiceTier string        `json:"service_tier,omitempty"`
	Choices     []ChunkChoice `json:"choices"`
	Usage       *ChunkUsage   `json:"usage,omitempty"`
}

ChatCompletionChunk is one SSE delta from /chat/completions in stream mode. Mirrors the OpenAI-compatible shape OpenRouter emits between `data:` and `\n\n`. Most chunks carry a single Choices[0].Delta with partial content/reasoning/tool_calls; the final chunk before `data: [DONE]` additionally carries Usage.

type ChatCompletionRequest 

type ChatCompletionRequest struct {
	Model              string           `json:"model"`
	Messages           []Message        `json:"messages"`
	Plugins            []Plugin         `json:"plugins,omitempty"`
	Tools              []Tool           `json:"tools,omitempty"`
	ToolChoice         any              `json:"tool_choice,omitempty"`
	ResponseFormat     interface{}      `json:"response_format,omitempty"`
	N                  int              `json:"n,omitempty"`
	Temperature        *float64         `json:"temperature,omitempty"`
	MaxTokens          int              `json:"max_tokens,omitempty"`
	Reasoning          *ReasoningConfig `json:"reasoning,omitempty"`
	ChatTemplateKwargs map[string]any   `json:"chat_template_kwargs,omitempty"`
	// Modalities enables image-output models. Use ["image","text"] for Gemini
	// image models that return both text and images. Required for image generation;
	// see TestAllImageGenerationRequestsSetModalities.
	Modalities []string `json:"modalities,omitempty"`
	// ImageConfig is model-specific image-generation configuration
	// (aspect ratio, size). Only used when Modalities includes "image".
	ImageConfig *ImageConfig `json:"image_config,omitempty"`
	// Provider overrides OpenRouter's provider routing for this request.
	// When nil, the client-level default (if any) is applied before sending.
	Provider *ProviderRouting `json:"provider,omitempty"`
	// User is the OpenAI/OpenRouter-standard end-user id, used for abuse
	// signals on the provider side and as user.id on OR-emitted Broadcast
	// spans. Auto-populated from UserID in CreateChatCompletion when empty.
	User string `json:"user,omitempty"`
	// Trace carries OpenRouter's Broadcast metadata. Keys with special
	// meaning: trace_id, parent_span_id, trace_name, span_name,
	// generation_name. CreateChatCompletion auto-fills trace_id and
	// parent_span_id from the current span context so OR-emitted spans
	// nest under our local trace when Broadcast is configured. When
	// Broadcast is disabled on the OR side, the field is ignored.
	Trace map[string]any `json:"trace,omitempty"`

	// Stream switches the request to SSE streaming. Set internally by
	// CreateChatCompletionStream; callers leave this false and use the
	// dedicated streaming method.
	Stream bool `json:"stream,omitempty"`

	// UserID is used for metrics tracking only, not sent to API
	UserID string `json:"-"`
}

type ChatCompletionResponse 

type ChatCompletionResponse struct {
	ID       string           `json:"id"`
	Model    string           `json:"model"`
	Provider string           `json:"provider,omitempty"` // Actual provider that served the request (e.g. "Google", "Google AI Studio")
	Choices  []ResponseChoice `json:"choices"`
	Usage    Usage            `json:"usage"`

	// DebugRequestBody contains the raw JSON request body sent to the API.
	// Not part of API response - populated by client for debugging purposes.
	DebugRequestBody string `json:"-"`

	// DebugResponseBody contains the raw JSON response body from the API.
	// Not part of API response - populated by client for debugging purposes.
	DebugResponseBody string `json:"-"`
}

type ChatCompletionStream 

type ChatCompletionStream struct {
	Events           <-chan StreamEvent
	DebugRequestBody string
}

ChatCompletionStream is the result of opening a streaming /chat/completions request. Events delivers the SSE deltas; DebugRequestBody mirrors the buffered ChatCompletionResponse field so callers can record the raw JSON request body alongside the synthesized response.

type ChunkChoice 

type ChunkChoice struct {
	Index              int              `json:"index"`
	Delta              ChunkDelta       `json:"delta"`
	FinishReason       string           `json:"finish_reason,omitempty"`
	NativeFinishReason string           `json:"native_finish_reason,omitempty"`
	Error              *orErrorEnvelope `json:"error,omitempty"` // mid-stream provider failure injected by OpenRouter
}

type ChunkDelta 

type ChunkDelta struct {
	Role             string          `json:"role,omitempty"`
	Content          string          `json:"content,omitempty"`
	Reasoning        string          `json:"reasoning,omitempty"`
	ReasoningDetails interface{}     `json:"reasoning_details,omitempty"`
	ToolCalls        []ChunkToolCall `json:"tool_calls,omitempty"`
}

type ChunkToolCall 

type ChunkToolCall struct {
	Index    int                   `json:"index"`
	ID       string                `json:"id,omitempty"`
	Type     string                `json:"type,omitempty"`
	Function ChunkToolCallFunction `json:"function,omitempty"`
}

type ChunkToolCallFunction 

type ChunkToolCallFunction struct {
	Name      string `json:"name,omitempty"`
	Arguments string `json:"arguments,omitempty"`
}

type ChunkUsage 

type ChunkUsage = Usage

ChunkUsage is the usage block on a streaming chunk. It aliases Usage so the streaming and non-streaming paths share the same polymorphic-cost decoding.

type Citation added in v0.10.2 

type Citation struct {
	URL   string
	Title string
}

Citation is the trimmed source (URL + title) passed up from a search tool to the agent layer, so the orchestrator can ground/verify links without re-parsing the raw API annotation shape.

type Client 

type Client interface {
	CreateChatCompletion(ctx context.Context, req ChatCompletionRequest) (ChatCompletionResponse, error)
	CreateChatCompletionStream(ctx context.Context, req ChatCompletionRequest) (*ChatCompletionStream, error)
	CreateEmbeddings(ctx context.Context, req EmbeddingRequest) (EmbeddingResponse, error)
}

func NewClient 

func NewClient(logger *slog.Logger, apiKey, proxyURL, baseURL string, defaultProvider *ProviderRouting) (Client, error)

NewClient builds a client against any OpenAI-compatible chat/embeddings endpoint (OpenRouter, litellm, vLLM, …) given by baseURL. OpenRouter-specific extensions (provider routing, plugins, usage-based cost) are sent regardless; compatible backends simply ignore the fields they don't recognise.

type EmbeddingObject 

type EmbeddingObject struct {
	Object    string    `json:"object"`
	Embedding []float32 `json:"embedding"`
	Index     int       `json:"index"`
}

type EmbeddingRequest 

type EmbeddingRequest struct {
	Model      string                 `json:"model"`
	Input      []string               `json:"input"`
	Dimensions int                    `json:"dimensions,omitempty"`
	Provider   *ProviderRouting       `json:"provider,omitempty"`
	LogMeta    map[string]interface{} `json:"-"`
	// User mirrors the OpenAI-standard end-user id. See the identical
	// field on ChatCompletionRequest for semantics.
	User string `json:"user,omitempty"`
	// Trace carries OpenRouter Broadcast metadata. See
	// ChatCompletionRequest.Trace for semantics and auto-populated keys.
	Trace map[string]any `json:"trace,omitempty"`
}

type EmbeddingResponse 

type EmbeddingResponse struct {
	Object string            `json:"object"`
	Data   []EmbeddingObject `json:"data"`
	Model  string            `json:"model"`
	Usage  Usage             `json:"usage"`
}

type ErrorKind added in v0.10.3 

type ErrorKind int

ErrorKind is a coarse, stable classification of an upstream provider error. It exists so retry gates and span attributes agree on one vocabulary: a typo in a raw string comparison compiles and silently never matches, whereas a misspelled ErrorKind constant does not compile. The wire strings (see String) are span-attribute values TraceQL queries depend on — treat them as a public contract and never change them for existing kinds. 

const (
	// KindNone means there is no error envelope to classify.
	KindNone ErrorKind = iota
	// KindUnknown is an envelope whose shape matched no known class; the raw
	// message surfaces on the span for triage.
	KindUnknown
	// KindInvalidArgument is a deterministic rejection of this exact request
	// (malformed/oversized input).
	KindInvalidArgument
	// KindSafety is a content-policy rejection (Gemini PROHIBITED_CONTENT /
	// RECITATION, OpenAI content_filter, ...).
	KindSafety
	// KindRateLimited is a gateway or provider throttle.
	KindRateLimited
	// KindContextLength means the request exceeded the model's context window.
	KindContextLength
	// KindThoughtSignature is Gemini's "Corrupted thought signature" — a
	// transient reasoning-state failure on multi-turn tool calls.
	KindThoughtSignature
	// KindUpstream5xx is a provider-side server error without a finer class.
	KindUpstream5xx
	// KindUpstream4xx is a provider-side client error without a finer class.
	KindUpstream4xx
)

func (ErrorKind) IsRetryable added in v0.10.3 

func (k ErrorKind) IsRetryable() bool

IsRetryable reports whether an identical retry can plausibly succeed. Deterministic rejections (content policy, malformed/oversized input) are permanent: the provider will reject the same request again, so a retry only burns latency and tokens. Transient classes (rate limits, 5xx, thought signature, unknown) stay retryable. Motivated by an enricher PROHIBITED_CONTENT 403 that was retried 3x over ~21s on the user's critical path before failing anyway.

func (ErrorKind) String added in v0.10.3 

func (k ErrorKind) String() string

String returns the stable wire form recorded as the error.upstream_kind span attribute. KindNone renders as "" and is never emitted.

type File 

type File struct {
	FileName string `json:"filename"`
	FileData string `json:"file_data"` // data URL format: "data:mime/type;base64,..."
}

File represents a file for multimodal content (v0.6.0: unified format).

type FilePart 

type FilePart struct {
	Type string `json:"type"` // "file"
	File File   `json:"file"`
}

FilePart represents a file part in multimodal messages (v0.6.0: unified format). This is the recommended format for all file types: images, PDFs, audio, video.

type ImageConfig 

type ImageConfig struct {
	// AspectRatio: "1:1","2:3","3:2","3:4","4:3","4:5","5:4","9:16","16:9","21:9"
	// Extended on nano banana: "1:4","4:1","1:8","8:1".
	AspectRatio string `json:"aspect_ratio,omitempty"`
	// ImageSize: "1K", "2K", "4K". OpenRouter's validator advertises "0.5K"
	// as a fourth value but Google rejects that upstream as INVALID_ARGUMENT;
	// the actual Gemini enum value "512" is in turn rejected by OR's validator.
	// Don't advertise either to the LLM — see docs/bugs/2026-04-30-nano-banana-*.
	ImageSize string `json:"image_size,omitempty"`
}

ImageConfig controls image generation output for models that support it (e.g. google/gemini-3.1-flash-image-preview). Only meaningful when the request sets Modalities to include "image".

type ImageOutput 

type ImageOutput struct {
	Type     string        `json:"type"` // "image_url"
	ImageURL ImageURLValue `json:"image_url"`
}

ImageOutput represents an image emitted by image-generation models in the response. The URL is a base64 data URL, e.g. "data:image/png;base64,...".

type ImageURLPart 

type ImageURLPart struct {
	Type     string        `json:"type"` // "image_url"
	ImageURL ImageURLValue `json:"image_url"`
}

ImageURLPart represents an image reference for MODEL INPUT, in the OpenAI-compatible shape. Use this when calling image-generation or image-editing models (e.g. google/gemini-3.1-flash-image-preview) — they reject the "file" FilePart shape with "Invalid file type: image/…". Regular multimodal text models accept either shape, but for image models this is the only format the provider accepts for input images.

type ImageURLValue 

type ImageURLValue struct {
	URL string `json:"url"`
}

ImageURLValue is the inner object of an ImageOutput; kept as a named type so struct literals can be written without repeating the anonymous shape.

type JSONSchema 

type JSONSchema struct {
	Name   string                 `json:"name"`
	Strict bool                   `json:"strict,omitempty"`
	Schema map[string]interface{} `json:"schema"`
}

type Message 

type Message struct {
	Role             string      `json:"role"`
	Content          interface{} `json:"content"`
	ToolCalls        []ToolCall  `json:"tool_calls,omitempty"`
	ToolCallID       string      `json:"tool_call_id,omitempty"`
	ReasoningDetails interface{} `json:"reasoning_details,omitempty"`
}

type PDFConfig 

type PDFConfig struct {
	Engine string `json:"engine"`
}

PDFConfig selects OpenRouter's PDF parsing engine for the pdf plugin.

type Plugin 

type Plugin struct {
	ID  string    `json:"id"`
	PDF PDFConfig `json:"pdf,omitempty"`
}

Plugin is an OpenRouter request plugin (e.g. "response-healing", "pdf").

type ProviderRouting 

type ProviderRouting struct {
	Order          []string `json:"order,omitempty"`
	AllowFallbacks *bool    `json:"allow_fallbacks,omitempty"`
}

ProviderRouting controls OpenRouter's provider selection for a request. Order lists preferred providers (tried in sequence). AllowFallbacks is a pointer so callers can distinguish "unset" (default true on OpenRouter's side) from an explicit false (strict routing — fail instead of falling back to providers outside the order list).

type ReasoningConfig 

type ReasoningConfig struct {
	Effort    string `json:"effort,omitempty"`     // Gemini 3: "minimal", "low", "medium", "high"
	MaxTokens int    `json:"max_tokens,omitempty"` // Other models: token budget for reasoning
	Exclude   bool   `json:"exclude,omitempty"`    // Suppress reasoning from response
}

ReasoningConfig controls the model's internal reasoning behavior. For Gemini 3: effort levels "minimal", "low", "medium", "high". For other models: max_tokens (1024-128000) controls reasoning depth.

NOTE: response-healing plugin breaks reasoning visibility when combined with json_object format.

func ReasoningFor 

func ReasoningFor(level string) *ReasoningConfig

ReasoningFor maps a config thinking level to a request ReasoningConfig. Returns nil — meaning the reasoning field is omitted from the request — for "auto" and "off". On Gemini an absent reasoning field enables dynamic thinking (the model picks its own budget), so "off" does NOT disable reasoning there; "auto" is the honest name for that behavior. An empty level also returns nil so callers apply their own defaults before calling.

type ResponseChoice 

type ResponseChoice struct {
	Message      ResponseMessage `json:"message"`
	FinishReason string          `json:"finish_reason,omitempty"`
	Index        int             `json:"index"`
}

ResponseChoice is one choice on a ChatCompletionResponse. Named for the same reason as ResponseMessage.

type ResponseFormat 

type ResponseFormat struct {
	Type       string      `json:"type"`
	JSONSchema *JSONSchema `json:"json_schema,omitempty"`
}

type ResponseFormatJSONSchema 

type ResponseFormatJSONSchema struct {
	Type       string     `json:"type"` // "json_schema"
	JSONSchema JSONSchema `json:"json_schema"`
}

type ResponseMessage 

type ResponseMessage struct {
	Role             string        `json:"role"`
	Content          string        `json:"content"`
	ToolCalls        []ToolCall    `json:"tool_calls,omitempty"`
	Reasoning        string        `json:"reasoning,omitempty"`         // Gemini 3: raw reasoning text
	ReasoningDetails interface{}   `json:"reasoning_details,omitempty"` // Structured reasoning details
	Images           []ImageOutput `json:"images,omitempty"`            // Generated images (image-output models)
	Annotations      []Annotation  `json:"annotations,omitempty"`       // Source citations (Perplexity/Sonar web search)
}

ResponseMessage is the assistant message on a ChatCompletionResponse choice. Extracted as a named type so test fixtures and helpers don't have to restate the anonymous struct shape every time a field is added.

type StreamEvent 

type StreamEvent struct {
	Chunk *ChatCompletionChunk
	Err   error
}

StreamEvent carries either a decoded chunk or a terminal error. When Err != nil the channel is about to close; callers should treat this as the stream's final event and not look for further chunks. Successful streams terminate by closing the channel without an Err event.

type TextPart 

type TextPart struct {
	Type string `json:"type"`
	Text string `json:"text"`
}

TextPart represents a text part in messages.

type Tool 

type Tool struct {
	Type     string       `json:"type"`
	Function ToolFunction `json:"function"`
}

type ToolCall 

type ToolCall struct {
	ID       string `json:"id"`
	Type     string `json:"type"`
	Function struct {
		Name      string `json:"name"`
		Arguments string `json:"arguments"`
	} `json:"function"`
	ExtraContent interface{} `json:"extra_content,omitempty"`
}

type ToolFunction 

type ToolFunction struct {
	Name        string `json:"name"`
	Description string `json:"description"`
	Parameters  any    `json:"parameters"`
}

type URLCitation added in v0.10.2 

type URLCitation struct {
	URL   string `json:"url"`
	Title string `json:"title"`
}

URLCitation is the payload of a url_citation annotation.

type Usage 

type Usage struct {
	PromptTokens     int      `json:"prompt_tokens"`
	CompletionTokens int      `json:"completion_tokens"`
	TotalTokens      int      `json:"total_tokens"`
	Cost             *float64 `json:"-"`
}

Usage is the token/cost accounting block on a completion or embedding response. Cost is kept as *float64 (nil when absent), but populated via a custom UnmarshalJSON because backends disagree on its wire shape: OpenRouter and most litellm models return a bare number, while litellm returns an object (e.g. {"total_cost": 0.006, ...}) for some providers such as Perplexity.

func (*Usage) UnmarshalJSON 

func (u *Usage) UnmarshalJSON(data []byte) error

type VideoURLPart 

type VideoURLPart struct {
	Type     string        `json:"type"` // "video_url"
	VideoURL ImageURLValue `json:"video_url"`
}

VideoURLPart is the OpenAI/vLLM-compatible video input part, analogous to ImageURLPart. litellm/vLLM accept "video_url" for video input; OpenRouter's "file" part is used instead when ImageInputFormatFile is selected.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.