llm

package
v0.51.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 3, 2026 License: MIT Imports: 13 Imported by: 2

Documentation

Overview

ABOUTME: Detects provider credit/quota exhaustion and builds an actionable, ABOUTME: account-attributed message — never printing the raw API key (#487 Phase 1).

ABOUTME: Model catalog providing a registry of known LLM models and their capabilities. ABOUTME: Supports lookup by ID/alias and listing by provider.

ABOUTME: Core LLM client that routes requests to provider adapters. ABOUTME: Supports multiple providers, default routing, middleware chains, and env-based config.

ABOUTME: Error type hierarchy for the unified LLM client library. ABOUTME: Defines provider errors, retryability, and HTTP status code mapping.

ABOUTME: Provider/model failover — switch lanes on a billing/quota exhaustion so ABOUTME: one dead upstream account doesn't end a run when another lane is configured (#486).

ABOUTME: Pretty-print formatting for Response and Usage types via fmt.Stringer. ABOUTME: Produces concise, human-readable output for CLI and TUI display.

ABOUTME: Middleware types for the unified LLM client's request/response pipeline. ABOUTME: Defines CompleteHandler and the Middleware interface.

ABOUTME: Cost estimation for LLM usage based on the model catalog. ABOUTME: Prices input, output, and cache tokens using per-model rates from the catalog.

ABOUTME: Defines the ProviderAdapter interface for LLM provider implementations. ABOUTME: Each provider (OpenAI, Anthropic, Gemini, etc.) implements this interface.

ABOUTME: Retry middleware with exponential backoff for the LLM client. ABOUTME: Retries only errors that implement Retryable() bool returning true; respects context and RetryAfter.

ABOUTME: Streaming event types and accumulator for incremental LLM responses. ABOUTME: Defines StreamEvent, StreamEventType enum, and StreamAccumulator.

ABOUTME: Middleware that accumulates per-provider token usage across LLM calls. ABOUTME: Thread-safe; used by the TUI dashboard header for real-time token counts.

ABOUTME: Per-provider cost rollup for the TokenTracker middleware. ABOUTME: Maps accumulated Usage to dollar cost via a caller-supplied model resolver.

ABOUTME: Structured trace events for live LLM introspection across console and TUI surfaces.

ABOUTME: Console logger for structured LLM trace events. ABOUTME: Batches sequential text/reasoning deltas into a single summary line per stream.

ABOUTME: Core data types for the unified LLM client library. ABOUTME: Defines Message, ContentPart, Request, Response, Usage, and related types.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func BillingHelp added in v0.47.0

func BillingHelp(err error) (string, bool)

BillingHelp returns an actionable, account-attributed message for a billing error (and true), or ("", false) if err is not one. The message identifies the failing provider, which env var supplied the key, and a masked fingerprint of that key — so a user with multiple accounts knows exactly where to add credit — plus the provider's billing URL. It NEVER includes the raw key.

func EmitRequestSent added in v0.50.0

func EmitRequestSent(ch chan<- StreamEvent, body []byte, tracing bool)

EmitRequestSent records the verbatim wire body on the event stream. Adapters call it immediately after building the body and before the HTTP call, because that is the only place the body is in scope — the provider's own stream-start arrives several layers deeper, inside SSE parsing.

tracing gates the emit: the body is telemetry, so an untraced caller driving an adapter directly sees the provider event sequence unchanged. A leading synthetic event would otherwise shift every position-indexed assertion in adapter tests and in any embedder reading Stream() by index. Adapters pass the same per-request flag that gates provider events, which the client sets on every request it traces.

func ErrorFromStatusCode

func ErrorFromStatusCode(statusCode int, message, provider string) error

ErrorFromStatusCode maps an HTTP status code to the appropriate error type.

func EstimateCost

func EstimateCost(model string, usage Usage) float64

EstimateCost returns the estimated dollar cost for the given model and token usage. Looks up pricing from the model catalog (supports ID and aliases). Returns 0 for unknown models, logging a single warning per unknown name so operators notice that their --max-cost ceiling will not apply to usage priced at an unknown rate.

Cached prompt tokens are priced from the model's own multipliers, since the discount varies by model rather than by provider — see ModelInfo. Assumes InputTokens excludes the cache buckets, the llm.Usage invariant every adapter normalizes to.

func FormatTraceLine

func FormatTraceLine(evt TraceEvent, verbose bool) string

FormatTraceLine formats one trace event for console or TUI rendering.

func IsBillingError added in v0.47.0

func IsBillingError(err error) bool

IsBillingError reports whether err is a provider credit/quota exhaustion — a recoverable operational condition ("add funds and retry"), distinct from a code bug, an auth failure, or a bad request. It matches the typed QuotaExceededError (OpenAI/compat) and any error in the chain whose message carries a billing signal (Anthropic's credit-balance case).

func NewCallID added in v0.50.0

func NewCallID() string

NewCallID returns an identifier for one LLM request. Callers stamp it into TraceOptions so every trace event from that request shares it; see TraceEvent.CallID for why grouping matters.

Locally generated rather than taken from Response.ID: the provider's ID only exists once the response arrives, so events emitted before then — including the request-start that carries the wire body — would have nothing to group on. Collision resistance only has to hold within one run's log.

func RequestIsTraced added in v0.50.0

func RequestIsTraced(req *Request) bool

RequestIsTraced reports whether the client is tracing this request. The client sets the flag on every request it routes through the streaming trace path; adapters use it to gate telemetry-only emissions (provider events and the wire body) so an untraced caller sees neither.

Types

type AbortError

type AbortError struct {
	SDKError
}

AbortError indicates the operation was explicitly aborted.

type AccessDeniedError

type AccessDeniedError struct {
	ProviderError
}

AccessDeniedError indicates insufficient permissions.

func (*AccessDeniedError) Retryable

func (e *AccessDeniedError) Retryable() bool

type AudioData

type AudioData struct {
	URL       string `json:"url,omitempty"`
	Data      []byte `json:"data,omitempty"`
	MediaType string `json:"media_type,omitempty"`
}

AudioData holds audio content.

type AuthenticationError

type AuthenticationError struct {
	ProviderError
}

AuthenticationError indicates invalid or missing credentials.

func (*AuthenticationError) Retryable

func (e *AuthenticationError) Retryable() bool

type Client

type Client struct {
	// contains filtered or unexported fields
}

Client manages LLM provider adapters and routes requests through a middleware chain.

func NewClient

func NewClient(opts ...ClientOption) (*Client, error)

NewClient creates a Client from the given options.

func NewClientFromEnv

func NewClientFromEnv(constructors map[string]func(apiKey string) (ProviderAdapter, error)) (*Client, error)

func (*Client) AddMiddleware

func (c *Client) AddMiddleware(mw Middleware)

AddMiddleware appends a middleware to the client's existing middleware chain. This allows post-construction addition of middleware (e.g., adding a TokenTracker after the client is built from environment variables).

func (*Client) AddTraceObserver

func (c *Client) AddTraceObserver(obs TraceObserver)

AddTraceObserver appends a live trace observer after client construction.

func (*Client) Close

func (c *Client) Close() error

Close releases resources for all registered provider adapters.

func (*Client) Complete

func (c *Client) Complete(ctx context.Context, req *Request) (*Response, error)

Complete sends a completion request through the middleware chain and returns the response.

func (*Client) CompleteFailover added in v0.47.0

func (c *Client) CompleteFailover(ctx context.Context, req *Request, fallbacks []Target, onFailover func(FailoverEvent)) (*Response, error)

CompleteFailover tries req against its own provider/model first, then each fallback Target in order, switching lanes on a failover-class error. onFailover (optional) is called before each switch so the caller can emit an audit event.

Only a *failover-class* error switches lanes (see isFailoverClass): a billing/ quota exhaustion means the current account is out of credit, so another lane may succeed. A transient error was already retried by the retry middleware; a code/auth/context error would fail identically on every lane, so those are returned as-is without burning the other lanes.

func (*Client) DefaultProvider

func (c *Client) DefaultProvider() string

DefaultProvider returns the name of the default provider, or empty string.

func (*Client) HasMiddleware added in v0.46.0

func (c *Client) HasMiddleware(mw Middleware) bool

HasMiddleware reports whether mw is already in the chain, compared by identity. Safe to call with a pointer-typed middleware (e.g. *TokenTracker): interface equality short-circuits on differing dynamic types, so it never compares — and never panics on — a non-comparable value already in the chain.

func (*Client) Stream

func (c *Client) Stream(ctx context.Context, req *Request) <-chan StreamEvent

Stream sends a streaming request to the resolved provider adapter. Middleware is NOT applied to streaming requests (middleware only wraps Complete). On provider resolution failure, it returns a channel containing a single error event.

type ClientOption

type ClientOption func(*clientConfig)

ClientOption configures a Client during construction.

func WithDefaultProvider

func WithDefaultProvider(name string) ClientOption

WithDefaultProvider sets the provider name used when a request does not specify one.

func WithMiddleware

func WithMiddleware(mw Middleware) ClientOption

WithMiddleware appends a middleware to the client's middleware chain.

func WithProvider

func WithProvider(adapter ProviderAdapter) ClientOption

WithProvider registers a provider adapter with the client.

type CompleteHandler

type CompleteHandler func(ctx context.Context, req *Request) (*Response, error)

CompleteHandler is a function that processes a completion request and returns a response. It is used as the unit of composition in the middleware chain.

type ConfigurationError

type ConfigurationError struct {
	SDKError
}

ConfigurationError indicates a problem with the client configuration.

type ContentFilterError

type ContentFilterError struct {
	ProviderError
}

ContentFilterError indicates the request or response was blocked by a content filter.

func (*ContentFilterError) Retryable

func (e *ContentFilterError) Retryable() bool

type ContentKind

type ContentKind string

ContentKind discriminates ContentPart variants.

const (
	KindText             ContentKind = "text"
	KindImage            ContentKind = "image"
	KindAudio            ContentKind = "audio"
	KindDocument         ContentKind = "document"
	KindToolCall         ContentKind = "tool_call"
	KindToolResult       ContentKind = "tool_result"
	KindThinking         ContentKind = "thinking"
	KindRedactedThinking ContentKind = "redacted_thinking"
)

type ContentPart

type ContentPart struct {
	Kind       ContentKind     `json:"kind"`
	Text       string          `json:"text,omitempty"`
	Image      *ImageData      `json:"image,omitempty"`
	Audio      *AudioData      `json:"audio,omitempty"`
	Document   *DocumentData   `json:"document,omitempty"`
	ToolCall   *ToolCallData   `json:"tool_call,omitempty"`
	ToolResult *ToolResultData `json:"tool_result,omitempty"`
	Thinking   *ThinkingData   `json:"thinking,omitempty"`
}

ContentPart is a tagged union representing one piece of message content.

type ContextLengthError

type ContextLengthError struct {
	ProviderError
}

ContextLengthError indicates the request exceeded the model's context window.

func (*ContextLengthError) Retryable

func (e *ContextLengthError) Retryable() bool

type DocumentData

type DocumentData struct {
	URL       string `json:"url,omitempty"`
	Data      []byte `json:"data,omitempty"`
	MediaType string `json:"media_type,omitempty"`
	FileName  string `json:"file_name,omitempty"`
}

DocumentData holds document content.

type FailoverEvent added in v0.47.0

type FailoverEvent struct {
	From Target // the exhausted lane
	To   Target // the lane being tried next
	Err  error  // the error that triggered the switch
}

FailoverEvent describes a single lane switch, for the caller to surface in the audit trail (which target served which node).

type FinishReason

type FinishReason struct {
	Reason string `json:"reason"`
	Raw    string `json:"raw,omitempty"`
}

FinishReason indicates why generation stopped.

type ImageData

type ImageData struct {
	URL       string `json:"url,omitempty"`
	Data      []byte `json:"data,omitempty"`
	MediaType string `json:"media_type,omitempty"`
	Detail    string `json:"detail,omitempty"`
}

ImageData holds image content as URL or raw bytes.

type InvalidRequestError

type InvalidRequestError struct {
	ProviderError
}

InvalidRequestError indicates a malformed or invalid request.

func (*InvalidRequestError) Retryable

func (e *InvalidRequestError) Retryable() bool

type InvalidToolCallError

type InvalidToolCallError struct {
	SDKError
}

InvalidToolCallError indicates the model produced an invalid tool call.

type Message

type Message struct {
	Role       Role          `json:"role"`
	Content    []ContentPart `json:"content"`
	Name       string        `json:"name,omitempty"`
	ToolCallID string        `json:"tool_call_id,omitempty"`
}

Message is the fundamental unit of conversation.

func AssistantMessage

func AssistantMessage(text string) Message

AssistantMessage creates a message with the assistant role.

func SystemMessage

func SystemMessage(text string) Message

SystemMessage creates a message with the system role.

func ToolResultMessage

func ToolResultMessage(toolCallID, content string, isError bool) Message

ToolResultMessage creates a tool result message.

func UserMessage

func UserMessage(text string) Message

UserMessage creates a message with the user role.

func (Message) Text

func (m Message) Text() string

Text returns concatenated text from all text content parts.

func (Message) ToolCalls

func (m Message) ToolCalls() []ToolCallData

ToolCalls extracts all tool call content parts from the message.

type Middleware

type Middleware interface {
	WrapComplete(next CompleteHandler) CompleteHandler
}

Middleware wraps a CompleteHandler to add cross-cutting behavior (logging, retry, etc.).

func NewRetryMiddleware

func NewRetryMiddleware(opts ...RetryOption) Middleware

NewRetryMiddleware creates a retry Middleware with the given options.

type ModelInfo

type ModelInfo struct {
	ID                string   `json:"id"`
	Provider          string   `json:"provider"`
	DisplayName       string   `json:"display_name"`
	ContextWindow     int      `json:"context_window"`
	MaxOutput         int      `json:"max_output"`
	SupportsTools     bool     `json:"supports_tools"`
	SupportsVision    bool     `json:"supports_vision"`
	SupportsReasoning bool     `json:"supports_reasoning"`
	InputCostPerM     float64  `json:"input_cost_per_m"`
	OutputCostPerM    float64  `json:"output_cost_per_m"`
	Aliases           []string `json:"aliases,omitempty"`
	// CacheReadMultiplier and CacheWriteMultiplier price cached prompt tokens
	// as a fraction of InputCostPerM. They live per model because the discount
	// is not a provider-wide convention: cached reads are 0.1x on Anthropic,
	// Gemini, and the GPT-5 family, 0.25x on GPT-4.1, and 0.5x on gpt-4o-mini.
	// Zero means "use the default" — see defaultCacheReadMultiplier — so a new
	// catalog entry prices cache traffic sanely without having to state both.
	CacheReadMultiplier  float64 `json:"cache_read_multiplier,omitempty"`
	CacheWriteMultiplier float64 `json:"cache_write_multiplier,omitempty"`
}

ModelInfo describes a known LLM model and its capabilities.

func GetModelInfo

func GetModelInfo(modelID string) *ModelInfo

GetModelInfo looks up a model by ID or alias. Returns nil if not found.

func ListModels

func ListModels(provider string) []ModelInfo

ListModels returns all known models, optionally filtered by provider. Pass an empty string to return all models.

type ModelResolver

type ModelResolver func(provider string) string

ModelResolver returns the model name that should be used for cost estimation for a given provider. Return "" when unknown — the entry is still included in the result with USD=0.

type NetworkError

type NetworkError struct {
	SDKError
}

NetworkError indicates a network-level failure (DNS, TCP, TLS).

func (*NetworkError) Retryable

func (e *NetworkError) Retryable() bool

type NoObjectGeneratedError

type NoObjectGeneratedError struct {
	SDKError
}

NoObjectGeneratedError indicates structured output generation failed.

type NotFoundError

type NotFoundError struct {
	ProviderError
}

NotFoundError indicates the requested resource does not exist.

func (*NotFoundError) Retryable

func (e *NotFoundError) Retryable() bool

type ProviderAdapter

type ProviderAdapter interface {
	// Name returns the provider's identifier (e.g. "openai", "anthropic").
	Name() string

	// Complete sends a request and returns the full response.
	Complete(ctx context.Context, req *Request) (*Response, error)

	// Stream sends a request and returns a channel of streaming events.
	Stream(ctx context.Context, req *Request) <-chan StreamEvent

	// Close releases any resources held by the adapter.
	Close() error
}

ProviderAdapter is the interface that LLM provider implementations must satisfy. It supports both synchronous completion and streaming responses.

type ProviderCost

type ProviderCost struct {
	Usage Usage
	Model string
	USD   float64
}

ProviderCost is the per-provider cost rollup returned by TokenTracker.CostByProvider. Usage and USD are aggregated across every model the provider ran; Model names the model used for display (the provider's last-observed model — see the note on ModelForProvider). USD is the sum of each (provider, model) bucket priced at its own rate, so a multi-model provider is no longer mispriced by a single last-write-wins model (#527).

type ProviderError

type ProviderError struct {
	SDKError
	Provider   string
	StatusCode int
	ErrorCode  string
	RetryAfter *float64
	RawBody    json.RawMessage
}

ProviderError is the base type for errors returned by LLM providers.

func (*ProviderError) GetProvider

func (e *ProviderError) GetProvider() string

func (*ProviderError) GetStatusCode

func (e *ProviderError) GetStatusCode() int

type ProviderErrorInterface

type ProviderErrorInterface interface {
	error
	Retryable() bool
	GetProvider() string
	GetStatusCode() int
}

ProviderErrorInterface describes errors originating from an LLM provider.

type QuotaExceededError

type QuotaExceededError struct {
	ProviderError
}

QuotaExceededError indicates the account's quota has been exhausted.

func (*QuotaExceededError) Retryable

func (e *QuotaExceededError) Retryable() bool

type RateLimitError

type RateLimitError struct {
	ProviderError
}

RateLimitError indicates the request was rate limited by the provider.

func (*RateLimitError) Retryable

func (e *RateLimitError) Retryable() bool

type RateLimitInfo

type RateLimitInfo struct {
	RequestsRemaining *int       `json:"requests_remaining,omitempty"`
	RequestsLimit     *int       `json:"requests_limit,omitempty"`
	TokensRemaining   *int       `json:"tokens_remaining,omitempty"`
	TokensLimit       *int       `json:"tokens_limit,omitempty"`
	ResetAt           *time.Time `json:"reset_at,omitempty"`
}

RateLimitInfo holds rate limit metadata from provider headers.

type Request

type Request struct {
	Model           string            `json:"model"`
	Messages        []Message         `json:"messages"`
	Provider        string            `json:"provider,omitempty"`
	Tools           []ToolDefinition  `json:"tools,omitempty"`
	ToolChoice      *ToolChoice       `json:"tool_choice,omitempty"`
	ResponseFormat  *ResponseFormat   `json:"response_format,omitempty"`
	Temperature     *float64          `json:"temperature,omitempty"`
	TopP            *float64          `json:"top_p,omitempty"`
	MaxTokens       *int              `json:"max_tokens,omitempty"`
	StopSequences   []string          `json:"stop_sequences,omitempty"`
	ReasoningEffort string            `json:"reasoning_effort,omitempty"`
	Metadata        map[string]string `json:"metadata,omitempty"`
	ProviderOptions map[string]any    `json:"provider_options,omitempty"`
	TraceObservers  []TraceObserver   `json:"-"`
}

Request is the input for Complete() and Stream().

type RequestTimeoutError

type RequestTimeoutError struct {
	ProviderError
}

RequestTimeoutError indicates the provider timed out processing the request.

func (*RequestTimeoutError) Retryable

func (e *RequestTimeoutError) Retryable() bool

type Response

type Response struct {
	ID           string          `json:"id"`
	Model        string          `json:"model"`
	Provider     string          `json:"provider"`
	Message      Message         `json:"message"`
	FinishReason FinishReason    `json:"finish_reason"`
	Usage        Usage           `json:"usage"`
	Latency      time.Duration   `json:"latency"`
	Raw          json.RawMessage `json:"raw,omitempty"`
	Warnings     []Warning       `json:"warnings,omitempty"`
	RateLimit    *RateLimitInfo  `json:"rate_limit,omitempty"`
}

Response is the output of Complete().

func (Response) Reasoning

func (r Response) Reasoning() string

Reasoning returns concatenated reasoning/thinking text.

func (Response) String

func (r Response) String() string

String formats a Response as a concise, human-readable summary.

func (Response) Text

func (r Response) Text() string

Text returns concatenated text from the response message.

func (Response) ToolCalls

func (r Response) ToolCalls() []ToolCallData

ToolCalls returns tool calls from the response message.

type ResponseFormat

type ResponseFormat struct {
	Type       string          `json:"type"`
	JSONSchema json.RawMessage `json:"json_schema,omitempty"`
	Strict     bool            `json:"strict,omitempty"`
}

ResponseFormat specifies output format constraints.

type RetryMiddleware

type RetryMiddleware struct {
	// contains filtered or unexported fields
}

RetryMiddleware retries failed requests with exponential backoff when the error is retryable. Non-retryable and unknown errors are returned immediately.

func (*RetryMiddleware) WrapComplete

func (rm *RetryMiddleware) WrapComplete(next CompleteHandler) CompleteHandler

WrapComplete implements the Middleware interface.

type RetryOption

type RetryOption func(*RetryMiddleware)

RetryOption configures a RetryMiddleware.

func WithBaseDelay

func WithBaseDelay(d time.Duration) RetryOption

WithBaseDelay sets the base delay for exponential backoff (default 1s). Actual delay is baseDelay * 2^attempt.

func WithMaxRetries

func WithMaxRetries(n int) RetryOption

WithMaxRetries sets the maximum number of retry attempts (default 3).

type Role

type Role string

Role represents who produced a message.

const (
	RoleSystem    Role = "system"
	RoleUser      Role = "user"
	RoleAssistant Role = "assistant"
	RoleTool      Role = "tool"
	RoleDeveloper Role = "developer"
)

type SDKError

type SDKError struct {
	Msg   string
	Cause error
}

SDKError is the base error type for all errors in the LLM package.

func (*SDKError) Error

func (e *SDKError) Error() string

func (*SDKError) Unwrap

func (e *SDKError) Unwrap() error

type ServerError

type ServerError struct {
	ProviderError
}

ServerError indicates a server-side failure from the provider.

func (*ServerError) Retryable

func (e *ServerError) Retryable() bool

type StreamAccumulator

type StreamAccumulator struct {
	// contains filtered or unexported fields
}

StreamAccumulator collects streaming events into a complete Response.

func NewStreamAccumulator

func NewStreamAccumulator() *StreamAccumulator

NewStreamAccumulator creates a new StreamAccumulator ready to process events.

func (*StreamAccumulator) Process

func (a *StreamAccumulator) Process(event StreamEvent)

Process handles a single StreamEvent, updating the accumulator state.

func (*StreamAccumulator) Response

func (a *StreamAccumulator) Response() Response

Response builds a complete Response from the accumulated events.

type StreamError

type StreamError struct {
	SDKError
}

StreamError indicates a failure during response streaming.

func (*StreamError) Retryable

func (e *StreamError) Retryable() bool

type StreamEvent

type StreamEvent struct {
	Type               StreamEventType `json:"type"`
	Delta              string          `json:"delta,omitempty"`
	TextID             string          `json:"text_id,omitempty"`
	ReasoningDelta     string          `json:"reasoning_delta,omitempty"`
	ReasoningSignature string          `json:"reasoning_signature,omitempty"`
	ToolCall           *ToolCallData   `json:"tool_call,omitempty"`
	FinishReason       *FinishReason   `json:"finish_reason,omitempty"`
	Usage              *Usage          `json:"usage,omitempty"`
	FullResponse       *Response       `json:"full_response,omitempty"`
	Err                error           `json:"-"`
	Raw                json.RawMessage `json:"raw,omitempty"`
	// RequestRaw is the verbatim wire body the adapter sent, carried on the
	// EventRequestSent event that EmitRequestSent emits below. EventStreamStart
	// only ever sees it via TraceBuilder's fallback, never set by an adapter.
	// The normalized Request records what tracker asked for; this records what
	// actually went out after provider-specific translation and ProviderOptions
	// merging — which is what a post-hoc reader needs to reproduce the call.
	RequestRaw json.RawMessage `json:"request_raw,omitempty"`
}

StreamEvent represents a single event in a streaming response.

type StreamEventType

type StreamEventType string

StreamEventType discriminates the kind of streaming event.

const (
	EventStreamStart        StreamEventType = "stream_start"
	EventTextStart          StreamEventType = "text_start"
	EventTextDelta          StreamEventType = "text_delta"
	EventTextEnd            StreamEventType = "text_end"
	EventReasoningStart     StreamEventType = "reasoning_start"
	EventReasoningDelta     StreamEventType = "reasoning_delta"
	EventReasoningSignature StreamEventType = "reasoning_signature"
	EventReasoningEnd       StreamEventType = "reasoning_end"
	EventRedactedThinking   StreamEventType = "redacted_thinking"
	EventToolCallStart      StreamEventType = "tool_call_start"
	EventToolCallDelta      StreamEventType = "tool_call_delta"
	EventToolCallEnd        StreamEventType = "tool_call_end"
	EventFinish             StreamEventType = "finish"
	EventError              StreamEventType = "error"
	EventProviderEvent      StreamEventType = "provider_event"
	// EventRequestSent carries the verbatim wire body in RequestRaw. Adapters
	// emit it once, immediately after building the body and before the HTTP
	// call, because that is the only place the body is in scope — the
	// provider's own stream-start arrives several layers deeper, inside SSE
	// parsing. TraceBuilder folds it into the TraceRequestStart it emits, and
	// StreamAccumulator ignores it, so it adds no event to either output.
	EventRequestSent StreamEventType = "request_sent"
)

type Target added in v0.47.0

type Target struct {
	Provider string
	Model    string
}

Target is one provider+model lane for failover.

type ThinkingData

type ThinkingData struct {
	Text      string `json:"text"`
	Signature string `json:"signature,omitempty"`
	Redacted  bool   `json:"redacted,omitempty"`
}

ThinkingData holds model reasoning content.

type TokenTracker

type TokenTracker struct {
	// contains filtered or unexported fields
}

TokenTracker is a middleware that accumulates token usage per (provider, model) pair. It implements Middleware and can be passed to NewClient via WithMiddleware.

func NewTokenTracker

func NewTokenTracker() *TokenTracker

NewTokenTracker creates a new, zeroed token tracking middleware.

func (*TokenTracker) AddUsage

func (t *TokenTracker) AddUsage(provider string, usage Usage, model ...string)

AddUsage manually adds token usage for a provider. Used by backends that bypass the LLM client middleware (e.g., claude-code subprocess backend). The model parameter is optional; pass "" to leave the provider's model unchanged. Model strings are normalized through the catalog so versioned provider-returned IDs resolve to canonical pricing entries — matching WrapComplete's behavior.

func (*TokenTracker) AllProviderUsage

func (t *TokenTracker) AllProviderUsage() map[string]Usage

AllProviderUsage returns a copy of the accumulated usage aggregated per provider (summed across each provider's models).

func (*TokenTracker) CostByProvider

func (t *TokenTracker) CostByProvider(resolve ModelResolver) map[string]ProviderCost

CostByProvider returns a per-provider cost rollup. Each (provider, model) bucket is priced at its own model's rate and the results are summed per provider, so a provider that ran multiple models is priced correctly and the total no longer depends on call ordering (#527). The caller-supplied resolver only prices buckets whose model was never observed ("" — e.g. subscription backends); a nil resolver treats those as $0.

func (*TokenTracker) ModelForProvider

func (t *TokenTracker) ModelForProvider(provider string) string

ModelForProvider returns the last-seen model for a provider. Returns "" if unknown. NOTE: a provider that ran multiple models has multiple buckets; this reports only the most-recently-observed one and is retained for back-compat (fallback resolution / display). For accurate multi-model pricing use CostByProvider, which prices each model bucket at its own rate.

func (*TokenTracker) ObservedModelResolver

func (t *TokenTracker) ObservedModelResolver(fallback string) ModelResolver

ObservedModelResolver returns a ModelResolver that uses the tracker's last-observed per-provider model, falling back to the provided fallback model for providers where no model was observed. Since CostByProvider now prices each (provider, model) bucket at its own rate, this resolver only supplies a price for buckets whose model is unknown ("" — e.g. subscription-auth backends). For a provider that ran multiple models it reports the last one.

func (*TokenTracker) ProviderUsage

func (t *TokenTracker) ProviderUsage(provider string) Usage

ProviderUsage returns the accumulated usage for a specific provider, summed across every model that provider ran. Returns a zero Usage if the provider has not been seen.

func (*TokenTracker) Providers

func (t *TokenTracker) Providers() []string

Providers returns a sorted, de-duplicated slice of provider names that have recorded usage (a provider that ran multiple models appears once).

func (*TokenTracker) TotalCostUSD

func (t *TokenTracker) TotalCostUSD(resolve ModelResolver) float64

TotalCostUSD sums CostByProvider to a single dollar figure using the same resolver.

func (*TokenTracker) TotalUsage

func (t *TokenTracker) TotalUsage() Usage

TotalUsage returns accumulated usage summed across all providers and models.

func (*TokenTracker) WrapComplete

func (t *TokenTracker) WrapComplete(next CompleteHandler) CompleteHandler

WrapComplete implements the Middleware interface. It calls the next handler and, on success, adds the response's token usage to the per-(provider, model) accumulator.

type ToolCallData

type ToolCallData struct {
	ID             string          `json:"id"`
	Name           string          `json:"name"`
	Arguments      json.RawMessage `json:"arguments"`
	ThoughtSigData string          `json:"thought_signature,omitempty"`
}

ToolCallData represents a model-initiated tool invocation.

type ToolChoice

type ToolChoice struct {
	Mode     string `json:"mode"`
	ToolName string `json:"tool_name,omitempty"`
}

ToolChoice controls how the model uses tools.

func ToolChoiceAuto

func ToolChoiceAuto() ToolChoice

ToolChoiceAuto returns a ToolChoice that lets the model decide whether to call tools.

func ToolChoiceNamed

func ToolChoiceNamed(name string) ToolChoice

ToolChoiceNamed creates a ToolChoice that forces the model to call a specific tool.

func ToolChoiceNone

func ToolChoiceNone() ToolChoice

ToolChoiceNone returns a ToolChoice that prevents the model from calling tools.

func ToolChoiceRequired

func ToolChoiceRequired() ToolChoice

ToolChoiceRequired returns a ToolChoice that forces the model to call a tool.

type ToolDefinition

type ToolDefinition struct {
	Name        string          `json:"name"`
	Description string          `json:"description"`
	Parameters  json.RawMessage `json:"parameters"`
}

ToolDefinition defines a tool the model can call.

type ToolResultData

type ToolResultData struct {
	ToolCallID     string `json:"tool_call_id"`
	Name           string `json:"name,omitempty"`
	Content        string `json:"content"`
	IsError        bool   `json:"is_error"`
	ImageData      []byte `json:"image_data,omitempty"`
	ImageMediaType string `json:"image_media_type,omitempty"`
}

ToolResultData represents the result of executing a tool call.

type TraceBuilder

type TraceBuilder struct {
	// contains filtered or unexported fields
}

TraceBuilder converts streaming events into normalized trace events.

func NewTraceBuilder

func NewTraceBuilder(opts TraceOptions) *TraceBuilder

NewTraceBuilder creates a trace builder for one request.

func (*TraceBuilder) Events

func (b *TraceBuilder) Events() []TraceEvent

Events returns the trace events emitted so far.

func (*TraceBuilder) Process

func (b *TraceBuilder) Process(evt StreamEvent)

Process ingests one stream event and emits any corresponding trace events.

type TraceEvent

type TraceEvent struct {
	Kind          TraceKind
	Provider      string
	Model         string
	ToolName      string
	Preview       string
	ProviderEvent string
	RawPreview    string
	FinishReason  string
	Usage         Usage
	// CallID groups every event belonging to one LLM request. Two log paths
	// record LLM activity — the agent session re-emits trace events as agent
	// llm_* events, and the client-level writer catches calls no session sees
	// (the autopilot interviewer, for one) — so one call can legitimately
	// appear twice. Both paths are kept because their coverage differs; this
	// is what lets a reader collapse the overlap rather than double-count
	// usage when aggregating.
	CallID string
	// RequestRaw is the verbatim wire body, set on TraceRequestStart. The
	// normalized Request says what tracker asked for; this says what actually
	// went out after provider translation and ProviderOptions merging.
	RequestRaw json.RawMessage
	// ToolArguments is the untruncated tool-call arguments, set on
	// TraceToolPrepare. Preview holds the same value clipped to 80 chars,
	// which is enough to render but not enough to reconstruct the call.
	ToolArguments json.RawMessage
	// ProviderRaw is the untruncated provider chunk, set on TraceProviderRaw
	// (verbose only). RawPreview holds the clipped form.
	ProviderRaw json.RawMessage
	// SessionOwned is true when the originating request carried
	// request-level TraceObservers — in tracker only the agent session
	// registers those, and it re-emits every trace event as an agent
	// llm_* event. Activity-log writers listening at the client level
	// skip SessionOwned events to avoid logging the same stream twice
	// (issue #354); non-session calls (e.g. the autopilot interviewer)
	// stay SessionOwned=false so the trace path remains their only,
	// and therefore kept, log surface.
	SessionOwned bool
}

TraceEvent is a normalized event for rendering live LLM activity.

Preview and RawPreview are display fields: previewText collapses newlines and clips to tracePreviewLimit (80 chars) so a TUI line stays a TUI line. The Raw* / Arguments fields alongside them carry the same content untruncated, for the activity log. Text and reasoning deltas are not clipped in either place — preserveSpacingText keeps them whole because coalescing clipped deltas would corrupt the reconstructed text.

type TraceKind

type TraceKind string

TraceKind identifies a normalized LLM trace event.

const (
	TraceRequestStart TraceKind = "request_start"
	TraceReasoning    TraceKind = "reasoning"
	TraceText         TraceKind = "text"
	TraceToolPrepare  TraceKind = "tool_prepare"
	TraceFinish       TraceKind = "finish"
	TraceProviderRaw  TraceKind = "provider_raw"
)

type TraceLogger

type TraceLogger struct {
	// contains filtered or unexported fields
}

TraceLogger writes trace events to an io.Writer. Sequential text and reasoning deltas are accumulated and flushed as a single summary line when the stream finishes or a different event kind arrives.

func NewTraceLogger

func NewTraceLogger(w io.Writer, opts TraceLoggerOptions) *TraceLogger

NewTraceLogger creates a console trace logger.

func (*TraceLogger) Flush

func (l *TraceLogger) Flush()

Flush forces any pending batched output to be written. Callers should invoke this when a stream terminates without a normal finish event (e.g. on error) to avoid losing buffered trace output.

func (*TraceLogger) HandleTraceEvent

func (l *TraceLogger) HandleTraceEvent(evt TraceEvent)

HandleTraceEvent implements TraceObserver.

type TraceLoggerOptions

type TraceLoggerOptions struct {
	Verbose bool
}

TraceLoggerOptions configure console trace logging.

type TraceObserver

type TraceObserver interface {
	HandleTraceEvent(evt TraceEvent)
}

TraceObserver receives normalized LLM trace events.

type TraceObserverFunc

type TraceObserverFunc func(evt TraceEvent)

TraceObserverFunc adapts a function into a TraceObserver.

func (TraceObserverFunc) HandleTraceEvent

func (f TraceObserverFunc) HandleTraceEvent(evt TraceEvent)

HandleTraceEvent implements TraceObserver.

type TraceOptions

type TraceOptions struct {
	Provider string
	Model    string
	Verbose  bool
	// CallID identifies the one request this builder traces. It is stamped
	// onto every event the builder emits so a reader can group them, and —
	// more importantly — collapse the overlap between the two log paths that
	// both record LLM activity (see TraceEvent.CallID).
	CallID string
}

TraceOptions configure trace building behavior.

type Usage

type Usage struct {
	InputTokens      int     `json:"input_tokens"`
	OutputTokens     int     `json:"output_tokens"`
	TotalTokens      int     `json:"total_tokens"`
	ReasoningTokens  *int    `json:"reasoning_tokens,omitempty"`
	CacheReadTokens  *int    `json:"cache_read_tokens,omitempty"`
	CacheWriteTokens *int    `json:"cache_write_tokens,omitempty"`
	EstimatedCost    float64 `json:"estimated_cost,omitempty"`
	Raw              any     `json:"raw,omitempty"`
}

Usage tracks token consumption.

The field names read as provider-neutral but the accounting rules are not obvious, and providers disagree about what nests inside what. Adapters are responsible for normalizing into the two invariants below; EstimateCost depends on them, so getting one wrong misprices every run on that provider.

InputTokens EXCLUDES cached tokens. CacheReadTokens and CacheWriteTokens
are separate additive buckets, so InputTokens + CacheReadTokens +
CacheWriteTokens is the full prompt. Anthropic reports this shape natively.
OpenAI and Gemini do not — their prompt counts INCLUDE the cached portion,
so those adapters must subtract it out. Wiring their cached count straight
into CacheReadTokens double-counts the prompt.

OutputTokens INCLUDES reasoning tokens. ReasoningTokens is an informational
subset, never priced separately — pricing reasoning on top of OutputTokens
would bill it twice. OpenAI reports this shape natively. Gemini does not:
its candidate count excludes thinking tokens that it nonetheless bills, so
that adapter must add them in.

func (Usage) Add

func (u Usage) Add(other Usage) Usage

Add combines two Usage values.

func (Usage) String

func (u Usage) String() string

String formats Usage as a concise token summary.

type Warning

type Warning struct {
	Message string `json:"message"`
	Code    string `json:"code,omitempty"`
}

Warning represents a non-fatal issue.

Directories

Path Synopsis
ABOUTME: Anthropic Messages API adapter implementing the ProviderAdapter interface.
ABOUTME: Anthropic Messages API adapter implementing the ProviderAdapter interface.
ABOUTME: Google Gemini API adapter implementing the ProviderAdapter interface.
ABOUTME: Google Gemini API adapter implementing the ProviderAdapter interface.
ABOUTME: OpenAI Responses API adapter implementing the ProviderAdapter interface.
ABOUTME: OpenAI Responses API adapter implementing the ProviderAdapter interface.
ABOUTME: OpenAI Chat Completions compatible adapter implementing the ProviderAdapter interface.
ABOUTME: OpenAI Chat Completions compatible adapter implementing the ProviderAdapter interface.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL