llm

package
v1.4.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 27, 2026 License: Apache-2.0 Imports: 18 Imported by: 0

Documentation

Index

Constants

View Source
const (
	// DefaultToolOutputMaxBytes is the default cap for tool output content.
	DefaultToolOutputMaxBytes = 50_000
)

Variables

ModelContextLimits defines the maximum prompt token budget per model. These are set conservatively below the absolute API limits to leave headroom for token-estimation inaccuracy and output tokens.

View Source
var ModelLimitsMap = map[constants.Model]ModelLimits{
	constants.ModelClaudeOpus48:   {ContextLimit: 180000, OutputReserve: 8192},
	constants.ModelClaudeSonnet46: {ContextLimit: 180000, OutputReserve: 8192},
	constants.ModelClaudeSonnet4:  {ContextLimit: 180000, OutputReserve: 8192},
	constants.ModelClaudeHaiku45:  {ContextLimit: 180000, OutputReserve: 8192},
	constants.ModelGPT55:          {ContextLimit: 115000, OutputReserve: 4096},
	constants.ModelGPT4o:          {ContextLimit: 115000, OutputReserve: 4096},
	constants.ModelGPT4oMini:      {ContextLimit: 115000, OutputReserve: 4096},
}

ModelLimitsMap provides structured limits per model, including output reservation.

View Source
var ToolOutputLimits = map[string]int{
	"fetch_url": 30_000,
	"read_doc":  30_000,
}

ToolOutputLimits maps tool names to their specific output byte caps. Tools not listed here use DefaultToolOutputMaxBytes.

Functions

func ContextBudgetRemaining

func ContextBudgetRemaining(system string, messages []Message, tools []ToolDefinition, model string) int

ContextBudgetRemaining returns the estimated tokens remaining before hitting the proactive compaction threshold (85% of context limit).

func EstimateAllMessages

func EstimateAllMessages(messages []Message) int

func EstimateContextUsage

func EstimateContextUsage(system string, messages []Message, tools []ToolDefinition) int

EstimateContextUsage estimates the total token usage for a set of messages, system prompt, and tool definitions.

func EstimateTokenCostCents

func EstimateTokenCostCents(inputTokens, outputTokens int, model string) int64

EstimateTokenCostCents returns the estimated cost in cents for the given token counts and model. Uses conservative (most expensive) pricing as fallback for unknown models.

func EstimateTokens

func EstimateTokens(s string) int

EstimateTokens returns a rough token count for a string. Uses ~4 characters per token, which is conservative for English/code.

func ExtractErrorMessage

func ExtractErrorMessage(body string) string

ExtractErrorMessage attempts to extract a human-readable error message from the gateway response body. Falls back to the raw body if parsing fails.

func GatewayModelName

func GatewayModelName(model string) string

GatewayModelName prefixes a model name with its provider for the Stripe AI Gateway. Model names already match Stripe's naming convention (e.g. "claude-sonnet-4", "gpt-4o").

func MarkedUpTokenCostCents

func MarkedUpTokenCostCents(inputTokens, outputTokens int, gatewayModel string, rates map[TokenRateKey]float64) (int64, bool)

MarkedUpTokenCostCents prices a turn's tokens at the plan's marked-up rate card rates — the same figures Stripe bills — so the in-run spending-cap gate matches the customer's actual bill. gatewayModel is the provider-prefixed name the rate card is keyed on (see GatewayModelName). Returns (cost, true) when both input and output rates are present; (0, false) when a rate is missing so the caller can fall back to EstimateTokenCostCents. Cached input/output are billed cheaper by Stripe but aren't tracked per turn, so charging them at the (higher) input/output rate keeps the gate conservative.

func NeedsProactiveCompaction

func NeedsProactiveCompaction(system string, messages []Message, tools []ToolDefinition, model string) bool

NeedsProactiveCompaction returns true if the estimated token usage exceeds 85% of the model's context limit, indicating compaction should be triggered before the next LLM call to avoid a hard context overflow error.

func StripeCustomerIDFromContext

func StripeCustomerIDFromContext(ctx context.Context) string

func TruncateToolOutput

func TruncateToolOutput(content string, toolName string) string

TruncateToolOutput caps the content of a tool result at the given byte limit. When truncated, it preserves the first (limit - tail) bytes and the last tail bytes, inserting a truncation notice in between.

func WithStripeCustomerID

func WithStripeCustomerID(ctx context.Context, id string) context.Context

Types

type AnthropicMessagesProvider

type AnthropicMessagesProvider struct {
	// contains filtered or unexported fields
}

AnthropicMessagesProvider calls Anthropic's native Messages API through the Stripe AI Gateway passthrough (https://llm.stripe.com/v1/messages). Going native — rather than the OpenAI-compatible /chat/completions path used by GatewayProvider — gives us real thinking blocks: a distinct reasoning channel that streams token-by-token and carries signatures, so reasoning can be replayed across tool-loop turns. Stripe still meters usage per customer via the X-Stripe-Customer-ID header, so billing is unchanged.

func NewAnthropicMessagesProvider

func NewAnthropicMessagesProvider(stripeAPIKey string) *AnthropicMessagesProvider

func (*AnthropicMessagesProvider) CompleteWithTools

func (p *AnthropicMessagesProvider) CompleteWithTools(ctx context.Context, req *ToolRequest) (*ToolResponse, error)

func (*AnthropicMessagesProvider) Name

func (*AnthropicMessagesProvider) StreamCompleteWithTools

func (p *AnthropicMessagesProvider) StreamCompleteWithTools(ctx context.Context, req *ToolRequest, callback func(StreamEvent)) (*ToolResponse, error)

type GatewayError

type GatewayError struct {
	StatusCode int
	Body       string
	Headers    http.Header
	Retryable  bool
}

GatewayError is a structured error returned when the LLM gateway responds with a non-200 status code. It classifies errors as retryable or not and extracts the Retry-After header when present.

func NewGatewayError

func NewGatewayError(statusCode int, body string, headers http.Header) *GatewayError

NewGatewayError creates a GatewayError from an HTTP response, classifying whether the error is retryable based on the status code and response body.

func (*GatewayError) Error

func (e *GatewayError) Error() string

func (*GatewayError) IsBillingLimitError

func (e *GatewayError) IsBillingLimitError() bool

IsBillingLimitError reports whether the gateway rejected the call because the customer's billing/spend limit was reached (a 402 Payment Required, or a 4xx whose body indicates a billing/quota/budget problem). These are account-wide: every model and provider routes through the same Stripe customer, so retrying or failing over to another model won't help — the run must stop.

func (*GatewayError) IsContextLengthError

func (e *GatewayError) IsContextLengthError() bool

IsContextLengthError returns true if the error body indicates the input exceeded the model's context window. These errors should not be retried.

func (*GatewayError) RetryAfter

func (e *GatewayError) RetryAfter() time.Duration

RetryAfter returns the duration to wait before retrying, based on the Retry-After header. Returns 0 if the header is absent or unparseable.

type GatewayProvider

type GatewayProvider struct {
	// contains filtered or unexported fields
}

GatewayProvider routes all LLM calls through the Stripe AI Gateway, which uses the OpenAI-compatible /chat/completions endpoint for all providers.

func NewGatewayProvider

func NewGatewayProvider(stripeAPIKey string) *GatewayProvider

func (*GatewayProvider) CompleteWithTools

func (p *GatewayProvider) CompleteWithTools(ctx context.Context, req *ToolRequest) (*ToolResponse, error)

func (*GatewayProvider) Name

func (p *GatewayProvider) Name() string

func (*GatewayProvider) StreamCompleteWithTools

func (p *GatewayProvider) StreamCompleteWithTools(ctx context.Context, req *ToolRequest, callback func(StreamEvent)) (*ToolResponse, error)

type LLMProvider

type LLMProvider interface {
	CompleteWithTools(ctx context.Context, req *ToolRequest) (*ToolResponse, error)
	Name() string
}

LLMProvider abstracts multi-turn LLM interactions with tool use.

type Message

type Message struct {
	Role        string            `json:"role"`
	Content     string            `json:"content,omitempty"`
	Thinking    []ThinkingBlock   `json:"thinking,omitempty"`
	ToolUse     []ToolUseBlock    `json:"tool_use,omitempty"`
	ToolResults []ToolResultBlock `json:"tool_results,omitempty"`
}

Message represents a single conversation turn.

func CopyMessages

func CopyMessages(messages []Message) []Message

func TruncateMessages

func TruncateMessages(system string, messages []Message, tools []ToolDefinition, model string) []Message

TruncateMessages ensures the total prompt fits within the model's context limit.

Truncation is applied in priority order to minimize information loss:

  1. Truncate assistant message content (thinking/reasoning — most expendable)
  2. Cap oversized tool result content
  3. Cap oversized tool call inputs
  4. Drop old assistant+tool messages from the middle (user messages preserved)
  5. Drop old messages from the middle as a last resort

type ModelLimits

type ModelLimits struct {
	ContextLimit  int // maximum prompt tokens
	OutputReserve int // tokens reserved for output generation
}

ModelLimits defines the context window and output token reservation per model.

func GetModelLimits

func GetModelLimits(model string) ModelLimits

GetModelLimits returns the ModelLimits for the given model, falling back to defaults.

type StreamEvent

type StreamEvent struct {
	Type           string // "reasoning_delta", "content_delta", "tool_call_delta", "done"
	ContentDelta   string
	ReasoningDelta string
	ToolIndex      int
	ToolCallID     string
	ToolName       string
	ArgumentsDelta string
	FinishReason   string
	InputTokens    int
	OutputTokens   int
}

StreamEvent represents a single chunk from a streaming LLM response. "reasoning_delta" carries native provider reasoning (Anthropic thinking blocks, or the OpenAI-compatible reasoning_content field) in ReasoningDelta — a channel distinct from "content_delta", which now carries only the user-facing answer text. "tool_call_delta" is emitted both at a tool block's start — carrying ToolName/ToolCallID so a live indicator can name the tool — and for each subsequent argument chunk in ArgumentsDelta (where ToolName is empty, as the arguments stream after the name on both providers).

type StreamingLLMProvider

type StreamingLLMProvider interface {
	LLMProvider
	StreamCompleteWithTools(ctx context.Context, req *ToolRequest, callback func(StreamEvent)) (*ToolResponse, error)
}

StreamingLLMProvider extends LLMProvider with streaming support.

type ThinkingBlock

type ThinkingBlock struct {
	Text      string `json:"text"`
	Signature string `json:"signature,omitempty"`
}

ThinkingBlock carries a provider reasoning block together with its signature so it can be replayed to the model on later tool-loop turns. Anthropic requires preserved, unmodified thinking blocks when continuing a turn after tool_use under interleaved thinking; providers that don't return signatures (the OpenAI-compatible path) leave Signature empty, and such blocks are not replayed.

type TokenRateKey

type TokenRateKey struct {
	Model     string
	TokenType string
}

TokenRateKey identifies a marked-up rate by the gateway model name and token type it applies to.

type ToolCall

type ToolCall struct {
	ID    string
	Name  string
	Input json.RawMessage
}

ToolCall represents a single tool invocation from the LLM.

type ToolDefinition

type ToolDefinition struct {
	Name        string          `json:"name"`
	Description string          `json:"description"`
	InputSchema json.RawMessage `json:"input_schema"`
}

ToolDefinition describes a tool available to the LLM.

type ToolRequest

type ToolRequest struct {
	Model       string
	System      string
	Messages    []Message
	Tools       []ToolDefinition
	MaxTokens   int
	Temperature float64
	// EnableReasoning turns on the provider's reasoning output — adaptive thinking on the native Anthropic path. Chat runs set it so the live thinking panel has content; other runs leave it off to avoid the extra thinking-token cost.
	EnableReasoning bool
	// ReasoningEffort, when set ("low"/"medium"/"high"), asks reasoning-capable models on the OpenAI-compatible path to emit reasoning. Ignored by the native Anthropic provider.
	ReasoningEffort string
}

ToolRequest is the input to a provider's CompleteWithTools call.

type ToolResponse

type ToolResponse struct {
	Content      string
	Thinking     []ThinkingBlock // native reasoning blocks (Anthropic); empty for the OpenAI-compat path
	ToolCalls    []ToolCall
	InputTokens  int
	OutputTokens int
	StopReason   string // "end_turn", "tool_use", "max_tokens"
}

ToolResponse is the normalized output from a provider.

type ToolResultBlock

type ToolResultBlock struct {
	ToolUseID string `json:"tool_use_id"`
	Content   string `json:"content"`
	IsError   bool   `json:"is_error,omitempty"`
}

ToolResultBlock represents the result of a tool execution.

type ToolUseBlock

type ToolUseBlock struct {
	ID    string          `json:"id"`
	Name  string          `json:"name"`
	Input json.RawMessage `json:"input"`
}

ToolUseBlock represents an assistant's request to use a tool.

type TruncationResult

type TruncationResult struct {
	Content      string
	WasTruncated bool
	FullLength   int
}

TruncationResult holds the outcome of truncating a tool output.

func TruncateToolOutputResult

func TruncateToolOutputResult(content string, toolName string) TruncationResult

TruncateToolOutputResult is like TruncateToolOutput but returns a TruncationResult indicating whether truncation occurred and the original content length.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL