Documentation
¶
Index ¶
- Constants
- Variables
- func ContextBudgetRemaining(system string, messages []Message, tools []ToolDefinition, model string) int
- func EstimateAllMessages(messages []Message) int
- func EstimateContextUsage(system string, messages []Message, tools []ToolDefinition) int
- func EstimateTokenCostCents(inputTokens, outputTokens int, model string) int64
- func EstimateTokens(s string) int
- func ExtractErrorMessage(body string) string
- func GatewayModelName(model string) string
- func MarkedUpTokenCostCents(inputTokens, outputTokens int, gatewayModel string, ...) (int64, bool)
- func NeedsProactiveCompaction(system string, messages []Message, tools []ToolDefinition, model string) bool
- func StripeCustomerIDFromContext(ctx context.Context) string
- func TruncateToolOutput(content string, toolName string) string
- func WithStripeCustomerID(ctx context.Context, id string) context.Context
- type AnthropicMessagesProvider
- func (p *AnthropicMessagesProvider) CompleteWithTools(ctx context.Context, req *ToolRequest) (*ToolResponse, error)
- func (p *AnthropicMessagesProvider) Name() string
- func (p *AnthropicMessagesProvider) StreamCompleteWithTools(ctx context.Context, req *ToolRequest, callback func(StreamEvent)) (*ToolResponse, error)
- type GatewayError
- type GatewayProvider
- type LLMProvider
- type Message
- type ModelLimits
- type StreamEvent
- type StreamingLLMProvider
- type ThinkingBlock
- type TokenRateKey
- type ToolCall
- type ToolDefinition
- type ToolRequest
- type ToolResponse
- type ToolResultBlock
- type ToolUseBlock
- type TruncationResult
Constants ¶
const (
// DefaultToolOutputMaxBytes is the default cap for tool output content.
DefaultToolOutputMaxBytes = 50_000
)
Variables ¶
var ModelContextLimits = map[constants.Model]int{ constants.ModelClaudeOpus48: 180000, constants.ModelClaudeSonnet46: 180000, constants.ModelClaudeSonnet4: 180000, constants.ModelClaudeHaiku45: 180000, constants.ModelGPT55: 115000, constants.ModelGPT4o: 115000, constants.ModelGPT4oMini: 115000, }
ModelContextLimits defines the maximum prompt token budget per model. These are set conservatively below the absolute API limits to leave headroom for token-estimation inaccuracy and output tokens.
var ModelLimitsMap = map[constants.Model]ModelLimits{ constants.ModelClaudeOpus48: {ContextLimit: 180000, OutputReserve: 8192}, constants.ModelClaudeSonnet46: {ContextLimit: 180000, OutputReserve: 8192}, constants.ModelClaudeSonnet4: {ContextLimit: 180000, OutputReserve: 8192}, constants.ModelClaudeHaiku45: {ContextLimit: 180000, OutputReserve: 8192}, constants.ModelGPT55: {ContextLimit: 115000, OutputReserve: 4096}, constants.ModelGPT4o: {ContextLimit: 115000, OutputReserve: 4096}, constants.ModelGPT4oMini: {ContextLimit: 115000, OutputReserve: 4096}, }
ModelLimitsMap provides structured limits per model, including output reservation.
var ToolOutputLimits = map[string]int{
"fetch_url": 30_000,
"read_doc": 30_000,
}
ToolOutputLimits maps tool names to their specific output byte caps. Tools not listed here use DefaultToolOutputMaxBytes.
Functions ¶
func ContextBudgetRemaining ¶
func ContextBudgetRemaining(system string, messages []Message, tools []ToolDefinition, model string) int
ContextBudgetRemaining returns the estimated tokens remaining before hitting the proactive compaction threshold (85% of context limit).
func EstimateAllMessages ¶
func EstimateContextUsage ¶
func EstimateContextUsage(system string, messages []Message, tools []ToolDefinition) int
EstimateContextUsage estimates the total token usage for a set of messages, system prompt, and tool definitions.
func EstimateTokenCostCents ¶
EstimateTokenCostCents returns the estimated cost in cents for the given token counts and model. Uses conservative (most expensive) pricing as fallback for unknown models.
func EstimateTokens ¶
EstimateTokens returns a rough token count for a string. Uses ~4 characters per token, which is conservative for English/code.
func ExtractErrorMessage ¶
ExtractErrorMessage attempts to extract a human-readable error message from the gateway response body. Falls back to the raw body if parsing fails.
func GatewayModelName ¶
GatewayModelName prefixes a model name with its provider for the Stripe AI Gateway. Model names already match Stripe's naming convention (e.g. "claude-sonnet-4", "gpt-4o").
func MarkedUpTokenCostCents ¶
func MarkedUpTokenCostCents(inputTokens, outputTokens int, gatewayModel string, rates map[TokenRateKey]float64) (int64, bool)
MarkedUpTokenCostCents prices a turn's tokens at the plan's marked-up rate card rates — the same figures Stripe bills — so the in-run spending-cap gate matches the customer's actual bill. gatewayModel is the provider-prefixed name the rate card is keyed on (see GatewayModelName). Returns (cost, true) when both input and output rates are present; (0, false) when a rate is missing so the caller can fall back to EstimateTokenCostCents. Cached input/output are billed cheaper by Stripe but aren't tracked per turn, so charging them at the (higher) input/output rate keeps the gate conservative.
func NeedsProactiveCompaction ¶
func NeedsProactiveCompaction(system string, messages []Message, tools []ToolDefinition, model string) bool
NeedsProactiveCompaction returns true if the estimated token usage exceeds 85% of the model's context limit, indicating compaction should be triggered before the next LLM call to avoid a hard context overflow error.
func TruncateToolOutput ¶
TruncateToolOutput caps the content of a tool result at the given byte limit. When truncated, it preserves the first (limit - tail) bytes and the last tail bytes, inserting a truncation notice in between.
Types ¶
type AnthropicMessagesProvider ¶
type AnthropicMessagesProvider struct {
// contains filtered or unexported fields
}
AnthropicMessagesProvider calls Anthropic's native Messages API through the Stripe AI Gateway passthrough (https://llm.stripe.com/v1/messages). Going native — rather than the OpenAI-compatible /chat/completions path used by GatewayProvider — gives us real thinking blocks: a distinct reasoning channel that streams token-by-token and carries signatures, so reasoning can be replayed across tool-loop turns. Stripe still meters usage per customer via the X-Stripe-Customer-ID header, so billing is unchanged.
func NewAnthropicMessagesProvider ¶
func NewAnthropicMessagesProvider(stripeAPIKey string) *AnthropicMessagesProvider
func (*AnthropicMessagesProvider) CompleteWithTools ¶
func (p *AnthropicMessagesProvider) CompleteWithTools(ctx context.Context, req *ToolRequest) (*ToolResponse, error)
func (*AnthropicMessagesProvider) Name ¶
func (p *AnthropicMessagesProvider) Name() string
func (*AnthropicMessagesProvider) StreamCompleteWithTools ¶
func (p *AnthropicMessagesProvider) StreamCompleteWithTools(ctx context.Context, req *ToolRequest, callback func(StreamEvent)) (*ToolResponse, error)
type GatewayError ¶
GatewayError is a structured error returned when the LLM gateway responds with a non-200 status code. It classifies errors as retryable or not and extracts the Retry-After header when present.
func NewGatewayError ¶
func NewGatewayError(statusCode int, body string, headers http.Header) *GatewayError
NewGatewayError creates a GatewayError from an HTTP response, classifying whether the error is retryable based on the status code and response body.
func (*GatewayError) Error ¶
func (e *GatewayError) Error() string
func (*GatewayError) IsBillingLimitError ¶
func (e *GatewayError) IsBillingLimitError() bool
IsBillingLimitError reports whether the gateway rejected the call because the customer's billing/spend limit was reached (a 402 Payment Required, or a 4xx whose body indicates a billing/quota/budget problem). These are account-wide: every model and provider routes through the same Stripe customer, so retrying or failing over to another model won't help — the run must stop.
func (*GatewayError) IsContextLengthError ¶
func (e *GatewayError) IsContextLengthError() bool
IsContextLengthError returns true if the error body indicates the input exceeded the model's context window. These errors should not be retried.
func (*GatewayError) RetryAfter ¶
func (e *GatewayError) RetryAfter() time.Duration
RetryAfter returns the duration to wait before retrying, based on the Retry-After header. Returns 0 if the header is absent or unparseable.
type GatewayProvider ¶
type GatewayProvider struct {
// contains filtered or unexported fields
}
GatewayProvider routes all LLM calls through the Stripe AI Gateway, which uses the OpenAI-compatible /chat/completions endpoint for all providers.
func NewGatewayProvider ¶
func NewGatewayProvider(stripeAPIKey string) *GatewayProvider
func (*GatewayProvider) CompleteWithTools ¶
func (p *GatewayProvider) CompleteWithTools(ctx context.Context, req *ToolRequest) (*ToolResponse, error)
func (*GatewayProvider) Name ¶
func (p *GatewayProvider) Name() string
func (*GatewayProvider) StreamCompleteWithTools ¶
func (p *GatewayProvider) StreamCompleteWithTools(ctx context.Context, req *ToolRequest, callback func(StreamEvent)) (*ToolResponse, error)
type LLMProvider ¶
type LLMProvider interface {
CompleteWithTools(ctx context.Context, req *ToolRequest) (*ToolResponse, error)
Name() string
}
LLMProvider abstracts multi-turn LLM interactions with tool use.
type Message ¶
type Message struct {
Role string `json:"role"`
Content string `json:"content,omitempty"`
Thinking []ThinkingBlock `json:"thinking,omitempty"`
ToolUse []ToolUseBlock `json:"tool_use,omitempty"`
ToolResults []ToolResultBlock `json:"tool_results,omitempty"`
}
Message represents a single conversation turn.
func CopyMessages ¶
func TruncateMessages ¶
func TruncateMessages(system string, messages []Message, tools []ToolDefinition, model string) []Message
TruncateMessages ensures the total prompt fits within the model's context limit.
Truncation is applied in priority order to minimize information loss:
- Truncate assistant message content (thinking/reasoning — most expendable)
- Cap oversized tool result content
- Cap oversized tool call inputs
- Drop old assistant+tool messages from the middle (user messages preserved)
- Drop old messages from the middle as a last resort
type ModelLimits ¶
type ModelLimits struct {
ContextLimit int // maximum prompt tokens
OutputReserve int // tokens reserved for output generation
}
ModelLimits defines the context window and output token reservation per model.
func GetModelLimits ¶
func GetModelLimits(model string) ModelLimits
GetModelLimits returns the ModelLimits for the given model, falling back to defaults.
type StreamEvent ¶
type StreamEvent struct {
Type string // "reasoning_delta", "content_delta", "tool_call_delta", "done"
ContentDelta string
ReasoningDelta string
ToolIndex int
ToolCallID string
ToolName string
ArgumentsDelta string
FinishReason string
InputTokens int
OutputTokens int
}
StreamEvent represents a single chunk from a streaming LLM response. "reasoning_delta" carries native provider reasoning (Anthropic thinking blocks, or the OpenAI-compatible reasoning_content field) in ReasoningDelta — a channel distinct from "content_delta", which now carries only the user-facing answer text. "tool_call_delta" is emitted both at a tool block's start — carrying ToolName/ToolCallID so a live indicator can name the tool — and for each subsequent argument chunk in ArgumentsDelta (where ToolName is empty, as the arguments stream after the name on both providers).
type StreamingLLMProvider ¶
type StreamingLLMProvider interface {
LLMProvider
StreamCompleteWithTools(ctx context.Context, req *ToolRequest, callback func(StreamEvent)) (*ToolResponse, error)
}
StreamingLLMProvider extends LLMProvider with streaming support.
type ThinkingBlock ¶
type ThinkingBlock struct {
Text string `json:"text"`
Signature string `json:"signature,omitempty"`
}
ThinkingBlock carries a provider reasoning block together with its signature so it can be replayed to the model on later tool-loop turns. Anthropic requires preserved, unmodified thinking blocks when continuing a turn after tool_use under interleaved thinking; providers that don't return signatures (the OpenAI-compatible path) leave Signature empty, and such blocks are not replayed.
type TokenRateKey ¶
TokenRateKey identifies a marked-up rate by the gateway model name and token type it applies to.
type ToolCall ¶
type ToolCall struct {
ID string
Name string
Input json.RawMessage
}
ToolCall represents a single tool invocation from the LLM.
type ToolDefinition ¶
type ToolDefinition struct {
Name string `json:"name"`
Description string `json:"description"`
InputSchema json.RawMessage `json:"input_schema"`
}
ToolDefinition describes a tool available to the LLM.
type ToolRequest ¶
type ToolRequest struct {
Model string
System string
Messages []Message
Tools []ToolDefinition
MaxTokens int
Temperature float64
// EnableReasoning turns on the provider's reasoning output — adaptive thinking on the native Anthropic path. Chat runs set it so the live thinking panel has content; other runs leave it off to avoid the extra thinking-token cost.
EnableReasoning bool
// ReasoningEffort, when set ("low"/"medium"/"high"), asks reasoning-capable models on the OpenAI-compatible path to emit reasoning. Ignored by the native Anthropic provider.
ReasoningEffort string
}
ToolRequest is the input to a provider's CompleteWithTools call.
type ToolResponse ¶
type ToolResponse struct {
Content string
Thinking []ThinkingBlock // native reasoning blocks (Anthropic); empty for the OpenAI-compat path
ToolCalls []ToolCall
InputTokens int
OutputTokens int
StopReason string // "end_turn", "tool_use", "max_tokens"
}
ToolResponse is the normalized output from a provider.
type ToolResultBlock ¶
type ToolResultBlock struct {
ToolUseID string `json:"tool_use_id"`
Content string `json:"content"`
IsError bool `json:"is_error,omitempty"`
}
ToolResultBlock represents the result of a tool execution.
type ToolUseBlock ¶
type ToolUseBlock struct {
ID string `json:"id"`
Name string `json:"name"`
Input json.RawMessage `json:"input"`
}
ToolUseBlock represents an assistant's request to use a tool.
type TruncationResult ¶
TruncationResult holds the outcome of truncating a tool output.
func TruncateToolOutputResult ¶
func TruncateToolOutputResult(content string, toolName string) TruncationResult
TruncateToolOutputResult is like TruncateToolOutput but returns a TruncationResult indicating whether truncation occurred and the original content length.