llm

package
v0.54.6 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 9, 2026 License: MIT Imports: 24 Imported by: 0

Documentation

Overview

Package llm implements first-party model providers for Suga.

Index

Constants

View Source
const (
	ClaudeThinkingBudget   = "budget"
	ClaudeThinkingAdaptive = "adaptive"
	ClaudeThinkingNone     = "none"
)

Thinking styles accepted in ClaudeConfig.ThinkingStyle (providers.<id>[.models.<model>].thinking_style). They pin the request shape that would otherwise be guessed from the model name; "" keeps that guess.

View Source
const (
	// ErrorKindTransient: the stream broke or was cut off, or the API reported a
	// temporary failure (overloaded, rate limited, a server error), and the
	// retries ran out (streamReadError, apiTransientError). The request itself
	// was fine, so a later attempt or a continued session is likely to work; the
	// text of a cut answer was already shown and is void (retryStream restarted
	// the message, see the stream-json "retry" lines with restart=true).
	ErrorKindTransient = "provider_transient"
	// ErrorKindBadResponse: the provider answered, but with something unusable -
	// a body or stream event that is not valid JSON, an answer that is missing
	// its content, or an error the API reported itself. Requesting it again
	// would get the same answer, so it is not retried.
	ErrorKindBadResponse = "provider_bad_response"
	// ErrorKindRejected: the provider refused the request with an HTTP status that
	// is about the request or this provider (400/401/403/404/422 ...); whether
	// another provider may help is ShouldFallback's call.
	ErrorKindRejected = "provider_rejected"
	// ErrorKindQuota means the provider account or billing allowance is exhausted.
	// It is checked before transient because weekly quota responses are often
	// labelled rate_limit_error/RATE_LIMITED by APIs.
	ErrorKindQuota = "provider_quota"
)

Error kinds a host reports for a turn that failed on a provider error.

View Source
const (
	// SessionIDPlaceholder is the suga session id, e.g. a gateway's
	// per-conversation routing header.
	SessionIDPlaceholder = "{{session_id}}"
	// TurnIDPlaceholder is the current turn id ("" outside a turn).
	TurnIDPlaceholder = "{{turn_id}}"
	// ModelPlaceholder is the model id the request is sent with.
	ModelPlaceholder = "{{model}}"
	// ProviderPlaceholder is the configured provider id.
	ProviderPlaceholder = "{{provider}}"
)

Placeholders of an llm.headers value (providers.<id>.headers). A header whose value uses a placeholder with no value (a request outside a session or a turn) is left out, so a gateway never receives an empty routing value.

View Source
const (
	FinishFinal     = agentcoreapi.FinishFinal
	FinishToolCalls = agentcoreapi.FinishToolCalls
)
View Source
const (
	DefaultStreamIdleTimeout          = 120 * time.Second
	DefaultReasoningStreamIdleTimeout = 300 * time.Second
)

A streamed body is the one part of a request no other timeout bounds (see timeouts.go): a long answer may stream for minutes, so a deadline would cut a healthy stream. A provider that stops sending data in the middle of an answer - a hung gateway, an upstream connection the reader never sees die - would then hold the turn until the caller cancels it. The idle timeout covers exactly that case: the time between two reads of a streamed body, any data counted (text deltas, tool arguments, keep-alive comments alike).

The default depends on the request: an ordinary answer is expected to stream continuously, while a reasoning model may think for minutes before its first visible delta, with nothing but its own keep-alives in between.

View Source
const (
	DefaultRequestTimeout = 120 * time.Second
	// DefaultResponseHeaderTimeout is deliberately shorter than the request
	// timeout: a gateway that never sends response headers is usually a hang, and
	// a streamed request cannot be bounded once the headers are in.
	DefaultResponseHeaderTimeout = 60 * time.Second
)

Request timeouts. A provider request is bounded in two places:

  • RequestTimeout bounds one non-streamed request as a whole (http.Client.Timeout: dial, TLS, headers and body).
  • ResponseHeaderTimeout bounds the wait for the response headers of a streamed request (http.Transport.ResponseHeaderTimeout). The streamed body itself has no deadline - a long answer may stream for minutes - and ctx cancellation still stops it.

The zero value of OpenAIConfig/ClaudeConfig takes the default below; NoTimeout switches the bound off (the config's `off`).

View Source
const ClaudeThinkingMetaKey = "suga.claude_thinking"

ClaudeThinkingMetaKey is the GenerateResult / Message metadata key holding a reply's thinking blocks as Anthropic returned them ([]any of maps: {type: thinking, thinking, signature} or {type: redacted_thinking, data}). They are sent back unchanged before the tool_use blocks of that reply, as extended thinking with tools requires.

View Source
const ConnectFailureCooldown = 60 * time.Second

ConnectFailureCooldown is how long a candidate whose endpoint failed to connect is skipped by the Pool: the following steps of the same run then ask the working model first instead of paying the connect timeout again on every step. The state is process-wide (not per turn) and keyed by provider name and endpoint, so a second provider configuration pointing at another host is not affected.

View Source
const DefaultConnectTimeout = 10 * time.Second

DefaultConnectTimeout bounds the connection phase of a provider request - dialing the endpoint and the TLS handshake - on both provider clients (see newProviderTransport). It is deliberately independent of the overall request timeout (timeouts.go): http.DefaultTransport dials for up to 30s, which a turn with a fallback model should not pay in front of every step when the endpoint is unreachable. 10s is generous for a healthy endpoint and short enough that a dead one is noticed quickly.

View Source
const DefaultUserAgent = "suga"

DefaultUserAgent identifies suga to model APIs; llm.headers can override it.

View Source
const NoTimeout time.Duration = -1

NoTimeout disables a request timeout (OpenAIConfig/ClaudeConfig RequestTimeout and ResponseHeaderTimeout).

View Source
const ReasoningFoldedMetaKey = "suga.reasoning_folded"

ReasoningFoldedMetaKey is the Message.Metadata key (bool true) that marks an assistant message whose reasoning was removed from the request (history folding). The OpenAI provider still treats the request as a thinking-mode session then, so reasoning_content keeps being echoed on every assistant message even when no remaining step carries reasoning.

View Source
const ResponsesReasoningMetaKey = "suga.responses_reasoning"

ResponsesReasoningMetaKey is the GenerateResult / Message metadata key holding a reply's reasoning items as the Responses API returned them ([]any of maps, without id and status). They are sent back first in that reply's turn on later requests.

View Source
const SegmentContextMetaKey = "suga.segment_context"

SegmentContextMetaKey is the Message.Metadata key (bool true) that marks the segment context message. With PromptCache on, the Claude provider puts a cache breakpoint at the end of that message so the stable prefix (system + tools + segment context) is cached across turns. The application layer must set the same key when it builds that message.

Variables

View Source
var DefaultRetryPolicy = RetryPolicy{MaxAttempts: 6, BaseDelay: time.Second, MaxDelay: 60 * time.Second, ResetAfter: 10 * time.Minute, MaxElapsed: 3 * time.Minute}

Functions

func ErrorKind

func ErrorKind(err error) string

ErrorKind classifies a failed model request for the hosts: ErrorKindQuota (the account or billing allowance is exhausted; checked first, since quota answers often come labelled as rate limits), ErrorKindTransient (a broken or cut stream, a temporary API error or a retryable HTTP status whose retries ran out), ErrorKindRejected (an HTTP status that refuses the request itself or a context that cannot fit), ErrorKindBadResponse (a provider answer that cannot be used, or a permanent error the API reported) or "" for everything else - a cancelled turn, a storage, tool or hook failure, or an error that did not come from a provider. The value goes on the turn event's attrs and on the stream-json error line.

func GenerateObserved

GenerateObserved calls prov.Generate and reports the attempt to obs (may be nil).

func NewQuotaError

func NewQuotaError(err error, resetAt string) error

NewQuotaError marks err as an exhausted quota (ErrorKindQuota) with an optional RFC3339 reset time, for a host whose own provider reports one and for tests.

func ResetAt

func ResetAt(err error) string

ResetAt returns the provider-supplied quota reset time, if one was present.

func SessionIDFrom deprecated

func SessionIDFrom(ctx context.Context) string

SessionIDFrom returns the session id set by WithSessionID, or "".

Deprecated: use agentcoreapi.SessionIDFrom.

func ShouldFallback

func ShouldFallback(err error) bool

ShouldFallback reports whether another provider may succeed. Quota is allowed here, while Pool separately prevents it from selecting another model of the same provider (the account allowance would be shared). A context-length rejection may succeed on a different provider with a larger window, but Pool skips sibling models of the same provider.

func StreamObserved

StreamObserved calls prov.Stream, measuring time-to-first-delta, and reports the attempt to obs (may be nil). The emit callback is passed through unchanged.

func WithSessionID deprecated

func WithSessionID(ctx context.Context, id string) context.Context

WithSessionID marks ctx with the suga session the model call belongs to.

Deprecated: use agentcoreapi.WithSessionID. The session id belongs to the runtime contract, not to the provider layer (the tasks manager reads it too), so the key moved to api; this wrapper forwards to it and stays for existing embedders.

Types

type Attempt

type Attempt struct {
	Provider  string
	Model     string // empty when the provider does not expose Model()
	Index     int    // 1-based attempt number within one logical request
	Streamed  bool
	StartedAt time.Time
	Duration  time.Duration
	// TTFT is the time to the first streamed text delta; zero for non-streaming
	// calls and for streams that produced no delta.
	TTFT  time.Duration
	Usage agentcoreapi.Usage
	Err   error
	// Estimated is true when Usage was filled by a provider-side estimator.
	Estimated bool
	// EstimatedInputTokens and EstimatedOutputTokens are the estimator values
	// carried in result metadata; zero means that side had no estimate metadata.
	EstimatedInputTokens  int
	EstimatedOutputTokens int
	// ToolCalls, ReasoningChars and TextChars describe the response: how many
	// tool calls it issued (more than one = a parallel step) and how long
	// its reasoning and text were. Zero when the attempt failed.
	ToolCalls, ReasoningChars, TextChars int
	// Finish is how the response ended: "tool_calls", "final", or
	// "truncated" when the output limit cut it off; "" when it failed.
	Finish string
	// OutputLimit is the output token limit the request asked for when Finish is
	// "truncated" (0 = the provider's or server's own default applied); 0 as
	// well for every other reply.
	OutputLimit int
	// EmptyFinal is true when this was the final retry attempt and the provider
	// returned the empty response accepted after successful tool execution.
	EmptyFinal bool
	// Retries is how often the provider retried the request after a
	// transient failure (RetryPolicy), RetryWait the time spent waiting and
	// RetryReason the reason of the last retry ("HTTP 503", "stream broken").
	Retries     int
	RetryWait   time.Duration
	RetryReason string
	// RequestID is the provider's correlation id for the final HTTP request.
	RequestID string
	// Fallback is what the pool that owns the attempt did with its failure:
	// "next, <reason>" when another provider is asked, "stop, <reason>" when the
	// error goes back to the caller, "" for a successful attempt and for a
	// provider used directly (not in a pool). The reason is the error kind and,
	// for a rejection, the HTTP status (see ErrorKind, ShouldFallback).
	Fallback string
}

Attempt describes one call to one provider, including fallback attempts made by a Pool. It is delivered to an AttemptObserver after the call returns.

type AttemptObserver

type AttemptObserver func(ctx context.Context, a Attempt)

AttemptObserver receives every provider attempt. It must not block.

type ClaudeConfig

type ClaudeConfig struct {
	// Name is the provider name reported by Name(); default: anthropic.
	Name   string
	APIKey string
	// BaseURL defaults to https://api.anthropic.com. A URL ending in /v1 gets
	// /messages appended, any other gets /v1/messages.
	BaseURL string
	Model   string // default: claude-sonnet-4-6
	// Headers are extra request headers; see applyHeaders for SessionIDPlaceholder.
	Headers map[string]string
	// PromptCache adds cache_control breakpoints (at most 2 of Anthropic's 4):
	// the segment context message's last block and the request's last block.
	// Default off, since gateways may reject cache_control.
	PromptCache bool
	// Thinking is the thinking level (off, minimal, low, medium, high, xhigh,
	// max, auto; see applyClaudeThinking). "", "auto" and "off" send no
	// thinking parameters.
	Thinking string
	// ThinkingStyle pins the thinking request shape (budget, adaptive or none,
	// see ClaudeThinking* constants and applyClaudeThinking); "" keeps the
	// model-name heuristic. "none" never sends thinking parameters.
	ThinkingStyle string
	// StreamRequired sends every request as a streamed one: Generate is routed
	// through Stream. Some gateways answer only stream=true requests, the
	// internal summary requests included.
	StreamRequired bool
	// NoImages marks a provider or model without vision: pictures a message
	// carries are sent as a text note instead of image blocks. It is the
	// fallback path for a model whose configuration says vision: false.
	NoImages bool
	// Retry is the retry policy; zero fields take DefaultRetryPolicy's.
	Retry RetryPolicy
	// RequestTimeout bounds one non-streamed request as a whole
	// (http.Client.Timeout); zero takes DefaultRequestTimeout and NoTimeout
	// disables the bound. See timeouts.go.
	RequestTimeout time.Duration
	// ResponseHeaderTimeout bounds the wait for a streamed request's response
	// headers; zero takes DefaultResponseHeaderTimeout and NoTimeout disables
	// the bound.
	ResponseHeaderTimeout time.Duration
	// StreamIdleTimeout bounds the silence between two reads of a streamed body
	// (see stream_idle.go). Zero takes DefaultStreamIdleTimeout - or
	// DefaultReasoningStreamIdleTimeout for a request that asks the model to
	// think first - and NoTimeout disables the bound.
	StreamIdleTimeout time.Duration
	// Proxy routes requests through the global network.proxy selector; nil keeps
	// the default transport, which follows the environment proxies.
	Proxy netproxy.ProxyFunc
}

ClaudeConfig configures a ClaudeProvider.

type ClaudeProvider

type ClaudeProvider struct {
	// contains filtered or unexported fields
}

ClaudeProvider implements Provider for Anthropic Claude.

func NewClaudeProvider

func NewClaudeProvider(cfg ClaudeConfig) *ClaudeProvider

NewClaudeProvider creates a ClaudeProvider.

func (*ClaudeProvider) Endpoint

func (p *ClaudeProvider) Endpoint() string

Endpoint is the messages URL of this provider; with the provider name it keys the connect-failure cooldown (cooldown.go).

func (*ClaudeProvider) Generate

Generate implements the public agentcore provider contract.

func (*ClaudeProvider) Model

func (p *ClaudeProvider) Model() string

func (*ClaudeProvider) Name

func (p *ClaudeProvider) Name() string

Name implements Provider.

func (*ClaudeProvider) Stream

Stream implements the public streaming contract with Anthropic SSE: text deltas are emitted as they arrive; tool_use input, thinking and usage are accumulated into the returned result. The connection is retried by openStream; a stream that breaks while being read is requested again by retryStream (after a StreamEventRetry when text was already emitted).

func (*ClaudeProvider) SupportsToolChoice

func (p *ClaudeProvider) SupportsToolChoice() bool

SupportsToolChoice implements api.ToolChooser: /v1/messages takes tool_choice as an object ({"type": "none"}), which is also allowed with extended thinking.

type Compat

type Compat struct {
	// Names lists the adapters applied, for logs and the manifest.
	Names []string
	// ReasoningSummary asks the Responses API for a reasoning summary
	// (reasoning.summary "auto").
	ReasoningSummary bool
	// EncryptedReasoning asks the Responses API for the encrypted reasoning
	// (include reasoning.encrypted_content), sent back on later requests.
	EncryptedReasoning bool
	// ReasoningEveryTurn: in thinking mode every assistant turn must carry a
	// reasoning item with its text when replayed (DeepSeek rejects a turn
	// without it); a missing one is filled in.
	ReasoningEveryTurn bool
	// NoToolChoice leaves tool_choice out of the request and makes the
	// provider report no ToolChooser support (the runtime skips its tool-less
	// wrap-up retry then).
	NoToolChoice bool
	// MaxOutputTokens caps the output when the request sets no limit; 0
	// sends none.
	MaxOutputTokens int
	// CacheHitField names a usage field with the cached prompt tokens, read
	// when the standard one is missing or 0.
	CacheHitField string
	// ToolOutputImages sends a tool result's pictures inside the Responses
	// function_call_output (the API accepts an array of input_text and
	// input_image there). Without it the pictures follow in a user item after
	// the tool batch, which is what the gateways accept.
	ToolOutputImages bool
}

Compat describes how a gateway or a model family departs from the plain OpenAI API. Each adapter lives in its own compat_<name>.go file and is matched by the base URL host and the model id (ResolveCompat), so the request builders never name a gateway; a new gateway is a new file.

func ResolveCompat

func ResolveCompat(baseURL, model string, extra ...string) Compat

ResolveCompat returns the adapters matching baseURL and model, plus the ones named in extra (providers.<id>.compat), applied in name order so the result does not depend on registration order. Unknown names are ignored.

type FinishReason

type FinishReason = agentcoreapi.FinishReason

type GenerateRequest

type GenerateRequest = agentcoreapi.GenerateRequest

type GenerateResult

type GenerateResult = agentcoreapi.GenerateResult

type OpenAIConfig

type OpenAIConfig struct {
	APIKey  string
	BaseURL string // default: https://api.openai.com/v1
	Model   string // default: gpt-4o
	Name    string // default: "openai"
	// Headers are extra request headers; see applyHeaders for SessionIDPlaceholder.
	Headers map[string]string
	// ReasoningEffort is sent as the reasoning_effort request field (e.g.
	// "low", "high", "none"); empty leaves the provider's default.
	ReasoningEffort string
	// Retry is the retry policy; zero fields take DefaultRetryPolicy's.
	Retry RetryPolicy
	// StreamRequired sends every request as a streamed one: Generate is routed
	// through Stream. Some gateways answer only stream=true requests, the
	// internal summary requests included.
	StreamRequired bool
	// NoImages marks a provider or model without vision: pictures a message
	// carries are sent as a text note instead of image parts. It is the
	// fallback path for a model whose configuration says vision: false.
	NoImages bool
	// RequestTimeout bounds one non-streamed request as a whole
	// (http.Client.Timeout); zero takes DefaultRequestTimeout and NoTimeout
	// disables the bound. See timeouts.go.
	RequestTimeout time.Duration
	// ResponseHeaderTimeout bounds the wait for a streamed request's response
	// headers; zero takes DefaultResponseHeaderTimeout and NoTimeout disables
	// the bound.
	ResponseHeaderTimeout time.Duration
	// StreamIdleTimeout bounds the silence between two reads of a streamed body
	// (see stream_idle.go). Zero takes DefaultStreamIdleTimeout - or
	// DefaultReasoningStreamIdleTimeout for a request that asks the model to
	// think first - and NoTimeout disables the bound.
	StreamIdleTimeout time.Duration
	// Proxy routes requests through the global network.proxy selector; nil keeps
	// the default transport, which follows the environment proxies.
	Proxy netproxy.ProxyFunc
}

OpenAIConfig configures an OpenAIProvider.

type OpenAIProvider

type OpenAIProvider struct {
	// contains filtered or unexported fields
}

OpenAIProvider implements Provider for OpenAI-compatible APIs (OpenAI, DeepSeek, Ollama, etc.).

func NewOpenAIProvider

func NewOpenAIProvider(cfg OpenAIConfig) *OpenAIProvider

NewOpenAIProvider creates an OpenAIProvider from cfg.

func (*OpenAIProvider) Endpoint

func (p *OpenAIProvider) Endpoint() string

Endpoint is the base URL of this provider; with the provider name it keys the connect-failure cooldown (cooldown.go).

func (*OpenAIProvider) Generate

Generate implements the public agentcore provider contract.

func (*OpenAIProvider) Model

func (p *OpenAIProvider) Model() string

func (*OpenAIProvider) Name

func (p *OpenAIProvider) Name() string

Name implements Provider.

func (*OpenAIProvider) Stream

Stream implements the public streaming contract with Server-Sent Events: text deltas are emitted as they arrive; tool calls, reasoning_content and usage are accumulated into the returned result. The connection is retried by openStream; a stream that breaks while being read is requested again by retryStream (after a StreamEventRetry when text was already emitted).

func (*OpenAIProvider) SupportsToolChoice

func (p *OpenAIProvider) SupportsToolChoice() bool

SupportsToolChoice implements api.ToolChooser: chat/completions takes the tool_choice field ("none" included).

type Pool

type Pool struct {
	// contains filtered or unexported fields
}

Pool manages multiple LLM providers with automatic failover. It tries each provider in order until one succeeds, and only while another model still has a chance (see ShouldFallback).

func NewPool

func NewPool(providers []Provider) *Pool

NewPool creates a Pool from an ordered list of providers. The first provider is the primary; subsequent ones are fallbacks.

func (*Pool) AddProvider

func (p *Pool) AddProvider(prov Provider)

AddProvider appends a provider to the pool (thread-safe).

func (*Pool) Generate

Generate tries each provider in turn and returns the first success. It moves on only when another model has a chance (ShouldFallback): a transient failure (broken stream, network error, 408/429/5xx), a context that does not fit this provider's window, or a rejection that names this provider (401/403/404 - its credentials or its models). A request the provider refused on the request's own account (400/422), a bad response or an unclassified failure is returned as it is, so the remaining models - and the timeouts and tokens they would spend - are not wasted on an error they would hit just the same.

func (*Pool) Name

func (p *Pool) Name() string

Name implements Provider.

func (*Pool) SetObserver

func (p *Pool) SetObserver(obs AttemptObserver)

SetObserver installs a callback invoked once per provider attempt (fallbacks included). Pass nil to remove it.

func (*Pool) Stream

Stream implements the public streaming contract.

func (*Pool) SupportsToolChoice

func (p *Pool) SupportsToolChoice() bool

SupportsToolChoice implements api.ToolChooser: it reports true only when every provider honours tool_choice, since one fallback rejecting the field would fail the request the runtime sends it for.

type Provider

type Provider = agentcoreapi.Provider

type ResponsesConfig

type ResponsesConfig struct {
	OpenAIConfig
	// Compat names extra adapters (providers.<id>.compat) on top of the ones
	// matched by base URL and model.
	Compat []string
	// PromptCacheRetention is sent as prompt_cache_retention when set.
	PromptCacheRetention string
}

ResponsesConfig configures a ResponsesProvider.

type ResponsesProvider

type ResponsesProvider struct {
	*OpenAIProvider
	// contains filtered or unexported fields
}

ResponsesProvider implements Provider for the OpenAI Responses API (POST {base_url}/responses). It always sends the whole history with store false (third-party gateways often ignore previous_response_id, and segments and folding rewrite history anyway). HTTP, retries and headers are the OpenAI provider's; gateway differences come from Compat.

func NewResponsesProvider

func NewResponsesProvider(cfg ResponsesConfig) *ResponsesProvider

NewResponsesProvider creates a ResponsesProvider from cfg.

func (*ResponsesProvider) Compat

func (p *ResponsesProvider) Compat() Compat

Compat returns the adapters this provider applies.

func (*ResponsesProvider) Generate

Generate implements Provider.

func (*ResponsesProvider) Stream

Stream implements the streaming contract over the Responses API's Server-Sent Events: output_text deltas are emitted as they arrive; the result comes from the terminal event's response object (or, without one, the items of response.output_item.done). A stream that ends without a terminal event is a broken stream, requested again by retryStream.

func (*ResponsesProvider) SupportsToolChoice

func (p *ResponsesProvider) SupportsToolChoice() bool

SupportsToolChoice implements api.ToolChooser; a gateway whose compat says it has no usable tool_choice (Compat.NoToolChoice) never gets one, and the runtime skips its tool-less wrap-up retry then.

type RetryPolicy

type RetryPolicy struct {
	MaxAttempts                     int
	BaseDelay, MaxDelay, ResetAfter time.Duration
	// MaxElapsed is the wall-clock budget of one request's retries: once the
	// attempts and waits since the request started pass it, no further attempt is
	// made and the request fails (the caller's fallback_models then take over).
	// A provider that keeps failing would otherwise retry away minutes while the
	// process holds its session. Zero takes the default, NoTimeout disables it.
	MaxElapsed time.Duration
}

type RetryReport

type RetryReport struct {
	Retries   int
	Wait      time.Duration
	Last      string
	RequestID string
}

type StreamingProvider

type StreamingProvider = agentcoreapi.StreamingProvider

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL