Documentation
¶
Overview ¶
Package provider — Anthropic prompt-cache breakpoint helpers.
caching.go stamps Anthropic prompt-cache markers on a request before it goes on the wire. The cache is opt-in and prefix-based: a cache_control marker on a block makes the API cache everything up to and including that block, and a later request with a byte-identical prefix reads it back at ~10% of the input price. Setting a marker is always safe: prefixes shorter than the model's minimum cacheable length (1024–4096 tokens) are silently not cached, with no error and no extra cost.
Anthropic caches in the order tools → system → messages and allows at most 4 breakpoints per request. We set 3:
The last tool definition. Tool schemas are static per agent, so this prefix survives every request of a session.
The last system block. Static too, and separate from (1) so editing the prompt (e.g. a hot reload) does not invalidate the tools cache.
The last cacheable block of the last message. Turn N's history becomes turn N+1's cached prefix; this is where the savings come from in agentic loops, which re-send the whole history on every tool round-trip. Thinking and redacted-thinking blocks cannot carry a marker, so this one walks backwards past them to the nearest eligible block.
Call applyAnthropicCacheControl as the LAST step of wire construction so every section is final.
Package provider — provider-neutral extension point for prompt caching.
cache_apply.go provides the shared helper that any provider's wire builder may call to check whether the user has opted out of prompt caching. Today only the Anthropic provider honors it; the openai, gemini, mistral, xai, ollama, and azure builders ignore the option entirely.
Index ¶
- Constants
- Variables
- func AzureAPIKey(apiKey string) string
- func AzureAPIVersion(apiVersion string) string
- func AzureEndpoint(endpoint string) string
- func AzureProbePaths(deployment, apiVersion, endpoint string) []string
- func BuildHTTPClient(opts *LLMOptions, timeout time.Duration) (*http.Client, error)
- func BuildTransport(opts *LLMOptions) (http.RoundTripper, error)
- func CheckOllama(baseURL string) error
- func ContextWindowSize(modelName string) int64
- func ContextWindowSizeFor(providerName, modelName string) int64
- func IsAzureCompatProxyEndpoint(endpoint string) bool
- func NewAnthropic(_ context.Context, modelName, apiKey, baseURL, thinkingLevel string, ...) (model.LLM, error)
- func NewAzureOpenAI(_ context.Context, deploymentName, apiKey, endpoint, apiVersion string, ...) (model.LLM, error)
- func NewGemini(ctx context.Context, modelName, baseURL string, opts *LLMOptions) (model.LLM, error)
- func NewLLM(ctx context.Context, info Info, apiKey, baseURL, thinkingLevel string, ...) (model.LLM, error)
- func NewMistral(_ context.Context, modelName, apiKey, baseURL string, llmOpts *LLMOptions) (model.LLM, error)
- func NewOllama(_ context.Context, modelName, apiKey, baseURL, thinkingLevel string, ...) (model.LLM, error)
- func NewOpenAI(_ context.Context, modelName, apiKey, baseURL string, llmOpts *LLMOptions) (model.LLM, error)
- func NewOpenCode(ctx context.Context, modelName, apiKey, baseURL, thinkingLevel string, ...) (model.LLM, error)
- func NewXAI(_ context.Context, modelName, apiKey, baseURL, thinkingLevel string, ...) (model.LLM, error)
- func OllamaContextWindowSize(ctx context.Context, baseURL, modelName string) int64
- func OllamaListModels(ctx context.Context, baseURL string) ([]string, error)
- func ValidateModel(info Info) error
- type Info
- type LLMOptions
- type ListModelsOptions
- type ModelInfo
- type ModelWindow
Constants ¶
const DefaultAzureAPIVersion = "2025-04-01-preview"
DefaultAzureAPIVersion is the api-version used when neither the caller nor OPENAI_API_VERSION supplies one.
Variables ¶
var KnownModels = mustLoadKnownModels()
KnownModels lists recognized model names per provider. The check is prefix-based: a model is valid if it starts with any entry. Ollama models are not validated here (they are dynamic).
OpenAI and Anthropic IDs are loaded from embedded llm-prices snapshots under modeldata/. Gemini and Mistral IDs are maintained in model_catalog.go. Update context-window metadata in modeldata/context-windows.json in the same change when official limits change.
var OllamaModelPrefixes = []string{}
OllamaModelPrefixes are model prefixes that previously auto-routed to Ollama. Bare Ollama model names are intentionally not auto-detected; use the explicit ollama/ prefix, or the :cloud tag for Ollama cloud models.
Functions ¶
func AzureAPIKey ¶ added in v0.0.52
AzureAPIKey resolves the Azure credential from an explicit value, then the three accepted environment variables, in the order NewAzureOpenAI uses.
Exported so `pi ping` authenticates exactly as a real run does rather than reimplementing the fallback chain and disagreeing about which var wins.
func AzureAPIVersion ¶ added in v0.0.52
AzureAPIVersion resolves the api-version, falling back to OPENAI_API_VERSION and then DefaultAzureAPIVersion.
func AzureEndpoint ¶ added in v0.0.52
AzureEndpoint resolves the Azure resource endpoint, falling back to AZURE_OPENAI_ENDPOINT.
func AzureProbePaths ¶ added in v0.0.52
AzureProbePaths returns the paths `pi ping` should GET for an Azure endpoint, most informative first, built from the same rules NewAzureOpenAI applies to real traffic. The caller tries them in order and keeps the first that answers 2xx.
There are two because neither alone is both reliable and meaningful:
- /openai/models?api-version=… is the data-plane list route. A 200 proves the resource is reachable AND that the Api-Key is accepted, which is the part worth knowing. It does not name the deployment.
- /openai/deployments/{deployment}?api-version=… names the deployment, but it is the legacy Deployments_Get route and current api-versions may not serve it at all. It is the fallback, not the primary.
Neither one establishes that the deployment can serve a completion — only the model ping that follows does. Pinging the bare host, which is what ping did before, proved nothing beyond a TLS handshake to Azure's front door.
A compat proxy gets /models alone, since deployment paths and api-version break routing there for the same reason they are skipped on real requests.
func BuildHTTPClient ¶ added in v0.0.9
BuildHTTPClient creates an *http.Client with the TLS trust, extra headers, connect timeout and request timeout from opts. Returns a client with the default transport when opts asks for no customization.
func BuildTransport ¶ added in v0.0.9
func BuildTransport(opts *LLMOptions) (http.RoundTripper, error)
BuildTransport creates an http.RoundTripper carrying the TLS trust, connect timeout and extra headers from opts. It returns (nil, nil) when opts asks for no customization, so callers can leave the SDK's own client in place.
Every variant starts from a clone of http.DefaultTransport rather than a fresh &http.Transport{}: the default carries ProxyFromEnvironment, connection pooling and HTTP/2, and building from scratch silently drops HTTPS_PROXY — which is exactly the environment where a custom CA or a TLS skip is needed in the first place.
func CheckOllama ¶ added in v0.0.4
CheckOllama verifies that the Ollama server at baseURL is reachable. It first checks TCP connectivity on the port, then issues a GET to the root endpoint (Ollama returns "Ollama is running").
func ContextWindowSize ¶ added in v0.0.18
ContextWindowSize returns the context window size for a model (in tokens). Returns 0 if the model is unknown.
This searches every vendor except Azure. Prefer ContextWindowSizeFor when the provider is known — an Azure deployment shares its name with an OpenAI model but not necessarily its window, so only the provider-aware lookup separates them.
func ContextWindowSizeFor ¶ added in v0.0.52
ContextWindowSizeFor returns the context window for a model as served by a specific provider, falling back to the vendor-agnostic table when that provider publishes no window of its own.
The fallback is what keeps a newly provisioned Azure deployment usable when an OpenAI entry matches the name it was provisioned from — that beats returning 0, since zero disables auto-compaction and lets the session grow unchecked until the API rejects it. A name that matches nothing anywhere still returns 0; there is no window to guess.
func IsAzureCompatProxyEndpoint ¶ added in v0.0.52
IsAzureCompatProxyEndpoint reports whether the endpoint is an OpenAI-compatible gateway rather than a native Azure resource, in which case deployment-path rewriting and the api-version parameter must be skipped.
func NewAnthropic ¶
func NewAnthropic(_ context.Context, modelName, apiKey, baseURL, thinkingLevel string, llmOpts *LLMOptions) (model.LLM, error)
NewAnthropic creates an Anthropic model.LLM. If baseURL is non-empty, it overrides the default API endpoint. When baseURL is set, the API key is optional (for Ollama compatibility). thinkingLevel controls extended thinking: "none", "low", "medium", "high". llmOpts.AdvisorModel enables the advisor tool (beta). See specs/features/LLM/001-claude-models/advisor-tool.md
func NewAzureOpenAI ¶ added in v0.0.18
func NewAzureOpenAI(_ context.Context, deploymentName, apiKey, endpoint, apiVersion string, llmOpts *LLMOptions) (model.LLM, error)
NewAzureOpenAI creates an Azure OpenAI model.LLM. It uses AZURE_OPENAI_API_KEY, AZURE_OPENAI_ENDPOINT, and OPENAI_API_VERSION (defaults to 2025-04-01-preview). The deploymentName is the Azure deployment name (not the model ID).
func NewGemini ¶
NewGemini creates a Gemini model.LLM using ADK Go's native Gemini support. It reads the API key from GEMINI_API_KEY or GOOGLE_API_KEY env vars. If neither is set, it falls back to Application Default Credentials. If baseURL is non-empty, it overrides the default API endpoint.
func NewLLM ¶
func NewLLM(ctx context.Context, info Info, apiKey, baseURL, thinkingLevel string, opts *LLMOptions) (model.LLM, error)
NewLLM creates a model.LLM for the given provider info, API key, optional base URL, thinking level, and options.
func NewMistral ¶ added in v0.0.18
func NewMistral(_ context.Context, modelName, apiKey, baseURL string, llmOpts *LLMOptions) (model.LLM, error)
NewMistral creates a Mistral model.LLM. If baseURL is empty, the default Mistral API endpoint is used.
func NewOllama ¶ added in v0.0.7
func NewOllama(_ context.Context, modelName, apiKey, baseURL, thinkingLevel string, opts *LLMOptions) (model.LLM, error)
NewOllama creates an Ollama model.LLM using the native Ollama Go client. The server is chosen by resolveOllamaEndpoint: an explicit baseURL if given, otherwise api.ollama.com for a :cloud/-cloud tagged model and localhost for everything else. thinkingLevel controls extended thinking: "none", "low", "medium", "high".
func NewOpenAI ¶
func NewOpenAI(_ context.Context, modelName, apiKey, baseURL string, llmOpts *LLMOptions) (model.LLM, error)
NewOpenAI creates an OpenAI model.LLM. If baseURL is non-empty, it overrides the default API endpoint. When apiKey is recognized as a codex ChatGPT OAuth token and no explicit baseURL is provided, the client is pointed at the ChatGPT backend (/codex/responses) and the `chatgpt-account-id` + `originator` headers pi-mono sends are injected — the platform /v1/responses endpoint rejects codex tokens with 401 "Missing scopes: api.responses.write".
func NewOpenCode ¶ added in v0.0.47
func NewOpenCode(ctx context.Context, modelName, apiKey, baseURL, thinkingLevel string, opts *LLMOptions) (model.LLM, error)
NewOpenCode creates an OpenCode Go model.LLM, routing the model to the client that serves its API family based on the hardcoded catalog.
- "chat" and "responses" → NewOpenAI (OpenAI-compatible; NewOpenAI picks the chat/completions vs responses endpoint internally from the model ID)
- "messages" → NewAnthropic (Anthropic messages, x-api-key)
Unknown models return an error. The opencode/ prefix must already be stripped.
func NewXAI ¶ added in v0.0.71
func NewXAI(_ context.Context, modelName, apiKey, baseURL, thinkingLevel string, llmOpts *LLMOptions) (model.LLM, error)
NewXAI creates an xAI model.LLM. If baseURL is empty, the default xAI API endpoint is used. thinkingLevel controls reasoning effort: "none", "low", "medium", "high", "max".
func OllamaContextWindowSize ¶ added in v0.0.34
OllamaContextWindowSize queries the Ollama server for the context window size of the given model. Returns 0 if the size cannot be determined.
func OllamaListModels ¶ added in v0.0.7
OllamaListModels lists available models from the Ollama server.
func ValidateModel ¶ added in v0.0.18
ValidateModel checks whether the model name is recognized for its provider. Returns an error with suggestions if the model is unknown. Ollama and custom endpoint models are always considered valid (they are dynamic).
Types ¶
type Info ¶
type Info struct {
Provider string
Model string
Ollama bool // true when model is served by Ollama
Custom bool // true when using an explicit custom OpenAI-compatible endpoint
// BaseURL is the endpoint finally selected for this model, recorded so a
// session transcript identifies the backend and not just the model name.
// The same name served by ollama, by a gateway, and by a vendor API behaves
// differently, and a transcript without it cannot be reproduced.
BaseURL string
}
Info describes a provider and the model to use.
func Resolve ¶
Resolve determines the provider from a model name. Ollama models are routed to the native "ollama" provider.
func ResolveWithBaseURL ¶ added in v0.0.31
ResolveWithBaseURL determines the provider from a model name. When baseURL is provided and the model is not otherwise recognized, it routes the model to an OpenAI-compatible custom endpoint. The optional openai/ prefix is stripped so `--model openai/foo` sends `foo` to the custom endpoint.
type LLMOptions ¶ added in v0.0.9
type LLMOptions struct {
ExtraHeaders map[string]string
InsecureSkipTLS bool
// CACertPath is a PEM bundle to trust in addition to the system roots —
// the answer for a TLS-intercepting corporate proxy, which otherwise
// forces InsecureSkipTLS and drops verification for every endpoint.
// Ignored when InsecureSkipTLS is set.
CACertPath string
// DisableSystemCAs narrows trust to CACertPath alone. Only useful when
// the endpoint must not be reachable through any public root.
DisableSystemCAs bool
// ConnectTimeout bounds connection establishment only. Zero leaves the Go
// default in place. Distinct from the per-client request timeout, which
// has to stay generous for streaming completions.
ConnectTimeout time.Duration
AdvisorModel string // Advisor model (e.g., "claude-opus-4-7")
AdvisorMaxUses int // Max advisor calls per request (0 = unlimited)
AdvisorCaching bool // Enable ephemeral prompt caching for advisor
// DisablePromptCaching turns OFF the cache_control breakpoints the
// Anthropic provider stamps on every request. Caching is on by default
// because it only ever lowers the bill: requests whose prefix is below
// Anthropic's minimum cacheable length (1024-4096 tokens) are simply
// not cached, with no error and no extra cost. Other providers ignore
// this flag today.
DisablePromptCaching bool
// TraceHTTP records every LLM request and response — method, URL, full
// headers and body — to the session log and to OTel span events. Set by
// --trace-http. Credentials are masked (see httplog.Redact), but prompts
// and completions are not: the log holds the entire conversation in
// cleartext, which is the point and also the reason it is off by default.
TraceHTTP bool
// EnableXAITools opts into xAI server-side tools (web search, X search,
// and code interpreter) for xAI Responses API requests.
EnableXAITools bool
}
LLMOptions holds optional configuration for LLM provider creation.
type ListModelsOptions ¶ added in v0.0.39
type ListModelsOptions struct {
APIKey string
BaseURL string // override the default API endpoint
Insecure bool // skip TLS certificate verification
}
ListModelsOptions controls how models are fetched from a provider.
type ModelInfo ¶ added in v0.0.39
ModelInfo holds a model ID and optional metadata returned by a provider's model listing API.
func ListModels ¶ added in v0.0.39
func ListModels(ctx context.Context, providerName string, opts ListModelsOptions) ([]ModelInfo, error)
ListModels calls the given provider's model listing API and returns available model IDs. Supported providers: anthropic, openai, gemini, mistral, xai, ollama.
type ModelWindow ¶ added in v0.0.52
ModelWindow pairs a model or deployment name with its context window.
func AzureDeployments ¶ added in v0.0.52
func AzureDeployments() []ModelWindow
AzureDeployments returns the cataloged Azure OpenAI deployments and the context window each was provisioned with, sorted by name.
Azure has no listing endpoint reachable with only an API key — enumerating deployments needs ARM credentials and the resource ID — so this is the embedded catalog, not a live query. It is therefore a description of one subscription's deployments and may not match another's.