Documentation
¶
Overview ¶
Package models is the adapter layer between core-agent's configuration and concrete LLM backends. The Provider interface keeps the rest of the codebase free of provider-specific imports so additional backends plug in behind the same contract.
Built-in providers:
- "gemini" / "vertex" — google.golang.org/adk/v2/model/gemini
- "anthropic" — Claude via github.com/anthropics/anthropic-sdk-go
- any provider profile — github.com/go-steer/core-models, through pkg/models/profiles (OpenAI-compatible endpoints, vLLM, SGLang, Ollama, Vertex AI partner models)
Each backend's package init() calls Register (RegisterProfiles for pkg/models/profiles) so importing the subpackage is enough to make the provider available.
Index ¶
- Constants
- Variables
- func Adapt(m llm.LLM) adkmodel.LLM
- func AsSideCall(ctx context.Context, name string) context.Context
- func AutoDetectProvider() string
- func BackendOf(p Provider) string
- func BridgeUsage(resp *adkmodel.LLMResponse)
- func BuiltinsSuppressed(ctx context.Context) bool
- func PromptCacheSuppressed(ctx context.Context) bool
- func Register(name string, c Constructor)
- func RegisterProfiles(c Constructor)
- func ResolveMCPSmallModel(p Provider, mcpSpecific, agenticGeneral string) string
- func ResolveSmallModel(p Provider, override string) string
- func SideCallName(ctx context.Context) string
- func WithPriorSuccess(ctx context.Context, rec *PriorSuccess) context.Context
- func WithoutBuiltins(ctx context.Context) context.Context
- func WithoutPromptCache(ctx context.Context) context.Context
- type AnthropicAPI
- type AnthropicVertex
- type BuiltinToolsReporter
- type Constructor
- type GeminiAPI
- type GeminiVertex
- type PriorSuccess
- type Provider
- type ProviderOptions
- type RetryError
- type RetryPolicy
- type SmallModelDefaulter
Constants ¶
const ( DefaultRetryBackoff = 2 * time.Second DefaultRetryCooldown = 30 * time.Second )
Default retry timings. Both are deliberately modest; see RetryPolicy for why the cooldown matters more than the backoff.
const ProviderRetryMetadataKey = "provider_retry"
ProviderRetryMetadataKey marks the response a retry recovered with (#1206). Its value is a map with "outcome" ("recovered"), "attempts" and "error" (the rejection the retry fired for). It is stamped on the first NON-partial response after recovery, because that is the one ADK persists as the session event; a partial chunk never reaches the transcript, and the stream aggregator builds its final response fresh, so a stamp on a chunk would be lost.
const RetryBurst = 3
RetryBurst is how many retries the shared budget holds at once (#1039). The budget refills at one per Cooldown, so the steady-state added load is unchanged from the single-timestamp guard this replaces; the burst is what a correlated rejection can draw down.
Three, because 429s arrive correlated by construction. A provider shedding load rejects several concurrent callers within seconds of each other, and a one-rescue-per-window budget hands that rescue to whichever caller was rejected first and abandons the rest — measured on a live cluster as thirteen retries that rescued their caller and one suppression that killed a delegation, which is the exact failure #935 was opened to remove. Three covers a parent plus two subagents, the largest correlated burst the drill archive shows, at a worst case of three extra requests per window rather than one.
Variables ¶
var ErrEmptyResponse = errors.New("model returned no usable content")
ErrEmptyResponse is the provider-agnostic sentinel for "the call succeeded and the model produced nothing usable". Provider adapters that synthesize such an error wrap this so callers can recognize the condition without importing the adapter — gemini.ErrEmptyResponse does, and anything added later should.
It exists because "no content" needs opposite handling in the two places it shows up. Inside the agentic loop it is a fault: a turn that produces nothing leaves the loop with no next action and the session goes idle forever, so the Gemini adapter retries it once and then surfaces it as an error (#220). For a one-shot side question (/btw) the same condition is simply the answer — the model declined, and the operator should be told that rather than shown a stack of provider prose. AskSideQuestion converts it; the loop doesn't.
Match with errors.Is. Never compare messages: the adapters wrap this with their own diagnostic text, which is the part that changes.
Functions ¶
func Adapt ¶
Adapt turns a core-models model into the ADK v2 model.LLM core-agent runs, for every backend built on the library: the provider profiles, and the gemini, vertex, anthropic and anthropic-vertex facades (docs/model-support-design.md).
Beyond core-models' adkv2 shim it does two things core-agent needs:
- Copies the usage record's cache-write counts into the CustomMetadata sidecar keys pkg/usage reads (usage.CacheCreationTokensMetadataKey and the 1-hour share's key), so cost accounting and usage.Rebuild over old event logs see what the library reports. The genai usage fields the library already fills (prompt, completion, cache reads, reasoning) pass through untouched.
- Keeps the WithoutBuiltins unwrap: a library model that can drop its server-side built-ins (Gemini's) comes back with a WithoutBuiltins() model.LLM method, the duck type RunSubtask looks for, so a subtask still drives the model with exactly its own tools.
func AsSideCall ¶
AsSideCall marks ctx as a one-shot internal call — the approver, the compaction summarizer, a session title, a /btw question — so a retry it triggers logs as one (#1206). A side call's response is never a session event and its error never a turn error, so its retry has no transcript surface by construction; labelling the log line lets box A2's counter tell those retries from the ones a transcript must show, instead of failing the run on them.
The mark is core-models' callctx one, so the library adapters read the same key: a side call neither uses nor seeds a Vertex context cache, and never counts as evidence for the bare-400 retry.
func AutoDetectProvider ¶
func AutoDetectProvider() string
AutoDetectProvider walks the env to pick a default backend. Exported alias of autoDetectProvider for callers that need the provider name BEFORE constructing a Provider (e.g. cmd/core-agent's --task flag resolution needs the provider name to pick a model for a given tier without paying for full provider construction).
Returns "" when no env-based default is detectable. Returns the same canonical name strings as Resolve would route to.
func BackendOf ¶
BackendOf returns the backend that bills p's tokens, the name the shared price catalog keys "<backend>/<model>" rows on: a provider profile's backend (profile.Profile.BackendName), or the core provider's own name ("gemini", "vertex", "anthropic", "anthropic-vertex"), which is already the core-models backend name. cmd/core-agent hands it to usage.SetBackend.
func BridgeUsage ¶
func BridgeUsage(resp *adkmodel.LLMResponse)
BridgeUsage copies the parts of core-models' usage record that genai's UsageMetadata cannot hold into the CustomMetadata sidecar pkg/usage reads: the cache-write bucket, and the 1-hour share of it that bills at 2x rather than 1.25x (#770). ToolUseTokens is deliberately not mapped: genai's field for it sits outside the prompt count, so guessing a mapping risks counting tokens twice.
A count the library reports as zero writes nothing, matching what the pre-library Anthropic adapter wrote; pkg/usage reads an absent key as zero.
func BuiltinsSuppressed ¶
BuiltinsSuppressed reports whether WithoutBuiltins was applied to ctx. Backends consult it per request, since one model.LLM serves both the agentic loop and the one-shot side calls.
func PromptCacheSuppressed ¶
PromptCacheSuppressed reports whether WithoutPromptCache was applied to ctx. Backends consult it per request, since one model.LLM serves both the agentic loop and the one-shot side calls.
func Register ¶
func Register(name string, c Constructor)
Register installs a Constructor under its provider name. Idiomatically called from package init() in each backend implementation.
func RegisterProfiles ¶
func RegisterProfiles(c Constructor)
RegisterProfiles installs the Constructor for provider profiles. Idiomatically called from pkg/models/profiles' init().
func ResolveMCPSmallModel ¶
ResolveMCPSmallModel is ResolveSmallModel's MCP-wrap sibling — layers a per-surface override (mcp.json's agentic_wrap_model or --mcp-agentic-wrap-model) in front of the general chain so operators can pick a different tier for MCP responses than for built-in-tool wrappers. Rationale: MCP responses can be shaped differently enough (structured GKE tables vs. arbitrary fetch_url bodies) that one tier works well for one surface but not the other.
Precedence: mcpSpecific → agenticGeneral → provider default → "".
func ResolveSmallModel ¶
ResolveSmallModel picks the model ID that agentic subtasks should run on. Operator override (a non-empty explicit --agentic-small-model value) always wins. Otherwise: if p implements SmallModelDefaulter, return whatever it reports; if not, return "" — agentic wrappers treat empty as "inherit the parent's model."
func SideCallName ¶
SideCallName returns the name AsSideCall gave ctx, or "".
func WithPriorSuccess ¶
func WithPriorSuccess(ctx context.Context, rec *PriorSuccess) context.Context
WithPriorSuccess puts rec on ctx for the model calls made under it.
A nil rec SHADOWS any record ctx already carries. That is the form a nested run uses — a subtask or a sync subagent runs its own model, instruction and tools on a context derived from the parent's tool call, and the parent's success says nothing about whether the child's request is well formed.
func WithoutBuiltins ¶
WithoutBuiltins marks ctx as a request that must go to the model with EXACTLY the tools the caller put on it: no provider-injected server-side built-ins (Gemini's google_search / url_context / code_execution) and no context-cache reference stamped on top.
The /btw side question is the motivating caller. It is documented as tool-less — the operator asks about what already happened, and the answer should come from the transcript, not from a web search the model decided to run. Building the request with a nil Config isn't enough to get that: the Gemini wrapper creates the Config itself and appends its built-ins, and on a Vertex cached turn it also stamps CachedContent onto a request that already carries the full history.
This is the per-call sibling of the model-level builtinsLLM.WithoutBuiltins() unwrap that RunSubtask uses. The unwrap is the right tool when a caller drives the inner model for a whole subtask; the context marker is the right tool for one call on the SHARED model, because it keeps the wrapper's other behavior — notably the empty-response retry — in place. For a feature whose reported symptom is blank answers, dropping that retry to get tool-lessness would trade one bug for another.
Both Gemini and Anthropic honor it. It can only take tools away.
func WithoutPromptCache ¶
WithoutPromptCache marks ctx as a one-shot request: a call whose prompt prefix is not expected to recur, so a provider that would normally write a prompt-cache entry should skip it.
Prompt caching trades a write premium now (Anthropic bills 1.25x on the tokens it stores) against cheap reads later (~0.1x). The trade pays off from the second request that carries the same prefix. Some calls are structurally never that second request — core-agent's summarizer and checkpointer send their own system instruction and no tools, the /btw side question sends neither, and a tight-budget subtask runs in a session whose ID never recurs. Their prefixes diverge from the agentic loop's at the first block, so the byte-exact match can't hit even where the history is identical, and writing an entry is a pure 25% surcharge on a call that is already expensive because it re-sends the whole conversation.
This is a hint, not a guarantee: providers without prompt caching (Gemini today) ignore it, and it can only turn caching OFF — a provider whose caching is disabled by config or CLI stays disabled. It lives here rather than on a provider-specific option so callers in pkg/agent can set it without importing a backend.
Types ¶
type AnthropicAPI ¶
type AnthropicAPI struct {
// APIKey overrides ANTHROPIC_API_KEY. Empty falls back to the
// environment, same as the config path.
APIKey string
}
AnthropicAPI selects the Anthropic API backend (api-key auth).
type AnthropicVertex ¶
type AnthropicVertex struct {
// Project and Region identify the Model Garden deployment. Empty
// values fall back to the same env chain as the config path:
// ANTHROPIC_VERTEX_PROJECT_ID then GOOGLE_CLOUD_PROJECT for the
// project; CLOUD_ML_REGION then GOOGLE_CLOUD_LOCATION then the
// "us-east5" default for the region.
Project string
Region string
}
AnthropicVertex selects Anthropic models served through Vertex AI Model Garden (ADC auth).
type BuiltinToolsReporter ¶
type BuiltinToolsReporter interface {
BuiltinToolNames() []string
}
BuiltinToolsReporter is an optional Provider extension for backends that inject the provider's own server-side built-in tools (web search, URL context, code execution) into every request. It reports the effective set — after config.ModelConfig.BuiltinTools and any construction options — under the provider-neutral names the config block uses.
Purpose is visibility: these tools never surface as a tool call core-agent can see, so without a report there is no way to confirm from the outside whether a `builtin_tools` key took effect or was discarded as a typo. Backends with no server-side tools (echo, scripted) simply don't implement it.
The report is provider-level and therefore an upper bound: a backend may still drop a built-in from an individual request (model compatibility, an explicitly suppressed one-shot). Implementations must only ever subtract that way, so an absent name means a tool that cannot be reached at all.
type Constructor ¶
Constructor builds a Provider from validated config. Tests register alternates via Register so resolution stays decoupled from the imports of any single backend.
type GeminiAPI ¶
type GeminiAPI struct {
// APIKey overrides GOOGLE_API_KEY / GEMINI_API_KEY. Empty falls
// back to the environment, same as the config path.
APIKey string
}
GeminiAPI selects the Gemini API backend (api-key auth). The model ID is NOT part of routing — pass it to Provider.Model(ctx, name) afterwards, exactly as with Resolve.
type GeminiVertex ¶
type GeminiVertex struct {
// Project and Location identify the Vertex deployment. Empty
// values fall back to GOOGLE_CLOUD_PROJECT / GOOGLE_CLOUD_LOCATION,
// same as the config path.
Project string
Location string
}
GeminiVertex selects Gemini via Vertex AI (ADC auth).
type PriorSuccess ¶
type PriorSuccess = callctx.PriorSuccess
PriorSuccess records that a model call in one agent session has already succeeded (#1247). It is the evidence that licenses RetryPolicy.IsTransientAfterSuccess: a rejection that is ambiguous on its own — Vertex's bare 400 INVALID_ARGUMENT with no details — is worth one retry only when the same session, model and configuration have already been served, because then the request cannot be malformed from the start.
The agent owns one per session and marks it from its event loop when a model response arrives; the retry policy only reads it. The record travels on the call's context (WithPriorSuccess), the same way AsSideCall does. A nil *PriorSuccess is valid and never succeeded.
It is core-models' callctx.PriorSuccess, so the library's HTTP-layer retry and this package's RetryPolicy read the same record.
func NewPriorSuccess ¶
func NewPriorSuccess() *PriorSuccess
NewPriorSuccess returns an unmarked record.
func PriorSuccessFrom ¶
func PriorSuccessFrom(ctx context.Context) *PriorSuccess
PriorSuccessFrom returns the record WithPriorSuccess put on ctx, or nil.
type Provider ¶
type Provider interface {
// Name reports the provider identity ("gemini", "vertex", "anthropic").
// Used for telemetry and diagnostic messages.
Name() string
// Model returns a usable model.LLM for the given model ID. The same
// Provider may be asked for several models over its lifetime.
Model(ctx context.Context, modelID string) (model.LLM, error)
}
Provider constructs concrete model.LLM instances on demand. A Provider is bound to one credential source (API key, Vertex project, etc.) at construction time; callers ask for specific models by ID through Model.
func New ¶
func New(opts ProviderOptions) (Provider, error)
New constructs a Provider from per-provider options, routing through the same registry as Resolve — remember to blank-import the backend package (e.g. pkg/models/gemini) exactly as with Resolve. For auto-detection from the environment, use Resolve with a default config (or AutoDetectProvider for the name alone); New is deliberately explicit about the backend.
type ProviderOptions ¶
type ProviderOptions interface {
// contains filtered or unexported methods
}
ProviderOptions is the programmatic counterpart of the config file's `model` block (#492): before it, the only way through the provider registry — Resolve — was to fabricate the on-disk *config.Config struct, which coupled every programmatic embedder to the file schema. Each per-provider struct below carries exactly the fields its backend routes on; New translates it through the same registry Resolve uses, so both paths construct identical providers. The MODEL ID is deliberately not routing state: pass it to Provider.Model(ctx, name) on the returned provider, exactly as with Resolve.
Sealed (the toModelConfig method is unexported) so the set of option shapes evolves with the registry rather than by third-party implementation. Backends with richer knobs (thinking budgets, server-side tools, cache flags) keep exposing them on their own constructors and functional options — gemini.NewAPIKey / gemini.NewVertex / anthropic.New / anthropic.NewVertex remain the full-control path; these structs cover the routing layer's job: pick a backend and point it at credentials. Pass structs by value (a nil *GeminiAPI etc. would panic in the value-receiver call).
Resolve remains the config-file adapter, unchanged.
type RetryError ¶
type RetryError struct {
// Outcome is "persisted", "abandoned", "skipped", "failed" or
// "interrupted".
Outcome string
Err error
}
RetryError is how a retry that did not rescue its call reaches the caller (#1206): the rejection, persisted after one retry, abandoned during the backoff, or not retried because the budget was spent.
Before it, the daemon log was the only place a retry existed: a turn error read the same whether or not a retry had been tried, and the 2026-09-13 batch counted 13 retries in the log and 0 in 21 transcripts. The outcome is a prefix, not a suffix, because the turn-error frame keeps only the first 240 characters and a provider error is often longer than that; and it names no word the turn-error classifier keys on, so the frame's kind and code are the rejection's own. Unwrap keeps errors.As / errors.Is working on the provider error underneath.
Two outcomes are not the rejection at all, and still name the retry because the daemon log counted it: "failed" is a retry answered by a different error (a 400, an empty response), and "interrupted" is a retry that recovered and whose stream then failed before the final response that would have carried the ProviderRetryMetadataKey stamp.
func (*RetryError) Error ¶
func (e *RetryError) Error() string
func (*RetryError) RetryPrefix ¶
func (e *RetryError) RetryPrefix() string
RetryPrefix returns the text Error puts before the provider error, so a surface that replaces the message — the turn-error frame's fixed text for a timeout or a cancel — can keep it.
func (*RetryError) Unwrap ¶
func (e *RetryError) Unwrap() error
type RetryPolicy ¶
type RetryPolicy struct {
// IsTransient reports whether err is a transient provider
// rejection worth one retry. Nil disables retrying entirely.
IsTransient func(error) bool
// IsTransientAfterSuccess reports whether err is a rejection that
// is ambiguous on its own and is worth one retry only after the
// calling session has already been served (#1247). It is consulted
// only for a call whose context carries a marked PriorSuccess and
// is not a side call; anywhere else the error is judged by
// IsTransient alone, which is today's behaviour. A retry it licenses
// spends the same budget, logs the same lines and surfaces the same
// way as any other. Nil disables it.
IsTransientAfterSuccess func(error) bool
// Backoff is how long to wait before the retry. Zero means
// DefaultRetryBackoff.
Backoff time.Duration
// Cooldown is the refill interval of this policy's retry budget:
// one retry is returned to the bucket per Cooldown, up to a
// ceiling of RetryBurst. Zero means DefaultRetryCooldown. A
// negative value disables the guard, which is not recommended and
// exists so a test can exercise back-to-back retries.
Cooldown time.Duration
// Log receives one line per retry decision, so a recovered failure
// is visible in the daemon log rather than silent. Nil discards.
Log func(format string, args ...any)
// contains filtered or unexported fields
}
RetryPolicy retries a streaming model call once when the provider rejects it transiently, and does nothing otherwise.
Why this exists ¶
The evidence is the GKE drill archive (#935). Across 38 archived runs against a live cluster, four lost a subagent delegation to a Vertex 429 RESOURCE_EXHAUSTED — three of them inside a single 20-run batch, so ~15% of that batch. In three of the four the child died on its very first model call, having recorded a plan and read nothing, and the parent then quietly did the cluster work itself. So the visible symptom is not a failed run. It is a run that silently loses the context isolation delegation exists to provide, at full parent-context cost, and in one of the four the operator was never told.
Why retrying is safe here, which is not obvious ¶
A retry that fires while the provider is shedding load adds requests at exactly the wrong moment, and that objection is the reason this was split out of #898 rather than shipped with it. The archive answers it: in 4 of 4 cases the parent's next model-backed call — same session, same model, 3 to 4 seconds later — succeeded. These are momentary rejections inside a window the provider is otherwise serving, which is the case a single retry is for.
The counter-evidence is real and shapes the guard: two of the four arrived 8 minutes apart at the tail of a sustained batch, so cumulative pressure exists even though no individual rejection persisted. That is why the budget is not optional. It bounds the extra load this can generate to a steady state of one additional request per Cooldown per policy, whatever the provider is doing — so a genuine shed, where every call fails, costs a few retries and then degrades to plain pass-through instead of doubling traffic.
The budget is a token bucket, not a single timestamp (#1039). A timestamp is right for the steady state and wrong for the burst: 429s are correlated, several concurrent callers get rejected within seconds of each other, and a one-rescue-per-window budget gives that rescue to whichever was rejected first and abandons the rest. See RetryBurst.
What it will not do ¶
It retries only when NO usable content reached the caller. Once a response with content has been yielded, a retry would duplicate it, so from that point the stream is pass-through and a late error surfaces unchanged. Buffer-and-drop-partials is lifted from the Gemini cache-eviction wrapper, which solved the same problem for a different trigger: chunks are held until the retry decision is moot, and discarded if the retry fires, so the retry supersedes any partial content rather than appending to it.
One retry. Two attempts total, never more.
Per-provider predicates over a shared mechanism ¶
IsTransient is supplied by the adapter because providers do not spell transience the same way and some cannot distinguish it at all. The mechanism — buffering, the attempt cap, the cooldown, the backoff — is identical for everyone, and a Gemini-only version of it would leave the second shipped provider permanently less resilient.
A zero RetryPolicy, or one with a nil IsTransient, is a no-op pass-through. That is the intended behaviour for a provider that cannot tell a transient rejection from a permanent one: guessing is worse than not retrying.
A RetryPolicy is safe for concurrent use and is meant to be shared across the calls of one provider instance — the budget is only meaningful if it is.
func (*RetryPolicy) Wrap ¶
func (p *RetryPolicy) Wrap(ctx context.Context, fn func() iter.Seq2[*adkmodel.LLMResponse, error]) iter.Seq2[*adkmodel.LLMResponse, error]
Wrap returns an iterator that runs fn, and on a transient failure that produced no usable content, runs it once more.
fn is an iterator FACTORY rather than an iterator: a retry has to re-invoke the underlying call, and an already-constructed iter.Seq2 cannot be replayed.
type SmallModelDefaulter ¶
type SmallModelDefaulter interface {
DefaultSmallModel() string
}
SmallModelDefaulter is an optional Provider extension. A Provider that implements this declares its preferred cheap-tier model — used by core-agent as the default for --agentic-small-model when the operator hasn't pinned one explicitly. Providers without a cheap-tier concept (echo, scripted) simply don't implement this; ResolveSmallModel returns "" for them and callers fall back to inheriting the parent's model.
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
Package anthropic implements models.Provider for Anthropic / Claude, on the first-party API and on Vertex AI.
|
Package anthropic implements models.Provider for Anthropic / Claude, on the first-party API and on Vertex AI. |
|
Package gemini implements models.Provider for Gemini, on the Gemini Developer API ("gemini") and on Vertex AI ("vertex").
|
Package gemini implements models.Provider for Gemini, on the Gemini Developer API ("gemini") and on Vertex AI ("vertex"). |
|
Package mock ships two credential-free LLM providers and a recording wrapper that pair for offline testing of agent flows:
|
Package mock ships two credential-free LLM providers and a recording wrapper that pair for offline testing of agent flows: |
|
Package profiles is the models.Provider for provider profiles: every model.provider beyond core-agent's own gemini / vertex / anthropic / anthropic-vertex / echo / scripted.
|
Package profiles is the models.Provider for provider profiles: every model.provider beyond core-agent's own gemini / vertex / anthropic / anthropic-vertex / echo / scripted. |