Documentation
¶
Overview ¶
Package profile declares how to reach a model: endpoint, credential, wire dialect, capabilities, and the models served.
A model is named by (profile, model id), not by a model id alone. The same id is served by several backends at several prices with several credentials — gpt-oss-20b by Vertex AI and by an operator's vLLM — so the id cannot pick the transport, the credential or the price, and a profile does (docs/design.md §5).
This package owns the schema and the built-in profiles. Where an operator's profiles are read from is each product's decision; the struct carries json and yaml tags so either format decodes into it, and DecodeJSON is the strict JSON reader.
A profile is checked at two moments, and both fail before any request is sent (R1): Validate checks its shape, and Resolve checks it against the environment — variables set, credentials found.
Index ¶
- Variables
- func BuiltinNames() []string
- func Has(c *bool) bool
- type Capabilities
- type Dialect
- type MetricsFormat
- type Model
- type Options
- type Platform
- type PricingMode
- type Profile
- func (p Profile) BackendName() string
- func (p Profile) CapabilitiesFor(id string) Capabilities
- func (p Profile) ContextWindowFor(id string) (int, bool)
- func (p Profile) DefaultSmallModel() string
- func (p Profile) ExtraBodyFor(id string) map[string]any
- func (p Profile) IsolatedPricing() bool
- func (p Profile) Open() bool
- func (p Profile) RatesFor(id string) (Rates, bool)
- func (p Profile) Serves(id string) bool
- func (p Profile) Validate() error
- type Rates
- type ReasoningFormat
- type Reliability
- type Resolved
- type Tier
- type UsageOverrides
Constants ¶
This section is empty.
Variables ¶
var ReservedBodyKeys = []string{
"model", "messages", "tools", "tool_choice", "stream", "stream_options",
"max_tokens", "max_completion_tokens", "temperature", "top_p", "stop",
"seed", "presence_penalty", "frequency_penalty", "response_format", "n",
}
ReservedBodyKeys are the request fields an adapter sets itself, which ExtraBody may not override.
Functions ¶
Types ¶
type Capabilities ¶
type Capabilities struct {
// ResponseSchema: the server enforces a JSON-schema response format.
// Many self-hosted servers accept the field and ignore it unless a
// guided-decoding backend is enabled — declare it only when it is.
ResponseSchema *bool `json:"response_schema,omitempty" yaml:"response_schema,omitempty"`
// ReasoningEcho: reasoning from one turn is carried into the next.
ReasoningEcho *bool `json:"reasoning_echo,omitempty" yaml:"reasoning_echo,omitempty"`
// ParallelToolCalls: the model may call several tools in one turn.
ParallelToolCalls *bool `json:"parallel_tool_calls,omitempty" yaml:"parallel_tool_calls,omitempty"`
// ServerTools: the provider runs built-in tools (search, code).
ServerTools *bool `json:"server_tools,omitempty" yaml:"server_tools,omitempty"`
// Streaming: the server streams responses.
Streaming *bool `json:"streaming,omitempty" yaml:"streaming,omitempty"`
// ForcedToolChoice: the model accepts a forced tool call — "required",
// or one named tool. Unset means it does. Declared false (gpt-oss on
// Vertex AI, per Google's function-calling notes), a forced choice is
// sent as "auto" instead and the response is marked; the alternative
// is a request the server rejects or silently ignores.
ForcedToolChoice *bool `json:"forced_tool_choice,omitempty" yaml:"forced_tool_choice,omitempty"`
}
Capabilities are declared, never probed. Nil means not declared, which a product reads as absent.
type Dialect ¶
type Dialect string
Dialect is the wire format a profile's server speaks.
const ( // OpenAIChat is POST /v1/chat/completions: Vertex AI partner models, // xAI, vLLM, SGLang, Ollama, llama.cpp, NIM and the managed long // tail. OpenAIChat Dialect = "openai-chat" // OpenAIResponses is POST /v1/responses: OpenAI first-party, xAI. OpenAIResponses Dialect = "openai-responses" // Anthropic is the Messages API: first-party, Vertex, Bedrock. Anthropic Dialect = "anthropic" // Gemini is genai's generateContent: the Developer API and Vertex. Gemini Dialect = "gemini" )
type MetricsFormat ¶ added in v0.10.0
type MetricsFormat string
MetricsFormat names the metric vocabulary a server's Prometheus endpoint uses, for package kvmetrics.
const ( // MetricsVLLM: vllm:prefix_cache_{queries,hits}_total and // vllm:kv_cache_usage_perc. MetricsVLLM MetricsFormat = "vllm" // MetricsSGLang: sglang:cache_hit_rate, sglang:token_usage, and the // prompt/cached token counters where the build has them. MetricsSGLang MetricsFormat = "sglang" // MetricsLlamaCpp: llama.cpp server's llamacpp:* metrics. MetricsLlamaCpp MetricsFormat = "llamacpp" // MetricsNone: the server exposes no KV-cache statistics (Ollama). MetricsNone MetricsFormat = "none" )
type Model ¶
type Model struct {
ID string `json:"id" yaml:"id"`
Tier Tier `json:"tier,omitempty" yaml:"tier,omitempty"`
// ContextWindow is the model's input window in tokens, as this
// server serves it, which can be smaller than the model's own (vLLM
// --max-model-len). Hosts size compaction from it
// (ContextWindowFor); zero means not declared.
ContextWindow int `json:"context_window,omitempty" yaml:"context_window,omitempty"`
// Capabilities override the profile's for this model only.
Capabilities Capabilities `json:"capabilities,omitzero" yaml:"capabilities,omitempty"`
// ExtraBody adds vendor fields to this model's requests, over the
// profile's ExtraBody.
ExtraBody map[string]any `json:"extra_body,omitempty" yaml:"extra_body,omitempty"`
// Rates are the operator's declared price for this model, for a
// backend no published catalog covers (a self-hosted server). A
// product prices from its catalog first and falls back to these.
Rates *Rates `json:"rates,omitempty" yaml:"rates,omitempty"`
}
Model is one model a profile serves.
type Options ¶
type Options struct {
// Getenv replaces os.Getenv, for ${VAR} expansion and credentials.
Getenv func(string) string
// Auth is passed to auth.Config.Resolve.
Auth auth.Options
}
Options adjusts Resolve.
type Platform ¶ added in v0.6.0
type Platform string
Platform is where a vendor's API is hosted; see Profile.Platform.
const PlatformSAPOrchestration Platform = "sap-orchestration"
PlatformSAPOrchestration is SAP AI Core's orchestration service: Chat Completions wrapped in SAP's envelope, sent to a deployment found at startup (package openaichat, docs/design.md §7.1). openai-chat only.
const PlatformVertex Platform = "vertex"
PlatformVertex is Google Cloud's Vertex AI.
type PricingMode ¶ added in v0.15.0
type PricingMode string
PricingMode says which published prices apply to a profile's backend.
const ( // first, then the model's bare row, as package pricing resolves them. PricingShared PricingMode = "shared" // PricingIsolated: only rows keyed with the backend apply, so a model // a gateway also offers first-party is never billed at that list // price. An operator prices such a backend by declaring models[].rates // or a "<backend>/<model>" row in pricing.json. PricingIsolated PricingMode = "isolated" )
type Profile ¶
type Profile struct {
// Name identifies the profile: what --provider names and what a
// product's config refers to. Lowercase letters, digits and '-'.
Name string `json:"name" yaml:"name"`
// Extends names a built-in profile this one starts from; every field
// set here overrides the built-in's.
Extends string `json:"extends,omitempty" yaml:"extends,omitempty"`
// Dialect is the server's wire format.
Dialect Dialect `json:"dialect,omitempty" yaml:"dialect,omitempty"`
// Platform is where a vendor's API is hosted, when that changes the
// request: empty for the vendor's own service, PlatformVertex for
// Vertex AI, which takes the model in the URL path,
// PlatformSAPOrchestration for SAP AI Core. Said explicitly
// rather than inferred from the credential or the backend name,
// because a proxy in front of Vertex may take a bearer token, and a
// backend can be renamed for pricing alone.
Platform Platform `json:"platform,omitempty" yaml:"platform,omitempty"`
// BaseURL is the API root. It may contain {param} placeholders,
// filled from Params.
BaseURL string `json:"base_url,omitempty" yaml:"base_url,omitempty"`
// Params fill BaseURL's placeholders. A value may be ${VAR} or
// ${VAR:-default}, read from the environment at Resolve.
Params map[string]string `json:"params,omitempty" yaml:"params,omitempty"`
// Auth is the credential.
Auth auth.Config `json:"auth,omitzero" yaml:"auth,omitempty"`
// Headers are sent on every request to the model endpoint, for a
// gateway that needs one that is not the credential: SAP AI Core's
// AI-Resource-Group, an API-management subscription key, a tenant or
// routing header, an anthropic-beta opt-in. A value may be ${VAR} or
// ${VAR:-default}, read at Resolve; one that comes out empty is not
// sent. Credentials and the headers the dialects set themselves
// (Content-Type, Accept, User-Agent, …) cannot be set here. Put a
// secret value in the environment and reference it, rather than
// writing it into the profile.
Headers map[string]string `json:"headers,omitempty" yaml:"headers,omitempty"`
// Backend is the identity prices are keyed on, beside the model id.
// Defaults to Name.
Backend string `json:"backend,omitempty" yaml:"backend,omitempty"`
// Pricing says whether published list prices apply to Backend.
// "isolated" is for a gateway that bills in its own units (SAP AI
// Core's capacity units): package pricing then prices the backend
// only from rows keyed "<backend>/<model>", never from a model's
// first-party or bare row. Unset inherits; "shared" is the default.
// pricing honors built-in profiles' modes on its own, and a
// product's profiles once they are passed in pricing.Options.Profiles.
Pricing PricingMode `json:"pricing,omitempty" yaml:"pricing,omitempty"`
// OpenModels admits any model id, not only those in Models: a
// self-hosted server serves whatever weights the operator loaded.
OpenModels *bool `json:"open_models,omitempty" yaml:"open_models,omitempty"`
// Capabilities are what the server honors. Products refuse at
// startup a workload that needs one the profile lacks.
Capabilities Capabilities `json:"capabilities,omitzero" yaml:"capabilities,omitempty"`
// Usage overrides how reported usage fields are trusted.
Usage UsageOverrides `json:"usage,omitzero" yaml:"usage,omitempty"`
// ReasoningFormat says where the server puts reasoning.
ReasoningFormat ReasoningFormat `json:"reasoning_format,omitempty" yaml:"reasoning_format,omitempty"`
// ExtraBody adds vendor-specific fields to every request: switches a
// server takes that the dialect does not model, such as
// chat_template_kwargs {enable_thinking: true} for Gemma 4 on vLLM,
// reasoning_effort, or top_k. A field the adapter sets itself (model,
// messages, tools, …) cannot be set here.
ExtraBody map[string]any `json:"extra_body,omitempty" yaml:"extra_body,omitempty"`
// MetricsURL is the server's Prometheus endpoint, for the opt-in
// KV-cache sampler (docs/design.md §8, package kvmetrics).
MetricsURL string `json:"metrics_url,omitempty" yaml:"metrics_url,omitempty"`
// Metrics names which server's metric names MetricsURL exposes, so
// kvmetrics knows how to read it. The vllm, sglang and ollama
// built-ins set it, and a profile that extends one inherits it.
Metrics MetricsFormat `json:"metrics,omitempty" yaml:"metrics,omitempty"`
// Models lists the models the profile serves.
Models []Model `json:"models,omitempty" yaml:"models,omitempty"`
// Tiers names the model for each tier this profile can fill.
Tiers map[Tier]string `json:"tiers,omitempty" yaml:"tiers,omitempty"`
// contains filtered or unexported fields
}
Profile is one declared way to reach models.
func DecodeJSON ¶
DecodeJSON reads a JSON array of profiles, refusing unknown fields: a misspelled key ("base_ulr") is an error, not a silently missing value.
func Expand ¶
Expand applies Extends: the result is the named built-in with every field p sets laid over it. Params, Headers and Tiers merge key by key (Headers case-insensitively, so p's spelling of a name replaces the built-in's, and an empty value drops it); Models and Auth replace as a whole; each capability overrides on its own. A profile without Extends is returned unchanged.
func Find ¶
Find returns the profile named name: one of declared if any has that name, else a built-in. A name declared twice is an error, since the second would silently shadow the first.
func (Profile) BackendName ¶
BackendName is Backend, defaulting to Name.
func (Profile) CapabilitiesFor ¶ added in v0.2.0
func (p Profile) CapabilitiesFor(id string) Capabilities
CapabilitiesFor is the profile's capabilities with the listed model's own overrides laid over them. One profile serves many models, and what a server honors can differ between two models it serves (Kimi wants reasoning echoed, gpt-oss refuses a forced tool choice), so a capability is declared per profile and refined per model.
func (Profile) ContextWindowFor ¶ added in v0.8.0
ContextWindowFor returns the input window a profile declares for model id, in tokens, and false when it declares none. A host sizing compaction falls back to its own table on false: core-models has no catalog of windows.
func (Profile) DefaultSmallModel ¶
DefaultSmallModel is the profile's small-tier model, or "".
func (Profile) ExtraBodyFor ¶ added in v0.4.0
ExtraBodyFor is the profile's ExtraBody with the listed model's laid over it, key by key. Nil when neither sets any.
func (Profile) IsolatedPricing ¶ added in v0.15.0
IsolatedPricing reports whether p's backend is priced only from rows keyed with it (Pricing "isolated"). Read it on an expanded profile.
type Rates ¶ added in v0.5.0
type Rates struct {
InputPerMTok float64 `json:"input_per_mtok,omitempty" yaml:"input_per_mtok,omitempty"`
CachedInputPerMTok float64 `json:"cached_input_per_mtok,omitempty" yaml:"cached_input_per_mtok,omitempty"`
CacheWritePerMTok float64 `json:"cache_write_per_mtok,omitempty" yaml:"cache_write_per_mtok,omitempty"`
OutputPerMTok float64 `json:"output_per_mtok,omitempty" yaml:"output_per_mtok,omitempty"`
}
Rates are USD per million tokens. Declare what the model costs you, for example GPU-hours divided by measured throughput. All zero is refused: a free model makes a cost ceiling a ceiling that never trips.
type ReasoningFormat ¶
type ReasoningFormat string
ReasoningFormat says where a server puts a model's reasoning.
const ( // ReasoningField (the default): in reasoning_content or reasoning, // apart from the answer, or not at all. ReasoningField ReasoningFormat = "" // ThinkTags: inline at the start of the answer as <think>…</think> // (Ollama; vLLM and SGLang without a reasoning parser). The adapter // splits the block out, so it never reaches the product as answer // text. ThinkTags ReasoningFormat = "think_tags" )
type Reliability ¶
type Reliability string
Reliability says whether a usage field a server reports can be trusted.
const ( // Reported (the default) records the field when the server sends it. Reported Reliability = "" // Unreliable records the field as not reported even when the server // sends it — for a server build known to send nonsense, such as the // vLLM V1 builds that null or zero prompt_tokens_details. Unreliable Reliability = "unreliable" )
type Resolved ¶
type Resolved struct {
Profile Profile
// Params are the profile's params with every ${VAR} read and the
// derived ones (vertex-maas's {host}) added.
Params map[string]string
BaseURL string
MetricsURL string
Credential *auth.Credential
// contains filtered or unexported fields
}
Resolved is a profile checked against the environment: every placeholder filled, every variable read, the credential found.
func Resolve ¶
Resolve expands, validates and resolves p. Any failure names the profile and what is missing; nothing here sends a request.
func (*Resolved) HeaderNames ¶ added in v0.14.0
HeaderNames lists the headers the profile sends, canonical and sorted, without their values, which may be secrets.
func (*Resolved) HeaderTransport ¶ added in v0.14.0
func (r *Resolved) HeaderTransport(base http.RoundTripper) http.RoundTripper
HeaderTransport returns base with the profile's headers set on every request, or base itself when the profile has none. Nil base means http.DefaultTransport.
Open puts it under every dialect's transport, so the headers reach the model endpoint and nothing else: not a token endpoint, not another host a redirect points to, and a metrics endpoint only when it shares the API's origin (package kvmetrics). A host that builds a dialect itself rather than through Open (gemini.New, to share one client for its context cache) wraps the HTTPClient it passes with this.
type UsageOverrides ¶
type UsageOverrides struct {
CachedTokens Reliability `json:"cached_tokens,omitempty" yaml:"cached_tokens,omitempty"`
}
UsageOverrides adjust how reported usage is trusted.