profile

package
v0.15.3 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 14, 2026 License: Apache-2.0 Imports: 13 Imported by: 0

Documentation

Overview

Package profile declares how to reach a model: endpoint, credential, wire dialect, capabilities, and the models served.

A model is named by (profile, model id), not by a model id alone. The same id is served by several backends at several prices with several credentials — gpt-oss-20b by Vertex AI and by an operator's vLLM — so the id cannot pick the transport, the credential or the price, and a profile does (docs/design.md §5).

This package owns the schema and the built-in profiles. Where an operator's profiles are read from is each product's decision; the struct carries json and yaml tags so either format decodes into it, and DecodeJSON is the strict JSON reader.

A profile is checked at two moments, and both fail before any request is sent (R1): Validate checks its shape, and Resolve checks it against the environment — variables set, credentials found.

Index

Constants

This section is empty.

Variables

View Source
var ReservedBodyKeys = []string{
	"model", "messages", "tools", "tool_choice", "stream", "stream_options",
	"max_tokens", "max_completion_tokens", "temperature", "top_p", "stop",
	"seed", "presence_penalty", "frequency_penalty", "response_format", "n",
}

ReservedBodyKeys are the request fields an adapter sets itself, which ExtraBody may not override.

Functions

func BuiltinNames

func BuiltinNames() []string

BuiltinNames lists the built-in profiles, sorted.

func Has

func Has(c *bool) bool

Has reports whether a declared capability is true.

Types

type Capabilities

type Capabilities struct {
	// ResponseSchema: the server enforces a JSON-schema response format.
	// Many self-hosted servers accept the field and ignore it unless a
	// guided-decoding backend is enabled — declare it only when it is.
	ResponseSchema *bool `json:"response_schema,omitempty" yaml:"response_schema,omitempty"`
	// ReasoningEcho: reasoning from one turn is carried into the next.
	ReasoningEcho *bool `json:"reasoning_echo,omitempty" yaml:"reasoning_echo,omitempty"`
	// ParallelToolCalls: the model may call several tools in one turn.
	ParallelToolCalls *bool `json:"parallel_tool_calls,omitempty" yaml:"parallel_tool_calls,omitempty"`
	// ServerTools: the provider runs built-in tools (search, code).
	ServerTools *bool `json:"server_tools,omitempty" yaml:"server_tools,omitempty"`
	// Streaming: the server streams responses.
	Streaming *bool `json:"streaming,omitempty" yaml:"streaming,omitempty"`
	// ForcedToolChoice: the model accepts a forced tool call — "required",
	// or one named tool. Unset means it does. Declared false (gpt-oss on
	// Vertex AI, per Google's function-calling notes), a forced choice is
	// sent as "auto" instead and the response is marked; the alternative
	// is a request the server rejects or silently ignores.
	ForcedToolChoice *bool `json:"forced_tool_choice,omitempty" yaml:"forced_tool_choice,omitempty"`
}

Capabilities are declared, never probed. Nil means not declared, which a product reads as absent.

type Dialect

type Dialect string

Dialect is the wire format a profile's server speaks.

const (
	// OpenAIChat is POST /v1/chat/completions: Vertex AI partner models,
	// xAI, vLLM, SGLang, Ollama, llama.cpp, NIM and the managed long
	// tail.
	OpenAIChat Dialect = "openai-chat"
	// OpenAIResponses is POST /v1/responses: OpenAI first-party, xAI.
	OpenAIResponses Dialect = "openai-responses"
	// Anthropic is the Messages API: first-party, Vertex, Bedrock.
	Anthropic Dialect = "anthropic"
	// Gemini is genai's generateContent: the Developer API and Vertex.
	Gemini Dialect = "gemini"
)

type MetricsFormat added in v0.10.0

type MetricsFormat string

MetricsFormat names the metric vocabulary a server's Prometheus endpoint uses, for package kvmetrics.

const (
	// MetricsVLLM: vllm:prefix_cache_{queries,hits}_total and
	// vllm:kv_cache_usage_perc.
	MetricsVLLM MetricsFormat = "vllm"
	// MetricsSGLang: sglang:cache_hit_rate, sglang:token_usage, and the
	// prompt/cached token counters where the build has them.
	MetricsSGLang MetricsFormat = "sglang"
	// MetricsLlamaCpp: llama.cpp server's llamacpp:* metrics.
	MetricsLlamaCpp MetricsFormat = "llamacpp"
	// MetricsNone: the server exposes no KV-cache statistics (Ollama).
	MetricsNone MetricsFormat = "none"
)

type Model

type Model struct {
	ID   string `json:"id" yaml:"id"`
	Tier Tier   `json:"tier,omitempty" yaml:"tier,omitempty"`
	// ContextWindow is the model's input window in tokens, as this
	// server serves it, which can be smaller than the model's own (vLLM
	// --max-model-len). Hosts size compaction from it
	// (ContextWindowFor); zero means not declared.
	ContextWindow int `json:"context_window,omitempty" yaml:"context_window,omitempty"`
	// Capabilities override the profile's for this model only.
	Capabilities Capabilities `json:"capabilities,omitzero" yaml:"capabilities,omitempty"`
	// ExtraBody adds vendor fields to this model's requests, over the
	// profile's ExtraBody.
	ExtraBody map[string]any `json:"extra_body,omitempty" yaml:"extra_body,omitempty"`
	// Rates are the operator's declared price for this model, for a
	// backend no published catalog covers (a self-hosted server). A
	// product prices from its catalog first and falls back to these.
	Rates *Rates `json:"rates,omitempty" yaml:"rates,omitempty"`
}

Model is one model a profile serves.

type Options

type Options struct {
	// Getenv replaces os.Getenv, for ${VAR} expansion and credentials.
	Getenv func(string) string
	// Auth is passed to auth.Config.Resolve.
	Auth auth.Options
}

Options adjusts Resolve.

type Platform added in v0.6.0

type Platform string

Platform is where a vendor's API is hosted; see Profile.Platform.

const PlatformSAPOrchestration Platform = "sap-orchestration"

PlatformSAPOrchestration is SAP AI Core's orchestration service: Chat Completions wrapped in SAP's envelope, sent to a deployment found at startup (package openaichat, docs/design.md §7.1). openai-chat only.

const PlatformVertex Platform = "vertex"

PlatformVertex is Google Cloud's Vertex AI.

type PricingMode added in v0.15.0

type PricingMode string

PricingMode says which published prices apply to a profile's backend.

const (
	// PricingShared (the default, also spelled ""): a backend's own rows
	// first, then the model's bare row, as package pricing resolves them.
	PricingShared PricingMode = "shared"
	// PricingIsolated: only rows keyed with the backend apply, so a model
	// a gateway also offers first-party is never billed at that list
	// price. An operator prices such a backend by declaring models[].rates
	// or a "<backend>/<model>" row in pricing.json.
	PricingIsolated PricingMode = "isolated"
)

type Profile

type Profile struct {
	// Name identifies the profile: what --provider names and what a
	// product's config refers to. Lowercase letters, digits and '-'.
	Name string `json:"name" yaml:"name"`
	// Extends names a built-in profile this one starts from; every field
	// set here overrides the built-in's.
	Extends string `json:"extends,omitempty" yaml:"extends,omitempty"`
	// Dialect is the server's wire format.
	Dialect Dialect `json:"dialect,omitempty" yaml:"dialect,omitempty"`
	// Platform is where a vendor's API is hosted, when that changes the
	// request: empty for the vendor's own service, PlatformVertex for
	// Vertex AI, which takes the model in the URL path,
	// PlatformSAPOrchestration for SAP AI Core. Said explicitly
	// rather than inferred from the credential or the backend name,
	// because a proxy in front of Vertex may take a bearer token, and a
	// backend can be renamed for pricing alone.
	Platform Platform `json:"platform,omitempty" yaml:"platform,omitempty"`
	// BaseURL is the API root. It may contain {param} placeholders,
	// filled from Params.
	BaseURL string `json:"base_url,omitempty" yaml:"base_url,omitempty"`
	// Params fill BaseURL's placeholders. A value may be ${VAR} or
	// ${VAR:-default}, read from the environment at Resolve.
	Params map[string]string `json:"params,omitempty" yaml:"params,omitempty"`
	// Auth is the credential.
	Auth auth.Config `json:"auth,omitzero" yaml:"auth,omitempty"`
	// Headers are sent on every request to the model endpoint, for a
	// gateway that needs one that is not the credential: SAP AI Core's
	// AI-Resource-Group, an API-management subscription key, a tenant or
	// routing header, an anthropic-beta opt-in. A value may be ${VAR} or
	// ${VAR:-default}, read at Resolve; one that comes out empty is not
	// sent. Credentials and the headers the dialects set themselves
	// (Content-Type, Accept, User-Agent, …) cannot be set here. Put a
	// secret value in the environment and reference it, rather than
	// writing it into the profile.
	Headers map[string]string `json:"headers,omitempty" yaml:"headers,omitempty"`
	// Backend is the identity prices are keyed on, beside the model id.
	// Defaults to Name.
	Backend string `json:"backend,omitempty" yaml:"backend,omitempty"`
	// Pricing says whether published list prices apply to Backend.
	// "isolated" is for a gateway that bills in its own units (SAP AI
	// Core's capacity units): package pricing then prices the backend
	// only from rows keyed "<backend>/<model>", never from a model's
	// first-party or bare row. Unset inherits; "shared" is the default.
	// pricing honors built-in profiles' modes on its own, and a
	// product's profiles once they are passed in pricing.Options.Profiles.
	Pricing PricingMode `json:"pricing,omitempty" yaml:"pricing,omitempty"`
	// OpenModels admits any model id, not only those in Models: a
	// self-hosted server serves whatever weights the operator loaded.
	OpenModels *bool `json:"open_models,omitempty" yaml:"open_models,omitempty"`
	// Capabilities are what the server honors. Products refuse at
	// startup a workload that needs one the profile lacks.
	Capabilities Capabilities `json:"capabilities,omitzero" yaml:"capabilities,omitempty"`
	// Usage overrides how reported usage fields are trusted.
	Usage UsageOverrides `json:"usage,omitzero" yaml:"usage,omitempty"`
	// ReasoningFormat says where the server puts reasoning.
	ReasoningFormat ReasoningFormat `json:"reasoning_format,omitempty" yaml:"reasoning_format,omitempty"`
	// ExtraBody adds vendor-specific fields to every request: switches a
	// server takes that the dialect does not model, such as
	// chat_template_kwargs {enable_thinking: true} for Gemma 4 on vLLM,
	// reasoning_effort, or top_k. A field the adapter sets itself (model,
	// messages, tools, …) cannot be set here.
	ExtraBody map[string]any `json:"extra_body,omitempty" yaml:"extra_body,omitempty"`
	// MetricsURL is the server's Prometheus endpoint, for the opt-in
	// KV-cache sampler (docs/design.md §8, package kvmetrics).
	MetricsURL string `json:"metrics_url,omitempty" yaml:"metrics_url,omitempty"`
	// Metrics names which server's metric names MetricsURL exposes, so
	// kvmetrics knows how to read it. The vllm, sglang and ollama
	// built-ins set it, and a profile that extends one inherits it.
	Metrics MetricsFormat `json:"metrics,omitempty" yaml:"metrics,omitempty"`
	// Models lists the models the profile serves.
	Models []Model `json:"models,omitempty" yaml:"models,omitempty"`
	// Tiers names the model for each tier this profile can fill.
	Tiers map[Tier]string `json:"tiers,omitempty" yaml:"tiers,omitempty"`
	// contains filtered or unexported fields
}

Profile is one declared way to reach models.

func Builtin

func Builtin(name string) (Profile, bool)

Builtin returns the built-in profile named name.

func DecodeJSON

func DecodeJSON(data []byte) ([]Profile, error)

DecodeJSON reads a JSON array of profiles, refusing unknown fields: a misspelled key ("base_ulr") is an error, not a silently missing value.

func Expand

func Expand(p Profile) (Profile, error)

Expand applies Extends: the result is the named built-in with every field p sets laid over it. Params, Headers and Tiers merge key by key (Headers case-insensitively, so p's spelling of a name replaces the built-in's, and an empty value drops it); Models and Auth replace as a whole; each capability overrides on its own. A profile without Extends is returned unchanged.

func Find

func Find(name string, declared []Profile) (Profile, error)

Find returns the profile named name: one of declared if any has that name, else a built-in. A name declared twice is an error, since the second would silently shadow the first.

func (Profile) BackendName

func (p Profile) BackendName() string

BackendName is Backend, defaulting to Name.

func (Profile) CapabilitiesFor added in v0.2.0

func (p Profile) CapabilitiesFor(id string) Capabilities

CapabilitiesFor is the profile's capabilities with the listed model's own overrides laid over them. One profile serves many models, and what a server honors can differ between two models it serves (Kimi wants reasoning echoed, gpt-oss refuses a forced tool choice), so a capability is declared per profile and refined per model.

func (Profile) ContextWindowFor added in v0.8.0

func (p Profile) ContextWindowFor(id string) (int, bool)

ContextWindowFor returns the input window a profile declares for model id, in tokens, and false when it declares none. A host sizing compaction falls back to its own table on false: core-models has no catalog of windows.

func (Profile) DefaultSmallModel

func (p Profile) DefaultSmallModel() string

DefaultSmallModel is the profile's small-tier model, or "".

func (Profile) ExtraBodyFor added in v0.4.0

func (p Profile) ExtraBodyFor(id string) map[string]any

ExtraBodyFor is the profile's ExtraBody with the listed model's laid over it, key by key. Nil when neither sets any.

func (Profile) IsolatedPricing added in v0.15.0

func (p Profile) IsolatedPricing() bool

IsolatedPricing reports whether p's backend is priced only from rows keyed with it (Pricing "isolated"). Read it on an expanded profile.

func (Profile) Open

func (p Profile) Open() bool

Open reports whether the profile admits model ids it does not list.

func (Profile) RatesFor added in v0.5.0

func (p Profile) RatesFor(id string) (Rates, bool)

RatesFor returns the declared rates for a listed model.

func (Profile) Serves

func (p Profile) Serves(id string) bool

Serves reports whether the profile can serve model id.

func (Profile) Validate

func (p Profile) Validate() error

Validate checks the profile's shape without touching the environment. Every problem is reported, joined, each naming the profile, so one run shows them all.

type Rates added in v0.5.0

type Rates struct {
	InputPerMTok       float64 `json:"input_per_mtok,omitempty" yaml:"input_per_mtok,omitempty"`
	CachedInputPerMTok float64 `json:"cached_input_per_mtok,omitempty" yaml:"cached_input_per_mtok,omitempty"`
	CacheWritePerMTok  float64 `json:"cache_write_per_mtok,omitempty" yaml:"cache_write_per_mtok,omitempty"`
	OutputPerMTok      float64 `json:"output_per_mtok,omitempty" yaml:"output_per_mtok,omitempty"`
}

Rates are USD per million tokens. Declare what the model costs you, for example GPU-hours divided by measured throughput. All zero is refused: a free model makes a cost ceiling a ceiling that never trips.

type ReasoningFormat

type ReasoningFormat string

ReasoningFormat says where a server puts a model's reasoning.

const (
	// ReasoningField (the default): in reasoning_content or reasoning,
	// apart from the answer, or not at all.
	ReasoningField ReasoningFormat = ""
	// ThinkTags: inline at the start of the answer as <think>…</think>
	// (Ollama; vLLM and SGLang without a reasoning parser). The adapter
	// splits the block out, so it never reaches the product as answer
	// text.
	ThinkTags ReasoningFormat = "think_tags"
)

type Reliability

type Reliability string

Reliability says whether a usage field a server reports can be trusted.

const (
	// Reported (the default) records the field when the server sends it.
	Reported Reliability = ""
	// Unreliable records the field as not reported even when the server
	// sends it — for a server build known to send nonsense, such as the
	// vLLM V1 builds that null or zero prompt_tokens_details.
	Unreliable Reliability = "unreliable"
)

type Resolved

type Resolved struct {
	Profile Profile
	// Params are the profile's params with every ${VAR} read and the
	// derived ones (vertex-maas's {host}) added.
	Params     map[string]string
	BaseURL    string
	MetricsURL string
	Credential *auth.Credential
	// contains filtered or unexported fields
}

Resolved is a profile checked against the environment: every placeholder filled, every variable read, the credential found.

func Resolve

func Resolve(ctx context.Context, p Profile, opts Options) (*Resolved, error)

Resolve expands, validates and resolves p. Any failure names the profile and what is missing; nothing here sends a request.

func (*Resolved) HeaderNames added in v0.14.0

func (r *Resolved) HeaderNames() []string

HeaderNames lists the headers the profile sends, canonical and sorted, without their values, which may be secrets.

func (*Resolved) HeaderTransport added in v0.14.0

func (r *Resolved) HeaderTransport(base http.RoundTripper) http.RoundTripper

HeaderTransport returns base with the profile's headers set on every request, or base itself when the profile has none. Nil base means http.DefaultTransport.

Open puts it under every dialect's transport, so the headers reach the model endpoint and nothing else: not a token endpoint, not another host a redirect points to, and a metrics endpoint only when it shares the API's origin (package kvmetrics). A host that builds a dialect itself rather than through Open (gemini.New, to share one client for its context cache) wraps the HTTPClient it passes with this.

type Tier

type Tier string

Tier is the three-rung vocabulary products route on.

const (
	Small    Tier = "small"
	Mid      Tier = "mid"
	Frontier Tier = "frontier"
)

type UsageOverrides

type UsageOverrides struct {
	CachedTokens Reliability `json:"cached_tokens,omitempty" yaml:"cached_tokens,omitempty"`
}

UsageOverrides adjust how reported usage is trusted.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL