Documentation
¶
Overview ¶
Package model defines model alias resolution and model metadata services.
Index ¶
- Variables
- func Cost(u provider.Usage, p provider.Pricing, embedding bool) (*money.USD, usage.PricingStatus)
- type Alias
- type AliasRegistry
- type AliasTarget
- type Message
- type OverflowStrategy
- type PriceBook
- func (b *PriceBook) Lists(ctx context.Context, providerName, modelID string) bool
- func (b *PriceBook) OverridesErr() error
- func (b *PriceBook) Price(ctx context.Context, providerName string, modelIDs ...string) (provider.Pricing, bool)
- func (b *PriceBook) Priced(ctx context.Context, providerName, modelID string) bool
- func (b *PriceBook) SetOverrides(overrides []PriceOverride) error
- type PriceOverride
- type RequestTokenCounter
- type Service
- type TokenCounter
- type TokenEstimate
Constants ¶
This section is empty.
Variables ¶
var ErrUnpricedLongContext = errors.New("nexus: long-context tier has no input or output price")
ErrUnpricedLongContext rejects a price override whose long-context tier lacks an input price or an output price. With neither, Cost would charge a long prompt at the base price, which is not what the override says. With only one, it would charge the other side at $0 and a budget would undercount. SetOverrides wraps it with the provider, the model and the missing price; match it with errors.Is.
Functions ¶
func Cost ¶
Cost prices usage at a model's list prices, exactly. A completion charges its uncached prompt tokens at the input price, cache reads and writes at their own prices (see provider.Pricing for the fallbacks) and completion tokens at the output price. A request whose prompt is longer than the model's LongContextAbove is charged entirely at the LongContext tier. A tier with neither an input nor an output price is ignored and the request is charged at the base price, so a missing tier never turns a priced model into an unknown cost.
An embedding charges prompt tokens at the embedding price. An embedding provider that reports only a total (Voyage does) is priced from the total, since every token of an embedding is input.
When the model has no price for the tokens used it returns nil and PricingUnpricedModel, so an unknown cost is never reported as $0. A Free price list costs exactly $0.
The result is only as good as the token counts the provider reported: a provider that does not report its cache tokens has them charged as uncached input.
Types ¶
type Alias ¶
type Alias struct {
Name string `json:"name"` // "fast", "smart", "code", "embed"
Targets []AliasTarget `json:"targets"` // ordered fallback chain
TenantOverrides map[string][]AliasTarget `json:"tenant_overrides,omitempty"`
}
Alias maps a virtual model name to one or more concrete models.
type AliasRegistry ¶
type AliasRegistry interface {
// Resolve returns the concrete targets for a model name.
// If the name is already concrete, returns it as-is.
Resolve(ctx context.Context, name, tenantID string) ([]AliasTarget, error)
// Register adds or updates an alias.
Register(alias *Alias) error
// List returns all registered aliases.
List() []Alias
}
AliasRegistry resolves aliases at request time.
func NewAliasRegistry ¶
func NewAliasRegistry() AliasRegistry
NewAliasRegistry creates an in-memory alias registry.
type AliasTarget ¶
type AliasTarget struct {
Provider string `json:"provider"` // "openai", "anthropic"
Model string `json:"model"` // "gpt-4o-mini", "claude-3.5-haiku"
Weight float64 `json:"weight,omitempty"` // for weighted distribution
}
AliasTarget maps to a specific provider + model.
type Message ¶
Message is a minimal message type for token counting. It mirrors the essential fields from provider.Message.
type OverflowStrategy ¶
type OverflowStrategy string
OverflowStrategy defines how to handle context overflow.
const ( // OverflowError returns an error when context overflows. OverflowError OverflowStrategy = "error" // OverflowTruncateOldest truncates oldest messages to fit. OverflowTruncateOldest OverflowStrategy = "truncate_oldest" // OverflowTruncateMiddle removes middle messages, keeping first and last. OverflowTruncateMiddle OverflowStrategy = "truncate_middle" )
type PriceBook ¶
type PriceBook struct {
// contains filtered or unexported fields
}
PriceBook looks up list prices by provider and model. The same model ID can be sold by several providers at different prices (gpt-4o through openai, azureopenai and openrouter), so a price is only meaningful at the provider that served the request. Each provider's price list is read once and kept. Overrides (SetOverrides) are consulted before the lists.
func NewPriceBook ¶
NewPriceBook creates a price book over the given providers.
func (*PriceBook) Lists ¶
Lists reports whether the provider named providerName lists modelID, or has an override for it. It reads the same cached list as Price. Routing uses it to find the providers that serve a model, and telemetry to name only the models a provider lists, so a client's arbitrary model string never becomes a metric series or a span attribute.
func (*PriceBook) OverridesErr ¶
OverridesErr is the error of the latest SetOverrides call, or nil when it succeeded or none was made. A later successful call clears it.
func (*PriceBook) Price ¶
func (b *PriceBook) Price(ctx context.Context, providerName string, modelIDs ...string) (provider.Pricing, bool)
Price returns the price at providerName of the first of modelIDs with an override there, else the list price of the first of modelIDs the provider lists. A provider that implements provider.FreeOfCharge prices every model it has no override for at exactly $0. ok is false when no override matches and the provider is unknown, its list cannot be read, or it lists none of the models.
func (*PriceBook) Priced ¶
Priced reports whether modelID has a price at providerName: Free, or any of an input, output or embedding price that is not zero. A model the provider lists with no price is not priced.
func (*PriceBook) SetOverrides ¶
func (b *PriceBook) SetOverrides(overrides []PriceOverride) error
SetOverrides replaces every override with overrides. An entry with no provider or no model is ignored; of two entries for the same model at the same provider the later wins. It is safe to call while requests are being priced, so a host can reload its prices at runtime.
An entry whose LongContext tier lacks an input price or an output price fails the whole call with ErrUnpricedLongContext, naming the model and the provider, and the overrides already set stay as they were. A Free price, or a Free tier, may leave either price at zero. OverridesErr reports the latest failure until a call succeeds.
type PriceOverride ¶
PriceOverride sets the price of Model at Provider, the registered name of the provider that serves it. It wins over the provider's own list price and prices a model the provider does not list, such as one served through an OpenAI-compatible endpoint.
type RequestTokenCounter ¶
type RequestTokenCounter interface {
TokenCounter
// EstimateRequest returns a full estimate for a completion request,
// including context window limits and overflow strategy.
EstimateRequest(ctx context.Context, messages []Message, model string, maxTokens int) (*TokenEstimate, error)
}
RequestTokenCounter extends TokenCounter with the ability to estimate tokens for a full completion request (used by the pipeline middleware).
type Service ¶
type Service interface {
// ListModels returns all available models across providers.
ListModels(ctx context.Context) ([]provider.Model, error)
// Get returns model metadata by ID.
Get(ctx context.Context, modelID string) (*provider.Model, error)
// ResolveAlias resolves a model alias to a concrete model target.
// Returns the original modelID if no alias exists.
ResolveAlias(ctx context.Context, alias, tenantID string) (string, string, error)
}
Service combines alias resolution and model catalog.
func NewService ¶
func NewService(aliases AliasRegistry, providers provider.Registry) Service
NewService creates a model service with the given alias registry and provider registry.
type TokenCounter ¶
type TokenCounter interface {
// Count returns the estimated token count for a text.
Count(ctx context.Context, text string, model string) (int, error)
// CountMessages returns the estimated token count for a message list.
CountMessages(ctx context.Context, messages []Message, model string) (int, error)
}
TokenCounter estimates token counts for requests.
type TokenEstimate ¶
type TokenEstimate struct {
InputTokens int `json:"input_tokens"`
OutputMax int `json:"output_max"`
Total int `json:"total"`
ContextWindow int `json:"context_window"`
Overflow bool `json:"overflow"` // true if InputTokens > ContextWindow
OverflowStrategy OverflowStrategy `json:"overflow_strategy,omitempty"`
}
TokenEstimate is the result of a token estimation.
Source Files