Documentation
¶
Overview ¶
Package llmcost holds the model-pricing primitives shared by the LLM budget and cost cap plugins: glob matching of model slugs, price resolution, model downgrade, and the stateless cost cap decision engine.
Index ¶
- Constants
- func ApplyDowngrade(req *infracontext.RequestContext, orig, target string) (string, []byte, map[string][]string, bool)
- func BestMatch[T any](m map[string]T, s string) (T, bool)
- func CostCapError(dec Decision) *appplugins.PluginError
- func GlobMatch(pattern, s string) bool
- func MicroUSD(costUSD float64) int64
- func Per1k(perToken float64) float64
- func PriceFor(ctx context.Context, resolver appcatalog.PricingResolver, ...) (float64, float64, bool)
- func ResolveDowngrade(provider, target string, allowed []string) (string, bool)
- type CapConfig
- type Ceiling
- type CustomPrice
- type Decision
- type DecisionKind
- type Telemetry
Constants ¶
const ( BehaviorReject = "reject" BehaviorDowngrade = "downgrade" )
Behaviors taken when a model exceeds its cost ceiling.
const ( UnknownReject = "reject" UnknownPassThrough = "pass_through" UnknownAssumeMax = "assume_max" )
Policies for models whose price cannot be resolved.
const CostCapErrorType = "model_too_expensive"
CostCapErrorType is the error type emitted when a model exceeds its cost cap.
const DowngradeHeader = "X-NeuralTrust-Model-Downgraded"
DowngradeHeader flags a response whose model was downgraded, carrying the "original→target" transition as its value.
Variables ¶
This section is empty.
Functions ¶
func ApplyDowngrade ¶
func ApplyDowngrade(req *infracontext.RequestContext, orig, target string) (string, []byte, map[string][]string, bool)
ApplyDowngrade rewrites the request body to target when the downgrade is valid, returning the new model, the rewritten body, and the downgrade header.
func BestMatch ¶
BestMatch returns the value whose key best matches s. An exact key wins over any glob; among globs the most specific pattern (most literal characters) wins, ties broken lexicographically.
func CostCapError ¶
func CostCapError(dec Decision) *appplugins.PluginError
CostCapError builds the 403 plugin error for a cost cap violation.
func GlobMatch ¶
GlobMatch reports whether s matches pattern, where '*' matches any run of characters (including the empty string).
func PriceFor ¶
func PriceFor(ctx context.Context, resolver appcatalog.PricingResolver, custom map[string]CustomPrice, provider string, models ...string) (float64, float64, bool)
PriceFor resolves per-token input and output USD prices for the first matching model. A custom overlay is consulted before the builtin resolver.
Types ¶
type CapConfig ¶
type CapConfig struct {
Enabled bool `mapstructure:"enabled"`
MaxInputCostPer1k float64 `mapstructure:"max_input_cost_per_1k_tokens"`
MaxOutputCostPer1k float64 `mapstructure:"max_output_cost_per_1k_tokens"`
PerModelOverrides map[string]Ceiling `mapstructure:"per_model_overrides"`
BehaviorOnViolation string `mapstructure:"behavior_on_violation"`
DowngradeTo string `mapstructure:"downgrade_to"`
UnknownModel string `mapstructure:"unknown_model"`
}
CapConfig is the stateless per-request cost cap configuration.
type Ceiling ¶
type Ceiling struct {
MaxInputCostPer1k float64 `mapstructure:"max_input_cost_per_1k_tokens"`
MaxOutputCostPer1k float64 `mapstructure:"max_output_cost_per_1k_tokens"`
}
Ceiling is a per-1k-token USD price ceiling.
func EvaluateCeiling ¶
EvaluateCeiling resolves the effective ceiling for a model, preferring the most specific per-model override before the global ceiling.
type CustomPrice ¶
type CustomPrice struct {
Input float64 `mapstructure:"input"`
Output float64 `mapstructure:"output"`
}
CustomPrice is a per-token USD overlay rate for a model slug or pattern.
type Decision ¶
type Decision struct {
Kind DecisionKind
Unknown bool
Model string
InputPrice float64
OutputPrice float64
MaxInput float64
MaxOutput float64
}
Decision is the outcome of evaluating a model against its cost ceiling.
func Decide ¶
func Decide(ctx context.Context, resolver appcatalog.PricingResolver, custom map[string]CustomPrice, cc *CapConfig, provider string, models ...string) Decision
Decide evaluates a model against its cost cap, resolving its price first.
type DecisionKind ¶
type DecisionKind int
DecisionKind classifies the outcome of a cost cap evaluation.
const ( DecisionAllow DecisionKind = iota DecisionViolation )
Source Files