Documentation
¶
Overview ¶
Package routing plans deterministic provider attempts from immutable inputs. It owns no connectors, network operations, clocks, randomness, or mutable health state.
Index ¶
- Constants
- Variables
- type AccountPolicy
- type Attempt
- type Candidate
- type Endpoint
- type Modality
- type Operation
- type OperationSet
- type OptimizationPolicy
- type Plan
- type Planner
- type ProviderAccess
- type ProviderPolicy
- type Rejection
- type RejectionCode
- type Request
- type Route
- type SelectionEvidence
- type Snapshot
- type TokenCost
Constants ¶
const DefaultSpreadRatio = 1.25
DefaultSpreadRatio bounds the spread band when the request does not name a ratio: a candidate joins the band while its ranking metric stays within 25% of the best candidate's metric.
Variables ¶
var ( // ErrNoCandidate reports that policy rejected every requested route. ErrNoCandidate = errors.New("no route candidate satisfies the request") // ErrInvalidRequest reports invalid route-planning input. ErrInvalidRequest = errors.New("invalid route-planning request") // ErrInvalidSnapshot reports invalid or generation-inconsistent candidates. ErrInvalidSnapshot = errors.New("invalid route-planning snapshot") // ErrInvalidPlan reports incomplete or duplicate attempt identities. ErrInvalidPlan = errors.New("invalid route plan") // ErrModalityUnsupported reports that the request carries a modality no // route accepts. It wraps beside ErrNoCandidate so a caller that answers // every routing failure with a 503 can still separate this one, which is // a caller mistake and answers 400. ErrModalityUnsupported = errors.New("model does not accept a requested modality") // ErrOperationUnsupported reports that every offering the request reached // serves other operations. A chat model asked to rerank is the case it // exists for: without it the planner reports "no candidate", the caller // hears a gateway problem, and the real cause stays inside the plan. It // wraps beside ErrNoCandidate so a caller answering every routing failure // with a 503 can still separate this one, which no retry changes. ErrOperationUnsupported = errors.New("model does not serve the requested operation") )
Functions ¶
This section is empty.
Types ¶
type AccountPolicy ¶ added in v1.1.0
type AccountPolicy struct {
AllowedModels []string
AllowedProviders []string
ModelOverrides map[string]string
// Access names the providers this account may reach, each entry
// optionally narrowed to specific models. A nil or empty list grants
// every provider and every model, so a policy-free account plans
// exactly as before the field existed.
Access []ProviderAccess
}
AccountPolicy defines the caller's hard model and provider boundaries.
type Attempt ¶
type Attempt struct {
Route Route
Evidence SelectionEvidence
}
Attempt is one ordered provider attempt in a route plan.
type Candidate ¶
type Candidate struct {
Route Route
Operations []Operation
Endpoints map[Operation]Endpoint
PromptCache *bool
Capabilities []string
// InputModalities lists what the model reads. An empty list means the
// catalog states nothing, which the planner reads as silence rather than
// as a refusal.
InputModalities []Modality
ContextWindow int
// MaxDocuments is the longest document list this offering accepts. The
// planner reads it nowhere: it carries the fact to the chosen route, which
// is where the operation that has documents can enforce it.
MaxDocuments int
Cost *TokenCost
Latency *time.Duration
Unhealthy bool
}
Candidate contains facts and runtime measurements for one route. The planner does not change this value.
func (Candidate) ServesOperation ¶ added in v1.1.0
ServesOperation reports whether the candidate declares the operation. An empty operation means the caller named none, which every candidate serves.
type Modality ¶ added in v1.1.0
type Modality string
Modality names one payload family a route accepts. Route planning keeps its own vocabulary instead of importing the canonical message types, because a plan stays a pure function of the values handed to it.
const ( // ModalityText is written or spoken language as characters. ModalityText Modality = "text" // ModalityImage is a still picture. ModalityImage Modality = "image" // ModalityAudio is recorded sound. ModalityAudio Modality = "audio" // ModalityDocument is a paged document, such as a PDF. ModalityDocument Modality = "document" // ModalityVideo is moving pictures. ModalityVideo Modality = "video" )
type Operation ¶
type Operation string
Operation is one provider inference operation selected from catalog facts. The values match Starmap's provider operation names exactly, because the catalog is the only source that names an operation.
const ( // OperationChatCompletions generates chat completions. OperationChatCompletions Operation = "chat-completions" // OperationEmbeddings generates vector embeddings. OperationEmbeddings Operation = "embeddings" // OperationImagesGenerations generates an image from a prompt. OperationImagesGenerations Operation = "images-generations" // OperationImagesEdits generates an image from a prompt and a source image. OperationImagesEdits Operation = "images-edits" // OperationAudioSpeech generates speech from text. OperationAudioSpeech Operation = "audio-speech" // OperationAudioTranscriptions writes recorded speech as text in its own // language. OperationAudioTranscriptions Operation = "audio-transcriptions" // OperationAudioTranslations writes recorded speech as English text. OperationAudioTranslations Operation = "audio-translations" // OperationVideosGenerations generates a video from a prompt. A provider // answers it with a job rather than a video, so a caller submits, polls, // and collects. OperationVideosGenerations Operation = "videos-generations" // OperationDocumentsRecognition reads the text off a document page that // carries none. A document with a text layer needs no provider at all, so // this operation names the case the in-process reader cannot answer. OperationDocumentsRecognition Operation = "documents-recognition" // OperationRerank scores a list of documents against one query and returns // them in relevance order. It generates nothing, so a model that serves it // serves no other operation. OperationRerank Operation = "rerank" // OperationModerations classifies text against a fixed set of harm // categories and answers with a score for each one. Like rerank, it // generates nothing, so a model that serves it serves no other operation. OperationModerations Operation = "moderations" )
type OperationSet ¶ added in v1.1.0
type OperationSet struct {
// contains filtered or unexported fields
}
OperationSet is an immutable set of operation names. One set answers the three separate questions the gateway asks about an operation: whether a caller may request it, whether a catalog fact that names it can reach a route, and whether a compiled transport may declare it. Three answers from one set is what keeps a widened catalog from disagreeing with a narrower build.
func NewOperationSet ¶ added in v1.1.0
func NewOperationSet(operations ...Operation) OperationSet
NewOperationSet builds a set from the named operations. It ignores the empty name, which means "the caller stated no operation" rather than an operation.
func ServedOperations ¶ added in v1.1.0
func ServedOperations() OperationSet
ServedOperations returns the operations this build can plan.
func (OperationSet) Contains ¶ added in v1.1.0
func (s OperationSet) Contains(operation Operation) bool
Contains reports whether the set names the operation.
func (OperationSet) Len ¶ added in v1.1.0
func (s OperationSet) Len() int
Len returns how many operations the set names.
func (OperationSet) Members ¶ added in v1.1.0
func (s OperationSet) Members() []Operation
Members returns the names in sorted order, so an error message built from a set reads the same on every run.
type OptimizationPolicy ¶
type OptimizationPolicy struct {
PreferLowestCost bool
PreferLowestLatency bool
// Spread reorders the leading candidates whose ranking metric sits
// within SpreadRatio of the best, weighted toward the better metric.
// Candidates outside the band keep the deterministic order.
Spread bool
// SpreadSeed seeds the weighted selection. The same request with the
// same seed produces the same plan.
SpreadSeed uint64
// SpreadRatio bounds the band as a multiple of the best metric. A value
// below one falls back to DefaultSpreadRatio.
SpreadRatio float64
}
OptimizationPolicy defines soft ordering preferences. Every field keeps the plan a pure function of the request: the spread fields carry the caller's seed instead of drawing randomness inside the planner.
type Plan ¶
type Plan struct {
// contains filtered or unexported fields
}
Plan is an immutable ordered attempt list with rejection evidence.
func NewPlan ¶
func NewPlan( catalogGenerationID string, availabilityRevision uint64, attempts []Attempt, rejections []Rejection, ) (*Plan, error)
NewPlan creates an immutable plan from an already ordered attempt set. Composition adapters use it when no catalog-backed planner is available.
func (*Plan) AvailabilityRevision ¶
AvailabilityRevision returns the runtime revision used for this plan.
func (*Plan) CatalogGenerationID ¶
CatalogGenerationID returns the generation that supplied every planned route.
func (*Plan) Rejections ¶
Rejections returns a caller-owned copy in stable route order.
type Planner ¶
type Planner struct{}
Planner converts one request and snapshot into a route plan. The plan is a pure function of its inputs: even spread ordering draws from the seed the request carries, so the same request plans the same way twice.
type ProviderAccess ¶ added in v1.1.0
ProviderAccess grants the account one provider, and optionally narrows which of its models. An empty Models list grants every model the provider serves. The pairing matters: a flat model set cannot say "every model on provider A, only one model on provider B" without denying A's other models.
type ProviderPolicy ¶
type ProviderPolicy struct {
Order []string
Only []string
Ignore []string
AllowFallbacks bool
// MaxPromptPricePerToken and MaxCompletionPricePerToken cap the accepted
// per-token price. Zero means no cap. A capped request rejects routes
// whose price is unknown: a cap is a promise the planner can only keep
// with known prices.
MaxPromptPricePerToken float64
MaxCompletionPricePerToken float64
}
ProviderPolicy defines request-scoped provider constraints and order.
type Rejection ¶
type Rejection struct {
Route Route
Code RejectionCode
Detail string
}
Rejection records why one considered route was not planned.
type RejectionCode ¶
type RejectionCode string
RejectionCode is a stable reason that excluded one route.
const ( RejectionUnavailable RejectionCode = "unavailable" // RejectionUnhealthy means runtime health disabled the offering. RejectionUnhealthy RejectionCode = "unhealthy" // RejectionAccountModel means account policy denied the model. RejectionAccountModel RejectionCode = "account_model" // RejectionAccountProvider means account policy denied the provider. RejectionAccountProvider RejectionCode = "account_provider" // RejectionProviderPolicy means request provider policy denied the route. RejectionProviderPolicy RejectionCode = "provider_policy" // RejectionMissingCapability means the route lacks a required capability. RejectionMissingCapability RejectionCode = "missing_capability" // RejectionMissingModality means the model does not read a modality the // request carries. RejectionMissingModality RejectionCode = "missing_modality" // RejectionMissingOperation means the exact offering or adapter cannot perform the request. RejectionMissingOperation RejectionCode = "missing_operation" // RejectionMissingEndpoint means the exact offering has no usable operation endpoint. RejectionMissingEndpoint RejectionCode = "missing_endpoint" // RejectionInsufficientContext means the route cannot accept the required context. RejectionInsufficientContext RejectionCode = "insufficient_context" // RejectionPriceExceeded means the route price violates a request price cap. RejectionPriceExceeded RejectionCode = "price_exceeded" // RejectionUnknownModel means a requested model matched no catalog offering. // The rejection carries only the model identity, not a full route. RejectionUnknownModel RejectionCode = "unknown_model" )
type Request ¶
type Request struct {
Models []string
Operation Operation
AllowModelFallbacks bool
AllowAnyModelFallback bool
RequiredCapabilities []string
RequiredModalities []Modality
RequiredContextTokens int
EstimatedInputTokens int
EstimatedOutputTokens int
Account AccountPolicy
Providers ProviderPolicy
AffinityProvider string
Optimization OptimizationPolicy
// ZeroPriceModels lists requested model IDs that only accept offerings
// with a known zero token price (the ":free" variant).
ZeroPriceModels []string
}
Request contains all policy and requirements used by the pure planner.
type Route ¶
type Route struct {
CatalogGenerationID string
ModelID string
ProviderID string
ProviderModelID string
Operation Operation
Endpoint Endpoint
PromptCacheKnown bool
PromptCache bool
// MaxDocuments is the longest document list this offering accepts, and
// zero means the catalog states no bound. It rides on the route rather
// than on a table beside it, because the bound belongs to the offering the
// planner chose and differs between two offerings of one model.
MaxDocuments int
}
Route identifies one provider offering in one immutable catalog generation.
type SelectionEvidence ¶
type SelectionEvidence struct {
ModelRank int
ProviderRank int
AffinityMatched bool
EstimatedCost float64
HasCost bool
EstimatedLatency time.Duration
HasLatency bool
}
SelectionEvidence records the pure ranks and measurements used to order an attempt.