routing

package
v1.2.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 11, 2026 License: AGPL-3.0 Imports: 6 Imported by: 0

Documentation

Overview

Package routing plans deterministic provider attempts from immutable inputs. It owns no connectors, network operations, clocks, randomness, or mutable health state.

Index

Constants

View Source
const DefaultSpreadRatio = 1.25

DefaultSpreadRatio bounds the spread band when the request does not name a ratio: a candidate joins the band while its ranking metric stays within 25% of the best candidate's metric.

Variables

View Source
var (
	// ErrNoCandidate reports that policy rejected every requested route.
	ErrNoCandidate = errors.New("no route candidate satisfies the request")
	// ErrInvalidRequest reports invalid route-planning input.
	ErrInvalidRequest = errors.New("invalid route-planning request")
	// ErrInvalidSnapshot reports invalid or generation-inconsistent candidates.
	ErrInvalidSnapshot = errors.New("invalid route-planning snapshot")
	// ErrInvalidPlan reports incomplete or duplicate attempt identities.
	ErrInvalidPlan = errors.New("invalid route plan")
	// ErrModalityUnsupported reports that the request carries a modality no
	// route accepts. It wraps beside ErrNoCandidate so a caller that answers
	// every routing failure with a 503 can still separate this one, which is
	// a caller mistake and answers 400.
	ErrModalityUnsupported = errors.New("model does not accept a requested modality")
	// ErrOperationUnsupported reports that every offering the request reached
	// serves other operations. A chat model asked to rerank is the case it
	// exists for: without it the planner reports "no candidate", the caller
	// hears a gateway problem, and the real cause stays inside the plan. It
	// wraps beside ErrNoCandidate so a caller answering every routing failure
	// with a 503 can still separate this one, which no retry changes.
	ErrOperationUnsupported = errors.New("model does not serve the requested operation")
)

Functions

This section is empty.

Types

type AccountPolicy added in v1.1.0

type AccountPolicy struct {
	AllowedModels    []string
	AllowedProviders []string
	ModelOverrides   map[string]string
	// Access names the providers this account may reach, each entry
	// optionally narrowed to specific models. A nil or empty list grants
	// every provider and every model, so a policy-free account plans
	// exactly as before the field existed.
	Access []ProviderAccess
}

AccountPolicy defines the caller's hard model and provider boundaries.

type Attempt

type Attempt struct {
	Route    Route
	Evidence SelectionEvidence
}

Attempt is one ordered provider attempt in a route plan.

type Candidate

type Candidate struct {
	Route        Route
	Operations   []Operation
	Endpoints    map[Operation]Endpoint
	PromptCache  *bool
	Capabilities []string

	// InputModalities lists what the model reads. An empty list means the
	// catalog states nothing, which the planner reads as silence rather than
	// as a refusal.
	InputModalities []Modality

	ContextWindow int

	// MaxDocuments is the longest document list this offering accepts. The
	// planner reads it nowhere: it carries the fact to the chosen route, which
	// is where the operation that has documents can enforce it.
	MaxDocuments int

	Cost        *TokenCost
	Latency     *time.Duration
	Unavailable bool
	Unhealthy   bool
}

Candidate contains facts and runtime measurements for one route. The planner does not change this value.

func (Candidate) ServesOperation added in v1.1.0

func (c Candidate) ServesOperation(operation Operation) bool

ServesOperation reports whether the candidate declares the operation. An empty operation means the caller named none, which every candidate serves.

type Endpoint

type Endpoint struct {
	Protocol  string
	URL       string
	StreamURL string
}

Endpoint is the exact offering endpoint and wire protocol selected for an attempt.

type Modality added in v1.1.0

type Modality string

Modality names one payload family a route accepts. Route planning keeps its own vocabulary instead of importing the canonical message types, because a plan stays a pure function of the values handed to it.

const (
	// ModalityText is written or spoken language as characters.
	ModalityText Modality = "text"
	// ModalityImage is a still picture.
	ModalityImage Modality = "image"
	// ModalityAudio is recorded sound.
	ModalityAudio Modality = "audio"
	// ModalityDocument is a paged document, such as a PDF.
	ModalityDocument Modality = "document"
	// ModalityVideo is moving pictures.
	ModalityVideo Modality = "video"
)

type Operation

type Operation string

Operation is one provider inference operation selected from catalog facts. The values match Starmap's provider operation names exactly, because the catalog is the only source that names an operation.

const (
	// OperationChatCompletions generates chat completions.
	OperationChatCompletions Operation = "chat-completions"
	// OperationEmbeddings generates vector embeddings.
	OperationEmbeddings Operation = "embeddings"
	// OperationImagesGenerations generates an image from a prompt.
	OperationImagesGenerations Operation = "images-generations"
	// OperationImagesEdits generates an image from a prompt and a source image.
	OperationImagesEdits Operation = "images-edits"
	// OperationAudioSpeech generates speech from text.
	OperationAudioSpeech Operation = "audio-speech"
	// OperationAudioTranscriptions writes recorded speech as text in its own
	// language.
	OperationAudioTranscriptions Operation = "audio-transcriptions"
	// OperationAudioTranslations writes recorded speech as English text.
	OperationAudioTranslations Operation = "audio-translations"
	// OperationVideosGenerations generates a video from a prompt. A provider
	// answers it with a job rather than a video, so a caller submits, polls,
	// and collects.
	OperationVideosGenerations Operation = "videos-generations"
	// OperationDocumentsRecognition reads the text off a document page that
	// carries none. A document with a text layer needs no provider at all, so
	// this operation names the case the in-process reader cannot answer.
	OperationDocumentsRecognition Operation = "documents-recognition"
	// OperationRerank scores a list of documents against one query and returns
	// them in relevance order. It generates nothing, so a model that serves it
	// serves no other operation.
	OperationRerank Operation = "rerank"
	// OperationModerations classifies text against a fixed set of harm
	// categories and answers with a score for each one. Like rerank, it
	// generates nothing, so a model that serves it serves no other operation.
	OperationModerations Operation = "moderations"
)

type OperationSet added in v1.1.0

type OperationSet struct {
	// contains filtered or unexported fields
}

OperationSet is an immutable set of operation names. One set answers the three separate questions the gateway asks about an operation: whether a caller may request it, whether a catalog fact that names it can reach a route, and whether a compiled transport may declare it. Three answers from one set is what keeps a widened catalog from disagreeing with a narrower build.

func NewOperationSet added in v1.1.0

func NewOperationSet(operations ...Operation) OperationSet

NewOperationSet builds a set from the named operations. It ignores the empty name, which means "the caller stated no operation" rather than an operation.

func ServedOperations added in v1.1.0

func ServedOperations() OperationSet

ServedOperations returns the operations this build can plan.

func (OperationSet) Contains added in v1.1.0

func (s OperationSet) Contains(operation Operation) bool

Contains reports whether the set names the operation.

func (OperationSet) Len added in v1.1.0

func (s OperationSet) Len() int

Len returns how many operations the set names.

func (OperationSet) Members added in v1.1.0

func (s OperationSet) Members() []Operation

Members returns the names in sorted order, so an error message built from a set reads the same on every run.

type OptimizationPolicy

type OptimizationPolicy struct {
	PreferLowestCost    bool
	PreferLowestLatency bool

	// Spread reorders the leading candidates whose ranking metric sits
	// within SpreadRatio of the best, weighted toward the better metric.
	// Candidates outside the band keep the deterministic order.
	Spread bool
	// SpreadSeed seeds the weighted selection. The same request with the
	// same seed produces the same plan.
	SpreadSeed uint64
	// SpreadRatio bounds the band as a multiple of the best metric. A value
	// below one falls back to DefaultSpreadRatio.
	SpreadRatio float64
}

OptimizationPolicy defines soft ordering preferences. Every field keeps the plan a pure function of the request: the spread fields carry the caller's seed instead of drawing randomness inside the planner.

type Plan

type Plan struct {
	// contains filtered or unexported fields
}

Plan is an immutable ordered attempt list with rejection evidence.

func NewPlan

func NewPlan(
	catalogGenerationID string,
	availabilityRevision uint64,
	attempts []Attempt,
	rejections []Rejection,
) (*Plan, error)

NewPlan creates an immutable plan from an already ordered attempt set. Composition adapters use it when no catalog-backed planner is available.

func (*Plan) Attempts

func (p *Plan) Attempts() []Attempt

Attempts returns a caller-owned copy in execution order.

func (*Plan) AvailabilityRevision

func (p *Plan) AvailabilityRevision() uint64

AvailabilityRevision returns the runtime revision used for this plan.

func (*Plan) CatalogGenerationID

func (p *Plan) CatalogGenerationID() string

CatalogGenerationID returns the generation that supplied every planned route.

func (*Plan) Rejections

func (p *Plan) Rejections() []Rejection

Rejections returns a caller-owned copy in stable route order.

type Planner

type Planner struct{}

Planner converts one request and snapshot into a route plan. The plan is a pure function of its inputs: even spread ordering draws from the seed the request carries, so the same request plans the same way twice.

func NewPlanner

func NewPlanner() Planner

NewPlanner creates a stateless route planner.

func (Planner) Plan

func (Planner) Plan(request Request, snapshot Snapshot) (*Plan, error)

Plan applies hard constraints, records rejections, and orders eligible routes.

type ProviderAccess added in v1.1.0

type ProviderAccess struct {
	Provider string
	Models   []string
}

ProviderAccess grants the account one provider, and optionally narrows which of its models. An empty Models list grants every model the provider serves. The pairing matters: a flat model set cannot say "every model on provider A, only one model on provider B" without denying A's other models.

type ProviderPolicy

type ProviderPolicy struct {
	Order          []string
	Only           []string
	Ignore         []string
	AllowFallbacks bool

	// MaxPromptPricePerToken and MaxCompletionPricePerToken cap the accepted
	// per-token price. Zero means no cap. A capped request rejects routes
	// whose price is unknown: a cap is a promise the planner can only keep
	// with known prices.
	MaxPromptPricePerToken     float64
	MaxCompletionPricePerToken float64
}

ProviderPolicy defines request-scoped provider constraints and order.

type Rejection

type Rejection struct {
	Route  Route
	Code   RejectionCode
	Detail string
}

Rejection records why one considered route was not planned.

type RejectionCode

type RejectionCode string

RejectionCode is a stable reason that excluded one route.

const (
	// RejectionUnavailable means runtime state disabled the offering.
	RejectionUnavailable RejectionCode = "unavailable"
	// RejectionUnhealthy means runtime health disabled the offering.
	RejectionUnhealthy RejectionCode = "unhealthy"
	// RejectionAccountModel means account policy denied the model.
	RejectionAccountModel RejectionCode = "account_model"
	// RejectionAccountProvider means account policy denied the provider.
	RejectionAccountProvider RejectionCode = "account_provider"
	// RejectionProviderPolicy means request provider policy denied the route.
	RejectionProviderPolicy RejectionCode = "provider_policy"
	// RejectionMissingCapability means the route lacks a required capability.
	RejectionMissingCapability RejectionCode = "missing_capability"
	// RejectionMissingModality means the model does not read a modality the
	// request carries.
	RejectionMissingModality RejectionCode = "missing_modality"
	// RejectionMissingOperation means the exact offering or adapter cannot perform the request.
	RejectionMissingOperation RejectionCode = "missing_operation"
	// RejectionMissingEndpoint means the exact offering has no usable operation endpoint.
	RejectionMissingEndpoint RejectionCode = "missing_endpoint"
	// RejectionInsufficientContext means the route cannot accept the required context.
	RejectionInsufficientContext RejectionCode = "insufficient_context"
	// RejectionPriceExceeded means the route price violates a request price cap.
	RejectionPriceExceeded RejectionCode = "price_exceeded"
	// RejectionUnknownModel means a requested model matched no catalog offering.
	// The rejection carries only the model identity, not a full route.
	RejectionUnknownModel RejectionCode = "unknown_model"
)

type Request

type Request struct {
	Models                []string
	Operation             Operation
	AllowModelFallbacks   bool
	AllowAnyModelFallback bool
	RequiredCapabilities  []string
	RequiredModalities    []Modality
	RequiredContextTokens int
	EstimatedInputTokens  int
	EstimatedOutputTokens int
	Account               AccountPolicy
	Providers             ProviderPolicy
	AffinityProvider      string
	Optimization          OptimizationPolicy

	// ZeroPriceModels lists requested model IDs that only accept offerings
	// with a known zero token price (the ":free" variant).
	ZeroPriceModels []string
}

Request contains all policy and requirements used by the pure planner.

type Route

type Route struct {
	CatalogGenerationID string
	ModelID             string
	ProviderID          string
	ProviderModelID     string
	Operation           Operation
	Endpoint            Endpoint
	PromptCacheKnown    bool
	PromptCache         bool

	// MaxDocuments is the longest document list this offering accepts, and
	// zero means the catalog states no bound. It rides on the route rather
	// than on a table beside it, because the bound belongs to the offering the
	// planner chose and differs between two offerings of one model.
	MaxDocuments int
}

Route identifies one provider offering in one immutable catalog generation.

func (Route) ID

func (r Route) ID() string

ID returns Starport's provider-scoped route ID.

type SelectionEvidence

type SelectionEvidence struct {
	ModelRank        int
	ProviderRank     int
	AffinityMatched  bool
	EstimatedCost    float64
	HasCost          bool
	EstimatedLatency time.Duration
	HasLatency       bool
}

SelectionEvidence records the pure ranks and measurements used to order an attempt.

type Snapshot

type Snapshot struct {
	CatalogGenerationID  string
	AvailabilityRevision uint64
	Candidates           []Candidate
}

Snapshot binds all planning candidates to one catalog generation and one runtime availability revision.

type TokenCost

type TokenCost struct {
	InputPerToken  float64
	OutputPerToken float64
}

TokenCost contains provider prices per input and output token.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL