virtualmodels

package
v0.1.94 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 18, 2026 License: MIT Imports: 27 Imported by: 0

Documentation

Overview

Package virtualmodels unifies model aliases (redirects) and model access overrides (policies) behind one entity, the virtual model, persisted in a single virtual_models table.

A row with Targets is a REDIRECT: Source is a new addressable name that rewrites to one or more real models. A redirect with a single target is a plain alias; a redirect with several targets is load balanced: Strategy picks the target a request is sent to first, and the remaining available targets are its failover chain when that attempt fails. A target may name another virtual model (a chain) or the redirect's own Source, which stands for the concrete model the redirect shadows. A row without Targets is an ACCESS POLICY: Source is a scoped selector over existing models, gated by UserPaths.

The Service is a single native engine: it operates directly on VirtualModel rows behind one in-memory snapshot, serving both redirect resolution and policy authorization without composing other engines.

Index

Constants

View Source 
const (
	// StrategyRoundRobin rotates across targets, honoring per-target Weight. It is
	// the default when Strategy is empty.
	StrategyRoundRobin = "round_robin"
	// StrategyCost always routes to the cheapest currently-available target, ranked
	// by the model registry's per-token pricing.
	StrategyCost = "cost"
	// StrategyAdaptive delegates target choice to a route selector registered
	// through the ext package (a pro extension). Without a registered selector
	// it behaves exactly like round_robin, so configs stay portable between
	// core and extended builds.
	StrategyAdaptive = "adaptive"
	// StrategyFailover always routes to the first currently-available target
	// in declared order: the target list is a priority list, and lower legs
	// serve only while every leg above them is unavailable or fails.
	StrategyFailover = "failover"
	// StrategyPlugin delegates target choice to the routing-strategy plugin
	// named by VirtualModel.StrategyPlugin, configured per virtual model
	// through StrategyConfig. Target weights are ignored: the plugin decides.
	// When the plugin is missing, misconfigured, declines, panics, or times
	// out, the redirect falls back to weighted round robin.
	StrategyPlugin = "plugin"
)

Load-balancing strategies for multi-target redirects.

View Source 
const (
	KindRedirect = "redirect"
	KindPolicy   = "policy"
)

Role kinds for the admin view.

View Source 
const (
	MinSlowdownFactor = 0.1
	MaxSlowdownFactor = 10.0
)
View Source 
const MaxChainDepth = 8

MaxChainDepth caps how many virtual models a redirect may pass through before reaching a concrete model. A redirect whose targets are all concrete has depth 1; each virtual model hop adds one.

Variables

View Source 
var ErrNotFound = errors.New("virtual model not found")

ErrNotFound indicates a requested virtual model was not found.

Functions

func IsValidationError 

func IsValidationError(err error) bool

IsValidationError reports whether err is a validation error.

Types

type AccessPolicy added in v0.1.84 

type AccessPolicy interface {
	AllowsModel(ctx context.Context, selector core.ModelSelector) bool
}

AccessPolicy narrows model access per request from the subject side.

type BatchPreparer 

type BatchPreparer struct {
	// contains filtered or unexported fields
}

BatchPreparer rewrites redirect (alias) sources for native batch subrequests and validates model access before provider submission. It is the combined replacement for the two old preparers (alias rewrite then access validation).

func NewBatchPreparer 

func NewBatchPreparer(provider core.RoutableProvider, service *Service) *BatchPreparer

NewBatchPreparer creates the combined redirect-rewrite + access-validation batch preparer.

func (*BatchPreparer) PrepareBatchRequest 

func (p *BatchPreparer) PrepareBatchRequest(ctx context.Context, providerType string, req *core.BatchRequest) (*core.BatchRewriteResult, error)

PrepareBatchRequest rewrites redirect sources for inline and file-backed batch items and validates model access for each resolved selector.

type Catalog 

type Catalog interface {
	Supports(model string) bool
	// ModelAvailable is Supports narrowed to providers whose inventory is
	// fresh: target selection uses it so load balancing routes around a
	// provider whose latest model refresh failed.
	ModelAvailable(model string) bool
	GetProviderType(model string) string
	LookupModel(model string) (*core.Model, bool)
	ProviderNames() []string
}

Catalog is the combined catalog surface the native engine needs.

type ChatExecutor 

type ChatExecutor struct {
	// contains filtered or unexported fields
}

ChatExecutor applies virtual-model redirects to transport-free chat completions before delegating to the wrapped provider. It is the narrow replacement for the former redirect-aware Provider decorator, whose only production role was serving as the guardrail auxiliary-LLM executor (guardrails.ChatCompletionExecutor is a single-method interface).

func NewChatExecutor 

func NewChatExecutor(inner ChatExecutorProvider, service *Service) *ChatExecutor

NewChatExecutor creates a redirect-aware chat executor over inner.

func (*ChatExecutor) ChatCompletion 

func (e *ChatExecutor) ChatCompletion(ctx context.Context, req *core.ChatRequest) (*core.ChatResponse, error)

ChatCompletion resolves the request's redirect (user-path aware) and delegates the rewritten request to the wrapped provider.

type ChatExecutorProvider 

type ChatExecutorProvider interface {
	ChatCompletion(ctx context.Context, req *core.ChatRequest) (*core.ChatResponse, error)
	// contains filtered or unexported methods
}

ChatExecutorProvider is the slice of the router the executor needs: model support checks for redirect validation and chat dispatch.

type EffectiveState 

type EffectiveState struct {
	Selector       string   `json:"selector"`
	ProviderName   string   `json:"provider_name,omitempty"`
	Model          string   `json:"model,omitempty"`
	DefaultEnabled bool     `json:"default_enabled"`
	Enabled        bool     `json:"enabled"`
	UserPaths      []string `json:"user_paths,omitempty"`
}

EffectiveState is the compiled access decision for one concrete selector.

type MongoDBStore 

type MongoDBStore struct {
	// contains filtered or unexported fields
}

MongoDBStore stores virtual models in MongoDB.

func NewMongoDBStore 

func NewMongoDBStore(database *mongo.Database) (*MongoDBStore, error)

NewMongoDBStore creates collection indexes if needed.

func (*MongoDBStore) Close 

func (s *MongoDBStore) Close() error

func (*MongoDBStore) Delete 

func (s *MongoDBStore) Delete(ctx context.Context, source string) error

func (*MongoDBStore) Get 

func (s *MongoDBStore) Get(ctx context.Context, source string) (*VirtualModel, error)

func (*MongoDBStore) List 

func (s *MongoDBStore) List(ctx context.Context) ([]VirtualModel, error)

func (*MongoDBStore) Upsert 

func (s *MongoDBStore) Upsert(ctx context.Context, vm VirtualModel) error

type Option added in v0.1.90 

type Option func(*Service)

Option customizes the service New builds.

func WithRouteResolver added in v0.1.90 

func WithRouteResolver(resolver RouteResolver) Option

WithRouteResolver installs the routing-strategy plugin resolver before the declarative virtual models are validated, so a managed redirect with an invalid strategy_config fails startup loudly.

type Resolution 

type Resolution struct {
	Requested core.ModelSelector
	Resolved  core.ModelSelector
	Source    string
}

Resolution captures the requested selector and the concrete selector chosen after redirect resolution. Source is the redirect name that matched, if any.

type Result 

type Result struct {
	Service *Service
	Store   Store
	// contains filtered or unexported fields
}

Result holds the initialized virtual models service and any owned resources.

func New 

func New(ctx context.Context, cfg *config.Config, shared storage.Storage, catalog Catalog, declaredProviders []string, opts ...Option) (*Result, error)

New creates a virtual models subsystem using an existing storage connection.

func (*Result) Close 

func (r *Result) Close() error

Close releases resources held by the virtual models subsystem.

type RouteResolver added in v0.1.90 

type RouteResolver interface {
	// Strategy returns the initialized strategy registered under name, or an
	// error when no such route plugin is loaded or its instance failed to
	// initialize.
	Strategy(name string) (pluginapi.RouteStrategy, *plugins.Instance, error)
	// Release hands back the instance Strategy returned once the call that
	// used it is over, so a replaced instance can be closed.
	Release(inst *plugins.Instance)
	// ValidateRouteConfig checks a virtual model's strategy_config against
	// the plugin's route-scoped fields and returns the canonical JSON with
	// defaults applied.
	ValidateRouteConfig(name string, cfg map[string]any) (json.RawMessage, error)
}

RouteResolver serves routing-strategy plugins to redirects using the plugin strategy. plugins.RouteResolver implements it.

type SQLStore added in v0.1.60 

type SQLStore struct {
	// contains filtered or unexported fields
}

SQLStore stores virtual models in a SQL database.

func NewSQLStore added in v0.1.60 

func NewSQLStore(ctx context.Context, db sqlx.DB) (*SQLStore, error)

NewSQLStore creates the virtual_models table and indexes if needed.

func (*SQLStore) Close added in v0.1.60 

func (s *SQLStore) Close() error

func (*SQLStore) Delete added in v0.1.60 

func (s *SQLStore) Delete(ctx context.Context, source string) error

func (*SQLStore) Get added in v0.1.60 

func (s *SQLStore) Get(ctx context.Context, source string) (*VirtualModel, error)

func (*SQLStore) List added in v0.1.60 

func (s *SQLStore) List(ctx context.Context) ([]VirtualModel, error)

func (*SQLStore) Upsert added in v0.1.60 

func (s *SQLStore) Upsert(ctx context.Context, vm VirtualModel) error

type Service 

type Service struct {
	// contains filtered or unexported fields
}

Service is the single native engine over the virtual_models store. It serves both redirect resolution (alias behavior) and policy authorization (access override behavior) from one atomically swapped in-memory snapshot.

func NewService 

func NewService(store Store, catalog Catalog, defaultEnabled bool) (*Service, error)

NewService creates a virtual models service backed by the store and catalog. defaultEnabled is the process-wide model availability default consulted when no policy matches.

func (*Service) AllowsModel 

func (s *Service) AllowsModel(ctx context.Context, selector core.ModelSelector) bool

AllowsModel reports whether selector is available for the effective request user path.

func (*Service) Delete 

func (s *Service) Delete(ctx context.Context, source string) error

Delete removes one virtual model and refreshes the in-memory snapshot.

func (*Service) EffectiveState 

func (s *Service) EffectiveState(selector core.ModelSelector) EffectiveState

EffectiveState resolves the compiled access state for one concrete selector.

func (*Service) EnabledByDefault 

func (s *Service) EnabledByDefault() bool

EnabledByDefault reports the process-wide model availability default.

func (*Service) ExposedModels 

func (s *Service) ExposedModels() []core.Model

ExposedModels returns enabled redirects projected as model-list entries.

func (*Service) ExposedModelsFiltered 

func (s *Service) ExposedModelsFiltered(allow func(core.ModelSelector) bool) []core.Model

ExposedModelsFiltered returns enabled redirects projected as model-list entries, filtered by the concrete target selector.

func (*Service) ExposedModelsForUserPath 

func (s *Service) ExposedModelsForUserPath(userPath string, allow func(core.ModelSelector) bool) []core.Model

ExposedModelsForUserPath is ExposedModelsFiltered plus per-redirect user_path scoping: a redirect carrying user_paths is hidden from callers it would not apply to, so a scoped alias is not listed (its name exposed) to callers outside its scope even though resolution would fall through for them.

func (*Service) FilterPublicModels 

func (s *Service) FilterPublicModels(ctx context.Context, models []core.Model) []core.Model

FilterPublicModels removes models that are unavailable for the effective request user path.

func (*Service) Get 

func (s *Service) Get(source string) (*VirtualModel, bool)

Get returns one cached virtual model by source.

func (*Service) List 

func (s *Service) List() []VirtualModel

List returns all cached virtual models sorted by source.

func (*Service) ListViews 

func (s *Service) ListViews() []View

ListViews returns all virtual models (redirects and policies) for the admin UI.

func (*Service) Refresh 

func (s *Service) Refresh(ctx context.Context) error

Refresh reloads virtual models from storage and atomically swaps the snapshot.

func (*Service) Rename 

func (s *Service) Rename(ctx context.Context, oldSource string, vm VirtualModel) error

Rename moves an existing virtual model to a new source: it stores the row under the new source and removes the old one, validating and refreshing like Upsert with rollback on failure. A no-op rename (old == new after normalization) delegates to Upsert. The new source must be free — renaming onto an existing row is rejected rather than silently overwriting it, since source is the primary key on every store backend.

func (*Service) ResolveFailovers added in v0.1.84 

func (s *Service) ResolveFailovers(resolution *core.RequestModelResolution, _ core.Operation) []core.ModelSelector

ResolveFailovers returns the failover chain for a request that resolved through a redirect: the concrete models behind the redirect's remaining available targets, in declared order and descending chained virtual models, minus the model the request was sent to first. The redirect's strategy only chooses that first target; every other available target is a failover leg, so a load balancer and a priority list fail over the same way. A redirect with failover switched off, and requests that did not go through a redirect, have no chain; a redirect whose single target is a chained redirect keeps that subtree's leaves as its chain — except a request that named its provider explicitly for a model shadowed by a redirect listing that model among its targets (see snapshot.failoverEntry).

func (*Service) ResolveModel 

func (s *Service) ResolveModel(requested core.RequestedModelSelector) (core.ModelSelector, bool, error)

ResolveModel resolves a requested selector and returns the concrete selector chosen for execution. It does not consult user_paths; scoped redirects are applied by ResolveModelForUserPath on the request path.

func (*Service) ResolveModelForUserPath 

func (s *Service) ResolveModelForUserPath(ctx context.Context, requested core.RequestedModelSelector) (core.ModelSelector, bool, error)

ResolveModelForUserPath resolves a requested selector honoring per-redirect user_paths against the effective request user path. A redirect scoped to user_paths the caller does not match falls through to the literal model name.

func (*Service) ResolveRefreshTarget 

func (s *Service) ResolveRefreshTarget(requested core.RequestedModelSelector) (core.ModelSelector, bool, error)

ResolveRefreshTarget returns a redirect target without consulting the current catalog so callers can refresh an unavailable target provider before normal resolution is retried.

func (*Service) ResolveSlowdown added in v0.1.74 

func (s *Service) ResolveSlowdown(
	ctx context.Context,
	requested core.RequestedModelSelector,
	resolved core.ModelSelector,
) float64

ResolveSlowdown returns the request-scoped extra-time factor for a resolved model. A matching alias setting takes precedence over its concrete target; otherwise the normal exact/provider/model/global policy precedence applies. The context makes the lookup ready for user-path-specific settings without changing the request execution interface later.

func (*Service) ResolveUpsertEnabled 

func (s *Service) ResolveUpsertEnabled(source, oldSource string, requested *bool) bool

ResolveUpsertEnabled returns the enabled flag an upsert should persist when the request may omit it: the requested value when present; otherwise the stored value for source (or, on a rename, for oldSource, since the new source does not exist yet); defaulting to true for new rows.

func (*Service) SetAccessPolicy added in v0.1.84 

func (s *Service) SetAccessPolicy(policy AccessPolicy)

SetAccessPolicy installs the subject-side access policy consulted by AllowsModel and ValidateModelAccess. Must be called before serving.

func (*Service) SetConfigModels 

func (s *Service) SetConfigModels(models []VirtualModel)

SetConfigModels installs the declarative (config.yaml / VIRTUAL_MODELS) virtual models that override store rows of the same source. Call it before the first Refresh, then ValidateManagedConfig to reject invalid declarations at startup.

func (*Service) SetRouteResolver added in v0.1.90 

func (s *Service) SetRouteResolver(resolver RouteResolver)

SetRouteResolver installs the routing-strategy plugin resolver consulted by redirects using the plugin strategy. Must be called before the service starts resolving requests.

func (*Service) SetRouteSelector added in v0.1.65 

func (s *Service) SetRouteSelector(selector ext.RouteSelector)

SetRouteSelector installs the extension route selector consulted by redirects using the adaptive strategy. Must be called before the service starts resolving requests.

func (*Service) SetTargetCapacity 

func (s *Service) SetTargetCapacity(capacity func(qualifiedModel string) bool)

SetTargetCapacity installs the rate-limit capacity probe consulted by load balancing. Must be called before the service starts resolving requests.

func (*Service) StartBackgroundRefresh 

func (s *Service) StartBackgroundRefresh(interval time.Duration) func()

StartBackgroundRefresh periodically reloads virtual models until stopped.

func (*Service) Upsert 

func (s *Service) Upsert(ctx context.Context, vm VirtualModel) error

Upsert validates and stores one virtual model, replacing any existing row at the same source even when its kind changes. A policy with no metadata whose access state is identical to what it would inherit is deleted instead of persisting a redundant row.

func (*Service) ValidateManagedConfig 

func (s *Service) ValidateManagedConfig(declaredProviders []string) error

ValidateManagedConfig checks that every declarative config redirect satisfies the catalog-independent redirect invariants (valid selector, not solely a self-target, no misspelled target provider), so a malformed IaC entry fails startup loudly. Chain cycles and depth are rejected earlier, by the initial Refresh. declaredProviders lists the names present in the providers configuration even when they did not register — e.g. their credentials are unset in this environment — so a config shared across environments still boots; such targets only warn and stay unavailable. Call it once after the initial Refresh.

It deliberately does NOT require targets to be catalog-supported: the provider model catalog loads asynchronously and may still be warming when this runs, and an unavailable target is skipped at resolve time like any other redirect target (the background ticker also skips this gate so a transient provider-catalog gap cannot freeze the snapshot). Gating startup on availability would abort an otherwise-valid declaration on a cold cache or a momentarily-unreachable provider — availability is runtime state, not a property of the declaration.

func (*Service) ValidateModelAccess 

func (s *Service) ValidateModelAccess(ctx context.Context, selector core.ModelSelector) error

ValidateModelAccess returns a typed request error when selector is not available.

type Store 

type Store interface {
	List(ctx context.Context) ([]VirtualModel, error)
	Get(ctx context.Context, source string) (*VirtualModel, error)
	Upsert(ctx context.Context, vm VirtualModel) error
	Delete(ctx context.Context, source string) error
	Close() error
}

Store defines persistence operations for virtual models.

type Target 

type Target struct {
	Provider string  `json:"provider,omitempty" bson:"provider,omitempty"`
	Model    string  `json:"model" bson:"model"`
	Weight   float64 `json:"weight,omitempty" bson:"weight,omitempty"`
}

Target is one concrete (provider, model) destination of a redirect.

Weight biases the round-robin strategy: a target with weight 2 receives twice the share of a target with weight 1. A non-positive or unset weight is treated as 1, so single-target and unweighted redirects behave identically to before. Weight is ignored by the cost strategy, which always picks the cheapest target.

type View 

type View struct {
	Source          string         `json:"source"`
	Kind            string         `json:"kind"`
	Targets         []Target       `json:"targets,omitempty"`
	Strategy        string         `json:"strategy,omitempty"`
	StrategyPlugin  string         `json:"strategy_plugin,omitempty"`
	StrategyConfig  map[string]any `json:"strategy_config,omitempty"`
	SessionAffinity *bool          `json:"session_affinity,omitempty"`
	Failover        *bool          `json:"failover,omitempty"`
	ProviderName    string         `json:"provider_name,omitempty"`
	Model           string         `json:"model,omitempty"`
	UserPaths       []string       `json:"user_paths,omitempty"`
	Description     string         `json:"description,omitempty"`
	// Slowdown is an extra-time factor from 0.1 to 10; zero disables it.
	Slowdown      *float64  `json:"slowdown,omitempty"`
	Enabled       bool      `json:"enabled"`
	Managed       bool      `json:"managed,omitempty"`
	ResolvedModel string    `json:"resolved_model,omitempty"`
	ProviderType  string    `json:"provider_type,omitempty"`
	Valid         bool      `json:"valid,omitempty"`
	ScopeKind     string    `json:"scope_kind,omitempty"`
	CreatedAt     time.Time `json:"created_at"`
	UpdatedAt     time.Time `json:"updated_at"`
}

View is the admin-facing representation of one virtual model.

type VirtualModel 

type VirtualModel struct {
	Source   string   `json:"source" bson:"_id"`
	Targets  []Target `json:"targets,omitempty" bson:"targets,omitempty"`
	Strategy string   `json:"strategy,omitempty" bson:"strategy,omitempty"`
	// StrategyPlugin names the routing-strategy plugin consulted when Strategy
	// is "plugin"; StrategyConfig is that plugin's per-virtual-model
	// configuration, validated against its route-scoped fields. Both are
	// cleared for every other strategy.
	StrategyPlugin string         `json:"strategy_plugin,omitempty" bson:"strategy_plugin,omitempty"`
	StrategyConfig map[string]any `json:"strategy_config,omitempty" bson:"strategy_config,omitempty"`
	ProviderName   string         `json:"provider_name,omitempty" bson:"provider_name,omitempty"`
	Model          string         `json:"model,omitempty" bson:"model,omitempty"`
	UserPaths      []string       `json:"user_paths,omitempty" bson:"user_paths,omitempty"`
	Description    string         `json:"description,omitempty" bson:"description,omitempty"`
	// Slowdown is an extra-time factor from 0.1 to 10; zero disables it. Nil
	// leaves the setting unspecified so an alias can inherit its target model.
	Slowdown *float64 `json:"slowdown,omitempty" bson:"slowdown,omitempty"`
	Enabled  bool     `json:"enabled" bson:"enabled"`

	// SessionAffinity keeps requests of one detected session on the target that
	// served it before, while that target stays available. Tri-state: nil means
	// enabled (the default); explicit false restores stateless balancing.
	SessionAffinity *bool `json:"session_affinity,omitempty" bson:"session_affinity,omitempty"`

	// Failover retries a failed request against the redirect's remaining
	// targets. Tri-state: nil means enabled (the default); explicit false
	// serves the chosen target only. The failover strategy always fails over.
	Failover *bool `json:"failover,omitempty" bson:"failover,omitempty"`

	CreatedAt time.Time `json:"created_at" bson:"created_at"`
	UpdatedAt time.Time `json:"updated_at" bson:"updated_at"`

	// Managed marks a virtual model supplied declaratively through config.yaml or
	// the VIRTUAL_MODELS env var rather than the admin store. It is an in-memory
	// flag only: stores never read or write it. Managed rows override store rows
	// of the same Source and are read-only to the admin API.
	Managed bool `json:"managed,omitempty" bson:"-"`
}

VirtualModel is one operator-defined model entry.

func ConfigModels 

func ConfigModels(entries []config.VirtualModelConfig) []VirtualModel

ConfigModels converts declarative config.yaml / VIRTUAL_MODELS entries into virtual model rows marked as managed. The rows are fully validated when the service builds its snapshot, so an invalid declaration fails startup loudly.

func FailoverConfigModels added in v0.1.84 

func FailoverConfigModels(cfg config.FailoverConfig, declared, stored []VirtualModel) []VirtualModel

FailoverConfigModels translates the deprecated `failover` rules block (failover.rules, manual_rules_path, FAILOVER_RULES_JSON) into managed failover-strategy virtual models, so a configuration written for the standalone failover feature keeps routing the same way. Each rule becomes a redirect that shadows its primary model: the primary is the first target and the fallbacks follow in order. A primary listed in disabled_models is skipped. Sources already declared under virtual_models are left to that declaration, and sources that exist in the store are left to the stored virtual model, with a warning that names it, so the rule's fallbacks can be moved into it by hand.

func (VirtualModel) IsRedirect 

func (v VirtualModel) IsRedirect() bool

IsRedirect reports whether this row redirects (has at least one target).

func (VirtualModel) Kind 

func (v VirtualModel) Kind() string

Kind returns the derived role: "redirect" or "policy".

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.