Documentation
¶
Overview ¶
Package vtc implements the Virtual Token Counter routing algorithms focused on fairness and utilization
Index ¶
- Constants
- func NewVTCBasicRouter() (types.Router, error)
- type BasicVTCRouter
- type InMemorySlidingWindowTokenTracker
- func (t *InMemorySlidingWindowTokenTracker) GetMaxTokenCount(ctx context.Context) (float64, error)
- func (t *InMemorySlidingWindowTokenTracker) GetMinTokenCount(ctx context.Context) (float64, error)
- func (t *InMemorySlidingWindowTokenTracker) GetTokenCount(ctx context.Context, user string) (float64, error)
- func (t *InMemorySlidingWindowTokenTracker) UpdateTokenCount(ctx context.Context, user string, inputTokens, outputTokens float64) error
- type SimpleTokenEstimator
- type TimeUnit
- type TokenEstimator
- type TokenTracker
- type TokenTrackerOption
- type VTCConfig
Constants ¶
View Source
const ( VTC_TOKEN_TRACKER_WINDOW_SIZE = "AIBRIX_ROUTER_VTC_TOKEN_TRACKER_WINDOW_SIZE" VTC_TOKEN_TRACKER_TIME_UNIT = "AIBRIX_ROUTER_VTC_TOKEN_TRACKER_TIME_UNIT" VTC_TOKEN_TRACKER_MIN_TOKENS = "AIBRIX_ROUTER_VTC_TOKEN_TRACKER_MIN_TOKENS" VTC_TOKEN_TRACKER_MAX_TOKENS = "AIBRIX_ROUTER_VTC_TOKEN_TRACKER_MAX_TOKENS" )
View Source
const ( VTC_MAX_POD_LOAD = "AIBRIX_ROUTER_VTC_BASIC_MAX_POD_LOAD" VTC_INPUT_TOKEN_WEIGHT = "AIBRIX_ROUTER_VTC_BASIC_INPUT_TOKEN_WEIGHT" VTC_OUTPUT_TOKEN_WEIGHT = "AIBRIX_ROUTER_VTC_BASIC_OUTPUT_TOKEN_WEIGHT" VTC_FAIRNESS_WEIGHT = "AIBRIX_ROUTER_VTC_BASIC_FAIRNESS_WEIGHT" VTC_UTILIZATION_WEIGHT = "AIBRIX_ROUTER_VTC_BASIC_UTILIZATION_WEIGHT" )
View Source
const RouterVTCBasic types.RoutingAlgorithm = "vtc-basic"
Variables ¶
This section is empty.
Functions ¶
func NewVTCBasicRouter ¶
Types ¶
type BasicVTCRouter ¶
type BasicVTCRouter struct {
// contains filtered or unexported fields
}
BasicVTCRouter implements the VTC routing algorithm
func NewBasicVTCRouter ¶
func NewBasicVTCRouter(tokenTracker TokenTracker, tokenEstimator TokenEstimator, config *VTCConfig) (*BasicVTCRouter, error)
NewBasicVTCRouter creates a new BasicVTCRouter with the provided token tracker and estimator
func (*BasicVTCRouter) Route ¶
func (r *BasicVTCRouter) Route(ctx *types.RoutingContext, readyPodList types.PodList) (string, error)
Route implements the VTC routing algorithm
func (*BasicVTCRouter) SubscribedMetrics ¶
func (r *BasicVTCRouter) SubscribedMetrics() []string
type InMemorySlidingWindowTokenTracker ¶
type InMemorySlidingWindowTokenTracker struct {
// contains filtered or unexported fields
}
InMemorySlidingWindowTokenTracker tracks tokens per user in a fixed-size sliding window (in-memory, thread-safe).
func (*InMemorySlidingWindowTokenTracker) GetMaxTokenCount ¶
func (t *InMemorySlidingWindowTokenTracker) GetMaxTokenCount(ctx context.Context) (float64, error)
Time: O(1) | Space: O(1)
func (*InMemorySlidingWindowTokenTracker) GetMinTokenCount ¶
func (t *InMemorySlidingWindowTokenTracker) GetMinTokenCount(ctx context.Context) (float64, error)
Time: O(1) | Space: O(1)
func (*InMemorySlidingWindowTokenTracker) GetTokenCount ¶
func (t *InMemorySlidingWindowTokenTracker) GetTokenCount(ctx context.Context, user string) (float64, error)
Time: Avg O(1) (amortized), Worst O(B_u) where B_u = buckets for user u | Space: O(1)
func (*InMemorySlidingWindowTokenTracker) UpdateTokenCount ¶
func (t *InMemorySlidingWindowTokenTracker) UpdateTokenCount(ctx context.Context, user string, inputTokens, outputTokens float64) error
Time: Avg O(1) (amortized), Worst O(B_u) where B_u = buckets for user u | Space: O(1)
type SimpleTokenEstimator ¶
func (*SimpleTokenEstimator) EstimateInputTokens ¶
func (e *SimpleTokenEstimator) EstimateInputTokens(message string) float64
func (*SimpleTokenEstimator) EstimateOutputTokens ¶
func (e *SimpleTokenEstimator) EstimateOutputTokens(message string) float64
type TokenEstimator ¶
type TokenEstimator interface {
EstimateInputTokens(message string) float64
EstimateOutputTokens(message string) float64
}
TokenEstimator estimates token counts for messages
func NewSimpleTokenEstimator ¶
func NewSimpleTokenEstimator() TokenEstimator
TODO: advanced token estimators
type TokenTracker ¶
type TokenTracker interface {
GetTokenCount(ctx context.Context, user string) (float64, error)
UpdateTokenCount(ctx context.Context, user string, inputTokens, outputTokens float64) error
GetMinTokenCount(ctx context.Context) (float64, error)
GetMaxTokenCount(ctx context.Context) (float64, error)
}
TokenTracker tracks token usage per user
func NewInMemorySlidingWindowTokenTracker ¶
func NewInMemorySlidingWindowTokenTracker(config *VTCConfig, opts ...TokenTrackerOption) TokenTracker
TODO: add redis token tracker so that state is shared across plugin instances NewInMemorySlidingWindowTokenTracker creates a new token tracker with configurable options
type TokenTrackerOption ¶
type TokenTrackerOption func(*InMemorySlidingWindowTokenTracker)
TokenTrackerOption is a function that configures a token tracker
func WithTimeUnit ¶
func WithTimeUnit(unit TimeUnit) TokenTrackerOption
func WithWindowSize ¶
func WithWindowSize(size int) TokenTrackerOption
type VTCConfig ¶
type VTCConfig struct {
Variant types.RoutingAlgorithm
InputTokenWeight float64
OutputTokenWeight float64
}
func DefaultVTCConfig ¶
func DefaultVTCConfig() VTCConfig
Source Files
Click to show internal directories.
Click to hide internal directories.