README
¶
KV Cache Utilization Scorer Plugin
Type: kv-cache-utilization-scorer
This plugin scores candidate endpoints using each endpoint's current KV-cache utilization.
What it does
For each candidate endpoint, the plugin computes:
{score(endpoint)} = 1 - {kvCacheUsagePercent}
Where kvCacheUsagePercent is read from endpoint metrics.
This means:
- lower KV-cache usage -> higher score
- higher KV-cache usage -> lower score
Scheduling intent
The scorer returns category Distribution, so it helps spread traffic away from endpoints with high KV-cache pressure.
Inputs consumed
The plugin consumes:
metrics.KVCacheUsagePercentKey(float64)
Configuration
This scorer currently has no runtime parameters.
Configuration Example:
plugins:
- type: kv-cache-utilization-scorer
name: kv-cache-util
schedulingProfiles:
- name: default
plugins:
- pluginRef: kv-cache-util
weight: 1
Documentation
¶
Index ¶
- Constants
- func KvCacheUtilizationScorerFactory(name string, _ *json.Decoder, _ fwkplugin.Handle) (fwkplugin.Plugin, error)
- type KVCacheUtilizationScorer
- func (s *KVCacheUtilizationScorer) Category() fwksched.ScorerCategory
- func (s *KVCacheUtilizationScorer) Consumes() map[string]any
- func (s *KVCacheUtilizationScorer) Score(_ context.Context, _ *fwksched.InferenceRequest, endpoints []fwksched.Endpoint) map[fwksched.Endpoint]float64
- func (s *KVCacheUtilizationScorer) TypedName() fwkplugin.TypedName
- func (s *KVCacheUtilizationScorer) WithName(name string) *KVCacheUtilizationScorer
Constants ¶
const (
KvCacheUtilizationScorerType = "kv-cache-utilization-scorer"
)
Variables ¶
This section is empty.
Functions ¶
Types ¶
type KVCacheUtilizationScorer ¶
type KVCacheUtilizationScorer struct {
// contains filtered or unexported fields
}
KVCacheUtilizationScorer scores list of candidate endpoints based on KV cache utilization.
func NewKVCacheUtilizationScorer ¶
func NewKVCacheUtilizationScorer() *KVCacheUtilizationScorer
NewKVCacheUtilizationScorer initializes a new KVCacheUtilizationScorer and returns its pointer.
func (*KVCacheUtilizationScorer) Category ¶
func (s *KVCacheUtilizationScorer) Category() fwksched.ScorerCategory
Category returns the preference the scorer applies when scoring candidate endpoints.
func (*KVCacheUtilizationScorer) Consumes ¶
func (s *KVCacheUtilizationScorer) Consumes() map[string]any
Consumes returns the list of data that is consumed by the plugin.
func (*KVCacheUtilizationScorer) Score ¶
func (s *KVCacheUtilizationScorer) Score(_ context.Context, _ *fwksched.InferenceRequest, endpoints []fwksched.Endpoint) map[fwksched.Endpoint]float64
Score returns the scoring result for the given list of endpoints based on context.
func (*KVCacheUtilizationScorer) TypedName ¶
func (s *KVCacheUtilizationScorer) TypedName() fwkplugin.TypedName
TypedName returns the type and name tuple of this plugin instance.
func (*KVCacheUtilizationScorer) WithName ¶
func (s *KVCacheUtilizationScorer) WithName(name string) *KVCacheUtilizationScorer
WithName sets the name of the scorer.
Source Files