README
¶
Multimodal Embeddings Cache Scorer Plugin
Type: mm-embeddings-cache-scorer
Scores candidate endpoints using multimodal embeddings cache match data produced
by mm-embeddings-cache-producer.
What It Does
For each candidate endpoint, the scorer reads EncoderCacheMatchInfo and computes:
score = matchedItemSize / totalRequestItemSize
For the unweighted producer path in this PR, every unique multimodal item has size
1, so the score is the fraction of unique request multimodal hashes that are
likely cached on the endpoint.
This produces a normalized score in the range [0, 1]:
- higher score: more request multimodal content is expected to reuse endpoint-local embeddings cache
- lower score: less multimodal cache reuse is expected
If the attribute is missing, has the wrong type, or total request item size is zero,
the endpoint receives score 0.
Inputs Consumed
This scorer consumes:
MultiModalEncoderCacheMatchInfoKey(EncoderCacheMatchInfo)
The attribute is produced by mm-embeddings-cache-producer before scheduling.
Configuration
This plugin does not define any plugin-specific parameters.
Configuration Example:
plugins:
- type: mm-embeddings-cache-producer
parameters:
cacheSizeInMBPerServer: 2048
- type: mm-embeddings-cache-scorer
- type: max-score-picker
schedulingProfiles:
- name: decode
plugins:
- pluginRef: mm-embeddings-cache-scorer
weight: 1
- pluginRef: max-score-picker
Operational Notes
- The scorer does not hash request media and does not maintain cache state.
- It only converts producer-generated match data into endpoint scores.
- KV-prefix cache affinity remains owned by
precise-prefix-cache-producer.
Documentation
¶
Overview ¶
Package mmcacheaffinity scores endpoints from multimodal encoder-cache match info produced by the request-control multimodal data producer.
Index ¶
Constants ¶
const (
// Type is the type name used to register the multimodal encoder-cache scorer.
Type = "mm-embeddings-cache-scorer"
)
Variables ¶
This section is empty.
Functions ¶
Types ¶
type Config ¶
type Config struct {
// ProducerName scopes the data key to a specific named producer instance.
// Leave empty to consume from the default (unnamed) producer.
ProducerName string `json:"producerName,omitempty"`
}
Config holds optional configuration for the scorer.
type Scorer ¶
type Scorer struct {
// contains filtered or unexported fields
}
Scorer computes normalized endpoint affinity from produced multimodal match data.
func New ¶
New creates a Scorer. producerName scopes the data key to a specific producer instance; pass an empty string to use the default producer's key.
func (*Scorer) Category ¶
func (s *Scorer) Category() scheduling.ScorerCategory
Category returns the scorer category.
func (*Scorer) Consumes ¶
func (s *Scorer) Consumes() plugin.DataDependencies
Consumes returns the endpoint data consumed by this scorer.
func (*Scorer) Score ¶
func (s *Scorer) Score(ctx context.Context, req *scheduling.InferenceRequest, endpoints []scheduling.Endpoint) map[scheduling.Endpoint]float64
Score scores endpoints by matched multimodal encoder-cache item size divided by total multimodal request item size.
Source Files