README
¶
Multimodal Embeddings Cache Producer Plugin
Type: mm-embeddings-cache-producer
Produces multimodal embeddings cache match data for downstream scheduling plugins.
What It Does
For each request, the producer extracts stable multimodal item hashes from:
TokenizedPrompt.MultiModalFeatures, when atoken-produceris configured- typed OpenAI chat-completions structured media blocks, as a lightweight fallback
It keeps an in-memory LRU map from multimodal hash to the set of pods that recently
handled that item. During scheduling, it attaches EncoderCacheMatchInfo to each
endpoint so scorers can prefer pods that are likely to have already processed the
same image, video, or audio input.
Repeated references to the same multimodal hash within one request count once.
Inputs Consumed
This plugin declares:
TokenizedPrompt
When token-producer is present, this orders tokenization before multimodal match
data production. If tokenized prompt data is absent at runtime, the producer falls
back to typed structured chat-completions media blocks.
Data Produced
This plugin produces:
MultiModalEncoderCacheMatchInfoKey(EncoderCacheMatchInfo)
Configuration
The producer supports the following runtime parameters:
cacheSizeInMBPerServer(integer, default:2048, 2 GiB): per-endpoint memory budget in mebibytes (MiB) for the best-effort pod-affinity LRU.
Configuration Examples:
plugins:
- type: mm-embeddings-cache-producer
parameters:
cacheSizeInMBPerServer: 2048
- type: mm-embeddings-cache-scorer
schedulingProfiles:
- name: encoder-cache-aware
plugins:
- pluginRef: mm-embeddings-cache-scorer
weight: 4
- pluginRef: kv-cache-utilization-scorer
weight: 2
- pluginRef: queue-scorer
weight: 2
plugins:
- type: token-producer
parameters:
modelName: Qwen/Qwen2.5-1.5B-Instruct
vllm:
http: http://localhost:8000
- type: mm-embeddings-cache-producer
parameters:
cacheSizeInMBPerServer: 2048
- type: mm-embeddings-cache-scorer
schedulingProfiles:
- name: decode
plugins:
- pluginRef: mm-embeddings-cache-scorer
weight: 4
Operational Notes
- The cache is a best-effort routing signal, not a correctness dependency.
- The producer remains tokenizer-free for request shapes where typed media blocks are
sufficient;
token-produceris only required when relying on upstream multimodal metadata.
Documentation
¶
Overview ¶
Package multimodal provides a data producer for multimodal encoder-cache affinity. It extracts request media identifiers once, matches them against recent pod placements, and stores reusable match data on endpoints.
Index ¶
- Constants
- Variables
- func ExtractMMItems(request *scheduling.InferenceRequest) []attrmm.MatchItem
- func Factory(name string, rawParameters *json.Decoder, handle plugin.Handle) (plugin.Plugin, error)
- type Parameters
- type Producer
- func (p *Producer) Consumes() plugin.DataDependencies
- func (p *Producer) Extract(ctx context.Context, event fwkdl.EndpointEvent) error
- func (p *Producer) PluginState() *plugin.PluginState
- func (p *Producer) PreRequest(ctx context.Context, request *scheduling.InferenceRequest, ...)
- func (p *Producer) Produce(ctx context.Context, request *scheduling.InferenceRequest, ...) error
- func (p *Producer) Produces() map[plugin.DataKey]any
- func (p *Producer) TypedName() plugin.TypedName
Constants ¶
const (
// ProducerType is the type name used to register the multimodal data producer.
ProducerType = "mm-embeddings-cache-producer"
)
Variables ¶
var ( // ProducedKey is the data key emitted by this producer. ProducedKey = attrmm.EncoderCacheMatchInfoKey )
Functions ¶
func ExtractMMItems ¶
func ExtractMMItems(request *scheduling.InferenceRequest) []attrmm.MatchItem
ExtractMMItems returns deterministic, unique multimodal encoder-cache items derived from the tokenized prompt's multimodal features.
Types ¶
type Parameters ¶
type Parameters struct {
// CacheSizeInMBPerServer is the per-endpoint LRU memory budget in mebibytes (MiB).
CacheSizeInMBPerServer int `json:"cacheSizeInMBPerServer"`
}
Parameters configures the multimodal encoder-cache data producer.
type Producer ¶
type Producer struct {
// contains filtered or unexported fields
}
Producer tracks multimodal content hashes and the pods that likely hold their encoder-cache entries. Each pod has its own LRU cache of hashes, so eviction is scoped per endpoint rather than global.
func New ¶
func New(ctx context.Context, name string, params *Parameters, podList func() []k8stypes.NamespacedName) (*Producer, error)
New creates a Producer.
func (*Producer) Consumes ¶
func (p *Producer) Consumes() plugin.DataDependencies
Consumes declares the TokenizedPrompt dependency so the data-layer DAG orders the token-producer before this producer runs and auto-creates one when none is configured; multimodal features come from the tokenizer output.
func (*Producer) Extract ¶
Extract removes deleted endpoints from the best-effort multimodal cache-affinity state when endpoint lifecycle events are wired through the data layer.
func (*Producer) PluginState ¶
func (p *Producer) PluginState() *plugin.PluginState
PluginState returns request-scoped state shared between producer extension points.
func (*Producer) PreRequest ¶
func (p *Producer) PreRequest(ctx context.Context, request *scheduling.InferenceRequest, schedulingResult *scheduling.SchedulingResult)
PreRequest records the selected endpoint(s) for each hash in the current request.
func (*Producer) Produce ¶
func (p *Producer) Produce(ctx context.Context, request *scheduling.InferenceRequest, endpoints []scheduling.Endpoint) error
Produce attaches multimodal encoder-cache match data to endpoints.
Source Files