README
¶
Request-Control Data Producer Plugins
Data producer plugins run in the PrepareRequestData phase of the request-control pipeline, before filters and scorers execute. Each plugin implements requestcontrol.DataProducer and writes typed attributes onto endpoints or the InferenceRequest itself. Downstream plugins (scorers, filters, admission controllers) read these attributes without re-doing the computation.
The following diagram shows how producers are connected to the request control and scheduling plugins:
Producers may also implement additional lifecycle hooks:
PreRequest— called after a routing decision is made; used to persist bookkeeping state (e.g., update a cache index, increment an in-flight counter).ResponseHeader/ResponseBody— called as response data arrives; used to collect training data or release in-flight counters.
Available Producers
| Plugin type | Package | Produces | Summary |
|---|---|---|---|
token-producer |
tokenizer |
TokenizedPrompt |
Tokenizes the request prompt via vLLM /render; required by precise-prefix-cache-producer and context-length-aware scorers. |
approx-prefix-cache-producer |
approximateprefix |
PrefixCacheMatchInfo |
Hashes the prompt into blocks and matches against a per-pod LRU index for approximate prefix-cache affinity. |
precise-prefix-cache-producer |
preciseprefixcache |
PrefixCacheMatchInfo |
Maintains a precise KV-block index by subscribing to vLLM KV-events; requires token-producer upstream. |
inflight-load-producer |
inflightload |
InFlightLoad |
Tracks real-time in-flight request and token counts per endpoint across the full request lifecycle. |
predicted-latency-producer |
predictedlatency |
LatencyPredictionInfo |
Trains XGBoost models via a sidecar and generates per-endpoint TTFT/TPOT predictions. |
session-id-producer |
sessionid |
SessionID |
Extracts a session identifier from a request header or cookie and publishes it for affinity-aware plugins. |
mm-embeddings-cache-producer |
multimodal |
EncoderCacheMatchInfo |
Tracks which pods recently processed each multimodal input hash and scores encoder-cache affinity. |
Plugin ordering and dependencies
The framework resolves a DAG from each plugin's Produces and Consumes declarations and runs producers in dependency order. Explicit dependencies to be aware of:
precise-prefix-cache-producerrequirestoken-producerupstream (it consumesTokenizedPrompt).mm-embeddings-cache-produceroptionally consumesTokenizedPrompt; configuretoken-producerfirst when multimodal features need tokenizer-derived hashes.inflight-load-produceroptionally consumesPrefixCacheMatchInfofrom an approx or precise prefix producer; prefix-discounting is applied automatically when the attribute is present.predicted-latency-produceroptionally consumesPrefixCacheMatchInfo; setprefixMatchInfoProducerNamein its config to the name of the prefix producer instance.
Related documentation
Directories
¶
| Path | Synopsis |
|---|---|
|
Package multimodal provides a data producer for multimodal encoder-cache affinity.
|
Package multimodal provides a data producer for multimodal encoder-cache affinity. |
|
Package preciseprefixcache hashes TokenizedPrompt into KV-block keys, looks them up in the index, and writes per-endpoint PrefixCacheMatchInfo.
|
Package preciseprefixcache hashes TokenizedPrompt into KV-block keys, looks them up in the index, and writes per-endpoint PrefixCacheMatchInfo. |
|
Package requestcontrol contains helpers to decouple latency-predictor logic.
|
Package requestcontrol contains helpers to decouple latency-predictor logic. |
|
latencypredictorclient/tests
command
|
|
|
Package sessionid provides a DataProducer that extracts a session identifier from a configured request header or named cookie and publishes it as a SessionID attribute on the InferenceRequest attribute store, so that affinity-aware scorers and filters can read it without knowing how the session was carried on the wire.
|
Package sessionid provides a DataProducer that extracts a session identifier from a configured request header or named cookie and publishes it as a SessionID attribute on the InferenceRequest attribute store, so that affinity-aware scorers and filters can read it without knowing how the session was carried on the wire. |
|
Package tokenizer provides a DataProducer plugin that tokenizes the request prompt and publishes the result on InferenceRequestBody.TokenizedPrompt for downstream consumers (scorers, filters, other data producers).
|
Package tokenizer provides a DataProducer plugin that tokenizes the request prompt and publishes the result on InferenceRequestBody.TokenizedPrompt for downstream consumers (scorers, filters, other data producers). |
Click to show internal directories.
Click to hide internal directories.