README
¶
Precise Prefix Cache Producer
Type: precise-prefix-cache-producer
DataProducer that owns the precise KV-block index and publishes
per-endpoint PrefixCacheMatchInfo. Pairs with the generic
prefix-cache-scorer; the scorer
must reference this producer by name:
- type: prefix-cache-scorer
parameters:
prefixMatchInfoProducerName: precise-prefix-cache-producer
Without the prefixMatchInfoProducerName field, the scorer falls back
to the auto-spawned approx producer.
Pipeline per request:
- Consume
TokenizedPromptfromtoken-producer. - Hash tokens → KV-block keys →
kvblock.Index.Lookup. - Write
PrefixCacheMatchInfo(matchBlocks, totalBlocks, blockSizeTokens)per endpoint. - (
PreRequest) Speculative-index the selected endpoint(s) with TTL eviction. - (
EndpointExtractor) Per-pod ZMQ subscriber lifecycle on add/delete.
Requires TokenizedPrompt on the request — set by a token-producer
upstream. No-op otherwise.
Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
tokenProcessorConfig |
object | kvblock.DefaultTokenProcessorConfig() |
KV-block hashing for the EPP-recomputed keys (block size, hash seed). |
indexerConfig |
object | kvcache.NewDefaultConfig() |
kvcache.Indexer config. |
kvEventsConfig |
object | kvevents.DefaultConfig() |
KV-events pool config. |
speculativeIndexing |
bool | false |
Seed predicted entries on routing decisions. |
speculativeTTL |
duration | 2s |
TTL for speculative entries. |
See llm-d-kv-cache/docs/configuration.md for nested parameter details.
Engine compatibility
Block keys are recomputed by the EPP from TokenizedPrompt (tokens, model,
multimodal features, cache salt) on both the lookup path and the KV-event
ingestion path, using this plugin's tokenProcessorConfig. The engine's own
block hashes serve only as opaque keys for the engine-to-request mapping, so
blockSize/hashSeed need not match the engine.
The cross-engine requirement is that the engine emits, in its KV-events, the
hash-affecting inputs the EPP hashes: token_ids, and extra_keys carrying
multimodal identifiers and cache_salt. An input the engine omits from
extra_keys is absent on the event side, so requests carrying it do not
correlate.
| Engine | extra_keys in KV-events |
cache_salt |
|---|---|---|
| vLLM | emitted | in block-0 extra_keys; salted prefixes isolated and precise-routed |
| SGLang | not emitted | baked into engine block hashes but not surfaced; salted requests are precise-cache misses until SGLang emits extra_keys |
Salt isolation is enforced by the engine regardless; the above affects only routing accuracy for salted requests.
Documentation
¶
Overview ¶
Package preciseprefixcache hashes TokenizedPrompt into KV-block keys, looks them up in the index, and writes per-endpoint PrefixCacheMatchInfo.
Index ¶
- Constants
- func PluginFactory(name string, rawParameters *json.Decoder, handle plugin.Handle) (plugin.Plugin, error)
- type PluginConfig
- type Producer
- func (p *Producer) Consumes() plugin.DataDependencies
- func (p *Producer) Extract(ctx context.Context, event fwkdl.EndpointEvent) error
- func (p *Producer) PreRequest(ctx context.Context, request *scheduling.InferenceRequest, ...)
- func (p *Producer) Produce(ctx context.Context, request *scheduling.InferenceRequest, ...) error
- func (p *Producer) Produces() map[plugin.DataKey]any
- func (p *Producer) TypedName() plugin.TypedName
Constants ¶
const PluginType = "precise-prefix-cache-producer"
PluginType is the registered type name of the precise-prefix-cache-producer.
Variables ¶
This section is empty.
Functions ¶
func PluginFactory ¶
func PluginFactory(name string, rawParameters *json.Decoder, handle plugin.Handle) (plugin.Plugin, error)
PluginFactory parses the raw plugin configuration and returns a configured Producer. Rejects configs with indexerConfig.tokenizersPoolConfig set, since this producer is tokens-only and requires an upstream token-producer.
Types ¶
type PluginConfig ¶
type PluginConfig struct {
TokenProcessorConfig *kvblock.TokenProcessorConfig `json:"tokenProcessorConfig"`
IndexerConfig *kvcache.Config `json:"indexerConfig"`
KVEventsConfig *kvevents.Config `json:"kvEventsConfig"`
// SpeculativeIndexing seeds predicted cache entries for the selected
// endpoint(s) immediately after a routing decision, so the next
// same-prefix request hits without waiting for engine confirmation.
SpeculativeIndexing bool `json:"speculativeIndexing"`
// SpeculativeTTL bounds how long speculative entries live before
// eviction. Go duration string; defaults to defaultSpeculativeTTL when
// empty.
SpeculativeTTL string `json:"speculativeTTL"`
}
PluginConfig configures the precise-prefix-cache-producer. Nested fields mirror the llm-d-kv-cache configuration shape (see that repo's docs/configuration.md for details on TokenProcessorConfig, IndexerConfig, and KVEventsConfig).
type Producer ¶
type Producer struct {
// contains filtered or unexported fields
}
Producer is a DataProducer plugin that maintains a KV-block prefix-cache index by subscribing to vLLM KV-events and writes per-endpoint PrefixCacheMatchInfo for each request. Operators pair it with the generic prefix-cache-scorer (set prefixMatchInfoProducerName to this producer's instance name) to route requests by precise cache locality.
Speculative-indexing logic lives in prerequest.go; per-pod ZMQ subscriber lifecycle in extractor.go.
func New ¶
New constructs a precise-prefix-cache-producer. The instance name becomes the producer name on PrefixCacheMatchInfoDataKey, which downstream consumers must match (see prefix-cache-scorer's prefixMatchInfoProducerName). The kvcache indexer, KV-events pool, and any local ZMQ subscriber start in background goroutines bound to ctx.
func (*Producer) Consumes ¶
func (p *Producer) Consumes() plugin.DataDependencies
Consumes declares the TokenizedPrompt dependency from token-producer so the data-layer DAG orders tokenization before this producer runs.
func (*Producer) Extract ¶
Extract processes endpoint lifecycle events emitted by the endpoint-notification-source: add/update installs a per-pod ZMQ KV-events subscriber, delete tears one down. No-op unless per-pod discovery is enabled.
func (*Producer) PreRequest ¶
func (p *Producer) PreRequest(ctx context.Context, request *scheduling.InferenceRequest, schedulingResult *scheduling.SchedulingResult, )
PreRequest seeds speculative KV-block index entries for the endpoint(s) selected by the scheduler, so the next same-prefix request hits without waiting for confirmed KV-events from the engine. Entries are tracked in a TTL cache and evicted automatically. No-op when speculativeIndexing is disabled.
func (*Producer) Produce ¶
func (p *Producer) Produce(ctx context.Context, request *scheduling.InferenceRequest, endpoints []scheduling.Endpoint, ) error
Produce hashes the request's TokenizedPrompt into KV-block keys, looks them up in the per-endpoint KV-block index, and writes PrefixCacheMatchInfo to each candidate endpoint. No-op when the request carries no tokens. With speculativeIndexing enabled, the computed block keys are stashed for PreRequest to seed the index after a routing decision is made.
Source Files