framework/

Package flowcontrol defines the core plugin interfaces for extending the Flow Control layer.

Package session declares the SessionID attribute that carries per-request session identity for affinity scoring and filtering.

Package file provides a file-based EndpointDiscovery implementation that reads a YAML (or JSON) file listing inference endpoints.

Package httptest provides test helpers for HTTPDataSource parser implementations.

Package filtering provides built-in EvictionFilterPolicy implementations.

Package fairness provides the standard implementations of the FairnessPolicy interface.

Package globalstrict implements a greedy fairness policy that enforces a strict global ordering across all flows within a priority band, ignoring flow boundaries.

Package programaware implements a flow-control fairness policy that schedules per-program queues using a swappable scoring strategy.

Package roundrobin implements a fairness policy that selects queues from a priority band using a round-robin strategy, cycling through active flows one by one to guarantee that no single flow can starve others.

Package ordering provides the standard implementations of the OrderingPolicy interface.

Package edf implements an ordering policy that selects requests based on their absolute deadlines (Earliest Deadline First).

Package fcfs implements an ordering policy that selects requests based on their arrival order at the Flow Control layer (First-Come, First-Served).

Package slodeadline implements an ordering policy that selects requests based on an SLO-based deadline derived from request headers.

Package requestcontrol holds definitions shared by all requestcontrol framework plugins.

Package probabilisticadmitter implements a binary-tier probabilistic admission control plugin.

Package multimodal provides a data producer for multimodal encoder-cache affinity.

Package preciseprefixcache hashes TokenizedPrompt into KV-block keys, looks them up in the index, and writes per-endpoint PrefixCacheMatchInfo.

Package requestcontrol contains helpers to decouple latency-predictor logic.

Package sessionid provides a DataProducer that extracts a session identifier from a configured request header or named cookie and publishes it as a SessionID attribute on the InferenceRequest attribute store, so that affinity-aware scorers and filters can read it without knowing how the session was carried on the wire.

Package tokenizer provides a DataProducer plugin that tokenizes the request prompt and publishes the result on InferenceRequestBody.TokenizedPrompt for downstream consumers (scorers, filters, other data producers).

Package agentidentity provides a PreAdmitter plugin that resolves agent identity from provider-specific headers into the FairnessID field.

Package vllmhttp provides a request parser for vLLM HTTP endpoints that are not part of the OpenAI-compatible API surface — specifically /inference/v1/generate.

Package scheduling holds definitions shared by all scheduling framework plugins.

Package bylabel provides filter plugins for the epp.

Package prefixcacheaffinity provides a probabilistic filter that narrows candidates to "sticky" endpoints (those with high prefix cache scores).

Package sessionaffinity provides a session affinity filter plugin for the epp.

Package headroomtier provides a probabilistic filter that selects endpoints based on predicted latency headroom (SLO - predicted).

Package picker provides the standard implementations of the Picker interface for the scheduling framework.

Package maxscore implements a scheduling picker that selects the endpoint(s) with the highest score calculated during the scoring phase.

Package random implements a scheduling picker that selects endpoints uniformly at random, ignoring any scores calculated by scorer plugins.

Package weightedrandom implements a scheduling picker that selects endpoints randomly, where the probability of an endpoint being selected is proportional to its score.

Package dataparallel provides a data-parallel profile handler plugin for the epp.

Package disagg provides profile handler plugins for the epp.

Package activerequest provides an active request scorer plugin for the epp.

Package contextlengthaware provides plugins that implement multiple extension points (e.g., both Filter and Scorer interfaces).

Package latency provides a scorer that scores endpoints based on predicted latency headroom.

Package loadaware provides a load-aware scorer plugin for the epp.

Package mmcacheaffinity scores endpoints from multimodal encoder-cache match info produced by the request-control multimodal data producer.

Package nohitlru provides a no-hit LRU scorer plugin for the epp.

Package preciseprefixcache provides a precise prefix cache scorer plugin for the epp.

Package sessionaffinity provides a session affinity scorer plugin for the epp.

Package sessionaffinity provides the session header helpers shared by the session-affinity scorer and filter plugins.