predictedlatency

package

v0.9.2 Latest Latest Go to latest Published: Jun 23, 2026 License: Apache-2.0 Imports: 36 Imported by: 0

Details

Valid go.mod file
Redistributable license
Tagged version
Stable version
Learn more about best practices

Repository

github.com/llm-d/llm-d-router

Links

Open Source Insights

README ¶

Predicted Latency Producer (`predicted-latency-producer`)

Type: predicted-latency-producer

Trains XGBoost models via a sidecar and generates per-endpoint TTFT/TPOT predictions.

Interfaces

DataProducer, PreRequest, ResponseHeader, ResponseBody, ProducerPlugin, ConsumerPlugin

Responsibilities

Bulk predictions during Produce (writes LatencyPredictionInfo to endpoint attributes)
SLO headroom calculation per endpoint: headroom = SLO - predicted_latency (used by downstream scorer and admission plugins)
TTFT training data collection on first token / EOS
TPOT training data collection at EOS (streaming mode)
Per-endpoint running request queue tracking (TPOT SLO priority queue)
Prefix cache score forwarding from PrefixCacheMatchInfo attributes
TPOT neutralization for prefill endpoints in disaggregated serving
E2E latency metrics when streamingMode=false

Config

Parameter	Default	Description
`samplingMean`	`1000`	Mean interval for decode token sampling
`maxDecodeTokenSamplesForPrediction`	`0`	Max tokens to sample for TPOT prediction (0 = disabled)
`sloBufferFactor`	`1.0`	Multiplier for SLO headroom calculation
`contextTTL`	`5m`	TTL for per-request context in the cache
`streamingMode`	`false`	Record TTFT on first chunk (true) vs EOS (false)
`endpointRoleLabel`	`""`	Label key for disaggregated serving roles
`predictInProduce`	`true`	Enable/disable bulk predictions. Set false for training-only mode

Default Behavior (`streamingMode: false`)

By default, the system assumes no SLO headers and trains for end-to-end request latency. TTFT is recorded at EOS and represents the full e2e latency (reported as request_e2e_latency_seconds in metrics). TPOT is not trained because there is no per-token streaming. The scorer routes based on e2e latency predictions only, with TPOT automatically neutralized.

Streaming Mode (`streamingMode: true`)

Set this when clients send "stream": true and you want to train separate TTFT (time to first token) and TPOT (time per output token) models. TTFT is recorded on the first streaming chunk, and TPOT is sampled across subsequent tokens.

Disaggregated Serving

Set endpointRoleLabel to the label distinguishing prefill from decode pods. TPOT is automatically neutralized for prefill endpoints (TPOTValid=true, TPOTHeadroom=0), ensuring TPOT doesn't affect scoring, admission, or tier classification for prefill pods.

Files

File	Purpose
`plugin.go`	Struct, factory, config, per-request context, queue helpers
`requestcontrol_hooks.go`	PreRequest, ResponseHeader, ResponseBody hooks
`dataproducer_hooks.go`	Produce, Produces, Consumes
`training.go`	buildTrainingEntry, buildPredictionRequest, bulkPredict
`prediction.go`	generatePredictions, validatePrediction, TPOT neutralization
`decode_token_sampler.go`	Poisson-distributed token sampling for TPOT
`running_request_queue.go`	Per-pod request priority queue

Documentation ¶

Overview ¶

Package requestcontrol contains helpers to decouple latency-predictor logic.

running_request_tpot_slo_queue.go tracks in-flight requests per endpoint, ordered by their TPOT SLO. This allows the predictor to quickly determine the tightest (minimum) TPOT SLO among all running requests on an endpoint, which is used as input to the latency prediction model. The count of running requests is also used by the scorer for idle pod detection.

Requests are added in PreRequest (after scheduling) and removed in ResponseBody at EOS or on TTL eviction.

Index ¶

Constants
Variables
func PredictedLatencyFactory(name string, rawParameters *json.Decoder, handle plugin.Handle) (plugin.Plugin, error)
type Config
type PredictedLatency
- func NewPredictedLatency(name string, config Config, predictor latencypredictor.PredictorInterface) *PredictedLatency

Constants ¶

View Source

const (
	// LatencyDataProviderPluginType is the plugin type for the latency predictor.
	// It trains XGBoost models via the sidecar and generates predictions for scoring.
	LatencyDataProviderPluginType = latencyproducerconstants.LatencyDataProviderPluginType

	// TTFTSLOHeaderKey is the header key for the TTFT SLO.
	TTFTSLOHeaderKey = metadata.TTFTSLOHeaderKey
	// TPOTSLOHeaderKey is the header key for the TPOT SLO.
	TPOTSLOHeaderKey = metadata.TPOTSLOHeaderKey

	// ExperimentalDefaultPrefillProfile is the default profile name for prefill endpoints in disaggregated serving.
	ExperimentalDefaultPrefillProfile = "prefill"
)

Variables ¶

View Source

var DefaultConfig = Config{
	SamplingMean:                       1000,
	MaxDecodeTokenSamplesForPrediction: 0,
	SLOBufferFactor:                    1,
	ContextTTL:                         5 * time.Minute,
	StreamingMode:                      false,
	PredictInProduce:                   true,
}

Functions ¶

func PredictedLatencyFactory ¶

func PredictedLatencyFactory(name string, rawParameters *json.Decoder, handle plugin.Handle) (plugin.Plugin, error)

Types ¶

type Config ¶

type Config struct {
	SamplingMean                       float64       `json:"samplingMean,omitempty"`
	MaxDecodeTokenSamplesForPrediction int           `json:"maxDecodeTokenSamplesForPrediction,omitempty"`
	SLOBufferFactor                    float64       `json:"sloBufferFactor,omitempty"`
	ContextTTL                         time.Duration `json:"contextTTL,omitempty"`
	StreamingMode                      bool          `json:"streamingMode,omitempty"`
	EndpointRoleLabel                  string        `json:"endpointRoleLabel,omitempty"`
	// PredictInProduce controls whether bulk predictions are generated during
	// Produce. Set to false to disable predictions (training-only mode).
	// When false, the predictor still collects training data but does not call the
	// sidecar for predictions. Default: true.
	PredictInProduce            bool   `json:"predictInProduce,omitempty"`
	PrefixMatchInfoProducerName string `json:"prefixMatchInfoProducerName,omitempty"`
}

type PredictedLatency ¶

type PredictedLatency struct {
	// contains filtered or unexported fields
}

PredictedLatency is the latency data provider plugin. It handles:

Produce: bulk predictions via the latency predictor sidecar
PreRequest: dispatch-time bookkeeping (token counters, request queues)
ResponseHeader/ResponseBody: training data collection (TTFT/TPOT)
Produces/Consumes: endpoint attribute declarations

Scoring, picking, and admission are handled by separate sub-plugins: LatencyScorer, AffinityWeightedPicker, and LatencyAdmission.

func NewPredictedLatency ¶

func NewPredictedLatency(name string, config Config, predictor latencypredictor.PredictorInterface) *PredictedLatency

func (*PredictedLatency) Consumes ¶

func (pl *PredictedLatency) Consumes() plugin.DataDependencies

func (*PredictedLatency) PreRequest ¶

func (pl *PredictedLatency) PreRequest(ctx context.Context, request *fwksched.InferenceRequest, schedulingResult *fwksched.SchedulingResult)

func (*PredictedLatency) Produce ¶

func (pl *PredictedLatency) Produce(ctx context.Context, request *fwksched.InferenceRequest, endpoints []fwksched.Endpoint) error

Produce prepares the SLO context for the request, including parsing SLO headers, gathering prefix cache scores, and generating predictions.

func (*PredictedLatency) Produces ¶

func (pl *PredictedLatency) Produces() map[plugin.DataKey]any

func (*PredictedLatency) ResponseBody ¶

func (pl *PredictedLatency) ResponseBody(ctx context.Context, request *fwksched.InferenceRequest, response *requestcontrol.Response, targetMetadata *fwkdl.EndpointMetadata)

ResponseBody handles both per-chunk processing and request completion logic.

func (*PredictedLatency) ResponseHeader ¶

func (pl *PredictedLatency) ResponseHeader(ctx context.Context, request *fwksched.InferenceRequest, response *requestcontrol.Response, targetMetadata *fwkdl.EndpointMetadata)

func (*PredictedLatency) TypedName ¶

func (pl *PredictedLatency) TypedName() plugin.TypedName

Source Files ¶

View all Source files

Directories ¶

Path	Synopsis
constants
latencypredictorclient
tests command

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL