latency

Latency Attributes

This package defines the data structures for latency predictions and performance tracking.

LatencyPredictionInfo

Contains latency predictions for an endpoint, typically generated by a predictor or extracted from model server metrics.

• Key: LatencyPredictionInfoDataKey
• Fields:
  • TTFT: Predicted Time To First Token (ms).
  • TPOT: Predicted Time Per Output Token (ms).
  • TTFTValid: Boolean indicating if TTFT is within SLO.
  • TPOTValid: Boolean indicating if TPOT is within SLO.
  • TTFTHeadroom: TTFT headroom relative to SLO.
  • TPOTHeadroom: TPOT headroom relative to SLO.
  • DispatchedRequestCount: Number of requests dispatched by this EPP instance.

Producers

The following plugins produce this attribute:

• predicted-latency-producer (Request Control): Uses a sidecar-based machine learning model to generate TTFT and TPOT predictions for each candidate endpoint based on current load and request characteristics.