README
¶
Load Aware Scorer
Type: load-aware-scorer
Interfaces: scheduling.Scorer
Category: Distribution
Scores pods based on their current load, measured by the number of requests waiting in each pod's queue, so the scheduler can route new traffic away from busy endpoints.
What it does
For each candidate endpoint, the scorer reads the pod's current waiting-queue size and assigns a score in [0, 0.5]:
- Pods with an empty waiting queue score
0.5. - Pods with requests queued score between
0.5and0.0— the more queued, the lower the score. - Pods with queue depth at or beyond
thresholdscore0.0.
How It Works
The score is computed from the live WaitingQueueSize metric using a piecewise linear function:
waitingRequests == 0 → score = 0.5
waitingRequests >= threshold → score = 0.0
0 < waitingRequests < threshold → score = 0.5 * (1 - waitingRequests / threshold)
Note that the maximum score is capped at 0.5, not 1.0. This reflects that an empty queue is the best observable signal of availability given current metrics, but does not necessarily indicate spare capacity. A future extension could raise idle pods above 0.5 once capacity-headroom information becomes available.
Inputs consumed
WaitingQueueSize— live pod metric read fromendpoint.GetMetrics(), populated by the metrics data pipeline (metrics-data-source+core-metrics-extractor).
Configuration
Location: schedulingProfiles[N].plugins
Parameters
| Name | Type | Required | Default | Description |
|---|---|---|---|---|
threshold |
int |
No | 128 |
Queue depth at which the score reaches 0.0. Non-positive values are rejected and replaced by the default. |
Example
plugins:
- type: load-aware-scorer
name: load-balancer
config:
threshold: 100
schedulingProfiles:
- name: default
plugins:
- pluginRef: load-balancer
weight: 5
Limitations
- Its maximum of 0.5 (vs. the usual 1.0 ceiling of other scorers) means its effective pull under weighted aggregation is roughly half its configured weight — raise weight to compensate.
- Treats all queued requests equally regardless of size or remaining work — a queue of one long request scores the same as a queue of one short request.
- Depends on accurate, fresh
WaitingQueueSizemetrics; stale or missing metrics make the scorer degenerate (all pods tied at0.5).
Related Documentation
- Active Request Scorer — alternative distribution scorer based on in-flight request counts instead of queue depth
Documentation
¶
Overview ¶
Package loadaware provides a load-aware scorer plugin for the epp.
Index ¶
Constants ¶
const ( // LoadAwareType is the type of the LoadAware scorer LoadAwareType = "load-aware-scorer" // QueueThresholdDefault defines the default queue threshold value QueueThresholdDefault = 128 )
Variables ¶
This section is empty.
Functions ¶
Types ¶
type LoadAware ¶
type LoadAware struct {
// contains filtered or unexported fields
}
LoadAware scorer that is based on load
func NewLoadAware ¶
NewLoadAware creates a new load based scorer
func (*LoadAware) Category ¶
func (s *LoadAware) Category() scheduling.ScorerCategory
Category returns the preference the scorer applies when scoring candidate endpoints.
func (*LoadAware) Score ¶
func (s *LoadAware) Score(_ context.Context, _ *scheduling.InferenceRequest, endpoints []scheduling.Endpoint) map[scheduling.Endpoint]float64
Score scores the given pod in range of 0-1 Currently metrics contains number of requests waiting in the queue, there is no information about number of requests that can be processed in the given pod immediately. Pod with empty waiting requests queue is scored with 0.5 Pod with requests in the queue will get score between 0.5 and 0. Score 0 will get pod with number of requests in the queue equal to the threshold used in load-based filter In the future, pods with additional capacity will get score higher than 0.5
Source Files