README
¶
Active Request Scorer
Type: active-request-scorer
Scores pods based on the number of active requests being served per pod. It consumes request counts produced by the inflight-load-producer.
Pods at or below idleThreshold active requests receive the maximum score (1.0). Busier pods are scored proportionally in the range [0, maxBusyScore], with the most-loaded pod scoring lowest.
Parameters:
idleThreshold(int, optional, default:0): Request count at or below which a pod is considered idle and scores1.0.maxBusyScore(float, optional, default:1.0): Maximum score assigned when a pod is busy.requestTimeout(duration string, optional): Deprecated and ignored. Accepted only for backward compatibility; request lifecycle is tracked byinflight-load-producer.
Configuration Example:
plugins:
- type: active-request-scorer
name: active-tracker
parameters:
idleThreshold: 2
maxBusyScore: 0.5
schedulingProfiles:
- name: default
plugins:
- pluginRef: active-tracker
weight: 5
Documentation
¶
Overview ¶
Package activerequest provides an active request scorer plugin for the epp.
Index ¶
- Constants
- func Factory(name string, rawParameters *json.Decoder, handle plugin.Handle) (plugin.Plugin, error)
- type ActiveRequest
- func (s *ActiveRequest) Category() scheduling.ScorerCategory
- func (s *ActiveRequest) Consumes() plugin.DataDependencies
- func (s *ActiveRequest) Score(ctx context.Context, _ *scheduling.InferenceRequest, ...) map[scheduling.Endpoint]float64
- func (s *ActiveRequest) TypedName() plugin.TypedName
- func (s *ActiveRequest) WithName(name string) *ActiveRequest
- type Parameters
Constants ¶
const (
// ActiveRequestType is the type of the ActiveRequest scorer.
ActiveRequestType = "active-request-scorer"
)
Variables ¶
This section is empty.
Functions ¶
Types ¶
type ActiveRequest ¶
type ActiveRequest struct {
// contains filtered or unexported fields
}
ActiveRequest scores endpoints based on in-flight request counts produced by the inflight-load-producer data producer.
func NewActiveRequest ¶
func NewActiveRequest(ctx context.Context, params *Parameters) *ActiveRequest
NewActiveRequest creates a new ActiveRequest scorer.
func (*ActiveRequest) Category ¶
func (s *ActiveRequest) Category() scheduling.ScorerCategory
Category returns the preference the scorer applies when scoring candidate endpoints.
func (*ActiveRequest) Consumes ¶
func (s *ActiveRequest) Consumes() plugin.DataDependencies
Consumes returns the in-flight load attribute required for scoring.
func (*ActiveRequest) Score ¶
func (s *ActiveRequest) Score(ctx context.Context, _ *scheduling.InferenceRequest, endpoints []scheduling.Endpoint) map[scheduling.Endpoint]float64
Score scores the given endpoints based on the number of active requests being served by each endpoint. The score is normalized to a range of 0-1.
func (*ActiveRequest) TypedName ¶
func (s *ActiveRequest) TypedName() plugin.TypedName
TypedName returns the typed name of the plugin.
func (*ActiveRequest) WithName ¶
func (s *ActiveRequest) WithName(name string) *ActiveRequest
WithName sets the name of the plugin.
type Parameters ¶
type Parameters struct {
// Deprecated: RequestTimeout is ignored. In-flight request lifecycle is
// tracked by the inflight-load-producer data producer.
RequestTimeout string `json:"requestTimeout"`
// IdleThreshold defines the maximum number of active requests for a pod
// to be considered "idle". Pods with request count <= idleThreshold
// will receive a score of 1.0.
// Default: 0 (only pods with zero requests are considered idle)
IdleThreshold int `json:"idleThreshold"`
// MaxBusyScore defines the maximum score that can be assigned to busy pods
// (pods with request count > idleThreshold). This creates a scoring gap
// between idle and busy pods.
// Range: 0.0 to 1.0
// Default: 1.0 (no gap, current behavior). Pointer so an unset value is
// distinguishable from an explicit 0.0.
// Example: 0.5 means idle pods get 1.0, busiest pod gets 0.0, least busy gets 0.5
MaxBusyScore *float64 `json:"maxBusyScore,omitempty"`
InFlightLoadProducerName string `json:"inFlightLoadProducerName,omitempty"`
}
Parameters defines the parameters for ActiveRequest.
Source Files