README
¶
Session Affinity Scorer
Type: session-affinity-scorer
Scores candidate pods by giving a higher score to the pod that was previously used for the same session, and zero to the rest. Enables sticky routing for stateful workloads where reusing the same pod reduces latency or preserves context.
The session is carried in a request header whose value is the base64-encoded namespace/name of the previously selected pod. As a ResponseHeaderProcessor, the scorer writes that same header on the response so the client can echo it back on the next request.
Parameters
| Name | Type | Default | Description |
|---|---|---|---|
headerName |
string | x-session-token |
Request and response header carrying the session token. When set, only this header is read; the default is ignored. |
profileName |
string | The name of the profile this instance is associated with. When set (e.g. prefill), the plugin looks up the target pod from the results of that profile in SchedulingResult during the response received phase. When empty, it defaults to the primary (decode) pod. |
Default Configuration (without PD disaggregation)
- type: session-affinity-scorer
parameters:
headerName: x-session-token
PD Disaggregation Configuration
To support session affinity with PD disaggregation, configure two separate instances of the scorer: one for decode and one for prefill.
# Instance for the decode profile (pins decode requests)
- name: session-affinity-decode
type: session-affinity-scorer
parameters:
headerName: x-session-token
# Instance for the prefill profile (pins prefill requests)
- name: session-affinity-prefill
type: session-affinity-scorer
parameters:
headerName: x-session-token-prefill
profileName: prefill
The decode instance uses the default behavior (writing the decode pod to x-session-token). The prefill instance uses profileName: prefill to look up the prefill pod from the scheduling results and write it to x-session-token-prefill. This ensures that subsequent requests in the same session target both the same prefill pod and the same decode pod.
Relationship to the session affinity filter
The session affinity filter (session-affinity-filter) provides the same affinity behavior as a hard constraint and writes the same response header. Configuring both alongside the scorer is unnecessary and can be misleading; see Relationship to the session affinity scorer for details. Use the scorer for a soft preference that can be outweighed by other scorers, or the filter for a hard pin.
Documentation
¶
Overview ¶
Package sessionaffinity provides a session affinity scorer plugin for the epp.
Index ¶
- Constants
- func Factory(name string, rawParameters *json.Decoder, _ plugin.Handle) (plugin.Plugin, error)
- type SessionAffinity
- func (s *SessionAffinity) Category() scheduling.ScorerCategory
- func (s *SessionAffinity) ResponseHeader(ctx context.Context, request *scheduling.InferenceRequest, ...)
- func (s *SessionAffinity) Score(ctx context.Context, request *scheduling.InferenceRequest, ...) map[scheduling.Endpoint]float64
- func (s *SessionAffinity) TypedName() plugin.TypedName
Constants ¶
const (
// SessionAffinityType is the type of the SessionAffinity scorer.
SessionAffinityType = "session-affinity-scorer"
)
Variables ¶
This section is empty.
Functions ¶
Types ¶
type SessionAffinity ¶
type SessionAffinity struct {
// contains filtered or unexported fields
}
SessionAffinity is a routing scorer that routes subsequent requests in a session to the same pod as the first request in the session was sent to, by giving that pod the specified weight and assigning zero score to the rest of the targets
func NewSessionAffinity ¶
func NewSessionAffinity(name, sessionHeader, profileName string) *SessionAffinity
NewSessionAffinity returns a scorer. When sessionHeader is empty the default x-session-token header is used.
func (*SessionAffinity) Category ¶
func (s *SessionAffinity) Category() scheduling.ScorerCategory
Category returns the preference the scorer applies when scoring candidate endpoints.
func (*SessionAffinity) ResponseHeader ¶
func (s *SessionAffinity) ResponseHeader(ctx context.Context, request *scheduling.InferenceRequest, response *requestcontrol.Response, targetPod *datalayer.EndpointMetadata)
ResponseHeader sets the session header on the response sent to the client.
func (*SessionAffinity) Score ¶
func (s *SessionAffinity) Score(ctx context.Context, request *scheduling.InferenceRequest, endpoints []scheduling.Endpoint) map[scheduling.Endpoint]float64
Score assign a high score to the pod used in previous requests and zero to others
func (*SessionAffinity) TypedName ¶
func (s *SessionAffinity) TypedName() plugin.TypedName
TypedName returns the typed name of the plugin.
Source Files