disagg

package
v0.9.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 23, 2026 License: Apache-2.0 Imports: 24 Imported by: 0

README

Disaggregated Profile Handler, PreRequest, and Decider Plugins

Plugins for disaggregated inference scheduling: a profile handler that selects the active stages: EPD (no disaggregation), P/D (Prefill/Decode), E/P/D (Encode/Prefill/Decode), or E/PD (Encode/Prefill-Decode), legacy headers handlers (deprecated) kept for backward compatibility, and decider plugins that control whether each disaggregation stage runs per request.

Contents

Profile Handlers

DisaggProfileHandler

Type: disagg-profile-handler Interfaces: scheduling.ProfileHandler

Orchestrates up to three scheduling stages per request — decode (always), and optionally encode and prefill — based on which decider plugins are configured.

What it does

Runs each scheduling stage in sequence and assembles the final result from all stages that ran.

  1. Run the decode profile (always).
  2. If an encode decider is configured and approves the request, run the encode profile.
  3. If a prefill decider is configured and approves the request, run the prefill profile.
  4. Return the assembled scheduling result with decode as the primary profile.
How It Works

The handler is invoked repeatedly by the framework until all stages are complete. Each optional stage is gated by a decider: if the decider returns false for a request, the stage is marked as skipped so the handler doesn't revisit it on the next invocation. If the decode stage finds no suitable endpoint, all remaining stages are skipped and the request fails.

Inputs consumed
  • PrefixCacheMatchInfo — endpoint attribute from approx-prefix-cache-producer, read by the configured prefill decider (e.g. prefix-based-pd-decider) when deciding whether to run the prefill stage.
Configuration
Parameters
Name Type Required Default Description
profiles.decode string No "decode" Name of the decode scheduling profile.
profiles.prefill string No "prefill" Name of the prefill scheduling profile.
profiles.encode string No "encode" Name of the encode scheduling profile.
deciders.prefill string No — Name of the prefill decider plugin. When set, enables P/D disaggregation.
deciders.encode string No — Name of the encode decider plugin. When set, enables E disaggregation.
Example

Decode-only (no disaggregation):

plugins:
  - type: disagg-profile-handler

P/D disaggregation:

plugins:
  - type: disagg-profile-handler
    parameters:
      deciders:
        prefill: prefix-based-pd-decider

E/P/D disaggregation:

plugins:
  - type: disagg-profile-handler
    parameters:
      deciders:
        prefill: prefix-based-pd-decider
        encode: always-disagg-multimodal-decider
Limitations
  • Without a configured decider, the corresponding stage is disabled for all requests — this is a static decision at startup, not per-request.
  • The names in deciders.prefill and deciders.encode must match plugin names declared earlier in the same configuration.
  • When using P/D disaggregation, a PrefixCachePlugin must be configured in the prefill and decode scheduling profiles.
PdProfileHandler (Deprecated)

Type: pd-profile-handler Interfaces: scheduling.ProfileHandler

Deprecated: Use disagg-profile-handler instead.

PreRequest Plugins

DisaggHeadersHandler (Deprecated)

Type: disagg-headers-handler Interfaces: requestcontrol.PreRequest

Deprecated: Use disagg-profile-handler instead.

disagg-profile-handler now implements requestcontrol.PreRequest natively.

Planned removal: v0.11.

Sets HTTP routing headers on the outgoing request so the inference proxy can forward prefill and encode work to the selected disaggregated pods.

What it does

Reads the scheduling result and writes pod addresses as request headers for each disaggregated stage that ran.

  1. If a prefill endpoint was selected, write its ip:port to x-prefiller-host-port.
  2. If one or more encode endpoints were selected, write their comma-separated ip:port list to x-encoder-hosts-ports.
  3. If a stage did not run or found no endpoints, that header is omitted.
Inputs consumed
  • SchedulingResult.ProfileResults — per-profile endpoint selections produced by disagg-profile-handler.
Output produced
  • x-prefiller-host-port request header — <ip:port> of the selected prefill pod; absent when P/D disaggregation was skipped.
  • x-encoder-hosts-ports request header — comma-separated <ip:port> list of selected encode pods; absent when encode disaggregation was skipped.
Configuration
Parameters
Name Type Required Default Description
prefillProfile string No "prefill" Name of the profile used for prefill scheduling. Only needed if the prefill profile is not named prefill.
encodeProfile string No "encode" Name of the profile used for encode scheduling. Only needed if the encode profile is not named encode.
Example
plugins:
  - type: disagg-headers-handler

Custom profile names:

plugins:
  - type: disagg-headers-handler
    parameters:
      prefillProfile: "my-prefill"
      encodeProfile: "my-encode"
PrefillHeaderHandler (Deprecated)

Type: prefill-header-handler Interfaces: requestcontrol.PreRequest

Deprecated: Use disagg-profile-handler instead.

Planned removal: v0.11.

Decider Plugins

PrefixBasedPDDecider

Type: prefix-based-pd-decider

Decides per-request whether P/D disaggregation should run, based on how much of the prompt is already cached on the selected decode pod.

What it does

Compares the uncached portion of the request prompt against a configurable threshold, triggering P/D disaggregation only when the uncached suffix is long enough to justify the overhead.

  1. Read the prompt token count as len(request.Body.TokenizedPrompt.TokenIDs).
  2. Read PrefixCacheMatchInfo from the decode endpoint attributes.
  3. Compute uncached suffix length.
  4. Return true (disaggregate) if uncached tokens ≥ nonCachedTokens.
How It Works

The prompt token count is len(request.Body.TokenizedPrompt.TokenIDs), populated by a token-producer — auto-created with the tokenizer-free estimate backend when none is configured. Prefix cache state is read from the PrefixCacheMatchInfo attribute on the decode endpoint, populated by approx-prefix-cache-producer. If the attribute is absent or malformed, disaggregation is skipped. Setting nonCachedTokens: 0 disables the decider entirely (always returns false).

Inputs consumed
  • PrefixCacheMatchInfo — endpoint attribute from approx-prefix-cache-producer, read from the decode endpoint.
  • request.Body.TokenizedPrompt.TokenIDs — token IDs from a token-producer plugin; their count is the prompt token count.
Configuration
Parameters
Name Type Required Default Description
nonCachedTokens int No 0 Uncached token threshold above which P/D disaggregation is triggered. 0 disables the decider.
Example
plugins:
  - type: prefix-based-pd-decider
    parameters:
      nonCachedTokens: 512
  - type: disagg-profile-handler
    parameters:
      deciders:
        prefill: prefix-based-pd-decider
Limitations
  • nonCachedTokens: 0 disables disaggregation entirely (the decider always returns false).
  • A token-producer populates TokenizedPrompt; when none is configured the framework auto-creates one with the estimate backend, so disaggregation works without extra setup.
  • Requires PrefixCacheMatchInfo on the decode endpoint; if absent, disaggregation is skipped with an error log.
AlwaysDisaggPDDecider

Type: always-disagg-pd-decider

Unconditionally approves P/D disaggregation for every request, regardless of cache state or prompt length.

What it does

Returns true for every request. Useful for testing or environments where P/D disaggregation should always run.

Inputs consumed

None — ignores request content and endpoint state.

Configuration
Parameters

None.

AlwaysDisaggMultimodalDecider

Type: always-disagg-multimodal-decider

Approves encode disaggregation for requests that contain multimodal content (images, audio, video); passes text-only requests through without disaggregation.

What it does

Inspects the chat completions message content blocks for image_url, video_url, or input_audio types and returns true when any such block is found.

Inputs consumed
  • Request body (ChatCompletions.Messages) — inspected for multimodal content blocks.
Configuration
Parameters

None.

Documentation

Overview

Package disagg provides profile handler plugins for the epp.

Package disagg provides pre-request plugins for GIE.

Package disagg provides profile handler plugins for the epp.

Package disagg provides profile handler plugin for the epp.

Index

Constants

View Source 
const (
	// DisaggHeadersHandlerType is the type of the HeadersHandler.
	//
	// Deprecated: Use DisaggProfileHandlerType instead, disagg-profile-handler now implements PreRequest natively.
	DisaggHeadersHandlerType = "disagg-headers-handler"

	// PrefillHeaderHandlerType is a deprecated alias for DisaggHeadersHandlerType.
	//
	// Deprecated: Use DisaggProfileHandlerType instead, disagg-profile-handler now implements PreRequest natively.
	PrefillHeaderHandlerType = "prefill-header-handler"
)
View Source 
const (
	// DecisionTypeDecodeOnly is for requests that are routed to decode instance only.
	DecisionTypeDecodeOnly = "decode-only"
	// DecisionTypePrefillDecode is for requests that are gone through P/D or EP/D.
	DecisionTypePrefillDecode = "prefill-decode"
	// DecisionTypeEncodeDecode is for requests that are gone through E/PD.
	DecisionTypeEncodeDecode = "encode-decode"
	// DecisionTypeEncodePrefillDecode is for requests that are gone through E/P/D.
	DecisionTypeEncodePrefillDecode = "encode-prefill-decode"
)
View Source 
const (
	// AlwaysDisaggMulimodalPluginType is the type-name of the AlwaysDisaggMultimodalDecider plugin.
	AlwaysDisaggMulimodalPluginType = "always-disagg-multimodal-decider"
)
View Source 
const (
	// AlwaysDisaggPDDeciderPluginType is the type-name of the alwaysDisaggPDDecider plugin.
	AlwaysDisaggPDDeciderPluginType = "always-disagg-pd-decider"
)
View Source 
const (
	// DisaggProfileHandlerType is the canonical type for the unified disaggregation profile handler.
	DisaggProfileHandlerType = "disagg-profile-handler"
)
View Source 
const (
	// PdProfileHandlerType is a legacy alias for DisaggProfileHandlerType.
	PdProfileHandlerType = "pd-profile-handler"
)
View Source 
const (
	// PrefixBasedPDDeciderPluginType is the type-name of the prefixBasedPDDecider plugin.
	PrefixBasedPDDeciderPluginType = "prefix-based-pd-decider"
)

Variables

View Source 
var (
	// SchedulerPDDecisionCount records request P/D decision.
	//
	// Deprecated: Use LlmdPDDecisionCount instead.
	// Tracked in: https://github.com/llm-d/llm-d-inference-scheduler/issues/1070
	SchedulerPDDecisionCount = prometheus.NewCounterVec(
		prometheus.CounterOpts{
			Subsystem: eppmetrics.SchedulerSubsystem,
			Name:      "pd_decision_total",
			Help:      metricsutil.HelpMsgWithStability("[Deprecated: Use llm_d_epp_pd_decision_total] Total number of P/D disaggregation decisions made", compbasemetrics.ALPHA),
		},
		[]string{"model_name", "decision_type"},
	)

	// LlmdPDDecisionCount records request P/D decision.
	LlmdPDDecisionCount = prometheus.NewCounterVec(
		prometheus.CounterOpts{
			Subsystem: eppmetrics.LLMDRouterEndpointPickerSubsystem,
			Name:      "pd_decision_total",
			Help:      metricsutil.HelpMsgWithStability("Total number of P/D disaggregation decisions made", compbasemetrics.ALPHA),
		},
		[]string{"plugin_name", "plugin_type", "model_name", "decision_type"},
	)

	// SchedulerDisaggDecisionCount records disaggregation routing decisions,
	// covering all stages: decode-only, prefill-decode, encode-decode, encode-prefill-decode.
	//
	// Deprecated: Use llm_d_epp_disagg_decision_total instead.
	// Tracked in: https://github.com/llm-d/llm-d-inference-scheduler/issues/1070
	SchedulerDisaggDecisionCount = prometheus.NewCounterVec(
		prometheus.CounterOpts{
			Subsystem: eppmetrics.SchedulerSubsystem,
			Name:      "disagg_decision_total",
			Help:      metricsutil.HelpMsgWithStability("[Deprecated: Use llm_d_epp_disagg_decision_total] Total number of disaggregation routing decisions made", compbasemetrics.ALPHA),
		},
		[]string{"model_name", "decision_type"},
	)

	// LlmdDisaggDecisionCount records disaggregation routing decisions.
	LlmdDisaggDecisionCount = prometheus.NewCounterVec(
		prometheus.CounterOpts{
			Subsystem: eppmetrics.LLMDRouterEndpointPickerSubsystem,
			Name:      "disagg_decision_total",
			Help:      metricsutil.HelpMsgWithStability("Total number of disaggregation routing decisions made", compbasemetrics.ALPHA),
		},
		[]string{"plugin_name", "plugin_type", "model_name", "decision_type"},
	)
)

Functions

func AlwaysDisaggMulimodalDeciderPluginFactory 

func AlwaysDisaggMulimodalDeciderPluginFactory(name string, _ *json.Decoder, _ plugin.Handle) (plugin.Plugin, error)

AlwaysDisaggMulimodalDeciderPluginFactory defines the factory function for creating a new instance of the AlwaysDisaggEncodeDecider.

func AlwaysDisaggPDDeciderPluginFactory 

func AlwaysDisaggPDDeciderPluginFactory(name string, _ *json.Decoder,
	_ plugin.Handle) (plugin.Plugin, error)

AlwaysDisaggPDDeciderPluginFactory defines the factory function for creating a new instance of the AlwaysDisaggPDDecider.

func DisaggDecisionType 

func DisaggDecisionType(encodeUsed, prefillUsed bool) string

DisaggDecisionType returns the DecisionType* constant corresponding to which disaggregation stages were used for a request.

func HandlerFactory 

func HandlerFactory(name string, rawParameters *json.Decoder, handle plugin.Handle) (plugin.Plugin, error)

HandlerFactory is the unified factory for all disaggregation profile handlers. 

if parameters.deciders.prefill is set - P disaggregation will be supported
if parameters.deciders.encode is set - E disaggregation will be supported

func HeadersHandlerFactory deprecated 

func HeadersHandlerFactory(name string, rawParameters *json.Decoder, _ plugin.Handle) (plugin.Plugin, error)

HeadersHandlerFactory defines the factory function for the HeadersHandler.

Deprecated: Use HandlerFactory instead, disagg-profile-handler now implements PreRequest natively.

func PdProfileHandlerFactory deprecated 

func PdProfileHandlerFactory(name string, rawParameters *json.Decoder, handle plugin.Handle) (plugin.Plugin, error)

PdProfileHandlerFactory defines the factory function for the PdProfileHandler.

Deprecated: Use HandlerFactory instead.

func PrefixBasedPDDeciderPluginFactory 

func PrefixBasedPDDeciderPluginFactory(name string, rawParameters *json.Decoder,
	handle plugin.Handle) (plugin.Plugin, error)

PrefixBasedPDDeciderPluginFactory defines the factory function for creating a new instance of the prefixBasedPDDecider.

func RecordDisaggDecision 

func RecordDisaggDecision(pluginName, pluginType, modelName, decisionType string)

RecordDisaggDecision increments the counter for a disaggregation routing decision. The decisionType must be one of the DecisionType* constants (DecisionTypeDecodeOnly, DecisionTypePrefillDecode, DecisionTypeEncodeDecode, DecisionTypeEncodePrefillDecode). The model parameter should be the target model name; if empty, "unknown" is used.

func RecordPDDecision deprecated 

func RecordPDDecision(pluginName, pluginType, modelName, decisionType string)

RecordPDDecision increments the counter for a specific P/D routing decision.

Deprecated: Use RecordDisaggDecision instead.

Types

type AlwaysDisaggMultimodalDecider 

type AlwaysDisaggMultimodalDecider struct {
	// contains filtered or unexported fields
}

AlwaysDisaggMultimodalDecider is an EP decider plugin which always decides to encode.

func (*AlwaysDisaggMultimodalDecider) TypedName 

func (d *AlwaysDisaggMultimodalDecider) TypedName() plugin.TypedName

TypedName returns the typed name of the plugin.

func (*AlwaysDisaggMultimodalDecider) WithName 

func (d *AlwaysDisaggMultimodalDecider) WithName(name string) *AlwaysDisaggMultimodalDecider

WithName sets the name of the plugin.

type AlwaysDisaggPDDecider 

type AlwaysDisaggPDDecider struct {
	// contains filtered or unexported fields
}

AlwaysDisaggPDDecider is a PD decider plugin which always decide to disaggregate PD

func (*AlwaysDisaggPDDecider) TypedName 

func (d *AlwaysDisaggPDDecider) TypedName() plugin.TypedName

TypedName returns the typed name of the plugin.

func (*AlwaysDisaggPDDecider) WithName 

func (d *AlwaysDisaggPDDecider) WithName(name string) *AlwaysDisaggPDDecider

WithName sets the name of the plugin.

type Handler 

type Handler struct {
	// contains filtered or unexported fields
}

Handler is the unified disaggregation profile handler. It drives one or more of the following stages, each optional except decode:

  • Encode (E): schedules encoder pods for multimodal content
  • Prefill (P): schedules a prefill pod for KV-cache disaggregation
  • Decode (D): schedules the decode pod (always runs first)

All four handler types (D, P/D, E/PD, E/P/D) share this single implementation; active stages are selected by setting encodeProfile / prefillProfile.

func NewDisaggProfileHandler 

func NewDisaggProfileHandler(decodeProfile, prefillProfile, encodeProfile string, pdDecider, encodeDecider deciderPlugin) *Handler

NewDisaggProfileHandler creates a Handler directly. Active stages are determined by non-empty deciders.

func (*Handler) Consumes 

func (*Handler) Consumes() plugin.DataDependencies

Consumes defines data types consumed by this plugin (through the PD decider).

func (*Handler) Pick 

func (h *Handler) Pick(ctx context.Context, request *scheduling.InferenceRequest, profiles map[string]scheduling.SchedulerProfile,
	profileResults map[string]*scheduling.ProfileRunResult) map[string]scheduling.SchedulerProfile

Pick implements scheduling.ProfileHandler. Stages run in order: decode → encode (optional) → prefill (optional). Returns the next profile to execute, or an empty map when all stages are done.

func (*Handler) PreRequest 

func (h *Handler) PreRequest(ctx context.Context, request *scheduling.InferenceRequest, schedulingResult *scheduling.SchedulingResult)

PreRequest wires prefill and encode SchedulerProfile results into headers so the sidecar knows which pods to contact for disaggregated work.

func (*Handler) ProcessResults 

func (h *Handler) ProcessResults(
	_ context.Context,
	request *scheduling.InferenceRequest,
	profileResults map[string]*scheduling.ProfileRunResult,
) (*scheduling.SchedulingResult, error)

ProcessResults implements scheduling.ProfileHandler. Builds the final SchedulingResult from whichever stages ran successfully.

func (*Handler) TypedName 

func (h *Handler) TypedName() plugin.TypedName

TypedName returns the typed name of the plugin.

func (*Handler) WithName 

func (h *Handler) WithName(name string) *Handler

WithName sets the instance name of the plugin.

type HeadersHandler deprecated 

type HeadersHandler struct {
	// contains filtered or unexported fields
}

HeadersHandler PreRequest plugin that sets both prefill and encode disaggregation headers.

Deprecated: Use Handler instead, disagg-profile-handler now implements PreRequest natively.

func NewHeadersHandler deprecated 

func NewHeadersHandler(prefillProfile, encodeProfile string) *HeadersHandler

NewHeadersHandler initializes a new HeadersHandler and returns its pointer.

Deprecated: Use NewDisaggProfileHandler instead, disagg-profile-handler now implements PreRequest natively.

func (*HeadersHandler) PreRequest 

func (p *HeadersHandler) PreRequest(ctx context.Context, request *scheduling.InferenceRequest, schedulingResult *scheduling.SchedulingResult)

PreRequest wires prefill and encode SchedulerProfile results into headers to indicate disaggregation workers.

func (*HeadersHandler) TypedName 

func (p *HeadersHandler) TypedName() plugin.TypedName

TypedName returns the typed name of the plugin.

func (*HeadersHandler) WithName 

func (p *HeadersHandler) WithName(name string) *HeadersHandler

WithName sets the name of the plugin.

type PdProfileHandler deprecated 

type PdProfileHandler struct {
	// contains filtered or unexported fields
}

PdProfileHandler handles scheduler profiles for PD.

Deprecated: Use Handler instead.

func NewPdProfileHandler deprecated 

func NewPdProfileHandler(name string, parameters pdProfileHandlerParameters, deciderPlugin deciderPlugin) (*PdProfileHandler, error)

NewPdProfileHandler initializes a new PdProfileHandler and returns its pointer.

Deprecated: Use NewDisaggProfileHandler instead.

func (*PdProfileHandler) Consumes 

func (h *PdProfileHandler) Consumes() plugin.DataDependencies

Consumes defines data types consumed by this plugin (through the PD decider).

func (*PdProfileHandler) Pick 

func (h *PdProfileHandler) Pick(ctx context.Context, request *scheduling.InferenceRequest, profiles map[string]scheduling.SchedulerProfile,
	profileResults map[string]*scheduling.ProfileRunResult) map[string]scheduling.SchedulerProfile

Pick selects the SchedulingProfiles to run from the list of candidate profiles, while taking into consideration the request properties and the previously executed cycles along with their results.

func (*PdProfileHandler) ProcessResults 

func (h *PdProfileHandler) ProcessResults(_ context.Context, request *scheduling.InferenceRequest,
	profileResults map[string]*scheduling.ProfileRunResult) (*scheduling.SchedulingResult, error)

ProcessResults handles the outcome of the profile runs after the selected profiles ran. In case of an error in any of the profiles, the matching entry in the profileResults will contain nil, to indicate there was an error while running the profile.

func (*PdProfileHandler) TypedName 

func (h *PdProfileHandler) TypedName() plugin.TypedName

TypedName returns the typed name of the plugin.

func (*PdProfileHandler) WithName 

func (h *PdProfileHandler) WithName(name string) *PdProfileHandler

WithName sets the name of the plugin.

type PrefixBasedPDDecider 

type PrefixBasedPDDecider struct {
	// contains filtered or unexported fields
}

PrefixBasedPDDecider is a PD decider plugin which decision is based prefix aware

func NewPrefixBasedPDDecider 

func NewPrefixBasedPDDecider(config PrefixBasedPDDeciderConfig) (*PrefixBasedPDDecider, error)

NewPrefixBasedPDDecider initializes a NewPrefixBasedPDDecider prefix based PD decider Plugin and returns its pointer. If the configuration is invalid an error is returned.

func (*PrefixBasedPDDecider) TypedName 

func (d *PrefixBasedPDDecider) TypedName() plugin.TypedName

TypedName returns the typed name of the plugin.

func (*PrefixBasedPDDecider) WithName 

func (d *PrefixBasedPDDecider) WithName(name string) *PrefixBasedPDDecider

WithName sets the name of the plugin.

type PrefixBasedPDDeciderConfig 

type PrefixBasedPDDeciderConfig struct {
	// NonCachedTokens non cached minimum tokens that triggers disaggregated PD
	NonCachedTokens int `json:"nonCachedTokens"`
}

PrefixBasedPDDeciderConfig holds the configuration for the prefixBasedPDDecider plugin.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.