README
¶
DataParallel Profile Handler
Type: data-parallel-profile-handler
Deprecated: Use
single-profile-handlerwith Istio >= 1.28.1 instead. See single/.
Provides a profile handler for data-parallel inference routing, where a request is scheduled to one pod among multiple replicas serving the same model. Injects the X-Data-Parallel-Endpoint header pointing to the selected pod and rewrites the target port to primaryPort.
Constraints:
- Requires exactly one scheduling profile in the config.
Parameters:
primaryPort(int, optional, default:8000): Primary service port (1–65535).
Configuration Example:
plugins:
- type: data-parallel-profile-handler
name: dp-handler
parameters:
primaryPort: 8000
Migration: Replace with single-profile-handler (requires Istio >= 1.28.1):
Before:
plugins:
- type: data-parallel-profile-handler
parameters:
primaryPort: 8000
After:
plugins:
- type: single-profile-handler
Related Documentation
Documentation
¶
Overview ¶
Package dataparallel provides a data-parallel profile handler plugin for the epp.
Index ¶
- Constants
- func ProfileHandlerFactory(name string, rawParameters *json.Decoder, handle plugin.Handle) (plugin.Plugin, error)
- type ProfileHandler
- func (h *ProfileHandler) Pick(ctx context.Context, _ *scheduling.InferenceRequest, ...) map[string]scheduling.SchedulerProfile
- func (h *ProfileHandler) ProcessResults(_ context.Context, request *scheduling.InferenceRequest, ...) (*scheduling.SchedulingResult, error)
- func (h *ProfileHandler) TypedName() plugin.TypedName
- func (h *ProfileHandler) WithName(name string) *ProfileHandler
Constants ¶
const (
// DataParallelProfileHandlerType is the type of the ProfileHandler
DataParallelProfileHandlerType = "data-parallel-profile-handler"
)
Variables ¶
This section is empty.
Functions ¶
Types ¶
type ProfileHandler ¶
type ProfileHandler struct {
// contains filtered or unexported fields
}
ProfileHandler handles scheduler profiles for Data Parallel.
func NewProfileHandler ¶
func NewProfileHandler(primaryPort int) *ProfileHandler
NewProfileHandler initializes a new PdProfileHandler and returns its pointer.
func (*ProfileHandler) Pick ¶
func (h *ProfileHandler) Pick(ctx context.Context, _ *scheduling.InferenceRequest, profiles map[string]scheduling.SchedulerProfile, profileResults map[string]*scheduling.ProfileRunResult) map[string]scheduling.SchedulerProfile
Pick selects the SchedulingProfiles to run from the list of candidate profiles, while taking into consideration the request properties and the previously executed cycles along with their results.
func (*ProfileHandler) ProcessResults ¶
func (h *ProfileHandler) ProcessResults(_ context.Context, request *scheduling.InferenceRequest, profileResults map[string]*scheduling.ProfileRunResult) (*scheduling.SchedulingResult, error)
ProcessResults handles the outcome of the profile runs after all profiles ran. It may aggregate results, log test profile outputs, or apply custom logic. It specifies in the SchedulingResult the key of the primary profile that should be used to get the request selected destination. When a profile run fails, its result in the profileResults map is nil.
func (*ProfileHandler) TypedName ¶
func (h *ProfileHandler) TypedName() plugin.TypedName
TypedName returns the typed name of the plugin.
func (*ProfileHandler) WithName ¶
func (h *ProfileHandler) WithName(name string) *ProfileHandler
WithName sets the name of the plugin.
Source Files