requestcontrol

type Admitter ¶

type Admitter interface {
	plugin.Plugin
	// Admit returns the denial reason, wrapped as error if the request is denied.
	// If the request is allowed, it returns nil.
	Admit(ctx context.Context, request *fwksched.InferenceRequest, pods []fwksched.Endpoint) error
}

Admitter is called by the director after the data producer and before scheduling. When a request has to go through multiple Admitter, the request is admitted only if all plugins say that the request should be admitted.

type DataProducer ¶

type DataProducer interface {
	plugin.ProducerPlugin
	Produce(ctx context.Context, request *fwksched.InferenceRequest, pods []fwksched.Endpoint) error
}

DataProducer is implemented by data producers which produce data from different sources. Produce is called by the director before scheduling requests.

type PreAdmitter ¶

type PreAdmitter interface {
	plugin.Plugin
	PreAdmit(ctx context.Context, request *fwksched.InferenceRequest) error
}

PreAdmitter runs after InferenceRequest creation but before admission control. It can mutate InferenceRequest fields such as FairnessID and Headers.

type PreRequest ¶

type PreRequest interface {
	plugin.Plugin
	PreRequest(ctx context.Context, request *fwksched.InferenceRequest, schedulingResult *fwksched.SchedulingResult)
}

PreRequest is called by the director after a getting result from scheduling layer and before a request is sent to the selected model server.

type Response ¶

type Response struct {
	// RequestID is the Envoy generated Id for the request being processed
	RequestID string
	// Headers is a map of the response headers. Nil during body processing
	Headers map[string]string
	// StartOfStream when true indicates that this invocation contains the first chunk of the response
	StartOfStream bool
	// EndOfStream when true indicates that this invocation contains the last chunk of the response
	EndOfStream bool
	// ReqMetadata is a map of metadata that can be passed from Envoy.
	// It is populated with Envoy's dynamic metadata when ext_proc is processing ProcessingRequest_ResponseHeaders.
	// Currently, this is only used by conformance test.
	ReqMetadata map[string]any
	// Token usage counts parsed from the response body.
	Usage requesthandling.Usage
	// DynamicMetadata is a map of metadata that can be passed to the Envoy. It is populated into the dynamic
	// metadata when processing ProcessingResponse_RequestHeaders.
	DynamicMetadata *structpb.Struct
}

Response contains information from the response received to be passed to the Response requestcontrol plugins

type ResponseBodyProcessor ¶

type ResponseBodyProcessor interface {
	plugin.Plugin
	ResponseBody(ctx context.Context, request *fwksched.InferenceRequest, response *Response, targetEndpoint *datalayer.EndpointMetadata)
}

ResponseBodyProcessor is the primary hook for processing response data. It is called by the director for every data chunk in a streaming response, or exactly once for non-streaming responses.

Lifecycle & Termination:

• For streams: Invoked multiple times. The final call will have response.EndOfStream set to true.
• For non-streaming: Invoked once with response.EndOfStream set to true.
• Plugins must treat the call where response.EndOfStream == true as the final lifecycle hook to perform cleanup or final logging.

TODO(https://github.com/kubernetes-sigs/gateway-api-inference-extension/issues/2079): Update signature to pass error/termination state. This is a breaking change required for plugins to distinguish between success, errors, and disconnects.

type ResponseHeaderProcessor ¶

type ResponseHeaderProcessor interface {
	plugin.Plugin
	ResponseHeader(ctx context.Context, request *fwksched.InferenceRequest, response *Response, targetEndpoint *datalayer.EndpointMetadata)
}

ResponseHeaderProcessor is called by the director after the response headers are successfully received which indicates the beginning of the response handling by the model server. The given pod argument is the pod that served the request.

type TimeoutAwareProducer ¶

type TimeoutAwareProducer interface {
	ProduceTimeout() time.Duration
}

TimeoutAwareProducer is an optional interface a DataProducer may implement to declare its own execution timeout, overriding the default. A non-positive value selects the default.