Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type AddressGetter ¶
type AddressGetter interface {
GetLLMAddress(modelID string) (string, error)
BlacklistLLMAddress(modelID, address string) error
}
AddressGetter gets an address of a model.
type ModelSyncer ¶
type ModelSyncer interface {
PullModel(ctx context.Context, modelID string) error
DeleteModel(ctx context.Context, modelID string) error
ListModels() []*v1.EngineStatus_Model
}
ModelSyncer syncs models.
type P ¶
type P struct {
// contains filtered or unexported fields
}
P processes tasks.
func NewP ¶
func NewP( engineID string, clientFactory processTasksClientFactory, addrGetter AddressGetter, modelSyncer ModelSyncer, logger logr.Logger, collector metrics.Collector, gracefulShutdownTimeout time.Duration, engineHeartbeatConfig config.EngineHeartbeatConfig, nimModels map[string]bool, ) *P
NewP returns a new processor.
func (*P) IsReady ¶
IsReady returns true if the processor is ready. If not, it returns a message describing why it is not ready.
func (*P) NeedLeaderElection ¶
NeedLeaderElection implements LeaderElectionRunnable
func (*P) SetupWithManager ¶
SetupWithManager sets up the processor with the manager.
func (*P) Start ¶
Start runs the processor.
TODO(kenji): Gracefully handle an error from the server.
func (*P) StartGracefulShutdown ¶ added in v1.39.0
func (p *P) StartGracefulShutdown()
StartGracefulShutdown starts graceful shutdown of the processor.
type ProcessTasksClient ¶ added in v1.25.0
type ProcessTasksClient interface {
ProcessTasks(ctx context.Context, opts ...grpc.CallOption) (v1.InferenceWorkerService_ProcessTasksClient, error)
}
ProcessTasksClient is a client for the ProcessTasks RPC.
Source Files
Click to show internal directories.
Click to hide internal directories.