Documentation
¶
Overview ¶
Package nodemonitor provides interfaces and implementations for detecting hardware failures on Kubernetes nodes.
Index ¶
- Constants
- type DetectionResult
- type NodeDiscoverer
- func (d *NodeDiscoverer) DiscoverNodesForJob(ctx context.Context, namespace, jobName string) ([]string, error)
- func (d *NodeDiscoverer) DiscoverPlacedNodesForJob(ctx context.Context, namespace, jobName string) ([]string, error)
- func (d *NodeDiscoverer) GetNode(ctx context.Context, name string) (*corev1.Node, error)
- func (d *NodeDiscoverer) GetNodes(ctx context.Context, names []string) ([]*corev1.Node, []error)
- type NodeFailureDetector
Constants ¶
const ( // NVCREJobLabel is the label used to identify pods belonging to an NVCRE Job. NVCREJobLabel = "nvcre.nvidia.com/job" // PodNVCREJobIndexField is the field index for pod lookups by NVCRE job label. // This enables efficient cache-based queries when discovering nodes for a Job. PodNVCREJobIndexField = "metadata.labels.nvcre.nvidia.com/job" )
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type DetectionResult ¶
type DetectionResult struct {
// Failed indicates whether the node has a hardware failure.
Failed bool
// NodeName is the name of the node that was evaluated.
NodeName string
// Reason is a short, machine-readable reason code for the failure.
Reason string
// Message provides human-readable details about the failure.
Message string
}
DetectionResult represents the outcome of evaluating a node for hardware failures.
type NodeDiscoverer ¶
type NodeDiscoverer struct {
// contains filtered or unexported fields
}
NodeDiscoverer finds nodes running pods for a given workload.
func NewNodeDiscoverer ¶
func NewNodeDiscoverer(c client.Client) *NodeDiscoverer
NewNodeDiscoverer creates a new node discoverer.
func (*NodeDiscoverer) DiscoverNodesForJob ¶
func (d *NodeDiscoverer) DiscoverNodesForJob(ctx context.Context, namespace, jobName string) ([]string, error)
DiscoverNodesForJob finds all nodes running pods associated with an NVCRE Job. It uses a field index on the nvcre.nvidia.com/job label for efficient cache-based lookups. Only returns nodes where pods are Running or Pending (i.e., scheduled).
Use this for questions about live health. For questions about where a job ran, including after its pods terminated, use DiscoverPlacedNodesForJob.
func (*NodeDiscoverer) DiscoverPlacedNodesForJob ¶ added in v0.7.0
func (d *NodeDiscoverer) DiscoverPlacedNodesForJob(ctx context.Context, namespace, jobName string) ([]string, error)
DiscoverPlacedNodesForJob finds every node a Job's pods were bound to, whatever phase those pods are now in.
This is the attribution question, not the health question, and the two need different filters. A job that fails fast leaves behind Failed pods, and a job whose pods have already been cleaned up to Succeeded leaves those; both still record the node they ran on in spec.nodeName. Filtering to Running or Pending the way DiscoverNodesForJob does would report no nodes in exactly the case attribution exists for, so a failed run would name no failed nodes.
A pod that was never bound has no placement to report and is skipped.
type NodeFailureDetector ¶
type NodeFailureDetector interface {
// Name returns the unique identifier for this detector type.
Name() string
// Detect evaluates the given node and returns whether it has a hardware failure.
// The context can be used for cancellation and timeouts.
Detect(ctx context.Context, node *corev1.Node) (DetectionResult, error)
// Validate checks if the detector configuration is valid.
// This should be called during setup to catch configuration errors early.
Validate() error
}
NodeFailureDetector is the interface for detecting hardware failures on nodes. Implementations can use different mechanisms such as CEL expressions, external APIs, or custom logic.
Source Files
Directories