Documentation
¶
Overview ¶
Package nvlink monitors the NVIDIA per-GPU nvlink devices.
Index ¶
- Constants
- func HasPostRxDetectFailure(line string) bool
- func Match(line string) (eventName string, message string)
- func New(gpudInstance *components.GPUdInstance) (components.Component, error)
- func SetDefaultExpectedLinkStates(states ExpectedLinkStates)
- type ExpectedLinkStates
- type NVLink
- type NVLinkState
- type NVLinkStates
Constants ¶
const ( // EventNamePostRxDetectFailure is emitted when the NVIDIA driver cannot // update an NVLink post-RX-detection link mask. EventNamePostRxDetectFailure = "nvlink_post_rx_detect_failure" // NVIDIA's published GH100 discovery path calls // knvlinkUpdatePostRxDetectLinkMask and reports the peer-mask failure when // that update fails. Some driver builds also emit the paired update failure. // The source does not classify this error as a driver hang. // ref. https://github.com/NVIDIA/open-gpu-kernel-modules/blob/452cec62d827034798072827d3866d1881662b77/src/nvidia/src/kernel/gpu/nvlink/arch/hopper/kernel_nvlink_gh100.c#L173-L249 RegexPostRxDetectFailureKMessage = `` /* 193-byte string literal not displayed */ )
const Name = "accelerator-nvidia-nvlink"
Name is the name of the NVIDIA NVLink component.
const SubSystem = "accelerator_nvidia_nvlink"
SubSystem is the Prometheus subsystem name for the NVIDIA NVLink component.
Variables ¶
This section is empty.
Functions ¶
func HasPostRxDetectFailure ¶ added in v0.12.10
HasPostRxDetectFailure returns true if the line reports an NVLink post-RX-detection failure.
func Match ¶ added in v0.12.10
Match returns the normalized event name and message for a matching kmsg line.
func New ¶
func New(gpudInstance *components.GPUdInstance) (components.Component, error)
New creates a NVIDIA NVLink component.
func SetDefaultExpectedLinkStates ¶ added in v0.9.0
func SetDefaultExpectedLinkStates(states ExpectedLinkStates)
SetDefaultExpectedLinkStates updates the process-wide default NVLink thresholds.
Types ¶
type ExpectedLinkStates ¶ added in v0.9.0
type ExpectedLinkStates struct {
// AtLeastGPUsWithAllLinksFeatureEnabled is the expected/minimum number of GPUs with all links feature enabled.
// This is useful to detect the following scenarios:
// - DeviceGetNvLinkState() returns SUCCESS (not ERROR_NOT_SUPPORTED)
// - Each link returns state != FEATURE_ENABLED (they're FEATURE_DISABLED or inactive)
// - Thus, safe to assume that NVLink is supported (no NOT_SUPPORTED error)
// - But ALL links are inactive/disabled
// - (e.g., nvidia-smi returns "Unable to retrieve NVLink information as all links are inActive")
// e.g., if set to 8 and one GPU has some nvlinks feature disabled, it will be considered as unhealthy.
AtLeastGPUsWithAllLinksFeatureEnabled int `json:"at_least_gpus_with_all_links_feature_enabled"`
}
ExpectedLinkStates defines the minimum healthy NVLink state across GPUs.
func GetDefaultExpectedLinkStates ¶ added in v0.9.0
func GetDefaultExpectedLinkStates() ExpectedLinkStates
GetDefaultExpectedLinkStates returns the process-wide default NVLink thresholds.
func (ExpectedLinkStates) IsZero ¶ added in v0.9.0
func (s ExpectedLinkStates) IsZero() bool
IsZero returns true if the expected link states are not set. Treats non-positive values as unset to prevent malformed configs from silently disabling NVLink health checks.
type NVLink ¶ added in v0.9.0
type NVLink struct {
// Represents the GPU UUID.
UUID string `json:"uuid"`
// BusID is the GPU bus ID from the nvml API.
// e.g., "0000:0f:00.0"
BusID string `json:"bus_id"`
// States is the list of nvlink states.
States NVLinkStates `json:"states"`
// Supported is true if the NVLink is supported by the device.
Supported bool `json:"supported"`
}
NVLink reports NVLink state for a single GPU.
type NVLinkState ¶ added in v0.9.0
type NVLinkState struct {
// Link is the nvlink link number.
Link int `json:"link"`
// FeatureEnabled is true if the nvlink feature is enabled.
FeatureEnabled bool `json:"feature_enabled"`
// ReplayErrors is the number of replay errors.
ReplayErrors uint64 `json:"replay_errors"`
// RecoveryErrors is the number of recovery errors.
RecoveryErrors uint64 `json:"recovery_errors"`
// CRCErrors is the number of crc errors.
CRCErrors uint64 `json:"crc_errors"`
// ThroughputRawTxBytes is the NVLink TX Data throughput + protocol overhead in bytes.
ThroughputRawTxBytes uint64 `json:"throughput_raw_tx_bytes"`
// ThroughputRawRxBytes is the NVLink RX Data throughput + protocol overhead in bytes.
ThroughputRawRxBytes uint64 `json:"throughput_raw_rx_bytes"`
}
NVLinkState reports the state and error counters for a single link.
type NVLinkStates ¶ added in v0.9.0
type NVLinkStates []NVLinkState
NVLinkStates is the collection of per-link NVLink states for a GPU.
func (NVLinkStates) AllFeatureEnabled ¶ added in v0.9.0
func (s NVLinkStates) AllFeatureEnabled() bool
AllFeatureEnabled reports whether every discovered link is enabled.
func (NVLinkStates) TotalCRCErrors ¶ added in v0.9.0
func (s NVLinkStates) TotalCRCErrors() uint64
TotalCRCErrors sums CRC errors across all links.
func (NVLinkStates) TotalRecoveryErrors ¶ added in v0.9.0
func (s NVLinkStates) TotalRecoveryErrors() uint64
TotalRecoveryErrors sums recovery errors across all links.
func (NVLinkStates) TotalReplayErrors ¶ added in v0.9.0
func (s NVLinkStates) TotalReplayErrors() uint64
TotalReplayErrors sums replay errors across all links.