nvlink

package
v0.12.13 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 21, 2026 License: Apache-2.0 Imports: 26 Imported by: 0

Documentation

Overview

Package nvlink monitors the NVIDIA per-GPU nvlink devices.

Index

Constants

View Source
const (
	// EventNamePostRxDetectFailure is emitted when the NVIDIA driver cannot
	// update an NVLink post-RX-detection link mask.
	EventNamePostRxDetectFailure = "nvlink_post_rx_detect_failure"

	// NVIDIA's published GH100 discovery path calls
	// knvlinkUpdatePostRxDetectLinkMask and reports the peer-mask failure when
	// that update fails. Some driver builds also emit the paired update failure.
	// The source does not classify this error as a driver hang.
	// ref. https://github.com/NVIDIA/open-gpu-kernel-modules/blob/452cec62d827034798072827d3866d1881662b77/src/nvidia/src/kernel/gpu/nvlink/arch/hopper/kernel_nvlink_gh100.c#L173-L249
	RegexPostRxDetectFailureKMessage = `` /* 193-byte string literal not displayed */

)
View Source
const Name = "accelerator-nvidia-nvlink"

Name is the name of the NVIDIA NVLink component.

View Source
const SubSystem = "accelerator_nvidia_nvlink"

SubSystem is the Prometheus subsystem name for the NVIDIA NVLink component.

Variables

This section is empty.

Functions

func HasPostRxDetectFailure added in v0.12.10

func HasPostRxDetectFailure(line string) bool

HasPostRxDetectFailure returns true if the line reports an NVLink post-RX-detection failure.

func Match added in v0.12.10

func Match(line string) (eventName string, message string)

Match returns the normalized event name and message for a matching kmsg line.

func New

func New(gpudInstance *components.GPUdInstance) (components.Component, error)

New creates a NVIDIA NVLink component.

func SetDefaultExpectedLinkStates added in v0.9.0

func SetDefaultExpectedLinkStates(states ExpectedLinkStates)

SetDefaultExpectedLinkStates updates the process-wide default NVLink thresholds.

Types

type ExpectedLinkStates added in v0.9.0

type ExpectedLinkStates struct {
	// AtLeastGPUsWithAllLinksFeatureEnabled is the expected/minimum number of GPUs with all links feature enabled.
	// This is useful to detect the following scenarios:
	// - DeviceGetNvLinkState() returns SUCCESS (not ERROR_NOT_SUPPORTED)
	// - Each link returns state != FEATURE_ENABLED (they're FEATURE_DISABLED or inactive)
	// - Thus, safe to assume that NVLink is supported (no NOT_SUPPORTED error)
	// - But ALL links are inactive/disabled
	// - (e.g., nvidia-smi returns "Unable to retrieve NVLink information as all links are inActive")
	// e.g., if set to 8 and one GPU has some nvlinks feature disabled, it will be considered as unhealthy.
	AtLeastGPUsWithAllLinksFeatureEnabled int `json:"at_least_gpus_with_all_links_feature_enabled"`
}

ExpectedLinkStates defines the minimum healthy NVLink state across GPUs.

func GetDefaultExpectedLinkStates added in v0.9.0

func GetDefaultExpectedLinkStates() ExpectedLinkStates

GetDefaultExpectedLinkStates returns the process-wide default NVLink thresholds.

func (ExpectedLinkStates) IsZero added in v0.9.0

func (s ExpectedLinkStates) IsZero() bool

IsZero returns true if the expected link states are not set. Treats non-positive values as unset to prevent malformed configs from silently disabling NVLink health checks.

type NVLink struct {
	// Represents the GPU UUID.
	UUID string `json:"uuid"`

	// BusID is the GPU bus ID from the nvml API.
	//  e.g., "0000:0f:00.0"
	BusID string `json:"bus_id"`

	// States is the list of nvlink states.
	States NVLinkStates `json:"states"`

	// Supported is true if the NVLink is supported by the device.
	Supported bool `json:"supported"`
}

NVLink reports NVLink state for a single GPU.

func GetNVLink(uuid string, dev device.Device) (NVLink, error)

GetNVLink queries the NVLink information for a device.

type NVLinkState added in v0.9.0

type NVLinkState struct {
	// Link is the nvlink link number.
	Link int `json:"link"`

	// FeatureEnabled is true if the nvlink feature is enabled.
	FeatureEnabled bool `json:"feature_enabled"`
	// ReplayErrors is the number of replay errors.
	ReplayErrors uint64 `json:"replay_errors"`
	// RecoveryErrors is the number of recovery errors.
	RecoveryErrors uint64 `json:"recovery_errors"`
	// CRCErrors is the number of crc errors.
	CRCErrors uint64 `json:"crc_errors"`

	// ThroughputRawTxBytes is the NVLink TX Data throughput + protocol overhead in bytes.
	ThroughputRawTxBytes uint64 `json:"throughput_raw_tx_bytes"`
	// ThroughputRawRxBytes is the NVLink RX Data throughput + protocol overhead in bytes.
	ThroughputRawRxBytes uint64 `json:"throughput_raw_rx_bytes"`
}

NVLinkState reports the state and error counters for a single link.

type NVLinkStates added in v0.9.0

type NVLinkStates []NVLinkState

NVLinkStates is the collection of per-link NVLink states for a GPU.

func (NVLinkStates) AllFeatureEnabled added in v0.9.0

func (s NVLinkStates) AllFeatureEnabled() bool

AllFeatureEnabled reports whether every discovered link is enabled.

func (NVLinkStates) TotalCRCErrors added in v0.9.0

func (s NVLinkStates) TotalCRCErrors() uint64

TotalCRCErrors sums CRC errors across all links.

func (NVLinkStates) TotalRecoveryErrors added in v0.9.0

func (s NVLinkStates) TotalRecoveryErrors() uint64

TotalRecoveryErrors sums recovery errors across all links.

func (NVLinkStates) TotalReplayErrors added in v0.9.0

func (s NVLinkStates) TotalReplayErrors() uint64

TotalReplayErrors sums replay errors across all links.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL