Documentation
¶
Overview ¶
Package temperature tracks the NVIDIA per-GPU temperatures.
Index ¶
Constants ¶
const Name = "accelerator-nvidia-temperature"
Name is the ID of the NVIDIA temperature component.
const SubSystem = "accelerator_nvidia_temperature"
SubSystem is the Prometheus subsystem name for the NVIDIA temperature component.
const ThresholdCelsiusSlowdownMargin int32 = 0
ThresholdCelsiusSlowdownMargin is the default thermal margin threshold. A value of 0 effectively disables margin-based degraded detection, since only margins <= 0 will trigger the check.
For GB200/Blackwell GPUs, the slowdown threshold is approximately 87°C. A margin of 5°C corresponds to an alert at ~82°C. A margin of 10°C corresponds to an alert at ~77°C.
Variables ¶
This section is empty.
Functions ¶
func New ¶
func New(gpudInstance *components.GPUdInstance) (components.Component, error)
New creates a NVIDIA temperature component.
func SetDefaultMarginThreshold ¶ added in v0.10.0
func SetDefaultMarginThreshold(threshold Thresholds)
SetDefaultMarginThreshold sets the default margin threshold configuration.
Types ¶
type Temperature ¶ added in v0.9.0
type Temperature struct {
// Represents the GPU UUID.
UUID string `json:"uuid"`
// BusID is the GPU bus ID from the nvml API.
// e.g., "0000:0f:00.0"
BusID string `json:"bus_id"`
CurrentCelsiusGPUCore uint32 `json:"current_celsius_gpu_core"`
CurrentCelsiusHBM uint32 `json:"current_celsius_hbm"`
// HBMTemperatureSupported indicates whether NVML provided a memory temperature reading.
HBMTemperatureSupported bool `json:"hbm_temperature_supported"`
// ThresholdCelsiusSlowdownMargin is the thermal headroom (in °C) to the nearest
// slowdown threshold as defined by NVML. NVML does not specify whether the threshold
// is for GPU core or HBM; it is whichever slowdown threshold is nearest (driver-defined).
ThresholdCelsiusSlowdownMargin int32 `json:"threshold_celsius_slowdown_margin"`
// MarginTemperatureSupported indicates whether NVML provided a margin temperature reading.
MarginTemperatureSupported bool `json:"margin_temperature_supported"`
// Threshold at which the GPU starts to shut down to prevent hardware damage.
ThresholdCelsiusShutdown uint32 `json:"threshold_celsius_shutdown"`
// Threshold at which the GPU starts to throttle its performance.
ThresholdCelsiusSlowdown uint32 `json:"threshold_celsius_slowdown"`
// Maximum safe operating temperature for the GPU's memory.
ThresholdCelsiusMemMax uint32 `json:"threshold_celsius_mem_max"`
// Maximum safe operating temperature for the GPU core.
ThresholdCelsiusGPUMax uint32 `json:"threshold_celsius_gpu_max"`
UsedPercentShutdown string `json:"used_percent_shutdown"`
UsedPercentSlowdown string `json:"used_percent_slowdown"`
UsedPercentMemMax string `json:"used_percent_mem_max"`
UsedPercentGPUMax string `json:"used_percent_gpu_max"`
}
Temperature represents the NVIDIA temperature readings and thresholds for a device.
func GetTemperature ¶ added in v0.9.0
func GetTemperature(uuid string, dev device.Device) (Temperature, error)
GetTemperature returns the temperature readings and thresholds for a device.
func (Temperature) GetUsedPercentGPUMax ¶ added in v0.9.0
func (temp Temperature) GetUsedPercentGPUMax() (float64, error)
GetUsedPercentGPUMax returns the GPU max threshold usage percentage.
func (Temperature) GetUsedPercentMemMax ¶ added in v0.9.0
func (temp Temperature) GetUsedPercentMemMax() (float64, error)
GetUsedPercentMemMax returns the HBM memory max threshold usage percentage.
func (Temperature) GetUsedPercentShutdown ¶ added in v0.9.0
func (temp Temperature) GetUsedPercentShutdown() (float64, error)
GetUsedPercentShutdown returns the shutdown threshold usage percentage.
func (Temperature) GetUsedPercentSlowdown ¶ added in v0.9.0
func (temp Temperature) GetUsedPercentSlowdown() (float64, error)
GetUsedPercentSlowdown returns the slowdown threshold usage percentage.
type Thresholds ¶ added in v0.10.0
type Thresholds struct {
// CelsiusSlowdownMargin is the minimum thermal margin (°C) before marking the GPU as degraded.
CelsiusSlowdownMargin int32 `json:"celsius_slowdown_margin"`
}
Thresholds defines the temperature thresholds used for health evaluation.
func GetDefaultThresholds ¶ added in v0.10.0
func GetDefaultThresholds() Thresholds
GetDefaultThresholds returns the default temperature thresholds.