Documentation
¶
Overview ¶
Package gpumetrics collects NVIDIA GPU metrics via NVML and renders them as OTLP metrics for the metricexport egress path. It is a port of Polar Signals' standalone gpu-metrics-agent NVML producer, adapted to the metricexport.Producer interface.
Metrics follow the OTel hardware semantic conventions (hw.*) where those model what NVML reports, and use the nvidia.* namespace where they don't. Device metrics sit on the host's resource; per-process metrics sit on one resource per process, built by the same code as the process's profiles so both carry the same identity.
The NVML collectors (subpackage nvidia) require cgo and a dynamically linked binary so go-nvml can dlopen libnvidia-ml at runtime, so parca-agent only links them when built with the gpumetrics build tag. This package is pure Go.
Index ¶
- Constants
- func CountNVIDIAGPUs(procRoot string) int
- type Collector
- type DeclaredLookup
- type GPU
- type MetricBuffer
- type ProcessResourcer
- type Producer
- func (p *Producer) Collect(ctx context.Context) error
- func (p *Producer) Produce(host pmetric.MetricSlice, resources pmetric.ResourceMetricsSlice) error
- func (p *Producer) RecordHost(fn func(*MetricBuffer))
- func (p *Producer) RecordProcess(pid uint32, fn func(*MetricBuffer))
- func (p *Producer) SetProcessResourcer(r ProcessResourcer)
Constants ¶
const ( MetricGPUUtilization = "hw.gpu.utilization" MetricGPUMemoryUsage = "hw.gpu.memory.usage" MetricGPUMemoryLimit = "hw.gpu.memory.limit" MetricGPUMemoryUtilization = "hw.gpu.memory.utilization" MetricPower = "hw.power" MetricTemperature = "hw.temperature" MetricMemoryCopyUtilization = "nvidia.gpu.memory_copy.utilization" MetricPowerLimit = "nvidia.gpu.power.limit" MetricClockSpeed = "nvidia.gpu.clock.speed" MetricPCIeThroughput = "nvidia.gpu.pcie.throughput" MetricProcessUtilization = "nvidia.gpu.process.utilization" MetricProcessMemoryCopyUtilization = "nvidia.gpu.process.memory_copy.utilization" )
Metric names. hw.* follow the OTel hardware semantic conventions; nvidia.* cover what NVML reports that those conventions don't model.
const ( AttrHwGPUTask = "hw.gpu.task" AttrNetworkIODirection = "network.io.direction" AttrClockDomain = "nvidia.gpu.clock.domain" )
Attribute keys. The hw.* and network.* keys are semconv's.
const ScopeName = "parca.nvidia_gpu_metrics"
ScopeName is the OTLP instrumentation scope GPU metrics are reported under.
Variables ¶
This section is empty.
Functions ¶
func CountNVIDIAGPUs ¶
CountNVIDIAGPUs counts the GPUs the NVIDIA kernel driver has bound, one directory per GPU under <procRoot>/driver/nvidia/gpus. This needs neither NVML nor /dev/nvidia* in the container, only a loaded driver, which NVML requires anyway.
Types ¶
type DeclaredLookup ¶
type DeclaredLookup func(pid libpf.PID) (d processlabels.Declared, ok bool)
DeclaredLookup returns what pid declared about itself as the profiler captured it: its process context and the OTel environment variables read when the profiler synchronized the process. ok is false if the profiler does not track pid.
type GPU ¶
GPU identifies a device on the metrics about it.
func (GPU) PutIDAttrs ¶
PutIDAttrs sets just enough to say which GPU a per-process metric is about; the rest of the device's attributes are on its device metrics.
type MetricBuffer ¶
type MetricBuffer struct {
// contains filtered or unexported fields
}
MetricBuffer accumulates data points by metric name for one resource until the next export.
func (*MetricBuffer) Points ¶
func (b *MetricBuffer) Points(name string) pmetric.NumberDataPointSlice
Points returns name's data points, creating the metric from its spec on first use. name must be one of the Metric* constants.
type ProcessResourcer ¶
type ProcessResourcer interface {
// PutResource sets pid's resource attributes on attrs. It returns false
// if the process must not be reported, because a relabel rule dropped it.
PutResource(pid uint32, attrs pcommon.Map) bool
}
ProcessResourcer renders the OTLP resource attributes of a process using a GPU.
func NewProcessResourcer ¶
func NewProcessResourcer(ctx context.Context, resolver *processlabels.Resolver, declared DeclaredLookup, nodeName string, inKubernetes bool, ) ProcessResourcer
NewProcessResourcer returns a ProcessResourcer backed by resolver. declared may be nil, in which case the environment is read from procfs.
type Producer ¶
type Producer struct {
// contains filtered or unexported fields
}
Producer implements metricexport.Producer for GPU metrics. Its Collector records into per-resource buffers; Produce drains them.
func NewProducer ¶
NewProducer returns a Producer whose metrics come from c.
func (*Producer) Produce ¶
func (p *Producer) Produce(host pmetric.MetricSlice, resources pmetric.ResourceMetricsSlice) error
Produce moves the metrics collected so far into host, and appends one ResourceMetrics per process that used a GPU since the last call.
func (*Producer) RecordHost ¶
func (p *Producer) RecordHost(fn func(*MetricBuffer))
RecordHost appends to the host resource's metrics.
func (*Producer) RecordProcess ¶
func (p *Producer) RecordProcess(pid uint32, fn func(*MetricBuffer))
RecordProcess appends to the metrics of pid's resource.
func (*Producer) SetProcessResourcer ¶
func (p *Producer) SetProcessResourcer(r ProcessResourcer)
SetProcessResourcer sets how per-process metrics get their resource. Call before Collect starts. Without one, a process's resource carries only process.pid.
Source Files
Directories