Documentation
¶
Overview ¶
Package allocwatch turns Kubernetes GPU allocation into mock NVML memory readings.
Issue #506 asks that `memory.used_bytes` respond to allocation: a pod holding an nvidia.com/gpu claim should move the number, and the number should return when the pod goes away. Nothing in the deployment observed allocation, so every profile reported a constant zero and any dashboard built on the mock was visibly fake.
The write path already existed. The engine hot-reloads <driver_root>/config/overrides.yaml within one TTL, and that file sits beside the config the DaemonSet stages into every injected container. This package supplies the missing producer: it reads the node's current allocation from the kubelet pod-resources API and renders the memory half of that override document.
The design is deliberately level-triggered. Every poll recomputes all readings from the full current allocation set rather than applying deltas, so a missed or duplicated event cannot accumulate drift, and "returns when the pod is removed" needs no separate teardown path — it is what an empty claim list already computes.
What this does NOT claim: the byte count is synthetic. The mock runs no kernel and allocates no device memory, so the number reflects that a claim EXISTS, not what a workload actually touched. See the Metric Fidelity section of docs/configuration.md.
Index ¶
Constants ¶
const DefaultInterval = 2 * time.Second
DefaultInterval is how often the node's allocation is re-read. The engine re-reads the override file at most once per its own TTL (1s by default), so polling much faster only burns kubelet CPU without being observable.
const DefaultSocketPath = "/var/lib/kubelet/pod-resources/kubelet.sock"
DefaultSocketPath is where the kubelet serves the pod-resources API. The socket is node-local and read-only, so it needs a hostPath mount rather than any API-server RBAC.
const GPUResourceName = "nvidia.com/gpu"
GPUResourceName is the extended resource the NVIDIA device plugin advertises. pod-resources reports device IDs per resource name and a node commonly serves others (SR-IOV NICs, hugepages, other vendors' accelerators), so this name is the filter that keeps the watcher to GPUs.
Variables ¶
This section is empty.
Functions ¶
func OverrideDoc ¶
OverrideDoc renders readings as the `devices` map of a config override document, matching what `nvml-mock-ctl set --gpu N memory.used_bytes=...` writes.
Only used_bytes and free_bytes are emitted. total_bytes is a hardware fact that allocation does not change, and emitting any non-memory key would clobber a concurrent nvml-mock-ctl override of temperature, utilization or failure injection when the two documents merge.
func Publish ¶
Publish merges readings into the override document at path.
Read-modify-write under the same lock nvml-mock-ctl takes, so a human running `nvml-mock-ctl temp` while the watcher runs does not lose their edit and the watcher does not lose its reading. Only the memory block of each device is touched; everything else in the document is carried through untouched.
Types ¶
type Claim ¶
type Claim struct {
DeviceUUID string
// Pod, Namespace and Container identify the holder. They are unused by the
// memory reconciliation and carried for logging and for the `processes`
// surface tracked separately in #506 item 2.
Namespace string
Pod string
Container string
}
Claim is a single container's hold on one device. The same DeviceUUID appears once per holding container, so time-sliced GPUs shared by several containers produce several claims.
type Device ¶
Device is one GPU the mock exposes on this node, as resolved from the loaded profile. UUID is the identifier the device plugin advertises and therefore the key pod-resources reports claims against.
func DevicesFromConfig ¶
DevicesFromConfig resolves the GPU set the mock exposes from the loaded profile, keyed by the UUID the device plugin advertises — which is what pod-resources reports claims against.
type Lister ¶
Lister reads the node's current GPU allocation.
func NewPodResourcesLister ¶
NewPodResourcesLister dials the kubelet pod-resources socket.
Chosen over the NRI plugin because this API is level-triggered: one call returns the node's complete current allocation, so a reader that misses a window recovers on the next poll rather than carrying a stale delta forever. It also reports device-plugin, DRA and NRI allocations alike, and keeps working with nri.enabled=false.
The dial blocks so a missing or unmountable socket fails at startup. The alternative — a lazy connection — would leave the watcher reporting an idle node indefinitely, which is indistinguishable from a genuinely idle node.
type Policy ¶
type Policy struct {
// UsedFractionPerClaim is the share of the usable aperture
// (total - reserved) attributed to each claim. Claims on one device add up
// and clamp at the aperture.
UsedFractionPerClaim float64
}
Policy tunes how much memory a single claim is reported to consume.
func DefaultPolicy ¶
func DefaultPolicy() Policy
DefaultPolicy attributes half the usable aperture per claim, so a single exclusive claim reads as a substantial but not saturated card, and two time-sliced containers fill it. The value is arbitrary in the sense that no real allocation backs it; it is chosen to look plausible on a dashboard and to leave headroom that a second claim visibly consumes.
type Reading ¶
Reading is the reconciled memory state of one device.
func Reconcile ¶
Reconcile computes a Reading for every device from the full current claim set. Devices with no claim are included and report the idle values, so the emitted override always covers the whole node: writing only the busy devices would leave a previously busy GPU pinned at its last value once its pod went away.
type Watcher ¶
type Watcher struct {
Lister Lister
Devices []Device
Policy Policy
OverridePath string
Interval time.Duration
}
Watcher polls the node's GPU allocation and republishes the memory readings.
func (*Watcher) Run ¶
Run polls until ctx is cancelled. Cancellation is a clean shutdown, not an error.
A failed poll is logged and skipped rather than fatal, and publishes nothing. Both halves of that matter: exiting would freeze every GPU at its last reading for the life of the pod, and publishing zeros on a failed read would report the whole node idle every time the kubelet is briefly unreachable.
Source Files