README
¶
Mokka
Simulate your GPU infrastructure on CPU nodes.
Mokka simulates the software contracts around NVIDIA devices rather than the devices themselves. The rule of thumb: reach for Mokka when your system reads hardware state and reacts to it, and for real hardware when it executes work, moves data, or measures performance.
That makes it a good fit for:
- Kubernetes discovery and allocation through the device plugin or DRA — scheduling, ResourceClaims, CDI visibility, topology attributes.
- Software that consumes NVML,
nvidia-smi, DCGM or DCGM Exporter. - Monitoring dashboards, parsers, alerting, and remediation logic that cordons, drains, reschedules and recovers.
- Repeatable fault injection — device loss, Xid errors, ECC errors, temperature, power, utilisation, clocks, and GPU-side NVLink errors.
- Code that interprets declared PCI, NUMA, NVLink, fabric UUID or clique topology.
- IMEX peer readiness and liveness over the pod network.
Use real hardware for CUDA execution, NCCL, GPUDirect and RDMA data paths, any throughput or thermal measurement, driver and firmware lifecycle, switch management planes, physically faithful fault timing, MIG partition lifecycle, and Confidential Computing.
Quick start
kind create cluster --name mokka
helm install nvml-mock oci://ghcr.io/nvidia/k8s-test-infra/chart/nvml-mock \
--namespace mokka --create-namespace
Every node now reports four mock GB300 GPUs. Swap in a100, b200, gb200,
h100, l40s or t4 with --set gpu.profile=<name>.
Simulation depth by area breaks this down per surface — GPU and NVML, Kubernetes allocation, metrics, PCI and NUMA, NVLink, NVSwitch and Fabric Manager, and InfiniBand — with what each one does and does not prove.
Tested consumers
| Consumer | What works |
|---|---|
| Node Feature Discovery | PCI vendor labels derived from the feature file Mokka writes |
| GPU Feature Discovery | Node labels derived from NVML |
| NVIDIA Device Plugin | Allocatable nvidia.com/gpu matches the profile, and workloads schedule against it |
| NVIDIA DRA Driver | ResourceSlices report the right GPUs, and a ResourceClaimTemplate pod reaches Running |
| NVIDIA GPU Operator | The full operand stack installs and its validator starts |
| DCGM / dcgm-exporter | Telemetry, time-varying power, and injected Xid errors |
| Run:ai fake-gpu-operator | Profile ConfigMaps published in the shape its discovery expects |
Documentation
Visit our documentation website to find all details about how Mokka works.
| Quick Start | Install and see simulated GPUs |
| Architecture | The moving parts and how the system behaves |
| Guides | Device plugin, DRA, GPU Operator, failure injection, node-wide injection |
| Configuration | Every profile knob |
| FAQ | What Mokka simulates, and what it does not |
Contributing
See the contributing guide for local development with Tilt, the test suites, and how to submit a change. Substantial changes start with a Mokka Enhancement Proposal.
Report vulnerabilities privately — see SECURITY.md, not the issue tracker.
Credits
- Logo designed by Roman Hlushko with the assistance of OpenAI's ChatGPT.
License
Apache License 2.0 — see LICENSE.
Directories
¶
| Path | Synopsis |
|---|---|
|
api
|
|
|
v1alpha1
Package v1alpha1 contains API types for the mokka.nvidia.com group.
|
Package v1alpha1 contains API types for the mokka.nvidia.com group. |
|
cmd
|
|
|
check-fabric
command
check-fabric prints the NVLink fabric attributes of every visible GPU by calling nvmlDeviceGetGpuFabricInfo / nvmlDeviceGetGpuFabricInfoV through go-nvml.
|
check-fabric prints the NVLink fabric attributes of every visible GPU by calling nvmlDeviceGetGpuFabricInfo / nvmlDeviceGetGpuFabricInfoV through go-nvml. |
|
control-plane
command
control-plane serves probes and reconciles declarative sGPU inventory.
|
control-plane serves probes and reconciles declarative sGPU inventory. |
|
generate-bridge
command
Package main provides a stub generator for the mock NVML bridge.
|
Package main provides a stub generator for the mock NVML bridge. |
|
node-agent
command
Package main is the entry point for the node-agent binary.
|
Package main is the entry point for the node-agent binary. |
|
nri-plugin
command
nri-plugin injects the nvml-mock overlay into containers through containerd's Node Resource Interface, so a workload sees mock GPUs without being modified.
|
nri-plugin injects the nvml-mock overlay into containers through containerd's Node Resource Interface, so a workload sees mock GPUs without being modified. |
|
nvml-mock-ctl
command
nvml-mock-ctl mutates the nvml-mock runtime config override so a running node's simulated GPU state can be changed without a Helm upgrade or pod restart.
|
nvml-mock-ctl mutates the nvml-mock runtime config override so a running node's simulated GPU state can be changed without a Helm upgrade or pod restart. |
|
hack
|
|
|
tools/imex-archives
command
Command imex-archives places the IMEX archives pinned in internal/agent/imex/imex.lock.json in a local directory, laid out as the build context of deployments/imex-binaries/Dockerfile:
|
Command imex-archives places the IMEX archives pinned in internal/agent/imex/imex.lock.json in a local directory, laid out as the build context of deployments/imex-binaries/Dockerfile: |
|
internal
|
|
|
agent
Package agent implements the Mokka Node Agent: a level-triggered reconciler and supervisor that compiles desired GPU simulation state and fans it out to per-component Simulators in parallel.
|
Package agent implements the Mokka Node Agent: a level-triggered reconciler and supervisor that compiles desired GPU simulation state and fans it out to per-component Simulators in parallel. |
|
agent/cpulocality
Package cpulocality decides which host CPUs are local to a PCI root complex.
|
Package cpulocality decides which host CPUs are local to a PCI root complex. |
|
agent/cri
Package cri implements the container runtime simulator: it owns what the node's container runtime needs from Mokka to inject mock GPU devices into workload containers.
|
Package cri implements the container runtime simulator: it owns what the node's container runtime needs from Mokka to inject mock GPU devices into workload containers. |
|
agent/fabricmanager
Package fabricmanager simulates nv-fabricmanager, the service that registers GPUs with the NVSwitches on an NVSwitch platform before they are usable.
|
Package fabricmanager simulates nv-fabricmanager, the service that registers GPUs with the NVSwitches on an NVSwitch platform before they are usable. |
|
agent/gpudriver
Package gpudriver implements the GPU driver footprint simulator: chardevs, NVML/CUDA shims, nvidia-smi, procfs entries, engine config, and a bind mount of that tree onto /run/nvidia/driver, the path GPU Operator's own operands default to.
|
Package gpudriver implements the GPU driver footprint simulator: chardevs, NVML/CUDA shims, nvidia-smi, procfs entries, engine config, and a bind mount of that tree onto /run/nvidia/driver, the path GPU Operator's own operands default to. |
|
agent/host
Package host abstracts on-host filesystem roots so simulators can retarget to t.TempDir() in tests without mounting a real /host.
|
Package host abstracts on-host filesystem roots so simulators can retarget to t.TempDir() in tests without mounting a real /host. |
|
agent/ib
Package ib implements the InfiniBand HCA simulator: a fake /sys/class/infiniband tree, the LD_PRELOAD shims and IB CLI tools that read it, and the mock-ib daemon serving UMAD and verbs over a Unix socket.
|
Package ib implements the InfiniBand HCA simulator: a fake /sys/class/infiniband tree, the LD_PRELOAD shims and IB CLI tools that read it, and the mock-ib daemon serving UMAD and verbs over a Unix socket. |
|
agent/imex
Package imex implements the IMEX simulator: the capability surface the NVIDIA kernel module provides (channel character devices, a /proc/devices overlay and the fabric-imex-mgmt capability file), read by the NVIDIA DRA compute-domain kubelet plugin, and the IMEX node software (nvidia-imex and nvidia-imex-ctl) its ComputeDomain daemon runs.
|
Package imex implements the IMEX simulator: the capability surface the NVIDIA kernel module provides (channel character devices, a /proc/devices overlay and the fabric-imex-mgmt capability file), read by the NVIDIA DRA compute-domain kubelet plugin, and the IMEX node software (nvidia-imex and nvidia-imex-ctl) its ComputeDomain daemon runs. |
|
agent/kernellog
Package kernellog simulates the kernel side of an Xid: the printk a real driver emits when a GPU faults.
|
Package kernellog simulates the kernel side of an Xid: the printk a real driver emits when a GPU faults. |
|
agent/migcaps
Package migcaps implements the MIG capability surface simulator: the /dev/nvidia-caps character devices, the capabilities/gpu*/mig procfs tree, and the mig-minors table that maps between them.
|
Package migcaps implements the MIG capability surface simulator: the /dev/nvidia-caps character devices, the capabilities/gpu*/mig procfs tree, and the mig-minors table that maps between them. |
|
agent/nvlink
Package nvlink stages the ComputeDomain topology document, which gives each node its NVLink fabric identity.
|
Package nvlink stages the ComputeDomain topology document, which gives each node its NVLink fabric identity. |
|
agent/pcibus
Package pcibus implements the PCI bus simulator: it renders a fake /sys/bus/pci/devices tree and stages libmockfs.so so that lspci and topology-aware schedulers see mock GPU BDFs.
|
Package pcibus implements the PCI bus simulator: it renders a fake /sys/bus/pci/devices tree and stages libmockfs.so so that lspci and topology-aware schedulers see mock GPU BDFs. |
|
agent/source
Package source provides StateSource implementations for the Mokka Node Agent.
|
Package source provides StateSource implementations for the Mokka Node Agent. |
|
controlplane
Package controlplane wires the Mokka HTTP and controller processes.
|
Package controlplane wires the Mokka HTTP and controller processes. |
|
controlplane/controller
Package controller wires informer snapshots and keyed workqueues to the rack, projection, and status reconcilers, and compiles effective runtime state from runtime policies.
|
Package controller wires informer snapshots and keyed workqueues to the rack, projection, and status reconcilers, and compiles effective runtime state from runtime policies. |
|
cpulist
Package cpulist reads and writes the kernel's cpulist format ("0-3,8,10-11"), used by sysfs files such as /sys/devices/system/cpu/online and a PCI device's local_cpulist.
|
Package cpulist reads and writes the kernel's cpulist format ("0-3,8,10-11"), used by sysfs files such as /sys/devices/system/cpu/online and a PCI device's local_cpulist. |
|
cri
Package cri sets up a node's container runtime for Mokka's CDI specs, as MEP-0006 describes: it installs the NVIDIA Container Toolkit's runtime and hook on the node, and has the toolkit's nvidia-ctk register the nvidia handler with the container runtime.
|
Package cri sets up a node's container runtime for Mokka's CDI specs, as MEP-0006 describes: it installs the NVIDIA Container Toolkit's runtime and hook on the node, and has the toolkit's nvidia-ctk register the nvidia handler with the container runtime. |
|
cri/containerd
Package containerd registers Mokka's runtime handler with containerd: the toolkit's own nvidia-ctk writes Mokka's config file into containerd's config dir, Mokka adds NRI to it, and systemd restarts containerd to load it.
|
Package containerd registers Mokka's runtime handler with containerd: the toolkit's own nvidia-ctk writes Mokka's config file into containerd's config dir, Mokka adds NRI to it, and systemd restarts containerd to load it. |
|
fabricmanager
Package fabricmanager defines the marker-file protocol that stands in for NVSwitch fabric registration.
|
Package fabricmanager defines the marker-file protocol that stands in for NVSwitch fabric registration. |
|
features
Package features declares Mokka's feature gates on top of the Kubernetes component-base feature gate, so operators use the same syntax and lifecycle they know from Kubernetes components.
|
Package features declares Mokka's feature gates on top of the Kubernetes component-base feature gate, so operators use the same syntax and lifecycle they know from Kubernetes components. |
|
fsutil
Package fsutil holds the filesystem mutations the mock trees are staged with.
|
Package fsutil holds the filesystem mutations the mock trees are staged with. |
|
gpuarch
Package gpuarch orders NVIDIA GPU architecture generations so a feature gate can be written as the question it is asking — "Hopper and newer", "before Ampere", "between Ampere and Hopper" — rather than as an integer comparison re-derived at each call site.
|
Package gpuarch orders NVIDIA GPU architecture generations so a feature gate can be written as the question it is asking — "Hopper and newer", "before Ampere", "between Ampere and Hopper" — rather than as an integer comparison re-derived at each call site. |
|
health
Package health serves the /healthz, /readyz, and /stagedz probes shared by Mokka binaries.
|
Package health serves the /healthz, /readyz, and /stagedz probes shared by Mokka binaries. |
|
ib/config
Package config defines the InfiniBand block embedded in mock-nvml profile YAML.
|
Package config defines the InfiniBand block embedded in mock-nvml profile YAML. |
|
ib/daemon
Package daemon serves the InfiniBand control plane the LD_PRELOAD shims talk to: UMAD and verbs over a Unix socket, plus a TCP relay that exchanges port registrations with peer pods so diagnostics can see a multi-node fabric.
|
Package daemon serves the InfiniBand control plane the LD_PRELOAD shims talk to: UMAD and verbs over a Unix socket, plus a TCP relay that exchanges port registrations with peer pods so diagnostics can see a multi-node fabric. |
|
ib/daemon/madtest
Package madtest provides shared MAD/UMAD fixtures for mock-ib daemon tests.
|
Package madtest provides shared MAD/UMAD fixtures for mock-ib daemon tests. |
|
ib/fabric
Package fabric models the mock InfiniBand subnet for SMP synthesis.
|
Package fabric models the mock InfiniBand subnet for SMP synthesis. |
|
ib/gid
Package gid provides InfiniBand GID / port-GUID string helpers shared by sysfs scanning and the mock-ib daemon.
|
Package gid provides InfiniBand GID / port-GUID string helpers shared by sysfs scanning and the mock-ib daemon. |
|
ib/protocol
Package protocol defines length-prefixed JSON framing for mock-ib traffic on Unix sockets (libibmockumad shim) and TCP (cross-node fabric).
|
Package protocol defines length-prefixed JSON framing for mock-ib traffic on Unix sockets (libibmockumad shim) and TCP (cross-node fabric). |
|
ib/registry
Package registry maps InfiniBand port GUIDs to peers for mock-ib fabric routing.
|
Package registry maps InfiniBand port GUIDs to peers for mock-ib fabric routing. |
|
ib/subnet
Package subnet synthesizes subnet management MAD responses for iblinkinfo.
|
Package subnet synthesizes subnet management MAD responses for iblinkinfo. |
|
ib/sysfs
Package sysfs owns both halves of the fake InfiniBand sysfs tree: Render writes it from a config.Infiniband spec, Scan reads it back.
|
Package sysfs owns both halves of the fake InfiniBand sysfs tree: Render writes it from a config.Infiniband spec, Scan reads it back. |
|
imex
Package imex generates the mock IMEX capability surface that the NVIDIA DRA driver's compute-domain kubelet plugin needs on a node with no NVIDIA kernel driver.
|
Package imex generates the mock IMEX capability surface that the NVIDIA DRA driver's compute-domain kubelet plugin needs on a node with no NVIDIA kernel driver. |
|
kmod
Package kmod stages mock kernel module state, and mirrors the node's own modules beside it.
|
Package kmod stages mock kernel module state, and mirrors the node's own modules beside it. |
|
logging
Package logging provides a shared logger constructor for all Mokka binaries.
|
Package logging provides a shared logger constructor for all Mokka binaries. |
|
migcaps
Package migcaps renders the MIG capability tables a MIG-enabled NVIDIA driver publishes, for a node that has no kernel driver at all.
|
Package migcaps renders the MIG capability tables a MIG-enabled NVIDIA driver publishes, for a node that has no kernel driver at all. |
|
nfd
Package nfd writes feature files for the local source of Node Feature Discovery (NFD).
|
Package nfd writes feature files for the local source of Node Feature Discovery (NFD). |
|
nri
Package nri is the containerd Node Resource Interface plugin that injects the nvml-mock overlay into containers as they are created.
|
Package nri is the containerd Node Resource Interface plugin that injects the nvml-mock overlay into containers as they are created. |
|
nri/inject
Package inject decides what the nvml-mock overlay adds to a container.
|
Package inject decides what the nvml-mock overlay adds to a container. |
|
pcisysfs
Package pcisysfs defines the schema for the PCI sysfs tree and renders it as a fake /sys/bus/pci/devices + /sys/devices/pciDDDD:BB tree under a configurable root directory.
|
Package pcisysfs defines the schema for the PCI sysfs tree and renders it as a fake /sys/bus/pci/devices + /sys/devices/pciDDDD:BB tree under a configurable root directory. |
|
sgpu/allocate
Package allocate plans stable Kubernetes-Node-to-logical-Node bindings from immutable snapshots.
|
Package allocate plans stable Kubernetes-Node-to-logical-Node bindings from immutable snapshots. |
|
sgpu/assignment
Package assignment owns the versioned Node assignment annotation value.
|
Package assignment owns the versioned Node assignment annotation value. |
|
sgpu/inventory
Package inventory reconciles inventories into durable rack resources using informer snapshots.
|
Package inventory reconciles inventories into durable rack resources using informer snapshots. |
|
sgpu/metadata
Package metadata defines the Kubernetes metadata surface owned by Mokka.
|
Package metadata defines the Kubernetes metadata surface owned by Mokka. |
|
sgpu/nodecatalog
Package nodecatalog maintains the controller's rebuildable, allocation- oriented view of eligible Kubernetes Nodes.
|
Package nodecatalog maintains the controller's rebuildable, allocation- oriented view of eligible Kubernetes Nodes. |
|
sgpu/policy
Package policy decides which SGPURuntimePolicies apply to an inventory and compiles the effective runtime state of its simulated GPUs.
|
Package policy decides which SGPURuntimePolicies apply to an inventory and compiles the effective runtime state of its simulated GPUs. |
|
sgpu/projection
Package projection derives the small controller-owned Node metadata surface from durable SGPURack bindings.
|
Package projection derives the small controller-owned Node metadata surface from durable SGPURack bindings. |
|
sgpu/rackrender
Package rackrender renders immutable rack topology and identities from API inputs without Kubernetes clients, mutable global state, or time.
|
Package rackrender renders immutable rack topology and identities from API inputs without Kubernetes clients, mutable global state, or time. |
|
sgpu/release
Package release defines why a durable binding is released and the Node projection cleanup that must be acknowledged before the release completes.
|
Package release defines why a durable binding is released and the Node projection cleanup that must be acknowledged before the release completes. |
|
sgpu/status
Package status computes aggregate inventory and rack status, and runtime policy status, from cached controller snapshots and writes only semantic changes.
|
Package status computes aggregate inventory and rack status, and runtime policy status, from cached controller snapshots and writes only semantic changes. |
|
staginggate
Package staginggate coordinates node-agent staging with NRI adjustments.
|
Package staginggate coordinates node-agent staging with NRI adjustments. |
|
version
Package version carries build-time-injected identity strings.
|
Package version carries build-time-injected identity strings. |
|
pkg
|
|
|
diagnostics
Package diagnostics collects failure diagnostics from an e2e cluster (pod logs, kubectl describes, DaemonSet state) into an artifact directory when a spec fails.
|
Package diagnostics collects failure diagnostics from an e2e cluster (pod logs, kubectl describes, DaemonSet state) into an artifact directory when a spec fails. |
|
generated/clientset/versioned/fake
This package has the automatically generated fake clientset.
|
This package has the automatically generated fake clientset. |
|
generated/clientset/versioned/scheme
This package contains the scheme of the automatically generated clientset.
|
This package contains the scheme of the automatically generated clientset. |
|
generated/clientset/versioned/typed/api/v1alpha1
This package has the automatically generated typed clients.
|
This package has the automatically generated typed clients. |
|
generated/clientset/versioned/typed/api/v1alpha1/fake
Package fake has the automatically generated clients.
|
Package fake has the automatically generated clients. |
|
gpu/allocwatch
Package allocwatch turns Kubernetes GPU allocation into mock NVML memory readings.
|
Package allocwatch turns Kubernetes GPU allocation into mock NVML memory readings. |
|
gpu/mockctl
Package mockctl holds the pure config override-mutation logic used by the nvml-mock-ctl CLI.
|
Package mockctl holds the pure config override-mutation logic used by the nvml-mock-ctl CLI. |
|
gpu/mocknvml/bridge
command
Package main provides the NVML per-process accounting entry points.
|
Package main provides the NVML per-process accounting entry points. |
|
gpu/mocknvml/engine
Package engine is the pure-Go mock NVML runtime driving the mocknvml bridge.
|
Package engine is the pure-Go mock NVML runtime driving the mocknvml bridge. |
|
shims
|
|
|
libcuda/bridge
command
Package main provides CUDA Runtime/Driver API bridge functions.
|
Package main provides CUDA Runtime/Driver API bridge functions. |
|
libcuda/engine
Package engine is the pure-Go mock CUDA runtime driving the libcuda bridge.
|
Package engine is the pure-Go mock CUDA runtime driving the libcuda bridge. |
|
nvidia-imex-shim
command
nvidia-imex-shim is a drop-in argv wrapper for the real `nvidia-imex` daemon.
|
nvidia-imex-shim is a drop-in argv wrapper for the real `nvidia-imex` daemon. |
|
tests
|
|
|
e2e/go/assertions/nvidiasmi
Package nvidiasmi asserts what `nvidia-smi -q -x` reports.
|
Package nvidiasmi asserts what `nvidia-smi -q -x` reports. |
|
e2e/go/ibutil
Package ibutil holds the pure LID/GUID normalizers used by the ibping assertion.
|
Package ibutil holds the pure LID/GUID normalizers used by the ibping assertion. |
|
e2e/go/profile
Package profile is the single, pure source of truth for the GPU-profile facts the e2e harness asserts against.
|
Package profile is the single, pure source of truth for the GPU-profile facts the e2e harness asserts against. |
|
kwok/cmd/kwok-assert
command
Command kwok-assert checks the scale scenario's bounded controller contract.
|
Command kwok-assert checks the scale scenario's bounded controller contract. |
Click to show internal directories.
Click to hide internal directories.