cuda

package
v0.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 8, 2026 License: Apache-2.0 Imports: 33 Imported by: 0

Documentation

Overview

Package cuda provides CUDA checkpoint and restore operations.

Index

Constants

View Source
const (

	// HelperBinaryName is the cuda-checkpoint-helper executable name.
	HelperBinaryName = "cuda-checkpoint-helper"
	// DefaultHelperBinaryPath is the agent-side cuda-checkpoint-helper absolute path.
	// In the placeholder namespace pass filepath.Join(bundleDir, HelperBinaryName) instead.
	DefaultHelperBinaryPath = "/usr/local/bin/" + HelperBinaryName
)
View Source
const (
	CoordinatorBinaryName        = "cuinterpose-coordinator"
	DefaultCoordinatorBinaryPath = "/usr/local/bin/" + CoordinatorBinaryName

	CuInterposeSocketPattern = cuInterposeSocketDir + "/cuinterpose-*.sock"
)
View Source
const (
	StateRunning      = "running"
	StateLocked       = "locked"
	StateCheckpointed = "checkpointed"
	StateFailed       = "failed"
)

States reported by the checkpoint helper.

View Source
const JobFileEnv = "CUDA_CHECKPOINT_JOB_FILE"

JobFileEnv is the CUDA launch-job environment variable consumed by the driver.

Variables

This section is empty.

Functions

func BuildDeviceMap

func BuildDeviceMap(sourceUUIDs, targetUUIDs []string, log logr.Logger) (string, error)

BuildDeviceMap creates a cuda-checkpoint-helper --device-map value from source and target GPU UUID lists. The i-th source GPU maps to the i-th target GPU, so each process keeps its position among the pod's GPUs. DiscoverGPUs lists every pod's GPUs in PCI bus order, so that position is the GPU's bus position, and GPU Memory Service places each GPU's restored weights by bus position. A restore onto the same set of GPUs returns an empty string instead, so it uses the default CUDA restore path rather than the GPU migration path even when kubelet lists the set in another order.

func CheckJobFile added in v0.2.0

func CheckJobFile(jobFile string, gpuCount int, usesCoordinator bool) error

CheckJobFile applies the same launch-job rule at capture and restore. Native multi-GPU restore needs the file, while a cuinterpose coordinator restores shared IPC and multicast state itself.

func DiscoverGPUUUIDs

func DiscoverGPUUUIDs(ctx context.Context, clientset kubernetes.Interface, podName, podNamespace, containerName, hostProcPath string, pid int, log logr.Logger) ([]string, error)

DiscoverGPUUUIDs resolves GPU UUIDs in the container's runtime ordinal order.

func DiscoverGPUs added in v0.2.0

func DiscoverGPUs(ctx context.Context, clientset kubernetes.Interface, podName, podNamespace, containerName string, env []string, visible VisibleGPULister, log logr.Logger) (compat.GPUInfo, error)

DiscoverGPUs resolves the same GPUs as DiscoverGPUUUIDs, in the same order, described by model and driver version wherever nvidia-smi can be reached. Whichever path finds the GPUs, they come out the same shape, so what gets recorded does not depend on how this cluster allocates GPUs. Explicit environment selections are resolved on the host; disabled legacy selections inspect only container visibility, which can include CDI devices. visible is what the container sees, from ListVisibleGPUs.

func FilterProcesses

func FilterProcesses(ctx context.Context, allPIDs []int, log logr.Logger) []int

FilterProcesses returns the candidate PIDs with a CUDA driver restore thread. Driver initialization can create that thread without a cuInterpose core runtime. Uses --get-restore-tid (the same technique as the CRIU CUDA plugin) instead of --get-state, because --get-state incorrectly matches coordinator processes like cuda-checkpoint --launch-job that share a /proc namespace with CUDA processes but do not have their own CUDA driver restore thread.

func GPUUUIDs added in v0.2.0

func GPUUUIDs(env compat.GPUInfo) []string

GPUUUIDs returns device UUIDs in discovery order.

func GetGPUUUIDsViaDRAAPI

func GetGPUUUIDsViaDRAAPI(ctx context.Context, clientset kubernetes.Interface, podName, podNamespace, containerName string, log logr.Logger) ([]string, bool, error)

GetGPUUUIDsViaDRAAPI resolves GPU UUIDs for a pod by querying the Kubernetes API: Pod (resource claim refs) -> ResourceClaim (allocation results) -> ResourceSlice (device attributes). A non-empty containerName restricts resolution to the claims (and claim requests) that container references via resources.claims. On success, the returned bool reports whether NVIDIA DRA GPU allocations were found for that container. With a non-nil error, it reports whether the container was established to reference resource claims, in which case the caller must fail instead of falling back to other discovery paths.

func GetPodGPUUUIDs

func GetPodGPUUUIDs(ctx context.Context, podName, podNamespace, containerName string) ([]string, error)

GetPodGPUUUIDs resolves GPU UUIDs for a pod/container from kubelet PodResources (nvidia.com/gpu entries in GetDevices()).

func InspectCuInterpose added in v0.2.0

func InspectCuInterpose(ctx context.Context, target CuInterposeTarget) error

InspectCuInterpose only reads state and requires every participant to reply and pass topology validation before the caller enters the phase where failure requires source termination.

func JobFileFromCheckpoint

func JobFileFromCheckpoint(checkpointDir string) (string, error)

JobFileFromCheckpoint returns the staged job file when an artifact contains one.

func PrepareCuInterpose added in v0.2.0

func PrepareCuInterpose(ctx context.Context, target CuInterposeTarget, checkpointDir string) error

PrepareCuInterpose removes shared mappings, so the caller must terminate the source on failure.

func PrepareLiveJobFile

func PrepareLiveJobFile(stagedJobFile string) (string, error)

PrepareLiveJobFile materializes the immutable capture-time launch-job state at the fixed launch-job path. It runs inside the restore container's namespaces before CRIU recreates processes. The returned path is the per-restore working copy that CUDA helpers must use; the staged artifact remains immutable so it can seed later restores.

func ProcessStates added in v0.3.0

func ProcessStates(ctx context.Context, pids []int, helperBinaryPath string) (map[int]string, error)

ProcessStates reads the GPU state of each process.

func ResolveDevicePaths added in v0.2.0

func ResolveDevicePaths(hostProc string, pid int, uuids []string) (map[string]string, error)

ResolveDevicePaths associates physical UUIDs with host minor numbers and verifies that those exact devices are exposed in the workload namespace.

func RestoreCuInterpose added in v0.2.0

func RestoreCuInterpose(ctx context.Context, checkpointDir string, namespacePIDs []int, binary string) error

RestoreCuInterpose runs inside the restored namespaces through a binary descriptor opened before CRIU.

func ReviveProcessTree added in v0.3.0

func ReviveProcessTree(ctx context.Context, pids []int, helperBinaryPath string, log logr.Logger) error

ReviveProcessTree hands a suspended workload back the GPUs it still holds, so a capture that could not be completed leaves it running rather than dead.

Checkpointing is a reversible state machine, and a tree can be part way through it: a suspend that failed leaves some processes merely locked while others are already checkpointed, so each process is dispatched on its own state. The first failure ends the attempt: the tree is only usable if every process comes back, and the caller terminates it either way. No device remapping is needed, because the workload is being handed back the GPUs it never released.

func SetLiveJobFileOwner

func SetLiveJobFileOwner(jobFile string, uid, gid int) error

SetLiveJobFileOwner makes the restore-time working copy accessible to the restored workload without following a replacement symlink.

func StageJobFile

func StageJobFile(sourceRootPath, checkpointDir string) (string, error)

StageJobFile copies a launch-job file into the checkpoint artifact and returns the host-visible path to the source Pod's live file, or "" when the source has none. Capture helpers must use the live file to join the targets' CUDA job, while the artifact copy is only for later restore Pods. The launch wrapper saves the driver-created file at a fixed path before starting the workload.

func VerifyCuInterposeLibraryIdentity added in v0.2.0

func VerifyCuInterposeLibraryIdentity(directory string, identity *types.CuInterposeManifest) error

VerifyCuInterposeLibraryIdentity checks that the restore bundle matches the libraries recorded at capture. Compatibility overrides do not bypass this check.

func VisibleDevicesValue added in v0.2.0

func VisibleDevicesValue(env []string) *string

VisibleDevicesValue distinguishes absent from explicitly empty. Scan backwards to match the last-assignment-wins convention used for process environments.

Types

type CheckpointPhaseTimings

type CheckpointPhaseTimings struct {
	TotalDuration time.Duration
}

func CheckpointProcessTree

func CheckpointProcessTree(ctx context.Context, cudaPIDs []int, jobFile, checkpointDir string, log logr.Logger) (CheckpointPhaseTimings, error)

CheckpointProcessTree locks and checkpoints CUDA state for all given PIDs, then persists the launch-job state needed to restore them. On failure, the caller is expected to fail the operation and terminate the workload.

type CuInterposeInspection added in v0.2.0

type CuInterposeInspection struct {
	// contains filtered or unexported fields
}

CuInterposeInspection is what capture found in the CUDA processes. It becomes the manifest once host PIDs are translated to namespace PIDs.

func InspectCuInterposeLibraries added in v0.2.0

func InspectCuInterposeLibraries(procRoot string, hostPIDs []int, requested bool) (*CuInterposeInspection, error)

InspectCuInterposeLibraries verifies that every CUDA process loaded the same shim bundle. It returns nil when no process loaded the shim, which is an error for a workload that requested it.

func (*CuInterposeInspection) Manifest added in v0.2.0

func (i *CuInterposeInspection) Manifest(cudaHostPIDs, cudaNamespacePIDs []int) *types.CuInterposeManifest

Manifest translates coordinator host PIDs through the parallel CUDA PID lists.

type CuInterposeTarget added in v0.2.0

type CuInterposeTarget struct {
	ProcRoot string
	// HostPID indexes ProcRoot and selects the namespaces the coordinator enters.
	HostPID int
	// NamespacePIDs are the participants' innermost namespace PIDs, which name their
	// coordinator endpoints.
	NamespacePIDs []int
	Binary        string
}

CuInterposeTarget selects the source container a coordinator command runs in.

type RestorePhaseTimings

type RestorePhaseTimings struct {
	TotalDuration time.Duration
}

func RestoreAndUnlockProcessTree

func RestoreAndUnlockProcessTree(ctx context.Context, cudaPIDs []int, deviceMap, helperBinaryPath string, concurrency int, log logr.Logger) (RestorePhaseTimings, error)

RestoreAndUnlockProcessTree restores and unlocks CUDA state for the given PIDs. Restores run concurrently, up to `concurrency` at a time; unlocks stay serial. It runs inside nsrestore and shares its process group so cancellation reaches the CUDA helper. helperBinaryPath identifies the binary in that namespace, including through an inherited descriptor opened before CRIU restores the workload filesystem.

type VisibleGPULister added in v0.3.0

type VisibleGPULister func() (VisibleGPUs, error)

VisibleGPULister answers what one container sees of the node's GPUs.

func ListVisibleGPUs added in v0.3.0

func ListVisibleGPUs(ctx context.Context, hostProcPath string, pid int, spec *specs.Spec) VisibleGPULister

ListVisibleGPUs returns a lister for the container whose host PID is pid. On its first call it reads the container's cards from the driver's procfs, or, when that cannot answer exactly (MIG, or no device cgroup rules), runs nvidia-smi -L inside the container's mount and PID namespaces and sorts its cards by the bus IDs in the host's driver procfs. Every later call gets the same answer. nvidia-smi spends most of its time initializing NVML, 0.1 to 0.2 s on an idle eight-GPU node and seconds while other processes start CUDA, so a capture or restore lists once for both its MIG check and GPU discovery.

The model comes from the same source as the UUIDs, and the driver version from the kernel module's version file: nothing else on the restore path gets to look at the source node's GPUs, so what is not read here cannot be compared later. nvidia-smi runs only against the module version it was built for, so that file holds the version nvidia-smi would report, and every container on the node shares it.

Under DRA this is the only path that reports GPUs at all, because the kubelet publishes no nvidia.com/gpu devices when the NVIDIA DRA driver allocates them instead of the device plugin.

type VisibleGPUs added in v0.3.0

type VisibleGPUs struct {
	// GPUs are the cards in PCI bus order, described by model and driver
	// version. A card carved into MIG slices is listed as
	// the card.
	GPUs compat.GPUInfo
	// MIGSlices are the slices listed under those cards. No DiscoverGPUs path
	// can stand in for them: DRA drops a slice's MIG- UUID.
	MIGSlices []string
	// contains filtered or unexported fields
}

VisibleGPUs is what a container sees of the node's GPUs.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL