Documentation
¶
Overview ¶
Package cuda provides CUDA checkpoint and restore operations.
Index ¶
- Constants
- func BuildDeviceMap(sourceUUIDs, targetUUIDs []string, log logr.Logger) (string, error)
- func CheckJobFile(jobFile string, gpuCount int, usesCoordinator bool) error
- func DiscoverGPUUUIDs(ctx context.Context, clientset kubernetes.Interface, ...) ([]string, error)
- func DiscoverGPUs(ctx context.Context, clientset kubernetes.Interface, ...) (compat.GPUInfo, error)
- func FilterProcesses(ctx context.Context, allPIDs []int, log logr.Logger) []int
- func GPUUUIDs(env compat.GPUInfo) []string
- func GetGPUUUIDsViaDRAAPI(ctx context.Context, clientset kubernetes.Interface, ...) ([]string, bool, error)
- func GetPodGPUUUIDs(ctx context.Context, podName, podNamespace, containerName string) ([]string, error)
- func InspectCuInterpose(ctx context.Context, target CuInterposeTarget) error
- func JobFileFromCheckpoint(checkpointDir string) (string, error)
- func PrepareCuInterpose(ctx context.Context, target CuInterposeTarget, checkpointDir string) error
- func PrepareLiveJobFile(stagedJobFile string) (string, error)
- func ProcessStates(ctx context.Context, pids []int, helperBinaryPath string) (map[int]string, error)
- func ResolveDevicePaths(hostProc string, pid int, uuids []string) (map[string]string, error)
- func RestoreCuInterpose(ctx context.Context, checkpointDir string, namespacePIDs []int, binary string) error
- func ReviveProcessTree(ctx context.Context, pids []int, helperBinaryPath string, log logr.Logger) error
- func SetLiveJobFileOwner(jobFile string, uid, gid int) error
- func StageJobFile(sourceRootPath, checkpointDir string) (string, error)
- func VerifyCuInterposeLibraryIdentity(directory string, identity *types.CuInterposeManifest) error
- func VisibleDevicesValue(env []string) *string
- type CheckpointPhaseTimings
- type CuInterposeInspection
- type CuInterposeTarget
- type RestorePhaseTimings
- type VisibleGPULister
- type VisibleGPUs
Constants ¶
const ( // HelperBinaryName is the cuda-checkpoint-helper executable name. HelperBinaryName = "cuda-checkpoint-helper" // DefaultHelperBinaryPath is the agent-side cuda-checkpoint-helper absolute path. // In the placeholder namespace pass filepath.Join(bundleDir, HelperBinaryName) instead. DefaultHelperBinaryPath = "/usr/local/bin/" + HelperBinaryName )
const ( CoordinatorBinaryName = "cuinterpose-coordinator" DefaultCoordinatorBinaryPath = "/usr/local/bin/" + CoordinatorBinaryName CuInterposeSocketPattern = cuInterposeSocketDir + "/cuinterpose-*.sock" )
const ( StateRunning = "running" StateLocked = "locked" StateCheckpointed = "checkpointed" StateFailed = "failed" )
States reported by the checkpoint helper.
const JobFileEnv = "CUDA_CHECKPOINT_JOB_FILE"
JobFileEnv is the CUDA launch-job environment variable consumed by the driver.
Variables ¶
This section is empty.
Functions ¶
func BuildDeviceMap ¶
BuildDeviceMap creates a cuda-checkpoint-helper --device-map value from source and target GPU UUID lists. The i-th source GPU maps to the i-th target GPU, so each process keeps its position among the pod's GPUs. DiscoverGPUs lists every pod's GPUs in PCI bus order, so that position is the GPU's bus position, and GPU Memory Service places each GPU's restored weights by bus position. A restore onto the same set of GPUs returns an empty string instead, so it uses the default CUDA restore path rather than the GPU migration path even when kubelet lists the set in another order.
func CheckJobFile ¶ added in v0.2.0
CheckJobFile applies the same launch-job rule at capture and restore. Native multi-GPU restore needs the file, while a cuinterpose coordinator restores shared IPC and multicast state itself.
func DiscoverGPUUUIDs ¶
func DiscoverGPUUUIDs(ctx context.Context, clientset kubernetes.Interface, podName, podNamespace, containerName, hostProcPath string, pid int, log logr.Logger) ([]string, error)
DiscoverGPUUUIDs resolves GPU UUIDs in the container's runtime ordinal order.
func DiscoverGPUs ¶ added in v0.2.0
func DiscoverGPUs(ctx context.Context, clientset kubernetes.Interface, podName, podNamespace, containerName string, env []string, visible VisibleGPULister, log logr.Logger) (compat.GPUInfo, error)
DiscoverGPUs resolves the same GPUs as DiscoverGPUUUIDs, in the same order, described by model and driver version wherever nvidia-smi can be reached. Whichever path finds the GPUs, they come out the same shape, so what gets recorded does not depend on how this cluster allocates GPUs. Explicit environment selections are resolved on the host; disabled legacy selections inspect only container visibility, which can include CDI devices. visible is what the container sees, from ListVisibleGPUs.
func FilterProcesses ¶
FilterProcesses returns the candidate PIDs with a CUDA driver restore thread. Driver initialization can create that thread without a cuInterpose core runtime. Uses --get-restore-tid (the same technique as the CRIU CUDA plugin) instead of --get-state, because --get-state incorrectly matches coordinator processes like cuda-checkpoint --launch-job that share a /proc namespace with CUDA processes but do not have their own CUDA driver restore thread.
func GetGPUUUIDsViaDRAAPI ¶
func GetGPUUUIDsViaDRAAPI(ctx context.Context, clientset kubernetes.Interface, podName, podNamespace, containerName string, log logr.Logger) ([]string, bool, error)
GetGPUUUIDsViaDRAAPI resolves GPU UUIDs for a pod by querying the Kubernetes API: Pod (resource claim refs) -> ResourceClaim (allocation results) -> ResourceSlice (device attributes). A non-empty containerName restricts resolution to the claims (and claim requests) that container references via resources.claims. On success, the returned bool reports whether NVIDIA DRA GPU allocations were found for that container. With a non-nil error, it reports whether the container was established to reference resource claims, in which case the caller must fail instead of falling back to other discovery paths.
func GetPodGPUUUIDs ¶
func GetPodGPUUUIDs(ctx context.Context, podName, podNamespace, containerName string) ([]string, error)
GetPodGPUUUIDs resolves GPU UUIDs for a pod/container from kubelet PodResources (nvidia.com/gpu entries in GetDevices()).
func InspectCuInterpose ¶ added in v0.2.0
func InspectCuInterpose(ctx context.Context, target CuInterposeTarget) error
InspectCuInterpose only reads state and requires every participant to reply and pass topology validation before the caller enters the phase where failure requires source termination.
func JobFileFromCheckpoint ¶
JobFileFromCheckpoint returns the staged job file when an artifact contains one.
func PrepareCuInterpose ¶ added in v0.2.0
func PrepareCuInterpose(ctx context.Context, target CuInterposeTarget, checkpointDir string) error
PrepareCuInterpose removes shared mappings, so the caller must terminate the source on failure.
func PrepareLiveJobFile ¶
PrepareLiveJobFile materializes the immutable capture-time launch-job state at the fixed launch-job path. It runs inside the restore container's namespaces before CRIU recreates processes. The returned path is the per-restore working copy that CUDA helpers must use; the staged artifact remains immutable so it can seed later restores.
func ProcessStates ¶ added in v0.3.0
func ProcessStates(ctx context.Context, pids []int, helperBinaryPath string) (map[int]string, error)
ProcessStates reads the GPU state of each process.
func ResolveDevicePaths ¶ added in v0.2.0
ResolveDevicePaths associates physical UUIDs with host minor numbers and verifies that those exact devices are exposed in the workload namespace.
func RestoreCuInterpose ¶ added in v0.2.0
func RestoreCuInterpose(ctx context.Context, checkpointDir string, namespacePIDs []int, binary string) error
RestoreCuInterpose runs inside the restored namespaces through a binary descriptor opened before CRIU.
func ReviveProcessTree ¶ added in v0.3.0
func ReviveProcessTree(ctx context.Context, pids []int, helperBinaryPath string, log logr.Logger) error
ReviveProcessTree hands a suspended workload back the GPUs it still holds, so a capture that could not be completed leaves it running rather than dead.
Checkpointing is a reversible state machine, and a tree can be part way through it: a suspend that failed leaves some processes merely locked while others are already checkpointed, so each process is dispatched on its own state. The first failure ends the attempt: the tree is only usable if every process comes back, and the caller terminates it either way. No device remapping is needed, because the workload is being handed back the GPUs it never released.
func SetLiveJobFileOwner ¶
SetLiveJobFileOwner makes the restore-time working copy accessible to the restored workload without following a replacement symlink.
func StageJobFile ¶
StageJobFile copies a launch-job file into the checkpoint artifact and returns the host-visible path to the source Pod's live file, or "" when the source has none. Capture helpers must use the live file to join the targets' CUDA job, while the artifact copy is only for later restore Pods. The launch wrapper saves the driver-created file at a fixed path before starting the workload.
func VerifyCuInterposeLibraryIdentity ¶ added in v0.2.0
func VerifyCuInterposeLibraryIdentity(directory string, identity *types.CuInterposeManifest) error
VerifyCuInterposeLibraryIdentity checks that the restore bundle matches the libraries recorded at capture. Compatibility overrides do not bypass this check.
func VisibleDevicesValue ¶ added in v0.2.0
VisibleDevicesValue distinguishes absent from explicitly empty. Scan backwards to match the last-assignment-wins convention used for process environments.
Types ¶
type CheckpointPhaseTimings ¶
func CheckpointProcessTree ¶
func CheckpointProcessTree(ctx context.Context, cudaPIDs []int, jobFile, checkpointDir string, log logr.Logger) (CheckpointPhaseTimings, error)
CheckpointProcessTree locks and checkpoints CUDA state for all given PIDs, then persists the launch-job state needed to restore them. On failure, the caller is expected to fail the operation and terminate the workload.
type CuInterposeInspection ¶ added in v0.2.0
type CuInterposeInspection struct {
// contains filtered or unexported fields
}
CuInterposeInspection is what capture found in the CUDA processes. It becomes the manifest once host PIDs are translated to namespace PIDs.
func InspectCuInterposeLibraries ¶ added in v0.2.0
func InspectCuInterposeLibraries(procRoot string, hostPIDs []int, requested bool) (*CuInterposeInspection, error)
InspectCuInterposeLibraries verifies that every CUDA process loaded the same shim bundle. It returns nil when no process loaded the shim, which is an error for a workload that requested it.
func (*CuInterposeInspection) Manifest ¶ added in v0.2.0
func (i *CuInterposeInspection) Manifest(cudaHostPIDs, cudaNamespacePIDs []int) *types.CuInterposeManifest
Manifest translates coordinator host PIDs through the parallel CUDA PID lists.
type CuInterposeTarget ¶ added in v0.2.0
type CuInterposeTarget struct {
ProcRoot string
// HostPID indexes ProcRoot and selects the namespaces the coordinator enters.
HostPID int
// NamespacePIDs are the participants' innermost namespace PIDs, which name their
// coordinator endpoints.
NamespacePIDs []int
Binary string
}
CuInterposeTarget selects the source container a coordinator command runs in.
type RestorePhaseTimings ¶
func RestoreAndUnlockProcessTree ¶
func RestoreAndUnlockProcessTree(ctx context.Context, cudaPIDs []int, deviceMap, helperBinaryPath string, concurrency int, log logr.Logger) (RestorePhaseTimings, error)
RestoreAndUnlockProcessTree restores and unlocks CUDA state for the given PIDs. Restores run concurrently, up to `concurrency` at a time; unlocks stay serial. It runs inside nsrestore and shares its process group so cancellation reaches the CUDA helper. helperBinaryPath identifies the binary in that namespace, including through an inherited descriptor opened before CRIU restores the workload filesystem.
type VisibleGPULister ¶ added in v0.3.0
type VisibleGPULister func() (VisibleGPUs, error)
VisibleGPULister answers what one container sees of the node's GPUs.
func ListVisibleGPUs ¶ added in v0.3.0
func ListVisibleGPUs(ctx context.Context, hostProcPath string, pid int, spec *specs.Spec) VisibleGPULister
ListVisibleGPUs returns a lister for the container whose host PID is pid. On its first call it reads the container's cards from the driver's procfs, or, when that cannot answer exactly (MIG, or no device cgroup rules), runs nvidia-smi -L inside the container's mount and PID namespaces and sorts its cards by the bus IDs in the host's driver procfs. Every later call gets the same answer. nvidia-smi spends most of its time initializing NVML, 0.1 to 0.2 s on an idle eight-GPU node and seconds while other processes start CUDA, so a capture or restore lists once for both its MIG check and GPU discovery.
The model comes from the same source as the UUIDs, and the driver version from the kernel module's version file: nothing else on the restore path gets to look at the source node's GPUs, so what is not read here cannot be compared later. nvidia-smi runs only against the module version it was built for, so that file holds the version nvidia-smi would report, and every container on the node shares it.
Under DRA this is the only path that reports GPUs at all, because the kubelet publishes no nvidia.com/gpu devices when the NVIDIA DRA driver allocates them instead of the device plugin.
type VisibleGPUs ¶ added in v0.3.0
type VisibleGPUs struct {
// GPUs are the cards in PCI bus order, described by model and driver
// version. A card carved into MIG slices is listed as
// the card.
GPUs compat.GPUInfo
// MIGSlices are the slices listed under those cards. No DiscoverGPUs path
// can stand in for them: DRA drops a slice's MIG- UUID.
MIGSlices []string
// contains filtered or unexported fields
}
VisibleGPUs is what a container sees of the node's GPUs.