runtime

package
v0.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 8, 2026 License: Apache-2.0 Imports: 37 Imported by: 0

Documentation

Overview

Package runtime provides low-level host and container-runtime primitives for snapshot execution.

Index

Constants

View Source
const (
	ContainerdSocket = "/run/containerd/containerd.sock"
	CRIOSocket       = "/var/run/crio/crio.sock"

	RuntimeContainerd = "containerd"
	RuntimeCRIO       = "crio"
)

Default socket paths and runtime-type identifiers.

View Source
const HostCgroupPath = "/sys/fs/cgroup"
View Source
const HostProcPath = "/host/proc"

HostProcPath is the mount point for the host's /proc in DaemonSet pods.

View Source
const KubeletPodsDir = HostProcPath + "/1/root/var/lib/kubelet/pods"

KubeletPodsDir is kubelet's per-Pod directory, seen through the node's root filesystem: the agent runs with hostPID, so /host/proc/1 is the node's init. Unlike a hostPath mount it is not rewritten by vCluster, and it sees emptyDirs that kubelet mounts after the agent started, such as medium: Memory.

Variables

View Source
var ErrKeptFilesMissing = errors.New("rootfs diff may lack files that must be kept")

ErrKeptFilesMissing marks a CaptureRootfsDiff failure that may have left a Keep path out of the rootfs diff, because the path is in the upperdir or could not be inspected there.

Functions

func ApplyDeletedFiles

func ApplyDeletedFiles(checkpointPath, targetRoot string, log logr.Logger) error

ApplyDeletedFiles removes files marked as deleted in the checkpoint.

func ApplyRootfsDiff

func ApplyRootfsDiff(checkpointPath, targetRoot, bundleDir string, log logr.Logger) error

ApplyRootfsDiff extracts rootfs-diff.tar into the target root.

tar reads the archive one 1 MiB record at a time. On NFS that leaves one read in flight per round trip, so prefetchRootfsDiff reads ahead in parallel and tar finds the archive in the page cache.

func BuildMountPolicy

func BuildMountPolicy(mounts []types.MountInfo, rootFS string, maskedPaths []string) (map[string]string, []string)

BuildMountPolicy classifies mounts and masked paths for CRIU dump. Mounts must already have IsOCIManaged set by ClassifyMounts.

Policy (evaluated top to bottom):

  1. Skip: non-OCI /proc/*, /sys/*, /run/* submounts (virtual/runtime, not in placeholder) and NVIDIA device plugin allocation markers (named after the source's GPUs)
  2. Native: the container's procfs (CRIU mounts it for the restored PID namespace)
  3. Native: /dev/shm tmpfs (CRIU saves and restores content)
  4. Masked: OCI masked non-directory paths that exist under rootFS → /dev/null
  5. Externalize: everything else (OCI mounts the runtime recreates in placeholder)

func CaptureDeletedFiles

func CaptureDeletedFiles(upperDir, checkpointDir string) (bool, error)

CaptureDeletedFiles finds whiteout files and saves them to a JSON file.

func CheckProcessHandles added in v0.2.0

func CheckProcessHandles(files []*os.File) error

CheckProcessHandles rejects targets that exited while their host PIDs were resolved.

func ClassifyMounts

func ClassifyMounts(mounts []types.MountInfo, ociSpec *specs.Spec, rootFS string) []types.MountInfo

ClassifyMounts sets IsOCIManaged on each mount by matching against the container's OCI spec (mounts, masked paths, readonly paths). Handles /run/ ↔ /var/run/ aliasing since some images symlink one to the other.

func CommandInNamespaces added in v0.2.0

func CommandInNamespaces(ctx context.Context, pid int, mountNS *os.File, rootPath, binaryPath string) (*exec.Cmd, func(), error)

CommandInNamespaces runs a host binary in the container's namespaces and root. mountNS keeps the mount namespace available if its original process exits, but the other namespaces still use pid and require that process to stay alive until entry. The caller retains ownership of mountNS and must call closeFiles after the command finishes to release the binary and root descriptors opened here.

func ControlSentinelExists

func ControlSentinelExists(hostPID int, name string) (bool, error)

ControlSentinelExists reports whether a sentinel exists in the workload container's snapshot-control volume. It returns an error when the container's control mount cannot be inspected, so callers do not mistake an inaccessible volume for a missing sentinel. Only a regular file counts; anything else under the name is an error.

func Dumpable added in v0.3.0

func Dumpable(procRoot string, recorded TreeSnapshot) (bool, string)

Dumpable reports whether CRIU can dump the recorded tree as it stands, and names the first signal that says it cannot: CRIU released it, and the tree is still the one that was suspended.

func GPUStateUniform added in v0.3.0

func GPUStateUniform(states map[int]string, want string) (bool, string)

GPUStateUniform reports whether every GPU process reads want. A suspend that failed partway leaves some merely locked, and that tree is not the one the dump was meant to capture.

func GetNetNSInode

func GetNetNSInode(pid int) (uint64, error)

GetNetNSInode returns the network namespace inode for a container process via /host/proc.

func GetOverlayUpperDir

func GetOverlayUpperDir(pid int) (string, error)

GetOverlayUpperDir extracts the overlay upperdir from mountinfo.

func GetRootFS

func GetRootFS(pid int) (string, error)

GetRootFS returns the container's root filesystem path via /host/proc.

func InheritFile added in v0.2.0

func InheritFile(cmd *exec.Cmd, file *os.File) string

InheritFile passes file to cmd's child as an extra descriptor and returns the /proc/self/fd path that refers to it there. The caller retains file ownership.

func MinimumImageSize added in v0.3.0

func MinimumImageSize(procRoot string, pids []int) (int64, error)

MinimumImageSize reports a floor on the bytes a dump of this tree must write.

It is the largest single process's resident anonymous memory. Those pages are private and dirty, so they cannot be reconstructed from a file and must appear in the image; sharing them with another process does not make the image smaller. Summing the tree instead would count shared pages once per process and overstate, and resident memory as a whole counts file-backed pages that a dump records as mappings rather than writing out — either would make this an over-estimate, and an over-estimate refuses captures that fit.

func OpenPodControlDir added in v0.3.0

func OpenPodControlDir(podUID, containerName string) (*os.File, error)

OpenPodControlDir opens a container's snapshot-control directory as seen from the node: the container's subPath directory inside the Pod's control emptyDir, created if kubelet has not. Unlike the /host/proc/<pid>/root view, it needs no process in the container, so it works before the container starts and after its process is gone. Write into it with WriteFileInDir.

The volume directory belongs to kubelet, but everything inside it can be changed by the Pod. So the container's directory is opened without following symlinks, and files are created relative to it, never by path.

func OpenProcessHandles added in v0.2.0

func OpenProcessHandles(pids []int) ([]*os.File, error)

OpenProcessHandles opens pidfds in the caller's PID namespace. The caller must keep them open through host PID resolution and request submission.

func ParseProcExitCode

func ParseProcExitCode(statLine string) (syscall.WaitStatus, error)

ParseProcExitCode extracts and decodes the exit_code field (field 52) from a /proc/<pid>/stat line.

func ProcessTreePIDs

func ProcessTreePIDs(rootPID int) []int

ProcessTreePIDs walks the process tree rooted at rootPID and returns all PIDs. Used during checkpoint to enumerate the source process tree before CUDA filtering.

func ReadCPUFeatures added in v0.3.0

func ReadCPUFeatures(hostProcPath string) ([]string, error)

ReadCPUFeatures reads the instruction sets the host's CPU offers from the flags in /proc/cpuinfo, reduced by compat.InstructionSetFeatures. A flag only some processors report is left out, since a restored process can be scheduled onto any of them.

func ReadKernelVersion added in v0.2.0

func ReadKernelVersion(hostProcPath string) (string, error)

ReadKernelVersion reads the host's /proc/sys/kernel/osrelease entry documented at https://man7.org/linux/man-pages/man5/proc_sys_kernel.5.html.

func ReadMountInfo

func ReadMountInfo(pid int) ([]types.MountInfo, error)

ReadMountInfo reads and parses mountinfo for a container process via /host/proc.

func ReadProcessFilesystemIDs

func ReadProcessFilesystemIDs(procRoot string, pid int) (int, int, error)

ReadProcessFilesystemIDs returns the filesystem UID and GID from a proc status entry.

func Released added in v0.3.0

func Released(procRoot string, pids []int) (bool, string)

Released reports whether CRIU has let go of every task, and names the first one it has not.

CRIU seizes every task and injects a parasite, undoing both when it finishes. A run that was interrupted can leave a task attached or stopped, and such a task cannot resume on its own.

Uninterruptible sleep is deliberately not disqualifying here: it ends by itself, and a thread sitting in a driver call is ordinary right after a GPU restore. Only a dump cares, because CRIU cannot seize such a task.

func RemountProcSys

func RemountProcSys(rw bool) error

RemountProcSys remounts /proc/sys read-write or read-only.

func RemoveControlSentinel

func RemoveControlSentinel(dir, name string) error

RemoveControlSentinel removes a sentinel from the snapshot-control mount at dir. The directory must exist: a missing file is already removed, but a missing mount is an error so callers do not confuse "cannot see the volume" with "volume is clean".

func ResolveCgroupRootFromHostPID

func ResolveCgroupRootFromHostPID(pid int) (string, error)

ResolveCgroupRootFromHostPID reads the unified cgroup v2 path for a PID via /host/proc.

func ResolveHostPIDs added in v0.2.0

func ResolveHostPIDs(hostProcRoot, procRoot string, pids []int) ([]int, error)

ResolveHostPIDs maps PIDs observed through procRoot to the host proc view. Match the process's own namespace because CRIU can restore it below nsrestore.

func ResolveManifestPIDsToObservedPIDs

func ResolveManifestPIDsToObservedPIDs(processes []ProcessDetails, restoredPID int, manifestPIDs []int) ([]int, error)

ResolveManifestPIDsToObservedPIDs is the restore-side remap from checkpoint-time innermost namespace PIDs onto the current observed PIDs in the restored subtree rooted at restoredPID.

func SendSignalToPID

func SendSignalToPID(log logr.Logger, pid int, sig syscall.Signal, reason string) error

SendSignalToPID sends a signal to a host-visible PID via syscall.Kill.

func SetProcessGroupCancellation added in v0.2.0

func SetProcessGroupCancellation(cmd *exec.Cmd, waitDelay time.Duration)

SetProcessGroupCancellation makes cancellation kill a helper and its children, including children that keep the command's output pipes open. After the kill, Wait returns once waitDelay passes even if descendants still hold those pipes.

func StagingCapacity added in v0.3.0

func StagingCapacity(path string) (total int64, err error)

StagingCapacity reports the total size of the staging volume.

Deliberately the total rather than what is free: free space moves as other captures on the node take and release it, so a shortfall against it says only that the node is busy. Exceeding the total is a property of this image alone, and it is the only thing that can never change.

func StagingShortfall added in v0.3.0

func StagingShortfall(minimum, total int64) int64

StagingShortfall reports how far an image exceeds what staging could ever hold, or zero when it does not provably exceed it.

func StripCRIScheme

func StripCRIScheme(id string) string

StripCRIScheme trims the kubelet-format scheme prefix from a ContainerStatus.ContainerID. Returns id unchanged if no known scheme matches.

func ValidateProcessState

func ValidateProcessState(procRoot string, pid int) error

ValidateProcessState checks that a process is alive and not a zombie.

func WithDetail added in v0.3.0

func WithDetail(err error, detail string) error

WithDetail appends detail to err's message when there is one.

func WriteControlSentinel

func WriteControlSentinel(hostPID int, name string, contents []byte) error

WriteControlSentinel writes a sentinel file into the workload container's snapshot-control volume at SnapshotControlMountPath/<name>, accessed through the agent's /host/proc/<pid>/root view of the container's mount namespace.

hostPID must be a PID inside the container's mount namespace (the container task PID is the canonical choice). The sentinel is observed by the workload via inotify on the control directory; it replaces the SIGUSR1/SIGCONT agent-to-workload signals that previously required the workload to run as PID 1.

It writes through a handle on the control directory (openContainerControlDir), with WriteFileInDir.

func WriteFileInDir added in v0.3.0

func WriteFileInDir(dir *os.File, name string, perm os.FileMode, write func(io.Writer) error) (retErr error)

WriteFileInDir writes name in dir: write fills a new file under a random name, which is then renamed over name. Readers never see a partial file, and the rename replaces whatever name was, without following it.

func WritePodControlSentinel added in v0.3.0

func WritePodControlSentinel(podUID, containerName, name string, contents []byte) error

WritePodControlSentinel writes a sentinel into OpenPodControlDir.

Types

type CRIORuntime

type CRIORuntime struct {
	// contains filtered or unexported fields
}

CRIORuntime resolves container identity via the CRI-O CRI gRPC socket.

func NewCRIORuntime

func NewCRIORuntime(socket string) (*CRIORuntime, error)

func (*CRIORuntime) Close

func (r *CRIORuntime) Close() error

func (*CRIORuntime) ExecSync added in v0.3.0

func (r *CRIORuntime) ExecSync(ctx context.Context, containerID string, cmd []string, timeout time.Duration) ([]byte, []byte, error)

func (*CRIORuntime) ResolveContainer

func (r *CRIORuntime) ResolveContainer(ctx context.Context, id string) (int, *specs.Spec, error)

func (*CRIORuntime) ResolveContainerByPod

func (r *CRIORuntime) ResolveContainerByPod(ctx context.Context, podName, podNamespace, containerName string) (int, *specs.Spec, error)

ResolveContainerByPod picks the first RUNNING container matching the pod + container-name label filter; errors if none qualify.

func (*CRIORuntime) ResolveContainerIDByPod

func (r *CRIORuntime) ResolveContainerIDByPod(ctx context.Context, podName, podNamespace, containerName string) (string, error)

func (*CRIORuntime) ResolveContainerImageID added in v0.2.0

func (r *CRIORuntime) ResolveContainerImageID(ctx context.Context, containerID string) (string, error)

func (*CRIORuntime) ResolveContainerPodUID added in v0.3.0

func (r *CRIORuntime) ResolveContainerPodUID(ctx context.Context, containerID string) (string, error)

func (*CRIORuntime) TerminateContainer added in v0.2.0

func (r *CRIORuntime) TerminateContainer(ctx context.Context, containerID string) error

type ContainerdRuntime

type ContainerdRuntime struct {
	// contains filtered or unexported fields
}

func NewContainerdRuntime

func NewContainerdRuntime(socket string) (*ContainerdRuntime, error)

func (*ContainerdRuntime) Close

func (r *ContainerdRuntime) Close() error

func (*ContainerdRuntime) ExecSync added in v0.3.0

func (r *ContainerdRuntime) ExecSync(ctx context.Context, containerID string, cmd []string, timeout time.Duration) ([]byte, []byte, error)

func (*ContainerdRuntime) ResolveContainer

func (r *ContainerdRuntime) ResolveContainer(ctx context.Context, containerID string) (int, *specs.Spec, error)

func (*ContainerdRuntime) ResolveContainerByPod

func (r *ContainerdRuntime) ResolveContainerByPod(ctx context.Context, podName, podNamespace, containerName string) (int, *specs.Spec, error)

func (*ContainerdRuntime) ResolveContainerIDByPod

func (r *ContainerdRuntime) ResolveContainerIDByPod(ctx context.Context, podName, podNamespace, containerName string) (string, error)

func (*ContainerdRuntime) ResolveContainerImageID added in v0.2.0

func (r *ContainerdRuntime) ResolveContainerImageID(ctx context.Context, containerID string) (string, error)

func (*ContainerdRuntime) ResolveContainerPodUID added in v0.3.0

func (r *ContainerdRuntime) ResolveContainerPodUID(ctx context.Context, containerID string) (string, error)

func (*ContainerdRuntime) TerminateContainer added in v0.2.0

func (r *ContainerdRuntime) TerminateContainer(ctx context.Context, containerID string) error

type ProcessDetails

type ProcessDetails struct {
	ObservedPID   int
	ParentPID     int
	OutermostPID  int
	InnermostPID  int
	NamespacePIDs []int
	Cmdline       string
}

ProcessDetails captures the parent link plus the observed, outermost, and innermost PID views for one proc entry. ObservedPID is relative to the proc root being read.

func ReadProcessDetails

func ReadProcessDetails(procRoot string, pid int) (ProcessDetails, error)

ReadProcessDetails reads one proc entry from a proc root.

func ReadProcessDetailsOrDefault

func ReadProcessDetailsOrDefault(procRoot string, pid int) ProcessDetails

ReadProcessDetailsOrDefault preserves pid-scoped logging even when proc parsing fails.

func ReadProcessTable

func ReadProcessTable(procRoot string) ([]ProcessDetails, error)

ReadProcessTable snapshots every numeric proc entry under procRoot. Used by restore-side PID remap and diagnostics after CRIU restore.

type ProcessOutput added in v0.3.0

type ProcessOutput struct {
	// contains filtered or unexported fields
}

ProcessOutput holds what a child process wrote: all of stdout and a bounded tail of stderr, with stderr also mirrored to another writer so the agent log keeps the full stream. Detail reduces what was kept to one line fit for a condition message.

func NewProcessOutput added in v0.3.0

func NewProcessOutput(mirror io.Writer, causePrefixes ...string) ProcessOutput

NewProcessOutput mirrors stderr to mirror. The first stderr line starting with one of causePrefixes is remembered as it is written, so it survives however much the process prints after it, and is preferred as the detail over the last line written.

func (*ProcessOutput) Attach added in v0.3.0

func (o *ProcessOutput) Attach(cmd *exec.Cmd)

Attach points cmd's stdout and stderr at this collector.

func (*ProcessOutput) CauseLine added in v0.3.0

func (o *ProcessOutput) CauseLine() string

CauseLine returns the first stderr line that started with a cause prefix, or "" when none did.

func (*ProcessOutput) Detail added in v0.3.0

func (o *ProcessOutput) Detail() string

Detail is the one line that best explains a failed run: the cause line or the last stderr line, then any stdout, each cut to a bounded length.

func (*ProcessOutput) DetailFrom added in v0.3.0

func (o *ProcessOutput) DetailFrom(stderrLine string) string

DetailFrom builds the detail around a caller-chosen stderr line, for callers that know how to read their process's stderr better than the last line heuristic does.

func (*ProcessOutput) LastStderrLine added in v0.3.0

func (o *ProcessOutput) LastStderrLine() string

LastStderrLine returns the last non-empty line in the kept stderr tail.

func (*ProcessOutput) Stderr added in v0.3.0

func (o *ProcessOutput) Stderr() []byte

Stderr returns the kept tail of what the process wrote to stderr.

func (*ProcessOutput) Stdout added in v0.3.0

func (o *ProcessOutput) Stdout() []byte

Stdout returns everything the process wrote to stdout.

type RootfsDiff added in v0.3.0

type RootfsDiff struct {
	// Path is the published archive.
	Path string
	// Kept lists the Keep paths that the exclusions left out and that were
	// appended to the archive.
	Kept []string
	// KeptBytes is the total size of the regular files in Kept.
	KeptBytes int64
}

RootfsDiff describes a captured rootfs diff.

func CaptureRootfsDiff

func CaptureRootfsDiff(upperDir, checkpointDir string, opts RootfsDiffOptions) (RootfsDiff, error)

CaptureRootfsDiff captures the overlay upperdir to a tar file.

Files and directories in opts.Keep that are in the upperdir are kept even when they match an exclusion. Keep paths missing from the upperdir, or that would be reached through a symlink there, are skipped. Once a Keep path is found in the upperdir or cannot be inspected, any failure wraps ErrKeptFilesMissing.

The archive is written to a temporary file first and renamed atomically on success, so a partial or failed capture never leaves a corrupt archive at the final path.

type RootfsDiffOptions added in v0.3.0

type RootfsDiffOptions struct {
	// Exclusions are tar patterns left out of the archive.
	Exclusions types.OverlaySettings
	// BindMountDests are the container's bind mount destinations, left out of
	// the archive.
	BindMountDests []string
	// Keep lists container paths, such as criu.ReadReferencedPaths returns,
	// that are archived even when an exclusion matches them.
	Keep []string
}

RootfsDiffOptions selects what CaptureRootfsDiff archives from the upperdir.

type Runtime

type Runtime interface {
	ResolveContainer(ctx context.Context, id string) (int, *specs.Spec, error)
	ResolveContainerIDByPod(ctx context.Context, pod, ns, ctr string) (string, error)
	ResolveContainerByPod(ctx context.Context, pod, ns, ctr string) (int, *specs.Spec, error)
	ResolveContainerImageID(ctx context.Context, id string) (string, error)
	// ResolveContainerPodUID returns the UID of the container's Pod on the
	// node, which names the Pod's directories under kubelet's.
	ResolveContainerPodUID(ctx context.Context, id string) (string, error)
	// TerminateContainer requests zero-grace termination through CRI using the
	// runtime container ID. Implementations validate any supplied runtime scheme
	// before dispatch.
	TerminateContainer(ctx context.Context, id string) error
	// ExecSync runs cmd in the container through CRI, the way kubelet runs an
	// exec probe: as the container's user, with its capabilities, seccomp and
	// LSM profile, inside its mounts. A non-zero exit is an error with an
	// ExitStatus method; a timeout wraps remote.ErrCommandTimedOut.
	ExecSync(ctx context.Context, id string, cmd []string, timeout time.Duration) (stdout, stderr []byte, err error)
	Close() error
}

Runtime abstracts the container-identity APIs behind a two-backend switch. Resolve methods return non-nil *specs.Spec with PID > 0 on success, or an error.

func New

func New(runtimeType, socket string) (Runtime, error)

New constructs a Runtime backend for the given type and socket. Pass an empty socket to use the per-type default.

type TaskProbe added in v0.3.0

type TaskProbe struct {
	PID       int
	TracerPID int
	// State is the first field of the status line: R, S, D, T and so on.
	State   string
	Threads int
}

TaskProbe is what /proc says about one task's availability to CRIU.

func ProbeTask added in v0.3.0

func ProbeTask(procRoot string, pid int) (TaskProbe, error)

ProbeTask reads the three fields that decide whether CRIU can seize a task, for a process's thread-group leader.

type TreeSnapshot added in v0.3.0

type TreeSnapshot map[int]int

TreeSnapshot records the tree at suspend as PID to thread count, so a later probe can tell whether anything exited or forked since.

func SnapshotTree added in v0.3.0

func SnapshotTree(procRoot string, pids []int) (TreeSnapshot, error)

SnapshotTree records the tree so a later probe can compare against it.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL