forkvm

VM Forking: Hypervisor Behavior

This document describes hypervisor-specific fork behavior and how fork is made to work across implementations.

Common fork model

• Stopped source: clone VM data and start a new VM from copied state.
• Standby source: clone data + snapshot artifacts, then adapt snapshot identity for the fork (paths, network, vsock behavior varies by hypervisor).
• Running source: transition source to standby, fork from that standby snapshot, then restore the source.

For networked forks, the fork gets a fresh host/guest identity (IP, MAC, TAP) instead of reusing the source identity.

Fork data copy behavior

• Guest directory copy is sparse-only for regular files.
• Copy uses SEEK_DATA/SEEK_HOLE to preserve sparse holes and avoid de-sparsifying large overlay images.
• If sparse seeking is unsupported on the underlying filesystem, fork fails with an explicit sparse-capability error (no dense-copy fallback).
• Non-essential runtime artifacts are skipped during copy: logs/ subtree and runtime socket files.

Cloud Hypervisor

• Snapshot-based forks are supported by rewriting snapshot configuration before restore.
• Path rewrites are constrained to exact source-directory matches or source-dir path prefixes to avoid mutating unrelated values.
• Serial log path, vsock socket path, and network fields are updated for the fork.
• Vsock CID is intentionally kept stable for snapshot restore compatibility.
• Running-source fork works by standby -> fork -> restore source, with source and fork separated by rewritten runtime endpoints.

QEMU

• Snapshot-based forks are supported by rewriting QEMU snapshot VM config.
• Rewrites are explicit and path-safe (source-dir exact/prefix replacement), applied to disk/kernel/initrd/serial/vsock socket paths.
• Kernel arguments are left unchanged (not blanket-rewritten), to avoid accidental mutation of non-path text.
• Network identity is updated in snapshot config for the fork.
• Vsock CID updates are supported for snapshot state, so running-source fork can rotate source CID when needed to avoid CID collision after restore.

Firecracker

• Firecracker snapshot restore supports network overrides but does not expose a full snapshot-config rewrite surface for arbitrary embedded paths.
• To make standby/running fork work, fork preparation stores desired network override data and source->target data-directory mapping.
• During restore, the source data path is temporarily aliased to the fork data path so embedded snapshot paths resolve for the fork, then aliasing is cleaned up.
• Network override fields are supplied at snapshot load to bind the fork to its own TAP device.
• Vsock CID remains stable for snapshot-based flows.
• When the Firecracker snapshot memory backend is configured as UFFD, UFFD is used as a one-shot acceleration for the first restore of a newly forked standby snapshot. The fork initially reuses the source snapshot memory as the pager backing file instead of cloning the large memory file during fanout.
• That deferred memory clone is paid when the fork later enters standby. Before Firecracker writes the fork's diff snapshot, Hypeman materializes the fork's own snapshot-latest/memory file from the original backing memory. After that point the fork has a normal on-disk snapshot base, independent from the source.
• Subsequent direct restores of that same fork use Firecracker's normal file-backed memory backend. If that standby fork is itself forked again, the new child gets its own one-shot UFFD restore.
• This keeps UFFD on the high-fanout path where shared snapshot cache is most useful, while preserving the normal Firecracker diff-snapshot lifecycle for per-instance standby/resume cycles.

VZ (Virtualization.framework)

• Stopped-source fork is supported (directory clone, no snapshot rewrite).
• Standby-source fork is supported (snapshot copy + VZ manifest rewrite + restore).
• Running-source fork is supported (standby source -> fork from standby -> restore source).
• VZ fork preparation rewrites instance-local paths in serialized shim config: disks, kernel/initrd, serial log, control socket, vsock socket, shim log.
• VZ keeps snapshotted NIC identity unchanged during fork prep because save/restore validation can reject machine-state restore when NIC identity fields are mutated.
• For forked standby restores with networking, a fresh network allocation is applied post-restore via the generic restore networking flow.
• Vsock socket naming is resolved generically through hypervisor registration (vz.vsock for VZ), so no instance-layer VZ-specific branching is required.
• Vsock CID rewrites are not required for VZ fork flows because VZ routing is socket-path based.

Operational constraints

• Writable attached volumes are rejected for fork to prevent concurrent cross-VM writes to the same backing data.
• If a post-fork target-state transition fails, the partially created fork is cleaned up rather than left orphaned.