vz-container

vz-container

A pure Go container runtime using Apple's Virtualization.framework (VZ). Run Linux containers in lightweight VMs on macOS with strong isolation and fast startup times.

Features

• Pure Go - No CGO, uses purego for Objective-C runtime calls
• Lightweight VMs - Each container runs in a dedicated Linux VM via Virtualization.framework
• Fast startup - Optimized kernel and minimal init for sub-second container boot
• Strong isolation - Hardware-level VM isolation between containers
• Docker-like CLI - Familiar run, ps, stop, rm commands

Architecture

┌─────────────────────────────────────────────────────────────────────────┐
│                           HOST (macOS)                                  │
│                                                                         │
│  ┌──────────────────┐          ┌────────────────────────────────────┐  │
│  │  vz-container    │          │     Virtualization.framework       │  │
│  │  (Go CLI)        │─────────▶│     (VZVirtualMachine)             │  │
│  │                  │          │                                    │  │
│  │  • VM lifecycle  │          │  ┌──────────────────────────────┐  │  │
│  │  • GRPC client   │◀─vsock──▶│  │        Linux VM              │  │  │
│  │  • Image mgmt    │          │  │                              │  │  │
│  └──────────────────┘          │  │  ┌────────────────────────┐  │  │  │
│                                │  │  │  vminitd (PID 1)       │  │  │  │
│                                │  │  │  • GRPC server :1024   │  │  │  │
│                                │  │  │  • Process supervisor  │  │  │  │
│                                │  │  │  • Mount/network ops   │  │  │  │
│                                │  │  └──────────┬─────────────┘  │  │  │
│                                │  │             │                │  │  │
│                                │  │  ┌──────────▼─────────────┐  │  │  │
│                                │  │  │  Container Process     │  │  │  │
│                                │  │  │  (your workload)       │  │  │  │
│                                │  │  └────────────────────────┘  │  │  │
│                                │  └──────────────────────────────┘  │  │
│                                └────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────────────────┘

Component Overview

VZ Framework Integration

The host uses these Virtualization.framework APIs:

Communication Flow

Host                              Guest (vminitd)
────                              ───────────────

1. Create VM config
2. Boot VM ──────────────────────▶ Boot kernel
                                   Mount /proc, /sys, /run
                                   Start GRPC server on vsock:1024

3. Connect to vsock:1024 ────────▶ Accept connection
4. GRPC: CreateProcess ──────────▶ Store process config
5. GRPC: StartProcess ───────────▶ Fork/exec, return PID
                                   Relay stdio via vsock
6. GRPC: WaitProcess ────────────▶ Wait for exit, return code
7. Stop VM

Project Structure

vz-container/
├── main.go              # CLI entry point
├── commands.go          # Command implementations (run, ps, stop, etc.)
├── vm.go                # VM configuration and lifecycle
├── run.go               # Container run logic
├── blocks.go            # Dispatch queue and runloop helpers
├── objc_helpers.go      # VZ API helpers (CreateVMWithQueue, etc.)
├── filemount.go         # Host directory mount handling
├── vsock.go             # Vsock communication helpers
├── doc.go               # Package documentation
├── entitlements.plist   # Code signing entitlements
├── Makefile             # Build targets
│
├── cmd/vminitd/         # Guest init daemon
│   ├── main.go          # Init entry point
│   ├── server.go        # GRPC service implementation
│   ├── supervisor.go    # Process management
│   └── doc.go           # Package documentation
├── cmd/vmexec/          # Guest execution helper
│
└── proto/               # GRPC definitions
    ├── sandbox.proto    # Service definition
    ├── sandbox.pb.go    # Generated message types
    └── sandbox_grpc.pb.go # Generated GRPC client/server

Requirements

• macOS 13+ (Ventura or later)
• Apple Silicon (arm64)
• Go 1.21+
• Linux kernel Image with virtio support (for guest)

Building

# Build vz-container (host binary)
go build -o vz-container .

# Sign with virtualization entitlement
codesign --entitlements entitlements.plist -s - ./vz-container

# Build vminitd (guest binary) and create initrd
make all

make all does not fetch a guest kernel. Before booting or running containers, install one with make kernel or place a compatible raw Linux kernel Image at ~/.vz-container/kernel/kata-vmlinux or ~/.vz-container/kernel/vmlinuz. Apple's Virtualization framework does not boot EFI/PE vmlinuz files through VZLinuxBootLoader. The bundled kernel bootstrap refuses to install an unverifiable download; when the mirror has no .sha256 sidecar, pass a trusted ALPINE_APK_SHA256.

# Or step by step:
make proto      # Generate GRPC code
make vminitd    # Build Linux/arm64 binary
make initrd     # Create initrd.img

Usage

Boot a VM directly (for testing)

# Boot with kernel and initrd
./vz-container boot -kernel /path/to/vmlinuz -initrd /path/to/initrd.img

# With custom options
./vz-container boot \
  -kernel ~/.vz-container/kernel/kata-vmlinux \
  -initrd initrd.img \
  -m 1024 \
  -c 2 \
  -cmdline "console=hvc0 rdinit=/init"

Run a container

# Run alpine with shell
./vz-container run alpine /bin/sh

# Run with port mapping
./vz-container run -p 8080:80 nginx

# Run with volume mount
./vz-container run -v /host/path:/container/path alpine

# Run with network and guest configuration
./vz-container run --net nat --dns 1.1.1.1 --hostname web alpine

# Give the container its own address instead of NAT + port forwarding
./vz-container run --network host-only alpine
./vz-container run --network bridged:en0 alpine

# Use an ext4 block-device rootfs instead of the default virtiofs share
# (built by the pure-Go formatter; no external tools needed)
./vz-container run --rootfs ext4 alpine

# Run detached
./vz-container run -d redis

Rootfs modes

--rootfs selects how the container root filesystem is presented to the guest:

• dir (default): the unpacked OCI layers are shared with the guest over virtiofs (tag containerfs) and mounted at /rootfs. No external tools, works on a stock macOS host.
• ext4: an ext4 block-device image is built from the unpacked rootfs by the pure-Go formatter in internal/ext4 (unprivileged, no mount, no external tools) and attached as /dev/vda. This matches Apple's block-rootfs model and gives conventional writable-layer semantics. The formatter writes a conservative, read-write-mountable layout (classic indirect block maps, 256-byte inodes, FILETYPE + LARGE_FILE features, no journal) preserving mode, ownership, nanosecond timestamps, symlinks, and hardlinks. If it ever fails and a mkfs.ext4/mke2fs is on PATH, the build falls back to it. The image is cached as rootfs.ext4 next to the image and reused across runs.

Networking modes

--network (alias --net) selects the VZ network attachment:

• nat (default): shared NAT network. Combine with -p host:container to forward host ports to the container. Stable on all supported macOS versions.
• host-only, bridged:IFACE, vmnet (🧪 experimental): the container receives its own reachable address; host-side port forwarding does not apply and -p is rejected with a clear error. These modes depend on host capabilities and entitlements and are not yet verified end-to-end:
  • host-only and bridged:IFACE work on recent macOS but may fail on restricted accounts or when the named interface is unavailable; a failed attachment surfaces as a VZ "Internal Virtualization error" at VM start.
  • vmnet additionally requires a newer macOS and a networking entitlement (the exact entitlement key still needs confirmation against Apple's Virtualization.framework docs); when unavailable it fails at start.
• none: no network device.

Use nat with -p for predictable, well-tested networking; reach for the dedicated-address modes only when you understand these constraints.

Container management

# List running containers
./vz-container ps

# List all containers
./vz-container ps -a

# Stop a container
./vz-container stop <container-id>

# Remove a container
./vz-container rm <container-id>

# Execute in a running container
./vz-container exec -i -t <container-id> /bin/sh

# Show resource statistics
./vz-container stats <container-id>

# Copy a regular file
./vz-container cp ./file.txt <container-id>:/tmp/file.txt

GRPC API

vminitd exposes a GRPC service on vsock port 1024:

Process Lifecycle

• CreateProcess - Configure a new process
• StartProcess - Start and return PID
• WaitProcess - Wait for exit with optional timeout
• KillProcess - Send signal to process
• DeleteProcess - Clean up process resources
• ResizeProcess - Resize PTY (terminal mode)
• CloseProcessStdin - Signal EOF on stdin

Filesystem

• Mount - Mount filesystem
• Umount - Unmount filesystem
• Mkdir - Create directory
• WriteFile - Write a file in the guest
• Copy - Stream files or directories between host and guest

Environment

• Setenv / Getenv - Environment variables
• Sysctl - Kernel parameters via /proc/sys
• SetTime - Set system time
• SetupEmulator - Configure a binary-format emulator such as Rosetta

Networking

• IpLinkSet - Configure interface (up/down, MTU)
• IpAddrAdd - Add IP address
• IpRouteAddLink - Add link-scoped route
• IpRouteAddDefault - Set default gateway
• ConfigureDns - Write /etc/resolv.conf
• ConfigureHosts - Write /etc/hosts entries

Runtime

• ContainerStatistics - Report cgroup-backed CPU, memory, I/O, and PID stats
• ProxyVsock / StopVsockProxy - Bridge vsock ports and guest Unix sockets

Utility

• Kill - Send signal by PID
• Sync - Filesystem sync

Design Decisions

Why VMs instead of containers?

Traditional Linux containers (namespaces + cgroups) aren't available on macOS. Using Virtualization.framework provides:

1. Strong isolation - Hardware-enforced VM boundaries
2. Full Linux compatibility - Real Linux kernel, not emulation
3. Apple Silicon optimization - Native hypervisor performance
4. Security - Each container is a separate VM

Why pure Go?

• No CGO complexity - Simpler builds and cross-compilation
• Single binary - Easy distribution
• purego magic - Direct Objective-C runtime calls without C

Why vsock?

• Fast - Direct VM-to-host communication, no network stack
• Secure - No network exposure, kernel-managed
• Simple - Just port numbers, no IP configuration

Kernel Setup

vz-container needs a Linux kernel with these options:

CONFIG_VIRTIO=y
CONFIG_VIRTIO_PCI=y
CONFIG_VIRTIO_MMIO=y
CONFIG_VIRTIO_BLK=y
CONFIG_VIRTIO_NET=y
CONFIG_VIRTIO_CONSOLE=y
CONFIG_VSOCKETS=y
CONFIG_VIRTIO_VSOCKETS=y
CONFIG_9P_FS=y
CONFIG_NET_9P=y
CONFIG_NET_9P_VIRTIO=y

Fetching a kernel

The bundled bootstrap downloads and verifies an Alpine Linux virt kernel (arm64) into ~/.vz-container/kernel/vmlinuz:

make kernel            # or: ./scripts/bootstrap-kernel.sh

The script reads the package version from Alpine's published APKINDEX and verifies the downloaded .apk before installing. If the selected mirror publishes a per-file .sha256 sidecar, that digest is used. Otherwise the script refuses to install until you provide an expected package digest:

ALPINE_VERSION=3.21 \
ALPINE_APK_SHA256=<trusted linux-virt apk sha256> \
make kernel

No checksum is hardcoded, and an unverifiable download is refused rather than installed. Use FORCE=1 to refresh an existing kernel. A KERNEL.json manifest recording the source, version, and extracted kernel sha256 is written alongside the kernel.

Virtualization.framework may reject EFI-stub vmlinuz files with an "invalid boot loader" error. When that happens, use a raw arm64 Linux Image such as ~/.vz-container/kernel/kata-vmlinux, or pass an explicit kernel path to ./vz-container boot or make boot-test BOOT_KERNEL=/path/to/Image.

Alternatives:

• Kata Containers kernel
• A custom minimal kernel built with the options above
• Any kernel placed manually at ~/.vz-container/kernel/vmlinuz

Status

This table tracks implemented surfaces and local code paths. It is not a fresh live-boot certification for every host/kernel/network combination; check the build, initrd, kernel digest, and boot commands on the target machine before treating a row as deployment proof.

Alignment with Apple Containerization

vz-container tracks Apple's containerization design (one VM per container, an init daemon as PID 1, gRPC over vsock, per-container cgroups). Intentional differences that remain, deferred as follow-ups:

• Rootfs storage. Apple centers block-image workflows. vz-container defaults to unpacked OCI layers shared over virtiofs and offers --rootfs ext4 as an opt-in block-device mode built by the pure-Go formatter in internal/ext4, with mkfs.ext4/mke2fs only as a fallback.

Related Projects

• Apple Containerization - Apple's Swift-based container runtime
• Lima - Linux VMs on macOS
• Colima - Container runtimes on macOS
• Finch - AWS container development tool

License

See LICENSE file.