sandboxd

sandboxd

sandboxd is the Linux sandbox lifecycle service used by AKernel. It exposes a small gRPC API, manages sandbox resources, and runs sandboxes with gVisor, Kata Containers, Firecracker, or the optional host-kernel runc adapter.

Responsibilities

• Start, wait for, inspect, measure, and delete sandboxes.
• Prepare local, OCI, Nydus, and S3-backed rootfs and mounts.
• Extract and inject OCI image startup metadata for in-sandbox runtimes.
• Allocate cgroups and pooled TAP or ephemeral veth endpoints, and configure iptables or eBPF for NAT.
• Discover, validate, and lease scheduler-selected accelerator devices.

Architecture

gRPC service
    |
sandbox lifecycle manager
    ├── sandbox runtime adapter ──> gVisor / Kata / Firecracker / runc
    ├── image manager ────────────> distill-fs / OCI
    └── resource managers ────────> cgroup / TAP or veth / iptables or eBPF

API / CLI

The public protobuf contract is api/runtime/v1/sandbox-api.proto. Runtime isolation, host requirements, and lifecycle differences are summarized in doc/runtime.md. The inherited OCI startup contract is documented in doc/image-process.md.

The sbox binary is an administrative CLI for managing sandboxes.

NVIDIA GPU sandboxes

GPU support is experimental and currently uses gVisor runsc with nvproxy. The scheduler passes concrete node-local device IDs through StartRequest.xpu_allocations; sandboxd resolves them to NVIDIA UUIDs and maintains a local exclusive lease:

sbox start \
  --rootfs /path/to/directory-rootfs \
  --xpu-allocation gpu:0,2 \
  /bin/sleep 300

When [plugin.node_resource] is enabled, /resource includes the stable XPU inventory:

{"cpu":32,"mem":68719476736,"xpu":[{"type":"gpu","product_model":"l20","device_ids":[0,1]}]}

Kubernetes remains the default CPU and memory provider. A standalone process can report the effective limits of its own cgroup without Kubernetes access:

[plugin.node_resource]
provider = "cgroup"
sock_path = "/run/sandboxd/resource.sock"

The cgroup provider is read-only and works with both cgroup v1 and v2. It does not enable controllers or create child cgroups, including when experimental cgroup-disabled execution is selected.

See test/e2e/README.md for the validated GPU debug image and manual CUDA test.

Network backends

Sandboxd supports two NAT backends:

• iptables is the default backend.
• bpfnat is an experimental embedded TC eBPF backend for Linux 5.10 or newer.

Select the backend with plugin.network.nat_backend. See the bpfnat implementation notes for its host prerequisites and build workflow.

Sandboxd owns TAP devices in the host network namespace whose names use its tap.<encoded-ip> format and decode to an address within plugin.network.ip_range. Startup recovery may reattach, reconfigure, and reuse those devices. Other software must not create TAP devices in this reserved namespace.

An optional per-sandbox ACL uses native netfilter and ipset with the iptables backend and TC eBPF with the bpfnat backend. Both support prioritized stateful IPv4 CIDR, domain, protocol, and port rules, fragments, and managed DNS policies. See the network ACL guide for its API, host requirements, limitations, and recovery behavior.

Build and test

make
make test
make vet
make networkacl-test
make bpfnat-test

networkacl-test runs one backend-neutral conformance suite against native iptables and TC eBPF enforcement in isolated network namespaces. It covers allow and deny precedence, exact and wildcard peers, peer and sandbox ports, stateful TCP, UDP, ICMP and related errors, IPv4 fragments, DNS redirection, policy replacement, restart recovery, and policy removal. bpfnat-test adds backend-specific NAT, map lifecycle, and garbage-collection coverage.

The privileged E2E suite covers runsc, runc, Kata Containers, and Firecracker. Runsc and runc use the default image; the two VM runtimes are selected separately with their KVM artifacts:

RUNSC_BINARY=/usr/local/bin/runsc \
RUNC_BINARY=/usr/local/bin/runc \
make e2e

E2E_RUNTIME=firecracker \
FIRECRACKER_BINARY=/usr/local/bin/firecracker \
FIRECRACKER_KERNEL=/opt/firecracker/vmlinux \
FIRECRACKER_INITRD=/opt/firecracker/initrd.img \
make e2e

See test/e2e/README.md for all runtime commands and coverage, including the complete adjacent gVisor bundle required by RUNSC_BINARY. AKernel integration is validated through its all-in-one node image and standalone deployment.

Protobuf development

Protobuf generation uses a pinned Docker image defined by the Makefile and tools/protobuf.Dockerfile, independent of host-installed protobuf tools:

make protos
make check-protos

Commit the generated Go bindings with the corresponding protobuf change.

Project layout

api/                 public protobuf API and generated Go code
cmd/sandboxd/        sandboxd daemon
cmd/sbox/            administrative CLI
config/              configuration types and defaults
configs/             AKernel integration configuration templates
internal/server/     gRPC service and daemon orchestration
pkg/runtime/         sandbox runtime abstraction and runtime adapters
pkg/imagemanager/    rootfs and mount integration
pkg/networkmanager/  TAP/veth, iptables, and eBPF network integration
pkg/cgroupmanager/   transparent cgroup v1/v2 integration and cache
pkg/xpumanager/      accelerator discovery, inventory, and local leases
test/e2e/            privileged runtime E2E
tools/               pinned protobuf code-generation image

Known limitations

• Kata Containers and Firecracker require a usable /dev/kvm; nodes without KVM continue to support gVisor. Firecracker additionally requires a compatible guest kernel/initrd and the ext4 image tool. Its root filesystem may be an immutable EROFS image or a directory exported through virtio-fs; directory-backed OCI/Nydus roots require virtiofs_enabled and a compatible virtiofsd.
• NVIDIA GPU sandboxes require runsc, a directory/lisafs-backed rootfs, nvidia-container-cli, accessible NVIDIA devices and userspace driver libraries, and a host driver supported by the pinned runsc nvproxy. Kata, Firecracker, runc, MIG, fractional GPUs, and regular-file/EROFS rootfs are not supported.
• sandboxd detects the local cgroup mode at startup. Legacy and hybrid hosts use cgroup v1; unified hosts use cgroup v2. The gRPC API and resource-cache behavior are identical in both modes.
• [plugin.resource].disable_cgroup = true enables an experimental/debug compatibility mode for environments where sandboxd cannot write the delegated hierarchy. sandboxd, distill-fs, and runsc then perform no cgroup writes; runsc sandboxes inherit the sandboxd process cgroup. Per-sandbox CPU, memory, and pids requests are accepted but not enforced, per-sandbox Stats returns FailedPrecondition, and non-runsc runtimes are not advertised. Do not use this mode when per-sandbox resource isolation is required.
• To minimize sandbox startup latency, sandboxd caches and reuses physical cgroups across sandbox leases. Cumulative CPU accounting and peak-memory statistics therefore cover the physical cgroup lifetime, and reclaimable charges such as page cache may remain until kernel pressure reclaims them; CPU utilization calculated from sampling deltas and configured resource limits remain correct for the active sandbox.
• The direct OCI registry client currently skips TLS certificate verification and should only be used with trusted registries.

License

Copyright (c) 2026 Ant Group Corporation.

Licensed under the Apache License, Version 2.0. See LICENSE. Third-party Go modules are declared in go.mod and retain their respective licenses.