container

package
v0.24.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 12, 2026 License: Apache-2.0 Imports: 20 Imported by: 0

Documentation

Overview

Package container is the isolation backend that runs the node stack in a privileged OCI container with systemd as PID 1. The agent unpacks the rootfs image, generates an OCI runtime bundle (config.json), and an agent-owned systemd unit supervises it with runc. The host's docker and containerd are not used.

PROVISIONAL: the Azure Container Linux node facts in the design (section 13) have not been collected. The backend is exercised only against fake runtimes and stubbed hosts, never on a real node. Comments marked PROVISIONAL name assumptions that have not been verified there.

The agent wires the backend in its mode composition table. Its operations on a live host (unpack, start, repave, reset) are composed by cmd/agent/internal/containerops, which the daemon and bootstrap use.

Index

Constants

View Source
const (
	DefaultGoneTimeout  = 30 * time.Second
	DefaultPollInterval = 500 * time.Millisecond
)

Defaults for Stop's check that the container is gone.

View Source
const (
	// ImageArchiveDir is where staged offline container image archives are
	// visible inside the node, and the base of their host staging directory.
	ImageArchiveDir = "/var/lib/unbounded/container-images"
	// ImageArchiveHostDir is the host symlink to the current staging directory.
	ImageArchiveHostDir = ImageArchiveDir + "/current"
	// KubeletDir is the kubelet's state directory, shared with the host.
	KubeletDir = "/var/lib/kubelet"
	// KernelModulesDir is the host's kernel modules directory, bound read-only
	// as the nspawn machine does so modprobe inside the node sees the host
	// kernel's modules. PROVISIONAL: Azure Container Linux may keep modules
	// elsewhere (for example under /usr/lib/modules); unverified.
	KernelModulesDir = "/lib/modules"
)

Host locations the nspawn machine also binds. They are local constants so this package does not import goalstates; a test pins them to goalstates.

View Source
const (
	// DefaultBase holds one directory per slot (the unpacked rootfs).
	DefaultBase = "/var/lib/unbounded/containers"
	// DefaultRuntimePath is the runc executable. Only runc is supported.
	DefaultRuntimePath = "/opt/unbounded/runtime/runc"
	// SystemdDir is where the agent writes the supervising unit.
	SystemdDir = "/etc/systemd/system"
)

Defaults for the host layout. The agent downloads runc into an agent-owned directory under /opt/unbounded (rootfs.InstallHostRuntime, run by the containerops composition). PROVISIONAL: that the node image allows executing from there is unverified on Azure Container Linux.

View Source
const (
	StartLimitInterval = time.Hour
	StartLimitBurst    = 10
)

The start rate limit: a unit whose start keeps failing (for example a GPU hook that cannot succeed) stops being restarted after StartLimitBurst failed starts within StartLimitInterval, and stays failed until the agent starts it again, instead of restarting forever.

View Source
const BPFFSDir = "/run/bpffs"

BPFFSDir holds one private bpffs mount per slot, at the same host path the nspawn machine of the slot uses, so each node gets its own pinned-object namespace and reset removes either mode's with one task.

View Source
const MinSystemdVersion = 254

MinSystemdVersion is the oldest systemd the container unit works on. It uses DelegateSubgroup=, RestartSteps= and RestartMaxDelaySec= (systemd 254). Older systemd ignores those keys, so runc stays in the unit's own cgroup and the container starts without the memory and io controllers.

View Source
const SELinuxLabel = "system_u:system_r:spc_t:s0"

SELinuxLabel is the process label applied when SELinux is enforcing. PROVISIONAL: spc_t is the design's assumption for Azure Container Linux; it has not been verified on a node.

StartTimeout bounds the supervising unit's start: the pre-start hook, runc starting the container, and the post-start hook's wait for the node stack and its GPU step, plus a margin. It is derived from the post-start budget so the two cannot drift apart again.

Variables

View Source
var DefaultHostFS = HostFS{
	CgroupControllers: "/sys/fs/cgroup/cgroup.controllers",
	SELinuxEnforce:    "/sys/fs/selinux/enforce",
	SELinuxContext:    "/sys/fs/selinux/context",
}

DefaultHostFS is the real host.

View Source
var MaskedUnits = []string{
	"systemd-udevd.service",
	"systemd-udevd-control.socket",
	"systemd-udevd-kernel.socket",
	"systemd-udev-trigger.service",
	"systemd-udev-settle.service",
}

MaskedUnits are masked in the container's rootfs. udev belongs to the host: the container sees the host's devices but must not run a second udev that processes their events and rewrites their nodes and permissions.

View Source
var SamePathMounts = []SameMount{
	{Path: "/run/containerd"},
	{Path: "/etc/containerd/certs.d"},
	{Path: KubeletDir, Shared: true},
	{Path: "/opt/cni"},
}

SamePathMounts are the paths shared with the host.

Pod volume mounts that the kubelet and CSI drivers make under /var/lib/kubelet must be visible on the host, so that bind is rshared and the container's root propagation is rshared too: mounts made in the container propagate back through the host's /var/lib/kubelet, which the agent makes a shared mount of its own before the container starts (see RenderUnit). The bpffs at /sys/fs/bpf is not shared: each slot gets its own (BPFFSPath), as each nspawn machine does.

PROVISIONAL: that propagation through the Azure Container Linux root mount behaves as on other distributions is unverified.

Functions

func BPFFSPath

func BPFFSPath(slot string) string

BPFFSPath is the host bpffs mount a slot's container sees at /sys/fs/bpf.

func CgroupPath

func CgroupPath(slot string) string

CgroupPath is the absolute cgroup (below the cgroup2 root) the container of a slot runs in.

func ContainerID

func ContainerID(slot string) string

ContainerID is the OCI container id for a slot.

func OtherSlot

func OtherSlot(slot string) string

OtherSlot is the blue/green slot that is not slot.

func RenderConfig

func RenderConfig(in BundleInput) ([]byte, error)

RenderConfig returns the OCI runtime config.json for the input. The output is deterministic.

func RenderUnit

func RenderUnit(s UnitSpec) []byte

RenderUnit returns the systemd unit that supervises the container.

Before runc starts, the unit makes the host's /var/lib/kubelet a shared mount (a bind of the directory on itself when nothing is mounted there), so the container's rshared bind of it joins its peer group and pod volume mounts the node stack makes there are visible on the host. It runs at every start, so it holds after a host reboot too. The slot's bpffs is mounted the same way.

The unit conflicts with the other slot's unit, so systemd never runs both slots' containers at once, and the pre-start hook refuses to start while the other slot's container is still alive.

PROVISIONAL: KillSignal SIGRTMIN+3 is the signal systemd as PID 1 treats as poweroff; runc forwards signals it receives to the container init. DelegateSubgroup= needs systemd 254. Neither has been exercised on Azure Container Linux.

func UnitName

func UnitName(slot string) string

UnitName is the agent-owned supervising unit for a slot.

func ValidSlot

func ValidSlot(slot string) error

ValidSlot reports whether slot is one of the blue/green slots.

Types

type Backend

type Backend struct {
	Log    *slog.Logger
	Layout Layout
	// RuntimePath is the runc executable (only runc is supported). Empty means
	// DefaultRuntimePath.
	RuntimePath string
	// CheckRuntime verifies the runtime executable exists before the unit is
	// started, so a missing runtime is a clear error instead of a unit systemd
	// quietly skips. Nil skips the check; the agent sets it.
	CheckRuntime func(path string) error
	// KernelRelease reports the running kernel's release string, for choosing
	// recursive read-only binds. Nil reads <SourceRoot>/proc/sys/kernel/osrelease;
	// an unknown kernel keeps plain "ro".
	KernelRelease func() string
	// HelperPath is the agent binary providing node-lifecycle hooks.
	HelperPath string
	// SystemdDir is where the unit is written. Empty means SystemdDir.
	SystemdDir string
	// Host is used by preflight and SELinux detection. Nil means DefaultHostFS.
	Host *HostFS
	// Run executes host commands. Nil uses executil.OutputCmd.
	Run Runner
	// Stream starts streaming commands (logs). Nil uses os/exec.
	Stream Streamer
	// Unpack materializes the rootfs. Nil requires the rootfs to exist already.
	Unpack Unpacker
	// StatDevice describes a granted device node. Nil stats the real device.
	StatDevice func(path string) (DeviceInfo, error)
	// MountInfoPath is read before removing a rootfs. Empty means /proc/self/mountinfo.
	MountInfoPath string
	// SourceRoot prefixes same-path source directories created by Prepare.
	// Empty means the real host root; tests use a temporary directory.
	SourceRoot string
	// WritableCheck overrides the writable path probe. Nil uses none.CheckWritable.
	WritableCheck func(paths ...string) error
	// RuntimeProvisioned tells Preflight that the agent installs the runtime
	// later in bootstrap, so a missing runtime file is expected on a clean host.
	// The runtime is still required to be executable when it is present.
	RuntimeProvisioned bool
	// RefreshGrants, when set, replaces the recorded grants at Regenerate. The
	// pre-start hook uses it to follow devices and driver libraries that a host
	// reboot or driver change moved since Prepare. It receives the grants
	// recorded by Prepare and returns the ones to render and record from now on.
	RefreshGrants func(ctx context.Context, recorded []isolation.Grant) ([]isolation.Grant, error)
	// UnitRequires lists host units the supervising unit requires and orders
	// after, such as the LocalDNS network unit.
	UnitRequires []string
	// GoneTimeout bounds how long Stop waits for the runtime to report the
	// container gone, before and again after killing it. Zero is
	// DefaultGoneTimeout.
	GoneTimeout time.Duration
	// PollInterval is the wait between runtime state checks. Zero is
	// DefaultPollInterval.
	PollInterval time.Duration
}

Backend runs the node stack in a privileged OCI container.

func NewDefault

func NewDefault(log *slog.Logger) *Backend

NewDefault returns a backend with production defaults, which is what a caller that only needs Status, Exec or Logs wants.

func (*Backend) Destroy

func (b *Backend) Destroy(ctx context.Context, r isolation.SlotRef) error

Destroy stops the container, then removes the unit, the bundle, the rootfs and the slot's bpffs mount. Nothing is removed while the runtime still has a live container for the slot: it owns its rootfs, and the binds inside its private mount namespace are not in the host mount table that removeRootfs checks, so removing the files could delete host data through them. A later destroy removes them once the container is gone. The rootfs is also kept while anything is mounted below it on the host.

func (*Backend) Exec

Exec runs a command in the container via `<runtime> exec`.

func (*Backend) Logs

Logs streams the container journal by running journalctl in the container.

func (*Backend) Mode

func (*Backend) Mode() isolation.Mode

Mode implements isolation.Backend.

func (*Backend) NodeRoot

func (b *Backend) NodeRoot(slot string) (isolation.NodeRoot, error)

NodeRoot returns the node root of a slot without preparing it.

func (*Backend) Preflight

func (b *Backend) Preflight() error

Preflight fails when the host cannot run the container backend: a missing or non-executable runtime, no cgroup v2, systemd older than MinSystemdVersion, an SELinux label the kernel does not accept while enforcing, or a read only required path. It mutates nothing.

func (*Backend) Prepare

Prepare materializes the slot: rootfs, same-path source directories, config.json, the recorded node spec and the supervising unit. It is safe to repeat.

func (*Backend) Regenerate

func (b *Backend) Regenerate(ctx context.Context, slot string) error

Regenerate re-renders config.json from the NodeSpec recorded by Prepare. The node-lifecycle pre-start hook calls it so SELinux or device changes since the last start are picked up. When RefreshGrants is set the grants it returns replace the recorded ones, and are recorded, so the next start begins from them.

func (*Backend) RenderBundle

func (b *Backend) RenderBundle(slot string, grants []isolation.Grant) error

RenderBundle writes config.json for the slot from the grants, the current SELinux state and the devices on the host. It is idempotent and atomic.

func (*Backend) Requirements

func (b *Backend) Requirements() isolation.Requirements

Requirements implements isolation.Backend. The runtime artifact is installed under /opt/unbounded by the agent; cgroup v2 and the SELinux label are checked by Preflight rather than expressed here.

func (*Backend) Restart

func (b *Backend) Restart(ctx context.Context, r isolation.SlotRef) error

Restart restarts the container only, never the host.

func (*Backend) RewriteUnit

func (b *Backend) RewriteUnit(ctx context.Context, slot string) error

RewriteUnit re-renders the supervising unit of a prepared slot and reloads systemd, without restarting a running container. It is how a moved helper path reaches the hooks of an existing unit.

func (*Backend) RuntimeStatus

func (b *Backend) RuntimeStatus(ctx context.Context, r isolation.SlotRef) (isolation.Status, error)

RuntimeStatus reports whether the OCI runtime considers the container running, without requiring the supervising unit to be active. The node-lifecycle post-start hook needs this form: it runs as ExecStartPost, when the unit is still activating and `systemctl is-active` fails, so Status would never report running from inside the hook. PROVISIONAL: the activating state during ExecStartPost is systemd's documented behavior, not observed here.

func (*Backend) SlotExists

func (b *Backend) SlotExists(slot string) bool

SlotExists reports whether the slot has a rootfs or a supervising unit on the host, even a half built one.

func (*Backend) Start

func (b *Backend) Start(ctx context.Context, r isolation.SlotRef) error

Start enables and starts the unit. Starting a running unit is a no-op. The other slot's unit is disabled first, so only this slot comes back at boot, and a failed state left by the unit's start limit is cleared, so the agent can always start a slot again.

func (*Backend) Status

Status reports running when the unit is active and the runtime reports the container running.

func (*Backend) Stop

func (b *Backend) Stop(ctx context.Context, r isolation.SlotRef) error

Stop stops and disables the unit, so the slot does not come back at boot, and returns only once the runtime reports the container gone. A container that outlives the stop is killed and deleted; one that survives that too is an error, because it still holds the host paths both slots share. Stopping a stopped container succeeds.

type BundleInput

type BundleInput struct {
	// Slot selects the slot's bpffs and cgroup.
	Slot      string
	RootfsDir string
	SELinux   SELinuxMode
	Grants    []isolation.Grant
	// Devices maps a granted device path to its node description.
	Devices map[string]DeviceInfo
	// ExtraBinds are bound after the same-path mounts and before grants.
	ExtraBinds []ExtraBind
	// RecursiveReadOnly makes read-only binds "rro" instead of "ro". With "ro"
	// a recursive bind keeps its submounts writable. The runtime (runc 1.1) and
	// the kernel (5.12, mount_setattr) must both support it, so the backend sets
	// this only after checking them.
	RecursiveReadOnly bool
}

BundleInput is everything config.json depends on, so generation is a pure function.

type DeviceInfo

type DeviceInfo struct {
	Type  string // "c" or "b"
	Major int64
	Minor int64
	Mode  uint32
}

DeviceInfo describes a host device node.

type ExtraBind

type ExtraBind struct {
	Source   string
	Target   string
	ReadOnly bool
}

ExtraBind is a host path bound into the container at a different path than the same-path contract covers, such as the staged offline image archives.

type HostFS

type HostFS struct {
	// CgroupControllers exists only on a cgroup v2 unified hierarchy.
	CgroupControllers string
	// SELinuxEnforce holds "1" when enforcing and "0" when permissive. Absent
	// means SELinux is disabled.
	SELinuxEnforce string
	// SELinuxContext is the kernel's context validation interface. Writing a
	// context to it and reading it back fails when the type is unknown.
	SELinuxContext string
}

HostFS locates the host files preflight inspects. Tests point the fields at a temporary tree; production uses DefaultHostFS.

func (HostFS) CheckLabel

func (h HostFS) CheckLabel() (retErr error)

CheckLabel verifies the kernel accepts the spc_t label. PROVISIONAL: the context interface behavior is assumed from the SELinux kernel API and has not been checked on Azure Container Linux. A failure to close the handle is reported only when every check passed, so the first failure keeps precedence.

func (HostFS) SELinuxMode

func (h HostFS) SELinuxMode() SELinuxMode

SELinuxMode reads the host SELinux state.

type Layout

type Layout struct {
	// Base is the directory holding the slots. Empty means DefaultBase.
	Base string
}

Layout is the on-disk arrangement for container slots.

func (Layout) BundleDir

func (l Layout) BundleDir(slot string) string

BundleDir is the OCI bundle directory holding config.json. It is kept apart from the rootfs so the rootfs stays a pure image tree. "bundles" is not a valid slot so the two never collide.

func (Layout) ConfigPath

func (l Layout) ConfigPath(slot string) string

ConfigPath is the generated OCI runtime config.

func (Layout) RootfsDir

func (l Layout) RootfsDir(slot string) string

RootfsDir is the unpacked rootfs of a slot, which is what the NodeRoot maps.

func (Layout) SpecPath

func (l Layout) SpecPath(slot string) string

SpecPath records the NodeSpec so the lifecycle pre-start hook can regenerate config.json without the original caller.

type Root

type Root struct {
	SlotName string
	// Dir is the unpacked rootfs.
	Dir string
	// HostRoot prefixes the same-path host directories. Empty is the real host;
	// tests use a temporary tree.
	HostRoot string
}

Root is the NodeRoot of a container slot. Most in-node paths resolve below the slot's unpacked rootfs, but the same-path contract (design invariant 3) bind-mounts host directories over some of them, for example /var/lib/kubelet and /opt/cni. A file the agent writes below the rootfs at one of those paths would be hidden by the mount, so Path maps them to the host directory itself, which is what the container sees there.

func (Root) Path

func (r Root) Path(inNode string) string

Path implements isolation.NodeRoot.

func (Root) Slot

func (r Root) Slot() string

Slot implements isolation.NodeRoot.

type Runner

type Runner = none.Runner

Runner runs a host command and returns its stdout. It is a seam for tests, shared with the none backend.

type SELinuxMode

type SELinuxMode string

SELinuxMode is the host SELinux state.

const (
	SELinuxDisabled   SELinuxMode = "disabled"
	SELinuxPermissive SELinuxMode = "permissive"
	SELinuxEnforcing  SELinuxMode = "enforcing"
)

type SameMount

type SameMount struct {
	Path   string
	Shared bool // rshared propagation
}

SameMount is a host path that must resolve identically in the container (design invariant 3).

type Streamer

type Streamer func(ctx context.Context, name string, args ...string) (io.ReadCloser, error)

Streamer starts a long running host command and returns its stdout.

type UnitSpec

type UnitSpec struct {
	Slot string
	// RuntimePath is the OCI runtime executable.
	RuntimePath string
	// BundleDir is the OCI bundle directory.
	BundleDir string
	// HelperPath is the agent binary that provides the node-lifecycle hooks.
	// Empty omits the hooks.
	HelperPath string
	// After lists host-global units to order after.
	After []string
	// Requires lists host-global units the container needs. Each is also
	// ordered before the container, as the nspawn machine's LocalDNS drop-in
	// does.
	Requires []string
}

UnitSpec is the input for the supervising unit.

type Unpacker

type Unpacker func(ctx context.Context, n isolation.NodeSpec, dir string) error

Unpacker materializes the image rootfs into dir. It is injected because the OCI unpack code lives in the phase packages, which depend on this package.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL