Documentation
¶
Overview ¶
Package containerops is the composition root for the container isolation backend. The backend package cannot import the phase packages that unpack the OS image, provision the rootfs and configure the node stack, so this package binds them to a goal state and wires them into container.Backend.
PROVISIONAL: the container mode is experimental. It is implemented against the design (section 8.3) and stubbed hosts; nothing here has run against a real runc, systemd or Azure Container Linux node. Comments marked PROVISIONAL name assumptions that are not verified.
Layout of a container node. A slot (kube1 or kube2, blue/green as for nspawn) is an unpacked OS image under /var/lib/unbounded/containers/<slot> that runs systemd as PID 1 in an OCI container supervised by an agent-owned unit. The in-rootfs configuration phases are the nspawn ones, pointed at the container root through the goal state's RootOverride and, for commands, ExecOverride (which runs "<runtime> exec"). Paths under the same-path bind mounts (/var/lib/kubelet, /opt/cni, /etc/containerd/certs.d, /run/containerd, /sys/fs/bpf) resolve to the host directory so the agent writes where the container reads; see container.Root.
Shared state. Because those directories are shared by both slots, a repave rewrites the kubelet configuration, kubeconfig and CNI binaries the previous slot also sees. Upgrade saves the shared configuration files first and puts them back before the previous slot starts again on a rollback. The previous slot's rootfs-private state (containerd data, unit drop-ins, binaries) is not touched until the new slot is healthy.
Index ¶
- Constants
- func Present() bool
- func ReconcileNVIDIA(ctx context.Context, log *slog.Logger, b *container.Backend, slot string, ...) error
- func RefreshNVIDIAGrants(h NVIDIAHooks) func(context.Context, []isolation.Grant) ([]isolation.Grant, error)
- type NVIDIAHooks
- type Node
- func (n *Node) Activate(ctx context.Context) error
- func (n *Node) Backend() *container.Backend
- func (n *Node) Configure(ctx context.Context) (containerdChanged, kubeletChanged bool, err error)
- func (n *Node) Destroy(ctx context.Context) error
- func (n *Node) GoalState() (*goalstates.MachineGoalState, error)
- func (n *Node) Prepare(ctx context.Context) (isolation.NodeRoot, error)
- func (n *Node) Restart(ctx context.Context) error
- func (n *Node) Resume(ctx context.Context) error
- func (n *Node) RewriteUnit(ctx context.Context) error
- func (n *Node) Running(ctx context.Context) (bool, error)
- func (n *Node) Start(ctx context.Context, wasRunning, containerdChanged, kubeletChanged bool) error
- func (n *Node) Stop(ctx context.Context) error
- func (n *Node) Upgrade(ctx context.Context, old *Node) (err error)
- type Overrides
- type Params
- type Teardown
Constants ¶
const Slot = goalstates.NSpawnMachineKube1
Slot is the slot a first install builds.
Variables ¶
This section is empty.
Functions ¶
func Present ¶
func Present() bool
Present is the package level form of Teardown.Present for the real host.
func ReconcileNVIDIA ¶
func ReconcileNVIDIA(ctx context.Context, log *slog.Logger, b *container.Backend, slot string, h NVIDIAHooks) error
ReconcileNVIDIA is the container node's post-start GPU flow. On a host with no NVIDIA devices it does nothing. Otherwise it waits for systemd in the container, then stops containerd and kubelet, rebuilds the NVIDIA state inside, and starts them again, as nspawn-lifecycle does for an nspawn machine.
func RefreshNVIDIAGrants ¶
func RefreshNVIDIAGrants(h NVIDIAHooks) func(context.Context, []isolation.Grant) ([]isolation.Grant, error)
RefreshNVIDIAGrants returns a container.Backend.RefreshGrants that rediscovers the NVIDIA devices and driver libraries at pre-start, so a host reboot or driver change since Prepare is followed the way nspawn's pre-start hook follows it. Grants of every other purpose are kept as recorded and in place. A node recorded without NVIDIA grants is left alone: adding a GPU to a node provisioned with the CPU rootfs needs a new rootfs, not a refresh.
Types ¶
type NVIDIAHooks ¶
type NVIDIAHooks struct {
// Resolve discovers NVIDIA state on the host. Nil is goalstates.ResolveNvidiaHost.
Resolve func(arch string) (goalstates.NvidiaHost, error)
// Execute runs a task. Nil is phases.ExecuteTask.
Execute func(ctx context.Context, log *slog.Logger, task phases.Task) error
// PollInterval and ReadyTimeout bound the wait for systemd in the container.
PollInterval, ReadyTimeout time.Duration
// ImageArchiveDir is where the staged container image archives the
// rebuild reimports are listed; see Overrides.ImageArchiveDir. Empty is the
// default.
ImageArchiveDir string
}
NVIDIAHooks are the seams of the GPU post-start flow. Zero values do the real work.
type Node ¶
type Node struct {
// contains filtered or unexported fields
}
Node is a container node bound to one configuration and slot.
func New ¶
New returns a Node. Goal state resolution is deferred to first use so that operations that never need it (Stop, Status, Destroy) work without a configuration.
func (*Node) Activate ¶
Activate brings a prepared slot up as the node's only running slot: it resets the shared kubelet state so the new kubelet bootstraps fresh credentials, configures the node, starts it and waits for kubelet to be active. The caller has stopped whatever ran before and cleaned host networking. It is the tail of Upgrade, shared with an isolation migration into this mode, and removes nothing on failure: the caller decides how to roll back.
func (*Node) Backend ¶
Backend is the container backend wired to this node. Its Unpack is bound on first use of the goal state.
func (*Node) Configure ¶
Configure writes the node configuration into the container root (and the same-path host directories) and prepares the shared LocalDNS network. It reports whether containerd or kubelet configuration differed from what was there, so a reapply against a running node restarts only what changed.
func (*Node) GoalState ¶
func (n *Node) GoalState() (*goalstates.MachineGoalState, error)
GoalState is the resolved goal state with the container overrides applied.
func (*Node) Prepare ¶
Prepare builds the slot: installs the host runtime and lifecycle helper, unpacks the image, renders the bundle and unit, and provisions the rootfs (Kubernetes, CRI and CNI binaries and OS configuration). It does not start anything. It is safe to repeat on a slot that is not running.
func (*Node) Restart ¶
Restart restarts the container only (soft reboot): the unit's hooks regenerate the bundle, systemd inside starts the enabled containerd and kubelet, and this waits for LocalDNS and an active kubelet. The host is not rebooted.
func (*Node) Resume ¶
Resume starts a slot that was stopped, for rollback: it restores the shared LocalDNS network, starts the container and waits for kubelet to be active. The slot's rootfs and unit are expected to be intact.
func (*Node) RewriteUnit ¶
RewriteUnit re-renders the supervising unit for the current helper path and reloads systemd without restarting the container.
func (*Node) Start ¶
Start brings the node up: starts the container, waits for systemd inside it, then starts containerd and kubelet and waits for kubelet to bootstrap. When the container was already running and its configuration changed, the changed service is restarted, containerd first.
func (*Node) Stop ¶
Stop gracefully stops the node: kubelet and containerd inside first so the container stop does not have to kill them, then the container. Stopping a stopped node succeeds.
func (*Node) Upgrade ¶
Upgrade moves the node from old to this node's slot (blue/green): the new slot is built beside the running old one, then the old slot is stopped, host networking is cleaned as repave does, the shared kubelet state is reset, and the new slot is configured, started and waited on. The caller has already drained the node and removed it from the cluster, as for an nspawn repave, persists the applied config after a nil return, and then retires old.
If anything fails the new slot is removed and the old one is started again, and the returned error says whether that worked. The files both slots share at the same host paths, such as the kubelet identity, are saved first (see confsnap) and restored before the old slot starts again.
type Overrides ¶
type Overrides struct {
// Layout, SystemdDir, RuntimePath and HelperPath are as in container.Backend.
// An empty HelperPath is the agent's rollback-stable lifecycle helper.
Layout container.Layout
SystemdDir string
RuntimePath string
HelperPath string
// Run executes host commands. Nil uses the real host.
Run container.Runner
// MountInfoPath is the mount table. Empty means /proc/self/mountinfo.
MountInfoPath string
// Host locates the SELinux and cgroup interfaces. Nil is the real host.
Host *container.HostFS
// Unpack, InstallRuntime, InstallHelper and ProvisionRootfs replace the
// real artifact work. Nil does it for real.
Unpack func(ctx context.Context, dir string) error
InstallRuntime func(ctx context.Context) error
InstallHelper func(ctx context.Context) error
ProvisionRootfs func(ctx context.Context) error
// PollInterval is the wait between readiness polls. Zero means one second.
PollInterval time.Duration
// GoneTimeout bounds the wait for a stopped container to be gone; see
// container.Backend.GoneTimeout. PollInterval is its poll interval too.
GoneTimeout time.Duration
// ReadyTimeout bounds the wait for systemd in the container. Zero means two
// minutes.
ReadyTimeout time.Duration
// ImageArchiveDir is the host directory listed for the staged container
// image archives to import after containerd starts. Empty means
// goalstates.ContainerImageArchiveHostDir, the directory the container
// bind-mounts; the archives are imported at their path inside it.
ImageArchiveDir string
}
Overrides are the container-specific seams of Params.
type Params ¶
type Params struct {
Log *slog.Logger
// Config is the agent configuration goal states are resolved from.
Config *config.AgentConfig
// Downloads optionally overrides artifact download sources.
Downloads *goalstates.DownloadOverrides
// Slot is kube1 or kube2. Empty means kube1.
Slot string
// GoalState, when set, is used instead of resolving one. Bootstrap passes
// the goal state it has already resolved so GPU and device discovery happens
// once.
GoalState *goalstates.MachineGoalState
// Overrides and Hooks are left zero in production, which is the real host.
// Tests set them to run a node against a temporary tree and a fake host.
Overrides
hostnode.Hooks
}
Params bind a Node to one agent configuration and one slot.
type Teardown ¶
type Teardown struct {
Log *slog.Logger
// Layout, SystemdDir, RuntimePath and Run are as in Params.
Layout container.Layout
SystemdDir string
RuntimePath string
Run container.Runner
// Root prefixes every host path the teardown removes outside the layout and
// systemd directory. Empty is the real host; tests use a temporary tree.
Root string
// MountInfo is the mount table. Empty means /proc/self/mountinfo.
MountInfo string
// GoneTimeout and PollInterval are as in Params.
GoneTimeout time.Duration
PollInterval time.Duration
}
Teardown removes a container node. Its tasks slot into the same positions in the reset order as the nspawn machine tasks they replace: stop, then config, then the node itself, with the host-global cleanup the caller owns in between and after.
func (*Teardown) Destroy ¶
Destroy returns the task that removes the nodes: for each slot the unit, bundle and rootfs through the backend, then the shared data the node left on the host (pod mounts, kubelet state, CNI binaries and configuration), the layout directory and the host runtime.
The shared data goes only once every slot is destroyed, which the backend reports only after the runtime confirmed its container gone: a container still running holds those paths, and its pod volumes are mounted below /var/lib/kubelet. The mounts there are unmounted after the slots are gone, so none of them is recreated behind the unmount, and the kubelet directory is kept if one stays, since RemoveAll would delete a volume's data through it.
func (*Teardown) Present ¶
Present reports whether the host has any trace of a container node: the layout directory or an agent-owned unit. Reset uses it to choose the container teardown even after the applied config is gone.
func (*Teardown) RemoveConfig ¶
RemoveConfig returns the task that removes the configuration the agent wrote outside the slots: the host kernel modules list (written by bootstrap).
Source Files