goalstates

package
v0.24.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 12, 2026 License: Apache-2.0 Imports: 31 Imported by: 0

Documentation

Index

Constants

View Source
const (
	// ConfigDir is the host-level configuration directory for unbounded-kube.
	ConfigDir = "/etc/unbounded/kube"

	// AgentConfigDir is the host-level configuration directory for the
	// unbounded-agent. Applied config files are persisted here.
	AgentConfigDir = "/etc/unbounded/agent"

	SystemdNSpawnDir = "/etc/systemd/nspawn"
	SystemdSystemDir = "/etc/systemd/system"
	BPFFSMountDir    = "/run/bpffs"

	// DaemonUnit is the systemd unit name for the unbounded-agent daemon.
	DaemonUnit = "unbounded-agent-daemon.service"

	// DaemonRecoveryUnit is the systemd recovery unit for the agent daemon.
	DaemonRecoveryUnit = "unbounded-agent-daemon-recovery.service"

	// NFTablesFlushUnit is the oneshot unit that flushes the host's nftables
	// ruleset before the node starts, on every boot while it is enabled.
	NFTablesFlushUnit = "nftables-flush.service"
	// NFTablesClearPath is the rules file NFTablesFlushUnit loads.
	NFTablesClearPath = ConfigDir + "/nftables-clear.nft"

	// The agent's host-side files under hostroot.LegacyPath, where agents
	// released before hostroot.Path installed them.
	//
	// Deprecated: on hosts installed since, these files are elsewhere. Use
	// ResolvedAgentUpgradePaths and ResolveHostPaths, which follow the host root.
	DaemonBinaryPath = "/usr/local/bin/unbounded-agent"
	// Deprecated: use ResolvedAgentUpgradePaths.
	DaemonBinaryBluePath = "/usr/local/bin/unbounded-agent-blue"
	// Deprecated: use ResolvedAgentUpgradePaths.
	DaemonBinaryGreenPath = "/usr/local/bin/unbounded-agent-green"
	// Deprecated: use ResolvedAgentUpgradePaths.
	DaemonBinaryCurrentPath = "/usr/local/bin/unbounded-agent-current"
	// Deprecated: use ResolvedAgentUpgradePaths.
	DaemonBinaryLastGoodPath = "/usr/local/bin/unbounded-agent-last-good"
	// Deprecated: use ResolveHostPaths.
	NSpawnLifecycleBinaryPath = "/usr/local/bin/unbounded-agent-nspawn-lifecycle"
	// Deprecated: use ResolveHostPaths.
	DaemonRecoveryScriptPath = "/usr/local/bin/unbounded-agent-daemon-recovery.sh"

	// AllocationLaunchRefusedExitCode is the code the agent's start guards,
	// the nspawn pre-start and the none-mode kubelet launch check, exit with
	// when the allocation launch guard refuses for good. Both guard units name
	// it in RestartPreventExitStatus= so a refused start fails rather than
	// loops. It must never be in SuccessExitStatus=: the node would then start.
	//
	// 77 is EX_NOPERM.
	AllocationLaunchRefusedExitCode = 77

	// NSpawnPreStartRefusedExitCode is AllocationLaunchRefusedExitCode, as the
	// nspawn pre-start exits with it.
	NSpawnPreStartRefusedExitCode = AllocationLaunchRefusedExitCode

	DaemonAgentUpgradeSignalPath = AgentConfigDir + "/agent-upgrade-signal"
	DaemonAgentUpgradeLockPath   = "/run/unbounded-agent-upgrade.lock"

	EnvDaemonBinary                 = "UNBOUNDED_AGENT_DAEMON_BINARY"
	EnvDaemonBinaryBlue             = "UNBOUNDED_AGENT_DAEMON_BINARY_BLUE"
	EnvDaemonBinaryGreen            = "UNBOUNDED_AGENT_DAEMON_BINARY_GREEN"
	EnvDaemonBinaryCurrent          = "UNBOUNDED_AGENT_DAEMON_BINARY_CURRENT"
	EnvDaemonBinaryLastGood         = "UNBOUNDED_AGENT_DAEMON_BINARY_LAST_GOOD"
	EnvDaemonAgentUpgradeSignalPath = "UNBOUNDED_AGENT_DAEMON_AGENT_UPGRADE_SIGNAL_PATH"
)
View Source
const (
	NSpawnMachineKube1 = "kube1"
	NSpawnMachineKube2 = "kube2"
)

NSpawn machine names used for alternating in-place upgrades.

  • NSpawnMachineKube1 is the initial (default) machine name.
  • NSpawnMachineKube2 will be used for the next upgrade cycle.

The pattern alternates between the two so an in-place upgrade can bring up the new machine before tearing down the old one:

kube1  ← initial
kube2  ← after operation 1
kube1  ← after operation 2
…
View Source
const (
	ContainerdVersion = "2.1.8"
	RunCVersion       = "1.5.0"
	CNIPluginVersion  = "1.5.1"

	ContainerdMetricsAddress = "0.0.0.0:10257"
	SandboxImage             = "mcr.microsoft.com/oss/v2/kubernetes/pause:3.9"
	KubeProxyImageRepository = "mcr.microsoft.com/oss/v2/kubernetes/kube-proxy"

	// OfflineArtifactArchiveHostDir stores HTTPS offline artifact archives
	// after download and extraction.
	OfflineArtifactArchiveHostDir = "/var/lib/unbounded/offline-artifacts"

	// ContainerImageArchiveDir is the path inside the nspawn machine where
	// staged container image archives are mounted.
	ContainerImageArchiveDir = "/var/lib/unbounded/container-images"
	// ContainerImageArchiveHostDir is the stable host-side symlink bind-mounted
	// read-only at ContainerImageArchiveDir inside nspawn machines.
	ContainerImageArchiveHostDir = ContainerImageArchiveDir + "/current"
	// ContainerImageArchiveHostSourceDir stores source-specific archive staging
	// directories pointed to by ContainerImageArchiveHostDir.
	ContainerImageArchiveHostSourceDir = ContainerImageArchiveDir

	CNIBinDir    = "/opt/cni/bin"
	CNIConfigDir = "/etc/cni/net.d"

	// BinDir is the standard binary directory relative to the machine root.
	// Use filepath.Join(machineDir, BinDir) for host-side rootfs paths, or
	// "/"+BinDir for absolute paths inside the running machine container.
	BinDir = "usr/local/bin"

	// NvidiaHostLibDir is the base directory inside the nspawn container where
	// host NVIDIA library directories are bind-mounted read-only. Each unique
	// host directory gets a numbered subdirectory (e.g. /run/host-nvidia/0/).
	NvidiaHostLibDir = "/run/host-nvidia"
	// NvidiaHostI386LibDir is the base directory for optional i386 NVIDIA
	// library bind mounts.
	NvidiaHostI386LibDir = "/run/host-nvidia-i386"
	// NvidiaHostBinDir exposes the host directory containing NVIDIA helper binaries.
	NvidiaHostBinDir = "/run/host-nvidia-bin"
	// NvidiaDriverDir is the driver-root layout used by NVIDIA tooling and
	// device plugins inside the nspawn machine.
	NvidiaDriverDir = "/run/nvidia/driver"

	NvidiaContainerRuntimePath = "/usr/bin/nvidia-container-runtime"
	NvidiaRuntimeClassName     = "nvidia"
	NvidiaCTKPath              = "/usr/bin/nvidia-ctk"

	// CDISpecDir is the directory where CDI specifications are stored inside the
	// nspawn machine. containerd reads specs from this directory when CDI is enabled.
	CDISpecDir = "/etc/cdi"

	// CDISpecFile is the path to the NVIDIA CDI specification file inside the
	// nspawn machine. This is generated by nvidia-ctk cdi generate.
	CDISpecFile = "/etc/cdi/nvidia.yaml"

	// Default OCI images for the nspawn rootfs when no image is explicitly
	// configured or set via AGENT_OCI_IMAGE.
	DefaultOCIImage                  = "ghcr.io/azure/agent-ubuntu2404:v20261031"
	DefaultNvidiaOCIImage            = "ghcr.io/azure/agent-ubuntu2404-nvidia:v20261031"
	DefaultUbuntu2604OCIImage        = "ghcr.io/azure/agent-ubuntu2604:v20261031"
	DefaultUbuntu2604NvidiaOCIImage  = "ghcr.io/azure/agent-ubuntu2604-nvidia:v20261031"
	DefaultAzureLinux3OCIImage       = "ghcr.io/azure/agent-azlinux3:v20261031"
	DefaultAzureLinux3NvidiaOCIImage = "ghcr.io/azure/agent-azlinux3-nvidia:v20261031"

	SystemdUnitContainerd      = "containerd.service"
	ContainerdConfigPath       = "/etc/containerd/config.toml"
	ContainerdConfDropInDir    = "/etc/containerd/conf.d"
	NvidiaRuntimeDropInPath    = ContainerdConfDropInDir + "/99-nvidia-runtime.toml"
	ContainerdCertsDir         = "/etc/containerd/certs.d"
	ContainerdDefaultHostsDir  = ContainerdCertsDir + "/_default"
	ContainerdDefaultHostsPath = ContainerdDefaultHostsDir + "/hosts.toml"

	SystemdUnitKubelet             = "kubelet.service"
	KubeletConfigurationPath       = "/var/lib/kubelet/config.yaml"
	KubeletKubeconfigPath          = "/var/lib/kubelet/kubeconfig"
	KubeletBootstrapKubeconfigPath = "/var/lib/kubelet/bootstrap-kubeconfig"
	KubeletPKIDir                  = "/etc/kubernetes/pki"
	KubeletAPIServerCACertPath     = "/etc/kubernetes/pki/apiserver-client-ca.crt"
	KubeletServiceDropInDir        = "/etc/systemd/system/kubelet.service.d"
	KubeletStaticPodManifestsDir   = "/etc/kubernetes/manifests"
	LocalDNSResolvConfPath         = "/etc/unbounded/localdns/resolv.conf"
)
View Source
const (
	CoreDNSVersion            = "1.12.3"
	LocalDNSInterfaceName     = "localdns"
	LocalDNSCorefilePath      = "/etc/unbounded/localdns/Corefile"
	LocalDNSUpstreamsPath     = "/etc/unbounded/localdns/node-upstreams"
	LocalDNSServiceUnit       = "localdns.service"
	LocalDNSSliceUnit         = "localdns.slice"
	LocalDNSReadinessPort     = 8181
	LocalDNSMetricsPort       = 9253
	LocalDNSRuleComment       = "unbounded-localdns: skip conntrack"
	LocalDNSNFTTable          = "unbounded_localdns"
	LocalDNSNetworkUnit       = "unbounded-localdns-network.service"
	LocalDNSSupervisorPath    = "/usr/local/libexec/unbounded-localdns-supervisor"
	LocalDNSCoreDNSBinaryPath = "/usr/local/bin/coredns"
)
View Source
const AgentUpgradeBinaryName = "unbounded-agent"
View Source
const OfflineArtifactManifestFileName = bootstrapartifacts.ManifestFileName

OfflineArtifactManifestFileName is the manifest file name in an offline artifact bundle.

View Source
const RunCLibpathrsFallbackVersion = "1.4.3"

RunCLibpathrsFallbackVersion is the runc release used when the host kernel release cannot be parsed by libpathrs. runc 1.5 links libpathrs, which panics on startup for such kernels, so every container create fails. runc 1.4 does not link libpathrs.

Variables

View Source
var ErrChecksumMismatch = errors.New("applied config checksum mismatch")

ErrChecksumMismatch is returned when the sidecar checksum does not match the config file content, indicating possible on-disk corruption.

View Source
var ErrNVIDIAStateUnavailable = errors.New("required NVIDIA host state is unavailable")

ErrNVIDIAStateUnavailable indicates that discovered NVIDIA devices do not have the complete host state needed for safe startup.

Functions

func AdditionalHostMountGrants added in v0.23.0

func AdditionalHostMountGrants(mounts []config.AdditionalHostMount) []isolation.Grant

AdditionalHostMountGrants converts configured host mounts to grants.

func AlternateMachine

func AlternateMachine(current string) string

AlternateMachine returns the other machine name in the pair. kube1 -> kube2, kube2 -> kube1.

func AppliedConfigChecksumPath

func AppliedConfigChecksumPath(machineName string) string

AppliedConfigChecksumPath returns the path to the SHA-256 sidecar file for the given nspawn machine's applied config, e.g. /etc/unbounded/agent/kube1-applied-config.json.sha256.

func AppliedConfigName added in v0.24.0

func AppliedConfigName(mode isolation.Mode, slot string) string

AppliedConfigName is the name the applied config of slot is stored under (AppliedConfigPath of the name) for a node in mode. An nspawn node keeps the slot itself, kube1 or kube2, which is what every agent has always read. A node in any other mode uses "<mode>-<slot>". Agents that predate that mode read only kube1 and kube2, so they find no applied config on such a host and refuse to operate it, instead of reading a host-native node as an nspawn machine and running nspawn operations against it.

func AppliedConfigPath

func AppliedConfigPath(machineName string) string

AppliedConfigPath returns the path to the applied config file for the given nspawn machine name, e.g. /etc/unbounded/agent/kube1-applied-config.json.

func BPFFSMountPath added in v0.1.11

func BPFFSMountPath(machineName string) string

BPFFSMountPath returns the host-side bpffs mount path for an nspawn machine.

func ComputeChecksum

func ComputeChecksum(data []byte) string

ComputeChecksum returns the lowercase hex-encoded SHA-256 digest of data.

func ConfigRegenerationUnit added in v0.2.4

func ConfigRegenerationUnit(machineName string) string

ConfigRegenerationUnit returns the systemd unit that regenerates host-side configuration before the named machine starts.

func ContainerImageArchivePath added in v0.1.21

func ContainerImageArchivePath(index int) string

ContainerImageArchivePath returns the path inside the nspawn machine where a preloaded container image archive is staged before importing into containerd.

func HostDistroIsImageManaged added in v0.9.0

func HostDistroIsImageManaged(hostDistro string) bool

HostDistroIsImageManaged reports whether a detected host distro is image-managed. Split out from HostIsImageManaged so it can be tested without a real /etc/os-release.

func HostIsImageManaged added in v0.9.0

func HostIsImageManaged() bool

HostIsImageManaged reports whether this host's OS content is delivered as an image rather than as packages.

On such a host the required tools are a property of the image: /usr is a read-only dm-verity mount, so nothing can be installed into it, and the presence of a package manager binary says nothing about whether package installation is supported. Callers use this to validate prerequisites and report a missing one, instead of trying to remediate it.

func KubeProxyImage added in v0.1.20

func KubeProxyImage(kubernetesVersion string) string

func NVIDIAStateAvailable added in v0.2.4

func NVIDIAStateAvailable(nvidia NvidiaHost) bool

NVIDIAStateAvailable reports whether discovery produced all host state needed to render mounts and perform in-machine setup.

func PathsExcluding added in v0.23.0

func PathsExcluding(paths, excluded []string) []string

PathsExcluding returns paths without the entries in excluded, preserving order. It returns paths unchanged when either input is empty.

func RenderOfflineSource added in v0.1.21

func RenderOfflineSource(sourceTemplate, kubernetesVersion string) (string, error)

RenderOfflineSource renders an OfflineArtifacts.Source template for the given Kubernetes version.

func ResolveDownloadOverridesWithOfflineArtifacts added in v0.1.21

func ResolveDownloadOverridesWithOfflineArtifacts(ctx context.Context, cfg *config.AgentConfig, downloads *DownloadOverrides) (*DownloadOverrides, *ContainerImageArchiveStaging, error)

ResolveDownloadOverridesWithOfflineArtifacts resolves AgentConfig.OfflineArtifacts and returns download overrides plus container image archive staging metadata that point at the offline artifact source. When OfflineArtifacts is not configured, the input downloads are returned unchanged and staging points at the host-side empty archive directory.

func ResolveOCIImage

func ResolveOCIImage(log *slog.Logger, configImage string, nvidiaGPUAvailable bool) string

ResolveOCIImage determines the OCI image to use for the nspawn rootfs.

Priority (highest to lowest):

  1. configImage from the agent config
  2. AGENT_DISABLE_OCI_IMAGE env var (truthy value disables OCI, returns "")
  3. AGENT_OCI_IMAGE env var
  4. Built-in default selected by host distro and GPU presence

func VerifyChecksum

func VerifyChecksum(data []byte, checksumPath string) error

VerifyChecksum compares the SHA-256 digest of data against the content of the sidecar checksum file at checksumPath.

Integrity assumptions:

  • Each file (config and sidecar) is written atomically via renameio, so an individual file is never half-written.
  • A missing sidecar is not an error: the config may have been written by an older agent version that did not produce checksums, or the sidecar write may not have completed before a crash. In this case VerifyChecksum returns nil so the caller can proceed.
  • A present sidecar whose digest does not match the config content indicates on-disk corruption (e.g. bitflip). This returns ErrChecksumMismatch.

Types

type AMDHost added in v0.1.19

type AMDHost struct {
	// GPUDevicePaths lists AMD GPU device paths discovered on the host
	// (e.g. /dev/kfd, /dev/dri/card*, /dev/dri/renderD*). When non-empty the
	// nspawn configuration will bind-mount these devices and grant cgroup access
	// so the AMD Kubernetes device plugin can detect GPUs inside the machine.
	GPUDevicePaths []string

	// SysFSPaths lists host sysfs paths the AMD Kubernetes device plugin reads
	// to discover AMD GPUs and KFD topology inside the nspawn machine.
	SysFSPaths []string
}

AMDHost aggregates AMD GPU host state discovered at agent startup.

func ResolveAMDHost added in v0.1.19

func ResolveAMDHost() AMDHost

ResolveAMDHost probes the host for AMD GPU device nodes. Returns an empty struct when the host does not have AMD GPUs or the driver is not loaded.

func (AMDHost) Grants added in v0.23.0

func (a AMDHost) Grants(exclude []string) []isolation.Grant

Grants returns the backend-neutral resource requests for the discovered AMD state, skipping any device path in exclude (devices NVIDIA already owns).

type AgentUpgradePaths added in v0.1.4

type AgentUpgradePaths struct {
	BinaryPath        string
	BluePath          string
	GreenPath         string
	CurrentPath       string
	LastGoodPath      string
	SignalPath        string
	CurrentTargetPath string
}

AgentUpgradePaths describes the host-side blue-green agent binary layout.

func PlannedAgentUpgradePaths added in v0.13.0

func PlannedAgentUpgradePaths() (AgentUpgradePaths, error)

PlannedAgentUpgradePaths returns the paths ResolvedAgentUpgradePaths will return once the host root is migrated, without migrating it; see PlannedHostPaths.

func ResolvedAgentUpgradePaths added in v0.1.4

func ResolvedAgentUpgradePaths() (AgentUpgradePaths, error)

ResolvedAgentUpgradePaths returns the host-side agent binary paths under the resolved host root, after applying environment overrides. Overrides name a specific file and so take precedence over the root.

The AgentUpgrade signal path is not under the host root. It is state about an upgrade rather than part of the installed layout, and lives in the agent config directory.

func (AgentUpgradePaths) InitialDaemonBinaryTarget added in v0.1.4

func (p AgentUpgradePaths) InitialDaemonBinaryTarget() (string, error)

InitialDaemonBinaryTarget returns the first executable binary that can seed the current daemon binary link.

func (AgentUpgradePaths) NextTargetPath added in v0.1.4

func (p AgentUpgradePaths) NextTargetPath() string

NextTargetPath returns the inactive blue-green binary path.

type AppliedConfigLocation added in v0.24.0

type AppliedConfigLocation struct {
	// Name is the stem passed to AppliedConfigPath.
	Name string
	// Slot is the blue/green slot the config belongs to.
	Slot string
	// Mode is the isolation mode a config stored under Name runs in.
	Mode isolation.Mode
}

AppliedConfigLocation is one place an applied config can be stored: the file name stem and the mode a config stored there must record.

func AppliedConfigLocations added in v0.24.0

func AppliedConfigLocations() []AppliedConfigLocation

AppliedConfigLocations lists every place this agent looks for the applied config of the running node, in search order: the nspawn slots first, as older agents search them, then the slots of the other modes.

type ContainerImageArchiveStaging added in v0.1.21

type ContainerImageArchiveStaging struct {
	// HostDir is the source-specific host directory containing staged archives.
	HostDir string
	// URLs lists archive sources to download into HostDir.
	URLs []string
}

ContainerImageArchiveStaging describes host-side staged container image archives.

type Containerd

type Containerd struct {
	SandboxImage      string
	ContainerdBinPath string
	RuncBinaryPath    string
	CNIBinDir         string
	CNIConfDir        string
	MetricsAddress    string
	NvidiaRuntime     NvidiaRuntime
}

Containerd describes the containerd configuration goal state.

func ResolveContainerd

func ResolveContainerd(opts ContainerdOptions) Containerd

ResolveContainerd returns the containerd configuration goal state.

type ContainerdOptions added in v0.2.4

type ContainerdOptions struct {
	SandboxImage   string
	NvidiaRequired bool
	// NvidiaToolkit locates the toolkit for the nvidia runtime handler. The
	// zero value is the default install paths.
	NvidiaToolkit NvidiaToolkit
}

ContainerdOptions controls containerd goal-state resolution.

type DownloadOverrides added in v0.1.3

type DownloadOverrides struct {
	// Kubernetes overrides the source for kubelet/kubectl/kube-proxy.
	Kubernetes *DownloadSource
	// Containerd overrides the source for the containerd release tarball.
	Containerd *DownloadSource
	// Runc overrides the source for the runc binary.
	Runc *DownloadSource
	// CNI overrides the source for the CNI plugins release tarball.
	CNI *DownloadSource
	// Crictl overrides the source for the crictl release tarball.
	Crictl *DownloadSource
	// CoreDNS overrides the source for the CoreDNS release tarball.
	CoreDNS *DownloadSource
}

DownloadOverrides optionally overrides the upstream download sources for binaries the agent installs into the nspawn rootfs. Each field is optional; nil or zero-value fields fall back to the compiled-in defaults.

type DownloadSource added in v0.1.3

type DownloadSource struct {
	BaseURL string
	URL     string
	Version string
}

DownloadSource configures the override for a single binary download. BaseURL replaces the upstream host + path prefix; URL replaces the entire URL template. When both are unset the default template is used. Version overrides the version that would otherwise be derived from the cluster Kubernetes version or compiled-in defaults.

type Gantry added in v0.1.23

type Gantry struct {
	Disabled bool
}

func ResolveGantry added in v0.1.23

func ResolveGantry(cfg *config.GantryConfig) Gantry

type HostDevices added in v0.1.19

type HostDevices struct {
	// KVM holds the KVM character device path (/dev/kvm) when present.
	KVM []string
	// MSHV holds the Microsoft Hypervisor device path (/dev/mshv) when
	// present. Virtual machine monitors such as Cloud Hypervisor use it in
	// place of /dev/kvm when Linux runs as the MSHV root partition.
	MSHV []string
	// Network holds host networking device paths used by virtualization
	// workloads (for example /dev/net/tun and /dev/vhost-net) when present.
	Network []string
	// Block holds storage block device node paths derived from
	// /sys/class/block (whole disks and partitions, excluding virtual
	// devices such as loop/ram/zram).
	Block []string
	// DeviceDirectories holds whole-directory device mounts (for example
	// /dev/infiniband) that are bind-mounted as directories, with cgroup
	// device permissions covering nodes created after the machine starts.
	DeviceDirectories []config.AdditionalHostMount
	// Additional holds extra host device node paths and systemd device group
	// specifiers requested by config.
	Additional []string
}

HostDevices groups host device node paths by category so that callers can report per-type discovery results while still bind-mounting the union of all paths into the nspawn container.

func DiscoverHostDevices added in v0.1.19

func DiscoverHostDevices(additional []string) HostDevices

DiscoverHostDevices probes the host for device nodes that should be bind-mounted into the nspawn container, grouped by category. Repeated calls produce the same output because each group is returned in a stable order.

func (HostDevices) DeviceGroupSpecifiers added in v0.1.22

func (d HostDevices) DeviceGroupSpecifiers() []string

DeviceGroupSpecifiers returns additional systemd DeviceAllow device group specifiers in a de-duplicated, sorted slice.

func (HostDevices) Grants added in v0.23.0

func (d HostDevices) Grants() []isolation.Grant

Grants returns the backend-neutral resource requests for the host devices: one device grant per node path, then one per device group specifier, then one mount grant per device directory.

func (HostDevices) Paths added in v0.1.19

func (d HostDevices) Paths() []string

Paths returns every discovered device node path merged into a single de-duplicated, sorted slice. This is the form the nspawn templates consume to emit Bind= directives.

type HostPaths added in v0.13.0

type HostPaths struct {
	// Root is the host root these paths are under.
	Root string
	// BinDir is <Root>/bin.
	BinDir string
	// NSpawnLifecycleBinary is the rollback-stable helper invoked by the
	// generated nspawn hook units.
	NSpawnLifecycleBinary string
	// DaemonRecoveryScript is executed by the daemon recovery unit.
	DaemonRecoveryScript string
	// LocalDNSNetworkHelper backs unbounded-localdns-network.service.
	LocalDNSNetworkHelper string
}

HostPaths is the host-side layout of the agent's own files under a host root; see the hostroot package.

These are paths on the host. Files inside the nspawn machine are always resolved relative to the machine directory.

func LegacyHostPaths added in v0.13.0

func LegacyHostPaths() HostPaths

LegacyHostPaths returns the layout under hostroot.LegacyPath, where releases that predate the host root installed it.

func PlannedHostPaths added in v0.13.0

func PlannedHostPaths() HostPaths

PlannedHostPaths returns the layout ResolveHostPaths will return once the host root is migrated, without migrating it. It is for code that must not change the host, such as preflight.

func ResolveHostPaths added in v0.13.0

func ResolveHostPaths() HostPaths

ResolveHostPaths returns the agent's host-side layout on this host.

type ImageCredentialProvider added in v0.2.3

type ImageCredentialProvider struct {
	ConfigPath string
	BinDir     string
}

ImageCredentialProvider holds paths inside the nspawn machine for kubelet's exec image credential provider configuration and binaries.

type Kubelet

type Kubelet struct {
	// KubeletBinPath is the absolute path to the kubelet binary inside the
	// machine container (e.g. /usr/local/bin/kubelet).
	KubeletBinPath string

	// KubeletAuthInfo holds the authentication configuration for the
	// kubelet when KubeconfigData is empty.
	config.KubeletAuthInfo

	// KubeconfigData is a complete kubeconfig written verbatim for direct
	// kubelet authentication.
	KubeconfigData []byte

	// APIServer is the HTTPS endpoint of the Kubernetes API server
	// (e.g. "https://my-cluster.hcp.eastus.azmk8s.io:443").
	APIServer string

	// CACertData is the PEM-encoded CA certificate of the API server.
	CACertData []byte

	// ClusterDNS is the ClusterIP of the kube-dns service.
	ClusterDNS string

	// ResolvConf is the resolver file kubelet uses for dnsPolicy: Default pods.
	ResolvConf string

	// NodeIP overrides kubelet --node-ip. Supports a single IP or a
	// comma-separated dual-stack pair.
	NodeIP string

	// NodeLabels are key=value labels applied to the node at registration.
	NodeLabels map[string]string

	// RegisterWithTaints are taints applied to the node at registration.
	// Each entry uses the kubelet format: "key=value:effect"
	// (e.g. "dedicated=gpu:NoSchedule").
	RegisterWithTaints []string

	// Configuration is merged over the agent's baseline and rendered as a
	// kubelet.config.k8s.io/v1beta1 KubeletConfiguration.
	Configuration map[string]any

	// ImageCredentialProvider holds optional kubelet exec image credential
	// provider paths inside the nspawn machine.
	ImageCredentialProvider *ImageCredentialProvider
}

Kubelet defines the goal state for the kubelet configuration.

type LocalDNS added in v0.2.3

type LocalDNS struct {
	Enabled                bool
	CoreDNSVersion         string
	NodeListenerIP         netip.Addr
	ClusterListenerIP      netip.Addr
	NodeUpstreamIPs        []netip.Addr
	ClusterDNSServiceIP    netip.Addr
	MetricsAddress         string
	CPULimitInMilliCores   int
	MemoryLimitInMB        int
	RequiredPlugins        []string
	Corefile               []byte
	OriginalHostResolvConf []byte
}

LocalDNS is the fully resolved machine-local DNS goal state.

type LocalDNSCorefileTemplateData added in v0.2.3

type LocalDNSCorefileTemplateData struct {
	NodeListenerIP        string
	ClusterListenerIP     string
	NodeUpstreamIPs       []string
	NodeUpstreamIPsJoined string
	ClusterDNSServiceIP   string
	MetricsAddress        string
}

LocalDNSCorefileTemplateData contains validated runtime values available to Corefile templates.

type MachineGoalState

type MachineGoalState struct {
	RootFS    *RootFS
	NodeStart *NodeStart
}

MachineGoalState holds the fully resolved goal state for provisioning and starting an nspawn machine. Callers use RootFS for the rootfs provisioning phases and NodeStart for the service configuration and boot phases.

func ResolveMachine

func ResolveMachine(log *slog.Logger, cfg *config.AgentConfig, machineName string, downloads *DownloadOverrides) (*MachineGoalState, error)

ResolveMachine probes the host (kernel version, hostname, GPU hardware) and resolves the complete goal state for the named nspawn machine from an agent config and caller-provided download overrides.

func ResolveMachineForPreflight added in v0.17.0

func ResolveMachineForPreflight(log *slog.Logger, cfg *config.AgentConfig, machineName string, downloads *DownloadOverrides) (*MachineGoalState, error)

ResolveMachineForPreflight resolves diagnostic inputs without enforcing NVIDIA readiness or initializing devices. The named host checks report driver failures. NVIDIA fields are best-effort and may be empty until devices are initialized. Its result must not be used to provision or start a worker.

type NodeStart

type NodeStart struct {
	// MachineName is the local systemd-nspawn machine name (e.g. "kube1").
	// Used by nspawn service management.
	MachineName string

	// KubeMachineName is the Kubernetes Machine CR name (e.g. "agent-e2e").
	// This is the name that appears in the cluster and may differ from
	// the local nspawn machine name.
	KubeMachineName string

	// NodeName is the Kubernetes Node name used by kubelet and host-side daemon watches.
	NodeName string

	// MachineDir is the nspawn backend's root directory. It is an intentional
	// nspawn-only goal state field; phases reach it through Root().
	MachineDir string // e.g. /var/lib/machines/node
	Containerd Containerd
	Gantry     Gantry
	Kubelet    Kubelet
	LocalDNS   LocalDNS

	// Nvidia holds NVIDIA GPU state discovered on the host. After the nspawn
	// boots, the setup-nvidia-libraries task uses LibMappings to create
	// symlinks inside the container's library path pointing into the
	// bind-mounted /run/host-nvidia/ directories.
	Nvidia NvidiaHost

	// NvidiaToolkit locates the NVIDIA Container Toolkit tools on the host. It
	// is set only for isolation mode none; the zero value is the default paths.
	NvidiaToolkit NvidiaToolkit

	// RootOverride replaces the nspawn machine root for backends that do not
	// run the node stack in a machine, such as the none backend. Nil means the
	// nspawn root.
	RootOverride isolation.NodeRoot

	// ExecOverride replaces nspawn.Exec for the same backends. Nil means
	// commands run inside the nspawn machine. A non-nil value also marks the
	// node stack as host native, so phases skip nspawn-only files, unless
	// InContainer is also set.
	ExecOverride func(ctx context.Context, log *slog.Logger, args ...string) (string, error)

	// InContainer marks a node stack that runs in its own root filesystem
	// inside an OCI container (the container backend). Its overrides replace
	// the nspawn machine like the none backend's, but the node stack still has
	// a private root, so phases keep writing the rootfs files and use the
	// in-node archive path. PROVISIONAL: see the container backend.
	InContainer bool
}

func (*NodeStart) Exec added in v0.23.0

func (n *NodeStart) Exec(ctx context.Context, log *slog.Logger, args ...string) (string, error)

Exec runs args inside the node stack and returns its stdout.

func (*NodeStart) HostNative added in v0.24.0

func (n *NodeStart) HostNative() bool

HostNative reports whether the node stack runs directly on the host, so phases that write nspawn-only files or masquerade as the machine skip them. A node stack in an OCI container has its own root and is not host native.

func (*NodeStart) NSpawnMachine added in v0.24.0

func (n *NodeStart) NSpawnMachine() bool

NSpawnMachine reports whether the node stack runs in an nspawn machine, which is the case when no backend replaced the machine root.

func (*NodeStart) Root added in v0.23.0

func (n *NodeStart) Root() isolation.NodeRoot

Root returns the node root phases use to place files in the node stack's filesystem. Phases call Path on it instead of joining the machine directory themselves. This file is the intentional bridge from the nspawn goal state fields to the isolation seam.

type NvidiaHost

type NvidiaHost struct {
	// Required records whether NVIDIA capability was selected when the machine
	// was provisioned. Discovery refreshes must preserve this value.
	Required bool

	// GPUDevicePaths lists NVIDIA GPU device paths discovered on the host
	// (e.g. /dev/nvidia0, /dev/nvidiactl, /dev/nvidia-caps,
	// /dev/nvidia-caps-imex-channels, /dev/dri/*).
	// When non-empty the nspawn configuration will bind-mount these devices
	// and grant the container cgroup access to them.
	GPUDevicePaths []string

	// ContainerLibDir is the architecture-specific multiarch library
	// directory inside the nspawn container (e.g. /usr/lib/x86_64-linux-gnu
	// on amd64, /usr/lib/aarch64-linux-gnu on arm64). Symlinks to
	// bind-mounted host NVIDIA libraries are created here.
	ContainerLibDir string

	// LibMappings contains NVIDIA userspace libraries discovered from
	// ldconfig -p and filesystem scans. These are used to create symlinks
	// inside the nspawn container so that the host's NVIDIA driver libraries
	// are accessible.
	LibMappings []NvidiaLibMapping

	// LibDirMounts lists unique host directories containing NVIDIA libraries
	// to be bind-mounted read-only into the nspawn container at
	// /run/host-nvidia/<index>/. After boot, symlinks from the container's
	// standard library path are created by the setup-nvidia-libraries task.
	LibDirMounts []NvidiaLibDirMount

	// I386LibMappings contains optional i386 NVIDIA and GLVND libraries with a
	// driver version matching the active 64-bit driver.
	I386LibMappings []NvidiaLibMapping

	// I386LibDirMounts lists the host directories for I386LibMappings.
	I386LibDirMounts []NvidiaLibDirMount

	// NvidiaSMIPath is the host path to nvidia-smi, when available.
	NvidiaSMIPath string

	// NvidiaIMEXPath is the host path to nvidia-imex, when available.
	NvidiaIMEXPath string

	// NvidiaIMEXCtlPath is the host path to nvidia-imex-ctl, when available.
	NvidiaIMEXCtlPath string

	// DriverVersion is the active NVIDIA kernel driver version. It is used to
	// provide versioned library names when a host installation exposes only
	// unversioned or SONAME aliases through ldconfig.
	DriverVersion string
}

NvidiaHost aggregates all NVIDIA-related host state discovered at agent startup: GPU device paths, driver library mappings, and the derived bind-mount specifications for the nspawn container.

func ResolveNvidiaHost

func ResolveNvidiaHost(arch string) (NvidiaHost, error)

ResolveNvidiaHost probes the host for NVIDIA GPU devices and driver libraries and requires usable GPU enumeration when NVIDIA hardware or device paths are present, returning a fully populated NvidiaHost. The arch parameter is a GOARCH value (e.g. "amd64", "arm64") used to select the correct multiarch library path and ldconfig filter. Returns an error for unsupported architectures. On a non-GPU host the returned struct has all nil/empty fields (except ContainerLibDir).

func (NvidiaHost) Grants added in v0.23.0

func (n NvidiaHost) Grants() []isolation.Grant

Grants returns the backend-neutral resource requests for the discovered NVIDIA state: device nodes, driver library directories and the helper binary directory. Each backend decides how to apply them.

func (NvidiaHost) HostBinDir added in v0.23.0

func (n NvidiaHost) HostBinDir() string

HostBinDir returns the host directory that holds the NVIDIA helper binaries, or the empty string when none was discovered.

type NvidiaLibDirMount

type NvidiaLibDirMount = isolation.LibDirMount

NvidiaLibDirMount represents a read-only bind mount of a host directory containing NVIDIA libraries into the nspawn container.

type NvidiaLibMapping

type NvidiaLibMapping struct {
	HostPath         string // e.g. "/usr/lib/x86_64-linux-gnu/libcuda.so.1", original discovered name
	ResolvedHostPath string // e.g. "/usr/lib/nvidia/libcuda.so.580.126.09", real bind-mount source
	ContainerPath    string // e.g. "/run/host-nvidia/0/libcuda.so.580.126.09", bind-mount source
	LinkPath         string // e.g. "/usr/lib/x86_64-linux-gnu/libcuda.so.1", symlink in container
}

NvidiaLibMapping maps a host NVIDIA library to its corresponding paths inside the nspawn container.

type NvidiaRuntime

type NvidiaRuntime struct {
	Enabled                    bool
	RuntimeClassName           string
	RuntimePath                string
	DisableSetAsDefaultRuntime bool
}

NvidiaRuntime describes the NVIDIA container runtime configuration for containerd. When Enabled is true the runtime is registered as a handler so that GPU workloads can be scheduled.

type NvidiaToolkit added in v0.24.0

type NvidiaToolkit struct {
	// CTKPath is the nvidia-ctk executable. Empty means NvidiaCTKPath.
	CTKPath string
	// ContainerRuntimePath is the nvidia-container-runtime executable. Empty
	// means NvidiaContainerRuntimePath.
	ContainerRuntimePath string
}

NvidiaToolkit locates the NVIDIA Container Toolkit tools for a node stack that runs them from the host (isolation mode none). The zero value means the paths the rootfs images ship them at, which is what every other mode uses.

func (NvidiaToolkit) CTK added in v0.24.0

func (t NvidiaToolkit) CTK() string

CTK returns the nvidia-ctk path.

func (NvidiaToolkit) ContainerRuntime added in v0.24.0

func (t NvidiaToolkit) ContainerRuntime() string

ContainerRuntime returns the nvidia-container-runtime path.

type OfflineArtifactManifest added in v0.1.21

type OfflineArtifactManifest = bootstrapartifacts.Manifest

OfflineArtifactManifest describes an offline artifact bundle.

type OfflineArtifactVersions added in v0.1.21

type OfflineArtifactVersions = bootstrapartifacts.Versions

OfflineArtifactVersions records the component versions included in an offline artifact bundle.

type OfflineTemplateData added in v0.1.21

type OfflineTemplateData struct {
	KubernetesVersion    string
	KubernetesVersionNoV string
}

OfflineTemplateData is the data available when rendering OfflineArtifacts.Source.

type ResolvedOfflineArtifacts added in v0.1.21

type ResolvedOfflineArtifacts struct {
	SourceRoot string
	Manifest   OfflineArtifactManifest
	// contains filtered or unexported fields
}

ResolvedOfflineArtifacts is a validated offline artifact source and manifest.

type RootFS

type RootFS struct {
	// MachineDir is the nspawn backend's root directory. It is an intentional
	// nspawn-only goal state field; phases reach it through Root().
	MachineDir             string
	NSpawnConfigFile       string // e.g. /etc/systemd/nspawn/node.nspawn
	ServiceOverrideFile    string // e.g. /etc/systemd/system/systemd-nspawn@node.service.d/override.conf
	ConfigRegenerationFile string // host systemd pre-start unit
	HostArch               string
	HostKernel             string // running kernel version from uname -r, e.g. "6.8.0-45-generic"
	Hostname               string // host hostname, written into the rootfs so the nspawn container inherits it
	ContainerdVersion      string
	RunCVersion            string
	CNIPluginVersion       string
	KubernetesVersion      string
	LocalDNS               LocalDNS

	// Downloads optionally overrides the download sources for binaries
	// the agent installs into the nspawn rootfs (kubelet, containerd,
	// runc, CNI plugins, crictl). Nil means upstream defaults apply.
	Downloads *DownloadOverrides

	// OCIImage is an OCI registry reference, local OCI layout, or HTTPS URL
	// to a tarred OCI image layout used to bootstrap the machine rootfs. The
	// single tagged image reference in an HTTPS archive is selected automatically.
	// The image must use OCI media types and include a platform manifest
	// matching the host architecture.
	OCIImage string

	// Nvidia holds NVIDIA GPU state discovered on the host: device paths,
	// driver library mappings, and bind-mount specifications for the nspawn
	// container. Empty on non-GPU hosts.
	Nvidia NvidiaHost

	// AMD holds AMD GPU state discovered on the host. Empty on non-AMD GPU hosts.
	AMD AMDHost

	// HostDevices holds host device nodes to be bind-mounted into the
	// nspawn container, grouped by category (KVM, network virtualization,
	// block storage, InfiniBand HCA). Device nodes are discovered at agent
	// startup. Empty on hosts without any supported devices.
	HostDevices HostDevices

	// AdditionalHostMounts holds configured non-device host paths to be
	// bind-mounted into the nspawn container.
	AdditionalHostMounts []config.AdditionalHostMount

	// NSpawnLifecycle holds the configured lifecycle hook commands. Nil means
	// the hooks run the lifecycle helper, which provisioning installs.
	NSpawnLifecycle *config.NSpawnLifecycleConfig

	// RootOverride replaces the nspawn machine root for backends that install
	// onto the host instead of into a machine. Nil means the nspawn root.
	RootOverride isolation.NodeRoot
}

RootFS defines the goal state of the machine root fs. This goal state produces a rootfs that is ready for running a Kubernetes node via systemd-nspawn from the machine directory.

func ResolveNSpawnConfig added in v0.2.4

func ResolveNSpawnConfig(cfg *config.AgentConfig, machineName string) (*RootFS, error)

ResolveNSpawnConfig probes only the host state needed to render the systemd-nspawn configuration for a machine. Unlike ResolveMachine, it does not resolve network-dependent node services such as LocalDNS.

func (*RootFS) Grants added in v0.23.0

func (r *RootFS) Grants() []isolation.Grant

Grants returns every device, mount and library request the machine needs, in the order host devices (with device directories), additional mounts, NVIDIA, AMD. AMD devices that NVIDIA already owns are not requested twice.

func (*RootFS) Root added in v0.23.0

func (r *RootFS) Root() isolation.NodeRoot

Root returns the node root phases use to place files in the node stack's filesystem. The slot is the base name of the machine directory.

func (*RootFS) UsesLifecycleHelper added in v0.17.0

func (r *RootFS) UsesLifecycleHelper() bool

UsesLifecycleHelper reports whether the machine's hooks run the lifecycle helper, which must then be installed on the host.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL