devicestate

package
v1.7.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 20, 2026 License: Apache-2.0 Imports: 57 Imported by: 0

Documentation

Overview

Package devicestate is the node-agent side of device state persistence: it follows the snapshot-backed device containers of a node, detects changes of their writable layer, and turns them into snapshot images in the platform registry. The container runtime, the cluster API and the registry are interfaces, so the logic runs (and is tested) without containerd.

Index

Constants

View Source
const (
	CaptureGuardAnnotation = "laboratory.cybericebox.com/capture-guard"
)
View Source
const DefaultMaxWatchDirs = 2000

DefaultMaxWatchDirs is how many directories of one layer are watched with inotify. A watch costs kernel memory and the limit (fs.inotify.max_user_watches) is shared by everything on the node, so a device that makes a million directories must not be able to use it all: past the cap the layer is only polled.

View Source
const DefaultPollInterval = 30 * time.Second

DefaultPollInterval is how often the layer is scanned as a safety net next to the filesystem events.

Variables

View Source
var ErrDeleting = errors.New("capture Pod deletion has been accepted; quiescence must remain held")
View Source
var ErrNoFreezer = errors.New("cgroup v2 freezer not available")

ErrNoFreezer is returned when the container cgroup cannot be frozen (cgroup v1 or an unknown cgroup path). The snapshot is then taken without freezing.

View Source
var ErrSnapshotsBlocked = errors.New("AppArmor blocks snapshots")

ErrSnapshotsBlocked is returned instead of a snapshot when the node-agent's AppArmor profile blocks the snapshot mounts (Engine.SnapshotsBlocked). Nothing is diffed or frozen: every attempt would fail the same way until the node-agent restarts under another profile.

View Source
var ErrStale = errors.New("pod is not the current incarnation of its device")

ErrStale is returned by Cluster writes for a pod that is no longer the current one of its device (reset, or a newer incarnation): its state is obsolete.

Functions

func AcquireCaptureOwner added in v1.1.0

func AcquireCaptureOwner(workDir string) (*os.File, error)

AcquireCaptureOwner serializes startup recovery, API invalidation and thaw across node-agent processes. Hold it until all capture workers have exited.

func AppArmorBlocksSnapshots added in v1.7.0

func AppArmorBlocksSnapshots(label string) bool

AppArmorBlocksSnapshots reports whether label, the AppArmor label of the node-agent (the content of /proc/self/attr/current), is the container runtime's default profile in enforce mode. That profile has `deny mount`, and the snapshot filter mounts read-only overlay views of the device's layers, so every snapshot would fail with "permission denied".

func AppArmorLabel added in v1.7.0

func AppArmorLabel() string

AppArmorLabel is the AppArmor label of this process; empty when AppArmor is off or the label cannot be read.

func CaptureCommittedDeletePreconditions added in v1.1.0

func CaptureCommittedDeletePreconditions(p *corev1.Pod, req *api.DeviceCaptureRequest, result *api.DeviceCaptureResult) (*metav1.Preconditions, error)

func CaptureCommittedGuardMatches added in v1.1.0

func CaptureCommittedGuardMatches(p *corev1.Pod, req *api.DeviceCaptureRequest, result *api.DeviceCaptureResult) bool

func CaptureDeletePreconditions added in v1.1.0

func CaptureDeletePreconditions(p *corev1.Pod, req *api.DeviceCaptureRequest, result *api.DeviceCaptureResult) (*metav1.Preconditions, error)

CaptureDeletePreconditions supplies the exact current Pod fence after the caller direct-reads the current lifecycle, Device and Pod. On Conflict, reread those objects and repeat; never substitute the capture-time RV.

func CaptureGuardMatches added in v1.1.0

func CaptureGuardMatches(p *corev1.Pod, req *api.DeviceCaptureRequest, result *api.DeviceCaptureResult) bool

CaptureGuardMatches is the controller's barrier predicate. Call it only on direct-read current Pod and Device. Delete uses that current Pod UID/RV, not the result's capture-time resourceVersion (which is audit information).

func CgroupDir

func CgroupDir(root, cgroupsPath string) string

CgroupDir turns the cgroup path from a container's OCI spec into its directory under the cgroup v2 mount at root. It understands both layouts the kubelet produces: the systemd driver ("kubepods-burstable-pod<uid>.slice:cri-containerd:<id>", whose slice name expands into nested slice directories) and cgroupfs ("/kubepods/burstable/pod<uid>/<id>"). It returns "" when the directory does not exist.

func Debounce

func Debounce(ctx context.Context, in <-chan struct{}, quiet, maxWait time.Duration, fire func())

Debounce turns a stream of change signals into snapshot triggers: one trigger after the signals stayed quiet for quiet, or after maxWait of continuous signals (so a constantly written layer is still snapshotted). It returns when ctx ends or in is closed.

func FindCgroup

func FindCgroup(root, containerID string) string

FindCgroup searches the cgroup tree for the directory of a container by id. The kubelet places it under a kubepods hierarchy; it is used when the path from the spec does not resolve.

func Fingerprint

func Fingerprint(root string, pol snapshot.Policy) string

Fingerprint summarises a layer directory: every non-excluded entry with its size, modification time and mode. Two scans differ when something changed.

func Forward

func Forward(ctx context.Context, listen, target string, log logr.Logger, opts ...ForwardOption) error

Forward listens on listen (a loopback address) and relays every connection to target. The node-agent uses it to make the cluster-internal snapshot registry reachable as localhost:<port> from the node's container runtime, which treats localhost registries as plain HTTP, so no host configuration (registries.yaml, hosts.toml, certificates, DNS) is needed. It returns when ctx ends.

With a reader (WithReader) it is an HTTP-aware relay instead: the registry lets only the platform read the snapshots of the labs (`lab/**`) and the shared `base` repository (everything else is anonymous, the public image cache), so the node's runtime, which pulls a snapshot as localhost:<port>/lab/..., has the reader account added for it here, on GET and HEAD requests of those repositories that carry no credentials of their own. Writes are never given the reader.

func Freeze

func Freeze(ctx context.Context, dir string) (thaw func(), err error)

Freeze freezes every process of the cgroup and returns the function that thaws it. The freeze is confirmed through cgroup.events before it returns.

func FreezeRequired added in v1.1.0

func FreezeRequired(ctx context.Context, dir string) (thaw func() error, err error)

FreezeRequired retains ownership of the requested freeze even if confirmation fails. Its caller must fence the API before thawing, or wait for actual death. Ordinary snapshots keep using Freeze's automatic error cleanup.

func NewPodSourceKeychain added in v1.1.0

func NewPodSourceKeychain(reader client.Reader, nodeName string) func(context.Context, Container, PodInfo) (authn.Keychain, error)

NewPodSourceKeychain reads only the current device pod's own pull Secrets. Snapshot-registry write credentials are not part of this source credential path.

func ThawOrphans

func ThawOrphans(root string) []string

ThawOrphans thaws container cgroups left frozen, which happens when the node-agent dies in the middle of a snapshot. Nothing else freezes them. It returns the directories it thawed.

func ThawOrphansExcept added in v1.1.0

func ThawOrphansExcept(root string, protected map[string]bool) []string

ThawOrphansExcept preserves required captures that still own live tasks.

func UpperDir

func UpperDir(mounts []mount.Mount) string

UpperDir returns the overlay upperdir of a snapshot's mounts, "" if none.

func Watch

func Watch(ctx context.Context, root string, pol snapshot.Policy, poll time.Duration, maxDirs int) (<-chan struct{}, error)

Watch reports changes of a writable layer directory. It combines two signals: inotify events on the directory tree (low latency; directories are added as they appear, excluded paths are not watched) and a periodic scan that compares a fingerprint of the tree. The scan is the safety net for what events miss: the kernel's inotify watch limit, and overlayfs, which does not always raise events on the upper directory for changes made through the overlay mount. The returned channel is closed when ctx ends.

Types

type CaptureCluster added in v1.1.0

type CaptureCluster interface {
	SetCaptureGuard(context.Context, PodInfo, api.DeviceCaptureRequest, string) error
	RecordCapture(context.Context, PodInfo, api.DeviceCaptureResult) error
	InvalidateCapture(context.Context, PodInfo, api.DeviceCaptureResult) error
	CheckCapture(context.Context, PodInfo, api.DeviceCaptureResult) (bool, error)
}

CaptureCluster uses direct reads and optimistic Pod locks, independent of legacy exit markers.

type CaptureCommitCluster added in v1.1.0

type CaptureCommitCluster interface {
	CaptureCommitRequested(context.Context, PodInfo, api.DeviceCaptureResult) (bool, error)
	AcknowledgeCaptureCommit(context.Context, PodInfo, api.DeviceCaptureResult) error
}

CaptureCommitCluster is the minimal durable final-capture-to-stop seam. Required capture remains compatible; an operator never deletes until this exact capture's committed acknowledgement is present in status and guard.

type Cluster

type Cluster interface {
	// Pods lists the snapshot-backed device pods scheduled on this node.
	Pods(ctx context.Context) ([]PodInfo, error)
	// Record stores a successful snapshot and clears the warning.
	Record(ctx context.Context, p PodInfo, s Snapshot) error
	// Warn stores a warning and keeps the last good snapshot.
	Warn(ctx context.Context, p PodInfo, msg string) error
	// MarkExit records that the exit snapshot of the pod is finished.
	MarkExit(ctx context.Context, p PodInfo) error
	// TenantBytes is what the snapshots of the tenant's devices take in the registry (the sizes recorded in their status),
	// without the device `except`.
	TenantBytes(ctx context.Context, tenant string, except types.NamespacedName) (int64, error)
	// LiveRepos is the set of snapshot repositories of the devices that exist (every tenant): any other repository of the
	// registry belongs to a lab that is gone and is only waiting out its retention.
	LiveRepos(ctx context.Context) (map[string]bool, error)
}

Cluster is the Kubernetes side: which pods to follow and where to report. Every write is guarded by the reset epoch: it is dropped (ErrStale) when the device has been reset since the pod was created.

type Container

type Container struct {
	ID string
	// ImageRef is the image the container was created from.
	ImageRef string
	// UpperDir is the writable layer directory on the node.
	UpperDir string
	// Cgroup is the container's cgroup directory (cgroup v2), "" if unknown.
	Cgroup string
	// Snapshotter and SnapshotKey name the container's writable snapshot in the
	// runtime; they are private to the Runtime implementation.
	Snapshotter string
	SnapshotKey string
	// IDs are the user and group id maps of the container's user namespace (empty without one): the diff of its writable layer
	// holds host ids, which a snapshot must not keep.
	IDs snapshot.IDMaps
	// OwnershipKnown distinguishes a read OCI no-userns map from missing metadata.
	OwnershipKnown bool
}

Container is the runtime view of a device container.

type ContainerdRuntime

type ContainerdRuntime struct {

	// SourceKeychain binds fallback reads to the current pod's pull credentials.
	SourceKeychain func(context.Context, Container, PodInfo) (authn.Keychain, error)
	// contains filtered or unexported fields
}

ContainerdRuntime is the Runtime backed by the node's containerd, reached through the CRI socket (containerd serves its native API on the same one).

func NewContainerdRuntime

func NewContainerdRuntime(sock, namespace, cgroupRoot string, log logr.Logger) (*ContainerdRuntime, error)

NewContainerdRuntime connects to containerd at sock; namespace is the one the kubelet's CRI uses ("k8s.io"); cgroupRoot is where the host's cgroup v2 tree is mounted.

func (*ContainerdRuntime) Close

func (r *ContainerdRuntime) Close() error

Close releases the connection.

func (*ContainerdRuntime) Diff

func (r *ContainerdRuntime) Diff(ctx context.Context, c Container, freeze bool, pol snapshot.Policy) (io.ReadCloser, error)

Diff implements Runtime: the containerd diff service compares the container's active snapshot with its parent. Containerd provides whiteouts, opaque markers and security.capability; rooted metadata reads preserve user.* attributes and supplement attribute-only changes. Directory attribute removals fail explicitly because containerd's directory merge cannot restore them faithfully.

func (*ContainerdRuntime) Exits

func (r *ContainerdRuntime) Exits(ctx context.Context) (<-chan string, error)

Exits implements Runtime: init-process exit events of the CRI namespace. The subscription is re-established after a connection error.

func (*ContainerdRuntime) Inspect

func (r *ContainerdRuntime) Inspect(ctx context.Context, id string) (Container, error)

Inspect implements Runtime.

func (*ContainerdRuntime) LoadImage

func (r *ContainerdRuntime) LoadImage(ctx context.Context, ref string) (v1.Image, error)

LoadImage implements Runtime: the image the kubelet pulled for this container is read out of the content store, so no registry is contacted.

func (*ContainerdRuntime) LoadImageForPod added in v1.1.0

func (r *ContainerdRuntime) LoadImageForPod(ctx context.Context, c Container, p PodInfo) (v1.Image, error)

func (*ContainerdRuntime) Quiesce added in v1.1.0

func (r *ContainerdRuntime) Quiesce(ctx context.Context, c Container) (func() error, error)

Quiesce uses the native freezer and treats sync failures as required-capture failures. On sync failure the caller still owns the thaw function and must invalidate first.

func (*ContainerdRuntime) TaskAlive added in v1.1.0

func (r *ContainerdRuntime) TaskAlive(ctx context.Context, c Container) (bool, error)

func (*ContainerdRuntime) Thaw added in v1.1.0

Thaw verifies the task's current cgroup identity; callers own API fencing.

type Engine

type Engine struct {
	Runtime        Runtime
	NodeAgentEpoch string
	Cluster        Cluster
	Pusher         Pusher
	// RegistryHost is host:port under which this node reaches the registry; an
	// image reference starting with it is a snapshot image.
	RegistryHost string
	// WorkDir holds the temporary layer files of a snapshot.
	WorkDir string
	// MaxWatchDirs is how many directories of one writable layer are watched with inotify (the kernel's watch limit is shared
	// by everything on the node); past it the layer is only polled. Zero means DefaultMaxWatchDirs.
	MaxWatchDirs int
	// Poll is the writable-layer scan interval (DefaultPollInterval when zero).
	Poll time.Duration
	// Resync is how often the pods are re-listed (5s when zero).
	Resync time.Duration
	// ExitTimeout bounds the exit snapshot so it ends before the controller
	// stops waiting for it (25s when zero).
	ExitTimeout time.Duration
	// RetryBase and RetryMax bound the retry of a failed snapshot (10s doubling to 5m when zero).
	RetryBase, RetryMax time.Duration

	// The registry is shared by every tenant, so what a device may push is bounded (zero values switch the bound off):
	// MinPushInterval is the least time between two pushes of one device (the exit snapshot is exempt: it must not be lost),
	// PushBudget the most state it may push within PushBudgetWindow (an hour when zero). A change that comes sooner or over the
	// budget waits; nothing is lost, the next snapshot has it.
	MinPushInterval  time.Duration
	PushBudget       int64
	PushBudgetWindow time.Duration
	// SupersededGrace is how long the manifest a new snapshot replaced is kept before it and its blobs are deleted from the
	// registry (a pod being created from it a moment ago must still be able to pull it). Zero deletes at once.
	SupersededGrace time.Duration
	// Space refuses a push when the registry is nearly full; Retained adds the state the tenant still has in repositories of labs that
	// are gone to the tenant's quota. Both are optional.
	Space    SpaceGuard
	Retained RetainedCounter
	Log      logr.Logger
	// Now is the clock; nil means time.Now.
	Now func() time.Time
	// SnapshotsBlocked, when set, is why no snapshot can be taken on this node (an AppArmor profile that denies mount): devices are
	// still followed, but each gets it as its warning and no snapshot is attempted.
	SnapshotsBlocked string
	// contains filtered or unexported fields
}

Engine follows the snapshot-backed device containers of one node.

func (*Engine) CaptureRequired added in v1.1.0

func (e *Engine) CaptureRequired(ctx context.Context, p PodInfo, req api.DeviceCaptureRequest) (result api.DeviceCaptureResult, err error)

CaptureRequired explicitly captures a live incarnation and keeps it frozen until actual task death or API-fenced invalidation. It never uses ExitSnapshotPod.

func (*Engine) RecoverCaptureHolds added in v1.1.0

func (e *Engine) RecoverCaptureHolds(ctx context.Context) (map[string]bool, error)

RecoverCaptureHolds invalidates prior live acknowledgements before startup thaw and preserves deleting/missing Pod holds until actual runtime task death.

func (*Engine) Run

func (e *Engine) Run(ctx context.Context) error

Run follows the node until ctx ends.

func (*Engine) Sync

func (e *Engine) Sync(ctx context.Context)

Sync aligns the followed containers with the pods of the node: starts following new running ones, drops the gone, and finishes the exit snapshot of pods that ended without one (the container exited while nobody was watching).

type ForwardOption

type ForwardOption func(*forwardOptions)

ForwardOption configures Forward.

func WithHost

func WithHost(host string) ForwardOption

WithHost sets the exact Host header a request must carry to be given the reader account: the address the node's runtime pulls the snapshots from (localhost:<port>). A request that arrives under any other Host is relayed without it.

func WithReader

func WithReader(user, password string) ForwardOption

WithReader makes the forwarder add the reader account to the reads of the snapshot repositories.

type KubeCluster

type KubeCluster struct {
	// Client reads from the informer cache.
	Client client.Client
	// Reader reads straight from the API server; every status write re-reads
	// through it so the epoch guard never acts on a stale cache.
	Reader   client.Reader
	NodeName string
}

KubeCluster is the Cluster backed by the Kubernetes API: the device pods of this node are found through their annotations and the Device CR carries the policy and receives the snapshot status.

func (*KubeCluster) AcknowledgeCaptureCommit added in v1.1.0

func (k *KubeCluster) AcknowledgeCaptureCommit(ctx context.Context, p PodInfo, result api.DeviceCaptureResult) error

func (*KubeCluster) CaptureCommitRequested added in v1.1.0

func (k *KubeCluster) CaptureCommitRequested(ctx context.Context, p PodInfo, result api.DeviceCaptureResult) (bool, error)

func (*KubeCluster) CapturePods added in v1.1.0

func (k *KubeCluster) CapturePods(ctx context.Context) ([]PodInfo, error)

CapturePods bypasses the informer cache during crash recovery.

func (*KubeCluster) CheckCapture added in v1.1.0

func (k *KubeCluster) CheckCapture(ctx context.Context, p PodInfo, result api.DeviceCaptureResult) (bool, error)

func (*KubeCluster) InvalidateCapture added in v1.1.0

func (k *KubeCluster) InvalidateCapture(ctx context.Context, p PodInfo, result api.DeviceCaptureResult) error

InvalidateCapture first changes the current Pod RV. Any deletion using the previous guard/RV now conflicts. A deleting Pod is never modified or thawed.

func (*KubeCluster) LiveRepos

func (k *KubeCluster) LiveRepos(ctx context.Context) (map[string]bool, error)

LiveRepos implements Cluster: the snapshot repositories of every existing device.

func (*KubeCluster) MarkExit

func (k *KubeCluster) MarkExit(ctx context.Context, p PodInfo) error

MarkExit implements Cluster.

func (*KubeCluster) Pods

func (k *KubeCluster) Pods(ctx context.Context) ([]PodInfo, error)

Pods implements Cluster.

func (*KubeCluster) Record

func (k *KubeCluster) Record(ctx context.Context, p PodInfo, s Snapshot) error

Record implements Cluster.

func (*KubeCluster) RecordCapture added in v1.1.0

func (k *KubeCluster) RecordCapture(ctx context.Context, p PodInfo, result api.DeviceCaptureResult) error

func (*KubeCluster) SetCaptureGuard added in v1.1.0

func (k *KubeCluster) SetCaptureGuard(ctx context.Context, p PodInfo, req api.DeviceCaptureRequest, boot string) error

func (*KubeCluster) TenantBytes

func (k *KubeCluster) TenantBytes(ctx context.Context, tenant string, except types.NamespacedName) (int64, error)

TenantBytes implements Cluster: the sum of the snapshot sizes of the tenant's devices. The default tenant also owns the devices that carry no tenant label (created before tenancy).

func (*KubeCluster) Warn

func (k *KubeCluster) Warn(ctx context.Context, p PodInfo, msg string) error

Warn implements Cluster.

type PodInfo

type PodInfo struct {
	// Device is the Device CR the pod belongs to.
	Device          types.NamespacedName
	Pod             string
	UID             string
	ResourceVersion string
	CaptureRequest  *api.DeviceCaptureRequest
	Capture         *api.DeviceCaptureResult
	Guard           string
	Deleting        bool

	// Incarnation is the pod's incarnation number; only the current incarnation
	// of a device records snapshots.
	Incarnation int32
	// ContainerID is the runtime id of the device container; empty until started.
	ContainerID string
	Running     bool
	// Ended is true once the pod finished (container exited).
	Ended bool
	// Epoch is the reset epoch the pod was created in; DeviceEpoch is the
	// device's current one. A pod of an older epoch is never snapshotted.
	Epoch       int32
	DeviceEpoch int32
	// ExitDone is true when the exit snapshot of this pod is already finished.
	ExitDone bool
	Policy   snapshot.Policy
	Repo     string
	// Tenant is the tenant the device belongs to (names.DefaultTenant when the device carries no tenant label);
	// TenantQuota is the most all of that tenant's snapshots may take in the registry together (0 = no limit).
	Tenant      string
	TenantQuota int64
	// contains filtered or unexported fields
}

PodInfo is what the engine needs to know about one snapshot-backed device pod on this node.

type Pusher

type Pusher interface {
	Push(ctx context.Context, repo string, img v1.Image, baseLayers int, sourceRepo string) (ref string, digest v1.Hash, err error)
}

Pusher publishes snapshot images; *snapshot.Registry implements it.

type RequiredRuntime added in v1.1.0

type RequiredRuntime interface {
	// Quiesce returns an owned thaw handle whenever freeze was requested, even on error.
	Quiesce(context.Context, Container) (func() error, error)
	// Thaw releases a known current container hold only after caller API invalidation.
	Thaw(context.Context, Container) error
	TaskAlive(context.Context, Container) (bool, error)
}

RequiredRuntime must fail when strict quiescence or task liveness cannot be established.

type RetainedCounter

type RetainedCounter interface {
	RetainedBytes(ctx context.Context, tenant string, live map[string]bool) (int64, error)
}

RetainedCounter is the state a tenant keeps in the repositories of labs that are gone; *snapshot.Registry implements it.

type Runtime

type Runtime interface {
	// Inspect finds a container by id.
	Inspect(ctx context.Context, containerID string) (Container, error)
	// Diff returns the uncompressed layer tar of the container's writable layer
	// against its parent snapshot. With freeze the container is frozen and its
	// filesystem synced while the diff is computed, so the result is consistent;
	// without it (the container has exited) nothing runs and nothing is frozen.
	// Closing the reader releases everything the diff held.
	Diff(ctx context.Context, c Container, freeze bool, policy snapshot.Policy) (io.ReadCloser, error)
	// LoadImage opens the image a container was created from out of the local
	// image store, with the snapshot chain already on the node.
	LoadImage(ctx context.Context, imageRef string) (v1.Image, error)
	// Exits reports the ids of containers whose task exited, until ctx ends.
	Exits(ctx context.Context) (<-chan string, error)
}

Runtime is the container runtime facade.

type Snapshot

type Snapshot struct {
	// Image is the pullable reference with digest; empty when the writable
	// layer holds nothing beyond the image the device started from and that
	// image is the base image.
	Image     string
	At        time.Time
	SizeBytes int64
	Layers    int32
}

Snapshot is the outcome of a successful snapshot.

type SourceImageRuntime added in v1.1.0

type SourceImageRuntime interface {
	LoadImageForPod(context.Context, Container, PodInfo) (v1.Image, error)
}

SourceImageRuntime optionally binds source image reads to the current device pod. Runtimes without this capability retain the local-only LoadImage behavior.

type SpaceGuard

type SpaceGuard interface {
	Check(ctx context.Context, bytes int64) error
}

SpaceGuard says whether the registry has room for a push of about this many bytes; *snapshot.Capacity implements it.

type Superseder

type Superseder interface {
	Supersede(ctx context.Context, repo, oldRef string, keep v1.Image) error
}

Superseder removes what a new snapshot replaced; *snapshot.Registry implements it. A Pusher that does not is never asked.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL