Documentation
¶
Overview ¶
Package devicestate is the node-agent side of device state persistence: it follows the snapshot-backed device containers of a node, detects changes of their writable layer, and turns them into snapshot images in the platform registry. The container runtime, the cluster API and the registry are interfaces, so the logic runs (and is tested) without containerd.
Index ¶
- Constants
- Variables
- func CgroupDir(root, cgroupsPath string) string
- func Debounce(ctx context.Context, in <-chan struct{}, quiet, maxWait time.Duration, ...)
- func FindCgroup(root, containerID string) string
- func Fingerprint(root string, pol snapshot.Policy) string
- func Forward(ctx context.Context, listen, target string, log logr.Logger, ...) error
- func Freeze(ctx context.Context, dir string) (thaw func(), err error)
- func ThawOrphans(root string) []string
- func UpperDir(mounts []mount.Mount) string
- func Watch(ctx context.Context, root string, pol snapshot.Policy, poll time.Duration, ...) (<-chan struct{}, error)
- type Cluster
- type Container
- type ContainerdRuntime
- func (r *ContainerdRuntime) Close() error
- func (r *ContainerdRuntime) Diff(ctx context.Context, c Container, freeze bool) (io.ReadCloser, error)
- func (r *ContainerdRuntime) Exits(ctx context.Context) (<-chan string, error)
- func (r *ContainerdRuntime) Inspect(ctx context.Context, id string) (Container, error)
- func (r *ContainerdRuntime) LoadImage(ctx context.Context, ref string) (v1.Image, error)
- type Engine
- type ForwardOption
- type KubeCluster
- func (k *KubeCluster) LiveRepos(ctx context.Context) (map[string]bool, error)
- func (k *KubeCluster) MarkExit(ctx context.Context, p PodInfo) error
- func (k *KubeCluster) Pods(ctx context.Context) ([]PodInfo, error)
- func (k *KubeCluster) Record(ctx context.Context, p PodInfo, s Snapshot) error
- func (k *KubeCluster) TenantBytes(ctx context.Context, tenant string, except types.NamespacedName) (int64, error)
- func (k *KubeCluster) Warn(ctx context.Context, p PodInfo, msg string) error
- type PodInfo
- type Pusher
- type RetainedCounter
- type Runtime
- type Snapshot
- type SpaceGuard
- type Superseder
Constants ¶
const DefaultMaxWatchDirs = 2000
DefaultMaxWatchDirs is how many directories of one layer are watched with inotify. A watch costs kernel memory and the limit (fs.inotify.max_user_watches) is shared by everything on the node, so a device that makes a million directories must not be able to use it all: past the cap the layer is only polled.
const DefaultPollInterval = 30 * time.Second
DefaultPollInterval is how often the layer is scanned as a safety net next to the filesystem events.
Variables ¶
var ErrNoFreezer = errors.New("cgroup v2 freezer not available")
ErrNoFreezer is returned when the container cgroup cannot be frozen (cgroup v1 or an unknown cgroup path). The snapshot is then taken without freezing.
var ErrStale = errors.New("pod is not the current incarnation of its device")
ErrStale is returned by Cluster writes for a pod that is no longer the current one of its device (reset, or a newer incarnation): its state is obsolete.
Functions ¶
func CgroupDir ¶
CgroupDir turns the cgroup path from a container's OCI spec into its directory under the cgroup v2 mount at root. It understands both layouts the kubelet produces: the systemd driver ("kubepods-burstable-pod<uid>.slice:cri-containerd:<id>", whose slice name expands into nested slice directories) and cgroupfs ("/kubepods/burstable/pod<uid>/<id>"). It returns "" when the directory does not exist.
func Debounce ¶
Debounce turns a stream of change signals into snapshot triggers: one trigger after the signals stayed quiet for quiet, or after maxWait of continuous signals (so a constantly written layer is still snapshotted). It returns when ctx ends or in is closed.
func FindCgroup ¶
FindCgroup searches the cgroup tree for the directory of a container by id. The kubelet places it under a kubepods hierarchy; it is used when the path from the spec does not resolve.
func Fingerprint ¶
Fingerprint summarises a layer directory: every non-excluded entry with its size, modification time and mode. Two scans differ when something changed.
func Forward ¶
func Forward(ctx context.Context, listen, target string, log logr.Logger, opts ...ForwardOption) error
Forward listens on listen (a loopback address) and relays every connection to target. The node-agent uses it to make the cluster-internal snapshot registry reachable as localhost:<port> from the node's container runtime, which treats localhost registries as plain HTTP, so no host configuration (registries.yaml, hosts.toml, certificates, DNS) is needed. It returns when ctx ends.
With a reader (WithReader) it is an HTTP-aware relay instead: the registry lets only the platform read the snapshots of the labs (`lab/**`) and the shared `base` repository (everything else is anonymous, the public image cache), so the node's runtime, which pulls a snapshot as localhost:<port>/lab/..., has the reader account added for it here, on GET and HEAD requests of those repositories that carry no credentials of their own. Writes are never given the reader.
func Freeze ¶
Freeze freezes every process of the cgroup and returns the function that thaws it. The freeze is confirmed through cgroup.events before it returns.
func ThawOrphans ¶
ThawOrphans thaws container cgroups left frozen, which happens when the node-agent dies in the middle of a snapshot. Nothing else freezes them. It returns the directories it thawed.
func Watch ¶
func Watch(ctx context.Context, root string, pol snapshot.Policy, poll time.Duration, maxDirs int) (<-chan struct{}, error)
Watch reports changes of a writable layer directory. It combines two signals: inotify events on the directory tree (low latency; directories are added as they appear, excluded paths are not watched) and a periodic scan that compares a fingerprint of the tree. The scan is the safety net for what events miss: the kernel's inotify watch limit, and overlayfs, which does not always raise events on the upper directory for changes made through the overlay mount. The returned channel is closed when ctx ends.
Types ¶
type Cluster ¶
type Cluster interface {
// Pods lists the snapshot-backed device pods scheduled on this node.
Pods(ctx context.Context) ([]PodInfo, error)
// Record stores a successful snapshot and clears the warning.
Record(ctx context.Context, p PodInfo, s Snapshot) error
// Warn stores a warning and keeps the last good snapshot.
Warn(ctx context.Context, p PodInfo, msg string) error
// MarkExit records that the exit snapshot of the pod is finished.
MarkExit(ctx context.Context, p PodInfo) error
// TenantBytes is what the snapshots of the tenant's devices take in the registry (the sizes recorded in their status),
// without the device `except`.
TenantBytes(ctx context.Context, tenant string, except types.NamespacedName) (int64, error)
// LiveRepos is the set of snapshot repositories of the devices that exist (every tenant): any other repository of the
// registry belongs to a lab that is gone and is only waiting out its retention.
LiveRepos(ctx context.Context) (map[string]bool, error)
}
Cluster is the Kubernetes side: which pods to follow and where to report. Every write is guarded by the reset epoch: it is dropped (ErrStale) when the device has been reset since the pod was created.
type Container ¶
type Container struct {
ID string
// ImageRef is the image the container was created from.
ImageRef string
// UpperDir is the writable layer directory on the node.
UpperDir string
// Cgroup is the container's cgroup directory (cgroup v2), "" if unknown.
Cgroup string
// Snapshotter and SnapshotKey name the container's writable snapshot in the
// runtime; they are private to the Runtime implementation.
Snapshotter string
SnapshotKey string
// IDs are the user and group id maps of the container's user namespace (empty without one): the diff of its writable layer
// holds host ids, which a snapshot must not keep.
IDs snapshot.IDMaps
}
Container is the runtime view of a device container.
type ContainerdRuntime ¶
type ContainerdRuntime struct {
// contains filtered or unexported fields
}
ContainerdRuntime is the Runtime backed by the node's containerd, reached through the CRI socket (containerd serves its native API on the same one).
func NewContainerdRuntime ¶
func NewContainerdRuntime(sock, namespace, cgroupRoot string, log logr.Logger) (*ContainerdRuntime, error)
NewContainerdRuntime connects to containerd at sock; namespace is the one the kubelet's CRI uses ("k8s.io"); cgroupRoot is where the host's cgroup v2 tree is mounted.
func (*ContainerdRuntime) Close ¶
func (r *ContainerdRuntime) Close() error
Close releases the connection.
func (*ContainerdRuntime) Diff ¶
func (r *ContainerdRuntime) Diff(ctx context.Context, c Container, freeze bool) (io.ReadCloser, error)
Diff implements Runtime: the containerd diff service compares the container's active snapshot with its parent, so whiteouts, opaque directories, xattrs and ownership come out exactly as an image layer would hold them.
func (*ContainerdRuntime) Exits ¶
func (r *ContainerdRuntime) Exits(ctx context.Context) (<-chan string, error)
Exits implements Runtime: init-process exit events of the CRI namespace. The subscription is re-established after a connection error.
type Engine ¶
type Engine struct {
Runtime Runtime
Cluster Cluster
Pusher Pusher
// RegistryHost is host:port under which this node reaches the registry; an
// image reference starting with it is a snapshot image.
RegistryHost string
// WorkDir holds the temporary layer files of a snapshot.
WorkDir string
// MaxWatchDirs is how many directories of one writable layer are watched with inotify (the kernel's watch limit is shared
// by everything on the node); past it the layer is only polled. Zero means DefaultMaxWatchDirs.
MaxWatchDirs int
// Poll is the writable-layer scan interval (DefaultPollInterval when zero).
Poll time.Duration
// Resync is how often the pods are re-listed (5s when zero).
Resync time.Duration
// ExitTimeout bounds the exit snapshot so it ends before the controller
// stops waiting for it (25s when zero).
ExitTimeout time.Duration
// RetryBase and RetryMax bound the retry of a failed snapshot (10s doubling to 5m when zero).
RetryBase, RetryMax time.Duration
// The registry is shared by every tenant, so what a device may push is bounded (zero values switch the bound off):
// MinPushInterval is the least time between two pushes of one device (the exit snapshot is exempt: it must not be lost),
// PushBudget the most state it may push within PushBudgetWindow (an hour when zero). A change that comes sooner or over the
// budget waits; nothing is lost, the next snapshot has it.
MinPushInterval time.Duration
PushBudget int64
PushBudgetWindow time.Duration
// SupersededGrace is how long the manifest a new snapshot replaced is kept before it and its blobs are deleted from the
// registry (a pod being created from it a moment ago must still be able to pull it). Zero deletes at once.
SupersededGrace time.Duration
// Space refuses a push when the registry is nearly full; Retained adds the state the tenant still has in repositories of labs that
// are gone to the tenant's quota. Both are optional.
Space SpaceGuard
Retained RetainedCounter
Log logr.Logger
// Now is the clock; nil means time.Now.
Now func() time.Time
// contains filtered or unexported fields
}
Engine follows the snapshot-backed device containers of one node.
type ForwardOption ¶
type ForwardOption func(*forwardOptions)
ForwardOption configures Forward.
func WithHost ¶
func WithHost(host string) ForwardOption
WithHost sets the exact Host header a request must carry to be given the reader account: the address the node's runtime pulls the snapshots from (localhost:<port>). A request that arrives under any other Host is relayed without it.
func WithReader ¶
func WithReader(user, password string) ForwardOption
WithReader makes the forwarder add the reader account to the reads of the snapshot repositories.
type KubeCluster ¶
type KubeCluster struct {
// Client reads from the informer cache.
Client client.Client
// Reader reads straight from the API server; every status write re-reads
// through it so the epoch guard never acts on a stale cache.
Reader client.Reader
NodeName string
}
KubeCluster is the Cluster backed by the Kubernetes API: the device pods of this node are found through their annotations and the Device CR carries the policy and receives the snapshot status.
func (*KubeCluster) LiveRepos ¶
LiveRepos implements Cluster: the snapshot repositories of every existing device.
func (*KubeCluster) MarkExit ¶
func (k *KubeCluster) MarkExit(ctx context.Context, p PodInfo) error
MarkExit implements Cluster.
func (*KubeCluster) Pods ¶
func (k *KubeCluster) Pods(ctx context.Context) ([]PodInfo, error)
Pods implements Cluster.
func (*KubeCluster) TenantBytes ¶
func (k *KubeCluster) TenantBytes(ctx context.Context, tenant string, except types.NamespacedName) (int64, error)
TenantBytes implements Cluster: the sum of the snapshot sizes of the tenant's devices. The default tenant also owns the devices that carry no tenant label (created before tenancy).
type PodInfo ¶
type PodInfo struct {
// Device is the Device CR the pod belongs to.
Device types.NamespacedName
Pod string
// Incarnation is the pod's incarnation number; only the current incarnation
// of a device records snapshots.
Incarnation int32
// ContainerID is the runtime id of the device container; empty until started.
ContainerID string
Running bool
// Ended is true once the pod finished (container exited).
Ended bool
// Epoch is the reset epoch the pod was created in; DeviceEpoch is the
// device's current one. A pod of an older epoch is never snapshotted.
Epoch int32
DeviceEpoch int32
// ExitDone is true when the exit snapshot of this pod is already finished.
ExitDone bool
Policy snapshot.Policy
Repo string
// Tenant is the tenant the device belongs to (names.DefaultTenant when the device carries no tenant label);
// TenantQuota is the most all of that tenant's snapshots may take in the registry together (0 = no limit).
Tenant string
TenantQuota int64
}
PodInfo is what the engine needs to know about one snapshot-backed device pod on this node.
type Pusher ¶
type Pusher interface {
Push(ctx context.Context, repo string, img v1.Image, baseLayers int, sourceRepo string) (ref string, digest v1.Hash, err error)
}
Pusher publishes snapshot images; *snapshot.Registry implements it.
type RetainedCounter ¶
type RetainedCounter interface {
RetainedBytes(ctx context.Context, tenant string, live map[string]bool) (int64, error)
}
RetainedCounter is the state a tenant keeps in the repositories of labs that are gone; *snapshot.Registry implements it.
type Runtime ¶
type Runtime interface {
// Inspect finds a container by id.
Inspect(ctx context.Context, containerID string) (Container, error)
// Diff returns the uncompressed layer tar of the container's writable layer
// against its parent snapshot. With freeze the container is frozen and its
// filesystem synced while the diff is computed, so the result is consistent;
// without it (the container has exited) nothing runs and nothing is frozen.
// Closing the reader releases everything the diff held.
Diff(ctx context.Context, c Container, freeze bool) (io.ReadCloser, error)
// LoadImage opens the image a container was created from out of the local
// image store, with the snapshot chain already on the node.
LoadImage(ctx context.Context, imageRef string) (v1.Image, error)
// Exits reports the ids of containers whose task exited, until ctx ends.
Exits(ctx context.Context) (<-chan string, error)
}
Runtime is the container runtime facade.
type Snapshot ¶
type Snapshot struct {
// Image is the pullable reference with digest; empty when the writable
// layer holds nothing beyond the image the device started from and that
// image is the base image.
Image string
At time.Time
SizeBytes int64
Layers int32
}
Snapshot is the outcome of a successful snapshot.
type SpaceGuard ¶
SpaceGuard says whether the registry has room for a push of about this many bytes; *snapshot.Capacity implements it.