snapshot

package
v1.7.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 20, 2026 License: Apache-2.0 Imports: 27 Imported by: 0

Documentation

Overview

Package snapshot holds the platform-independent logic of device state persistence: which paths are kept, the size quota, the layer chain rules and the tar plumbing that filters and squashes snapshot layers. It has no containerd or Kubernetes dependency in its pure parts; the registry side is in registry.go and image.go.

Index

Constants

View Source
const (
	DefaultDebounce   = 5 * time.Second
	DefaultWriteQuota = int64(512 << 20)
	DefaultMaxLayers  = 10
	// DefaultMaxFileSize: a regular file larger than this is left out of a snapshot.
	DefaultMaxFileSize = int64(256 << 20)
	// DefaultMaxEntries is the most entries (files, directories, links) one snapshot layer may hold. The byte quota counts
	// only the bytes of regular files, so a million empty files cost nothing against it but a lot of memory and time to
	// merge and unpack; this is the cap on them.
	DefaultMaxEntries = 100000
	// MaxEntryHeaderBytes is the most the names and extended attributes of one entry may take; an entry over it is left out.
	MaxEntryHeaderBytes = 8192
)

Defaults of the policy; the chart values and the operator configuration use the same numbers.

View Source
const (
	AnnotationTenant     = "cybericebox.com/tenant"
	AnnotationStateBytes = "cybericebox.com/state-bytes"
)

The annotations of a snapshot manifest: whose it is and how much state it holds (the uncompressed bytes the quota counts). They let the registry itself say what a repository of a lab that is gone still takes from whom.

View Source
const AnnotationLayerSize = "com.cybericebox.state.size"

AnnotationLayerSize marks a manifest layer as a snapshot layer and records its uncompressed size. Layers without it belong to the base image.

View Source
const BaseRepo = "base"

BaseRepo is the registry repository that holds the base image layers shared by all devices, so a device's first snapshot mounts them instead of uploading.

View Source
const Latest = "latest"

Latest is the tag every snapshot of a device is pushed under; the status records the digest, so older manifests are simply untagged and left to the registry's garbage collection.

View Source
const MaxSkippedListed = 10

MaxSkippedListed caps the skipped files named in a warning.

View Source
const RepoPrefix = "lab"

RepoPrefix is the top-level repository path of every lab's snapshots.

Variables

View Source
var DefaultExcludePaths = []string{"/tmp", "/var/tmp", "/run"}

DefaultExcludePaths are never snapshotted unless the operator overrides the list.

View Source
var ErrEntries = errors.New("too many files in the snapshot")

ErrEntries is returned when a snapshot layer (or the squashed chain) holds more entries than the policy allows.

View Source
var ErrNoChange = errors.New("no changes to snapshot")

ErrNoChange tells that the filtered layer holds nothing.

View Source
var ErrQuota = errors.New("write quota exceeded")

ErrQuota is returned when a snapshot would exceed the device's size quota.

View Source
var ErrRegistryFull = errors.New("the snapshot registry is nearly full")

ErrRegistryFull is the refusal of a push while the registry has too little room left.

View Source
var SystemExcludePaths = []string{
	"/dev", "/proc", "/sys",
	"/etc/hosts", "/etc/hostname", "/etc/resolv.conf", "/etc/mtab",
	"/var/run/secrets", "/run/secrets",
}

SystemExcludePaths are always excluded, whatever the operator configures. They are the mount points and files the container runtime and kubelet create or bind over at every start (runtime filesystems, hostname and resolver files, the service account token directory), so a snapshot of them would only hold empty placeholders. Flag and secret values reach a device as environment variables from a Secret (envFrom) and never as files in the writable layer; a future file-based injection must add its path here.

Functions

func Annotated

func Annotated(img v1.Image, tenant string, stateBytes int64) v1.Image

Annotated is img with the tenant and state size annotated on its manifest.

func CheckQuota

func CheckQuota(existing, added, max int64) error

CheckQuota reports ErrQuota (wrapped with the numbers) when the state already kept plus the new layer exceeds max. A zero max means no limit.

func CleanPath

func CleanPath(p string) string

CleanPath returns the absolute, cleaned form of a path inside the container.

func LabRepoPrefix

func LabRepoPrefix(namespace, lab string) string

LabRepoPrefix is the repository prefix shared by all devices of one lab.

func LoadImage

func LoadImage(ctx context.Context, src BlobSource, manifest v1.Hash) (v1.Image, error)

LoadImage exposes an image whose single-platform manifest (and everything it references) is present in src as a v1.Image.

func NeedSquash

func NeedSquash(deltaLayers, maxLayers int) bool

NeedSquash reports whether adding one more snapshot layer to a chain that has deltaLayers of them must squash the chain into one layer.

func Repo

func Repo(namespace, lab, device string) string

Repo names the repository that holds one device's snapshots.

Types

type BlobSource

type BlobSource interface {
	// ReadBlob returns the blob and its size.
	ReadBlob(ctx context.Context, digest v1.Hash) (io.ReadCloser, int64, error)
}

BlobSource reads immutable image blobs. The native source prefers containerd content and can recover missing manifest-listed packed layers from the original registry when containerd discards them after unpacking.

type Capacity

type Capacity struct {
	Registry        *Registry
	Total           int64
	ReserveFraction float64
	TTL             time.Duration
	// contains filtered or unexported fields
}

Capacity says whether the registry has room for a push: its volume is Total bytes, and a push is refused once what is stored plus what the push adds would take more than all but ReserveFraction of it. Usage is measured at most once per TTL (it reads every manifest).

func (*Capacity) Check

func (c *Capacity) Check(ctx context.Context, bytes int64) error

Check returns ErrRegistryFull (wrapped with the numbers) when adding bytes would pass the line. With no Total it never refuses.

type Chain

type Chain struct {
	// Base is the number of leading layers that belong to the base image.
	Base int
	// Sizes are the uncompressed sizes of the snapshot layers, oldest first.
	Sizes []int64
}

Chain describes the snapshot layers of an image.

func Build

func Build(run v1.Image, newTar string, newBytes int64, pol Policy, workDir string) (v1.Image, Chain, error)

Build returns the image that a snapshot of a container started from run should publish: run plus the layer in newTar (an uncompressed, already filtered layer tar of newBytes bytes). Over the quota it returns ErrQuota; when the chain would outgrow pol.MaxLayers it squashes the snapshot layers of run and the new one into a single layer. Temporary files go to workDir, which must outlive the use of the returned image.

func ChainOf

func ChainOf(img v1.Image) (Chain, error)

ChainOf reads the snapshot chain of an image from its manifest annotations. Snapshot layers are always the trailing ones.

func (Chain) Bytes

func (c Chain) Bytes() int64

Bytes is the total uncompressed size of the snapshot layers.

func (Chain) Layers

func (c Chain) Layers() int

Layers is the number of snapshot layers.

type IDMap

type IDMap []IDMapping

IDMap is the id mapping of a user namespace. An empty map (no user namespace) changes nothing.

func (IDMap) ToContainer

func (m IDMap) ToContainer(host int) (id int, ok bool)

ToContainer translates a host id to the id inside the container. ok is false when the id is outside the map.

type IDMapping

type IDMapping struct {
	ContainerID uint32
	HostID      uint32
	Size        uint32
}

IDMapping is one line of a uid_map or gid_map: the Size ids from ContainerID inside the container are HostID.. on the host.

type IDMaps

type IDMaps struct {
	UID, GID IDMap
}

IDMaps are the user and group maps of one container.

type Opener

type Opener func() (io.ReadCloser, error)

Opener returns a fresh reader of an uncompressed layer tar.

type Policy

type Policy struct {
	Debounce     time.Duration
	ExcludePaths []string
	WriteQuota   int64
	// MaxFileSize: a regular file over it is skipped (like an excluded path, for that file only).
	MaxFileSize int64
	MaxLayers   int
	// MaxEntries caps the entries of one snapshot layer (and of a squashed chain); over it the snapshot is refused (the last good
	// one is kept and the status carries a warning), like the byte quota.
	MaxEntries int
}

Policy is the per-device snapshot policy.

func NewPolicy

func NewPolicy(debounce time.Duration, exclude []string, maxBytes int64, maxLayers int) Policy

NewPolicy fills unset fields with the defaults and normalises the exclude list (absolute, cleaned, no duplicates), adding the system paths.

func (Policy) Excluded

func (p Policy) Excluded(name string) bool

Excluded reports whether p, or one of its ancestors, is an excluded path.

func (Policy) WithMaxEntries

func (p Policy) WithMaxEntries(n int) Policy

WithMaxEntries sets the entry cap (n <= 0 keeps the default).

func (Policy) WithMaxFileSize

func (p Policy) WithMaxFileSize(n int64) Policy

WithMaxFileSize sets the per-file size limit (n <= 0 keeps the default).

type Registry

type Registry struct {
	// Host is host:port of the registry, e.g. "localhost:5035". Hosts on the
	// loopback or a private network are spoken to over plain HTTP.
	Host string
	Auth authn.Authenticator
	// Transport overrides the HTTP transport (tests).
	Transport http.RoundTripper
}

Registry is a client of the snapshot registry (zot) with write access.

func (*Registry) DeleteRepo

func (r *Registry) DeleteRepo(ctx context.Context, repo string) error

DeleteRepo deletes every tagged manifest of a repository; the registry's garbage collection then frees the blobs and the untagged manifests.

func (*Registry) DeleteSnapshot added in v1.1.0

func (r *Registry) DeleteSnapshot(ctx context.Context, repo, image string) error

DeleteSnapshot removes only the immutable manifest referenced by an owned Device. No cache/base/shared blob or physical GC is touched.

func (*Registry) HasBlob

func (r *Registry) HasBlob(ctx context.Context, repo name.Repository, d v1.Hash) bool

func (*Registry) HasBlobIn

func (r *Registry) HasBlobIn(ctx context.Context, repo string, d v1.Hash) bool

HasBlobIn reports whether the blob is stored in the repository named by its path.

func (*Registry) Image

func (r *Registry) Image(ctx context.Context, repo string) (v1.Image, error)

Image reads the manifest of repo:latest back (for tests and diagnostics).

func (*Registry) Options

func (r *Registry) Options(ctx context.Context) []remote.Option

func (*Registry) Push

func (r *Registry) Push(ctx context.Context, repo string, img v1.Image, baseLayers int, sourceRepo string) (string, v1.Hash, error)

Push makes img available as repo:latest. The base layers of img (the first baseLayers) are placed in the shared base repository once and mounted into repo, so a device's first snapshot does not upload the whole base image. sourceRepo, when set, is a repository that already holds the base layers (the image cache's repository of the base image): they are mounted from it, so nothing is uploaded at all. It returns the pullable reference by digest.

func (*Registry) Ref

func (r *Registry) Ref(repo string, digest v1.Hash) string

Ref is the pullable reference of a pushed manifest.

func (*Registry) Repo

func (r *Registry) Repo(repo string) (name.Repository, error)

Repo and Options expose the repository name and request options to tests of other packages.

func (*Registry) Repos

func (r *Registry) Repos(ctx context.Context) ([]string, error)

Repos lists the repositories of the registry.

func (*Registry) RepositoryEmpty added in v1.1.0

func (r *Registry) RepositoryEmpty(ctx context.Context, repo string) (bool, error)

RepositoryEmpty confirms actual absence of tagged manifests after DeleteRepo. Untagged history/shared blobs remain physical unknown until verified GC.

func (*Registry) RetainedBytes

func (r *Registry) RetainedBytes(ctx context.Context, tenant string, live map[string]bool) (int64, error)

RetainedBytes is the state a tenant still has in the repositories no live device owns (live is the set of repository names of the existing devices): the snapshots of labs that are gone and wait out their retention. It reads the annotations of each repository's latest manifest, so a repository pushed before they existed counts for nothing.

func (*Registry) Supersede

func (r *Registry) Supersede(ctx context.Context, repo, oldRef string, keep v1.Image) error

Supersede removes what a new snapshot has replaced: the old manifest of the repository, and the blobs only it used, at once instead of after the registry's garbage collection (which keeps unreferenced blobs for hours). oldRef is the pullable reference ("host/repo@sha256:...") of the manifest that was current before keep. A missing manifest or blob is fine, and so is a registry that does not allow deleting blobs: the garbage collection then does it.

func (*Registry) Usage

func (r *Registry) Usage(ctx context.Context) (int64, error)

Usage is what the registry stores for the images it has a name for: every blob once, however many repositories mount it, summed over every tag of every repository (the snapshots, the shared base, the image cache).

type SkippedFile

type SkippedFile struct {
	Path string
	Size int64
}

SkippedFile is a regular file left out of a snapshot for its size.

type Stats

type Stats struct {
	// Entries is the number of tar entries written.
	Entries int
	// Bytes is the total size of the regular files written (uncompressed).
	Bytes int64
	// Dropped is the number of entries left out.
	Dropped int
	// RefusedEntries counts omitted data outside explicitly excluded paths.
	RefusedEntries int
	// Skipped are the first MaxSkippedListed regular files left out for being larger than
	// the policy's MaxFileSize; SkippedTotal counts all of them.
	Skipped      []SkippedFile
	SkippedTotal int
	// Unmapped counts the owner ids that were outside the user namespace map (written as 0).
	Unmapped int
}

Stats describe a filtered or merged layer.

func FilterLayer

func FilterLayer(in io.Reader, out io.Writer, pol Policy) (Stats, error)

FilterLayer copies an uncompressed layer tar from in to out, leaving out the entries (and whiteouts) of excluded paths and the regular files larger than the policy's MaxFileSize (with the hard links to them). Whiteouts are empty files and never skipped for size. The result is a complete tar.

func FilterLayerMapped

func FilterLayerMapped(in io.Reader, out io.Writer, pol Policy, ids IDMaps) (Stats, error)

FilterLayerMapped is FilterLayer for a container in a user namespace: the diff holds the host ids of the files (the shifted ids of that pod's namespace), and the snapshot must hold the ids inside the container, so every owner is translated through ids. An id outside the map is written as 0 and counted in Stats.Unmapped. Containerd's synthetic root-owned whiteout markers already carry container-relative zero ids and are not translated. Empty maps change nothing.

func MergeLayers

func MergeLayers(layers []Opener, out io.Writer, maxEntries ...int) (Stats, error)

MergeLayers squashes uncompressed layer tars (oldest first) into one tar, keeping the effect of applying them in order on top of any lower layer: the newest version of every path wins, deleted paths disappear, and the whiteout and opaque markers that still have to hide lower layers are kept. Markers are written first, so an unpacker that removes on a whiteout never removes a file of the same layer.

maxEntries (optional, 0 = no limit) bounds the entries of the result: the merge keeps a map of every path in memory, so it stops with ErrEntries as soon as the survivors pass the cap.

func (Stats) SkippedWarning

func (s Stats) SkippedWarning(maxFileSize int64) string

SkippedWarning is the status warning for the files skipped for size; empty when none.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL