Documentation
¶
Overview ¶
Package snapimport implements the `d8 snapshot upload` command: it reconstructs a snapshot tree in a target namespace from a local archive produced by `d8 snapshot download`, walking the tree bottom-up and, per node, creating an import-mode CR, importing volume data for data leaves (via SVDM DataImport), and POSTing the node's manifests plus its direct child refs to the state-snapshotter manifests-and-children-refs-upload aggregated subresource.
Index ¶
- Constants
- Variables
- func Run(ctx context.Context, cfg Config) error
- func WithSkipUnsupportedFSEntries(ctx context.Context) context.Context
- type ChildRef
- type Config
- type ManifestUploader
- type PlanLimits
- type PlannedNode
- func BuildPlan(rootDir string) ([]PlannedNode, error)
- func BuildPlanWithLimits(rootDir string, limits PlanLimits) ([]PlannedNode, error)
- func BuildPlanWithLimitsAndOptions(rootDir string, limits PlanLimits, options archive.SnapshotYAMLReadOptions) ([]PlannedNode, error)
- func BuildPlanWithOptions(rootDir string, options archive.SnapshotYAMLReadOptions) ([]PlannedNode, error)
- type VolumeImporter
Constants ¶
const DefaultControlRequestTimeout = 30 * time.Second
DefaultControlRequestTimeout bounds one Kubernetes or aggregated-API request.
Variables ¶
var ErrForeignDataImport = errors.New("foreign DataImport collision")
ErrForeignDataImport is returned when a shared DataImport name is occupied by an object whose content identity or normalized upload spec belongs to another archive.
var ErrPlanBudget = errors.New("snapshot import plan budget exceeded")
ErrPlanBudget is returned when archive planning exceeds a configured resource limit.
var ErrRawBlockSizeMismatch = errors.New("raw block size mismatch")
ErrRawBlockSizeMismatch is returned by blockTotalSize when a raw (codec none) data.bin file's on-disk size does not match the size captured in the archive's VolumeInfo. Unlike a compressed payload, a raw payload has no separate decompressed size to fall back on — stat size and captured size are the SAME quantity — so any disagreement means a truncated, corrupted, or mismatched archive. Checking this before any HEAD/PUT keeps the failure deterministic and sends zero HTTP requests, instead of streaming a wrong byte count to the importer and only discovering the mismatch mid-transfer.
Functions ¶
func Run ¶
Run imports a local snapshot archive into the target namespace. It plans the tree bottom-up, then:
- creates every import-mode CR TOP-DOWN (parents first) so each child carries a child->parent ownerRef stamped with the parent's server-assigned UID (the API server requires a non-empty uid on ownerReferences, and the state-snapshotter import binders resolve a leaf's parent SnapshotContent through that ownerRef);
- waits for the bind-first contract: the manifests upload is refused with 409 ImportContentNotBound until a node's status.boundSnapshotContentName is set, so Run blocks (waitForBinds) until EVERY planned node reports a bound SnapshotContent before uploading anything. The binder creates and binds contents top-down from the markers of pass 1, independent of the upload, so the CLI can wait for all nodes collectively;
- uploads EVERY node's manifests plus its direct child refs (pass 2a) BEFORE importing any volume bytes (pass 2b). The two are sequenced, not interleaved, because a data leaf's SVDM DataImport stays Pending ("awaiting target leaf bound SnapshotContent") until the leaf VolumeSnapshot is bound, which needs the parent SnapshotContent, which needs the parent's manifests upload — so finishing a leaf (including waiting on its DataImport) before its ancestors' manifests are up would deadlock until timeout;
- pass 2b runs data-leaf uploads with bounded concurrency (cfg.Workers goroutines via errgroup.SetLimit); the first leaf error cancels all in-flight siblings via the derived ctx;
finally waiting for the root Snapshot and its bound SnapshotContent to become Ready.
Types ¶
type ChildRef ¶
type ChildRef struct {
APIVersion string `json:"apiVersion"`
Kind string `json:"kind"`
Name string `json:"name"`
}
ChildRef is a direct-child reference for a manifests-and-children-refs-upload payload. The child namespace is implicit (it is always the upload target namespace), mirroring the server-side SnapshotChildRef shape.
type Config ¶
type Config struct {
// Namespace is the target namespace the snapshot tree is reconstructed into.
Namespace string
// InputDir is the root archive directory produced by `d8 snapshot download`.
InputDir string
// TTL is the DataImport TTL used for data-leaf imports.
TTL string
// Timeout bounds the per-node readiness/completion waits.
Timeout time.Duration
// ControlRequestTimeout bounds each Kubernetes and aggregated-API request
// independently of the longer readiness/completion polling budget.
ControlRequestTimeout time.Duration
// PollInterval is the readiness polling cadence.
PollInterval time.Duration
// SelectedNodeKind restricts the import to a single node subtree when non-empty.
// After BuildPlan the plan is filtered to the selected node and its descendants.
// The selected node becomes the import root for waitRootReady; it must be a core
// Snapshot or a CSI VolumeSnapshot data leaf (domain aggregators are rejected).
SelectedNodeKind string
// SelectedNodeName is the name of the selected node. Required when SelectedNodeKind is set.
SelectedNodeName string
// Uploader posts manifests-and-children-refs-upload (aggregated API).
Uploader ManifestUploader
// Volumes imports data-leaf volume bytes (DataImport + HTTP upload).
Volumes VolumeImporter
// Dynamic creates import-mode CRs and reads readiness status.
Dynamic dynamic.Interface
// Workers is the maximum number of data-leaf volume uploads to run concurrently in
// pass 2b. Defaults to 5 when zero. Block-volume uploads stream-decode directly into
// the PUT (see snapimport.putBlock), so raising Workers no longer multiplies temporary
// disk usage — only per-worker in-memory codec buffers.
Workers int
// AllowExisting, when true, downgrades the namespace preflight conflict check to a
// warning instead of an error. Import-mode markers from a prior run of this import
// are never treated as conflicts regardless of this flag. When false (default), the
// run aborts before any cluster mutation if conflicting non-import-mode objects exist.
AllowExisting bool
// AllowUnauthenticatedLegacy permits pre-version snapshot.yaml metadata that has no
// metadata checksum. The default rejects it as a possible version-downgrade attack.
AllowUnauthenticatedLegacy bool
// Mapper resolves node GVKs to resources.
Mapper meta.RESTMapper
// Log receives progress output.
Log *slog.Logger
// Progress, when non-nil, receives per-stream byte increments for each data-leaf
// volume upload. Each leaf gets its own Stream; nil disables progress reporting
// and leaves upload behaviour unchanged.
Progress progress.Sink
// contains filtered or unexported fields
}
Config holds all parameters for one import run.
type ManifestUploader ¶
type ManifestUploader interface {
UploadManifests(ctx context.Context, ref aggapi.NodeRef, body []byte) ([]byte, error)
}
ManifestUploader posts a node's manifests-and-children-refs-upload payload. It is satisfied by *aggapi.Client and stubbed in tests.
type PlanLimits ¶
type PlanLimits struct {
MaxDepth int
MaxNodes int
MaxManifestBytes int64
MaxTotalMetadataBytes int64
MaxManifestsPerNode int
}
PlanLimits bounds archive traversal and metadata retained by BuildPlanWithLimits. The root is at depth zero and counts toward MaxNodes. MaxManifestBytes applies independently to snapshot.yaml and each manifest file.
func DefaultPlanLimits ¶
func DefaultPlanLimits() PlanLimits
DefaultPlanLimits returns the limits used by BuildPlan. Planning visits at most 10,000 nodes and 64 child directories below the root. It retains at most 256 MiB of raw metadata, accepts at most 10,000 manifests per node, and reads at most 16 MiB from snapshot.yaml or one manifest.
type PlannedNode ¶
type PlannedNode struct {
// Dir is the absolute path of the node directory in the archive.
Dir string
// APIVersion/Kind/Name identify the snapshot CR for this node (from snapshot.yaml).
APIVersion string
Kind string
Name string
// SourceNamespace is the namespace recorded in the archive (informational; the import
// always targets the user-supplied namespace).
SourceNamespace string
// Manifests are the node's own captured manifests (from manifests/), the same shape
// the server returned from manifests-download.
Manifests []unstructured.Unstructured
// Children are the direct child snapshot refs (from snapshots/<child>/snapshot.yaml).
Children []ChildRef
// DataFile is the absolute path to the node's single-volume block data file
// (data.bin[.<ext>]) when present; empty when the node carries no importable
// block volume data.
DataFile string
// Ext is DataFile's codec extension, resolved by
// archive.ClassifyBlockPayload alongside DataFile: "" for the raw/none
// codec, ".zst", ".gz", or ".lz4" — matching compress.Codec.Ext. Callers
// MUST use this field instead of filepath.Ext(DataFile): filepath.Ext on
// the raw name "data.bin" returns ".bin", not "" (see
// archive.BlockPayload.Ext's doc comment). Empty when HasBlockData() is
// false.
Ext string
// FilesystemData is true when the node carries filesystem-volume data (data.tar).
FilesystemData bool
// TarFile is the absolute path to the node's filesystem-volume data file (data.tar).
// It is always set when FilesystemData is true.
TarFile string
// SourceObjectRef carries the structured spec.sourceRef from a domain snapshot CR
// ({apiVersion,kind,name} of the source object), read from snapshot.yaml. Nil for
// core Snapshot nodes and CSI VolumeSnapshot data leaves.
SourceObjectRef *archive.SourceObjectRef
// StorageClassName/Size/VolumeMode are the captured scratch-volume parameters of this
// leaf's volume, read from snapshot.yaml Volumes[0]. They feed the PopulateData
// DataImport spec.storageParams on re-import (storageClassName and size are required by
// the DataImport CRD; volumeMode is optional). Empty for structural/aggregator nodes
// that own no volume data.
StorageClassName string
Size string
VolumeMode string
// NodeChecksum is the full checksum verified by the archive integrity preflight.
NodeChecksum string
// SizeBytes is Size parsed once into its canonical byte count before cluster mutation.
SizeBytes int64
// PayloadKind and Codec are the classified on-disk upload representation.
PayloadKind string
Codec string
// DataImportIdentity is the versioned full content identity used to qualify and
// validate the shared DataImport object.
DataImportIdentity string
// contains filtered or unexported fields
}
PlannedNode is one archive node resolved for import. Nodes are returned by BuildPlan in post-order (deepest descendants first, root last) so that data leaves and child SnapshotContents materialise before their parents reference them.
func BuildPlan ¶
func BuildPlan(rootDir string) ([]PlannedNode, error)
BuildPlan walks the archive rooted at rootDir and returns its nodes in post-order (leaves first, root last). Each node's own manifests, direct child refs, and volume data file (if any) are resolved.
func BuildPlanWithLimits ¶
func BuildPlanWithLimits(rootDir string, limits PlanLimits) ([]PlannedNode, error)
BuildPlanWithLimits builds an import plan subject to explicit traversal and metadata limits.
func BuildPlanWithLimitsAndOptions ¶
func BuildPlanWithLimitsAndOptions( rootDir string, limits PlanLimits, options archive.SnapshotYAMLReadOptions, ) ([]PlannedNode, error)
BuildPlanWithLimitsAndOptions builds an import plan with explicit resource and compatibility policy.
func BuildPlanWithOptions ¶
func BuildPlanWithOptions( rootDir string, options archive.SnapshotYAMLReadOptions, ) ([]PlannedNode, error)
BuildPlanWithOptions builds an import plan under an explicit snapshot.yaml compatibility policy.
func (PlannedNode) HasBlockData ¶
func (n PlannedNode) HasBlockData() bool
HasBlockData reports whether the node carries a single-volume block data file.
type VolumeImporter ¶
type VolumeImporter interface {
// DataImportName returns the deterministic identity-qualified DataImport name for the leaf.
// The DataImport is created bottom-up immediately before its upload — its TTL is an idle
// timer that starts at importer-pod start, so a freshly created importer must not sit
// idle waiting for earlier siblings to finish.
DataImportName(leaf PlannedNode) string
// EnsureDataImport creates (idempotently) the DataImport for the leaf and returns its name.
EnsureDataImport(ctx context.Context, leaf PlannedNode, namespace string) (string, error)
// UploadVolumeData waits for the DataImport to become ready, streams the leaf's block
// or filesystem data, finalises, and waits for completion. onProgress, when non-nil, is
// called as raw bytes become known durable, including a validated server-side resume
// prefix and each chunk or file upload; nil disables progress reporting and leaves upload
// behaviour unchanged. setTotal, when non-nil, is
// called with the expected total byte count: once, before any bytes are sent, on the
// block path (the total is known up front from leaf.Size); progressively, with a
// growing running sum as each file's exact size becomes known, on the filesystem path
// (see sendVolumeData for why an accurate a-priori FS total is not free). activate, when
// non-nil, is called at least once but ONLY when a real transfer occurs — never on a
// leaf whose upload is entirely a server-side skip (offset==totalSize on the block HEAD,
// or every file already done on the filesystem HEAD) — so the caller's progress stream
// can distinguish a genuine transfer from a full resume-skip (see progress.Stream.Activate).
UploadVolumeData(ctx context.Context, leaf PlannedNode, diName, namespace string, setTotal func(int64), onProgress func(int), activate func()) error
}
VolumeImporter imports a data leaf's volume bytes by creating an SVDM DataImport, waiting for the importer to be ready, streaming the archive bytes, finalising the upload, and waiting for the durable artifact to be produced. It is satisfied by clusterVolumeImporter and stubbed in tests.
func NewClusterVolumeImporter ¶
func NewClusterVolumeImporter( dyn dynamic.Interface, sc *transport.Client, ttl string, wait, poll time.Duration, log *slog.Logger, ) VolumeImporter
NewClusterVolumeImporter builds the live VolumeImporter. ttl is the DataImport TTL, wait bounds the per-DataImport readiness/completion waits, and poll is the polling cadence. Block-volume uploads stream-decode directly into the PUT (see putBlock), so no scratch directory for decompressed temporary files is needed.