archive

package
v0.4.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 16, 2026 License: AGPL-3.0 Imports: 16 Imported by: 0

Documentation

Overview

Package archive turns scanned node trees into PXAR archive streams.

A planner determines the archive structure and entry order up front from scan-time metadata; byte offsets are bound as late as possible via fstat when a file is dispatched for reading, goodbye tables and hardlink offsets from actually-emitted positions. File contents are read by a pool of workers and reassembled in plan order through a bounded reorder buffer, so generation is asynchronous by default while the output stream stays byte-identical to synchronous mode.

Supports single-stream v1 output (plus a .pcat1 catalog) and split v2 output (.mpxar metadata + .ppxar payload). Exclude patterns given via Options.Scan.Exclude are recorded in the archive the way the official client records --exclude (a .pxarexclude-cli root file in v1, the prelude record in v2).

Index

Examples

Constants

View Source
const DefaultPaddingThreshold = 0.1

DefaultPaddingThreshold is the share of reused chunk bytes that may belong to files not referenced by the new archive (the previous chunks at the edges of a run of unchanged files carry bytes of neighbouring files). Above it the run is read and chunked again. Matches the upstream client.

Variables

This section is empty.

Functions

This section is empty.

Types

type Archive

type Archive struct {
	// contains filtered or unexported fields
}

Archive accumulates roots and generates PXAR streams. Not safe for concurrent use; scanning happens at generation time, so the filesystem state that matters is the one at Generate, not at Add.

func New

func New(opts Options) (*Archive, error)

New returns an empty Archive.

func (*Archive) AddDirectories

func (a *Archive) AddDirectories(paths []string) error

AddDirectories queues several directory trees.

func (*Archive) AddDirectoriesAs added in v0.1.5

func (a *Archive) AddDirectoriesAs(paths map[string]string) error

AddDirectoriesAs queues several directory trees, each under the name it maps to (path -> as). Entries are processed in sorted path order so the first error reported is deterministic; on error, paths sorting before the failing one have already been queued.

func (*Archive) AddDirectory

func (a *Archive) AddDirectory(path string) error

AddDirectory queues a directory tree. A single queued directory becomes the archive root; any other combination is placed under a virtual root named Options.Name, where the directory appears under its own basename.

func (*Archive) AddDirectoryAs added in v0.1.5

func (a *Archive) AddDirectoryAs(path, as string) error

AddDirectoryAs queues a directory tree like AddDirectory, but under the name as instead of the directory's basename. as must be a single path component: non-empty, not "." or "..", and free of '/' and NUL. Two queued roots may not share a name. When the directory is the archive's only root the PXAR root entry carries no name, so as only affects the multi-root layout.

func (*Archive) AddFile

func (a *Archive) AddFile(path string) error

AddFile queues a single non-directory entry, placed under the virtual root.

func (*Archive) AddFiles

func (a *Archive) AddFiles(paths []string) error

AddFiles queues several non-directory entries.

func (*Archive) AddStream

func (a *Archive) AddStream(name string, size int64, r io.Reader) error

AddStream queues a virtual file backed by r. size must be declared up front; if r yields a different byte count the output is padded or truncated to exactly size bytes and a WarnSizeChanged warning is issued. The reader is consumed during generation, so an Archive with streams generates once.

func (*Archive) AddTree added in v0.4.0

func (a *Archive) AddTree(root *scan.Node) error

AddTree queues a virtual directory tree the caller built: KindDirectory nodes with KindStream leaves (scan.StreamNode, or nodes with Stat filled in by the caller — mode, uid/gid, mtime and the declared size are what the archive records, and what metadata change detection compares against the previous snapshot). Children are sorted into archive order; names must be unique within a directory. A single tree and nothing else is the archive root (its own name is not recorded); otherwise it appears under the virtual root with root.Name. Scan options (exclusions, filters) do not apply: the caller decides what the tree contains. Readers are consumed during generation, so an Archive with a tree generates once.

func (*Archive) CatalogEntryName

func (a *Archive) CatalogEntryName() string

CatalogEntryName returns the name the catalog's top-level entry carries: the archive's dynamic index name. It must match the name the pxar index is uploaded under, or PBS's snapshot browser resolves nothing. Options.Name may be a plain name ("root"), an archive name ("root.pxar"), or the full index name ("root.pxar.didx"); the suffix is completed as needed.

func (*Archive) EstimatedSizeV1

func (a *Archive) EstimatedSizeV1() (int64, error)

EstimatedSizeV1 scans the queued roots and returns the archive size implied by scan-time metadata. An estimate: payload sizes are re-bound at emission, so files changed after the estimate shift the real total.

func (*Archive) EstimatedSizeV2

func (a *Archive) EstimatedSizeV2() (meta, payload int64, err error)

EstimatedSizeV2 scans the queued roots and returns the sizes of the two v2 streams implied by scan-time metadata. Estimates, like EstimatedSizeV1.

func (*Archive) GenerateCatalog

func (a *Archive) GenerateCatalog(ctx context.Context) (io.ReadCloser, error)

GenerateCatalog builds the node tree and returns the .pcat1 catalog as a stream. It uses scan-time metadata: sizes of files that change afterwards may differ from the archive's late-bound payload sizes (the catalog is a lookup aid, not the archive's source of truth).

func (*Archive) GenerateV1

func (a *Archive) GenerateV1(ctx context.Context) (io.ReadCloser, error)

GenerateV1 builds the node tree and returns the archive as a stream. Generation runs concurrently with consumption; errors surface from Read, and Close cancels generation.

Example

Archive-only use: generate a pxar v1 stream from a directory tree and write it to a file. The returned stream is produced concurrently with consumption, so it can just as well be piped, chunked, or uploaded elsewhere.

package main

import (
	"context"
	"io"
	"log"
	"os"

	"github.com/osshield/gopbs/archive"
)

func main() {
	a, err := archive.New(archive.Options{})
	if err != nil {
		log.Fatal(err)
	}
	if err := a.AddDirectory("/etc"); err != nil {
		log.Fatal(err)
	}

	stream, err := a.GenerateV1(context.Background())
	if err != nil {
		log.Fatal(err)
	}
	defer stream.Close()

	out, err := os.Create("/tmp/etc.pxar")
	if err != nil {
		log.Fatal(err)
	}
	defer out.Close()
	if _, err := io.Copy(out, stream); err != nil {
		log.Fatal(err)
	}
}

func (*Archive) GenerateV2

func (a *Archive) GenerateV2(ctx context.Context) (meta, payload io.ReadCloser, err error)

GenerateV2 builds the node tree and returns the split v2 archive as two streams: the metadata stream (.mpxar) and the payload stream (.ppxar). Both derive from one plan and are generated concurrently; the caller must consume them concurrently too (metadata progress can await payload dispatch progress, so draining one stream to the end before starting the other may stall) and close both. Errors surface from Read; closing a stream aborts that stream's generation.

func (*Archive) ReuseStats added in v0.3.0

func (a *Archive) ReuseStats() ReuseStats

ReuseStats reports the outcome of metadata change detection for the most recent GenerateV2 call (zero values when Options.Previous was nil).

type Options

type Options struct {
	// Name is the archive name; it becomes the virtual root directory's name
	// when more than one root is added. Required in that case.
	Name string
	// Workers is the payload-read parallelism: 0 = GOMAXPROCS, 1 = fully
	// synchronous generation. Both modes produce byte-identical output.
	Workers int
	// Buffer is the reorder-buffer memory budget in bytes (async mode);
	// 0 = 64 MiB.
	Buffer int64
	// OnWarn receives non-fatal events; nil discards them.
	OnWarn func(Warning)
	// Scan carries scanner policy (SkipOnError, custom reader, quota
	// lookups) and the exclusion rules: Scan.Exclude patterns are recorded
	// in the archive the way proxmox-backup-client records its --exclude
	// options (a .pxarexclude-cli file emitted last in the v1 root, the v2
	// prelude record), Scan.PxarExcludeFiles honours .pxarexclude files, and
	// Scan.Filter is a free-form veto that leaves no trace in the archive.
	// The .pxarexclude-cli file is a 0600 regular file owned by the
	// process's real uid/gid with mtime 0, exactly as upstream writes it.
	// Its OnWarn field is ignored; scan warnings arrive at OnWarn above as
	// WarnSkipped (or WarnBadPattern for unparsable .pxarexclude lines).
	Scan scan.Options

	// Previous enables metadata change detection for GenerateV2: regular
	// files whose metadata and size match the previous snapshot are not
	// read; the payload stream references the previous snapshot's chunks
	// instead. The payload stream is then framed (see package reuse) and
	// must be uploaded with pbs.UploadPXARv2, which understands the framing.
	// nil reads everything.
	Previous *Previous
	// PaddingThreshold bounds the share of reused chunk bytes that belong
	// to files not in the new archive; 0 = DefaultPaddingThreshold.
	PaddingThreshold float64
}

Options configures an Archive.

type PayloadChunk added in v0.3.0

type PayloadChunk struct {
	EndOffset uint64
	Digest    [32]byte
}

PayloadChunk is one entry of the previous payload stream's dynamic index: the chunk's end offset in that stream and its digest.

type Previous added in v0.3.0

type Previous struct {
	// contains filtered or unexported fields
}

Previous is the previous snapshot's split archive as needed for metadata change detection: every regular file's encoded metadata and payload reference, plus the payload stream's chunk layout.

func LoadPrevious added in v0.3.0

func LoadPrevious(meta io.Reader, chunks []PayloadChunk) (*Previous, error)

LoadPrevious indexes a previous v2 metadata stream (as uploaded, e.g. a local copy) against its payload stream's index. Files, sizes and offsets come from the metadata stream; chunk boundaries from the index.

func (*Previous) Files added in v0.3.0

func (p *Previous) Files() int

Files is the number of regular files known from the previous snapshot

type ReuseStats added in v0.3.0

type ReuseStats struct {
	// Files is the number of regular files in the archive
	Files int
	// ReusedFiles were not read: their payload was referenced from the
	// previous snapshot's chunks
	ReusedFiles int
	// ReusedBytes is the content size of the reused files
	ReusedBytes uint64
	// InjectedBytes is the size of the previous chunks referenced, including
	// padding (bytes of neighbouring files inside boundary chunks)
	InjectedBytes uint64
	// RejectedRuns counts runs of unchanged files that were read anyway
	// because the boundary padding exceeded the threshold
	RejectedRuns int
}

ReuseStats reports what metadata change detection achieved for one generation.

type WarnKind

type WarnKind int

WarnKind classifies non-fatal generation events.

const (
	// WarnSkipped: an entry could not be scanned and was omitted
	// (scan.Options.SkipOnError).
	WarnSkipped WarnKind = iota
	// WarnSizeChanged: a payload yielded a different byte count than its
	// bound size and was padded or truncated to keep the archive valid.
	WarnSizeChanged
	// WarnTorn: a file's size or mtime changed while its content was being
	// read; the archived bytes may interleave old and new content.
	WarnTorn
	// WarnBadPattern: a .pxarexclude line failed to parse and was ignored
	// (scan.Options.PxarExcludeFiles). Path is the .pxarexclude file, Err a
	// *scan.PatternError.
	WarnBadPattern
)

type Warning

type Warning struct {
	Kind   WarnKind
	Path   string
	Err    error // WarnSkipped: the scan error
	Bound  int64 // WarnSizeChanged: the size the archive committed to
	Actual int64 // WarnSizeChanged: the byte count the source yielded
}

Warning is a non-fatal event surfaced during generation.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL