Documentation
¶
Overview ¶
Package ingest builds content-addressed filesystem trees from a local directory or a single regular file: it walks the source, applies .amberignore filtering, splits file content by content-defined chunking, and streams every built object (children before parents) to the consumer. Objects exposes the raw object stream for consumers that route objects themselves; Dir writes the stream straight into a packstore.
Index ¶
- Constants
- func Dir(st *packstore.Store, path string, opts Opts) (key.Key, packstore.WriteStats, error)
- func Objects(path string, opts Opts) (iter.Seq2[fstree.Object, error], *key.Key, error)
- func Scan(dir string, noIgnore bool, jobs int) (files int64, bytes int64, err error)
- func ScanWith(dir string, opts Opts) (files int64, bytes int64, err error)
- type ChunkOpts
- type Opts
- type Progress
Constants ¶
const DefaultItemBits = 7
DefaultItemBits is the item-chunker bit width used when Opts leaves it zero: directory leaves and file index nodes average 2^7 entries.
const DefaultXattrInlineMax = 256
DefaultXattrInlineMax is the inline-xattr cap used when Opts leaves it zero: encoded xattrs larger than this many bytes spill to an XattrSet object.
Variables ¶
This section is empty.
Functions ¶
func Dir ¶
Dir builds the tree at path — a directory, or a single regular file — and stores every object in st via WriteParallel, returning the resolved root key and the write stats.
func Objects ¶
Objects builds the tree at path — a directory, or a single regular file — and returns the stream of every CAS object it produces plus a pointer to the root key, which is set once the stream has been consumed to completion without error. A directory root is a DirNode; a file root is the file's content key (Blob or FileNode).
Object order is unspecified — the store is a flat content-addressed bag and dedups by key — but per-file chunk order and per-directory entry order are preserved, so every object's key, and the root, are deterministic functions of the source tree and the chunking options.
func Scan ¶
Scan walks the directory tree at dir concurrently (ReadDir + Lstat only, no content reads) and returns the number of regular files and the total size of their content — the totals a progress display is sized by. Only regular-file bytes are counted, since only regular files are read during a build; symlinks, dirs and special files contribute nothing. The first stat/read error aborts the scan.
When noIgnore is false, .amberignore filtering rooted at dir is applied exactly as a build applies it, so the totals match what the build reads.
The fan-out mirrors the parallel build: each entry may run on a pooled goroutine, but when the pool is full the work runs inline on the current goroutine, so the recursion cannot deadlock waiting on a slot held by a descendant.
Types ¶
type ChunkOpts ¶
type ChunkOpts struct {
Byte *chunkers.ByteOpts // byte-chunker sizes; nil selects chunkers.Default{Min,Normal,Max}Size
ItemBits int // item chunker average run = 2^bits; 0 selects DefaultItemBits
XattrInlineMax int // xattr spill threshold in bytes; 0 selects DefaultXattrInlineMax
}
ChunkOpts sets the content-defined-chunking parameters. The zero value selects the library defaults; the parameters determine every object key, so two builds of the same tree agree only when their ChunkOpts agree.
type Opts ¶
type Opts struct {
// Jobs bounds the concurrent build workers; <1 selects GOMAXPROCS.
Jobs int
// Chunk sets the chunking parameters.
Chunk ChunkOpts
// NoIgnore disables .amberignore filtering (which only applies to
// directory builds).
NoIgnore bool
// Progress, when non-nil, receives build-progress events.
Progress Progress
// Exclude lists names directly under the root that are never ingested,
// whatever NoIgnore says. It applies to the root directory only; the
// same name deeper in the tree is ingested normally.
Exclude []string
}
Opts configures a tree build.
type Progress ¶
type Progress interface {
// FileDone records one regular file fully chunked.
FileDone()
// AddBytes records n source-content bytes chunked.
AddBytes(n int)
}
Progress receives build-progress events. Implementations must be safe for concurrent use; a nil Progress disables reporting. Totals for sizing a progress display come from Scan.
Source Files