Documentation
¶
Overview ¶
Package chunkers provides the two content-defined chunkers the fstree uses: a byte chunker (ultracdc) for file content, and an item chunker for the index/entry streams (architecture/fstree.md, "Content-defined chunking").
Index ¶
Constants ¶
const ( DefaultMinSize = 32 << 10 DefaultNormalSize = 512 << 10 DefaultMaxSize = 1 << 20 )
The default byte-chunker sizes: min 32 KiB, normal (target) 512 KiB, max 1 MiB. They determine every Blob boundary and so every file key, so two stores dedup against each other only when they chunk with the same sizes.
Variables ¶
This section is empty.
Functions ¶
func SplitBytes ¶
SplitBytes runs the ultracdc content-defined chunker over r and calls fn once per chunk, in order. Each chunk passed to fn is a fresh copy, so fn may retain it (the underlying chunker reuses its read buffer between calls). A nil opts uses the default sizes (DefaultMinSize, DefaultNormalSize, DefaultMaxSize), and so does every zero field of a non-nil opts. An empty reader yields zero chunks.
Types ¶
type ByteOpts ¶
type ByteOpts = gocdc.ChunkerOpts
ByteOpts configures the ultracdc byte chunker (re-exported so callers need not import the upstream package).
type ItemChunker ¶
ItemChunker decides chunk boundaries between whole items (a child key, a directory entry, or a [sepName, childKey] pair). A boundary ends the current run when the low k bits of BLAKE3(item encoding) are zero (average run 2^k), bounded by MinRun and MaxRun so an item is never split and variance is capped.
func NewItemChunker ¶
func NewItemChunker(bits int) ItemChunker
NewItemChunker returns an item chunker with average run 2^bits and derived bounds MinRun = 2^bits / 4 (at least 2) and MaxRun = 2^bits * 4.
func (ItemChunker) IsBoundary ¶
func (c ItemChunker) IsBoundary(enc []byte, runLen int) bool
IsBoundary reports whether the run should end after the item whose canonical encoding is enc, given the current run length (including this item).
Source Files