Documentation
¶
Overview ¶
Package chunk implements streaming block splitters. Splitters read data from a reader and provide byte slices (chunks) The size and contents of these slices depend on the splitting method used.
Index ¶
Constants ¶
const ( // BlockSizeLimit is the maximum block size defined by the bitswap spec. // https://specs.ipfs.tech/bitswap-protocol/#block-sizes BlockSizeLimit int = 2 * 1024 * 1024 // 2MiB // ChunkOverheadBudget is reserved for protobuf/UnixFS framing overhead // when chunks are wrapped in non-raw leaves (--raw-leaves=false). ChunkOverheadBudget int = 256 // ChunkSizeLimit is the maximum chunk size accepted by the chunker. // It is set below BlockSizeLimit to leave room for framing overhead // so that serialized blocks stay within the 2MiB wire limit. // // In practice this limit only matters for custom chunker sizes. // The CID-deterministic profiles defined in IPIP-499 use max 1MiB // chunks, well within this limit. ChunkSizeLimit int = BlockSizeLimit - ChunkOverheadBudget )
Variables ¶
var ( ErrRabinMin = errors.New("rabin min must be greater than 16") ErrSize = errors.New("chunker size must be greater than 0") ErrSizeMax = fmt.Errorf("chunker parameters may not exceed the maximum chunk size of %d", ChunkSizeLimit) )
var DefaultBlockSize int64 = 1024 * 256
DefaultBlockSize is the chunk size that splitters produce (or aim to). Can be modified to change the default for all subsequent chunker operations. For CID-deterministic imports, prefer using UnixFSProfile presets from ipld/unixfs/io/profile.go which set this and other related globals.
var IpfsRabinPoly = chunker.Pol(17437180132763653)
IpfsRabinPoly is the irreducible polynomial of degree 53 used by for Rabin.
Functions ¶
Types ¶
type Rabin ¶
type Rabin struct {
// contains filtered or unexported fields
}
Rabin implements the Splitter interface and splits content with Rabin fingerprints.
func NewRabinMinMax ¶
NewRabinMinMax returns a new Rabin splitter which uses the given min, average and max block sizes.
type Splitter ¶
A Splitter reads bytes from a Reader and creates "chunks" (byte slices) that can be used to build DAG nodes.
func DefaultSplitter ¶
DefaultSplitter returns a SizeSplitter with the DefaultBlockSize.
func FromString ¶
FromString returns a Splitter depending on the given string: it supports "default" (""), "size-{size}", "rabin", "rabin-{blocksize}", "rabin-{min}-{avg}-{max}" and "buzhash".
type SplitterGen ¶
SplitterGen is a splitter generator, given a reader.
func SizeSplitterGen ¶
func SizeSplitterGen(size int64) SplitterGen
SizeSplitterGen returns a SplitterGen function which will create a splitter with the given size when called.
Source Files
Directories