Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Chunker ¶
type Chunker interface {
Chunk(ctx context.Context, r io.Reader, fn func(*core.Chunk) error) error
}
Chunker splits an io.Reader into content-addressed chunks, invoking fn for each chunk as it is produced. This streaming interface avoids accumulating all chunks in memory for large files. The context lets callers cancel a long-running chunking operation mid-stream. If fn returns a non-nil error, Chunk stops and returns that error.
func BinaryChunkerFor ¶
BinaryChunkerFor returns a Chunker tuned for binary file content of the given size. It is the canonical building block for image, video, and binary filetype engines.
- size < 50 MB → FastCDC with default params (128/256/512 KB)
- 50 MB ≤ size < 500 MB → FastCDC with params scaled 4× (fewer, larger chunks)
- size ≥ 500 MB → FixedChunker at avgSize×8 to cap chunk-count overhead
type DefaultSelector ¶
type DefaultSelector struct{}
DefaultSelector is a convenience adapter that satisfies the filetype.ChunkerSelector interface by delegating to BinaryChunkerFor. Use it in engine scaffolding or for any engine whose chunking requirements match the binary defaults. Engines with custom size thresholds (e.g. text) should implement ChunkerFor directly instead of using DefaultSelector.
func (DefaultSelector) ChunkerFor ¶
func (DefaultSelector) ChunkerFor(fileSize int64) Chunker
ChunkerFor returns a Chunker appropriate for the given file size by delegating to BinaryChunkerFor.
type FastCDCChunker ¶
type FastCDCChunker struct {
// contains filtered or unexported fields
}
FastCDCChunker implements Chunker using the FastCDC content-defined chunking algorithm. Cut points depend on the data itself, so identical regions of two files produce the same chunks.
func NewFastCDCChunker ¶
func NewFastCDCChunker() *FastCDCChunker
NewFastCDCChunker creates a FastCDC chunker with the default 128KB/256KB/512KB min/avg/max sizes. Behavior is unchanged from before this constructor accepted parameters.
func NewFastCDCChunkerWithParams ¶
func NewFastCDCChunkerWithParams(minSize, avgSize, maxSize int) *FastCDCChunker
NewFastCDCChunkerWithParams creates a FastCDC chunker with custom min/avg/max chunk sizes. Parameters are clamped to satisfy the underlying FastCDC library constraints (MinSize >= 64B, NormalSize must be a power of two, MinSize < NormalSize < MaxSize), so any input produces a usable chunker without panicking.
func (*FastCDCChunker) Chunk ¶
Chunk splits r into content-defined chunks using FastCDC. Each chunk is BLAKE3-hashed and zero-length chunks are skipped. fn is invoked for every emitted chunk; if fn returns an error Chunk stops and returns it. The context is checked at each cut point so a cancelled context aborts the chunking loop promptly.
type FixedChunker ¶
type FixedChunker struct {
// contains filtered or unexported fields
}
FixedChunker implements Chunker by splitting the input into fixed-size chunks. Cut points are independent of the data, so identical bytes at different offsets produce different chunks.
func NewFixedChunker ¶
func NewFixedChunker(chunkSize int) *FixedChunker
NewFixedChunker creates a FixedChunker with the given chunk size. Sizes below fixedChunkMinSize or above fixedChunkMaxSize are clamped to the nearest bound.
func (*FixedChunker) Chunk ¶
Chunk splits r into consecutive fixed-size chunks. The final chunk may be shorter than the chunk size if the input length is not a multiple of it. Each chunk is BLAKE3-hashed. fn is invoked for every emitted chunk; if fn returns an error Chunk stops and returns it. The context is checked before each read so a cancelled context aborts the chunking loop promptly.
Source Files