chunker

package
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 8, 2026 License: Apache-2.0 Imports: 9 Imported by: 0

Documentation

Overview

Package chunker implements content-defined chunking (CDC) using the FastCDC algorithm with gear hashing.

Index

Constants

View Source
const (
	DefaultMinChunk = 256 * 1024      // 256 KiB
	DefaultAvgChunk = 1024 * 1024     // 1 MiB
	DefaultMaxChunk = 4 * 1024 * 1024 // 4 MiB
)

Default chunk size parameters, the format v1 defaults. Min/Avg/Max define the FastCDC boundaries in bytes.

The average decides how much a single edit costs on the next backup: one changed byte rewrites one chunk. A 1 MiB average is the same order restic and borg use. The maximum is the ceiling a restore cache has to budget for per worker, so it stays at four times the average rather than growing on its own.

Variables

This section is empty.

Functions

This section is empty.

Types

type Chunk

type Chunk struct {
	Start  uint64 // Byte offset of this chunk in the original stream
	Data   []byte // Raw chunk data
	Length int    // Same as len(Data)
	Hash   string // HMAC-SHA256 of Data under Config.IDKey, hex
}

Chunk represents a single chunk produced by the chunker.

type Chunker

type Chunker struct {
	// contains filtered or unexported fields
}

Chunker splits an input stream into variable-size chunks using FastCDC. It reads from an io.Reader and produces chunks via Next().

func New

func New(r io.Reader, cfg Config) *Chunker

New creates a new FastCDC chunker that reads from r.

func (*Chunker) Next

func (c *Chunker) Next() (Chunk, error)

Next returns the next chunk from the input stream. Returns io.EOF when the stream is exhausted.

func (*Chunker) Reset

func (c *Chunker) Reset(r io.Reader)

Reset points c at a new stream and keeps its buffer, so chunking many files does not allocate a MaxSize buffer per file.

type Config

type Config struct {
	MinSize int // Minimum chunk size in bytes
	AvgSize int // Target average chunk size in bytes
	MaxSize int // Maximum chunk size in bytes

	Seed  [32]byte // keys the gear table
	IDKey [32]byte // keys the HMAC-SHA256 that names each chunk
}

Config configures the FastCDC chunker.

Seed and IDKey are secrets of the repository, so cut points and chunk names of a known file cannot be predicted without them. A repository fills them in from its master key; the zero values are valid and only fit tests.

func DefaultConfig

func DefaultConfig() Config

DefaultConfig returns the default chunker configuration.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL