fastcdc

package
v0.0.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: May 25, 2023 License: AGPL-3.0, Apache-2.0 Imports: 5 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

View Source
var DefaultOpts = Options{
	AverageSize: 1024,
	Seed:        84372,
}

Functions

This section is empty.

Types

type ChunkHashWriter

type ChunkHashWriter func(ch *pb.ChunkHash) error

type Chunker

type Chunker struct {
	// contains filtered or unexported fields
}

Chunker implements the FastCDC content defined chunking algorithm. See https://www.usenix.org/system/files/conference/atc16/atc16-paper-xia.pdf.

func NewChunker

func NewChunker(rd io.Reader, opts Options) (*Chunker, error)

NewChunker returns a Chunker with the given Options.

func (*Chunker) ApplyDelta

func (c *Chunker) ApplyDelta(alignedTarget io.Writer, target io.ReadSeeker, ops chan *pb.Operation) error

func (*Chunker) CreateDelta

func (c *Chunker) CreateDelta(chunkHashes []*pb.ChunkHash, ops OperationWriter) error

Writes delta ops that are diffs from chunkhashes

func (*Chunker) CreateSignature

func (c *Chunker) CreateSignature(sw ChunkHashWriter) error

func (*Chunker) Next

func (c *Chunker) Next() (*pb.Chunk, error)

Next returns the next Chunk from the reader or io.EOF after the last chunk has been read. The chunk data is invalidated when Next is called again.

type OpType

type OpType byte

type OperationWriter

type OperationWriter func(op *pb.Operation) error

type Options

type Options struct {
	// NormalSize is the target chunk size. Typically a power of 2. It must be in the
	// range 64B to 1GiB.
	AverageSize int

	// (Optional) MinSize is the minimum allowed chunk size. By default, it's set to
	// AverageSize / 4.
	MinSize int

	// (Optional) MaxSize is the maximum allowed chunk size. By default, it's set to
	// AverageSize * 4.
	MaxSize int

	// (Optional) Sets the chunk normalization level. It may be set to 1, 2 or 3,
	// unless DisableNormalization is set, in which case it's ignored. By default,
	// it's set to 2.
	Normalization int

	// (Optional) DisableNormalization turns normalization off. By default, it's set to
	// false.
	DisableNormalization bool

	// (Optional) Seed alters the lookup table of the rolling hash algorithm to mitigate
	// chunk-size based fingerprinting attacks. It may be set to a random uint64.
	Seed uint64

	// (Optional) BufSize is the size of the internal buffer used while chunking. It has
	// no effect on the chuking output, but performance is improved with larger buffers.
	// It must be at least MaxSize. Recommended values are 1 to 3 times MaxSize. By
	// default it is set to MaxSize * 2.
	BufSize int
}

Options configures the options for the Chunker.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL