chunker

package
v1.64.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 13, 2026 License: Apache-2.0 Imports: 2 Imported by: 0

Documentation

Overview

Package chunker cuts byte streams into content-defined chunks with a FastCDC-style normalised Gear hash.

The profile is fixed so that every agent cuts identical bytes identically: minimum 16 KiB, target 64 KiB, maximum 256 KiB. The gear table entry for byte value v is the first eight bytes, big-endian, of SHA-256 of the single byte v. The masks have 17 (below the target) and 15 (above it) one bits at every third position from the top of the 64-bit fingerprint.

Index

Constants

View Source 
const (
	// MinSize is the smallest chunk cut before the end of a stream.
	MinSize = 16 << 10
	// NormalSize is the target chunk size.
	NormalSize = 64 << 10
	// MaxSize is the largest chunk; a cut is forced there.
	MaxSize = 256 << 10
)

Variables

This section is empty.

Functions

func Split 

func Split(data []byte) []int

Split cuts a whole buffer and returns the chunk lengths in order.

Types

type FastCDC 

type FastCDC struct {
	// contains filtered or unexported fields
}

FastCDC holds the fingerprint of the chunk currently being scanned.

func (*FastCDC) Reset 

func (c *FastCDC) Reset()

Reset forgets the current chunk, for use when the stream ends mid-chunk.

func (*FastCDC) Scan 

func (c *FastCDC) Scan(chunk []byte, scanned int) int

Scan looks at chunk[scanned:], where chunk holds the bytes of the current chunk collected so far, and returns the chunk length if a cut point was found, or 0 if more bytes are needed. A cut resets the fingerprint.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.