Help us shape the future of Go! Take the 2026 Go Developer Survey

chunker

package
v1.80.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 20, 2026 License: Apache-2.0 Imports: 2 Imported by: 0

Documentation

Overview

Package chunker cuts byte streams into content-defined chunks with a FastCDC-style normalised Gear hash.

The profile is fixed so that every agent cuts identical bytes identically: minimum 16 KiB, target 64 KiB, maximum 256 KiB. The gear table entry for byte value v is the first eight bytes, big-endian, of SHA-256 of the single byte v. The masks have 17 (below the target) and 15 (above it) one bits at every third position from the top of the 64-bit fingerprint.

Index

Constants

View Source 
const (
	// MinSize is the smallest chunk cut before the end of a stream.
	MinSize = 16 << 10
	// NormalSize is the target chunk size.
	NormalSize = 64 << 10
	// MaxSize is the largest chunk; a cut is forced there.
	MaxSize = 256 << 10
)

Variables

This section is empty.

Functions

func Split 

func Split(data []byte) []int

Split cuts a whole buffer and returns the chunk lengths in order.

Types

type FastCDC 

type FastCDC struct {
	// contains filtered or unexported fields
}

FastCDC holds the fingerprint of the chunk currently being scanned.

func (*FastCDC) Reset 

func (c *FastCDC) Reset()

Reset forgets the current chunk, for use when the stream ends mid-chunk.

func (*FastCDC) Scan 

func (c *FastCDC) Scan(chunk []byte, scanned int) int

Scan looks at chunk[scanned:], where chunk holds the bytes of the current chunk collected so far, and returns the chunk length if a cut point was found, or 0 if more bytes are needed. A cut resets the fingerprint.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.