gearhash64

package
v4.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jun 30, 2026 License: MIT Imports: 5 Imported by: 0

Documentation

Overview

Package rollinghash/gearhash64 implements the Gear rolling hash (64-bit). https://www.usenix.org/system/files/conference/atc16/atc16-paper-xia.pdf

For a window of w bytes [b0, b1, ..., b_{w-1}] the hash is:

h = gear[b0]<<(w-1) + gear[b1]<<(w-2) + ... + gear[b_{w-1}]

Rolling in byte c (dropping b0) uses: h = (h<<1) - (gear[b0]<<w) + gear[c]. When w >= 64 the subtraction term is zero — the oldest byte's contribution has already been shifted out of the 64-bit word — so the formula is correct for all window sizes.

Example
package main

import (
	"fmt"
	"hash"
	"log"

	"github.com/chmduquesne/rollinghash/v4/gearhash64"
)

func main() {
	s := []byte("The quick brown fox jumps over the lazy dog")

	classic := hash.Hash64(gearhash64.New())
	rolling := gearhash64.New()

	// Window len
	n := 16

	// You MUST load an initial window into the rolling hash before being
	// able to roll bytes
	if _, err := rolling.Write(s[:n]); err != nil {
		log.Fatal(err)
	}

	// Roll it and compare the result with full re-calculus every time
	for i := n; i < len(s); i++ {

		// Reset and write the window in classic
		classic.Reset()
		if _, err := classic.Write(s[i-n+1 : i+1]); err != nil {
			log.Fatal(err)
		}

		// Roll the incoming byte in rolling
		rolling.Roll(s[i])

		fmt.Printf("%v: checksum %x\n", string(s[i-n+1:i+1]), rolling.Sum64())

		// Compare the hashes
		if classic.Sum64() != rolling.Sum64() {
			log.Fatalf("%v: expected %x, got %x",
				s[i-n+1:i+1], classic.Sum64(), rolling.Sum64())
		}
	}

}
Output:
he quick brown f: checksum 8266e99517d74e3f
e quick brown fo: checksum edf51911f895457f
 quick brown fox: checksum 7970b7aee863e103
quick brown fox : checksum 6d4a0ff7400acc05
uick brown fox j: checksum f79e9fe5c47882c6
ick brown fox ju: checksum 8ade39ee795c46f7
ck brown fox jum: checksum 65fc71b8c0153151
k brown fox jump: checksum da4680437cf3619d
 brown fox jumps: checksum aad11e02c261fcb7
brown fox jumps : checksum d00adc9ef407036d
rown fox jumps o: checksum b2f67616d6eaafdb
own fox jumps ov: checksum 2523af83d09e7243
wn fox jumps ove: checksum bb20ce1eb0ef377d
n fox jumps over: checksum 462850369ce05f25
 fox jumps over : checksum f90a53341766c849
fox jumps over t: checksum a3b8dce97926c756
ox jumps over th: checksum 6debbb66c6a673d6
x jumps over the: checksum 4cb0e5e49cff3aa3
 jumps over the : checksum ef679b745d3b7f45
jumps over the l: checksum 782668a4e3e6ab54
umps over the la: checksum 2a53d6ac06fd86b4
mps over the laz: checksum ac0b238f3ee227f8
ps over the lazy: checksum 80783d6ee6ef1204
s over the lazy : checksum 50753a5948252e07
 over the lazy d: checksum b6940bee550b4687
over the lazy do: checksum 7bafdc44e28360f
ver the lazy dog: checksum 1eb77581ae7093

Index

Examples

Constants

View Source
const Size = 8

Size is the size of the checksum in bytes.

Variables

This section is empty.

Functions

func GenerateHashes

func GenerateHashes(seed int64) (res [256]uint64)

GenerateHashes generates a table of 256 random 64-bit values for use with GearHash64.

Types

type GearHash64

type GearHash64 struct {
	// contains filtered or unexported fields
}

GearHash64 is a digest which satisfies the rollinghash.Hash64 interface. It implements the Gear rolling hash used in FastCDC.

func New

func New() *GearHash64

New returns a GearHash64 using the default table seeded with 1.

func NewFromUint64Array

func NewFromUint64Array(g [256]uint64) *GearHash64

NewFromUint64Array returns a GearHash64 using the provided lookup table.

func (*GearHash64) BatchBoundaries

func (d *GearHash64) BatchBoundaries(a, b []int32, data []byte, window int, mask uint64) (na, nb int)

BatchBoundaries reports the window positions where the rolling checksum satisfies sum & mask == 0, fusing the boundary test into the hashing loop. It uses the same two-lane design as BatchRoll but avoids the register pressure that comes from carrying na, nb, a, and b into the hot loop: each block of 64 positions accumulates hits as bits in a uint64, then extracts positions via TrailingZeros64 in a rare outer step. The inner loop is 2x unrolled so loads for step k+1 can issue while computing step k. It does not modify the receiver.

func (*GearHash64) BatchRoll

func (d *GearHash64) BatchRoll(dst []uint64, data []byte, window int)

BatchRoll computes the rolling checksum of every window-sized slice of data in one pass and writes them to dst, which must have len(data)-window+1 elements. Two independent accumulator lanes let the CPU overlap their dependency chains and approach the ILP ceiling of the load-latency-bound step h = (h<<1) - shiftedGear[leaving] + gear[entering]. It does not modify the receiver.

func (*GearHash64) BlockSize

func (d *GearHash64) BlockSize() int

BlockSize returns 1 byte.

func (*GearHash64) Reset

func (d *GearHash64) Reset()

Reset resets the Hash to its initial state.

func (*GearHash64) Roll

func (d *GearHash64) Roll(c byte)

Roll updates the checksum as byte c enters the window and the oldest byte leaves. You MUST call Write before Roll.

func (*GearHash64) Size

func (d *GearHash64) Size() int

Size returns 8 bytes.

func (*GearHash64) Sum

func (d *GearHash64) Sum(b []byte) []byte

Sum returns the hash as a byte slice.

func (*GearHash64) Sum64

func (d *GearHash64) Sum64() uint64

Sum64 returns the hash as a uint64.

func (*GearHash64) Write

func (d *GearHash64) Write(data []byte) (int, error)

Write appends data to the rolling window and recomputes the digest. It never returns an error.

func (*GearHash64) WriteWindow

func (d *GearHash64) WriteWindow(w io.Writer) (n int, err error)

WriteWindow writes the current window contents to w.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL