Documentation
¶
Overview ¶
Package rollinghash/gearhash64 implements the Gear rolling hash (64-bit). https://www.usenix.org/system/files/conference/atc16/atc16-paper-xia.pdf
For a window of w bytes [b0, b1, ..., b_{w-1}] the hash is:
h = gear[b0]<<(w-1) + gear[b1]<<(w-2) + ... + gear[b_{w-1}]
Rolling in byte c (dropping b0) uses: h = (h<<1) - (gear[b0]<<w) + gear[c]. When w >= 64 the subtraction term is zero — the oldest byte's contribution has already been shifted out of the 64-bit word — so the formula is correct for all window sizes.
Example ¶
package main
import (
"fmt"
"hash"
"log"
"github.com/chmduquesne/rollinghash/v4/gearhash64"
)
func main() {
s := []byte("The quick brown fox jumps over the lazy dog")
classic := hash.Hash64(gearhash64.New())
rolling := gearhash64.New()
// Window len
n := 16
// You MUST load an initial window into the rolling hash before being
// able to roll bytes
if _, err := rolling.Write(s[:n]); err != nil {
log.Fatal(err)
}
// Roll it and compare the result with full re-calculus every time
for i := n; i < len(s); i++ {
// Reset and write the window in classic
classic.Reset()
if _, err := classic.Write(s[i-n+1 : i+1]); err != nil {
log.Fatal(err)
}
// Roll the incoming byte in rolling
rolling.Roll(s[i])
fmt.Printf("%v: checksum %x\n", string(s[i-n+1:i+1]), rolling.Sum64())
// Compare the hashes
if classic.Sum64() != rolling.Sum64() {
log.Fatalf("%v: expected %x, got %x",
s[i-n+1:i+1], classic.Sum64(), rolling.Sum64())
}
}
}
Output: he quick brown f: checksum 8266e99517d74e3f e quick brown fo: checksum edf51911f895457f quick brown fox: checksum 7970b7aee863e103 quick brown fox : checksum 6d4a0ff7400acc05 uick brown fox j: checksum f79e9fe5c47882c6 ick brown fox ju: checksum 8ade39ee795c46f7 ck brown fox jum: checksum 65fc71b8c0153151 k brown fox jump: checksum da4680437cf3619d brown fox jumps: checksum aad11e02c261fcb7 brown fox jumps : checksum d00adc9ef407036d rown fox jumps o: checksum b2f67616d6eaafdb own fox jumps ov: checksum 2523af83d09e7243 wn fox jumps ove: checksum bb20ce1eb0ef377d n fox jumps over: checksum 462850369ce05f25 fox jumps over : checksum f90a53341766c849 fox jumps over t: checksum a3b8dce97926c756 ox jumps over th: checksum 6debbb66c6a673d6 x jumps over the: checksum 4cb0e5e49cff3aa3 jumps over the : checksum ef679b745d3b7f45 jumps over the l: checksum 782668a4e3e6ab54 umps over the la: checksum 2a53d6ac06fd86b4 mps over the laz: checksum ac0b238f3ee227f8 ps over the lazy: checksum 80783d6ee6ef1204 s over the lazy : checksum 50753a5948252e07 over the lazy d: checksum b6940bee550b4687 over the lazy do: checksum 7bafdc44e28360f ver the lazy dog: checksum 1eb77581ae7093
Index ¶
- Constants
- func GenerateHashes(seed int64) (res [256]uint64)
- type GearHash64
- func (d *GearHash64) BatchBoundaries(a, b []int32, data []byte, window int, mask uint64) (na, nb int)
- func (d *GearHash64) BatchRoll(dst []uint64, data []byte, window int)
- func (d *GearHash64) BlockSize() int
- func (d *GearHash64) Reset()
- func (d *GearHash64) Roll(c byte)
- func (d *GearHash64) Size() int
- func (d *GearHash64) Sum(b []byte) []byte
- func (d *GearHash64) Sum64() uint64
- func (d *GearHash64) Write(data []byte) (int, error)
- func (d *GearHash64) WriteWindow(w io.Writer) (n int, err error)
Examples ¶
Constants ¶
const Size = 8
Size is the size of the checksum in bytes.
Variables ¶
This section is empty.
Functions ¶
func GenerateHashes ¶
GenerateHashes generates a table of 256 random 64-bit values for use with GearHash64.
Types ¶
type GearHash64 ¶
type GearHash64 struct {
// contains filtered or unexported fields
}
GearHash64 is a digest which satisfies the rollinghash.Hash64 interface. It implements the Gear rolling hash used in FastCDC.
func NewFromUint64Array ¶
func NewFromUint64Array(g [256]uint64) *GearHash64
NewFromUint64Array returns a GearHash64 using the provided lookup table.
func (*GearHash64) BatchBoundaries ¶
func (d *GearHash64) BatchBoundaries(a, b []int32, data []byte, window int, mask uint64) (na, nb int)
BatchBoundaries reports the window positions where the rolling checksum satisfies sum & mask == 0, fusing the boundary test into the hashing loop. It uses the same two-lane design as BatchRoll but avoids the register pressure that comes from carrying na, nb, a, and b into the hot loop: each block of 64 positions accumulates hits as bits in a uint64, then extracts positions via TrailingZeros64 in a rare outer step. The inner loop is 2x unrolled so loads for step k+1 can issue while computing step k. It does not modify the receiver.
func (*GearHash64) BatchRoll ¶
func (d *GearHash64) BatchRoll(dst []uint64, data []byte, window int)
BatchRoll computes the rolling checksum of every window-sized slice of data in one pass and writes them to dst, which must have len(data)-window+1 elements. Two independent accumulator lanes let the CPU overlap their dependency chains and approach the ILP ceiling of the load-latency-bound step h = (h<<1) - shiftedGear[leaving] + gear[entering]. It does not modify the receiver.
func (*GearHash64) Roll ¶
func (d *GearHash64) Roll(c byte)
Roll updates the checksum as byte c enters the window and the oldest byte leaves. You MUST call Write before Roll.
func (*GearHash64) Sum ¶
func (d *GearHash64) Sum(b []byte) []byte
Sum returns the hash as a byte slice.
func (*GearHash64) Write ¶
func (d *GearHash64) Write(data []byte) (int, error)
Write appends data to the rolling window and recomputes the digest. It never returns an error.
func (*GearHash64) WriteWindow ¶
func (d *GearHash64) WriteWindow(w io.Writer) (n int, err error)
WriteWindow writes the current window contents to w.