compression

package
v1.36.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Feb 23, 2026 License: BSD-3-Clause Imports: 2 Imported by: 0

Documentation

Index

Constants

View Source
const (
	PQEncoderTypeTile              = "tile"
	PQEncoderTypeKMeans            = "kmeans"
	PQEncoderDistributionLogNormal = "log-normal"
	PQEncoderDistributionNormal    = "normal"
)

Encoder constants for PQ encoder types.

View Source
const (
	DefaultFastRotationSeed = uint64(0x535ab5105169b1df)
)

Variables

This section is empty.

Functions

func FastWalshHadamardTransform64

func FastWalshHadamardTransform64(x []float32)

FastWalshHadamardTransform64 performs a fast Walsh-Hadamard transform on a 64-element vector. This explicit instantiation is about 10% faster.

func FastWalshHadamardTransform256

func FastWalshHadamardTransform256(x []float32)

Types

type BRQData

type BRQData struct {
	InputDim uint32
	Rotation FastRotation
	Rounding []float32
}

BRQData holds the serialization data for Binary Rotational Quantization compression.

type Encoder

type Encoder byte

Encoder represents the type of encoder used for product quantization.

const (
	UseTileEncoder   Encoder = 0
	UseKMeansEncoder Encoder = 1
)

type FastRotation

type FastRotation struct {
	OutputDim uint32      // The dimension of the output returned by Rotate().
	Rounds    uint32      // The number of rounds of random signs, swaps, and blocked transforms that the Rotate() function is going to apply.
	Swaps     [][]Swap    // Random swaps to apply each round prior to transforming.
	Signs     [][]float32 // Random signs to apply each round prior to transforming. We store these as float32 values for performance reasons, to avoid casts.
}

FastRotation holds the configuration for fast random rotation.

func NewFastRotation

func NewFastRotation(inputDim int, rounds int, seed uint64) *FastRotation

func RestoreFastRotation

func RestoreFastRotation(outputDim int, rounds int, swaps [][]Swap, signs [][]float32) *FastRotation

func (*FastRotation) Rotate

func (r *FastRotation) Rotate(x []float32) []float32

func (*FastRotation) UnRotate

func (r *FastRotation) UnRotate(rx []float32) []float32

func (*FastRotation) UnRotateInPlace

func (r *FastRotation) UnRotateInPlace(x []float32) []float32

type MuveraData

type MuveraData struct {
	KSim         uint32        // 4 bytes
	NumClusters  uint32        // 4 bytes
	Dimensions   uint32        // 4 bytes
	DProjections uint32        // 4 bytes
	Repetitions  uint32        // 4 bytes
	Gaussians    [][][]float32 // (repetitions, kSim, dimensions)
	S            [][][]float32 // (repetitions, dProjections, dimensions)
}

MuveraData holds the serialization data for Muvera multi-vector encoding.

type PQData

type PQData struct {
	Ks                  uint16
	M                   uint16
	Dimensions          uint16
	EncoderType         Encoder
	EncoderDistribution byte
	Encoders            []PQSegmentEncoder
	UseBitsEncoding     bool
	TrainingLimit       int
}

PQData holds the serialization data for Product Quantization compression.

type PQSegmentEncoder

type PQSegmentEncoder interface {
	Encode(x []float32) byte
	Centroid(b byte) []float32
	Add(x []float32)
	Fit(data [][]float32) error
	ExposeDataForRestore() []byte
}

PQSegmentEncoder is the interface for product quantization segment encoders. Implementations include TileEncoder and KMeansEncoder.

type RQData

type RQData struct {
	InputDim uint32
	Bits     uint32
	Rotation FastRotation
	Rounding []float32
}

RQData holds the serialization data for Rotational Quantization compression.

type SQData

type SQData struct {
	A          float32
	B          float32
	Dimensions uint16
}

SQData holds the serialization data for Scalar Quantization compression.

type State

type State struct {
	// Compressed indicates if the index uses compression.
	Compressed bool

	// PQData holds Product Quantization compression data.
	PQData *PQData

	// SQData holds Scalar Quantization compression data.
	SQData *SQData

	// RQData holds Rotational Quantization compression data.
	RQData *RQData

	// BRQData holds Binary Rotational Quantization compression data.
	BRQData *BRQData

	// MuveraEnabled indicates if Muvera multi-vector encoding is enabled.
	MuveraEnabled bool

	// MuveraData holds Muvera multi-vector encoding data.
	MuveraData *MuveraData
}

State holds all compression-related data for an HNSW index. This structure is optional - if nil, the index is uncompressed.

func (*State) HasCompression

func (s *State) HasCompression() bool

HasCompression returns true if any compression is enabled.

func (*State) HasMuvera

func (s *State) HasMuvera() bool

HasMuvera returns true if Muvera encoding is enabled.

type Swap

type Swap struct {
	I, J uint16
}

Swap represents a pair of indices to swap during fast rotation.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL