vectorindex

package
v0.0.0-debug-20260702 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 2, 2026 License: Apache-2.0 Imports: 12 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	HNSW    = "HNSW"
	IVFFLAT = "IVFFLAT"
	IVFPQ   = "IVFPQ"
	CAGRA   = "CAGRA"
)
View Source 
const (
	CDC_INSERT = "I"
	CDC_UPSERT = "U"
	CDC_DELETE = "D"
)
View Source 
const (
	DistributionMode_SINGLE_GPU_Str = "single"
	DistributionMode_SHARDED_Str    = "sharded"
	DistributionMode_REPLICATED_Str = "replicated"
)
View Source 
const CdcTailId = "cdc_tail"

CdcTailId is the literal index_id under which CDC writes tag=1 event-log rows in the storage table for cagra/ivfpq indexes. CDC is single-threaded and re-index quiesces it, so this fixed sentinel removes the per-sub-index "active id" coordination problem entirely — search reads it once at Load time alongside the real sub-index models.

View Source 
const (
	MaxChunkSize = 65536
)

Variables

This section is empty.

Functions

func CheckSum 

func CheckSum(path string) (string, error)

get the checksum of the file

func CheckSumFromBuffer 

func CheckSumFromBuffer(b []byte) string

func GetConcurrency 

func GetConcurrency(nthread int64) int64

nthread == 0, result will return NumCPU - 1

func GetConcurrencyForBuild 

func GetConcurrencyForBuild(nthread int64) int64

nthread == 0, result will return NumCPU

func SimulateDevices 

func SimulateDevices(devices []int, n int64) []int

SimulateDevices is a test-only seam for exercising SHARDED / REPLICATED distribution modes on a single-GPU host. When n >= 2 it returns a device list of physical device 0 repeated n times ([0,0,...]), so the cuVS worker pool spins up n logical ranks (each with its own stream/handle) all on device 0. When n < 2 the real device list is returned unchanged.

func ValidDistributionMode 

func ValidDistributionMode(val string) bool

Types

type CagraParam 

type CagraParam struct {
	M                      string `json:"m"`
	EfConstruction         string `json:"ef_construction"`
	OpType                 string `json:"op_type"`
	EfSearch               string `json:"ef_search"`
	Async                  string `json:"async"`
	Quantization           string `json:"quantization"`
	Distribution           string `json:"distribution_mode"`
	IntermediateGraphDegee string `json:"intermediate_graph_degree"`
	GraphDegee             string `json:"graph_degree"`
	ITopkSize              string `json:"itopk_size"`
	IncludedColumns        string `json:"included_columns"`
	MaxIndexCapacity       string `json:"max_index_capacity"`
}

CAGRA specified parameters

type ChunkTag 

type ChunkTag int64
const (
	Tag_ModelChunk ChunkTag = 0
	// Tag_CdcEvents stores an ordered event log of CDC mutations. Each chunk
	// is a sequence of op-tagged records; replay in chunk_id order produces
	// the (deleted, overflow) state used by the load path. Replaces the
	// earlier two-stream design (separate deleted-pkid list + insert
	// overflow), which lost temporal ordering between DELETE/INSERT events
	// for the same pkid. See cuvs_cdc.md for the full record format.
	Tag_CdcEvents ChunkTag = 1
)

type CuvsCagraIndexConfig 

type CuvsCagraIndexConfig struct {
	IntermediateGraphDegree uint64
	GraphDegree             uint64
	ITopkSize               uint64
	Metric                  uint16
	Dimensions              uint
	Version                 int64
	VectorType              int32
	Quantization            uint16
	DistributionMode        uint16
	IncludedColumns         []string
}

type CuvsIvfIndexConfig 

type CuvsIvfIndexConfig struct {
	Lists            uint
	Metric           uint16
	InitType         uint16
	Dimensions       uint
	Spherical        bool
	Version          int64
	VectorType       int32
	Quantization     uint16
	DistributionMode uint16
}

type CuvsIvfpqIndexConfig 

type CuvsIvfpqIndexConfig struct {
	Lists                  uint
	M                      uint
	BitsPerCode            uint
	Metric                 uint16
	Dimensions             uint
	Quantization           uint16
	DistributionMode       uint16
	Version                int64
	KmeansTrainsetFraction float64
	IncludedColumns        []string
}

type DistributionMode 

type DistributionMode uint16
const (
	DistributionMode_SINGLE_GPU DistributionMode = iota
	DistributionMode_SHARDED
	DistributionMode_REPLICATED
)

type FastMaxHeap 

type FastMaxHeap[T types.RealNumbers, K HeapKeyType] struct {
	// contains filtered or unexported fields
}

FastMaxHeap is a highly optimized, generic bounded max-heap designed specifically for vector search Top-K operations.

Benefits over standard container/heap:

  1. Zero Interface Boxing: By using generics and specific array layouts, it completely avoids the heap-escape "boxing" allocations caused by passing interface{} around.
  2. Struct of Arrays (SoA): Uses independent slices for keys and distances rather than an Array of Structs (AoS). This dramatically improves CPU cache locality during distance comparisons.
  3. Inline Array Reuse: Requires passing pre-allocated backing buffers to ensure zero allocations inside tight loops.
  4. Bounded Logic: Natively handles "Limit/K" bounded sizing directly during the push step, reducing structural overhead.

func NewFastMaxHeap 

func NewFastMaxHeap[T types.RealNumbers, K HeapKeyType](limit int, keysBuf []K, distsBuf []T) *FastMaxHeap[T, K]

NewFastMaxHeap initializes the FastMaxHeap using caller-provided buffer slices to guarantee zero-allocation operations during tight query loops.

func (*FastMaxHeap[T, K]) Pop 

func (h *FastMaxHeap[T, K]) Pop() (K, T, bool)

Pop extracts the element with the largest distance from the max-heap.

func (*FastMaxHeap[T, K]) Push 

func (h *FastMaxHeap[T, K]) Push(key K, dist T)

Push inserts a new element into the max-heap. If the heap is at its limit, it replaces the maximum (root) element if the new distance is smaller.

type FastMaxHeapSafe 

type FastMaxHeapSafe[T types.RealNumbers, K HeapKeyType] struct {
	// contains filtered or unexported fields
}

Thread-safe wrapper for FastMaxHeap

func NewFastMaxHeapSafe 

func NewFastMaxHeapSafe[T types.RealNumbers, K HeapKeyType](limit int, keysBuf []K, distsBuf []T) *FastMaxHeapSafe[T, K]

NewFastMaxHeapSafe creates a thread-safe FastMaxHeap

func (*FastMaxHeapSafe[T, K]) Pop 

func (s *FastMaxHeapSafe[T, K]) Pop() (K, T, bool)

func (*FastMaxHeapSafe[T, K]) Push 

func (s *FastMaxHeapSafe[T, K]) Push(key K, dist T)

type HeapKeyType 

type HeapKeyType interface {
	int64 | uint32 | int32
}

HeapKeyType is the constraint for keys stored in FastMaxHeap.

type HnswCdcParam 

type HnswCdcParam struct {
	DbName    string    `json:"db"`
	Table     string    `json:"table"`
	MetaTbl   string    `json:"meta"`
	IndexTbl  string    `json:"index"`
	Params    HnswParam `json:"params"`
	Dimension int32     `json:"dimension"`
	VecType   int32     `json:"type"`
}

type HnswParam 

type HnswParam struct {
	M                string `json:"m"`
	EfConstruction   string `json:"ef_construction"`
	OpType           string `json:"op_type"`
	EfSearch         string `json:"ef_search"`
	Async            string `json:"async"`
	MaxIndexCapacity string `json:"max_index_capacity"`
}

HNSW specified parameters

type IndexConfig 

type IndexConfig struct {
	Type          string
	OpType        string
	IndexCapacity int64
	Usearch       usearch.IndexConfig
	Ivfflat       IvfflatIndexConfig
	CuvsIvf       CuvsIvfIndexConfig
	CuvsCagra     CuvsCagraIndexConfig
	CuvsIvfpq     CuvsIvfpqIndexConfig
}

This is generalized index config and able to share between various algorithm types. Simply add your new configuration such as usearch.IndexConfig

type IndexTableConfig 

type IndexTableConfig struct {
	DbName        string `json:"db"`
	SrcTable      string `json:"src"`
	MetadataTable string `json:"metadata"`
	IndexTable    string `json:"index"`
	PKey          string `json:"pkey"`
	KeyPart       string `json:"part"`
	OrigFuncName  string `json:"orig_func_name"`
	ThreadsBuild  int64  `json:"threads_build"`
	ThreadsSearch int64  `json:"threads_search"`

	// IVF related
	EntriesTable   string  `json:"entries"`
	DataSize       int64   `json:"datasize"`
	Nprobe         uint    `json:"nprobe"`
	PKeyType       int32   `json:"pktype"`
	KeyPartType    int32   `json:"parttype"`
	Limit          uint64  `json:"limit"`
	LowerBoundType int8    `json:"lower_bound_type"`
	LowerBound     float64 `json:"lower_bound"`
	UpperBoundType int8    `json:"upper_bound_type"`
	UpperBound     float64 `json:"upper_bound"`

	// GPU related
	BatchWindow int64 `json:"batch_window"`
	// GpuMultiSimulation is a test-only knob: when >= 2, the device list is
	// replaced with physical device 0 repeated N times so SHARDED / REPLICATED
	// distribution modes can be exercised on a single-GPU host. 0/1 = real devices.
	GpuMultiSimulation int64 `json:"gpu_multi_simulation"`
}

HNSW have two secondary index tables, metadata and index storage. For new vector index algorithm that share the same secondary tables, can use the same IndexTableConfig struct

type IvfParam 

type IvfParam struct {
	Lists              string `json:"lists"`
	OpType             string `json:"op_type"`
	Async              string `json:"async"`
	Quantization       string `json:"quantization"`
	Distribution       string `json:"distribution_mode"`
	KmeansTrainPercent string `json:"kmeans_train_percent"`
	KmeansMaxIteration string `json:"kmeans_max_iteration"`
}

IVF specified parameters

type IvfflatIndexConfig 

type IvfflatIndexConfig struct {
	Lists              uint
	Metric             uint16
	InitType           uint16
	Dimensions         uint
	Spherical          bool
	Version            int64
	VectorType         int32
	KmeansTrainPercent float64
	KmeansMaxIteration int64
}

type IvfpqParam 

type IvfpqParam struct {
	Lists              string `json:"lists"`
	M                  string `json:"m"`
	BitsPerCode        string `json:"bits_per_code"`
	OpType             string `json:"op_type"`
	Quantization       string `json:"quantization"`
	Distribution       string `json:"distribution_mode"`
	IncludedColumns    string `json:"included_columns"`
	KmeansTrainPercent string `json:"kmeans_train_percent"`
	KmeansMaxIteration string `json:"kmeans_max_iteration"`
	MaxIndexCapacity   string `json:"max_index_capacity"`
}

IVF-PQ specified parameters

type RuntimeConfig 

type RuntimeConfig struct {
	Limit             uint
	Probe             uint
	OrigFuncName      string
	BackgroundQueries []*plan.Query
	NThreads          uint // Brute Force Index

	// FilterJSON is a JSON predicate array forwarded verbatim to CGo
	// (gpu_<idx>_search_with_filter). Empty → unfiltered search path.
	// Go never parses this payload; it's produced by the SQL layer and
	// consumed by the C++ eval_filter_bitmap_cpu.
	FilterJSON string
}

type SearchResult 

type SearchResult struct {
	Id       int64
	Distance float64
}

func (*SearchResult) GetDistance 

func (s *SearchResult) GetDistance() float64

type SearchResultAnyKey 

type SearchResultAnyKey struct {
	Id       any
	Distance float64
}

func (*SearchResultAnyKey) GetDistance 

func (s *SearchResultAnyKey) GetDistance() float64

type SearchResultHeap 

type SearchResultHeap []SearchResultIf

Non thread-safe heap struct

func (SearchResultHeap) Len 

func (h SearchResultHeap) Len() int

func (SearchResultHeap) Less 

func (h SearchResultHeap) Less(i, j int) bool

func (*SearchResultHeap) Pop 

func (h *SearchResultHeap) Pop() any

func (*SearchResultHeap) Push 

func (h *SearchResultHeap) Push(x any)

func (SearchResultHeap) Swap 

func (h SearchResultHeap) Swap(i, j int)

type SearchResultIf 

type SearchResultIf interface {
	GetDistance() float64
}

Priority Queue/Heap structure for getting N-Best results from multiple mini-indexes

type SearchResultMaxHeap 

type SearchResultMaxHeap []SearchResultIf

func (SearchResultMaxHeap) Len 

func (h SearchResultMaxHeap) Len() int

func (SearchResultMaxHeap) Less 

func (h SearchResultMaxHeap) Less(i, j int) bool

func (*SearchResultMaxHeap) Pop 

func (h *SearchResultMaxHeap) Pop() any

func (*SearchResultMaxHeap) Push 

func (h *SearchResultMaxHeap) Push(x any)

func (SearchResultMaxHeap) Swap 

func (h SearchResultMaxHeap) Swap(i, j int)

type SearchResultSafeHeap 

type SearchResultSafeHeap struct {
	// contains filtered or unexported fields
}

Thread-safe Heap struct

func NewSearchResultSafeHeap 

func NewSearchResultSafeHeap(size int) *SearchResultSafeHeap

func (*SearchResultSafeHeap) Len 

func (h *SearchResultSafeHeap) Len() int

func (*SearchResultSafeHeap) Pop 

func (h *SearchResultSafeHeap) Pop() SearchResultIf

func (*SearchResultSafeHeap) Push 

func (h *SearchResultSafeHeap) Push(srif SearchResultIf)

type VectorIndexCdc 

type VectorIndexCdc[T types.RealNumbers] struct {
	// Start string                   `json:"start"`
	// End   string                   `json:"end"`
	Data []VectorIndexCdcEntry[T] `json:"cdc"`
}

func NewVectorIndexCdc 

func NewVectorIndexCdc[T types.RealNumbers](capacity int) *VectorIndexCdc[T]

func (*VectorIndexCdc[T]) Delete 

func (h *VectorIndexCdc[T]) Delete(key int64)

func (*VectorIndexCdc[T]) Empty 

func (h *VectorIndexCdc[T]) Empty() bool

func (*VectorIndexCdc[T]) Full 

func (h *VectorIndexCdc[T]) Full() bool

func (*VectorIndexCdc[T]) Insert 

func (h *VectorIndexCdc[T]) Insert(key int64, v []T, includeBytes []byte)

Insert appends a CDC INSERT event. includeBytes carries the row's INCLUDE column values for cuvs CAGRA / IVF-PQ in row-major + trailing null-mask layout (see EncodeEventRecord in cuvs_cdc.go for the format). Pass nil for HNSW or for indexes without INCLUDE columns.

func (*VectorIndexCdc[T]) Reset 

func (h *VectorIndexCdc[T]) Reset()

func (*VectorIndexCdc[T]) ToJson 

func (h *VectorIndexCdc[T]) ToJson() (string, error)

func (*VectorIndexCdc[T]) Upsert 

func (h *VectorIndexCdc[T]) Upsert(key int64, v []T, includeBytes []byte)

Upsert appends a CDC UPSERT event. See Insert for includeBytes semantics.

type VectorIndexCdcEntry 

type VectorIndexCdcEntry[T types.RealNumbers] struct {
	Type string `json:"t"` // I - INSERT, D - DELETE, U - UPSERT
	PKey int64  `json:"pk"`
	Vec  []T    `json:"v,omitempty"`
	// IncludeBytes carries this row's INCLUDE column values (row-major
	// in column-meta order + trailing null mask). Empty/omitted for HNSW
	// and for cuvs indexes with no INCLUDE columns.
	IncludeBytes []byte `json:"i,omitempty"`
}

Source Files

View all Source files

Directories

Path Synopsis
cagra
plugin
Package plugin is the CAGRA vector index integration.
 Package plugin is the CAGRA vector index integration.
plugin/compile
Package compile implements the CAGRA plugin's compile-layer (DDL) hooks.
 Package compile implements the CAGRA plugin's compile-layer (DDL) hooks.
plugin/idxcron
Package idxcron is CAGRA's idxcron hook implementation.
 Package idxcron is CAGRA's idxcron hook implementation.
plugin/plan
Package plan implements the Cagra plugin's plan-layer hooks.
 Package plan implements the Cagra plugin's plan-layer hooks.
plugin/runtime
Package runtime holds the CAGRA algorithm's catalog-side metadata.
 Package runtime holds the CAGRA algorithm's catalog-side metadata.
Package cuvs carries shared cuvs-specific helpers — currently the CDC wire format used by CAGRA and IVF-PQ for tag=1 event chunks.
 Package cuvs carries shared cuvs-specific helpers — currently the CDC wire format used by CAGRA and IVF-PQ for tag=1 event chunks.
idxcron
Package idxcron carries the shared cuvs idxcron Updatable body.
 Package idxcron carries the shared cuvs idxcron Updatable body.
plugin
Package plugin is the HNSW vector index integration.
 Package plugin is the HNSW vector index integration.
plugin/compile
Package compile implements the HNSW plugin's compile-layer (DDL) hooks.
 Package compile implements the HNSW plugin's compile-layer (DDL) hooks.
plugin/idxcron
Package idxcron is HNSW's idxcron hook implementation.
 Package idxcron is HNSW's idxcron hook implementation.
plugin/iscp
Package iscp provides HNSW's ISCP hook layer: writer construction and consumer-loop dispatch.
 Package iscp provides HNSW's ISCP hook layer: writer construction and consumer-loop dispatch.
plugin/plan
Package plan implements the HNSW plugin's plan-layer hooks.
 Package plan implements the HNSW plugin's plan-layer hooks.
plugin/runtime
Package runtime holds the HNSW algorithm's catalog-side metadata.
 Package runtime holds the HNSW algorithm's catalog-side metadata.
kmeans
kmeans/balanced
kmeans/device
kmeans/elkans
plugin
Package plugin is the IVF-FLAT vector index plugin registration point.
 Package plugin is the IVF-FLAT vector index plugin registration point.
plugin/compile
Package compile implements the IVF-FLAT plugin's compile-layer (DDL) hooks.
 Package compile implements the IVF-FLAT plugin's compile-layer (DDL) hooks.
plugin/idxcron
Package idxcron is IVF-FLAT's idxcron hook implementation.
 Package idxcron is IVF-FLAT's idxcron hook implementation.
plugin/iscp
Package iscp provides IVF-FLAT's ISCP hook layer.
 Package iscp provides IVF-FLAT's ISCP hook layer.
plugin/plan
Package plan implements the Ivfflat plugin's plan-layer hooks.
 Package plan implements the Ivfflat plugin's plan-layer hooks.
plugin/runtime
Package runtime holds IVF-FLAT's catalog-side metadata: hidden-table types, parameter schema, op-type set, default options, sync descriptor.
 Package runtime holds IVF-FLAT's catalog-side metadata: hidden-table types, parameter schema, op-type set, default options, sync descriptor.
ivfpq
plugin
Package plugin is the IVF-PQ vector index integration AND the canonical reference for adding a new vector index algorithm to MatrixOne.
 Package plugin is the IVF-PQ vector index integration AND the canonical reference for adding a new vector index algorithm to MatrixOne.
plugin/compile
Package compile implements the IVF-PQ plugin's compile-layer (DDL) hooks.
 Package compile implements the IVF-PQ plugin's compile-layer (DDL) hooks.
plugin/idxcron
Package idxcron is IVF-PQ's idxcron hook implementation.
 Package idxcron is IVF-PQ's idxcron hook implementation.
plugin/plan
Package plan implements the Ivfpq plugin's plan-layer hooks.
 Package plan implements the Ivfpq plugin's plan-layer hooks.
plugin/runtime
Package runtime holds the IVF-PQ algorithm's catalog-side metadata: hidden-table types, parameter schema, op-type set, default options.
 Package runtime holds the IVF-PQ algorithm's catalog-side metadata: hidden-table types, parameter schema, op-type set, default options.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.