Documentation
¶
Index ¶
- Constants
- Variables
- func CosineDistance[T types.RealNumbers](p, q []T) (T, error)
- func CosineSimilarity[T types.RealNumbers](p, q []T) (T, error)
- func DistanceTransformHnsw(dist float64, origMetricType MetricType, metricType usearch.Metric) float64
- func DistanceTransformIvfflat(dist float64, origMetricType, metricType MetricType) float64
- func GoPairWiseDistance[T types.RealNumbers](x [][]T, y [][]T, metric MetricType) ([]float32, error)
- func InnerProduct[T types.RealNumbers](p, q []T) (T, error)
- func L1Distance[T types.RealNumbers](p, q []T) (T, error)
- func L2Distance[T types.RealNumbers](v1, v2 []T) (T, error)
- func L2DistanceSq[T types.RealNumbers](p, q []T) (T, error)
- func MaxFloat[T types.RealNumbers]() T
- func NormalizeL2[T types.RealNumbers](v1 []T, normalized []T) error
- func PairWiseDistance[T types.RealNumbers](x [][]T, y [][]T, metric MetricType, _ bool) ([]float32, error)
- func PairwiseDistanceWait(handle PairwiseJobHandle, metric MetricType) ([]float32, error)
- func PairwiseDistanceWaitCPU(handle PairwiseJobHandle, metric MetricType) ([]float32, error)
- func ScaleInPlace[T types.RealNumbers](v []T, scale T)
- func SphericalDistance[T types.RealNumbers](p, q []T) (T, error)
- func ValidQuantization(val string) bool
- type DistanceFunction
- func ResolveDistanceFn[T types.RealNumbers](metric MetricType) (DistanceFunction[T], error)
- func ResolveKmeansDistanceFn[T types.RealNumbers](metric MetricType, spherical bool) (DistanceFunction[T], bool, error)
- func ResolveKmeansDistanceFnForDense[T types.RealNumbers](metric MetricType) (DistanceFunction[T], bool, error)
- func ResolveKmeansDistanceFnForSparse[T types.RealNumbers](metric MetricType) (DistanceFunction[T], bool, error)
- type MetricType
- type PairwiseJobHandle
- type QuantizationType
Constants ¶
const ( OpType_L2Distance = "vector_l2_ops" OpType_L2sqDistance = "vector_l2sq_ops" OpType_InnerProduct = "vector_ip_ops" OpType_CosineDistance = "vector_cosine_ops" OpType_L1Distance = "vector_l1_ops" DistFn_L2Distance = "l2_distance" DistFn_L2sqDistance = "l2_distance_sq" DistFn_InnerProduct = "inner_product" DistFn_CosineDistance = "cosine_distance" DistFn_L1Distance = "l1_distance" DistIntFn_L2Distance = "l2_distance_sq" DistIntFn_InnerProduct = "inner_product" DistIntFn_CosineDistance = "cosine_distance" DistIntFn_L1Distance = "l1_distance" )
const ( Quantization_F32_Str = "float32" Quantization_F16_Str = "float16" Quantization_INT8_Str = "int8" Quantization_UINT8_Str = "uint8" Quantization_F64_Str = "float64" )
const GPUThresholdOverlapped = uint64(0)
const GPUThresholdSQL = GPUThresholdSync / 4
const GPUThresholdSync = uint64(4 * 1024 * 1024)
GPUThresholdSync and GPUThresholdOverlapped are defined here for non-gpu builds so that callers can reference them unconditionally.
Variables ¶
var ( DistFuncOpTypes = map[string]string{ DistFn_L2Distance: OpType_L2Distance, DistFn_L2sqDistance: OpType_L2Distance, DistFn_InnerProduct: OpType_InnerProduct, DistFn_CosineDistance: OpType_CosineDistance, } OpTypeToIvfMetric = map[string]MetricType{ OpType_L2Distance: Metric_L2sqDistance, OpType_L2sqDistance: Metric_L2sqDistance, OpType_InnerProduct: Metric_InnerProduct, OpType_CosineDistance: Metric_CosineDistance, OpType_L1Distance: Metric_L1Distance, } OpTypeToUsearchMetric = map[string]usearch.Metric{ OpType_L2Distance: usearch.L2sq, OpType_L2sqDistance: usearch.L2sq, OpType_InnerProduct: usearch.InnerProduct, OpType_CosineDistance: usearch.Cosine, } MetricTypeToUsearchMetric = map[MetricType]usearch.Metric{ Metric_L2Distance: usearch.L2sq, Metric_L2sqDistance: usearch.L2sq, Metric_InnerProduct: usearch.InnerProduct, Metric_CosineDistance: usearch.Cosine, } MetricTypeToDistFuncName = map[MetricType]string{ Metric_L2Distance: DistFn_L2Distance, Metric_L2sqDistance: DistFn_L2sqDistance, Metric_InnerProduct: DistFn_InnerProduct, Metric_CosineDistance: DistFn_CosineDistance, Metric_L1Distance: DistFn_L1Distance, } DistFuncNameToMetricType = map[string]MetricType{ DistFn_L2Distance: Metric_L2Distance, DistFn_L2sqDistance: Metric_L2sqDistance, DistFn_InnerProduct: Metric_InnerProduct, DistFn_CosineDistance: Metric_CosineDistance, DistFn_L1Distance: Metric_L1Distance, } )
var CuvsQuantizationNameToType = map[string]QuantizationType{ Quantization_F32_Str: Quantization_F32, Quantization_F16_Str: Quantization_F16, Quantization_INT8_Str: Quantization_INT8, Quantization_UINT8_Str: Quantization_UINT8, }
CuvsQuantizationNameToType is the analogous map for the cuvs (CAGRA / IVF-PQ) backend. cuvs does NOT support float64, so f64 is intentionally omitted; including it here would let CREATE INDEX pass the validator and then fail downstream in the GPU code path.
var UsearchQuantizationNameToType = map[string]QuantizationType{ Quantization_F32_Str: Quantization_F32, Quantization_F16_Str: Quantization_F16, Quantization_F64_Str: Quantization_F64, Quantization_INT8_Str: Quantization_INT8, Quantization_UINT8_Str: Quantization_UINT8, }
UsearchQuantizationNameToType maps a SQL quantization name to its enum value for the usearch (HNSW) backend, which supports float32, float16, float64, int8, and uint8.
Functions ¶
func CosineDistance ¶
func CosineDistance[T types.RealNumbers](p, q []T) (T, error)
CosineDistance calculates the cosine distance between two vectors using generics.
Formula: Cosine Distance = 1 - Cosine Similarity Cosine Similarity = (v1 · v2) / (||v1|| * ||v2||)
This implementation uses loop unrolling to optimize the calculation of the dot product (v1 · v2) and the squared L2 norms (||v1||², ||v2||²) in a single pass. This improves performance by reducing loop overhead and maximizing CPU cache efficiency.
func CosineSimilarity ¶
func CosineSimilarity[T types.RealNumbers](p, q []T) (T, error)
CosineSimilarity calculates the cosine similarity between two vectors using generics.
Formula: Cosine Distance = 1 - Cosine Similarity Cosine Similarity = (v1 · v2) / (||v1|| * ||v2||)
This implementation uses loop unrolling to optimize the calculation of the dot product (v1 · v2) and the squared L2 norms (||v1||², ||v2||²) in a single pass. This improves performance by reducing loop overhead and maximizing CPU cache efficiency.
func DistanceTransformHnsw ¶
func DistanceTransformHnsw(dist float64, origMetricType MetricType, metricType usearch.Metric) float64
func DistanceTransformIvfflat ¶
func DistanceTransformIvfflat(dist float64, origMetricType, metricType MetricType) float64
func GoPairWiseDistance ¶
func GoPairWiseDistance[T types.RealNumbers]( x [][]T, y [][]T, metric MetricType, ) ([]float32, error)
func InnerProduct ¶
func InnerProduct[T types.RealNumbers](p, q []T) (T, error)
InnerProductUnrolled calculates the inner product using loop unrolling. This can significantly improve performance for large vectors by reducing loop overhead and enabling better CPU instruction scheduling.
func L1Distance ¶
func L1Distance[T types.RealNumbers](p, q []T) (T, error)
L1DistanceUnrolled calculates the L1 distance using loop unrolling for optimization. It processes 8 elements per iteration to reduce loop overhead and improve performance on large vectors. It also uses an inline 'abs' for potential speed gains.
func L2Distance ¶
func L2Distance[T types.RealNumbers](v1, v2 []T) (T, error)
func L2DistanceSq ¶
func L2DistanceSq[T types.RealNumbers](p, q []T) (T, error)
L2SquareDistanceUnrolled calculates the L2 square distance using loop unrolling. This optimization can improve performance for large vectors by reducing loop overhead and allowing for better instruction-level parallelism.
func MaxFloat ¶
func MaxFloat[T types.RealNumbers]() T
func NormalizeL2 ¶
func NormalizeL2[T types.RealNumbers](v1 []T, normalized []T) error
func PairWiseDistance ¶
func PairWiseDistance[T types.RealNumbers]( x [][]T, y [][]T, metric MetricType, _ bool, ) ([]float32, error)
gpuMode is accepted-but-ignored in non-gpu builds — CPU is the only option here. The signature matches the gpu.go variant so callers pass the flag uniformly.
func PairwiseDistanceWait ¶
func PairwiseDistanceWait(handle PairwiseJobHandle, metric MetricType) ([]float32, error)
func PairwiseDistanceWaitCPU ¶
func PairwiseDistanceWaitCPU(handle PairwiseJobHandle, metric MetricType) ([]float32, error)
PairwiseDistanceWaitCPU returns the results of the pairwise distance calculation performed on the CPU.
func ScaleInPlace ¶
func ScaleInPlace[T types.RealNumbers](v []T, scale T)
func SphericalDistance ¶
func SphericalDistance[T types.RealNumbers](p, q []T) (T, error)
SphericalDistance is used for InnerProduct and CosineDistance in Spherical Kmeans. NOTE: spherical distance between two points on a sphere is equal to the angular distance between the two points, scaled by pi. Refs: https://en.wikipedia.org/wiki/Great-circle_distance#Vector_version
func ValidQuantization ¶
ValidQuantization gates the QUANTIZATION='X' option in CREATE INDEX for CAGRA / IVF-PQ — both cuvs-backed, so the cuvs map is the source of truth. See pkg/catalog/secondary_index_utils.go.
Types ¶
type DistanceFunction ¶
type DistanceFunction[T types.RealNumbers] func(v1, v2 []T) (T, error)
DistanceFunction is a function that computes the distance between two vectors NOTE: clusterer already ensures that the all the input vectors are of the same length, so we don't need to check for that here again and return error if the lengths are different.
func ResolveDistanceFn ¶
func ResolveDistanceFn[T types.RealNumbers](metric MetricType) (DistanceFunction[T], error)
ResolveDistanceFn is used for similarity score for search and assign vector to centroids (CENTROIDX JOIN / ProductL2). IMPORTANT: Don't use it for Elkans Kmeans. NOTE: Metric_L2Distance returns L2DistanceSq (squared distance). Callers that need true L2 must apply sqrt to each result afterwards (as GoPairWiseDistance does).
func ResolveKmeansDistanceFn ¶
func ResolveKmeansDistanceFn[T types.RealNumbers](metric MetricType, spherical bool) (DistanceFunction[T], bool, error)
func ResolveKmeansDistanceFnForDense ¶
func ResolveKmeansDistanceFnForDense[T types.RealNumbers](metric MetricType) (DistanceFunction[T], bool, error)
func ResolveKmeansDistanceFnForSparse ¶
func ResolveKmeansDistanceFnForSparse[T types.RealNumbers](metric MetricType) (DistanceFunction[T], bool, error)
IMPORTANT: Spherical Kmeans always use Spherical Distance / Cosine Similarity for Sparse vector or text embedding (TD-IDF). After getting the centroids, we can use other distance function specified by user to assign vector to corresponding centroids (CENTROIDX JOIN / ProductL2).
type MetricType ¶
type MetricType uint16
const ( Metric_L2Distance MetricType = iota Metric_L2sqDistance Metric_InnerProduct Metric_CosineDistance Metric_L1Distance Metric_TypeCount )
type PairwiseJobHandle ¶
type PairwiseJobHandle uint64
PairwiseJobHandle identifies a pending pairwise-distance computation. It is a plain uint64 to avoid heap allocation:
bit 63 = 1 → CPU job bit 63 = 0, value ≠ 0 → GPU job value = 0 → invalid (zero value)
The map key in the CPU job store is the full handle value (CPU bit included), so no masking is needed on the wait side.
func PairwiseDistanceLaunch ¶
func PairwiseDistanceLaunch[T types.RealNumbers]( x [][]T, y [][]T, metric MetricType, dist []float32, _ uint64, _ bool, ) (PairwiseJobHandle, error)
func PairwiseDistanceLaunchCPU ¶
func PairwiseDistanceLaunchCPU[T types.RealNumbers]( x [][]T, y [][]T, metric MetricType, dist []float32, ) (PairwiseJobHandle, error)
PairwiseDistanceLaunchCPU captures parameters for a pairwise distance calculation on CPU. While this is currently synchronous for CPU (it performs the calculation in Launch), it follows the asynchronous interface to support the pipelined execution model used in the block reader.
func (PairwiseJobHandle) IsValid ¶
func (h PairwiseJobHandle) IsValid() bool
IsValid reports whether the handle refers to a real pending job.
type QuantizationType ¶
type QuantizationType uint16
const ( Quantization_F32 QuantizationType = iota Quantization_F16 Quantization_INT8 Quantization_UINT8 Quantization_F64 )
Source Files