vectors

package
v0.12.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 2, 2026 License: MIT, MIT Imports: 18 Imported by: 0

README

vectors

Zero-dependency primitive vector math for embedding and retrieval systems in Go.

Cosine similarity, normalization, binary quantization, in-memory search indexes, pooling, rank fusion, and clustering — float32 for hot embedding paths, float64 where precision governs. Standard library only.

go get github.com/dotcommander/reliquary/vector

package main

import (
	"fmt"

	"github.com/dotcommander/reliquary/vector"
)

func main() {
	a := []float32{0.10, 0.20, 0.30}
	b := []float32{0.20, 0.10, 0.40}

	fmt.Println(vectors.Cosine32(a, b)) // cosine similarity in [-1, 1]
}

Why

Embedding pipelines need the same handful of operations over and over: compare two vectors, normalize a batch, shrink vectors to bits for fast candidate generation, fuse rankings from multiple retrievers. vectors provides those as small, allocation-conscious functions with no dependency footprint — drop it into any module without dragging in a math framework.

What's inside

Area Highlights
Similarity & distance Cosine32/64, CosineDistance32/64, Dot32/64, Euclidean64, JaccardWords
Normalization in-place Normalize32/64, non-mutating NormalizeTo32/64, IsUnit32/64
Binary quantization Quantize, ComputeMedians, HammingDistance, BinaryVector
Blob storage EncodeFloat32Vec / DecodeFloat32Vec, DotFromBlob and TopKFromBlob for on-disk vectors
In-memory search ExactIndex (exact cosine), SearchKeys rerank, BinaryIndex (Hamming candidate generation), checked build reports
Pooling MeanPool32, WeightedMeanPool32
Rank fusion RRF — reciprocal rank fusion across multiple ranked lists
Clustering root KMeans (float32) and the clustering subpackage (k-means, HAC, silhouette)
Semantic boundaries sliding-window similarity, elbow/curvature detection, adaptive thresholds
Artifact identity index manifests, ANN profile result records, and transform/profile digests
Binary quantization at a glance
medians := vectors.ComputeMedians(corpus)        // per-dimension thresholds
q, _ := vectors.Quantize(query, medians)         // []float32 -> BinaryVector
hamming := vectors.HammingDistance(q, candidate) // fast, branch-light
Near-duplicate detection

Group embeddings that are close in cosine space — the semantic complement to the lexical SimHash near-dup primitive in the dedup module.

groups := vectors.NearDuplicateGroups(corpus, 0.92) // [][]int of mutual near-dups
for _, g := range groups {
	fmt.Println("near-duplicate cluster:", g) // indices into corpus, size >= 2
}

pairs := vectors.NearDuplicatePairs(corpus, 0.92) // [][2]int linked (i<j) pairs

NearDuplicateGroups returns connected components (singletons omitted); NearDuplicatePairs returns the raw linked pairs. v1 verifies every pair with exact Cosine32 (a brute-force-correct baseline), so no true positive is dropped; binary quantization is wired in as the O(n) screen for a future radius-bounded prefilter.

Index and ANN profile identity

IndexManifest records caller-owned embedding, chunking, and profile identity. HashIndexProfile remains the legacy stable hex helper for persisted profile hashes. New callers can use HashIndexProfileIdentity(fields).String() for an ordered transform digest and store it in IndexProfileHash.

ANNProfile and ANNProfileResult describe candidate limits, oversampling, quantization labels, exact-rescore flags, memory estimates, recall@K, and latency summaries without importing an ANN engine or vector database.

Vector index knee harness

Use the opt-in test harness to compare the exact index with binary candidate generation followed by exact SearchKeys reranking:

RELIQUARY_VECTOR_INDEX_KNEE=1 \
GOWORK=off go test ./vector \
  -run '^TestVectorIndexKneeHarness$' -count=1 -v

Use strict mode only when a comparison needs explicit binary recall or latency gates. This deterministic quality gate raises the candidate limit because the default 100-candidate evidence run does not meet these recall thresholds:

RELIQUARY_VECTOR_INDEX_KNEE=1 \
RELIQUARY_VECTOR_INDEX_KNEE_CANDIDATES=700 \
RELIQUARY_VECTOR_INDEX_KNEE_STRICT=1 \
RELIQUARY_VECTOR_INDEX_KNEE_APPROX_MIN_RECALL_AT_1=0.99 \
RELIQUARY_VECTOR_INDEX_KNEE_APPROX_MIN_RECALL_AT_5=0.98 \
GOWORK=off go test ./vector \
  -run '^TestVectorIndexKneeHarness$' -count=1 -v

Use the reduced cohort to verify concurrent query execution under the race detector:

RELIQUARY_VECTOR_INDEX_KNEE=1 \
RELIQUARY_VECTOR_INDEX_KNEE_SIZES=200 \
RELIQUARY_VECTOR_INDEX_KNEE_DIM=64 \
RELIQUARY_VECTOR_INDEX_KNEE_QUERIES=16 \
RELIQUARY_VECTOR_INDEX_KNEE_CONCURRENCY=4 \
GOWORK=off go test -race ./vector \
  -run '^TestVectorIndexKneeHarness$' -count=1

The harness does not select or recommend an index default.

Reciprocal rank fusion
// Fuse rankings from two retrievers (e.g. dense + lexical).
fused, _ := vectors.RRF([][]int{denseRanks, lexicalRanks}, 60)
for _, s := range fused {
	fmt.Println(s.Index, s.Score)
}

float32 vs float64

Use float32 for embedding and retrieval — it halves memory and is faster on modern hardware. Reach for float64 only when numerical precision is the governing constraint (e.g. iterative clustering math). The two spaces are not interchangeable; pick one per pipeline.

Documentation

  • docs/api-reference.md — every exported symbol in the root package, with semantics and edge cases.
  • docs/clustering.md — the clustering subpackage (ClusterService, k-means, HAC, silhouette analysis).
  • go doc github.com/dotcommander/reliquary/vector — package overview from source.

License

MIT © DotCommander contributors

Documentation

Overview

Package vectors provides small vector math helpers for embedding and retrieval systems.

Package vectors provides zero-dependency primitive vector math for embedding and retrieval systems.

Similarity

Cosine similarity and distance are available in float32 and float64 variants (Cosine32, Cosine64, CosineDistance32, CosineDistance64). Use float32 for embedding pipelines — it halves memory and is faster on modern hardware. Use float64 only when numerical precision is the governing constraint. Dot product helpers (Dot32, Dot64, DotFromBlob, TopKFromBlob) are provided for callers that normalize up front and want raw dot as a proxy for cosine. Euclidean64 and Jaccard (JaccardWords) cover additional similarity regimes.

Normalization

Normalize32/Normalize64 L2-normalize a vector in place and return the original magnitude. NormalizeTo32/NormalizeTo64 return a new normalized slice. These normalization helpers use scaled accumulation so extreme finite magnitudes remain normalizable. NormSquared32/NormSquared64 and IsUnit32/IsUnit64 are low-level helpers for callers that need to pre-check or validate unit vectors.

Binary quantization

Quantize encodes a float32 embedding as a BinaryVector ([]uint64 packed bits) using per-dimension thresholds; QuantizeInto is the allocation-free variant. HammingDistance counts differing bits between two BinaryVectors in O(dims/64) time. ComputeMedians/ComputeMediansChecked derive thresholds from a corpus of embeddings. Together these support compact storage and fast approximate pre-filtering: Hamming distance on quantized vectors is ~10x faster than float32 cosine, making it a practical first-pass filter before exact re-ranking.

Pooling and aggregation

MeanPool32 averages a slice of vectors into one. WeightedMeanPool32 weights the average by a parallel float64 slice. AverageSimilarity summarizes pairwise similarity across a set of embeddings.

In-memory indexes

ExactIndex is a flat brute-force cosine index suitable for corpora up to ~100 K chunks. SearchKeys reranks selected (group, chunk) candidates without importing caller storage policy. BinaryIndex is a Hamming-distance pre-filter index backed by packed bit blobs, intended for larger corpora or memory-constrained environments. Checked constructors return IndexBuildReport values so callers can log skipped rows without changing compatibility paths. IndexManifest records dimensions, embedding identity, median thresholds, and chunking identity so persisted indexes can be rejected when stale.

Auxiliary utilities

K-means clustering (KMeans, KMeans64, FindOptimalK, FindOptimalK64, SilhouetteScore, AverageSilhouetteScore64), sliding-window and semantic boundary detection (SlidingWindowSimilarity, FindSemanticBoundaries, FindElbowCurvature), Reciprocal Rank Fusion (RRF), score normalization (MinMaxNormalize, CosineToUnit, Clamp01), statistical helpers (MeanStddev, GaussianSmooth, Gradient), and binary blob encoding/decoding (EncodeFloat32Vec, DecodeFloat32Vec, EncodeFloat64Vec, DecodeFloat64Vec) are also provided.

Dependencies

This package's production code uses the standard library only. The enclosing reliquary module has dependencies for sibling packages such as chunking.

Index

Examples

Constants

View Source 
const IndexManifestVersion = 1

IndexManifestVersion is the current IndexManifest schema version.

Variables

This section is empty.

Functions

func AdaptiveThreshold 

func AdaptiveThreshold(similarities []float32) float32

AdaptiveThreshold calculates an adaptive threshold based on similarity distribution.

func AverageSilhouetteScore64 

func AverageSilhouetteScore64(points [][]float64, assignments []int) float64

AverageSilhouetteScore64 computes the average silhouette score.

func AverageSimilarity 

func AverageSimilarity(embeddings [][]float32) float32

AverageSimilarity calculates the average cosine similarity between multiple vector pairs. Returns 1 when fewer than two embeddings are provided or no pairs exist.

func BinaryWords 

func BinaryWords(dim int) int

BinaryWords returns the number of uint64 words required to represent dim bits.

func Clamp01 

func Clamp01(x float64) float64

Clamp01 clamps x into [0,1]. NaN and negative values collapse to 0; values above 1 collapse to 1. The NaN/negative check is FIRST and deliberate: a naive `x < 0` / `x > 1` pair lets NaN fall through (both comparisons are false), returning NaN and poisoning any weighted sum downstream.

func ClusterSilhouetteScores64 

func ClusterSilhouetteScores64(points [][]float64, assignments []int) map[int]float64

ClusterSilhouetteScores64 computes the mean silhouette score per cluster.

func ComputeCentroid64 

func ComputeCentroid64(points [][]float64) []float64

ComputeCentroid64 returns the mean vector of the given points. Returns nil if points is empty.

func ComputeClusterCentroids64 

func ComputeClusterCentroids64(points [][]float64, assignments []int, k int) [][]float64

ComputeClusterCentroids64 computes k centroids from assignments. Ragged points are skipped because a fixed-width centroid has no valid slot for them.

func ComputeMedians 

func ComputeMedians(vecs [][]float32) []float32

ComputeMedians returns the per-dimension median across a set of float32 vectors. The result is suitable for use as the thresholds argument to Quantize. Returns nil if vectors is empty. For a single vector, returns a copy of it.

func ComputeMediansChecked 

func ComputeMediansChecked(vecs [][]float32) ([]float32, error)

ComputeMediansChecked computes the per-dimension median across a set of float32 vectors with validation. Returns an error on dimension mismatch.

func Cosine32 

func Cosine32(a, b []float32) float32

Cosine32 computes cosine similarity between two float32 vectors. Returns 0 if either vector has zero magnitude.

Example 
a := []float32{1, 2, 3}
b := []float32{1, 2, 4}

vectors.Normalize32(a)
vectors.Normalize32(b)

fmt.Printf("%.3f\n", vectors.Cosine32(a, b))

Output:
0.991

func Cosine64 

func Cosine64(a, b []float64) float64

Cosine64 computes cosine similarity between two float64 vectors using scaled accumulation to avoid overflow and underflow for finite inputs. It returns 0 if either vector has zero magnitude.

func CosineDistance32 

func CosineDistance32(a, b []float32) float32

CosineDistance32 returns L2 distance on cosine space: 1 - cosine(a, b). Mismatch, empty, and zero-magnitude comparisons return 1.

func CosineDistance64 

func CosineDistance64(a, b []float64) float64

CosineDistance64 returns L2 distance on cosine space: 1 - cosine(a, b). Mismatch, empty, and zero-magnitude comparisons return 1.

func CosineToUnit 

func CosineToUnit(score float64) float64

CosineToUnit remaps a cosine similarity in [-1,1] onto [0,1] via (x+1)/2. A NaN input (e.g. a zero-vector cosine) maps to 0 BEFORE the arithmetic, so it cannot propagate through the remap. The result is Clamp01-guarded against out-of-range cosine inputs.

func DecodeFloat32Batch 

func DecodeFloat32Batch(blobs [][]byte) ([][]float32, error)

DecodeFloat32Batch decodes a slice of little-endian float32 blobs, preserving input order: out[i] corresponds to blobs[i]. It fails fast with an error naming the first blob whose length is not a multiple of 4.

func DecodeFloat32Vec 

func DecodeFloat32Vec(blob []byte) []float32

DecodeFloat32Vec decodes a raw little-endian byte slice into []float32. Returns nil if len(blob) is not a multiple of 4.

func DecodeFloat64Vec 

func DecodeFloat64Vec(data []byte) []float64

DecodeFloat64Vec decodes a raw little-endian byte slice into []float64. Returns nil when len(data) is not a multiple of 8.

func Dot32 

func Dot32(a, b []float32) float64

Dot32 computes the dot product of two float32 vectors, accumulating in float64 to reduce rounding error on high-dimensional inputs. For L2-normalized vectors the result equals cosine similarity.

func Dot64 

func Dot64(a, b []float64) float64

Dot64 returns the dot product of two float64 vectors. Returns 0 on length mismatch.

func DotFromBlob 

func DotFromBlob(query []float32, blob []byte) float32

DotFromBlob computes the dot product between a pre-normalized query vector and a raw little-endian float32 BLOB without intermediate allocation. Both vectors MUST be L2-normalized (as guaranteed by the embedder); under that invariant, dot product == cosine similarity. Callers must also ensure both operands contain only finite values. Returns 0 if dimensions do not match.

func EncodeFloat32Vec 

func EncodeFloat32Vec(v []float32) []byte

EncodeFloat32Vec encodes a []float32 as a raw little-endian byte slice. Each float32 occupies 4 bytes; 512 dimensions × 4 bytes = 2048 bytes per vector.

func EncodeFloat64Vec 

func EncodeFloat64Vec(v []float64) []byte

EncodeFloat64Vec encodes a []float64 as a raw little-endian byte slice. Each float64 occupies 8 bytes.

func Euclidean64 

func Euclidean64(a, b []float64) float64

Euclidean64 computes the L2 distance between two float64 vectors. Returns math.Inf(1) if vectors are empty or have different lengths.

func FindElbowCurvature 

func FindElbowCurvature(scores []float32, minKeep int) int

FindElbowCurvature returns the index at which the smoothed curvature peaks. The curvature is |d²/di²| of the Gaussian-smoothed score vector. minKeep guarantees a minimum index regardless of curvature shape — the search for the peak starts at minKeep so we never cut before that point. Returns len(scores)-1 when no meaningful peak is found (flat or trivially short).

func FindOptimalK 

func FindOptimalK(points [][]float32, minK, maxK int, rng *rand.Rand) (int, float64)

FindOptimalK runs K-means for each candidate k in [minK, maxK] and returns the k with the highest silhouette score.

func FindOptimalK64 

func FindOptimalK64(points [][]float64, minK, maxK int) (bestK int, bestScore float64, assignments []int, centroids [][]float64, scores []float64, kValues []int)

FindOptimalK64 runs K-means for each candidate k and returns the best result.

func FindSemanticBoundaries 

func FindSemanticBoundaries(similarities []float32, threshold float32) []int

FindSemanticBoundaries identifies positions where semantic similarity drops below threshold.

func GaussianSmooth 

func GaussianSmooth(scores []float32, sigma float64) []float32

GaussianSmooth applies a 1D Gaussian kernel to scores. sigma controls smoothing width; typical value 1.0. Short vectors and invalid sigma values return a copy unchanged.

func Gradient 

func Gradient(scores []float32) []float32

Gradient computes the discrete gradient of a score slice using central differences. Forward difference at index 0, backward at index n-1.

func HammingDistance 

func HammingDistance(a, b BinaryVector) int

HammingDistance returns the number of differing bits between two binary vectors. Returns 0 if the vectors have different lengths.

func HashANNProfile 

func HashANNProfile(profile ANNProfile) hash.Digest

HashANNProfile returns a stable profile digest for ANN comparison inputs.

func HashFloat32Slice 

func HashFloat32Slice(values []float32) string

HashFloat32Slice returns a stable SHA-256 hash for float32 identity data such as binary-index median thresholds.

func HashIndexProfile 

func HashIndexProfile(fields map[string]string) string

HashIndexProfile returns a stable hash for caller-owned index profile fields such as candidate limits, quantization settings, or external ANN parameters.

func HashIndexProfileIdentity 

func HashIndexProfileIdentity(fields map[string]string) supporthash.Digest

HashIndexProfileIdentity returns a transform identity digest for caller-owned index profile fields. HashIndexProfile is retained for compatibility with existing persisted profile hashes.

func IsUnit32 

func IsUnit32(v []float32, tolerance float64) bool

IsUnit32 validates whether v is a unit vector within tolerance. Returns false for invalid tolerances, empty vectors, zero vectors, and NaN/Inf inputs.

func IsUnit64 

func IsUnit64(v []float64, tolerance float64) bool

IsUnit64 validates whether v is a unit vector within tolerance. Returns false for invalid tolerances, empty vectors, zero vectors, and NaN/Inf inputs.

func JaccardWords 

func JaccardWords(text1, text2 string) float64

JaccardWords computes Jaccard similarity between meaningful words in two texts. Returns 0 for empty inputs.

func MeanPool32 

func MeanPool32(vecs [][]float32) []float32

MeanPool32 averages a set of float32 vectors componentwise and L2-normalizes the result, so the output is a unit vector suitable for cosine-via-dot scoring. Members whose length differs from the first vector's length are skipped. Special cases: an empty input returns nil; a single-vector input is returned unchanged (not normalized).

func MeanStddev 

func MeanStddev(vals []float64) (mean, stddev float64)

MeanStddev returns the mean and POPULATION standard deviation (divisor N, not N-1) of vals. An empty slice returns (0, 0) rather than NaN. The population convention is deliberate so derived thresholds are reproducible across runs.

func MinMaxNormalize 

func MinMaxNormalize(vals []float64) []float64

MinMaxNormalize rescales vals into [0,1] via (v-min)/(max-min), returning a new slice. Degenerate input — fewer than 2 elements, or a spread (max-min) below 1e-10 — returns all 0.5: a (max-min) of ~0 would divide to NaN, and 0.5 keeps such a signal weight-neutral rather than poisoning a downstream weighted sum. An empty input returns an empty (non-nil) slice.

func NearDuplicateGroups 

func NearDuplicateGroups(vecs [][]float32, cosineThreshold float32) [][]int

NearDuplicateGroups groups input vectors into clusters of mutual near-duplicates by cosine similarity. Two vectors are linked when their cosine >= threshold; groups are the connected components of that graph (singletons omitted). Returns indices into vecs. Uses binary quantization as an O(n) Hamming prefilter, then verifies candidate pairs with exact Cosine32.

Guards: nil/empty input or fewer than 2 usable vectors returns an empty result; nil or zero-length member vectors are skipped (never linked).

func NearDuplicatePairs 

func NearDuplicatePairs(vecs [][]float32, cosineThreshold float32) [][2]int

NearDuplicatePairs returns the linked index pairs (i<j, cosine>=threshold) rather than connected-component groups. Pairs are returned in ascending (i, j) order. Same guards as NearDuplicateGroups.

func NewBinaryIndexChecked 

func NewBinaryIndexChecked(blobs [][]byte, groups []string, chunkIndices []int, dims int) (*BinaryIndex, IndexBuildReport)

NewBinaryIndexChecked builds a BinaryIndex and reports skipped rows and build errors. Rows are validated before their keys are reserved, so the first valid row for each (Group, ChunkIndex) key is retained.

func NewExactIndexChecked 

func NewExactIndexChecked(dims int, chunks []IndexChunk, arena []byte) (*ExactIndex, IndexBuildReport)

NewExactIndexChecked constructs an ExactIndex and reports skipped or suspicious rows. Rows containing non-finite vector values are skipped and counted in SkippedBadBlob. It snapshots arena, so callers may mutate or release their slice after this function returns.

func NormSquared32 

func NormSquared32(v []float32) float64

NormSquared32 returns the squared L2 norm of v. Accumulates in float64 to avoid intermediate rounding error.

func NormSquared64 

func NormSquared64(v []float64) float64

NormSquared64 returns the squared L2 norm of v.

func Normalize32 

func Normalize32(v []float32) float32

Normalize32 L2-normalizes a float32 vector in place using scaled accumulation to avoid overflow and underflow for finite inputs. It returns the original magnitude, which may be +Inf when the mathematical magnitude exceeds the float32 range.

func Normalize64 

func Normalize64(v []float64) float64

Normalize64 L2-normalizes a float64 vector in place using scaled accumulation to avoid overflow and underflow for finite inputs. It returns the original magnitude, which may be +Inf when the mathematical magnitude exceeds the float64 range.

func NormalizeTo32 

func NormalizeTo32(v []float32) []float32

NormalizeTo32 returns a new L2-normalized copy of v without mutating the input. If v has zero magnitude, the input slice is returned unchanged (identity preserved) rather than a fresh zero slice. Scaled accumulation avoids overflow and underflow for finite inputs.

func NormalizeTo64 

func NormalizeTo64(v []float64) []float64

NormalizeTo64 is the float64 twin of NormalizeTo32. Zero-magnitude input is returned unchanged.

func QuantizeInto 

func QuantizeInto(dst BinaryVector, vec, thresholds []float32) error

QuantizeInto encodes a float32 embedding into an existing BinaryVector buffer. Bits are cleared before writing to avoid stale bits from prior calls.

func SilhouetteCoefficient64 

func SilhouetteCoefficient64(pointIdx int, points [][]float64, assignments []int) float64

SilhouetteCoefficient64 computes the silhouette coefficient for one point.

func SilhouetteScore 

func SilhouetteScore(points [][]float32, assignments []int, k int) float64

SilhouetteScore computes the average silhouette coefficient for the given clustering. For large datasets (n > silhouetteSampleN), a deterministic subsample is used to keep computation tractable.

func SlidingWindowSimilarity 

func SlidingWindowSimilarity(embeddings [][]float32, windowSize int) []float32

SlidingWindowSimilarity calculates similarity scores using a sliding window. Returns similarity scores between consecutive windows.

func SmoothSimilarities 

func SmoothSimilarities(similarities []float32, windowSize int) []float32

SmoothSimilarities applies smoothing to reduce noise in similarity scores.

func TopKMaxIndices 

func TopKMaxIndices(scores []float32, k int) []int

TopKMaxIndices returns the indices of the k largest values in scores, ordered largest-first. k is clamped to len(scores); k<=0 returns an empty slice. Equal scores are ordered by ascending index (stable). Runs in O(n log k) time, O(k) space.

func TopKMinIndices 

func TopKMinIndices(scores []float32, k int) []int

TopKMinIndices returns the indices of the k smallest values in scores, ordered smallest-first. Same clamping and stability rules as TopKMaxIndices.

func WeightedMeanPool32 

func WeightedMeanPool32(vecs [][]float32, weights []float64) ([]float32, error)

WeightedMeanPool32 computes a weighted mean of vectors, then L2-normalizes the result. It rejects weight mismatches, ragged vectors, and invalid weights.

Types

type ANNProfile 

type ANNProfile struct {
	ID              string
	Kind            IndexKind
	CandidateLimit  int
	Oversampling    float64
	ExactRescore    bool
	Quantization    string
	MemoryEstimate  MemoryEstimate
	TransformDigest hash.Digest
	ProfileDigest   hash.Digest
}

ANNProfile describes one approximate-nearest-neighbor benchmark profile. It is data-only and does not imply a specific ANN implementation.

type ANNProfileResult 

type ANNProfileResult struct {
	Profile        ANNProfile
	RecallAtK      float64
	Latency        LatencySummary
	IndexedRows    int
	EvaluatedQuery int
}

ANNProfileResult records quality and latency observations for a profile.

Example 
transform := hash.HashIdentity(
	hash.IdentityPart{Kind: "embedding_model", ID: "demo-hash", Version: "1"},
	hash.IdentityPart{Kind: "chunker", Version: "semantic-v1", ConfigHash: "chunk-cfg"},
)
exact := vectors.ANNProfile{
	ID:              "exact-baseline",
	Kind:            vectors.IndexKindExact,
	CandidateLimit:  100,
	TransformDigest: transform,
}
binary := vectors.ANNProfile{
	ID:              "binary-screen",
	Kind:            vectors.IndexKindBinary,
	CandidateLimit:  300,
	Oversampling:    3,
	ExactRescore:    true,
	Quantization:    "median_binary",
	TransformDigest: transform,
}
binary.ProfileDigest = vectors.HashANNProfile(binary)

results := []vectors.ANNProfileResult{
	{Profile: exact, RecallAtK: 1, Latency: vectors.LatencySummary{Samples: 20, P95MS: 4.2}},
	{Profile: binary, RecallAtK: 0.92, Latency: vectors.LatencySummary{Samples: 20, P95MS: 1.1}},
}

fmt.Println(results[0].Profile.ID, results[1].Profile.Quantization, results[1].Profile.ProfileDigest.String() != "")

Output:
exact-baseline median_binary true

type BinaryIndex 

type BinaryIndex struct {
	// contains filtered or unexported fields
}

BinaryIndex is a thread-safe, in-memory pre-filter for semantic search. It stores packed bit vectors for all chunks and provides fast Hamming-distance pre-filtering.

func NewBinaryIndex 

func NewBinaryIndex(blobs [][]byte, groups []string, chunkIndices []int, dims int) *BinaryIndex

NewBinaryIndex builds a BinaryIndex from raw little-endian float32 blobs. Rows with invalid or non-finite blobs, missing group/chunk metadata, or duplicate keys are skipped. For duplicate keys, the first valid row wins.

func (*BinaryIndex) Clear 

func (idx *BinaryIndex) Clear()

Clear clears the index.

func (*BinaryIndex) Len 

func (idx *BinaryIndex) Len() int

Len returns the number of entries in the index.

func (*BinaryIndex) Manifest 

func (idx *BinaryIndex) Manifest(identity IndexManifestIdentity) IndexManifest

Manifest returns a compatibility manifest for the binary index, merging caller-owned embedding/chunking identity with index-owned dimensions, row count, and median thresholds hash.

func (*BinaryIndex) SearchCandidates 

func (idx *BinaryIndex) SearchCandidates(queryVec []float32) ([]HammingCandidate, bool)

SearchCandidates returns the best Hamming-distance candidates for exact re-ranking.

func (*BinaryIndex) SearchCandidatesLimit 

func (idx *BinaryIndex) SearchCandidatesLimit(queryVec []float32, limit int) ([]HammingCandidate, bool)

SearchCandidatesLimit returns up to limit best Hamming-distance candidates for exact re-ranking. A limit larger than the index size is clamped; a non-positive limit returns an empty candidate slice for a valid query. Non-finite queries are rejected.

type BinaryIndexEntry 

type BinaryIndexEntry struct {
	Group      string
	ChunkIndex int
}

BinaryIndexEntry maps a binary vector back to its source ID and chunk index.

type BinaryVector 

type BinaryVector []uint64

BinaryVector is a packed bit representation of a float32 vector. For a 768-dimension embedding, this is 12 uint64s (768/64 = 12). For a 1024-dimension embedding, this is 16 uint64s.

func Quantize 

func Quantize(vec []float32, thresholds []float32) (BinaryVector, error)

Quantize converts a float32 embedding to a BinaryVector using per-dimension thresholds. For dimension i: if vec[i] > thresholds[i], the bit is 1; else 0. Bits are packed into uint64s with dimension 0 at bit 0 of uint64[0]. Returns an error if len(vec) != len(thresholds) — typically a sign that the embedder dimension changed between runs and the binary index is stale.

type ExactIndex 

type ExactIndex struct {
	// contains filtered or unexported fields
}

ExactIndex is a thread-safe, in-memory flat vector index. It stores L2-normalized float32 vectors packed in a single contiguous byte arena to minimize GC allocation overhead.

func NewExactIndex 

func NewExactIndex(dims int, chunks []IndexChunk, arena []byte) *ExactIndex

NewExactIndex constructs an ExactIndex. It snapshots arena, so callers may mutate or release their slice after this function returns.

func (*ExactIndex) Clear 

func (idx *ExactIndex) Clear()

Clear clears all internal slices and maps.

func (*ExactIndex) Len 

func (idx *ExactIndex) Len() int

Len returns the number of chunks stored.

func (*ExactIndex) Manifest 

func (idx *ExactIndex) Manifest(identity IndexManifestIdentity) IndexManifest

Manifest returns a compatibility manifest for the exact index, merging caller-owned embedding/chunking identity with index-owned dimensions and row count.

func (*ExactIndex) Search 

func (idx *ExactIndex) Search(queryVec []float32, limit int, minSimilarity float64) ([]SearchResult, bool)

Search scores all vectors against the query and returns the top-K matches. Returns false if the index is empty or the query is invalid.

func (*ExactIndex) SearchFiltered 

func (idx *ExactIndex) SearchFiltered(
	queryVec []float32,
	limit int,
	minSimilarity float64,
	groups []string,
) ([]SearchResult, bool)

SearchFiltered scores only vectors belonging to the specified groups. Returns false if the index is empty or the query is invalid.

func (*ExactIndex) SearchGroupsByMaxPool 

func (idx *ExactIndex) SearchGroupsByMaxPool(queryVec []float32, limit int) ([]SearchResult, bool)

SearchGroupsByMaxPool pools the best similarity score per group across all chunks and returns the top-K best matching groups. Returns false if the index is empty or the query is invalid.

func (*ExactIndex) SearchKeys 

func (idx *ExactIndex) SearchKeys(
	queryVec []float32,
	limit int,
	minSimilarity float64,
	keys []IndexKey,
) ([]SearchResult, bool)

SearchKeys scores only the chunks identified by keys. Duplicate and missing keys are ignored. Returns false if the index is empty or the query is invalid.

type HammingCandidate 

type HammingCandidate struct {
	Group      string
	ChunkIndex int
	Hamming    int // Lower = more similar
}

HammingCandidate holds a candidate from the Hamming pre-filter stage.

type IndexBuildReport 

type IndexBuildReport struct {
	InputRows              int
	IndexedRows            int
	SkippedBadSpan         int
	SkippedBadBlob         int
	SkippedMissingMetadata int
	SkippedDuplicateKey    int
	DimensionMismatch      int
	MedianError            string
	QuantizeError          string
}

IndexBuildReport describes rows accepted or skipped while building an index.

type IndexChunk 

type IndexChunk struct {
	Group      string // e.g. EntryID
	ChunkIndex int    // 0-based index of this chunk within the Group
	Offset     int    // Byte offset into the contiguous arena slice
	Length     int    // Byte length of this vector in the arena
}

IndexChunk defines the location of a chunk's vector in the index arena.

type IndexKey 

type IndexKey struct {
	Group      string
	ChunkIndex int
}

IndexKey identifies one indexed chunk without tying vectors to caller storage.

type IndexKind 

type IndexKind string

IndexKind identifies the index implementation described by a manifest. 

const (
	IndexKindExact  IndexKind = "exact"
	IndexKindBinary IndexKind = "binary"
)

type IndexManifest 

type IndexManifest struct {
	Version            int
	CreatedAt          time.Time
	Kind               IndexKind
	Dims               int
	IndexedRows        int
	EmbeddingModelID   string
	EmbeddingModelHash string
	MediansHash        string
	IndexProfileHash   string
	ChunkStrategy      string
	ChunkSize          int
	ChunkOverlap       int
}

IndexManifest records the compatibility identity and provenance for a built vector index. It is intentionally data-only so callers can persist it in any format they already use.

func NewIndexManifest 

func NewIndexManifest(identity IndexManifestIdentity, indexedRows int) IndexManifest

NewIndexManifest builds a manifest from caller-owned identity fields.

func (IndexManifest) Validate 

func (m IndexManifest) Validate(expected IndexManifestIdentity) error

Validate returns an actionable error when a manifest is incompatible with the expected identity. A fully zero expected identity is valid for backward-compatible callers that have not opted into manifest enforcement.

type IndexManifestIdentity 

type IndexManifestIdentity struct {
	Kind               IndexKind
	Dims               int
	EmbeddingModelID   string
	EmbeddingModelHash string
	MediansHash        string
	IndexProfileHash   string
	ChunkStrategy      string
	ChunkSize          int
	ChunkOverlap       int
}

IndexManifestIdentity describes compatibility inputs that can make a vector index stale when they change. Zero-valued fields are treated as unspecified by Validate so callers can enforce only the identity dimensions they own.

type KMeans64Config 

type KMeans64Config struct {
	K             int
	MaxIterations int
	Tolerance     float64
	Seed          int64
}

KMeans64Config holds configuration for float64 K-means clustering.

type KMeans64Result 

type KMeans64Result struct {
	Assignments []int
	Centroids   [][]float64
	K           int
	Iterations  int
	Converged   bool
}

KMeans64Result holds the result of a float64 K-means clustering run.

func KMeans64 

func KMeans64(points [][]float64, cfg KMeans64Config) *KMeans64Result

KMeans64 performs K-means clustering with K-means++ initialization using cosine distance. It preserves the legacy float64 clustering semantics used by the higher-level clustering service.

type KMeansResult 

type KMeansResult struct {
	K           int
	Assignments []int       // Assignments[i] = cluster ID for points[i]
	Centroids   [][]float32 // Centroids[j] = centroid vector for cluster j
	Iterations  int
}

KMeansResult holds the output of a K-means clustering run.

func KMeans 

func KMeans(points [][]float32, k int, rng *rand.Rand) *KMeansResult

KMeans runs K-means clustering with K-means++ initialization on the given points using cosine distance. Points must be non-empty, uniform-dimensional, and L2-normalized. Invalid point shapes return an empty result.

type LatencySummary 

type LatencySummary struct {
	Samples int
	MinMS   float64
	P50MS   float64
	P95MS   float64
	MaxMS   float64
}

LatencySummary records deterministic benchmark sample summaries. Tests should compare explicit values, not wall-clock thresholds.

type MemoryEstimate 

type MemoryEstimate struct {
	Bytes int64
	Label string
}

MemoryEstimate records caller-measured or estimated index memory.

type Scored 

type Scored struct {
	Index int
	Score float64
}

Scored pairs an index with its fused score. Sorted by descending score, then ascending index for deterministic tie-breaks.

func RRF 

func RRF(ranked [][]int, k float64) ([]Scored, float64)

RRF fuses ranked index lists via Reciprocal Rank Fusion. Each list is a ranking where position i has rank i+1.

Each index contribution is: 

score += 1/(k+rank)

where rank is 1-based.

Raw metric scores are not used; only rank positions contribute.

Returns the fused ranking sorted by descending score and ascending index for ties, and the maximum score (or 0 for empty input).

type ScoredIndex 

type ScoredIndex struct {
	Index int
	Score float32
}

ScoredIndex identifies one scored input row.

func TopKFromBlob 

func TopKFromBlob(query []float32, blobs [][]byte, limit int, minScore float32) []ScoredIndex

TopKFromBlob scores raw little-endian float32 blobs against query and returns the top limit input indexes. Invalid or dimension-mismatched blobs are skipped, as are blobs whose score is non-finite. A non-finite query returns no results. Equal scores are ordered by ascending input index.

type SearchResult 

type SearchResult struct {
	Group      string
	ChunkIndex int
	Score      float64
}

SearchResult holds a query match.

Source Files

View all Source files

Directories

Path Synopsis
Package clustering performs online and offline clustering over caller-supplied vector embeddings.
 Package clustering performs online and offline clustering over caller-supplied vector embeddings.
Package pq implements Product Quantization for efficient vector compression.
 Package pq implements Product Quantization for efficient vector compression.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.