vectorstore

package
v0.0.0-...-7871f83 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Dec 23, 2025 License: Apache-2.0 Imports: 4 Imported by: 0

Documentation

Overview

Package vectorstore provides interfaces and implementations for vector storage backends.

Overview

The vectorstore package defines the VectorStore interface for storing and searching documents by their vector embeddings. This enables semantic search, RAG (Retrieval-Augmented Generation), and similarity matching workflows.

Core Concepts

  • Document: A storable item with content, embedding, and metadata
  • VectorStore: Interface for Add/Search/Delete operations
  • EmbeddingStore: Helper that auto-generates embeddings from text
  • Filter: Metadata-based filtering for search queries

Available Implementations

  • memory.Store: In-memory store for testing and development
  • qdrant.Store: Qdrant vector database (gRPC-based)
  • pgvector.Store: PostgreSQL with pgvector extension
  • pinecone.Store: Pinecone managed vector database
  • weaviate.Store: Weaviate vector search engine
  • s3vectors.Store: Amazon S3 Vectors native storage

Basic Usage

// Create an in-memory store
store := memory.New()

// Add documents with embeddings
docs := []vectorstore.Document{
    {ID: "1", Content: "Hello world", Embedding: []float32{0.1, 0.2, ...}},
    {ID: "2", Content: "Goodbye world", Embedding: []float32{0.3, 0.4, ...}},
}
store.Add(ctx, docs)

// Search by embedding vector
results, _ := store.Search(ctx, queryEmbedding, vectorstore.SearchOptions{K: 5})

Using EmbeddingStore

For convenience, EmbeddingStore auto-generates embeddings:

embedder := openai.NewEmbedder()
store := memory.New()
es := vectorstore.NewEmbeddingStore(store, embedder)

// Add texts (embeddings generated automatically)
es.AddTexts(ctx, []string{"doc1", "doc2"}, nil)

// Search by text query
results, _ := es.SearchText(ctx, "find similar docs", vectorstore.SearchOptions{K: 10})

Metadata Filtering

Filter search results by metadata:

results, _ := store.Search(ctx, embedding, vectorstore.SearchOptions{
    K:      10,
    Filter: vectorstore.Eq("category", "science"),
})

Namespaces

Partition data for multi-tenant scenarios:

store.Add(ctx, docs, func(o *vectorstore.AddOptions) {
    o.Namespace = "tenant-123"
})

results, _ := store.Search(ctx, embedding, vectorstore.SearchOptions{
    Namespace: "tenant-123",
})

Index

Constants

View Source
const DefaultK = 10

DefaultK is the default number of results if K is not specified.

Variables

This section is empty.

Functions

This section is empty.

Types

type AddOptions

type AddOptions struct {
	// Namespace partitions the store.
	Namespace string

	// Upsert controls whether to update existing documents. Default: true
	Upsert bool
}

AddOptions configures document addition behavior.

type Document

type Document struct {
	// ID is the unique identifier. If empty on Add, one will be generated.
	ID string

	// Content is the original text content.
	Content string

	// Embedding is the vector representation.
	Embedding embedding.Vector

	// Metadata contains arbitrary key-value pairs for filtering.
	Metadata Metadata

	// Score is the similarity score (populated on search results).
	Score float64

	// Timestamp is when this document was added/updated.
	Timestamp time.Time
}

Document represents a storable/retrievable item with vector and metadata.

type EmbeddingStore

type EmbeddingStore struct {
	// contains filtered or unexported fields
}

EmbeddingStore wraps a VectorStore with automatic embedding generation. It simplifies common workflows by handling embedding creation internally.

func NewEmbeddingStore

func NewEmbeddingStore(store VectorStore, embedder embedding.Embedder) *EmbeddingStore

NewEmbeddingStore creates a store that auto-embeds content.

func (*EmbeddingStore) AddDocuments

func (es *EmbeddingStore) AddDocuments(ctx context.Context, docs []Document, opts ...func(*AddOptions)) error

AddDocuments adds pre-structured documents, generating embeddings from their content. Documents with existing embeddings are added as-is.

func (*EmbeddingStore) AddTexts

func (es *EmbeddingStore) AddTexts(ctx context.Context, texts []string, metadata []Metadata, opts ...func(*AddOptions)) error

AddTexts embeds and stores text documents. If metadata slice is shorter than texts, remaining documents get nil metadata.

func (*EmbeddingStore) Close

func (es *EmbeddingStore) Close() error

Close releases resources (delegates to underlying store).

func (*EmbeddingStore) Delete

func (es *EmbeddingStore) Delete(ctx context.Context, ids []string, namespace string) error

Delete removes documents by ID (delegates to underlying store).

func (*EmbeddingStore) Embedder

func (es *EmbeddingStore) Embedder() embedding.Embedder

Embedder returns the underlying Embedder.

func (*EmbeddingStore) SearchText

func (es *EmbeddingStore) SearchText(ctx context.Context, query string, opts SearchOptions) ([]Document, error)

SearchText embeds query and performs similarity search.

func (*EmbeddingStore) Store

func (es *EmbeddingStore) Store() VectorStore

Store returns the underlying VectorStore.

type Filter

type Filter map[string]any

Filter represents a metadata filter for search queries. Simple map-based approach - backends translate as needed.

func And

func And(filters ...Filter) Filter

And combines multiple filters (merged map).

func Eq

func Eq(field string, value any) Filter

Eq creates an equality filter for a field.

func In

func In(field string, values ...any) Filter

In creates a filter matching any of the provided values.

type FusionAlgorithm

type FusionAlgorithm string

FusionAlgorithm specifies how to combine keyword and vector search results.

const (
	// FusionRRF uses Reciprocal Rank Fusion to combine results.
	// Score = sum(1 / (k + rank)) for each result list.
	FusionRRF FusionAlgorithm = "rrf"

	// FusionRelativeScore normalizes and combines scores from both searches.
	FusionRelativeScore FusionAlgorithm = "relativeScore"
)

type HybridSearchOptions

type HybridSearchOptions struct {
	SearchOptions

	// Alpha controls the balance between keyword and vector search.
	// 0.0 = pure keyword (BM25/sparse), 1.0 = pure vector (dense).
	// Default: 0.5 (equal weighting)
	Alpha float64

	// FusionAlgorithm specifies how to combine results.
	// Default: RRF (Reciprocal Rank Fusion)
	FusionAlgorithm FusionAlgorithm
}

HybridSearchOptions configures hybrid search behavior.

func (*HybridSearchOptions) Normalize

func (o *HybridSearchOptions) Normalize()

Normalize ensures hybrid options have sensible defaults.

type Indexer

type Indexer interface {
	VectorStore

	// CreateIndex creates a new index/collection.
	CreateIndex(ctx context.Context, name string, dims int, metric embedding.Metric) error

	// DeleteIndex removes an index and all its data.
	DeleteIndex(ctx context.Context, name string) error

	// ListIndexes returns all available indexes.
	ListIndexes(ctx context.Context) ([]string, error)
}

Indexer extends VectorStore with index lifecycle management.

type Metadata

type Metadata = map[string]any

Metadata is a map of string keys to any values.

type SearchOptions

type SearchOptions struct {
	// K is the maximum number of results to return. Default: 10
	K int

	// MinScore filters results below this similarity threshold (0.0-1.0).
	MinScore float64

	// Filter applies metadata-based filtering.
	Filter Filter

	// IncludeEmbeddings controls whether embeddings are returned in results.
	IncludeEmbeddings bool

	// Namespace partitions the store (for multi-tenant scenarios).
	Namespace string
}

SearchOptions configures similarity search behavior.

func (*SearchOptions) Normalize

func (o *SearchOptions) Normalize()

Normalize ensures options have sensible defaults.

type Stats

type Stats struct {
	DocumentCount int64
	IndexSize     int64 // bytes
	Dimensions    int
}

Stats provides storage statistics.

type StatsProvider

type StatsProvider interface {
	Stats(ctx context.Context, namespace string) (Stats, error)
}

StatsProvider optionally exposes storage statistics.

type TextSearcher

type TextSearcher interface {
	VectorStore

	// SearchHybrid combines keyword and vector search.
	// The alpha parameter in HybridSearchOptions controls the balance:
	// 0.0 = pure keyword search, 1.0 = pure vector search.
	SearchHybrid(ctx context.Context, query string, embedding embedding.Vector, opts HybridSearchOptions) ([]Document, error)
}

TextSearcher extends VectorStore with hybrid (keyword + vector) search.

type VectorStore

type VectorStore interface {
	// Add inserts or updates documents in the store.
	Add(ctx context.Context, docs []Document, opts ...func(*AddOptions)) error

	// Search finds documents similar to the query embedding.
	Search(ctx context.Context, embedding embedding.Vector, opts SearchOptions) ([]Document, error)

	// Delete removes documents by ID.
	Delete(ctx context.Context, ids []string, namespace string) error

	// Close releases resources.
	Close() error
}

VectorStore defines the contract for vector storage backends.

Directories

Path Synopsis
Package memory provides an in-memory VectorStore implementation.
Package memory provides an in-memory VectorStore implementation.
Package pgvector provides a PostgreSQL pgvector-backed VectorStore implementation.
Package pgvector provides a PostgreSQL pgvector-backed VectorStore implementation.
Package pinecone provides a Pinecone-backed VectorStore implementation.
Package pinecone provides a Pinecone-backed VectorStore implementation.
Package qdrant provides a Qdrant-backed VectorStore implementation.
Package qdrant provides a Qdrant-backed VectorStore implementation.
Package s3vectors provides an Amazon S3 Vectors-backed VectorStore implementation.
Package s3vectors provides an Amazon S3 Vectors-backed VectorStore implementation.
Package weaviate provides a Weaviate-backed VectorStore implementation.
Package weaviate provides a Weaviate-backed VectorStore implementation.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL