vector

package
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 14, 2026 License: Apache-2.0 Imports: 22 Imported by: 0

Documentation

Overview

This file is excluded from Windows builds: github.com/coder/hnsw depends on github.com/google/renameio for atomic index-file writes, and renameio has no Windows support. See hnsw_store_windows.go for the stub that keeps the rest of the package compiling there — vector.BackendSQLite and vector.BackendMemory remain fully available on every platform.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Backend

type Backend string

Backend identifies which vector store implementation to use.

const (
	BackendSQLite   Backend = "sqlite"
	BackendPgVector Backend = "pgvector"
	BackendQdrant   Backend = "qdrant"
	BackendChroma   Backend = "chroma"
	BackendMemory   Backend = "memory" // in-process, for tests and dev
	BackendHNSW     Backend = "hnsw"   // embedded HNSW, no CGO, no external service
)

type ChromaConfig

type ChromaConfig struct {
	// BaseURL is the root URL of the Chroma server, e.g. "http://localhost:8000".
	BaseURL string

	// APIKey is sent as "Authorization: Bearer <key>" when non-empty.
	APIKey string

	// Tenant and Database are used by the v2 API.
	// Defaults: "default_tenant" / "default_database".
	Tenant   string
	Database string
}

ChromaConfig holds connection parameters for the Chroma HTTP API.

type ChromaStore

type ChromaStore struct {
	// contains filtered or unexported fields
}

ChromaStore is a vector.Store backed by the Chroma HTTP API (v2).

Namespace mapping: each Seshat namespace becomes a Chroma collection. Collection IDs (UUIDs) are cached in-memory after first use.

Chroma stores the original Seshat key as the document id. Text and metadata are stored as Chroma document + metadata fields.

func NewChromaStore

func NewChromaStore(cfg ChromaConfig) *ChromaStore

NewChromaStore creates a ChromaStore. No network call is made at construction time.

func (*ChromaStore) DeleteKeys

func (s *ChromaStore) DeleteKeys(ctx context.Context, namespace string, keys []string) error

func (*ChromaStore) DeleteNamespace

func (s *ChromaStore) DeleteNamespace(ctx context.Context, namespace string) error

func (*ChromaStore) Get

func (s *ChromaStore) Get(ctx context.Context, namespace string, keys []string) ([]Record, error)

func (*ChromaStore) HasNamespace

func (s *ChromaStore) HasNamespace(ctx context.Context, namespace string) (bool, error)

func (*ChromaStore) Search

func (s *ChromaStore) Search(ctx context.Context, query Query) ([]SearchResult, error)

func (*ChromaStore) Upsert

func (s *ChromaStore) Upsert(ctx context.Context, records []Record) error

type Config

type Config struct {
	// Backend selects the implementation.
	Backend Backend

	// DB is used by the SQLite and pgvector backends.
	// For SQLite:   must be a SQLite DB (DriverSQLite).
	// For pgvector: must be a Postgres DB (DriverPostgres).
	DB *dbpkg.DB

	// Dim is the vector dimension required by pgvector and Qdrant when
	// creating a new collection/table. Defaults to 1536 (OpenAI ada-002).
	Dim int

	// PgVectorCreateExtension controls whether the pgvector extension is created
	// automatically at store initialization time.
	PgVectorCreateExtension *bool

	// PgVectorIndexMethod selects the ANN index type for pgvector. Supported:
	// "hnsw" and "ivfflat". Empty defaults to "hnsw".
	PgVectorIndexMethod string

	// PgVectorHNSWM tunes HNSW index creation when PgVectorIndexMethod=hnsw.
	PgVectorHNSWM int

	// PgVectorHNSWEfConstruction tunes HNSW ef_construction.
	PgVectorHNSWEfConstruction int

	// PgVectorIVFFlatLists tunes IVFFlat index creation when
	// PgVectorIndexMethod=ivfflat.
	PgVectorIVFFlatLists int

	// QdrantHost / QdrantPort / QdrantAPIKey for the Qdrant gRPC client.
	QdrantHost   string
	QdrantPort   int // defaults to 6334 (gRPC)
	QdrantAPIKey string

	// QdrantPrefix is prepended to every collection name (useful for
	// multi-tenant deployments sharing one Qdrant instance).
	QdrantPrefix string

	// ChromaURL is the base URL of the Chroma HTTP API (e.g. "http://localhost:8000").
	ChromaURL string

	// ChromaAPIKey is sent as "Authorization: Bearer <key>" when non-empty.
	ChromaAPIKey string

	// ChromaTenant and ChromaDatabase are used with the Chroma v2 API.
	// Defaults to "default_tenant" / "default_database".
	ChromaTenant   string
	ChromaDatabase string

	// HNSWDir is the directory where HNSW index files are stored.
	// Each namespace gets its own pair of files (<slug>.hnsw + <slug>.meta.json).
	// Defaults to <runtime_root>/data/hnsw when not set.
	HNSWDir string
}

Config describes how to open a vector store. Not all fields are used by all backends — see field comments.

type HNSWStore

type HNSWStore struct {
	// contains filtered or unexported fields
}

HNSWStore is an embedded, persistent vector store backed by HNSW graphs.

Each namespace gets two files in dir:

  • <slug>.hnsw — binary HNSW index (vectors + graph topology)
  • <slug>.meta.json — text and metadata per key

Search is O(log n) via HNSW. No external service or CGO required. Designed as the default RAG backend for the Seshat CLI.

func NewHNSWStore

func NewHNSWStore(dir string) (*HNSWStore, error)

NewHNSWStore creates an HNSWStore that persists files in dir. The directory is created if it does not exist.

func (*HNSWStore) Close

func (s *HNSWStore) Close() error

Close releases in-memory graphs. All data is already persisted to disk.

func (*HNSWStore) DeleteKeys

func (s *HNSWStore) DeleteKeys(ctx context.Context, namespace string, keys []string) error

DeleteKeys removes specific records within a namespace.

func (*HNSWStore) DeleteNamespace

func (s *HNSWStore) DeleteNamespace(ctx context.Context, namespace string) error

DeleteNamespace removes all records for a namespace and its index files.

func (*HNSWStore) Get

func (s *HNSWStore) Get(ctx context.Context, namespace string, keys []string) ([]Record, error)

Get retrieves records by key (without their vectors — use Search for ANN retrieval). If keys is nil or empty, all records in the namespace are returned.

func (*HNSWStore) HasNamespace

func (s *HNSWStore) HasNamespace(ctx context.Context, namespace string) (bool, error)

HasNamespace reports whether the namespace contains at least one record.

func (*HNSWStore) Search

func (s *HNSWStore) Search(ctx context.Context, query Query) ([]SearchResult, error)

Search performs HNSW ANN search (O(log n)) over the namespace. When query.HybridWeight > 0 and query.QueryText is set, keyword scores are blended with vector scores using linear interpolation.

func (*HNSWStore) Upsert

func (s *HNSWStore) Upsert(ctx context.Context, records []Record) error

Upsert inserts or replaces records. Saves index and metadata atomically after each namespace batch.

type MemoryStore

type MemoryStore struct {
	// contains filtered or unexported fields
}

func NewMemoryStore

func NewMemoryStore() *MemoryStore

func (*MemoryStore) DeleteKeys

func (s *MemoryStore) DeleteKeys(_ context.Context, namespace string, keys []string) error

func (*MemoryStore) DeleteNamespace

func (s *MemoryStore) DeleteNamespace(_ context.Context, namespace string) error

func (*MemoryStore) Get

func (s *MemoryStore) Get(_ context.Context, namespace string, keys []string) ([]Record, error)

func (*MemoryStore) HasNamespace

func (s *MemoryStore) HasNamespace(_ context.Context, namespace string) (bool, error)

func (*MemoryStore) Search

func (s *MemoryStore) Search(_ context.Context, query Query) ([]SearchResult, error)

func (*MemoryStore) Upsert

func (s *MemoryStore) Upsert(_ context.Context, records []Record) error

type PgVectorOptions

type PgVectorOptions struct {
	Dim                int
	CreateExtension    bool
	IndexMethod        string
	HNSWM              int
	HNSWEfConstruction int
	IVFFlatLists       int
}

type PgVectorStore

type PgVectorStore struct {
	// contains filtered or unexported fields
}

PgVectorStore is a persistent vector.Store backed by PostgreSQL + pgvector extension.

Schema: a single `vector_chunks` table shared by all namespaces. Each row: (collection_name TEXT, id TEXT, text TEXT, embedding VECTOR(dim), metadata JSONB) Primary key: (collection_name, id). An HNSW cosine index is created on `embedding` at initialization.

Vectors are passed as formatted strings (e.g. "[0.1,0.2,0.3]") and cast with ::vector, avoiding the need for any codec registration.

func NewPgVectorStore

func NewPgVectorStore(ctx context.Context, database *dbpkg.DB, options PgVectorOptions) (*PgVectorStore, error)

NewPgVectorStore opens a pgvector store using an already-open Postgres DB. It creates the vector extension and the vector_chunks table if they don't exist.

func (*PgVectorStore) DeleteKeys

func (s *PgVectorStore) DeleteKeys(ctx context.Context, namespace string, keys []string) error

func (*PgVectorStore) DeleteNamespace

func (s *PgVectorStore) DeleteNamespace(ctx context.Context, namespace string) error

func (*PgVectorStore) Get

func (s *PgVectorStore) Get(ctx context.Context, namespace string, keys []string) ([]Record, error)

func (*PgVectorStore) HasNamespace

func (s *PgVectorStore) HasNamespace(ctx context.Context, namespace string) (bool, error)

func (*PgVectorStore) Search

func (s *PgVectorStore) Search(ctx context.Context, query Query) ([]SearchResult, error)

func (*PgVectorStore) Upsert

func (s *PgVectorStore) Upsert(ctx context.Context, records []Record) error

type QdrantConfig

type QdrantConfig struct {
	Host       string
	Port       int
	APIKey     string
	CollPrefix string // prefix prepended to every collection name
	DefaultDim int    // vector dimension used when creating new collections
}

QdrantConfig holds connection parameters for the Qdrant gRPC client.

type QdrantStore

type QdrantStore struct {
	// contains filtered or unexported fields
}

QdrantStore is a vector.Store backed by Qdrant (gRPC).

Namespace mapping: each Seshat namespace becomes a Qdrant collection named "{prefix}{namespace}". Seshat string keys are converted to deterministic uint64 point IDs via SHA-256 (first 8 bytes), with the original key stored in the point payload under _seshat_id so it can be round-tripped.

Metadata is stored in the point payload as individual string fields (Qdrant does not support nested JSONB natively in gRPC filters).

func NewQdrantStore

func NewQdrantStore(_ context.Context, cfg QdrantConfig) (*QdrantStore, error)

NewQdrantStore dials Qdrant and returns a ready QdrantStore.

func (*QdrantStore) Close

func (s *QdrantStore) Close() error

Close releases the underlying gRPC connection.

func (*QdrantStore) DeleteKeys

func (s *QdrantStore) DeleteKeys(ctx context.Context, namespace string, keys []string) error

func (*QdrantStore) DeleteNamespace

func (s *QdrantStore) DeleteNamespace(ctx context.Context, namespace string) error

func (*QdrantStore) Get

func (s *QdrantStore) Get(ctx context.Context, namespace string, keys []string) ([]Record, error)

func (*QdrantStore) HasNamespace

func (s *QdrantStore) HasNamespace(ctx context.Context, namespace string) (bool, error)

func (*QdrantStore) Search

func (s *QdrantStore) Search(ctx context.Context, query Query) ([]SearchResult, error)

func (*QdrantStore) Upsert

func (s *QdrantStore) Upsert(ctx context.Context, records []Record) error

type Query

type Query struct {
	Namespace string
	Vector    []float32
	TopK      int
	Filter    map[string]any // optional; nil = no filter

	// HybridWeight blends BM25 keyword search with vector similarity.
	//   0   (default) → pure vector
	//   1             → pure BM25
	//   0 < w < 1    → linear blend: (1-w)*vector + w*bm25
	// Only the SQLite and pgvector backends implement BM25; others ignore it.
	HybridWeight float32

	// QueryText is the raw query string used for BM25 scoring.
	// Must be non-empty when HybridWeight > 0.
	QueryText string
}

Query describes a vector similarity search. Filter applies optional metadata predicates server-side (where supported) or client-side (SQLite / in-memory backends).

Simple equality: {"source_file": "doc.txt"} IN operator: {"source_file": {"$in": ["a.txt", "b.txt"]}}

type Record

type Record struct {
	Namespace string            `json:"namespace"`
	Key       string            `json:"key"`
	Text      string            `json:"text,omitempty"`
	Vector    []float32         `json:"vector"`
	Metadata  map[string]string `json:"metadata,omitempty"`
}

Record is one vectorized chunk stored in a vector backend.

type SQLiteStore

type SQLiteStore struct {
	// contains filtered or unexported fields
}

SQLiteStore is a persistent vector.Store backed by SQLite.

Vectors are stored as raw IEEE-754 little-endian float32 BLOBs for compact storage and fast decode. Cosine similarity search runs in Go after loading the namespace's vectors from SQLite — adequate for typical RAG corpora (thousands of chunks). The table schema is registered as migration 005_rag_vector_store in internal/db/schema.go.

func NewSQLiteStore

func NewSQLiteStore(database *dbpkg.DB) (*SQLiteStore, error)

NewSQLiteStore wraps an already-open, already-migrated DB handle.

func OpenSQLiteStore

func OpenSQLiteStore(path string) (*SQLiteStore, error)

OpenSQLiteStore opens (or creates) a SQLite file at path and initializes the schema, returning a ready-to-use SQLiteStore.

func (*SQLiteStore) Close

func (s *SQLiteStore) Close() error

Close releases the underlying database connection.

func (*SQLiteStore) CountNamespace

func (s *SQLiteStore) CountNamespace(ctx context.Context, namespace string) (int, error)

CountNamespace returns the number of records stored under a namespace. Not part of Store interface — useful for tests and diagnostics.

func (*SQLiteStore) DeleteKeys

func (s *SQLiteStore) DeleteKeys(ctx context.Context, namespace string, keys []string) error

DeleteKeys removes specific keys within a namespace.

func (*SQLiteStore) DeleteNamespace

func (s *SQLiteStore) DeleteNamespace(ctx context.Context, namespace string) error

DeleteNamespace removes all records for a namespace.

func (*SQLiteStore) Get

func (s *SQLiteStore) Get(ctx context.Context, namespace string, keys []string) ([]Record, error)

Get retrieves records by key. If keys is nil/empty, all records in the namespace are returned (without their vector blobs decoded, which would be expensive).

func (*SQLiteStore) HasNamespace

func (s *SQLiteStore) HasNamespace(ctx context.Context, namespace string) (bool, error)

HasNamespace reports whether at least one record exists in the namespace.

func (*SQLiteStore) Search

func (s *SQLiteStore) Search(ctx context.Context, query Query) ([]SearchResult, error)

Search performs cosine similarity search over the given namespace. When query.HybridWeight > 0 and query.QueryText is set, BM25 scores from the FTS5 index are blended with vector scores using linear interpolation.

func (*SQLiteStore) Upsert

func (s *SQLiteStore) Upsert(ctx context.Context, records []Record) error

Upsert inserts or replaces records in the vector_records table.

type SearchResult

type SearchResult struct {
	Record Record  `json:"record"`
	Score  float32 `json:"score"`
}

SearchResult is one ranked vector search hit.

type Store

type Store interface {
	// Upsert inserts or replaces records (namespace + key = primary key).
	Upsert(ctx context.Context, records []Record) error

	// Search performs vector similarity search within a namespace.
	// Query.Filter is applied as an AND predicate on record metadata.
	Search(ctx context.Context, query Query) ([]SearchResult, error)

	// Get retrieves records by key. If keys is nil or empty, all records in
	// the namespace are returned.
	Get(ctx context.Context, namespace string, keys []string) ([]Record, error)

	// HasNamespace reports whether the namespace contains at least one record.
	HasNamespace(ctx context.Context, namespace string) (bool, error)

	// DeleteNamespace removes all records for a namespace.
	DeleteNamespace(ctx context.Context, namespace string) error

	// DeleteKeys removes specific records within a namespace.
	DeleteKeys(ctx context.Context, namespace string, keys []string) error
}

Store abstracts the vector backend used by RAG. Concrete implementations target SQLite, pgvector, Qdrant, Chroma, or any future provider — the rag.Service and knowledge.Service call only this interface.

func NewStore

func NewStore(ctx context.Context, cfg Config) (Store, error)

NewStore creates and returns a ready-to-use Store from cfg. The caller is responsible for closing any resources (e.g. Qdrant gRPC connection).

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL