core

package
v2.111.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 15, 2026 License: MIT Imports: 28 Imported by: 2

Documentation

Overview

Package core provides advanced search capabilities

Package core provides the core storage and retrieval engine for cortexdb.

It implements vector storage using SQLite as the primary backend, supported by specialized in-memory indexes (HNSW, IVF) for high-performance approximate nearest neighbor (ANN) search.

Key Components

  • SQLiteStore: The main entry point for data operations, managing both persistent SQL data and memory indexes.
  • Store Interface: Defines the standard operations for vector storage, document management, and chat memory.
  • DimensionAdapter: Automatically handles vector dimension mismatches based on configurable policies.
  • Hybrid Search: Combines HNSW/IVF vector search with SQLite FTS5 keyword search.
  • Metadata Filtering: Efficiently filters results using JSON-extract SQL push-down.
  • ACL: Provides row-level security for multi-user/department environments.

Observability

Since v2.0.0, the core engine supports pluggable structured logging through the Logger interface.

Package core provides faceted search capabilities

Package core provides multi-vector entity support

Index

Constants

View Source 
const PostgresLexicalExtension = "pg_trgm"

PostgresLexicalExtension is what the trigram index below needs installed.

It is named separately rather than returned as the first statement of PostgresLexicalDDL because creating an extension is not the same kind of operation as creating an index: CREATE EXTENSION IF NOT EXISTS loses a race against a concurrent creation, so it has to go through sqldialect.EnsureExtension, which checks the catalogue instead of trusting the guard. Run as one statement in a list whose loop broke on first error, a lost race silently cost both indexes below.

View Source 
const TrigramFloor = 3

TrigramFloor is the shortest run the trigram tokenizer can make a token from.

Variables

View Source 
var (
	// ErrInvalidDimension is returned when vector dimension doesn't match expected
	ErrInvalidDimension = errors.New("invalid vector dimension")

	// ErrNotFound is returned when an embedding is not found
	ErrNotFound = errors.New("embedding not found")

	// ErrInvalidVector is returned when vector data is invalid
	ErrInvalidVector = errors.New("invalid vector data")

	// ErrStoreClosed is returned when trying to use a closed store
	ErrStoreClosed = errors.New("store is closed")

	// ErrInvalidConfig is returned when configuration is invalid
	ErrInvalidConfig = errors.New("invalid configuration")

	// ErrEmptyQuery is returned when search query is empty
	ErrEmptyQuery = errors.New("empty query vector")
)

Common errors

View Source 
var (
	// CosineSimilarity calculates cosine similarity between two vectors
	CosineSimilarity = cosineSimilarity

	// DotProduct calculates dot product between two vectors
	DotProduct = dotProduct

	// EuclideanDist calculates negative Euclidean distance (higher = more similar)
	EuclideanDist = euclideanDistance
)

Predefined similarity functions for backward compatibility

View Source 
var ErrPostgresStoreUnimplemented = fmt.Errorf("cortexdb: not implemented by the PostgreSQL store")

ErrPostgresStoreUnimplemented is returned by the parts of Store this backend does not cover yet. It always names the method, so a log line is enough to know what to reach for.

Functions

func BelowTrigramFloor added in v2.60.0 

func BelowTrigramFloor(query string) bool

BelowTrigramFloor reports whether a CJK query is too short for the trigram index to match it.

Two characters is a whole word in Chinese, and a great many of the words a lesson is about are exactly two: 乘法, 除法, 分数, 面积, 周长. MATCH against a trigram index returns nothing for all of them, so a caller who does not notice reports "no results" for a term the corpus is full of. Falling back to a substring scan is slower than an index, but it is the difference between an answer and a silent zero.

func BuildSQLFromFilter 

func BuildSQLFromFilter(filter *FilterExpression, paramIndex *int) (string, []interface{})

BuildSQLFromFilter converts FilterExpression to SQL WHERE clause

func CJKAwareIndex added in v2.58.0 

func CJKAwareIndex(wordIndex, query string) string

CJKAwareIndex returns the FTS index to MATCH against for this query: the trigram companion for CJK text, otherwise the unicode61 word index.

Note the trigram floor: a query of one or two CJK characters produces no trigrams and therefore matches nothing. Callers that may be handed such a query should route it past MATCH entirely — see BelowTrigramFloor.

func ContainsCJK added in v2.58.0 

func ContainsCJK(text string) bool

ContainsCJK reports whether the text holds any Han, Hiragana, Katakana or Hangul character — the scripts whose text is not space-delimited.

func IsStaleTypeCache added in v2.82.1 

func IsStaleTypeCache(err error) bool

IsStaleTypeCache reports whether err is a connection whose cached statement refers to a type the catalog no longer has under that OID.

Matched on SQLSTATE *and* message text: XX000 is PostgreSQL's catch-all internal error and 0A000 covers plenty of unrelated unsupported features, so either code alone would swallow errors that deserve to surface.

func MatchExpression added in v2.68.0 

func MatchExpression(query string) string

MatchExpression makes a query safe to hand to FTS5 MATCH, rewriting it only when the raw text would be read as syntax rather than as words.

FTS5's MATCH argument is a query language, not a string: ":" filters a column, "*" is a prefix, parentheses group, and a stray "-" or quote is a parse error. A query typed by a person — or generated by a model — is none of those things, so ordinary technical vocabulary becomes a syntax error: "on-drbd-demote-failure" fails with `no such column: drbd`, and so does every other hyphenated identifier — al-extents, peer-disk, no-quorum, drbd-reactor. In a hybrid search the failure is silent, because the vector arm still answers; the keyword arm simply stops contributing, precisely for the exact technical terms it is best at.

A query of plain words is returned UNCHANGED. That is the important half: those queries already work, FTS5's own operators (AND, OR, NOT, NEAR) still mean what they meant, and the retrieval evals that pin lexical quality keep measuring the same thing. Rewriting them "harmlessly" is not harmless — quoting every token cost recall@10 0.85 → 0.27 on cortexdb-retrieval-v2.

Only when the query carries characters FTS5 would interpret is it rebuilt from its indexable runs, each quoted so nothing can be read as an operator. The result is empty when there is nothing indexable left; callers must skip the keyword arm rather than pass "" to MATCH, which is a syntax error of its own.

func MergeStreamResults 

func MergeStreamResults(ctx context.Context, channels ...<-chan StreamingResult) <-chan StreamingResult

MergeStreamResults merges multiple streaming result channels into one

func PgVectorLiteral added in v2.82.0 

func PgVectorLiteral(vec []float32) string

PgVectorLiteral renders a vector the way pgvector parses it: [1,2,3]. Always passed as a bound parameter, so it is a value and never SQL.

func PostgresLexicalCondition added in v2.82.0 

func PostgresLexicalCondition(column, query string) (sql string, args []any, indexed bool)

PostgresLexicalCondition builds the WHERE fragment for one query, with `?` placeholders for the dialect to rebind, and the values to bind.

indexed reports whether an index can serve it. False is not a failure — the row still gets found — but it is linear in table size, and a caller that wants to log or refuse that needs to be told rather than left to infer it from a slow query log.

The CJK arm matches TERMS, not the whole string. That was wrong for a while and wrong in a way no unit test saw: every test passed a single word, while the real caller passes a whole question. `LIKE '%pkg/agentmem 那个 bug 是 什么？%'` matches nothing however much of the corpus is about it, so search returned an empty result and the model above it correctly answered "the material does not say" — a wrong answer that looks like an honest one. FTS5's MATCH tokenises, so this has to as well, or the two backends are answering different questions.

func PostgresLexicalDDL added in v2.82.0 

func PostgresLexicalDDL(table, column string) []string

PostgresLexicalDDL returns the index statements that make lexical search on a column fast, in the order they must run.

Create PostgresLexicalExtension first. pg_trgm is a contrib extension: present in the standard images, but a managed instance may refuse CREATE EXTENSION to this account. The caller should treat a failure as "unindexed, still correct" rather than fatal — every query below works without any of these, just linearly.

func PostgresLexicalRank added in v2.82.0 

func PostgresLexicalRank(column, query string) (expr string, args []any)

PostgresLexicalRank is the ORDER BY expression that goes with PostgresLexicalCondition, negated so that lower is better — the convention bm25 already sets on the FTS5 side and what every caller's ORDER BY expects.

It has to be built from the same parse as the condition, and it has to bind exactly as many values. Ordering by id was worse than no ranking at all: it HAD one, an arbitrary one, so the wrong chunk came first and the model above answered from it — which reads as a retrieval that found nothing useful rather than as a sort order nobody chose.

func RegisterStore added in v2.82.0 

func RegisterStore(name string, factory StoreFactory) error

RegisterStore adds a backend under a name.

Refuses to replace one that is already registered: a silent replacement would mean the backend a process uses depends on package initialisation order, which is not something anybody wants to debug.

func RegisteredStores added in v2.82.0 

func RegisteredStores() []string

RegisteredStores lists the backends this binary was built with, sorted.

func SubstringPattern added in v2.60.0 

func SubstringPattern(literal string) string

SubstringPattern turns a literal into a LIKE pattern matching it anywhere, escaping the wildcards LIKE would otherwise read as syntax. Use it with ESCAPE '\'.

Types

type AdaptPolicy 

type AdaptPolicy int

AdaptPolicy defines how to handle vector dimension mismatches 

const (
	StrictMode   AdaptPolicy = iota // Error on dimension mismatch (default)
	SmartAdapt                      // Intelligent adaptation based on data distribution
	AutoTruncate                    // Always truncate to smaller dimension
	AutoPad                         // Always pad to larger dimension
	WarnOnly                        // Only warn, don't auto-adapt
)

type AdvancedSearchOptions 

type AdvancedSearchOptions struct {
	SearchOptions
	PreFilter     *FilterExpression // Applied before vector search
	PostFilter    *FilterExpression // Applied after vector search
	ArraySupport  bool              // Enable array field filtering
	NumericRanges bool              // Enable numeric range optimization
}

AdvancedSearchOptions extends SearchOptions with advanced filtering

type AggregationMethod 

type AggregationMethod string

AggregationMethod for combining multi-vector scores 

const (
	AggregateMax      AggregationMethod = "max"
	AggregateMin      AggregationMethod = "min"
	AggregateAverage  AggregationMethod = "average"
	AggregateSum      AggregationMethod = "sum"
	AggregateWeighted AggregationMethod = "weighted"
)

type AggregationRequest 

type AggregationRequest struct {
	Type       AggregationType        `json:"type"`
	Field      string                 `json:"field"`      // Metadata field to aggregate
	GroupBy    []string               `json:"group_by"`   // Fields to group by
	Filters    map[string]interface{} `json:"filters"`    // Optional filters
	Collection string                 `json:"collection"` // Optional collection filter
	Having     map[string]interface{} `json:"having"`     // Post-aggregation filters
	OrderBy    string                 `json:"order_by"`   // Field to order results by
	Limit      int                    `json:"limit"`      // Max results
}

AggregationRequest defines parameters for aggregation queries

type AggregationResponse 

type AggregationResponse struct {
	Request AggregationRequest  `json:"request"`
	Results []AggregationResult `json:"results"`
	Total   int                 `json:"total"`
}

AggregationResponse contains the aggregation results

type AggregationResult 

type AggregationResult struct {
	GroupKeys map[string]interface{} `json:"group_keys"` // Group by field values
	Value     interface{}            `json:"value"`      // Aggregated value
	Count     int                    `json:"count"`      // Number of items in group
}

AggregationResult represents a single aggregation result

type AggregationType 

type AggregationType string

AggregationType defines the type of aggregation 

const (
	AggregationCount   AggregationType = "count"
	AggregationSum     AggregationType = "sum"
	AggregationAvg     AggregationType = "avg"
	AggregationMin     AggregationType = "min"
	AggregationMax     AggregationType = "max"
	AggregationGroupBy AggregationType = "group_by"
)

type AutoSaveConfig added in v2.8.0 

type AutoSaveConfig struct {
	Enabled     bool          `json:"enabled"`     // Enable auto-save (default: true)
	Interval    time.Duration `json:"interval"`    // Save interval (default: 5 minutes)
	SaveOnClose bool          `json:"saveOnClose"` // Save on database close (default: true)
	MinChanges  int           `json:"minChanges"`  // Minimum changes before saving (default: 100)
}

AutoSaveConfig defines configuration for automatic index snapshot saving

func DefaultAutoSaveConfig added in v2.8.0 

func DefaultAutoSaveConfig() AutoSaveConfig

DefaultAutoSaveConfig returns default auto-save configuration

type Backupper added in v2.91.0 

type Backupper interface {
	// Backup writes a consistent copy of the whole store to path. It must not
	// require quiescing writers: a backup that needs the server stopped is a
	// different operational thing, and callers of this one are not stopping it.
	Backup(ctx context.Context, path string) error
}

Backupper is a store that can copy itself. It is deliberately not part of BrainStore.

BrainStore is the set of methods DB must have to be a brain at all: without UpdateDocument or GetDB there is nothing to run. Backup is not like that. A brain on PostgreSQL is perfectly complete without it, because backing up PostgreSQL is pg_dump or a base backup — a job for the operations team and its retention policy, not something this process should imitate by writing a file next to a database it does not own. Folding Backup into BrainStore would make every backend either implement that pretence or carry a permanent "not supported" stub, and the compile error that greets the next backend would be asking it for a capability it may have no business having.

So it is optional, and callers type-assert. The cost is that the failure moves from compile time to run time; the payment for that is an error that names the backend, so an operator reading it learns which one it is and where its backups actually come from.

type BrainStore added in v2.82.0 

type BrainStore interface {
	Store

	// GetSimilarityFunc is how this store compares two vectors. The graph
	// layer scores its own candidates and must score them the same way, or a
	// hybrid result would rank differently from a plain search over the same
	// rows.
	GetSimilarityFunc() SimilarityFunc

	// Config reports how this store was built — vector width above all, which
	// callers compare against an embedder's output.
	Config() Config

	// GetDB hands out the raw handle. Sibling packages (agentmem, graphflow,
	// the graph itself) keep their own tables in the same database and manage
	// them directly; this is how they reach it. Callers must not close it.
	GetDB() *sql.DB

	// UpdateDocument replaces a document in place. Store has Create, Get and
	// Delete but not this one; DB needs it, which is the whole reason this
	// interface exists rather than Store being widened.
	UpdateDocument(ctx context.Context, doc *Document) error

	// GetByID and GetByDocID read embeddings back out rather than searching
	// for them — an id is not a query.
	GetByID(ctx context.Context, id string) (*Embedding, error)
	GetByDocID(ctx context.Context, docID string) ([]*Embedding, error)

	// UpsertBatchTx joins a transaction the caller already opened, so a write
	// that spans the vector store and a sibling package's tables is one
	// transaction rather than two that can half-fail.
	UpsertBatchTx(ctx context.Context, tx *sql.Tx, embs []*Embedding) error

	// UpsertBatchWithAdapt writes vectors that may not match the store's
	// dimension, adapting them to it. Separate from UpsertBatch because
	// silently reshaping a vector is a decision, not a detail.
	UpsertBatchWithAdapt(ctx context.Context, embs []*Embedding) error

	// SearchChatHistoryScored is SearchChatHistory with the scores kept.
	SearchChatHistoryScored(ctx context.Context, queryVec []float32, sessionID string, limit int) ([]ScoredMessage, error)

	// The dimension bookkeeping. An embedder swap leaves rows of the old width
	// behind, and a store that cannot find them cannot be repaired — which is
	// how a brain ends up silently unable to search part of itself.
	DimensionReport(ctx context.Context) (*DimensionReport, error)
	MismatchedEmbeddings(ctx context.Context, wantDim, limit int) ([]*Embedding, error)
	ReconcileCollectionDimensions(ctx context.Context, dim int) (int, error)

	// SyncDeletedEmbeddingIDs and SyncUpsertedEmbeddings keep an in-process
	// index in step with rows written by another path. They take no error
	// because an index that has drifted is a performance problem, not a
	// correctness one: the rows are already right.
	SyncDeletedEmbeddingIDs(ctx context.Context, ids []string)
	SyncUpsertedEmbeddings(ctx context.Context, embs []*Embedding)
}

BrainStore is Store plus everything else cortexdb.DB reaches for.

func OpenBrainStore added in v2.82.0 

func OpenBrainStore(dsn string, config Config) (BrainStore, error)

OpenBrainStore opens a store that can back a brain.

Same DSNs as OpenStore, one extra requirement: the backend has to implement everything cortexdb.DB reaches for, not only the vector contract. A backend that opens but cannot be a brain is refused here, by name, rather than panicking on a type assertion somewhere in the middle of a request.

type Collection 

type Collection struct {
	ID          int                    `json:"id"`
	Name        string                 `json:"name"`
	Dimensions  int                    `json:"dimensions"`
	Description string                 `json:"description,omitempty"`
	Metadata    map[string]interface{} `json:"metadata,omitempty"`
	CreatedAt   time.Time              `json:"created_at"`
	UpdatedAt   time.Time              `json:"updated_at"`
}

Collection represents a logical grouping of embeddings

type CollectionDimensions added in v2.59.0 

type CollectionDimensions struct {
	Collection string           `json:"collection"`
	Declared   int              `json:"declared"`   // dimension the collection was created with
	Rows       int              `json:"rows"`       // rows holding a vector
	Dimensions []DimensionCount `json:"dimensions"` // stored sizes, smallest first
	Mismatched int              `json:"mismatched"` // rows whose dimension is not Declared
}

CollectionDimensions describes the vector sizes actually stored in one collection.

func (CollectionDimensions) RowsWithDim added in v2.59.0 

func (c CollectionDimensions) RowsWithDim(dim int) int

RowsWithDim reports how many rows hold vectors of exactly dim.

type CollectionStats 

type CollectionStats struct {
	Name           string    `json:"name"`
	Count          int64     `json:"count"`
	Dimensions     int       `json:"dimensions"`
	Size           int64     `json:"size"`
	CreatedAt      time.Time `json:"created_at"`
	LastInsertedAt time.Time `json:"last_inserted_at,omitempty"`
}

CollectionStats represents statistics for a collection

type Config 

type Config struct {
	Path           string               `json:"path"`                     // Database file path
	VectorDim      int                  `json:"vectorDim"`                // Expected vector dimension, 0 = auto-detect
	AutoDimAdapt   AdaptPolicy          `json:"autoDimAdapt"`             // How to handle dimension mismatches
	SimilarityFn   SimilarityFunc       `json:"-"`                        // Similarity function
	IndexType      IndexType            `json:"indexType"`                // Index type to use
	HNSW           HNSWConfig           `json:"hnsw,omitempty"`           // HNSW index configuration
	IVF            IVFConfig            `json:"ivf,omitempty"`            // IVF index configuration
	TextSimilarity TextSimilarityConfig `json:"textSimilarity,omitempty"` // Text similarity configuration
	Quantization   QuantizationConfig   `json:"quantization,omitempty"`   // Quantization configuration
	Logger         Logger               `json:"-"`                        // Logger instance (defaults to nop logger)
	AutoSave       AutoSaveConfig       `json:"autoSave,omitempty"`       // Auto-save configuration
}

Config represents configuration options for the vector store

func DefaultConfig 

func DefaultConfig() Config

DefaultConfig returns a default configuration

type CustomReranker 

type CustomReranker struct {
	// ScoreFunc computes a custom score for each result
	ScoreFunc func(ctx context.Context, query string, result ScoredEmbedding) float64
}

CustomReranker allows users to provide a custom scoring function

func NewCustomReranker 

func NewCustomReranker(scoreFunc func(ctx context.Context, query string, result ScoredEmbedding) float64) *CustomReranker

NewCustomReranker creates a new custom reranker

func (*CustomReranker) Rerank 

func (r *CustomReranker) Rerank(ctx context.Context, query string, results []ScoredEmbedding) ([]ScoredEmbedding, error)

Rerank applies custom scoring function

type DimensionAdapter 

type DimensionAdapter struct {
	// contains filtered or unexported fields
}

DimensionAdapter handles vector dimension adaptation

func NewDimensionAdapter 

func NewDimensionAdapter(policy AdaptPolicy) *DimensionAdapter

NewDimensionAdapter creates a new dimension adapter with the given policy

func (*DimensionAdapter) AdaptVector 

func (da *DimensionAdapter) AdaptVector(vector []float32, sourceDim, targetDim int) ([]float32, error)

AdaptVector adapts a vector from source dimension to target dimension

type DimensionAnalysis 

type DimensionAnalysis struct {
	PrimaryDim     int         `json:"primaryDim"`     // Most common dimension
	PrimaryCount   int         `json:"primaryCount"`   // Count of primary dimension
	Dimensions     map[int]int `json:"dimensions"`     // Map of dimension -> count
	TotalVectors   int         `json:"totalVectors"`   // Total number of vectors
	NeedsMigration bool        `json:"needsMigration"` // Whether migration is recommended
}

DimensionAnalysis contains information about vector dimensions in the store

func AnalyzeDimensions 

func AnalyzeDimensions(vectors [][]float32) *DimensionAnalysis

AnalyzeDimensions analyzes dimension distribution in the given vectors

type DimensionCount added in v2.59.0 

type DimensionCount struct {
	Dim  int `json:"dim"`
	Rows int `json:"rows"`
}

DimensionCount is how many rows hold vectors of one size.

type DimensionReport added in v2.59.0 

type DimensionReport struct {
	Collections []CollectionDimensions `json:"collections"`
	Mismatched  int                    `json:"mismatched"` // total rows needing repair
}

DimensionReport summarises vector dimensionality across the store.

func (*DimensionReport) NeedsRepair added in v2.59.0 

func (r *DimensionReport) NeedsRepair() bool

NeedsRepair reports whether any stored vector disagrees with its collection.

type DiversityMethod 

type DiversityMethod string

DiversityMethod for result diversification 

const (
	// Maximal Marginal Relevance
	DiversityMMR DiversityMethod = "mmr"

	// Determinantal Point Process
	DiversityDPP DiversityMethod = "dpp"

	// Simple distance-based
	DiversityDistance DiversityMethod = "distance"

	// Random sampling
	DiversityRandom DiversityMethod = "random"
)

type DiversityReranker 

type DiversityReranker struct {
	// Lambda controls the diversity vs relevance trade-off
	// 0.0 = maximum diversity, 1.0 = maximum relevance
	Lambda float64
	// SimilarityFunc computes similarity between two embeddings
	SimilarityFunc SimilarityFunc
}

DiversityReranker promotes diverse results using Maximal Marginal Relevance (MMR)

func NewDiversityReranker 

func NewDiversityReranker(lambda float64, simFunc SimilarityFunc) *DiversityReranker

NewDiversityReranker creates a new diversity-based reranker

func (*DiversityReranker) Rerank 

func (r *DiversityReranker) Rerank(ctx context.Context, query string, results []ScoredEmbedding) ([]ScoredEmbedding, error)

Rerank applies MMR to promote diverse results

type DiversitySearchOptions 

type DiversitySearchOptions struct {
	// Lambda parameter for MMR (0 = max diversity, 1 = max relevance)
	Lambda float32

	// Diversity method
	Method DiversityMethod

	// Minimum distance between results
	MinDistance float32

	// Base search options
	SearchOptions
}

DiversitySearchOptions for diverse result sampling

type Document 

type Document struct {
	ID        string                 `json:"id"`
	Title     string                 `json:"title"`
	SourceURL string                 `json:"source_url,omitempty"`
	Content   string                 `json:"content,omitempty"` // Full document content
	Version   int                    `json:"version"`
	Author    string                 `json:"author,omitempty"`
	Metadata  map[string]interface{} `json:"metadata,omitempty"`
	ACL       []string               `json:"acl,omitempty"` // Allowed user IDs or groups
	CreatedAt time.Time              `json:"created_at"`
	UpdatedAt time.Time              `json:"updated_at"`
}

Document represents a high-level document containing multiple embeddings (chunks)

type DocumentInfo 

type DocumentInfo struct {
	DocID          string  `json:"docId"`
	EmbeddingCount int     `json:"embeddingCount"`
	FirstCreated   *string `json:"firstCreated,omitempty"`
	LastUpdated    *string `json:"lastUpdated,omitempty"`
}

DocumentInfo provides information about a document in the store

type DumpFormat 

type DumpFormat string

DumpFormat represents the format for data export 

const (
	// DumpFormatJSON exports data as JSON
	DumpFormatJSON DumpFormat = "json"
	// DumpFormatJSONL exports data as JSON Lines (one JSON object per line)
	DumpFormatJSONL DumpFormat = "jsonl"
	// DumpFormatCSV exports data as CSV (vectors as base64 encoded strings)
	DumpFormatCSV DumpFormat = "csv"
)

type DumpOptions 

type DumpOptions struct {
	Format         DumpFormat      // Export format
	IncludeVectors bool            // Include vector data (can be large)
	IncludeIndex   bool            // Include index data (HNSW, IVF)
	Filter         *MetadataFilter // Optional filter for selective export
	BatchSize      int             // Batch size for export (default: 1000)
}

DumpOptions defines options for data export

func DefaultDumpOptions 

func DefaultDumpOptions() DumpOptions

DefaultDumpOptions returns default dump options

type DumpStats 

type DumpStats struct {
	TotalEmbeddings  int   `json:"total_embeddings"`
	TotalDocuments   int   `json:"total_documents"`
	TotalCollections int   `json:"total_collections"`
	BytesWritten     int64 `json:"bytes_written"`
}

DumpStats provides statistics about the export operation

type Embedding 

type Embedding struct {
	ID           string            `json:"id"`
	CollectionID int               `json:"collection_id,omitempty"`
	Collection   string            `json:"collection,omitempty"`
	Vector       []float32         `json:"vector"`
	Content      string            `json:"content"`
	DocID        string            `json:"docId,omitempty"`
	Metadata     map[string]string `json:"metadata,omitempty"`
	ACL          []string          `json:"acl,omitempty"` // Allowed user IDs or groups
}

Embedding represents a vector embedding with associated metadata

type ExportMetadata 

type ExportMetadata struct {
	Version    string `json:"version"`
	Dimensions int    `json:"dimensions"`
	Count      int    `json:"count"`
	ExportedAt string `json:"exported_at"`
	Config     Config `json:"config"`
}

ExportMetadata contains metadata about the export

type FTS5Query added in v2.82.0 

type FTS5Query struct {
	// Terms are the words and phrases to match, quotes and prefix stars gone.
	Terms []string
	// Any is true when the expression joined its terms with OR.
	Any bool
}

FTS5Query is an FTS5 MATCH expression taken apart far enough that a database without FTS5 can answer it.

func ParseFTS5 added in v2.82.0 

func ParseFTS5(query string) FTS5Query

ParseFTS5 reads an FTS5 MATCH expression as terms plus a connective.

It exists because the query the retrieval layer builds is FTS5 *syntax*, not text: sanitizeFTSQuery double-quotes every token, and the keyword expansion emits `owner OR name` and `owner* OR name*`. SQLite reads those as a query. PostgreSQL was handed the same string as if a person had typed it, so the quotes became part of the words, the star became part of the word, and OR became a word — a search for a document containing the English word "or".

The result is an approximation on purpose. NOT and NEAR are dropped rather than translated: they never appear in what this codebase generates, and silently mistranslating them would be worse than matching a little too much.

type FacetFilter 

type FacetFilter struct {
	// Type of filter
	Type FacetFilterType

	// Values for equality/inclusion filters
	Values []interface{}

	// Range for numeric filters
	Min interface{}
	Max interface{}

	// Pattern for text filters
	Pattern string

	// Nested filters for complex conditions
	Nested []FacetFilter

	// Logical operator for nested filters
	Operator LogicalOperator
}

FacetFilter defines filtering for a specific facet

type FacetFilterType 

type FacetFilterType string

FacetFilterType defines the type of facet filter 

const (
	FilterTypeEquals   FacetFilterType = "equals"
	FilterTypeIn       FacetFilterType = "in"
	FilterTypeRange    FacetFilterType = "range"
	FilterTypeContains FacetFilterType = "contains"
	FilterTypePrefix   FacetFilterType = "prefix"
	FilterTypeExists   FacetFilterType = "exists"
	FilterTypeNested   FacetFilterType = "nested"
)

type FacetResult 

type FacetResult struct {
	Field  string
	Values map[string]int
	Total  int
}

FacetResult contains facet counts

type FacetedSearchOptions 

type FacetedSearchOptions struct {
	SearchOptions

	// Facets to filter by
	Facets map[string]FacetFilter

	// Whether to return facet counts
	ReturnFacets bool

	// Maximum number of facet values to return
	MaxFacetValues int
}

FacetedSearchOptions extends SearchOptions with faceted filtering

type FilterExpression 

type FilterExpression struct {
	Operator FilterOperator
	Field    string
	Value    interface{}
	Children []*FilterExpression
}

FilterExpression represents a complex filter expression

func ParseFilterString 

func ParseFilterString(filterStr string) (*FilterExpression, error)

ParseFilterString parses a string filter expression into FilterExpression Example: "(tag:ai OR tag:ml) AND date>2024 AND price BETWEEN 100 AND 500"

type FilterOperator 

type FilterOperator string

FilterOperator represents logical operators for filters 

const (
	FilterAND     FilterOperator = "AND"
	FilterOR      FilterOperator = "OR"
	FilterNOT     FilterOperator = "NOT"
	FilterEQ      FilterOperator = "="
	FilterNE      FilterOperator = "!="
	FilterGT      FilterOperator = ">"
	FilterGTE     FilterOperator = ">="
	FilterLT      FilterOperator = "<"
	FilterLTE     FilterOperator = "<="
	FilterIN      FilterOperator = "IN"
	FilterBETWEEN FilterOperator = "BETWEEN"
	FilterLIKE    FilterOperator = "LIKE"
	FilterREGEX   FilterOperator = "REGEX"
)

type HNSWConfig 

type HNSWConfig struct {
	Enabled        bool `json:"enabled"`
	M              int  `json:"m"`              // Maximum connections per node (default: 16)
	EfConstruction int  `json:"efConstruction"` // Candidates during construction (default: 64)
	EfSearch       int  `json:"efSearch"`       // Candidates during search (default: 50)
	NumWorkers     int  `json:"numWorkers"`     // Number of parallel workers for index building (default: 4)
	Incremental    bool `json:"incremental"`    // Enable incremental indexing (default: true)
}

HNSWConfig represents configuration options for HNSW indexing

func DefaultHNSWConfig 

func DefaultHNSWConfig() HNSWConfig

DefaultHNSWConfig returns default HNSW configuration

type HybridReranker 

type HybridReranker struct {
	Rerankers []Reranker
	Weights   []float64
}

HybridReranker combines multiple rerankers with weighted scores

func NewHybridReranker 

func NewHybridReranker(rerankers []Reranker, weights []float64) *HybridReranker

NewHybridReranker creates a new hybrid reranker

func (*HybridReranker) Rerank 

func (r *HybridReranker) Rerank(ctx context.Context, query string, results []ScoredEmbedding) ([]ScoredEmbedding, error)

Rerank combines multiple rerankers

type HybridSearchOptions 

type HybridSearchOptions struct {
	SearchOptions
	// Fusion parameter for RRF (default 60)
	RRFK float64
}

HybridSearchOptions for combined vector + keyword search

type IVFConfig 

type IVFConfig struct {
	Enabled    bool `json:"enabled"`
	NCentroids int  `json:"nCentroids"` // Number of centroids (default: 100)
	NProbe     int  `json:"nProbe"`     // Number of clusters to search (default: 10)
}

IVFConfig represents configuration options for IVF indexing

func DefaultIVFConfig 

func DefaultIVFConfig() IVFConfig

DefaultIVFConfig returns default IVF configuration

type ImportStats 

type ImportStats struct {
	TotalEmbeddings int `json:"total_embeddings"`
	TotalDocuments  int `json:"total_documents"`
	FailedCount     int `json:"failed_count"`
	SkippedCount    int `json:"skipped_count"`
}

ImportStats provides statistics about the import operation

func (*ImportStats) String 

func (s *ImportStats) String() string

Helper to convert import stats to string

type IncrementalIndex 

type IncrementalIndex struct {
	// contains filtered or unexported fields
}

IncrementalIndex allows adding vectors while searching continues

func NewIncrementalIndex 

func NewIncrementalIndex(store *SQLiteStore) *IncrementalIndex

NewIncrementalIndex creates a new incremental index

func (*IncrementalIndex) AddAsync 

func (idx *IncrementalIndex) AddAsync(emb *Embedding) error

AddAsync adds a vector asynchronously

func (*IncrementalIndex) Close 

func (idx *IncrementalIndex) Close()

Close shuts down the incremental index

func (*IncrementalIndex) SearchWithUpdates 

func (idx *IncrementalIndex) SearchWithUpdates(ctx context.Context, query []float32, opts SearchOptions) ([]ScoredEmbedding, error)

SearchWithUpdates performs search while considering ongoing updates

type IndexType 

type IndexType int

IndexType defines the type of index to use 

const (
	IndexTypeHNSW IndexType = iota
	IndexTypeIVF
	IndexTypeFlat
)

type KeywordMatchReranker 

type KeywordMatchReranker struct {
	// Boost is the score multiplier for keyword matches
	Boost float64
	// CaseSensitive enables case-sensitive matching
	CaseSensitive bool
}

KeywordMatchReranker boosts results that contain the query keywords

func NewKeywordMatchReranker 

func NewKeywordMatchReranker(boost float64) *KeywordMatchReranker

NewKeywordMatchReranker creates a new keyword match reranker

func (*KeywordMatchReranker) Rerank 

func (r *KeywordMatchReranker) Rerank(ctx context.Context, query string, results []ScoredEmbedding) ([]ScoredEmbedding, error)

Rerank boosts results containing query keywords

type LoadOptions 

type LoadOptions struct {
	Format       DumpFormat // Import format
	SkipExisting bool       // Skip existing embeddings (by ID)
	Replace      bool       // Replace existing embeddings
	BatchSize    int        // Batch size for import (default: 100)
	Upsert       bool       // Use upsert instead of insert
}

LoadOptions defines options for data import

func DefaultLoadOptions 

func DefaultLoadOptions() LoadOptions

DefaultLoadOptions returns default load options

type LogLevel 

type LogLevel int

LogLevel represents the severity level of a log message 

const (
	// LevelDebug is for detailed debugging information
	LevelDebug LogLevel = iota
	// LevelInfo is for general informational messages
	LevelInfo
	// LevelWarn is for warning messages
	LevelWarn
	// LevelError is for error messages
	LevelError
)

func (LogLevel) String 

func (l LogLevel) String() string

String returns the string representation of the log level

type Logger 

type Logger interface {
	// Debug logs a debug message
	Debug(msg string, keyvals ...any)
	// Info logs an informational message
	Info(msg string, keyvals ...any)
	// Warn logs a warning message
	Warn(msg string, keyvals ...any)
	// Error logs an error message
	Error(msg string, keyvals ...any)
	// With returns a new logger with additional key-value pairs
	With(keyvals ...any) Logger
}

Logger is the interface for logging operations

func NewLogger 

func NewLogger(writer io.Writer, minLevel LogLevel) Logger

NewLogger creates a new logger that writes to the given writer

func NewStdLogger 

func NewStdLogger(minLevel LogLevel) Logger

NewStdLogger creates a new logger that writes to stdout

func NopLogger 

func NopLogger() Logger

NopLogger returns a logger that discards all messages

type LogicalOperator 

type LogicalOperator string

LogicalOperator for combining filters 

const (
	OperatorAND LogicalOperator = "AND"
	OperatorOR  LogicalOperator = "OR"
	OperatorNOT LogicalOperator = "NOT"
)

type Message 

type Message struct {
	ID        string                 `json:"id"`
	SessionID string                 `json:"session_id"`
	Role      string                 `json:"role"` // 'user', 'assistant', 'system'
	Content   string                 `json:"content"`
	Vector    []float32              `json:"vector,omitempty"` // Embedding for long-term memory
	Metadata  map[string]interface{} `json:"metadata,omitempty"`
	CreatedAt time.Time              `json:"created_at"`
}

Message represents a single message in a chat session

type MetadataFilter 

type MetadataFilter struct {
	// contains filtered or unexported fields
}

MetadataFilter helper for building filter expressions

func NewMetadataFilter 

func NewMetadataFilter() *MetadataFilter

NewMetadataFilter creates a new metadata filter builder

func (*MetadataFilter) And 

func (f *MetadataFilter) And(other *MetadataFilter) *MetadataFilter

And combines with another filter using AND

func (*MetadataFilter) Between 

func (f *MetadataFilter) Between(field string, min, max interface{}) *MetadataFilter

Between adds a BETWEEN condition

func (*MetadataFilter) Build 

func (f *MetadataFilter) Build() *FilterExpression

Build returns the underlying expression (for tests)

func (*MetadataFilter) BuildSQL 

func (f *MetadataFilter) BuildSQL() (string, []interface{})

BuildSQL returns the SQL WHERE clause for the filter

func (*MetadataFilter) Equal 

func (f *MetadataFilter) Equal(field string, value interface{}) *MetadataFilter

Equal adds an equality condition

func (*MetadataFilter) GreaterThan 

func (f *MetadataFilter) GreaterThan(field string, value interface{}) *MetadataFilter

GreaterThan adds a greater than condition

func (*MetadataFilter) GreaterThanOrEqual 

func (f *MetadataFilter) GreaterThanOrEqual(field string, value interface{}) *MetadataFilter

GreaterThanOrEqual adds a greater than or equal condition

func (*MetadataFilter) In 

func (f *MetadataFilter) In(field string, values ...interface{}) *MetadataFilter

In adds an IN condition

func (*MetadataFilter) IsEmpty 

func (f *MetadataFilter) IsEmpty() bool

IsEmpty checks if the filter is empty

func (*MetadataFilter) LessThan 

func (f *MetadataFilter) LessThan(field string, value interface{}) *MetadataFilter

LessThan adds a less than condition

func (*MetadataFilter) LessThanOrEqual 

func (f *MetadataFilter) LessThanOrEqual(field string, value interface{}) *MetadataFilter

LessThanOrEqual adds a less than or equal condition

func (*MetadataFilter) Like 

func (f *MetadataFilter) Like(field string, pattern string) *MetadataFilter

Like adds a LIKE condition

func (*MetadataFilter) NotEqual 

func (f *MetadataFilter) NotEqual(field string, value interface{}) *MetadataFilter

NotEqual adds a non-equality condition

func (*MetadataFilter) NotIn 

func (f *MetadataFilter) NotIn(field string, values ...interface{}) *MetadataFilter

NotIn adds a NOT IN condition

func (*MetadataFilter) Or 

func (f *MetadataFilter) Or(other *MetadataFilter) *MetadataFilter

Or combines with another filter using OR

func (*MetadataFilter) StringIn 

func (f *MetadataFilter) StringIn(field string, values ...string) *MetadataFilter

StringIn is alias for In for string values

func (*MetadataFilter) ToSQL 

func (f *MetadataFilter) ToSQL() (string, []interface{})

ToSQL is an alias for BuildSQL

type MultiVectorEntity 

type MultiVectorEntity struct {
	// Entity ID
	ID string

	// Vectors associated with this entity
	Vectors map[string][]float32

	// Metadata for the entity
	Metadata map[string]interface{}

	// Content/text for the entity
	Content string
}

MultiVectorEntity represents an entity with multiple vectors

type MultiVectorSearchOptions 

type MultiVectorSearchOptions struct {
	// Which vector fields to search
	VectorFields []string

	// Weights for each vector field
	FieldWeights map[string]float32

	// Aggregation method for combining scores
	Aggregation AggregationMethod

	// Standard search options
	SearchOptions
}

MultiVectorSearchOptions for multi-vector search

type NegativeSearchOptions 

type NegativeSearchOptions struct {
	// Positive examples (find similar to these)
	PositiveVectors [][]float32

	// Negative examples (avoid similar to these)
	NegativeVectors [][]float32

	// Weight for negative examples (higher = stronger avoidance)
	NegativeWeight float32

	// Base search options
	SearchOptions
}

NegativeSearchOptions for "not like this" queries

type PostgresStore added in v2.82.0 

type PostgresStore struct {
	// contains filtered or unexported fields
}

PostgresStore implements the vector core of Store on PostgreSQL + pgvector.

func NewPostgresStore added in v2.82.0 

func NewPostgresStore(db *sql.DB, config Config) *PostgresStore

NewPostgresStore wraps an open database. The caller owns the pool: a deployment usually has one and wants its own limits on it.

func (*PostgresStore) AddMessage added in v2.82.0 

func (s *PostgresStore) AddMessage(ctx context.Context, msg *Message) error

AddMessage adds a message to a session.

The timestamp is the caller's clock (time.Now().UTC()), not the database's, because that is what chat.go does and history ordering depends on it.

func (*PostgresStore) Aggregate added in v2.104.0 

func (s *PostgresStore) Aggregate(ctx context.Context, req AggregationRequest) (*AggregationResponse, error)

Aggregate performs aggregation queries on embeddings metadata.

func (*PostgresStore) BatchRangeSearch added in v2.104.0 

func (s *PostgresStore) BatchRangeSearch(ctx context.Context, queries [][]float32, radius float32, opts SearchOptions) ([][]ScoredEmbedding, error)

BatchRangeSearch performs range search for multiple queries.

Results stay grouped by input query and in input order — the index into the outer slice is the index of the query that produced it, which is the only thing tying a result back to its query. Same loop as the SQLite store, including that the first failing query fails the batch: a partial batch whose gaps are indistinguishable from empty result sets is worse than none.

func (*PostgresStore) Close added in v2.82.0 

func (s *PostgresStore) Close() error

func (*PostgresStore) Config added in v2.82.0 

func (s *PostgresStore) Config() Config

Config reports how this store was built. SQLiteStore takes its read lock here; this store keeps no state behind a lock, so the copy is the whole of it.

func (*PostgresStore) CreateCollection added in v2.82.0 

func (s *PostgresStore) CreateCollection(ctx context.Context, name string, dimensions int) (*Collection, error)

func (*PostgresStore) CreateDocument added in v2.82.0 

func (s *PostgresStore) CreateDocument(ctx context.Context, doc *Document) error

func (*PostgresStore) CreateSession added in v2.82.0 

func (s *PostgresStore) CreateSession(ctx context.Context, session *Session) error

CreateSession creates a new chat session.

created_at and updated_at come from the database, as they do on SQLite: the row's clock is the database's, not the caller's.

func (*PostgresStore) Delete added in v2.82.0 

func (s *PostgresStore) Delete(ctx context.Context, id string) error

func (*PostgresStore) DeleteBatch added in v2.82.0 

func (s *PostgresStore) DeleteBatch(ctx context.Context, ids []string) error

func (*PostgresStore) DeleteByDocID added in v2.82.0 

func (s *PostgresStore) DeleteByDocID(ctx context.Context, docID string) error

func (*PostgresStore) DeleteByFilter added in v2.82.0 

func (s *PostgresStore) DeleteByFilter(ctx context.Context, filter *MetadataFilter) error

DeleteByFilter deletes the rows a filter selects.

It waited for the real filter compiler rather than getting an approximate one of its own. A partial translation that quietly ignored a clause would delete rows the caller meant to keep, and unlike a wrong search result there is nothing to notice afterwards — data loss is not a degradation. So this shares pgFilterSQL with SearchWithAdvancedFilter: one translation, checked by that method's tests, and an operator it refuses to compile is refused here too rather than silently widening the delete.

func (*PostgresStore) DeleteCollection added in v2.82.0 

func (s *PostgresStore) DeleteCollection(ctx context.Context, name string) error

func (*PostgresStore) DeleteDocument added in v2.82.0 

func (s *PostgresStore) DeleteDocument(ctx context.Context, id string) error

DeleteDocument removes a document. Its embeddings go with it, by the foreign key — the same cascade SQLite declares.

func (*PostgresStore) DimensionReport added in v2.82.0 

func (s *PostgresStore) DimensionReport(ctx context.Context) (*DimensionReport, error)

DimensionReport groups stored vectors by collection and length, exactly as the SQLite store does: a collection whose declared dimension is 0 has never had one recorded, so its rows are counted but never called mismatched.

func (*PostgresStore) GetByDocID added in v2.82.0 

func (s *PostgresStore) GetByDocID(ctx context.Context, docID string) ([]*Embedding, error)

GetByDocID returns every embedding of one document, oldest first.

An unknown document is an empty result and no error — the document simply has no chunks — while an empty docID is an error, because that is a caller bug rather than an answer. Both are SQLite's choices. The five columns are SQLite's five: no collection name, no ACL, since the SQLite query does not select them and a caller that got them from one backend and not the other would be looking at exactly the divergence this file exists to prevent.

func (*PostgresStore) GetByID added in v2.82.0 

func (s *PostgresStore) GetByID(ctx context.Context, id string) (*Embedding, error)

GetByID reads one embedding back by id.

A missing id is ErrNotFound, not (nil, nil): SQLite decided that and callers test for it. The collection name comes from the join, matching the SQLite query column for column — including that CollectionID is left unset there, so it is left unset here rather than helpfully filled in.

func (*PostgresStore) GetCollection added in v2.82.0 

func (s *PostgresStore) GetCollection(ctx context.Context, name string) (*Collection, error)

func (*PostgresStore) GetCollectionStats added in v2.82.0 

func (s *PostgresStore) GetCollectionStats(ctx context.Context, name string) (*CollectionStats, error)

GetCollectionStats counts what is in a collection and how much room it takes.

Size is pg_column_size rather than SQLite's LENGTH(vector). LENGTH on a pgvector column is its dimension count, not its bytes, so reusing the same SQL would have returned a number that looked plausible and meant something else — the worst kind of wrong for a statistic nobody double-checks. pg_column_size reports the stored width including TOAST compression, which is the closest true analogue of what SQLite is measuring.

func (*PostgresStore) GetDB added in v2.82.0 

func (s *PostgresStore) GetDB() *sql.DB

GetDB hands out the raw handle for the sibling packages that keep their own tables in the same database. The caller must not close it — NewPostgresStore did not open it either.

func (*PostgresStore) GetDocument added in v2.82.0 

func (s *PostgresStore) GetDocument(ctx context.Context, id string) (*Document, error)

func (*PostgresStore) GetSession added in v2.82.0 

func (s *PostgresStore) GetSession(ctx context.Context, id string) (*Session, error)

GetSession retrieves a session by ID.

A session that is not there is an error, not a zero value — the same wrapped ErrNotFound SQLite returns, so `errors.Is(err, ErrNotFound)` works against either backend.

func (*PostgresStore) GetSessionHistory added in v2.82.0 

func (s *PostgresStore) GetSessionHistory(ctx context.Context, sessionID string, limit int) ([]*Message, error)

GetSessionHistory retrieves recent messages from a session.

The newest `limit` messages, returned oldest-first — the window is taken from the end and then read forwards, which is what a model wants in a prompt. A session with no messages is an empty result and not an error; a session that does not exist is indistinguishable from one that is empty, here as on SQLite.

func (*PostgresStore) GetSimilarityFunc added in v2.82.0 

func (s *PostgresStore) GetSimilarityFunc() SimilarityFunc

GetSimilarityFunc is how this store compares two vectors.

The graph layer scores its own candidates in Go and has to score them the same way, or a hybrid result would rank differently from a plain search over the same rows. Ranking here happens in the database, so this is the conversion that keeps the two agreeing: pgvector's <=> is cosine distance and Search returns 1 - distance, which is what CosineSimilarity computes.

func (*PostgresStore) HybridSearch added in v2.82.0 

func (s *PostgresStore) HybridSearch(ctx context.Context, vectorQuery []float32, textQuery string, opts HybridSearchOptions) ([]ScoredEmbedding, error)

HybridSearch combines a vector arm and a keyword arm with Reciprocal Rank Fusion, the same fusion and the same default k the SQLite store uses.

RRF fuses ranks, not scores, which is why nothing here has to reconcile a cosine similarity with a ts_rank — two quantities that share no scale and whose weighted sum would mean whatever the corpus happened to make it mean. A row's contribution from an arm is 1/(k + its rank in that arm), summed across the arms that found it. Consequently a hybrid score is NOT a similarity and must not be compared against a Threshold; the threshold applies inside the vector arm, where it still means what it always did.

The failure policy is SQLite's, and it is asymmetric on purpose. When the keyword arm is the only arm — no vector supplied — a failure is returned, because swallowing it reports "nothing matched" for a query that never ran. When a vector arm is also going to answer, the search degrades to vector-only and says so in the log, because silently vector-only results are indistinguishable from ordinary ones.

func (*PostgresStore) Indexed added in v2.82.0 

func (s *PostgresStore) Indexed() (bool, string)

Indexed reports what search will actually do, and why if it is not what you would want. See the note on the field.

func (*PostgresStore) Init added in v2.82.0 

func (s *PostgresStore) Init(ctx context.Context) error

func (*PostgresStore) ListCollections added in v2.82.0 

func (s *PostgresStore) ListCollections(ctx context.Context) ([]*Collection, error)

func (*PostgresStore) ListDocumentsWithFilter added in v2.82.0 

func (s *PostgresStore) ListDocumentsWithFilter(ctx context.Context, author string, limit int) ([]*Document, error)

func (*PostgresStore) MismatchedEmbeddings added in v2.82.0 

func (s *PostgresStore) MismatchedEmbeddings(ctx context.Context, wantDim, limit int) ([]*Embedding, error)

MismatchedEmbeddings returns rows whose stored vector width differs from wantDim, newest first, capped by limit (limit <= 0 means no cap). Content comes with them so a caller holding an embedder can re-embed the text.

What "mismatched" can mean here is narrower than on SQLite, and worth being precise about, because the difference is in the column type rather than in this query.

Init picks the column type from the configured dimension: `vector(N)` when one is known, bare `vector` when it is not (see store_postgres.go). Those two cases behave differently:

  • Bare `vector`. Width is per row, so a store that was filled by one model and then by another holds both widths side by side, exactly as SQLite does, and this returns the rows of the wrong one. This is the case the method exists for.

  • `vector(N)`. PostgreSQL enforces the width at write time, so a row of any other width was never accepted and drift *within* the table cannot arise. What can still arise is drift between the table and the model: the table was created as vector(768), the embedder now produces 1024, and every row is mismatched — which this reports faithfully, because it compares against wantDim rather than against the column.

The second case is worth a caller's attention for a reason that belongs to the schema and not to this method: the repair pass writes the new vectors back with UpsertBatch, and a `vector(768)` column will refuse a 1024-wide value. On this backend widening the column (ALTER TABLE ... TYPE vector(N), then rebuilding the ANN index) is a migration the operator has to run; the report is honest about the state either way, which is what lets them know the migration is needed.

func (*PostgresStore) RangeSearch added in v2.82.0 

func (s *PostgresStore) RangeSearch(ctx context.Context, query []float32, radius float32, opts SearchOptions) ([]ScoredEmbedding, error)

RangeSearch returns everything within `radius` of the query, closest first.

It used to set opts.Threshold = 1 - radius and hand the whole thing to Search, which looked like reuse and was three divergences from the SQLite store at once: the score came back a similarity where SQLite returned a distance, TopK was silently defaulted to 1000 so a wider match set was truncated with nothing said, and a non-positive radius — an error on SQLite — became a threshold that quietly matched everything. The score is now the similarity on both backends and the truncation is gone; see (*PostgresStore).rangeSearch for what replaced it.

func (*PostgresStore) ReconcileCollectionDimensions added in v2.82.0 

func (s *PostgresStore) ReconcileCollectionDimensions(ctx context.Context, dim int) (int, error)

ReconcileCollectionDimensions brings each collection's declared dimension in line with what it actually stores, for collections whose vectors are now uniformly dim. Returns the number of collections updated.

Re-embedding rewrites vectors but cannot know whether the recorded dimension was deliberate, so it is left alone until every row in the collection agrees. Without this the drift report keeps flagging rows that are in fact correct.

func (*PostgresStore) Search added in v2.82.0 

func (s *PostgresStore) Search(ctx context.Context, query []float32, opts SearchOptions) ([]ScoredEmbedding, error)

Search returns the nearest embeddings, ranked by the database.

Cosine distance, converted back to a similarity so the score means the same thing it does on SQLite — a caller comparing against a threshold must not have to know which backend answered.

func (*PostgresStore) SearchChatHistory added in v2.82.0 

func (s *PostgresStore) SearchChatHistory(ctx context.Context, queryVec []float32, sessionID string, limit int) ([]*Message, error)

SearchChatHistory returns the messages alone, for callers that never needed the scores.

func (*PostgresStore) SearchChatHistoryScored added in v2.82.0 

func (s *PostgresStore) SearchChatHistoryScored(ctx context.Context, queryVec []float32, sessionID string, limit int) ([]ScoredMessage, error)

SearchChatHistoryScored performs semantic search over a session's messages, keeping the similarity.

The ranking happens in the database — `ORDER BY vector <=> $1` — where SQLite scans the session and sorts in Go. The score is converted back from cosine distance to cosine similarity so a threshold tuned against one backend is right on the other; an exact match scores ~1 on both.

func (*PostgresStore) SearchWithACL added in v2.82.0 

func (s *PostgresStore) SearchWithACL(ctx context.Context, query []float32, acl []string, opts SearchOptions) ([]ScoredEmbedding, error)

SearchWithACL performs vector search restricted to what the caller may see.

The rule is SQLite's, unchanged: a row with no acl is public and visible to everyone, and a row that has one is visible only to a caller holding at least one of its entries. An empty caller ACL is therefore not "an administrator" — it is an anonymous reader, who sees the public rows and nothing else.

`jsonb_exists_any` is the function behind the `?|` operator, spelled out rather than written as `?|` so no driver can mistake it for a placeholder. It matches against the top-level array elements, which is exactly what SQLite's `EXISTS (SELECT 1 FROM json_each(acl) WHERE value IN (…))` walks.

func (*PostgresStore) SearchWithAdvancedFilter added in v2.82.0 

func (s *PostgresStore) SearchWithAdvancedFilter(ctx context.Context, query []float32, opts AdvancedSearchOptions) ([]ScoredEmbedding, error)

SearchWithAdvancedFilter performs vector search with a FilterExpression tree on either side of the scoring.

PreFilter is compiled to SQL and narrows what the database looks at; PostFilter is evaluated in Go by evaluateFilter — the same function the SQLite store calls, so the two backends cannot drift on what a tree means. Both are checked for compilability up front: a PostFilter carrying an operator nothing evaluates would otherwise reject every row and look like an empty corpus.

func (*PostgresStore) SearchWithFacets added in v2.104.0 

func (s *PostgresStore) SearchWithFacets(ctx context.Context, query []float32, opts FacetedSearchOptions) ([]ScoredEmbedding, []FacetResult, error)

SearchWithFacets performs vector search with faceted filtering.

The facet conditions narrow the scan, the database ranks what survives, and the counts — when asked for — are a second pass. That second pass is deliberately as wide as SQLite's: computeFacetCounts ignores the collection and the metadata filter and counts the whole table. It is a strange choice (see the note there) but it is the choice the other backend makes, and a facet sidebar that adds up differently depending on the DSN would be worse than one that is consistently too generous.

func (*PostgresStore) Stats added in v2.82.0 

func (s *PostgresStore) Stats(ctx context.Context) (StoreStats, error)

func (*PostgresStore) SyncDeletedEmbeddingIDs added in v2.82.0 

func (s *PostgresStore) SyncDeletedEmbeddingIDs(context.Context, []string)

SyncDeletedEmbeddingIDs is a no-op: pgvector's index is maintained by PostgreSQL, in the same transaction as the delete. See the note above.

func (*PostgresStore) SyncUpsertedEmbeddings added in v2.82.0 

func (s *PostgresStore) SyncUpsertedEmbeddings(context.Context, []*Embedding)

SyncUpsertedEmbeddings is a no-op: pgvector's index is maintained by PostgreSQL, in the same transaction as the write. See the note above.

func (*PostgresStore) TrainIndex added in v2.82.0 

func (s *PostgresStore) TrainIndex(context.Context, int) error

func (*PostgresStore) TrainQuantizer added in v2.82.0 

func (s *PostgresStore) TrainQuantizer(context.Context) error

TrainQuantizer has nothing to train here, and says so rather than pretending.

SQLiteStore trains a quantizer for the index it keeps in process. pgvector has no such object: compression is a column type chosen at DDL time — halfvec for 16-bit, bit for binary — so it is a migration, not a training run. A silent no-op would be the wrong answer, because a caller reaching for this wants their vectors to get smaller and would be told they had.

Contrast TrainIndex, which IS a no-op here: the index genuinely exists and PostgreSQL maintains it. Nothing is being skipped there. Here it would be.

func (*PostgresStore) UpdateDocument added in v2.82.0 

func (s *PostgresStore) UpdateDocument(ctx context.Context, doc *Document) error

UpdateDocument replaces a document's content and bumps its version.

Not part of the Store interface, but cortexdb.DB calls it, and a store that cannot answer it cannot back a brain.

func (*PostgresStore) Upsert added in v2.82.0 

func (s *PostgresStore) Upsert(ctx context.Context, emb *Embedding) error

func (*PostgresStore) UpsertBatch added in v2.82.0 

func (s *PostgresStore) UpsertBatch(ctx context.Context, embs []*Embedding) error

UpsertBatch writes in one transaction: a half-written batch would leave the caller with no way to know which half.

func (*PostgresStore) UpsertBatchTx added in v2.82.0 

func (s *PostgresStore) UpsertBatchTx(ctx context.Context, tx *sql.Tx, embs []*Embedding) error

UpsertBatchTx writes inside a transaction the caller already opened.

The point of it is that a write spanning the vector store and a sibling package's tables is one transaction rather than two that can half-fail — so it neither commits nor rolls back, and a caller who rolls back must find nothing left behind. Like SQLite's, it also touches no index: the rows are not durable until the caller commits, which is what SyncUpsertedEmbeddings is for afterwards.

func (*PostgresStore) UpsertBatchWithAdapt added in v2.82.0 

func (s *PostgresStore) UpsertBatchWithAdapt(ctx context.Context, embs []*Embedding) error

UpsertBatchWithAdapt writes vectors that may not be the store's width, reshaping them to it first.

Separate from UpsertBatch because silently reshaping a vector is a decision: the policy in Config makes it, and the default policy is StrictMode, which refuses. A store with no dimension yet takes the first vector's width as its own, which is how auto-detection settles.

func (*PostgresStore) VectorAggregate added in v2.104.0 

func (s *PostgresStore) VectorAggregate(ctx context.Context, req VectorAggregateRequest) (*VectorAggregateResponse, error)

VectorAggregate reduces the matching vectors to one vector (or one member) per group. Same contract as the SQLite implementation, including which absences are errors and which are empty results.

type QuantizationConfig 

type QuantizationConfig struct {
	Enabled bool   `json:"enabled"` // Enable quantization
	Type    string `json:"type"`    // "scalar" (SQ8) or "binary" (BQ)
	NBits   int    `json:"nBits"`   // Bits per component (default 8 for SQ8)
}

QuantizationConfig represents configuration for vector quantization

func DefaultQuantizationConfig 

func DefaultQuantizationConfig() QuantizationConfig

DefaultQuantizationConfig returns default quantization configuration

type ReciprocalRankFusionReranker 

type ReciprocalRankFusionReranker struct {
	// K is the RRF constant (typically 60)
	K float64
	// VectorWeight is the weight for vector similarity scores
	VectorWeight float64
	// TextWeight is the weight for text match scores
	TextWeight float64
}

ReciprocalRankFusionReranker combines multiple ranking signals using RRF

func NewReciprocalRankFusionReranker 

func NewReciprocalRankFusionReranker(k float64) *ReciprocalRankFusionReranker

NewReciprocalRankFusionReranker creates a new RRF reranker

func (*ReciprocalRankFusionReranker) Rerank 

func (r *ReciprocalRankFusionReranker) Rerank(ctx context.Context, query string, results []ScoredEmbedding) ([]ScoredEmbedding, error)

Rerank combines vector and text ranking using RRF

type RerankOptions 

type RerankOptions struct {
	// TopK is the number of results to return after reranking
	TopK int
	// Threshold is the minimum score threshold after reranking
	Threshold float64
	// PreserveOriginalScore keeps the original vector similarity score
	// If false, the score is replaced by the reranker's score
	PreserveOriginalScore bool
}

RerankOptions defines options for reranking

func DefaultRerankOptions 

func DefaultRerankOptions() RerankOptions

DefaultRerankOptions returns default reranking options

type Reranker 

type Reranker interface {
	// Rerank reorders the scored embeddings based on the query
	// Returns a new slice with the same embeddings, but potentially different scores/order
	Rerank(ctx context.Context, query string, results []ScoredEmbedding) ([]ScoredEmbedding, error)
}

Reranker defines the interface for re-ranking search results A reranker takes the initial search results and reorders them based on additional relevance signals beyond vector similarity

type RerankerFunc 

type RerankerFunc func(ctx context.Context, query string, results []ScoredEmbedding) ([]ScoredEmbedding, error)

RerankerFunc is a function adapter that implements Reranker interface

func (RerankerFunc) Rerank 

func (f RerankerFunc) Rerank(ctx context.Context, query string, results []ScoredEmbedding) ([]ScoredEmbedding, error)

Rerank implements the Reranker interface

type SQLiteStore 

type SQLiteStore struct {
	// contains filtered or unexported fields
}

SQLiteStore implements the Store interface using SQLite as backend

func New 

func New(path string, vectorDim int) (*SQLiteStore, error)

New creates a new SQLite vector store with the given configuration

func NewWithConfig 

func NewWithConfig(config Config) (*SQLiteStore, error)

NewWithConfig creates a new SQLite vector store with custom configuration

func (*SQLiteStore) AddMessage 

func (s *SQLiteStore) AddMessage(ctx context.Context, msg *Message) error

AddMessage adds a message to a session If vector is provided, it can be used for semantic search over chat history

func (*SQLiteStore) Aggregate 

func (s *SQLiteStore) Aggregate(ctx context.Context, req AggregationRequest) (*AggregationResponse, error)

Aggregate performs aggregation queries on embeddings metadata

func (*SQLiteStore) Backup 

func (s *SQLiteStore) Backup(ctx context.Context, filepath string) error

Backup writes a consistent copy of the whole database to filepath, without stopping writers: VACUUM INTO runs inside a read transaction, so the copy includes everything committed to the WAL at the moment it starts and nothing committed after. That is the reason it is preferred over copying the file — a .db taken on its own while the server runs is missing whatever is still in the -wal, and looks fine until it is restored.

SQLite refuses an existing destination, so this never overwrites a backup.

The destination is a bound parameter rather than interpolated into the SQL. VACUUM INTO takes an expression, so the driver accepts one, and it must: the gRPC AdminService lets a remote caller choose this path, the driver executes multiple statements from one string, and a quote in the filename would otherwise let that caller append SQL of their own to a statement running against the brain.

func (*SQLiteStore) BatchRangeSearch 

func (s *SQLiteStore) BatchRangeSearch(ctx context.Context, queries [][]float32, radius float32, opts SearchOptions) ([][]ScoredEmbedding, error)

BatchRangeSearch performs range search for multiple queries.

Results stay grouped by input query and in input order — the index into the outer slice is the index of the query that produced it, which is the only thing tying a result back to its query.

func (*SQLiteStore) Clear 

func (s *SQLiteStore) Clear(ctx context.Context) error

Clear removes all embeddings from the store

func (*SQLiteStore) ClearByDocID 

func (s *SQLiteStore) ClearByDocID(ctx context.Context, docIDs []string) error

ClearByDocID removes all embeddings for specific document IDs

func (*SQLiteStore) Close 

func (s *SQLiteStore) Close() error

Close closes the database connection and releases resources

func (*SQLiteStore) Config 

func (s *SQLiteStore) Config() Config

Config returns the current configuration of the store

func (*SQLiteStore) CreateCollection 

func (s *SQLiteStore) CreateCollection(ctx context.Context, name string, dimensions int) (*Collection, error)

CreateCollection creates a new collection

func (*SQLiteStore) CreateDocument 

func (s *SQLiteStore) CreateDocument(ctx context.Context, doc *Document) error

CreateDocument creates a new document record

func (*SQLiteStore) CreateSession 

func (s *SQLiteStore) CreateSession(ctx context.Context, session *Session) error

CreateSession creates a new chat session

func (*SQLiteStore) Delete 

func (s *SQLiteStore) Delete(ctx context.Context, id string) error

Delete removes an embedding by ID

func (*SQLiteStore) DeleteBatch 

func (s *SQLiteStore) DeleteBatch(ctx context.Context, ids []string) error

DeleteBatch removes multiple embeddings by their IDs in a single operation

func (*SQLiteStore) DeleteByDocID 

func (s *SQLiteStore) DeleteByDocID(ctx context.Context, docID string) error

DeleteByDocID removes all embeddings for a document

func (*SQLiteStore) DeleteByFilter 

func (s *SQLiteStore) DeleteByFilter(ctx context.Context, filter *MetadataFilter) error

DeleteByFilter removes embeddings matching the given metadata filter

func (*SQLiteStore) DeleteCollection 

func (s *SQLiteStore) DeleteCollection(ctx context.Context, name string) error

DeleteCollection deletes a collection and all its embeddings

func (*SQLiteStore) DeleteDocument 

func (s *SQLiteStore) DeleteDocument(ctx context.Context, id string) error

DeleteDocument deletes a document and all its associated embeddings (chunks)

func (*SQLiteStore) DeleteMultiVectorEntity 

func (s *SQLiteStore) DeleteMultiVectorEntity(ctx context.Context, entityID string) error

DeleteMultiVectorEntity deletes all vectors for an entity

func (*SQLiteStore) DimensionReport added in v2.59.0 

func (s *SQLiteStore) DimensionReport(ctx context.Context) (*DimensionReport, error)

DimensionReport groups stored vectors by collection and length. A collection whose `declared` dimension is 0 has never had one recorded; its rows are counted but not treated as mismatched.

func (*SQLiteStore) Dump 

func (s *SQLiteStore) Dump(ctx context.Context, w io.Writer, opts DumpOptions) (*DumpStats, error)

Dump exports all embeddings to a writer in the specified format

func (*SQLiteStore) DumpToFile 

func (s *SQLiteStore) DumpToFile(ctx context.Context, filepath string, opts DumpOptions) (*DumpStats, error)

DumpToFile exports data to a file

func (*SQLiteStore) ExportIndex 

func (s *SQLiteStore) ExportIndex(ctx context.Context, filepath string) error

ExportIndex exports the index data (HNSW/IVF) to a file

func (*SQLiteStore) FindAnomalies 

func (s *SQLiteStore) FindAnomalies(ctx context.Context, opts SearchOptions) ([]ScoredEmbedding, error)

FindAnomalies finds vectors that are outliers

func (*SQLiteStore) GetByDocID 

func (s *SQLiteStore) GetByDocID(ctx context.Context, docID string) ([]*Embedding, error)

GetByDocID returns all embeddings for a specific document ID

func (*SQLiteStore) GetByID 

func (s *SQLiteStore) GetByID(ctx context.Context, id string) (*Embedding, error)

GetByID gets an embedding by its ID

func (*SQLiteStore) GetCollection 

func (s *SQLiteStore) GetCollection(ctx context.Context, name string) (*Collection, error)

GetCollection retrieves a collection by name

func (*SQLiteStore) GetCollectionStats 

func (s *SQLiteStore) GetCollectionStats(ctx context.Context, name string) (*CollectionStats, error)

GetCollectionStats returns statistics for a collection

func (*SQLiteStore) GetDB 

func (s *SQLiteStore) GetDB() *sql.DB

GetDB returns the underlying database connection

func (*SQLiteStore) GetDocument 

func (s *SQLiteStore) GetDocument(ctx context.Context, id string) (*Document, error)

GetDocument retrieves a document by ID

func (*SQLiteStore) GetDocumentsByType 

func (s *SQLiteStore) GetDocumentsByType(ctx context.Context, docType string) ([]*Embedding, error)

GetDocumentsByType returns documents filtered by metadata type

func (*SQLiteStore) GetMultiVectorEntity 

func (s *SQLiteStore) GetMultiVectorEntity(ctx context.Context, entityID string) (*MultiVectorEntity, error)

GetMultiVectorEntity retrieves a multi-vector entity

func (*SQLiteStore) GetSession 

func (s *SQLiteStore) GetSession(ctx context.Context, id string) (*Session, error)

GetSession retrieves a session by ID

func (*SQLiteStore) GetSessionHistory 

func (s *SQLiteStore) GetSessionHistory(ctx context.Context, sessionID string, limit int) ([]*Message, error)

GetSessionHistory retrieves recent messages from a session

func (*SQLiteStore) GetSimilarityFunc 

func (s *SQLiteStore) GetSimilarityFunc() SimilarityFunc

GetSimilarityFunc returns the similarity function

func (*SQLiteStore) HybridSearch 

func (s *SQLiteStore) HybridSearch(ctx context.Context, vectorQuery []float32, textQuery string, opts HybridSearchOptions) ([]ScoredEmbedding, error)

HybridSearch performs combined vector and keyword search using RRF fusion

func (*SQLiteStore) ImportIndex 

func (s *SQLiteStore) ImportIndex(ctx context.Context, filepath string) error

ImportIndex imports index data from a file

func (*SQLiteStore) IncrementChanges added in v2.8.0 

func (s *SQLiteStore) IncrementChanges()

IncrementChanges increments the change counter for auto-save tracking

func (*SQLiteStore) Init 

func (s *SQLiteStore) Init(ctx context.Context) error

Init initializes the SQLite database and creates necessary tables

func (*SQLiteStore) KeywordSearchMessages 

func (s *SQLiteStore) KeywordSearchMessages(ctx context.Context, query, userID, excludeSessionID string, limit int) ([]*Message, error)

KeywordSearchMessages performs BM25 full-text search over all messages belonging to a user. It uses the SQLite FTS5 virtual table (messages_fts) for efficient keyword matching. excludeSessionID may be empty to search across all sessions.

func (*SQLiteStore) ListCollections 

func (s *SQLiteStore) ListCollections(ctx context.Context) ([]*Collection, error)

ListCollections lists all collections

func (*SQLiteStore) ListDocuments 

func (s *SQLiteStore) ListDocuments(ctx context.Context) ([]string, error)

ListDocuments returns all unique document IDs in the store

func (*SQLiteStore) ListDocumentsWithFilter 

func (s *SQLiteStore) ListDocumentsWithFilter(ctx context.Context, author string, limit int) ([]*Document, error)

ListDocumentsWithFilter lists documents matching specific criteria TODO: Add more filter options as needed

func (*SQLiteStore) ListDocumentsWithInfo 

func (s *SQLiteStore) ListDocumentsWithInfo(ctx context.Context) ([]DocumentInfo, error)

ListDocumentsWithInfo returns detailed information about documents

func (*SQLiteStore) Load 

func (s *SQLiteStore) Load(ctx context.Context, r io.Reader, opts LoadOptions) (*ImportStats, error)

Load imports embeddings from a reader

func (*SQLiteStore) LoadFromFile 

func (s *SQLiteStore) LoadFromFile(ctx context.Context, filepath string, opts LoadOptions) (*ImportStats, error)

LoadFromFile imports data from a file

func (*SQLiteStore) MismatchedEmbeddings added in v2.59.0 

func (s *SQLiteStore) MismatchedEmbeddings(ctx context.Context, wantDim, limit int) ([]*Embedding, error)

MismatchedEmbeddings returns rows whose stored vector length differs from wantDim, newest first, capped by limit (limit <= 0 means no cap). Content is included so a caller with an embedder can re-embed it.

func (*SQLiteStore) ParallelStreamSearch 

func (s *SQLiteStore) ParallelStreamSearch(ctx context.Context, queries [][]float32, opts StreamingOptions) ([]<-chan StreamingResult, error)

ParallelStreamSearch performs parallel streaming search across multiple queries

func (*SQLiteStore) RangeSearch 

func (s *SQLiteStore) RangeSearch(ctx context.Context, query []float32, radius float32, opts SearchOptions) ([]ScoredEmbedding, error)

RangeSearch returns every vector within `radius` of the query, closest first.

Score is the similarity the store's similarityFn returns — the same quantity Search puts there, on the same scale, sorted the same way. It used to be the distance instead, which inverts the meaning: 0 became "identical" and the sort ran ascending. Nothing in this package noticed, because nothing calls this; what would have noticed is the first caller to hand a range result to a reranker or to the RRF fusion in pkg/cortexdb, both of which read ScoredEmbedding.Score as a similarity and would have ranked the nearest vectors last.

The radius is still a distance, in whatever metric the store was configured with. rangeDistance is the conversion, and the PostgreSQL store applies the same one.

func (*SQLiteStore) RecommendSimilar 

func (s *SQLiteStore) RecommendSimilar(ctx context.Context, positiveIDs []string, negativeIDs []string, opts SearchOptions) ([]ScoredEmbedding, error)

RecommendSimilar finds items similar to given examples

func (*SQLiteStore) ReconcileCollectionDimensions added in v2.59.1 

func (s *SQLiteStore) ReconcileCollectionDimensions(ctx context.Context, dim int) (int, error)

ReconcileCollectionDimensions brings each collection's declared dimension in line with what it actually stores, for collections whose vectors are now uniformly dim.

Re-embedding rewrites vectors but cannot know whether the collection's recorded dimension was deliberate, so it is left alone until every row agrees. Without this the drift report keeps flagging rows that are in fact correct. Returns the number of collections updated.

func (*SQLiteStore) Search 

func (s *SQLiteStore) Search(ctx context.Context, query []float32, opts SearchOptions) ([]ScoredEmbedding, error)

Search performs vector similarity search

func (*SQLiteStore) SearchChatHistory 

func (s *SQLiteStore) SearchChatHistory(ctx context.Context, queryVec []float32, sessionID string, limit int) ([]*Message, error)

SearchChatHistory returns the messages alone, for callers that never needed the scores.

func (*SQLiteStore) SearchChatHistoryScored added in v2.75.0 

func (s *SQLiteStore) SearchChatHistoryScored(ctx context.Context, queryVec []float32, sessionID string, limit int) ([]ScoredMessage, error)

SearchChatHistoryScored is SearchChatHistory with the similarity kept.

The score was always computed and then thrown away, which forced callers to rank by list position and left them nothing to threshold on — and a vector search without a floor returns its nearest neighbours to every query, including queries the store holds nothing about.

func (*SQLiteStore) SearchMessagesByUser 

func (s *SQLiteStore) SearchMessagesByUser(ctx context.Context, userID string, queryVec []float32, excludeSessionID string, limit int) ([]*Message, error)

SearchMessagesByUser performs semantic (vector similarity) search across all sessions for a user, optionally excluding a specific session (e.g., current session already covered by short-term memory).

func (*SQLiteStore) SearchMultiVector 

func (s *SQLiteStore) SearchMultiVector(ctx context.Context, queryVectors map[string][]float32, opts MultiVectorSearchOptions) ([]ScoredEmbedding, error)

SearchMultiVector performs multi-vector search

func (*SQLiteStore) SearchWithACL 

func (s *SQLiteStore) SearchWithACL(ctx context.Context, query []float32, acl []string, opts SearchOptions) ([]ScoredEmbedding, error)

SearchWithACL performs vector search with access control filtering

func (*SQLiteStore) SearchWithAdvancedFilter 

func (s *SQLiteStore) SearchWithAdvancedFilter(ctx context.Context, query []float32, opts AdvancedSearchOptions) ([]ScoredEmbedding, error)

SearchWithAdvancedFilter performs vector search with advanced filtering

func (*SQLiteStore) SearchWithDiversity 

func (s *SQLiteStore) SearchWithDiversity(ctx context.Context, query []float32, opts DiversitySearchOptions) ([]ScoredEmbedding, error)

SearchWithDiversity performs search with result diversification

func (*SQLiteStore) SearchWithFacets 

func (s *SQLiteStore) SearchWithFacets(ctx context.Context, query []float32, opts FacetedSearchOptions) ([]ScoredEmbedding, []FacetResult, error)

func (*SQLiteStore) SearchWithFilter 

func (s *SQLiteStore) SearchWithFilter(ctx context.Context, query []float32, opts SearchOptions, metadataFilters map[string]interface{}) ([]ScoredEmbedding, error)

SearchWithFilter performs vector similarity search with advanced metadata filtering

func (*SQLiteStore) SearchWithNegatives 

func (s *SQLiteStore) SearchWithNegatives(ctx context.Context, opts NegativeSearchOptions) ([]ScoredEmbedding, error)

SearchWithNegatives performs search with negative examples

func (*SQLiteStore) SearchWithReranker 

func (s *SQLiteStore) SearchWithReranker(ctx context.Context, queryVec []float32, queryText string, reranker Reranker, opts RerankOptions) ([]ScoredEmbedding, error)

SearchWithReranker performs vector search and then reranks the results

func (*SQLiteStore) SetAutoDimAdapt 

func (s *SQLiteStore) SetAutoDimAdapt(policy AdaptPolicy)

SetAutoDimAdapt sets the dimension adaptation policy

func (*SQLiteStore) SetLogger 

func (s *SQLiteStore) SetLogger(logger Logger)

SetLogger sets the logger for the store

func (*SQLiteStore) Stats 

func (s *SQLiteStore) Stats(ctx context.Context) (StoreStats, error)

Stats returns statistics about the store

func (*SQLiteStore) StreamSearch 

func (s *SQLiteStore) StreamSearch(ctx context.Context, query []float32, opts StreamingOptions) (<-chan StreamingResult, error)

StreamSearch performs incremental vector search with results streaming

func (*SQLiteStore) SyncDeletedEmbeddingIDs added in v2.14.1 

func (s *SQLiteStore) SyncDeletedEmbeddingIDs(_ context.Context, ids []string)

SyncDeletedEmbeddingIDs removes committed embedding deletions from in-memory indexes.

func (*SQLiteStore) SyncUpsertedEmbeddings added in v2.14.1 

func (s *SQLiteStore) SyncUpsertedEmbeddings(_ context.Context, embs []*Embedding)

SyncUpsertedEmbeddings updates in-memory indexes after embeddings were committed through UpsertBatchTx.

func (*SQLiteStore) TrainIndex 

func (s *SQLiteStore) TrainIndex(ctx context.Context, numCentroids int) error

TrainIndex trains the index with existing data

func (*SQLiteStore) TrainQuantizer 

func (s *SQLiteStore) TrainQuantizer(ctx context.Context) error

TrainQuantizer trains the quantizer on existing vectors

func (*SQLiteStore) UpdateDocument 

func (s *SQLiteStore) UpdateDocument(ctx context.Context, doc *Document) error

UpdateDocument updates an existing document's metadata and other fields

func (*SQLiteStore) Upsert 

func (s *SQLiteStore) Upsert(ctx context.Context, emb *Embedding) error

Upsert inserts or updates a single embedding

func (*SQLiteStore) UpsertBatch 

func (s *SQLiteStore) UpsertBatch(ctx context.Context, embs []*Embedding) error

UpsertBatch inserts or updates multiple embeddings in a transaction

func (*SQLiteStore) UpsertBatchTx added in v2.14.1 

func (s *SQLiteStore) UpsertBatchTx(ctx context.Context, tx *sql.Tx, embs []*Embedding) error

UpsertBatchTx writes embeddings inside an existing transaction. Call SyncUpsertedEmbeddings after commit.

func (*SQLiteStore) UpsertBatchWithAdapt added in v2.13.0 

func (s *SQLiteStore) UpsertBatchWithAdapt(ctx context.Context, embs []*Embedding) error

UpsertBatchWithAdapt inserts or updates multiple embeddings with automatic dimension adaptation. Unlike UpsertBatch, this function handles dimension mismatches by adapting vectors to match the store's configured dimension before insertion.

func (*SQLiteStore) UpsertMultiVector 

func (s *SQLiteStore) UpsertMultiVector(ctx context.Context, entity *MultiVectorEntity) error

UpsertMultiVector inserts or updates a multi-vector entity

func (*SQLiteStore) VectorAggregate added in v2.104.0 

func (s *SQLiteStore) VectorAggregate(ctx context.Context, req VectorAggregateRequest) (*VectorAggregateResponse, error)

VectorAggregate reduces the matching vectors to one vector (or one member) per group.

type ScoreNormalizationReranker 

type ScoreNormalizationReranker struct {
	MinScore float64
	MaxScore float64
}

ScoreNormalizationReranker normalizes scores to a specific range

func NewScoreNormalizationReranker 

func NewScoreNormalizationReranker(min, max float64) *ScoreNormalizationReranker

NewScoreNormalizationReranker creates a new score normalizer

func (*ScoreNormalizationReranker) Rerank 

func (norm *ScoreNormalizationReranker) Rerank(ctx context.Context, query string, results []ScoredEmbedding) ([]ScoredEmbedding, error)

Rerank normalizes all scores to the specified range

type ScoredEmbedding 

type ScoredEmbedding struct {
	Embedding
	Score float64 `json:"score"`
}

ScoredEmbedding represents an embedding with similarity score

func CollectTopKFromStream 

func CollectTopKFromStream(ctx context.Context, stream <-chan StreamingResult, k int) ([]ScoredEmbedding, error)

CollectTopKFromStream collects top-k results from a streaming channel

type ScoredMessage added in v2.75.0 

type ScoredMessage struct {
	Message *Message
	Score   float64
}

SearchChatHistory performs semantic search over messages This requires messages to have vectors stored ScoredMessage pairs a message with its cosine similarity to the query.

type SearchOptions 

type SearchOptions struct {
	Collection string            `json:"collection,omitempty"` // Collection name to search in
	TopK       int               `json:"topK"`
	Filter     map[string]string `json:"filter,omitempty"`
	Threshold  float64           `json:"threshold,omitempty"`
	QueryText  string            `json:"queryText,omitempty"`  // Optional query text for enhanced matching
	TextWeight float64           `json:"textWeight,omitempty"` // Weight for text similarity (0.0-1.0, default 0.3)
}

SearchOptions defines options for vector search

type Session 

type Session struct {
	ID        string                 `json:"id"`
	UserID    string                 `json:"user_id"`
	Metadata  map[string]interface{} `json:"metadata,omitempty"`
	CreatedAt time.Time              `json:"created_at"`
	UpdatedAt time.Time              `json:"updated_at"`
}

Session represents a chat session or conversation thread

type SimilarityFunc 

type SimilarityFunc func(a, b []float32) float64

SimilarityFunc defines a function that calculates similarity between two vectors

func GetCosineSimilarity 

func GetCosineSimilarity() SimilarityFunc

GetCosineSimilarity returns the cosine similarity function

func GetDotProduct 

func GetDotProduct() SimilarityFunc

GetDotProduct returns the dot product function

func GetEuclideanDist 

func GetEuclideanDist() SimilarityFunc

GetEuclideanDist returns the euclidean distance function

type Store 

type Store interface {
	// Init initializes the store, creates necessary tables, and builds/loads indexes.
	// It must be called before any other operation.
	Init(ctx context.Context) error

	// Upsert inserts or updates a single embedding.
	// If the vector dimension doesn't match the store's dimension, it applies the adaptation policy.
	Upsert(ctx context.Context, emb *Embedding) error

	// UpsertBatch inserts or updates multiple embeddings in a single database transaction.
	// This is significantly faster than calling Upsert multiple times.
	UpsertBatch(ctx context.Context, embs []*Embedding) error

	// Search performs a vector similarity search.
	// It uses the configured index (HNSW or IVF) if available, otherwise falls back to linear search.
	Search(ctx context.Context, query []float32, opts SearchOptions) ([]ScoredEmbedding, error)

	// RangeSearch finds all vectors within a specified distance (radius) from
	// the query: everything above a bar rather than a fixed K, which is the
	// answer a threshold decision needs. Score is the similarity, descending,
	// like Search; radius is a distance in this store's metric; TopK 0 means
	// every match.
	RangeSearch(ctx context.Context, query []float32, radius float32, opts SearchOptions) ([]ScoredEmbedding, error)

	// BatchRangeSearch answers several range queries in one pass, keeping each
	// result list with the query that produced it, by index.
	BatchRangeSearch(ctx context.Context, queries [][]float32, radius float32, opts SearchOptions) ([][]ScoredEmbedding, error)

	// SearchWithFacets filters a vector search by metadata facets and can
	// return the facet counts beside the hits.
	SearchWithFacets(ctx context.Context, query []float32, opts FacetedSearchOptions) ([]ScoredEmbedding, []FacetResult, error)

	// Aggregate counts, sums, averages and groups over a metadata field.
	Aggregate(ctx context.Context, req AggregationRequest) (*AggregationResponse, error)

	// VectorAggregate reduces the vectors themselves — centroid, geometric
	// median, or the medoid, which unlike the other two is a row that exists.
	VectorAggregate(ctx context.Context, req VectorAggregateRequest) (*VectorAggregateResponse, error)

	// Delete removes an embedding by its unique ID.
	Delete(ctx context.Context, id string) error

	// DeleteByDocID removes all embeddings associated with a specific document ID.
	DeleteByDocID(ctx context.Context, docID string) error

	// DeleteBatch removes multiple embeddings by their IDs in a single operation.
	DeleteBatch(ctx context.Context, ids []string) error

	// DeleteByFilter removes embeddings matching the given metadata filter criteria.
	DeleteByFilter(ctx context.Context, filter *MetadataFilter) error

	// Close closes the store, releases database connections, and persists memory indexes.
	Close() error

	// Stats returns global statistics about the vector store (count, dimensions, size).
	Stats(ctx context.Context) (StoreStats, error)

	// CreateCollection creates a new named collection for multi-tenant isolation.
	CreateCollection(ctx context.Context, name string, dimensions int) (*Collection, error)
	// GetCollection retrieves collection information by name.
	GetCollection(ctx context.Context, name string) (*Collection, error)
	// ListCollections lists all available collections.
	ListCollections(ctx context.Context) ([]*Collection, error)
	// DeleteCollection deletes a collection and all its associated data.
	DeleteCollection(ctx context.Context, name string) error
	// GetCollectionStats returns statistics for a specific collection.
	GetCollectionStats(ctx context.Context, name string) (*CollectionStats, error)

	// TrainIndex learns cluster centroids for IVF indexes from existing data.
	TrainIndex(ctx context.Context, numCentroids int) error
	// TrainQuantizer learns value ranges for scalar quantization from existing data.
	TrainQuantizer(ctx context.Context) error

	// CreateDocument creates a document record for source tracking and versioning.
	CreateDocument(ctx context.Context, doc *Document) error
	// GetDocument retrieves a document record by its ID.
	GetDocument(ctx context.Context, id string) (*Document, error)
	// UpdateDocument replaces a document record that already exists.
	//
	// Both backends had it and the interface did not, which is the gap
	// parity_aggregate_test.go is about: a caller holding the store as an
	// interface — which cortexdb.DB does — could create a document and delete
	// one but never write over one, so replacing a record meant delete then
	// create, with a window where it is neither.
	UpdateDocument(ctx context.Context, doc *Document) error
	// DeleteDocument deletes a document and all its linked embeddings (cascading).
	DeleteDocument(ctx context.Context, id string) error
	// ListDocumentsWithFilter lists documents matching specific criteria like author.
	ListDocumentsWithFilter(ctx context.Context, author string, limit int) ([]*Document, error)

	// CreateSession starts a new conversation thread for chat memory.
	CreateSession(ctx context.Context, session *Session) error
	// GetSession retrieves a chat session by its ID.
	GetSession(ctx context.Context, id string) (*Session, error)
	// AddMessage appends a new message (user or assistant) to a session.
	AddMessage(ctx context.Context, msg *Message) error
	// GetSessionHistory returns the chronological message history for a session.
	GetSessionHistory(ctx context.Context, sessionID string, limit int) ([]*Message, error)
	// SearchChatHistory performs semantic search over previous messages in a session.
	SearchChatHistory(ctx context.Context, queryVec []float32, sessionID string, limit int) ([]*Message, error)

	// SearchWithACL performs vector search while enforcing access control rules.
	SearchWithACL(ctx context.Context, query []float32, acl []string, opts SearchOptions) ([]ScoredEmbedding, error)
	// HybridSearch combines vector similarity with FTS5 keyword matching using RRF fusion.
	HybridSearch(ctx context.Context, vectorQuery []float32, textQuery string, opts HybridSearchOptions) ([]ScoredEmbedding, error)
	// SearchWithAdvancedFilter performs vector search with complex boolean and range metadata filters.
	SearchWithAdvancedFilter(ctx context.Context, query []float32, opts AdvancedSearchOptions) ([]ScoredEmbedding, error)
}

Store defines the core interface for vector storage operations. It provides a high-level API for managing embeddings, documents, chat history, and collections.

Everything here is implemented by both backends, and that is the point of it being here. BatchRangeSearch, SearchWithFacets, Aggregate and VectorAggregate used to sit on *SQLiteStore alone and outside this interface, which made them unreachable from anything holding a Store — pkg/cortexdb.DB holds one — and made any code that did reach for them break the moment the store behind it was PostgreSQL. A capability that only one backend has does not belong on the type both are addressed through.

func OpenStore added in v2.82.0 

func OpenStore(dsn string, config Config) (Store, error)

OpenStore opens the backend a DSN asks for. 

/var/lib/cortexdb/brain.db                 -> SQLite
postgres://user:pw@host:5432/cortex        -> PostgreSQL + pgvector

A bare path has always meant a SQLite file here, so it still does: an existing configuration keeps working without being told about any of this. The caller still has to call Init.

type StoreError 

type StoreError struct {
	Op  string // Operation name
	Err error  // Underlying error
}

StoreError wraps errors with operation context

func (*StoreError) Error 

func (e *StoreError) Error() string

Error implements the error interface

func (*StoreError) Is 

func (e *StoreError) Is(target error) bool

Is checks if the error matches the target

func (*StoreError) Unwrap 

func (e *StoreError) Unwrap() error

Unwrap returns the underlying error

type StoreFactory added in v2.82.0 

type StoreFactory func(dsn string, config Config) (Store, error)

StoreFactory opens one backend from a DSN.

type StoreStats 

type StoreStats struct {
	Count      int64 `json:"count"`
	Dimensions int   `json:"dimensions"`
	Size       int64 `json:"size"`
}

StoreStats provides statistics about the vector store

type StreamingOptions 

type StreamingOptions struct {
	SearchOptions
	BatchSize        int                        // Number of vectors to process per batch
	MaxLatency       time.Duration              // Maximum time to wait before sending partial results
	EarlyTerminate   bool                       // Stop when enough good results are found
	QualityThreshold float64                    // Score threshold for early termination
	ProgressCallback func(processed, total int) // Optional progress reporting
}

StreamingOptions configures streaming search behavior

type StreamingResult 

type StreamingResult struct {
	ScoredEmbedding
	Timestamp time.Time
	BatchID   int
}

StreamingResult represents a single result in streaming search

type TextSimilarity 

type TextSimilarity interface {
	CalculateSimilarity(query, text string) float64
}

TextSimilarity interface for text-based similarity calculations

type TextSimilarityConfig 

type TextSimilarityConfig struct {
	Enabled       bool    `json:"enabled"`       // Enable text similarity matching
	DefaultWeight float64 `json:"defaultWeight"` // Default weight for text similarity (0.0-1.0)
}

TextSimilarityConfig represents configuration for text-based similarity

func DefaultTextSimilarityConfig 

func DefaultTextSimilarityConfig() TextSimilarityConfig

DefaultTextSimilarityConfig returns default text similarity configuration

type VectorAggregateGroup added in v2.104.0 

type VectorAggregateGroup struct {
	Group    string    `json:"group"`               // the GroupBy value; "" for the single ungrouped case
	Count    int       `json:"count"`               // vectors that went into this group
	Vector   []float32 `json:"vector,omitempty"`    // centroid / geometric_median; nil for medoid
	MemberID string    `json:"member_id,omitempty"` // medoid only: the id of the representative record
	// MemberContent is that record's text. It rides along because the rows
	// were already read to find it: returning an id alone would make every
	// caller ask for the one thing the question was about in a second round
	// trip, and a tool caller may have no tool that takes this kind of id.
	MemberContent string  `json:"member_content,omitempty"` // medoid only
	Score         float64 `json:"score,omitempty"`          // medoid only: its mean similarity to the rest of the group
}

VectorAggregateGroup is the answer for one group.

type VectorAggregateKind added in v2.104.0 

type VectorAggregateKind string

VectorAggregateKind names one of the three reductions. 

const (
	VectorCentroid        VectorAggregateKind = "centroid"
	VectorGeometricMedian VectorAggregateKind = "geometric_median"
	VectorMedoid          VectorAggregateKind = "medoid"
)

type VectorAggregateRequest added in v2.104.0 

type VectorAggregateRequest struct {
	Kind        VectorAggregateKind `json:"kind"`
	Collection  string              `json:"collection,omitempty"`
	Filter      map[string]string   `json:"filter,omitempty"`        // metadata equality filter
	GroupBy     string              `json:"group_by,omitempty"`      // metadata field; empty means one unnamed group over everything matched
	MaxPerGroup int                 `json:"max_per_group,omitempty"` // 0 = no cap
}

VectorAggregateRequest selects the rows and says what to do with them.

type VectorAggregateResponse added in v2.104.0 

type VectorAggregateResponse struct {
	Request VectorAggregateRequest `json:"request"`
	Groups  []VectorAggregateGroup `json:"groups"`
}

VectorAggregateResponse echoes the request beside the groups, so a result travelling on its own still says what produced it.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.