postgres

package
v0.17.3 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 20, 2026 License: MIT Imports: 22 Imported by: 0

Documentation

Overview

Package postgres provides a hybrid index backed by PostgreSQL, combining native full-text search (tsvector + ts_rank) with vector similarity search (pgvector, cosine distance). Results from both legs are fused with Reciprocal Rank Fusion.

The vector leg is only active when an llm.Client is provided; without one the index degrades gracefully to full-text search only. In both cases the target database must have the `vector` and `unaccent` extensions available (creating them requires sufficient privileges, e.g. the pgvector/pgvector Docker images).

Index

Constants

View Source
const (
	// RankNormalizationNone is PostgreSQL's default: the score ignores document
	// length entirely.
	RankNormalizationNone = 0
	// RankNormalizationLogLength divides the rank by 1 + the logarithm of the
	// document length. The gentlest length correction, and the closest in
	// spirit to BM25's b parameter.
	RankNormalizationLogLength = 1
	// RankNormalizationLength divides the rank by the document length.
	RankNormalizationLength = 2
	// RankNormalizationUniqueWords divides the rank by the number of unique
	// words in the document.
	RankNormalizationUniqueWords = 8
	// RankNormalizationScale maps the rank into [0,1) as rank/(rank+1). It does
	// not correct for length on its own, but makes scores comparable.
	RankNormalizationScale = 32
)

ts_rank normalization flags, as documented by PostgreSQL. They are a bit mask: combine with |.

View Source
const (
	// DefaultEFSearchFactor multiplies the requested limit to size the HNSW
	// result heap. Scanning more candidates than are returned is what lets the
	// graph traversal recover the neighbours it pruned early; 2 is the usual
	// starting point in pgvector's own tuning guidance.
	DefaultEFSearchFactor = 2
)
View Source
const DefaultVectorSize int = 768

DefaultVectorSize is the default dimension of the pgvector column, consistent with the sqlitevec backend.

Variables

This section is empty.

Functions

This section is empty.

Types

type Index

type Index struct {
	// contains filtered or unexported fields
}

func NewIndex

func NewIndex(pool *pgxpool.Pool, client llm.Client, funcs ...OptionFunc) *Index

NewIndex creates a hybrid PostgreSQL index on top of the given pool. The pool remains owned by the caller. A nil llm.Client disables the vector leg (full-text search only).

func (*Index) All

func (i *Index) All(ctx context.Context, yield func(model.SectionID) bool) error

All implements index.Index.

func (*Index) DeleteByID

func (i *Index) DeleteByID(ctx context.Context, ids ...model.SectionID) error

DeleteByID implements index.Index.

func (*Index) DeleteBySource

func (i *Index) DeleteBySource(ctx context.Context, source *url.URL) error

DeleteBySource implements index.Index.

func (*Index) GenerateSnapshot

func (i *Index) GenerateSnapshot(ctx context.Context) (io.ReadCloser, error)

GenerateSnapshot implements backup.Snapshotable.

func (*Index) Index

func (i *Index) Index(ctx context.Context, document model.Document, funcs ...index.OptionFunc) error

Index implements index.Index.

func (*Index) RestoreSnapshot

func (i *Index) RestoreSnapshot(ctx context.Context, r io.Reader) error

RestoreSnapshot implements backup.Snapshotable.

func (*Index) Search

func (i *Index) Search(ctx context.Context, query string, opts index.SearchOptions) ([]*index.SearchResult, error)

Search implements index.Index.

func (*Index) SearchFiltered added in v0.7.0

func (i *Index) SearchFiltered(ctx context.Context, query string, filter index.Filter, opts index.SearchOptions) ([]*index.SearchResult, error)

SearchFiltered implements index.FilterableIndex: the metadata filter is applied inside both legs, before their fusion. Filtering after the fusion would let a selective filter empty a leg's top-k — the very problem the capability exists to solve — so each leg returns its own k matching rows.

The translation is validated against the shared conformance suite (index/filtertest), which is what allows this index to advertise the capability.

type OptionFunc

type OptionFunc func(opts *Options)

func WithEFSearchFactor added in v0.15.0

func WithEFSearchFactor(factor int) OptionFunc

WithEFSearchFactor sets the multiplier applied to a query's limit to size the HNSW result heap (see Options.EFSearchFactor). A value below 1 is ignored.

func WithEmbeddingsModel

func WithEmbeddingsModel(model string) OptionFunc

func WithMaxWords

func WithMaxWords(maxWords int) OptionFunc

func WithRankNormalization added in v0.15.0

func WithRankNormalization(flags int) OptionFunc

WithRankNormalization sets the ts_rank normalization bit mask (see Options.RankNormalization). Negative values are ignored.

func WithSimpleConfigUnion added in v0.15.0

func WithSimpleConfigUnion(union bool) OptionFunc

WithSimpleConfigUnion controls whether the tsvector unions the detected language's lexemes with the 'simple' ones (see Options.SimpleConfigUnion).

It must match between indexing and querying: turning it off only at query time searches for stemmed lexemes in an index that also stores unstemmed ones, which changes the ranking without changing what was stored. Switching it therefore requires a reindex.

func WithTextSearchConfig

func WithTextSearchConfig(config string) OptionFunc

func WithVectorSize

func WithVectorSize(size int) OptionFunc

type Options

type Options struct {
	// EmbeddingsModel identifies the embeddings model; snapshots generated
	// with a different model are rejected on restore.
	EmbeddingsModel string
	// VectorSize is the dimension of the pgvector column. Larger embeddings
	// are truncated then re-normalized (matryoshka-style).
	VectorSize int
	// MaxWords bounds the size of the chunks sent to the embeddings model.
	MaxWords int
	// TextSearchConfig is the regconfig used when language detection is
	// inconclusive (default "simple").
	TextSearchConfig string
	// EFSearchFactor multiplies a query's limit to size the HNSW result heap
	// (hnsw.ef_search). Raising it trades vector search latency for recall;
	// lowering it to 1 keeps the heap at the strict minimum. Defaults to
	// DefaultEFSearchFactor.
	EFSearchFactor int
	// RankNormalization is the third argument of ts_rank, a bit mask deciding
	// how the score accounts for document length. PostgreSQL's default of 0
	// means *no* length normalization at all: a long chunk accumulates matches
	// and outranks a short, precise one, where BM25 would have divided by
	// length. See RankNormalization* for the flags.
	RankNormalization int
	// SimpleConfigUnion keeps the historical tsvector construction, which
	// unions the detected language's tsvector with the 'simple' one, both when
	// indexing and when querying:
	//
	//	to_tsvector('english', …) || to_tsvector('simple', …)
	//
	// It buys recall — an unstemmed query term still matches — at a ranking
	// cost that is easy to miss: a word whose stem equals its surface form
	// contributes one lexeme, a word with irregular morphology contributes two,
	// so ts_rank silently weighs irregular words double. Defaults to true, the
	// behaviour every existing index was built with.
	SimpleConfigUnion bool
}

func NewOptions

func NewOptions(funcs ...OptionFunc) *Options

type SnapshottedMetadata

type SnapshottedMetadata struct {
	Model      string
	VectorSize int
}

type SnapshottedRecord

type SnapshottedRecord struct {
	Source     string
	SectionID  string
	ChunkIndex int
	Content    string
	Lang       string
	// Embedding is the pgvector text literal ("[0.1,...]"), empty when the
	// chunk has no embedding.
	Embedding   string
	Collections []string
}

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL