Documentation
¶
Overview ¶
Package postgres provides a hybrid index backed by PostgreSQL, combining native full-text search (tsvector + ts_rank) with vector similarity search (pgvector, cosine distance). Results from both legs are fused with Reciprocal Rank Fusion.
The vector leg is only active when an llm.Client is provided; without one the index degrades gracefully to full-text search only. In both cases the target database must have the `vector` and `unaccent` extensions available (creating them requires sufficient privileges, e.g. the pgvector/pgvector Docker images).
Index ¶
- Constants
- type Index
- func (i *Index) All(ctx context.Context, yield func(model.SectionID) bool) error
- func (i *Index) DeleteByID(ctx context.Context, ids ...model.SectionID) error
- func (i *Index) DeleteBySource(ctx context.Context, source *url.URL) error
- func (i *Index) GenerateSnapshot(ctx context.Context) (io.ReadCloser, error)
- func (i *Index) Index(ctx context.Context, document model.Document, funcs ...index.OptionFunc) error
- func (i *Index) RestoreSnapshot(ctx context.Context, r io.Reader) error
- func (i *Index) Search(ctx context.Context, query string, opts index.SearchOptions) ([]*index.SearchResult, error)
- func (i *Index) SearchFiltered(ctx context.Context, query string, filter index.Filter, ...) ([]*index.SearchResult, error)
- type OptionFunc
- func WithEFSearchFactor(factor int) OptionFunc
- func WithEmbeddingsModel(model string) OptionFunc
- func WithMaxWords(maxWords int) OptionFunc
- func WithRankNormalization(flags int) OptionFunc
- func WithSimpleConfigUnion(union bool) OptionFunc
- func WithTextSearchConfig(config string) OptionFunc
- func WithVectorSize(size int) OptionFunc
- type Options
- type SnapshottedMetadata
- type SnapshottedRecord
Constants ¶
const ( // RankNormalizationNone is PostgreSQL's default: the score ignores document // length entirely. RankNormalizationNone = 0 // RankNormalizationLogLength divides the rank by 1 + the logarithm of the // document length. The gentlest length correction, and the closest in // spirit to BM25's b parameter. RankNormalizationLogLength = 1 // RankNormalizationLength divides the rank by the document length. RankNormalizationLength = 2 // RankNormalizationUniqueWords divides the rank by the number of unique // words in the document. RankNormalizationUniqueWords = 8 // RankNormalizationScale maps the rank into [0,1) as rank/(rank+1). It does // not correct for length on its own, but makes scores comparable. RankNormalizationScale = 32 )
ts_rank normalization flags, as documented by PostgreSQL. They are a bit mask: combine with |.
const ( // DefaultEFSearchFactor multiplies the requested limit to size the HNSW // result heap. Scanning more candidates than are returned is what lets the // graph traversal recover the neighbours it pruned early; 2 is the usual // starting point in pgvector's own tuning guidance. DefaultEFSearchFactor = 2 )
const DefaultVectorSize int = 768
DefaultVectorSize is the default dimension of the pgvector column, consistent with the sqlitevec backend.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Index ¶
type Index struct {
// contains filtered or unexported fields
}
func NewIndex ¶
NewIndex creates a hybrid PostgreSQL index on top of the given pool. The pool remains owned by the caller. A nil llm.Client disables the vector leg (full-text search only).
func (*Index) DeleteByID ¶
DeleteByID implements index.Index.
func (*Index) DeleteBySource ¶
DeleteBySource implements index.Index.
func (*Index) GenerateSnapshot ¶
GenerateSnapshot implements backup.Snapshotable.
func (*Index) Index ¶
func (i *Index) Index(ctx context.Context, document model.Document, funcs ...index.OptionFunc) error
Index implements index.Index.
func (*Index) RestoreSnapshot ¶
RestoreSnapshot implements backup.Snapshotable.
func (*Index) Search ¶
func (i *Index) Search(ctx context.Context, query string, opts index.SearchOptions) ([]*index.SearchResult, error)
Search implements index.Index.
func (*Index) SearchFiltered ¶ added in v0.7.0
func (i *Index) SearchFiltered(ctx context.Context, query string, filter index.Filter, opts index.SearchOptions) ([]*index.SearchResult, error)
SearchFiltered implements index.FilterableIndex: the metadata filter is applied inside both legs, before their fusion. Filtering after the fusion would let a selective filter empty a leg's top-k — the very problem the capability exists to solve — so each leg returns its own k matching rows.
The translation is validated against the shared conformance suite (index/filtertest), which is what allows this index to advertise the capability.
type OptionFunc ¶
type OptionFunc func(opts *Options)
func WithEFSearchFactor ¶ added in v0.15.0
func WithEFSearchFactor(factor int) OptionFunc
WithEFSearchFactor sets the multiplier applied to a query's limit to size the HNSW result heap (see Options.EFSearchFactor). A value below 1 is ignored.
func WithEmbeddingsModel ¶
func WithEmbeddingsModel(model string) OptionFunc
func WithMaxWords ¶
func WithMaxWords(maxWords int) OptionFunc
func WithRankNormalization ¶ added in v0.15.0
func WithRankNormalization(flags int) OptionFunc
WithRankNormalization sets the ts_rank normalization bit mask (see Options.RankNormalization). Negative values are ignored.
func WithSimpleConfigUnion ¶ added in v0.15.0
func WithSimpleConfigUnion(union bool) OptionFunc
WithSimpleConfigUnion controls whether the tsvector unions the detected language's lexemes with the 'simple' ones (see Options.SimpleConfigUnion).
It must match between indexing and querying: turning it off only at query time searches for stemmed lexemes in an index that also stores unstemmed ones, which changes the ranking without changing what was stored. Switching it therefore requires a reindex.
func WithTextSearchConfig ¶
func WithTextSearchConfig(config string) OptionFunc
func WithVectorSize ¶
func WithVectorSize(size int) OptionFunc
type Options ¶
type Options struct {
// EmbeddingsModel identifies the embeddings model; snapshots generated
// with a different model are rejected on restore.
EmbeddingsModel string
// VectorSize is the dimension of the pgvector column. Larger embeddings
// are truncated then re-normalized (matryoshka-style).
VectorSize int
// MaxWords bounds the size of the chunks sent to the embeddings model.
MaxWords int
// TextSearchConfig is the regconfig used when language detection is
// inconclusive (default "simple").
TextSearchConfig string
// EFSearchFactor multiplies a query's limit to size the HNSW result heap
// (hnsw.ef_search). Raising it trades vector search latency for recall;
// lowering it to 1 keeps the heap at the strict minimum. Defaults to
// DefaultEFSearchFactor.
EFSearchFactor int
// RankNormalization is the third argument of ts_rank, a bit mask deciding
// how the score accounts for document length. PostgreSQL's default of 0
// means *no* length normalization at all: a long chunk accumulates matches
// and outranks a short, precise one, where BM25 would have divided by
// length. See RankNormalization* for the flags.
RankNormalization int
// SimpleConfigUnion keeps the historical tsvector construction, which
// unions the detected language's tsvector with the 'simple' one, both when
// indexing and when querying:
//
// to_tsvector('english', …) || to_tsvector('simple', …)
//
// It buys recall — an unstemmed query term still matches — at a ranking
// cost that is easy to miss: a word whose stem equals its surface form
// contributes one lexeme, a word with irregular morphology contributes two,
// so ts_rank silently weighs irregular words double. Defaults to true, the
// behaviour every existing index was built with.
SimpleConfigUnion bool
}
func NewOptions ¶
func NewOptions(funcs ...OptionFunc) *Options