pgvector

package
v2.4.19 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 9, 2026 License: MIT Imports: 14 Imported by: 0

Documentation

Overview

Package pgvector implements the second external VectorIndex adapter (ADR-05, REQ-VEC-001/002). It uses the pgx pure-Go driver (zero-CGO) to talk to a PostgreSQL database with the pgvector extension.

SERVER-TRACK BOUNDARY: this package is SERVER-track (arch_test.go bans it from local composition). The local composition path (internal/app) wires the sqlite_blob zero-CGO default and MUST NOT import this package. Provider selection (pgvector.New) happens ONLY in the server/external composition path. This preserves REQ-FOUND-001: CGO_ENABLED=0 local build with zero external vector dependencies.

ISOLATED REPLICA: SQLite remains authoritative for observation data. The pgvector table is a vector REPLICA keyed by observation ID (BIGINT PK from SQLite). This is NOT a full Postgres source-of-truth (W11) — it is a read-optional dense candidate source for vector similarity search. No observation metadata (content, timestamps, lifecycle) is replicated; only the vector embedding and the filter columns needed for PreFilter/PostFilter search.

Schema/table isolation: the adapter creates its own schema (default cortex_vector) and table (default embeddings). These identifiers are validated against a safe regex before interpolation into SQL (schema/table names cannot be parameterized in PostgreSQL DDL). All data values use parameterized queries ($N) — no string interpolation of user data.

Dimension/model mismatch are rejected FAIL-CLOSED before any DB call (REQ-VEC-001 dim-mismatch corruption pin; model-namespace mismatch).

No plaintext secrets: the DSN password is extracted at construction time and scrubbed from every error message via redactWith (REQ-CP-002). The DSN is NEVER logged or surfaced in error strings.

Statement/query timeouts: every operation runs under a context with a configurable deadline, AND each transaction sets statement_timeout via set_config (server-side kill on slow queries).

Index

Constants

View Source
const (
	// EnvMaxParallelWorkersPerGather enables SET LOCAL
	// max_parallel_workers_per_gather = N inside the search transaction.
	// A positive integer enables the tuning; unset/invalid/<= 0 leaves the
	// default (no SET LOCAL, pool-level query, no transaction wrapper).
	EnvMaxParallelWorkersPerGather = "CORTEX_VECTOR_PGVECTOR_MAX_PARALLEL_WORKERS_PER_GATHER"

	// EnvDistanceMode selects the search distance operator. "cosine" (default,
	// also on unset) keeps the exact '<=>' cosine scan; "ip" switches to the
	// normalized inner-product '<#>' distance for pre-normalized corpora.
	EnvDistanceMode = "CORTEX_VECTOR_PGVECTOR_DISTANCE_MODE"

	// DistanceModeCosine / DistanceModeIP are the recognized mode values.
	DistanceModeCosine = "cosine"
	DistanceModeIP     = "ip"
)

Exact-scan tuning knobs for the pgvector adapter (REQ-RET-102). Both are config-gated: with the env unset the adapter behavior is byte-identical in semantics to the pre-tuning exact '<=>' cosine scan.

Variables

This section is empty.

Functions

This section is empty.

Types

type Adapter

type Adapter struct {
	// contains filtered or unexported fields
}

Adapter implements domain.VectorIndex over a PostgreSQL database with the pgvector extension via the pgx pure-Go driver.

func New

func New(ctx context.Context, cfg AdapterConfig) (*Adapter, error)

New constructs a pgvector Adapter with a real pgxpool connection pool. The caller MUST be in the server/external composition path (this package is server-track). The returned Adapter owns its pool; Close closes it.

Bootstrap: New connects with a raw pgx.Conn first to create the extension, schema, table, and index, then creates the pool. This ensures the vector type exists before any pooled connection uses it.

func NewWithDB

func NewWithDB(db pgvectorDB, cfg AdapterConfig) (*Adapter, error)

NewWithDB constructs an Adapter over a pre-built (or fake) DB. Used by tests and by compositions that manage the pool lifecycle externally. The caller controls whether Close closes the pool (see ownDB; default false).

func (*Adapter) Capabilities

func (a *Adapter) Capabilities(_ context.Context) (domain.Capabilities, error)

Capabilities declares the adapter's supported features for capability-driven strategy selection (ADR-05).

func (*Adapter) Close

func (a *Adapter) Close() error

Close releases resources. If the adapter owns its pool (factory-built via New), the underlying pool is closed.

func (*Adapter) Delete

func (a *Adapter) Delete(ctx context.Context, ids []int64) error

Delete removes vectors by observation ID using a parameterized ANY($1) clause. Missing IDs are tolerated (idempotent delete).

func (*Adapter) Health

func (a *Adapter) Health(ctx context.Context) domain.Health

Health reports the adapter's current health by pinging the PostgreSQL pool. The DSN/password is never included in the health message.

func (*Adapter) ID

func (a *Adapter) ID() string

ID returns the stable adapter identifier.

func (*Adapter) Search

Search translates a domain.VectorQuery into a parameterized SELECT against the pgvector table. Filters are mapped to WHERE conditions (PostFilter). The score threshold is applied client-side after receiving results.

func (*Adapter) Upsert

func (a *Adapter) Upsert(ctx context.Context, points []domain.VectorPoint) error

Upsert stores a batch of vectors. Dimension and model-namespace mismatch are rejected FAIL-CLOSED before any DB call (REQ-VEC-001 dim-mismatch pin; model mismatch). The batch is chunked at maxBatchSize within a transaction, and each chunk is written as ONE multi-row INSERT ... ON CONFLICT (id) DO UPDATE statement (pgvector batch pattern), removing the per-point round-trips. Duplicate ids inside a chunk are collapsed to the last occurrence (latest write wins) because ON CONFLICT DO UPDATE cannot affect the same row twice within a single statement.

type AdapterConfig

type AdapterConfig struct {
	DSN                string        // PostgreSQL runtime connection string
	BootstrapDSN       string        // optional privileged DDL connection; empty falls back to DSN
	Schema             string        // schema name (default cortex_vector)
	Table              string        // table name (default embeddings)
	Dimension          int           // expected vector dimension
	ModelName          string        // model name for namespace enforcement (empty = skip)
	IndexType          string        // hnsw or ivfflat (default hnsw)
	HNSWM              int           // HNSW max connections per node (default 16, range 2-100)
	HNSWEfConstruction int           // HNSW dynamic candidate list for build (default 64, range 1-1000)
	IVFFlatLists       int           // IVFFlat number of inverted lists (default 100, range 1-50000)
	MaxBatchSize       int           // upsert batch ceiling (default 256)
	Timeout            time.Duration // per-operation timeout (default 30s)
	MaxConns           int32         // pool max connections (default 10)
	StatementTimeoutMs int           // PostgreSQL statement_timeout in ms (default 5000)

	// GrantRoles names additional PostgreSQL roles to receive USAGE on the
	// schema and SELECT on the table during bootstrap. The server composition
	// passes the storage runtime role so the background embedding worker's
	// RLS-bound observation query can check embedding existence through
	// NOT EXISTS on this table (issue #115). Best-effort: a vector-dedicated
	// database may legitimately not contain the storage role.
	GrantRoles []string

	// Tuning carries the config-gated exact-scan tuning knobs (REQ-RET-102).
	// The zero value is fully disabled (byte-identical default cosine exact
	// scan). The factory constructor New() populates it from the
	// CORTEX_VECTOR_PGVECTOR_* environment pattern; NewWithDB callers set it
	// explicitly (tests and external compositions stay deterministic).
	Tuning SearchTuning
}

AdapterConfig holds the parameters to construct a pgvector Adapter. It mirrors config.PGVectorConfig but is declared here so the adapter package is self-contained (the server composition path maps config.PGVectorConfig → AdapterConfig). Index tuning uses typed validated integers — there is NO raw SQL string surface for index options (prevents injection via DDL).

type SearchTuning added in v2.4.14

type SearchTuning struct {
	// MaxParallelWorkersPerGather > 0 enables SET LOCAL
	// max_parallel_workers_per_gather scoped to the search transaction only.
	MaxParallelWorkersPerGather int
	// InnerProduct enables the normalized inner-product distance mode
	// ('<#>') for pre-normalized corpora. Scores stay 1 - distance per the
	// retrieval-engine contract; on unit vectors the ordering is identical to
	// cosine ordering.
	InnerProduct bool
}

SearchTuning holds the config-gated exact-scan tuning knobs. The zero value is fully disabled: default cosine operator, no SET LOCAL, no transaction wrapper (REQ-RET-102 "default behavior unchanged").

func SearchTuningFromEnv added in v2.4.14

func SearchTuningFromEnv(getenv func(string) string) SearchTuning

SearchTuningFromEnv builds SearchTuning from the CORTEX_VECTOR_PGVECTOR_* environment pattern (same style as internal/store/postgres CORTEX_VECTOR_PGVECTOR_DIMENSION handling). getenv is injected for deterministic unit testing; SearchTuningFromOS reads the real environment.

func SearchTuningFromOS added in v2.4.14

func SearchTuningFromOS() SearchTuning

SearchTuningFromOS reads the tuning env vars from the process environment.

func (SearchTuning) Enabled added in v2.4.14

func (t SearchTuning) Enabled() bool

Enabled reports whether any tuning knob is active. When false the search path is untouched (no transaction, default SQL).

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL