sqlite_blob

Overview ¶

Package sqlite_blob implements the always-available zero-CGO VectorIndex adapter (ADR-05, REQ-VEC-001).

It wraps the SQLite BLOB vector store (internal/store/sqlite) behind the domain.VectorIndex port. This is the single highest-leverage change of the vector modernization: bundle.Stores.Vectors changes from the concrete *sqlite.VectorStore to domain.VectorIndex, unblocking every future adapter (qdrant, pgvector) without touching MCP/HTTP/CLI/TUI.

Search is the REQ-VEC-SCAN-001 two-pass coarse-to-fine scan:

1. A Matryoshka prefix pass scores every candidate row over the leading 1-1/coarseDivisor of the stored dimensions and shortlists the strongest max(limit*shortlistFactor, minShortlist) rows.
2. An int8 refinement pass rescores only that shortlist across the full stored prefix.
3. Both passes shard disjoint row ranges across goroutines and the merge orders by (score desc, observation id asc), so concurrency never changes the returned sequence.

Metadata-filtered searches (REQ-VEC-FILTER-001) switch plans on a deterministic cardinality probe: a highly selective filter refines every matching row (exact plan, full top-k guaranteed), a broad filter keeps the two-pass scan (constrained plan), and zero matches short-circuits to an empty result before any corpus row is read. The unfiltered path never runs the probe, so its scan behavior is unchanged.

Storage uses the same scheme: Upsert writes a pre-normalized int8 payload holding the shortest leading prefix that retains energyRetain of the vector's L2 energy. Matryoshka embeddings concentrate that energy in the leading dimensions, which is where the storage reduction comes from; a fully unstructured vector degrades gracefully to the int8 floor instead of losing recall.

Build-tag semantics are PRESERVED EXACTLY:

• Default build (cortex_vectors NOT set, zero-CGO): the underlying sqlite.VectorStore is the stub that returns ErrVectorSearchDisabled. The adapter reports degraded and passes the disabled error through without issuing any SQL. No external service, no CGO.
• cortex_vectors build tag: the underlying sqlite.VectorStore is available and the adapter runs the two-pass scan above.

Dimension-mismatch corruption is FIXED (REQ-VEC-001 error scenario): the legacy cosine path logged a warning and scored mismatched vectors 0 (silent corruption). This adapter REJECTS any upsert whose vector dimension does not match the declared ModelInfo.Dimension with domain.ErrDimensionMismatch — the mismatched vector is never stored. At scan time a row whose stored dimension differs from the query still scores 0 with a warning, matching the legacy scan instead of silently comparing misaligned coordinates.

A non-finite vector (NaN/Inf) keeps the legacy float32 blob so its scan semantics are unchanged: it still scores NaN and is filtered out, rather than being corrupted into an int8 code.