Documentation
¶
Overview ¶
Package retrieval provides shared retrieval-pipeline orchestration that bridges the domain.VectorIndex port (W8.1, ADR-05) to the full search-result types that MCP, bench, CLI, and TUI consumers require.
The VectorIndex port returns lightweight VectorCandidate results carrying only an observation ID and a similarity score. Consumers need full observation data to format responses and to fuse vector results with FTS5 results via Reciprocal Rank Fusion. This package centralizes that post-fetch orchestration so it is NOT duplicated across consumer packages.
Dependency direction: this package imports ONLY internal/domain. It defines a narrow ObservationLookup interface (satisfied structurally by every concrete observation store) so it never reaches into a store package. Both internal/mcp and bench/locomo import this package; neither duplicates the helpers anymore.
The functions here are a pure extraction of logic that was previously duplicated verbatim in internal/mcp/tools_cortex.go and bench/locomo/runner.go. The extraction preserves byte-for-byte behavior: same RRF constant (k=60), same 1-based rank indexing, same tie-breaking (sort.Slice, NOT stable — matching the original), same soft-delete drop discipline.
Index ¶
- Constants
- func FuseResults(ftsResults []*domain.SearchResult, vecResults []*domain.VectorSearchResult, ...) []*domain.SearchResult
- func RevalidateCandidates(ctx context.Context, obs ObservationLookup, ...) []*domain.VectorSearchResult
- func SearchVectors(ctx context.Context, idx domain.VectorIndex, q domain.VectorQuery, ...) ([]*domain.VectorSearchResult, error)
- type BatchObservationLookup
- type ObservationLookup
Constants ¶
const PostFilterPoolMultiplier = 3
PostFilterPoolMultiplier is the factor by which the retrieval pool is expanded when the adapter declares PostFilter or none. A multiplier of 3 gives in-engine filtering enough headroom to recover candidates the adapter's post-filter removed while keeping the pool bounded. This is a heuristic; the engine truncates to the requested limit after in-engine filtering.
Variables ¶
This section is empty.
Functions ¶
func FuseResults ¶
func FuseResults(ftsResults []*domain.SearchResult, vecResults []*domain.VectorSearchResult, limit int) []*domain.SearchResult
FuseResults combines FTS5 full-text search results with vector similarity search results using Reciprocal Rank Fusion (k=60).
Each input list is treated as a TRUE ranked list: only the 1-based POSITION (rank) contributes to the RRF score — a raw relevance score (BM25, cosine similarity) is NEVER fed into RRF as a rank input. The RRF formula is:
score(id) = 1/(k + rank_fts) + 1/(k + rank_vec)
where rank is the 1-based position in the respective list. An ID appearing in BOTH lists accumulates RRF credit from each (additive). An ID appearing in only one list gets credit only from that list.
The output is sorted by descending RRF score, truncated to limit. When scores are tied, sort.Slice (NOT stable) is used — matching the original behavior in every consumer. Callers must not depend on tie-breaking order.
For vector-only results (no FTS5 match), the VectorSearchResult.Similarity score is carried onto the SearchResult.Rank field so downstream consumers can inspect it.
func RevalidateCandidates ¶
func RevalidateCandidates(ctx context.Context, obs ObservationLookup, candidates []domain.VectorCandidate) []*domain.VectorSearchResult
RevalidateCandidates converts lightweight VectorCandidate results (ID + score from a domain.VectorIndex) into full VectorSearchResult entries by looking up the observation data via the provided ObservationLookup.
Candidates whose observation cannot be loaded (soft-deleted, missing, store error) are DROPPED — the same revalidation discipline the store-layer pipeline applies to fused candidates. A nil observation from the store is treated the same as an error: the candidate is dropped.
Batch fast path (VEC-01): when obs also implements BatchObservationLookup, the unique candidate IDs are hydrated with ONE GetByIDs call and the results are rebuilt by iterating the original candidate sequence. If the batch call fails, the error is swallowed and the unchanged per-ID loop runs instead — outputs are byte-equivalent either way.
The output preserves the INPUT ORDER of candidates (NOT re-sorted by score). Callers that need score-sorted output should sort the returned slice or rely on FuseResults, which re-sorts via RRF.
func SearchVectors ¶
func SearchVectors(ctx context.Context, idx domain.VectorIndex, q domain.VectorQuery, obs ObservationLookup) ([]*domain.VectorSearchResult, error)
SearchVectors executes a vector similarity search with capability-driven strategy selection. It reads idx.Capabilities, selects the appropriate filter strategy, retrieves candidates, revalidates them against the live observation store, applies in-engine filter safety-net when needed, and truncates to the requested limit.
Returns full VectorSearchResult entries (observation + similarity score). Soft-deleted, missing, or filter-mismatched candidates are dropped.
Types ¶
type BatchObservationLookup ¶
type BatchObservationLookup interface {
ObservationLookup
GetByIDs(ctx context.Context, ids []int64) (map[int64]*domain.Observation, error)
}
BatchObservationLookup is the OPTIONAL batch-capable superset of ObservationLookup (VEC-01). RevalidateCandidates detects it via type assertion and, when hydration succeeds, replaces the per-candidate N+1 GetByID loop with a single GetByIDs call over the unique candidate IDs.
It is deliberately retrieval-local: it is NOT added to domain.ObservationRepository (which has multiple implementors), so stores opt in simply by exposing the method — *sqlite.Store does. A lookup that does not implement it keeps the exact legacy per-ID behavior.
GetByIDs contract:
- Empty/nil ids MUST issue no SQL and return an empty map.
- Live rows MUST be keyed by observation ID.
- Soft-deleted and missing IDs MUST be absent from the map (or mapped to nil), which the engine treats as a drop — identical legacy semantics.
- Rows for IDs that were not requested MAY be present and are ignored.
type ObservationLookup ¶
type ObservationLookup interface {
GetByID(ctx context.Context, id int64) (*domain.Observation, error)
}
ObservationLookup is the observation-store subset needed for candidate revalidation. Every concrete observation store (*sqlite.Store, test fakes, any domain.ObservationRepository) satisfies this structurally. Defining the narrow interface here keeps this package free of any store import while remaining compatible with every backend.
Source Files