Documentation
¶
Overview ¶
Package retrieval implements hybrid, multi-signal, and adaptive retrieval pipelines for Cortex.
Package retrieval implements hybrid, multi-signal, and adaptive retrieval pipelines for Cortex.
Package retrieval implements hybrid, multi-signal, and adaptive retrieval pipelines for Cortex.
Package retrieval provides shared retrieval-pipeline orchestration that bridges the domain.VectorIndex port (W8.1, ADR-05) to the full search-result types that MCP, bench, CLI, and TUI consumers require.
The VectorIndex port returns lightweight VectorCandidate results carrying only an observation ID and a similarity score. Consumers need full observation data to format responses and to fuse vector results with FTS5 results via Reciprocal Rank Fusion. This package centralizes that post-fetch orchestration so it is NOT duplicated across consumer packages.
Dependency direction: this package imports ONLY internal/domain. It defines a narrow ObservationLookup interface (satisfied structurally by every concrete observation store) so it never reaches into a store package. Both internal/mcp and bench/locomo import this package; neither duplicates the helpers anymore.
The functions here are a pure extraction of logic that was previously duplicated verbatim in internal/mcp/tools_cortex.go and bench/locomo/runner.go. The extraction preserves byte-for-byte behavior: same RRF constant (k=60), same 1-based rank indexing, same tie-breaking (sort.Slice, NOT stable — matching the original), same soft-delete drop discipline.
Index ¶
- Constants
- Variables
- func ComputeMaxSimScore(queryTokens, docTokens []string) float64
- func ExpandQuerySynonyms(query string) string
- func FuseResults(ftsResults []*domain.SearchResult, vecResults []*domain.VectorSearchResult, ...) []*domain.SearchResult
- func NormalizeQuery(query string) string
- func ReRankWithLateInteraction(query string, results []*domain.SearchResult) []*domain.SearchResult
- func RefineAgentCRAGQuery(query string) string
- func RevalidateCandidates(ctx context.Context, obs ObservationLookup, ...) []*domain.VectorSearchResult
- func SearchVectors(ctx context.Context, idx domain.VectorIndex, q domain.VectorQuery, ...) ([]*domain.VectorSearchResult, error)
- func TokenSimilarity(a, b string) float64
- func TokenizeLateInteraction(text string) []string
- type AdaptiveSearchOptions
- type AdaptiveSearchResult
- type AgentScore
- type AgentScoreSignal
- type BatchObservationLookup
- type CRAGConfig
- type CRAGEvaluation
- type CacheEntry
- type ConfidenceGrade
- type ObservationLookup
- type QueryTier
- type ScopedCache
Constants ¶
const AgentCRAGMaxQueryRunes = 320
const PostFilterPoolMultiplier = 3
PostFilterPoolMultiplier is the factor by which the retrieval pool is expanded when the adapter declares PostFilter or none. A multiplier of 3 gives in-engine filtering enough headroom to recover candidates the adapter's post-filter removed while keeping the pool bounded. This is a heuristic; the engine truncates to the requested limit after in-engine filtering.
Variables ¶
var ErrInvalidAgentCRAGScore = errors.New("agent CRAG score must be finite and normalized")
var ErrInvalidAgentScore = errors.New("invalid agent retrieval score")
ErrInvalidAgentScore marks a score that cannot safely participate in retrieval ranking or confidence evaluation.
Functions ¶
func ComputeMaxSimScore ¶ added in v2.3.0
ComputeMaxSimScore computes the ColBERT-style Late-Interaction MaxSim score between query and document.
MaxSim(Q, D) = sum_{q in Q} max_{d in D} sim(q, d)
func ExpandQuerySynonyms ¶ added in v2.3.0
ExpandQuerySynonyms returns an OR-expanded query string containing relevant synonyms.
func FuseResults ¶
func FuseResults(ftsResults []*domain.SearchResult, vecResults []*domain.VectorSearchResult, limit int) []*domain.SearchResult
FuseResults combines FTS5 full-text search results with vector similarity search results using Reciprocal Rank Fusion (k=60).
Each input list is treated as a TRUE ranked list: only the 1-based POSITION (rank) contributes to the RRF score — a raw relevance score (BM25, cosine similarity) is NEVER fed into RRF as a rank input. The RRF formula is:
score(id) = 1/(k + rank_fts) + 1/(k + rank_vec)
where rank is the 1-based position in the respective list. An ID appearing in BOTH lists accumulates RRF credit from each (additive). An ID appearing in only one list gets credit only from that list.
The output is sorted by descending RRF score, truncated to limit. When scores are tied, sort.Slice (NOT stable) is used — matching the original behavior in every consumer. Callers must not depend on tie-breaking order.
For vector-only results (no FTS5 match), the VectorSearchResult.Similarity score is carried onto the SearchResult.Rank field so downstream consumers can inspect it.
func NormalizeQuery ¶ added in v2.3.0
NormalizeQuery cleans and corrects common typos in the search query.
func ReRankWithLateInteraction ¶ added in v2.3.0
func ReRankWithLateInteraction(query string, results []*domain.SearchResult) []*domain.SearchResult
ReRankWithLateInteraction re-ranks search results using ColBERT-inspired Late-Interaction MaxSim.
func RefineAgentCRAGQuery ¶ added in v2.3.0
func RevalidateCandidates ¶
func RevalidateCandidates(ctx context.Context, obs ObservationLookup, candidates []domain.VectorCandidate) []*domain.VectorSearchResult
RevalidateCandidates converts lightweight VectorCandidate results (ID + score from a domain.VectorIndex) into full VectorSearchResult entries by looking up the observation data via the provided ObservationLookup.
Candidates whose observation cannot be loaded (soft-deleted, missing, store error) are DROPPED — the same revalidation discipline the store-layer pipeline applies to fused candidates. A nil observation from the store is treated the same as an error: the candidate is dropped.
Batch fast path (VEC-01): when obs also implements BatchObservationLookup, the unique candidate IDs are hydrated with ONE GetByIDs call and the results are rebuilt by iterating the original candidate sequence. If the batch call fails, the error is swallowed and the unchanged per-ID loop runs instead — outputs are byte-equivalent either way.
The output preserves the INPUT ORDER of candidates (NOT re-sorted by score). Callers that need score-sorted output should sort the returned slice or rely on FuseResults, which re-sorts via RRF.
func SearchVectors ¶
func SearchVectors(ctx context.Context, idx domain.VectorIndex, q domain.VectorQuery, obs ObservationLookup) ([]*domain.VectorSearchResult, error)
SearchVectors executes a vector similarity search with capability-driven strategy selection. It reads idx.Capabilities, selects the appropriate filter strategy, retrieves candidates, revalidates them against the live observation store, applies in-engine filter safety-net when needed, and truncates to the requested limit.
Returns full VectorSearchResult entries (observation + similarity score). Soft-deleted, missing, or filter-mismatched candidates are dropped.
func TokenSimilarity ¶ added in v2.3.0
TokenSimilarity calculates token-level string overlap similarity (Jaro-Winkler / trigram-like).
func TokenizeLateInteraction ¶ added in v2.3.0
TokenizeLateInteraction cleans and splits text into normalized word and subword tokens.
Types ¶
type AdaptiveSearchOptions ¶ added in v2.3.0
type AdaptiveSearchOptions struct {
Mode string // "auto", "direct", "semantic", "multi_hop"
Project string
Scope string
Types []string
Limit int
GraphNodes []graph.GraphAnalyticsNode
GraphEdges []graph.GraphAnalyticsEdge
CRAGConfig *CRAGConfig
}
AdaptiveSearchOptions controls the adaptive retrieval engine.
type AdaptiveSearchResult ¶ added in v2.3.0
type AdaptiveSearchResult struct {
Tier QueryTier `json:"tier"`
Confidence ConfidenceGrade `json:"confidence"`
ConfidenceScore float64 `json:"confidence_score"`
NeedsRefinement bool `json:"needs_refinement"`
Results []*domain.SearchResult `json:"results"`
}
AdaptiveSearchResult represents the enriched search output with RAG metadata.
func ExecuteAdaptiveSearch ¶ added in v2.3.0
func ExecuteAdaptiveSearch( ctx context.Context, query string, opts AdaptiveSearchOptions, lexicalSearch func(ctx context.Context, q domain.SearchOptions) ([]*domain.SearchResult, error), vectorSearch func(ctx context.Context, q domain.VectorQuery) ([]*domain.VectorSearchResult, error), ) (*AdaptiveSearchResult, error)
ExecuteAdaptiveSearch runs the adaptive RAG pipeline, dynamically selecting between direct lexical search, semantic hybrid vectors, and HippoRAG graph propagation.
type AgentScore ¶ added in v2.3.0
type AgentScore struct {
Signal AgentScoreSignal
Raw *float64
Normalized float64
SourceKind string
PublicHandle string
}
AgentScore carries one native retrieval signal and its stable public identity. Raw is a pointer so a structurally missing score cannot be confused with the valid value zero.
func NormalizeAgentScores ¶ added in v2.3.0
func NormalizeAgentScores(scores []AgentScore) ([]AgentScore, error)
NormalizeAgentScores validates and normalizes scores, then orders them by descending normalized relevance. Bounded dense, MaxSim, PPR and summary inputs are clamped to one. Non-negative unbounded lexical inputs use x/(1+x) (expressed in an overflow-safe form). Missing, negative and non-finite values are rejected instead of receiving a confidence-bearing default.
Equal scores are ordered by source kind and public handle. Stable sorting preserves input order only when both public tie breakers are identical.
type AgentScoreSignal ¶ added in v2.3.0
type AgentScoreSignal string
AgentScoreSignal identifies the native scale used by one retrieval signal.
const ( AgentScoreLexical AgentScoreSignal = "lexical" AgentScoreDense AgentScoreSignal = "dense" AgentScoreMaxSim AgentScoreSignal = "maxsim" AgentScorePPR AgentScoreSignal = "ppr" AgentScoreSummary AgentScoreSignal = "summary" )
type BatchObservationLookup ¶
type BatchObservationLookup interface {
ObservationLookup
GetByIDs(ctx context.Context, ids []int64) (map[int64]*domain.Observation, error)
}
BatchObservationLookup is the OPTIONAL batch-capable superset of ObservationLookup (VEC-01). RevalidateCandidates detects it via type assertion and, when hydration succeeds, replaces the per-candidate N+1 GetByID loop with a single GetByIDs call over the unique candidate IDs.
It is deliberately retrieval-local: it is NOT added to domain.ObservationRepository (which has multiple implementors), so stores opt in simply by exposing the method — *sqlite.Store does. A lookup that does not implement it keeps the exact legacy per-ID behavior.
GetByIDs contract:
- Empty/nil ids MUST issue no SQL and return an empty map.
- Live rows MUST be keyed by observation ID.
- Soft-deleted and missing IDs MUST be absent from the map (or mapped to nil), which the engine treats as a drop — identical legacy semantics.
- Rows for IDs that were not requested MAY be present and are ignored.
type CRAGConfig ¶ added in v2.3.0
type CRAGConfig struct {
HighThreshold float64 // typically 0.65
LowThreshold float64 // typically 0.30
MinScoreFloor float64 // noise floor, results below this are stripped
}
CRAGConfig defines thresholds for Corrective RAG evaluation and filtering.
func DefaultCRAGConfig ¶ added in v2.3.0
func DefaultCRAGConfig() CRAGConfig
DefaultCRAGConfig returns standard CRAG evaluation parameters.
type CRAGEvaluation ¶ added in v2.3.0
type CRAGEvaluation struct {
Grade ConfidenceGrade `json:"grade"`
Confidence float64 `json:"confidence"`
NeedsRefinement bool `json:"needs_refinement"`
FilteredResults []*domain.SearchResult `json:"filtered_results"`
}
CRAGEvaluation encapsulates the confidence evaluation of retrieved results.
func EvaluateCRAG ¶ added in v2.3.0
func EvaluateCRAG(results []*domain.SearchResult, cfg CRAGConfig) CRAGEvaluation
EvaluateCRAG evaluates retrieved search results against CRAG confidence thresholds, filtering out noisy or irrelevant low-scoring candidates to protect downstream agents from hallucinations.
type CacheEntry ¶ added in v2.3.0
type CacheEntry[T any] struct { // contains filtered or unexported fields }
CacheEntry holds a cached value along with its expiration timestamp.
type ConfidenceGrade ¶ added in v2.3.0
type ConfidenceGrade string
ConfidenceGrade represents the categorical confidence of retrieved context.
const ( ConfidenceGradeHigh ConfidenceGrade = "high" ConfidenceGradeMedium ConfidenceGrade = "medium" ConfidenceGradeLow ConfidenceGrade = "low" )
func EvaluateAgentCRAG ¶ added in v2.3.0
func EvaluateAgentCRAG(scores []float64, cfg CRAGConfig) (ConfidenceGrade, error)
type ObservationLookup ¶
type ObservationLookup interface {
GetByID(ctx context.Context, id int64) (*domain.Observation, error)
}
ObservationLookup is the observation-store subset needed for candidate revalidation. Every concrete observation store (*sqlite.Store, test fakes, any domain.ObservationRepository) satisfies this structurally. Defining the narrow interface here keeps this package free of any store import while remaining compatible with every backend.
type QueryTier ¶ added in v2.3.0
type QueryTier string
QueryTier defines the complexity routing level in Adaptive-RAG.
const ( // TierDirectFactual represents direct lookups (symbols, IDs, exact keywords) ~1-5ms. TierDirectFactual QueryTier = "direct_factual" // TierSemanticHybrid represents conceptual queries combining lexical + dense vectors. TierSemanticHybrid QueryTier = "semantic_hybrid" // TierMultiHopGraph represents complex relational questions resolved via HippoRAG PPR. TierMultiHopGraph QueryTier = "multi_hop_graph" // TierArchitecturalGlobal represents macro architectural questions resolved via LightRAG Community Summaries. TierArchitecturalGlobal QueryTier = "architectural_global" )
func ClassifyQueryComplexity ¶ added in v2.3.0
ClassifyQueryComplexity routes a query to the optimal retrieval tier in < 0.1ms.
type ScopedCache ¶ added in v2.3.0
type ScopedCache[T any] struct { // contains filtered or unexported fields }
ScopedCache is a generic, thread-safe LRU cache with TTL expiration.
func NewScopedCache ¶ added in v2.3.0
func NewScopedCache[T any](capacity int, ttl time.Duration) *ScopedCache[T]
NewScopedCache creates a new ScopedCache with a maximum capacity and TTL.
func (*ScopedCache[T]) Clear ¶ added in v2.3.0
func (c *ScopedCache[T]) Clear()
Clear empties the cache completely.
func (*ScopedCache[T]) Get ¶ added in v2.3.0
func (c *ScopedCache[T]) Get(key string) (T, bool)
Get retrieves a value from the cache if present and not expired.
func (*ScopedCache[T]) Len ¶ added in v2.3.0
func (c *ScopedCache[T]) Len() int
Len returns the current number of items in the cache.
func (*ScopedCache[T]) PurgePrefix ¶ added in v2.3.0
func (c *ScopedCache[T]) PurgePrefix(prefix string)
PurgePrefix removes all entries whose key starts with the given prefix.
func (*ScopedCache[T]) Set ¶ added in v2.3.0
func (c *ScopedCache[T]) Set(key string, value T)
Set adds or updates a value in the cache.
Source Files