Documentation
¶
Overview ¶
Package retrieval implements hybrid, multi-signal, and adaptive retrieval pipelines for Cortex.
Package retrieval implements hybrid, multi-signal, and adaptive retrieval pipelines for Cortex.
Package retrieval implements hybrid, multi-signal, and adaptive retrieval pipelines for Cortex.
Package retrieval provides shared retrieval-pipeline orchestration that bridges the domain.VectorIndex port (W8.1, ADR-05) to the full search-result types that MCP, bench, CLI, and TUI consumers require.
The VectorIndex port returns lightweight VectorCandidate results carrying only an observation ID and a similarity score. Consumers need full observation data to format responses and to fuse vector results with FTS5 results via Reciprocal Rank Fusion. This package centralizes that post-fetch orchestration so it is NOT duplicated across consumer packages.
Dependency direction: this package imports ONLY internal/domain. It defines a narrow ObservationLookup interface (satisfied structurally by every concrete observation store) so it never reaches into a store package. Both internal/mcp and bench/locomo import this package; neither duplicates the helpers anymore.
The functions here are a pure extraction of logic that was previously duplicated verbatim in internal/mcp/tools_cortex.go and bench/locomo/runner.go. The extraction preserves byte-for-byte behavior: same RRF constant (k=60), same 1-based rank indexing, same tie-breaking (sort.Slice, NOT stable — matching the original), same soft-delete drop discipline.
Index ¶
- Constants
- Variables
- func ComputeMaxSimScore(queryTokens, docTokens []string) float64
- func ComputeWeightedMaxSimScore(queryTokens, docTokens []string) float64
- func ExpandQuerySynonyms(query string) string
- func FuseResults(ftsResults []*domain.SearchResult, vecResults []*domain.VectorSearchResult, ...) []*domain.SearchResult
- func FuseResultsWithOptions(ftsResults []*domain.SearchResult, vecResults []*domain.VectorSearchResult, ...) []*domain.SearchResult
- func NormalizeQuery(query string) string
- func ReRankWithLateInteraction(query string, results []*domain.SearchResult) []*domain.SearchResult
- func RefineAgentCRAGQuery(query string) string
- func RevalidateCandidates(ctx context.Context, obs ObservationLookup, ...) []*domain.VectorSearchResult
- func SearchVectors(ctx context.Context, idx domain.VectorIndex, q domain.VectorQuery, ...) ([]*domain.VectorSearchResult, error)
- func TokenSimilarity(a, b string) float64
- func TokenSpecificityWeight(token string) float64
- func TokenizeLateInteraction(text string) []string
- type AdaptiveSearchOptions
- type AdaptiveSearchResult
- type AgentScore
- type AgentScoreSignal
- type BatchObservationLookup
- type CRAGConfig
- type CRAGEvaluation
- type CacheEntry
- type ConfidenceGrade
- type FuseOptions
- type ObservationLookup
- type QueryTier
- type ScopedCache
Constants ¶
const AgentCRAGMaxQueryRunes = 320
const PostFilterPoolMultiplier = 3
PostFilterPoolMultiplier is the factor by which the retrieval pool is expanded when the adapter declares PostFilter or none. A multiplier of 3 gives in-engine filtering enough headroom to recover candidates the adapter's post-filter removed while keeping the pool bounded. This is a heuristic; the engine truncates to the requested limit after in-engine filtering.
Variables ¶
var ErrInvalidAgentCRAGScore = errors.New("agent CRAG score must be finite and normalized")
var ErrInvalidAgentScore = errors.New("invalid agent retrieval score")
ErrInvalidAgentScore marks a score that cannot safely participate in retrieval ranking or confidence evaluation.
Functions ¶
func ComputeMaxSimScore ¶ added in v2.3.0
ComputeMaxSimScore computes the ColBERT-style Late-Interaction MaxSim score between query and document.
MaxSim(Q, D) = sum_{q in Q} w(q) * max_{d in D} sim(q, d) / sum_{q in Q} w(q)
func ComputeWeightedMaxSimScore ¶ added in v2.3.8
ComputeWeightedMaxSimScore computes the specificity-weighted ColBERT MaxSim score.
func ExpandQuerySynonyms ¶ added in v2.3.0
ExpandQuerySynonyms returns an OR-expanded query string containing relevant synonyms.
func FuseResults ¶
func FuseResults(ftsResults []*domain.SearchResult, vecResults []*domain.VectorSearchResult, limit int) []*domain.SearchResult
FuseResults combines FTS5 full-text search results with vector similarity search results using Reciprocal Rank Fusion (k=60).
Preserves byte-for-byte behavior for all callers by delegating to FuseResultsWithOptions.
func FuseResultsWithOptions ¶ added in v2.3.8
func FuseResultsWithOptions(ftsResults []*domain.SearchResult, vecResults []*domain.VectorSearchResult, opts FuseOptions) []*domain.SearchResult
FuseResultsWithOptions combines FTS5 full-text search results with vector similarity search results using Reciprocal Rank Fusion (k=60) with support for configurable exponential temporal decay (Time-Decayed RRF).
func NormalizeQuery ¶ added in v2.3.0
NormalizeQuery cleans and corrects common typos in the search query.
func ReRankWithLateInteraction ¶ added in v2.3.0
func ReRankWithLateInteraction(query string, results []*domain.SearchResult) []*domain.SearchResult
ReRankWithLateInteraction re-ranks search results using ColBERT-inspired Late-Interaction MaxSim.
func RefineAgentCRAGQuery ¶ added in v2.3.0
func RevalidateCandidates ¶
func RevalidateCandidates(ctx context.Context, obs ObservationLookup, candidates []domain.VectorCandidate) []*domain.VectorSearchResult
RevalidateCandidates converts lightweight VectorCandidate results (ID + score from a domain.VectorIndex) into full VectorSearchResult entries by looking up the observation data via the provided ObservationLookup.
Candidates whose observation cannot be loaded (soft-deleted, missing, store error) are DROPPED — the same revalidation discipline the store-layer pipeline applies to fused candidates. A nil observation from the store is treated the same as an error: the candidate is dropped.
Batch fast path (VEC-01): when obs also implements BatchObservationLookup, the unique candidate IDs are hydrated with ONE GetByIDs call and the results are rebuilt by iterating the original candidate sequence. If the batch call fails, the error is swallowed and the unchanged per-ID loop runs instead — outputs are byte-equivalent either way.
The output preserves the INPUT ORDER of candidates (NOT re-sorted by score). Callers that need score-sorted output should sort the returned slice or rely on FuseResults, which re-sorts via RRF.
func SearchVectors ¶
func SearchVectors(ctx context.Context, idx domain.VectorIndex, q domain.VectorQuery, obs ObservationLookup) ([]*domain.VectorSearchResult, error)
SearchVectors executes a vector similarity search with capability-driven strategy selection. It reads idx.Capabilities, selects the appropriate filter strategy, retrieves candidates, revalidates them against the live observation store, applies in-engine filter safety-net when needed, and truncates to the requested limit.
Returns full VectorSearchResult entries (observation + similarity score). Soft-deleted, missing, or filter-mismatched candidates are dropped.
func TokenSimilarity ¶ added in v2.3.0
TokenSimilarity calculates token-level string overlap similarity (Jaro-Winkler / trigram-like).
func TokenSpecificityWeight ¶ added in v2.3.8
TokenSpecificityWeight computes the information-theoretic importance weight of a token. Common stop words receive reduced weight, while technical identifiers, tokens containing underscores or digits, and longer terms receive elevated weights.
func TokenizeLateInteraction ¶ added in v2.3.0
TokenizeLateInteraction cleans and splits text into normalized word and subword tokens.
Types ¶
type AdaptiveSearchOptions ¶ added in v2.3.0
type AdaptiveSearchOptions struct {
Mode string // "auto", "direct", "semantic", "multi_hop"
Project string
Scope string
Types []string
Limit int
GraphNodes []graph.GraphAnalyticsNode
GraphEdges []graph.GraphAnalyticsEdge
CRAGConfig *CRAGConfig
}
AdaptiveSearchOptions controls the adaptive retrieval engine.
type AdaptiveSearchResult ¶ added in v2.3.0
type AdaptiveSearchResult struct {
Tier QueryTier `json:"tier"`
Confidence ConfidenceGrade `json:"confidence"`
ConfidenceScore float64 `json:"confidence_score"`
NeedsRefinement bool `json:"needs_refinement"`
Results []*domain.SearchResult `json:"results"`
}
AdaptiveSearchResult represents the enriched search output with RAG metadata.
func ExecuteAdaptiveSearch ¶ added in v2.3.0
func ExecuteAdaptiveSearch( ctx context.Context, query string, opts AdaptiveSearchOptions, lexicalSearch func(ctx context.Context, q domain.SearchOptions) ([]*domain.SearchResult, error), vectorSearch func(ctx context.Context, q domain.VectorQuery) ([]*domain.VectorSearchResult, error), ) (*AdaptiveSearchResult, error)
ExecuteAdaptiveSearch runs the adaptive RAG pipeline, dynamically selecting between direct lexical search, semantic hybrid vectors, and HippoRAG graph propagation.
type AgentScore ¶ added in v2.3.0
type AgentScore struct {
Signal AgentScoreSignal
Raw *float64
Normalized float64
SourceKind string
PublicHandle string
}
AgentScore carries one native retrieval signal and its stable public identity. Raw is a pointer so a structurally missing score cannot be confused with the valid value zero.
func NormalizeAgentScores ¶ added in v2.3.0
func NormalizeAgentScores(scores []AgentScore) ([]AgentScore, error)
NormalizeAgentScores validates and normalizes scores, then orders them by descending normalized relevance. Bounded dense, MaxSim, PPR and summary inputs are clamped to one. Non-negative unbounded lexical inputs use x/(1+x) (expressed in an overflow-safe form). Missing, negative and non-finite values are rejected instead of receiving a confidence-bearing default.
Equal scores are ordered by source kind and public handle. Stable sorting preserves input order only when both public tie breakers are identical.
type AgentScoreSignal ¶ added in v2.3.0
type AgentScoreSignal string
AgentScoreSignal identifies the native scale used by one retrieval signal.
const ( AgentScoreLexical AgentScoreSignal = "lexical" AgentScoreDense AgentScoreSignal = "dense" AgentScoreMaxSim AgentScoreSignal = "maxsim" AgentScorePPR AgentScoreSignal = "ppr" AgentScoreSummary AgentScoreSignal = "summary" )
type BatchObservationLookup ¶
type BatchObservationLookup interface {
ObservationLookup
GetByIDs(ctx context.Context, ids []int64) (map[int64]*domain.Observation, error)
}
BatchObservationLookup is the OPTIONAL batch-capable superset of ObservationLookup (VEC-01). RevalidateCandidates detects it via type assertion and, when hydration succeeds, replaces the per-candidate N+1 GetByID loop with a single GetByIDs call over the unique candidate IDs.
It is deliberately retrieval-local: it is NOT added to domain.ObservationRepository (which has multiple implementors), so stores opt in simply by exposing the method — *sqlite.Store does. A lookup that does not implement it keeps the exact legacy per-ID behavior.
GetByIDs contract:
- Empty/nil ids MUST issue no SQL and return an empty map.
- Live rows MUST be keyed by observation ID.
- Soft-deleted and missing IDs MUST be absent from the map (or mapped to nil), which the engine treats as a drop — identical legacy semantics.
- Rows for IDs that were not requested MAY be present and are ignored.
type CRAGConfig ¶ added in v2.3.0
type CRAGConfig struct {
HighThreshold float64 // typically 0.65
LowThreshold float64 // typically 0.30
MinScoreFloor float64 // noise floor, results below this are stripped
DynamicMode bool // enables dynamic score gap & relative elbow calibration (for RRF/dense distributions)
}
CRAGConfig defines thresholds for Corrective RAG evaluation and filtering.
func DefaultCRAGConfig ¶ added in v2.3.0
func DefaultCRAGConfig() CRAGConfig
DefaultCRAGConfig returns standard CRAG evaluation parameters.
func DynamicCRAGConfig ¶ added in v2.3.8
func DynamicCRAGConfig() CRAGConfig
DynamicCRAGConfig returns CRAG configuration optimized for dynamic/RRF score distributions.
type CRAGEvaluation ¶ added in v2.3.0
type CRAGEvaluation struct {
Grade ConfidenceGrade `json:"grade"`
Confidence float64 `json:"confidence"`
NeedsRefinement bool `json:"needs_refinement"`
FilteredResults []*domain.SearchResult `json:"filtered_results"`
}
CRAGEvaluation encapsulates the confidence evaluation of retrieved results.
func EvaluateCRAG ¶ added in v2.3.0
func EvaluateCRAG(results []*domain.SearchResult, cfg CRAGConfig) CRAGEvaluation
EvaluateCRAG evaluates retrieved search results against CRAG confidence thresholds, filtering out noisy or irrelevant low-scoring candidates to protect downstream agents from hallucinations.
type CacheEntry ¶ added in v2.3.0
type CacheEntry[T any] struct { // contains filtered or unexported fields }
CacheEntry holds a cached value along with its expiration timestamp.
type ConfidenceGrade ¶ added in v2.3.0
type ConfidenceGrade string
ConfidenceGrade represents the categorical confidence of retrieved context.
const ( ConfidenceGradeHigh ConfidenceGrade = "high" ConfidenceGradeMedium ConfidenceGrade = "medium" ConfidenceGradeLow ConfidenceGrade = "low" )
func EvaluateAgentCRAG ¶ added in v2.3.0
func EvaluateAgentCRAG(scores []float64, cfg CRAGConfig) (ConfidenceGrade, error)
type FuseOptions ¶ added in v2.3.8
type FuseOptions struct {
// Limit is the maximum number of fused search results to return.
Limit int
// DecayHalfLifeDays specifies the half-life in days for exponential recency decay.
// When <= 0, no temporal decay is applied (standard RRF).
DecayHalfLifeDays float64
// ReferenceTime is the anchor time for decay calculation. If zero, time.Now() is used.
ReferenceTime time.Time
}
FuseOptions configures Reciprocal Rank Fusion and optional temporal decay.
type ObservationLookup ¶
type ObservationLookup interface {
GetByID(ctx context.Context, id int64) (*domain.Observation, error)
}
ObservationLookup is the observation-store subset needed for candidate revalidation. Every concrete observation store (*sqlite.Store, test fakes, any domain.ObservationRepository) satisfies this structurally. Defining the narrow interface here keeps this package free of any store import while remaining compatible with every backend.
type QueryTier ¶ added in v2.3.0
type QueryTier string
QueryTier defines the complexity routing level in Adaptive-RAG.
const ( // TierDirectFactual represents direct lookups (symbols, IDs, exact keywords) ~1-5ms. TierDirectFactual QueryTier = "direct_factual" // TierSemanticHybrid represents conceptual queries combining lexical + dense vectors. TierSemanticHybrid QueryTier = "semantic_hybrid" // TierMultiHopGraph represents complex relational questions resolved via HippoRAG PPR. TierMultiHopGraph QueryTier = "multi_hop_graph" // TierArchitecturalGlobal represents macro architectural questions resolved via LightRAG Community Summaries. TierArchitecturalGlobal QueryTier = "architectural_global" )
func ClassifyQueryComplexity ¶ added in v2.3.0
ClassifyQueryComplexity routes a query to the optimal retrieval tier in < 0.1ms.
type ScopedCache ¶ added in v2.3.0
type ScopedCache[T any] struct { // contains filtered or unexported fields }
ScopedCache is a generic, thread-safe LRU cache with TTL expiration.
func NewScopedCache ¶ added in v2.3.0
func NewScopedCache[T any](capacity int, ttl time.Duration) *ScopedCache[T]
NewScopedCache creates a new ScopedCache with a maximum capacity and TTL.
func (*ScopedCache[T]) Clear ¶ added in v2.3.0
func (c *ScopedCache[T]) Clear()
Clear empties the cache completely.
func (*ScopedCache[T]) Get ¶ added in v2.3.0
func (c *ScopedCache[T]) Get(key string) (T, bool)
Get retrieves a value from the cache if present and not expired.
func (*ScopedCache[T]) Len ¶ added in v2.3.0
func (c *ScopedCache[T]) Len() int
Len returns the current number of items in the cache.
func (*ScopedCache[T]) PurgePrefix ¶ added in v2.3.0
func (c *ScopedCache[T]) PurgePrefix(prefix string)
PurgePrefix removes all entries whose key starts with the given prefix.
func (*ScopedCache[T]) Set ¶ added in v2.3.0
func (c *ScopedCache[T]) Set(key string, value T)
Set adds or updates a value in the cache.
Source Files