retrieval

package
v2.4.13 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 8, 2026 License: MIT Imports: 23 Imported by: 0

Documentation

Overview

Package retrieval implements hybrid, multi-signal, and adaptive retrieval pipelines for Cortex.

Package retrieval implements hybrid, multi-signal, and adaptive retrieval pipelines for Cortex.

Package retrieval implements hybrid, multi-signal, and adaptive retrieval pipelines for Cortex.

Package retrieval provides shared retrieval-pipeline orchestration that bridges the domain.VectorIndex port (W8.1, ADR-05) to the full search-result types that MCP, bench, CLI, and TUI consumers require.

The VectorIndex port returns lightweight VectorCandidate results carrying only an observation ID and a similarity score. Consumers need full observation data to format responses and to fuse vector results with FTS5 results via Reciprocal Rank Fusion. This package centralizes that post-fetch orchestration so it is NOT duplicated across consumer packages.

Dependency direction: this package imports ONLY internal/domain. It defines a narrow ObservationLookup interface (satisfied structurally by every concrete observation store) so it never reaches into a store package. Both internal/mcp and bench/locomo import this package; neither duplicates the helpers anymore.

The functions here are a pure extraction of logic that was previously duplicated verbatim in internal/mcp/tools_cortex.go and bench/locomo/runner.go. The extraction preserves byte-for-byte behavior: same RRF constant (k=60), same 1-based rank indexing, same tie-breaking (sort.Slice, NOT stable — matching the original), same soft-delete drop discipline.

Index

Constants

View Source 
const AgentCRAGMaxQueryRunes = 320
View Source 
const PostFilterPoolMultiplier = 3

PostFilterPoolMultiplier is the factor by which the retrieval pool is expanded when the adapter declares PostFilter or none. A multiplier of 3 gives in-engine filtering enough headroom to recover candidates the adapter's post-filter removed while keeping the pool bounded. This is a heuristic; the engine truncates to the requested limit after in-engine filtering.

Variables

View Source 
var ErrInvalidAgentCRAGScore = errors.New("agent CRAG score must be finite and normalized")
View Source 
var ErrInvalidAgentScore = errors.New("invalid agent retrieval score")

ErrInvalidAgentScore marks a score that cannot safely participate in retrieval ranking or confidence evaluation.

Functions

func ApplyRecognitionFilter added in v2.4.0 

func ApplyRecognitionFilter(results []*domain.SearchResult, cfg RecognitionConfig) []*domain.SearchResult

ApplyRecognitionFilter prunes post-PPR candidates below the recognition floors, then refills from the strongest pruned candidates so the requested TopK is always filled (and never exceeded), preserving descending-score order with ties broken by descending ID to mirror the RRF contract.

func ComputeMaxSimScore added in v2.3.0 

func ComputeMaxSimScore(queryTokens, docTokens []string) float64

ComputeMaxSimScore computes the ColBERT-style Late-Interaction MaxSim score between query and document.

MaxSim(Q, D) = sum_{q in Q} w(q) * max_{d in D} sim(q, d) / sum_{q in Q} w(q)

func ComputeWeightedMaxSimScore added in v2.3.8 

func ComputeWeightedMaxSimScore(queryTokens, docTokens []string) float64

ComputeWeightedMaxSimScore computes the specificity-weighted ColBERT MaxSim score.

func ExpandQuerySynonyms added in v2.3.0 

func ExpandQuerySynonyms(query string) string

ExpandQuerySynonyms returns an OR-expanded query string containing relevant synonyms.

func FuseResults 

func FuseResults(ftsResults []*domain.SearchResult, vecResults []*domain.VectorSearchResult, limit int) []*domain.SearchResult

FuseResults combines FTS5 full-text search results with vector similarity search results using Reciprocal Rank Fusion (k=60).

Preserves byte-for-byte behavior for all callers by delegating to FuseResultsWithOptions.

func FuseResultsWithNotes added in v2.4.0 

func FuseResultsWithNotes(ftsResults []*domain.SearchResult, vecResults []*domain.VectorSearchResult, noteRanked []*domain.SearchResult, opts FuseOptions) []*domain.SearchResult

FuseResultsWithNotes fuses FTS5, vector, and Chain-of-Note ranked lists (REQ-LME-001). The note-ranked list is a third accumulate input: its contribution is position-only — a note's Relevance score NEVER becomes an RRF rank input (score-as-rank pin, REQ-RET-002). Temporal decay stays a final multiplicative pass over the accumulated scores.

func FuseResultsWithOptions added in v2.3.8 

func FuseResultsWithOptions(ftsResults []*domain.SearchResult, vecResults []*domain.VectorSearchResult, opts FuseOptions) []*domain.SearchResult

FuseResultsWithOptions combines FTS5 full-text search results with vector similarity search results using Reciprocal Rank Fusion (k=60) with support for configurable exponential temporal decay (Time-Decayed RRF) and lexical/vector weighting.

func NormalizeQuery added in v2.3.0 

func NormalizeQuery(query string) string

NormalizeQuery cleans and corrects common typos in the search query.

func ReRankWithLateInteraction added in v2.3.0 

func ReRankWithLateInteraction(query string, results []*domain.SearchResult) []*domain.SearchResult

ReRankWithLateInteraction re-ranks search results using ColBERT-inspired Late-Interaction MaxSim.

func RefineAgentCRAGQuery added in v2.3.0 

func RefineAgentCRAGQuery(query string) string

func RevalidateCandidates 

func RevalidateCandidates(ctx context.Context, obs ObservationLookup, candidates []domain.VectorCandidate) []*domain.VectorSearchResult

RevalidateCandidates converts lightweight VectorCandidate results (ID + score from a domain.VectorIndex) into full VectorSearchResult entries by looking up the observation data via the provided ObservationLookup.

Candidates whose observation cannot be loaded (soft-deleted, missing, store error) are DROPPED — the same revalidation discipline the store-layer pipeline applies to fused candidates. A nil observation from the store is treated the same as an error: the candidate is dropped.

Batch fast path (VEC-01): when obs also implements BatchObservationLookup, the unique candidate IDs are hydrated with ONE GetByIDs call and the results are rebuilt by iterating the original candidate sequence. If the batch call fails, the error is swallowed and the unchanged per-ID loop runs instead — outputs are byte-equivalent either way.

The output preserves the INPUT ORDER of candidates (NOT re-sorted by score). Callers that need score-sorted output should sort the returned slice or rely on FuseResults, which re-sorts via RRF.

func SearchVectors 

func SearchVectors(ctx context.Context, idx domain.VectorIndex, q domain.VectorQuery, obs ObservationLookup) ([]*domain.VectorSearchResult, error)

SearchVectors executes a vector similarity search with capability-driven strategy selection. It reads idx.Capabilities, selects the appropriate filter strategy, retrieves candidates, revalidates them against the live observation store, applies in-engine filter safety-net when needed, and truncates to the requested limit.

Returns full VectorSearchResult entries (observation + similarity score). Soft-deleted, missing, or filter-mismatched candidates are dropped.

func TokenSimilarity added in v2.3.0 

func TokenSimilarity(a, b string) float64

TokenSimilarity calculates token-level string overlap similarity (Jaro-Winkler / trigram-like).

func TokenSpecificityWeight added in v2.3.8 

func TokenSpecificityWeight(token string) float64

TokenSpecificityWeight computes the information-theoretic importance weight of a token. Common stop words receive reduced weight, while technical identifiers, tokens containing underscores or digits, and longer terms receive elevated weights.

func TokenizeLateInteraction added in v2.3.0 

func TokenizeLateInteraction(text string) []string

TokenizeLateInteraction cleans and splits text into normalized word and subword tokens.

Types

type AdaptiveSearchOptions added in v2.3.0 

type AdaptiveSearchOptions struct {
	Mode string // "auto", "direct", "semantic", "multi_hop"
	// FusionScores are the RRF fusion scores of a prior retrieval stage. When
	// present and well-formed they drive SkewRoute-style tier routing in auto
	// mode; otherwise the heuristic classifier runs unchanged.
	FusionScores []float64
	// QueryVector is the query embedding for the multi-hop tier's dense leg.
	// When empty the tier seeds propagation from lexical evidence only and
	// omits the dense fusion input instead of failing.
	QueryVector []float32
	Project     string
	Scope       string
	Types       []string
	Limit       int
	GraphNodes  []graph.GraphAnalyticsNode
	GraphEdges  []graph.GraphAnalyticsEdge
	CRAGConfig  *CRAGConfig
}

AdaptiveSearchOptions controls the adaptive retrieval engine.

type AdaptiveSearchResult added in v2.3.0 

type AdaptiveSearchResult struct {
	Tier            QueryTier              `json:"tier"`
	Confidence      ConfidenceGrade        `json:"confidence"`
	ConfidenceScore float64                `json:"confidence_score"`
	NeedsRefinement bool                   `json:"needs_refinement"`
	Results         []*domain.SearchResult `json:"results"`
}

AdaptiveSearchResult represents the enriched search output with RAG metadata.

func ExecuteAdaptiveSearch added in v2.3.0 

func ExecuteAdaptiveSearch(
	ctx context.Context,
	query string,
	opts AdaptiveSearchOptions,
	lexicalSearch func(ctx context.Context, q domain.SearchOptions) ([]*domain.SearchResult, error),
	vectorSearch func(ctx context.Context, q domain.VectorQuery) ([]*domain.VectorSearchResult, error),
) (*AdaptiveSearchResult, error)

ExecuteAdaptiveSearch runs the adaptive RAG pipeline, dynamically selecting between direct lexical search, semantic hybrid vectors, and HippoRAG graph propagation.

type AgentScore added in v2.3.0 

type AgentScore struct {
	Signal       AgentScoreSignal
	Raw          *float64
	Normalized   float64
	SourceKind   string
	PublicHandle string
}

AgentScore carries one native retrieval signal and its stable public identity. Raw is a pointer so a structurally missing score cannot be confused with the valid value zero.

func NormalizeAgentScores added in v2.3.0 

func NormalizeAgentScores(scores []AgentScore) ([]AgentScore, error)

NormalizeAgentScores validates and normalizes scores, then orders them by descending normalized relevance. Bounded dense, MaxSim, PPR and summary inputs are clamped to one. Non-negative unbounded lexical inputs use x/(1+x) (expressed in an overflow-safe form). Missing, negative and non-finite values are rejected instead of receiving a confidence-bearing default.

Equal scores are ordered by source kind and public handle. Stable sorting preserves input order only when both public tie breakers are identical.

type AgentScoreSignal added in v2.3.0 

type AgentScoreSignal string

AgentScoreSignal identifies the native scale used by one retrieval signal. 

const (
	AgentScoreLexical AgentScoreSignal = "lexical"
	AgentScoreDense   AgentScoreSignal = "dense"
	AgentScoreMaxSim  AgentScoreSignal = "maxsim"
	AgentScorePPR     AgentScoreSignal = "ppr"
	AgentScoreSummary AgentScoreSignal = "summary"
)

type BatchObservationLookup 

type BatchObservationLookup interface {
	ObservationLookup
	GetByIDs(ctx context.Context, ids []int64) (map[int64]*domain.Observation, error)
}

BatchObservationLookup is the OPTIONAL batch-capable superset of ObservationLookup (VEC-01). RevalidateCandidates detects it via type assertion and, when hydration succeeds, replaces the per-candidate N+1 GetByID loop with a single GetByIDs call over the unique candidate IDs.

It is deliberately retrieval-local: it is NOT added to domain.ObservationRepository (which has multiple implementors), so stores opt in simply by exposing the method — *sqlite.Store does. A lookup that does not implement it keeps the exact legacy per-ID behavior.

GetByIDs contract:

  • Empty/nil ids MUST issue no SQL and return an empty map.
  • Live rows MUST be keyed by observation ID.
  • Soft-deleted and missing IDs MUST be absent from the map (or mapped to nil), which the engine treats as a drop — identical legacy semantics.
  • Rows for IDs that were not requested MAY be present and are ignored.

type CRAGConfig added in v2.3.0 

type CRAGConfig struct {
	HighThreshold float64 // typically 0.65
	LowThreshold  float64 // typically 0.30
	MinScoreFloor float64 // noise floor, results below this are stripped
	DynamicMode   bool    // enables dynamic score gap & relative elbow calibration (for RRF/dense distributions)
}

CRAGConfig defines thresholds for Corrective RAG evaluation and filtering.

func DefaultCRAGConfig added in v2.3.0 

func DefaultCRAGConfig() CRAGConfig

DefaultCRAGConfig returns standard CRAG evaluation parameters.

func DynamicCRAGConfig added in v2.3.8 

func DynamicCRAGConfig() CRAGConfig

DynamicCRAGConfig returns CRAG configuration optimized for dynamic/RRF score distributions.

type CRAGEvaluation added in v2.3.0 

type CRAGEvaluation struct {
	Grade           ConfidenceGrade        `json:"grade"`
	Confidence      float64                `json:"confidence"`
	NeedsRefinement bool                   `json:"needs_refinement"`
	FilteredResults []*domain.SearchResult `json:"filtered_results"`
}

CRAGEvaluation encapsulates the confidence evaluation of retrieved results.

func EvaluateCRAG added in v2.3.0 

func EvaluateCRAG(results []*domain.SearchResult, cfg CRAGConfig) CRAGEvaluation

EvaluateCRAG evaluates retrieved search results against CRAG confidence thresholds, filtering out noisy or irrelevant low-scoring candidates to protect downstream agents from hallucinations.

type CacheEntry added in v2.3.0 

type CacheEntry[T any] struct {
	// contains filtered or unexported fields
}

CacheEntry holds a cached value along with its expiration timestamp.

type ConfidenceGrade added in v2.3.0 

type ConfidenceGrade string

ConfidenceGrade represents the categorical confidence of retrieved context. 

const (
	ConfidenceGradeHigh   ConfidenceGrade = "high"
	ConfidenceGradeMedium ConfidenceGrade = "medium"
	ConfidenceGradeLow    ConfidenceGrade = "low"
)

func EvaluateAgentCRAG added in v2.3.0 

func EvaluateAgentCRAG(scores []float64, cfg CRAGConfig) (ConfidenceGrade, error)

type DefaultLateInteractionReranker added in v2.4.0 

type DefaultLateInteractionReranker struct{}

DefaultLateInteractionReranker is the zero-network local default. It delegates to ReRankWithLateInteraction so the revived 60/40 rank/MaxSim blend keeps exactly one source of ordering semantics.

func (DefaultLateInteractionReranker) Rerank added in v2.4.0 

func (DefaultLateInteractionReranker) Rerank(query string, docs []*domain.SearchResult) ([]*domain.SearchResult, error)

Rerank reorders docs with the late-interaction blend and never fails.

type FuseOptions added in v2.3.8 

type FuseOptions struct {
	// Limit is the maximum number of fused search results to return.
	Limit int
	// DecayHalfLifeDays specifies the half-life in days for exponential recency decay.
	// When <= 0, no temporal decay is applied (standard RRF).
	DecayHalfLifeDays float64
	// ReferenceTime is the anchor time for decay calculation. If zero, time.Now() is used.
	ReferenceTime time.Time
	// LexicalWeight specifies the multiplier for FTS5 lexical score (defaults to 1.0 if <= 0).
	LexicalWeight float64
	// VectorWeight specifies the multiplier for dense vector similarity score (defaults to 1.0 if <= 0).
	VectorWeight float64
	// NoteWeight specifies the multiplier for the Chain-of-Note ranked list
	// (defaults to 1.0 if <= 0). Note relevance never becomes an RRF input —
	// only the note-ranked list POSITION contributes (REQ-LME-001).
	NoteWeight float64
}

FuseOptions configures Reciprocal Rank Fusion and optional temporal decay.

type FusionRouteFeatures added in v2.4.0 

type FusionRouteFeatures struct {
	Skewness   float64 `json:"skewness"`
	Top1Margin float64 `json:"top1_margin"`
	Available  bool    `json:"available"`
}

FusionRouteFeatures holds training-free routing signals derived from an RRF fusion score distribution (SkewRoute style, REQ-ROUTE-001).

func ComputeFusionFeatures added in v2.4.0 

func ComputeFusionFeatures(scores []float64) FusionRouteFeatures

ComputeFusionFeatures derives training-free SkewRoute routing signals from an RRF fusion score distribution: Pearson skewness of the score distribution and the relative top-1 margin (s1 - s2) / s1. Scores are copied before sorting so the caller's slice is never reordered. Non-finite scores or fewer than two candidates yield unavailable features, which callers treat as "fall back to the heuristic".

type ObservationLookup 

type ObservationLookup interface {
	GetByID(ctx context.Context, id int64) (*domain.Observation, error)
}

ObservationLookup is the observation-store subset needed for candidate revalidation. Every concrete observation store (*sqlite.Store, test fakes, any domain.ObservationRepository) satisfies this structurally. Defining the narrow interface here keeps this package free of any store import while remaining compatible with every backend.

type PassageNote added in v2.4.0 

type PassageNote struct {
	ObservationID int64
	Note          string
	Relevance     float64
	AnswerBearing bool
}

PassageNote is one Chain-of-Note reading note for a candidate passage.

func ChainOfNoteStage added in v2.4.0 

func ChainOfNoteStage(query string, candidates []*domain.SearchResult) ([]PassageNote, []*domain.SearchResult)

ChainOfNoteStage reads every candidate passage (Title + TopicKey + Content), scores each sentence against the query with the late-interaction MaxSim scorer, and returns the per-passage notes plus the candidates re-ordered by note relevance (Relevance DESC, ID ASC, stable). An empty or untokenizable query returns (nil, candidates) untouched — no notes, no reordering.

type QueryTier added in v2.3.0 

type QueryTier string

QueryTier defines the complexity routing level in Adaptive-RAG. 

const (
	// TierDirectFactual represents direct lookups (symbols, IDs, exact keywords) ~1-5ms.
	TierDirectFactual QueryTier = "direct_factual"
	// TierSemanticHybrid represents conceptual queries combining lexical + dense vectors.
	TierSemanticHybrid QueryTier = "semantic_hybrid"
	// TierMultiHopGraph represents complex relational questions resolved via HippoRAG PPR.
	TierMultiHopGraph QueryTier = "multi_hop_graph"
	// TierArchitecturalGlobal represents macro architectural questions resolved via LightRAG Community Summaries.
	TierArchitecturalGlobal QueryTier = "architectural_global"
)

func ClassifyQueryComplexity added in v2.3.0 

func ClassifyQueryComplexity(query string) QueryTier

ClassifyQueryComplexity routes a query to the optimal retrieval tier in < 0.1ms.

func ClassifyQueryWithFusion added in v2.4.0 

func ClassifyQueryWithFusion(query string, features FusionRouteFeatures) QueryTier

ClassifyQueryWithFusion routes a query using fusion features when available, otherwise it falls back to the text heuristic unchanged.

type RecognitionConfig added in v2.4.0 

type RecognitionConfig struct {
	TopK          int     // requested final candidate count; <= 0 disables prune-only
	RelativeFloor float64 // keep candidates scoring at least RelativeFloor * top score
	MinScoreFloor float64 // absolute noise floor on Rank
}

RecognitionConfig configures the post-PPR recognition-memory filter that prunes weak candidates before final fusion while still filling the requested top-k.

func DefaultRecognitionConfig added in v2.4.0 

func DefaultRecognitionConfig() RecognitionConfig

DefaultRecognitionConfig returns standard recognition-memory parameters.

type RerankConfig added in v2.4.0 

type RerankConfig struct {
	Provider string
	Model    string
	BaseURL  string
}

RerankConfig is the resolved rerank preset handed to the factories. It has no API-key field by design: the key resolves only from CORTEX_RERANK_API_KEY at construction (REQ-CFG-001), so it can never be serialized into a config file or passed in from an untrusted source.

type Reranker added in v2.4.0 

type Reranker interface {
	Rerank(query string, docs []*domain.SearchResult) ([]*domain.SearchResult, error)
}

Reranker orders one fused candidate list for a query before the final limit is applied (REQ-RNK-001). Implementations preserve the input set exactly — the same *domain.SearchResult pointers with the same multiplicity — and order by descending score with ties broken by descending observation ID (the RRF convention in fuseInputs). A non-nil error means the returned slice must be discarded wholesale: partially reranked output is never valid.

func NewReranker added in v2.4.0 

func NewReranker(cfg RerankConfig) (Reranker, error)

NewReranker builds the local-mode reranker. The disabled default returns (nil, nil) so the local composition constructs zero machinery, and the remote preset is refused fail-closed: only NewSecureReranker may ever build the HTTP reranker.

func NewSecureReranker added in v2.4.0 

func NewSecureReranker(cfg RerankConfig, policy embedding.OutboundPolicy) (Reranker, error)

NewSecureReranker is the sole construction path for the HTTP reranker: the base URL must be approved on the shared embedding OutboundPolicy, and the API key resolves only from CORTEX_RERANK_API_KEY (missing => construction error). Local composition never reaches this dial.

type ScopedCache added in v2.3.0 

type ScopedCache[T any] struct {
	// contains filtered or unexported fields
}

ScopedCache is a generic, thread-safe LRU cache with TTL expiration.

func NewScopedCache added in v2.3.0 

func NewScopedCache[T any](capacity int, ttl time.Duration) *ScopedCache[T]

NewScopedCache creates a new ScopedCache with a maximum capacity and TTL.

func (*ScopedCache[T]) Clear added in v2.3.0 

func (c *ScopedCache[T]) Clear()

Clear empties the cache completely.

func (*ScopedCache[T]) Get added in v2.3.0 

func (c *ScopedCache[T]) Get(key string) (T, bool)

Get retrieves a value from the cache if present and not expired.

func (*ScopedCache[T]) Len added in v2.3.0 

func (c *ScopedCache[T]) Len() int

Len returns the current number of items in the cache.

func (*ScopedCache[T]) PurgePrefix added in v2.3.0 

func (c *ScopedCache[T]) PurgePrefix(prefix string)

PurgePrefix removes all entries whose key starts with the given prefix.

func (*ScopedCache[T]) Set added in v2.3.0 

func (c *ScopedCache[T]) Set(key string, value T)

Set adds or updates a value in the cache.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.