retrieve

package
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 8, 2026 License: MIT Imports: 17 Imported by: 0

Documentation

Index

Constants

View Source
const EvidenceGapMarker = "// ..."

EvidenceGapMarker separates non-adjacent windows in a trimmed body.

View Source
const WeakMatchRelevance = 0.70

WeakMatchRelevance is the cross-encoder score below which the top result is reported as a weak match regardless of how far it leads the rest. On the jina-tiny sigmoid scale; calibrated by cmd/confcal and pinned by a parity test — a different reranker needs a new calibration run.

Variables

View Source
var DefaultBodyFTSWeight float32 = 1.0

DefaultBodyFTSWeight scales the lossless-body channel, swept on prometheus deep-content recall (cmd/deepprobe) against the 20-repo gate:

0.0  recall 0.47  (channel off)
0.8  recall 0.53
1.0  recall 0.68  gen corpus R@5 0.88->0.90; cockroach Hit@10 0.96->0.93
1.5  recall 0.72  breaks Hit@1 outright (prometheus 0.93->0.88)

DECISION(2026-08): ship 1.0. A controlled 0.8-vs-1.0 gate — same binary, the weight the only difference — returns bit-identical false confidence on all twenty repos, and confcal shows the score distributions barely move (negatives p50 0.312 either way), so the channel does not disturb the confidence landscape. Hit@1 and Hit@5 are flat within a case in both directions; the one real cost is cockroach Hit@10, and the served default returns five results, not ten. ASSUMES: max_results stays small enough that rank 6-10 placements are not what the agent reads. REVISIT IF: a repo shows false confidence moving with this weight rather than with a code change.

View Source
var DefaultChunkVecWeight float32 = 0

DefaultChunkVecWeight scales the chunk-vector channel, and 0 switches the feature off end to end: no chunks are built at index time and no KNN runs per query. CONTEXTMAXXER_CHUNK_VEC_WEIGHT turns it on.

DECISION(2026-08): ship it off. The channel is the semantic counterpart of the body FTS one — the per-symbol vector is built from the capped excerpt, so code past the cap has no vector at all — but the benefit did not survive measurement: prometheus deep recall 0.68 -> 0.73 (41 -> 44 of 60) against cockroach 0.60 -> 0.58 (36 -> 35), which is +2 cases in 120. The costs are not in doubt: ~30% longer indexing (cockroach 1938s), +4-7% index size, a full reindex to benefit at all, +2% query latency.

ASSUMES: the null result is real and not an artifact of how it was measured. That assumption is weak — cmd/deepprobe builds its query as a bag of identifiers from one code line, which is a lexical query, and a semantic channel should earn its keep on paraphrases that the probe never generates. REVISIT IF: a paraphrastic deep-content case set exists (an LLM rewriting probe lines as questions would do it), or the embedder changes — ft2 moved recall on large corpora where the base model was saturated.

View Source
var DefaultFTSWeight float32 = 1.0

DefaultFTSWeight scales the FTS list's contribution in hybrid RRF fusion (1.0 = classic equal-weight RRF). Package-level so eval can sweep it; promote to a Request field once the value is settled.

View Source
var DefaultIdentFTSWeight float32 = 0

The identifier channel's knobs. Off by default: it is a measured hypothesis, not a shipped default, and CONTEXTMAXXER_IDENT_FTS_WEIGHT turns it on the way CONTEXTMAXXER_CHUNK_VEC_WEIGHT does for the chunk-vector channel.

Functions

func LooksLikeLocator

func LooksLikeLocator(query string) bool

LooksLikeLocator reports whether a query is written as a position rather than a sentence. Exported so adoption reporting classifies queries by the same rule the pipeline routes them with: two copies of this question disagreeing is how a measurement ends up describing something the product does not do.

It answers the shape only. Whether the path resolves is a question for the index, and the pipeline asks it separately.

func PersonalizedPageRank

func PersonalizedPageRank(g *Graph, seeds map[int64]float32, damping float32, maxIter int) (map[int64]float32, int)

PersonalizedPageRank runs PPR with the given seed distribution. Seeds map symbolID → initial weight (need not sum to 1). damping=0.85, maxIter=30 are standard choices. Returns map symbolID → rank score (scores sum ≈ 1.0).

Types

type BodySegment

type BodySegment struct {
	StartLine int
	Lines     int
}

BodySegment is one contiguous window of a trimmed body.

type Embedder

type Embedder interface {
	Embed(ctx context.Context, texts []string) ([][]float32, error)
}

type FlowRef

type FlowRef struct {
	QualifiedName string
	File          string
	Lines         string
	Kind          string
	Distance      int
	Via           string
}

type Graph

type Graph struct {
	NumNodes int
	NodeIdx  map[int64]int
	NodeIDs  []int64
	OutEdges [][]int
}

func BuildGraph

func BuildGraph(symbolIDs []int64, edges []store.Edge) *Graph

DECISION: using undirected edges (both A→B and B→A per edge) for retrieval. Callers are equally relevant context as callees — if you query a helper function, you want to see its callers too. Bidirectionality makes PPR seeds propagate through the full local neighbourhood rather than only downstream.

type IntentRanker

type IntentRanker interface {
	Ranker
	IsIntentRanker()
}

IntentRanker is a marker interface implemented by the symbolic intent ranker.

type Mode

type Mode int
const (
	ModeHybrid     Mode = iota
	ModeVectorOnly Mode = 1
)

type NextStepsHints

type NextStepsHints struct {
	IfTopCorrect string
	IfUnsure     string
	ToExplore    []string
}

type OutputMode

type OutputMode int
const (
	OutputModeAnswer  OutputMode = iota
	OutputModeMinimal OutputMode = 1
	OutputModeExplore OutputMode = 2
)

func ParseOutputMode

func ParseOutputMode(s string) (OutputMode, error)

func (OutputMode) String

func (m OutputMode) String() string

type Ranker

type Ranker interface {
	Rank(query string, candidates []ScoredResult) []ScoredResult
}

type RankingFeatures

type RankingFeatures struct {
	VectorSeed       float32
	FTSSeed          float32
	PPR              float32
	EffectiveAlpha   float32
	SeedScore        float32
	PPRScore         float32
	SeedRank         int
	PPRRank          int
	ShortNameOverlap float32
	NameOverlap      float32
	PathOverlap      float32
	KindOverlap      float32
	SignatureOverlap float32
	BodyOverlap      float32
	IsConstructor    float32
	KindFunction     float32
	KindMethod       float32
	HubPenalty       float32
}

func (RankingFeatures) AsMap

func (f RankingFeatures) AsMap() map[string]float32

AsMap flattens the feature vector for logging. It is the single source of truth for which ranking signals reach the feedback log (and thus a future learned ranker) — add a field here when you add one to the pipeline.

type Request

type Request struct {
	Query        string
	BudgetTokens int
	SeedK        int
	MaxResults   int
	RerankK      int
	// AnchorExpand adds this many 1-hop graph neighbours of the top seeds to the
	// candidate pool (0 = off).
	AnchorExpand int
	// TestFloor reserves this many top answer slots for non-test files (0 = off).
	// Tests keep their order behind them, so they fill what implementation
	// candidates leave empty. See internal/retrieve/testfloor.go.
	TestFloor int
	// LiteralSlots hands this many of the last answer slots to files where
	// several of the query's identifiers occur together (0 = off). Worth
	// switching on when the index holds test files; see internal/retrieve/literal.go.
	LiteralSlots   int
	AdaptiveRerank bool
	// LazyRerank inverts the reranking default: the cross-encoder runs only
	// when the fused ranking is ambiguous (small gap / several near-ties),
	// instead of on every query.
	LazyRerank bool
	// FullBodyResults caps how many top results keep their full body in the
	// packed response (0 = default 3, negative = all results keep bodies).
	FullBodyResults int
	// PreserveFullBodies disables query-relevant evidence trimming. Serving
	// paths set it only for an explicit full_bodies compatibility override;
	// normal navigation uses excerpts and expand_context for exact hydration.
	PreserveFullBodies bool
	Mode               Mode
	OutputMode         OutputMode
	Alpha              float32
	AlphaSet           bool
	IncludeTrivial     bool
	SkipRerank         bool
	SkipIntent         bool
}

type Reranker

type Reranker interface {
	Rerank(ctx context.Context, query string, candidates []ScoredResult) ([]ScoredResult, error)
}

type Result

type Result struct {
	Symbols         []ScoredResult
	TotalTokens     int
	Stats           Stats
	Structure       *StructureView
	NextSteps       *NextStepsHints
	RetrievalHealth *RetrievalHealth
	// ExpansionSymbols is an internal snapshot of the selected ranked symbols
	// before tiering and evidence trimming. MCP caches it by request_id so
	// expand_context can hydrate an already-found symbol without rerunning
	// embedding, graph ranking, or reranking.
	ExpansionSymbols []ScoredResult
}

type RetrievalHealth

type RetrievalHealth struct {
	CandidatesSeen int
	TopScoreGap    float32
	TiedCandidates int
	Confidence     string
	Suggestion     string
}

type Retriever

type Retriever struct {
	// contains filtered or unexported fields
}

func NewRetriever

func NewRetriever(s Store, e Embedder, log *slog.Logger) *Retriever

func NewRetrieverWithRankers

func NewRetrieverWithRankers(s Store, e Embedder, reranker Reranker, ranker Ranker, log *slog.Logger) *Retriever

func NewRetrieverWithReranker

func NewRetrieverWithReranker(s Store, e Embedder, reranker Reranker, log *slog.Logger) *Retriever

func (*Retriever) GetSymbolBody

func (r *Retriever) GetSymbolBody(ctx context.Context, symbolID int64) (store.SymbolBody, error)

func (*Retriever) Retrieve

func (r *Retriever) Retrieve(ctx context.Context, req Request) (Result, error)

func (*Retriever) SetEscalator

func (r *Retriever) SetEscalator(esc Reranker)

SetEscalator installs a stronger fallback reranker used only for low-confidence queries. Nil disables escalation.

type ScoredResult

type ScoredResult struct {
	SymbolID      int64
	File          string
	QualifiedName string
	Kind          string
	Signature     string
	Docstring     string
	StartLine     int
	EndLine       int
	Score         float32
	Body          string
	Why           string
	AlsoVia       []string
	Features      RankingFeatures

	// Relevance is the cross-encoder's own score (sigmoid, 0..1) for this
	// query/document pair — set only when the reranker ran. Score, by
	// contrast, accumulates ranking bonuses (intent priors add up to ~+4),
	// which makes it useless as an absolute "is this actually a match"
	// signal. 0 means "no cross-encoder verdict" (rerank skipped or failed).
	Relevance float32

	// Detail marks how much of the symbol the packed Body carries:
	// "full" (complete indexed body), "excerpt" (a query-relevant source
	// window), or "compact" (signature + doc line).
	Detail string

	// BodyStartLine is the real file line number of the first line of Body. It
	// equals StartLine unless an evidence-span trim moved the body window down,
	// in which case it lets the output number the trimmed lines correctly.
	BodyStartLine int
	// BodyEndLine is the real file line number of the last visible Body line.
	// It differs from EndLine when Detail is "excerpt".
	BodyEndLine int

	// CallersTotal and CalleesTotal are how many graph edges exist before the
	// display cap. Showing five of thirty without saying so is how the hop an
	// agent came for goes missing in silence.
	CallersTotal int
	CalleesTotal int

	// BodySegments describes the windows Body carries when the evidence trim
	// keeps more than one. Body joins them with evidenceGapMarker; the segments
	// carry each window's real first line so numbering stays honest across the
	// gap. Empty means one contiguous span starting at BodyStartLine.
	BodySegments []BodySegment

	Confidence string
	Callers    []SymbolRef
	Callees    []SymbolRef

	Visibility     string
	Tests          []SymbolRef
	Siblings       []SymbolRef
	FlowContext    []FlowRef
	CompanionFiles []string
}

func Pack

func Pack(symbols []ScoredResult, budgetTokens int, fullBodyCount int) ([]ScoredResult, int)

Pack preserves ranking order and selects symbols until budget is exhausted. DECISION(2026-06): tiered packing — only the top fullBodyCount results carry the full body; the tail is compacted to signature + first docstring line (~10-20x cheaper per result). The consumer goal is information per token: a correct answer at rank 5 should cost the agent a glance, not a screenful. fullBodyCount < 0 disables tiering (every result keeps its body).

type Stats

type Stats struct {
	SeedCount         int
	GraphNodes        int
	GraphEdges        int
	PPRIterations     int
	EmbedDuration     time.Duration
	SeedDuration      time.Duration
	GraphDuration     time.Duration
	PPRDuration       time.Duration
	RerankDuration    time.Duration
	IntentDuration    time.Duration
	EscalateDuration  time.Duration
	Escalated         bool
	RerankLazySkipped bool
	PackDuration      time.Duration
	EvidenceDuration  time.Duration
	// GraphCtxDuration covers the callers/callees/tests/siblings SQL
	// enrichment of the selected results (not the PPR graph itself).
	GraphCtxDuration time.Duration
	Total            time.Duration
	EffectiveAlpha   float32
}

type Store

type Store interface {
	SearchByVectorScored(ctx context.Context, vec []float32, k int) ([]store.ScoredSymbol, error)
	SearchByText(ctx context.Context, query string, k int) ([]store.ScoredSymbol, error)
	SearchByBodyText(ctx context.Context, query string, k int) ([]store.ScoredSymbol, error)
	SearchByChunkVector(ctx context.Context, vec []float32, k int) ([]store.ScoredSymbol, error)
	GetSymbolsByIDs(ctx context.Context, ids []int64) ([]store.Symbol, error)
	GetSymbolBody(ctx context.Context, symbolID int64) (store.SymbolBody, error)
	GetFilesByIDs(ctx context.Context, ids []int64) (map[int64]string, error)
	ListAllSymbolIDs(ctx context.Context) ([]int64, error)
	ListSymbolMeta(ctx context.Context) ([]store.Symbol, error)
	ListAllEdges(ctx context.Context) ([]store.Edge, error)
	GetEmbeddingsByIDs(ctx context.Context, ids []int64) (map[int64][]float32, error)
	GetCallerEdges(ctx context.Context, dstIDs []int64, limitPerSymbol int) (map[int64][]int64, error)
	GetCalleeEdges(ctx context.Context, srcIDs []int64, limitPerSymbol int) (map[int64][]int64, error)
}

type StructureView

type StructureView struct {
	Summary     string
	PackageView map[string][]string
}

type SymbolRef

type SymbolRef struct {
	QualifiedName string
	File          string
	Lines         string
	Kind          string
	// CallLine is the file line of the call site: for a callee, where the result
	// symbol calls it; for a caller, where that caller calls the result symbol.
	// 0 when not located. Lets the agent follow a call chain without opening files.
	CallLine int
	// PathStatus is "static_unverified" for call-graph edges. Extractors prove
	// that the call exists in source, not that its surrounding branch executes.
	PathStatus string
	// CallSite is a bounded, line-numbered source window around CallLine. It is
	// populated for top-result graph refs so agents can inspect nearby branch or
	// dispatch conditions without paying for every related symbol body.
	CallSite string
}

type TextScorer

type TextScorer interface {
	ScoreTexts(ctx context.Context, query string, docs []string) ([]float32, error)
}

TextScorer is an optional Reranker capability: score free-form texts against the query. Evidence-window selection uses it when the reranker offers it, because a cross-encoder reads the query and the window together, while the bi-encoder can only compare two vectors built in ignorance of each other.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL