Affected by 
Documentation
¶
Overview ¶
Package reranking is the single rerank stage shared by the chat pipeline, the agent search_knowledge tool and the retrieval APIs: passage building, model scoring, threshold filtering with degradation, composite scoring and MMR selection.
Index ¶
- Constants
- func CleanPassage(text string) string
- func CompositeScore(r *types.SearchResult, modelScore, baseScore float64) float64
- func EnrichedPassage(ctx context.Context, result *types.SearchResult) string
- func FallbackMinScore(explicitScope bool) float64
- func ImageKeepScoreFor(vectorThreshold float64) float64
- func ModelPassage(ctx context.Context, result *types.SearchResult) string
- func PreBoostScore(r *types.SearchResult) float64
- func SelectMMR(ctx context.Context, results []*types.SearchResult, k int, lambda float64) []int
- type Options
- type Result
Constants ¶
const ( // DefaultThreshold is the minimum rerank score used when no configuration // supplies one. It matches RetrievalConfig.GetEffectiveRerankThreshold. DefaultThreshold = 0.2 // DefaultFallbackMinScore is the score the best candidate needs to be kept // when nothing passes the threshold. Below it, returning nothing beats // forcing an irrelevant result on the caller. DefaultFallbackMinScore = 0.15 // DefaultMaxCandidates bounds how many candidates the chat pipeline and // the agent tool send to the rerank model. Query expansion and // per-document or per-tag targets each add a full retrieval list, and // every candidate is a billed passage and a share of the latency. DefaultMaxCandidates = 200 )
const CompositeScoreKey = "composite_score"
CompositeScoreKey is the Metadata key holding a reranked row's composite score before any boost (FAQ boost here, wiki or memory boosts later in the chat pipeline) changes Score.
const DefaultMMRLambda = 0.7
DefaultMMRLambda weighs relevance against redundancy in SelectMMR.
const ( // MinImageKeepScore is the least vector similarity that keeps a pictorial // image hit over a text reranker. It sits above the 0.2 image recall // threshold: recall only lets an image compete, keeping it needs more. // Empirical, on the cosine scale most engines report. MinImageKeepScore = 0.25 )
An image found by its own vector is judged by what it shows. A reranker that reads images is sent the image. A text reranker only sees the caption and OCR text, which speak for the image when the image is mostly text, but not when it is a photo, chart or diagram the caption can only summarize: there its rejection means "the caption does not say", not "irrelevant".
Variables ¶
This section is empty.
Functions ¶
func CleanPassage ¶
CleanPassage strips markdown/structural noise from text to produce a clean semantic passage for the rerank model. The cleaning is designed to preserve all meaningful semantic content while removing formatting that would confuse text-similarity scoring.
func CompositeScore ¶
func CompositeScore(r *types.SearchResult, modelScore, baseScore float64) float64
CompositeScore blends the rerank model score with the retrieval score and a source weight, clamped to [0, 1]. Graph hits carry no retrieval score (they come from entity lookups, not similarity search), so the model score stands in for it rather than a made-up constant.
func EnrichedPassage ¶
func EnrichedPassage(ctx context.Context, result *types.SearchResult) string
EnrichedPassage is the cleaned chunk body plus the image captions, OCR text and generated questions attached to it. MMR compares candidates on this text, so chunks of one document are not made similar by a shared title.
func FallbackMinScore ¶
FallbackMinScore returns the score the best candidate needs to survive an all-rejecting threshold. When the caller explicitly scoped the search to a tag or document set, its best candidate is kept regardless, rather than letting a global threshold erase the whole authoritative scope. Model scores can be negative, so "regardless" is -Inf rather than 0.
func ImageKeepScoreFor ¶
ImageKeepScoreFor is Options.ImageKeepScore for a search whose text hits needed vectorThreshold: an image is kept over a text reranker only when its vector was at least as close as a text hit had to be, and never below MinImageKeepScore.
func ModelPassage ¶
func ModelPassage(ctx context.Context, result *types.SearchResult) string
ModelPassage is the text the rerank model scores: the document title followed by the enriched chunk. A chunk rarely restates what its document is about, so without the title a passage from "Show HN: Echo - ... using open-weight models" scored 0.002 against "Echo open-weight models reduce cost" and 0.45 with it. FAQ entries carry their own question instead.
An empty enriched body stays empty: a title alone is not evidence that the chunk answers anything.
The chunk's heading breadcrumb (ContextHeader) goes between the two for the same reason: it was embedded with the chunk, so vector search can find "7 天内可申请" under "## 退款政策" while a model scoring the bare body rejects it.
func PreBoostScore ¶
func PreBoostScore(r *types.SearchResult) float64
PreBoostScore returns the score to compare against absolute thresholds such as the FAQ direct-answer threshold: the composite score before any boost when the row was reranked, and its retrieval score otherwise.
Types ¶
type Options ¶
type Options struct {
// Threshold is the minimum model score a candidate needs.
Threshold float64
// TopK, when positive, MMR-selects at most TopK results. Otherwise every
// candidate that passed the threshold is returned, best first.
TopK int
// FallbackMinScore is the score the best candidate needs to be kept when
// nothing passes the threshold. See FallbackMinScore().
FallbackMinScore float64
// MaxCandidates, when positive, reranks only the MaxCandidates rows with
// the highest retrieval score. Retrieval scores share one [0, 1] scale
// across searches, so the cut keeps the strongest candidates of each.
MaxCandidates int
// FAQScoreBoost, when above 1, multiplies the composite score of FAQ
// entries, capped at 1 so scores stay on the [0, 1] scale. FAQs tied at
// the cap are ordered by their pre-boost score. Compare absolute
// thresholds against PreBoostScore.
FAQScoreBoost float64
// LoadImage reads the image an image_vector hit shows. With it, a model
// that reads images scores those hits by their image; without it they
// are scored by their text like any other row.
LoadImage func(ctx context.Context, r *types.SearchResult) ([]byte, error)
// ImageKeepScore, when positive, keeps a pictorial image_vector hit a
// text reranker rejected if its VectorScore reaches this, at most
// maxKeptImages of them, after the ranked results and marked with
// types.MetadataKeptBy. A text reranker cannot judge what a caption leaves
// out; the vector can. Callers pass the text vector threshold or more.
// Kept results ride outside TopK; see types.MetadataKeptBy.
ImageKeepScore float64
}
Options configures one rerank run.
type Result ¶
type Result struct {
// Results are the returned rows, best first. They are copies carrying the
// composite score and base_score / model_score metadata; the input rows
// are never modified. On a model error they are the input rows unchanged.
Results []*types.SearchResult
// Indices[i] is the position of Results[i] in the input slice.
Indices []int
// Diagnostics summarises the run for API callers and logs.
Diagnostics types.RerankDiagnostics
// Candidates are the input rows with a non-empty passage, in input
// order; Passages are what the model scored for them. ModelScores index
// into Candidates. Scored are the rows that passed the threshold, with
// composite scores, before MMR. Kept for tracing.
Candidates []*types.SearchResult
Passages []string
ModelScores []rerank.RankResult
Scored []*types.SearchResult
}
Result is the outcome of a rerank run.
func Rerank ¶
func Rerank( ctx context.Context, model rerank.Reranker, query string, results []*types.SearchResult, opts Options, ) *Result
Rerank scores results against query with model and returns the rows that pass opts, best first. A failed model call is not an error: the result carries the input rows unchanged and a model_error outcome, so callers can degrade to the retrieval order.
Source Files