Documentation
¶
Overview ¶
Package bm25 provides candidate-set BM25 rerank for CKV's query path.
The Okapi implementation + code-aware tokenizer are adapted from the sibling repo github.com/0xmhha/code-knowledge-graph at pkg/bm25 (2026-05-26 snapshot). Per that package's header note, the algorithm is hand-written from the BM25 description (Robertson, Walker 1994) and is not derivative of any third-party library; we copy rather than depend across repos so the CKV build does not require a CKG checkout.
CKV-specific surface (this package, not in CKG):
- rerank.go candidate-set Rerank + RRF fusion (k=60 default)
CKV remains dense-only at the schema layer; this is a candidate-rerank overlay measured for impact alongside the vector-only baseline.
Index ¶
Constants ¶
const ( DefaultK1 = 1.5 DefaultB = 0.75 )
Default Okapi BM25 hyperparameters. K1 controls term-frequency saturation (higher = more weight to repeated matches); B controls length normalization (0 = ignore length, 1 = full normalization). 1.5 / 0.75 is the most widely cited starting point in the literature and matches both bleve and rank_bm25 defaults.
const DefaultRRFK = 60
DefaultRRFK is the canonical RRF k constant (Cormack, Clarke, Buettcher 2009 — "Reciprocal Rank Fusion outperforms Condorcet and individual rank learning methods"). 60 is the value the original paper proposed and the value Elasticsearch / OpenSearch default to.
Variables ¶
This section is empty.
Functions ¶
func BuildCorpusText ¶
BuildCorpusText is the canonical BM25 corpus shape: symbol_name + the first non-empty line of the chunk text (typically the signature). Centralized so engine.Search and tests don't drift on the spec.
When the chunk has no symbol name (rare for code chunks, common for doc / header chunks), the first text line stands in for symbol.
func Rerank ¶
Rerank applies candidate-set BM25 to the input hits and combines with the existing vector ordering via Reciprocal Rank Fusion. Inputs are assumed already sorted by vector score descending (the contract the store layer satisfies); each input's vector rank is taken as its position + 1.
The IDF in this BM25 is computed over the candidate set only, not the global chunk corpus — this is a candidate-set bias by design. The signal is still useful as a *rerank within candidates* because IDF rewards terms that vary across the local set, which is what differentiates one candidate from another.
On empty input, empty intent, or zero query tokens the function returns the input unchanged (each result carries its original index as HybridRank, BM25Score = 0). Callers can detect the no-op via Stats.BM25Disabled.
func Tokenize ¶
Tokenize converts an arbitrary string into BM25 tokens with code-aware splitting. The output preserves the joined identifier (lowercased) AND its sub-words so a query for either form scores. Concretely:
- "parseFile" → ["parsefile", "parse", "file"]
- "HTTPServer" → ["httpserver", "http", "server"]
- "read_file" → ["read_file", "read", "file"]
- "pkg.Type.Method" → ["pkg", "type", "method"]
- "json-rpc/v1" → ["json", "rpc", "v1"]
All tokens are lowercased; tokens of length < 2 are dropped (they add noise without helping rank). Caller-supplied tokens are not deduped — repetition is meaningful to BM25's TF term.
Types ¶
type Candidate ¶
Candidate pairs a vector-retrieved Hit with the BM25 corpus text for that candidate. CKV's caller (engine.Search) builds Corpus from chunk.SymbolName + signature first line. Whole chunk.Text is not used because tokenizing 30 long bodies dominates the rerank latency and the noise overwhelms the symbol-level signal.
type Document ¶
Document is one indexable record. Tokens are pre-tokenized — call Tokenize for the package's standard code-aware splitter, or supply custom tokens for a domain-specific corpus.
type Okapi ¶
Okapi is the standard BM25Okapi scorer. Concurrent reads (Score, TopK) after a completed Index are safe; do NOT call Index concurrently with a reader. Build once, query many.
In CKV the corpus is built per-Search call from the vector-retrieved candidate set (~30 documents), so "concurrent reads" simply means the Engine.Search owner runs Score in a single goroutine; the parameters remain useful documentation for any future consumer.
func NewOkapi ¶
func NewOkapi() *Okapi
NewOkapi returns a scorer with default hyperparameters. Override K1/B directly on the returned struct before calling Index.
func (*Okapi) Index ¶
Index registers the corpus. Empty input is allowed — the scorer will return 0 for all queries until a non-empty Index call replaces state.
func (*Okapi) Score ¶
Score returns the BM25 score for one document under the given query. Both K1 and B come from the Okapi struct so callers can tune at runtime without recreating the scorer.
func (*Okapi) TopK ¶
TopK returns the top-k documents by score, descending. Documents with score 0 (no matching terms) are excluded. k <= 0 means "all matches".
CKV's Rerank uses Score directly rather than TopK because it needs every candidate's rank to feed RRF — including zero-scoring ones — not just the top-k. TopK is kept for parity with the CKG interface and for ad-hoc debugging.
type Result ¶
type Result struct {
Hit types.Hit
BM25Score float64
HybridRank int // 1-based, after RRF
OriginalIdx int // 0-based position in the input (= vector rank - 1)
}
Result is one row in the reordered output. Hit.Score has BM25Score and HybridRank populated; the wrapping fields duplicate them for callers that prefer the rerank-package shape (e.g. footprint summarization).
type ScoredDoc ¶
ScoredDoc pairs a document ID with its BM25 score for the most recent query. Score is always >= 0; documents with no matching terms are not returned by TopK.
type Scorer ¶
type Scorer interface {
// Index registers the full corpus. Repeated calls overwrite earlier
// state — Scorer is not append-only.
Index(docs []Document)
// Score returns the BM25 score for one document under one query.
// Returns 0 when docID is unknown or no query term matches.
Score(query []string, docID string) float64
// TopK returns the top-k documents by score, descending. k <= 0
// returns every matching document.
TopK(query []string, k int) []ScoredDoc
}
Scorer is the contract every BM25 implementation in this package satisfies. The two-phase pattern (Index then Score / TopK) lets callers build the corpus once per query and reuse for many lookups.
type Stats ¶
type Stats struct {
CandidatesIn int // = len(input)
CandidatesOut int // = len(output); should equal input
RankChanges int // count of candidates whose HybridRank != OriginalIdx+1
Top1ScoreDelta float64 // Normalized score: final top-1 minus original top-1; positive when BM25 pulled a still-strong vector match to top, negative when it demoted vector
Top1ChunkID string // first 12 hex chars of new top-1 chunk_id, for fingerprint drift detection
BM25Disabled bool // true when Rerank was a no-op (empty input / empty intent / no query tokens)
}
Stats summarizes the rerank effect for footprint logging. Designed for single-fingerprint log lines: every field is a scalar an operator can grep / aggregate without joining multiple log entries.
Source Files