Documentation
¶
Overview ¶
Package benchdata loads BEIR-format retrieval benchmark data (the layout CORE-Bench ships: per-repo corpus.jsonl / queries.jsonl / qrels/test.tsv) and manages the on-disk embedding-matrix cache shared by cmd/corebench and cmd/ftdata.
Index ¶
- func BM25Tokenize(s string) []string
- func Dot(a, b []float32) float32
- func LoadMatrix(path string, count, dim int) ([]float32, bool)
- func LoadQrels(path string) (map[string]map[string]int, error)
- func PeekMatrix(path string) (count, dim int, ok bool)
- func ReadJSONL(path string, fn func([]byte) error) error
- func Sanitize(s string) string
- func SaveMatrix(path string, mat []float32, count, dim int) error
- type BM25Index
- type CorpusDoc
- type Query
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func BM25Tokenize ¶
func LoadMatrix ¶
LoadMatrix reads a cached row-major float32 embedding matrix written by SaveMatrix, validating the (count, dim) header against expectations.
func PeekMatrix ¶
PeekMatrix reads only the (count, dim) header, so callers can validate a cache without loading gigabytes.
Types ¶
type BM25Index ¶
type BM25Index struct {
// contains filtered or unexported fields
}
BM25Index is a small in-memory BM25 (k1=1.2, b=0.75) with the same tokenization idea as the store's FTS layer: runs of letters/digits/ underscore, lowercased. Used for seed fusion in corebench and for mining hard negatives in ftdata.
Source Files