corebench

Overview ¶

Command corebench evaluates an embedding model on a downloaded subset of CORE-Bench (arXiv:2606.11864, HF: zhangfw123/CORE-Bench) in BEIR format: each repo dir holds corpus.jsonl, queries.jsonl and qrels/test.tsv.

It measures the EMBEDDER in our own runtime (tokenizer + pooling + ORT), not the full find_context pipeline: corpus chunks are embedded as-is, queries are scored against the query's filtered_corpus_id subset (the benchmark's temporal filter), and NDCG@10 / Recall@100 are reported per repo plus pooled. Corpus embeddings are cached on disk per (repo, model) so an A/B between two models only pays each corpus once.