Documentation
¶
Overview ¶
Package beir loads datasets in the BEIR format (the de-facto information retrieval benchmark) into an amoxtli evaluation corpus + query set. Unlike the SQuAD-style extractive-QA datasets (where questions are written from the passage and share its vocabulary, favouring lexical search), several BEIR datasets — FiQA, Quora, ArguAna, SciFact — exhibit a real query↔document vocabulary gap, so they are where semantic (dense) retrieval is expected to help over BM25.
BEIR layout (three files):
corpus.jsonl one JSON per line: {"_id","title","text"}
queries.jsonl one JSON per line: {"_id","text"}
qrels/*.tsv header "query-id\tcorpus-id\tscore" then rows
A query's relevant documents are the corpus ids with a positive qrel score.
Index ¶
- func Load(corpusPath, queriesPath, qrelsPath, name, lang string) (*eval.Corpus, *eval.Dataset, error)
- func LoadHotpotAnswers(path string) (map[string][]string, error)
- func LoadSubsample(corpusPath, queriesPath, qrelsPath, name, lang string, maxQueries, maxDocs int) (*eval.Corpus, *eval.Dataset, error)
- func Source(dataset, id string) string
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func Load ¶
func Load(corpusPath, queriesPath, qrelsPath, name, lang string) (*eval.Corpus, *eval.Dataset, error)
Load reads a BEIR dataset from the three given files, tagging documents and queries with lang (may be empty). name labels the corpus/dataset and namespaces the source identifiers. Only queries that have at least one relevant document present in the corpus are kept.
func LoadHotpotAnswers ¶
LoadHotpotAnswers reads the gold answers from a native HotpotQA distribution file (e.g. hotpot_dev_fullwiki_v1.json) — a JSON array of objects with at least "_id" and "answer" — and returns them keyed by question id. BEIR's hotpotqa query ids are the original HotpotQA question ids, so the result can be joined onto a BEIR-loaded dataset to enable the generation (reader) EM/F1 evaluation, which the three BEIR files (corpus/queries/qrels) cannot provide on their own.
func LoadSubsample ¶
func LoadSubsample(corpusPath, queriesPath, qrelsPath, name, lang string, maxQueries, maxDocs int) (*eval.Corpus, *eval.Dataset, error)
LoadSubsample is a memory-bounded, gold-aware alternative to Load followed by eval.Subsample, for BEIR datasets whose corpus is too large to hold in memory (FEVER, HotpotQA, DBPedia, Climate-FEVER, NQ… — millions of documents). It reads the (small) queries and qrels first, selects a deterministic query subset, then streams corpus.jsonl exactly once, retaining only the gold documents of the selected queries plus up to maxDocs-minus-gold distractors — so peak memory is bounded by maxDocs, not by the full corpus.
Selection is deterministic: queries are sorted by id and the first maxQueries with at least one qrel are chosen (maxQueries <= 0 keeps them all). Gold documents are always kept even if that exceeds maxDocs; distractors are the first non-gold documents in corpus-file order until the budget is spent (maxDocs <= 0 keeps every document — only viable for a small corpus). A selected query is dropped if any of its relevant documents is absent from the corpus, so every kept query stays answerable.
Types ¶
This section is empty.
Source Files