ir

package
v1.2.3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 20, 2026 License: MIT Imports: 14 Imported by: 0

Documentation

Overview

Package ir provides standard information-retrieval metrics for benchmark runners.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func ReadDocuments 

func ReadDocuments(reader io.Reader, visit func(Document) error) error

ReadDocuments streams official BEIR corpus.jsonl entries to visit.

func ReadRun 

func ReadRun(reader io.Reader) (map[string][]string, error)

ReadRun parses a standard six-column TREC run file into ranked document IDs.

func SelectQueryIDs 

func SelectQueryIDs(queryIDs []string, limit int, seed uint64) ([]string, error)

SelectQueryIDs chooses at most limit unique query IDs without replacement. It sorts the input before sampling so output is independent of ingestion order.

func WriteRun 

func WriteRun(writer io.Writer, queryID string, results []RunResult, tag string) error

WriteRun writes a standard six-column TREC run. Results must already be in final rank order and use evaluation-facing BEIR document IDs.

Types

type Comparison 

type Comparison struct {
	Queries       int      `json:"queries"`
	Baseline      Metrics  `json:"baseline"`
	Candidate     Metrics  `json:"candidate"`
	AbsoluteDelta Metrics  `json:"absolute_delta"`
	RecallAt10CI  Interval `json:"recall_at_10_ci_95"`
	RecallAt100CI Interval `json:"recall_at_100_ci_95"`
	NDCGAt10CI    Interval `json:"ndcg_at_10_ci_95"`
}

Comparison is a paired evaluation of candidate results against a baseline.

func Compare 

func Compare(qrels Qrels, baseline, candidate map[string][]string, seed uint64, resamples int) (Comparison, error)

Compare evaluates two runs on identical qrels and calculates deterministic paired-bootstrap 95% confidence intervals for the primary metrics.

type Document 

type Document struct {
	ID       string         `json:"_id"`
	Title    string         `json:"title"`
	Text     string         `json:"text"`
	Metadata map[string]any `json:"metadata,omitempty"`
}

Document is the logical BEIR corpus retrieval unit.

type DocumentSink 

type DocumentSink interface {
	StoreDocument(context.Context, Document) error
}

DocumentSink accepts a BEIR document while preserving its retrieval identity.

type IngestStats 

type IngestStats struct {
	Documents int64
}

IngestStats reports the number of corpus documents accepted by a sink.

func IngestDocuments 

func IngestDocuments(ctx context.Context, reader io.Reader, sink DocumentSink) (IngestStats, error)

IngestDocuments streams corpus.jsonl into sink without retaining the corpus in memory. The sink is responsible for transactional batching and index life cycle.

type Interval 

type Interval struct {
	Lower float64 `json:"lower"`
	Upper float64 `json:"upper"`
}

Interval is a percentile confidence interval for an absolute metric delta.

type Metrics 

type Metrics struct {
	RecallAt10  float64 `json:"recall_at_10"`
	RecallAt100 float64 `json:"recall_at_100"`
	NDCGAt10    float64 `json:"ndcg_at_10"`
	MRRAt10     float64 `json:"mrr_at_10"`
	MAPAt100    float64 `json:"map_at_100"`
}

Metrics is the standard metric set used by retrieval benchmarks.

func Compute 

func Compute(results []string, qrels map[string]int) Metrics

Compute calculates graded nDCG and binary relevance metrics for one query. Results are ordered document IDs; qrels maps document ID to its relevance grade.

func Evaluate 

func Evaluate(qrels Qrels, run map[string][]string) Metrics

Evaluate calculates macro-average metrics, counting each qrels query once.

type NodeIndexer 

type NodeIndexer interface {
	IndexNode(*storage.Node) error
}

NodeIndexer is the search index operation required after a benchmark write.

type NornicDocumentSink 

type NornicDocumentSink struct {
	Engine  storage.Engine
	Indexer NodeIndexer
}

NornicDocumentSink imports one BEIR document as one NornicDB retrieval unit. The beir_id property is the lossless persistent map when storage namespaces rewrite the physical node ID.

func (*NornicDocumentSink) StoreDocument 

func (s *NornicDocumentSink) StoreDocument(ctx context.Context, document Document) error

StoreDocument persists and indexes a BEIR document with its original ID.

type NornicRetriever 

type NornicRetriever struct {
	Service  *search.Service
	Embedder QueryEmbedder
	Options  *search.SearchOptions
}

NornicRetriever adapts the existing search service to the benchmark retriever. A nil Embedder intentionally executes BM25-only retrieval.

func (*NornicRetriever) Retrieve 

func (r *NornicRetriever) Retrieve(ctx context.Context, query string, topK int) ([]RunResult, error)

Retrieve executes one NornicDB retrieval query and returns BEIR document IDs.

type Qrels 

type Qrels map[string]map[string]int

Qrels maps each query ID to its graded relevance judgments.

func ReadQrels 

func ReadQrels(reader io.Reader) (Qrels, error)

ReadQrels parses the standard four-column TREC qrels format.

type Query 

type Query struct {
	ID       string         `json:"_id"`
	Text     string         `json:"text"`
	Metadata map[string]any `json:"metadata,omitempty"`
}

Query is an official BEIR query.

func FilterQueriesWithQrels 

func FilterQueriesWithQrels(queries []Query, qrels Qrels) []Query

FilterQueriesWithQrels retains only queries that have relevance judgments.

func ReadQueries 

func ReadQueries(reader io.Reader) ([]Query, error)

ReadQueries loads official BEIR queries.jsonl and rejects duplicate IDs.

type QueryEmbedder 

type QueryEmbedder func(context.Context, string) ([]float32, error)

QueryEmbedder produces a query embedding for vector and hybrid benchmark variants.

type QueryManifest 

type QueryManifest struct {
	Dataset  string   `json:"dataset"`
	Split    string   `json:"split"`
	Seed     uint64   `json:"seed"`
	QueryIDs []string `json:"query_ids"`
	SHA256   string   `json:"sha256"`
}

QueryManifest records the exact query subset shared by all benchmark variants.

func NewQueryManifest 

func NewQueryManifest(dataset, split string, queries []Query, limit int, seed uint64) (QueryManifest, error)

NewQueryManifest deterministically selects query IDs and records their digest.

type Retriever 

type Retriever interface {
	Retrieve(context.Context, string, int) ([]RunResult, error)
}

Retriever executes a benchmark query and returns evaluation-facing BEIR IDs.

type RunResult 

type RunResult struct {
	DocumentID string
	Score      float64
}

RunResult is one ranked retrieval result expressed with its BEIR document ID.

func RunResultsFromSearchResults 

func RunResultsFromSearchResults(results []search.SearchResult) ([]RunResult, error)

RunResultsFromSearchResults converts hydrated NornicDB results into evaluation-facing BEIR IDs. Every benchmark result must carry beir_id.

type RunStats 

type RunStats struct {
	Queries int64
}

RunStats reports one manifest-driven retrieval execution.

func RunManifest 

func RunManifest(ctx context.Context, manifest QueryManifest, queries []Query, retriever Retriever, topK int, tag string, writer io.Writer) (RunStats, error)

RunManifest executes only the query IDs in manifest and writes a TREC run in manifest order. Queries must contain every selected official query ID.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.