Documentation
¶
Overview ¶
Package eval is a retrieval-quality evaluation harness for CortexDB. It runs a labeled query set against a retriever and reports standard information- retrieval metrics (recall@k, precision@k, MRR, nDCG@k), so retrieval quality is measured and regression-guarded rather than assumed.
Index ¶
- func NDCGAtK(retrieved, relevant []string, k int) float64
- func PrecisionAtK(retrieved, relevant []string, k int) float64
- func RecallAtK(retrieved, relevant []string, k int) float64
- func ReciprocalRank(retrieved, relevant []string) float64
- type Dataset
- type Document
- type Query
- type QueryResult
- type Report
- type Retriever
- type RetrieverFunc
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func NDCGAtK ¶
NDCGAtK is the normalized discounted cumulative gain at k, with binary relevance. Returns 0 when there are no relevant documents.
func PrecisionAtK ¶
PrecisionAtK is the fraction of the top k retrieved documents that are relevant. Returns 0 when k <= 0.
func RecallAtK ¶
RecallAtK is the fraction of relevant documents retrieved within the top k. Returns 0 when there are no relevant documents.
func ReciprocalRank ¶
ReciprocalRank is 1/rank of the first relevant document (rank starting at 1), or 0 if none is retrieved. Averaged across queries this is MRR.
Types ¶
type Dataset ¶
type Dataset struct {
Name string `json:"name"`
Description string `json:"description"`
Documents []Document `json:"documents"`
Queries []Query `json:"queries"`
}
Dataset is a corpus plus a labeled query set.
type Document ¶
type Document struct {
ID string `json:"id"`
Title string `json:"title"`
Content string `json:"content"`
}
Document is one corpus document to index.
type Query ¶
type Query struct {
ID string `json:"id"`
Text string `json:"text"`
Relevant []string `json:"relevant"`
}
Query is one labeled query: its text plus the ids of relevant documents.
type QueryResult ¶
type QueryResult struct {
QueryID string `json:"query_id"`
Retrieved []string `json:"retrieved"`
RR float64 `json:"rr"`
}
QueryResult is a single query's retrieval and its reciprocal rank.
type Report ¶
type Report struct {
Dataset string `json:"dataset"`
NumQueries int `json:"num_queries"`
Ks []int `json:"ks"`
RecallAtK map[int]float64 `json:"recall_at_k"`
PrecisionAtK map[int]float64 `json:"precision_at_k"`
NDCGAtK map[int]float64 `json:"ndcg_at_k"`
MRR float64 `json:"mrr"`
PerQuery []QueryResult `json:"per_query,omitempty"`
}
Report holds aggregate metrics over a query set, plus per-query detail.
Source Files