Documentation
¶
Overview ¶
Package eval provides an evaluation harness for testing and validating NornicDB's search quality, ranking accuracy, and performance.
The eval harness allows you to:
- Define test cases with queries and expected results
- Run evaluations and compute standard IR metrics
- Compare before/after changes to verify improvements
- Track performance over time
Metrics computed:
- Precision@K: What fraction of top-K results are relevant?
- Recall@K: What fraction of all relevant docs appear in top-K?
- MRR (Mean Reciprocal Rank): Where does the first relevant result appear?
- NDCG (Normalized Discounted Cumulative Gain): Ranking quality
- Diversity: How different are the results from each other?
Example usage:
harness := eval.NewHarness(searchService)
harness.AddTestCase(eval.TestCase{
Name: "ML concepts",
Query: "machine learning algorithms",
Expected: []string{"node-1", "node-2", "node-3"},
})
results, err := harness.Run(ctx)
if err != nil {
log.Fatal(err)
}
fmt.Printf("Precision@10: %.2f\n", results.Precision10)
fmt.Printf("MRR: %.2f\n", results.MRR)
ELI12 (Explain Like I'm 12):
Think of it like grading a test:
- You have questions (search queries)
- You have answer keys (expected results)
- The harness checks how well the search did
- It gives you a score so you know if changes help or hurt
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type EvalResult ¶
type EvalResult struct {
// Suite info
SuiteName string `json:"suite_name"`
Timestamp time.Time `json:"timestamp"`
Duration time.Duration `json:"duration"`
// Aggregate metrics (averaged across all test cases)
Aggregate Metrics `json:"aggregate"`
// Per-test results
Results []TestResult `json:"results"`
// Summary statistics
TotalTests int `json:"total_tests"`
PassedTests int `json:"passed_tests"`
FailedTests int `json:"failed_tests"`
// Thresholds used for pass/fail
Thresholds Thresholds `json:"thresholds"`
}
EvalResult contains the complete evaluation results.
type Harness ¶
type Harness struct {
// contains filtered or unexported fields
}
Harness is the main evaluation harness.
func NewHarness ¶
NewHarness creates a new evaluation harness.
func (*Harness) AddTestCase ¶
AddTestCase adds a single test case.
func (*Harness) AddTestCases ¶
AddTestCases adds multiple test cases.
func (*Harness) Run ¶
func (h *Harness) Run(ctx context.Context) (*EvalResult, error)
Run executes the evaluation and returns results.
func (*Harness) SetThresholds ¶
func (h *Harness) SetThresholds(t Thresholds)
SetThresholds sets the pass/fail thresholds.
type Metrics ¶
type Metrics struct {
// Precision at various K values
Precision1 float64 `json:"precision@1"`
Precision5 float64 `json:"precision@5"`
Precision10 float64 `json:"precision@10"`
// Recall at various K values
Recall5 float64 `json:"recall@5"`
Recall10 float64 `json:"recall@10"`
Recall50 float64 `json:"recall@50"`
// Mean Reciprocal Rank - where does first relevant result appear?
MRR float64 `json:"mrr"`
// Normalized Discounted Cumulative Gain
NDCG5 float64 `json:"ndcg@5"`
NDCG10 float64 `json:"ndcg@10"`
// Mean Average Precision
MAP float64 `json:"map"`
// Diversity - how different are results from each other? (0-1)
Diversity float64 `json:"diversity"`
// Hit Rate - fraction of queries with at least one relevant result
HitRate float64 `json:"hit_rate"`
}
Metrics contains all computed evaluation metrics.
type Reporter ¶
type Reporter struct {
// contains filtered or unexported fields
}
Reporter formats and outputs evaluation results.
func NewReporter ¶
NewReporter creates a new reporter that writes to the given writer.
func (*Reporter) PrintCompact ¶
func (r *Reporter) PrintCompact(result *EvalResult)
PrintCompact prints a one-line summary.
func (*Reporter) PrintDetails ¶
func (r *Reporter) PrintDetails(result *EvalResult)
PrintDetails prints detailed per-test results.
func (*Reporter) PrintJSON ¶
func (r *Reporter) PrintJSON(result *EvalResult) error
PrintJSON outputs results as JSON.
func (*Reporter) PrintSummary ¶
func (r *Reporter) PrintSummary(result *EvalResult)
PrintSummary prints a human-readable summary of results.
type TestCase ¶
type TestCase struct {
// Name is a human-readable identifier for this test
Name string `json:"name"`
// Query is the search query text
Query string `json:"query"`
// Embedding is optional pre-computed query embedding
// If nil, the harness will use text-only search
Embedding []float32 `json:"embedding,omitempty"`
// Expected is the list of node IDs that should be returned
// Order matters for ranking metrics (first = most relevant)
Expected []string `json:"expected"`
// RelevanceGrades allows graded relevance (0-3 scale)
// If nil, binary relevance is assumed (in Expected = relevant)
// Map of nodeID -> grade (0=not relevant, 1=marginal, 2=relevant, 3=highly relevant)
RelevanceGrades map[string]int `json:"relevance_grades,omitempty"`
// Tags for grouping and filtering test cases
Tags []string `json:"tags,omitempty"`
// Options overrides default search options for this test
Options *search.SearchOptions `json:"options,omitempty"`
}
TestCase defines a single evaluation test case.
type TestResult ¶
type TestResult struct {
TestCase TestCase `json:"test_case"`
Metrics Metrics `json:"metrics"`
Returned []string `json:"returned"`
Duration time.Duration `json:"duration"`
Error string `json:"error,omitempty"`
SearchMethod string `json:"search_method"`
}
TestResult contains results for a single test case.
type TestSuite ¶
type TestSuite struct {
Name string `json:"name"`
Description string `json:"description"`
Version string `json:"version"`
Created time.Time `json:"created"`
TestCases []TestCase `json:"test_cases"`
}
TestSuite is a collection of test cases.
type Thresholds ¶
type Thresholds struct {
Precision10 float64 `json:"precision@10"`
Recall10 float64 `json:"recall@10"`
MRR float64 `json:"mrr"`
NDCG10 float64 `json:"ndcg@10"`
HitRate float64 `json:"hit_rate"`
}
Thresholds define minimum acceptable metric values.
func DefaultThresholds ¶
func DefaultThresholds() Thresholds
DefaultThresholds returns sensible default thresholds.
Source Files
Click to show internal directories.
Click to hide internal directories.