Documentation
¶
Overview ¶
Package eval implements the OKF held-out retrieval evaluation: it derives example-query/expected-document pairs from a bundle, builds a held-out copy of the corpus with those queries stripped from their host chunks, ingests the copy into an ephemeral database, runs each query through the Retriever, and computes doc-level retrieval metrics against a versioned baseline.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Metrics ¶
type Metrics struct {
N int `json:"n"`
HitAt1 float64 `json:"hit_at_1"`
RecallAtK float64 `json:"recall_at_k"`
MRRAtK float64 `json:"mrr_at_k"`
ExactChunk float64 `json:"exact_chunk"`
K int `json:"k"`
}
Metrics holds the doc-level retrieval scores over all evaluated pairs. All rates are in [0,1]. N is the number of example-query pairs evaluated.
func Report ¶
func Report(ctx context.Context, logger *slog.Logger, out io.Writer, opts Options, baselinePath string, save bool) (Metrics, error)
Report runs an evaluation and renders the metrics table to out, loading the baseline at baselinePath for delta annotation when present and saving the fresh metrics back when save is true. It returns the computed Metrics so callers can reuse them.
func Run ¶
Run executes the full held-out evaluation: derive example-query pairs, build a held-out copy with those queries stripped, ingest the copy into an ephemeral SQLite database, search each query, and compute doc-level metrics. The user's real database is never touched — everything happens under a temp directory that is removed on return. The Retriever is the FTS5 engine over the temp DB.
type Options ¶
type Options struct {
Bundle string
K int
Include []string
Exclude []string
Chunking chunk.Config
// Semantic, when true, evaluates the hybrid (bm25 + vector) retriever instead
// of the lexical engine, so the run can be compared against the FTS baseline.
Semantic bool
// ModelDir is the embedding model directory used when Semantic is true.
ModelDir string
}
Options configures one evaluation run. Bundle is the path to the OKF bundle. K is the retrieval depth (defaults to 12). Include/Exclude/Chunking mirror the ingest configuration so the held-out copy is indexed exactly like production.
Source Files