Documentation
¶
Overview ¶
Package eval is the Gate-1 measurement harness. An adversarial review of the first version found two fatal defects: it compared a 1-body Mesh arm against a 3-body baseline (so the "saving" was mostly body-count, not fusion), and it mixed two recall definitions (candidates-surfaced for Mesh vs bodies-read for the baseline). This version fixes both: three arms with matched costs, and surfacing-recall (at equal candidate K) reported separately from answer@1 (the single body each arm actually reads). Both arms use the same tokenizer.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type CaseResult ¶
type CaseResult struct {
Query string
// Surfacing recall at equal K: does a relevant id appear in the candidate set?
MeshSurfaced bool
FTSSurfaced bool
// Answer@1: is the single body the arm reads (top card / top FTS hit) relevant?
MeshAnswer1 bool
FTSAnswer1 bool
// Tokens for the body(ies) each arm actually reads, plus Mesh's cards.
MeshTokens int // cards + top-1 body (what Mesh costs)
FTSTop1Tokens int // 1 body (matched single-read baseline)
FTSTop3Tokens int // 3 bodies (naive baseline)
}
CaseResult holds the per-query outcome across the three arms.
type Report ¶
type Report struct {
Cases []CaseResult
N int
MeshSurfaced, FTSSurfaced int // surfacing recall (equal K)
MeshAnswer1, FTSAnswer1 int // answer@1
MeshMean, FTSTop1Mean, FTSTop3Mean float64
MeshMedian, FTSTop1Median, FTSTop3Median float64
// The three defensible sub-claims.
SurfacingWin bool // Mesh surfaces relevant at equal K at least as often
AnswerWin bool // Mesh's single read is relevant at least as often
NaiveCostWin bool // Mesh median tokens < naive read-top-3 median
Pass bool // all three hold
}
Report aggregates the run.
type Score ¶
type Score struct{ Answer1, Recall, N int }
Score holds answer@1 and surfacing-recall counts for a weight set on a case set.
type TuneReport ¶
type TuneReport struct {
Candidates int
Default, Best WeightSet
DefaultTrain, BestTrain Score
DefaultTest, BestTest Score
}
TuneReport is the outcome of a weight search: the best set found on the train split and how it and the built-in default score on both train and test, so the reader can see at a glance whether the learned weights generalize or overfit.
func TuneWeights ¶
TuneWeights grid-searches the weight simplex on train, maximizing answer@1 (surfacing recall as the tiebreak), and reports how the winner and the built-in default score on both train and test. The retriever must already have vectors enabled if vectors is true. The search never touches the rerank stage.
Source Files