Documentation
¶
Overview ¶
Package evalrepo evaluates a frozen Corvint retrieval corpus without mutation.
Index ¶
- Constants
- Variables
- func BudgetYield(hitsAtBudget []int, mustTotal int) map[string]any
- func CaseSplit(id string) string
- func Encode(value any) ([]byte, error)
- func Evaluate(ctx context.Context, root, goldenPath string, fixturePaths ...string) (map[string]any, error)
- func EvaluateComparable(ctx context.Context, root, goldenPath string, purpose Purpose, ...) (map[string]any, map[string]any, error)
- func MergeSplitMetrics(parts []map[string]any) map[string]any
- func NotRecordedOutcome() map[string]any
- func ValidateDownstreamOutcome(value any) error
- type ErrHeldoutRowInTuning
- type Purpose
- type SplitCorpus
- type SplitManifest
- type SplitRow
Constants ¶
const ( SplitRule = "atlas-eval-split-v0" SplitDev = "dev" SplitHeldout = "heldout" // HeldoutProvenance labels every row of the v0 manifest: each row had been // observed during development before it was assigned (REC-V0-007). HeldoutProvenance = "previously-observed" OutcomeNotRecorded = "not-recorded" )
Variables ¶
var BudgetLadderBytes = []int{512, 1024, 2048, 4096, 8192, 16384}
BudgetLadderBytes is the fixed recall@budget ladder (REC-V0-005).
Functions ¶
func BudgetYield ¶
BudgetYield reports recall@budget over BudgetLadderBytes and its area (REC-V0-005).
func Evaluate ¶
func Evaluate(ctx context.Context, root, goldenPath string, fixturePaths ...string) (map[string]any, error)
Evaluate returns the Python-compatible eval report for one pinned corpus.
func EvaluateComparable ¶
func EvaluateComparable(ctx context.Context, root, goldenPath string, purpose Purpose, fixturePaths ...string) (map[string]any, map[string]any, error)
EvaluateComparable returns the Evaluate report over the rows the purpose may read, plus the REC-V0 comparability block for the baseline arm. For PurposeScore the report is identical to Evaluate's.
func MergeSplitMetrics ¶
MergeSplitMetrics sums one split's metrics across corpora and recomputes its ratios and budget yield from the summed counts.
func NotRecordedOutcome ¶
NotRecordedOutcome is the explicit absent downstream outcome (REC-V0-006).
func ValidateDownstreamOutcome ¶
ValidateDownstreamOutcome accepts exactly {"state":"not-recorded"} or a recorded passed/failed outcome naming its evidence (REC-V0-006).
Types ¶
type ErrHeldoutRowInTuning ¶
type ErrHeldoutRowInTuning struct{ ID string }
ErrHeldoutRowInTuning is returned when a tuning run reaches a held-out row.
func (ErrHeldoutRowInTuning) Error ¶
func (err ErrHeldoutRowInTuning) Error() string
type SplitCorpus ¶
type SplitManifest ¶
type SplitManifest struct {
SchemaVersion int `json:"schema_version"`
Rule string `json:"rule"`
HeldoutModulus int `json:"heldout_modulus"`
HeldoutProvenance string `json:"heldout_provenance"`
Corpora []SplitCorpus `json:"corpora"`
}
SplitManifest is the committed partition manifest (REC-V0-002).
func NewSplitManifest ¶
func NewSplitManifest(corpora map[string][]byte) (SplitManifest, error)
NewSplitManifest builds the manifest for corpora given as path -> bytes.
func (SplitManifest) VerifyCorpus ¶
func (manifest SplitManifest) VerifyCorpus(path string, raw []byte) (bool, error)
VerifyCorpus refuses a corpus whose rows differ from the committed manifest in ID, order, assignment or content digest. It reports false when the manifest does not register the corpus at all.
Source Files