Documentation
¶
Overview ¶
Package retrieval implements the LLM-free retrieval-accuracy measurement (EV1 Phase 2) — load YAML probe fixtures, dispatch each to the matching MCP tool through StoreReader, score result symbols against an expected set with recall / precision / F1.
Why this layer exists separately from internal/eval (LLM-driven): retrieval tests are deterministic, fast (no API calls), and gate every code change. The LLM eval lives next door with its own runner.go — they share the task YAML idiom but not the execution path.
Fixture lifecycle:
- eval/retrieval/*.yaml is committed
- ckg eval-retrieval --graph=eval/.synthetic-data --fixtures=eval/retrieval loads each fixture, executes the probe, scores the result
- Output is JSON for diffing against eval/baseline/retrieval.json
New tool support requires extending dispatchProbe in runner.go; the fixture format stays stable (map[string]any args).
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Expect ¶
type Expect struct {
Symbols []string `yaml:"symbols"`
}
Expect carries the gold-set symbols. The set is unordered; order is not part of the assertion. If a downstream test cares about ranking (top-K MRR, NDCG), that's a Phase 2.5 addition, not what this layer scores.
type Fixture ¶
type Fixture struct {
ID string `yaml:"id"`
Description string `yaml:"description"`
Probe Probe `yaml:"probe"`
Expected Expect `yaml:"expected"`
Scoring Scoring `yaml:"scoring"`
}
Fixture is one retrieval probe specification.
Schema is intentionally close to the LLM-eval task YAML (id, description, expected.symbols, scoring.threshold) so a future reader recognises the shape, but adds `probe.{tool, args}` to make the call deterministic — there's no LLM to interpret natural-language `description` here.
func LoadFixtures ¶
LoadFixtures reads every *.yaml file under dir and returns them sorted by ID for deterministic execution order. A malformed file (missing ID, unknown tool, missing expected.symbols) is a hard error — the eval gate must not silently skip a broken fixture.
type Probe ¶
Probe identifies the StoreReader/MCP tool to invoke and its arguments. Args is map[string]any (not a typed struct) so the schema can carry every tool's parameters without a Go enum — dispatchProbe routes by Tool and casts each arg at call time.
type Result ¶
type Result struct {
Fixture Fixture
Got []string // unique qualified_names returned by the probe
Score Score
// PassRecall / PassPrecision report whether the per-fixture
// thresholds were met. Aggregator combines them into the overall
// gate result.
PassRecall bool
PassPrecision bool
}
Result is the outcome of running one Fixture against a Reader.
func Run ¶
func Run(reader persist.StoreReader, f Fixture) (Result, error)
Run executes a single fixture against the given Reader and produces a scored Result. A dispatch error (unknown tool, missing arg) is returned separately from a low-score Result — the caller distinguishes "ran and scored badly" from "could not run at all".
func RunAll ¶
func RunAll(reader persist.StoreReader, fixtures []Fixture) ([]Result, error)
RunAll runs every fixture sequentially and returns one Result per fixture in input order. Sequential rather than parallel because the per-probe cost is microseconds on the synthetic fixture — parallelism would add ordering noise to the JSON output for no real speedup.
type Score ¶
type Score struct {
Recall float64 `json:"recall"`
Precision float64 `json:"precision"`
F1 float64 `json:"f1"`
// Diagnostic — the set differences, useful when a fixture fails.
// Missing = expected \ got (what the probe failed to return)
// Extra = got \ expected (what the probe over-returned)
Missing []string `json:"missing,omitempty"`
Extra []string `json:"extra,omitempty"`
}
Score is the result of comparing got symbols against an expected set. Recall and Precision follow the textbook definitions:
Recall = |expected ∩ got| / |expected| Precision = |expected ∩ got| / |got| F1 = harmonic mean of the two
Edge cases (documented because retrieval probes routinely produce empty result sets):
- expected empty: rejected at fixture load (validateFixture)
- got empty + nonempty expected: Recall=0, Precision=0 (no TP, no FP), F1=0
- intersection empty: same as above
- perfect match: Recall=Precision=F1=1.0
func ComputeScore ¶
ComputeScore returns recall/precision/F1 plus the symbol-level set differences. Both inputs are deduped internally — duplicate IDs in expected or got do not skew the ratio.
type Scoring ¶
type Scoring struct {
RecallMin float64 `yaml:"recall_min"`
PrecisionMin float64 `yaml:"precision_min"`
}
Scoring carries per-fixture pass/fail thresholds. Defaults (when a field is omitted in YAML) are:
RecallMin = 0.0 (no recall gate — the test is informational) PrecisionMin = 0.0 (no precision gate)
In practice every committed fixture sets at least RecallMin so regressions surface as test failures rather than silent diffs.
Source Files