eval

Overview ¶

Package main — eval harness for go-code retrieval quality.

This file: SPLADE A/B go/no-go gate for flipping RRF_WEIGHT_SPARSE from 0.

Gate rule (research-mandated, documented in the SPLADE plan Phase 6):

PASS  iff nDCG@10 improves with p < 0.05 (paired t-test) AND
       Recall@20 is non-inferior (delta >= -nonInferiorMargin OR p_recall20 >= 0.05).
FAIL  if nDCG@10 shows no statistically significant improvement (p >= 0.05)
       OR nDCG@10 delta is non-positive (no gain even when nominally significant).
FAIL  if Recall@20 significantly regresses (delta < -nonInferiorMargin AND p < 0.05).

Future online validation: Team Draft Interleaving (TDI) provides higher sensitivity via live traffic interleaving. This offline harness is the prerequisite gate; TDI is the recommended follow-up to confirm with real query distribution before a permanent default change.

Package main — eval harness for go-code retrieval quality.

This file: HTTP client that drives go-code's REST bridge (POST /api/tools/semantic_search) and parses the XML response into a flat list of (file, symbol) pairs ranked top-1 first.

Package main — eval harness for go-code retrieval quality.

This file: load golden dataset records from JSONL files.

Command eval — offline retrieval-quality harness for go-code.

Replays a labeled (query, expected_top_3) golden dataset against a running go-code MCP server's REST bridge, computes nDCG@10, Recall@10/@20, and MRR, and writes a JSON report. Optional --baseline runs a paired t-test against a prior report and reports per-metric significance.

SPLADE A/B gate mode (Phase P6):

Run the harness twice — once with RRF_WEIGHT_SPARSE=0 (baseline) and once with the candidate weight — then use --baseline + --splade-weight to emit a go/no-go verdict for flipping the production env var:

# Step 1: baseline run (RRF_WEIGHT_SPARSE=0 on the server)
go-code-eval --golden-dir eval/golden \
             --target-url http://127.0.0.1:8897 \
             --output /tmp/eval-baseline.json

# Step 2: candidate run (RRF_WEIGHT_SPARSE=0.3 on the server)
go-code-eval --golden-dir eval/golden \
             --target-url http://127.0.0.1:8897 \
             --output /tmp/eval-cand.json \
             --baseline /tmp/eval-baseline.json \
             --splade-weight 0.3

# The report's "splade_gate" field contains the PASS/FAIL verdict.
jq .splade_gate /tmp/eval-cand.json

Graph-arm A/B gate mode (Phase 1 graph-first retrieval):

Same pattern as SPLADE but use --graph-weight and the report's "graph_gate":

# Step 1: baseline run (RRF_WEIGHT_GRAPH=0 on the server)
go-code-eval --golden-dir eval/golden \
             --target-url http://127.0.0.1:8897 \
             --output /tmp/eval-baseline.json

# Step 2: candidate run (RRF_WEIGHT_GRAPH=0.2 on the server)
go-code-eval --golden-dir eval/golden \
             --target-url http://127.0.0.1:8897 \
             --output /tmp/eval-graph-cand.json \
             --baseline /tmp/eval-baseline.json \
             --graph-weight 0.2

jq .graph_gate /tmp/eval-graph-cand.json

Gate: PASS iff nDCG@10 improves at p<0.05 AND Recall@20 is non-inferior (delta >= -2% OR p >= 0.05). See abgate.go for the full rule.

Future online step: Team Draft Interleaving (TDI) on live traffic provides higher sensitivity; this offline harness is the prerequisite gate.

The harness is read-only against the target server: every query is a semantic_search call. Use against a non-prod target for fair benchmarking.

Package main — eval harness for go-code retrieval quality.

This file: nDCG@10, Recall@K, MRR over a single (golden, retrieved) pair. All inputs are 1-based ranked lists; relevance is binary (1 if a retrieved hit matches any expected symbol, else 0).

Package main — eval harness for go-code retrieval quality.

This file: JSON output schema and writer.

Package main — eval harness for go-code retrieval quality.

This file: concurrent dispatch of golden queries against a running go-code.

Package main — eval harness for go-code retrieval quality.

This file: paired Student's t-test with a stdlib-only Student's t two-tailed p-value. Implementation mirrors Numerical Recipes §6.4 (regularized incomplete beta via continued fraction, Abramowitz-Stegun 26.5.8/26.5.20).

Why no gonum: harness is a single-binary tool; pulling gonum into go-code's vendor tree forces a `go mod vendor` round-trip that risks the recurring tree-sitter PHP-header strip. ~80 LOC of pure-Go beta is cheaper.