eval

Overview ¶

Package main — eval harness for go-code retrieval quality.

This file: HTTP client that drives go-code's REST bridge (POST /api/tools/semantic_search) and parses the XML response into a flat list of (file, symbol) pairs ranked top-1 first.

Package main — eval harness for go-code retrieval quality.

This file: load golden dataset records from JSONL files.

Command eval — offline retrieval-quality harness for go-code.

Replays a labeled (query, expected_top_3) golden dataset against a running go-code MCP server's REST bridge, computes nDCG@10, Recall@10/@20, and MRR, and writes a JSON report. Optional --baseline runs a paired t-test against a prior report and reports per-metric significance.

Usage:

go-code-eval \
  --golden-dir eval/golden \
  --target-url http://127.0.0.1:8897 \
  --output     /tmp/eval-candidate.json \
  --baseline   /tmp/eval-baseline.json   # optional

The harness is read-only against the target server: every query is a semantic_search call. Use against a non-prod target for fair benchmarking.

Package main — eval harness for go-code retrieval quality.

This file: nDCG@10, Recall@K, MRR over a single (golden, retrieved) pair. All inputs are 1-based ranked lists; relevance is binary (1 if a retrieved hit matches any expected symbol, else 0).

Package main — eval harness for go-code retrieval quality.

This file: JSON output schema and writer.

Package main — eval harness for go-code retrieval quality.

This file: concurrent dispatch of golden queries against a running go-code.

Package main — eval harness for go-code retrieval quality.

This file: paired Student's t-test with a stdlib-only Student's t two-tailed p-value. Implementation mirrors Numerical Recipes §6.4 (regularized incomplete beta via continued fraction, Abramowitz-Stegun 26.5.8/26.5.20).

Why no gonum: harness is a single-binary tool; pulling gonum into go-code's vendor tree forces a `go mod vendor` round-trip that risks the recurring tree-sitter PHP-header strip. ~80 LOC of pure-Go beta is cheaper.