bench/

bench/ — baseline harness for feature PR validation

Repo-level benchmark harness that captures measured pre/post metrics for incoming feature PRs (graphify-inspired improvements: confidence tags, orientation report, rationale extraction, graph diff, similarity edges).

Scope: this directory is not the historical language coverage benchmark in docs/benchmarks.md. This directory tests repo-level metrics against locally pinned fixture repositories (declared in an uncommitted fixtures.json, see below), and is designed to be re-run after each feature PR with compare.py diffing before/after.

Quick start

# Rebuild the binary first (required — deployed binary may lag)
cd /path/to/code-graph
CGO_ENABLED=1 go build -o bin/code-graph.exe ./cmd/code-graph/

# One-time fixture setup: create the main-pinned worktree for the self-hosting fixture
git worktree add "$HOME/worktrees/code-graph-main" main
./bin/code-graph.exe cli --raw index_repository \
  '{"repo_path":"'$HOME'/worktrees/code-graph-main"}'

# Full baseline across all 4 fixtures
python bench/harness.py --output bench/baseline_$(date +%F).json \
  --binary ./bin/code-graph.exe

# Skip re-index (use whatever's cached in ~/.cache/code-graph/)
python bench/harness.py --output bench/baseline_$(date +%F).json \
  --binary ./bin/code-graph.exe --skip-index

# Smoke test (Q1+Q2 only on one repo, fast feedback)
python bench/harness.py --output bench/smoke.json --smoke --repo mcp-servers \
  --binary ./bin/code-graph.exe --skip-index

# Diff two baselines (before vs after a feature PR)
python bench/compare.py bench/baseline_2026-04-22.json bench/after_pr1.json

Opt-in: Voyage embeddings during indexing

By default the harness sets VOYAGE_API_KEY="" during indexing to skip the embeddings pass. This avoids a known indefinite stall in the embeddings HTTP loop (see Known issues below). Semantic-search (search_code_semantic) will return empty until a fresh index with embeddings runs.

# Opt in — only when semantic_search data is needed (still subject to stall risk)
python bench/harness.py --output bench/baseline.json \
  --binary ./bin/code-graph.exe --with-embeddings

Files

What the harness captures per question

• Wall-clock latency (ms)
• Grade: PASS / PARTIAL / FAIL / N/A / ERROR
• Result hash (SHA256 prefix) — detects silent behavior changes between runs
• Short preview of the raw result

Grading notes

• N/A is used when:
  • A feature PR hasn't merged yet (for Q13-Q20)
  • The installed binary doesn't contain a tool the source does (binary-lag case)
  • The question doesn't apply to the repo
• ERROR is reserved for unexpected tool errors on tools that should work
• PASS requires the question's expect_contains / expect_min_results / expect_non_empty predicate to hold

Known issues + workarounds

1. Embeddings-pass indefinite stall (VOYAGE_API_KEY set)

Symptom: on larger repos (~4k+ nodes), the indexer logs phase=embeddings pct=97 detail="generating embeddings" and produces no further output for many minutes. The process is alive but no pass.embeddings.progress log fires. Observed 2026-04-22 on rmf-corsair at 8+ minutes.

Root cause (not confirmed — deferred to a separate investigation):

• Not DNS/IPv6 — reproduced with GODEBUG=netdns=go+v4.
• The pass has no overall timeout; only a per-request 120s http.Client.Timeout in voyage_client.go. If requests return slowly but successfully, the pass can run for an unbounded time on a large batch queue with no user-visible progress.

Harness workaround: default VOYAGE_API_KEY="" during index_repository calls (Harness.with_embeddings=False by default). Semantic-search index is not populated; everything else indexes fine.

Opt-in: --with-embeddings flag. Still subject to stall risk.

Proper fix (not in this PR): add a bounded context.WithTimeout around the embeddings pass in pass_embeddings.go; emit progress log per inner batch (currently only per outer batch of 128); expose a CODE_GRAPH_SKIP_EMBEDDINGS env var for unambiguous opt-out.

2. Binary version lag

The harness runs against whatever --binary path points to. If you pass ~/bin/code-graph.exe, it may be stale. Always build fresh from the current code-graph source before capturing a baseline (see Quick Start above). Stale binary produces false N/As for tools that exist in source (explain_symbol, get_review_context, etc.) and polluted comparisons.

3. Git worktree path mangling on Windows Git Bash

git worktree add ~/tmp/foo main produces a worktree at C:/c/Users/user/tmp/foo (the ~ expansion confuses git's path handling on Windows). Use explicit Windows-style absolute paths: git worktree add "$HOME/worktrees/code-graph-main" main (the $HOME env var expands correctly because it's shell-expanded before git sees it).

4. Fixture paths are portable

fixtures.json uses $HOME-prefixed paths ($HOME/Documents/GitHub/...). The harness expands both ~ and $VAR via os.path.expandvars(os.path.expanduser(...)) so the same fixture file works on any machine with the expected directory layout. Don't hard-code absolute paths.

5. Stale code-graph processes after a stall kill

If a stalled index is killed via taskkill /F, orphan processes can linger. Clean with:

MSYS_NO_PATHCONV=1 taskkill /F /IM code-graph.exe

Then delete_project to clear any 0-node stub left in the registry:

./bin/code-graph.exe cli --raw delete_project \
  '{"project_name":"c-Users-user-Documents-GitHub-<repo>"}'

The harness itself has a stall watchdog (--index-timeout, default 600s) that surfaces a clean error instead of orphaning the subprocess.

Fixture repo SHAs

Captured in fixtures.json. A baseline run prints a WARN if any fixture's HEAD SHA has drifted from the pinned value, and a WARN if any fixture has uncommitted changes. Baselines from drifted/dirty repos are still written but flagged — do not use them as reference points for PR diffing.

Question ID coverage matrix

Baseline findings (2026-04-22, freshly-built binary from main + feat/phase0)

Clean baselines captured:

• mcp-servers @ 81fa7d5 (2,920 nodes / 7,318 edges): 15 PASS / 5 N/A (N/As are the 5 feature-PR probes)
• mcp-infra @ 8173017 (668 nodes / 1,096 edges): 15 PASS / 5 N/A

Pending baselines (needs index_repository first — harness was run with --skip-index):

• rmf-corsair @ f545976: not yet indexed
• code-graph @ c9b1195: on this feature branch (dirty) — baseline should be captured from main

Findings that reshape the feature PR plan

1. PR 2 scope is substantially smaller than planned. Current get_architecture already returns clusters with per-cluster cohesion scores. Sample from mcp-servers: {id:1, label:"msgraph_mcp", members:94, cohesion:0.96875}. 157 Community nodes exist in the graph. Remaining PR 2 work: verify the oversized-community split threshold (>25% of graph) actually fires. May drop from M effort to XS, or be removed entirely if split logic is already present.

2. Tool surface has shifted since plan was written.

   • list_directory removed — replaced by search_graph(label="File") in Q12.
   • Schema key edge_types renamed to relationship_types in get_graph_schema output.
   • get_architecture output shape changed — no top-level node_labels; see clusters, boundaries, hotspots, packages keys instead.
   • Cypher parser rejects <> (not-equal) — Q10 adjusted to not use it.

3. Q19 and Q20 already pass on current source. explain_symbol and get_review_context exist. PR 4 and PR 5's scope is additive enhancement to these existing tools (rationale extraction; graph-diff inclusion) rather than building from scratch.

Phase 0b — transcript replay corpus

Reads Claude Code session transcripts from ~/.claude/projects/ and summarizes every session that invoked mcp__code-search__* or mcp__code-graph__* tools. One JSONL record per qualifying session.

Usage

# default: last 14 days, write bench/transcripts_YYYY-MM-DD.jsonl, print stats
python bench/transcripts.py --stats

# dry-run — count without writing
python bench/transcripts.py --count-only

# custom window + output
python bench/transcripts.py --days 30 --output bench/scan.jsonl

# sample N random sessions (seed=42) for hand-labeling outcomes
python bench/transcripts.py --sample 30 --output bench/sample_hand-label.jsonl

Baseline captured 2026-04-22 (14-day window, 299 transcripts scanned)

What pre_graph_grep measures

For each session we compute first_graph_call_index = the position of the first mcp__code-search__* or mcp__code-graph__* tool call. If any Glob/Grep/Read call precedes that index, the session counts as "pre-graph grep" (the behavior PR 3's PreToolUse hook aims to reduce). On the current main-branch baseline, 76.7% of sessions exhibit this.

A/B usage (post-PR-3)

After PR 3 ships, re-run transcripts.py over the next 14-day window and compare pre_graph_grep rate. Target delta: ≥40 pp reduction (baseline 76.7% → target ≤ 37%). Stop-ship criterion for PR 3 is met only if the delta is observable across a meaningfully-sized sample (≥30 sessions).

Measuring PR 3 (PreToolUse orientation hook) — A/B protocol

Script: bench/compare_pregrep.py — compares the pre_graph_grep rate between a baseline corpus (pre-install) and a post-install corpus.

Protocol

1. Capture a baseline (already done in PR #51 — 76.7% on 14-day window, 30 qualifying sessions):

   python bench/transcripts.py --output bench/transcripts_pre.jsonl
   

2. Install the hook in your live Claude Code environment:

   code-graph install
   # writes ~/.claude/hooks/codebase-memory-orientation.sh and
   # registers a PreToolUse matcher Glob|Grep in ~/.claude/settings.json
   

3. Restart Claude Code.
4. Use Claude Code normally for ≥14 days (comparable sample to baseline).
5. Capture the post-install corpus:

   python bench/transcripts.py --output bench/transcripts_post.jsonl
   

6. Run the comparison:

   python bench/compare_pregrep.py \
     --baseline bench/transcripts_pre.jsonl \
     --post     bench/transcripts_post.jsonl
   

   Or without a pre-corpus file, against the flat baseline rate:

   python bench/compare_pregrep.py \
     --post bench/transcripts_post.jsonl \
     --baseline-rate 0.767
   

Stop-ship criteria (PR 3)

• Post pre_graph_grep rate ≤ 37% AND
• Delta ≥ 40 percentage points AND
• Post corpus size ≥ 30 qualifying sessions

Script exits 0 on PASS, 1 on FAIL, and prints a per-session-length breakdown (short / medium / long) so you can see whether the hook helps long sessions more than short ones — an expected pattern, since long sessions have more Glob/Grep opportunities for the hook to fire on.

Follow-up PRs

• Phase 0c: PR ground-truth set (separate PR)
• CI integration: gate feature PRs on compare.py output (future)