runner

Overview ¶

Command runner is the codegraph benchmark harness: it drives the freshly-built Go `codegraph` binary — and, in headtohead mode, the installed TS `codegraph@1.3.1` binary too — through a shared, external measurement path and reports internal/bench.Metrics.

Two modes (-mode):

• headtohead: for each tools/bench/realcorpus entry, runs BOTH the Go and TS binaries over the real, pinned repo and prints raw per-repo/per-subject Metrics as JSON to stdout (PERF-01). This is a published comparison, not a gate — it never fails the process on its own.

• regression: materializes the deterministic, network-free synthetic corpus (tools/bench/gencorpus) and runs ONLY the Go binary over it, then checks the result against the committed baseline (-baseline) via internal/bench.CheckRegression (PERF-02 + the INDX-06 absolute peak-RSS ceiling, -ceiling-bytes). Exits non-zero when CheckRegression returns an error — this is the blocking CI gate. -rebless is the ONLY flag that causes this mode to overwrite -baseline instead of gating against it.

Every measured command is run 5 times (medianRuns) and each metric's own median is reported, per CONTEXT.md D-05. Regression mode adds a second, outer level of repetition on top of that: -trials N (default defaultTrials) repeats the whole materialize+init+measure session N times and medians each metric across sessions, because median-of-5 inside one session does not touch session-to-session variance — see defaultTrials. Peak RSS is ALWAYS read from the completed child process via bench.PeakRSSBytes — never via this process's own in-process memory statistics, which cannot be compared fairly against the TS Node process. Only fixed, pinned repo paths or the generated corpus's own scratch directory are ever passed as subprocess arguments (V5) — this tool never forwards attacker-controlled input to either shelled-out binary.