runner

Overview ¶

Command runner is the codegraph benchmark harness: it drives the freshly-built Go `codegraph` binary through a shared, external measurement path and reports internal/bench.Metrics.

Two modes (-mode):

• publish: for each tools/bench/realcorpus entry, runs the Go binary over the real, pinned repo and prints raw per-repo, absolute Metrics as JSON to stdout — every record's subject is always "go". This is a published measurement, not a gate — it never fails the process on its own.

• regression: materializes the deterministic, network-free synthetic corpus (tools/bench/gencorpus) and runs ONLY the Go binary over it, then checks the result against the committed baseline (-baseline) via internal/bench.CheckRegression (PERF-02 + the INDX-06 absolute peak-RSS ceiling, -ceiling-bytes). Exits non-zero when CheckRegression returns an error — this is the blocking CI gate. -rebless is the ONLY flag that causes this mode to overwrite -baseline instead of gating against it.

Every measured command is run 5 times (medianRuns) and each metric's own median is reported, per CONTEXT.md D-05. Regression mode adds a second, outer level of repetition on top of that: -trials N (default defaultTrials) repeats the whole materialize+init+measure session N times and medians each metric across sessions, because median-of-5 inside one session does not touch session-to-session variance — see defaultTrials. Peak RSS is ALWAYS read from the completed child process via bench.PeakRSSBytes — never via this process's own in-process memory statistics — because an externally-observed child-process number is the only figure that stays comparable run to run. Only fixed, pinned repo paths or the generated corpus's own scratch directory are ever passed as subprocess arguments (V5) — this tool never forwards attacker-controlled input to the shelled-out binary.