bench

Overview ¶

Command bench runs the mcpproxy benchmark.

Legacy offline mode (no profiler flags) scores the committed Spec 065 frozen corpus for token reduction and writes the v1 JSON report plus a static HTML dashboard:

go run ./bench/cmd/bench [-corpus PATH] [-out DIR] [-encoding NAME]

Profiler mode (Spec 083) measures deterministic encoding arms on frozen corpora and writes the v2 report + dashboard:

go run ./bench/cmd/bench -corpus-v2 specs/083-discovery-profiler/datasets/corpus_v2.tools.json \
  -arms all -out bench/results
go run ./bench/cmd/bench -toolret bench/results/cache/toolret -subset 250 -seed 42 \
  -arms baseline_json,compact_sig -out bench/results
go run ./bench/cmd/bench -livemcptool specs/083-discovery-profiler/datasets/livemcptool_snapshot \
  -arms baseline_json,compact_sig,tscg,toon_listing -out bench/results

Live mode boots against a running proxy (see bench/docker-compose.yml) to add the exact-token comparison (full schemas), retrieval accuracy, search latency, and — Spec 083 US1 — the retrieve_tools RESPONSE cost over the real MCP protocol with break-even analysis:

go run ./bench/cmd/bench -live [-proxy URL] [-api-key KEY] [-golden PATH] \
  [-corpus-v2 PATH] [-expected-tools N]

In live mode -corpus-v2 supplies the full-definition schemas for the naive full-menu count (joined to live tools by id — GET /api/v1/tools can serve stub schemas), and -expected-tools surfaces corpus drift (FR-021).

A LAP lint artifact (`uvx --from lap-score==0.8.0 lap lint --json`) merges into either report via -lap-json. Reports land in bench/results/ (gitignored — reports are never committed, per the Spec 065 CN-003 rule).