Documentation
¶
Overview ¶
Command bench runs the mcpproxy benchmark.
Legacy offline mode (no profiler flags) scores the committed Spec 065 frozen corpus for token reduction and writes the v1 JSON report plus a static HTML dashboard:
go run ./bench/cmd/bench [-corpus PATH] [-out DIR] [-encoding NAME]
Profiler mode (Spec 083) measures deterministic encoding arms on frozen corpora and writes the v2 report + dashboard:
go run ./bench/cmd/bench -corpus-v2 specs/083-discovery-profiler/datasets/corpus_v2.tools.json \ -arms all -out bench/results go run ./bench/cmd/bench -toolret bench/results/cache/toolret -subset 250 -seed 42 \ -arms baseline_json,compact_sig -out bench/results go run ./bench/cmd/bench -livemcptool specs/083-discovery-profiler/datasets/livemcptool_snapshot \ -arms baseline_json,compact_sig,tscg,toon_listing -out bench/results
Live mode boots against a running proxy (see bench/docker-compose.yml) to add the exact-token comparison (full schemas), retrieval accuracy, search latency, and — Spec 083 US1 — the retrieve_tools RESPONSE cost over the real MCP protocol with break-even analysis:
go run ./bench/cmd/bench -live [-proxy URL] [-api-key KEY] [-golden PATH] \ [-corpus-v2 PATH] [-expected-tools N]
In live mode -corpus-v2 supplies the full-definition schemas for the naive full-menu count (joined to live tools by id — GET /api/v1/tools can serve stub schemas), and -expected-tools surfaces corpus drift (FR-021).
Replay mode (Spec 103 US1) recomputes cost over a REAL recorded workload: an activity export plus a MANDATORY fleet input, deterministic, no model spend:
go run ./bench/cmd/bench -replay /path/outside/the/repo/activity.jsonl \ -corpus-v2 specs/083-discovery-profiler/datasets/corpus_v2.tools.json -out bench/results
The fleet may instead be read live from a running proxy's direct MCP surface, so an operator can price their OWN fleet without exporting a corpus first (Spec 103):
go run ./bench/cmd/bench -replay /path/outside/the/repo/activity.jsonl \ -proxy http://127.0.0.1:8080 -api-key KEY -out bench/results
The fleet input is not optional, and it is not plural. A menu is a property of the tool definitions and the activity export carries no fleet snapshot, so a recording on its own computes nothing — `-replay` with no fleet flag at all is an error, not a degraded run. Supplying TWO (say -corpus-v2 and -proxy) is also an error: every figure is quoted for exactly one fleet shape, and silently preferring one would make the reported fleet_id false.
A LAP lint artifact (`uvx --from lap-score==0.8.0 lap lint --json`) merges into either report via -lap-json. Reports land in bench/results/ (gitignored — reports are never committed, per the Spec 065 CN-003 rule).