bench

command
v0.73.0-rc.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 7, 2026 License: MIT Imports: 13 Imported by: 0

Documentation

Overview

Command bench runs the mcpproxy benchmark.

Legacy offline mode (no profiler flags) scores the committed Spec 065 frozen corpus for token reduction and writes the v1 JSON report plus a static HTML dashboard:

go run ./bench/cmd/bench [-corpus PATH] [-out DIR] [-encoding NAME]

Profiler mode (Spec 083) measures deterministic encoding arms on frozen corpora and writes the v2 report + dashboard:

go run ./bench/cmd/bench -corpus-v2 specs/083-discovery-profiler/datasets/corpus_v2.tools.json \
  -arms all -out bench/results
go run ./bench/cmd/bench -toolret bench/results/cache/toolret -subset 250 -seed 42 \
  -arms baseline_json,compact_sig -out bench/results
go run ./bench/cmd/bench -livemcptool specs/083-discovery-profiler/datasets/livemcptool_snapshot \
  -arms baseline_json,compact_sig,tscg,toon_listing -out bench/results

Live mode boots against a running proxy (see bench/docker-compose.yml) to add the exact-token comparison (full schemas), retrieval accuracy, search latency, and — Spec 083 US1 — the retrieve_tools RESPONSE cost over the real MCP protocol with break-even analysis:

go run ./bench/cmd/bench -live [-proxy URL] [-api-key KEY] [-golden PATH] \
  [-corpus-v2 PATH] [-expected-tools N]

In live mode -corpus-v2 supplies the full-definition schemas for the naive full-menu count (joined to live tools by id — GET /api/v1/tools can serve stub schemas), and -expected-tools surfaces corpus drift (FR-021).

Replay mode (Spec 103 US1) recomputes cost over a REAL recorded workload: an activity export plus a MANDATORY fleet input, deterministic, no model spend:

go run ./bench/cmd/bench -replay /path/outside/the/repo/activity.jsonl \
  -corpus-v2 specs/083-discovery-profiler/datasets/corpus_v2.tools.json -out bench/results

The fleet may instead be read live from a running proxy's direct MCP surface, so an operator can price their OWN fleet without exporting a corpus first (Spec 103):

go run ./bench/cmd/bench -replay /path/outside/the/repo/activity.jsonl \
  -proxy http://127.0.0.1:8080 -api-key KEY -out bench/results

The fleet input is not optional, and it is not plural. A menu is a property of the tool definitions and the activity export carries no fleet snapshot, so a recording on its own computes nothing — `-replay` with no fleet flag at all is an error, not a degraded run. Supplying TWO (say -corpus-v2 and -proxy) is also an error: every figure is quoted for exactly one fleet shape, and silently preferring one would make the reported fleet_id false.

A LAP lint artifact (`uvx --from lap-score==0.8.0 lap lint --json`) merges into either report via -lap-json. Reports land in bench/results/ (gitignored — reports are never committed, per the Spec 065 CN-003 rule).

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL