bench

package
v0.64.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 2, 2026 License: MIT Imports: 29 Imported by: 0

README

mcpproxy benchmark harness

The reproducible numbers behind mcpproxy's marketing claims — token reduction, discovery accuracy, and latency — comparing three ways an agent can be wired to upstream MCP tools, plus the discovery-effectiveness profiler (Spec 083): encoding-arm comparisons, real-MCP response cost, break-even analysis, session-cost estimates, public-corpus validation, and an independent LAP verdict.

Roadmap item #19 (MCP-42) + Spec 083. In-repo (bench/), reproducible, intended to be refreshed on release. Reports are never committed (Spec 065 CN-003); only code, fixtures, and this methodology are versioned.

The three modes

Mode What the agent sees in context mcpproxy server
baseline Every upstream tool definition, loaded directly (no proxy discovery)
retrieve_tools retrieve_tools + call_tool_read/write/destructive + read_cache + code_execution + management tools; tools found on demand via BM25 callToolServer
code_execution code_execution + retrieve_tools + management tools; many tools orchestrated from sandboxed JS in one round-trip codeExecServer

Both proxy modes also append the shared management tool set — upstream_servers, quarantine_security, search_servers, list_registries — that the live routing-mode servers expose. These count against the proxy context cost: omitting them undercounts that cost and inflates the savings.

The per-mode catalog is derived directly from the live tool builders (buildCallToolModeTools / buildCodeExecModeTools in internal/server/mcp_routing.go, via server.ProxyModeToolDefs), so it can never drift from production.

What ships today (deterministic, offline)

The token-reduction measurement is fully deterministic and runs with no network or LLM:

go run ./bench/cmd/bench            # scores the committed Spec 065 corpus (v1 report)
make bench-discovery                # Spec 083 profiler: all encoding arms on corpus_v2 (v2 report)
go test ./bench/...                 # unit + invariant tests

It counts the context-token cost of each mode over a frozen tool corpus and reports the savings of each proxy mode versus the baseline. Output: a report.json and a self-contained dashboard.html in bench/results/ (gitignored).

Current deterministic result

Over the 45-tool Spec 065 reference corpus, counting tool name + description only (schemas excluded uniformly — see limitations), cl100k_base:

Mode Context tools Tokens Savings vs. baseline
baseline 45 1730 —
retrieve_tools 10 1431 ~17%
code_execution 6 986 ~43%

These are deliberately modest: the proxy context here is the full per-mode tool set (discovery + call-tool variants + management tools), and the corpus is small. Savings grow toward the asymptote as the upstream tool count rises (the baseline grows linearly while the proxy context stays fixed) — always quote the corpus size alongside a percentage. Reproduce with go run ./bench/cmd/bench.

Scoring rubric — token reduction
  • Tool universe: the frozen Spec 065 snapshot specs/065-evaluation-foundation/datasets/corpus_v1.tools.json — 45 tools across 7 no-auth reference servers. Frozen + versioned so scoring never runs against a drifting corpus (CN-002).
  • Tokenizer: tiktoken cl100k_base, a widely-used reproducible BPE (already a repo dependency). It is a model-agnostic estimator — see the tokenizer caveat under "Known limitations" before quoting an absolute number.
  • Proxy-mode tools: the complete per-mode catalog, derived from the live server builders — discovery, the call-tool variants, code_execution, and the shared management tool set (upstream_servers, quarantine_security, search_servers, list_registries). Nothing the agent actually sees is dropped from the proxy cost.
  • Cost of a tool: name + "\n" + description. JSON input schemas are excluded uniformly across all modes (the committed corpus snapshot does not carry schemas). The Spec 083 profiler measures schema-bearing renderings on corpus_v2 instead.
  • Savings for a mode m: 1 - tokens(m) / tokens(baseline).

Discovery-effectiveness profiler (Spec 083)

The profiler answers three questions the v1 harness could not: what does a retrieve_tools response actually cost over the real MCP protocol, which tool-definition encoding is cheapest without wrecking retrieval quality, and do the in-house numbers survive external corpora and an independent linter. Output is a versioned v2 report (report.json conforming to specs/083-discovery-profiler/contracts/report-v2.schema.json) plus a self-contained dashboard.html, where every headline number carries a provenance badge: measured (observed on the wire), computed (deterministic arithmetic over measurements), or estimated (a model with documented assumptions).

Quickstart

One command (SC-008):

make bench-discovery
# = npm ci --prefix bench/tscg  (pinned @tscg/core, one-time)
#   + go run ./bench/cmd/bench -corpus-v2 ... -arms all -out bench/results
# → bench/results/report.json (v2) + bench/results/dashboard.html
# CI runs this same target.

Offline (deterministic, no network):

# All mandatory arms on the schema-bearing frozen corpus
go run ./bench/cmd/bench \
  -corpus-v2 specs/083-discovery-profiler/datasets/corpus_v2.tools.json \
  -arms baseline_json,compact_sig,tscg,toon_listing \
  -out bench/results

TSCG arm prerequisite (covered by make bench-discovery): npm ci --prefix bench/tscg (Node ≥20). Without it the arm reports skipped: node runtime unavailable (allowed locally, fails SC-002 in CI).

Live (response cost + break-even + latency):

# 1. Boot the snapshot proxy (same as existing bench live mode)
go build -o mcpproxy ./cmd/mcpproxy
./mcpproxy serve --config specs/065-evaluation-foundation/datasets/snapshot-servers.config.json \
  --listen 127.0.0.1:8092 &   # MCPPROXY_API_KEY=eval-corpus-snapshot

# 2. Run live measurement (real MCP retrieve_tools calls per golden query)
go run ./bench/cmd/bench -live \
  -proxy http://127.0.0.1:8092 -api-key eval-corpus-snapshot \
  -golden specs/065-evaluation-foundation/datasets/retrieval_golden_v1.json \
  -out bench/results

Public corpora:

# ToolRet (runtime fetch — never committed; license unstated upstream)
./scripts/fetch-toolret.sh                # uv-run parquet→JSON into bench/results/cache/toolret/
go run ./bench/cmd/bench -toolret bench/results/cache/toolret \
  -subset 250 -seed 42 -arms baseline_json,compact_sig -out bench/results

# LiveMCPTool (committed Apache-2.0 snapshot) — token/scale measurement
go run ./bench/cmd/bench \
  -livemcptool specs/083-discovery-profiler/datasets/livemcptool_snapshot \
  -arms baseline_json,compact_sig,tscg,toon_listing -out bench/results

Independent LAP verdict (what CI runs):

uvx --from lap-score==0.8.0 lap lint \
  --mcp-url "http://127.0.0.1:8092/mcp?apikey=eval-corpus-snapshot" --json > bench/results/lap.json

Regenerating fixtures (maintainers only):

./scripts/gen-corpus-v2.sh     # schema-bearing frozen corpus from booted snapshot proxy
# result_fixtures_v1.json: captured once from reference servers; see datasets/README

Verifying success criteria:

  • SC-003 gate: baseline arm on in-house golden set must report recall@5 = 0.68 ± 0.05.
  • SC-001: report.json .response_cost.p50/.p95, .break_even.break_even_calls populated after a live run.
  • SC-005: dashboard shows provenance badge + tokenizer caveat on every headline number.
Flags
Flag Default Meaning
-corpus specs/065-evaluation-foundation/datasets/corpus_v1.tools.json Legacy v1 corpus (name+description only); used when no profiler flag is set
-out bench/results Output directory for report.json + dashboard.html
-encoding cl100k_base tiktoken encoding name
-live off Live benchmark against a running proxy: full-schema tokens, accuracy, latency, and real-MCP response cost + break-even
-proxy http://127.0.0.1:8092 Live proxy base URL
-api-key eval-corpus-snapshot Live proxy API key (X-API-Key; the MCP session uses ?apikey=)
-golden specs/065-evaluation-foundation/datasets/retrieval_golden_v1.json Retrieval golden set (the SC-003 gate corpus)
-arms (empty) Comma-separated encoding arms, or all — enables profiler mode. toon_results is selectable here as a fixture-driven pseudo-arm
-corpus-v2 (empty) Spec 083 schema-bearing frozen corpus (corpus_v2.tools.json)
-toolret (empty) ToolRet cache dir from scripts/fetch-toolret.sh (revision dir, or its parent with exactly one revision) — runtime fetch, never committed
-livemcptool (empty) LiveMCPTool committed snapshot directory (or its tools.json)
-subset 250 Seeded ToolRet query-subset size (FR-014)
-seed 42 Seed for the deterministic query subset — same revision+seed+size ⇒ same subset
-lap-json (empty) Path to a LAP lint artifact to merge as the independent verdict
-result-fixtures specs/083-discovery-profiler/datasets/result_fixtures_v1.json Tool-result fixture set for the toon_results arm

Any of -arms / -corpus-v2 / -toolret / -livemcptool switches the CLI into profiler mode (v2 report); -live runs the live benchmark and also emits a v2 report; with none of them the legacy v1 offline run executes.

Encoding arms

An arm is one deterministic way of rendering tool definitions into agent-context text (bench/arms/; behavioral contract in specs/083-discovery-profiler/contracts/arm-interface.md). Arms are byte-deterministic (FR-010), fail explicitly instead of truncating silently (an unencodable tool is a counted skip, with examples in the report), and declare whether they alter what the retrieval index ingests — index-altering arms are scored for retrieval quality through the production Bleve index (bench/armindex.go), where the baseline arm must reproduce the golden-set recall@5 = 0.68 ± 0.05 (SC-003 gate).

Arm What it renders (one-line example) Index-altering Lower-bound
baseline_json Full definition, canonical JSON schema — fetch ⏎ Fetches a URL from the internet… ⏎ {"properties":{"url":{…}},"required":["url"],…}. THE canonical renderer: it is also the naive-menu count, every savings denominator, and the break-even input no no
compact_sig Flat signature fetch(url:string, max_length?:int, raw?:bool, start_index?:int)|Fetches a URL from the internet… — required vs opt? params, nested objects collapse to obj, descriptions preserved verbatim yes no
tscg Reference TSCG compiler (pinned @tscg/core@1.4.3 via the committed Node shim bench/tscg/shim.mjs, JSONL protocol) — fetch: Fetches a URL… ⏎ max_length (integer): Maximum number of characters… | raw (bool): …. Rewrites/elides description filler, so savings are a lower bound. Without node/node_modules the whole arm is a skip-with-reason row (never per-tool failures) yes yes
toon_listing Official TOON encoding (toon-go) of {name, description, inputSchema} per tool — - name: fetch ⏎ description: "Fetches a URL…" ⏎ inputSchema: ⏎ properties: … yes no
tron_dedup In-tree TRON-style named-class schema dedup (no upstream Go impl): byte-identical canonical schemas declared once in the listing preamble — class Cbae0cfe3 = {…canonical schema…} — and referenced per tool as fetch|Fetches a URL…|Cbae0cfe3. Content-addressed class names, exact-bytes dedup key: deliberately conservative vs. the paper (deviations documented in bench/arms/tron.go) yes no
toon_results Pseudo-arm over tool-call outputs, not definitions: the committed result_fixtures_v1.json payloads encoded as TOON vs. a compact-JSON baseline of the same payloads, split tabular vs. non-tabular. Selectable via -arms; included automatically by -arms all when the fixture file exists n/a no

Listing-level amortization (TOON header, TRON class preamble) lives only in EncodeListing; per-tool means stay comparable across arms. Each arm row also reports skip counts with examples, the top-N heaviest tools, and the corpus's degenerate-description count (FR-020: description missing, shorter than 20 runes, or matching stub patterns) so cheap-but-useless corpora are visible.

Live response cost, break-even, and session estimates (US1)

The -live run now measures what the v1 REST-based run could not — the response an agent pays for on every discovery call:

  • Real MCP protocol. bench/mcpcall.go speaks streamable-HTTP MCP (mark3labs/mcp-go) against the proxy's /mcp endpoint and calls retrieve_tools for every golden query — capturing the exact response text an agent's context ingests (REST /api/v1/index/search misses usage_instructions, call_with, annotations, session_risk), plus client-measured latency. Provenance: measured.
  • Span-based component attribution (bench/respcost.go). Per-query tokens split into input_schemas / descriptions / usage_instructions / metadata / other. BPE is not additive across concatenation boundaries, so the response is partitioned into labeled byte spans, tokenized once, and each token attributed to the span owning its starting byte — components sum exactly to the total (FR-002), by construction. Percentiles: p50/p95/max + mean.
  • Break-even (bench/breakeven.go): break_even_calls = (naive_full_menu_tokens − proxy_menu_tokens) / mean_response_tokens, all three inputs echoed in the row so the number is recomputable. Both menus come from the same canonical renderer. Numerator ≤ 0 ⇒ explicit no_break_even row; break-even is withheld when menu token counts are non-authoritative (see the MCP-3161 safety valve below). Provenance: computed.
  • Session-cost estimator (bench/session.go) — an ESTIMATE, not a measurement: session_cost(arm, calls) = proxy_menu + calls × mean_response(arm) × (1 + retry_rate(arm)) for calls ∈ {1,3,5,10}. retry_rate defaults are literature-derived (0.0 for format-native JSON/compact/TSCG/TRON; 0.05 for toon_listing, per the parsing-cascade evidence in arXiv:2605.29676 §5) and echoed in every row. Provenance: estimated, always.
LAP independent verdict (US4)

CI runs the version-pinned external linter lap-score==0.8.0 against the booted proxy (command in the quickstart) and merges its artifact via -lap-json: the report's lap block carries LAP's letter grade and menu-token count, compared against the in-house count of the same surface. Both use cl100k_base but frame the menu differently, so divergence within ±15% is expected; beyond it the report shows a non-blocking warning (FR-016). LAP being absent or broken never fails the benchmark (FR-015): every failure path yields executed: false with an explicit skip reason.

Live run — full schemas + accuracy + latency

The live run boots mcpproxy over the Spec 065 reference-server config and measures the headline claims against a running proxy. Everything here is still deterministic and LLM-free.

# 1. Boot the reproducible substrate (proxy + 7 no-auth reference servers)
docker compose -f bench/docker-compose.yml up --build -d

# 2. Score against the running proxy (writes bench/results/live_report.json + v2 report)
go run ./bench/cmd/bench -live -proxy http://127.0.0.1:8092 -api-key eval-corpus-snapshot

What it adds over the offline token run:

  • Exact token number (full schemas). Pulls GET /api/v1/tools for the upstream tools with their full JSON input schemas and counts them against the proxy modes — whose management-tool schemas come from the same live builders as the offline run (server.ProxyModeToolDefs). Because schemas are counted on both sides, the savings is authoritative.
    • Safety valve (MCP-3161): if any proxy tool is missing a schema, counting the baseline's schemas alone would overstate savings, so the run withholds the headline % and reports raw token totals only (authoritative_headline: false) — and withholds break-even, which depends on the same menus. Never quote a withheld run.
  • Accuracy. Replays retrieval_golden_v1.json through the proxy's BM25 search (GET /api/v1/index/search) and scores Recall@{1,3,5,10}, MRR, nDCG@10, MAP against the graded labels. Deterministic (BM25), so a single run is reported (runs_averaged: 1). The emitted retrieval block conforms to the Spec 065 score-report.schema.json shape — nested metrics + gate (verified by a schema-validation test). A standalone live run has no stored baseline to regress against, so gate.passed is true by construction; CI regression-gating against a committed baseline is the MCP-3133 lane.
  • Latency. Client-measured per-query search latency (p50/p95/p99/max) vs. the one-shot cost of loading all tools. Measured client-side on purpose: the server's SearchToolsResponse.took field is currently a "0ms" stub.
  • Response cost + break-even (Spec 083 US1) — see the profiler section above. The live run emits both the v1 live_report.json and the v2 report.json/dashboard.html.
Compact arm / flip gates (Spec 085, -flip-gates)

-live -flip-gates additionally replays the golden set through the proxy's MCP retrieve_tools tool twice per query — once with detail=full, once with detail=compact (same pipeline, FR-017) — and emits the Spec 085 flip-gate metrics under flip_gates in live_report.json (FR-018):

  • per-query ranked-ID identity across modes (gate: 100%, SC-002 — any mismatch is listed and fails the gate),
  • full vs compact response-token distributions (p50/p95/max) and the median reduction (SC-001 gates on ≥50% on a real corpus),
  • the lossy-signature rate via the shared internal/toolsig grammar (gate: <20%, SC-005), with the heaviest signatures for triage,
  • describe_tool usage per completed task (informational; populated by the E2E suite, not a live run).
go run ./bench/cmd/bench -live -flip-gates -proxy http://127.0.0.1:8092 -api-key eval-corpus-snapshot

Real-workload replay + agent loop (Spec 103)

Everything above scores a frozen corpus. Spec 103 adds the two entry points that score a real workload: a deterministic replay of recorded activity (no model spend, US1) and a live agent loop under a pinned model (US2). The replay entry point lands with US1; the agent loop is gated on an operator decision (pinned model + spend ceiling) and is tracked under "not yet built" below until that is made.

Both cross the five-cell mode matrix — retrieve_full, retrieve_compact, direct_full, direct_deferred, code_exec. The three axes (routing mode × response mode × schema detail) have twelve combinations, but only five are distinct behaviours; the other seven are configurable, redundant, and reported as skip rows naming the cell they collapse onto.

Replay — recompute a recorded workload (deterministic, no model spend)

1. Export a real workload, from a machine that has been using mcpproxy for real work:

mcpproxy activity export --format json > ~/replay-corpus.jsonl

CSV is not a valid input: it drops work_session_id, the byte fields and the call arguments, so it can neither group units of work nor account tokens.

2. Recompute across the matrix. A fleet input is MANDATORY — a menu cost is a property of the tool definitions the agent was shown, and the activity export carries no fleet snapshot. The recording contributes the call shape (sequence, tool mix, call counts); it does not contribute fleet size. A recording-only invocation is a hard error, not a degraded run.

# Frozen fleet (reproducible):
go run ./bench/cmd/bench \
  -replay ~/replay-corpus.jsonl \
  -corpus-v2 specs/083-discovery-profiler/datasets/corpus_v2.tools.json \
  -out bench/results

# ...or read today's fleet from a live proxy instead:
go run ./bench/cmd/bench -replay ~/replay-corpus.jsonl \
  -proxy http://127.0.0.1:8092 -api-key "$KEY" -out bench/results

Either way the workload is scored against the supplied fleet, not the fleet as it stood when the session was recorded. That is internally valid across cells and is not a historical reconstruction — the report says so on every figure.

3. Bodies-off is the DEFAULT, and it decides which figures exist. The export path does not mask (masking is wired into the list and detail handlers only), so a bodies-on export is raw and unmasked by design. Bodies-on is therefore a separate, explicit opt-in that prints a warning, and content is tokenized inside the loader — only counts cross the boundary, never text.

bodies-off (default) bodies-on (opt-in)
Menu cost, all five cells measured (from the fleet input) measured
Absolute complete-workload cost unavailable for every cell — reported unavailable, never as zero measurable where bodies survive
direct_full vs direct_deferred delta measured — their call responses are identical and cancel measured
retrieve_full / retrieve_compact / code_exec deltas unavailable — the mode changes the response body itself measured
Response cost generally estimated at best (request_bytes/response_bytes are byte lengths, not token counts) measured

4. Read the exclusion report before the headline. Truncated, bodies-missing, sensitive and unreplayable records are each detected and counted; a truncated record never contributes silently. A corpus that is 80% truncated yields a real number over a small slice, and the report says so. Two caveats worth internalizing: has_sensitive_data is derived from detection metadata added asynchronously after persistence, so exclude-by-flag is a best-effort reducer and never a guarantee; and the activity log stores the FULL pre-truncation retrieve_tools response while the agent consumed truncated text, so a truncated record tokenized as-is overstates cost.

5. Verify determinism (SC-002) — two runs are byte-identical once the wall-clock stamp is removed:

CORPUS=specs/083-discovery-profiler/datasets/corpus_v2.tools.json
go run ./bench/cmd/bench -replay ~/replay-corpus.jsonl -corpus-v2 $CORPUS -out /tmp/run-a
go run ./bench/cmd/bench -replay ~/replay-corpus.jsonl -corpus-v2 $CORPUS -out /tmp/run-b
diff <(jq 'del(.generated_at)' /tmp/run-a/report.json) \
     <(jq 'del(.generated_at)' /tmp/run-b/report.json) \
  && echo "byte-identical modulo generated_at (SC-002)"

6. Delete the input when you are finished.

rm -f ~/replay-corpus.jsonl

A replay input is raw user traffic. Keep it outside the repository, never commit it, and delete it when the run is done — it is gitignored nowhere precisely because it should never be inside the repo in the first place.

Agent loop — tokens per completed task (costs model spend)

The replay half is a counterfactual over recorded traffic: what the same call shape would have cost per cell. It is not observed agent behaviour, and no replay figure may be published without that label. The agent loop is the half that observes behaviour, and it is the only part that costs spend.

  • One instance, whole matrix. All three routing-mode servers are mounted at startup, so a cell is selected by endpoint URL; only the two serialization axes need config, and both hot-reload. Enable code execution if code_exec is in scope — without it that cell is degenerate and is skipped with a reason.
  • Stand up the full fleet, even when running one service's tasks. A single server's toolset is too small a fleet for proxy modes to differ from baseline, and the full fleet is also the honest baseline arm: same agent, same tasks, all tools loaded directly.
  • k ≥ 4 runs per cell. A single agentic run is noise; every model-dependent figure is an average over at least four runs with its spread, or it is not a headline.
  • Four figures per cell: tokens per completed task, completion rate, first-attempt success and retry rate. The retry classification rule (corrective vs infrastructure) must be applied identically to the baseline arm and the proxy arms, or the comparison is biased toward whichever arm carries richer error signal.
  • Accounting sources are never summed. Deterministic figures come from the local tiktoken estimator; live figures come from provider-reported usage. A cross-source aggregate is withheld with a stated reason instead — and a mode that costs less while completing less is marked a regression, whatever its token figure says.
Offline prerequisite (both halves)

tiktoken downloads its vocabulary on first use unless the cache directory is both named and populated:

export TIKTOKEN_CACHE_DIR="$HOME/.cache/tiktoken"
go run ./bench/cmd/bench -corpus-v2 specs/083-discovery-profiler/datasets/corpus_v2.tools.json \
  -out /tmp/warm   # warms the cache; needs network exactly once

Setting the variable only names a cache; it does not fill one. A genuinely offline reproduction needs the cache pre-populated — warm it once with network access, or restore a known-good copy (which is what .github/workflows/bench.yml does via its actions/cache step).

Reproducing a published figure (SC-004)

Someone with no prior context reproduces the deterministic figures exactly, and the model-dependent ones within the stated numeric tolerance (FR-022 — see the tokenizer caveat under "Known limitations"), by following these steps and nothing else.

1. Populate the tokenizer cache. Naming it is not filling it.

export TIKTOKEN_CACHE_DIR="$HOME/.cache/tiktoken"
go run ./bench/cmd/bench \
  -corpus-v2 specs/083-discovery-profiler/datasets/corpus_v2.tools.json \
  -out /tmp/warm             # downloads the vocabulary once — needs network
ls "$TIKTOKEN_CACHE_DIR"     # non-empty ⇒ every later run is genuinely offline

TIKTOKEN_CACHE_DIR only names a directory. A first run with the variable set and the directory empty still reaches the network, so a machine "configured for offline" but never warmed fails at the first token count rather than at configuration time — and it fails the same way on an air-gapped reviewer's laptop as it does in a network-restricted CI job. Warm it once with network access, or restore a known-good copy (which is what .github/workflows/bench.yml does via its actions/cache step).

2. Supply a fleet input. There is no default, and there is no fallback.

A menu cost is a property of the tool definitions the agent was shown. A recording carries the call shape — sequence, tool mix, call counts — and no fleet snapshot whatsoever, so a recording-only invocation has nothing to price:

go run ./bench/cmd/bench -replay ~/replay-corpus.jsonl -out bench/results
# bench: replay requires a fleet input as well as a recording: a menu is a
# property of the tool definitions and the activity export carries no fleet
# snapshot, so a recording on its own computes nothing — pass a frozen corpus
# or a live-proxy catalog (this is an error, not a degraded run)
# exit status 1

That is a hard error, not a degraded run. The alternative — quietly scoring against an empty or assumed fleet — would produce a plausible-looking number with no fleet shape behind it, which is exactly the failure every percentage in this report carries a fleet_shape to prevent. Pass the frozen corpus for a reproducible figure, or a live -proxy for today's fleet.

3. Run the pinned command, twice. The deterministic half is byte-identical across runs once the wall-clock stamp is removed — the diff recipe under "Verify determinism" above is the check, and it is what SC-002 means. If two runs on the same machine differ by anything other than generated_at, stop: nothing downstream is worth comparing yet.

4. Read the three honesty marks before quoting anything. They are fields in report.json, not prose in this file, because a report travels without its README:

Field Says If it is missing
run_status.completeness complete, or partial with a reason and the unmeasured cells named Undeclared completeness — the run is not publishable as a complete comparison (FR-032)
<block>.inputs.independently_reproducible whether an outsider can obtain the inputs at all Undeclared availability — treated as not reproducible, and blocks publication (FR-030)
<block>.records where this block's raw per-run records sit, as a path relative to the run directory No records key at all blocks publication; retention: not_retained does not — the figure is real, just no longer traceable to its inputs (FR-029)

bench.ReportV2.PublicationCheck() applies all three and returns blockers plus caveats. A partial run blocks. An undeclared input blocks. A report in which every block rests on a private recording blocks — see step 5.

5. Know what you cannot reproduce. A replay run over someone's own exported activity is scored from private recorded sessions: the recording is raw user traffic, it is never published (FR-006), and no procedure exists that would hand it to you. Such a block is marked independently_reproducible: false with the limitation stated in the document, and it may never be the sole support for a published claim — a report whose only figures come from one operator's recording is a claim nobody outside can check, however honestly each row is labelled. Publish it beside a block an outsider can reproduce (the committed corpora, or a pinned public task suite), and keep the caveat attached.

6. Raw records are run-local and not durable. They are written under the gitignored bench/results/ tree — the same tree SC-011 forbids committing and the same tree a cleanup empties. The report references them by a path relative to its own directory, never an absolute one: an absolute path would carry the operator's filesystem into a published document and would dangle on every other machine. Copy the run directory elsewhere before relying on those records; once they are gone the reference degrades to records not retained rather than pointing at nothing.

Each mark is written by the run that produces the block: the replay entry point sets replay.inputs (private-recording for an operator's own export) and replay.records; the agent loop sets its own pair plus the report-level run_status, derived from the cells it planned against the cells it actually measured. WriteJSON re-checks every records reference against the directory it is writing into, so a reference cannot be published pointing at a file that is already gone.

Before publishing a replay or agent-loop number
  1. Recompute the achievable ceiling for each fleet shape; never carry a previous fleet's ceiling forward.
  2. Quote the fleet shape beside every percentage.
  3. Confirm the run is not partial, and that no private-recording figure is standing alone — PublicationCheck() answers both, and returns the caveats that must travel with the number even when it publishes.
  4. Confirm no report is tracked:
    test -z "$(git ls-files bench/results)" && echo "clean (SC-011)"
    
    Use git ls-files, not git status --porcelain — porcelain does not list ignored files and would stay silent about an already-tracked report. .github/workflows/bench-results-untracked.yml runs this same assertion on every pull request.

What is scoped but not yet built (follow-ups)

These require decisions and/or other roles, so they are tracked as child issues rather than landed here:

  • End-to-end task success with a pinned LLM — requires a pinned model + an LLM-call budget; this is the only part that costs spend. Until then, the session-cost estimator above is the honest substitute — and it is labeled estimated, never measured.
  • CI publish-on-release-tag → public static dashboard — Release/DevOps lane.

Dataset sources & provenance

Full details and regeneration procedures: specs/083-discovery-profiler/datasets/README.md (immutability rule: a refresh is *_v3.*, never an edit of a committed *_v2.* file).

  • Spec 065 datasets (specs/065-evaluation-foundation/datasets/): the 45-tool corpus_v1 snapshot (name+description) and retrieval_golden_v1.json, generated from 7 permissively reachable no-auth reference servers (filesystem, git, memory, sqlite, fetch, time, sequential-thinking).
  • corpus_v2.tools.json (committed): the same 45 tools exported with full JSON input schemas — the universe for all encoding-arm comparisons. Captured 2026-07-14 (corpus_v2@2026-07-14), canonicalized (sorted tool IDs, sorted object keys). Provenance note: the export reads the Bleve index (scripts/gen-corpus-v2.sh), not GET /api/v1/tools — that REST endpoint currently serves stub schemas (supervisor StateView never parses ParamsJSON); the index is the authoritative record of what the retrieval funnel ingests, which is exactly what the arm comparison must measure.
  • result_fixtures_v1.json (committed): 6 deterministic tool-call outputs captured once through the proxy from 5 reference servers for the toon_results arm — 2 tabular, 4 non-tabular; wall-clock values and capture-host specifics stripped. License-clean: outputs of our own reference servers over our own seed data (fetch_fixture.html is the committed fetch input).
  • LiveMCPTool snapshot (specs/083-discovery-profiler/datasets/livemcptool_snapshot/, committed): frozen, normalized copy of the LiveMCPBench tool corpus — 70 servers / 527 tools, pinned Hugging Face revision ddea2d24 (ICIP/LiveMCPBench, arXiv:2508.01780). Apache-2.0, redistributed with the mandatory attribution in the snapshot's ATTRIBUTION.md. Used for token/scale measurement only: its task annotations name tools as unqualified free text (5/150 names resolve to no corpus tool, 13/150 to multiple servers), so relevance labels are not derivable — the loader records that absence explicitly rather than guessing (FR-011).
  • ToolRet (never committed): the upstream mangopy/ToolRet-Tools + ToolRet-Queries datasets carry no stated license, so redistribution is not clearly permitted — the harness fetches at runtime only (scripts/fetch-toolret.sh, pinned revision, uv run parquet→JSON) into the gitignored bench/results/cache/ and never writes ToolRet bytes into git (FR-013). Retrieval scoring runs on a seeded query subset (-subset/-seed): same revision + seed + size ⇒ byte-identical subset on every machine. Attribution: ToolRet tool-retrieval benchmark (Hugging Face; ACL 2025).
  • Proxy + management tool definitions: derived at run time from the live server tool builders (internal/server/mcp_routing.go → buildCallToolModeTools / buildCodeExecModeTools, exposed via internal/server.ProxyModeToolDefs). No hand-maintained fixture — the benchmark cannot drift from the tools the proxy actually serves.
  • TSCG shim (bench/tscg/): committed shim.mjs + package-lock.json pinning @tscg/core@1.4.3; npm ci --prefix bench/tscg (Node ≥20) reproduces the exact runtime everywhere, including CI.

Known limitations (read before quoting a number)

  • Tokenizer caveat — absolute numbers are estimates. All token counts use tiktoken cl100k_base as a reproducible, model-agnostic, offline estimator. It can underestimate other tokenizers (e.g. Claude's) by up to ~60%, so never quote an absolute token count as a Claude cost — but relative savings between arms and modes are stable across vocabularies, and those are the headline numbers. The v2 dashboard renders this caveat next to every absolute number (SC-005).
  • The session estimator is an ESTIMATE, not a measurement. The multi-turn session cost is a model — proxy_menu + calls × mean_response × (1 + retry_rate) — whose retry rates are literature-derived defaults, not observations of a live agent loop. Every estimator row is provenance-labeled estimated and echoes its inputs; treat it as a decision aid, not a benchmark result. A measured agent loop is the pinned-LLM follow-up above.
  • Lower-bound arms. Arms that rewrite or elide description text (TSCG) report savings as a lower bound: what they drop cannot be priced back in.
  • v1 offline run: schemas excluded — direction is not clean. In the legacy name+description-only run, input schemas are dropped from both sides, so that number is its own well-defined metric, not unambiguously conservative. The live run adds full schemas for the exact headline number; the Spec 083 arms measure schema-bearing renderings on corpus_v2. Quote those, not the v1 offline estimate.
  • Savings scale with tool count. The 45-tool reference corpus is small; the 527-tool LiveMCPTool snapshot and the ToolRet subset exist precisely to test scale. Quote the corpus size alongside any percentage.
  • LiveMCPTool has no relevance labels (see provenance above): it validates token/scale claims only, never retrieval quality.
  • LAP divergence is expected. LAP frames the tool menu its own way; within ±15% of the in-house count is normal, beyond it is a non-blocking warning — never a hidden discrepancy, never a build failure.

Reproducible live run

docker-compose.yml boots mcpproxy over the frozen reference-server config so the corpus and live tool list are reproducible across machines. The live accuracy/latency/full-schema/response-cost scorers attach to it via -live (see "Live run" above). Pin the upstream-server images before publishing headline numbers (image drift can change the tool corpus).

Reviewer contact

Methodology questions / disputes: open an issue in smart-mcp-proxy/mcpproxy-go and tag the maintainers, or comment on the roadmap benchmark ticket (MCP-42).

Warm it with the one-line command rather than by hand — the same step CI runs before its parallel test sweep:

export TIKTOKEN_CACHE_DIR="$HOME/.cache/tiktoken"
go run ./bench/cmd/warmtiktoken

Why a dedicated step: go test ./... runs packages in parallel, and several test binaries race to download the same vocabulary into one cache dir. On Windows the loser fails its whole package on the atomic rename — it surfaced as bench/arms flaking with no code change near it. Naming the cache does not fix that; populating it first does.

Documentation

Overview

agentloop.go — Spec 103 US2: the LIVE agent-loop driver. This is where the feature's actual claim — tokens per *completed* task — becomes measurable.

Why this file exists at all

Every other measurement in this harness prices a MENU: how many tokens the tool definitions cost an agent before it does any work. That number is real and it is reproducible, but it is not the number a user cares about. A mode that halves the menu and doubles the number of failed attempts costs more, not less. Only a live loop can tell the difference, because the difference is made of decisions a model takes — which call it attempts, whether the first attempt was right, whether it finished. A recording cannot answer any of that (see the Replay Boundary in the spec), which is exactly why US1's replay and this file are separate stories with separate accounting.

Why it is parameterised over function types

A live run costs real money and needs a pinned model, a patched suite and credentials. If the arithmetic that decides what gets PUBLISHED could only be exercised by such a run, it would never be exercised: the tests would be skipped in CI, the classification rules would drift, and the first time anybody checked the numbers would be the day they were published.

So the driver takes RunTaskFunc and UsageFunc (the bench/flipgate.go precedent, where RetrieveToolsFunc makes the flip-gate maths testable with no proxy running). Every rule below — the binding Definitions, the FR-021 headline bar, the FR-019 regression flag, the never-sum-across-sources refusal — is decided in pure functions over records, and the production caller supplies real records instead of fakes. Nothing about the arithmetic changes between the two.

The classification is deliberately source-blind

ClassifyIntents sees an attempt sequence and nothing else — not the cell, not the endpoint, not whether mcpproxy was in the path. If the baseline arm and the proxy arms were classified by different rules, the comparison would be biased toward whichever surface reports failure more legibly, and that bias would be invisible in the published figure. One rule, both arms.

What this file will NOT do

It will not sum a provider-reported figure with a tokenizer-counted one. It will not publish a percentage without a fleet shape. It will not treat a missing completion verdict as a failure, a missing cache-read count as a zero, or a single run as a headline. Each of those would understate cost or overstate success in this project's favour, which is the failure class the whole feature exists to prevent.

armindex.go — temp-dir retrieval index for arm-aware quality scoring (Spec 083 US2, FR-008, research D7).

For index-altering arms the profiler must answer "does this encoding move recall?", which is only meaningful if the index under test IS the production index: same field mappings, same analyzers, same six-branch SearchTools query shape. So the builder funnels through the production internal/index.Manager (BatchIndexTools → BleveIndex.BatchIndex) and the SearchFunc calls the production search path (Manager.SearchTools → BleveIndex.SearchTools) — nothing here re-implements the mapping or the query. The SC-003 parity test (bench/armindex_test.go) proves the baseline arm through this funnel reproduces the recorded golden-set baseline (recall@5 = 0.68 ± 0.05) before any other arm's score is trusted.

armrun.go — the encoding-arm runner (Spec 083 US2, T024): orchestrates arms over one corpus and assembles the ArmResult report rows (FR-005/006/007/ 008/009/020).

Per arm it measures: total tokens over the amortized whole-corpus listing (EncodeListing, so shared preambles/dictionaries are paid once — contract rule 6), per-tool mean and p95 from EncodeTool (comparable across arms), savings % against the baseline_json arm's listing total (the single denominator, research D7b), per-tool skip counting with capped examples (FR-009), the heaviest-tools top-N (FR-020), and — for index-altering arms when a golden set is supplied — retrieval quality through the production index funnel (armindex.go, FR-008).

Determinism (FR-010): results follow the caller's arm order, per-tool iteration is corpus order, skip examples are the first failures in corpus order, and heaviest-tools ties break on tool_id — no map iteration anywhere.

breakeven.go — break-even analysis for proxy-mediated tool discovery (Spec 083 US1, FR-003/FR-004, research D8).

LAP independent verdict (Spec 083 US4, FR-015/016, SC-006).

CI runs the version-pinned external linter against the booted proxy:

uvx --from 'lap-score[mcp]==0.8.0' lap lint --mcp-url "$PROXY/mcp" --json > lap.json

and this file turns that artifact into the report's LapVerdict: parse the JSON, extract LAP's menu-token count (bucket A) and letter grade, and compare against the in-house count for the same surface. LAP being absent or broken must never fail the benchmark (FR-015): every failure path yields Executed=false with a skip reason instead of an error.

mcpcall.go — real MCP retrieve_tools invocation (Spec 083 US1, FR-001, research D1).

The REST /api/v1/index/search endpoint used by live.go is NOT the payload an agent pays for: it misses usage_instructions, call_with, annotations, and session_risk. This file speaks the actual MCP protocol (streamable-http, mark3labs/mcp-go client) against the proxy's /mcp endpoint and returns the raw text content of each retrieve_tools call — the exact bytes an agent's context ingests — plus the client-measured latency (FR-023; the server-side timing fields are stubs, same rationale as LiveClient.Search).

mcptools.go — the live MCP tools/list primitive (Spec 103).

What this is for, and what it is NOT for

The obvious justification for reading tools/list is "REST serves stub schemas". On this branch that is FALSE and must not be repeated: GET /api/v1/tools returns real upstream input schemas (MCP-3167 is fixed in internal/contracts/converters.go, which now reads "inputSchema" first and falls back to the legacy "schema"). Measured against the 7-server reference fleet, 42 of 45 REST schemas are real and the 3 empties are genuinely parameter-less tools serving identical bytes on both surfaces.

The real justification is FIDELITY. tools/list is the only surface that shows what an agent actually receives:

  • it is the MENU, not a catalog. REST /api/v1/tools is a different population (it skips quarantined servers but includes disabled servers and non-callable tools) and a different rendering (no "server " prefix, no deferred signature suffix).
  • it reflects the deployment. bench's in-process derivation (internal/server.ProxyModeToolDefs) pins EnableCodeExecution:true unconditionally, so it always counts code_execution's 2420-char description even for a deployment with code execution off — the shipped default — overstating the proxy menu by 1049 tokens.
  • it is verifiably the same arithmetic. A captured tools/list run through bench's own tokenizer reproduces the in-process built-in menu EXACTLY (5783 tokens for the retrieve_tools surface, 3781 for code_execution).

The stub guard is the point of this file

A fleet of stub schemas prices every upstream definition at a fraction of its real size. That shrinks the naive baseline, which is the denominator of every published savings percentage, so the error lands as INFLATED SAVINGS — the one direction of error this benchmark exists to prevent. Two systematic sources are known:

  1. The Spec 102 deferred direct surface. With `direct_tool_response_mode: "deferred"`, /mcp/all serves the placeholder `{"type":"object"}` for EVERY upstream tool (internal/server deferredDirectInputSchema / renderDeferredDirectTool) and folds the real schema into the description as a compact signature. It is served with no error and no flag, and REST keeps serving real schemas under the very same config — so this is the live hazard, and it is silent.

    Measured end to end on the 7-server / 45-tool reference fleet, with this guard removed: direct_full's menu collapses from 4368 to 2502 tokens (-42.7%), and the published headline — the direct_full <-> direct_deferred delta — FLIPS SIGN, from +1300 tokens (29.8% saved) to -806 (-32.2%, i.e. deferral reported as COSTING more). That is the failure this file exists to make impossible: not a slightly-off number, but a report that asserts the opposite of the truth while looking entirely well-formed.

  2. The supervisor StateView stub `{"type":"object","properties":{}}` (MCP-3132/MCP-3167). Fixed today; the guard is what keeps a regression from silently re-inflating the headline.

The guard is stated per POPULATION rather than per tool, because a per-tool rule would be useless: real fleets contain genuinely parameter-less tools whose honest schema is `{"properties":{},"required":[],"type":"object"}`, and the deferred surface leaves describe_tool's schema real, so "does anything here have a schema" waves the deferred case straight through. The rule is instead: within each population PRESENT in the listing — upstream tools and mcpproxy's own built-ins — at least one tool must carry a schema with real properties. Both known failure modes stub an entire population; no healthy fleet stubs one.

Why the RAW wire bytes

The schema must come off the wire as json.RawMessage, never through mcp-go's typed decode. ToolArgumentsSchema keeps only {$defs, type, properties, required, additionalProperties} and drops every other top-level keyword, and its marshaller ALWAYS emits "properties" and "required" — so a round trip rewrites the deferred placeholder `{"type":"object"}` as `{"properties":{},"required":[],"type":"object"}` and erases the exact distinction the guard depends on. The transport-level path is therefore not an optimization; it is what makes the guard possible.

The "[server] " prefix is stripped

The direct surface renders every description as `fmt.Sprintf("[%s] %s", ...)` (internal/server/mcp_routing.go). Isolated on the reference fleet, that prefix costs 136 tokens across 45 tools — 3.0% (4595 with, 4459 without). It is stripped here, and the reason is not that 3% is small: the encoding arms SYNTHESIZE each cell's rendering from a canonical definition, and bench/arms.DirectDeferredArm prepends that same "server " itself. Keeping the prefix would double-charge it in the direct cells and make a live fleet non-interchangeable with every frozen corpus the published figures use.

Known limitation: server attribution is a first-"__" split

Attribution uses the production parser (server.ParseDirectToolName), which splits on the FIRST "__" and is knowingly ambiguous: a server whose name contains "__" mis-splits. The proxy resolves this with server-side state (mcp_direct_catalog.go keeps byDisplayName AND byCanonical maps) that is not recoverable over the wire, and it WITHHOLDS both tools on a display-name collision rather than picking a winner — so a live listing can also under- count relative to GET /api/v1/tools. Reconciling a fleet's count against the REST catalog is the fix for that, and is deliberately left to the caller rather than smuggled in here.

modematrix.go — the Spec 103 mode matrix (specs/103-token-bench/contracts/mode-matrix.md, FR-015/016/017).

The matrix is what an agent actually sees, crossed over the three configuration axes that determine it: routing mode, discovery-surface serialization, and direct-surface serialization. The naive product is 3 x 2 x 2 = 12 combinations, but only 5 of them are distinct BEHAVIOURS — on each surface one or both serialization axes have no consumer, so the other 7 configurations produce a run identical to one of the 5.

Those 7 are the reason this file exists as data rather than as a loop. They are configurable and behaviourally redundant, NOT impossible: calling them impossible would be wrong, and rendering them as zeros would be worse, because a zero reads as "measured, and it cost nothing". Each is emitted as a skip row naming the cell it collapses onto and the reason code for the collapse (FR-017).

This file composes; it does not reimplement. Cell tool catalogs come from ProxyToolsForMode (proxytools.go, itself derived from the production tool builders), the arm that renders a cell's menu is named for resolution through the existing bench/arms registry — bench cannot import bench/arms, since arms imports bench — and skip rows go through the existing SkippedArmResult constructor rather than a second skip shape.

offline_report.go — assembly of the versioned v2 report for offline profiler runs (Spec 083, research D12): encoding-arm rows over one or more frozen corpora, corpus descriptors with license/attribution provenance, and provenance labels for every headline number (SC-005). The same assembly path serves the bench CLI and the end-to-end report-validation test (bench/reportv2_e2e_test.go), so what CI validates is what the CLI ships.

replay.go — Spec 103 US1: deterministic cost recomputation over a REAL recorded workload (specs/103-token-bench/contracts/replay-input.md).

What replay is. An operator exports their own activity log (`mcpproxy activity export --format json`) and supplies a fleet. Replay then prices, per mode cell, the tool menu that fleet would put in the agent's context, and reports the recorded call shape it was priced against. It costs nothing to run, spends no model tokens, and replaces frozen-corpus numbers with numbers from a fleet somebody actually uses.

A fleet input is MANDATORY

A menu is a property of the TOOL DEFINITIONS the agent was shown, and the activity export carries no fleet snapshot — not even implicitly, because under the direct routing mode the menu is the whole upstream fleet rather than a fixed built-in list. So a recording on its own can compute nothing at all. That is why `-replay <jsonl>` without a fleet is ErrReplayFleetRequired and not a degraded run: there is no reduced figure to degrade TO, and a run that quietly produced menu-less output would be a report with no menu cost in it, which is the only thing US1 exists to produce.

What bodies-off can and cannot yield

The default posture reads no request or response bodies (privacy: the activity EXPORT path does not mask, so an export is raw user traffic). That posture fixes what is knowable, per the contract's table:

  • MENU COST is measured for every cell, from the fleet input.
  • ABSOLUTE COMPLETE-WORKLOAD COST is available for NO cell, because a complete workload includes every consumed response and that text is absent. It is reported as withheld WITH a reason — never as a zero, which would read as "measured, and it cost nothing".
  • The CROSS-MODE DELTA between direct_full and direct_deferred IS measurable, because those two cells serve identical call responses and the response term cancels out of the difference. That delta is the honest bodies-off headline. The retrieve cells get no such delta: their serialization axis changes the RESPONSE body, which is exactly the text bodies-off does not have.

Byte lengths do NOT rescue this. `request_bytes` / `response_bytes` are pre-truncation BYTE measurements, and the loader deliberately reports them as Cost{Basis: CostEstimated, Bytes: n} with NO token count: converting bytes to tokens would require a fudge factor nobody sanctioned. So a response figure is emitted only when every contributing response was genuinely tokenized (bodies-on), and otherwise not at all.

Every figure is a counterfactual

FR-004 is the spec's most load-bearing constraint and the easiest to lose in a headline: a replay recomputes a RECORDED call shape against the SUPPLIED fleet. No agent ran under these modes; no model was asked. And because the fleet comes from the fleet input rather than the recording, the figures score recorded work against TODAY's fleet, not the fleet as it stood when the session was recorded — internally valid across modes, not a historical reconstruction. ValidateCounterfactual refuses to let a block carrying figures leave without saying so.

Determinism

Two runs over the same inputs must be byte-identical (SC-002), so replay reports pin GeneratedAt to ReplayGeneratedAt rather than stamping the wall clock. Nothing else here reads a clock, sorts a map, or depends on iteration order; the loader already orders sessions deterministically.

respcost.go — span-based component attribution for retrieve_tools responses (Spec 083 US1, FR-001/FR-002, research D7b).

The FR-002 invariant — component token counts sum EXACTLY to the response total — is mathematically unsatisfiable if each field is tokenized on its own, because BPE is not additive across concatenation boundaries. Instead the response text is partitioned into contiguous labeled byte spans, the WHOLE text is tokenized once, and each token is attributed to the span that contains its starting byte. The sum then equals the total by construction.

Package bench is the mcpproxy benchmark harness (roadmap #19 / MCP-42).

It produces the reproducible numbers behind mcpproxy's marketing claims — token reduction, discovery accuracy, and latency — by comparing three ways an agent can be wired to upstream MCP tools:

  • baseline: every upstream tool definition is loaded directly into the agent's context (no proxy discovery).
  • retrieve_tools: only mcpproxy's discovery + call_tool variants occupy the context; tools are found on demand via BM25 search.
  • code_execution: only code_execution + retrieve_tools occupy the context; the agent orchestrates many tools from sandboxed JS in one round-trip.

The token-reduction measurement in this file is fully deterministic and offline: it counts the context cost of each mode over a frozen tool corpus using the tiktoken cl100k_base encoding (a reproducible, model-agnostic estimator). It reuses the Spec 065 frozen corpus (specs/065-evaluation-foundation/datasets/corpus_v1.tools.json) as its tool universe so the benchmark scores a versioned, non-drifting snapshot (CN-002).

Methodology, limitations, and the live (docker-compose) run that adds full JSON input schemas and end-to-end accuracy/latency are documented in bench/README.md.

Index

Constants

View Source
const (
	CompletionPass     = "pass"
	CompletionFail     = "fail"
	CompletionNoSignal = "no-signal"
)

Completion verdicts. TASK COMPLETION is the suite's own pass verdict and is NEVER inferred. Where a suite emits no verdict the unit is reported as CompletionNoSignal and excluded from completion-dependent figures — not counted as a success (which would flatter us) and not counted as a failure (which would flatter a competing mode). Silence is not evidence either way.

View Source
const (
	AttemptOutcomeOK             = "ok"
	AttemptOutcomeError          = "error"
	AttemptOutcomeInfrastructure = "infrastructure"
	AttemptOutcomeUnknown        = "unknown"
)

Attempt outcomes.

The distinction that carries weight is AttemptOutcomeInfrastructure versus AttemptOutcomeError. A transport failure measures the NETWORK, not the mode: counting it against a cell would mean a flaky hop during one arm's run shows up as that mode confusing the agent. So transport failures are counted on their own axis and leave the intent indeterminate rather than scoring it.

AttemptOutcomeUnknown exists because a trajectory can end mid-call. An unknown outcome is never silently read as success; it makes its intent indeterminate, and the record that carries it is partial.

View Source
const (
	// CellBaseline is the FR-020 denominator, not an mcpproxy mode: the same
	// agent doing the same tasks with every upstream tool loaded inline.
	CellBaseline = "baseline"
	// CellRetrieveFull is the default: retrieve_tools surface, schema-bearing
	// discovery entries.
	CellRetrieveFull = "retrieve_full"
	// CellRetrieveCompact is the Spec 085 compact discovery serialization.
	CellRetrieveCompact = "retrieve_compact"
	// CellDirectFull is the direct enumeration surface with full schemas.
	CellDirectFull = "direct_full"
	// CellDirectDeferred is the Spec 102 deferred-schema direct surface.
	CellDirectDeferred = "direct_deferred"
	// CellCodeExec is the code-execution surface, which forces full discovery.
	CellCodeExec = "code_exec"
)

Cell identifiers. These are a stability contract (FR-028): a report key that changed between releases would silently break comparability with every earlier run, so these strings are frozen even if the internal naming moves.

View Source
const (
	SerializationFull          = config.ToolResponseModeFull
	SerializationCompact       = config.ToolResponseModeCompact
	SerializationDeferred      = config.DirectToolResponseModeDeferred
	SerializationNotApplicable = "not_applicable"
)

Serialization-axis values. SerializationNotApplicable is deliberately its own value rather than an empty string or a default: contract rule 3 requires an ignored axis to be recorded as not-applicable, because recording it as "full" would imply a measurement of a serialization that was never in play.

View Source
const (
	// SkipReasonAxisIgnored: the configured axis governs a different surface,
	// so this combination runs identically to the cell it collapses onto.
	SkipReasonAxisIgnored = "axis-ignored"
	// SkipReasonForcedFull: the code-execution surface overwrites the response
	// mode with full and blanks the detail parameter. An override, not an
	// ignored axis, hence its own code.
	SkipReasonForcedFull = "forced-full"
	// SkipReasonDegenerate: code_execution with enable_code_execution:false.
	// Not part of the product at all — nothing is callable, so there is no
	// behaviour to collapse onto.
	SkipReasonDegenerate = "degenerate"
)

Skip reason codes (contract "The other 7 combinations" + "a degenerate configuration"). They are distinct because they describe different things: an axis with no consumer on that surface, a surface that actively overrides the axis, and a configuration in which the surface can discover tools and call none of them.

View Source
const (
	// CapabilityBatching is describe_tool's batch form (up to 5 ids in one
	// call, Spec 085). Registered on the retrieve_tools surface and, since
	// Spec 102, on the direct surface; deliberately absent from the
	// code-execution surface, where a compact response would reference a
	// second stage that does not exist there.
	CapabilityBatching = "batching"
	// CapabilityStoredScripts is Spec 097 stored-script execution, a
	// code_execution feature with no analogue on the other surfaces.
	CapabilityStoredScripts = "stored_scripts"
	// CapabilityValidateBeforeDispatch is Spec 085 FR-013 pre-dispatch
	// argument validation with its self-healing invalid_params error. It runs
	// on the call_tool_* variants and, since Spec 102 US3, on direct dispatch;
	// the code-execution surface dispatches from the sandbox and does not.
	CapabilityValidateBeforeDispatch = "validate_before_dispatch"
)

Capability identifiers (FR-016). These are binary conditions applied to the cells where each is available — measure a cell with the condition on and with it off — NOT a fourth axis of the product.

View Source
const (
	EndpointRetrieveTools = "/mcp/call"
	EndpointDirect        = "/mcp/all"
	EndpointCodeExec      = "/mcp/code"
)

Endpoint paths for the routing-mode axis.

These are mounted permanently at startup and stay mounted regardless of config (internal/server/server.go, the "Routing mode dedicated endpoints" block): each endpoint always serves its own routing mode. That is what makes the routing axis free of config changes and restarts — and it is why two cells that differ only in serialization legitimately share one URL.

View Source
const (
	PayloadComponentName        = "name"
	PayloadComponentDescription = "description"
	PayloadComponentSchema      = "schema"
	// PayloadComponentStructural is the rendering's own overhead — the
	// separators between components. Small, but it belongs to some component
	// or the shares do not partition the payload, and folding it into a
	// content component would overstate that component.
	PayloadComponentStructural = "structural"
)

PayloadComponent labels the parts of a rendered tool definition.

View Source
const (
	ProvenanceMeasured  = "measured"
	ProvenanceComputed  = "computed"
	ProvenanceEstimated = "estimated"
)

Provenance labels for headline metrics (SC-005): a number is either measured (observed over the real protocol), computed (derived arithmetic over measured inputs), or estimated (model with documented assumptions).

View Source
const (
	// AccountingKindTokenizer marks figures counted by the pinned
	// deterministic estimator named in the document-level TokenizerInfo.
	AccountingKindTokenizer = "deterministic-tokenizer"
	// AccountingKindProvider marks figures taken from provider-reported
	// usage, which requires a pinned model to be interpretable at all.
	AccountingKindProvider = "provider-usage"
)

Accounting-source kinds. Figures from different kinds are NEVER summed: a cross-source aggregate is withheld with a stated reason (data-model.md invariant 2), reusing the harness's withhold-rather-than-compute pattern.

View Source
const (
	ReplayExclusionTruncated    = "truncated"
	ReplayExclusionBodiesMissed = "bodies_missing"
	ReplayExclusionSensitive    = "sensitive"
	ReplayExclusionUnreplayable = "unreplayable"
	ReplayExclusionUnattributed = "unattributed"
)

Exclusion reason codes for replay sessions (FR-003). Every supplied session that does not reach a figure is counted under exactly one of these — silence is never success (data-model.md invariant 3).

View Source
const (
	Spec102Confirmed = "confirmed"
	Spec102Corrected = "corrected"
)

Spec-102 verdicts (FR-025). The verdict is stated explicitly per fleet shape; an absent verdict is not "confirmed by default".

View Source
const (
	// InputAvailabilityRepository marks inputs committed in this repository —
	// the frozen corpora and fixtures. Anyone with a clone has them.
	InputAvailabilityRepository = "repository"
	// InputAvailabilityPinnedProcedure marks inputs an outsider fetches by
	// following a documented, version-pinned procedure (a pinned dataset
	// revision, a pinned task suite). Reproducible, just not in-tree.
	InputAvailabilityPinnedProcedure = "pinned-procedure"
	// InputAvailabilityPrivateRecording marks inputs that are one operator's
	// own recorded traffic. Not publishable, therefore not obtainable,
	// therefore not independently reproducible.
	InputAvailabilityPrivateRecording = "private-recording"
)

Input availability (FR-030). Every input is either in the repository, obtainable by a documented pinned procedure, or a private recording that an outsider simply cannot get.

The third value is not a lesser version of the first two. Repository and pinned-procedure inputs differ only in where the bytes live; a private recording differs in kind, because FR-006 forbids publishing the recording and there is no procedure that would hand it over. That is a permanent limitation of any figure built on it, and the report says so rather than leaving a reader to assume the usual.

View Source
const (
	RunCompletenessComplete = "complete"
	RunCompletenessPartial  = "partial"
)

Run completeness (FR-032). A run is one harness invocation producing one report; it is complete only when every planned cell either produced a figure or was skipped for a stated reason.

View Source
const (
	// RecordsRetained means the artifact exists in this run's output
	// directory, at RunLocalPath, right now.
	RecordsRetained = "retained"
	// RecordsNotRetained means there is no artifact to point at. It is the
	// honest terminal state of a reference whose file is gone, and it is
	// never a dangling path.
	RecordsNotRetained = "not_retained"
)

Raw per-run record retention (FR-029).

View Source
const (
	// ComponentInputSchemas is each result's "inputSchema" pair — the raw JSON
	// input schema, including nested property descriptions (schema cost, not
	// tool-description cost).
	ComponentInputSchemas = "input_schemas"
	// ComponentDescriptions is each result's tool-level "description" pair.
	ComponentDescriptions = "descriptions"
	// ComponentUsageInstructions is the top-level "usage_instructions" pair.
	ComponentUsageInstructions = "usage_instructions"
	// ComponentMetadata is every other pair inside a result object: name,
	// score, server, call_with, annotations, usage stats.
	ComponentMetadata = "metadata"
	// ComponentOther is the response envelope: query/total/session_risk/notice
	// and the structural braces, brackets, and separators between spans.
	ComponentOther = "other"
)

Component labels of a retrieve_tools response (data-model DiscoveryResponseMeasurement.Components). Every token of the response text lands in exactly one of these buckets.

View Source
const (
	ModeBaseline      = "baseline"
	ModeRetrieveTools = "retrieve_tools"
	ModeCodeExecution = "code_execution"
)

Routing modes the benchmark compares. The mode names mirror the mcpproxy MCP servers in internal/server/mcp.go (codeExecServer, callToolServer).

View Source
const DefaultCompletionRegressionThresholdPct = 5.0

DefaultCompletionRegressionThresholdPct is FR-019's "stated threshold", in PERCENTAGE POINTS of completion rate below the baseline arm. It is stated as a named constant rather than buried in a comparison precisely because FR-019 requires the threshold to be stated: a reader must be able to see what bar a mode had to clear before its cheapness was called a saving.

View Source
const DefaultEncoding = "cl100k_base"

DefaultEncoding is the tiktoken encoding used for token estimation. cl100k_base is a widely-used, reproducible BPE; exact counts for a specific pinned model (e.g. Claude) will differ, but the *relative* savings between modes are stable.

View Source
const DefaultHeaviestN = 10

DefaultHeaviestN is the FR-020 default size of the heaviest-tools list.

View Source
const LapDivergenceTolerancePct = 15.0

LapDivergenceTolerancePct is the documented tolerance (FR-016) between LAP's menu-token count and the in-house count of the same surface. Both use cl100k_base, but LAP frames the menu its own way, so small divergence is expected; beyond ±15% the report shows a (non-blocking) warning.

View Source
const LapPinnedVersion = "0.8.0"

LapPinnedVersion is the lap-score release the CI step installs (research D9). The lint artifact itself carries no version field, so the pin is recorded here and stamped into the verdict on successful parse.

View Source
const MCPToolsListMethod = "tools/list"

MCPToolsListMethod is the JSON-RPC method this primitive calls.

View Source
const MinHeadlineRuns = 4

MinHeadlineRuns is FR-021's bar: a model-dependent figure is an average over at least four runs together with a measure of consistency. A single run of a stochastic loop is an anecdote, and the difference between two modes is routinely smaller than the run-to-run variance of one of them.

View Source
const RecordsNotDurableNote = "" /* 142-byte string literal not displayed */

RecordsNotDurableNote is the standing caveat on every retained reference. The records live under the gitignored bench/results/ tree — the same tree a results cleanup empties, and the same tree SC-011 forbids committing — so "retained" means retained in THIS run directory, on THIS machine, until someone clears it. Anything stronger would be a promise the harness cannot keep.

View Source
const ReplayCounterfactualLabel = "COUNTERFACTUAL — not observed agent behaviour. " +
	"These figures recompute a RECORDED call shape (call sequence, tool mix and call counts, taken from an activity export) " +
	"against the SUPPLIED FLEET under each mode cell. No agent ran under these modes and no model was asked, " +
	"so nothing here measures how an agent behaves. The fleet comes from the fleet input, not from the recording, " +
	"so a replay scores recorded work against TODAY's fleet rather than the fleet as it stood when the session was recorded: " +
	"internally valid across modes, not a historical reconstruction."

ReplayCounterfactualLabel is the FR-004 label carried by every replay block.

It states three things a reader needs at once, because dropping any one of them turns the figures into a claim the data cannot support: that this is a counterfactual, that no agent behaviour was observed, and that recorded work is scored against the supplied fleet rather than the fleet of the day.

View Source
const ReplayGeneratedAt = "1970-01-01T00:00:00Z"

ReplayGeneratedAt is the pinned generated_at stamp of every replay report (T029). SC-002 asks for byte-identical output across runs, and a wall-clock stamp defeats that check on its own, silently — a diff that always differs in one line trains a reader to ignore diffs. The value is a real RFC 3339 timestamp so the report still validates against the schema's date-time format, and an obviously synthetic one so nobody mistakes it for a run time.

View Source
const ReportVersion2 = 2

ReportVersion2 is the report_version const of the v2 envelope.

View Source
const SessionEstimateProvenance = ProvenanceEstimated

SessionEstimateProvenance is the provenance label the report layer attaches to session-estimate rows (SC-005): always "estimated", never "measured" — the estimator is a model with documented assumptions, not an observation.

View Source
const TokenizerCaveatText = "" /* 283-byte string literal not displayed */

TokenizerCaveatText is the mandatory accuracy caveat rendered wherever absolute token numbers appear (research D11, SC-005).

Variables

View Source
var ErrAgentLoopBaselineMissing = errors.New("agent loop requires the FR-020 baseline arm")

ErrAgentLoopBaselineMissing is the FR-020 refusal: no baseline arm, no denominator, therefore no publishable percentage. It is an error rather than a degraded run because there is nothing to degrade TO — a savings figure measured against nothing is not a smaller figure, it is a fabricated one.

View Source
var ErrAgentLoopModelUnpinned = errors.New("provider-sourced agent-loop figures require a pinned model")

ErrAgentLoopModelUnpinned refuses provider-reported usage from an unpinned model. Usage numbers are only interpretable against the model that produced them: the same task under a different model has a different token shape, so an unpinned figure is not comparable to any later run and cannot honour FR-028's "a later run is comparable to an earlier one".

View Source
var ErrBaselineNotBypassing = errors.New("the baseline arm must bypass mcpproxy entirely (no endpoint)")

ErrBaselineNotBypassing guards the substance of FR-020 rather than its name. The baseline is the same agent doing the same tasks with EVERY upstream tool loaded directly, bypassing mcpproxy entirely. A cell called "baseline" that points at an mcpproxy endpoint is a sixth proxy cell wearing the denominator's name, and every percentage computed against it would be a proxy-versus-proxy comparison published as proxy-versus-nothing.

View Source
var ErrDriverCacheReadMissing = errors.New("driver usage hook returned no cache-read count")

ErrDriverCacheReadMissing fires when the driver's own usage hook returns no cache-read count. The hook exists for exactly one reason — the suite's per-task output has no cache-read field, so that axis can only come from the driver reading provider responses. A hook that returns nothing for it is a wiring bug, and treating it as "unavailable" would hide the bug behind the legitimate unavailable path.

View Source
var ErrReplayFleetAmbiguous = errors.New("replay takes exactly one fleet input")

ErrReplayFleetAmbiguous is the two-fleets refusal. Silently preferring one input over the other would make the report's fleet_id a lie: every figure is quoted for exactly one fleet shape, and the reader has no way to tell which.

View Source
var ErrReplayFleetRequired = errors.New("replay requires a fleet input as well as a recording")

ErrReplayFleetRequired is the T019 hard error. It is a sentinel rather than a formatted string so a caller (and a test) can assert the RULE, not the wording: a recording-only invocation is an error, not a degraded run.

View Source
var ErrStubToolSchemas = errors.New("mcp tools/list served stub input schemas")

ErrStubToolSchemas is the guard's sentinel: a tools/list listing whose schemas cannot be priced. It is a sentinel rather than a formatted string so a caller (and a test) can assert the RULE — a stub-schema listing is refused, never returned — independently of the wording.

Functions

func AttributeTokens added in v0.51.0

func AttributeTokens(tk *Tokenizer, text string, spans []Span) (int, map[string]int, error)

AttributeTokens tokenizes the WHOLE text once and attributes each token to the span containing its starting byte (research D7b). It returns the total token count and the per-label component counts; all five canonical components are always present in the map. Invariant, by construction: sum(components) == total.

Token byte offsets are recovered by decoding each token id individually — cl100k is a byte-level BPE, so per-token decoding is lossless and the concatenated pieces reproduce the input bytes exactly (verified at runtime).

func AveragePrecision

func AveragePrecision(ranked []string, labels []Label) float64

AveragePrecision is the mean of the precision values computed at each rank where a relevant tool is retrieved, divided by the total number of relevant tools (so unretrieved relevant tools lower the score). Binary relevance (relevance >= 1) is used, matching the standard MAP definition.

func CanonicalJSON added in v0.51.0

func CanonicalJSON(raw json.RawMessage) (string, error)

CanonicalJSON re-encodes raw JSON bytes into the canonical form used everywhere in the bench (research D7b): object keys sorted lexicographically at every depth, array order preserved, number literals preserved verbatim (via json.Number — no float round-trip), compact (no insignificant whitespace), and no HTML escaping. Identical JSON content in any key order canonicalizes to identical bytes (FR-010).

It lives in the bench package (not bench/arms) because canonicalization happens at every schema INGESTION boundary — LoadCorpusV2, the live /api/v1/tools fetch, and the corpusio loaders — so the plain Tokenizer.CountToolWithSchema and the arm renderers always count the same canonical bytes (contract parity). bench/arms re-exports it for the arm encoders (arms imports bench, never the reverse).

func CompletionRegressionThresholdPct added in v0.64.0

func CompletionRegressionThresholdPct(b *AgentLoopBlock) float64

CompletionRegressionThresholdPct is the stated FR-019 threshold for a block.

A named accessor rather than a bare constant so the value a verdict uses and the value AgentLoopBlockFor applied cannot drift apart — they must be the same number or a cell can be flagged by one and cleared by the other.

func DefaultCallsPerSession added in v0.51.0

func DefaultCallsPerSession() []int

DefaultCallsPerSession returns the default calls-per-session grid {1,3,5,10} (research D8) as a fresh slice, so callers cannot mutate the shared default.

func DefaultStubPatterns added in v0.51.0

func DefaultStubPatterns() []string

DefaultStubPatterns returns the FR-020 default stub-description pattern list as a fresh slice (callers may append their own patterns).

func FleetEndpointURL added in v0.64.0

func FleetEndpointURL(baseURL, apiKey string) (string, error)

FleetEndpointURL builds the URL a live fleet is read from.

The path comes from the direct_full mode cell rather than a literal, so the surface a fleet is captured from stays tied to the one cell that actually enumerates upstream tools; if the matrix ever remounts that surface, this follows it. The API key rides the ?apikey= query parameter, the same surface mcpEndpointURL uses, and never appears in any reported field.

func FleetResolver added in v0.64.0

func FleetResolver(fleet *Corpus) func(serverName, toolName string) bool

FleetResolver builds the loader's replayability predicate from a fleet input: does the (server, tool) a record names still exist in the fleet this replay scores against?

It answers true for mcpproxy's OWN built-in tools regardless of the fleet. A recording of a proxy session is full of retrieve_tools / call_tool_* / describe_tool records, and those are not upstream tools — treating them as vanished would mark essentially every real recording unreplayable and empty the report, which is a wiring bug dressed as a data finding.

It is exported because a caller assembling its own load (a test, or a future bodies-on path) must be able to wire the SAME predicate: the loader treats a nil resolver as "replayability was never evaluated", and a replay that silently took that branch would report "nothing was unreplayable" when it meant "nobody looked".

func GeneratedAtNow added in v0.51.0

func GeneratedAtNow() string

GeneratedAtNow returns the RFC 3339 UTC timestamp the CLI stamps into GeneratedAt. It lives at the CLI boundary on purpose: no library encoder below this call reads the wall clock (FR-010), and tests pass fixed values.

func InHouseProxyMenuTokens added in v0.51.0

func InHouseProxyMenuTokens(tk *Tokenizer) int

InHouseProxyMenuTokens is the in-house count of the proxy's retrieve_tools menu surface (management tools WITH schemas) — the same surface LAP lints, used as the divergence comparison base (FR-016) in offline runs where no live menu was measured.

func InvokeRetrieveTools added in v0.51.0

func InvokeRetrieveTools(ctx context.Context, baseURL, apiKey, query string) (string, time.Duration, error)

InvokeRetrieveTools is the one-shot form: open a session, perform a single retrieve_tools call with the proxy's default limit, and close. The live harness prefers one MCPCaller across the golden set; this entry point serves smoke tests and single-query probes.

func IsValidProvenance added in v0.64.0

func IsValidProvenance(s string) bool

IsValidProvenance reports whether s is one of the three closed-enum values shared by the section-level map and every per-row provenance field.

func NDCGAtK

func NDCGAtK(ranked []string, labels []Label, k int) float64

NDCGAtK is the normalized discounted cumulative gain at k using the graded relevance as the gain (linear gain, log2 position discount). 1.0 means the ranking is in ideal (relevance-descending) order; 0 means no gain in top-k.

func RecallAtK

func RecallAtK(ranked []string, labels []Label, k int) float64

RecallAtK is the fraction of the query's relevant tools (relevance >= 1) that appear in the top-k of the ranking. Returns 0 when there are no relevant tools (a degenerate query that should not be scored).

func ReciprocalRank

func ReciprocalRank(ranked []string, labels []Label) float64

ReciprocalRank is 1/rank of the first relevant tool in the ranking, or 0 if none of the ranked tools are relevant.

func ReplayArmNames added in v0.64.0

func ReplayArmNames() []string

ReplayArmNames returns the arm names a replay needs resolved, in a stable order. It is smaller than the set of arms the mode matrix names, and deliberately so — see replayMenuArmName for why the discovery-serialization arm never prices a menu.

func RequireEOF added in v0.51.0

func RequireEOF(dec *json.Decoder) error

RequireEOF verifies dec has no data left after the first decoded top-level value. dec.More() is NOT a valid trailing-data check at the top level: it merely peeks for a byte other than ']' or '}', so streams like `{}}` or `{}]` slip through (and `{} {}` is a second value, not "more of" the first). Draining one more token and requiring io.EOF rejects them all.

func ResolveRetryRate added in v0.64.0

func ResolveRetryRate(arm string, measured MeasuredOutcomeSource) (rate float64, provenance string)

ResolveRetryRate returns the retry rate to price an arm with, together with the provenance of that rate: ProvenanceMeasured when a live measurement supersedes the default, ProvenanceEstimated when the documented default stands. The provenance is the whole point of the second return value — the rate alone cannot tell a measured 0.0 from a defaulted 0.0.

func RetryRateForArm added in v0.51.0

func RetryRateForArm(arm string) float64

RetryRateForArm returns the documented retry-rate default for an arm. Unknown arms default to 0.0 (no retry evidence → no penalty; the rate is echoed in every estimate row, so the assumption stays visible).

func SessionCostRowsMixed added in v0.64.0

func SessionCostRowsMixed(rows []SessionCostRow) bool

SessionCostRowsMixed reports whether a table holds rows of more than one provenance. A caller rendering a section badge must check this: a single section badge over a mixed table is precisely the misrepresentation FR-013 forbids, so a mixed table must be badged per row instead.

Types

type AccountingSource added in v0.64.0

type AccountingSource struct {
	// Kind is AccountingKindTokenizer or AccountingKindProvider.
	Kind string `json:"kind"`
	// Identity names the estimator ("cl100k_base") or the provider.
	Identity string `json:"identity"`
	// Model pins the model for provider-sourced figures. Mandatory for
	// AccountingKindProvider, empty otherwise — usage numbers from an
	// unpinned model are not comparable to any later run.
	Model string `json:"model,omitempty"`
}

AccountingSource names where one block's numbers came from. It is deliberately a value, not a string: a provider-sourced block is only interpretable with the model pinned alongside the provider, and squeezing both into one enum string would make the "never sum across sources" comparison a string-parsing exercise.

func AgentLoopAccountingSource added in v0.64.0

func AgentLoopAccountingSource(provider, model string) (AccountingSource, error)

AgentLoopAccountingSource builds the block's accounting source (T043).

A provider-sourced block names the provider AND the pinned model, because the two together are what makes a usage number mean anything. This is the same refusal shape as ErrStubToolSchemas in mcptools.go: the guard returns an error rather than a plausible-looking default, so an unpinned run cannot produce a report that merely looks comparable to a pinned one.

func (AccountingSource) IsZero added in v0.64.0

func (a AccountingSource) IsZero() bool

IsZero reports whether the source was never populated. A block with a zero source cannot enter any aggregate (data-model.md: accounting_source is mandatory), so emission-time validation rejects it rather than defaulting.

type AgentLoopBlock added in v0.64.0

type AgentLoopBlock struct {
	AccountingSource AccountingSource `json:"accounting_source"`
	// CompletionRegressionThresholdPct is the stated FR-019 threshold this
	// block was evaluated against. Serialised so a reader knows which number
	// separated "saving" from "regression", and so a verdict recomputed from a
	// loaded report uses the SAME threshold the run applied rather than
	// whatever the current default happens to be.
	CompletionRegressionThresholdPct float64 `json:"completion_regression_threshold_pct,omitempty"`
	// Suite and SuiteVersion pin the task suite so a later run is comparable
	// to an earlier one (FR-028).
	Suite        string           `json:"suite,omitempty"`
	SuiteVersion string           `json:"suite_version,omitempty"`
	FleetShape   FleetShape       `json:"fleet_shape"`
	Cells        []AgentLoopCell  `json:"cells,omitempty"`
	Inputs       *InputProvenance `json:"inputs,omitempty"`
	Records      *RawRecordsRef   `json:"records,omitempty"`
}

AgentLoopBlock is the live measured block (US2). Its AccountingSource is provider-usage plus the pinned model, which is why the block-level field exists: this block and the deterministic ones sit in one document and must stay separable.

func AgentLoopBlockFor added in v0.64.0

func AgentLoopBlockFor(records []UnitRecord, opts AgentLoopOptions) (*AgentLoopBlock, error)

AgentLoopBlockFor aggregates raw records into the agent_loop report block (T043).

The arithmetic decisions worth stating plainly, because each one is a place a friendlier number was available:

  • A run in which ANY unit is partial is excluded WHOLESALE, not unit-by-unit. A run that died halfway completed a PREFIX of the task set, and averaging that prefix biases the figure toward whichever tasks the suite happens to order first.
  • Units with no completion signal are excluded from completion-dependent figures on BOTH sides of the ratio — their tokens leave the numerator with them. Keeping their cost while dropping their (absent) verdict would inflate cost per completed task; the reverse would deflate it.
  • Tokens per completed task is a POOLED ratio (total tokens over total completions), not a mean of per-run ratios, so a run that completed nothing still contributes the money it spent. Spread is measured across the per-run ratios that are defined.
  • A cell whose token accounting is missing an axis is not a headline. An incomplete total understates cost, and understating cost in this project's favour is the exact failure this feature exists to prevent.

func RunAgentLoop added in v0.64.0

func RunAgentLoop(driver AgentLoopDriver, opts AgentLoopOptions) (*AgentLoopBlock, error)

RunAgentLoop drives the fixed task set across every configured cell — the FR-020 baseline arm included — and aggregates the result into the report block (T039/T040/T043).

func (*AgentLoopBlock) SavingVsBaseline added in v0.64.0

func (b *AgentLoopBlock) SavingVsBaseline(cellID string) SavingVerdict

SavingVsBaseline is the only sanctioned way to turn two agent-loop cells into a percentage (FR-019/FR-020/SC-007).

It refuses in three situations, each for a reason that would otherwise be invisible in the published number:

  1. There is no usable baseline. FR-020's denominator is the same agent doing the same tasks with every tool loaded directly; without it a "saving" is measured against nothing.
  2. The cell is a REGRESSION — its completion rate fell more than the stated threshold below the baseline's. A mode that finishes fewer tasks is not cheap, it is broken, and describing it as a saving is precisely what SC-007 exists to prevent. Its token cost is still reported; it is the word "saving" that is refused.
  3. Either cell's token accounting is incomplete. Comparing a complete total against a lower bound produces a percentage of two different quantities.

The comparison inside one block is legitimate because both cells share the block's accounting source. Comparing a cell here against a figure from the deterministic blocks is NOT, which is what AggregateTokenFigures refuses.

type AgentLoopCell added in v0.64.0

type AgentLoopCell struct {
	CellID string `json:"cell_id"`
	// Provenance is measured for a cell backed by real runs; a row still
	// carrying an assumed default stays estimated and must be rendered as
	// such beside its measured neighbours (FR-013).
	Provenance string `json:"provenance"`
	// Runs and SpreadPct are the FR-021 consistency requirement: a
	// model-dependent figure needs at least four runs plus a spread before
	// it may be a headline.
	Runs int `json:"runs"`
	// SpreadPct is a POINTER: with fewer than two per-run figures the spread is
	// UNDEFINED, not zero. Serialising 0 there renders "±0.0%" — a claim of
	// perfect consistency from a single run, which is the opposite of what
	// FR-021 asks the field to convey.
	SpreadPct *float64 `json:"spread_pct"`
	// PartialRuns counts runs that did not finish. They are excluded from
	// the figures above and reported, never silently absorbed (FR-032).
	PartialRuns int `json:"partial_runs,omitempty"`
	// TokensPerCompletedTask is the FR-018 headline, and CompletionRatePct
	// travels with it at equal prominence — a cheap mode that finishes
	// fewer tasks is not a saving.
	TokensPerCompletedTask float64 `json:"tokens_per_completed_task"`
	CompletionRatePct      float64 `json:"completion_rate_pct"`
	// FirstAttemptSuccessPct is a POINTER for the same reason: with no
	// determinate intents there is no rate, and a serialised 0 reads as "the
	// agent never got a call right first time" rather than "nothing was
	// measured".
	FirstAttemptSuccessPct *float64 `json:"first_attempt_success_pct"`
	// Corrective and infrastructure retries are counted separately: only the
	// first kind says anything about how well the mode serves the agent.
	RetriesCorrective     int `json:"retries_corrective"`
	RetriesInfrastructure int `json:"retries_infrastructure"`
	// Input/output/cache-read consumption stays split (FR-014); collapsing
	// them hides the cache behaviour that dominates a long agent loop.
	InputTokens     int `json:"input_tokens"`
	OutputTokens    int `json:"output_tokens"`
	CacheReadTokens int `json:"cache_read_tokens"`
	// Headline is false whenever the FR-021 bar is unmet; the row is still
	// published, just not as a headline.
	Headline bool `json:"headline"`
	// Regression marks a cell whose completion rate falls more than the
	// stated threshold below the baseline's. Such a cell must never be
	// described as a saving regardless of its token cost (FR-019).
	Regression bool `json:"regression,omitempty"`
	// Withheld carries a figure that cannot be computed honestly — a
	// cross-source aggregate above all — with the reason stated.
	Withheld       bool   `json:"withheld,omitempty"`
	WithheldReason string `json:"withheld_reason,omitempty"`
}

AgentLoopCell is one mode cell measured over a live agent loop (US2). Its numbers come from provider-reported usage, so they may never be summed with the deterministic blocks' figures.

type AgentLoopDriver added in v0.64.0

type AgentLoopDriver struct {
	// RunTask is mandatory.
	RunTask RunTaskFunc
	// CaptureUsage is OPTIONAL. When nil, records keep the usage RunTask
	// returned — which, for a suite-derived record, has no cache-read count,
	// and the resulting cells are marked accordingly rather than zero-filled.
	CaptureUsage UsageFunc
}

AgentLoopDriver is the pair of seams a live run plugs into.

type AgentLoopOptions added in v0.64.0

type AgentLoopOptions struct {
	// Suite and SuiteVersion pin the task suite (FR-028). SuiteVersion should
	// be the commit SHA the run was executed at.
	Suite        string
	SuiteVersion string
	// Provider and Model name the accounting source. Both are mandatory: see
	// ErrAgentLoopModelUnpinned.
	Provider string
	Model    string
	// FleetShape is the tool set every percentage from this run holds for
	// (IC-004). A percentage without it does not get published.
	FleetShape FleetShape
	// Cells is the matrix under measurement. It MUST include the baseline
	// cell — that is the denominator, not an optional extra row.
	Cells []ModeCell
	// Units is the fixed task set, run identically under every cell.
	Units []string
	// Runs is k, the number of repetitions per cell (FR-021 wants >= 4).
	Runs int
	// CompletionRegressionThresholdPct overrides the stated FR-019 threshold.
	// Zero means DefaultCompletionRegressionThresholdPct.
	CompletionRegressionThresholdPct float64
}

AgentLoopOptions configures one live measurement.

type AggregateResult added in v0.64.0

type AggregateResult struct {
	Value          float64          `json:"value"`
	Source         AccountingSource `json:"source"`
	Withheld       bool             `json:"withheld,omitempty"`
	WithheldReason string           `json:"withheld_reason,omitempty"`
}

AggregateResult is an aggregate that either has a value or has a reason it does not. It mirrors the harness's existing withhold-rather-than-compute pattern (AuthoritativeHeadline / withholdHeadline in live_report.go): when the inputs cannot honestly be combined, the number is WITHHELD with a stated reason rather than computed and quietly caveated somewhere else.

func AggregateTokenFigures added in v0.64.0

func AggregateTokenFigures(figures []SourcedFigure) AggregateResult

AggregateTokenFigures sums figures ONLY when every one of them came from the same accounting source, and otherwise withholds with a stated reason (T038).

This is data-model.md invariant 2, and it is enforced here rather than left to callers because the failure it prevents is silent: tokenizer counts and provider-reported usage are both "tokens", both plausible magnitudes, and a summed hybrid looks exactly like a real number. There is no downstream check that would catch it. Two figures from the same provider but DIFFERENT pinned models are also different sources — the same task has a different token shape under a different model.

type ArmIndex added in v0.51.0

type ArmIndex struct {
	// contains filtered or unexported fields
}

ArmIndex is a scratch-directory retrieval index built through the production index code for one arm's encoding of one corpus. Close releases the underlying Bleve index; the scratch directory itself is owned by the caller (tests pass t.TempDir()).

func BuildArmIndex added in v0.51.0

func BuildArmIndex(dir string, metas []*config.ToolMetadata) (*ArmIndex, error)

BuildArmIndex creates a fresh production index.Manager rooted at dir (the index lands in dir/index.bleve) and batch-indexes the given per-tool metadata — the output of an arm's EncodeIndexMetadata over a corpus. dir must be an empty scratch directory: an existing index there would be silently reused and contaminate the measurement.

func (*ArmIndex) Close added in v0.51.0

func (ai *ArmIndex) Close() error

Close releases the underlying index.

func (*ArmIndex) SearchFunc added in v0.51.0

func (ai *ArmIndex) SearchFunc() SearchFunc

SearchFunc returns the ScoreRetrieval adapter over the production search path: each query runs through Manager.SearchTools (the exact BM25 query shape retrieve_tools uses) and the hits are returned as ranked "server:tool" IDs — the golden set's tool_id format, matching how the live REST path assembles IDs (LiveClient.Search).

type ArmResult added in v0.51.0

type ArmResult struct {
	Arm                  string           `json:"arm"`
	CorpusID             string           `json:"corpus_id"`
	Skipped              bool             `json:"skipped"`
	SkipReason           string           `json:"skip_reason,omitempty"`
	LowerBound           bool             `json:"lower_bound,omitempty"`
	IndexAltering        bool             `json:"index_altering"`
	PayloadClass         string           `json:"payload_class,omitempty"` // "listing" | "results"
	FixtureID            string           `json:"fixture_id,omitempty"`
	TabularCount         *int             `json:"tabular_count,omitempty"`
	NonTabularCount      *int             `json:"non_tabular_count,omitempty"`
	TotalTokens          int              `json:"total_tokens"`
	MeanTokens           float64          `json:"mean_tokens"`
	P95Tokens            int              `json:"p95_tokens"`
	SavingsVsBaselinePct float64          `json:"savings_vs_baseline_pct"`
	SkippedTools         int              `json:"skipped_tools"`
	SkipExamples         []SkipExample    `json:"skip_examples,omitempty"`
	HeaviestTools        []ToolTokenEntry `json:"heaviest_tools,omitempty"`
	Quality              *RetrievalScore  `json:"quality"`
}

ArmResult is one encoding arm measured on one corpus (FR-005/006/007/009).

Contract conditionals: a skipped row requires SkipReason; a non-skipped row requires the token stats; a results-class row requires FixtureID and the tabular/non-tabular split (pointers so an explicit 0 still serializes); a non-skipped index-altering row requires the quality key (nullable only when the corpus has no relevance labels, with MetricNote explaining the absence).

func RunArms added in v0.51.0

func RunArms(tk *Tokenizer, corpus *Corpus, armsToRun []EncodingArm, opts ArmRunOptions) ([]ArmResult, error)

RunArms measures every given arm on the corpus and returns one ArmResult per arm, in caller order. The baseline_json arm MUST be among armsToRun — its listing total is the savings denominator (research D7b). Arms whose runtime is absent never reach this function: the harness records them via SkippedArmResult after registry resolution fails (contract rule 5).

func SkippedArmResult added in v0.51.0

func SkippedArmResult(armName, corpusID, reason string) ArmResult

SkippedArmResult is the arm-level skip row (FR-006, contract rule 5): the arm's external runtime is absent, so no tool was processed and only the identity + reason fields are meaningful.

type ArmRunOptions added in v0.51.0

type ArmRunOptions struct {
	// CorpusID identifies the corpus in report rows (e.g. "corpus_v2@<sha>");
	// empty defaults to the corpus's own Version.
	CorpusID string
	// Golden enables retrieval-quality scoring for index-altering arms. nil
	// means the corpus has no relevance labels: index-altering rows then carry
	// a quality block with only an explanatory MetricNote (FR-008/011).
	Golden *GoldenSet
	// HeaviestN sizes the heaviest-tools list; <=0 defaults to DefaultHeaviestN.
	HeaviestN int
	// IndexDir is the scratch parent directory for per-arm retrieval indexes.
	// Empty creates (and removes) a temp directory per run.
	IndexDir string
}

ArmRunOptions configures one RunArms invocation.

type Attempt added in v0.64.0

type Attempt struct {
	// IntentID groups attempts aimed at the same goal. Attempts sharing an
	// IntentID form one intent; the tool name is the usual value.
	IntentID string `json:"intent_id"`
	// Seq orders attempts within an intent (1-based).
	Seq int `json:"seq"`
	// Outcome is one of the AttemptOutcome* values.
	Outcome string `json:"outcome"`
	// ArgsFingerprint identifies the arguments this attempt was issued with.
	ArgsFingerprint string `json:"args_fingerprint,omitempty"`
}

Attempt is ONE tool call issued toward a given intent — the spec's ATTEMPT, not its unit of work.

ArgsFingerprint is what makes the binding Definition's carve-out decidable. "A schema-valid call the agent immediately re-issues DIFFERENTLY is not a success" needs a notion of "differently", and the arguments are it: a re-issue with the same arguments after a transport failure is the network being retried, while a re-issue with changed arguments is the agent correcting itself, which is the thing a routing/serialization mode can actually be blamed for.

func LoadMCPMarkTrajectory added in v0.64.0

func LoadMCPMarkTrajectory(path string) (attempts []Attempt, truncated bool, err error)

LoadMCPMarkTrajectory reads the suite's messages.json and reconstructs the ordered ATTEMPT sequence that retry classification needs.

Why the trajectory and not just meta.json: meta.json says whether the task passed and what it cost, but first-attempt success and corrective-versus- infrastructure retries are properties of the SEQUENCE of calls. Without the trajectory, FR-010 and FR-011 are unmeasurable and the whole "does this mode help the agent succeed" question collapses back into token counting.

Two classification honesty notes:

  • The intent is the tool NAME and the arguments are the fingerprint. This is an approximation of "intent" — two calls to the same tool for genuinely different purposes read as one intent — and it is the approximation that errs toward counting MORE corrective retries, i.e. against mcpproxy rather than for it.
  • An error whose type is not recognisably a transport failure is classified as a MODE error, not an infrastructure one, for the same reason: the conservative direction is the one that does not flatter us.

A trailing tool call with no result marks the record truncated (the caller sets Partial); an unanswered call with later calls after it is a malformed trajectory and an error, because it means results were dropped rather than cut off.

type BreakEvenAnalysis added in v0.51.0

type BreakEvenAnalysis struct {
	NaiveFullMenuTokens int     `json:"naive_full_menu_tokens"`
	ProxyMenuTokens     int     `json:"proxy_menu_tokens"`
	MeanResponseTokens  float64 `json:"mean_response_tokens"`
	BreakEvenCalls      float64 `json:"break_even_calls"`
	NoBreakEven         bool    `json:"no_break_even,omitempty"`
}

BreakEvenAnalysis is the FR-003 break-even computation with its inputs echoed (FR-004): break_even_calls = (naive − proxy_menu) / mean_response.

func ComputeBreakEven added in v0.51.0

func ComputeBreakEven(naiveFullMenuTokens, proxyMenuTokens int, meanResponseTokens float64) (*BreakEvenAnalysis, error)

ComputeBreakEven computes the number of retrieve_tools calls at which the proxy's cumulative context cost (proxy menu + N discovery responses) equals the naive full-menu cost:

break_even_calls = (naive_full_menu_tokens − proxy_menu_tokens) / mean_response_tokens

Below break_even_calls discovery calls the proxy is strictly cheaper; each call beyond it spends part of the menu savings. All three inputs are echoed in the result (FR-004) so a reader can recompute the number from the row alone. Both menu inputs MUST come from the same canonical full-definition renderer (research D7b) or the numerator compares different token shapes.

When the numerator is ≤ 0 the proxy menu is already at least as expensive as the naive menu — there are no savings to amortize — and the result is a NoBreakEven row (BreakEvenCalls stays 0), whatever the response cost. Otherwise meanResponseTokens must be positive: a zero mean means no responses were measured, and dividing by it is undefined, so that is an error rather than a fabricated verdict.

type CapabilityCondition added in v0.64.0

type CapabilityCondition struct {
	ID        string   `json:"id"`
	AppliesTo []string `json:"applies_to"`
}

CapabilityCondition is one FR-016 binary toggle together with the cells it applies to. The enumeration is part of the contract: the report must say which rows a capability was measured on, so a reader never has to guess whether an absent figure means "off" or "not available here".

func CapabilityConditions added in v0.64.0

func CapabilityConditions() []CapabilityCondition

CapabilityConditions returns the FR-016 binary conditions with their applicable rows enumerated. They are conditions over existing cells, not a fourth axis: turning one on does not create a new cell, it re-measures the cells listed here.

type Combination added in v0.64.0

type Combination struct {
	RoutingMode            string
	ToolResponseMode       string
	DirectToolResponseMode string
	EnableCodeExecution    bool
}

Combination is a configured (routing mode, discovery serialization, direct serialization) triple plus the one config flag that can make the code-execution surface degenerate. It is what an operator can actually set; ResolveCombination maps it to the behaviour that would actually run.

type Corpus

type Corpus struct {
	Version string `json:"version"`
	Tools   []Tool `json:"tools"`
}

Corpus is a frozen, versioned set of tool definitions.

func FleetFromToolsList added in v0.64.0

func FleetFromToolsList(ctx context.Context, surface, fleetID string, page ToolsListPageFunc) (*Corpus, error)

FleetFromToolsList fetches a listing and wraps its UPSTREAM tools as a Corpus fit to be a replay fleet.

The built-ins are dropped, and that is a modelling decision worth stating: a fleet is the upstream catalog. Replay prices mcpproxy's own tools per cell from ProxyToolsForMode, the frozen corpora contain upstream tools only, and folding describe_tool into the fleet would both double-count it in the retrieve cells and move the direct cells off the basis every frozen-corpus figure is quoted on. (replay.go already records the reverse gap: the direct menu omits describe_tool, which biases the direct delta percentage slightly upward. Keeping the fleet upstream-only keeps that ONE stated gap rather than adding a second, opposite one that depends on the fleet's source.)

func LoadCorpus

func LoadCorpus(path string) (*Corpus, error)

LoadCorpus reads a frozen corpus snapshot (e.g. the Spec 065 corpus_v1.tools.json) from disk.

func LoadCorpusV2 added in v0.51.0

func LoadCorpusV2(path string) (*Corpus, error)

LoadCorpusV2 reads the Spec 083 schema-bearing frozen corpus (specs/083-discovery-profiler/datasets/corpus_v2.tools.json). On top of LoadCorpus it enforces the corpus_v2 contract: every tool MUST carry a non-empty JSON input schema (arm comparison over a schema-less corpus is meaningless — research D4), and each schema is CANONICALIZED at load (CanonicalJSON: sorted keys, compact, verbatim numbers) so the in-memory bytes match what the arm renderers produce — CountToolWithSchema and the baseline arm count identical bytes even for a non-canonical input file (the committed file is pretty-printed for git diffability; its keys are already sorted by the generator's jq -S, so this is a no-op for it).

func ResolveFleet added in v0.64.0

func ResolveFleet(frozen *Corpus, frozenID string, live FleetSource) (*Corpus, string, error)

ResolveFleet picks the ONE fleet a replay is priced over, from the two ways of supplying one: a frozen corpus, or a live proxy's own catalog read over MCP tools/list (mcptools.go).

Two rules, and they are the same rule seen from both sides.

NEITHER given is ErrReplayFleetRequired. A menu is a property of the tool definitions and the activity export carries no fleet snapshot, so a recording on its own computes nothing at all. Adding a live source does NOT soften this: it adds a second way to supply a fleet, not permission to omit one.

BOTH given is ErrReplayFleetAmbiguous. This one is easy to get wrong in the accommodating direction — prefer the live proxy, say, and warn. But every figure in the report is quoted for exactly one fleet shape, named by a single fleet_id, and a reader has no way to tell which input that id came from. A silent preference therefore does not produce a slightly-less-precise report; it produces a report whose provenance line is false. Refuse instead.

The live fetch happens HERE rather than at the call site so that a live fleet's failure lands on the same code path as every other fleet failure. A caller that fell back to a frozen corpus on a failed fetch would answer a different question than the one that was asked, using the same headline.

type CorpusDescriptor added in v0.51.0

type CorpusDescriptor struct {
	ID                     string                  `json:"id"`
	Name                   string                  `json:"name"`
	Version                string                  `json:"version"`
	ToolCount              int                     `json:"tool_count"`
	License                string                  `json:"license"`
	Attribution            string                  `json:"attribution,omitempty"`
	Committed              bool                    `json:"committed"`
	DegenerateDescriptions *DegenerateDescriptions `json:"degenerate_descriptions,omitempty"`
}

CorpusDescriptor identifies one measured corpus with license/attribution provenance (FR-011/012/013).

type CostOutcomeExclusion added in v0.64.0

type CostOutcomeExclusion struct {
	CellID string
	Reason string
}

CostOutcomeExclusion names a cell that could not be plotted, with the reason carried through verbatim. An unexplained absence is indistinguishable from an oversight.

type CostOutcomePoint added in v0.64.0

type CostOutcomePoint struct {
	CellID string
	// TokensPerCompletedTask is the FR-018 cost headline, and
	// CompletionRatePct is its equal-prominence companion.
	TokensPerCompletedTask float64
	CompletionRatePct      float64
	// PlotX/PlotY are viewBox coordinates from the axis mappings below.
	PlotX float64
	PlotY float64
	// Headline is false when the cell has not met the FR-021 bar; such a
	// point is drawn hollow, present but not load-bearing.
	Headline bool
	// Regression marks a cell that completes materially fewer tasks than the
	// baseline. It is drawn in the warning colour and must never be described
	// as a saving (FR-019).
	Regression bool
	// Provenance is the cell's own row provenance (FR-013): a cell still
	// resting on an assumption is badged as such beside its measured
	// neighbours.
	Provenance string
	// Runs and SpreadPct travel with the point so the plot's tooltip can say
	// how much evidence stands behind it.
	Runs      int
	SpreadPct float64
	// LabelAnchor and LabelDX place the cell name clear of its marker. A
	// point in the right half of the plot is labelled to its LEFT, or a long
	// cell name on the expensive end of the axis runs off the viewBox and
	// the reader loses the identity of the most costly cell — the one they
	// most need to identify.
	LabelAnchor string
	LabelDX     float64
}

CostOutcomePoint is one mode cell positioned in the cost/outcome plane. It carries both the raw figures (so the table and the plot cannot drift apart) and the pre-computed SVG coordinates (so the template does no arithmetic).

type CostOutcomeTick added in v0.64.0

type CostOutcomeTick struct {
	Label string
	Pos   float64
}

CostOutcomeTick is one labelled axis gridline, positioned in viewBox units.

type CostOutcomeView added in v0.64.0

type CostOutcomeView struct {
	Points   []CostOutcomePoint
	Excluded []CostOutcomeExclusion
	// CostAxisMaxTokens is the zero-based cost axis maximum: the most
	// expensive plotted cell plus headroom, never less than the data.
	CostAxisMaxTokens float64
	// CompletionAxisMaxPct is fixed at 100 — see the block comment.
	CompletionAxisMaxPct float64
	CostTicks            []CostOutcomeTick
	CompletionTicks      []CostOutcomeTick
}

CostOutcomeView is everything the dashboard needs to draw the FR-023 plot. It is built in Go rather than in the template so the axis arithmetic is unit-testable and so a degenerate input (one cell, all-zero costs, a nil block) cannot produce NaN coordinates inside an SVG attribute, where they would fail silently.

func NewCostOutcomeView added in v0.64.0

func NewCostOutcomeView(b *AgentLoopBlock) CostOutcomeView

NewCostOutcomeView builds the plot for one agent-loop block. A nil block yields an empty view rather than a panic, so the template can call this unconditionally.

func (CostOutcomeView) HasPlot added in v0.64.0

func (v CostOutcomeView) HasPlot() bool

HasPlot reports whether there is anything to draw. A block whose every cell was withheld still renders its exclusion list, but no empty axes.

func (CostOutcomeView) PlotX added in v0.64.0

func (v CostOutcomeView) PlotX(tokens float64) float64

PlotX maps a token cost onto the horizontal axis.

func (CostOutcomeView) PlotY added in v0.64.0

func (v CostOutcomeView) PlotY(pct float64) float64

PlotY maps a completion rate onto the vertical axis (0% at the bottom).

type DegenerateDescriptions added in v0.51.0

type DegenerateDescriptions struct {
	Count int      `json:"count"`
	Rules []string `json:"rules,omitempty"`
}

DegenerateDescriptions counts corpus tools whose descriptions trip the FR-020 quality rules, with the rule list echoed for reproducibility.

func CountDegenerateDescriptions added in v0.51.0

func CountDegenerateDescriptions(tools []Tool, stubPatterns []string) (*DegenerateDescriptions, error)

CountDegenerateDescriptions counts corpus tools whose descriptions trip any FR-020 rule — empty/whitespace-only, shorter than 20 characters, or matching a stub pattern — each tool counted once, with the applied rule list echoed for reproducibility. An invalid pattern is an explicit error.

type DescribeToolUsage added in v0.51.0

type DescribeToolUsage struct {
	Calls          int     `json:"calls"`
	CompletedTasks int     `json:"completed_tasks"`
	PerTask        float64 `json:"per_task"`
}

DescribeToolUsage is the informational FR-018 metric: describe_tool calls per completed task, collected by the E2E suite (not by a live bench run).

type DiscoveryResponseMeasurement added in v0.51.0

type DiscoveryResponseMeasurement struct {
	QueryID     string         `json:"query_id"`
	TotalTokens int            `json:"total_tokens"`
	ResultCount int            `json:"result_count,omitempty"`
	LatencyMs   float64        `json:"latency_ms,omitempty"`
	Components  map[string]int `json:"components"`
}

DiscoveryResponseMeasurement is one golden query's retrieve_tools response cost with its span-attributed component breakdown (FR-001/002). Invariant: the component values sum EXACTLY to TotalTokens — enforced by construction in the span attributor (bench/respcost.go), never by re-tokenizing fields.

func MeasureRetrieveToolsResponse added in v0.51.0

func MeasureRetrieveToolsResponse(tk *Tokenizer, queryID, raw string, latencyMs float64) (*DiscoveryResponseMeasurement, error)

MeasureRetrieveToolsResponse is the one-call US1 measurement: partition the raw response text, attribute tokens, and assemble the report row. queryID and latencyMs are caller-observed (the harness measures latency around the MCP call, FR-023).

type EncodingArm added in v0.51.0

type EncodingArm interface {
	Name() string
	IndexAltering() bool
	LowerBound() bool
	EncodeTool(t Tool) (string, error)
	EncodeListing(ts []Tool) (string, error)
	EncodeIndexMetadata(t Tool) (config.ToolMetadata, error)
}

EncodingArm is the bench-side mirror of the arms.Arm contract (specs/083-discovery-profiler/contracts/arm-interface.md). It is declared structurally here because bench cannot import bench/arms (arms imports bench); every arms.Arm value satisfies it as-is.

type FleetShape added in v0.64.0

type FleetShape struct {
	ID                   string  `json:"id"`
	ToolCount            int     `json:"tool_count"`
	MeanDefinitionTokens float64 `json:"mean_definition_tokens,omitempty"`
	P95DefinitionTokens  int     `json:"p95_definition_tokens,omitempty"`
	// SchemalessTools counts tools in this fleet that carried no priceable
	// input schema.
	//
	// A fleet is refused outright only when NO tool has a real schema. A
	// PARTIAL regression — 3 of 45, or 20 of 45 — passes that guard and prices
	// the stubbed ones at nothing, quietly shrinking the baseline and
	// inflating the headline. Without this count the report carries no trace
	// of it, so the reader cannot tell a clean fleet from a partly stubbed
	// one, and a drift from 3 to 20 between runs is invisible.
	SchemalessTools int `json:"schemaless_tools,omitempty"`
}

FleetShape is the tool count and definition-size distribution a figure holds for. Every published percentage carries one (IC-004): a saving quoted without its fleet shape is a saving quoted at an unstated fleet size, which is how spec 102 over-projected from a single corpus.

type FleetSource added in v0.64.0

type FleetSource interface {
	// FleetID names the fleet shape every figure is quoted for (IC-004). It
	// must never carry a credential: it is reported.
	FleetID() string
	// Fleet returns the tool definitions the menu cost is computed from.
	Fleet() (*Corpus, error)
}

FleetSource supplies a replay fleet on demand.

It is an interface for the reason EncodingArm is one (armrun.go): the seam must be satisfiable by a value a test writes inline, not only by a live client. It also keeps the "exactly one fleet input" rule (ResolveFleet) in package bench, where the fleet-is-mandatory rule already lives, instead of splitting the two halves of one rule across bench and the CLI.

type FlipGateReport added in v0.51.0

type FlipGateReport struct {
	Encoding       string                `json:"encoding"`
	RankedIdentity *RankedIdentityGate   `json:"ranked_identity"`
	Tokens         *TokenReductionReport `json:"tokens"`
	Lossy          *LossySignatureGate   `json:"lossy,omitempty"`
	DescribeTool   *DescribeToolUsage    `json:"describe_tool_usage,omitempty"`
	Notes          []string              `json:"notes,omitempty"`
}

FlipGateReport bundles every FR-018 flip-gate metric.

func RunFlipGates added in v0.51.0

func RunFlipGates(retrieve RetrieveToolsFunc, golden *GoldenSet, tk *Tokenizer, limit int) (*FlipGateReport, error)

RunFlipGates replays every golden query through retrieve_tools twice — once per detail mode, same pipeline (FR-017) — and computes the ranked-ID identity gate plus the full/compact token distributions. The lossy gate is computed separately (ComputeLossyGate) because it needs a schema-bearing corpus, which a golden set does not carry.

func RunLiveFlipGates added in v0.51.0

func RunLiveFlipGates(ctx context.Context, mcpURL string, golden *GoldenSet, upstream []Tool, corpusVersion string, tk *Tokenizer, limit int) (*FlipGateReport, error)

RunLiveFlipGates is the live compact arm (FR-017): it connects an MCP client to a running proxy, replays the golden set through retrieve_tools in both detail modes, and computes the full flip-gate report. The lossy gate runs over the live upstream tool corpus (with schemas from GET /api/v1/tools); the frozen corpus_v2 re-baseline follows the 083 rebase (tasks T040/T043).

type GoldenQuery

type GoldenQuery struct {
	ID     string  `json:"id"`
	Query  string  `json:"query"`
	Labels []Label `json:"labels"`
}

GoldenQuery is one labelled query -> relevant-tool(s) judgement.

type GoldenSet

type GoldenSet struct {
	CorpusVersion string        `json:"corpus_version"`
	Queries       []GoldenQuery `json:"queries"`
}

GoldenSet is the frozen Spec 065 retrieval golden set (retrieval_golden_v1.json).

func LoadGoldenSet

func LoadGoldenSet(path string) (*GoldenSet, error)

LoadGoldenSet reads the Spec 065 retrieval golden set (retrieval_golden_v1.json) from disk.

type HeaviestSignature added in v0.51.0

type HeaviestSignature struct {
	ToolID string `json:"tool_id"`
	Tokens int    `json:"tokens"`
	Sig    string `json:"sig"`
	Lossy  bool   `json:"lossy"`
}

HeaviestSignature is one corpus signature ranked by token cost, for triage of what compact mode still spends tokens on.

type IndexMetadataFunc added in v0.51.0

type IndexMetadataFunc func(t Tool) (config.ToolMetadata, error)

IndexMetadataFunc maps one corpus tool to the exact metadata the production retrieval index ingests for an arm. It is the bench-side shape of arms.Arm.EncodeIndexMetadata (bench cannot import bench/arms — arms imports bench — so the mapping crosses the boundary as a function value).

type InputProvenance added in v0.64.0

type InputProvenance struct {
	Availability              string `json:"availability"`
	IndependentlyReproducible bool   `json:"independently_reproducible"`
	// Limitation states, in the report itself, WHY an outsider cannot
	// reproduce this block. Mandatory whenever the block is not
	// independently reproducible: an unexplained "false" is only marginally
	// better than no mark, because the reader cannot tell whether the input
	// is private, expired, or merely undocumented.
	Limitation string `json:"limitation,omitempty"`
	// Procedure names the documented, pinned procedure that obtains the
	// inputs. Mandatory for pinned-procedure availability — a procedure that
	// is not named is not documented, and FR-030 asks for a documented one.
	Procedure string `json:"procedure,omitempty"`
}

InputProvenance is one block's FR-030 mark: where its inputs come from and, consequently, whether an outsider can reproduce its figures.

IndependentlyReproducible is deliberately redundant with Availability — it is exactly `Availability != private-recording`, and Validate enforces that. The redundancy is the point: a renderer, a dashboard or an outside reader consuming the JSON should not have to know which enum values imply reproducibility. The one-word answer is in the document, and the enum explains it. A mismatch between the two is a hard error, so the redundancy can never drift into a contradiction.

func PrivateRecordingInputs added in v0.64.0

func PrivateRecordingInputs(limitation string) *InputProvenance

PrivateRecordingInputs marks a block as built on one operator's own recorded sessions (T056/FR-030). It is a constructor rather than a struct literal so the not-reproducible flag cannot be forgotten at a call site: the whole value of this mark is that it is never omitted on the one block that needs it most.

func ReproducibleInputs added in v0.64.0

func ReproducibleInputs(availability, procedure string) *InputProvenance

ReproducibleInputs marks a block whose inputs an outsider can obtain, either from the repository or by following the named pinned procedure.

func (*InputProvenance) Validate added in v0.64.0

func (p *InputProvenance) Validate(block string) error

Validate reports whether the mark is internally consistent. block names the report block for the error message.

type IntentOutcome added in v0.64.0

type IntentOutcome struct {
	IntentID string `json:"intent_id"`
	// FirstAttemptSuccess implements the binding Definition verbatim: the
	// first attempt returned a non-error result AND was not followed by a
	// corrective retry for the same intent.
	FirstAttemptSuccess bool `json:"first_attempt_success"`
	// Indeterminate marks an intent whose first attempt failed for
	// infrastructure reasons or whose outcome is unknown. Such an intent is
	// EXCLUDED from the first-attempt-success denominator rather than counted
	// as a failure: it measures the network (or a truncated record), not the
	// mode, and scoring it would let network weather move a published figure.
	Indeterminate bool `json:"indeterminate"`
	// Corrective and infrastructure retries are counted separately (FR-011)
	// because only the first says anything about how well the mode serves the
	// agent.
	RetriesCorrective     int `json:"retries_corrective"`
	RetriesInfrastructure int `json:"retries_infrastructure"`
}

IntentOutcome is the classification of one intent's attempt sequence.

func ClassifyIntents added in v0.64.0

func ClassifyIntents(attempts []Attempt) []IntentOutcome

ClassifyIntents implements the spec's binding Definitions over one unit of work's attempt sequence, and NOTHING else is in scope for it: no cell, no endpoint, no knowledge of whether mcpproxy was in the path. That is the point — the same rule must apply to the baseline arm and the proxy arms, or the comparison is biased toward whichever carries richer error signal.

The rules, verbatim from the Definitions:

  • ATTEMPT: one tool call issued toward a given intent.
  • FIRST-ATTEMPT SUCCESS: the first attempt returned a non-error result AND was not followed by a corrective retry for the same intent. A schema-valid call the agent immediately re-issues differently is NOT a success — hence the arguments comparison rather than a status check.
  • RETRY: a subsequent attempt for the same intent. Transport/infrastructure retries are counted SEPARATELY because they measure the network.

A retry is INFRASTRUCTURE only when the attempt it follows failed for transport reasons AND the arguments are unchanged: that is the shape of a re-send. Change the arguments and the agent has decided the call was wrong, which is a correction no matter what failed first. Anything following a non-transport outcome is corrective — including a re-issue after a successful call, which means the agent was not satisfied by the result.

type Label

type Label struct {
	ToolID    string `json:"tool_id"`
	Relevance int    `json:"relevance"`
}

Label is a graded relevance judgement for one tool against one query, taken from the Spec 065 retrieval golden set (relevance 2 = primary, 1 = related, 0 / absent = irrelevant).

type LapVerdict added in v0.51.0

type LapVerdict struct {
	Executed          bool    `json:"executed"`
	SkipReason        string  `json:"skip_reason,omitempty"`
	Version           string  `json:"version,omitempty"`
	MenuTokens        int     `json:"menu_tokens,omitempty"`
	InHouseMenuTokens int     `json:"in_house_menu_tokens,omitempty"`
	DivergencePct     float64 `json:"divergence_pct,omitempty"`
	Grade             string  `json:"grade,omitempty"`
	ArtifactPath      string  `json:"artifact_path,omitempty"`
}

LapVerdict is the pinned independent LAP run (FR-015/016, SC-006).

func ParseLapJSON added in v0.51.0

func ParseLapJSON(path string) LapVerdict

ParseLapJSON reads a LAP lint artifact from disk and returns the verdict. It never returns an error: a missing, unreadable, corrupt, or menu-token-free artifact yields Executed=false with a SkipReason (SC-006 requires an explicit skip reason over a silent absence).

func (*LapVerdict) Compare added in v0.51.0

func (v *LapVerdict) Compare(inHouseMenuTokens int) (divergencePct float64, warn bool)

Compare records the in-house menu-token count for the same proxy surface on the verdict and computes the divergence of LAP's count from it, in percent (positive = LAP counted more). It returns the divergence and whether it exceeds LapDivergenceTolerancePct (FR-016 warning; non-blocking). When the verdict was not executed or the in-house count is not positive there is nothing to compare: divergence is 0 and no warning is raised.

type LatencyAggregate added in v0.51.0

type LatencyAggregate struct {
	P50Ms float64 `json:"p50_ms"`
	P95Ms float64 `json:"p95_ms"`
	P99Ms float64 `json:"p99_ms"`
	MaxMs float64 `json:"max_ms"`
}

LatencyAggregate is one nearest-rank percentile summary of client-measured latencies (FR-023) for a single measured surface.

type LatencyReport

type LatencyReport struct {
	Samples        int     `json:"samples"`
	P50ms          float64 `json:"p50_ms"`
	P95ms          float64 `json:"p95_ms"`
	P99ms          float64 `json:"p99_ms"`
	MaxMs          float64 `json:"max_ms"`
	LoadAllToolsMs float64 `json:"load_all_tools_ms"`
}

LatencyReport summarizes the REST /api/v1/index/search (BM25 search endpoint) round-trips replayed for retrieval scoring, versus the fixed one-shot cost of loading every tool. Times are client-measured (milliseconds); the server's SearchToolsResponse "took" field is a "0ms" stub. NOTE: this is NOT the MCP retrieve_tools discovery latency — that is aggregated separately in LiveReport.MCPDiscoveryLatency (FR-023).

type LatencyV2 added in v0.51.0

type LatencyV2 struct {
	P50Ms float64 `json:"p50_ms"`
	P95Ms float64 `json:"p95_ms"`
	P99Ms float64 `json:"p99_ms"`
	MaxMs float64 `json:"max_ms"`
	// RESTSearch labels the flat fields' surface explicitly:
	// GET /api/v1/index/search round-trips.
	RESTSearch *LatencyAggregate `json:"rest_search,omitempty"`
	// MCPDiscovery aggregates the MCP retrieve_tools call latencies from the
	// per-query DiscoveryResponseMeasurement rows.
	MCPDiscovery *LatencyAggregate `json:"mcp_discovery,omitempty"`
}

LatencyV2 is the client-measured latency block (FR-023). Two DIFFERENT surfaces are measured and must never be conflated: the REST /api/v1/index/search calls used for retrieval scoring, and the MCP retrieve_tools calls the discovery-response rows time. The flat fields summarize the REST search calls (kept additive for existing consumers and mirrored in RESTSearch); MCPDiscovery, when present, is the retrieve_tools aggregate over the real MCP protocol.

type LiveClient

type LiveClient struct {
	BaseURL string
	APIKey  string
	HTTP    *http.Client
}

LiveClient talks to a running mcpproxy instance (e.g. the bench docker-compose substrate on 127.0.0.1:8092) over its REST API. It is used by the live benchmark run to pull the exact tool definitions (with schemas) and to replay the retrieval golden set through the proxy's BM25 search.

func NewLiveClient

func NewLiveClient(baseURL, apiKey string) *LiveClient

NewLiveClient builds a LiveClient for baseURL (e.g. "http://127.0.0.1:8092") authenticating with apiKey via the X-API-Key header.

func (*LiveClient) FetchUpstreamTools

func (c *LiveClient) FetchUpstreamTools(ctx context.Context) ([]Tool, error)

FetchUpstreamTools pulls the consolidated tool list (GET /api/v1/tools) and returns every upstream tool with its full JSON input schema, ready to feed into schema-aware token counting for the baseline.

func (*LiveClient) Search

func (c *LiveClient) Search(ctx context.Context, query string, limit int) (ranked []string, latency time.Duration, err error)

Search replays one query through the proxy's BM25 tool search (GET /api/v1/index/search) and returns the ranked tool IDs (server:tool, best first) plus the client-measured round-trip latency.

Latency is measured client-side on purpose: the server's SearchToolsResponse "took" field is currently a hardcoded "0ms" stub (internal/httpapi handleSearchTools), so it cannot be trusted as the proxy-side timing.

type LiveFleetSource added in v0.64.0

type LiveFleetSource struct {
	// contains filtered or unexported fields
}

LiveFleetSource reads a fleet from a running proxy's direct MCP surface.

It holds the run's context rather than taking one per call, so FleetSource stays a two-method seam a test can satisfy with a literal. The value is a one-shot adapter constructed at the call site for a single run, which is the case where a stored context is honest rather than a leak.

func NewLiveFleetSource added in v0.64.0

func NewLiveFleetSource(ctx context.Context, baseURL, apiKey string) *LiveFleetSource

NewLiveFleetSource builds a fleet source against a running proxy's base URL (e.g. "http://127.0.0.1:8080"). Endpoint resolution failures are deferred to Fleet(), so a mis-typed URL fails where every other fleet failure does.

func (*LiveFleetSource) Fleet added in v0.64.0

func (s *LiveFleetSource) Fleet() (*Corpus, error)

Fleet implements FleetSource: one MCP session, one paginated tools/list, and the stub guard. A failure here is fatal to the run by design — falling back to a frozen corpus would answer a different question than the one asked.

func (*LiveFleetSource) FleetID added in v0.64.0

func (s *LiveFleetSource) FleetID() string

FleetID implements FleetSource.

type LiveModeResult

type LiveModeResult struct {
	Mode         string  `json:"mode"`
	ContextTools int     `json:"context_tools"`
	Tokens       int     `json:"tokens"`
	SavingsRatio float64 `json:"savings_vs_baseline,omitempty"`
}

LiveModeResult is the per-mode context-token cost from the live run.

type LiveReport

type LiveReport struct {
	Proxy     string            `json:"proxy"`
	Encoding  string            `json:"encoding"`
	Tokens    *LiveTokenReport  `json:"tokens"`
	Retrieval *RetrievalMetrics `json:"retrieval"`
	Latency   *LatencyReport    `json:"latency"`

	// ProxyInfo records the measured proxy's identity and discovery
	// configuration (FR-021): version, tool count, tools_limit, routing_mode.
	// Collected best-effort from /api/v1/info, /api/v1/config, /api/v1/status;
	// fields the proxy does not expose stay zero-valued.
	ProxyInfo *ProxyInfo `json:"proxy_info,omitempty"`
	// ResponseCost is the per-golden-query retrieve_tools response-cost
	// summary over the real MCP protocol (FR-001). nil when the measurement
	// was skipped — see ResponseCostNote.
	ResponseCost *ResponseCostSummary `json:"response_cost,omitempty"`
	// BreakEven is the FR-003 analysis over the SAME token report as the
	// headline (naive full menu vs proxy menu, both schema-counted). nil when
	// responses were not measured or the menu counts are non-authoritative.
	BreakEven *BreakEvenAnalysis `json:"break_even,omitempty"`
	// SessionEstimates are the FR-019 estimator rows for the measured live
	// encoding (baseline_json — the proxy's own full-JSON responses).
	SessionEstimates []SessionCostRow `json:"session_estimates,omitempty"`
	// MCPDiscoveryLatency aggregates the MCP retrieve_tools call latencies
	// from the per-query response-cost rows (FR-023). It is a DIFFERENT
	// surface than Latency, which times REST /api/v1/index/search calls —
	// the two are never conflated. nil when responses were not measured.
	MCPDiscoveryLatency *LatencyAggregate `json:"mcp_discovery_latency,omitempty"`
	// ResponseCostNote explains a skipped or degraded response-cost /
	// break-even measurement (never silent, SC-006 spirit). Empty on success.
	ResponseCostNote string `json:"response_cost_note,omitempty"`
	// FlipGates (Spec 085 FR-017/FR-018) is populated only when the live run
	// was invoked with the compact arm enabled (-flip-gates).
	FlipGates *FlipGateReport `json:"flip_gates,omitempty"`
}

LiveReport is the full live benchmark result: exact-token comparison, retrieval accuracy, and search latency, all gathered from one running proxy — plus, since Spec 083 (T016), the retrieve_tools RESPONSE cost measured over the real MCP protocol (per-golden-query rows with span-based component attribution, FR-001/002), the break-even analysis (FR-003/004), the FR-021 proxy identity fields, and the session-cost estimate rows for the measured live encoding. FlipGates (Spec 085 FR-017/FR-018) is populated only when the live run was invoked with the compact arm enabled (-flip-gates).

func RunLive

func RunLive(ctx context.Context, client *LiveClient, golden *GoldenSet, opts LiveRunOptions) (*LiveReport, error)

RunLive gathers the full live benchmark from a running proxy: it pulls the exact tool definitions (with schemas) for the token comparison, replays the golden set through the proxy's BM25 search for accuracy, and records the per-query search latency.

func (*LiveReport) ToReportV2 added in v0.51.0

func (r *LiveReport) ToReportV2(generatedAt string) *ReportV2

ToReportV2 projects a live run into the versioned v2 report envelope (research D12): the live proxy toolset becomes the single corpus row, arm rows stay empty (arms are offline-computable, not live-measured), and every headline number carries its provenance label (SC-005).

func (*LiveReport) WriteJSON

func (r *LiveReport) WriteJSON(dir string) (string, error)

WriteJSON writes the live report as indented JSON into dir/live_report.json (the dir is gitignored — reports are never committed, per Spec 065 CN-003).

type LiveRunOptions added in v0.51.0

type LiveRunOptions struct {
	// CorpusV2Path, when non-empty, is the Spec 083 schema-bearing frozen
	// corpus used as the full-definition schema SOURCE for the naive
	// full-menu count: live tools are joined by tool id with the corpus_v2
	// schemas. This exists because GET /api/v1/tools can serve stub schemas
	// ({"type":"object","properties":{}} — see scripts/gen-corpus-v2-dump),
	// which would silently hollow out the FR-004 baseline. Live tools missing
	// from the corpus fall back to name+description and the headline is
	// withheld (the MCP-3161/3167 guard pattern).
	CorpusV2Path string
	// ExpectedToolCount, when > 0, is the tool count the frozen corpus
	// documents; it is recorded in ProxyInfo.ExpectedToolCount and drives the
	// corpus-drift warning when the live catalog differs (FR-021).
	ExpectedToolCount int
}

LiveRunOptions are the optional knobs of a live benchmark run.

type LiveTokenReport

type LiveTokenReport struct {
	Encoding               string           `json:"encoding"`
	UpstreamTools          int              `json:"upstream_tools"`
	BaselineTokens         int              `json:"baseline_tokens"`
	Modes                  []LiveModeResult `json:"modes"`
	ProxySchemasCounted    bool             `json:"proxy_schemas_counted"`
	BaselineSchemasCounted bool             `json:"baseline_schemas_counted"`
	AuthoritativeHeadline  bool             `json:"authoritative_headline"`
	// SchemaSource records where the baseline upstream schemas came from
	// (provenance, SC-005): the live GET /api/v1/tools response, or a
	// corpus_v2 join when the live endpoint serves stub schemas (FR-004).
	SchemaSource string   `json:"schema_source,omitempty"`
	Notes        []string `json:"notes"`
}

LiveTokenReport is the exact-token comparison from a live proxy, with the baseline upstream tools counted WITH their full JSON input schemas.

AuthoritativeHeadline gates the savings percentage: it is only true when schemas were counted on BOTH sides — the proxy management tools carry schemas (ProxySchemasCounted) AND the baseline upstream tools carry schemas (BaselineSchemasCounted). Counting schemas on one side only overstates or distorts savings — the exact error corrected in MCP-3161 — so when either side is schema-less the savings ratio is withheld and only raw token totals are reported. BaselineSchemasCounted also guards against a /api/v1/tools response that silently dropped upstream schemas (MCP-3167).

type LossySignatureGate added in v0.51.0

type LossySignatureGate struct {
	CorpusVersion string              `json:"corpus_version"`
	Tools         int                 `json:"tools"`
	LossyTools    int                 `json:"lossy_tools"`
	Rate          float64             `json:"rate"`
	Pass          bool                `json:"pass"`
	Heaviest      []HeaviestSignature `json:"heaviest,omitempty"`
}

LossySignatureGate is the corpus lossy-rate result (gate: <20%, SC-005).

func ComputeLossyGate added in v0.51.0

func ComputeLossyGate(tk *Tokenizer, corpusVersion string, tools []Tool, topN int) *LossySignatureGate

ComputeLossyGate renders every corpus tool through the SHARED internal/toolsig grammar (FR-019) and reports the lossy-signature rate (gate: <20%) plus the topN heaviest signatures by token cost. A tool with no schema is a parameter-less tool (renders "()", never lossy) — matching the live normalizeSchema semantics — not an unparseable one.

type MCPCaller added in v0.51.0

type MCPCaller struct {
	// contains filtered or unexported fields
}

MCPCaller is one initialized MCP session against a running proxy. The live run reuses a single session for every golden query (one initialize, N retrieve_tools calls) so per-query latency measures the call, not session setup.

func NewMCPCaller added in v0.51.0

func NewMCPCaller(ctx context.Context, baseURL, apiKey string) (*MCPCaller, error)

NewMCPCaller connects to the proxy's /mcp streamable-http endpoint and performs the MCP initialize handshake. Callers must Close() the session.

func (*MCPCaller) Close added in v0.51.0

func (c *MCPCaller) Close() error

Close terminates the MCP session.

func (*MCPCaller) RetrieveTools added in v0.51.0

func (c *MCPCaller) RetrieveTools(ctx context.Context, query string, limit int) (string, time.Duration, error)

RetrieveTools calls the proxy's retrieve_tools tool with query and returns the raw text content of the result — the exact JSON string marshaled by internal/server/mcp.go handleRetrieveToolsWithMode — plus the client-side round-trip latency of the call itself (FR-023). limit <= 0 omits the limit argument so the proxy's configured tools_limit default applies, matching what a real agent pays per call.

type MCPMarkMeta added in v0.64.0

type MCPMarkMeta struct {
	TaskName        string `json:"task_name"`
	ExecutionResult struct {
		Success *bool `json:"success"`
	} `json:"execution_result"`
	TokenUsage *MCPMarkTokenUsage `json:"token_usage"`
	TurnCount  *int               `json:"turn_count"`

	// SourcePath is the file this was read from, kept so a report can
	// reference the raw per-run record it was computed from (FR-029).
	SourcePath string `json:"-"`
}

MCPMarkMeta is the subset of the suite's per-task meta.json this driver reads: the pass verdict, the token usage, and the turn count.

execution_result.success is a POINTER because its absence is NOT a failure. TASK COMPLETION is the suite's own verdict; where the suite emits none, the unit has NO COMPLETION SIGNAL and is excluded from completion-dependent figures. Decoding a missing bool as false would quietly convert "we do not know" into "it failed", which is an inference the Definitions forbid.

func LoadMCPMarkMeta added in v0.64.0

func LoadMCPMarkMeta(path string) (*MCPMarkMeta, error)

LoadMCPMarkMeta reads one per-task meta.json.

A missing or malformed file is a REPORTED ERROR, never a zero-valued record. The distinction matters more than it looks: a run over hundreds of tasks where a handful of meta files failed to write would, under a zero-filling loader, publish a cheaper-and-less-successful-looking number with no trace of why — and nothing downstream could recover the difference between "this task cost nothing" and "we could not read what it cost".

func (*MCPMarkMeta) Completion added in v0.64.0

func (m *MCPMarkMeta) Completion() string

Completion maps the suite's verdict onto the report vocabulary. An absent success field is no-signal, never a failure.

func (*MCPMarkMeta) UnitRecord added in v0.64.0

func (m *MCPMarkMeta) UnitRecord(cellID, unitID string, runIndex int) (*UnitRecord, error)

UnitRecord projects one meta.json onto a UnitRecord.

The resulting usage has NO cache-read count, and that is not an oversight to be tidied away with a zero: the suite's output has no such field at all (research.md §5). FR-014's third axis has to come from the driver's own capture of provider responses, and a record built from meta.json alone carries the gap honestly so the report can say so.

type MCPMarkTokenUsage added in v0.64.0

type MCPMarkTokenUsage struct {
	Input        *int `json:"input"`
	Output       *int `json:"output"`
	Total        *int `json:"total"`
	Reasoning    *int `json:"reasoning"`
	InputAlt     *int `json:"input_tokens"`
	OutputAlt    *int `json:"output_tokens"`
	TotalAlt     *int `json:"total_tokens"`
	ReasoningAlt *int `json:"reasoning_tokens"`
}

MCPMarkTokenUsage mirrors the pinned suite's per-task `token_usage` object.

Every field is a POINTER so an absent key is distinguishable from a reported zero — the same reason ReplayCellCost.ResponseTokens is a pointer. A missing count silently read as 0 would understate cost, and understating cost in this project's favour is the failure class FR-002 forbids for truncation and that this file must not reintroduce on a new axis.

Both key spellings are accepted. The suite reports input/output/total/ reasoning; the `_tokens`-suffixed spelling is common in provider payloads that the suite passes through, and refusing to read a file we can obviously read would trade a real measurement for a stylistic point. Which spelling the pinned SHA actually emits is recorded in bench/README.md alongside the pin, and is re-checked whenever the pin moves.

type MCPRetrieveCaller added in v0.51.0

type MCPRetrieveCaller struct {
	// contains filtered or unexported fields
}

MCPRetrieveCaller is the live compact arm's transport (Spec 085 FR-017): it drives a running proxy's MCP endpoint and replays retrieve_tools calls with a per-call `detail` override, so full and compact responses are measured through the SAME pipeline against the same live index. The serialized response text it returns is exactly what an agent would receive — that is what gets tokenized.

Spec 103 gave it a second job — ToolsListPage, the raw tools/list fetch the live fleet source is built on — because the handshake is the same one and a second connect+initialize would be a second thing to keep correct. The two jobs share the session; only ToolsListPage needs the transport handle.

func NewMCPRetrieveCaller added in v0.51.0

func NewMCPRetrieveCaller(ctx context.Context, mcpURL string) (*MCPRetrieveCaller, error)

NewMCPRetrieveCaller connects and initializes an MCP client against mcpURL (e.g. "http://127.0.0.1:8092/mcp"). Callers must Close it.

func (*MCPRetrieveCaller) Close added in v0.51.0

func (m *MCPRetrieveCaller) Close() error

Close shuts the underlying MCP client down.

func (*MCPRetrieveCaller) RetrieveFunc added in v0.51.0

func (m *MCPRetrieveCaller) RetrieveFunc(ctx context.Context) RetrieveToolsFunc

RetrieveFunc adapts the caller to the RetrieveToolsFunc signature RunFlipGates consumes.

func (*MCPRetrieveCaller) ToolsListPage added in v0.64.0

func (m *MCPRetrieveCaller) ToolsListPage(ctx context.Context, cursor string) (json.RawMessage, error)

ToolsListPage implements ToolsListPageFunc (mcptools.go): one tools/list round trip on the live session, returning the RAW JSON-RPC result.

It goes through the transport rather than client.ListTools deliberately. ListTools returns a decoded *mcp.ListToolsResult, and mcp-go's ToolArgumentsSchema keeps only {$defs, type, properties, required, additionalProperties} while its marshaller always re-emits "properties" and "required" — so re-serializing a decoded result rewrites the Spec 102 deferred placeholder {"type":"object"} into {"properties":{},"required":[],"type":"object"} and makes it indistinguish- able from a genuinely parameter-less tool. The stub guard exists precisely to tell those apart, so the raw bytes are load-bearing, not an optimization.

Pagination is NOT handled here: this is one page, and the cursor loop lives in FetchToolsList where it is testable without a proxy.

type MeasuredArmOutcome added in v0.64.0

type MeasuredArmOutcome struct {
	// RetryRate is the measured mean retries per call, on the same scale as
	// the armRetryRates defaults it supersedes.
	RetryRate float64
	// FirstAttemptSuccessPct is the measured first-attempt success rate
	// (FR-010), carried because FR-013 covers success figures as well as
	// retries.
	FirstAttemptSuccessPct float64
	// Runs is the number of completed runs behind the two figures above. A
	// zero here means no run produced them, which HasMeasurement treats as an
	// absence rather than a measurement of zero — the same
	// indistinguishable-zero hazard one level up. FR-021's four-run bar is a
	// PUBLICATION gate applied by the report layer, not a gate on
	// superseding: a two-run measurement is still better evidence than a
	// literature constant, and the row publishes Runs so the reader can apply
	// FR-021 themselves.
	Runs int
}

MeasuredArmOutcome is what a live agent loop observed for ONE arm (or mode cell): its retry rate, its first-attempt success rate, and how many runs stand behind them. Runs is not decoration — see HasMeasurement.

func (MeasuredArmOutcome) HasMeasurement added in v0.64.0

func (m MeasuredArmOutcome) HasMeasurement() bool

HasMeasurement reports whether this outcome is backed by at least one run. An outcome that reached the map without a run behind it (an arm the loop was configured for but never executed) is an absence, and must not supersede a documented default.

type MeasuredOutcomeSource added in v0.64.0

type MeasuredOutcomeSource interface {
	MeasuredOutcomeForArm(arm string) (MeasuredArmOutcome, bool)
}

MeasuredOutcomeSource supplies measured outcomes per arm. It is a structural interface on purpose: the live agent loop that produces these figures is a separate concern with its own accounting source, and the estimator must stay unit-testable with a fake and no suite running (the bench/flipgate.go and bench/armrun.go precedent). A nil source means "nothing was measured", which is the ordinary offline case.

type MeasuredOutcomes added in v0.64.0

type MeasuredOutcomes map[string]MeasuredArmOutcome

MeasuredOutcomes is the trivial map-backed MeasuredOutcomeSource, used by callers that already hold the figures and by tests.

func (MeasuredOutcomes) MeasuredOutcomeForArm added in v0.64.0

func (m MeasuredOutcomes) MeasuredOutcomeForArm(arm string) (MeasuredArmOutcome, bool)

MeasuredOutcomeForArm implements MeasuredOutcomeSource. It reports ok only for an entry that is actually backed by runs, so a zero-value entry cannot masquerade as a measured zero.

type ModeCell added in v0.64.0

type ModeCell struct {
	// ID is the stable report key (FR-028).
	ID string `json:"id"`
	// RoutingMode is the production routing-mode identifier
	// (config.RoutingMode*). Empty for the baseline, which is served by no
	// mcpproxy surface.
	RoutingMode string `json:"routing_mode,omitempty"`
	// Endpoint is the mounted path this cell is measured through. Empty for
	// the baseline and for skip rows.
	Endpoint string `json:"endpoint,omitempty"`
	// DiscoverySerialization is full / compact / not-applicable.
	DiscoverySerialization string `json:"discovery_serialization"`
	// DirectSerialization is full / deferred / not-applicable.
	DirectSerialization string `json:"direct_serialization"`
	// Capabilities lists the FR-016 binary conditions available on this cell.
	Capabilities []string `json:"capabilities,omitempty"`
	// Skipped marks the configurable-but-redundant and degenerate rows.
	Skipped bool `json:"skipped"`
	// SkipReasonCodes holds one or more of the reason codes above. It is a
	// slice because one combination genuinely carries two:
	// code_execution x compact x deferred is forced-full AND axis-ignored.
	SkipReasonCodes []string `json:"skip_reason_codes,omitempty"`
	// CollapsesOnto names the valid cell a redundant combination runs
	// identically to. Empty for valid cells and for the degenerate row, which
	// collapses onto nothing.
	CollapsesOnto string `json:"collapses_onto,omitempty"`
}

ModeCell is one point in the matrix: either one of the 5 distinct behaviours, the baseline denominator, or a skip row standing in for a configurable-but-redundant combination.

A cell is (URL, serialization-config), NOT a URL alone. The routing axis is chosen by endpoint; the two serialization axes are config. Both of those hot-reload — tool_response_mode is read from the live snapshot on every call and direct_tool_response_mode is rebuilt on config.reloaded — so the whole matrix crosses on ONE long-lived proxy instance, with a config apply between serialization cells rather than a restart.

func AllCells added in v0.64.0

func AllCells() []ModeCell

AllCells returns the baseline followed by the 5 mcpproxy cells — the full row set a comparison report renders, denominator first.

func BaselineCell added in v0.64.0

func BaselineCell() ModeCell

BaselineCell returns the FR-020 denominator cell.

func CellByID added in v0.64.0

func CellByID(id string) (ModeCell, bool)

CellByID looks up a valid cell (the baseline included) by its stable id. Skip rows are not addressable this way: they are outcomes of a configuration, not cells of the matrix.

func ModeCells added in v0.64.0

func ModeCells() []ModeCell

ModeCells returns the 5 distinct mcpproxy behaviours in contract order.

func RedundantCombinations added in v0.64.0

func RedundantCombinations() []ModeCell

RedundantCombinations returns the 7 skip rows of the collapse table, in contract order. They exist so the report can show that these configurations were considered and understood — reported as skipped with a reason, never as a zero and never as a missing row (FR-017).

func ResolveCombination added in v0.64.0

func ResolveCombination(comb Combination) ModeCell

ResolveCombination maps a configured combination to the behaviour that would actually run: either one of the 5 distinct cells, or a skip row naming the cell it collapses onto (or, for the degenerate configuration, naming nothing, because there is nothing equivalent to collapse onto).

An unrecognized combination — an axis value that is not a production constant — also yields a skip row rather than an error or a silent default, on the same principle: the matrix never invents a measurement it did not take.

func (ModeCell) ArmName added in v0.64.0

func (c ModeCell) ArmName() string

ArmName names the existing encoding arm that renders this cell's tool menu, for resolution through the bench/arms registry by a caller that may import it (package bench cannot: arms imports bench). Returning a NAME rather than an arm is what keeps every serialization in the one place it is already implemented instead of being re-derived here.

func (ModeCell) EndpointURL added in v0.64.0

func (c ModeCell) EndpointURL(baseURL string) (string, error)

EndpointURL resolves this cell's mounted MCP endpoint against a base URL (T015). The routing-mode axis needs no config change and no restart — all three routing-mode servers are built at startup and all three endpoints stay mounted regardless of config — so moving between routing modes is nothing more than changing this URL.

The two serialization cells that share an endpoint are NOT an error: a cell is (URL, serialization-config), and the serialization half is applied by a hot config reload on the same long-lived instance.

The returned URL is what NewMCPRetrieveCaller (mcpcaller.go) takes, so the transport is composed rather than rebuilt per cell.

func (ModeCell) ProxyTools added in v0.64.0

func (c ModeCell) ProxyTools() []Tool

ProxyTools returns the built-in mcpproxy tool definitions that occupy the agent's context window in this cell's routing mode, delegating to ProxyToolsForMode so the catalog stays derived from the production tool builders and can never drift from what the server really registers.

The direct routing mode returns none: its context cost is the rendered upstream catalog itself, which is measured by the cell's encoding arm rather than by a fixed built-in list.

func (ModeCell) SkipReasonText added in v0.64.0

func (c ModeCell) SkipReasonText() string

SkipReasonText renders a skip row's reason as the single string the existing report shape carries, naming both the code(s) and the cell collapsed onto. Contract rule 4 requires a skipped row to carry a reason AND its collapse target; the shape it must fit into (ArmResult.SkipReason) is one string, so they are joined here rather than by inventing a second skip shape.

func (ModeCell) SkipRow added in v0.64.0

func (c ModeCell) SkipRow(corpusID string) ArmResult

SkipRow renders this cell as the report's EXISTING skipped-row shape (ArmResult.Skipped / SkipReason via SkippedArmResult). Reusing that constructor is deliberate: a second skip shape would give consumers two places to look for "why is this row empty", and the whole point of FR-017 is that the answer is always in exactly one place.

type ModeResult

type ModeResult struct {
	Mode         string  `json:"mode"`
	ContextTools int     `json:"context_tools"`
	Tokens       int     `json:"tokens"`
	SavingsRatio float64 `json:"savings_vs_baseline"`
}

ModeResult is the per-mode context-cost outcome.

type OfflineSection added in v0.51.0

type OfflineSection struct {
	// Corpus is the frozen tool corpus to run Arms over; nil skips the arm
	// runner and keeps only Descriptor + SkippedArms + ExtraArmRows.
	Corpus *Corpus
	// Descriptor identifies the corpus in the report (FR-011/012/013).
	// ToolCount defaults to len(Corpus.Tools) and DegenerateDescriptions is
	// computed with the FR-020 default rules when unset.
	Descriptor CorpusDescriptor
	// Golden enables retrieval-quality scoring for index-altering arms; nil
	// means the corpus has no relevance labels (FR-011 explicit absence).
	Golden *GoldenSet
	// Arms are the resolved encoding arms to measure (must include
	// baseline_json when non-empty — the savings denominator).
	Arms []EncodingArm
	// SkippedArms are arm-level skip rows for arms whose runtime is absent
	// (contract rule 5), recorded with the section's corpus ID.
	SkippedArms []ArmResult
	// ExtraArmRows are pre-computed rows appended verbatim (e.g. the
	// toon_results fixture rows, which do not run over a tool corpus).
	ExtraArmRows []ArmResult
}

OfflineSection is one corpus measured by a set of encoding arms (or, with a nil Corpus, a container for pre-computed rows such as the fixture-driven toon_results measurement).

type PayloadDecomposition added in v0.64.0

type PayloadDecomposition struct {
	FleetID     string `json:"fleet_id"`
	ToolCount   int    `json:"tool_count"`
	TotalTokens int    `json:"total_tokens"`
	// ToolsWithSchema is how many of ToolCount actually carry an input schema.
	// Load-bearing: a corpus with none cannot measure a schema share at all,
	// and a 0% result there means "nothing to defer was present", not "schemas
	// are cheap". Reporting the second from the first is the failure this
	// field exists to make impossible.
	ToolsWithSchema int `json:"tools_with_schema"`

	NameTokens        int `json:"name_tokens"`
	DescriptionTokens int `json:"description_tokens"`
	SchemaTokens      int `json:"schema_tokens"`
	StructuralTokens  int `json:"structural_tokens"`

	NamePct        float64 `json:"name_pct"`
	DescriptionPct float64 `json:"description_pct"`
	SchemaPct      float64 `json:"schema_pct"`
	StructuralPct  float64 `json:"structural_pct"`

	// SchemaCeilingPct is the most any schema-deferring design could ever save
	// on THIS corpus: the schema share, since deleting the schema entirely is
	// the upper bound of deferring it. Recomputed per corpus — carrying a
	// ceiling forward is the error spec 102 made.
	SchemaCeilingPct float64 `json:"schema_ceiling_pct"`

	// Spec102Verdict is an explicit confirmed/corrected statement rather than
	// a number the reader has to interpret.
	Spec102Verdict string `json:"spec_102_verdict"`

	// Limitation names what this corpus shape cannot settle. Always populated:
	// a decomposition with no stated limits is claiming there are none.
	Limitation string `json:"limitation"`

	Provenance string `json:"provenance"`
}

PayloadDecomposition attributes one corpus's rendered payload to its components, and states what that corpus shape can and cannot settle.

func DecomposePayload added in v0.64.0

func DecomposePayload(tk *Tokenizer, corpus *Corpus, fleetID string) (*PayloadDecomposition, error)

DecomposePayload attributes the canonical rendering of every tool in the corpus to name, description, schema and structural overhead.

The rendering matches the baseline arm's canonical form — name + "\n" + description + "\n" + canonical(schema) — because that is what the savings figures are measured against. Decomposing a different rendering would produce shares that do not explain those figures.

type PayloadDecompositionBlock added in v0.64.0

type PayloadDecompositionBlock struct {
	AccountingSource AccountingSource          `json:"accounting_source"`
	Shapes           []PayloadDecompositionRow `json:"shapes,omitempty"`
	Inputs           *InputProvenance          `json:"inputs,omitempty"`
}

PayloadDecompositionBlock holds one row per fleet shape (at least two, per FR-024) so the reader can see how the attribution moves with fleet size rather than trusting a single-corpus projection.

func PayloadDecompositionBlockFor added in v0.64.0

func PayloadDecompositionBlockFor(ds []*PayloadDecomposition) (*PayloadDecompositionBlock, error)

PayloadDecompositionBlockFor builds the report block from one decomposition per fleet shape (T053).

FR-024 asks for at least two shapes, because the whole point is to show how the attribution MOVES with fleet size rather than to publish a single-corpus projection — which is the shape of mistake spec 102 made. Fewer than two is therefore an error rather than a smaller block.

type PayloadDecompositionRow added in v0.64.0

type PayloadDecompositionRow struct {
	FleetShape FleetShape `json:"fleet_shape"`
	Provenance string     `json:"provenance"`
	// The four attributions, as percentages of the definition payload.
	ShareNamesPct        float64 `json:"share_names_pct"`
	ShareDescriptionsPct float64 `json:"share_descriptions_pct"`
	// ShareAnnotationsPct is a POINTER because null and zero mean different
	// things here. The frozen corpora carry no annotations field, so a
	// corpus-based decomposition cannot measure this share at all — and a 0
	// would read as "measured, and annotations cost nothing", which is the
	// silent-zero failure the rest of this report works to avoid. nil marshals
	// to null: not measured.
	ShareAnnotationsPct *float64 `json:"share_annotations_pct"`
	ShareSchemasPct     float64  `json:"share_schemas_pct"`
	// AchievableCeilingPct is recomputed for THIS shape. Carrying a
	// previously published ceiling forward is precisely the error spec 102
	// made when it projected from a single corpus.
	AchievableCeilingPct float64 `json:"achievable_ceiling_pct"`
	// Spec102Verdict is an explicit confirmed/corrected call on spec 102's
	// conclusion, with the delta when corrected.
	Spec102Verdict  string   `json:"spec102_verdict"`
	Spec102DeltaPct *float64 `json:"spec102_delta_pct,omitempty"`
}

PayloadDecompositionRow attributes tool-definition cost for ONE fleet shape (FR-024/025).

type ProviderUsage added in v0.64.0

type ProviderUsage struct {
	InputTokens     int  `json:"input_tokens"`
	OutputTokens    int  `json:"output_tokens"`
	CacheReadTokens *int `json:"cache_read_tokens"`
	// ReasoningTokens is recorded because the suite reports it, and is
	// deliberately NOT added into the total: providers report it as a
	// component of output, so adding it would double-count.
	ReasoningTokens int `json:"reasoning_tokens,omitempty"`
	// Responses counts the provider responses this usage was summed from, so
	// a partially-captured unit is distinguishable from a genuinely cheap one
	// (data-model.md RunRecord rule).
	Responses int `json:"responses,omitempty"`
}

ProviderUsage is provider-reported consumption for one unit of work, with the FR-014 axes kept apart.

CacheReadTokens is a POINTER on purpose. The pinned suite's own per-task output records input / output / total / reasoning and has NO cache-read field, so a suite-derived record genuinely cannot know it — and a 0 there would read as "measured, and nothing was served from cache", which in a long agent loop is the difference between an honest total and one understated in this project's favour. nil means unavailable and is reported as such.

func (ProviderUsage) Total added in v0.64.0

func (u ProviderUsage) Total() (total int, complete bool)

Total is input + output + cache-read, and reports whether every axis was available. A caller that ignores the second return value and publishes the first has published an understated cost.

type ProxyInfo added in v0.51.0

type ProxyInfo struct {
	Version           string `json:"version,omitempty"`
	ToolCount         int    `json:"tool_count,omitempty"`
	ExpectedToolCount int    `json:"expected_tool_count,omitempty"`
	ToolsLimit        int    `json:"tools_limit,omitempty"`
	RoutingMode       string `json:"routing_mode,omitempty"`
}

ProxyInfo records the measured proxy's identity and discovery configuration (FR-021): interpreting response cost requires knowing tools_limit and routing_mode; tool_count vs expected_tool_count surfaces corpus drift.

type PublicationDecision added in v0.64.0

type PublicationDecision struct {
	Publishable bool     `json:"publishable"`
	Blockers    []string `json:"blockers,omitempty"`
	Caveats     []string `json:"caveats,omitempty"`
}

PublicationDecision is the answer to "may this report back a published claim?" — a blocker list and a caveat list, both prose meant for the human about to publish.

The two lists are not severity levels of one thing. A BLOCKER says the document cannot honestly support a published claim at all. A CAVEAT says it can, but something must travel with it: notably that a private-recording figure stays marked as not independently reproducible even in a report that publishes perfectly well. A caveat that quietly disappears once the blocker is resolved would defeat the mark entirely, so publication keeps them.

The decision is derived on demand and deliberately NOT stored in the report. A stored verdict is a verdict that can go stale against the blocks beside it, and a stale "publishable: true" is the single most dangerous field this document could carry.

type QueryModeMismatch added in v0.51.0

type QueryModeMismatch struct {
	QueryID    string   `json:"query_id"`
	Query      string   `json:"query"`
	FullIDs    []string `json:"full_ids"`
	CompactIDs []string `json:"compact_ids"`
}

QueryModeMismatch records one golden query whose ranked ids differ between modes — any entry here fails the SC-002 gate.

type RankedIdentityGate added in v0.51.0

type RankedIdentityGate struct {
	Queries    int                 `json:"queries"`
	Identical  int                 `json:"identical"`
	Pass       bool                `json:"pass"`
	Mismatches []QueryModeMismatch `json:"mismatches,omitempty"`
}

RankedIdentityGate is the per-query ranked-ID identity result (gate: 100%).

type RawRecordsRef added in v0.64.0

type RawRecordsRef struct {
	Retention string `json:"retention"`
	// RunLocalPath is relative to the report's own directory, and empty
	// whenever Retention is not_retained — a not-retained reference that
	// still carries a path IS the dangling reference this type replaces.
	RunLocalPath string `json:"run_local_path,omitempty"`
	Records      int    `json:"records,omitempty"`
	Durable      bool   `json:"durable"`
	// Note carries RecordsNotDurableNote on a retained reference, or the
	// reason nothing is retained on a degraded one.
	Note string `json:"note"`
}

RawRecordsRef is the report-side reference to a block's raw per-run records (T057/FR-029), so a headline figure can be traced back to its inputs without embedding any recorded content in the report (FR-006).

The path is RUN-LOCAL — relative to the directory the report itself was written into — for two reasons. An absolute path would carry the operator's filesystem layout into a document that may be published, and it would dangle on every other machine, which is worse than saying nothing. A relative path resolves for anyone holding the run directory and resolves nowhere else, which is exactly the scope of the claim.

Durable is always false and Validate enforces it. Like InputProvenance.IndependentlyReproducible, the redundancy with the note is deliberate: a renderer keys off the boolean, a human reads the note, and neither has to infer the other.

func NotRetainedRecords added in v0.64.0

func NotRetainedRecords(reason string) *RawRecordsRef

NotRetainedRecords is the degraded reference: no artifact, a stated reason, and deliberately no path. FR-029 wants a traceable figure; when the trace does not exist, the report says the trace does not exist.

func RetainedRecords added in v0.64.0

func RetainedRecords(runLocalPath string, records int) *RawRecordsRef

RetainedRecords references a per-run artifact written into this run's own output directory. runLocalPath must be relative to that directory.

func (*RawRecordsRef) Validate added in v0.64.0

func (r *RawRecordsRef) Validate(block string) error

Validate enforces the run-local, non-durable, never-dangling shape.

type ReplayBlock added in v0.64.0

type ReplayBlock struct {
	AccountingSource AccountingSource `json:"accounting_source"`
	// Counterfactual is the mandatory FR-004 label. It is a required field
	// rather than a rendering concern so a figure cannot be emitted without
	// it and then lose the caveat downstream.
	Counterfactual string `json:"counterfactual"`
	// BodiesIncluded records whether the export carried response bodies.
	// False is the default and the safe mode; true is an explicit opt-in
	// over raw unmasked traffic (the export path does not mask).
	BodiesIncluded bool       `json:"bodies_included"`
	FleetShape     FleetShape `json:"fleet_shape"`
	// SessionsSupplied/SessionsUsed plus Exclusions are the FR-003 accounting:
	// supplied minus used must be explained by the exclusion counts.
	SessionsSupplied int               `json:"sessions_supplied"`
	SessionsUsed     int               `json:"sessions_used"`
	Exclusions       []ReplayExclusion `json:"exclusions,omitempty"`
	// LoaderAccounting is the loader's own exclusion detail: what was dropped
	// before joining a unit of work, what had its cost withheld INSIDE an
	// admitted unit, what was admitted but flagged, and how many sub-calls
	// lost their parent. The session rows above cannot express any of that,
	// and a withheld cost that collapses a response total with no row saying
	// why is precisely the silent accounting SC-008 forbids.
	LoaderAccounting *ReplayLoaderAccounting `json:"loader_accounting,omitempty"`
	Cells            []ReplayCellCost        `json:"cells,omitempty"`
	// DirectDelta is absent when only one direct cell was measured.
	DirectDelta *ReplayDirectDelta `json:"direct_delta,omitempty"`
	// SensitiveFlagBestEffort restates, in the report itself, that the
	// sensitive-data flag is derived from detection metadata written
	// asynchronously AFTER persistence: its absence does not prove a record
	// is clean. It is a reducer, never a guarantee (Principle IV).
	SensitiveFlagBestEffort bool             `json:"sensitive_flag_best_effort,omitempty"`
	Inputs                  *InputProvenance `json:"inputs,omitempty"`
	Records                 *RawRecordsRef   `json:"records,omitempty"`
}

ReplayBlock is the deterministic recomputation over a real recorded workload (US1). It is a COUNTERFACTUAL: recorded call shape scored against the SUPPLIED fleet, not the fleet as it stood when the session was recorded, and never observed agent behaviour (FR-004). No recorded content reaches this struct — only counts, sizes and derived measurements (FR-006).

func RunReplay added in v0.64.0

func RunReplay(tk *Tokenizer, opts ReplayOptions) (*ReplayBlock, error)

RunReplay loads a recording, prices every mode cell's menu against the supplied fleet, and returns the report block.

The fleet is resolved FIRST, before the recording is even opened: refusing early is what makes the refusal a rule about the invocation rather than an accident of whichever file happened to be readable. Resolution also covers the live source (ResolveFleet), so a live fleet's stub guard fires before any raw recorded traffic is read off disk.

func (*ReplayBlock) ValidateCounterfactual added in v0.64.0

func (b *ReplayBlock) ValidateCounterfactual() error

ValidateCounterfactual refuses a block that carries figures without its FR-004 label (T030/T031).

It checks the label's SUBSTANCE, not merely its presence, because the failure mode is not an empty string — it is a well-meaning summary that replaces the caveat with something that reads better and claims more. A label must name itself a counterfactual and say the behaviour was not observed; a block with no figures needs no label at all.

func (*ReplayBlock) ValidateExclusionBalance added in v0.64.0

func (b *ReplayBlock) ValidateExclusionBalance() error

ValidateExclusionBalance asserts the FR-003 accounting closes: the sessions that were dropped account exactly for supplied minus used.

Without this the exclusion table is decorative — present, plausible, and free to disagree with the numbers beside it. SC-008 asks for every exclusion to be counted, and a count nobody reconciles is how one goes missing.

type ReplayCellCost added in v0.64.0

type ReplayCellCost struct {
	CellID string `json:"cell_id"`
	// Provenance is measured/computed/estimated for THIS row.
	Provenance string `json:"provenance"`
	Calls      int    `json:"calls"`
	// MenuTokens is the tool-surface cost of what the agent was shown under
	// this cell, computed from the supplied fleet input.
	MenuTokens int `json:"menu_tokens"`
	// ResponseTokens is present only when a response figure could be
	// produced at all; bodies-off it is an estimate from byte lengths, never
	// a measurement, and a pointer so a genuine 0 is distinguishable from
	// "not computed".
	ResponseTokens *int `json:"response_tokens,omitempty"`
	// AbsoluteWorkloadWithheld records that no absolute complete-workload
	// figure is available for this cell. Bodies-off that is true of EVERY
	// cell, because complete workload includes every consumed response and
	// that text is absent. It is reported as unavailable, never as zero.
	AbsoluteWorkloadWithheld bool   `json:"absolute_workload_withheld,omitempty"`
	WithheldReason           string `json:"withheld_reason,omitempty"`
}

type ReplayDirectDelta added in v0.64.0

type ReplayDirectDelta struct {
	FromCellID  string  `json:"from_cell_id"`
	ToCellID    string  `json:"to_cell_id"`
	Provenance  string  `json:"provenance"`
	DeltaTokens int     `json:"delta_tokens"`
	DeltaPct    float64 `json:"delta_pct"`
}

ReplayDirectDelta is the honest bodies-off headline: the cross-mode delta between the two direct cells, whose identical call responses cancel out of the comparison. It is a row in its own right and carries its own provenance (computed — it is arithmetic over two measured menu costs).

type ReplayExclusion added in v0.64.0

type ReplayExclusion struct {
	Reason   string `json:"reason"`
	Sessions int    `json:"sessions"`
}

ReplayExclusion counts the supplied sessions dropped for one reason (FR-003, SC-008). Reasons are the Replay* constants above.

type ReplayLoaderAccounting added in v0.64.0

type ReplayLoaderAccounting struct {
	RecordsDropped int                  `json:"records_dropped"`
	Dropped        []ReplayLoaderReason `json:"dropped,omitempty"`
	CostsWithheld  int                  `json:"costs_withheld"`
	Withheld       []ReplayLoaderReason `json:"withheld,omitempty"`
	RecordsFlagged int                  `json:"records_flagged"`
	Flagged        []ReplayLoaderReason `json:"flagged,omitempty"`
	// OrphanedSubCalls counts sub-calls whose parent fell outside the exported
	// window. They ARE counted in the workload, at top level; what was lost is
	// their attribution to a parent. Reported because a total that hides it
	// cannot be audited.
	OrphanedSubCalls int `json:"orphaned_sub_calls,omitempty"`
}

ReplayLoaderAccounting is the loader-level companion to ReplayBlock.Exclusions.

The three buckets are deliberately separate rather than one total. A dropped record never entered a unit of work; a withheld cost was suppressed on a record that DID contribute; a flagged record contributed and was counted but carries a usability caveat. "12 exclusions" spanning all three is uninterpretable, and an uninterpretable count satisfies the letter of "everything is counted" while destroying its purpose.

type ReplayLoaderReason added in v0.64.0

type ReplayLoaderReason struct {
	Reason string `json:"reason"`
	Count  int    `json:"count"`
}

ReplayCellCost is one mode cell's recomputed cost over the recorded workload. Bodies-off it carries a MEASURED menu cost and, at best, an ESTIMATED response cost derived from pre-truncation byte LENGTHS — which is exactly the mix that per-row provenance exists to express. ReplayLoaderReason is one reason code and its count, as the loader recorded it. The reason strings are the loader's own vocabulary rather than the report's closed enum: they name causes the report-level enum has no term for (a truncated internal response that would overstate, a byte-gap record), and flattening them into the closed set would destroy the very distinction that makes them actionable.

type ReplayOptions added in v0.64.0

type ReplayOptions struct {
	// RecordingPath is the activity JSONL export. It must live OUTSIDE the
	// repository working tree — the loader refuses an in-tree path, because a
	// replay input is raw recorded traffic and must never be committed.
	RecordingPath string

	// Fleet is the frozen half of the MANDATORY fleet input: the tool
	// definitions the menu cost is computed from, as a committed corpus.
	// Exactly one of Fleet and LiveFleet must be supplied.
	Fleet *Corpus

	// FleetID names the fleet shape every figure is quoted for (IC-004).
	// Empty falls back to the corpus version.
	FleetID string

	// LiveFleet is the live half of the fleet input: a running proxy's own
	// catalog, read over MCP tools/list (mcptools.go). It is an ALTERNATIVE to
	// Fleet, never a supplement — see ResolveFleet for why supplying both is
	// an error rather than a preference.
	//
	// It is a structural FleetSource for the same reason Arms are structural
	// EncodingArms: bench cannot import the packages that build the concrete
	// producers, and a test must be able to satisfy the seam with a literal.
	LiveFleet FleetSource

	// Bodies is the loader's privacy posture. The ZERO VALUE is bodies-off,
	// which is the safe default and the one this file's contract is written
	// for; bodies-on is an explicit opt-in that warns.
	Bodies replaycorpus.BodyPolicy

	// Encoding is the tiktoken encoding the loader counts bodies with on a
	// bodies-on run. Empty means the loader default.
	Encoding string

	// Arms resolves the encoding arms named by the mode matrix, keyed by arm
	// name. Package bench cannot import bench/arms (arms imports bench), so
	// arms arrive here as structural EncodingArm values the way armrun.go
	// takes them. ReplayArmNames lists exactly what must be present.
	Arms map[string]EncodingArm

	// Warnf receives the loader's operator-facing warnings. Nil means stderr.
	Warnf func(format string, args ...any)
}

ReplayOptions configures one replay run.

type Report

type Report struct {
	Encoding      string       `json:"encoding"`
	CorpusVersion string       `json:"corpus_version"`
	CorpusTools   int          `json:"corpus_tools"`
	Modes         []ModeResult `json:"modes"`
	Notes         []string     `json:"notes"`
}

Report is the full token-reduction benchmark result.

func ComputeReport

func ComputeReport(tk *Tokenizer, corpus *Corpus) *Report

ComputeReport computes the per-mode context-token cost over the corpus and the savings of each proxy mode versus the baseline (all tools loaded directly).

func (*Report) WriteHTML

func (r *Report) WriteHTML(path string) error

WriteHTML renders the report as a self-contained static dashboard. The output is a single file with no external assets so it can be published as-is to a static host (CI release-tag publishing is tracked as a follow-up).

func (*Report) WriteJSON

func (r *Report) WriteJSON(path string) error

WriteJSON writes the report as indented JSON to path.

func (*Report) WriteReports

func (r *Report) WriteReports(dir string) (jsonPath, htmlPath string, err error)

WriteReports writes both report.json and dashboard.html into dir.

type ReportV2 added in v0.51.0

type ReportV2 struct {
	ReportVersion int                  `json:"report_version"`
	GeneratedAt   string               `json:"generated_at"`
	Tokenizer     TokenizerInfo        `json:"tokenizer"`
	Proxy         *ProxyInfo           `json:"proxy,omitempty"`
	Corpora       []CorpusDescriptor   `json:"corpora"`
	Arms          []ArmResult          `json:"arms"`
	ResponseCost  *ResponseCostSummary `json:"response_cost,omitempty"`
	BreakEven     *BreakEvenAnalysis   `json:"break_even,omitempty"`
	// SessionEstimates carries SessionCostRow, not the bare estimate: FR-013
	// requires a measured retry rate and a defaulted one to be
	// distinguishable INSIDE one table, and RetryRateForArm returns 0.0 for an
	// unknown arm — indistinguishable from a measured 0.0 without the per-row
	// provenance the row type adds.
	SessionEstimates []SessionCostRow `json:"session_estimates,omitempty"`
	Latency          *LatencyV2       `json:"latency,omitempty"`
	Lap              *LapVerdict      `json:"lap,omitempty"`
	Subset           *SubsetInfo      `json:"subset,omitempty"`
	// Spec 103 blocks. Additive and optional: omitted they do not appear at
	// all, so existing offline reports and their consumers are byte-
	// unaffected and no report_version bump is needed. Each names its own
	// accounting source rather than borrowing the document-level Tokenizer.
	RunStatus            *RunStatus                 `json:"run_status,omitempty"`
	Replay               *ReplayBlock               `json:"replay,omitempty"`
	AgentLoop            *AgentLoopBlock            `json:"agent_loop,omitempty"`
	PayloadDecomposition *PayloadDecompositionBlock `json:"payload_decomposition,omitempty"`
	Provenance           map[string]string          `json:"provenance"`
}

ReportV2 is the versioned report envelope (research D12). Additive over the v1 report: existing consumers are unaffected (reports are never committed, Spec 065 CN-003).

func BuildOfflineReportV2 added in v0.51.0

func BuildOfflineReportV2(tk *Tokenizer, generatedAt string, sections []OfflineSection) (*ReportV2, error)

BuildOfflineReportV2 measures every section and assembles the v2 envelope. Sections are processed in caller order; within a section rows follow the caller's arm order, then skip rows, then extra rows — fully deterministic (FR-010). Lap/Subset merging stays with the caller (CLI flags own those).

func ReplayReport added in v0.64.0

func ReplayReport(tk *Tokenizer, block *ReplayBlock) *ReportV2

ReplayReport wraps a replay block in the v2 report envelope.

GeneratedAt is PINNED here rather than stamped (T029): a replay report's value is that two runs over the same inputs are byte-identical, and a wall-clock field defeats that check quietly. Corpora and Arms are present but empty — replay measures no encoding arms over a corpus; it crosses the mode matrix, which is why it is a block rather than one more OfflineSection.

func (*ReportV2) PublicationCheck added in v0.64.0

func (r *ReportV2) PublicationCheck() PublicationDecision

PublicationCheck applies FR-030 and FR-032 to a finished report.

It is strict about ABSENCE in a way ValidateAdditiveBlocks deliberately is not. Everywhere else in this harness an unset optional field means "this run did not produce that", which is fine; at the publication boundary it means "nobody said", and "nobody said" must not read as "fine". Undeclared run completeness, undeclared input availability and an undeclared records reference are each blockers here, while a report that never calls this function is unaffected — which is what keeps the addition additive.

The sole-support rule is the substance of FR-030's second sentence. Marking a private-recording figure is necessary but not sufficient: a report whose EVERY block rests on a private recording is a published claim supported by nothing an outsider can check, however honestly each block is labelled. One block backed by reproducible inputs is what turns the private figure into corroborating detail.

func (*ReportV2) ValidateAdditiveBlocks added in v0.64.0

func (r *ReportV2) ValidateAdditiveBlocks() error

ValidateAdditiveBlocks enforces, at emission time, the two properties the JSON schema cannot: that every present block names a populated accounting source, and that every row of every block carries a provenance from the closed enum.

The schema declares these blocks but has no additionalProperties:false and cannot require a field to be non-empty, so a block whose accounting source was never set would validate silently — which is the exact failure the contract calls out ("a field that exists but is never set does not satisfy the contract"). Callers that build a report should run this before writing.

func (*ReportV2) WriteHTML added in v0.51.0

func (r *ReportV2) WriteHTML(path string) error

WriteHTML renders the v2 report as a self-contained static dashboard (Spec 083 T035, FR-017/018, SC-005): arms table, corpora table, response-cost percentiles, break-even, session estimates, LAP row, provenance badges on every headline section, and the tokenizer caveat banner. Single file, inline CSS only, no external resource loads — bench/report_test.go asserts self-containment.

func (*ReportV2) WriteJSON added in v0.51.0

func (r *ReportV2) WriteJSON(dir string) (string, error)

WriteJSON writes the v2 report as indented JSON into dir/report.json.

Before marshaling it resolves every raw-record reference against dir, the run-local root those references are relative to: a reference whose artifact is no longer there degrades to "records not retained" instead of being written out as a path that resolves to nothing (T057/FR-029). This is the only moment both halves are in hand, and the report lands in the same gitignored tree the records do — so a reference that was accurate when the block was built can already be stale by the time it is serialized.

It stays deterministic (FR-010/SC-002): the same run directory produces the same bytes, and no wall-clock time is injected.

func (*ReportV2) WriteReports added in v0.51.0

func (r *ReportV2) WriteReports(dir string) (jsonPath, htmlPath string, err error)

WriteReports writes report.json and dashboard.html for a v2 run into dir.

type ResponseCostSummary added in v0.51.0

type ResponseCostSummary struct {
	P50      int                            `json:"p50"`
	P95      int                            `json:"p95"`
	Max      int                            `json:"max"`
	Mean     float64                        `json:"mean"`
	PerQuery []DiscoveryResponseMeasurement `json:"per_query,omitempty"`
}

ResponseCostSummary aggregates per-query response measurements (FR-001).

func SummarizeResponseCost added in v0.51.0

func SummarizeResponseCost(perQuery []DiscoveryResponseMeasurement) *ResponseCostSummary

SummarizeResponseCost aggregates per-query measurements into the FR-001 summary with nearest-rank percentiles over TotalTokens (same rank rule as the latency percentiles in live_report.go). PerQuery keeps the caller's order (golden-set order), so reports stay deterministic (FR-010).

type RetrievalGate

type RetrievalGate struct {
	Passed    bool    `json:"passed"`
	Metric    string  `json:"metric,omitempty"`
	Tolerance float64 `json:"tolerance,omitempty"`
}

RetrievalGate is the `retrieval.gate` object of the score-report contract.

A standalone live run has no stored baseline to regress against, so the gate cannot fail by construction: Passed is true and Metric/Tolerance are empty. Regression gating against a committed baseline is the CI lane's job (MCP-3133) — that run fills Metric/Tolerance and can set Passed=false.

type RetrievalMetricValues

type RetrievalMetricValues struct {
	RecallAt map[int]float64 `json:"recall_at"`
	MRR      float64         `json:"mrr"`
	NDCGAt10 float64         `json:"ndcg_at_10"`
	MAP      float64         `json:"map"`
}

RetrievalMetricValues holds the aggregated metric numbers. It is the `retrieval.metrics` object of the Spec 065 score-report.schema.json contract.

type RetrievalMetrics

type RetrievalMetrics struct {
	CorpusVersion string                `json:"corpus_version"`
	GoldenVersion string                `json:"golden_version,omitempty"`
	RunsAveraged  int                   `json:"runs_averaged"`
	QueryCount    int                   `json:"query_count,omitempty"`
	Metrics       RetrievalMetricValues `json:"metrics"`
	Gate          RetrievalGate         `json:"gate"`
}

RetrievalMetrics is the aggregated retrieval-quality report over a golden set. Its JSON shape IS the Spec 065 score-report.schema.json `retrieval` block (nested `metrics` + `gate`), so a live report's retrieval payload validates against that contract directly.

func ScoreRetrieval

func ScoreRetrieval(golden *GoldenSet, search SearchFunc, ks []int) (*RetrievalMetrics, error)

ScoreRetrieval replays every golden query through search and aggregates Recall@k (for each k in ks), MRR, nDCG@10 and MAP as the mean over all queries. The search is deterministic (BM25), so a single run is averaged.

type RetrievalScore added in v0.51.0

type RetrievalScore struct {
	RecallAt1  float64 `json:"recall_at_1"`
	RecallAt3  float64 `json:"recall_at_3"`
	RecallAt5  float64 `json:"recall_at_5"`
	RecallAt10 float64 `json:"recall_at_10"`
	MRR        float64 `json:"mrr"`
	NDCGAt10   float64 `json:"ndcg_at_10"`
	MAP        float64 `json:"map"`
	// MetricNote documents the gain formula (FR-012), or — for an
	// index-altering arm scored on a corpus without relevance labels —
	// explains the absence of numbers.
	MetricNote string `json:"metric_note,omitempty"`
}

RetrievalScore is the flat retrieval-quality DTO of the v2 contract. It is mapped from the existing nested RetrievalMetrics by MapRetrievalMetrics — the single conversion point — so the report schema stays flat and stable.

func MapRetrievalMetrics added in v0.51.0

func MapRetrievalMetrics(m *RetrievalMetrics) *RetrievalScore

MapRetrievalMetrics converts the existing nested bench.RetrievalMetrics (recall_at as a map) into the flat report DTO. nil in, nil out — a nil score marks a quality-neutral arm (FR-008).

type RetrieveToolsFunc added in v0.51.0

type RetrieveToolsFunc func(query string, limit int, detail string) (rankedIDs []string, responseText string, err error)

RetrieveToolsFunc replays one query through the proxy's MCP retrieve_tools tool with the given per-call detail override ("full" | "compact") and returns the ordered entry ids (server:tool, best first) plus the raw serialized response text — exactly what gets tokenized.

type RunStatus added in v0.64.0

type RunStatus struct {
	Completeness string `json:"completeness"`
	// Reason is mandatory on a partial run. A partial with no reason is the
	// same silence as no mark at all — the reader learns something is
	// missing but not whether it matters.
	Reason      string `json:"reason,omitempty"`
	Interrupted bool   `json:"interrupted,omitempty"`

	CellsPlanned  int      `json:"cells_planned"`
	CellsMeasured int      `json:"cells_measured"`
	CellsSkipped  int      `json:"cells_skipped,omitempty"`
	MissingCells  []string `json:"missing_cells,omitempty"`
}

RunStatus is the FR-032 mark (T058): whether this report covers the whole comparison it set out to make.

Why the three counts rather than one boolean. A cell can be absent for two very different reasons, and collapsing them would make the mark useless. A DELIBERATE skip — code_exec skipped because code execution is disabled — is accounted for, expected, and does not make the comparison partial; the harness already reports such cells as skip rows naming what they collapse onto. An UNEXPLAINED gap is a cell the run was supposed to measure and did not, which is exactly the subset-presented-as-the-whole failure FR-032 exists to catch. Planned − measured − skipped is that gap, and MissingCells names it, because "5 of 6" tells a reader nothing about which comparison they are missing.

Interrupted is separate again: a run killed during the fourth repetition of the last cell may have touched every cell and still be partial, since the per-cell figures are averages over a repetition count the operator chose.

func DeriveRunStatus added in v0.64.0

func DeriveRunStatus(planned, measured, skipped []string, interrupted bool, reason string) *RunStatus

DeriveRunStatus computes the completeness label from what the run actually did, rather than accepting one the caller asserts.

This direction matters. A producer that stamps its own "complete" stamps it from the same optimism that lost the cells; a label derived from the planned set and the measured set cannot disagree with them. Validate then re-checks the same arithmetic on the way out, so a status that was hand-edited (or deserialized from somewhere else) is caught at emission time too.

planned, measured and skipped are cell ids. reason is optional: when the run turns out to be partial and the caller supplied none, a factual one is synthesized, because a partial run must never reach the report unexplained.

func (*RunStatus) IsPartial added in v0.64.0

func (s *RunStatus) IsPartial() bool

IsPartial reports whether this run may not be published as a complete comparison. An unset status is NOT treated as complete here — callers that need that distinction check for nil themselves, and PublicationCheck does.

func (*RunStatus) Validate added in v0.64.0

func (s *RunStatus) Validate() error

Validate re-derives the completeness label from the counts and refuses a status that claims more than the counts support. The interesting case is a hand-stamped "complete" over an unexplained gap: that is the exact document FR-032 forbids, and it is well-formed JSON, so nothing else would catch it.

type RunTaskFunc added in v0.64.0

type RunTaskFunc func(cell ModeCell, unitID string, runIndex int) (*UnitRecord, error)

RunTaskFunc runs ONE unit of work under ONE mode cell and returns its raw record. It is a function type for the reason RetrieveToolsFunc is (flipgate.go): a concrete suite in the signature would pin every test to a paid run, and an untestable rule is an unenforced rule.

type SavingVerdict added in v0.64.0

type SavingVerdict struct {
	CellID         string     `json:"cell_id"`
	SavingPct      float64    `json:"saving_pct"`
	Regression     bool       `json:"regression,omitempty"`
	Withheld       bool       `json:"withheld,omitempty"`
	WithheldReason string     `json:"withheld_reason,omitempty"`
	FleetShape     FleetShape `json:"fleet_shape"`
}

SavingVerdict is a per-cell saving against the baseline arm, or the stated reason there is none to publish. FleetShape travels with it because a percentage without its fleet shape does not appear (IC-004/SC-005): the figure moves with fleet size, so quoting it bare is quoting it at an unstated size.

type SearchFunc

type SearchFunc func(query string, limit int) (ranked []string, err error)

SearchFunc replays one query through the retrieval system under test and returns the ranked tool IDs (most relevant first), limited to `limit`.

type SessionCostEstimate added in v0.51.0

type SessionCostEstimate struct {
	Arm             string  `json:"arm"`
	CallsPerSession int     `json:"calls_per_session"`
	RetryRate       float64 `json:"retry_rate"`
	EstimatedTokens int     `json:"estimated_tokens"`
}

SessionCostEstimate is one row of the FR-019 session estimator (provenance is always "estimated"; retry-rate defaults documented in research D8).

func EstimateSessionCost added in v0.51.0

func EstimateSessionCost(arm string, proxyMenuTokens int, meanResponseTokens float64, calls int) SessionCostEstimate

EstimateSessionCost computes one estimator row for a single arm and calls-per-session point (formula and rounding policy in the package comment). meanResponseTokens is the arm's mean retrieve_tools response cost — measured for the live baseline, derived from arm token ratios otherwise.

func EstimateSessionCosts added in v0.51.0

func EstimateSessionCosts(proxyMenuTokens int, meanResponseTokensByArm map[string]float64) []SessionCostEstimate

EstimateSessionCosts produces the full session-estimate table: one row per arm per default calls-per-session point, in deterministic order (arms sorted lexicographically, then calls ascending) regardless of map iteration order (FR-010). meanResponseTokensByArm maps arm name → mean retrieve_tools response tokens for that arm.

type SessionCostRow added in v0.64.0

type SessionCostRow struct {
	SessionCostEstimate

	// Provenance is this ROW's label, from the closed measured/computed/
	// estimated enum: computed when a measured retry rate was applied,
	// estimated when the row still rests on the literature default. Never
	// measured — the token magnitudes are tokenizer-counted (see the block
	// comment above).
	Provenance string `json:"provenance"`

	// RetryRateProvenance labels the retry rate specifically. It is the field
	// that makes a measured 0.0 distinguishable from a defaulted 0.0, and it
	// is separate from Provenance because a reader scanning the retry-rate
	// column needs the answer in that column.
	RetryRateProvenance string `json:"retry_rate_provenance"`

	// MeasuredRuns is how many runs back the measurement, 0 when none. It
	// travels with the row so a consumer can apply the FR-021 four-run
	// headline bar without re-joining against the agent-loop block.
	MeasuredRuns int `json:"measured_runs,omitempty"`

	// FirstAttemptSuccessPct is the measured first-attempt success rate, or
	// nil when nothing was measured. It is a POINTER for the same reason
	// PayloadDecompositionRow.ShareAnnotationsPct is: a 0 here would read as
	// "measured, and nothing ever succeeded on the first attempt", which is a
	// far stronger claim than "not measured". nil marshals to null.
	FirstAttemptSuccessPct *float64 `json:"first_attempt_success_pct"`
}

SessionCostRow is a session-cost estimate that knows where its inputs came from. It EMBEDS SessionCostEstimate rather than redefining it, so the serialized row is a strict superset of the existing shape (the embedded fields flatten into the same JSON object) and no existing consumer of a session-estimate row breaks.

func EstimateSessionCostRow added in v0.64.0

func EstimateSessionCostRow(arm string, proxyMenuTokens int, meanResponseTokens float64, calls int, measured MeasuredOutcomeSource) SessionCostRow

EstimateSessionCostRow computes one session-cost row, applying a measured retry rate for the arm when one exists and the documented default otherwise. The formula and rounding policy are unchanged (package comment); only the source of retry_rate can differ, and the row says which source it was.

func EstimateSessionCostRows added in v0.64.0

func EstimateSessionCostRows(proxyMenuTokens int, meanResponseTokensByArm map[string]float64, measured MeasuredOutcomeSource) []SessionCostRow

EstimateSessionCostRows produces the full self-describing session-estimate table: one row per arm per default calls-per-session point, in the same deterministic order as EstimateSessionCosts (arms lexicographic, then calls ascending — FR-010), with measured and estimated rows coexisting in the one slice. Pass a nil source offline; every row is then identical to what EstimateSessionCosts produces, plus its estimated badge.

type SkipExample added in v0.51.0

type SkipExample struct {
	ToolID string `json:"tool_id"`
	Error  string `json:"error"`
}

SkipExample is one (tool, error) pair from an arm's skipped tools (FR-009).

type SourcedFigure added in v0.64.0

type SourcedFigure struct {
	Value  float64
	Source AccountingSource
}

SourcedFigure is one value together with the accounting source that produced it. The pairing is the whole point: a bare float64 cannot be checked against the never-sum rule, and by the time figures are floats in a slice the source is exactly the information that has been lost.

type Span added in v0.51.0

type Span struct {
	Label string
	Start int
	End   int
}

Span is one contiguous labeled byte range [Start, End) of a response text. A valid span list is a partition: sorted, non-empty, gap-free, covering exactly [0, len(text)).

func PartitionRetrieveToolsResponse added in v0.51.0

func PartitionRetrieveToolsResponse(raw string) ([]Span, int, error)

PartitionRetrieveToolsResponse partitions the raw retrieve_tools MCP text content (the exact JSON string produced by internal/server/mcp.go handleRetrieveToolsWithMode) into labeled byte spans, and returns the number of tools in the response. The walk is offset-exact over the ORIGINAL bytes (json.Decoder.InputOffset) — the text is never re-marshaled, so the token total measured over these spans is the cost of the wire payload itself.

A key/value pair's span runs from the end of the previous value through the end of its own value, so it includes its preceding separator and its key bytes: the "inputSchema" bucket is the full cost of carrying schemas, key included. Structural bytes not owned by any pair land in "other".

type SubsetInfo added in v0.51.0

type SubsetInfo struct {
	Seed int `json:"seed"`
	Size int `json:"size"`
}

SubsetInfo records the seeded query subset of a public-corpus run (FR-014): same revision + seed + size ⇒ same subset.

type TokenDistribution added in v0.51.0

type TokenDistribution struct {
	Samples int `json:"samples"`
	P50     int `json:"p50"`
	P95     int `json:"p95"`
	Max     int `json:"max"`
}

TokenDistribution summarizes per-query response-token counts for one mode.

type TokenReductionReport added in v0.51.0

type TokenReductionReport struct {
	Encoding        string            `json:"encoding"`
	Full            TokenDistribution `json:"full"`
	Compact         TokenDistribution `json:"compact"`
	MedianReduction float64           `json:"median_reduction"`
}

TokenReductionReport compares full vs compact response-token distributions. MedianReduction = 1 - compact.P50/full.P50 (SC-001 gates on >= 0.50).

type Tokenizer

type Tokenizer struct {
	// contains filtered or unexported fields
}

Tokenizer wraps a tiktoken encoding for reproducible token estimation.

func NewTokenizer

func NewTokenizer(encoding string) (*Tokenizer, error)

NewTokenizer constructs a Tokenizer for the given tiktoken encoding name.

func (*Tokenizer) Count

func (t *Tokenizer) Count(text string) int

Count returns the number of tokens in text.

func (*Tokenizer) CountTool

func (t *Tokenizer) CountTool(tl Tool) int

CountTool returns the context-token cost of a single tool definition.

It counts the tool name and description only. Input JSON schemas are excluded uniformly across every mode because the committed Spec 065 corpus snapshot does not carry schemas. Schemas are dropped from BOTH sides — the baseline's upstream tools and the proxy modes' management tools (e.g. upstream_servers carries a large multi-field schema) — so this is a well-defined name+description-only metric, not an unambiguously conservative one. The live docker-compose run (README.md) adds full schemas from GET /api/v1/tools for the exact headline number.

func (*Tokenizer) CountToolWithSchema

func (t *Tokenizer) CountToolWithSchema(tl Tool) int

CountToolWithSchema returns the context-token cost of a tool definition INCLUDING its JSON input schema (name + description + schema). This is the authoritative per-tool context cost an agent actually pays. A tool with no schema counts identically to CountTool, so mixing schema-bearing (live) and schemaless tools in one report is well-defined. Used by the live run, where both the baseline upstream tools AND the proxy management tools carry their real schemas — counting schemas on BOTH sides is what keeps the headline savings honest rather than overstated.

type TokenizerInfo added in v0.51.0

type TokenizerInfo struct {
	Name   string `json:"name"`
	Caveat string `json:"caveat"`
}

TokenizerInfo names the token estimator and carries the mandatory accuracy caveat rendered wherever absolute numbers appear (research D11, SC-005).

type Tool

type Tool struct {
	ToolID      string          `json:"tool_id"`
	Server      string          `json:"server"`
	Name        string          `json:"tool"`
	Description string          `json:"description"`
	Schema      json.RawMessage `json:"schema,omitempty"`
}

Tool is a single tool definition the benchmark scores token cost over. It matches the shape of both the Spec 065 corpus snapshot and the embedded proxy-tool fixture. Schema is optional: the committed corpus snapshot is description-only (nil schema), while the live run (live.go) populates it with each tool's full JSON input schema for the exact-token headline.

func FetchToolsList added in v0.64.0

func FetchToolsList(ctx context.Context, surface string, page ToolsListPageFunc) ([]Tool, error)

FetchToolsList performs a full, paginated MCP tools/list against one surface and maps the result onto []Tool with REAL schemas, or fails.

surface is used only for diagnostics (e.g. "/mcp/all"); the transport is supplied by page. The returned order is the wire order, page by page, so two runs against an unchanged proxy produce identical fleets.

func ProxyToolsForMode

func ProxyToolsForMode(mode string) []Tool

ProxyToolsForMode returns the built-in mcpproxy proxy + management tool definitions that occupy the agent's context window in the given routing mode.

The catalog is derived directly from the live server tool builders (internal/server.ProxyModeToolDefs → buildCallToolModeTools / buildCodeExecModeTools in internal/server/mcp_routing.go). This is the single source of truth: both routing modes append the shared management tool set (upstream_servers, quarantine_security, search_servers, list_registries), so deriving from the builders guarantees the benchmark counts the real per-mode context cost and can never drift from production by re-introducing the undercount that inflated the headline savings (MCP-3161).

type ToolTokenEntry added in v0.51.0

type ToolTokenEntry struct {
	ToolID string `json:"tool_id"`
	Tokens int    `json:"tokens"`
}

ToolTokenEntry is one heaviest-tools row (FR-020).

type ToolsListPageFunc added in v0.64.0

type ToolsListPageFunc func(ctx context.Context, cursor string) (json.RawMessage, error)

ToolsListPageFunc performs ONE tools/list round trip and returns the RAW JSON-RPC result object for that page (the `{"tools":[...],"nextCursor":...}` value, not the envelope).

It is a function type for the reason RetrieveToolsFunc is (flipgate.go): the arithmetic that matters — identity mapping, prefix stripping, canonical- ization and above all the stub guard — must be testable with no proxy running, and a concrete client in the signature would pin every test to one. The raw json.RawMessage return is load-bearing; see the file header.

type UnitRecord added in v0.64.0

type UnitRecord struct {
	CellID   string `json:"cell_id"`
	UnitID   string `json:"unit_id"`
	RunIndex int    `json:"run_index"`
	// Completion is the suite's verdict: pass, fail, or no-signal.
	Completion string `json:"completion"`
	// Partial marks a run record that did not finish. Partial records are
	// excluded from every figure and counted separately (FR-032).
	Partial  bool          `json:"partial,omitempty"`
	Attempts []Attempt     `json:"attempts,omitempty"`
	Usage    ProviderUsage `json:"usage"`
	// Source names where THIS record's token figures came from.
	//
	// It lives on the record rather than only on the block because the block's
	// source is a LABEL applied at assembly, while the sum it labels is built
	// here, record by record. Without a per-record source, aggregateCell adds
	// integers whose origin nothing checked and AgentLoopBlockFor then stamps
	// the caller's provider onto the result — so a tokenizer-derived figure
	// mixed into a provider run produces a plausible hybrid wearing a provider
	// label. The never-sum rule has to be checkable at the point of addition,
	// not asserted at the point of naming.
	//
	// Zero value means "unset", which aggregation treats as the block's own
	// source: the common case is one driver producing every record, and
	// requiring every fake in every test to restate it would be noise.
	Source    AccountingSource `json:"source,omitzero"`
	TurnCount int              `json:"turn_count,omitempty"`
}

UnitRecord is the raw outcome of ONE UNIT OF WORK — one task from the fixed task set — under one mode cell on one run. It is the traceable unit behind every headline figure (FR-029). A unit is never an individual tool call, no matter how many attempts it took.

func CollectAgentLoop added in v0.64.0

func CollectAgentLoop(driver AgentLoopDriver, opts AgentLoopOptions) ([]UnitRecord, error)

CollectAgentLoop runs cells x runs x units through the driver.

The iteration is cell-major and deterministic so a partially completed run is a prefix of a full one rather than an arbitrary subset, and so a reader comparing two result sets is comparing the same order of work.

type UsageFunc added in v0.64.0

type UsageFunc func(cell ModeCell, unitID string, runIndex int) (ProviderUsage, error)

UsageFunc returns the provider-reported usage the DRIVER captured for one unit of work, read off the model responses themselves.

It exists because of a specific gap: the suite's per-task report has no cache-read field (research.md §5), so FR-014's third axis cannot come from the suite at all. Either the driver supplies it here, or the axis is recorded as unavailable — never as zero.

Directories

Path Synopsis
Package arms implements the encoding arms of the discovery-effectiveness profiler (spec 083).
Package arms implements the encoding arms of the discovery-effectiveness profiler (spec 083).
cmd
bench command
Command bench runs the mcpproxy benchmark.
Command bench runs the mcpproxy benchmark.
breakevencurve command
Command breakevencurve computes the fleet-size break-even curve: at how many upstream tools (and servers) does the proxy stop costing more than it saves?
Command breakevencurve computes the fleet-size break-even curve: at how many upstream tools (and servers) does the proxy stop costing more than it saves?
codeexecsaving command
Command codeexecsaving reports the RESPONSE-SIDE saving of code execution from a recorded activity export: what the sandbox's sub-call responses would have cost an agent that made those calls itself, against the script it sent and the single result it got back.
Command codeexecsaving reports the RESPONSE-SIDE saving of code execution from a recorded activity export: what the sandbox's sub-call responses would have cost an agent that made those calls itself, against the script it sent and the single result it got back.
iosplit command
Command iosplit reports code execution's cost split by BILLING DIRECTION.
Command iosplit reports code execution's cost split by BILLING DIRECTION.
warmtiktoken command
Command warmtiktoken populates the tiktoken vocabulary cache before a parallel test run.
Command warmtiktoken populates the tiktoken vocabulary cache before a parallel test run.
Package corpusio loads public tool-retrieval corpora into the bench types (spec 083, US3).
Package corpusio loads public tool-retrieval corpora into the bench types (spec 083, US3).
codeexec.go — the response-side saving of code execution.
codeexec.go — the response-side saving of code execution.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL