bench_resources

Overview ¶

Command bench_resources measures what this server costs to run, and draws the charts the documentation publishes.

Everything else measured in this repository is about tokens and tool counts. None of it tells an operator how much memory to give a container, how long a client waits before the first tool call answers, or what a second credential adds to a shared deployment. This command answers those, from the real binary, on both transports, and writes one record every downstream artifact is rendered from.

What it needs ¶

Nothing but a Go toolchain. GitLab is stood in for by an in-process HTTP server on loopback, and the OTLP collector by another, so a run is offline and a second machine measures the same thing rather than its own network. The tool surface is passed to the server explicitly and never read from the environment, for the reason CLAUDE.md gives about generators: a developer machine exporting GITLAB_MCP_TOOL_SURFACE would otherwise publish different numbers than CI.

Usage ¶

go run ./cmd/bench_resources/                 # measure, then render
go run ./cmd/bench_resources/ -render         # redraw from the record
go run ./cmd/bench_resources/ -check          # is the drawing current?
go run ./cmd/bench_resources/ -quick -json /tmp/x.json
go run ./cmd/bench_resources/ -fairness tools-call-rps  # is the quiet tenant better off?

fairness.go declares what a fairness comparison is: two populations of credentials driven differently, one bound, and two arms.

Every other scenario here drives every credential the same way and starts the server with the limiter off, so none of them can see fairness: a harness where every caller behaves alike has no quiet neighbor to protect, and with no bound in force there is nothing to protect it. This scenario answers the question those cannot, which is not "does the bound refuse the noisy tenant" but "is the quiet tenant better off". Those are different questions and the second one may honestly answer no.

Three things about the shape are load-bearing, and each of them is a way the measurement would otherwise lie.

The driver is open loop. A refused request comes back in about two milliseconds where a served one takes tens, so a closed-loop driver would send several times as many requests in the arm with the bound on, and the two arms would be different experiments whose processor-time difference is the sum of two opposite effects. Both populations therefore follow a schedule computed before the phase starts, identical in both arms, and a tick the driver could not fire is counted rather than deferred.

Served and refused are never one number. A refusal is cheap, so anything that pools them improves as the bound refuses more, which is exactly backwards. There are four terminal outcomes per population and per method, they are held to an arithmetic identity in code, and no field anywhere in the document carries a latency over all requests, so a merged percentile cannot be read out of the record by accident.

The bound is a value rather than a code path. A boundSpec carries the arguments and environment of both arms and the wire shape of its own refusal, so the listen ceilings and whatever the policy work produces are a literal in the table below rather than a copy of this file.