README
¶
loadtest -- peeringdb-plus operator load tool
SAFETY WARNING
NEVER point
--baseathttps://www.peeringdb.com. Upstream PeeringDB applies endpoint-specific rate limits and can block abusive traffic. The hourly anonymous sync cadence is not a universal one-request-per-hour limit. This tool is for the mirror athttps://peeringdb-plus.fly.dev(default) or your own local deployment via--base http://localhost:8080. The tool refuses to start when--baseor--targetnames peeringdb.com or a subdomain of it. beta.peeringdb.com is the only exception.Do NOT run this tool from CI. The package compiles as a normal
cmd/binary, but the binary is never invoked by CI / Dockerfiles / deployment scripts -- only by operators against deployed instances. Unit tests in this directory are hermetic (httptest-based, no outbound calls) and are safe to run in CI as part of the standardgo test ./...sweep.
What it does
Four modes drive read-only HTTP traffic against a peeringdb-plus mirror, exercising every entity type across five of the six API surfaces (pdbcompat /api, entrest /rest/v1, GraphQL /graphql, ConnectRPC /peeringdb.v1.*, Web UI /ui).
The MCP endpoint (/mcp) is not covered.
| mode | purpose |
|---|---|
endpoints |
One-shot inventory sweep (~114 distinct requests). Checks that each endpoint in the registry returns 2xx. |
sync |
Replays the 13-step FK-ordered type sequence across 3 depth bands (depth=0/1/2) → 39 GETs (full or incremental). Uses the worker type order, not its exact request shapes. |
soak |
Sustained QPS-capped mixed-surface load. Defaults to 30 s, 4 workers, and a global 5 req/s cap. |
ramp |
Per-surface concurrency ramp; finds the inflection point where p95/p99 latency or error rate degrades. Sequential per surface (no cross-surface contention). |
Read-only verbs only: GET for pdbcompat / entrest / Web UI, POST for GraphQL queries and ConnectRPC unary calls.
No PUT, PATCH, or DELETE anywhere -- peeringdb-plus is a read-only mirror.
Build
export PDBPLUS_LOADTEST_DIR="${TMPDIR:-/tmp}/peeringdb-plus/loadtest"
mkdir -p "$PDBPLUS_LOADTEST_DIR"
mise exec -- go build -o "$PDBPLUS_LOADTEST_DIR/loadtest" ./cmd/loadtest
Remove the task directory after the load test.
The binary is not committed and go build ./... only compiles it (it does not run it).
CI compiles + lints + tests this package along with the rest of the repository, but never invokes the resulting binary.
Modes
endpoints -- inventory sweep
"$PDBPLUS_LOADTEST_DIR/loadtest" endpoints --base http://localhost:8080
Sequentially hits every endpoint in the registry (~114 requests: 13 entities × ~8 shapes per entity, plus 3 surface-wide UI routes). Sequential by design: this is a once-per-run sweep for dashboard warmup and post-deploy validation, not concurrent stress.
| flag | default | description |
|---|---|---|
--base |
https://peeringdb-plus.fly.dev |
Target deployment URL. |
--timeout |
30s |
Per-request timeout. |
--verbose |
false |
Emit one log line per request. |
sync -- replay sync GET sequence
"$PDBPLUS_LOADTEST_DIR/loadtest" sync --mode=full --base http://localhost:8080
"$PDBPLUS_LOADTEST_DIR/loadtest" sync --mode=incremental --since=2026-04-27T12:00:00Z
"$PDBPLUS_LOADTEST_DIR/loadtest" sync --mode=incremental --since=1714219200
Issues 39 GETs.
It sends the 13 types in FK order (org, campus, fac, carrier, carrierfac, ix, ixlan, ixpfx, ixfac, net, poc, netfac, netixlan) at depth=0, then again at depth=1, then at depth=2.
The URL shape depends on the mode: full mode issues a bare /api/<short>?depth=N, incremental mode issues /api/<short>?limit=250&skip=0&depth=N&since=M.
This mode tests mirror API shapes.
Production sync streams one depth=0 response per type without limit or skip.
The order comes from pdbtypes.Names().
TestSync_OrderingMatchesWorker fails when that order differs from internal/sync.StepOrder().
| flag | default | description |
|---|---|---|
--mode |
full |
full or incremental. |
--since |
(empty) | Cursor for incremental mode. Accepts RFC3339 (2026-04-27T12:00:00Z) or unix seconds (1714219200). Defaults to now - 1h when unset. |
--base |
https://peeringdb-plus.fly.dev |
(as above) |
--timeout |
30s |
(as above) |
--verbose |
false |
(as above) |
soak -- sustained mixed-surface load
"$PDBPLUS_LOADTEST_DIR/loadtest" soak --duration=30s --concurrency=4 --qps=5
"$PDBPLUS_LOADTEST_DIR/loadtest" soak --duration=5m --concurrency=10 --qps=20 # heavier -- see notes
Spawns --concurrency workers behind a global golang.org/x/time/rate.Limiter capped at --qps requests per second (burst=1).
Each worker picks a random endpoint from the registry, hits it, and records latency.
After --duration elapses the soak terminates and prints per-surface p50/p95/p99 latency + success rates.
| flag | default | description |
|---|---|---|
--duration |
30s |
Total soak duration. |
--concurrency |
4 |
Number of concurrent workers (sharing the rate cap). |
--qps |
5 |
Global request-per-second cap. Combined across workers. |
--base |
https://peeringdb-plus.fly.dev |
(as above) |
--timeout |
30s |
(as above) |
--verbose |
false |
(as above) |
QPS rationale
The default is 5 requests per second total, shared by four workers.
The current Fly manifest uses shared-cpu-2x machines with 512 MB for both process groups.
These settings do not establish a measured capacity limit.
- Smoke / warmup:
--qps=5 --concurrency=4. - Higher load:
--qps=20 --concurrency=10, with operator coordination. Watch latency, errors, CPU throttling, and the GrafanaLive Heap by Instancepanel.
Coordinate production load tests before you increase the rate. PeeringDB Plus does not limit incoming request rates. When a machine reaches its concurrency soft limit, Fly Proxy prefers other machines for new requests.
ramp -- find inflection point per surface
Per-surface concurrency ramp that discovers, surface by surface, the concurrency level at which p95/p99 latency or error rate visibly degrades. Replaces ad-hoc operator probing with a deterministic, scriptable, per-surface capacity probe.
"$PDBPLUS_LOADTEST_DIR/loadtest" ramp --target http://localhost:8080 --entity net --max-concurrency 64
"$PDBPLUS_LOADTEST_DIR/loadtest" ramp --target https://peeringdb-plus.fly.dev --entity org --surfaces pdbcompat,graphql
Ramp loop per surface:
- Baseline at
--startconcurrency for--step-duration. - Multiply concurrency by
--growtheach step (capped at--max-concurrency). - Inflection triggers on the first step where:
- no request completed within the step (
no samples), OR p95 > baseline.p95 × --p95-multiplier, ORp99 > --p99-absolute, ORerror rate > --error-rate-threshold.
- no request completed within the step (
- Hold at the inflection concurrency for
--hold-durationto gather a stable p99 reading. - Past-inflection: 1-2 additional steps (concurrency permitting) so the operator can see how badly things degrade just past the knee.
- Print a markdown table for the surface to stdout, then move on.
If no request completes within the baseline step, the ramp has no latency to compare with.
It prints ABORTED for the surface and moves on.
In the table, a step without samples shows - in the latency and error columns.
A request completes when it gets a response, fails, or reaches --timeout.
Each request that completes within its step is a sample.
A non-2xx response, a failed request and a request that reaches --timeout count as errors, and their latencies go into the percentiles.
A request that fails while it reads the response body is a failed request.
A request that the end of the step cuts off is not a sample, also when it has its response headers.
A request that completes as the step ends is a sample.
Thus, against a target that does not answer, a step has a 100% error rate if --timeout is shorter than the step, and no samples if it is not.
Surfaces are exercised sequentially (in --surfaces order) so cross-surface contention does not bias results.
Each surface fetches the same prefetched ID list (round-robin) so requests don't all hit a single hot row.
| flag | default | description |
|---|---|---|
--target |
https://peeringdb-plus.fly.dev |
Alias for --base. Either flag works. |
--entity |
net |
Entity type for the ramp: net (uses /ui/asn/<asn> for webui) or org (uses /ui/org/<id>). |
--start |
1 |
Initial concurrency level (the baseline step). |
--growth |
1.5 |
Per-step concurrency multiplier (next = ceil(prev * growth)). |
--step-duration |
2s |
Wall-clock time at each ramp step. |
--hold-duration |
10s |
Wall-clock time held at the inflection step for stable p99. |
--max-concurrency |
256 |
Upper bound on per-step concurrency. Ramp terminates once a step would exceed this cap. |
--p95-multiplier |
2.0 |
Inflection trigger: step.p95 > baseline.p95 × this multiplier. |
--p99-absolute |
1s |
Inflection trigger: step.p99 > this absolute duration. |
--error-rate-threshold |
0.01 |
Inflection trigger: step error rate > this fraction (0.01 = 1%). |
--surfaces |
(all five) | Comma-separated surfaces in execution order. Valid: pdbcompat,entrest,graphql,connectrpc,webui. |
--prefetch-count |
20 |
Number of IDs to prefetch via /api/<entity>?limit=N for round-robin selection across ramp steps. |
Ramp vs soak -- when to use which
- Soak is for sustained known-rate validation: "does the mirror handle 20 req/s × 5 minutes without dropping responses?" You set the rate, run for a duration, and check the success percentage and latency tail. The traffic mix is randomly drawn from the full registry -- every surface contributes.
- Ramp is for capacity discovery: "at what concurrency does the pdbcompat surface fall over?" You don't know the answer in advance; ramp finds it. Per-surface ramps tell you that, e.g., the GraphQL surface tips at C=24 while the pdbcompat surface doesn't tip until C=64 -- useful for capacity planning and for knowing which surface dominates your incident response when load spikes hit.
Sample output
### pdbcompat (entity=net)
| label | C | p50 | p95 | p99 | err % | rps |
|--------------------|----:|--------:|--------:|--------:|-------:|--------:|
| baseline | 1 | 12ms | 29ms | 41ms | 0.00% | 54.2 |
| inflection | 8 | 74ms | 412ms | 1.21s | 0.00% | 189.3 |
| hold | 8 | 79ms | 428ms | 1.18s | 0.00% | 192.6 |
| inflection+1 | 12 | 156ms | 891ms | 2.04s | 0.42% | 201.0 |
inflection reason: p99 1.21s > 1s absolute
The output is markdown verbatim -- paste directly into incident reports, capacity-planning docs, or tee to a file:
"$PDBPLUS_LOADTEST_DIR/loadtest" ramp --target http://localhost:8080 | tee ramp-results.md
Auth
Set PDBPLUS_LOADTEST_AUTH_TOKEN to send Authorization: Bearer <token> on every request:
PDBPLUS_LOADTEST_AUTH_TOKEN=$(cat ~/.pdbplus-token) "$PDBPLUS_LOADTEST_DIR/loadtest" soak --duration=1m
Use it only when a proxy in front of the mirror requires a token. peeringdb-plus ignores the header.
Every caller sees the tier that PDBPLUS_PUBLIC_TIER sets.
Output
The endpoints, sync, and soak modes print an aligned summary table at the end.
The ramp mode prints its markdown tables instead (see Sample output).
=== loadtest soak summary ===
wall-clock 30.012s
observed-rps 4.96 req/s
SURFACE COUNT OK ERR SUCCESS% P50 P95 P99
pdbcompat 53 53 0 100.0% 8.412ms 24.13ms 48.007ms
entrest 31 31 0 100.0% 9.205ms 27.34ms 52.118ms
graphql 16 16 0 100.0% 18.031ms 89.26ms 89.26ms
connectrpc 29 29 0 100.0% 11.302ms 36.118ms 71.044ms
webui 20 20 0 100.0% 7.118ms 19.406ms 34.772ms
TOTAL 149 149 0 100.0% 10.087ms 34.105ms 71.044ms
Latency percentiles use the nearest-rank method on a sorted slice of observed latencies. p99 reports the 99th-percentile of the observed distribution; a single anomalous outlier in 100 requests will surface only at p100, which is intentionally not printed (read the ERR column to surface anomalies).
CI behavior
This package is a regular cmd/ binary -- no build tags.
CI:
go build ./...-- compilescmd/loadtest/loadtestalong with every other binary. The artifact is discarded.go test -race ./...-- runs the unit tests. They usehttptest.NewServerfor the server side; nothing reaches a real Fly.io endpoint.golangci-lint run-- lints the package as part of the default sweep..github/workflows/ci.yml-- never invokes the resulting binary. The loadtest binary never ships in Docker images and never runs in production deployments.
To run loadtest's own tests locally during development:
go test -race ./cmd/loadtest/...
Out of scope
- Distributed load -- single-process only.
Use
wrk,k6, orvegetafor multi-region distributed traffic. - Recording / replay -- no HAR or HTTP capture. The registry is declarative.
- Write endpoints -- peeringdb-plus is a read-only mirror; the upstream PeeringDB is the source of truth for writes.
- Custom queries -- GraphQL queries are minimal
{ id }selectors; ConnectRPC bodies are{"id":1}/{"limit":10}. If you need richer query payloads, extendcmd/loadtest/registry.go.
Files
main.go-- flag parsing + mode dispatchsurfaces.go--Surfaceenum +Hit()request executorregistry.go-- 13-entity × 5-surface inventory (~114 endpoints)discover.go--discoverIDs+discoverRampIDs(id/asn prefetch)endpoints.go--runEndpointssequential sweep driversync.go--runSync13-step ordered driver +syncOrderparity mirrorsoak.go--runSoakerrgroup × rate-limited worker poolramp.go--runRampper-surface concurrency ramp + inflection detector + markdown emitterreport.go-- per-surface aggregation, percentile calculation, table printer*_test.go-- hermetic httptest-based unit tests
Documentation
¶
Overview ¶
Command loadtest is an operator tool that sends read-only traffic to five interfaces of a peeringdb-plus deployment (default https://peeringdb-plus.fly.dev): /api, /rest/v1, /graphql, ConnectRPC, and /ui. It does not test /mcp. Use it for capacity validation, dashboard warmup, and load reproduction.
SAFETY: this binary is compiled by `go build ./...` but is NEVER invoked by CI, Dockerfiles, or deployment scripts — only by operators against a deployed peeringdb-plus instance. NEVER point --base at https://www.peeringdb.com — upstream PeeringDB enforces 1 req/hour per IP and will block your address.
Four modes are supported:
loadtest endpoints [flags] one-shot inventory sweep across the 5 surfaces loadtest sync [flags] replay the 13-step ordered sync sequence loadtest soak [flags] sustained QPS-capped mixed-surface load loadtest ramp [flags] per-surface inflection-point capacity probe
Build: go build -o loadtest ./cmd/loadtest