bench/

bench — live route-multiplexing measurements

One directory per <date>/<develop commit>; every TSV row is one hash-verified transfer through one proxy session, produced by bench.sh:

bench/bench.sh <socks host:port> <sink base url> <bytes> <down|up> [label]

down GETs <sink>/?bytes=N from a skywire cli proxy loadtest serve sink and checks the body's sha256 against the sink's X-Sha256 header; up POSTs N random bytes to <sink>/upload and checks the sink's sha256 of what it received. Columns: label dir bytes speed_Bps http got time_s hash_ok. The first line of each file names the exit, both ends' commits, the proxy session, its route and transport, and the sink, so a row can be reproduced.

Rules (docs/design/route-multiplexing-test-plan.md): frozen exit (its update timer stopped for the set), both ends on the same develop commit, one transfer at a time unless the row says otherwise, references before subjects, no mux change merges without a before/after row here.

Reference sets, unattended

bench/run-refs.sh <exit pk> bench/<date>/<commit> <pins dir> [trials] [sink]

produces ref-direct-stcpr, ref-direct-squicr (a --direct proxy each, with route settings --prefer selecting the carrier) and one ref-via-<short> per pin file <pins>/via-<short>.json (a proxy started with --route, so the session runs on exactly that two-hop leg). Each set also writes <set>.carrier.tsv: per row, the byte deltas of the transport the row is supposed to ride (and, for a pinned route, the exit-side deltas of hop 2 over the whole set), so the carrier claim in the header is checked against the transport counters rather than assumed.

Mux sets, unattended

bench/run-mux.sh <exit pk> <out dir> <pins dir> [trials] [sink] [pin order] takes the stream-level and packet-level sets after the references:

• mux-tunnels-N — proxy start --tunnels N: N independent tunnels, one route group each; the visor steers every extra tunnel onto a different first-hop transport.
• mux-legs-N — one routed proxy, then proxy mux set --legs --prune to exactly the first N pinned two-hop routes as legs of ONE route group. The group's src_port is checked before and after the set (it must not change).

<set>.legs.json is the mux info --json snapshot the set ran on (groups, legs, transport types, remote pks); <set>.carrier.tsv has the per-row byte deltas of every first-hop transport. TUNNELS and LEGS choose the counts (both default to 2 — see the knob table below); [pin order] lists pin shorts best route first.

Given no [pin order], run-mux.sh and run-compose.sh derive one from the run's own numbers instead of ls order (pin_order, bench/lib-pins.sh). A pin is ranked by its paired-ref.tsv probe median, or the suite rank that file records for a candidate it did not re-probe, or the median of its own ref-via-<tok>.tsv — best number first. It is dropped when a number says the route is broken: a probe that ran and did not verify every trial, a failed 50 MB reference cell, or a median under PIN_MIN_MBPS (1 MB/s). It is unranked when nothing in the out dir measured it at all, and an unranked pin is left out unless PIN_ALLOW_UNRANKED=1.

Alphabetical order is not a ranking: on 2026-09-18 it seated via-0255117b, a route whose own reference measured 0.05 MB/s and failed all three 50 MB downloads that hour, in the composition cell's second tunnel. Neither is "unranked": chain AL ran its smoke sets in a dir holding only a copied paired-ref.tsv and no ref-via-*.tsv, and the same route came back as the cell's fourth pin for want of a number. A runner short of ranked pins now fails the set loudly — "need 4 ranked pins for 2x2, have 3" — instead of taking a stray one, and pick-ref.sh records EVERY candidate it probed or skipped (failed ones with 0/N, unprobed ones with 0/0 and whatever rank is known) so that file ranks a bad route as bad rather than leaving it unmeasured. The chosen order and the reason for every pin are printed on stderr.

The carrier list is a union, re-read every row, not the list captured at set start: the app's route groups are read before and after each row and every transport they hold joins the tracked list, which is never shortened. campaign21's cut row left the session holding a replacement group (49172) whose first hop was a transport that did not exist when the list was taken (sudph 93139b56-acc0-00e5-8ca0-cd0bad229d62), so the three 50 MB uploads that rode it show no sender at all in mux-tunnels-2.carrier.tsv rows 14–16. A transport that appears mid-set simply becomes new carrier rows from the row it appeared at, so the columns summarize.sh and verdict.sh read are unchanged; <set>.tps.tsv names each tracked transport once — tp type remote_pk first_seen_row, where row 0 means held at set start — and the set's mux-events line names any route group created after row 1 with its port, first hop and transport type.

Paired references (the 2026-09-17 goal rules)

A reference set measured an hour before the subject is not a bar: the 50 MB download reference was 8.84, then 6.63, then 5.43 MB/s in three consecutive windows on 2026-09-16. So the campaign measures its reference next to every row:

PAIRED=1 (default)  one reference row of the SAME cell runs immediately
                    BEFORE each mux row, on ONE single route, in a second
                    proxy instance (`skysocks-client-ref` on :1081 — never
                    through :1080, which stays the instance under test).
                    The two never transfer at the same time, so a 20-row
                    set is 40 transfers.
PAIRED_REF=<pin|direct|auto>
                    the reference route. `auto` reads `paired-ref.txt`
                    (below), then the newest `ref-*` medians in the out
                    dir, then `drift.tsv`, then falls back to `direct`.
PAIRED=0            restores exactly the un-paired behaviour.

Each set writes <set>.paired.tsv — row, cell, ref MB/s, mux MB/s, ratio, ref_ok, mux_ok, ref_legs — and <set>.paired-rows.tsv, the reference transfers as plain bench.sh rows.

bench/verdict.sh <refs dir> <mux dir> scores the median paired ratio per cell when that file exists, and says paired in its mode column; with no paired file it falls back to the old best-reference bar and says bar. PASS needs

• a download cell at ratio >= 0.95 (1.0 within the goal's 5 % noise band),
• a single upload cell at ratio >= 0.95,
• a composition set (mux-compose-*) not worse than either mux-tunnels-2 or mux-legs-2 at 10 MB, and >= the better of the two at 50 MB and 100 MB — all three paired to the same route in the same run,
• every row hash-verified, and wire/goodput <= 1.2.

Which route the references ride

bench/pick-ref.sh <out dir> <pins dir> [exit pk] [sink]

takes the top three routes of the last full run-refs.sh suite, probes each with two 50 MB downloads now, and writes the winner to <out dir>/paired-ref.txt, which run-mux.sh and run-compose.sh read when PAIRED_REF is unset. Six transfers, about a minute.

That makes the eight-set reference suite a once-a-day route-ordering run rather than the campaign's bar, and it makes bench/drift-probe.sh unnecessary whenever PAIRED_REF is in play: drift asks whether a stale bar has moved, and a paired campaign has no stale bar. The probe is a download only — upload medians repeat within 5 % across runs (direct 50 MB up 9.52 / 9.60 / 9.65 / 10.32), downloads swing by 2x.

The endpoint ceiling (v4, 2026-09-18)

bench/run-ceiling.sh <exit pk> <out dir> [trials] [sink] [pins dir]

starts CEIL_N (3) proxy clients on their own SOCKS ports, warms each, then per trial runs one 50 MB transfer alone and then all N at the same instant — uploads (kind=uplink) first, then downloads (kind=downlink) — every transfer hash-verified against the sink. ceiling.tsv holds kind clients trial bytes sum_MBps route:rate_MBps hashes_ok/n plus a # ceiling line per kind saying whether the sum grew from one client to N, and the ceiling itself is the median of the concurrent sums. verdict.sh reads it from the mux dir (or the refs dir) automatically.

Which routes it rides. The uplink is the local card, so its clients are --direct (9.04 alone / 10.70 concurrent on 2026-09-16). The downlink is not: over three direct clients it read 4.72 alone / 5.04 concurrent while a single download via the best intermediate did 8–9.5 MB/s in the same hour — the direct stcpr path is itself the download bottleneck, and a downlink ceiling taken through it would let the composition criterion pass trivially. So the downlink clients are pinned to the N best distinct routes of the paired ranking (<dir>/paired-ref.tsv when pick-ref.sh has written one, else paired_rank's ref-*.tsv/drift.tsv medians, looked for in the out dir and then its parent): slot 1 the best, slot 2 the second best, and direct last when the ranking holds fewer than N routes. CEIL_ROUTES overrides the list; CEIL_PINS (or the 5th argument) says where the via-<short>.json pins are. Each rate in the file carries its route, and one extra uplink-via row per trial uploads over the best via route alone, so the file says whether the direct uplink is a bottleneck too.

The v4 bars that need it: the two-upload cell is scored against min(ref1 + ref2, 0.95 x the uplink ceiling) and the line names the binding bound; a composition's 50/100 MB download is scored against the better of mux-tunnels-2/mux-legs-2 while that better one stays below 0.9 x the downlink ceiling, and against 0.95 x the ceiling once it does not. Without a ceiling.tsv both keep the pre-v4 rule and print no ceiling row — the link moves by 2x inside an hour, so a ceiling from another campaign is not a bar.

The spread set (criterion 10)

SPREAD=1 bench/run-mux.sh … adds mux-spread-3: the default pool session — no pins, no --tunnels — with SETTINGS="spread.max_share=${SPREAD_SHARE:-0.4} spread.min_routes=${SPREAD_ROUTES:-3}" merged into whatever SETTINGS already holds, 50 MB down and up, paired like every other set, and no cut row. After the set bench/direction.sh runs over it and mux-spread-3.assert.tsv scores criterion 10 from both ends' per-leg counters:

• routes_active — the minimum number of in-scope legs over the set's rows, want >= SPREAD_ROUTES (3);
• max_share — the largest payload-direction share of any one route on any row (reverse bytes on a download, forward bytes on an upload), want <= SPREAD_SHARE + SPREAD_CHUNK_BYTES / SPREAD_SIZE = 0.484 at the defaults: one chunk of slack, because a scheduler placing whole 4 MiB chunks into a 50 MB transfer cannot land on 40.000 %;
• ratio_50down / ratio_50up — the median paired ratio of each cell, want >= SPREAD_RATIO_MIN (0.8);
• hashes — n/n;
• spread_policy — the knobs as proxy settings reported them; a REFUSED or still-pending knob is a FAIL, because that set measured the default policy under the spread set's name.

verdict.sh prints every <set>.assert.tsv it finds, this one included.

The rest of the campaign knobs

Degradation, and the exit resource gate

bench/run-degrade.sh (criterion 6) is a whole set of cut rows; run-mux.sh's CUT_ROW is one. Both take the cut from bench/lib-cut.sh, with the same fences: never the transport to the exit, never a first hop a second route group shares, and never one that cannot be restored to the same transport id. <set>.cut.tsv records the row, the transport, the first hop's full public key, the timestamp, ttfb_after_cut_s and the surviving route group ports before and after.

The cut also never targets the paired reference's own route. campaign21 chose Atlanta's first hop 95839ad0-b588-0b1d-8475-45fba6ae993c under CUT_FENCE=auto while paired-ref.txt named that same route (0371ab4b) for the :1081 reference instance, so the reference lost its route at row 11 and the paired ratios of rows 12–16 measure the mux session against a rebuilt reference rather than the one rows 1–10 saw. Fenced off now: the first-hop transport of the reference route (resolved from the same pin file lib-paired.sh starts the instance on) and every transport the skysocks-client-ref / -ref2 route groups hold. When that leaves no candidate the set carries no cut row — <set>.cut.tsv holds # cut=skipped:<reason> and verdict.sh prints it — because cutting the reference costs every later paired row of the set; CUT_REF_FENCE=0 restores the old choice.

bench/exit-resources.sh <out dir> <label>          # one reading
bench/exit-resources-check.sh <out dir> <set>      # score one set
bench/exit-resources-check.sh <out dir> --slope    # score the whole run

The reading is taken over pty exec on the exit: RssAnon and CPU seconds of the skywire unit's MainPID, the 1-minute load, the idle CPU rate sampled inside the reading, and the visor's own diag.runtime (sys_mb, heap_alloc_mb, num_gc, goroutines) from the same exec, so the OS number and the Go number share an instant. A run opens with one 50 MB warm-up transfer and a warmup reading; each set is then scored on a <set>-settled reading taken EXIT_RES_SETTLE_S (30 s) after its post, and FAILS on settled - pre above 64 MiB (EXIT_RES_RSS_KB), on sys_mb growth above 96 MB (EXIT_RES_SYS_MB), or on CPU per wall second more than half a core above the pre-set idle rate; the run then ends with --slope, which FAILS a least-squares RssAnon drift above EXIT_RES_SLOPE_KB_MIN (2048 KiB/min) over a span of at least 20 minutes.

Why that shape: the exit pays its cold-start heap high-water on whichever set runs first, which is the whole reason campaign21 and the dc23fd9ff smoke each failed set 1 by +66 / +76 MiB and passed every set after it, so the warm-up buys that high-water outside any set's bracket. Go's scavenger hands back what a set borrowed — the smoke returned 52 MiB in 29 idle seconds between two sets — so what a set KEPT is the settled reading, not the post. The per-set delta alone therefore flags warm-up and little else, while the drift that actually walks toward the cgroup ceiling (campaign21: 347 -> 479 MB, 2393 KiB/min over 43 minutes) appears in no single set and is what the slope check is for — and RssAnon over the bar with sys_mb flat underneath it is named in the verdict as the non-Go anonymous class, the mmap'd-bbolt kind that OOM-killed this exit seven times on 2026-09-16.

A failed check keeps every result and makes the run script exit non-zero at the end; a reading that could not be taken is SKIP, not FAIL.

Standby tunnel pool, unattended

bench/run-standby.sh <exit pk> <out dir> <pins dir> [trials] [sink] [pin order] measures the standby tunnel pool: the session holds every disjoint route to the exit as a dialed tunnel, and only --tunnels of them carry traffic.

One set per run, mux-standby-<pool>, where <pool> is the pool size the session SETTLED at — the script starts the default proxy instance exactly as run-mux.sh starts a mux-tunnels-N set (no pins, no extra flag: the pool is on by default), then polls visor state --select mux_route_groups until the group count to the exit has not moved for POOL_QUIET (20 s). The count it finds names the set, so a run whose pool never grew past the active set is recorded as mux-standby-2, marked INVALID, and cannot be mistaken for a full pool. <set>.legs.json is the settled pool in run-mux.sh's shape — groups, legs, hops with full public keys, remote_ip per leg — plus each group's tunnel_role when the visor reports one (STANDBY_POOL=0 is the control run).

PAIRED=1 gives the pool set the same contemporaneous reference every other mux set gets — one reference row of the same cell on one single route, in a second instance on :1081, immediately before each pool row — so verdict.sh scores it by ratio and its <set>.paired.tsv is written like any other. The reference's own route is fenced off the chaos cut. Unset, the set is scored against the day's bar as before; chain AL exported PAIRED=1, this script ignored it, and the pool set came back bar/NOBAR in a smoke dir that holds no ref-* rows.

Rows are run-mux.sh's twenty in the same order (1–5 10 MB down, 6–10 10 MB up, 11–15 50 MB down, 16–20 50 MB up). Row 11 — the first 50 MB download — is the chaos row: once the transfer has moved CHAOS_PCT (40 %) of its bytes, or the deadline cut_deadline derives from the download rate the set has already shown has passed — CHAOS_PCT of the time this row is expected to take, capped at CHAOS_AFTER_S (5 s) — the first-hop transport of one ACTIVE tunnel is removed with tp rm, and the row is timed to the first byte that arrives afterwards.

The deadline is scaled because the byte trigger alone cannot be relied on: the only progress signal is the size of curl's output file, and the client's in-order reassembly hands it multi-megabyte bursts, so a fast row can cross 40 % and 95 % inside one sampling gap. Every chaos row on record fired on the fixed 5 s deadline, and on 2026-09-18 that meant cutting a 9.44 MB/s row at 5.180 s with 48,955,392 of 50,000,000 bytes already in — nothing was left for a promoted tunnel to carry, and the row recorded "no byte arrived after the cut" as a FAIL. A cut that still lands with CHAOS_LATE_PCT (90 %) or more of the object delivered now records ttfb_after_s as late and the assert as INVALID: the row timed the cut, not the pool. The sampling cadence is CHAOS_POLL (0.1 s). The active tunnel is the one tunnel_role names, or, when the visor does not report that field, the one whose first hop actually carried the downloads of rows 1–10. What may be cut is fenced as in run-degrade.sh — one of the pinned hop-1 stcpr transports (so tp add -t stcpr <pk> rebuilds the same id), never the transport to the exit, never one a second group shares — and a pool with no such candidate is INVALID before a row is measured. The transport is re-added right after the row and every pin is swept again at the end; a pin the sweep cannot restore is named, and rig-restore.sh is then the next step.

<set>.chaos.tsv records what was cut (row, transport id, first-hop public key in full, timestamp, the row's before/after goodput, and — since the late-cut fix — cut_trigger (bytes or deadline), pct_at_cut and the measured ttfb_measured_s behind a late). <set>.assert.tsv is the set's pass/fail table: rows 11–15 all hash-verified; every pre-cut route group except the cut one still held afterwards (the no-rebuild test, with any port the pool refills recorded separately); at least one tunnel_promoted — or leg_promoted, and the row says which was found; time to first byte after the cut under CHAOS_TTFB_MAX_S (2) — INVALID, not FAIL, when the cut landed late; and zero reorder wedges at either end, read from the local event ring and both ends' recovery counters.

The standby pool changes what "shape" means

proxy start --standby-pool (#4986) holds more route groups than --tunnels N: the active set plus standby tunnels that are dialed, kept alive and measured on the same 5 s ping but carry no streams, so a tunnel that dies is replaced by a route that already exists instead of one set up from scratch. A live --tunnels 2 therefore comes up with three groups — two active, one standby — and the pool keeps filling one dial at a time after the active set is up.

So every shape check counts the active groups, from the tunnel_role field proxy mux info --json puts on each group. The field is omitempty: when no group carries it (any binary before the pool) every group counts as active and the checks behave exactly as they did. The set header records route_groups=<all> active=<n> standby=<m>, <set>.legs.json keeps the role per group, and run-mux.sh / run-compose.sh wait for the group count to hold still for 10 s (bounded at 60 s, POOL_STABLE_S / POOL_WAIT_S) before snapshotting, then print the pool size they settled at. The carriers are still named from every group, standby included, because a standby leg can be promoted mid-set; the cut row only ever targets an active one.

A tunnels set that finds a different number of ACTIVE groups than --tunnels N asked for now restarts the session once with the asked shape, waits for the pool to settle and judges the re-read, saying so in one line; only a second mismatch is INVALID. Leftover shape is not a measurement: on 2026-09-18 the UP2 cell left three actives behind and mux-tunnels-2 was thrown away over a shape it could simply have re-taken.

Knob work on one deploy: run-variants.sh and screen

A knob decision used to cost a build, a deploy and a campaign — about fifty minutes for one value — although the 115 router knobs (skywire cli route settings, catalogued in pkg/router/routersettings) and the 71 proxy knobs (skywire cli proxy settings, pkg/skysocks/skysettings) are all live. Two tools spend that deploy once.

bench/run-variants.sh <exit pk> <out dir> <pins> [trials] [sink] [order] [variants file] dials ONE session and measures many knob variants through it, writing knobs only between rows. The variants file is one variant per line — <name> <knob=value> … — and line 1, or the line named baseline, is variant A. For each cell (SWEEP_SIZES, default 50 MB, x SWEEP_DIRS, default down up) and each of SWEEP_TRIALS (3) trials it walks the B list and measures A, B, A, B' …, so every B row has a control taken seconds earlier on the same session. The verdict per variant and cell is median(B)/median(A) against SWEEP_BAR (1.0), with every per-pair ratio listed beside it, so no reference route and no second proxy instance are involved: A is the reference.

A key that is in the router catalog is written to the local visor and to the exit over --via dmsg://<exit> — a send window moved on one end only measures half a path; a key in the proxy catalog goes to proxy settings --app <app>. Membership is decided once at the start from route settings --json / proxy settings --json, never from the name, and a key in neither catalog stops the run before a row is measured. Every value is read back after each apply and recorded in the row; a knob whose catalog doc says it is read when a route group is BUILT or when a dial CHOOSES a route is flagged dialtime:<key> instead of being measured as if a live session could feel it. Every touched knob is snapshotted into <out>/sweep/knobs-restore.tsv and put back at the end and from the EXIT/TERM/INT traps.

Artefacts: <out>/<variant>.tsv and <variant>.carrier.tsv are ordinary bench rows, so summarize.sh and verdict.sh read them unchanged; <out>/sweep/ rows.tsv is one line per transfer and <out>/sweep/verdict.tsv the ratio table (ratio, medians, pairs, hash_ok, wire/goodput, PASS/FAIL). The exit RssAnon/CPU gate brackets the whole run as the set sweep.

SWEEP_DRY=1 runs the whole thing against bench/dry-stub-cli.sh and bench/dry-stub-bench.sh — no visor, no network — and bench/selftest-variants.sh asserts the interleaving order, that B's knobs never leak into an A row, the ratio arithmetic, the restore (including after a TERM mid-row), the dial-time note and the refusal of an unknown knob. Run it before touching either script.

bench/screen says which knobs are worth measuring at all:

go run ./bench/screen gen -factors bench/screen/factors-mux.tsv -runs 16 > variants.txt
bench/run-variants.sh <exit> <out dir> $S/pins 3 http://127.0.0.1:18080 "" variants.txt
go run ./bench/screen analyze variants.txt <out dir>

gen emits a Plackett-Burman (-runs 12 or 24) or fractional-factorial (-runs 16) design as a variants file: every column balanced and every pair of columns balanced, so each factor's main effect is estimated from all the runs with the others averaged out. analyze reads the sweep's rows back and prints, per cell, each factor's effect in ratio units, its standard error from the replicate scatter, whether it clears two of them, and which factors deserve bench/tune. It is a filter, not a verdict: a resolution-III design aliases main effects with two-factor interactions, and the verdict is still a paired campaign under verdict.sh.

bench/screen/factors-mux.tsv ships twelve factors — leg.starve_ratio, ecf.max_window_bytes, ecf.window_margin, band.demote_ratio_tight, rack.reorder_factor, reorder.window, tlp.pto_factor, hol.rtt_factor, sbd.enabled, chunk.tunnel_concurrency, chunk.per_tunnel, upload.concurrency — each with a line saying why it is in, and a closing list of the knobs deliberately left out (dial-time and shape knobs, which a single-session sweep cannot move).

bench/run-sweep.sh is the older, coarser form and still has its place: it spends a COMPLETE runner invocation per value of one knob, which is what a shape change (a different --tunnels, a different pin order) needs. run-variants.sh is for knobs that move on a running session.

SETTINGS may name a router knob now

bench/lib-settings.sh — the SETTINGS="k=v …" hook every runner calls once per set — applies the same catalog membership rule: a key that is in the router catalog goes to route settings on the local visor and, over --via dmsg://<exit> (SETTINGS_EXIT, or the runner's own $exit_pk), to the exit's; everything else goes to proxy settings --app <app> as before. Until now such a key was sent to the app, refused there, and took the whole SETTINGS line down with it. The router keys are snapshotted into <set>.route-knobs.tsv and restored on both ends by settings_restore, like ROUTE_SETTINGS. A SETTINGS line of proxy keys only behaves exactly as it did. bench/selftest-settings.sh asserts the split, the snapshot and the restore against the stub CLI.