bench/

lantern bench harness (testbed/bench/)

Reusable performance + memory-leak harness for the HA docker compose cluster (see deploy/compose/). Drives the cluster with ghz, captures Prometheus range queries + Go pprof snapshots, applies per-scenario leak / lifecycle-metric / semantic / producer-performance gates, and renders a Markdown report.

PR CI does not run the wall-clock scenarios. The default PR pipeline runs their schema/contract tests, but not a Compose load run. On a root vX.Y.Z tag, however, docker-publish.yml runs the short search_qualification scenario plus targeted real-h2c Search tests as a blocking qualification. A failed or skipped stage prevents image and GitHub Release publication. Its bounded artifact records the tag, full commit SHA, every stage status, and the scenario report.

Release CI runs a compact canonical sweep. On every vX.Y.Z tag push, the Release workflow runs ./testbed/bench/release.sh — a driver that sweeps the scenarios listed in release-scenarios.txt. Each scenario owns a fresh Compose lifecycle so data, retained high-water state, and scenario-specific cluster env cannot leak into its successor (#1097). To keep wall-time bounded without losing coverage, the sweep is seven scenarios: three fan-out scenarios (broad_rw, broad_mutate, broad_illuminate) that exercise many call paths inside a single steady window (read / write / batch / scan / edge / delete / ScanVertexKeys / CountVerticesByPrefix / idempotent AddEdge, and the Illuminate algorithm/reduction × objective × weighting axes) plus ttl_churn, many_subscribers, search_churn (index decay / thread-safety under churn, #703), and replication_apply_churn (vertexHLC map regression gate, #700 / #705). It produces a fixed-format bench-report.md that is spliced into the GitHub Release notes. This full profiling sweep remains continue-on-error, so a noisy runner cannot block a release; it is separate from the short blocking Search qualification above. See issues #256, #262, #573, #708, #1063, #1097.

Prerequisites

• Docker + docker compose v2
• ghz ≥ 0.117 — drives load over gRPC wire frames. The Lantern server is Connect-only (see #335), but the Connect-Go handlers accept gRPC, gRPC-Web, and Connect on the same h2c socket, and connectrpc.com/grpcreflect exposes the standard grpc.reflection.v1* service the harness uses for descriptor discovery. ghz keeps working unchanged. See #383 for the verification log.
• yq v4 (Go reimplementation)
• jq, curl, bash ≥ 4
• Go (matching go.mod toolchain) — used to build the report renderer

Quick start

# Run any scenario:
./testbed/bench/run.sh write_heavy

# Keep the cluster up after the run (useful for ad-hoc poking):
KEEP_UP=1 ./testbed/bench/run.sh read_heavy

# Reuse an already-running cluster (skip compose up / down):
SKIP_UP=1 KEEP_UP=1 ./testbed/bench/run.sh mixed_rw

# Also capture a 30s CPU profile per replica after the steady phase:
PPROF_CPU=1 ./testbed/bench/run.sh addedge_contention

The exit code folds together the leak gate and any declared metric, semantic, and perf gates (0 = all pass, 1 = at least one failed). Run artifacts are written under testbed/bench/out/<scenario>/<UTC-timestamp>/.

Scenarios

Each YAML declares the phases (warmup, steady, cooldown), the ghz target (call + data_template), optional subscribe and chaos blocks, and the leak-gate thresholds (goroutine_max_delta, heap_alloc_max_delta_mb). The gate evaluates against heap_alloc (post-GC live bytes), forcing a runtime.GC() via /debug/pprof/heap?gc=1 before each snapshot so the reading reflects live memory rather than span-level allocator headroom. The legacy field name heap_inuse_max_delta_mb is still accepted for backward compatibility but produced false-positive verdicts under sustained churn — see issue #248.

For fan-out targets, each target.calls[] entry may declare its own positive integer rps. Otherwise the phase RPS is divided equally. Use explicit RPS when one producer must not borrow throughput from another; search_churn does this so a fast PutVertex cannot conceal a slow Search family.

RPC benchmark payloads that omit Vertex.expiration / Edge.expiration write permanent entries. cluster.default_ttl_seconds configures the server's default TTL for cache-default paths; it does not rewrite explicit RPC payloads. Scenarios that need decay through ghz should include an expiration template in data_template.

Every data_template is schema-checked at ordinary go test ./... time by testbed/bench/scenarios_gate_test.go: it renders each template with ghz-style data and protojson-unmarshals the result against the request message resolved from the call name, so a proto change that orphans a scenario fails the schema PR instead of the next nightly (#934). If you retire or rename a wire field, migrate every scenario that sends it in the same PR.

broad_illuminate additionally has a semantic topology gate. Before warmup, the harness seeds a deterministic graph, then verifies the requested 64-way three-hop BFS shape, tuned PPR touch count, and a 32-member weak-bridge community reduced to a directed arborescence. Its six named producers are gated independently as well as in aggregate, so a cheap BFS cannot hide a PPR or community regression.

Perf gate (perf_gate: block)

A scenario MAY additionally declare throughput/latency floors over its steady-phase producers (#935):

perf_gate:
  min_steady_rps_total: 1500   # floor on SUM of producer rps
  max_p99_ms: 500              # ceiling on the WORST producer p99
  max_non_ok_ratio: 0.02       # ceiling on Σnon-OK / Σcount

Every key is individually optional — gate only the metrics that are stable for the scenario. The aggregation matches the release summary table (testbed/bench/release), the verdict lands in perf_gate.json, and a fail folds into run.sh's exit code exactly like the leak gate: the blocking nightly (bench-nightly.yml) enforces it, the release-time bench stays advisory (continue-on-error, #256/#394).

For a fan-out scenario, perf_gate.producers may define the same thresholds per named target.calls[] producer:

perf_gate:
  producers:
    ppr_tuned:
      min_steady_rps: 50
      max_p99_ms: 1000
      max_non_ok_ratio: 0.02

All named producer gates are conjunctive with the aggregate gate and appear as separate rows in perf_gate.json and the rendered report.

Lifecycle metric gate (metric_gate: block)

metric_gate.metrics compares each replica's unlabeled gauge in the pre/post runtime snapshots. Thresholds on a metric are conjunctive:

metric_gate:
  metrics:
    lantern_search_index_docs:
      max_increase: 256
      max_ratio: 1.25
    lantern_search_index_expiration_purged:
      min_increase: 1
    lantern_search_index_retained_ratio:
      max_post: 3
    lantern_search_index_healthy:
      min_post: 1
      max_post: 1

Available comparisons are max_increase, max_ratio, min_increase, min_post, and max_post. Missing or non-finite samples fail closed. Ratio growth from a zero baseline also fails when post is positive; pair ratio and absolute limits intentionally rather than using ratio as a noise filter. The verdict is written to metric_gate.json and folds into run.sh's exit code.

Search semantic gate (semantic_gate: block)

semantic_gate.kind: search seeds a deterministic fixture before warmup and checks every replica both before and after steady load. It proves a persistent hit, deleted/expired absence, stable ordering, key-prefix filtering, phrase, fuzzy-1/2, prefix-term, ALL, and MIN_SHOULD behavior. It also walks a five-page endpoint-sticky cursor chain with FULL_VERTEX snapshots, rejecting duplicate, gap, truncated-tail, or projection races under concurrent index churn. An HTTP-OK response with empty hits therefore fails. The bounded semantic_pre.json and semantic_post.json artifacts contain only phase, replica endpoint, check count, and verdict—never query text, prefixes, keys, or values.

Sizing rules (all seven release scenarios carry a block sized this way):

• Floors are ratchet floors with ≥2x headroom over the worst nightly baseline — they exist to catch step-change regressions (accidental O(n²), lock convoy, a dropped fan-out), NOT single-digit-% drift, which shared ubuntu-latest runners cannot resolve. Do not tighten them to "just below last night's number".
• Gate p99 only where it is stable. Scenarios whose offered load deliberately saturates the runner (broad_rw, broad_mutate) have queue-depth p99 — observed varying 7x night-over-night — so they gate rps and non-OK only.
• Re-baseline after changing a scenario's mix. Adding/removing a target.calls entry re-splits RPS for entries without an explicit rps and always shifts the offered mix; drop the affected ceilings in the same PR and restore them (from fresh nightly numbers, with headroom) after a few green nightlies.
• When a perf floor legitimately moves (accepted throughput/latency trade-off), adjust it in the same PR and say so in the PR body — same contract as the coverage-floor ratchet in CONTRIBUTING.md.

Output layout

testbed/bench/out/<scenario>/<ts>/
├── report.md                       # rendered summary (start here)
├── leak_gate.json                  # verdict + per-replica deltas + thresholds
├── metric_gate.json                # per-replica pre/post gauge contracts (when declared)
├── semantic_{pre,post}.json        # bounded Search semantic verdicts (when declared)
├── perf_gate.json                  # perf verdict + thresholds + observed (only when perf_gate: declared)
├── runtime_pre.json                # runtime values + unlabeled lifecycle gauges, after warmup
├── runtime_post.json               # same, after cooldown
├── ghz_warmup_<endpoint>.json      # raw ghz results, one per invocation
├── ghz_steady_<...>.json
├── ghz_sub*_<n>.json               # if subscribers were launched
├── prom/
│   ├── _index.ndjson               # {query, file} pairs
│   └── q_NN.json                   # raw Prom /api/v1/query_range output
└── pprof/
    └── <endpoint>__{pre,post}__{heap,goroutine,allocs,mutex,block[,profile]}.pb.gz

Drill-down workflow

1. Verdict & deltas. Open report.md. The leak-gate table shows the pre/post goroutines + heap_alloc + heap_inuse + heap_objects per replica with the Δ vs the scenario threshold. The verdict is driven by heap_alloc (post-GC live bytes); heap_inuse and heap_objects are shown for context. Any cell exceeding the threshold is the first place to look.

2. Allocation source. For a leaking replica, diff its heap profiles:

   go tool pprof -http=:0 \
     -base testbed/bench/out/<sc>/<ts>/pprof/<ep>__pre__heap.pb.gz \
            testbed/bench/out/<sc>/<ts>/pprof/<ep>__post__heap.pb.gz
   

3. Stuck goroutines. Open <ep>__post__goroutine.pb.gz and look for blocked stacks (e.g. on channel send/recv, on a sync.Mutex). For addedge_contention the mutex and block profiles (<ep>__post__mutex.pb.gz, <ep>__post__block.pb.gz) are the primary signal.

4. Tail latency. Cross-reference the ghz p99 column in report.md with the matching Prom histogram queries listed under "Prometheus range queries" (raw data in prom/q_NN.json).

5. Pubsub (#240). many_subscribers and replication_soak exercise the subscription queue. Inspect the per-policy lantern_subscription_dropped_total and lantern_subscription_queue_depth_bucket queries in prom/.

Why only the short qualification is blocking

The full sweep needs multi-minute steady phases, a 10-minute soak, pprof and Prometheus capture, a healthy Docker daemon, and stable host conditions. It is therefore advisory at release time and blocking only in the untruncated nightly. Root tags additionally run the deliberately short, tolerant search_qualification scenario: deterministic semantic and lifecycle failures block, while loose throughput/p99 ratchets reject only step changes and tolerate normal hosted-runner jitter.