hopperbench

Overview ¶

Command hopperbench measures hopper's hot paths against a real database and prints machine-readable results, one JSON object per scenario.

hopperbench -database-url postgres://... [-scenarios throughput,insert,copy,latency] [-jobs 50000] [-clients 4]
hopperbench -compare base.jsonl,head.jsonl [-threshold 0.10]

It works in a schema of its own ("hopperbench"), which it creates and drops, so it can point at any database. Numbers depend on the hardware, the Postgres configuration and the network between them; compare runs made on the same setup, such as a PR branch against canary. The -compare mode does that: it reads two files of results, averages the faster half of the runs of each scenario, and exits non-zero if the second is slower than the first by more than the threshold on any scenario. Noise on a shared machine only ever slows a run down, so the faster half is the better estimate; averaging it, rather than taking the single best run, keeps one lucky run from deciding the outcome.

Scenarios:

throughput  no-op jobs preloaded, then drained by -clients clients
mixed       as throughput, with all four priorities interleaved
scheduled   as throughput, with jobs scheduled up to 500ms in the future
tagged      as throughput, with two tags on every job
retry       as throughput, with every job failing its first attempt
workflow    as throughput, in workflows of ten steps (one, then eight after it, then one after those),
            each step recording a small output; -jobs is rounded down to a multiple of ten
workflow-outputs
            as workflow, with every step after the first claimed with its dependencies' outputs
            (StepOpts.Dependencies), which each worker checks
workflow-any
            as workflow, with the last step waiting for any one of the eight in the middle (AfterAny)
insert      InsertMany in batches of 100 (the unnest statement)
copy        InsertMany in batches of 5000 (the COPY path)
latency     commit-to-Work on an idle queue, local wake-up
loaded      commit-to-Work while -clients clients work a stream of inserts at -rate jobs/s for -seconds;
            with -report N it also prints one "loaded-window" line every N seconds (for soaks)
leases      jobs run under job leases, as on remote workers: a tenth of -jobs (at most 10,000), each claimed
            by ID, renewed three times, given a step and finished, by -clients x 16 goroutines; jobs_per_sec
            counts leased attempts. It is for the record: CI's gate does not run it

-history preloads that many rows of history first, to measure the live table's independence from accumulated history.