benchmark

Benchmark

Runs query scenarios against a real database and outputs markdown, JSON, or benchfmt timing data. Markdown reports include warm-up row counts, path-heavy scenarios can report distinct and duplicate returned path rows, and PostgreSQL explain capture includes translated SQL, plan text, and optimizer rule/lowering metadata in JSON output.

Usage

# Default datasets (base, fixed_suffix_expansion_fanout, and traversal_shapes)
go run ./cmd/benchmark -connection "postgresql://dawgs:dawgs@localhost:5432/dawgs"

# Traversal shape dataset only
go run ./cmd/benchmark -connection "..." -dataset traversal_shapes

# Fixed-suffix expansion fanout dataset with PostgreSQL EXPLAIN diagnostics
go run ./cmd/benchmark -connection "..." -dataset fixed_suffix_expansion_fanout -json-output report.json -explain

# Local dataset (not committed to repo)
go run ./cmd/benchmark -connection "..." -dataset local/phantom

# Default + local dataset
go run ./cmd/benchmark -connection "..." -local-dataset local/phantom

# Neo4j
go run ./cmd/benchmark -driver neo4j -connection "neo4j://neo4j:password@localhost:7687"

# PostgreSQL connection-local translation cache
go run ./cmd/benchmark -driver pg -connection "..." -iterations 10

# Cold and warm concurrent cache measurements (4 workers × 20 samples)
go run ./cmd/benchmark -driver pg -connection "..." -dataset traversal_shapes -pg-min-conns 0 -pg-max-conns 4 -workers 4 -warmup 0 -iterations 20
go run ./cmd/benchmark -driver pg -connection "..." -dataset traversal_shapes -pg-min-conns 0 -pg-max-conns 4 -workers 4 -warmup 2 -iterations 20

# Benchmark the guarded inline predecessor-DAG executor without enabling it in production
go run ./cmd/benchmark -driver pg -connection "..." -dataset traversal_shapes -pg-shortest-path-executor 'ASP-I1-U-DAG+MAT-M0' -pg-plan-cache-mode auto -iterations 20

# Exercise one verified manifest-authorized query through the real V2 policy path
go run ./cmd/benchmark -driver pg -connection "..." -dataset traversal_shapes -pg-traversal-policy-manifest .coverage/promotion.json -pg-traversal-policy-generation 7 -pg-plan-cache-mode auto -iterations 20

# Derive a non-promotional exact SQL anchor for one provisional traversal-policy bucket
go run ./cmd/benchmark -driver pg -connection "..." -dataset traversal_shapes -pg-traversal-policy-preflight-manifest .coverage/provisional.json -pg-traversal-policy-preflight-output .coverage/policy-preflight.json

# Save to file
go run ./cmd/benchmark -connection "..." -output report.md

# Save markdown and JSON for quality baseline comparison
go run ./cmd/benchmark -connection "..." -output report.md -json-output report.json

# Emit benchfmt for benchstat
go run ./cmd/benchmark -connection "..." -format benchfmt -output report.bench

Flags

Use -format benchfmt when comparing scenario timings with benchstat. Each timed scenario iteration is emitted as a separate ns/op sample so two benchmark runs can be compared directly.

pg constructs the connection-local runtime with the selected pool and cache settings, uses the default 64-entry cache per physical PostgreSQL connection, and supports PostgreSQL EXPLAIN capture. Its JSON and Markdown reports also include query-text-free translation-cache, traversal-workspace, and prepared-statement counters, structured PostgreSQL planning/execution timings, and configured pool limits. For shortest paths it also reports query-text-free parse, cache/bind, translation, formatting, and dispatch totals plus shared-template L2 activity. Use a cold (-warmup 0) and warm (-warmup 2) run with the same worker count and pool configuration to measure cache effectiveness; do not compare their latency distributions without accounting for the intentionally different warm-up state.

-pg-shortest-path-executor is deliberately a tool-only comparison mode: it retranslates at the benchmark boundary and bypasses production policy selection. -pg-traversal-policy-manifest instead installs the supplied document on the newly opened PostgreSQL driver, runs at Repeatable Read, and leaves translation, SQL-anchor verification, cache lookup, and fallback selection to the driver. The report identifies this as production_policy and records only the policy generation and manifest SHA-256, never its raw contents.

Before using manifest policy mode, verify the complete evidence closure with go run ./cmd/graphbench -promotion-manifest <path>. The driver fails closed on an invalid candidate, manifest, query set, snapshot, or SQL anchor, but the benchmark command does not treat a digest-shaped document as promotion proof. The current policy contract authorizes exactly one query digest, so policy mode requires -dataset and runs exactly one matching parameterized Cypher scenario. -explain is intentionally unavailable in this mode because the standalone explainer would otherwise bypass the live policy gate and report a different statement.

When a formal manifest needs its candidate SQL anchor, use -pg-traversal-policy-preflight-manifest with the same selected benchmark dataset. Its provisional manifest supplies the candidate, selector, caps, and single query bucket; the command loads the graph and renders that exact translation, then writes only the query and SQL SHA-256 values plus translation metadata. It does not install a traversal policy, execute the candidate SQL, or create verification evidence. Copy the SQL hash into the provisional manifest and recapture the complete GraphBench evidence closure before using the resulting document with policy mode. The output path must be new: preflight refuses to overwrite either the provisional manifest or an earlier record.

The committed default datasets are base, fixed_suffix_expansion_fanout, and traversal_shapes. traversal_shapes covers chain, fanout, bounded cycle, disconnected, edge-kind-selective, and multi-path shortest-path traversal shapes. Scenarios with declared expected row counts fail before reporting timings if a query returns the wrong result shape.

Example: Neo4j on local/phantom

$ go run ./cmd/benchmark -driver neo4j -connection "neo4j://neo4j:testpassword@localhost:7687" -dataset local/phantom

Example: PG on local/phantom

$ export CONNECTION_STRING="postgresql://dawgs:dawgs@localhost:5432/dawgs"
$ go run ./cmd/benchmark -dataset local/phantom