27_concurrent_txn

Example 27 — Concurrent Transaction Isolation

What it demonstrates

Transactional isolation, atomicity, and consistency of the WAL-backed Cypher engine, certified under concurrency and the race detector. Many writer goroutines move money between accounts while many reader goroutines continuously observe an invariant that can only hold if the engine isolates in-flight transactions from readers and never loses a concurrent update. It is the only example whose runtime exercises cypher.Engine.BeginTx (multi-statement explicit transactions), cypher.Engine.RunInTx (single-statement autocommit writes), and cypher.Engine.BeginReadTx (read-only transactions) together under contention.

Domain / scenario

A bank clearing-ledger. Each account is a (:ACCOUNT {id, balance}) node whose integer balance is held in cents and keyed by a string account number backed by a range index. A transfer debits one account and credits another by the same amount, so the sum of all balances is invariant — money is neither created nor destroyed. That conserved total is the observable the readers pin.

A seeded generator fixes the whole workload for a given -seed: the accounts and their initial balances, and every transfer (source, destination, amount) assigned to each writer. The ledger is fully capitalised — initial balances are validated to exceed the largest possible aggregate debit on any single account — so no account can ever go negative and every planned transfer commits. That keeps the committed set, and therefore the final per-account state, deterministic: because a transfer is a commutative delta on two accounts, replaying the committed transfers in any order yields the same final balances, which the run computes up front and asserts against after the concurrent phase.

How to run

go run ./examples/27_concurrent_txn                 # small deterministic default
go run ./examples/27_concurrent_txn \
    -accounts 5000 -writers 16 -readers 32 \
    -ops-per-writer 5000 -max-amount 1000 -seed 7   # observable-scale run

Run it under the race detector to use it as a data-race + isolation certification:

go test -race ./examples/27_concurrent_txn/...

Scale and flags

min-initial must be at least writers × ops-per-writer × max-amount so the no-overdraft invariant holds; validate rejects a configuration that violates it. At larger scales keep -max-amount small (as in the example above) so the guarantee is easy to satisfy.

Expected output

Bare lines are deterministic facts pinned by the regression test; lines prefixed with # are volatile telemetry that varies per run and machine.

config.accounts=32
config.writers=4
config.readers=4
config.ops_per_writer=150
config.seed=1
accounts=32
transfers.planned=600
transfers.multi_statement=300
transfers.single_statement=300
initial_total=46625986168
# plan.debit_index_seek=true
transfers.committed=600
final_total=46625986168
conservation.holds=1
lost_updates=0
no_negative_balances=1
total_balance_invariant_holds=1
# run.elapsed=2.28s
# writer.transfers_per_s=262
# writer.mean_acquire_wait=11.45ms
# reader.observations=2414
# reader.observations_per_s=1056
# mem.heap_alloc=1.69 MiB
# scale.writers_1.transfers_per_s=264
# scale.writers_2.transfers_per_s=267
# scale.writers_4.transfers_per_s=265

The headline facts are the ACID certification:

• total_balance_invariant_holds=1 — every reader observation equalled the seeded total; no reader ever saw a debit without its matching credit (isolation).
• conservation.holds=1 and final_total == initial_total — money was neither created nor destroyed; every transfer applied atomically (atomicity).
• lost_updates=0 — the final per-account state matched the deterministic replay; no concurrent read-modify-write interleaving lost a write (consistency / serialisability of writers). Under MVCC this is the fact that certifies write-write conflict DETECTION together with the physical rollback of a refused transaction: writers overlap, collisions happen (the telemetry counts them), and a colliding transfer must leave nothing behind.
• no_negative_balances=1 — the fully-capitalised ledger stayed non-negative.

A single torn observation, lost update, or conservation failure makes run return an error naming the violated property rather than reporting success — the example surfaces a module isolation bug, it never hides one.

Evidence it collects

For a concurrency subject (see docs/examples-standard.md):

• Writer throughput (# writer.transfers_per_s) and reader throughput (# reader.observations_per_s) of the mixed workload.
• Contention (# writer.mean_acquire_wait) — the mean time a writer blocked acquiring a MULTI-STATEMENT write transaction, which still takes the graph's schema barrier exclusively for its whole lifetime (retiring that is rmp #2305) and therefore still rises with the writer count. An autocommit statement does not appear here: it holds the barrier shared and queues behind nobody.
• Write-write conflicts (# writer.conflicts_retried, # writer.conflict_retries_max, # writer.conflict_wait_max) — how many transfer attempts were refused with mvcc.ErrSerializationConflict and retried, the deepest retry chain one transfer needed, and the longest WALL TIME any one transfer spent retrying. This is the observable cost of concurrent writers, and it is the load-bearing evidence that they genuinely overlap: a single-writer engine reports zero because the conflict cannot arise. The retry backoff is sized to a WAL fsync, not to a scheduler yield — a yield loop was measured spinning five attempts inside one fsync, all against the same in-flight version and all with the same stale snapshot.
• Scaling across worker counts (# scale.writers_N.transfers_per_s) — the identical workload at 1, 2, 4 … writers on a fresh store, each level asserting conservation on its own ledger.
• Index seek (# plan.debit_index_seek) — evidence the keyed lookup plans as a NodeByIndexSeek rather than a full label scan.
• Live heap (# mem.heap_alloc).

When scaling up, watch two things. Reader throughput and observation count grow with -readers and are never blocked by a writer at all: a read takes a snapshot and no lock. Writer throughput is bounded by the shape of the transfer — half the transfers are multi-statement and still serialise on the exclusive barrier — and by the conflict rate, since every refused attempt pays a retry. Raising -accounts spreads the transfers over more keys and drives # writer.conflicts_retried down.

The retry bound is a clock, and it says why it fired

The retry chain is bounded so that a persistent conflict fails loudly instead of spinning forever. That bound used to be an attempt count — 24 — and it was the wrong instrument, for the reason rmp #2330 made a project rule after finding the same mistake five times in one sprint:

a bound on waiting for another goroutine, process or network peer must be sized to catch a HANG, never to assert a latency.

An attempt count asserts a latency. The 24 waits summed to roughly 96 ms, while what a transfer actually waits for is the writers ahead of it clearing their fsyncs. #2330's own sweep for siblings covered bolt/server and stopped there, so this bound survived it — and was then caught exactly as the rule predicts. Running examples 04, 17, 20, 21, 27, 35, 36 and 37 together under go test -race, this example failed 2 runs in 6 with 25 serialization conflicts on one transfer, while passing 8 of 8 alone and 8 of 8 under fourteen CPU burners. CPU starvation does not reproduce it; competing fsyncs do, because the start timestamp a retry takes is the contiguous commit frontier (rmp #2298) and that frontier advances only as commits become durable.

The bound is now a wall-clock budget of 30 s, sized from measurement: under the same parallel-WAL load, with the bound lifted, the worst chain observed over eight runs was 20 attempts / 167 ms (# writer.conflict_wait_max reports it every run). The budget therefore sits roughly two orders of magnitude above contention and still catches a wedged object immediately — and a wedged object is not hypothetical: before rmp #2318 gave the vacuum an unconditional wake, "the FIRST transaction to abort on an object made that object permanently unwritable", and this example's writers were what exhausted their chain on it.

Exhausting the budget now prints the conflict chain rather than a bare count, because a bound that fires without evidence is what sent rmp #2333 hunting through 245 million observations for one unattributable number. Everything needed was already in the error — mvcc.Conflict carries the blocking version's timestamp, the losing transaction's snapshot and its id — and the loop used to discard all three. Keeping them classifies the exhaustion into four outcomes:

The first two are engine liveness defects. The last is a property of first-updater-wins itself: it has no queue and no age priority, so a loser's backoff grows while every fresh arrival starts at the floor. retry_test.go holds all four verdicts under test and — the direction that matters — proves a wedged head still fails while a chain of 40 progressing refusals no longer does, which is the case the old attempt count could not pass whatever the engine was doing.

The torn-total gate, and how it explains itself

The conserved total is the example's headline oracle, and a violation of it is rare enough that the run which produces one may be the only run that ever does. It is therefore built to be attributable, not merely loud.

• One instant per observation. A reader takes exactly one snapshot, runs exactly one aggregate, and compares it to the seeded total — a constant. It never samples the graph twice and compares the two, which is the unsound shape that made the rmp #2332 gate report violations that existed at no instant.
• A tear diagnoses itself. When a reader holding an explicit read transaction sees a wrong total, it re-reads every account balance inside the same transaction — therefore at the same pinned instant the aggregate was computed over — and the failure names a verdict:
  • the per-account state is consistent → the aggregate disagreed with the rows it summed, which is an execution defect and not an isolation one;
  • the per-account state is itself inconsistent → a genuine isolation violation, and the deviating accounts are listed with their deltas. A reader using Engine.Run holds no pinned instant to re-read, so it reports UNATTRIBUTED rather than guessing.
• A NULL is an error, not a zero, and an ungrouped sum() must return exactly one row. Both used to be absorbed silently — a NULL became a total of 0, and a multi-row result kept its last row — and both would then have been reported as an isolation violation the engine had not committed.
• The gate is validated on a deliberately broken build. TestTornGate_CatchesADeliberateTear sets an internal negative-control seam that commits a multi-statement transfer's debit and credit as two separate transactions. That is a real tear, and the test asserts the gate both catches it and attributes it to isolation rather than to the aggregate. A gate that has never been shown to fire is not evidence that anything passed.

Key APIs

• cypher.NewEngineWithStore — a WAL-backed engine over a txn.Store.
• cypher.Engine.BeginTx / cypher.ExplicitTx.Exec / Commit / Rollback — multi-statement explicit write transactions (the debit-then-credit transfer).
• cypher.Engine.RunInTx (via RunAny) — single-statement autocommit writes.
• cypher.Engine.BeginReadTx — read-only transactions for the invariant reads.
• cypher.Engine.Run — the concurrent read path used by the other readers.

Further reading

• cypher — package documentation (cypher/exectx.go holds the explicit-transaction and isolation contract).
• docs/isolation-design.md — the isolation model.
• docs/acid-audit.md — the ACID durability/atomicity design.
• Example 17 — Transactional Log — WAL, checkpointing, and crash recovery of the durable store.
• Example 20 — Concurrent Reads — the lock-free read contract of an immutable snapshot.
• Example 25 — Software House API — a full app that serialises engine read/write access behind an HTTP surface.