repair-cchain

Overview ¶

Command repair-cchain is a ONE-TIME, fail-closed, OFFLINE tool to roll a coreth C-Chain accepted tip DOWN to a consensus-finalized height, repairing a proposervm/EVM height inconsistency without re-genesis.

Motivation (Lux mainnet C-Chain, incident 1085xxx): the inner EVM ran AHEAD of consensus finality — the proposervm never created the outer wrappers for the heights above its finality tip (the swallowed-accept bug), so on restart the EVM reports a last-accepted height ABOVE the proposervm's. node v1.34.22 (consensus v1.35.29) CORRECTLY fail-closes on that: proposervm repairAcceptedChainByHeight classifies inner-tip > proposervm-tip as `heightBehind`, a hard fatal, rather than silently resetting its finality pointer (which would wedge the node forever). So the C-Chain will not mount.

The minimal-loss recovery is to roll the EVM accepted tip back DOWN to the proposervm finality floor N. Then on the next boot the proposervm reconciles as `heightMatch` (or `heightAhead`, which SELF-HEALS by rolling its own pointer back to the inner tip). Only the ~handful of never-finalized run-ahead blocks above N are dropped; every balance at height <= N is preserved (no re-genesis).

This is NOT a blind hex edit. On mainnet the C-Chain runs with useStandaloneDatabase=true, so the coreth EVM has its OWN db at the chain data dir (e.g. /data/chainData/network-1/<chainID>/db/<engine>); the shared node db holds only proposervm state (a DIFFERENT physical db this tool never touches — after the EVM rewinds to N, luxd's proposervm repairAcceptedChainByHeight sees its tip ABOVE the inner tip and self-heals via heightAhead). This tool opens the standalone EVM db with the SAME zapdb engine coreth uses (the luxfi/database factory routes zapdb/badgerdb to zapdb.New, which wraps BadgerDB with its own key encoding — never open it with raw badger) and derives coreth's keyspace directly on it (no chainID/"vm" prefix — that wraps only the shared node db). It applies the rewind through coreth's OWN tested boot path: it reconstructs the real core.BlockChain with lastAcceptedHash = N's hash, so loadLastState sets the finalized floor to N and reorgs the run-ahead tip away + reprocesses state@N — binding the C-Chain identity (networkID + blockchain id) so any identity-gated (0x9999) re-execution validates against each block header (a wrong target or identity fails LOUD rather than committing corrupt state). core.FinalizeRewind then commits state@N, repoints the snapshot, and cleans orphaned roots.

It sets BOTH pointers the node reads on boot: the coreth head/last-accepted on disk (via the loadLastState reorg + FinalizeRewind) AND the VM-level acceptedBlockDB[lastAcceptedKey] (via this tool). Since the inner EVM is byte-identical across the fleet, rewind ONE node's DB copy, snapshot it, and restore all nodes uniform.

The DB uses an exclusive LOCK; luxd MUST be stopped on the target before `rewind`. PROVE on a disposable copy first; never run against a live node.

Modes:

inspect : print current EVM last-accepted + head, and the highest committed
          state height at or below the floor (the zero-re-execution target).
plan    : inspect + resolve the exact target N and print the rewind plan
          (blocks dropped), without writing.
rewind  : fail-closed, read-write. Roll the EVM tip down to N, persist the
          VM last-accepted pointer, then re-open and verify head==N,
          last-accepted==N, state@N serveable. Requires --yes.
verify  : re-open read-write and assert the on-disk state is consistent at the
          current last-accepted (used after rewind / by the proof harness).