wal

func AnyCheckpointFileExists ¶

func AnyCheckpointFileExists(dir string, fileName string) (bool, error)

AnyCheckpointFileExists reports whether any file belonging to a checkpoint with the given fileName exists under dir: the checkpoint header, any of its part files, or any stray file sharing the same prefix (for example a leftover from an interrupted write). It is intended for callers that must refuse to overwrite the checkpoint output, since writing a checkpoint recreates all of these files.

No error returns are expected during normal operation.

func Checkpoints ¶ added in v0.28.1

func Checkpoints(dir string) ([]int, error)

Checkpoints returns all the checkpoint numbers in asc order

func ConvertCheckpointV6ToV7 ¶

func ConvertCheckpointV6ToV7(
	inputDir string,
	inputFileName string,
	outputDir string,
	outputFileName string,
	logger zerolog.Logger,
	nWorker uint,
	stream bool,
) error

ConvertCheckpointV6ToV7 reads a V6 checkpoint at (inputDir, inputFileName), converts it to a V7 (payloadless) checkpoint, and writes it to (outputDir, outputFileName).

Behavior:

• The input V6 part files (header + 17 part files) must all be present.
• The output filename must use the V7 suffix (e.g. "checkpoint.00000100.v7"); a missing or wrong suffix is rejected.
• No output file (including any part file) with the same name may already exist; otherwise the call is rejected and the existing output is left untouched.
• The conversion preserves trie root hashes: a V7 checkpoint round-tripped through this function matches the V6 root hashes exactly.
• On any failure after the checks above, the partially written output is removed.

`stream` selects the conversion strategy:

• false: read the entire V6 forest into memory, convert it, and write the V7 checkpoint. Peak memory is approximately the sum of the V6 trie set and the V7 trie set, so mainnet-scale checkpoints need a host with memory headroom.
• true: stream each part file node-by-node (see [convertCheckpointV6ToV7Stream]). Peak memory is independent of checkpoint size, at the cost of not re-deriving the trie root hashes from the converted nodes.

nWorker controls how many of the 16 subtrie part files are processed in parallel; valid range is [1, 16]. In the non-streaming mode, the V6 read step also reads the 16 subtrie part files concurrently using its own internal worker pool (this function does not gate that), so the total parallelism while running may exceed nWorker briefly during the read→write hand-off.

Expected error returns during normal operation:

• none — all error returns indicate a malformed input, a clobbering output, or a write failure, which are treated as exceptions.

func ConvertCheckpointV7ToV6 ¶

func ConvertCheckpointV7ToV6(
	v7Dir string,
	v7File string,
	execDir string,
	prevCheckpointNum int,
	walFrom int,
	walTo int,
	outputDir string,
	outputFile string,
	logger zerolog.Logger,
	nWorker uint,
) error

ConvertCheckpointV7ToV6 reconstructs a full V6 checkpoint from a V7 (payloadless) checkpoint by re-sourcing every leaf's payload from a previous full V6 checkpoint plus the WAL segments written between that previous checkpoint and the V7 checkpoint.

Rationale: a V7 checkpoint stores only a leaf hash per register, not the payload, so it cannot be turned back into a V6 checkpoint on its own. However, every payload referenced by the V7 checkpoint must exist either in the previous full checkpoint (if the register was not updated since) or in one of the WAL segments written since (if it was). By computing HashLeaf(path, value) for each candidate payload from those two sources and matching it against the leaf hash stored in the V7 checkpoint, the original payload is recovered.

Inputs:

• (v7Dir, v7File): the V7 checkpoint header file to convert, e.g. "checkpoint.00000100.v7". Its number N is parsed from the filename.
• execDir: the standard ledger WAL directory holding both the previous V6 checkpoint part files and the numbered WAL segment files.
• prevCheckpointNum: the previous full V6 checkpoint number M to source unchanged payloads from. If negative, it is auto-discovered as the latest V6 checkpoint in execDir with number strictly less than N. If no such numbered checkpoint exists, it falls back to the V6 root checkpoint, in which case the full WAL range [0, N] is replayed.
• walFrom, walTo: the inclusive WAL segment range to source updated payloads from. If negative, they default to (M+1, N] — i.e. all updates applied after the previous checkpoint up to and including the V7 checkpoint state.
• (outputDir, outputFile): where to write the reconstructed V6 checkpoint.

Memory: the checkpoint is processed one subtrie partition (first path nibble) at a time. For partition i, only that partition's payloads are held in memory while its V6 subtrie part file is written, then released before the next partition. nWorker partitions are processed concurrently (valid range [1, subtrieCount]), trading peak memory for speed.

Limitation: the per-trie regSize (AllocatedRegSize) field is metrics-only and is dropped by the V7 format; it is NOT reconstructed and is written as 0 in every trie root record. A warning is logged. This does not affect trie root hashes or any consensus-critical state; it only affects the LatestTrieRegSize metric until the node rebuilds tries.

The output filename must NOT carry the V7 suffix and no output part file may already exist; otherwise the call is rejected. On any failure, partially written output files are removed.

No error returns are expected during normal operation; all error returns indicate malformed input, missing source data, an unmatched leaf hash, a clobbering output, or an IO failure.

func CopyCheckpointFile ¶ added in v0.28.2

func CopyCheckpointFile(filename string, from string, to string) (
	[]string,
	error,
)

Copy the checkpoint file including the part files from the given `from` to the `to` directory it returns the path of all the copied files any error returned are exceptions

func CreateCheckpointWriterForFile ¶

func CreateCheckpointWriterForFile(dir, filename string, logger zerolog.Logger) (io.WriteCloser, error)

CreateCheckpointWriterForFile returns a file writer that will write to a temporary file and then move it to the checkpoint folder by renaming it.

func DeleteCheckpointFiles ¶

func DeleteCheckpointFiles(dir string, fileName string) error

DeleteCheckpointFiles removes the header file and all part files of the checkpoint with the given file name in `dir`. It is version-agnostic: `fileName` selects the checkpoint, so pass the V6 name ("checkpoint.00000100", "root.checkpoint") or the V7 name (same, suffixed with V7FileSuffix).

Deleting a checkpoint that isn't there is not an error. This makes the function suitable for clearing a partially-written checkpoint left behind by a process that died mid-write: a checkpoint is only complete once its header file exists, so an incomplete one can always be discarded and rewritten from its source.

No error returns are expected during normal operation.

func EncodeDelete ¶

func EncodeDelete(rootHash ledger.RootHash) []byte

func EncodeUpdate ¶

func EncodeUpdate(update *ledger.TrieUpdate) []byte

func FromV6LeafNode ¶

func FromV6LeafNode(v6 *node.Node) (*payloadless.Node, error)

FromV6LeafNode converts a V6 leaf node.Node into the equivalent V7 (payloadless) payloadless.Node. The conversion preserves the node's path, height, and computed hash; the payload value is replaced by the height-0 leaf hash HashLeaf(path, value).

For an unallocated leaf (empty or nil payload), the result is a payloadless leaf with leafHash == nil and the same default-for-height node hash.

Expected error returns during normal operation:

• none — the only failure mode is passing an interim node, which is treated as a programmer error rather than a benign error.

func FromV6Trie ¶

func FromV6Trie(v6 *trie.MTrie) (*payloadless.MTrie, error)

FromV6Trie converts a V6 trie.MTrie into the equivalent V7 (payloadless) payloadless.MTrie. Every node is converted via FromV6LeafNode (leaves) or fromV6InterimNode (interim), preserving the node hashes; consequently the resulting V7 trie has the same root hash as the input V6 trie.

Shared sub-tries in the input (e.g. across a forest of related tries) are converted only once thanks to the visited-node memoization.

No error returns are expected during normal operation.

func FromV6Tries ¶

func FromV6Tries(v6Tries []*trie.MTrie) ([]*payloadless.MTrie, error)

FromV6Tries converts a slice of V6 tries to V7 tries, preserving root hashes. Sub-tries shared across multiple input tries are converted once.

No error returns are expected during normal operation.

func HasRootCheckpoint ¶ added in v0.28.1

func HasRootCheckpoint(dir string) (bool, error)

func HasRootCheckpointV7 ¶

func HasRootCheckpointV7(dir string) (bool, error)

HasRootCheckpointV7 checks if a V7 (payloadless) root checkpoint exists.

func IterateCheckpointNodes ¶

func IterateCheckpointNodes(logger zerolog.Logger, dir string, fileName string, fn IterateNodeFunc) error

IterateCheckpointNodes streams every node of a checkpoint (V6 or V7), invoking fn once per node in descendants-first (post-order DFS) order, the same order in which nodes are written to disk. The whole checkpoint is never loaded into memory: each node is decoded from the raw byte stream and handed to fn without retaining child pointers or payloads.

The version is detected from the checkpoint header file's version bytes; each part file's magic+version bytes are additionally validated while reading. Per-part-file CRC32 checksums are verified, matching the regular checkpoint readers.

Counts produced by fn are over the unique nodes of the whole checkpoint forest (nodes shared between tries are stored, and therefore delivered, exactly once).

While streaming, the trie structure is verified:

• every interim node must reference only already-seen, in-range children (descendants-first ordering); and
• every node must be referenced by some parent interim node or trie root.

To perform these checks without retaining nodes, the iterator keeps two bits per node (a "default node" bit and a "referenced" bit), i.e. O(nodeCount) bits of memory — far smaller than the nodes themselves, but not constant.

Expected error returns during normal operation:

• ErrCheckpointIntegrity: when an interim node references an unknown/forward child, when a node is not referenced by any parent or trie root, or when a node declares an out-of-range height.
• os.ErrNotExist (wrapped): when a checkpoint part file is missing.

func ListCheckpoints ¶ added in v0.28.1

func ListCheckpoints(dir string) ([]int, int, error)

ListCheckpoints returns all the numbers of the checkpoint files (both V6 and V7), and the number of the last checkpoint. Note: it doesn't include the root checkpoint file. For version-specific listing, use ListV6Checkpoints or ListV7Checkpoints.

func ListCheckpointsWithInfo ¶

func ListCheckpointsWithInfo(dir string) ([]CheckpointInfo, *CheckpointInfo, error)

ListCheckpointsWithInfo returns all checkpoint infos and the latest checkpoint info. It detects both V6 and V7 checkpoints based on their filenames. Note: it doesn't include the root checkpoint file.

func ListV6Checkpoints ¶

func ListV6Checkpoints(dir string) ([]int, int, error)

ListV6Checkpoints returns all V6 checkpoint numbers (unsorted) and the latest V6 checkpoint number. Returns -1 as the latest if no V6 checkpoints exist.

func ListV7Checkpoints ¶

func ListV7Checkpoints(dir string) ([]int, int, error)

ListV7Checkpoints returns all V7 checkpoint numbers (unsorted) and the latest V7 checkpoint number. Returns -1 as the latest if no V7 checkpoints exist.

func LoadCheckpoint ¶

func LoadCheckpoint(filepath string, logger zerolog.Logger) (
	tries []*trie.MTrie,
	errToReturn error)

func NumberToFilename ¶

func NumberToFilename(n int) string

func NumberToFilenamePart ¶

func NumberToFilenamePart(n int) string

func NumberToFilenameV7 ¶

func NumberToFilenameV7(n int) string

NumberToFilenameV7 returns the V7 (payloadless) checkpoint filename for a given number. Example: 100 -> "checkpoint.00000100.v7"

func OpenAndReadCheckpointV6 ¶ added in v0.28.1

func OpenAndReadCheckpointV6(dir string, fileName string, logger zerolog.Logger) (
	triesToReturn []*trie.MTrie,
	errToReturn error,
)

OpenAndReadCheckpointV6 open the checkpoint file and read it with readCheckpointV6

func OpenAndReadCheckpointV7 ¶

func OpenAndReadCheckpointV7(dir string, fileName string, logger zerolog.Logger) (
	triesToReturn []*payloadless.MTrie,
	errToReturn error,
)

OpenAndReadCheckpointV7 opens a V7 (payloadless) checkpoint and returns the tries as []*payloadless.MTrie. The file must be a V7 checkpoint — V6 (and any other version) is rejected, both because the V7 reader explicitly validates the V7 magic+version at every part-file header and because V7 files use a different filename suffix (V7FileSuffix) so they're trivially distinguishable on disk.

func OpenAndReadLeafNodesFromCheckpointV6 ¶ added in v0.30.0

func OpenAndReadLeafNodesFromCheckpointV6(
	allLeafNodesCh chan<- *LeafNode,
	dir string,
	fileName string,
	expectedRootHash ledger.RootHash,
	logger zerolog.Logger) (
	errToReturn error)

OpenAndReadLeafNodesFromCheckpointV6 takes a channel for pushing the leaf nodes that are read from the given checkpoint file specified by dir and fileName. It returns when finish reading the checkpoint file and the input channel can be closed. It requires the checkpoint file only has one trie.

func ReadCheckpointFileSize ¶ added in v0.33.10

func ReadCheckpointFileSize(dir string, fileName string) (uint64, error)

ReadCheckpointFileSize returns the total size of the checkpoint file

func ReadCheckpointTriesRootHash ¶

func ReadCheckpointTriesRootHash(logger zerolog.Logger, dir, fileName string) ([]ledger.RootHash, error)

ReadCheckpointTriesRootHash reads the trie root hashes from either a V6 or V7 (payloadless) checkpoint, dispatching by the V7FileSuffix on `fileName`.

Both checkpoint versions may coexist in the same directory, so callers that only have a directory and filename (and not the version) should use this function rather than dispatching on the suffix themselves. Callers that already know which version they want should call ReadTriesRootHash or ReadTriesRootHashV7 directly.

No error returns are expected during normal operation.

func ReadTriesRootHash ¶ added in v0.32.2

func ReadTriesRootHash(logger zerolog.Logger, dir string, fileName string) (
	[]ledger.RootHash,
	error,
)

func ReadTriesRootHashV7 ¶

func ReadTriesRootHashV7(logger zerolog.Logger, dir string, fileName string) (
	[]ledger.RootHash,
	error,
)

ReadTriesRootHashV7 returns the trie root hashes recorded in a V7 (payloadless) checkpoint without decoding any node payloads. It first validates the part-file checksums and then reads only the per-trie metadata records at the tail of the top-trie file.

fileName is the V7 header filename (typically ending in V7FileSuffix).

func RootCheckpointFilenameV7 ¶

func RootCheckpointFilenameV7() string

RootCheckpointFilenameV7 returns the on-disk filename of the V7 (payloadless) root checkpoint. The V7 file lives alongside the V6 root (bootstrap.FilenameWALRootCheckpoint), with the V7 suffix appended, so the two can coexist while a node is being migrated between modes.

func SoftlinkCheckpointFile ¶ added in v0.33.37

func SoftlinkCheckpointFile(filename string, from string, to string) ([]string, error)

SoftlinkCheckpointFile creates soft links of the checkpoint file including the part files from the given `from` to the `to` directory

func StoreCheckpointV5 ¶ added in v0.28.1

func StoreCheckpointV5(dir string, fileName string, logger zerolog.Logger, tries ...*trie.MTrie) (

	errToReturn error,
)

StoreCheckpointV5 writes the given tries to checkpoint file, and also appends a CRC32 file checksum for integrity check. Checkpoint file consists of a flattened forest. Specifically, it consists of:

• a list of encoded nodes, where references to other nodes are by list index.
• a list of encoded tries, each referencing their respective root node by index.

Referencing to other nodes by index 0 is a special case, meaning nil.

As an important property, the nodes are listed in an order which satisfies Descendents-First-Relationship. The Descendents-First-Relationship has the following important property: When rebuilding the trie from the sequence of nodes, build the trie on the fly, as for each node, the children have been previously encountered. TODO: evaluate alternatives to CRC32 since checkpoint file is many GB in size. TODO: add concurrency if the performance gains are enough to offset complexity.

func StoreCheckpointV6 ¶ added in v0.28.1

func StoreCheckpointV6(
	tries []*trie.MTrie, outputDir string, outputFile string, logger zerolog.Logger, nWorker uint) error

StoreCheckpointV6 stores checkpoint file into a main file and 17 file parts. the main file stores:

• version
• checksum of each part file (17 in total)
• checksum of the main file itself the first 16 files parts contain the trie nodes below the subtrieLevel the last part file contains the top level trie nodes above the subtrieLevel and all the trie root nodes.

nWorker specifies how many workers to encode subtrie concurrently, valid range [1,16]

func StoreCheckpointV6Concurrently ¶ added in v0.28.1

func StoreCheckpointV6Concurrently(tries []*trie.MTrie, outputDir string, outputFile string, logger zerolog.Logger) error

StoreCheckpointV6Concurrently stores checkpoint file in v6 in max workers, useful during state extraction

func StoreCheckpointV6SingleThread ¶ added in v0.28.1

func StoreCheckpointV6SingleThread(tries []*trie.MTrie, outputDir string, outputFile string, logger zerolog.Logger) error

StoreCheckpointV6SingleThread stores checkpoint file in v6 in a single threaded manner, useful when EN is executing block.

func StoreCheckpointV7 ¶

func StoreCheckpointV7(
	tries []*payloadless.MTrie, outputDir string, outputFile string, logger zerolog.Logger, nWorker uint,
) error

StoreCheckpointV7 stores a payloadless checkpoint into a header file and 17 part files. The on-disk layout (header + 16 subtrie parts + top-trie part) mirrors V6, but each node and trie record is encoded by the payloadless flattener (payloadless.EncodeNode, payloadless.EncodeTrie) — leaves carry a 32-byte leaf hash, not a full payload.

nWorker specifies how many subtries to encode concurrently; valid range is [1,16].

func StoreCheckpointV7Concurrently ¶

func StoreCheckpointV7Concurrently(tries []*payloadless.MTrie, outputDir string, outputFile string, logger zerolog.Logger) error

StoreCheckpointV7Concurrently stores a V7 (payloadless) checkpoint using up to 16 worker goroutines to encode subtries in parallel.

func StoreCheckpointV7SingleThread ¶

func StoreCheckpointV7SingleThread(tries []*payloadless.MTrie, outputDir string, outputFile string, logger zerolog.Logger) error

StoreCheckpointV7SingleThread stores a V7 (payloadless) checkpoint in a single-threaded manner.

func VerifyCheckpointHashes ¶

func VerifyCheckpointHashes(logger zerolog.Logger, dir string, fileName string, nWorker uint) error

VerifyCheckpointHashes verifies the cryptographic integrity of every node in a checkpoint (V6 or V7) by recomputing each node's hash and comparing it against the hash stored alongside the node on disk:

• For a leaf node, the hash is recomputed from its content: the payload value (V6) or the stored leaf hash (V7). This is the streaming equivalent of the per-leaf check performed by trie.MTrie.IsAValidTrie / node.Node.VerifyCachedHash.
• For an interim node, the hash is recomputed as HashInterNode of its two children's hashes (using the height-appropriate default hash for an empty child).

Nodes are streamed in descendants-first (post-order DFS) order, so every child's hash is verified and recorded before its parent is checked. A correct subtrie root hash therefore transitively attests the whole subtrie. The full forest is never materialized: only one 32-byte hash per node is retained (no node objects, no payloads), which is the improvement over loading the checkpoint and calling trie.MTrie.IsAValidTrie.

The 16 subtrie part files are verified concurrently using up to nWorker goroutines; nWorker must be in [1, 16]. The (small) top-trie part file is then verified single-threaded using the subtrie node hashes. Per-part-file CRC32 checksums and magic/version bytes are validated while reading, matching the regular checkpoint readers.

Expected error returns during normal operation:

• ErrCheckpointHashMismatch: when a node's stored hash does not match its recomputed hash.
• ErrCheckpointIntegrity: when an interim node references an out-of-range or forward child index.
• os.ErrNotExist (wrapped): when a checkpoint part file is missing.