complete

package
v0.51.1-payloadless-2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 15, 2026 License: AGPL-3.0 Imports: 23 Imported by: 15

Documentation

Index

Constants

View Source 
const (
	DefaultCacheSize         = 1000
	DefaultPathFinderVersion = 1
)

Variables

This section is empty.

Functions

This section is empty.

Types

type Compactor added in v0.27.0 

type Compactor struct {
	// contains filtered or unexported fields
}

Compactor is a long-running goroutine responsible for: - writing WAL record from trie update, - starting checkpointing async when enough segments are finalized.

Compactor communicates with Ledger through channels to ensure that by the end of any trie update processing, update is written to WAL and new trie is pushed to trie queue.

Compactor stores pointers to tries in ledger state in a fix-sized checkpointing queue (FIFO). Checkpointing queue is decoupled from main ledger state to allow separate optimization and looser coupling, etc. CAUTION: If the forest LRU Cache is used for main state, then ledger state and checkpointing queue may contain different tries. This will be resolved automaticaly after the forest LRU Cache (code outside checkpointing) is replaced by something like a FIFO queue.

func NewCompactor added in v0.27.0 

func NewCompactor(
	l *Ledger,
	w realWAL.LedgerWAL,
	logger zerolog.Logger,
	checkpointCapacity uint,
	checkpointDistance uint,
	checkpointsToKeep uint,
	triggerCheckpointOnNextSegmentFinish *atomic.Bool,
	metrics module.WALMetrics,
) (*Compactor, error)

NewCompactor creates new Compactor which writes WAL record and triggers checkpointing asynchronously when enough segments are finalized. The checkpointDistance is a flag that specifies how many segments need to be finalized to trigger checkpointing. However, if a prior checkpointing is already running and not finished, then more segments than specified could be accumulated for the new checkpointing (to reduce memory). All returned errors indicate that Compactor can't be created. Since failure to create Compactor will end up blocking ledger updates, the caller should handle all returned errors as unrecoverable.

func (*Compactor) Done added in v0.27.0 

func (c *Compactor) Done() <-chan struct{}

Done returns channel which would be closed when Compactor goroutine exits.

func (*Compactor) Ready added in v0.27.0 

func (c *Compactor) Ready() <-chan struct{}

Ready returns channel which would be closed when Compactor goroutine starts.

func (*Compactor) Subscribe added in v0.27.0 

func (c *Compactor) Subscribe(observer observable.Observer)

Subscribe subscribes observer to Compactor.

func (*Compactor) Unsubscribe added in v0.27.0 

func (c *Compactor) Unsubscribe(observer observable.Observer)

Unsubscribe unsubscribes observer to Compactor.

type Ledger 

type Ledger struct {
	// contains filtered or unexported fields
}

Ledger (complete) is a fast memory-efficient fork-aware thread-safe trie-based key/value storage. Ledger holds an array of registers (key-value pairs) and keeps tracks of changes over a limited time. Each register is referenced by an ID (key) and holds a value (byte slice). Ledger provides atomic batched updates and read (with or without proofs) operation given a list of keys. Every update to the Ledger creates a new state which captures the state of the storage. Under the hood, it uses binary Merkle tries to generate inclusion and non-inclusion proofs. Ledger is fork-aware which means any update can be applied at any previous state which forms a tree of tries (forest). The forest is in memory but all changes (e.g. register updates) are captured inside write-ahead-logs for crash recovery reasons. In order to limit the memory usage and maintain the performance storage only keeps a limited number of tries and purge the old ones (FIFO-based); in other words, Ledger is not designed to be used for archival usage but make it possible for other software components to reconstruct very old tries using write-ahead logs.

func NewLedger 

func NewLedger(
	wal realWAL.LedgerWAL,
	capacity int,
	metrics module.LedgerMetrics,
	log zerolog.Logger,
	pathFinderVer uint8) (*Ledger, error)

NewLedger creates a new in-memory trie-backed ledger storage with persistence.

func (*Ledger) CollectStats added in v0.29.0 

func (l *Ledger) CollectStats(payloadCallBack func(payload *ledger.Payload)) (*LedgerStats, error)

func (*Ledger) Done 

func (l *Ledger) Done() <-chan struct{}

Done implements interface module.ReadyDoneAware

func (*Ledger) DumpTrieAsJSON added in v0.13.0 

func (l *Ledger) DumpTrieAsJSON(state ledger.State, writer io.Writer) error

DumpTrieAsJSON export trie at specific state as JSONL (each line is JSON encoding of a payload)

func (*Ledger) FindTrieByStateCommit added in v0.32.0 

func (l *Ledger) FindTrieByStateCommit(commitment flow.StateCommitment) (*trie.MTrie, error)

FindTrieByStateCommit iterates over the ledger tries and compares the root hash to the state commitment if a match is found it is returned, otherwise a nil value is returned indicating no match was found

func (*Ledger) ForestSize 

func (l *Ledger) ForestSize() int

ForestSize returns the number of tries stored in the forest

func (*Ledger) Get 

func (l *Ledger) Get(query *ledger.Query) (values []ledger.Value, err error)

Get read the values of the given keys at the given state it returns the values in the same order as given registerIDs and errors (if any)

func (*Ledger) GetSingleValue added in v0.26.2 

func (l *Ledger) GetSingleValue(query *ledger.QuerySingleValue) (value ledger.Value, err error)

GetSingleValue reads value of a single given key at the given state.

func (*Ledger) HasState added in v0.26.16 

func (l *Ledger) HasState(state ledger.State) (bool, error)

HasState returns true if the given state exists inside the ledger

No error returns are expected during normal operation.

func (*Ledger) InitialState added in v0.10.0 

func (l *Ledger) InitialState() ledger.State

InitialState returns the state of an empty ledger

func (*Ledger) MemSize 

func (l *Ledger) MemSize() (int64, error)

MemSize return the amount of memory used by ledger TODO implement an approximate MemSize method

func (*Ledger) MigrateAt added in v0.31.1 

func (l *Ledger) MigrateAt(
	state ledger.State,
	migration ledger.Migration,
	targetPathFinderVersion uint8,
) (*trie.MTrie, error)

func (*Ledger) MostRecentTouchedState added in v0.16.0 

func (l *Ledger) MostRecentTouchedState() (ledger.State, error)

MostRecentTouchedState returns a state which is most recently touched.

func (*Ledger) Prove 

func (l *Ledger) Prove(query *ledger.Query) (proof ledger.Proof, err error)

Prove provides proofs for a ledger query and errors (if any).

Proves are generally _not_ provided in the register order of the query. In the current implementation, proofs are sorted in a deterministic order specified by the forest and mtrie implementation.

func (*Ledger) Ready 

func (l *Ledger) Ready() <-chan struct{}

Ready implements interface module.ReadyDoneAware it starts the EventLoop's internal processing loop.

func (*Ledger) Set 

func (l *Ledger) Set(update *ledger.Update) (newState ledger.State, trieUpdate *ledger.TrieUpdate, err error)

Set updates the ledger given an update. It returns the state after update and errors (if any)

func (*Ledger) StateByIndex added in v0.47.0 

func (l *Ledger) StateByIndex(index int) (ledger.State, error)

StateByIndex returns the state at the given index -1 is the last index

func (*Ledger) StateCount added in v0.47.0 

func (l *Ledger) StateCount() int

StateCount returns the number of states (tries) stored in the forest

func (*Ledger) Trie added in v0.33.30 

func (l *Ledger) Trie(rootHash ledger.RootHash) (*trie.MTrie, error)

Trie returns the trie stored in the forest

func (*Ledger) TrieUpdateChan added in v0.27.0 

func (l *Ledger) TrieUpdateChan() <-chan *WALTrieUpdate

TrieUpdateChan returns a channel which is used to receive trie updates that needs to be logged in WALs. This channel is closed when ledger component shutdowns down.

func (*Ledger) Tries added in v0.27.0 

func (l *Ledger) Tries() ([]*trie.MTrie, error)

Tries returns the tries stored in the forest

func (*Ledger) ValueSizes added in v0.23.9 

func (l *Ledger) ValueSizes(query *ledger.Query) (valueSizes []int, err error)

ValueSizes read the values of the given keys at the given state. It returns value sizes in the same order as given registerIDs and errors (if any)

type LedgerStats added in v0.29.0 

type LedgerStats struct {
	TrieCount        uint64 `json:"tries_count"`
	NodeCount        uint64 `json:"node_count"`
	InterimNodeCount uint64 `json:"interim_node_count"`
	LeafNodeCount    uint64 `json:"leaf_node_count"`
}

func CollectStats added in v0.38.0 

func CollectStats(tries []*trie.MTrie, payloadCallBack func(payload *ledger.Payload)) (*LedgerStats, error)

type LedgerWithCompactor added in v0.47.0 

type LedgerWithCompactor struct {
	*Ledger
	// contains filtered or unexported fields
}

LedgerWithCompactor wraps a Ledger and its internal Compactor, managing both as a single component. This hides the compactor as an implementation detail. Embedding *Ledger allows automatic delegation of Ledger methods.

func NewLedgerWithCompactor added in v0.47.0 

func NewLedgerWithCompactor(
	diskWAL realWAL.LedgerWAL,
	ledgerCapacity int,
	compactorConfig *ledger.CompactorConfig,
	triggerCheckpoint *atomic.Bool,
	metrics module.LedgerMetrics,
	logger zerolog.Logger,
	pathFinderVersion uint8,
) (*LedgerWithCompactor, error)

NewLedgerWithCompactor creates a new ledger with an internal compactor. The compactor lifecycle is managed by this wrapper. Use Ready() to wait for the ledger and compactor to be ready. triggerCheckpoint is a runtime control signal to trigger checkpoint on next segment finish.

func (*LedgerWithCompactor) Done added in v0.47.0 

func (lwc *LedgerWithCompactor) Done() <-chan struct{}

Done manages shutdown of both ledger and compactor. Overrides the embedded Ledger.Done() to coordinate with the compactor.

func (*LedgerWithCompactor) Ready added in v0.47.0 

func (lwc *LedgerWithCompactor) Ready() <-chan struct{}

Ready manages lifecycle of both ledger and compactor. Signals when initialization (WAL replay) is complete and compactor is ready. Overrides the embedded Ledger.Ready() to coordinate with the compactor.

type PayloadlessCompactor 

type PayloadlessCompactor struct {
	// contains filtered or unexported fields
}

PayloadlessCompactor is the payloadless-mode counterpart of Compactor. It shares the same disk WAL with the full-mtrie compactor (both write the same ledger.TrieUpdate wire format) and produces V7 checkpoints at the configured cadence.

Responsibilities:

  • drain WALPayloadlessTrieUpdate from the ledger's trie-update channel
  • record each update to the shared WAL via realWAL.LedgerWAL.RecordUpdate
  • track an in-memory queue of recent payloadless tries
  • periodically snapshot the queue into a V7 checkpoint via realWAL.StoreCheckpointV7SingleThread
  • prune older V7 checkpoints per the [CheckpointsToKeep] policy
  • honor an external [triggerCheckpointOnNextSegmentFinish] flag for manual checkpointing on the next segment boundary

The implementation deliberately mirrors Compactor so reasoning about one transfers to the other.

func NewPayloadlessCompactor 

func NewPayloadlessCompactor(
	l *PayloadlessLedger,
	w realWAL.LedgerWAL,
	logger zerolog.Logger,
	checkpointCapacity uint,
	checkpointDistance uint,
	checkpointsToKeep uint,
	triggerCheckpointOnNextSegmentFinish *atomic.Bool,
	metrics module.WALMetrics,
) (*PayloadlessCompactor, error)

NewPayloadlessCompactor wires a PayloadlessLedger to a shared [LedgerWAL] for payloadless checkpoint generation. The ledger must have been constructed with a non-nil WAL so that PayloadlessLedger.TrieUpdateChan returns a non-nil channel — otherwise the compactor has no source of updates.

All returned errors indicate that the compactor can't be created and the caller should treat them as unrecoverable.

func (*PayloadlessCompactor) Done 

func (c *PayloadlessCompactor) Done() <-chan struct{}

Done stops the compactor goroutine and waits for the WAL to shut down.

func (*PayloadlessCompactor) Ready 

func (c *PayloadlessCompactor) Ready() <-chan struct{}

Ready starts the compactor goroutine.

func (*PayloadlessCompactor) Subscribe 

func (c *PayloadlessCompactor) Subscribe(observer observable.Observer)

Subscribe registers an observer for checkpoint-completion notifications.

func (*PayloadlessCompactor) Unsubscribe 

func (c *PayloadlessCompactor) Unsubscribe(observer observable.Observer)

Unsubscribe removes a previously-registered observer.

type PayloadlessLedger 

type PayloadlessLedger struct {
	// contains filtered or unexported fields
}

PayloadlessLedger is a fork-aware, in-memory trie-based key/leaf-hash storage.

Unlike Ledger, the underlying trie does not retain payload values: each leaf only retains its hash (HashLeaf(path, value)). Reads therefore return leaf hashes rather than the original values. Use this variant when the caller only needs commitment-level verification (e.g. payloadless execution) and does not need the values themselves.

PayloadlessLedger is fork-aware: any update can be applied at any previous state which forms a tree of tries (forest). The forest is kept entirely in memory and is bounded by `forestCapacity`. When more tries are added than the capacity, the Least Recently Added trie is removed (FIFO).

PayloadlessLedger persists updates to a write-ahead log when constructed with a non-nil realWAL.LedgerWAL; otherwise it operates purely in-memory.

func NewPayloadlessLedger 

func NewPayloadlessLedger(
	wal realWAL.LedgerWAL,
	capacity int,
	metrics module.LedgerMetrics,
	log zerolog.Logger,
	pathFinderVer uint8,
) (*PayloadlessLedger, error)

NewPayloadlessLedger creates a new payloadless trie-backed ledger.

When `wal` is non-nil the ledger:

  • serializes each [Set] update through a WALPayloadlessTrieUpdate sent over [TrieUpdateChan], blocking until the consumer (typically a PayloadlessCompactor) reports the WAL write outcome;
  • exposes a non-nil channel from [TrieUpdateChan].

When `wal` is nil the ledger is purely in-memory: [Set] applies updates synchronously and [TrieUpdateChan] returns nil. This mode is intended for tests and short-lived experimental nodes that don't need persistence.

`capacity` bounds the number of tries kept in the forest; the least-recently added trie is evicted once capacity is exceeded.

func (*PayloadlessLedger) Done 

func (l *PayloadlessLedger) Done() <-chan struct{}

Done implements module.ReadyDoneAware. When a WAL is attached, Done closes the trie-update channel so a compactor can drain pending updates before the WAL is shut down. The WAL itself is closed by the compactor (matching the V6 ordering), so Done returns once channel closure has been signaled.

func (*PayloadlessLedger) FindTrieByStateCommit 

func (l *PayloadlessLedger) FindTrieByStateCommit(commitment flow.StateCommitment) (*payloadless.MTrie, error)

FindTrieByStateCommit iterates over the ledger tries and compares the root hash to the state commitment. Returns a nil trie if no match is found.

func (*PayloadlessLedger) ForestSize 

func (l *PayloadlessLedger) ForestSize() int

ForestSize returns the number of tries stored in the forest.

func (*PayloadlessLedger) GetLeafHashes 

func (l *PayloadlessLedger) GetLeafHashes(query *ledger.Query) ([]*hash.Hash, error)

GetLeafHashes returns leaf hashes for the given keys at the given state, in the same order as `query.Keys()`. A nil entry indicates the path has no allocated register at the given state.

GetLeafHashes replaces the full ledger's Get for payloadless mode, since payload values are not retained.

func (*PayloadlessLedger) GetSingleLeafHash 

func (l *PayloadlessLedger) GetSingleLeafHash(query *ledger.QuerySingleValue) (*hash.Hash, error)

GetSingleLeafHash returns the leaf hash (HashLeaf(path, value)) for the given key at the given state. Returns nil if the path has no allocated register.

GetSingleLeafHash replaces the full ledger's GetSingleValue for payloadless mode, since payload values are not retained.

func (*PayloadlessLedger) HasPaths 

func (l *PayloadlessLedger) HasPaths(query *ledger.Query) ([]bool, error)

HasPaths reports, for each key in `query`, whether the key has an allocated register at the given state. The returned slice is in the same order as `query.Keys()`.

HasPaths replaces the full ledger's ValueSizes for payloadless mode, since the payloadless trie does not retain payload byte sizes.

func (*PayloadlessLedger) HasState 

func (l *PayloadlessLedger) HasState(state ledger.State) (bool, error)

HasState returns true if the given state exists inside the ledger.

No error returns are expected during normal operation.

func (*PayloadlessLedger) InitialState 

func (l *PayloadlessLedger) InitialState() ledger.State

InitialState returns the state of an empty ledger.

func (*PayloadlessLedger) MemSize 

func (l *PayloadlessLedger) MemSize() (int64, error)

MemSize returns the amount of memory used by the ledger. TODO implement an approximate MemSize method.

func (*PayloadlessLedger) MostRecentTouchedState 

func (l *PayloadlessLedger) MostRecentTouchedState() (ledger.State, error)

MostRecentTouchedState returns the state most recently touched.

Expected error returns during normal operation:

  • an error if no trie is stored in the forest

func (*PayloadlessLedger) Prove 

func (l *PayloadlessLedger) Prove(query *ledger.Query) (*ledger.PayloadlessTrieBatchProof, error)

Prove returns a payloadless batch proof for the given keys at the given state. The returned proofs carry leaf hashes rather than full payload values.

Proofs are generally _not_ provided in the register order of the query. In the current implementation, proofs follow the order specified by the underlying payloadless forest implementation.

func (*PayloadlessLedger) Ready 

func (l *PayloadlessLedger) Ready() <-chan struct{}

Ready implements module.ReadyDoneAware. When a WAL is attached, Ready gates on the WAL's own readiness; otherwise it returns an already-closed channel.

func (*PayloadlessLedger) Set 

func (l *PayloadlessLedger) Set(update *ledger.Update) (newState ledger.State, trieUpdate *ledger.TrieUpdate, err error)

Set applies the given update to the ledger and returns the new state and the trie update that was applied. The update payload's `value` bytes are hashed into the trie; the payload's key is not retained.

When the ledger was constructed with a WAL, Set publishes the trie update on [TrieUpdateChan] and waits for the consumer (compactor) to confirm the WAL write; the new trie is computed in parallel with the WAL write. When the ledger was constructed without a WAL, Set applies the update synchronously.

func (*PayloadlessLedger) StateByIndex 

func (l *PayloadlessLedger) StateByIndex(index int) (ledger.State, error)

StateByIndex returns the state at the given index. `-1` returns the last index.

Expected error returns during normal operation:

  • an error if no states are available in the forest
  • an error if the given index is out of range

func (*PayloadlessLedger) StateCount 

func (l *PayloadlessLedger) StateCount() int

StateCount returns the number of states (tries) stored in the forest.

func (*PayloadlessLedger) Trie 

func (l *PayloadlessLedger) Trie(rootHash ledger.RootHash) (*payloadless.MTrie, error)

Trie returns the trie stored in the forest with the given root hash.

Expected error returns during normal operation:

  • an error if no trie with the given root hash is stored in the forest

func (*PayloadlessLedger) TrieUpdateChan 

func (l *PayloadlessLedger) TrieUpdateChan() <-chan *WALPayloadlessTrieUpdate

TrieUpdateChan returns the channel that [Set] uses to publish trie updates to the consumer (typically a PayloadlessCompactor). Returns nil when the ledger was constructed without a WAL — in that case [Set] applies updates synchronously.

The returned channel is closed by PayloadlessLedger.Done so the consumer can drain any in-flight updates.

func (*PayloadlessLedger) Tries 

func (l *PayloadlessLedger) Tries() ([]*payloadless.MTrie, error)

Tries returns the tries stored in the forest.

type PayloadlessLedgerWithCompactor 

type PayloadlessLedgerWithCompactor struct {
	*PayloadlessLedger
	// contains filtered or unexported fields
}

PayloadlessLedgerWithCompactor bundles a PayloadlessLedger with its PayloadlessCompactor so callers can treat the pair as a single ReadyDoneAware component. It is the payloadless analog of LedgerWithCompactor.

Embedding *PayloadlessLedger automatically delegates the public ledger methods (Set, Get*, Has*, Prove, etc.). Ready and Done are overridden so the compactor's lifecycle is coordinated with the ledger's. Lifecycle logging goes through the embedded ledger's logger, so the bundle and the ledger it wraps share one set of log fields.

func NewPayloadlessLedgerWithCompactor 

func NewPayloadlessLedgerWithCompactor(
	diskWAL realWAL.LedgerWAL,
	ledgerCapacity int,
	compactorConfig *ledger.CompactorConfig,
	triggerCheckpoint *atomic.Bool,
	metrics module.LedgerMetrics,
	logger zerolog.Logger,
	pathFinderVersion uint8,
) (*PayloadlessLedgerWithCompactor, error)

NewPayloadlessLedgerWithCompactor constructs a payloadless ledger and a payloadless compactor wired together against the shared realWAL.LedgerWAL.

Boot-time recovery (loading the latest V7 checkpoint and replaying newer WAL segments) is performed by NewPayloadlessLedger, mirroring how the V6 NewLedgerWithCompactor delegates recovery to NewLedger.

Steady-state:

All returned errors indicate the bundle can't be created and the caller should treat them as unrecoverable.

func (*PayloadlessLedgerWithCompactor) Done 

func (lwc *PayloadlessLedgerWithCompactor) Done() <-chan struct{}

Done shuts the bundle down. The ledger closes its trie-update channel so the compactor can drain it; the compactor then closes the WAL. Overrides the embedded PayloadlessLedger.Done.

func (*PayloadlessLedgerWithCompactor) Ready 

func (lwc *PayloadlessLedgerWithCompactor) Ready() <-chan struct{}

Ready waits for both the ledger and the compactor to be ready. Overrides the embedded PayloadlessLedger.Ready so the compactor lifecycle is part of the readiness contract.

type WALPayloadlessTrieUpdate 

type WALPayloadlessTrieUpdate struct {
	Update   *ledger.TrieUpdate        // update to be encoded into the WAL
	ResultCh chan<- error              // compactor sends back the WAL write result
	TrieCh   <-chan *payloadless.MTrie // ledger sends the freshly-built trie to the compactor
}

WALPayloadlessTrieUpdate is the message sent from PayloadlessLedger.Set to a payloadless compactor over the trie-update channel. It mirrors WALTrieUpdate but carries the new *payloadless.MTrie back to the compactor on TrieCh so the compactor can enqueue it in its checkpoint queue.

type WALTrieUpdate added in v0.27.0 

type WALTrieUpdate struct {
	Update   *ledger.TrieUpdate // Update data needs to be encoded and saved in WAL.
	ResultCh chan<- error       // ResultCh channel is used to send WAL update result from Compactor to Ledger.
	TrieCh   <-chan *trie.MTrie // TrieCh channel is used to send new trie from Ledger to Compactor.
}

WALTrieUpdate is a message communicated through channel between Ledger and Compactor.

Source Files

View all Source files

Directories

Path Synopsis
flattener
node
trie
wal
fixtures

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.