Documentation
¶
Overview ¶
Package shadow provides block-by-block comparison between a shadow chain node and a canonical chain node. The comparison is layered:
- Layer 0: Block header hashes (AppHash, LastResultsHash, gas). If these match, the block is identical and deeper layers are skipped.
- Layer 1: Transaction receipt comparison (status, gas, logs, etc.). Run only when Layer 0 fails, to isolate which transactions diverged.
- Layer 2: State diff comparison — logical EVM state (storage/code/nonce) for the keys a block touched, read via EVM RPC on both sides. The load-bearing check for an AppHash-breaking migration shadow, where the committed root diverges by design and only logical state can be compared.
- Layer 3: Execution trace comparison (future).
Index ¶
- Variables
- func RenderMarkdown(r *DivergenceReport) string
- type ChainSnapshot
- type Comparator
- type CompareResult
- type DivergenceReport
- type FieldDivergence
- type KeySource
- type Layer0Result
- type Layer1Result
- type Layer2Result
- type Option
- type StateDivergence
- type StateReader
- type StaticKeySource
- type TouchedAccount
- type TraceKeySource
- type TxDivergence
Constants ¶
This section is empty.
Variables ¶
var ( // BlocksCompared counts blocks the comparator has processed. // rate(...)==0 indicates the comparator has stopped advancing — typically // shadow RPC unreachable or the local node has stopped producing blocks. // pod_name differentiates two shadow candidates for the same chain so // alerts can route to a specific candidate image. BlocksCompared = prometheus.NewCounterVec( prometheus.CounterOpts{ Name: "seictl_shadow_blocks_compared_total", Help: "Total blocks compared between the shadow node and the canonical chain.", }, []string{"chain_id", "pod_name"}, ) // Divergences counts app-hash divergences detected. Increments at most // once per process lifetime — the comparison loop exits on first divergence. // divergence_layer is "0" for header-hash mismatch, "1" when Layer 1 // isolated specific tx-receipt mismatches. Divergences = prometheus.NewCounterVec( prometheus.CounterOpts{ Name: "seictl_shadow_divergences_total", Help: "App-hash divergences detected by the shadow comparator. Increments once per process lifetime since the loop exits on first divergence.", }, []string{"chain_id", "pod_name", "divergence_layer"}, ) )
Functions ¶
func RenderMarkdown ¶ added in v0.0.25
func RenderMarkdown(r *DivergenceReport) string
RenderMarkdown produces a human-readable investigation report from a DivergenceReport. The output is designed to be consumed by engineers or LLM agents analyzing why a shadow node diverged from the canonical chain.
Types ¶
type ChainSnapshot ¶ added in v0.0.25
type ChainSnapshot struct {
Block json.RawMessage `json:"block"`
BlockResults json.RawMessage `json:"blockResults"`
}
ChainSnapshot captures the raw RPC responses from one chain at a specific height. The JSON is preserved verbatim for offline analysis.
type Comparator ¶
type Comparator struct {
// contains filtered or unexported fields
}
Comparator performs block-by-block comparison between a shadow node and a canonical chain node via their RPC endpoints.
func NewComparator ¶
func NewComparator(shadowRPC, canonicalRPC string, opts ...Option) *Comparator
NewComparator creates a Comparator that queries shadowRPC for the local shadow node and canonicalRPC for the reference chain.
func (*Comparator) BuildDivergenceReport ¶ added in v0.0.25
func (c *Comparator) BuildDivergenceReport(ctx context.Context, height int64, comparison CompareResult) (*DivergenceReport, error)
BuildDivergenceReport captures a complete investigation artifact at the given height. It pairs the comparison result with the full raw RPC responses from both chains so engineers can diagnose offline.
func (*Comparator) Close ¶ added in v0.0.59
func (c *Comparator) Close()
Close releases resources held by configured Layer 2 readers / key source. go-ethereum's *ethclient.Client and *rpc.Client expose Close() with NO return, so they do not satisfy io.Closer — assert the no-return shape instead, or the connections leak silently.
func (*Comparator) CompareBlock ¶
func (c *Comparator) CompareBlock(ctx context.Context, height int64) (*CompareResult, error)
CompareBlock performs a layered comparison for the given block height. Layer 0 (block headers) always runs. Layer 1 (transaction receipts) runs when a real divergence is detected, and always in migration mode — where AppHash, the cheap Layer 0 signal, is expected to differ, so the receipt check is the real correctness signal.
type CompareResult ¶
type CompareResult struct {
// Height is the block height that was compared.
Height int64 `json:"height"`
// Timestamp is the UTC time the comparison was performed.
Timestamp string `json:"timestamp"`
// Match is true when all checked layers agree between shadow and canonical.
// In migration mode an expected AppHash divergence does not clear Match.
Match bool `json:"match"`
// MigrationMode records that this comparison treated AppHash divergence as
// expected (an AppHash-breaking migration shadow), keying the verdict on
// execution-results equivalence instead.
MigrationMode bool `json:"migrationMode,omitempty"`
// DivergenceLayer is the first layer that detected a mismatch.
// Nil when Match is true.
DivergenceLayer *int `json:"divergenceLayer,omitempty"`
// Layer0 holds the block-level hash comparison. Always populated.
Layer0 Layer0Result `json:"layer0"`
// Layer1 holds the transaction receipt comparison.
// Only populated when Layer 0 detected a divergence.
Layer1 *Layer1Result `json:"layer1,omitempty"`
// Layer2 holds the logical state-diff comparison. Populated only when a
// state reader and key source are configured (see WithLayer2).
Layer2 *Layer2Result `json:"layer2,omitempty"`
}
CompareResult holds the comparison output for a single block.
func (*CompareResult) Diverged ¶
func (r *CompareResult) Diverged() bool
Diverged returns true when the comparison detected a mismatch at any layer.
type DivergenceReport ¶ added in v0.0.25
type DivergenceReport struct {
Height int64 `json:"height"`
Timestamp string `json:"timestamp"`
Comparison CompareResult `json:"comparison"`
Shadow ChainSnapshot `json:"shadow"`
Canonical ChainSnapshot `json:"canonical"`
}
DivergenceReport is a self-contained investigation artifact for a single app-hash divergence event. It includes the layered comparison result plus the full block and block_results from both chains, giving engineers all the context needed to diagnose why the shadow node diverged without querying external systems.
func FetchReport ¶ added in v0.0.30
func FetchReport(ctx context.Context, downloader seis3.Downloader, bucket, key string) (*DivergenceReport, error)
FetchReport downloads and decodes a DivergenceReport from S3.
type FieldDivergence ¶
type FieldDivergence struct {
Field string `json:"field"`
Shadow any `json:"shadow"`
Canonical any `json:"canonical"`
}
FieldDivergence records a single field-level mismatch in a tx receipt.
type KeySource ¶ added in v0.0.59
type KeySource interface {
TouchedAccounts(ctx context.Context, height int64) ([]TouchedAccount, error)
}
KeySource yields the accounts (and their slots) a block touched, so Layer 2 compares exactly the state real transactions read or wrote at that height.
type Layer0Result ¶
type Layer0Result struct {
AppHashMatch bool `json:"appHashMatch"`
LastResultsHashMatch bool `json:"lastResultsHashMatch"`
GasUsedMatch bool `json:"gasUsedMatch"`
// Raw values are included when there is a mismatch, for diagnostics.
ShadowAppHash string `json:"shadowAppHash,omitempty"`
CanonicalAppHash string `json:"canonicalAppHash,omitempty"`
ShadowLastResultsHash string `json:"shadowLastResultsHash,omitempty"`
CanonicalLastResultsHash string `json:"canonicalLastResultsHash,omitempty"`
ShadowGasUsed int64 `json:"shadowGasUsed,omitempty"`
CanonicalGasUsed int64 `json:"canonicalGasUsed,omitempty"`
}
Layer0Result compares block-level hashes. This is the cheapest check; if all fields match, the block is identical and no further comparison is needed.
func (Layer0Result) Match ¶
func (r Layer0Result) Match() bool
Match returns true when all Layer 0 fields agree.
type Layer1Result ¶
type Layer1Result struct {
// TotalTxs is the number of transactions in the block.
TotalTxs int `json:"totalTxs"`
// TxCountMatch is true when both chains have the same number of txs.
TxCountMatch bool `json:"txCountMatch"`
// Divergences lists the per-transaction differences found.
Divergences []TxDivergence `json:"divergences,omitempty"`
// Indeterminate is set when the receipt comparison could not run (RPC error).
// In migration mode Layer 1 is a load-bearing check, so an indeterminate
// Layer 1 forces the block to fail closed rather than pass silently.
Indeterminate bool `json:"indeterminate,omitempty"`
Error string `json:"error,omitempty"`
}
Layer1Result compares individual transaction receipts within a block. Only populated when Layer 0 fails.
type Layer2Result ¶ added in v0.0.59
type Layer2Result struct {
// AccountsChecked is the number of touched accounts compared.
AccountsChecked int `json:"accountsChecked"`
// KeysChecked is the number of individual state reads compared (storage
// slots plus per-account balance/code/nonce checks).
KeysChecked int `json:"keysChecked"`
// Divergences lists the logical-state mismatches found.
Divergences []StateDivergence `json:"divergences,omitempty"`
// Indeterminate is set when the layer could not be evaluated (a key source
// or state read failed). A migration shadow's load-bearing check could not
// run, so the block must NOT be counted clean — Error carries the cause.
Indeterminate bool `json:"indeterminate,omitempty"`
Error string `json:"error,omitempty"`
}
Layer2Result compares logical EVM state (storage slots, code, nonce) for the keys a block touched, read via EVM RPC on both chains. This is the logical- content truth check; it never compares the committed root, which is schedule-dependent for a migration shadow and so not a correctness oracle.
type Option ¶ added in v0.0.59
type Option func(*Comparator)
Option configures a Comparator.
func WithLayer2 ¶ added in v0.0.59
func WithLayer2(shadowState, canonicalState StateReader, keySource KeySource) Option
WithLayer2 enables logical state-diff comparison: the keySource yields the accounts/slots each block touched, and the two StateReaders (EVM RPC on the shadow and canonical chains) supply their logical values to compare.
func WithMigrationMode ¶ added in v0.0.59
func WithMigrationMode() Option
WithMigrationMode treats AppHash divergence as expected (not a mismatch) and keys the verdict on execution-results equivalence. Use for a shadow running an AppHash-breaking state migration against an un-migrated canonical chain.
type StateDivergence ¶ added in v0.0.59
type StateDivergence struct {
Kind string `json:"kind"` // storage | code | nonce
Addr string `json:"addr"`
Slot string `json:"slot,omitempty"` // set only for storage
Shadow string `json:"shadow"`
Canonical string `json:"canonical"`
}
StateDivergence records a single logical-state mismatch between the shadow and canonical chains. Values are hex for legibility in reports.
type StateReader ¶ added in v0.0.59
type StateReader interface {
StorageAt(ctx context.Context, account common.Address, key common.Hash, blockNumber *big.Int) ([]byte, error)
CodeAt(ctx context.Context, account common.Address, blockNumber *big.Int) ([]byte, error)
NonceAt(ctx context.Context, account common.Address, blockNumber *big.Int) (uint64, error)
BalanceAt(ctx context.Context, account common.Address, blockNumber *big.Int) (*big.Int, error)
}
StateReader reads logical EVM state at a height. Its method set matches go-ethereum's *ethclient.Client, so a real client satisfies it directly; the shadow and canonical sides are two instances. A nil blockNumber means latest.
type StaticKeySource ¶ added in v0.0.59
type StaticKeySource struct {
Accounts []TouchedAccount
}
StaticKeySource compares a fixed, curated set of accounts on every block — the fallback when prestate tracing (debug_) is unavailable, and the hook for a periodic cold-key / hot-contract sweep that the trace source does not cover.
func (StaticKeySource) TouchedAccounts ¶ added in v0.0.59
func (s StaticKeySource) TouchedAccounts(_ context.Context, _ int64) ([]TouchedAccount, error)
type TouchedAccount ¶ added in v0.0.59
type TouchedAccount struct {
Addr common.Address
Slots []common.Hash
CheckCode bool
CheckNonce bool
CheckBalance bool
}
TouchedAccount is the set of state a block touched for one address: the storage slots to compare, and whether the account's balance, code, and nonce should be checked. A KeySource produces these per block (e.g. from a trace).
type TraceKeySource ¶ added in v0.0.59
type TraceKeySource struct {
// contains filtered or unexported fields
}
TraceKeySource derives a block's touched accounts and storage slots from a diff-mode prestate trace (debug_traceBlockByNumber, prestateTracer with diffMode) on an EVM JSON-RPC endpoint. diffMode reports both the pre-state (slots read) and post-state (slots written), so the union covers keys the block read OR modified — not just those read. Requires the debug_ namespace enabled on the endpoint (a non-public, operator-owned node).
Coverage boundary: this is the per-block TOUCHED set. Keys migrated but never touched by any transaction (cold state), and non-EVM Cosmos-module state, are not covered here — that breadth is the corpus harness's job (Arm A) plus a periodic StaticKeySource sweep. Layer 2 over a trace source is a hot-state sampling oracle, not a full-keyspace check.
func NewTraceKeySource ¶ added in v0.0.59
func NewTraceKeySource(evmRPC string) (*TraceKeySource, error)
NewTraceKeySource dials the EVM JSON-RPC endpoint used to fetch prestate traces. The same touched set applies to both chains (identical transactions), so a single endpoint (typically the canonical node) is sufficient.
func (*TraceKeySource) Close ¶ added in v0.0.59
func (t *TraceKeySource) Close()
Close releases the underlying RPC connection. No return value, matching go-ethereum's *ethclient.Client / *rpc.Client Close() so Comparator.Close can treat all closeables uniformly.
func (*TraceKeySource) TouchedAccounts ¶ added in v0.0.59
func (t *TraceKeySource) TouchedAccounts(ctx context.Context, height int64) ([]TouchedAccount, error)
type TxDivergence ¶
type TxDivergence struct {
// TxIndex is the position of the transaction within the block.
TxIndex int `json:"txIndex"`
// Fields lists which receipt fields diverged.
Fields []FieldDivergence `json:"fields"`
}
TxDivergence records a mismatch for a single transaction within a block.