store

package
v1.9.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 23, 2026 License: Apache-2.0, MIT Imports: 14 Imported by: 0

Documentation

Overview

Package store implements Lantern's persistent header store.

The store is BadgerDB-backed. It keys block headers by their CID (canonical content-addressed lookup) and maintains:

  • Per-epoch indexes: h:epoch:<be8> → []byte (set of header CIDs)
  • Per-CID header blobs: h:cid:<bytes> → CBOR-encoded BlockHeader
  • Canonical tipset per epoch: h:can:<be8> → tipset-key bytes (sorted CIDs)
  • Head epoch: h:head → be8

"Canonical" means: the tipset key currently selected as part of the active chain. On reorg we rewrite the canonical pointers for the affected epochs.

The store cooperates with chain/header (per-block validation) and with a caller-provided beacon Verifier for BLS-signature replay on startup.

Written for Lantern. Not a direct lift; structural inspiration from lotus/chain/store but pure pure-Go and header-only.

Background head-sync agent for the persistent header store.

The Sync agent talks to a Lotus-compatible JSON-RPC source. It polls ChainHead at a configurable interval (default 6s, half of Filecoin's 30s block time), and on every new head epoch:

  1. Walks back from the new head to the most-recent epoch we already have canonical for (or HeadEpoch - MaxBacktrack, whichever is closer) fetching block headers along the way.
  2. For each fetched header, verifies the block CID and parent linkage.
  3. Calls Store.SetHead(newHead), which rewrites canonical pointers and fires OnHeadChange listeners. SetHead itself detects parent-CID mismatch against the prior canonical chain and acts as the reorg trigger.

We do not use a gossipsub or libp2p ChainNotify channel here — that's the job of net/libp2p. The Sync agent is the simple "poll for ChainHead" fallback that works against any Lotus-compatible RPC.

Index

Constants

This section is empty.

Variables

View Source
var ErrNotFound = errors.New("header/store: not found")

ErrNotFound is returned when a header or tipset is missing.

Functions

This section is empty.

Types

type Options

type Options struct {
	// BeaconConfig is an optional DRAND verifier used at startup to
	// re-validate beacon entries on the most-recent N headers before
	// accepting the persisted head as current.
	BeaconConfig *beacon.Config
	// StartupVerifyDepth: how many recent headers to re-verify on Open.
	// 0 means "no verification" (e.g. unit tests).
	StartupVerifyDepth int
}

Options configures Open.

type RPCSource

type RPCSource interface {
	// HeadEpoch returns the current chain head epoch.
	HeadEpoch(ctx context.Context) (abi.ChainEpoch, error)
	// TipsetCIDsByHeight returns the block CIDs that form the canonical
	// tipset at the given epoch.
	TipsetCIDsByHeight(ctx context.Context, h abi.ChainEpoch) ([]cid.Cid, error)
	// FetchBlock returns a single BlockHeader, CID-verified.
	FetchBlock(ctx context.Context, k cid.Cid) (*ltypes.BlockHeader, error)
}

RPCSource is the minimal RPC surface required by Sync. The Lantern glif client and gateway client both satisfy this interface.

type Stats

type Stats struct {
	HeadEpoch abi.ChainEpoch
	OpenedAt  time.Time
}

Stats captures store-level metrics.

type Store

type Store struct {
	// contains filtered or unexported fields
}

Store is a thread-safe persistent header store.

func Open

func Open(path string, opts Options) (*Store, error)

Open opens (or creates) a BadgerDB-backed header store at the given path. Pass path="" to open an in-memory store (useful for tests).

func (*Store) AllHeadersAtEpoch

func (s *Store) AllHeadersAtEpoch(epoch abi.ChainEpoch) ([]*ltypes.BlockHeader, error)

AllHeadersAtEpoch returns every header we've seen at the given epoch (canonical + competing forks). Useful for reorg debugging.

func (*Store) Close

func (s *Store) Close() error

Close closes the underlying BadgerDB.

func (*Store) Get

func (s *Store) Get(c cid.Cid) (*ltypes.BlockHeader, error)

Get returns the block header for the given CID.

func (*Store) GetTipSet

func (s *Store) GetTipSet(tsk ltypes.TipSetKey) (*ltypes.TipSet, error)

GetTipSet resolves a tipset directly by its key (the set of block CIDs). It reads each block header from the store and reassembles the tipset. Returns ErrNotFound if any constituent block is missing. Unlike GetTipSetByHeight this does NOT require the tipset to be on the canonical chain — it serves any tipset whose headers were persisted, which is what callers like ChainGetTipSet(key) need.

func (*Store) GetTipSetByHeight

func (s *Store) GetTipSetByHeight(epoch abi.ChainEpoch) (*ltypes.TipSet, error)

GetTipSetByHeight returns the canonical tipset at the given epoch. Walks backward for null-round epochs (returns the nearest canonical tipset with height <= epoch).

func (*Store) Head

func (s *Store) Head() *ltypes.TipSet

Head returns the current canonical head tipset, or nil if the store is empty.

func (*Store) HeadEpoch

func (s *Store) HeadEpoch() abi.ChainEpoch

HeadEpoch returns the persisted head epoch, or -1 if no head is set.

func (*Store) LatestBeaconEntry

func (s *Store) LatestBeaconEntry(bh *ltypes.BlockHeader) (*ltypes.BeaconEntry, error)

LatestBeaconEntry returns the most recent beacon entry at or before block bh: if bh carries entries, the last one; else walk parents (bounded) until a tipset with entries is found. Mirrors Lotus StateManager.GetLatestBeaconEntry, used by chain/fullvalidate to supply prevBeacon for entry-less blocks (#90). Returns ErrNotFound if none is found within the walk budget.

func (*Store) OnHeadChange

func (s *Store) OnHeadChange(cb func(*ltypes.TipSet))

OnHeadChange registers a listener invoked whenever the canonical head advances. Listeners run synchronously in the goroutine that called SetHead — keep them fast.

func (*Store) Put

func (s *Store) Put(bh *ltypes.BlockHeader) error

Put validates and persists a single BlockHeader. The caller is expected to have an established parent linkage; we verify that the header's declared parent CIDs exist in the store (if Height > 0).

func (*Store) SetHead

func (s *Store) SetHead(_ctx context.Context, ts *ltypes.TipSet) error

SetHead marks ts as the canonical head, atomically rewiring canonical tipset pointers for every epoch from ts down to the most recent common ancestor with the previous head.

On reorg (new head's parent chain diverges from current head's), all canonical-tipset pointers between the divergence epoch and the new head are overwritten.

func (*Store) Stats

func (s *Store) Stats() Stats

Stats returns simple counters for diagnostics.

func (*Store) Tipset

func (s *Store) Tipset(epoch abi.ChainEpoch) (*ltypes.TipSet, error)

Tipset returns the canonical tipset at exactly `epoch` (no walk-back). Returns ErrNotFound if there's no canonical tipset there (e.g. null round).

type Sync

type Sync struct {
	// contains filtered or unexported fields
}

Sync polls an RPCSource and feeds new heads into a Store.

func NewSync

func NewSync(s *Store, src RPCSource, opts SyncOptions) *Sync

NewSync returns a Sync agent that has not been started.

func (*Sync) PollOnce

func (s *Sync) PollOnce(ctx context.Context) error

PollOnce runs one synchronization cycle. Useful for tests + the Phase 6 demo where we want a deterministic single-shot sync.

func (*Sync) SetBlockValidator

func (s *Sync) SetBlockValidator(fn func(ctx context.Context, bh *ltypes.BlockHeader) error, fatal bool)

SetBlockValidator wires the Full-tier per-block consensus validator (#90). fn runs after the tipset-shape check on each assembled block; fatal decides whether a validation error rejects the tipset (true) or is only logged (false, observe mode). Passing nil disables it. Safe to call before/after Start. Light/PDP tiers never call this.

func (*Sync) SetGossipFresh

func (s *Sync) SetGossipFresh(fn func() bool)

SetGossipFresh wires the gossip-freshness callback after construction. The gossipsub ingestor is created after the Sync in both daemon paths, so the callback can't be passed to NewSync; this lets the caller wire it once the ingestor exists (#71). Safe to call before or after Start. Passing nil disables the gossip-aware poll skip.

func (*Sync) SetGossipObservedHead

func (s *Sync) SetGossipObservedHead(fn func() abi.ChainEpoch, tol abi.ChainEpoch)

SetGossipObservedHead wires the gossip-observed-tip callback after construction (#83). Pair with SetGossipFresh to make the #71 skip lag-aware. tol is the SkipLagTolerance in epochs; <=0 uses the default.

func (*Sync) Start

func (s *Sync) Start(ctx context.Context) error

Start launches the polling loop. Returns an error if already started.

func (*Sync) Stats

func (s *Sync) Stats() SyncStats

Stats returns a snapshot of activity counters.

func (*Sync) Stop

func (s *Sync) Stop()

Stop halts the polling loop.

type SyncOptions

type SyncOptions struct {
	// Interval between ChainHead polls. Default 6s.
	Interval time.Duration
	// MaxBacktrack caps how far back the agent walks on each poll
	// (defends against unbounded catch-up cost). Default 30.
	MaxBacktrack abi.ChainEpoch
	// BootstrapDepth is how many tipsets to ingest on the very first
	// poll when the store is empty. Defaults to MaxBacktrack. Operators
	// running against rate-limited public RPC providers (Glif) should
	// set this small (1–3) so startup is quick; subsequent polls catch
	// up incrementally.
	BootstrapDepth abi.ChainEpoch
	// CatchUpChunk caps how many epochs the warm-store catch-up advances
	// in a single poll. When the store falls behind by more than this,
	// each poll ingests a contiguous chunk of this size starting from
	// currentHead+1, so lag decreases monotonically without skipping
	// epochs (issue #33). 0 means "no cap" (advance straight to head).
	// Default 200 — large enough that normal operation (0–2 epochs
	// behind) is a single chunk, bounded enough that a deep backlog
	// against rate-limited RPC is paced over several polls.
	CatchUpChunk abi.ChainEpoch
	// GossipFresh, when set, reports whether an external head source
	// (the gossipsub block ingestor) has advanced the store's head
	// recently — i.e. within roughly the last poll Interval. When it
	// returns true the Sync skips its own RPCSource.HeadEpoch() poll for
	// that tick, because gossip is already supplying the live head with
	// no upstream-RPC dependency. The Sync then acts purely as a
	// catch-up fallback: it resumes polling the RPCSource only when
	// gossip goes quiet (stale store head). This is what keeps a node
	// with healthy gossipsub from hammering a rate-limited public RPC
	// (Glif 429) every Interval (#71). Nil disables the optimization
	// (the Sync polls every tick, the pre-#71 behavior).
	GossipFresh func() bool

	// GossipObservedHead, when set, reports the highest chain epoch the
	// gossip ingestor has observed/installed - i.e. the gossip layer's
	// view of the live tip (>= the canonical store head). It makes the
	// #71 GossipFresh skip LAG-AWARE (#83): GossipFresh alone only tells
	// us the head moved recently, not that it reached the tip. A node
	// whose gossip is fresh-but-lagging (installing some blocks while
	// skipping head+N>1 blocks it can't backfill) would otherwise have
	// Sync skip its catch-up poll forever, wedging the head ~10-20 epochs
	// behind the tip. With this set, Sync skips only when the store head
	// is within SkipLagTolerance of the observed tip; when further behind
	// it runs the catch-up poll regardless of GossipFresh. Returns a
	// negative epoch when no observation is available (treated as "can't
	// confirm we're at the tip" -> don't skip). Nil keeps the pre-#83
	// behavior (skip purely on GossipFresh).
	GossipObservedHead func() abi.ChainEpoch

	// SkipLagTolerance is how many epochs behind the gossip-observed tip
	// the store head may be and still skip the catch-up poll (#83). Only
	// consulted when GossipObservedHead is set. Default 2 (allows the
	// normal 0-1 epoch gossip latency without forcing an RPC poll).
	SkipLagTolerance abi.ChainEpoch

	// StaleResetThreshold is the lag (live head − store head, in epochs)
	// beyond which a warm store is considered hopelessly stale and is
	// re-bootstrapped near the live head instead of attempting a
	// contiguous backfill.
	//
	// This is the #51 "down for a week" fix. The contiguous catch-up
	// (issue #33) walks from currentHead+1 and relies on backfillParents
	// connecting each new tipset to the store's existing chain within
	// MaxBacktrack epochs. After a multi-day outage the gap is tens of
	// thousands of epochs — far beyond MaxBacktrack — so backfill never
	// connects, SetHead never fires, and the head pointer freezes at the
	// stale epoch (exactly the symptom reported 2026-06-18). Past this
	// threshold we stop trying to bridge the gap and re-anchor near live
	// head, the same way a cold start does. No keys, wallets, or other
	// secrets are touched — only rebuildable chain state.
	//
	// 0 disables the behaviour (pure contiguous catch-up, legacy).
	// Default 2880 (~1 day at 30s epochs): a node behind by more than a
	// day re-anchors instead of grinding.
	StaleResetThreshold abi.ChainEpoch
	// OnStaleReset, when set, is fired once when a stale warm store is
	// re-anchored, with (storeHead, liveHead). Used for logging/metrics.
	OnStaleReset func(storeHead, liveHead abi.ChainEpoch)
	// OnReorg is fired (after Store.SetHead) when the new head's parent
	// chain replaced canonical pointers at one or more epochs. The
	// argument is the divergence epoch (the deepest epoch whose
	// canonical pointer changed).
	OnReorg func(divergence abi.ChainEpoch)
}

SyncOptions configures a Sync agent.

type SyncStats

type SyncStats struct {
	Polls              uint64
	SkippedGossipFresh uint64 // #71: ticks where the Glif poll was skipped because gossip was fresh
	HeadAdvances       uint64
	Reorgs             uint64
	StaleResets        uint64
	HeadersAdded       uint64
	LastError          string
	LastHeadEpoch      abi.ChainEpoch
}

SyncStats reports observable Sync activity.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL