Documentation
¶
Overview ¶
Package blockingest consumes Filecoin block announcements from gossipsub (/fil/blocks/<network>) and installs them into the header store as new heads, giving Lantern 0-1 epoch head-tracking latency without polling an upstream RPC.
This logic previously lived in cmd/lantern (package main) and could not be reused by pkg/daemon (the embedded path used by curio-core). It is extracted here unchanged so both the standalone daemon (cmd/lantern) and the embedded daemon (pkg/daemon) mount the same gossipsub head-tracker. See lantern#40.
Trust model: validation here is intentionally narrow. blockpub already did the CBOR shape + signature-presence check and the gossipsub topic validator; we add a CID re-derive (defense in depth), dedupe, a height fence, and parent-linkage. We do NOT do full Lotus-style block validation (BLS sig / election proof / beacon). Lantern is an F3-anchored light client: the trust path is F3 anchor -> SetHead -> content-addressed state queries, not full block re-execution. The polling Sync agent (chain/header/store) remains the catch-up fallback for anything gossipsub misses.
Concurrency: blockpub's OnBlock callback fires from its own read goroutine. We funnel into a single processor goroutine via a bounded channel so head installs are serialized cleanly against the polling Sync and a runaway peer can't blow our memory.
Index ¶
- Constants
- func StatsLogger(ctx context.Context, ing *Ingestor, pub *blockpub.Publisher, ...)
- type BackfillSource
- type ChainFetcher
- type Ingestor
- func (g *Ingestor) Enqueue(blk *ltypes.BlockMsg)
- func (g *Ingestor) Fresh(within time.Duration) bool
- func (g *Ingestor) ObservedHead() abi.ChainEpoch
- func (g *Ingestor) Run(ctx context.Context)
- func (g *Ingestor) SetChainFetcher(f ChainFetcher)
- func (g *Ingestor) SetHeadAdoptionGate(fn func() bool)
- func (g *Ingestor) SetHeadCorroboration(fn func(cid.Cid) bool)
- func (g *Ingestor) SetParentWalkBackfill(on bool)
- func (g *Ingestor) Stats() Stats
- type Stats
Constants ¶
const DefaultBackfillCap abi.ChainEpoch = 3
DefaultBackfillCap bounds RPC-based inline backfill depth (in epochs) per gossipsub event before deferring to the polling Sync agent. The RPC path (glif) does one round-trip per epoch, so this stays small to avoid hammering the upstream on a single stale gossip arrival.
const DefaultParentWalkCap abi.ChainEpoch = 2880
DefaultParentWalkCap bounds parent-walk (bridge-off, #76) inline backfill depth in epochs. In bridge-off mode there is NO polling Sync catch-up (the polling src is a no-op without an upstream RPC), so this cap is the only path that stitches a cold-booted head from the seeded anchor to the live tip. It has to accommodate the natural anchor→live gap that accrues during init + daemon startup + peer discovery (easily 30-120 epochs on calibration; more if the anchor sat on disk overnight). The chainxchg path handles this efficiently (one iterative request per 900-tipset chunk, capped internally at maxBackfillTipsets); the per-CID bitswap fallback pays a linear FetchBlock cost, still bounded.
Variables ¶
This section is empty.
Functions ¶
func StatsLogger ¶
func StatsLogger(ctx context.Context, ing *Ingestor, pub *blockpub.Publisher, interval time.Duration, logf func(format string, args ...any))
StatsLogger runs a periodic one-line stats summary every interval until ctx is cancelled. Optional; standalone cmd/lantern uses it so operators can confirm gossipsub is carrying the load. logf is the sink (e.g. fmt.Fprintf to stderr or a logger).
Types ¶
type BackfillSource ¶
type BackfillSource interface {
TipsetCIDsByHeight(ctx context.Context, h abi.ChainEpoch) ([]cid.Cid, error)
FetchBlock(ctx context.Context, k cid.Cid) (*ltypes.BlockHeader, error)
}
BackfillSource is the minimal RPC surface the ingestor uses for inline backfill when a gossipsub block arrives at head+N with N>1. The Lantern glif client, gateway client, and combined fetcher all satisfy it.
type ChainFetcher ¶
type ChainFetcher interface {
FetchTipsetChain(ctx context.Context, head []cid.Cid, length int) ([][]*ltypes.BlockHeader, error)
}
ChainFetcher pulls a verified header chain (newest->oldest, level 0 = the tipset whose block CIDs are head) from the p2p swarm. Implemented by net/chainxchg.Client (#91).
type Ingestor ¶
type Ingestor struct {
// contains filtered or unexported fields
}
Ingestor consumes block announcements from gossipsub and installs them into the header store as new heads. One ingestor per daemon; owns the dedup state. Build with New, then call Run once and feed it via Enqueue (which is the blockpub OnBlock callback).
func New ¶
func New(store *hstore.Store, src BackfillSource) *Ingestor
New builds an ingestor wired to the header store. src may be nil; when nil, blocks at head+N>1 are skipped and the polling Sync's backfill path handles them on its next cycle.
func Start ¶
func Start(ctx context.Context, ps *pubsub.PubSub, store *hstore.Store, src BackfillSource, topic string) (*Ingestor, *blockpub.Publisher, error)
Start brings up the blockpub subscription + ingestor goroutine on the given gossipsub topic. Returns the ingestor (for stats/shutdown) and the blockpub publisher, or an error.
Side effects:
- Joins /fil/blocks/<network> on gossipsub
- Starts the ingestor goroutine
The caller owns ctx cancellation for shutdown and may start its own stats-logging loop using Ingestor.Stats() + Publisher.Stats().
func (*Ingestor) Enqueue ¶
Enqueue is the OnBlock callback handed to blockpub. Non-blocking: drops when the processor is behind so the gossipsub read loop never stalls (the polling Sync picks up any dropped head within its cycle).
func (*Ingestor) Fresh ¶
Fresh reports whether the ingestor installed a block within the last `within` duration. Used by the polling Sync (#71) to decide whether gossip is currently keeping the store head live, in which case the Sync skips its upstream-RPC HeadEpoch() poll for that tick. Returns false before the first install (zero timestamp).
func (*Ingestor) ObservedHead ¶
func (g *Ingestor) ObservedHead() abi.ChainEpoch
ObservedHead returns the highest block height the ingestor has successfully installed into the store. This is the gossip layer's view of the chain tip: it tracks the live head (>= the canonical store head, since individual high-epoch blocks can be installed before the canonical head advances contiguously to them). Returns -1 if nothing installed yet.
The polling Sync uses this to make its #71 gossip-fresh skip lag-aware (#83): gossip being "fresh" only means head moved recently, not that head is at the tip. Comparing the store head against ObservedHead lets Sync skip the catch-up poll only when actually at the tip, and run it when gossip is fresh-but-lagging - without paying an upstream HeadEpoch RPC call.
func (*Ingestor) SetChainFetcher ¶
func (g *Ingestor) SetChainFetcher(f ChainFetcher)
SetChainFetcher wires the ChainExchange client as the preferred parent-walk gap resolver (#91). Call before Run.
func (*Ingestor) SetHeadAdoptionGate ¶
SetHeadAdoptionGate wires a predicate consulted before each head adoption. While it returns false (e.g. headcheck reports the running head DIVERGES from the independent-source quorum), the ingestor holds head where it is: incoming blocks are still received and backfilled, but head is not advanced onto an uncorroborated tip. Passing nil re-opens the gate (always adopt). Safe to call before or after Run. (#79 item 2)
func (*Ingestor) SetHeadCorroboration ¶
SetHeadCorroboration wires the per-candidate corroboration predicate (#80 part 2): before adopting a block as the new head, the ingestor asks whether the block has been forwarded by enough distinct peers (see blockpub.CorroborationGate). While the predicate returns false the ingestor holds head where it is and schedules bounded retries (corroborating duplicates arrive within ~1s of first delivery on a healthy mesh). After the retries are exhausted the block is dropped; the polling Sync remains the safety net for head progress, so an uncorroboratable block can delay adoption but never wedge the node. Passing nil disables the check. Safe to call before or after Run.
func (*Ingestor) SetParentWalkBackfill ¶
SetParentWalkBackfill enables the #76 bridge-off backfill strategy, where a gap is filled by CID-walking the gossip block's Parents via bitswap FetchBlock instead of the RPC-shaped height->CID lookup. Call once at construction time before Run.
type Stats ¶
type Stats struct {
Received uint64
Dedup uint64
Installed uint64
Skipped uint64
Rejected uint64
RejectedLighter uint64 // #79: rejected by heaviest-ParentWeight fork choice
HeldDiverged uint64 // #79: head adoptions held while divergence gate closed
HeldUncorrob uint64 // #80: head adoptions held awaiting multi-peer corroboration
Backfilled uint64
BackfillFailed uint64
LastInstallEpoch abi.ChainEpoch
}
Stats is a snapshot of ingestor counters for observability.
Source Files