Documentation
¶
Overview ¶
Package devenv boots a complete local Substreams stack — a dummy blockchain in a container, plus tier1 and tier2 running in-process — against which requests can be issued.
The same setup backs both the end-to-end tests and `substreams tools devenv`, so what a developer watches locally is the code path CI exercises.
It lives here rather than in firehose-core — the natural home, since that is where the reader, merger and relayer come from — because firehose-core depends on substreams, so a command over there could only ever run the substreams version it pins, never the working tree. Tier1 and tier2 are built in this module, so this side of the dependency edge is the only one that can run your changes.
Index ¶
- Constants
- Variables
- func FindFreePort() (int, error)
- func RelayerEndpoint(ctx context.Context, container testcontainers.Container) (string, error)
- func StartDummyBlockchain(ctx context.Context, config ChainConfig) (testcontainers.Container, error)
- func StartTier1(ctx context.Context, config Tier1Config, logger *zap.Logger) (*app.Tier1App, string, error)
- func StartTier2(ctx context.Context, config Tier2Config, logger *zap.Logger) (*app.Tier2App, string, error)
- func WaitMergedBlocks(ctx context.Context, dataDir string, upTo uint64, timeout time.Duration) error
- type ChainConfig
- type Tier1Config
- type Tier2Config
Constants ¶
const BlockType = "sf.acme.type.v1.Block"
BlockType is the only chain this stack knows about: the dummy blockchain used across the end-to-end tests.
const DefaultImage = "ghcr.io/streamingfast/dummy-blockchain:1cea671"
DefaultImage is the dummy blockchain image the end-to-end tests are pinned to.
Variables ¶
var Cmd = &cobra.Command{ Use: "devenv", Short: "Run a local Substreams stack against a dummy blockchain", Long: `Boots a dummy blockchain, a tier1 and a tier2 built from this source tree, prints the tier1 endpoint and stays up until interrupted. Requires a working Docker daemon. A large genesis burst against a small state bundle size gives thousands of segments to chew through, which is what makes backprocessing long enough to observe. The state store lives under --data-dir, so a second run of the same request finds everything cached and returns instantly; delete that directory to get a cold backprocess back. The stack logs every block it reads at info level, which buries the endpoint banner. Set DLOG=warn to quiet it down, or DLOG=debug when the stack itself is what misbehaves.`, RunE: runDevenv, Args: cobra.NoArgs, SilenceUsage: true, }
Cmd boots the stack and blocks. It is registered under `substreams tools`.
Functions ¶
func FindFreePort ¶
FindFreePort asks the OS for an available TCP port.
func RelayerEndpoint ¶
RelayerEndpoint resolves the host-side address of the container's relayer.
func StartDummyBlockchain ¶
func StartDummyBlockchain(ctx context.Context, config ChainConfig) (testcontainers.Container, error)
StartDummyBlockchain runs a reader-node/merger/relayer container producing the dummy chain.
func StartTier1 ¶
func StartTier1(ctx context.Context, config Tier1Config, logger *zap.Logger) (*app.Tier1App, string, error)
StartTier1 boots a tier1 on a free port and waits for it to become ready.
func StartTier2 ¶
func StartTier2(ctx context.Context, config Tier2Config, logger *zap.Logger) (*app.Tier2App, string, error)
StartTier2 boots a tier2 on a free port and waits for it to become ready.
func WaitMergedBlocks ¶
func WaitMergedBlocks(ctx context.Context, dataDir string, upTo uint64, timeout time.Duration) error
WaitMergedBlocks blocks until the merger has bundled blocks covering upTo.
Tier1 only reports ready once its block hub can link incoming live blocks, and the hub bootstraps from merged blocks. The reader writes a genesis burst far faster than the merger bundles it — tens of thousands of one-block files against a merger doing roughly a hundred blocks a second — so starting tier1 straight after the container gives it a hub that cannot link anything, and it times out reporting "block not linkable after one-block lookup".
Types ¶
type ChainConfig ¶
type ChainConfig struct {
// Image defaults to DefaultImage.
Image string
// TmpDir is bind-mounted as the firehose storage directory, and is also where tier1 keeps
// its merged blocks and its state store. Wiping it between runs is what forces a cold
// backprocess.
TmpDir string
// Burst is the number of blocks produced immediately at genesis. This is the knob that
// decides how much there is to backprocess.
Burst int
// BlockRate is the number of blocks per minute produced after the burst.
BlockRate int
// ExtraReaderArgs are appended to the reader node arguments verbatim.
ExtraReaderArgs string
// StartupTimeout bounds the wait for the container to serve. A large Burst pushes this
// out — the node produces every genesis block before it starts serving — so it scales
// with the burst when left at zero.
StartupTimeout time.Duration
}
ChainConfig describes the dummy blockchain to run.
type Tier1Config ¶
type Tier1Config struct {
TmpDir string
// RelayerEndpoint is the block stream tier1 reads live blocks from, normally the mapped
// 10014 port of the dummy blockchain container.
RelayerEndpoint string
// Tier2Endpoint is where subrequests are sent.
Tier2Endpoint string
// Tier2Secret must match the tier2 secret when one is configured.
Tier2Secret string
// StateBundleSize is the segment size. Small values mean many small jobs, which is what
// makes a backprocess visible: 100 over a 200k block burst is 2000 segments per stage.
StateBundleSize uint64
// MaxSubrequests is how many tier2 jobs may run at once.
MaxSubrequests uint64
// MaxWorkersPerSession caps the workers a single request may hold, defaulting to
// MaxSubrequests. Lower it to observe a request that is throttled by the session pool
// rather than by the job scheduler.
MaxWorkersPerSession uint64
// MetricsPrefix distinguishes the metric set when several tier1s run in one process.
MetricsPrefix string
LiveBackFillerFinalBlockDelay uint64
// ReadyTimeout bounds the wait for the app to report ready, two minutes when left at zero.
// Tier1 bootstraps its block hub from the merged blocks on disk before it answers, so a
// large genesis burst pushes this well past the handful of seconds a small one needs.
ReadyTimeout time.Duration
// FoundationalStoresConfigPath is a JSON file of identifier → endpoint.
FoundationalStoresConfigPath string
// HostedStoreRegistryAddress is the control-plane gRPC address used to resolve
// hosted foundational stores that are not in the JSON file.
HostedStoreRegistryAddress string
}
Tier1Config describes the request tier.
type Tier2Config ¶
type Tier2Config struct {
TmpDir string
// Secret, when set, makes tier2 require "Authorization: Bearer <secret>" on subrequests.
Secret string
// ScratchSpace configures the store scratch space backend, empty for the default.
ScratchSpace string
// ReadyTimeout bounds the wait for the app to report ready, 30s when left at zero.
ReadyTimeout time.Duration
}
Tier2Config describes the worker tier.