devenv

package
v1.23.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 22, 2026 License: Apache-2.0 Imports: 28 Imported by: 0

Documentation

Overview

Package devenv boots a complete local Substreams stack — a dummy blockchain in a container, plus tier1 and tier2 running in-process — against which requests can be issued.

The same setup backs both the end-to-end tests and `substreams tools devenv`, so what a developer watches locally is the code path CI exercises.

It lives here rather than in firehose-core — the natural home, since that is where the reader, merger and relayer come from — because firehose-core depends on substreams, so a command over there could only ever run the substreams version it pins, never the working tree. Tier1 and tier2 are built in this module, so this side of the dependency edge is the only one that can run your changes.

Index

Constants

View Source
const BlockType = "sf.acme.type.v1.Block"

BlockType is the only chain this stack knows about: the dummy blockchain used across the end-to-end tests.

View Source
const DefaultImage = "ghcr.io/streamingfast/dummy-blockchain:1cea671"

DefaultImage is the dummy blockchain image the end-to-end tests are pinned to.

Variables

View Source
var Cmd = &cobra.Command{
	Use:   "devenv",
	Short: "Run a local Substreams stack against a dummy blockchain",
	Long: `Boots a dummy blockchain, a tier1 and a tier2 built from this source tree, prints the
tier1 endpoint and stays up until interrupted. Requires a working Docker daemon.

A large genesis burst against a small state bundle size gives thousands of segments to chew
through, which is what makes backprocessing long enough to observe. The state store lives
under --data-dir, so a second run of the same request finds everything cached and returns
instantly; delete that directory to get a cold backprocess back.

The stack logs every block it reads at info level, which buries the endpoint banner. Set
DLOG=warn to quiet it down, or DLOG=debug when the stack itself is what misbehaves.`,
	RunE:         runDevenv,
	Args:         cobra.NoArgs,
	SilenceUsage: true,
}

Cmd boots the stack and blocks. It is registered under `substreams tools`.

Functions

func FindFreePort

func FindFreePort() (int, error)

FindFreePort asks the OS for an available TCP port.

func RelayerEndpoint

func RelayerEndpoint(ctx context.Context, container testcontainers.Container) (string, error)

RelayerEndpoint resolves the host-side address of the container's relayer.

func StartDummyBlockchain

func StartDummyBlockchain(ctx context.Context, config ChainConfig) (testcontainers.Container, error)

StartDummyBlockchain runs a reader-node/merger/relayer container producing the dummy chain.

func StartTier1

func StartTier1(ctx context.Context, config Tier1Config, logger *zap.Logger) (*app.Tier1App, string, error)

StartTier1 boots a tier1 on a free port and waits for it to become ready.

func StartTier2

func StartTier2(ctx context.Context, config Tier2Config, logger *zap.Logger) (*app.Tier2App, string, error)

StartTier2 boots a tier2 on a free port and waits for it to become ready.

func WaitMergedBlocks

func WaitMergedBlocks(ctx context.Context, dataDir string, upTo uint64, timeout time.Duration) error

WaitMergedBlocks blocks until the merger has bundled blocks covering upTo.

Tier1 only reports ready once its block hub can link incoming live blocks, and the hub bootstraps from merged blocks. The reader writes a genesis burst far faster than the merger bundles it — tens of thousands of one-block files against a merger doing roughly a hundred blocks a second — so starting tier1 straight after the container gives it a hub that cannot link anything, and it times out reporting "block not linkable after one-block lookup".

Types

type ChainConfig

type ChainConfig struct {
	// Image defaults to DefaultImage.
	Image string
	// TmpDir is bind-mounted as the firehose storage directory, and is also where tier1 keeps
	// its merged blocks and its state store. Wiping it between runs is what forces a cold
	// backprocess.
	TmpDir string
	// Burst is the number of blocks produced immediately at genesis. This is the knob that
	// decides how much there is to backprocess.
	Burst int
	// BlockRate is the number of blocks per minute produced after the burst.
	BlockRate int
	// ExtraReaderArgs are appended to the reader node arguments verbatim.
	ExtraReaderArgs string
	// StartupTimeout bounds the wait for the container to serve. A large Burst pushes this
	// out — the node produces every genesis block before it starts serving — so it scales
	// with the burst when left at zero.
	StartupTimeout time.Duration
}

ChainConfig describes the dummy blockchain to run.

type Tier1Config

type Tier1Config struct {
	TmpDir string
	// RelayerEndpoint is the block stream tier1 reads live blocks from, normally the mapped
	// 10014 port of the dummy blockchain container.
	RelayerEndpoint string
	// Tier2Endpoint is where subrequests are sent.
	Tier2Endpoint string
	// Tier2Secret must match the tier2 secret when one is configured.
	Tier2Secret string
	// StateBundleSize is the segment size. Small values mean many small jobs, which is what
	// makes a backprocess visible: 100 over a 200k block burst is 2000 segments per stage.
	StateBundleSize uint64
	// MaxSubrequests is how many tier2 jobs may run at once.
	MaxSubrequests uint64
	// MaxWorkersPerSession caps the workers a single request may hold, defaulting to
	// MaxSubrequests. Lower it to observe a request that is throttled by the session pool
	// rather than by the job scheduler.
	MaxWorkersPerSession uint64
	// MetricsPrefix distinguishes the metric set when several tier1s run in one process.
	MetricsPrefix                 string
	LiveBackFillerFinalBlockDelay uint64
	// ReadyTimeout bounds the wait for the app to report ready, two minutes when left at zero.
	// Tier1 bootstraps its block hub from the merged blocks on disk before it answers, so a
	// large genesis burst pushes this well past the handful of seconds a small one needs.
	ReadyTimeout time.Duration
	// FoundationalStoresConfigPath is a JSON file of identifier → endpoint.
	FoundationalStoresConfigPath string
	// HostedStoreRegistryAddress is the control-plane gRPC address used to resolve
	// hosted foundational stores that are not in the JSON file.
	HostedStoreRegistryAddress string
	// CPUEviction configures the CPU-based request evictor. The zero value leaves it off.
	// Point SUBSTREAMS_CGROUP_DIR at hand-written cpu.max/cpu.stat files to drive it on a
	// host without cgroups.
	CPUEviction active_requests.EvictorConfig
}

Tier1Config describes the request tier.

type Tier2Config

type Tier2Config struct {
	TmpDir string
	// Secret, when set, makes tier2 require "Authorization: Bearer <secret>" on subrequests.
	Secret string
	// ScratchSpace configures the store scratch space backend, empty for the default.
	ScratchSpace string
	// ReadyTimeout bounds the wait for the app to report ready, 30s when left at zero.
	ReadyTimeout time.Duration
}

Tier2Config describes the worker tier.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL