firecore

package module
v0.0.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Apr 4, 2023 License: Apache-2.0 Imports: 48 Imported by: 24

README

Firehose Integrators Tool Kit

This repository contains all the boilerplate that is required to maintain the Go part of the Firehose stack for chain integrators. This repository can be seen as a Integrator Tool Kit for people maintaining Firehose version of a specific chain. It's essentially a chain agnostic shared library that is used to avoid duplication across all projects and ease maintenance work for the various teams. It contain no chain specific code and everything that is chain specific must be provided.

Note This repository is only useful for maintainers of firehose-<chain> repositories and new integrators looking to integrate Firehose into a new chain. If you are a developer using Firehose or Substreams technology, this repository is not for you.

Philosophy

Firehose maintenance cost comes from two side. First there is the chain integration that needs to be maintained. This is done within the chain's code directly by the chain's core developers. The second side of thing is the maintenance of the Golang part of the Firehose stack.

Each chain creates it's own Firehose Golang repository named firehose-<chain> (https://github.com/streamingfast/firehose-acme acts as template for this). Firehose is composed of multiple smaller components that can be run independently and each of them as a set of CLI flags and other configuration.

The initial https://github.com/streamingfast/firehose-acme "template" we had was containing a lot of boilerplate code to properly configure and run the Firehose Golang stack. This means that if we needed to add a new feature which required a new flags or change a flag default value or any kind of improvements, chain integrators that were maintaining their own firehose-<chain> repository were in the obligation of tracking changes made https://github.com/streamingfast/firehose-acme and apply those back on their own repository by hand.

This was true also for continuously tracking updating the various small library that form the Firehose stack. With Firehose starting to getting more and more streamlined across different chains, that was a recipe for a maintenance hell for every chain integration.

This repository aims at solving this maintenance burden by acting as a facade for all the Golang code required to have a functional and up to date Firehose stack. This way, we maintain the firehose-core project, adding/changing/removing flags, bumping dependencies, adding new features and you as a maintainer of firehose-<chain> repository, you simply need to track https://github.com/streamingfast/firehose-core for new releases and bump a single dependency to be up to date with latest changes.

Changelog

The CHANGELOG.md of this project will be re-written so that you can copy directly the various entries since your last update straight to your own release notes so that operators that are using your firehose-<chain> repository are made aware of deprecation notes, removal, changes and other important element.

Build & CI

The build and CI files are maintained for now in https://github.com/streamingfast/firehose-acme directly and should be updated manually from time to time from there.

Documentation

Index

Constants

This section is empty.

Variables

View Source
var (
	// Common ports
	MetricsListenAddr string = ":9102"

	// Firehose chain specific port
	ReaderNodeGRPCAddr       string = ":10010"
	ReaderNodeManagerAPIAddr string = ":10011"
	MergerServingAddr        string = ":10012"
	RelayerServingAddr       string = ":10014"
	FirehoseGRPCServingAddr  string = ":10015"

	// Data storage default locations
	BlocksCacheDirectory string = "file://{data-dir}/storage/blocks-cache"
	MergedBlocksStoreURL string = "file://{data-dir}/storage/merged-blocks"
	OneBlockStoreURL     string = "file://{data-dir}/storage/one-blocks"
	ForkedBlocksStoreURL string = "file://{data-dir}/storage/forked-blocks"
	IndexStoreURL        string = "file://{data-dir}/storage/index"
)

Those are `var` and globally available so that some chains to keep backward-compatibility can change them. This is not advertised and should **not** be used by new chain.

View Source
var Example = func(in string) string {
	return string(cli.Example(in))
}
View Source
var ExamplePrefixed = func(chain *Chain, prefix, in string) string {
	return string(cli.ExamplePrefixed(chain.BinaryName()+" "+prefix, in))
}

Functions

func GetCommonStoresURLs

func GetCommonStoresURLs(dataDir string) (mergedBlocksStoreURL, oneBlocksStoreURL, forkedBlocksStoreURL string, err error)

func GetIndexStore

func GetIndexStore(dataDir string) (indexStore dstore.Store, possibleIndexSizes []uint64, err error)

func Main

func Main(chain *Chain)

Main is the main entry point that configures everything and should be called from your Go 'main' entrypoint directly.

func MustReplaceDataDir

func MustReplaceDataDir(dataDir, in string) string

MustReplaceDataDir replaces `{data-dir}` from within the `in` received argument by the `dataDir` argument

func PrintOutputModeNames

func PrintOutputModeNames() []string

PrintOutputModeNames returns a list of possible string values of PrintOutputMode.

Types

type BlockPrinterFunc

type BlockPrinterFunc func(block *bstream.Block, alsoPrintTransactions bool, out io.Writer) error

BlockPrinterFunc takes a chain agnostic [block] and prints it to a human readable form.

See [ToolsConfig#BlockPrinter] for extra details about expected printing.

type Chain

type Chain struct {
	// ShortName is the short name for your Firehose on <Chain> and is usually how
	// your chain's name is represented as a diminitutive. If your chain's name is already
	// short, we suggest to keep [ShortName] and [LongName] the same.
	//
	// As an example, Firehose on Ethereum [ShortName] is `eth` while Firehose on NEAR
	// short name is `near`.
	//
	// The [ShortName] **must** be  non-empty, lower cased and must **not** contain any spaces.
	ShortName string

	// LongName is the full name of your chain and the case sensitivy of this value is respected.
	// It is used in description of command and some logging output.
	//
	// The [LongName] **must** be non-empty.
	LongName string

	// ExecutableName is the name of the binary that is used to launch a syncing node for this chain. For example,
	// on Ethereum, the binary by default is `geth`. This is used by the `reader-node` app to specify the
	// `reader-node-binary-name` flag.
	//
	// The [ExecutableName] **must** be non-empty.
	ExecutableName string

	// FullyQualifiedModule is the Go module of your actual `firehose-<chain>` repository and should
	// correspond to the `module` line of the `go.mod` file found at the root of your **own** `firehose-<chain>`
	// repository.
	//
	// The [FullyQualifiedModule] **must** be non-empty.
	FullyQualifiedModule string

	// Version represents the actual version for your Firehose on <Chain>. It should be injected
	// via and `ldflags` through your `main` package.
	//
	// The [Version] **must** be non-empty.
	Version string

	// FirstStreamableBlock represents the block number of the first block that is streamable using Firehose,
	// for example on Ethereum it's set to `0`, the genesis block's number while on Antelope it's
	// set to 2 (genesis block is 1 there but our instrumentation on this chain instruments
	// only from block #2).
	//
	// This is used in multiple places to determine if we reached the oldest block of the chain.
	FirstStreamableBlock uint64

	// Should be the number of blocks between two targets before we consider the
	// first as "near" the second. For example if a chain is at block #215 and another
	// source is at block #225, then there is a difference of 10 blocks which is <=
	// than `BlockDifferenceThresholdConsideredNear` which would mean it's "near".
	//
	// Must be greater than 0 and lower than 1024
	BlockDifferenceThresholdConsideredNear uint64

	// ConsoleReaderFactory is the function that should return the `ConsoleReader` that knowns
	// how to transform your your chain specific Firehose instrumentation logs into the proper
	// Block model of your chain.
	//
	// The [ConsoleReaderFactory] **must** be non-nil and must return a non-nil [mindreader.ConsolerReader] or an error.
	ConsoleReaderFactory func(lines chan string, logger *zap.Logger, tracer logging.Tracer) (mindreader.ConsolerReader, error)

	ChainSuperviserFactory func(
		chainShortName string,
		binary string,
		arguments []string,
		dataDir string,
		headBlockUpdateFunc nodeManager.HeadBlockUpdater,
		debugFirehose bool,
		logToZap bool,
		appLogger *zap.Logger,
		nodelogger *zap.Logger,
	) (nodeManager.ChainSuperviser, error)

	// Tools aggregate together all configuration options required for the various `fire<chain> tools`
	// to work properly for example to print block using chain specific information.
	//
	// The [Tools] element is optional and if not provided, sane defaults will be used.
	Tools *ToolsConfig
}

Chain is the omni config object for configuring your chain specific information. It contains various fields that are used everywhere to properly configure the `firehose-<chain>` binary.

Each field is documented about where it's used. Throughtout the different Chain option, we will use `Acme` as the chain's name placeholder, replace it with your chain name.

func (*Chain) BinaryName

func (c *Chain) BinaryName() string

BinaryName represents the binary name for your Firehose on <Chain> is the [ShortName] lowered appended to 'fire' prefix to before for example `fireacme`.

func (*Chain) BlockPrinter

func (c *Chain) BlockPrinter() BlockPrinterFunc

func (*Chain) LoggerPackageID

func (c *Chain) LoggerPackageID(subPackage string) string

LoggerPackageID computes a logger `packageID` value for a specific sub-package.

func (*Chain) RootLoggerPackageID

func (c *Chain) RootLoggerPackageID() string

RootLoggerPackageID is the `packageID` value when instantiating the root logger on the chain that is used by CLI command and other

func (*Chain) Validate

func (c *Chain) Validate()

Validate normalizes some aspect of the Chain values (spaces trimming essentially) and validates the chain by accumulating error an panic if all the error found along the way.

func (*Chain) VersionString

func (c *Chain) VersionString() string

VersionString computes the version string that will be display when calling `firexxx --version` and extract build information from Git via Golang `debug.ReadBuildInfo`.

type CommandExecutor

type CommandExecutor func(cmd *cobra.Command, args []string) (err error)

type PrintOutputMode

type PrintOutputMode uint

ENUM(

Text
JSON

)

const (
	// PrintOutputModeText is a PrintOutputMode of type Text.
	PrintOutputModeText PrintOutputMode = iota
	// PrintOutputModeJSON is a PrintOutputMode of type JSON.
	PrintOutputModeJSON
)

func ParsePrintOutputMode

func ParsePrintOutputMode(name string) (PrintOutputMode, error)

ParsePrintOutputMode attempts to convert a string to a PrintOutputMode

func (PrintOutputMode) MarshalText

func (x PrintOutputMode) MarshalText() ([]byte, error)

MarshalText implements the text marshaller method

func (PrintOutputMode) String

func (x PrintOutputMode) String() string

String implements the Stringer interface.

func (*PrintOutputMode) UnmarshalText

func (x *PrintOutputMode) UnmarshalText(text []byte) error

UnmarshalText implements the text unmarshaller method

type ToolsConfig

type ToolsConfig struct {
	// BlockPrinter represents a printing function that render a chain specific human readable
	// form of the receive chain agnostic [bstream.Block]. This block is expected to be rendered as
	// a single line for example on Ethereum rendering of a single block looks like:
	//
	// “`
	// Block #24924194 (01d6d349fbd3fa419182a2f0cf0b00714e101286650c239de8923caef6134b6c) 62 transactions, 607 calls
	// “`
	//
	// If the [alsoPrintTransactions] argument is true, each transaction of the block should also be printed, following
	// directly the block line. Each transaction should also be on a single line, usually prefixed with a `- ` to make
	// the rendering more appealing.
	//
	// For example on Ethereum rendering with [alsoPrintTransactions] being `true` looks like:
	//
	// “`
	// Block #24924194 (01d6d349fbd3fa419182a2f0cf0b00714e101286650c239de8923caef6134b6c) 62 transactions, 607 calls
	// - Transaction 0xc7e04240d6f2cc5f382c478fd0a0b5c493463498c64b31477b95bded8cd12ab4 (10 calls)
	// - Transaction 0xc7d8a698351eb1ac64acb76c8bf898365bb639865271add95d2c81650b2bd98c (4 calls)
	// “`
	//
	// The `out` parameter is used to write to the correct location. You can use [fmt.Fprintf] and [fmt.Fprintln]
	// and use `out` as the output writer in your implementation.
	//
	// The [BlockPrinter] is optional, if nil, a default block printer will be used. It's important to note
	// that the default block printer error out if `alsoPrintTransactions` is true.
	BlockPrinter BlockPrinterFunc
}

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL