Documentation
¶
Overview ¶
Package cryo wraps the third-party `cryo` binary (https://github.com/paradigmxyz/cryo) so EL cannon can extract execution-layer datasets over a block range and read them back as typed rows. The runner is dataset-agnostic: it invokes cryo for one dataset over an inclusive block range, writes parquet into a unique temp directory, and returns the file paths. Callers parse those files into dataset-specific structs with ReadParquet.
Two cryo options matter for round-tripping through parquet-go:
- --hex encodes binary columns as 0x-prefixed hex strings (large_string), which is exactly what the canonical_execution_* ClickHouse schema wants.
- --compression zstd avoids cryo's default lz4, which parquet-go decodes incorrectly (it produces garbage byte-array offsets at volume). zstd is unambiguous and read cleanly by parquet-go.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func ReadParquet ¶
ReadParquet reads every supplied parquet file into a slice of T. T must be a struct with `parquet:"<column>"` tags matching the cryo column names.
Types ¶
type Collection ¶
Collection is the result of a cryo invocation. Close removes the temp dir.
func (*Collection) Close ¶
func (c *Collection) Close() error
Close removes the temporary output directory.
type Config ¶
type Config struct {
// BinaryPath is the path to the cryo binary.
BinaryPath string `yaml:"binaryPath" default:"cryo"`
// OutputDir is the parent directory for per-invocation temp output.
OutputDir string `yaml:"outputDir" default:"/tmp/xatu-cannon-cryo"`
// MaxRangeSize is the default number of blocks per cryo invocation.
MaxRangeSize uint64 `yaml:"maxRangeSize" default:"50"`
// Compression is the cryo parquet compression spec (name plus optional
// level, e.g. "zstd 1"). Avoid cryo's lz4 default — parquet-go mis-decodes
// it. Set as space-separated tokens passed straight to `cryo --compression`.
Compression string `yaml:"compression" default:"zstd 1"`
// RequestsPerSecond rate-limits cryo's RPC calls (0 = cryo default).
RequestsPerSecond uint64 `yaml:"requestsPerSecond" default:"0"`
// MaxConcurrentChunks bounds cryo's internal chunk concurrency (0 = default).
MaxConcurrentChunks uint64 `yaml:"maxConcurrentChunks" default:"0"`
}
Config configures the cryo runner.
type Runner ¶
type Runner struct {
// contains filtered or unexported fields
}
Runner invokes cryo against a single execution RPC endpoint.
func New ¶
func New(log logrus.FieldLogger, cfg *Config, rpc string) *Runner
New creates a cryo runner. rpc is the execution-layer RPC URL (credentials may be embedded as https://user:pass@host).
func (*Runner) Collect ¶
func (r *Runner) Collect(ctx context.Context, dataset string, from, to uint64, columns []string) (*Collection, error)
Collect runs cryo for the given dataset over the inclusive block range [from, to], restricting output to the supplied columns (empty = cryo defaults). cryo's --blocks end is exclusive, so we pass to+1. The returned Collection must be Closed by the caller to clean up disk.
Source Files