block

package
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 26, 2026 License: Apache-2.0 Imports: 11 Imported by: 0

Documentation

Overview

Package block implements the L2 immutable columnar part format (DESIGN.md §3, §14 M1): per-column streams with min/max stats and constant-column collapse, a sparse granule mark index, and an atomic manifest written last. Not yet implemented (M1).

Index

Constants

This section is empty.

Variables

View Source 
var ErrCorrupt = errors.New("block: corrupt metadata")

ErrCorrupt is returned when a manifest (or any part metadata) fails to parse: bad magic, CRC mismatch, truncation, or an out-of-range field.

Functions

func WritePart 

func WritePart(ctx context.Context, b backend.Backend, prefix string, w *PartWriter) error

WritePart serializes the writer's columns and writes the part's objects under prefix on b. Column and marks objects are written first; the manifest is written LAST so the part only becomes readable once fully committed.

Types

type Column 

type Column struct {
	Name     string
	Kind     Kind
	Int64    []int64
	Float64  []float64
	Bytes    [][]byte
	Int128   []chunk.U128
	Codec    chunk.Codec
	Compress compress.Algorithm
}

Column is an input column for a part: a name, a physical Kind, and the matching typed slice (exactly one of Int64/Float64/Bytes per Kind). Codec and Compress are optional overrides; the zero values select the per-kind default codec and no block compression (the chunk codecs already compress well).

type ColumnDesc 

type ColumnDesc struct {
	Name     string
	Kind     Kind
	Codec    chunk.Codec
	Compress compress.Algorithm
	Const    bool

	// Constant value, set iff Const, by Kind.
	ConstInt64   int64
	ConstFloat64 float64
	ConstBytes   []byte

	// Numeric min/max (KindInt64/KindFloat64). Unused for KindBytes.
	MinInt64, MaxInt64     int64
	MinFloat64, MaxFloat64 float64
}

ColumnDesc describes one column in a part: its identity, codecs, constant value (if collapsed), and numeric min/max stats. Data offsets are absent — each column is its own backend object keyed by ordinal (DESIGN.md §14 M1 multi-key layout).

type ColumnReader 

type ColumnReader struct {
	// contains filtered or unexported fields
}

ColumnReader gives lazy, decode-on-demand access to one column of a part. Constant columns are synthesized from the manifest with no I/O; other columns are decompressed and decoded only when an accessor is called (DESIGN.md §7: decode only what the query touches). It is created by PartReader.Column.

func (*ColumnReader) Bytes 

func (r *ColumnReader) Bytes() (*chunk.DictColumn, error)

Bytes decodes the column into its split chunk.DictColumn form (unique entries + a per-row id array), deferring the per-row gather to chunk.DictColumn.At. A constant column is synthesized as a single-entry dictionary. It errors if the column is not KindBytes.

func (*ColumnReader) Const 

func (r *ColumnReader) Const() (any, bool)

Const returns the column's constant value (int64/float64/[]byte per Kind) and true if the column was constant-collapsed; otherwise (nil, false).

func (*ColumnReader) Float64 

func (r *ColumnReader) Float64(dst []float64) ([]float64, error)

Float64 decodes the column into dst (reusing its capacity) and returns the result. It errors if the column is not KindFloat64.

func (*ColumnReader) ID128 

func (r *ColumnReader) ID128(dst []chunk.U128) ([]chunk.U128, error)

ID128 decodes the column into dst (reusing its capacity) and returns the result. It errors if the column is not KindInt128. Id columns are never constant-collapsed, so the value always comes from the decoded RLE stream.

func (*ColumnReader) Int64 

func (r *ColumnReader) Int64(dst []int64) ([]int64, error)

Int64 decodes the column into dst (reusing its capacity) and returns the result. It errors if the column is not KindInt64.

func (*ColumnReader) Kind 

func (r *ColumnReader) Kind() Kind

Kind reports the column's physical type.

func (*ColumnReader) Len 

func (r *ColumnReader) Len() int

Len reports the column's row count.

type Granule 

type Granule struct {
	FirstRow int
	MinKey   int64
	MaxKey   int64
}

Granule is one entry of the sparse mark index: the first row of a fixed-size run of rows, plus the min/max of the part's sort-key column over that run. The min/max let a query prune whole granules whose key range cannot intersect its window (_ref/docs/storage-engine.md §2, ClickHouse marks / Parquet row-group stats).

type Kind 

type Kind uint8

Kind is the physical type of a column's values (DESIGN.md §6). It selects which typed slice a Column carries and which codec family applies. Values are persisted in the manifest; never reorder. 

const (
	// KindInt64 is an int64 column (timestamps, counters, series ids).
	KindInt64 Kind = iota
	// KindFloat64 is a float64 column (gauge/sum values).
	KindFloat64
	// KindBytes is a []byte column (low-cardinality attributes, strings).
	KindBytes
	// KindInt128 is a 128-bit id column ([chunk.U128]), e.g. the SeriesID sort key of a
	// metric part. RLE-coded; carries no min/max or constant value in the manifest.
	KindInt128
)

func (Kind) String 

func (k Kind) String() string

String returns a stable lower-case kind name.

type Manifest 

type Manifest struct {
	Version     uint32
	RowCount    int
	MinTime     int64
	MaxTime     int64
	GranuleSize int
	Columns     []ColumnDesc
}

Manifest is the part descriptor: format version, row count, time range, granule size, and the per-column descriptors. It is serialized to the `{prefix}/manifest` object, CRC32C-checked, and written last to commit the part.

func DecodeManifest 

func DecodeManifest(src []byte) (Manifest, error)

DecodeManifest parses a manifest object. It verifies the CRC and bounds-checks every field, returning an ErrCorrupt-wrapping error on any malformed input — it never panics, so it is safe to fuzz on arbitrary bytes.

func (Manifest) Encode 

func (m Manifest) Encode(dst []byte) []byte

Encode appends the binary manifest to dst and returns the extended slice. Layout: 

[u32 magic][uvarint version][uvarint rowCount][varint minTime][varint maxTime]
[uvarint granuleSize][uvarint colCount]
  per column: [uvarint nameLen][name][byte kind][byte codec][byte compress][byte flags]
              [numeric min/max per kind][const value per kind if flagConst]
[u32 CRC32C over all the above]

type Marks 

type Marks struct {
	GranuleSize int
	Granules    []Granule
}

Marks is the sparse granule index for a part: granules of GranuleSize rows over the sort-key column (timestamp for the metrics vertical). It is serialized to the `{prefix}/marks` object.

func BuildMarks 

func BuildMarks(sortKey []int64, granuleSize int) Marks

BuildMarks computes the sparse index over a sort-key column, chunked into granules of granuleSize rows. The min/max are computed by scanning each granule, so correctness does not depend on sortKey being sorted (though for the metrics vertical it is). granuleSize must be > 0.

func DecodeMarks 

func DecodeMarks(src []byte) (Marks, error)

DecodeMarks parses a marks object. It verifies the CRC and bounds-checks every field, returning an ErrCorrupt-wrapping error on malformed input; it never panics.

func (Marks) Encode 

func (m Marks) Encode(dst []byte) []byte

Encode appends the binary marks index to dst. Layout: 

[u32 magic][uvarint version][uvarint granuleSize][uvarint count]
  per granule: [uvarint firstRow][varint minKey-prevMinKey][varint maxKey-minKey]
[u32 CRC32C]

Keys are delta-encoded across granules (sort key is non-decreasing) for compactness.

func (Marks) Overlapping 

func (m Marks) Overlapping(lo, hi int64) []Granule

Overlapping returns the granules whose [MinKey, MaxKey] range intersects the inclusive window [lo, hi]. It is the pruning primitive the fetcher (M3) uses to skip granules.

type PartOption 

type PartOption func(*PartWriter)

PartOption configures a PartWriter.

func WithCompression 

func WithCompression(alg compress.Algorithm) PartOption

WithCompression sets the default block-compression algorithm for columns that do not set Column.Compress (default none — the chunk codecs already compress well).

func WithGranuleSize 

func WithGranuleSize(n int) PartOption

WithGranuleSize sets the sparse-index granularity in rows (default 8192).

func WithSortKey 

func WithSortKey(name string) PartOption

WithSortKey names the int64 column that the marks index and time range are built over. If unset, the first int64 column is used.

type PartReader 

type PartReader struct {
	// contains filtered or unexported fields
}

PartReader reads a part written by WritePart. It loads only the manifest up front; columns and marks are read lazily, so a query touches only the objects it references (DESIGN.md §7).

func OpenPart 

func OpenPart(ctx context.Context, b backend.Backend, prefix string) (*PartReader, error)

OpenPart reads a part's manifest from b under prefix and returns a reader. It returns an error (wrapping ErrCorrupt or backend.ErrNotExist) if the manifest is absent or malformed — an incompletely written part (no manifest) is therefore not readable.

func (*PartReader) Column 

func (r *PartReader) Column(ctx context.Context, name string) (*ColumnReader, error)

Column returns a lazy reader for the named column. A constant column is synthesized from the manifest with no I/O; otherwise its object is read from the backend.

func (*PartReader) ColumnNames 

func (r *PartReader) ColumnNames() []string

ColumnNames returns the column names in part order.

func (*PartReader) Manifest 

func (r *PartReader) Manifest() Manifest

Manifest returns the part's decoded manifest.

func (*PartReader) Marks 

func (r *PartReader) Marks(ctx context.Context) (Marks, error)

Marks reads and decodes the part's sparse granule index.

func (*PartReader) RowCount 

func (r *PartReader) RowCount() int

RowCount returns the number of rows in the part.

type PartWriter 

type PartWriter struct {
	// contains filtered or unexported fields
}

PartWriter accumulates columns and serializes them into a part's objects. Columns are added in order; their ordinal is their object key. The sort-key column (timestamp for metrics) drives the marks index and the manifest time range.

func NewPartWriter 

func NewPartWriter(opts ...PartOption) *PartWriter

NewPartWriter returns a PartWriter with the given options applied.

func (*PartWriter) AddColumn 

func (w *PartWriter) AddColumn(c Column) error

AddColumn appends a column. All columns in a part must have the same row count.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.