block

package
v0.28.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 2, 2026 License: Apache-2.0 Imports: 13 Imported by: 0

Documentation

Overview

Package block implements the L2 immutable columnar part format (DESIGN.md §3, §14 M1): per-column streams with min/max stats and constant-column collapse, a sparse granule mark index, and an atomic manifest written last. Not yet implemented (M1).

Index

Constants

This section is empty.

Variables

View Source 
var ErrCorrupt = errors.New("block: corrupt metadata")

ErrCorrupt is returned when a manifest (or any part metadata) fails to parse: bad magic, CRC mismatch, truncation, or an out-of-range field.

Functions

func WritePart 

func WritePart(ctx context.Context, b backend.Backend, prefix string, w *PartWriter) error

WritePart serializes the writer's columns and writes the part's objects under prefix on b. Column and marks objects are written first; the manifest is written LAST so the part only becomes readable once fully committed.

Types

type Column 

type Column struct {
	Name     string
	Kind     Kind
	Int64    []int64
	Float64  []float64
	Bytes    [][]byte
	Int128   []chunk.U128
	Codec    chunk.Codec
	Compress compress.Algorithm
	// BytesBlob/BytesOffsets are the blob+offsets alternative to Bytes for a KindBytes column:
	// cell i is BytesBlob[BytesOffsets[i]:BytesOffsets[i+1]], with len(BytesOffsets) == rows+1 and
	// BytesOffsets[0] == 0 — the head-buffer byte-column layout, accepted directly so a flush
	// encodes straight from the blob without materializing a [][]byte view per row. Used only when
	// Bytes is nil; the encoded object is byte-identical to the equivalent Bytes input.
	BytesBlob    []byte
	BytesOffsets []int32
	// AutoCodec, when set on a float64 column with no explicit Codec, picks the smaller of the
	// lossless float codecs (Gorilla XOR vs scaled-decimal+delta) by trial-encoding — so an
	// integer-valued or low-precision column (e.g. a counter) takes the far denser decimal path
	// while a high-entropy column keeps Gorilla. Lossless either way (see [chooseFloatCodec]).
	AutoCodec bool
	// FloatPrecisionBits, when in 1..63 on an AutoCodec float column, requests *lossy* encoding:
	// the scaled-decimal codec retains only this many significant mantissa bits (fewer ⇒ denser,
	// less accurate), competing against lossless Gorilla so the result is never worse than
	// lossless. 0 (or ≥64) means lossless. The budget is recorded in the descriptor so the merge
	// engine reaches a fixed point (it never re-applies a budget it has already met). Set per age
	// tier by the merge engine so only old data trades accuracy for size.
	FloatPrecisionBits uint8
	// Block requests block-framed encoding: the column is split into blockRows-row blocks (the
	// part's granule size), each an independent codec stream, so a reader can decode one block at a
	// time (sub-part seek). Only the per-row sequential codecs (DoD/T64 int64, Gorilla/decimal
	// float64) are blockable — the metric ts/value/sf columns. The zero value keeps the prior
	// single-stream layout. See blockcolumn.go.
	Block bool
}

Column is an input column for a part: a name, a physical Kind, and the matching typed slice (exactly one of Int64/Float64/Bytes per Kind). Codec and Compress are optional overrides; the zero values select the per-kind default codec and no block compression (the chunk codecs already compress well).

type ColumnDesc 

type ColumnDesc struct {
	Name     string
	Kind     Kind
	Codec    chunk.Codec
	Compress compress.Algorithm
	Const    bool

	// Constant value, set iff Const, by Kind.
	ConstInt64   int64
	ConstFloat64 float64
	ConstBytes   []byte

	// Numeric min/max (KindInt64/KindFloat64). Unused for KindBytes.
	MinInt64, MaxInt64     int64
	MinFloat64, MaxFloat64 float64

	// FloatPrecisionBits is the lossy precision budget a float column was encoded under (the
	// significant mantissa bits retained): 0 ⇒ lossless. Persisted only when non-zero (a
	// flag-gated byte), so lossless parts keep their byte-for-byte layout. The merge engine reads
	// it as the fixed point for age-tiered precision — it never re-coarsens a part already at or
	// below the target budget.
	FloatPrecisionBits uint8

	// Blocked marks a block-framed column: its object is a [blockDir] + per-block codec streams
	// (see blockcolumn.go) instead of a single stream, so a reader can decode one block at a time.
	// Persisted via [flagBlocked]; clear on the prior single-stream layout.
	Blocked bool
}

ColumnDesc describes one column in a part: its identity, codecs, constant value (if collapsed), and numeric min/max stats. Data offsets are absent — each column is its own backend object keyed by ordinal (DESIGN.md §14 M1 multi-key layout).

type ColumnReader 

type ColumnReader struct {
	// contains filtered or unexported fields
}

ColumnReader gives lazy, decode-on-demand access to one column of a part. Constant columns are synthesized from the manifest with no I/O; other columns are decompressed and decoded only when an accessor is called (DESIGN.md §7: decode only what the query touches). It is created by PartReader.Column.

func (*ColumnReader) BlockDecoder added in v0.20.0 

func (r *ColumnReader) BlockDecoder() (*Decoder, error)

BlockDecoder returns a per-block decoder over a blocked column (parsing the directory once), for a caller that decodes individual blocks. It errors on an unblocked column.

func (*ColumnReader) BlockRows added in v0.20.0 

func (r *ColumnReader) BlockRows() (int, error)

BlockRows returns the column's block size in rows, or 0 for an unblocked column.

func (*ColumnReader) Blocked added in v0.20.0 

func (r *ColumnReader) Blocked() bool

Blocked reports whether the column is block-framed (and so supports DecodeBlocks*/BlockRows).

func (*ColumnReader) Bytes 

func (r *ColumnReader) Bytes() (*chunk.DictColumn, error)

Bytes decodes the column into its split chunk.DictColumn form (unique entries + a per-row id array), deferring the per-row gather to chunk.DictColumn.At. A constant column is synthesized as a single-entry dictionary. It errors if the column is not KindBytes.

func (*ColumnReader) Const 

func (r *ColumnReader) Const() (any, bool)

Const returns the column's constant value (int64/float64/[]byte per Kind) and true if the column was constant-collapsed; otherwise (nil, false).

func (*ColumnReader) DecodeBlocksFloat64 added in v0.20.0 

func (r *ColumnReader) DecodeBlocksFloat64(dst []float64, blocks []int) ([]float64, error)

DecodeBlocksFloat64 is the float64 analog of ColumnReader.DecodeBlocksInt64.

func (*ColumnReader) DecodeBlocksInt64 added in v0.20.0 

func (r *ColumnReader) DecodeBlocksInt64(dst []int64, blocks []int) ([]int64, error)

DecodeBlocksInt64 decodes only the given block indices of a blocked column into dst (reused), and returns a full-length (Len()) slice with those blocks' row spans populated — the rest of dst is left as-is (a caller reads only the rows it selected, which fall in the requested blocks). An unblocked/const column has no blocks to skip, so it decodes whole. blocks must be in range.

func (*ColumnReader) Float64 

func (r *ColumnReader) Float64(dst []float64) ([]float64, error)

Float64 decodes the column into dst (reusing its capacity) and returns the result. It errors if the column is not KindFloat64.

func (*ColumnReader) FloatCursor added in v0.15.0 

func (r *ColumnReader) FloatCursor() (chunk.FloatDecoder, error)

FloatCursor returns a forward cursor over a KindFloat64 column (Gorilla or scaled-decimal). A constant-collapsed column yields a repeating cursor. It is the streaming-merge form of Float64.

func (*ColumnReader) ID128 

func (r *ColumnReader) ID128(dst []chunk.U128) ([]chunk.U128, error)

ID128 decodes the column into dst (reusing its capacity) and returns the result. It errors if the column is not KindInt128. Id columns are never constant-collapsed, so the value always comes from the decoded RLE stream.

func (*ColumnReader) Int64 

func (r *ColumnReader) Int64(dst []int64) ([]int64, error)

Int64 decodes the column into dst (reusing its capacity) and returns the result. It errors if the column is not KindInt64.

func (*ColumnReader) Kind 

func (r *ColumnReader) Kind() Kind

Kind reports the column's physical type.

func (*ColumnReader) Len 

func (r *ColumnReader) Len() int

Len reports the column's row count.

func (*ColumnReader) RangeFloat64 added in v0.20.0 

func (r *ColumnReader) RangeFloat64(dst []float64, lo, hi int) ([]float64, error)

RangeFloat64 decodes only rows [lo,hi) of a float64 column. See ColumnReader.RangeInt64.

func (*ColumnReader) RangeInt64 added in v0.20.0 

func (r *ColumnReader) RangeInt64(dst []int64, lo, hi int) ([]int64, error)

RangeInt64 decodes only rows [lo,hi) of an int64 column into a buffer reusing dst's capacity. For a blocked column it decodes just the blocks spanning the range (sub-part seek); for an unblocked one it decodes the whole column and slices, so callers get a uniform seek API regardless of layout. Requires 0 ≤ lo < hi ≤ Len().

func (*ColumnReader) TsCursor added in v0.15.0 

func (r *ColumnReader) TsCursor() (chunk.TsCursor, error)

TsCursor returns a forward cursor over a KindInt64 timestamp column (delta-of-delta). A constant-collapsed column yields a repeating cursor. It is the streaming-merge form of Int64: same decode, but one row at a time so a merge holds only one series range resident per part.

type Decoder added in v0.20.0 

type Decoder struct {
	// contains filtered or unexported fields
}

Decoder decodes individual blocks of a blocked column, parsing the directory once so a caller that decodes several blocks (e.g. a per-block cache filling its misses) does not re-parse it per block. Obtain one via [ColumnReader.Decoder]; it holds the column's already-read object.

func (*Decoder) BlockRows added in v0.20.0 

func (d *Decoder) BlockRows() int

BlockRows returns the column's nominal block size in rows.

func (*Decoder) BlockSpan added in v0.20.0 

func (d *Decoder) BlockSpan(blk int) (lo, hi int)

BlockSpan returns block blk's half-open row range [lo, hi) in the column.

func (*Decoder) DecodeFloat64 added in v0.20.0 

func (d *Decoder) DecodeFloat64(blk int) ([]float64, error)

DecodeFloat64 decodes block blk into a fresh slice (for a float64 column).

func (*Decoder) DecodeFloat64Into added in v0.23.0 

func (d *Decoder) DecodeFloat64Into(blk int, dst []float64) ([]float64, error)

DecodeFloat64Into is the float64 analog of Decoder.DecodeInt64Into.

func (*Decoder) DecodeInt64 added in v0.20.0 

func (d *Decoder) DecodeInt64(blk int) ([]int64, error)

DecodeInt64 decodes block blk into a fresh slice (for an int64 column).

func (*Decoder) DecodeInt64Into added in v0.23.0 

func (d *Decoder) DecodeInt64Into(blk int, dst []int64) ([]int64, error)

DecodeInt64Into decodes block blk into dst (for an int64 column), reusing dst's backing array when it has room for the block's rows — so a caller drawing dst from a pool decodes without allocating the output. Passing a nil dst is equivalent to Decoder.DecodeInt64.

func (*Decoder) NumBlocks added in v0.20.0 

func (d *Decoder) NumBlocks() int

NumBlocks returns the column's block count.

type Granule 

type Granule struct {
	FirstRow int
	MinKey   int64
	MaxKey   int64
}

Granule is one entry of the sparse mark index: the first row of a fixed-size run of rows, plus the min/max of the part's sort-key column over that run. The min/max let a query prune whole granules whose key range cannot intersect its window (_ref/docs/storage-engine.md §2, ClickHouse marks / Parquet row-group stats).

type Kind 

type Kind uint8

Kind is the physical type of a column's values (DESIGN.md §6). It selects which typed slice a Column carries and which codec family applies. Values are persisted in the manifest; never reorder. 

const (
	// KindInt64 is an int64 column (timestamps, counters, series ids).
	KindInt64 Kind = iota
	// KindFloat64 is a float64 column (gauge/sum values).
	KindFloat64
	// KindBytes is a []byte column (low-cardinality attributes, strings).
	KindBytes
	// KindInt128 is a 128-bit id column ([chunk.U128]), e.g. the SeriesID sort key of a
	// metric part. RLE-coded; carries no min/max or constant value in the manifest.
	KindInt128
)

func (Kind) String 

func (k Kind) String() string

String returns a stable lower-case kind name.

type Manifest 

type Manifest struct {
	Version     uint32
	RowCount    int
	MinTime     int64
	MaxTime     int64
	GranuleSize int
	Columns     []ColumnDesc
}

Manifest is the part descriptor: format version, row count, time range, granule size, and the per-column descriptors. It is serialized to the `{prefix}/manifest` object, CRC32C-checked, and written last to commit the part.

func DecodeManifest 

func DecodeManifest(src []byte) (Manifest, error)

DecodeManifest parses a manifest object. It verifies the CRC and bounds-checks every field, returning an ErrCorrupt-wrapping error on any malformed input — it never panics, so it is safe to fuzz on arbitrary bytes.

func (Manifest) Encode 

func (m Manifest) Encode(dst []byte) []byte

Encode appends the binary manifest to dst and returns the extended slice. Layout: 

[u32 magic][uvarint version][uvarint rowCount][varint minTime][varint maxTime]
[uvarint granuleSize][uvarint colCount]
  per column: [uvarint nameLen][name][byte kind][byte codec][byte compress][byte flags]
              [byte precisionBits if flagLossy][numeric min/max per kind][const value per kind if flagConst]
[u32 CRC32C over all the above]

type Marks 

type Marks struct {
	GranuleSize int
	Granules    []Granule
}

Marks is the sparse granule index for a part: granules of GranuleSize rows over the sort-key column (timestamp for the metrics vertical). It is serialized to the `{prefix}/marks` object.

func BuildMarks 

func BuildMarks(sortKey []int64, granuleSize int) Marks

BuildMarks computes the sparse index over a sort-key column, chunked into granules of granuleSize rows. The min/max are computed by scanning each granule, so correctness does not depend on sortKey being sorted (though for the metrics vertical it is). granuleSize must be > 0.

func DecodeMarks 

func DecodeMarks(src []byte) (Marks, error)

DecodeMarks parses a marks object. It verifies the CRC and bounds-checks every field, returning an ErrCorrupt-wrapping error on malformed input; it never panics.

func (Marks) Encode 

func (m Marks) Encode(dst []byte) []byte

Encode appends the binary marks index to dst. Layout: 

[u32 magic][uvarint version][uvarint granuleSize][uvarint count]
  per granule: [uvarint firstRow][varint minKey-prevMinKey][varint maxKey-minKey]
[u32 CRC32C]

Keys are delta-encoded across granules (sort key is non-decreasing) for compactness.

func (Marks) Overlapping 

func (m Marks) Overlapping(lo, hi int64) []Granule

Overlapping returns the granules whose [MinKey, MaxKey] range intersects the inclusive window [lo, hi]. It is the pruning primitive the fetcher (M3) uses to skip granules.

type PartOption 

type PartOption func(*PartWriter)

PartOption configures a PartWriter.

func WithCompression 

func WithCompression(alg compress.Algorithm) PartOption

WithCompression sets the default block-compression algorithm for columns that do not set Column.Compress (default none — the chunk codecs already compress well).

func WithCompressionLevel added in v0.4.0 

func WithCompressionLevel(level compress.Level) PartOption

WithCompressionLevel sets the compression level used by the block compressors (default compress.LevelDefault). It is decode-irrelevant — the reader reconstructs the decompressor from the per-column algorithm recorded in the manifest, regardless of the level data was written at — so a merge can rewrite cold parts at a higher ratio with no format change.

func WithGranuleSize 

func WithGranuleSize(n int) PartOption

WithGranuleSize sets the sparse-index granularity in rows (default 8192).

func WithSortKey 

func WithSortKey(name string) PartOption

WithSortKey names the int64 column that the marks index and time range are built over. If unset, the first int64 column is used.

type PartReader 

type PartReader struct {
	// contains filtered or unexported fields
}

PartReader reads a part written by WritePart. It loads only the manifest up front; columns and marks are read lazily, so a query touches only the objects it references (DESIGN.md §7).

func OpenPart 

func OpenPart(ctx context.Context, b backend.Backend, prefix string) (*PartReader, error)

OpenPart reads a part's manifest from b under prefix and returns a reader. It returns an error (wrapping ErrCorrupt or backend.ErrNotExist) if the manifest is absent or malformed — an incompletely written part (no manifest) is therefore not readable.

func (*PartReader) Column 

func (r *PartReader) Column(ctx context.Context, name string) (*ColumnReader, error)

Column returns a lazy reader for the named column. A constant column is synthesized from the manifest with no I/O; otherwise its object is read from the backend.

func (*PartReader) ColumnDescByName added in v0.20.0 

func (r *PartReader) ColumnDescByName(name string) (ColumnDesc, bool)

ColumnDescByName returns the named column's descriptor from the already-loaded manifest, without reading the column object — so a caller can check Const/Blocked/Codec before deciding whether to read and decode the column. ok is false for an unknown column.

func (*PartReader) ColumnNames 

func (r *PartReader) ColumnNames() []string

ColumnNames returns the column names in part order.

func (*PartReader) Manifest 

func (r *PartReader) Manifest() Manifest

Manifest returns the part's decoded manifest.

func (*PartReader) Marks 

func (r *PartReader) Marks(ctx context.Context) (Marks, error)

Marks reads and decodes the part's sparse granule index.

func (*PartReader) RowCount 

func (r *PartReader) RowCount() int

RowCount returns the number of rows in the part.

type PartWriter 

type PartWriter struct {
	// contains filtered or unexported fields
}

PartWriter accumulates columns and serializes them into a part's objects. Columns are added in order; their ordinal is their object key. The sort-key column (timestamp for metrics) drives the marks index and the manifest time range.

func NewPartWriter 

func NewPartWriter(opts ...PartOption) *PartWriter

NewPartWriter returns a PartWriter with the given options applied.

func (*PartWriter) AddColumn 

func (w *PartWriter) AddColumn(c Column) error

AddColumn appends a column. All columns in a part must have the same row count.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.