block

package
v0.40.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 22, 2026 License: Apache-2.0 Imports: 15 Imported by: 0

Documentation

Overview

Package block implements the L2 immutable columnar part format (DESIGN.md §3, §14 M1): per-column streams with min/max stats and constant-column collapse, a sparse granule mark index, and an atomic manifest written last. Not yet implemented (M1).

Index

Constants

This section is empty.

Variables

View Source 
var ErrCorrupt = errors.New("block: corrupt metadata")

ErrCorrupt is returned when a manifest (or any part metadata) fails to parse: bad magic, CRC mismatch, truncation, or an out-of-range field.

Functions

func WritePart 

func WritePart(ctx context.Context, b backend.Backend, prefix string, w *PartWriter) error

WritePart serializes the writer's columns and writes the part's objects under prefix on b. Column and marks objects are written first; the manifest is written LAST so the part only becomes readable once fully committed.

func WriteStreamPart added in v0.37.0 

func WriteStreamPart(ctx context.Context, b backend.Backend, prefix string, w *StreamWriter) error

WriteStreamPart serializes w and writes the part under prefix on b, in the same order and under the same keys as WritePart — manifest last, so the part becomes readable only once committed.

For a writer from NewStreamWriterTo, b and prefix must be the ones it was given: its column objects commit here, and only the marks and manifest are written from memory. A failure leaves the part uncommitted (no manifest) but may leave column objects behind, exactly as a failure partway through WritePart does.

Types

type Column 

type Column struct {
	Name     string
	Kind     Kind
	Int64    []int64
	Float64  []float64
	Bytes    [][]byte
	Int128   []chunk.U128
	Codec    chunk.Codec
	Compress compress.Algorithm
	// BytesBlob/BytesOffsets are the blob+offsets alternative to Bytes for a KindBytes column:
	// cell i is BytesBlob[BytesOffsets[i]:BytesOffsets[i+1]], with len(BytesOffsets) == rows+1 —
	// the head-buffer byte-column layout, accepted directly so a flush encodes straight from the
	// blob without materializing a [][]byte view per row. Offsets index the whole blob and need not
	// start at 0, so a row range of a larger column is a valid input. Used only when
	// Bytes is nil; the encoded object is byte-identical to the equivalent Bytes input.
	BytesBlob    []byte
	BytesOffsets []int32
	// BytesDict/BytesIDs are the split (dictionary) alternative to Bytes for a KindBytes column:
	// cell i is BytesDict[BytesIDs[i]] — the shape a merge already holds when it reads a
	// dictionary-encoded byte column, accepted directly so the writers build their dictionaries by
	// remapping entry indices instead of re-hashing every row. Used only when Bytes is nil and
	// BytesOffsets is empty; the encoded object is byte-identical to the equivalent Bytes input.
	//
	// BytesDict must be distinct by value: the encoders deduplicate by index, so two entries holding
	// equal bytes yield two dictionary entries — a valid stream, but not the same one. Only
	// [chunk.CodecDict] accepts this form.
	BytesDict [][]byte
	BytesIDs  []int32
	// AutoCodec, when set on a float64 column with no explicit Codec, picks the smaller of the
	// lossless float codecs (Gorilla XOR vs scaled-decimal+delta) by trial-encoding — so an
	// integer-valued or low-precision column (e.g. a counter) takes the far denser decimal path
	// while a high-entropy column keeps Gorilla. Lossless either way (see [chooseFloatCodec]).
	AutoCodec bool
	// FloatPrecisionBits, when in 1..63 on an AutoCodec float column, requests *lossy* encoding:
	// the scaled-decimal codec retains only this many significant mantissa bits (fewer ⇒ denser,
	// less accurate), competing against lossless Gorilla so the result is never worse than
	// lossless. 0 (or ≥64) means lossless. The budget is recorded in the descriptor so the merge
	// engine reaches a fixed point (it never re-applies a budget it has already met). Set per age
	// tier by the merge engine so only old data trades accuracy for size.
	FloatPrecisionBits uint8
	// Block requests block-framed encoding: the column is split into blockRows-row blocks (the
	// part's granule size), each an independent codec stream, so a reader can decode one block at a
	// time (sub-part seek). Only the per-row sequential codecs (DoD/T64 int64, Gorilla/decimal
	// float64) are blockable — the metric ts/value/sf columns. The zero value keeps the prior
	// single-stream layout. See blockcolumn.go.
	Block bool
}

Column is an input column for a part: a name, a physical Kind, and the matching typed slice (exactly one of Int64/Float64/Bytes per Kind). Codec and Compress are optional overrides; the zero values select the per-kind default codec and no block compression (the chunk codecs already compress well).

type ColumnDesc 

type ColumnDesc struct {
	Name     string
	Kind     Kind
	Codec    chunk.Codec
	Compress compress.Algorithm
	Const    bool

	// Constant value, set iff Const, by Kind.
	ConstInt64   int64
	ConstFloat64 float64
	ConstBytes   []byte

	// Numeric min/max (KindInt64/KindFloat64). Unused for KindBytes.
	MinInt64, MaxInt64     int64
	MinFloat64, MaxFloat64 float64

	// FloatPrecisionBits is the lossy precision budget a float column was encoded under (the
	// significant mantissa bits retained): 0 ⇒ lossless. Persisted only when non-zero (a
	// flag-gated byte), so lossless parts keep their byte-for-byte layout. The merge engine reads
	// it as the fixed point for age-tiered precision — it never re-coarsens a part already at or
	// below the target budget.
	FloatPrecisionBits uint8

	// Blocked marks a block-framed column: its object is a [blockDir] + per-block codec streams
	// (see blockcolumn.go) instead of a single stream, so a reader can decode one block at a time.
	// Persisted via [flagBlocked]; clear on the prior single-stream layout.
	Blocked bool

	// Framed marks a blocked column written with the frame-packed directory, where a compression
	// frame spans several decode granules so the compressor sees more than one granule of context.
	// Persisted via [flagFramed]; clear on the older one-block-per-granule layout, which is still
	// read. Set by the writer on every blocked column it produces.
	Framed bool

	// SharedDict marks a block-framed bytes column carrying one dictionary for the whole column
	// ahead of its frames, each granule holding ids into it or self-encoding. Persisted via
	// [flagSharedDict].
	SharedDict bool

	// Footer marks a framed column whose directory trails the frames instead of leading them, the
	// layout a streaming writer emits (see [flagFooter]). Persisted via [flagFooter]; clear on a
	// column written whole, which keeps the directory-first layout byte for byte.
	Footer bool

	// Bytes is the size of the column's backend object, 0 when unknown (a constant column has no
	// object, and a part written before [flagBytes] existed did not record one). Persisted only when
	// non-zero. It exists so opening a column for ranged reads costs no round trip of its own.
	Bytes int64

	// Checked reports that the column's object carries CRC32C checksums over its data — per
	// compression frame for a framed column, a trailing one over the whole object otherwise — which
	// the read path verifies. It is not a persisted field of its own: it follows from the enclosing
	// manifest's version, and the writer sets it on every column it produces.
	Checked bool

	// Level is the compression level the column's data was written at (0 ⇒ the algorithm default, or
	// no compression). Persisted only when non-zero, via [flagLevel]. Decode ignores it; the merge
	// engine reads it as the fixed point of the compression ladder.
	Level compress.Level
}

ColumnDesc describes one column in a part: its identity, codecs, constant value (if collapsed), and numeric min/max stats. Data offsets are absent — each column is its own backend object keyed by ordinal (DESIGN.md §14 M1 multi-key layout).

type ColumnReader 

type ColumnReader struct {
	// contains filtered or unexported fields
}

ColumnReader gives lazy, decode-on-demand access to one column of a part. Constant columns are synthesized from the manifest with no I/O; other columns are decompressed and decoded only when an accessor is called (DESIGN.md §7: decode only what the query touches). It is created by PartReader.Column.

func (*ColumnReader) BlockDecoder added in v0.20.0 

func (r *ColumnReader) BlockDecoder() (*Decoder, error)

BlockDecoder returns a per-block decoder over a blocked column (parsing the directory once), for a caller that decodes individual blocks. It errors on an unblocked column.

func (*ColumnReader) BlockRows added in v0.20.0 

func (r *ColumnReader) BlockRows() (int, error)

BlockRows returns the column's block size in rows, or 0 for an unblocked column.

func (*ColumnReader) Blocked added in v0.20.0 

func (r *ColumnReader) Blocked() bool

Blocked reports whether the column is block-framed (and so supports DecodeBlocks*/BlockRows).

func (*ColumnReader) Bytes 

func (r *ColumnReader) Bytes() (*chunk.DictColumn, error)

Bytes decodes the column into its split chunk.DictColumn form (unique entries + a per-row id array), deferring the per-row gather to chunk.DictColumn.At. A constant column is synthesized as a single-entry dictionary. It errors if the column is not KindBytes.

func (*ColumnReader) BytesRaw added in v0.32.0 

func (r *ColumnReader) BytesRaw() (blob []byte, width int, err error)

BytesRaw decodes a chunk.CodecBytesRaw-encoded KindBytes column into its flat fixed-width form: the contiguous rows×width blob and width, with no per-row [][]byte headers built over it (unlike ColumnReader.Bytes) — the shape a stride/SIMD equality scan (e.g. github.com/oteldb/storage/internal/simd.EqualFixed16) operates on directly. It errors if the column is not KindBytes, its codec is not chunk.CodecBytesRaw, or it is block-framed (no blocked encoder exists yet for byte columns — see ColumnReader.DecodeBlocksInt64 for the int64/float64 precedent this would follow).

func (*ColumnReader) Const 

func (r *ColumnReader) Const() (any, bool)

Const returns the column's constant value (int64/float64/[]byte per Kind) and true if the column was constant-collapsed; otherwise (nil, false).

func (*ColumnReader) DecodeBlocksBytes added in v0.37.0 

func (r *ColumnReader) DecodeBlocksBytes(blocks []int) (*chunk.DictColumn, error)

DecodeBlocksBytes decodes only the named granules of a block-framed bytes column, merged into one chunk.DictColumn over their concatenated rows. It is the bytes counterpart of ColumnReader.DecodeBlocksInt64 — the seek primitive a time-pruned query uses to read a fraction of a column instead of all of it.

The returned column covers the selected rows *packed together*, not their positions in the part. Use ColumnReader.DecodeBlocksBytesIntoColumn when part row indices must stay valid.

func (*ColumnReader) DecodeBlocksBytesIntoColumn added in v0.37.0 

func (r *ColumnReader) DecodeBlocksBytesIntoColumn(blocks []int) (*chunk.DictColumn, error)

DecodeBlocksBytesIntoColumn decodes the named granules into a column spanning *every* row of the part, each granule at its own row offset.

Unlike ColumnReader.DecodeBlocksBytes, which packs the selection, this keeps part row indices valid — the property the int64 path gets for free by decoding into the destination at absolute offsets. A fetch that located its rows through the part's row-range index and its marks can then prune the decode without renumbering anything it already resolved.

Rows outside the selected granules hold an **unspecified** value: the caller asked for these granules and reads only their rows. This matches ColumnReader.DecodeBlocksInt64, where unselected rows keep whatever the destination held. Zeroing them would cost a pass over the rows the pruning exists to avoid touching.

func (*ColumnReader) DecodeBlocksFloat64 added in v0.20.0 

func (r *ColumnReader) DecodeBlocksFloat64(dst []float64, blocks []int) ([]float64, error)

DecodeBlocksFloat64 is the float64 analog of ColumnReader.DecodeBlocksInt64.

func (*ColumnReader) DecodeBlocksInt64 added in v0.20.0 

func (r *ColumnReader) DecodeBlocksInt64(dst []int64, blocks []int) ([]int64, error)

DecodeBlocksInt64 decodes only the given block indices of a blocked column into dst (reused), and returns a full-length (Len()) slice with those blocks' row spans populated — the rest of dst is left as-is (a caller reads only the rows it selected, which fall in the requested blocks). An unblocked/const column has no blocks to skip, so it decodes whole. blocks must be in range.

func (*ColumnReader) Float64 

func (r *ColumnReader) Float64(dst []float64) ([]float64, error)

Float64 decodes the column into dst (reusing its capacity) and returns the result. It errors if the column is not KindFloat64.

func (*ColumnReader) FloatCursor added in v0.15.0 

func (r *ColumnReader) FloatCursor() (chunk.FloatDecoder, error)

FloatCursor returns a forward cursor over a KindFloat64 column (Gorilla or scaled-decimal). A constant-collapsed column yields a repeating cursor. It is the streaming-merge form of Float64.

func (*ColumnReader) Frames added in v0.37.0 

func (r *ColumnReader) Frames() ([]FrameExtent, error)

Frames maps the column's rows onto its compression frames. A frame is the unit compression is applied to, so it is the finest granularity at which a caller can attribute a column's *compressed* bytes to a subset of its rows — nothing below it is separable, since the entropy coder shares state across the whole frame.

A column that is not block-framed (constant, or written as one stream) reports a single extent covering every row and the whole object. The extents' Bytes sum to the frames' bytes, which is less than ColumnReader.ObjectBytes by the directory (and, for a shared-dictionary column, by the dictionary) — neither belongs to any single frame.

func (*ColumnReader) ID128 

func (r *ColumnReader) ID128(dst []chunk.U128) ([]chunk.U128, error)

ID128 decodes the column into dst (reusing its capacity) and returns the result. It errors if the column is not KindInt128. Id columns are never constant-collapsed, so the value always comes from the decoded RLE stream.

func (*ColumnReader) Int64 

func (r *ColumnReader) Int64(dst []int64) ([]int64, error)

Int64 decodes the column into dst (reusing its capacity) and returns the result. It errors if the column is not KindInt64.

func (*ColumnReader) Kind 

func (r *ColumnReader) Kind() Kind

Kind reports the column's physical type.

func (*ColumnReader) Len 

func (r *ColumnReader) Len() int

Len reports the column's row count.

func (*ColumnReader) ObjectBytes added in v0.37.0 

func (r *ColumnReader) ObjectBytes() int64

ObjectBytes is the column object's size as stored on the backend, 0 for a constant column (which has no object).

func (*ColumnReader) RangeFloat64 added in v0.20.0 

func (r *ColumnReader) RangeFloat64(dst []float64, lo, hi int) ([]float64, error)

RangeFloat64 decodes only rows [lo,hi) of a float64 column. See ColumnReader.RangeInt64.

func (*ColumnReader) RangeInt64 added in v0.20.0 

func (r *ColumnReader) RangeInt64(dst []int64, lo, hi int) ([]int64, error)

RangeInt64 decodes only rows [lo,hi) of an int64 column into a buffer reusing dst's capacity. For a blocked column it decodes just the blocks spanning the range (sub-part seek); for an unblocked one it decodes the whole column and slices, so callers get a uniform seek API regardless of layout. Requires 0 ≤ lo < hi ≤ Len().

func (*ColumnReader) TsCursor added in v0.15.0 

func (r *ColumnReader) TsCursor() (chunk.TsCursor, error)

TsCursor returns a forward cursor over a KindInt64 timestamp column (delta-of-delta). A constant-collapsed column yields a repeating cursor. It is the streaming-merge form of Int64: same decode, but one row at a time so a merge holds only one series range resident per part.

type Decoder added in v0.20.0 

type Decoder struct {
	// contains filtered or unexported fields
}

Decoder decodes individual blocks of a blocked column, parsing the directory once so a caller that decodes several blocks (e.g. a per-block cache filling its misses) does not re-parse it per block. Obtain one via [ColumnReader.Decoder]; it holds the column's already-read object.

func (*Decoder) BlockRows added in v0.20.0 

func (d *Decoder) BlockRows() int

BlockRows returns the column's nominal block size in rows.

func (*Decoder) BlockSpan added in v0.20.0 

func (d *Decoder) BlockSpan(blk int) (lo, hi int)

BlockSpan returns block blk's half-open row range [lo, hi) in the column.

func (*Decoder) DecodeFloat64 added in v0.20.0 

func (d *Decoder) DecodeFloat64(blk int) ([]float64, error)

DecodeFloat64 decodes block blk into a fresh slice (for a float64 column).

func (*Decoder) DecodeFloat64Into added in v0.23.0 

func (d *Decoder) DecodeFloat64Into(blk int, dst []float64) ([]float64, error)

DecodeFloat64Into is the float64 analog of Decoder.DecodeInt64Into.

func (*Decoder) DecodeInt64 added in v0.20.0 

func (d *Decoder) DecodeInt64(blk int) ([]int64, error)

DecodeInt64 decodes block blk into a fresh slice (for an int64 column).

func (*Decoder) DecodeInt64Into added in v0.23.0 

func (d *Decoder) DecodeInt64Into(blk int, dst []int64) ([]int64, error)

DecodeInt64Into decodes block blk into dst (for an int64 column), reusing dst's backing array when it has room for the block's rows — so a caller drawing dst from a pool decodes without allocating the output. Passing a nil dst is equivalent to Decoder.DecodeInt64.

func (*Decoder) NumBlocks added in v0.20.0 

func (d *Decoder) NumBlocks() int

NumBlocks returns the column's block count.

type FrameExtent added in v0.37.0 

type FrameExtent struct {
	StartRow int   // inclusive
	EndRow   int   // exclusive
	Bytes    int64 // compressed size of the frame
}

FrameExtent is one compression frame of a column: the row span it covers and the compressed byte span it occupies in the column object.

type Granule 

type Granule struct {
	FirstRow int
	MinKey   int64
	MaxKey   int64
}

Granule is one entry of the sparse mark index: the first row of a fixed-size run of rows, plus the min/max of the part's sort-key column over that run. The min/max let a query prune whole granules whose key range cannot intersect its window (_ref/docs/storage-engine.md §2, ClickHouse marks / Parquet row-group stats).

type Kind 

type Kind uint8

Kind is the physical type of a column's values (DESIGN.md §6). It selects which typed slice a Column carries and which codec family applies. Values are persisted in the manifest; never reorder. 

const (
	// KindInt64 is an int64 column (timestamps, counters, series ids).
	KindInt64 Kind = iota
	// KindFloat64 is a float64 column (gauge/sum values).
	KindFloat64
	// KindBytes is a []byte column (low-cardinality attributes, strings).
	KindBytes
	// KindInt128 is a 128-bit id column ([chunk.U128]), e.g. the SeriesID sort key of a
	// metric part. RLE-coded; carries no min/max or constant value in the manifest.
	KindInt128
)

func (Kind) String 

func (k Kind) String() string

String returns a stable lower-case kind name.

type Manifest 

type Manifest struct {
	Version     uint32
	RowCount    int
	MinTime     int64
	MaxTime     int64
	GranuleSize int
	Columns     []ColumnDesc
	// DiskBytes is the encoded size of the part's column and marks objects, excluding the manifest
	// itself (which cannot size itself) and the engine's sidecars. The merge cap is denominated in
	// it, so the cap means bytes on disk rather than a row estimate.
	//
	// Trailing, so a manifest without it reads as 0 and an older reader ignores it.
	DiskBytes int64
	// RawBytes is the part's *decoded* footprint: the bytes its column values occupy in memory,
	// which is what a merge's working set is made of. DiskBytes cannot stand in for it — the ratio
	// between them is the compression ratio, which varies per column and per dataset — so a merge
	// bounded by memory is denominated in this instead.
	//
	// Trailing, like DiskBytes, and 0 in a manifest written before it existed.
	RawBytes int64
}

Manifest is the part descriptor: format version, row count, time range, granule size, and the per-column descriptors. It is serialized to the `{prefix}/manifest` object, CRC32C-checked, and written last to commit the part.

func DecodeManifest 

func DecodeManifest(src []byte) (Manifest, error)

DecodeManifest parses a manifest object. It verifies the CRC and bounds-checks every field, returning an ErrCorrupt-wrapping error on any malformed input — it never panics, so it is safe to fuzz on arbitrary bytes.

func (Manifest) Encode 

func (m Manifest) Encode(dst []byte) []byte

Encode appends the binary manifest to dst and returns the extended slice. Layout: 

[u32 magic][uvarint version][uvarint rowCount][varint minTime][varint maxTime]
[uvarint granuleSize][uvarint colCount]
  per column: [uvarint nameLen][name][byte kind][byte codec][byte compress][byte flags]
              [byte precisionBits if flagLossy][byte level if flagLevel]
              [uvarint objectBytes if flagBytes]
              [numeric min/max per kind][const value per kind if flagConst]
[uvarint diskBytes][uvarint rawBytes]
[u32 CRC32C over all the above]

type Marks 

type Marks struct {
	GranuleSize int
	Granules    []Granule
}

Marks is the sparse granule index for a part: granules of GranuleSize rows over the sort-key column (timestamp for the metrics vertical). It is serialized to the `{prefix}/marks` object.

func BuildMarks 

func BuildMarks(sortKey []int64, granuleSize int) Marks

BuildMarks computes the sparse index over a sort-key column, chunked into granules of granuleSize rows. The min/max are computed by scanning each granule, so correctness does not depend on sortKey being sorted (though for the metrics vertical it is). granuleSize must be > 0.

func DecodeMarks 

func DecodeMarks(src []byte) (Marks, error)

DecodeMarks parses a marks object. It verifies the CRC and bounds-checks every field, returning an ErrCorrupt-wrapping error on malformed input; it never panics.

func (Marks) Encode 

func (m Marks) Encode(dst []byte) []byte

Encode appends the binary marks index to dst. Layout: 

[u32 magic][uvarint version][uvarint granuleSize][uvarint count]
  per granule: [uvarint firstRow][varint minKey-prevMinKey][varint maxKey-minKey]
[u32 CRC32C]

Keys are delta-encoded across granules (sort key is non-decreasing) for compactness.

func (Marks) Overlapping 

func (m Marks) Overlapping(lo, hi int64) []Granule

Overlapping returns the granules whose [MinKey, MaxKey] range intersects the inclusive window [lo, hi]. It is the pruning primitive the fetcher (M3) uses to skip granules.

type PartOption 

type PartOption func(*partConfig)

PartOption configures a PartWriter or a StreamWriter.

func WithCompressBlockBytes added in v0.37.0 

func WithCompressBlockBytes(n int) PartOption

WithCompressBlockBytes sets the minimum uncompressed bytes packed into one compression frame of a block-framed column (default [defaultCompressBlockBytes]). It decouples the compression unit from the decode granule (WithGranuleSize): a granule stays the smallest decodable slice, while a frame gathers enough consecutive granules to give the compressor real context. Decode-compatible either way — the directory records the packing.

func WithCompression 

func WithCompression(alg compress.Algorithm) PartOption

WithCompression sets the default block-compression algorithm for columns that do not set Column.Compress (default none — the chunk codecs already compress well).

func WithCompressionLevel added in v0.4.0 

func WithCompressionLevel(level compress.Level) PartOption

WithCompressionLevel sets the compression level used by the block compressors (default compress.LevelDefault). It is decode-irrelevant — the reader reconstructs the decompressor from the per-column algorithm recorded in the manifest, regardless of the level data was written at — so a merge can rewrite cold parts at a higher ratio with no format change.

func WithGranuleSize 

func WithGranuleSize(n int) PartOption

WithGranuleSize sets the sparse-index granularity in rows (default 8192).

func WithSortKey 

func WithSortKey(name string) PartOption

WithSortKey names the int64 column that the marks index and time range are built over. If unset, the first int64 column is used.

type PartReader 

type PartReader struct {
	// contains filtered or unexported fields
}

PartReader reads a part written by WritePart. It loads only the manifest up front; columns and marks are read lazily, so a query touches only the objects it references (DESIGN.md §7).

func OpenPart 

func OpenPart(ctx context.Context, b backend.Backend, prefix string) (*PartReader, error)

OpenPart reads a part's manifest from b under prefix and returns a reader. It returns an error (wrapping ErrCorrupt or backend.ErrNotExist) if the manifest is absent or malformed — an incompletely written part (no manifest) is therefore not readable.

func (*PartReader) Column 

func (r *PartReader) Column(ctx context.Context, name string) (*ColumnReader, error)

Column returns a lazy reader for the named column. A constant column is synthesized from the manifest with no I/O; otherwise its object is read from the backend.

func (*PartReader) ColumnBlocks added in v0.37.0 

func (r *PartReader) ColumnBlocks(ctx context.Context, name string) (*Decoder, error)

ColumnBlocks returns a per-block decoder for the named column that reads only the block directory up front and fetches each compression frame by ranged read as blocks are decoded.

It is the read counterpart of the streaming writer: the whole-object PartReader.Column is right when a caller decodes the whole column, and this is right when it decodes a fraction of it — the query path, where the matched series' rows lie in a handful of granules. The returned decoder holds ctx for its frame reads, so it must not outlive the operation that opened it.

The column must be block-framed and not constant-collapsed; PartReader.Column handles those.

func (*PartReader) ColumnDescByName added in v0.20.0 

func (r *PartReader) ColumnDescByName(name string) (ColumnDesc, bool)

ColumnDescByName returns the named column's descriptor from the already-loaded manifest, without reading the column object — so a caller can check Const/Blocked/Codec before deciding whether to read and decode the column. ok is false for an unknown column.

func (*PartReader) ColumnNames 

func (r *PartReader) ColumnNames() []string

ColumnNames returns the column names in part order.

func (*PartReader) Manifest 

func (r *PartReader) Manifest() Manifest

Manifest returns the part's decoded manifest.

func (*PartReader) Marks 

func (r *PartReader) Marks(ctx context.Context) (Marks, error)

Marks reads and decodes the part's sparse granule index.

func (*PartReader) RowCount 

func (r *PartReader) RowCount() int

RowCount returns the number of rows in the part.

type PartWriter 

type PartWriter struct {
	// contains filtered or unexported fields
}

PartWriter accumulates columns and serializes them into a part's objects. Columns are added in order; their ordinal is their object key. The sort-key column (timestamp for metrics) drives the marks index and the manifest time range.

func NewPartWriter 

func NewPartWriter(opts ...PartOption) *PartWriter

NewPartWriter returns a PartWriter with the given options applied.

func (*PartWriter) AddColumn 

func (w *PartWriter) AddColumn(c Column) error

AddColumn appends a column. All columns in a part must have the same row count.

type StreamWriter added in v0.37.0 

type StreamWriter struct {
	// contains filtered or unexported fields
}

StreamWriter builds a part incrementally: the schema is declared up front and each column encodes a granule as soon as one fills, so the working set is the encoded part rather than its uncompressed rows. See ARCH.md ("Two writers") for what that buys and what it costs.

Only encodings that restart per granule can stream: blocked KindInt64/KindFloat64, and KindInt128, whose RLE codec is fed runs. Output matches PartWriter's byte for byte except for an Column.AutoCodec column.

Constructed with NewStreamWriter it still accumulates the *encoded* part in memory, which caps the part at what the process can hold. NewStreamWriterTo removes that ceiling: sealed compression frames go to the backend as they seal, leaving only one frame per column resident.

func NewStreamWriter added in v0.37.0 

func NewStreamWriter(opts ...PartOption) *StreamWriter

NewStreamWriter returns a StreamWriter with the given options applied. It takes the same options as NewPartWriter and lays a part out identically.

func NewStreamWriterTo added in v0.37.0 

func NewStreamWriterTo(ctx context.Context, b backend.Backend, prefix string, opts ...PartOption) *StreamWriter

NewStreamWriterTo returns a StreamWriter that hands each column's compression frames to b under prefix as they seal, rather than holding the encoded part until the end. Resident memory becomes one unsealed frame per column plus the column directories (kilobytes per column, one entry per granule and frame) instead of the whole part, so the part's size stops being bounded by the process's memory.

The column objects it produces carry their directory as a footer (ColumnDesc.Footer) — with the directory leading, no byte of the object could be final until the last frame sealed. A column that turns out to be constant is written under neither layout: it collapses into the manifest, so it stays buffered until it is known non-constant (which for real data is the second row).

ctx spans the writer's whole life, appends included. Nothing is stored under prefix until the writer finishes; a writer that will not finish must be released with StreamWriter.Abort. WriteStreamPart must be called with the same b and prefix.

func (*StreamWriter) Abort added in v0.37.0 

func (w *StreamWriter) Abort()

Abort releases every object a streaming writer has under way without publishing any of them. It is idempotent, and a no-op on a writer that finished or never streamed, so `defer w.Abort()` is the correct cleanup.

func (*StreamWriter) AddColumn added in v0.37.0 

func (w *StreamWriter) AddColumn(c Column) error

AddColumn declares a column; only c's schema fields are read, since rows arrive through the Append methods. A column's ordinal — the index those methods take — is its object key.

An AutoCodec column streams under both candidate codecs and keeps the denser, so the choice is still made over the whole column rather than a prefix. It compares block-framed sizes where PartWriter compares whole-column ones, so the two can pick differently in a marginal case; both are lossless.

func (*StreamWriter) AppendFloat64 added in v0.37.0 

func (w *StreamWriter) AppendFloat64(i int, vals []float64) error

AppendFloat64 appends vals to the i-th column, encoding every granule they complete.

func (*StreamWriter) AppendInt64 added in v0.37.0 

func (w *StreamWriter) AppendInt64(i int, vals []int64) error

AppendInt64 appends vals to the i-th column, encoding every granule they complete.

func (*StreamWriter) AppendU128Run added in v0.37.0 

func (w *StreamWriter) AppendU128Run(i int, v chunk.U128, count int) error

AppendU128Run appends count copies of v to the i-th column as one run, which is how an id column streams without materializing its rows. A count ≤ 0 is a no-op.

func (*StreamWriter) EncodedBytes added in v0.37.0 

func (w *StreamWriter) EncodedBytes() int64

EncodedBytes returns the compressed bytes accumulated so far, what a caller seals a part on when the target is a size on disk.

It is a lower bound: low by at most one unsealed frame per column plus the id column's RLE stream, tens of KiB against a cap in the hundreds of MiB.

func (*StreamWriter) OmitConstColumn added in v0.37.0 

func (w *StreamWriter) OmitConstColumn(v float64) error

OmitConstColumn drops the last declared column from the part entirely if every row appended to it turned out to be v.

It covers a column the format leaves *absent* rather than constant — the sampling weight, which a reader defaults to 1. A present-but-constant column is not block-framed and would drop readers onto the whole-part decode path. Only the last column may be omitted; dropping an earlier one would renumber the object keys after it.

func (*StreamWriter) ResidentBytes added in v0.37.0 

func (w *StreamWriter) ResidentBytes() int64

ResidentBytes returns what the writer is holding in RAM right now: per column, the frame still being filled, the block directory it is building, and — for a buffered writer, or a column not yet proven non-constant — the frames sealed so far.

It is what a caller seals a part on when the bound is memory rather than a size on disk. For a NewStreamWriterTo writer it settles at a few hundred KiB per column and stops tracking the part; for a buffered one it tracks StreamWriter.EncodedBytes and grows without limit.

func (*StreamWriter) Rows added in v0.37.0 

func (w *StreamWriter) Rows() int

Rows returns the row count appended to the first declared column, or 0 if none is declared.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.