stats

package
v4.0.0-rc.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 4, 2026 License: AGPL-3.0 Imports: 23 Imported by: 0

Documentation

Overview

Package stats defines types for the data object stats section. The stats section holds per-section statistics for the data objects referenced in an index object.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func CheckSection

func CheckSection(section *dataobj.Section) bool

CheckSection returns true if the section is a stats section.

func Compare

func Compare(a, b Stat) int

Compare reports the canonical sort order of two Stat rows. It returns a negative value if a sorts before b, a positive value if a sorts after b, and zero if they share the full key.

Both rows must share the same SortSchema.

func FromRecordBatch

func FromRecordBatch(rec arrow.RecordBatch, dest []Stat) (int, error)

Types

type Builder

type Builder struct {
	// contains filtered or unexported fields
}

Builder accumulates Stat rows and encodes them into a section via a dataobj.SectionWriter.

Flush writes are done via Builder.Flush, which encodes all accumulated rows and writes them to the provided dataobj.SectionWriter.

func NewBuilder

func NewBuilder(metrics *Metrics, encode SectionEncoder) *Builder

NewBuilder creates a new Builder. encode is the SectionEncoder to use for encoding rows. metrics may be nil to disable instrumentation.

func (*Builder) Append

func (b *Builder) Append(stat Stat)

Append adds a Stat row to the builder.

func (*Builder) EstimatedSize

func (b *Builder) EstimatedSize() int

EstimatedSize returns an estimate of the encoded size of the accumulated rows in bytes, maintained incrementally by Builder.Append (see statSize).

func (*Builder) Flush

func (b *Builder) Flush(w dataobj.SectionWriter) (n int64, err error)

Flush sorts the accumulated rows with Compare, encodes them via the SectionEncoder, and writes the result to the provided dataobj.SectionWriter.

After a successful flush, the builder is reset.

func (*Builder) Reset

func (b *Builder) Reset()

Reset clears all accumulated rows and resets the builder to a fresh state.

func (*Builder) SetTenant

func (b *Builder) SetTenant(tenant string)

SetTenant sets the tenant for this builder.

func (*Builder) Tenant

func (b *Builder) Tenant() string

Tenant returns the tenant for this builder.

func (*Builder) Type

func (b *Builder) Type() dataobj.SectionType

Type returns the dataobj.SectionType of the stats builder.

type Column

type Column struct {
	Section *Section   // Section that contains the column.
	Name    string     // Optional name of the column (label name for dynamic columns).
	Type    ColumnType // Type of data in the column.
	// contains filtered or unexported fields
}

A Column represents one of the columns in the stats section. Valid columns can only be retrieved by calling Section.Columns.

Data in columns can be read by using a Reader.

type ColumnIndex

type ColumnIndex map[string]int

ColumnIndex maps Arrow field names to column indices within an arrow.RecordBatch. Build one with BuildColumnIndex and reuse it across rows decoded from the same schema.

func BuildColumnIndex

func BuildColumnIndex(schema *arrow.Schema) ColumnIndex

BuildColumnIndex constructs a ColumnIndex from an Arrow schema. The returned index can be passed to DecodeRow for efficient per-row decoding.

type ColumnStats

type ColumnStats struct {
	Name             string
	Type             string
	ValueType        string
	RowsCount        uint64
	Compression      string
	UncompressedSize uint64
	CompressedSize   uint64
	MetadataOffset   uint64
	MetadataSize     uint64
	ValuesCount      uint64
	Cardinality      uint64

	Pages []PageStats
}

ColumnStats provides statistics about a column in a section.

type ColumnType

type ColumnType int

ColumnType identifies the type of a column in the stats section.

const (
	ColumnTypeInvalid          ColumnType = iota // ColumnTypeInvalid is an invalid column type.
	ColumnTypeObjectPath                         // "object_path"
	ColumnTypeSectionIndex                       // "section_index"
	ColumnTypeSortSchema                         // "sort_schema"
	ColumnTypeMinTimestamp                       // "min_timestamp"
	ColumnTypeMaxTimestamp                       // "max_timestamp"
	ColumnTypeRowCount                           // "row_count"
	ColumnTypeUncompressedSize                   // "uncompressed_size"
	ColumnTypeLabel                              // "label" — dynamic; tag carries label name
	ColumnTypeShardBucket                        // "__shard_bucket__"
)

func ParseColumnType

func ParseColumnType(text string) (ColumnType, error)

ParseColumnType parses a ColumnType from a string. The expected string format is the same as what's returned by ColumnType.String.

func (ColumnType) String

func (ct ColumnType) String() string

String returns the human-readable name of [ct].

type Metrics

type Metrics struct {
	// contains filtered or unexported fields
}

Metrics instruments the stats section.

func NewMetrics

func NewMetrics() *Metrics

NewMetrics creates a new set of metrics for the stats section.

func (*Metrics) Observe

func (m *Metrics) Observe(ctx context.Context, section *Section) error

Observe observes section statistics for a given section.

func (*Metrics) Register

func (m *Metrics) Register(reg prometheus.Registerer) error

Register registers metrics to report to reg.

func (*Metrics) Unregister

func (m *Metrics) Unregister(reg prometheus.Registerer)

Unregister unregisters metrics from the provided Registerer.

type PageStats

type PageStats struct {
	UncompressedSize uint64
	CompressedSize   uint64
	CRC32            uint32
	RowsCount        uint64
	Encoding         string
	DataOffset       uint64
	DataSize         uint64
	ValuesCount      uint64
}

PageStats provides statistics about a page in a column.

type Reader

type Reader struct {
	// contains filtered or unexported fields
}

A Reader reads batches of rows from a stats Section. Returned arrow.RecordBatch values must be released by the caller after use.

func NewReader

func NewReader(opts ReaderOptions) *Reader

NewReader creates a new Reader. Options are not validated until the first call to Reader.Open.

func (*Reader) Close

func (r *Reader) Close() error

Close closes the Reader and releases any resources it holds.

func (*Reader) Columns

func (r *Reader) Columns() []*Column

Columns returns the [Column]s the Reader will read.

func (*Reader) Open

func (r *Reader) Open(ctx context.Context) error

Open initializes Reader resources. Must be called before Reader.Read. Safe to call multiple times.

func (*Reader) Read

func (r *Reader) Read(ctx context.Context, batchSize int) (arrow.RecordBatch, error)

Read reads up to batchSize rows from the section. At end of section returns (nil, io.EOF). May return a non-nil batch with io.EOF — callers should process the batch before checking the error. Returned batches must be released by the caller after use.

func (*Reader) Reset

func (r *Reader) Reset(opts ReaderOptions)

Reset reuses the Reader with new options.

func (*Reader) Schema

func (r *Reader) Schema() *arrow.Schema

Schema returns the arrow.Schema used by the Reader. Set on construction (via Reader.Reset) so it is valid before Reader.Open is called.

type ReaderOptions

type ReaderOptions struct {
	// Columns to read. Each column must belong to the same [Section].
	Columns []*Column

	// Allocator to use for allocating Arrow records. If nil,
	// [memory.DefaultAllocator] is used.
	Allocator memory.Allocator
}

ReaderOptions customizes the behavior of a Reader.

type RowReader

type RowReader struct {
	// contains filtered or unexported fields
}

RowReader reads Stat records from a stats Section one row at a time, in section order. It is a row-level cursor over the batch-level Reader. It implements iter.CloseIterator over Stat.

A RowReader is not safe for concurrent use.

func NewRowReader

func NewRowReader(ctx context.Context, sec *Section) *RowReader

NewRowReader creates a RowReader over all of sec's columns. The underlying reader is opened lazily on the first call to Next. The provided ctx governs all subsequent I/O (Open and Read).

func (*RowReader) At

func (r *RowReader) At() Stat

At returns the current record. Undefined if Next has not been called or if the last Next call returned false.

func (*RowReader) Close

func (r *RowReader) Close() error

Close releases the current batch and the underlying reader. Repeat calls return nil without re-closing.

func (*RowReader) Err

func (r *RowReader) Err() error

Err returns any error that caused iteration to end. nil on natural EOF.

func (*RowReader) Next

func (r *RowReader) Next() bool

Next advances the cursor. Returns false on exhaustion (natural EOF or any error). Subsequent calls continue to return false.

type Section

type Section struct {
	// contains filtered or unexported fields
}

Section represents an opened stats section.

func Open

func Open(ctx context.Context, section *dataobj.Section) (*Section, error)

Open opens a Section from an underlying dataobj.Section. Open returns an error if the section metadata could not be read or if the provided ctx is canceled.

func (*Section) Columns

func (s *Section) Columns() []*Column

Columns returns the set of Columns in the section. The slice of returned columns must not be mutated.

Unrecognized columns (e.g., when running older code against newer stats sections) are skipped.

type SectionEncoder

type SectionEncoder func(rows []Stat, enc *columnar.Encoder) error

SectionEncoder encodes a batch of sorted Stat rows into a columnar encoder.

func ColumnarSectionEncoder

func ColumnarSectionEncoder(pageSizeHint int, pageMaxRowCount int) SectionEncoder

ColumnarSectionEncoder returns a SectionEncoder that encodes Stat rows using [dataset.ColumnBuilder]s backed by a columnar.Encoder.

pageSizeHint and pageMaxRowCount control the page splitting behaviour of the underlying column builders.

type Stat

type Stat struct {
	ObjectPath       string
	SectionIndex     int64
	ShardBucket      uint32
	SortSchema       string
	Labels           map[string]string // Label values keyed by sort schema key name
	MinTimestamp     int64             // UnixNano
	MaxTimestamp     int64             // UnixNano
	RowCount         int64
	UncompressedSize int64
}

Stat represents a single row in the stats section. A row contains information about a single sortable bucket: (Shard, Tenant Sort Schema)

func DecodeRow

func DecodeRow(batch arrow.RecordBatch, columns ColumnIndex, rowIndex int) Stat

DecodeRow decodes a single row at rowIndex from an Arrow arrow.RecordBatch into a Stat. Dynamic label columns (fields with a ".label.utf8" suffix) are decoded into the Labels map.

Columns are looked up by field name via the provided ColumnIndex; missing or null values are left at their zero value.

type Stats

type Stats struct {
	UncompressedSize uint64
	CompressedSize   uint64

	Columns []ColumnStats
}

Stats provides statistics about a stats section.

func ReadStats

func ReadStats(ctx context.Context, section *Section) (Stats, error)

ReadStats returns statistics about the stats section. ReadStats returns an error if the stats section couldn't be inspected or if the provided ctx is canceled.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL