Documentation
¶
Overview ¶
Package stats defines types for the data object stats section. The stats section holds per-section statistics for the data objects referenced in an index object.
Index ¶
- func CheckSection(section *dataobj.Section) bool
- func Compare(a, b Stat) int
- func FromRecordBatch(rec arrow.RecordBatch, dest []Stat) (int, error)
- type Builder
- type Column
- type ColumnIndex
- type ColumnStats
- type ColumnType
- type Metrics
- type PageStats
- type Reader
- type ReaderOptions
- type RowReader
- type Section
- type SectionEncoder
- type Stat
- type Stats
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func CheckSection ¶
CheckSection returns true if the section is a stats section.
func Compare ¶
Compare reports the canonical sort order of two Stat rows. It returns a negative value if a sorts before b, a positive value if a sorts after b, and zero if they share the full key.
Both rows must share the same SortSchema.
func FromRecordBatch ¶
func FromRecordBatch(rec arrow.RecordBatch, dest []Stat) (int, error)
Types ¶
type Builder ¶
type Builder struct {
// contains filtered or unexported fields
}
Builder accumulates Stat rows and encodes them into a section via a dataobj.SectionWriter.
Flush writes are done via Builder.Flush, which encodes all accumulated rows and writes them to the provided dataobj.SectionWriter.
func NewBuilder ¶
func NewBuilder(metrics *Metrics, encode SectionEncoder) *Builder
NewBuilder creates a new Builder. encode is the SectionEncoder to use for encoding rows. metrics may be nil to disable instrumentation.
func (*Builder) EstimatedSize ¶
EstimatedSize returns an estimate of the encoded size of the accumulated rows in bytes, maintained incrementally by Builder.Append (see statSize).
func (*Builder) Flush ¶
func (b *Builder) Flush(w dataobj.SectionWriter) (n int64, err error)
Flush sorts the accumulated rows with Compare, encodes them via the SectionEncoder, and writes the result to the provided dataobj.SectionWriter.
After a successful flush, the builder is reset.
func (*Builder) Reset ¶
func (b *Builder) Reset()
Reset clears all accumulated rows and resets the builder to a fresh state.
func (*Builder) Type ¶
func (b *Builder) Type() dataobj.SectionType
Type returns the dataobj.SectionType of the stats builder.
type Column ¶
type Column struct {
Section *Section // Section that contains the column.
Name string // Optional name of the column (label name for dynamic columns).
Type ColumnType // Type of data in the column.
// contains filtered or unexported fields
}
A Column represents one of the columns in the stats section. Valid columns can only be retrieved by calling Section.Columns.
Data in columns can be read by using a Reader.
type ColumnIndex ¶
ColumnIndex maps Arrow field names to column indices within an arrow.RecordBatch. Build one with BuildColumnIndex and reuse it across rows decoded from the same schema.
func BuildColumnIndex ¶
func BuildColumnIndex(schema *arrow.Schema) ColumnIndex
BuildColumnIndex constructs a ColumnIndex from an Arrow schema. The returned index can be passed to DecodeRow for efficient per-row decoding.
type ColumnStats ¶
type ColumnStats struct {
Name string
Type string
ValueType string
RowsCount uint64
Compression string
UncompressedSize uint64
CompressedSize uint64
MetadataOffset uint64
MetadataSize uint64
ValuesCount uint64
Cardinality uint64
Pages []PageStats
}
ColumnStats provides statistics about a column in a section.
type ColumnType ¶
type ColumnType int
ColumnType identifies the type of a column in the stats section.
const ( ColumnTypeInvalid ColumnType = iota // ColumnTypeInvalid is an invalid column type. ColumnTypeObjectPath // "object_path" ColumnTypeSectionIndex // "section_index" ColumnTypeSortSchema // "sort_schema" ColumnTypeMinTimestamp // "min_timestamp" ColumnTypeMaxTimestamp // "max_timestamp" ColumnTypeRowCount // "row_count" ColumnTypeUncompressedSize // "uncompressed_size" ColumnTypeLabel // "label" — dynamic; tag carries label name ColumnTypeShardBucket // "__shard_bucket__" )
func ParseColumnType ¶
func ParseColumnType(text string) (ColumnType, error)
ParseColumnType parses a ColumnType from a string. The expected string format is the same as what's returned by ColumnType.String.
func (ColumnType) String ¶
func (ct ColumnType) String() string
String returns the human-readable name of [ct].
type Metrics ¶
type Metrics struct {
// contains filtered or unexported fields
}
Metrics instruments the stats section.
func NewMetrics ¶
func NewMetrics() *Metrics
NewMetrics creates a new set of metrics for the stats section.
func (*Metrics) Register ¶
func (m *Metrics) Register(reg prometheus.Registerer) error
Register registers metrics to report to reg.
func (*Metrics) Unregister ¶
func (m *Metrics) Unregister(reg prometheus.Registerer)
Unregister unregisters metrics from the provided Registerer.
type PageStats ¶
type PageStats struct {
UncompressedSize uint64
CompressedSize uint64
CRC32 uint32
RowsCount uint64
Encoding string
DataOffset uint64
DataSize uint64
ValuesCount uint64
}
PageStats provides statistics about a page in a column.
type Reader ¶
type Reader struct {
// contains filtered or unexported fields
}
A Reader reads batches of rows from a stats Section. Returned arrow.RecordBatch values must be released by the caller after use.
func NewReader ¶
func NewReader(opts ReaderOptions) *Reader
NewReader creates a new Reader. Options are not validated until the first call to Reader.Open.
func (*Reader) Open ¶
Open initializes Reader resources. Must be called before Reader.Read. Safe to call multiple times.
func (*Reader) Read ¶
Read reads up to batchSize rows from the section. At end of section returns (nil, io.EOF). May return a non-nil batch with io.EOF — callers should process the batch before checking the error. Returned batches must be released by the caller after use.
func (*Reader) Reset ¶
func (r *Reader) Reset(opts ReaderOptions)
Reset reuses the Reader with new options.
func (*Reader) Schema ¶
Schema returns the arrow.Schema used by the Reader. Set on construction (via Reader.Reset) so it is valid before Reader.Open is called.
type ReaderOptions ¶
type ReaderOptions struct {
// Columns to read. Each column must belong to the same [Section].
Columns []*Column
// Allocator to use for allocating Arrow records. If nil,
// [memory.DefaultAllocator] is used.
Allocator memory.Allocator
}
ReaderOptions customizes the behavior of a Reader.
type RowReader ¶
type RowReader struct {
// contains filtered or unexported fields
}
RowReader reads Stat records from a stats Section one row at a time, in section order. It is a row-level cursor over the batch-level Reader. It implements iter.CloseIterator over Stat.
A RowReader is not safe for concurrent use.
func NewRowReader ¶
NewRowReader creates a RowReader over all of sec's columns. The underlying reader is opened lazily on the first call to Next. The provided ctx governs all subsequent I/O (Open and Read).
func (*RowReader) At ¶
At returns the current record. Undefined if Next has not been called or if the last Next call returned false.
func (*RowReader) Close ¶
Close releases the current batch and the underlying reader. Repeat calls return nil without re-closing.
type Section ¶
type Section struct {
// contains filtered or unexported fields
}
Section represents an opened stats section.
func Open ¶
Open opens a Section from an underlying dataobj.Section. Open returns an error if the section metadata could not be read or if the provided ctx is canceled.
type SectionEncoder ¶
SectionEncoder encodes a batch of sorted Stat rows into a columnar encoder.
func ColumnarSectionEncoder ¶
func ColumnarSectionEncoder(pageSizeHint int, pageMaxRowCount int) SectionEncoder
ColumnarSectionEncoder returns a SectionEncoder that encodes Stat rows using [dataset.ColumnBuilder]s backed by a columnar.Encoder.
pageSizeHint and pageMaxRowCount control the page splitting behaviour of the underlying column builders.
type Stat ¶
type Stat struct {
ObjectPath string
SectionIndex int64
ShardBucket uint32
SortSchema string
Labels map[string]string // Label values keyed by sort schema key name
MinTimestamp int64 // UnixNano
MaxTimestamp int64 // UnixNano
RowCount int64
UncompressedSize int64
}
Stat represents a single row in the stats section. A row contains information about a single sortable bucket: (Shard, Tenant Sort Schema)
func DecodeRow ¶
func DecodeRow(batch arrow.RecordBatch, columns ColumnIndex, rowIndex int) Stat
DecodeRow decodes a single row at rowIndex from an Arrow arrow.RecordBatch into a Stat. Dynamic label columns (fields with a ".label.utf8" suffix) are decoded into the Labels map.
Columns are looked up by field name via the provided ColumnIndex; missing or null values are left at their zero value.
type Stats ¶
type Stats struct {
UncompressedSize uint64
CompressedSize uint64
Columns []ColumnStats
}
Stats provides statistics about a stats section.