streams

package
v4.0.0-rc.4 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 20, 2026 License: AGPL-3.0 Imports: 27 Imported by: 0

Documentation

Overview

Package streams defines types used for the data object streams section. The streams section holds a list of streams present in the data object.

Index

Constants

View Source
const DefaultShardBits = 5

DefaultShardBits is the shard bit count of DefaultShardLayout.

Do not change it. Readers assume that every object uses the default layout, so a new value makes them mis-read every existing object. The per-tenant shard factor limit rejects other factors for the same reason.

View Source
const DefaultShardBuckets = 1 << DefaultShardBits

DefaultShardBuckets is the bucket count of DefaultShardLayout.

Variables

This section is empty.

Functions

func CheckSection

func CheckSection(section *dataobj.Section) bool

CheckSection returns true if section is a streams section.

func CompareSortKey

func CompareSortKey(a, b SortKey) int

CompareSortKey compares stream keys by shard bucket, schema key, stable hash, and full labels.

func EncodeSchemaKey

func EncodeSchemaKey(values []string) string

EncodeSchemaKey joins sort-schema label values into one schema key. A NUL byte separates the values.

Stream IDs in written objects follow the byte order of this encoding. Code that must agree with stream ID order compares encoded keys, not the separate values: they order differently when a value contains a NUL byte.

func Iter

func Iter(ctx context.Context, obj *dataobj.Object, opts ...IterOption) result.Seq[Stream]

Iter iterates over streams in the provided decoder. All streams sections are iterated over in order.

func IterSection

func IterSection(ctx context.Context, section *Section, opts ...IterOption) result.Seq[Stream]

Types

type AndRowPredicate

type AndRowPredicate struct{ Left, Right RowPredicate }

An AndRowPredicate is a RowPredicate which requires both its Left and Right predicate to be true.

type Builder

type Builder struct {
	// contains filtered or unexported fields
}

Builder builds a streams section.

func NewBuilder

func NewBuilder(metrics *Metrics, pageSize, pageRowCount int, shardLayout ShardLayout) *Builder

NewBuilder creates a new streams section builder. pageSize is the target page size in bytes. shardLayout selects the shard bucket of each stream that the builder adds.

The builder records no layout until SetLayout is called, and again after Builder.Reset.

func (*Builder) AppendValue

func (b *Builder) AppendValue(val Stream)

AppendValue may only be used for copying streams from an existing section.

func (*Builder) EstimatedSize

func (b *Builder) EstimatedSize() int

EstimatedSize returns the estimated size of the Streams section in bytes.

func (*Builder) Flush

func (b *Builder) Flush(w dataobj.SectionWriter) (n int64, err error)

Flush flushes the streams section to the provided writer.

After successful encoding, b is reset to a fresh state and can be reused.

func (*Builder) Record

func (b *Builder) Record(streamLabels labels.Labels, ts time.Time, recordSize int64) int64

Record a stream record within the section. The provided timestamp is used to track the minimum and maximum timestamp of a stream. The number of calls to Record is used to track the number of rows for a stream. The recordSize is used to track the uncompressed size of the stream.

The stream ID of the recorded stream is returned.

func (*Builder) Reset

func (b *Builder) Reset()

Reset resets all state, allowing Streams to be reused.

func (*Builder) SetLayout

func (b *Builder) SetLayout(layout Layout)

SetLayout makes b record layout in the section metadata on Flush. The layout applies until the next Builder.Reset.

SetLayout panics if layout has another shard layout than b, because the shard buckets of the streams would not match the recorded layout. It also panics if layout is the zero value.

func (*Builder) SetTenant

func (b *Builder) SetTenant(tenant string)

SetTenant sets the tenant that owns the builder. A builder can be made multi-tenant by passing an empty string.

func (*Builder) StreamID

func (b *Builder) StreamID(streamLabels labels.Labels) int64

StreamID returns the stream ID for the provided streamLabels. If the stream has not been recorded, StreamID returns 0.

func (*Builder) Tenant

func (b *Builder) Tenant() string

Tenant returns the optional tenant that owns the builder.

func (*Builder) TimeRange

func (b *Builder) TimeRange() (time.Time, time.Time)

TimeRange returns the minimum and maximum timestamp across all streams.

func (*Builder) Type

func (b *Builder) Type() dataobj.SectionType

Type returns the dataobj.SectionType of the streams builder.

type Column

type Column struct {
	Section *Section   // Section that contains the column.
	Name    string     // Optional name of the column.
	Type    ColumnType // Type of data in the column.
	// contains filtered or unexported fields
}

A Column represents one of the columns in the streams section. Valid columns can only be retrieved by calling Section.Columns.

Data in columns can be read by using a RowReader.

type ColumnStats

type ColumnStats struct {
	Name             string
	Type             string
	ValueType        string
	RowsCount        uint64
	Compression      string
	UncompressedSize uint64
	CompressedSize   uint64
	MetadataOffset   uint64
	MetadataSize     uint64
	ValuesCount      uint64
	Cardinality      uint64

	Pages []PageStats
}

ColumnStats provides statistics about a column in a section.

type ColumnType

type ColumnType int

ColumnType represents the kind of information stored in a Column.

const (
	ColumnTypeInvalid      ColumnType = iota // ColumnTypeInvalid is an invalid column.
	ColumnTypeStreamID                       // ColumnTypeStreamID is a column containing a set of stream IDs.
	ColumnTypeShardBucket                    // ColumnTypeShardBucket contains the shard bucket derived from labels.StableHash(labels).
	ColumnTypeMinTimestamp                   // ColumnTypeMinTimestamp is a column containing minimum timestamps per stream.
	ColumnTypeMaxTimestamp                   // ColumnTypeMaxTimestamp is a column containing maximum timestamps per stream.

	// ColumnTypeLabel is a column containing a sequence of label values per
	// stream. There will be one ColumnTypeLabels per label name; the name of the
	// label is stored as the column name.
	ColumnTypeLabel

	ColumnTypeRows             // ColumnTypeRows is a column containing row counts per stream.
	ColumnTypeUncompressedSize // ColumnTypeUncompressedSize is a column containing uncompressed size per stream.
)

func ParseColumnType

func ParseColumnType(text string) (ColumnType, error)

ParseColumnType parses a ColumnType from a string. The expected string format is the same as what's returned by ColumnType.String.

func (ColumnType) String

func (ct ColumnType) String() string

String returns the human-readable name of ct.

type IterOption

type IterOption func(*iterConfig)

IterOption configures iteration behavior.

func WithReuseLabelsBuffer

func WithReuseLabelsBuffer() IterOption

WithReuseLabelsBuffer enables label buffer reuse to reduce memory allocations. When enabled, Stream.Labels is overwritten on each iteration. WARNING: Callers must extract needed data (hash, specific label values) before continuing iteration - do not retain the Labels reference.

func WithSymbolizer

func WithSymbolizer(sym *symbolizer.Symbolizer) IterOption

WithSymbolizer sets a symbolizer for deduplicating label value strings.

type LabelFilterRowPredicate

type LabelFilterRowPredicate struct {
	Name string
	Keep func(name, value string) bool
}

A LabelFilterRowPredicate is a RowPredicate that requires that labels with the provided name pass a Keep function.

The name is provided to the keep function to allow the same function to be used for multiple filter predicates.

Uses of LabelFilterRowPredicate are not eligible for page filtering and should only be used when a condition cannot be expressed by other basic predicates.

type LabelMatcherRowPredicate

type LabelMatcherRowPredicate struct{ Name, Value string }

A LabelMatcherRowPredicate is a RowPredicate which requires a label named Name to exist with a value of Value.

type Layout

type Layout struct {
	// contains filtered or unexported fields
}

Layout describes how the streams of one tenant in a data object are laid out: the shard layout of the shard bucket column, the schema labels of the stream key, and the stream key version that orders stream IDs.

The streams section records the layout and defines the stream IDs. The logs sections of the tenant use those IDs and sort rows by them, so all of them share one order. Stream IDs of two objects are comparable only if their layouts are equal.

The zero value means "no layout": it has one shard bucket, no schema labels, and an unspecified stream order. NewLayout never returns it.

func NewLayout

func NewLayout(shardLayout ShardLayout, schemaLabels []string) Layout

NewLayout returns the layout with shardLayout and schemaLabels. Stream IDs follow StreamOrderStableHashV1. An empty schemaLabels orders streams by shard bucket, then labels.StableHash.

NewLayout copies schemaLabels. It stores empty schemaLabels as nil, so layouts that Layout.Equal reports as equal are also deeply equal.

func (Layout) Equal

func (l Layout) Equal(other Layout) bool

Equal reports whether l and other describe the same layout.

func (Layout) SchemaLabels

func (l Layout) SchemaLabels() []string

SchemaLabels returns the fully-qualified schema labels of the stream key, in order. The caller must not modify the returned slice.

SchemaLabels does not copy, because writers call it once per appended stream.

func (Layout) ShardLayout

func (l Layout) ShardLayout() ShardLayout

ShardLayout returns the shard layout of the shard bucket column.

func (Layout) StreamOrder

func (l Layout) StreamOrder() StreamOrder

StreamOrder returns the stream key version that orders stream IDs.

type Metrics

type Metrics struct {
	// contains filtered or unexported fields
}

Metrics instruments the streams section.

func NewMetrics

func NewMetrics() *Metrics

NewMetrics creates a new set of metrics for the streams section.

func (*Metrics) Observe

func (m *Metrics) Observe(ctx context.Context, section *Section) error

Observe observes section statistics for a given section.

func (*Metrics) Register

func (m *Metrics) Register(reg prometheus.Registerer) error

Register registers metrics to report to reg.

func (*Metrics) Unregister

func (m *Metrics) Unregister(reg prometheus.Registerer)

Unregister unregisters metrics from the provided Registerer.

type NotRowPredicate

type NotRowPredicate struct{ Inner RowPredicate }

A NotRowPredicate is a RowPredicate which requires its Inner predicate to be false.

type OrRowPredicate

type OrRowPredicate struct{ Left, Right RowPredicate }

An OrRowPredicate is a RowPredicate which requires either its Left or Right predicate to be true.

type PageStats

type PageStats struct {
	UncompressedSize uint64
	CompressedSize   uint64
	CRC32            uint32
	RowsCount        uint64
	Encoding         string
	DataOffset       uint64
	DataSize         uint64
	ValuesCount      uint64
}

PageStats provides statistics about a page in a column.

type RowPredicate

type RowPredicate interface {
	// contains filtered or unexported methods
}

RowPredicate is an expression used to filter rows in a data object.

type RowReader

type RowReader struct {
	// contains filtered or unexported fields
}

RowReader reads the set of streams from an [Object].

func NewRowReader

func NewRowReader(sec *Section) *RowReader

NewRowReader creates a new RowReader that reads rows from the provided Section.

Call RowReader.Open before calling RowReader.Read.

func (*RowReader) Close

func (r *RowReader) Close() error

Close closes the RowReader and releases any resources it holds. Closed RowReaders can be reused by calling RowReader.Reset.

func (*RowReader) MatchStreams

func (r *RowReader) MatchStreams(ids iter.Seq[int64]) error

MatchStreams provides a sequence of stream IDs for the reader to match. RowReader.Read will only return the streams with the provided IDs.

MatchStreams may be called several times to match multiple sets of streams. An empty set, or never calling it, applies no stream-ID filter.

MatchStreams may only be called before reading begins or after a call to RowReader.Reset.

func (*RowReader) Open

func (r *RowReader) Open(ctx context.Context) error

Open initializes RowReader resources.

Open must be called before RowReader.Read. Open is safe to call multiple times. Open is a no-op when the reader has no section.

func (*RowReader) Read

func (r *RowReader) Read(ctx context.Context, s []Stream) (int, error)

Read reads up to the next len(s) streams from the reader and stores them into s. It returns the number of streams read and any error encountered. At the end of the stream section, Read returns 0, io.EOF.

func (*RowReader) Reset

func (r *RowReader) Reset(sec *Section)

Reset resets the RowReader with a new decoder to read from. Reset allows reusing a RowReader without allocating a new one.

Any set predicate is cleared when Reset is called.

Reset may be called with a nil object and a negative section index to clear the RowReader without needing a new object.

func (*RowReader) SetPredicate

func (r *RowReader) SetPredicate(p RowPredicate) error

SetPredicate sets the predicate to use for filtering logs. [LogsReader.Read] will only return logs for which the predicate passes.

SetPredicate returns an error if the predicate is not supported by LogsReader.

A predicate may only be set before reading begins or after a call to RowReader.Reset.

type Section

type Section struct {
	// contains filtered or unexported fields
}

Section represents an opened streams section.

func Open

func Open(ctx context.Context, section *dataobj.Section) (*Section, error)

Open opens a Section from an underlying dataobj.Section. Open returns an error if the section metadata could not be read or if the provided ctx is canceled.

func (*Section) Columns

func (s *Section) Columns() []*Column

Columns returns the set of Columns in the section. The slice of returned sections must not be mutated.

Unrecognized columns (e.g., when running older code against newer streams sections) are skipped.

func (*Section) HasColumn

func (s *Section) HasColumn(typ ColumnType) bool

HasColumn reports whether the section holds a column of the given type.

func (*Section) Layout

func (s *Section) Layout() (layout Layout, ok bool, err error)

Layout returns the layout that the section records. ok is false if the section records no layout.

Layout returns an error if the recorded layout is not one that NewLayout can produce: the shard count is not a valid shard layout, the stream order is not StreamOrderStableHashV1, or the section has column sorts.

func (*Section) NumRows

func (s *Section) NumRows() int

NumRows returns the number of rows in the section.

type ShardBucketRangeRowPredicate

type ShardBucketRangeRowPredicate struct{ From, To uint64 }

A ShardBucketRangeRowPredicate is a RowPredicate which requires a stream's shard bucket to be within the range [From, To], both ends inclusive.

A section that carries no shard-bucket column matches no stream, as for any other predicate whose column is absent.

type ShardLayout

type ShardLayout struct {
	// contains filtered or unexported fields
}

ShardLayout maps streams to shard buckets. A stream's bucket is the top bits of labels.StableHash of its labels.

The layout is part of the storage format. Writers persist bucket values, and readers trust them without a new hash check. A reader that uses a different layout than the writer returns wrong results.

The zero value has one bucket. Bits is never more than 16.

func DefaultShardLayout

func DefaultShardLayout() ShardLayout

DefaultShardLayout returns the layout that readers assume for every object when they map query shards to buckets. Writers use it unless a tenant sets another shard factor.

func ShardLayoutFromBuckets

func ShardLayoutFromBuckets(buckets uint32) (ShardLayout, error)

ShardLayoutFromBuckets returns the layout with the given bucket count. It returns an error if buckets is not a power of two from 1 to 65536.

func (ShardLayout) Bits

func (l ShardLayout) Bits() int

Bits returns the number of high hash bits that select a bucket.

func (ShardLayout) BucketFromHash

func (l ShardLayout) BucketFromHash(hash uint64) uint32

BucketFromHash returns the shard bucket for a stream hash, in [0, Buckets()).

func (ShardLayout) BucketFromLabels

func (l ShardLayout) BucketFromLabels(streamLabels labels.Labels) uint32

BucketFromLabels returns the shard bucket for streamLabels, in [0, Buckets()).

func (ShardLayout) Buckets

func (l ShardLayout) Buckets() uint32

Buckets returns the number of shard buckets, 2^Bits.

func (ShardLayout) NewSortKey

func (l ShardLayout) NewSortKey(streamLabels labels.Labels, schemaKey string) SortKey

NewSortKey returns the sort key of streamLabels. Keys are stable across objects that use the same layout and schemaKey.

type SortKey

type SortKey struct {
	ShardBucket uint32
	SchemaKey   string
	Hash        uint64
	Labels      labels.Labels
}

SortKey is the globally stable ordering key for one stream.

Streams are sorted by [shard_bucket, tenant sort-schema, stream hash] Full labels provide a deterministic tiebreaker when hashes collide so independently written objects assign compatible stream IDs.

func (SortKey) Compare

func (a SortKey) Compare(b SortKey) int

Compare reports the order of a and b by [shard, key, hash]. Labels are not compared directly. Use CompareSortKey for a full deduplication using labels.

func (SortKey) Prefix

func (a SortKey) Prefix() SortPrefix

Prefix returns the leading [shard, schema key] part of the sort key.

type SortPrefix

type SortPrefix struct {
	ShardBucket uint32
	SchemaKey   string
}

SortPrefix is the [shard bucket, schema key] prefix of a SortKey. Streams with the same prefix are contiguous in stream ID order, so a range of prefixes selects a contiguous range of stream IDs.

func NewSortPrefix

func NewSortPrefix(shardBucket uint32, schemaValues []string) SortPrefix

NewSortPrefix returns the prefix for a shard bucket and the stream's values of the sort-schema labels, in schema order.

func (SortPrefix) Compare

func (a SortPrefix) Compare(b SortPrefix) int

Compare reports the order of a and b by shard bucket, then schema key.

type Stats

type Stats struct {
	UncompressedSize uint64
	CompressedSize   uint64

	MinTimestamp          time.Time
	MaxTimestamp          time.Time
	TimestampDistribution []uint64 // Stream count per hour.

	Columns []ColumnStats
}

Stats provides statistics about a streams section.

func ReadStats

func ReadStats(ctx context.Context, section *Section) (Stats, error)

ReadStats returns statistics about the streams section. ReadStats returns an error if the streams section couldn't be inspected or if the provided ctx is canceled.

type Stream

type Stream struct {
	// ID to uniquely represent a stream in a data object. Valid IDs start at 1.
	// IDs are used to track streams across multiple sections in the same data
	// object.
	ID int64

	// MinTime and MaxTime denote the range of timestamps across all entries in
	// the stream.
	MinTimestamp, MaxTimestamp time.Time // Minimum timestamp in the stream.

	// Uncompressed size of the log lines and structured metadata values in the stream.
	UncompressedSize int64

	// Labels of the stream.
	Labels labels.Labels

	// Total number of log records in the stream.
	Rows int

	// ShardBucket is the bucket that the builder's [ShardLayout] assigned
	// to Labels, in [0, Buckets()).
	ShardBucket int64
}

A Stream is an individual stream within a data object.

func (*Stream) Reset

func (s *Stream) Reset()

Reset zeroes all values in the stream struct so it can be reused.

type StreamOrder

type StreamOrder int

StreamOrder identifies the stream key that orders the stream IDs of a section.

const (
	// StreamOrderUnspecified means that stream IDs follow no known key.
	StreamOrderUnspecified StreamOrder = 0

	// StreamOrderStableHashV1 orders stream IDs by shard bucket, then schema
	// key, then labels.StableHash, then labels.
	StreamOrderStableHashV1 StreamOrder = 2
)

type TimeRangeRowPredicate

type TimeRangeRowPredicate struct {
	StartTime, EndTime time.Time
	IncludeStart       bool // Whether StartTime is inclusive.
	IncludeEnd         bool // Whether EndTime is inclusive.
}

A TimeRangeRowPredicate is a RowPredicate which requires the timestamp of the entry to be within the range of StartTime and EndTime.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL