Documentation
¶
Overview ¶
Package streams defines types used for the data object streams section. The streams section holds a list of streams present in the data object.
Index ¶
- Constants
- func CheckSection(section *dataobj.Section) bool
- func CompareSortKey(a, b SortKey) int
- func EncodeSchemaKey(values []string) string
- func Iter(ctx context.Context, obj *dataobj.Object, opts ...IterOption) result.Seq[Stream]
- func IterSection(ctx context.Context, section *Section, opts ...IterOption) result.Seq[Stream]
- type AndRowPredicate
- type Builder
- func (b *Builder) AppendValue(val Stream)
- func (b *Builder) EstimatedSize() int
- func (b *Builder) Flush(w dataobj.SectionWriter) (n int64, err error)
- func (b *Builder) Record(streamLabels labels.Labels, ts time.Time, recordSize int64) int64
- func (b *Builder) Reset()
- func (b *Builder) SetLayout(layout Layout)
- func (b *Builder) SetTenant(tenant string)
- func (b *Builder) StreamID(streamLabels labels.Labels) int64
- func (b *Builder) Tenant() string
- func (b *Builder) TimeRange() (time.Time, time.Time)
- func (b *Builder) Type() dataobj.SectionType
- type Column
- type ColumnStats
- type ColumnType
- type IterOption
- type LabelFilterRowPredicate
- type LabelMatcherRowPredicate
- type Layout
- type Metrics
- type NotRowPredicate
- type OrRowPredicate
- type PageStats
- type RowPredicate
- type RowReader
- func (r *RowReader) Close() error
- func (r *RowReader) MatchStreams(ids iter.Seq[int64]) error
- func (r *RowReader) Open(ctx context.Context) error
- func (r *RowReader) Read(ctx context.Context, s []Stream) (int, error)
- func (r *RowReader) Reset(sec *Section)
- func (r *RowReader) SetPredicate(p RowPredicate) error
- type Section
- type ShardBucketRangeRowPredicate
- type ShardLayout
- type SortKey
- type SortPrefix
- type Stats
- type Stream
- type StreamOrder
- type TimeRangeRowPredicate
Constants ¶
const DefaultShardBits = 5
DefaultShardBits is the shard bit count of DefaultShardLayout.
Do not change it. Readers assume that every object uses the default layout, so a new value makes them mis-read every existing object. The per-tenant shard factor limit rejects other factors for the same reason.
const DefaultShardBuckets = 1 << DefaultShardBits
DefaultShardBuckets is the bucket count of DefaultShardLayout.
Variables ¶
This section is empty.
Functions ¶
func CheckSection ¶
CheckSection returns true if section is a streams section.
func CompareSortKey ¶
CompareSortKey compares stream keys by shard bucket, schema key, stable hash, and full labels.
func EncodeSchemaKey ¶
EncodeSchemaKey joins sort-schema label values into one schema key. A NUL byte separates the values.
Stream IDs in written objects follow the byte order of this encoding. Code that must agree with stream ID order compares encoded keys, not the separate values: they order differently when a value contains a NUL byte.
func Iter ¶
Iter iterates over streams in the provided decoder. All streams sections are iterated over in order.
func IterSection ¶
Types ¶
type AndRowPredicate ¶
type AndRowPredicate struct{ Left, Right RowPredicate }
An AndRowPredicate is a RowPredicate which requires both its Left and Right predicate to be true.
type Builder ¶
type Builder struct {
// contains filtered or unexported fields
}
Builder builds a streams section.
func NewBuilder ¶
func NewBuilder(metrics *Metrics, pageSize, pageRowCount int, shardLayout ShardLayout) *Builder
NewBuilder creates a new streams section builder. pageSize is the target page size in bytes. shardLayout selects the shard bucket of each stream that the builder adds.
The builder records no layout until SetLayout is called, and again after Builder.Reset.
func (*Builder) AppendValue ¶
AppendValue may only be used for copying streams from an existing section.
func (*Builder) EstimatedSize ¶
EstimatedSize returns the estimated size of the Streams section in bytes.
func (*Builder) Flush ¶
func (b *Builder) Flush(w dataobj.SectionWriter) (n int64, err error)
Flush flushes the streams section to the provided writer.
After successful encoding, b is reset to a fresh state and can be reused.
func (*Builder) Record ¶
Record a stream record within the section. The provided timestamp is used to track the minimum and maximum timestamp of a stream. The number of calls to Record is used to track the number of rows for a stream. The recordSize is used to track the uncompressed size of the stream.
The stream ID of the recorded stream is returned.
func (*Builder) Reset ¶
func (b *Builder) Reset()
Reset resets all state, allowing Streams to be reused.
func (*Builder) SetLayout ¶
SetLayout makes b record layout in the section metadata on Flush. The layout applies until the next Builder.Reset.
SetLayout panics if layout has another shard layout than b, because the shard buckets of the streams would not match the recorded layout. It also panics if layout is the zero value.
func (*Builder) SetTenant ¶
SetTenant sets the tenant that owns the builder. A builder can be made multi-tenant by passing an empty string.
func (*Builder) StreamID ¶
StreamID returns the stream ID for the provided streamLabels. If the stream has not been recorded, StreamID returns 0.
func (*Builder) Type ¶
func (b *Builder) Type() dataobj.SectionType
Type returns the dataobj.SectionType of the streams builder.
type Column ¶
type Column struct {
Section *Section // Section that contains the column.
Name string // Optional name of the column.
Type ColumnType // Type of data in the column.
// contains filtered or unexported fields
}
A Column represents one of the columns in the streams section. Valid columns can only be retrieved by calling Section.Columns.
Data in columns can be read by using a RowReader.
type ColumnStats ¶
type ColumnStats struct {
Name string
Type string
ValueType string
RowsCount uint64
Compression string
UncompressedSize uint64
CompressedSize uint64
MetadataOffset uint64
MetadataSize uint64
ValuesCount uint64
Cardinality uint64
Pages []PageStats
}
ColumnStats provides statistics about a column in a section.
type ColumnType ¶
type ColumnType int
ColumnType represents the kind of information stored in a Column.
const ( ColumnTypeInvalid ColumnType = iota // ColumnTypeInvalid is an invalid column. ColumnTypeStreamID // ColumnTypeStreamID is a column containing a set of stream IDs. ColumnTypeShardBucket // ColumnTypeShardBucket contains the shard bucket derived from labels.StableHash(labels). ColumnTypeMinTimestamp // ColumnTypeMinTimestamp is a column containing minimum timestamps per stream. ColumnTypeMaxTimestamp // ColumnTypeMaxTimestamp is a column containing maximum timestamps per stream. // ColumnTypeLabel is a column containing a sequence of label values per // stream. There will be one ColumnTypeLabels per label name; the name of the // label is stored as the column name. ColumnTypeLabel ColumnTypeRows // ColumnTypeRows is a column containing row counts per stream. ColumnTypeUncompressedSize // ColumnTypeUncompressedSize is a column containing uncompressed size per stream. )
func ParseColumnType ¶
func ParseColumnType(text string) (ColumnType, error)
ParseColumnType parses a ColumnType from a string. The expected string format is the same as what's returned by ColumnType.String.
func (ColumnType) String ¶
func (ct ColumnType) String() string
String returns the human-readable name of ct.
type IterOption ¶
type IterOption func(*iterConfig)
IterOption configures iteration behavior.
func WithReuseLabelsBuffer ¶
func WithReuseLabelsBuffer() IterOption
WithReuseLabelsBuffer enables label buffer reuse to reduce memory allocations. When enabled, Stream.Labels is overwritten on each iteration. WARNING: Callers must extract needed data (hash, specific label values) before continuing iteration - do not retain the Labels reference.
func WithSymbolizer ¶
func WithSymbolizer(sym *symbolizer.Symbolizer) IterOption
WithSymbolizer sets a symbolizer for deduplicating label value strings.
type LabelFilterRowPredicate ¶
A LabelFilterRowPredicate is a RowPredicate that requires that labels with the provided name pass a Keep function.
The name is provided to the keep function to allow the same function to be used for multiple filter predicates.
Uses of LabelFilterRowPredicate are not eligible for page filtering and should only be used when a condition cannot be expressed by other basic predicates.
type LabelMatcherRowPredicate ¶
type LabelMatcherRowPredicate struct{ Name, Value string }
A LabelMatcherRowPredicate is a RowPredicate which requires a label named Name to exist with a value of Value.
type Layout ¶
type Layout struct {
// contains filtered or unexported fields
}
Layout describes how the streams of one tenant in a data object are laid out: the shard layout of the shard bucket column, the schema labels of the stream key, and the stream key version that orders stream IDs.
The streams section records the layout and defines the stream IDs. The logs sections of the tenant use those IDs and sort rows by them, so all of them share one order. Stream IDs of two objects are comparable only if their layouts are equal.
The zero value means "no layout": it has one shard bucket, no schema labels, and an unspecified stream order. NewLayout never returns it.
func NewLayout ¶
func NewLayout(shardLayout ShardLayout, schemaLabels []string) Layout
NewLayout returns the layout with shardLayout and schemaLabels. Stream IDs follow StreamOrderStableHashV1. An empty schemaLabels orders streams by shard bucket, then labels.StableHash.
NewLayout copies schemaLabels. It stores empty schemaLabels as nil, so layouts that Layout.Equal reports as equal are also deeply equal.
func (Layout) SchemaLabels ¶
SchemaLabels returns the fully-qualified schema labels of the stream key, in order. The caller must not modify the returned slice.
SchemaLabels does not copy, because writers call it once per appended stream.
func (Layout) ShardLayout ¶
func (l Layout) ShardLayout() ShardLayout
ShardLayout returns the shard layout of the shard bucket column.
func (Layout) StreamOrder ¶
func (l Layout) StreamOrder() StreamOrder
StreamOrder returns the stream key version that orders stream IDs.
type Metrics ¶
type Metrics struct {
// contains filtered or unexported fields
}
Metrics instruments the streams section.
func NewMetrics ¶
func NewMetrics() *Metrics
NewMetrics creates a new set of metrics for the streams section.
func (*Metrics) Register ¶
func (m *Metrics) Register(reg prometheus.Registerer) error
Register registers metrics to report to reg.
func (*Metrics) Unregister ¶
func (m *Metrics) Unregister(reg prometheus.Registerer)
Unregister unregisters metrics from the provided Registerer.
type NotRowPredicate ¶
type NotRowPredicate struct{ Inner RowPredicate }
A NotRowPredicate is a RowPredicate which requires its Inner predicate to be false.
type OrRowPredicate ¶
type OrRowPredicate struct{ Left, Right RowPredicate }
An OrRowPredicate is a RowPredicate which requires either its Left or Right predicate to be true.
type PageStats ¶
type PageStats struct {
UncompressedSize uint64
CompressedSize uint64
CRC32 uint32
RowsCount uint64
Encoding string
DataOffset uint64
DataSize uint64
ValuesCount uint64
}
PageStats provides statistics about a page in a column.
type RowPredicate ¶
type RowPredicate interface {
// contains filtered or unexported methods
}
RowPredicate is an expression used to filter rows in a data object.
type RowReader ¶
type RowReader struct {
// contains filtered or unexported fields
}
RowReader reads the set of streams from an [Object].
func NewRowReader ¶
NewRowReader creates a new RowReader that reads rows from the provided Section.
Call RowReader.Open before calling RowReader.Read.
func (*RowReader) Close ¶
Close closes the RowReader and releases any resources it holds. Closed RowReaders can be reused by calling RowReader.Reset.
func (*RowReader) MatchStreams ¶
MatchStreams provides a sequence of stream IDs for the reader to match. RowReader.Read will only return the streams with the provided IDs.
MatchStreams may be called several times to match multiple sets of streams. An empty set, or never calling it, applies no stream-ID filter.
MatchStreams may only be called before reading begins or after a call to RowReader.Reset.
func (*RowReader) Open ¶
Open initializes RowReader resources.
Open must be called before RowReader.Read. Open is safe to call multiple times. Open is a no-op when the reader has no section.
func (*RowReader) Read ¶
Read reads up to the next len(s) streams from the reader and stores them into s. It returns the number of streams read and any error encountered. At the end of the stream section, Read returns 0, io.EOF.
func (*RowReader) Reset ¶
Reset resets the RowReader with a new decoder to read from. Reset allows reusing a RowReader without allocating a new one.
Any set predicate is cleared when Reset is called.
Reset may be called with a nil object and a negative section index to clear the RowReader without needing a new object.
func (*RowReader) SetPredicate ¶
func (r *RowReader) SetPredicate(p RowPredicate) error
SetPredicate sets the predicate to use for filtering logs. [LogsReader.Read] will only return logs for which the predicate passes.
SetPredicate returns an error if the predicate is not supported by LogsReader.
A predicate may only be set before reading begins or after a call to RowReader.Reset.
type Section ¶
type Section struct {
// contains filtered or unexported fields
}
Section represents an opened streams section.
func Open ¶
Open opens a Section from an underlying dataobj.Section. Open returns an error if the section metadata could not be read or if the provided ctx is canceled.
func (*Section) Columns ¶
Columns returns the set of Columns in the section. The slice of returned sections must not be mutated.
Unrecognized columns (e.g., when running older code against newer streams sections) are skipped.
func (*Section) HasColumn ¶
func (s *Section) HasColumn(typ ColumnType) bool
HasColumn reports whether the section holds a column of the given type.
func (*Section) Layout ¶
Layout returns the layout that the section records. ok is false if the section records no layout.
Layout returns an error if the recorded layout is not one that NewLayout can produce: the shard count is not a valid shard layout, the stream order is not StreamOrderStableHashV1, or the section has column sorts.
type ShardBucketRangeRowPredicate ¶
type ShardBucketRangeRowPredicate struct{ From, To uint64 }
A ShardBucketRangeRowPredicate is a RowPredicate which requires a stream's shard bucket to be within the range [From, To], both ends inclusive.
A section that carries no shard-bucket column matches no stream, as for any other predicate whose column is absent.
type ShardLayout ¶
type ShardLayout struct {
// contains filtered or unexported fields
}
ShardLayout maps streams to shard buckets. A stream's bucket is the top bits of labels.StableHash of its labels.
The layout is part of the storage format. Writers persist bucket values, and readers trust them without a new hash check. A reader that uses a different layout than the writer returns wrong results.
The zero value has one bucket. Bits is never more than 16.
func DefaultShardLayout ¶
func DefaultShardLayout() ShardLayout
DefaultShardLayout returns the layout that readers assume for every object when they map query shards to buckets. Writers use it unless a tenant sets another shard factor.
func ShardLayoutFromBuckets ¶
func ShardLayoutFromBuckets(buckets uint32) (ShardLayout, error)
ShardLayoutFromBuckets returns the layout with the given bucket count. It returns an error if buckets is not a power of two from 1 to 65536.
func (ShardLayout) Bits ¶
func (l ShardLayout) Bits() int
Bits returns the number of high hash bits that select a bucket.
func (ShardLayout) BucketFromHash ¶
func (l ShardLayout) BucketFromHash(hash uint64) uint32
BucketFromHash returns the shard bucket for a stream hash, in [0, Buckets()).
func (ShardLayout) BucketFromLabels ¶
func (l ShardLayout) BucketFromLabels(streamLabels labels.Labels) uint32
BucketFromLabels returns the shard bucket for streamLabels, in [0, Buckets()).
func (ShardLayout) Buckets ¶
func (l ShardLayout) Buckets() uint32
Buckets returns the number of shard buckets, 2^Bits.
func (ShardLayout) NewSortKey ¶
func (l ShardLayout) NewSortKey(streamLabels labels.Labels, schemaKey string) SortKey
NewSortKey returns the sort key of streamLabels. Keys are stable across objects that use the same layout and schemaKey.
type SortKey ¶
SortKey is the globally stable ordering key for one stream.
Streams are sorted by [shard_bucket, tenant sort-schema, stream hash] Full labels provide a deterministic tiebreaker when hashes collide so independently written objects assign compatible stream IDs.
func (SortKey) Compare ¶
Compare reports the order of a and b by [shard, key, hash]. Labels are not compared directly. Use CompareSortKey for a full deduplication using labels.
func (SortKey) Prefix ¶
func (a SortKey) Prefix() SortPrefix
Prefix returns the leading [shard, schema key] part of the sort key.
type SortPrefix ¶
SortPrefix is the [shard bucket, schema key] prefix of a SortKey. Streams with the same prefix are contiguous in stream ID order, so a range of prefixes selects a contiguous range of stream IDs.
func NewSortPrefix ¶
func NewSortPrefix(shardBucket uint32, schemaValues []string) SortPrefix
NewSortPrefix returns the prefix for a shard bucket and the stream's values of the sort-schema labels, in schema order.
func (SortPrefix) Compare ¶
func (a SortPrefix) Compare(b SortPrefix) int
Compare reports the order of a and b by shard bucket, then schema key.
type Stats ¶
type Stats struct {
UncompressedSize uint64
CompressedSize uint64
MinTimestamp time.Time
MaxTimestamp time.Time
TimestampDistribution []uint64 // Stream count per hour.
Columns []ColumnStats
}
Stats provides statistics about a streams section.
type Stream ¶
type Stream struct {
// ID to uniquely represent a stream in a data object. Valid IDs start at 1.
// IDs are used to track streams across multiple sections in the same data
// object.
ID int64
// MinTime and MaxTime denote the range of timestamps across all entries in
// the stream.
MinTimestamp, MaxTimestamp time.Time // Minimum timestamp in the stream.
// Uncompressed size of the log lines and structured metadata values in the stream.
UncompressedSize int64
// Labels of the stream.
Labels labels.Labels
// Total number of log records in the stream.
Rows int
// ShardBucket is the bucket that the builder's [ShardLayout] assigned
// to Labels, in [0, Buckets()).
ShardBucket int64
}
A Stream is an individual stream within a data object.
type StreamOrder ¶
type StreamOrder int
StreamOrder identifies the stream key that orders the stream IDs of a section.
const ( // StreamOrderUnspecified means that stream IDs follow no known key. StreamOrderUnspecified StreamOrder = 0 // StreamOrderStableHashV1 orders stream IDs by shard bucket, then schema // key, then labels.StableHash, then labels. StreamOrderStableHashV1 StreamOrder = 2 )
type TimeRangeRowPredicate ¶
type TimeRangeRowPredicate struct {
StartTime, EndTime time.Time
IncludeStart bool // Whether StartTime is inclusive.
IncludeEnd bool // Whether EndTime is inclusive.
}
A TimeRangeRowPredicate is a RowPredicate which requires the timestamp of the entry to be within the range of StartTime and EndTime.