streams

package
v3.6.7 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 23, 2026 License: AGPL-3.0 Imports: 26 Imported by: 1

Documentation

Overview

Package streams defines types used for the data object streams section. The streams section holds a list of streams present in the data object.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func CheckSection 

func CheckSection(section *dataobj.Section) bool

CheckSection returns true if section is a streams section.

func Iter 

func Iter(ctx context.Context, obj *dataobj.Object) result.Seq[Stream]

Iter iterates over streams in the provided decoder. All streams sections are iterated over in order.

func IterSection 

func IterSection(ctx context.Context, section *Section) result.Seq[Stream]

Types

type AndPredicate 

type AndPredicate struct{ Left, Right Predicate }

An AndPredicate is a Predicate which asserts that a row may only be included if both the Left and Right Predicate are true.

type AndRowPredicate 

type AndRowPredicate struct{ Left, Right RowPredicate }

An AndRowPredicate is a RowPredicate which requires both its Left and Right predicate to be true.

type Builder 

type Builder struct {
	// contains filtered or unexported fields
}

Builder builds a streams section.

func NewBuilder 

func NewBuilder(metrics *Metrics, pageSize, pageRowCount int) *Builder

NewBuilder creates a new sterams section builder. The pageSize argument specifies how large pages should be.

func (*Builder) AppendValue 

func (b *Builder) AppendValue(val Stream)

AppendValue may only be used for copying streams from an existing section.

func (*Builder) EstimatedSize 

func (b *Builder) EstimatedSize() int

EstimatedSize returns the estimated size of the Streams section in bytes.

func (*Builder) Flush 

func (b *Builder) Flush(w dataobj.SectionWriter) (n int64, err error)

Flush flushes the streams section to the provided writer.

After successful encoding, b is reset to a fresh state and can be reused.

func (*Builder) Record 

func (b *Builder) Record(streamLabels labels.Labels, ts time.Time, recordSize int64) int64

Record a stream record within the section. The provided timestamp is used to track the minimum and maximum timestamp of a stream. The number of calls to Record is used to track the number of rows for a stream. The recordSize is used to track the uncompressed size of the stream.

The stream ID of the recorded stream is returned.

func (*Builder) Reset 

func (b *Builder) Reset()

Reset resets all state, allowing Streams to be reused.

func (*Builder) SetTenant 

func (b *Builder) SetTenant(tenant string)

SetTenant sets the tenant that owns the builder. A builder can be made multi-tenant by passing an empty string.

func (*Builder) StreamID 

func (b *Builder) StreamID(streamLabels labels.Labels) int64

StreamID returns the stream ID for the provided streamLabels. If the stream has not been recorded, StreamID returns 0.

func (*Builder) Tenant 

func (b *Builder) Tenant() string

Tenant returns the optional tenant that owns the builder.

func (*Builder) TimeRange 

func (b *Builder) TimeRange() (time.Time, time.Time)

TimeRange returns the minimum and maximum timestamp across all streams.

func (*Builder) Type 

func (b *Builder) Type() dataobj.SectionType

Type returns the dataobj.SectionType of the streams builder.

type Column 

type Column struct {
	Section *Section   // Section that contains the column.
	Name    string     // Optional name of the column.
	Type    ColumnType // Type of data in the column.
	// contains filtered or unexported fields
}

A Column represents one of the columns in the streams section. Valid columns can only be retrieved by calling Section.Columns.

Data in columns can be read by using a Reader.

type ColumnStats 

type ColumnStats struct {
	Name             string
	Type             string
	ValueType        string
	RowsCount        uint64
	Compression      string
	UncompressedSize uint64
	CompressedSize   uint64
	MetadataOffset   uint64
	MetadataSize     uint64
	ValuesCount      uint64
	Cardinality      uint64

	Pages []PageStats
}

ColumnStats provides statistics about a column in a section.

type ColumnType 

type ColumnType int

ColumnType represents the kind of information stored in a Column. 

const (
	ColumnTypeInvalid      ColumnType = iota // ColumnTypeInvalid is an invalid column.
	ColumnTypeStreamID                       // ColumnTypeStreamID is a column containing a set of stream IDs.
	ColumnTypeMinTimestamp                   // ColumnTypeMinTimestamp is a column containing minimum timestamps per stream.
	ColumnTypeMaxTimestamp                   // ColumnTypeMaxTimestamp is a column containing maximum timestamps per stream.

	// ColumnTypeLabel is a column containing a sequence of label values per
	// stream. There will be one ColumnTypeLabels per label name; the name of the
	// label is stored as the column name.
	ColumnTypeLabel

	ColumnTypeRows             // ColumnTypeRows is a column containing row counts per stream.
	ColumnTypeUncompressedSize // ColumnTypeUncompressedSize is a column containing uncompressed size per stream.
)

func ParseColumnType 

func ParseColumnType(text string) (ColumnType, error)

ParseColumnType parses a ColumnType from a string. The expected string format is the same as what's returned by ColumnType.String.

func (ColumnType) String 

func (ct ColumnType) String() string

String returns the human-readable name of ct.

type EqualPredicate 

type EqualPredicate struct {
	Column *Column       // Column to check.
	Value  scalar.Scalar // Value to check equality for.
}

An EqualPredicate is a Predicate which asserts that a row may only be included if the Value of the Column is equal to the Value.

type FalsePredicate 

type FalsePredicate struct{}

FalsePredicate is a Predicate which always returns false.

type FuncPredicate 

type FuncPredicate struct {
	Column *Column // Column to check.

	// Keep is invoked with the column and value pair to check. Keep is given
	// the Column instance to allow for reusing the same function across
	// multiple columns, if necessary.
	//
	// If Keep returns true, the row is kept.
	Keep func(column *Column, value scalar.Scalar) bool
}

FuncPredicate is a Predicate which asserts that a row may only be included if the Value of the Column passes the Keep function.

Instances of FuncPredicate are ineligible for page filtering and should only be used when there isn't a more explicit Predicate implementation.

type GreaterThanPredicate 

type GreaterThanPredicate struct {
	Column *Column       // Column to check.
	Value  scalar.Scalar // Value for which rows in Column must be greater than.
}

A GreaterThanPredicate is a Predicate which asserts that a row may only be included if the Value of the Column is greater than the provided Value.

type InPredicate 

type InPredicate struct {
	Column *Column         // Column to check.
	Values []scalar.Scalar // Values to check for inclusion.
}

An InPredicate is a Predicate which asserts that a row may only be included if the Value of the Column is present in the provided Values.

type LabelFilterRowPredicate 

type LabelFilterRowPredicate struct {
	Name string
	Keep func(name, value string) bool
}

A LabelFilterRowPredicate is a RowPredicate that requires that labels with the provided name pass a Keep function.

The name is is provided to the keep function to allow the same function to be used for multiple filter predicates.

Uses of LabelFilterRowPredicate are not eligible for page filtering and should only be used when a condition cannot be expressed by other basic predicates.

type LabelMatcherRowPredicate 

type LabelMatcherRowPredicate struct{ Name, Value string }

A LabelMatcherRowPredicate is a RowPredicate which requires a label named Name to exist with a value of Value.

type LessThanPredicate 

type LessThanPredicate struct {
	Column *Column       // Column to check.
	Value  scalar.Scalar // Value for which rows in Column must be less than.
}

A LessThanPredicate is a Predicate which asserts that a row may only be included if the Value of the Column is less than the provided Value.

type Metrics 

type Metrics struct {
	// contains filtered or unexported fields
}

Metrics instruments the streams section.

func NewMetrics 

func NewMetrics() *Metrics

NewMetrics creates a new set of metrics for the streams section.

func (*Metrics) Observe 

func (m *Metrics) Observe(ctx context.Context, section *Section) error

Observe observes section statistics for a given section.

func (*Metrics) Register 

func (m *Metrics) Register(reg prometheus.Registerer) error

Register registers metrics to report to reg.

func (*Metrics) Unregister 

func (m *Metrics) Unregister(reg prometheus.Registerer)

Unregister unregisters metrics from the provided Registerer.

type NotPredicate 

type NotPredicate struct{ Inner Predicate }

A NotePredicate is a Predicate which asserts that a row may only be included if the inner Predicate is false.

type NotRowPredicate 

type NotRowPredicate struct{ Inner RowPredicate }

A NotRowPredicate is a RowPredicate which requires its Inner predicate to be false.

type OrPredicate 

type OrPredicate struct{ Left, Right Predicate }

An OrPredicate is a Predicate which asserts that a row may only be included if either the Left or Right Predicate are true.

type OrRowPredicate 

type OrRowPredicate struct{ Left, Right RowPredicate }

An OrRowPredicate is a RowPredicate which requires either its Left or Right predicate to be true.

type PageStats 

type PageStats struct {
	UncompressedSize uint64
	CompressedSize   uint64
	CRC32            uint32
	RowsCount        uint64
	Encoding         string
	DataOffset       uint64
	DataSize         uint64
	ValuesCount      uint64
}

PageStats provides statistics about a page in a column.

type Predicate 

type Predicate interface {
	// contains filtered or unexported methods
}

Predicate is an expression used to filter column values in a Reader.

type Reader 

type Reader struct {
	// contains filtered or unexported fields
}

A Reader reads batches of rows from a Section.

func NewReader 

func NewReader(opts ReaderOptions) *Reader

NewReader creates a new Reader from the provided options. Options are not validated until the first call to Reader.Read.

func (*Reader) Close 

func (r *Reader) Close() error

Close closes the Reader and releases any resources it holds. Closed Readers can be reused by calling Reader.Reset.

func (*Reader) Read 

func (r *Reader) Read(ctx context.Context, batchSize int) (arrow.Record, error)

Read reads the batch of rows from the section, returning them as an Arrow record.

If ReaderOptions has predicates, only rows that match the predicates are returned. If none of the next batchSize rows matched the predicate, Read returns a nil record with a nil error.

Read will return an error if the next batch of rows could not be read due to invalid options or I/O errors. At the end of the section, Read returns nil, io.EOF.

Read may return a non-nil record with a non-nil error, including if the error is io.EOF. Callers should always process the record before processing the error value.

When a record is returned, it will match the schema specified by Reader.Schema. These records must always be released after use.

func (*Reader) Reset 

func (r *Reader) Reset(opts ReaderOptions)

Reset discards any state and resets r with a new set of optiosn. This permits reusing a Reader rather than allocating a new one.

func (*Reader) Schema 

func (r *Reader) Schema() *arrow.Schema

Schema returns the arrow.Schema used by the Reader. Fields in the schema match the order of columns listed in ReaderOptions.

Names of fields in the schema are guaranteed to be unique per column but are not guaranteed to be stable.

The returned Schema must not be modified.

type ReaderOptions 

type ReaderOptions struct {
	// Columns to read. Each column must belong to the same [Section].
	Columns []*Column

	// Predicates holds a set of predicates to apply when reading the section.
	// Columns referenced in Predicates must be in the set of Columns.
	Predicates []Predicate

	// Allocator to use for allocating Arrow records. If nil,
	// [memory.DefaultAllocator] is used.
	Allocator memory.Allocator
}

ReaderOptions customizes the behavior of a Reader.

func (*ReaderOptions) Validate 

func (opts *ReaderOptions) Validate() error

Validate returns an error if the opts is not valid. ReaderOptions are only valid when:

  • Each Column in Columns belongs to the same Section.
  • Each Predicate in Predicates references a Column from Columns.
  • Scalar values used in predicates are of a supported type: an int64, uint64, timestamp, or a byte array.

type RowPredicate 

type RowPredicate interface {
	// contains filtered or unexported methods
}

RowPredicate is an expression used to filter rows in a data object.

type RowReader 

type RowReader struct {
	// contains filtered or unexported fields
}

RowReader reads the set of streams from an [Object].

func NewRowReader 

func NewRowReader(sec *Section) *RowReader

NewRowReader creates a new RowReader that reads rows from the provided Section.

func (*RowReader) Close 

func (r *RowReader) Close() error

Close closes the RowReader and releases any resources it holds. Closed RowReaders can be reused by calling RowReader.Reset.

func (*RowReader) Read 

func (r *RowReader) Read(ctx context.Context, s []Stream) (int, error)

Read reads up to the next len(s) streams from the reader and stores them into s. It returns the number of streams read and any error encountered. At the end of the stream section, Read returns 0, io.EOF.

func (*RowReader) Reset 

func (r *RowReader) Reset(sec *Section)

Reset resets the RowReader with a new decoder to read from. Reset allows reusing a RowReader without allocating a new one.

Any set predicate is cleared when Reset is called.

Reset may be called with a nil object and a negative section index to clear the RowReader without needing a new object.

func (*RowReader) SetPredicate 

func (r *RowReader) SetPredicate(p RowPredicate) error

SetPredicate sets the predicate to use for filtering logs. [LogsReader.Read] will only return logs for which the predicate passes.

SetPredicate returns an error if the predicate is not supported by LogsReader.

A predicate may only be set before reading begins or after a call to RowReader.Reset.

type Section 

type Section struct {
	// contains filtered or unexported fields
}

Section represents an opened streams section.

func Open 

func Open(ctx context.Context, section *dataobj.Section) (*Section, error)

Open opens a Section from an underlying dataobj.Section. Open returns an error if the section metadata could not be read or if the provided ctx is canceled.

func (*Section) Columns 

func (s *Section) Columns() []*Column

Columns returns the set of Columns in the section. The slice of returned sections must not be mutated.

Unrecognized columns (e.g., when running older code against newer streams sections) are skipped.

type Stats 

type Stats struct {
	UncompressedSize uint64
	CompressedSize   uint64

	MinTimestamp          time.Time
	MaxTimestamp          time.Time
	TimestampDistribution []uint64 // Stream count per hour.

	Columns []ColumnStats
}

Stats provides statistics about a streams section.

func ReadStats 

func ReadStats(ctx context.Context, section *Section) (Stats, error)

ReadStats returns statistics about the streams section. ReadStats returns an error if the streams section couldn't be inspected or if the provided ctx is canceled.

type Stream 

type Stream struct {
	// ID to uniquely represent a stream in a data object. Valid IDs start at 1.
	// IDs are used to track streams across multiple sections in the same data
	// object.
	ID int64

	// MinTime and MaxTime denote the range of timestamps across all entries in
	// the stream.
	MinTimestamp, MaxTimestamp time.Time // Minimum timestamp in the stream.

	// Uncompressed size of the log lines and structured metadata values in the stream.
	UncompressedSize int64

	// Labels of the stream.
	Labels labels.Labels

	// Total number of log records in the stream.
	Rows int
}

A Stream is an individual stream within a data object.

func (*Stream) Reset 

func (s *Stream) Reset()

Reset zeroes all values in the stream struct so it can be reused.

type TimeRangeRowPredicate 

type TimeRangeRowPredicate struct {
	StartTime, EndTime time.Time
	IncludeStart       bool // Whether StartTime is inclusive.
	IncludeEnd         bool // Whether EndTime is inclusive.
}

A TimeRangeRowPredicate is a RowPredicate which requires the timestamp of the entry to be within the range of StartTime and EndTime.

type TruePredicate 

type TruePredicate struct{}

TruePredicate is a Predicate which always returns true.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.