pointers

package
v3.7.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 26, 2026 License: AGPL-3.0 Imports: 32 Imported by: 1

Documentation

Overview

Package pointers defines types used for the data object pointers section. The pointers section holds a list of pointers to sections present in the data object.

Index

Constants

View Source 
const InternalLabelsFieldName = "__streamLabelNames__"

Variables

This section is empty.

Functions

func CheckSection 

func CheckSection(section *dataobj.Section) bool

CheckSection returns true if section is a streams section.

func FromRecordBatch added in v3.7.0 

func FromRecordBatch(
	rec arrow.RecordBatch,
	dest []SectionPointer,
	populate func(arrow.Field, ColumnType) bool,
) (int, error)

func InternalLabelsColumn added in v3.7.0 

func InternalLabelsColumn(rec arrow.RecordBatch) *array.String

func Iter 

func Iter(ctx context.Context, obj *dataobj.Object) result.Seq[SectionPointer]

Iter iterates over pointers in the provided decoder. All pointers sections are iterated over in order.

func IterSection 

func IterSection(ctx context.Context, section *Section) result.Seq[SectionPointer]

func PopulateSection added in v3.7.0 

func PopulateSection(_ arrow.Field, columnType ColumnType) bool

func PopulateSectionKey added in v3.7.0 

func PopulateSectionKey(_ arrow.Field, columnType ColumnType) bool

Types

type AndPredicate added in v3.7.0 

type AndPredicate struct{ Left, Right Predicate }

An AndPredicate is a Predicate which asserts that a row may only be included if both the Left and Right Predicate are true.

type AndRowPredicate 

type AndRowPredicate struct{ Left, Right RowPredicate }

An AndRowPredicate is a RowPredicate which requires both its Left and Right predicate to be true.

type BloomExistenceRowPredicate 

type BloomExistenceRowPredicate struct{ Name, Value string }

A BloomExistenceRowPredicate is a RowPredicate which requires a bloom filter column named Name to exist, and for the Value to pass the bloom filter.

type Builder 

type Builder struct {
	// contains filtered or unexported fields
}

Builder builds a pointers section.

func NewBuilder 

func NewBuilder(metrics *Metrics, pageSize, pageRows int) *Builder

NewBuilder creates a new pointers section builder. The pageSize argument specifies how large pages should be.

func (*Builder) EstimatedSize 

func (b *Builder) EstimatedSize() int

EstimatedSize returns the estimated size of the Pointers section in bytes.

func (*Builder) Flush 

func (b *Builder) Flush(w dataobj.SectionWriter) (n int64, err error)

Flush flushes the streams section to the provided writer.

After successful encoding, b is reset to a fresh state and can be reused.

func (*Builder) ObserveStream 

func (b *Builder) ObserveStream(path string, section int64, idInObject int64, idInIndex int64, ts time.Time, uncompressedSize int64)

ObserveStream observes a stream in the index by recording the start & end timestamps, line count, and uncompressed size per-section.

func (*Builder) RecordColumnIndex 

func (b *Builder) RecordColumnIndex(path string, section int64, columnName string, columnIndex int64, valuesBloomFilter []byte)

func (*Builder) Reset 

func (b *Builder) Reset()

Reset resets all state, allowing Pointers builder to be reused.

func (*Builder) SetTenant 

func (b *Builder) SetTenant(tenant string)

func (*Builder) Tenant 

func (b *Builder) Tenant() string

func (*Builder) Type 

func (b *Builder) Type() dataobj.SectionType

Type returns the dataobj.SectionType of the pointers builder.

type Column 

type Column struct {
	Section *Section   // Section that contains the column.
	Name    string     // Optional name of the column.
	Type    ColumnType // Type of data in the column.
	// contains filtered or unexported fields
}

A Column represents one of the columns in the streams section. Valid columns can only be retrieved by calling Section.Columns.

Data in columns can be read by using a Reader.

type ColumnStats 

type ColumnStats struct {
	Name             string
	Type             string
	ValueType        string
	RowsCount        uint64
	Compression      string
	UncompressedSize uint64
	CompressedSize   uint64
	MetadataOffset   uint64
	MetadataSize     uint64
	ValuesCount      uint64
	Cardinality      uint64

	Pages []PageStats
}

ColumnStats provides statistics about a column in a section.

type ColumnType 

type ColumnType int

ColumnType represents the kind of information stored in a Column. 

const (
	ColumnTypeInvalid ColumnType = iota // ColumnTypeInvalid is an invalid column.
	ColumnTypePath
	ColumnTypeSection
	ColumnTypePointerKind // ColumnTypePointerKind is a column containing the kind of pointer: stream or column.

	ColumnTypeStreamID         // ColumnTypeStreamID is a column containing a set of stream IDs.
	ColumnTypeStreamIDRef      // ColumnTypeStreamIDRef is a column containing a set of stream IDs from the referenced object.
	ColumnTypeMinTimestamp     // ColumnTypeMinTimestamp is a column containing minimum timestamps per stream.
	ColumnTypeMaxTimestamp     // ColumnTypeMaxTimestamp is a column containing maximum timestamps per stream.
	ColumnTypeRowCount         // ColumnTypeRowCount is a column containing row count per stream.
	ColumnTypeUncompressedSize // ColumnTypeUncompressedSize is a column containing uncompressed size per stream.

	ColumnTypeColumnName        // ColumnTypeColumnName is a column containing the name of the column in the referenced object.
	ColumnTypeColumnIndex       // ColumnTypeColumnIndex is a column containing the index of the column in the referenced object.
	ColumnTypeValuesBloomFilter // ColumnTypeValuesBloomFilter is a column containing a bloom filter of the values in the column in the referenced object.
)

func ColumnTypeFromField added in v3.7.0 

func ColumnTypeFromField(field arrow.Field) ColumnType

func ParseColumnType 

func ParseColumnType(text string) (ColumnType, error)

ParseColumnType parses a ColumnType from a string. The expected string format is the same as what's returned by ColumnType.String.

func (ColumnType) String 

func (ct ColumnType) String() string

String returns the human-readable name of ct.

type EqualPredicate added in v3.7.0 

type EqualPredicate struct {
	Column *Column       // Column to check.
	Value  scalar.Scalar // Value to check equality for.
}

An EqualPredicate is a Predicate which asserts that a row may only be included if the Value of the Column is equal to the Value.

type FalsePredicate added in v3.7.0 

type FalsePredicate struct{}

FalsePredicate is a Predicate which always returns false.

type FuncPredicate added in v3.7.0 

type FuncPredicate struct {
	Column *Column // Column to check.

	// Keep is invoked with the column and value pair to check. Keep is given
	// the Column instance to allow for reusing the same function across
	// multiple columns, if necessary.
	//
	// If Keep returns true, the row is kept.
	Keep func(column *Column, value scalar.Scalar) bool
}

FuncPredicate is a Predicate which asserts that a row may only be included if the Value of the Column passes the Keep function.

Instances of FuncPredicate are ineligible for page filtering and should only be used when there isn't a more explicit Predicate implementation.

type GreaterThanPredicate added in v3.7.0 

type GreaterThanPredicate struct {
	Column *Column       // Column to check.
	Value  scalar.Scalar // Value for which rows in Column must be greater than.
}

A GreaterThanPredicate is a Predicate which asserts that a row may only be included if the Value of the Column is greater than the provided Value.

type InPredicate added in v3.7.0 

type InPredicate struct {
	Column *Column         // Column to check.
	Values []scalar.Scalar // Values to check for inclusion.
}

An InPredicate is a Predicate which asserts that a row may only be included if the Value of the Column is present in the provided Values.

type LessThanPredicate added in v3.7.0 

type LessThanPredicate struct {
	Column *Column       // Column to check.
	Value  scalar.Scalar // Value for which rows in Column must be less than.
}

A LessThanPredicate is a Predicate which asserts that a row may only be included if the Value of the Column is less than the provided Value.

type Metrics 

type Metrics struct {
	// contains filtered or unexported fields
}

Metrics instruments the streams section.

func NewMetrics 

func NewMetrics() *Metrics

NewMetrics creates a new set of metrics for the pointers section.

func (*Metrics) Observe 

func (m *Metrics) Observe(ctx context.Context, section *Section) error

Observe observes section statistics for a given section.

func (*Metrics) Register 

func (m *Metrics) Register(reg prometheus.Registerer) error

Register registers metrics to report to reg.

func (*Metrics) Unregister 

func (m *Metrics) Unregister(reg prometheus.Registerer)

Unregister unregisters metrics from the provided Registerer.

type NotPredicate added in v3.7.0 

type NotPredicate struct{ Inner Predicate }

A NotePredicate is a Predicate which asserts that a row may only be included if the inner Predicate is false.

type OrPredicate added in v3.7.0 

type OrPredicate struct{ Left, Right Predicate }

An OrPredicate is a Predicate which asserts that a row may only be included if either the Left or Right Predicate are true.

type PageStats 

type PageStats struct {
	UncompressedSize uint64
	CompressedSize   uint64
	CRC32            uint32
	RowsCount        uint64
	Encoding         string
	DataOffset       uint64
	DataSize         uint64
	ValuesCount      uint64
}

PageStats provides statistics about a page in a column.

type PointerKind 

type PointerKind int
const (
	PointerKindInvalid     PointerKind = iota // PointerKindInvalid is an invalid pointer kind.
	PointerKindStreamIndex                    // PointerKindStreamIndex is a pointer for a stream index.
	PointerKindColumnIndex                    // PointerKindColumnIndex is a pointer for a column index.
)

type PopulateColumnFilter added in v3.7.0 

type PopulateColumnFilter func(arrow.Field, ColumnType) bool

type Predicate added in v3.7.0 

type Predicate interface {
	// contains filtered or unexported methods
}

Predicate is an expression used to filter column values in a Reader.

func WhereBloomFilterMatches added in v3.7.0 

func WhereBloomFilterMatches(
	colColumnName *Column,
	colBloom *Column,
	columnName scalar.Scalar,
	columnValue string,
) Predicate

func WhereTimeRangeOverlapsWith added in v3.7.0 

func WhereTimeRangeOverlapsWith(
	colMinTimestamp *Column,
	colMaxTimestamp *Column,
	start scalar.Scalar,
	end scalar.Scalar,
) Predicate

type Reader added in v3.7.0 

type Reader struct {
	// contains filtered or unexported fields
}

A Reader reads batches of rows from a Section.

func NewReader added in v3.7.0 

func NewReader(opts ReaderOptions) *Reader

NewReader creates a new Reader from the provided options. Options are not validated until the first call to Reader.Open.

func (*Reader) Close added in v3.7.0 

func (r *Reader) Close() error

Close closes the Reader and releases any resources it holds. Closed Readers can be reused by calling Reader.Reset.

func (*Reader) Columns added in v3.7.0 

func (r *Reader) Columns() []*Column

Columns returns the current [Column]s used by the Reader.

func (*Reader) Open added in v3.7.0 

func (r *Reader) Open(ctx context.Context) error

Open initializes Reader resources.

Open must be called before Reader.Read. Open is safe to call multiple times.

func (*Reader) Read added in v3.7.0 

func (r *Reader) Read(ctx context.Context, batchSize int) (arrow.RecordBatch, error)

Read reads the batch of rows from the section, returning them as an Arrow record.

If ReaderOptions has predicates, only rows that match the predicates are returned. If none of the next batchSize rows matched the predicate, Read returns a nil record with a nil error.

Read will return an error if the next batch of rows could not be read due to invalid options or I/O errors. At the end of the section, Read returns nil, io.EOF.

Read may return a non-nil record with a non-nil error, including if the error is io.EOF. Callers should always process the record before processing the error value.

When a record is returned, it will match the schema specified by Reader.Schema. These records must always be released after use.

func (*Reader) Reset added in v3.7.0 

func (r *Reader) Reset(opts ReaderOptions)

Reset discards any state and resets r with a new set of optiosn. This permits reusing a Reader rather than allocating a new one.

func (*Reader) Schema added in v3.7.0 

func (r *Reader) Schema() *arrow.Schema

Schema returns the arrow.Schema used by the Reader. Fields in the schema match the order of columns listed in ReaderOptions.

Names of fields in the schema are guaranteed to be unique per column but are not guaranteed to be stable.

The returned Schema must not be modified.

type ReaderOptions added in v3.7.0 

type ReaderOptions struct {
	// Columns to read. Each column must belong to the same [Section].
	Columns []*Column

	// Predicates holds a set of predicates to apply when reading the section.
	// Columns referenced in Predicates must be in the set of Columns.
	Predicates []Predicate

	// Allocator to use for allocating Arrow records. If nil,
	// [memory.DefaultAllocator] is used.
	Allocator memory.Allocator

	// An existing Stream ID to label names for the reader to decorate responses with.
	StreamIDToLabelNames map[int64][]string
}

ReaderOptions customizes the behavior of a Reader.

func (*ReaderOptions) Validate added in v3.7.0 

func (opts *ReaderOptions) Validate() error

Validate returns an error if the opts is not valid. ReaderOptions are only valid when:

  • Each Column in Columns belongs to the same Section.
  • Each Predicate in Predicates references a Column from Columns.
  • Scalar values used in predicates are of a supported type: an int64, uint64, timestamp, or a byte array.

type RowPredicate 

type RowPredicate interface {
	// contains filtered or unexported methods
}

RowPredicate is an expression used to filter rows in a data object.

type RowReader 

type RowReader struct {
	// contains filtered or unexported fields
}

RowReader reads the set of streams from an [Object].

func NewRowReader 

func NewRowReader(sec *Section) *RowReader

NewRowReader creates a new RowReader that reads rows from the provided Section.

Call RowReader.Open before calling RowReader.Read.

func (*RowReader) Close 

func (r *RowReader) Close() error

Close closes the RowReader and releases any resources it holds. Closed RowReaders can be reused by calling RowReader.Reset.

func (*RowReader) MatchStreams 

func (r *RowReader) MatchStreams(ids iter.Seq[int64]) error

MatchStreams provides a sequence of stream IDs for the logs reader to match. RowReader.Read will only return logs for the provided stream IDs.

MatchStreams may be called multiple times to match multiple sets of streams.

MatchStreams may only be called before reading begins or after a call to RowReader.Reset.

func (*RowReader) Open added in v3.7.0 

func (r *RowReader) Open(ctx context.Context) error

Open initializes RowReader resources.

Open must be called before RowReader.Read. Open is safe to call multiple times. Open is a no-op when the reader has no section.

func (*RowReader) Read 

func (r *RowReader) Read(ctx context.Context, s []SectionPointer) (int, error)

Read reads up to the next len(s) streams from the reader and stores them into s. It returns the number of streams read and any error encountered. At the end of the stream section, Read returns 0, io.EOF.

func (*RowReader) Reset 

func (r *RowReader) Reset(sec *Section)

Reset resets the RowReader with a new decoder to read from. Reset allows reusing a RowReader without allocating a new one.

Any set predicate is cleared when Reset is called.

Reset may be called with a nil object and a negative section index to clear the RowReader without needing a new object.

func (*RowReader) SetPredicate 

func (r *RowReader) SetPredicate(p RowPredicate) error

SetPredicate sets the predicate to use for filtering logs. [LogsReader.Read] will only return logs for which the predicate passes.

SetPredicate returns an error if the predicate is not supported by LogsReader.

A predicate may only be set before reading begins or after a call to RowReader.Reset.

type Section 

type Section struct {
	// contains filtered or unexported fields
}

Section represents an opened streams section.

func Open 

func Open(ctx context.Context, section *dataobj.Section) (*Section, error)

Open opens a Section from an underlying dataobj.Section. Open returns an error if the section metadata could not be read or if the provided ctx is canceled.

func (*Section) Columns 

func (s *Section) Columns() []*Column

Columns returns the set of Columns in the section. The slice of returned sections must not be mutated.

Unrecognized columns (e.g., when running older code against newer streams sections) are skipped.

type SectionPointer 

type SectionPointer struct {
	Path        string
	Section     int64
	PointerKind PointerKind

	// Stream indexing metadata
	StreamID         int64
	StreamIDRef      int64
	StartTs          time.Time
	EndTs            time.Time
	LineCount        int64
	UncompressedSize int64

	// Column indexing metadata
	ColumnIndex       int64
	ColumnName        string
	ValuesBloomFilter []byte
}

A SectionPointer is a pointer to an section within another object. It is a wide object containing two types of index information:

1. Stream indexing metadata 2. Column indexing metadata

The stream indexing metadata is used to lookup which stream is in the referenced section, and their ID within the object. The column indexing metadata is used to lookup which column values are present in the referenced section. Path & Section are mandatory fields, and are used to uniquely identify the section within the referenced object.

func (*SectionPointer) Reset 

func (p *SectionPointer) Reset()

type Stats 

type Stats struct {
	UncompressedSize uint64
	CompressedSize   uint64

	MinTimestamp          time.Time
	MaxTimestamp          time.Time
	TimestampDistribution []uint64 // Stream count per hour.

	Columns []ColumnStats
}

Stats provides statistics about a streams section.

func ReadStats 

func ReadStats(ctx context.Context, section *Section) (Stats, error)

ReadStats returns statistics about the streams section. ReadStats returns an error if the streams section couldn't be inspected or if the provided ctx is canceled.

type TimeRangeRowPredicate 

type TimeRangeRowPredicate struct{ Start, End time.Time }

A TimeRangeRowPredicate is a RowPredicate which requires the timestamp of the entry to be within the range of StartTime and EndTime.

type TruePredicate added in v3.7.0 

type TruePredicate struct{}

TruePredicate is a Predicate which always returns true.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.