Documentation
¶
Overview ¶
Package pointers defines types used for the data object pointers section. The pointers section holds a list of pointers to sections present in the data object.
Index ¶
- func CheckSection(section *dataobj.Section) bool
- func Iter(ctx context.Context, obj *dataobj.Object) result.Seq[SectionPointer]
- func IterSection(ctx context.Context, section *Section) result.Seq[SectionPointer]
- type AndRowPredicate
- type BloomExistenceRowPredicate
- type Builder
- func (b *Builder) EstimatedSize() int
- func (b *Builder) Flush(w dataobj.SectionWriter) (n int64, err error)
- func (b *Builder) ObserveStream(path string, section int64, idInObject int64, idInIndex int64, ts time.Time, ...)
- func (b *Builder) RecordColumnIndex(path string, section int64, columnName string, columnIndex int64, ...)
- func (b *Builder) Reset()
- func (b *Builder) SetTenant(tenant string)
- func (b *Builder) Tenant() string
- func (b *Builder) Type() dataobj.SectionType
- type Column
- type ColumnStats
- type ColumnType
- type Metrics
- type PageStats
- type PointerKind
- type RowPredicate
- type RowReader
- type Section
- type SectionPointer
- type Stats
- type TimeRangeRowPredicate
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func CheckSection ¶
CheckSection returns true if section is a streams section.
func Iter ¶
Iter iterates over pointers in the provided decoder. All pointers sections are iterated over in order.
func IterSection ¶
Types ¶
type AndRowPredicate ¶
type AndRowPredicate struct{ Left, Right RowPredicate }
An AndRowPredicate is a RowPredicate which requires both its Left and Right predicate to be true.
type BloomExistenceRowPredicate ¶
type BloomExistenceRowPredicate struct{ Name, Value string }
A BloomExistenceRowPredicate is a RowPredicate which requires a bloom filter column named Name to exist, and for the Value to pass the bloom filter.
type Builder ¶
type Builder struct {
// contains filtered or unexported fields
}
Builder builds a pointers section.
func NewBuilder ¶
NewBuilder creates a new pointers section builder. The pageSize argument specifies how large pages should be.
func (*Builder) EstimatedSize ¶
EstimatedSize returns the estimated size of the Pointers section in bytes.
func (*Builder) Flush ¶
func (b *Builder) Flush(w dataobj.SectionWriter) (n int64, err error)
Flush flushes the streams section to the provided writer.
After successful encoding, b is reset to a fresh state and can be reused.
func (*Builder) ObserveStream ¶
func (b *Builder) ObserveStream(path string, section int64, idInObject int64, idInIndex int64, ts time.Time, uncompressedSize int64)
ObserveStream observes a stream in the index by recording the start & end timestamps, line count, and uncompressed size per-section.
func (*Builder) RecordColumnIndex ¶
func (*Builder) Reset ¶
func (b *Builder) Reset()
Reset resets all state, allowing Pointers builder to be reused.
func (*Builder) Type ¶
func (b *Builder) Type() dataobj.SectionType
Type returns the dataobj.SectionType of the pointers builder.
type Column ¶
type Column struct {
Section *Section // Section that contains the column.
Name string // Optional name of the column.
Type ColumnType // Type of data in the column.
// contains filtered or unexported fields
}
A Column represents one of the columns in the streams section. Valid columns can only be retrieved by calling Section.Columns.
Data in columns can be read by using a [Reader].
type ColumnStats ¶
type ColumnStats struct {
Name string
Type string
ValueType string
RowsCount uint64
Compression string
UncompressedSize uint64
CompressedSize uint64
MetadataOffset uint64
MetadataSize uint64
ValuesCount uint64
Cardinality uint64
Pages []PageStats
}
ColumnStats provides statistics about a column in a section.
type ColumnType ¶
type ColumnType int
ColumnType represents the kind of information stored in a Column.
const ( ColumnTypeInvalid ColumnType = iota // ColumnTypeInvalid is an invalid column. ColumnTypePath ColumnTypeSection ColumnTypePointerKind // ColumnTypePointerKind is a column containing the kind of pointer: stream or column. ColumnTypeStreamID // ColumnTypeStreamID is a column containing a set of stream IDs. ColumnTypeStreamIDRef // ColumnTypeStreamIDRef is a column containing a set of stream IDs from the referenced object. ColumnTypeMinTimestamp // ColumnTypeMinTimestamp is a column containing minimum timestamps per stream. ColumnTypeMaxTimestamp // ColumnTypeMaxTimestamp is a column containing maximum timestamps per stream. ColumnTypeRowCount // ColumnTypeRowCount is a column containing row count per stream. ColumnTypeUncompressedSize // ColumnTypeUncompressedSize is a column containing uncompressed size per stream. ColumnTypeColumnName // ColumnTypeColumnName is a column containing the name of the column in the referenced object. ColumnTypeColumnIndex // ColumnTypeColumnIndex is a column containing the index of the column in the referenced object. ColumnTypeValuesBloomFilter // ColumnTypeValuesBloomFilter is a column containing a bloom filter of the values in the column in the referenced object. )
func ParseColumnType ¶
func ParseColumnType(text string) (ColumnType, error)
ParseColumnType parses a ColumnType from a string. The expected string format is the same as what's returned by ColumnType.String.
func (ColumnType) String ¶
func (ct ColumnType) String() string
String returns the human-readable name of ct.
type Metrics ¶
type Metrics struct {
// contains filtered or unexported fields
}
Metrics instruments the streams section.
func NewMetrics ¶
func NewMetrics() *Metrics
NewMetrics creates a new set of metrics for the pointers section.
func (*Metrics) Register ¶
func (m *Metrics) Register(reg prometheus.Registerer) error
Register registers metrics to report to reg.
func (*Metrics) Unregister ¶
func (m *Metrics) Unregister(reg prometheus.Registerer)
Unregister unregisters metrics from the provided Registerer.
type PageStats ¶
type PageStats struct {
UncompressedSize uint64
CompressedSize uint64
CRC32 uint32
RowsCount uint64
Encoding string
DataOffset uint64
DataSize uint64
ValuesCount uint64
}
PageStats provides statistics about a page in a column.
type PointerKind ¶
type PointerKind int
const ( PointerKindInvalid PointerKind = iota // PointerKindInvalid is an invalid pointer kind. PointerKindStreamIndex // PointerKindStreamIndex is a pointer for a stream index. PointerKindColumnIndex // PointerKindColumnIndex is a pointer for a column index. )
type RowPredicate ¶
type RowPredicate interface {
// contains filtered or unexported methods
}
RowPredicate is an expression used to filter rows in a data object.
type RowReader ¶
type RowReader struct {
// contains filtered or unexported fields
}
RowReader reads the set of streams from an [Object].
func NewRowReader ¶
NewRowReader creates a new RowReader that reads rows from the provided Section.
func (*RowReader) Close ¶
Close closes the RowReader and releases any resources it holds. Closed RowReaders can be reused by calling RowReader.Reset.
func (*RowReader) MatchStreams ¶
MatchStreams provides a sequence of stream IDs for the logs reader to match. RowReader.Read will only return logs for the provided stream IDs.
MatchStreams may be called multiple times to match multiple sets of streams.
MatchStreams may only be called before reading begins or after a call to RowReader.Reset.
func (*RowReader) Read ¶
Read reads up to the next len(s) streams from the reader and stores them into s. It returns the number of streams read and any error encountered. At the end of the stream section, Read returns 0, io.EOF.
func (*RowReader) Reset ¶
Reset resets the RowReader with a new decoder to read from. Reset allows reusing a RowReader without allocating a new one.
Any set predicate is cleared when Reset is called.
Reset may be called with a nil object and a negative section index to clear the RowReader without needing a new object.
func (*RowReader) SetPredicate ¶
func (r *RowReader) SetPredicate(p RowPredicate) error
SetPredicate sets the predicate to use for filtering logs. [LogsReader.Read] will only return logs for which the predicate passes.
SetPredicate returns an error if the predicate is not supported by LogsReader.
A predicate may only be set before reading begins or after a call to RowReader.Reset.
type Section ¶
type Section struct {
// contains filtered or unexported fields
}
Section represents an opened streams section.
func Open ¶
Open opens a Section from an underlying dataobj.Section. Open returns an error if the section metadata could not be read or if the provided ctx is canceled.
type SectionPointer ¶
type SectionPointer struct {
Path string
Section int64
PointerKind PointerKind
// Stream indexing metadata
StreamID int64
StreamIDRef int64
StartTs time.Time
EndTs time.Time
LineCount int64
UncompressedSize int64
// Column indexing metadata
ColumnIndex int64
ColumnName string
ValuesBloomFilter []byte
}
A SectionPointer is a pointer to an section within another object. It is a wide object containing two types of index information:
1. Stream indexing metadata 2. Column indexing metadata
The stream indexing metadata is used to lookup which stream is in the referenced section, and their ID within the object. The column indexing metadata is used to lookup which column values are present in the referenced section. Path & Section are mandatory fields, and are used to uniquely identify the section within the referenced object.
func (*SectionPointer) Reset ¶
func (p *SectionPointer) Reset()
type Stats ¶
type Stats struct {
UncompressedSize uint64
CompressedSize uint64
MinTimestamp time.Time
MaxTimestamp time.Time
TimestampDistribution []uint64 // Stream count per hour.
Columns []ColumnStats
}
Stats provides statistics about a streams section.
type TimeRangeRowPredicate ¶
A TimeRangeRowPredicate is a RowPredicate which requires the timestamp of the entry to be within the range of StartTime and EndTime.
Source Files