Documentation
¶
Overview ¶
Package columnar provides a base implementation for sections which store columnar data using github.com/grafana/loki/v3/pkg/dataobj/internal/dataset.
Index ¶
- Constants
- type Column
- type ColumnEncoder
- type Dataset
- func (ds *Dataset) Columns() []dataset.Column
- func (ds *Dataset) ListColumns(_ context.Context) result.Seq[dataset.Column]
- func (ds *Dataset) ListPages(ctx context.Context, columns []dataset.Column) result.Seq[dataset.Pages]
- func (ds *Dataset) ReadPages(ctx context.Context, pages []dataset.Page) result.Seq[dataset.PageData]
- type DatasetColumn
- type DatasetPage
- type Decoder
- func (dec *Decoder) Pages(ctx context.Context, columns []*datasetmd.ColumnDesc) result.Seq[[]*datasetmd.PageDesc]
- func (dec *Decoder) ReadPages(ctx context.Context, pages []*datasetmd.PageDesc) result.Seq[dataset.PageData]
- func (dec *Decoder) SectionMetadata(ctx context.Context) (*datasetmd.SectionMetadata, error)
- type Encoder
- func (enc *Encoder) Flush(w dataobj.SectionWriter) (int64, error)
- func (enc *Encoder) NumColumns() int
- func (enc *Encoder) OpenColumn(desc *dataset.ColumnDesc) (*ColumnEncoder, error)
- func (enc *Encoder) Reset()
- func (enc *Encoder) SetSortInfo(info *datasetmd.SortInfo)
- func (enc *Encoder) SetTenant(tenant string)
- type Metrics
- type Section
Constants ¶
const FormatVersion = 2
FormatVersion is the current version the encoding format for dataset-derived sections.
FormatVersion started at 2 to help consolidate all the duplicated encoding code across the original section implementations (which were all collectively at v1).
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Column ¶
type Column struct {
Section *Section // Section that contains this Column.
Type dataset.ColumnType // Type of the column.
Tag string // Optional tag of the column.
// contains filtered or unexported fields
}
A Column represents one of the columns in the section. Valid columns can only be retrieved by calling Section.Columns.
Data in columns can be read by using a [Reader].
func (*Column) Metadata ¶
func (c *Column) Metadata() *datasetmd.ColumnDesc
Metadata returns the protobuf metadata of the column.
type ColumnEncoder ¶
type ColumnEncoder struct {
// contains filtered or unexported fields
}
The ColumnEncoder encodes data for a single column in a section. ColumnEncoders must be created by an Encoder.
func (*ColumnEncoder) AppendPage ¶
func (enc *ColumnEncoder) AppendPage(page *dataset.MemPage) error
AppendPage appends a new page to enc. The page must not be modified after passing to AppendPage.
AppendPage fails if enc is closed.
func (*ColumnEncoder) Commit ¶
func (enc *ColumnEncoder) Commit() error
Commit completes the column, appending it to the section. After Commit is called, enc can no longer be used.
If no pages have been appnded, Commit appends an empty column to the section.
func (*ColumnEncoder) Discard ¶
func (enc *ColumnEncoder) Discard() error
Discard discards the column, discarding any data written to it. After Discard is called, enc can no longer be used.
type Dataset ¶
type Dataset struct {
// contains filtered or unexported fields
}
Dataset is a dataset.Dataset implementation that reads from a slice of Column.
func MakeDataset ¶
MakeDataset returns a Dataset from a section and a set of columns. MakeDataset returns an error if not all columns are from the provided section.
func (*Dataset) Columns ¶
Columns returns the set of [dataset.Column]s in the dataset. The order of returned columns matches the order from MakeDataset. The returned slice must not be modified.
func (*Dataset) ListColumns ¶
ListColumns returns an iterator over the columns in the dataset.
type DatasetColumn ¶
type DatasetColumn struct {
// contains filtered or unexported fields
}
DatasetColumn is a dataset.Column implementation that reads from a Column. It is automatically created when using MakeDataset.
func MakeDatasetColumn ¶
func MakeDatasetColumn(dec *Decoder, col *Column) *DatasetColumn
MakeDatasetColumn returns a DatasetColumn from a decoder and a column.
func (*DatasetColumn) ColumnDesc ¶
func (ds *DatasetColumn) ColumnDesc() *dataset.ColumnDesc
ColumnDesc returns the dataset.ColumnDesc for the column.
type DatasetPage ¶
type DatasetPage struct {
// contains filtered or unexported fields
}
DatasetPage is a dataset.Page implementation that reads from an individual page within a column.
func MakeDatasetPage ¶
func MakeDatasetPage(dec *Decoder, desc *datasetmd.PageDesc) *DatasetPage
MakeDatasetPage returns a DatasetPage from a decoder and page description.
func (*DatasetPage) PageDesc ¶
func (p *DatasetPage) PageDesc() *dataset.PageDesc
PageDesc returns the page description.
type Decoder ¶
type Decoder struct {
// contains filtered or unexported fields
}
A Decoder allows reading an encoded dataset-based section.
func NewDecoder ¶
func NewDecoder(reader dataobj.SectionReader, formatVersion uint32) (*Decoder, error)
NewDecoder creates a new Decoder for the given dataobj.SectionReader. The formatVersion argument must denote the format version of the data being decoded.
NewDecoder returns an error if the format version is not supported.
func (*Decoder) Pages ¶
func (dec *Decoder) Pages(ctx context.Context, columns []*datasetmd.ColumnDesc) result.Seq[[]*datasetmd.PageDesc]
Pages returns the set of pages for the provided columns. The order of slices of pages emitted by the iterator matches the order of the columns slice: the first slice corresponds to the first column, and so on.
func (*Decoder) ReadPages ¶
func (dec *Decoder) ReadPages(ctx context.Context, pages []*datasetmd.PageDesc) result.Seq[dataset.PageData]
ReadPages reads the provided set of pages, iterating over their data matching the argument order. If an error is encountered while retrieving pages, an error is emitted from the sequence and iteration stops.
func (*Decoder) SectionMetadata ¶
SectionMetadata returns the metadata for the section.
type Encoder ¶
type Encoder struct {
// contains filtered or unexported fields
}
Encoder encodes an individual dataset-based section in a data object.
The zero value of Encoder is ready for use.
func (*Encoder) Flush ¶
func (enc *Encoder) Flush(w dataobj.SectionWriter) (int64, error)
Flush writes the section to the given dataobj.SectionWriter. Flush returns an error if there is a currently open column.
After Flush is called successfully, the encoder is reset to a fresh state and can be reused.
func (*Encoder) NumColumns ¶
NumColumns returns the number of columns committed to the section.
func (*Encoder) OpenColumn ¶
func (enc *Encoder) OpenColumn(desc *dataset.ColumnDesc) (*ColumnEncoder, error)
OpenColumn opens a new column in the section. OpenColumn fails if there is another open column.
func (*Encoder) Reset ¶
func (enc *Encoder) Reset()
Reset resets the encoder to a fresh state, discarding any in-progress columns.
func (*Encoder) SetSortInfo ¶
SetSortInfo sets the sort order information for the section. This must be called before calling Encoder.Flush. The sort order information is reset after flushing or calling Encoder.Reset.
func (*Encoder) SetTenant ¶
SetTenant sets the tenant ID for the section. This must be called before calling Encoder.Flush. The set tenant is reset after flushing or calling Encoder.Reset.
type Metrics ¶
type Metrics struct {
// contains filtered or unexported fields
}
Metrics instruments the a columnar section. Metrics are only updated when calling Metrics.Observe.
func NewMetrics ¶
func NewMetrics(sectionType dataobj.SectionType) *Metrics
NewMetrics creates a new set of metrics for the streams section. Call Metrics.Observe to update the metrics for a given section.
func (*Metrics) Register ¶
func (m *Metrics) Register(reg prometheus.Registerer) error
Register registers metrics to report to reg.
func (*Metrics) Unregister ¶
func (m *Metrics) Unregister(reg prometheus.Registerer)
Unregister unregisters metrics from the provided Registerer.
type Section ¶
type Section struct {
// contains filtered or unexported fields
}
Section represents an opened dataset-based section.
func Open ¶
Open opens a Section from an underlying Decoder.
Open returns an error if the section metadata couldn't be read.
func (*Section) Columns ¶
Columns returns the set of Columns in the section. The slice of returned sections must not be mutated.
func (*Section) PrimarySortOrder ¶
func (s *Section) PrimarySortOrder() (dataset.ColumnType, datasetmd.SortDirection, error)
PrimarySortOrder returns the primary sort order information of the section as a tuple of [ColumnType] and [SortDirection].
Source Files