logs

package
v4.0.0-rc.4 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 20, 2026 License: AGPL-3.0 Imports: 30 Imported by: 0

Documentation

Overview

Package logs defines types used for the data object logs section. The logs section holds a list of log records across multiple streams.

Index

Constants

View Source
const (
	AppendUnordered = iota
	AppendOrdered
)

Variables

View Source
var RegionRead = "logs.Reader.Read"

RegionRead is the xcap region name that holds the statistics of reading rows from a logs section.

Functions

func CheckSection

func CheckSection(section *dataobj.Section) bool

CheckSection returns true if section is a logs section.

func CompareByStreamSchema

func CompareByStreamSchema(sortKeys []streams.SortKey) func(result.Result[dataset.Row], result.Result[dataset.Row]) bool

CompareByStreamSchema returns a comparison function for k-way merge of log lines using schema key ordering: [shard bucket ASC, schemaKey ASC, stream hash ASC, streamID ASC, timestamp DESC]. sortKeys map stream IDs to the corresponding schema information (0th element is unused). math.MaxInt64 is treated as a sentinel (loser-tree maxValue) and always compares greater.

func CompareForSortOrder

func CompareForSortOrder(sort SortOrder) func(result.Result[dataset.Row], result.Result[dataset.Row]) bool

CompareForSortOrder returns a comparison function for result rows for the given sort order.

func DecodeRow

func DecodeRow(columns []*Column, row dataset.Row, record *Record, sym *symbolizer.Symbolizer) error

DecodeRow decodes a record from a dataset.Row, using the provided columns to determine the column type. The list of columns must match the columns used to create the row.

DecodeRow resets record first, so a field whose column is absent from columns comes back as its zero value rather than keeping what a previous row left there.

The sym argument is used for reusing metadata strings between calls to DecodeRow. If sym is nil, metadata strings are always allocated.

func Iter

func Iter(ctx context.Context, obj *dataobj.Object) result.Seq[Record]

Iter iterates over records in the provided decoder. All logs sections are iterated over in order. Results objects returned to yield may be reused and must be copied for further use via DeepCopy().

func IterSection

func IterSection(ctx context.Context, section *Section) result.Seq[Record]

func LegacyLayoutOf

func LegacyLayoutOf(ctx context.Context, sections []*dataobj.Section) (streams.Layout, error)

LegacyLayoutOf returns the layout that every non-nil section of sections records with Section.LegacyLayout. Readers use it for objects whose streams section records no layout. It returns an error if a section records no legacy layout, if the sections disagree, or if no section is non-nil.

Types

type AndRowPredicate

type AndRowPredicate struct{ Left, Right RowPredicate }

An AndRowPredicate is a RowPredicate which requires both its Left and Right predicate to be true.

type AppendStrategy

type AppendStrategy int

type Builder

type Builder struct {
	// contains filtered or unexported fields
}

Builder accumulate a set of [Record]s within a data object.

func NewBuilder

func NewBuilder(metrics *Metrics, opts BuilderOptions) *Builder

Nwe creates a new logs section. The pageSize argument specifies how large pages should be.

func (*Builder) Append

func (b *Builder) Append(entry Record)

Append adds a new entry to b.

func (*Builder) EstimatedSize

func (b *Builder) EstimatedSize() int

EstimatedSize returns the estimated size of the Logs section in bytes.

When using AppendOrdered, records are held uncompressed in memory until flush. If BuilderOptions.EstimatedCompressionRatio is set (> 1), the uncompressed record size is divided by that ratio to approximate compressed output size.

func (*Builder) Flush

func (b *Builder) Flush(w dataobj.SectionWriter) (n int64, err error)

Flush flushes b to the provided writer.

After successful encoding, the b is reset and can be reused.

func (*Builder) Reset

func (b *Builder) Reset()

Reset resets all state, allowing b to be reused.

func (*Builder) SetTenant

func (b *Builder) SetTenant(tenant string)

SetTenant sets the tenant that owns the builder. A builder can be made multi-tenant by passing an empty string.

func (*Builder) Tenant

func (b *Builder) Tenant() string

Tenant returns the optional tenant that owns the builder.

func (*Builder) Type

func (b *Builder) Type() dataobj.SectionType

Type returns the dataobj.SectionType of the logs builder.

func (*Builder) UncompressedSize

func (b *Builder) UncompressedSize() int

UncompressedSize returns the current uncompressed size of the logs section in bytes.

type BuilderOptions

type BuilderOptions struct {
	// PageSizeHint is the size of pages to use when encoding the logs section.
	PageSizeHint int

	// PageMaxRowCount is the maximum amount of rows of pages to use when encoding the logs section.
	PageMaxRowCount int

	// BufferSize is the size of the buffer to use when accumulating log records.
	BufferSize int

	// StripeMergeLimit is the maximum number of stripes to merge at once when
	// flushing stripes into a section. StripeMergeLimit must be larger than 1.
	//
	// Lower values of StripeMergeLimit reduce the memory overhead of merging but
	// increase time spent merging. Higher values of StripeMergeLimit increase
	// memory overhead but reduce time spent merging.
	StripeMergeLimit int

	// AppendStrategy is allowed to control how the builder creates the section.
	// When appending logs to the section in strict sort order, the [AppendOrdered] can be used to avoid
	// creating and sorting of stripes.
	AppendStrategy AppendStrategy

	// EstimatedCompressionRatio is the expected compression ratio used by
	// [EstimatedSize] to approximate compressed output size from uncompressed
	// buffered records when using [AppendOrdered]. Only takes effect with
	// AppendOrdered; ignored for AppendUnordered where stripes are already
	// compressed. A value of either 0 or 1 disables the adjustment.
	EstimatedCompressionRatio int

	// SortOrder defines the order in which the rows of the logs sections are sorted.
	// They can either be sorted by [streamID ASC, timestamp DESC] ([SortStreamASC]) or [timestamp DESC, streamID ASC] ([SortTimestampDESC]).
	SortOrder SortOrder

	// DropDuplicates indicates whether duplicate records should be dropped or retained during Flush.
	// Records are considered duplicates if they have the same stream ID, timestamp, metadata, and line bytes.
	DropDuplicates bool
}

BuilderOptions configures the behavior of the logs section.

type Column

type Column struct {
	Section *Section   // Section that contains the column.
	Name    string     // Optional name of the column.
	Type    ColumnType // Type of data in the column.
	// contains filtered or unexported fields
}

A Column represents one of the columns in the logs section. Valid columns can only be retrieved by calling Section.Columns.

Data in columns can be read by using a RowReader.

type ColumnStats

type ColumnStats struct {
	Name             string
	Type             string
	ValueType        string
	RowsCount        uint64
	Compression      string
	UncompressedSize uint64
	CompressedSize   uint64
	MetadataOffset   uint64
	MetadataSize     uint64
	ValuesCount      uint64
	Cardinality      uint64
	ColumnIndex      int64

	Pages []PageStats
}

ColumnStats provides statistics about a column in a section.

type ColumnType

type ColumnType int

ColumnType represents the kind of information stored in a Column.

const (
	ColumnTypeInvalid   ColumnType = iota // ColumnTypeInvalid is an invalid column.
	ColumnTypeStreamID                    // ColumnTypeStreamID is a column that contains stream IDs.
	ColumnTypeTimestamp                   // ColumnTypeTimestamp is a column that contains timestamps per log record.

	// ColumnTypeMetadata is a column containing a sequence of structured
	// metadata values per log record. There will be one ColumnTypeMetadata per
	// structured metadata key; the name of the structured metadata key is stored
	// as the column name.
	ColumnTypeMetadata

	ColumnTypeMessage // ColumnTypeMessage is a column that contains log messages.
)

func ParseColumnType

func ParseColumnType(text string) (ColumnType, error)

ParseColumnType parses a ColumnType from a string. The expected string format is the same as the return value of ColumnType.String.

func (ColumnType) String

func (ct ColumnType) String() string

String returns the human-readable name of ct.

type ColumnarDataset

type ColumnarDataset = columnar.Dataset

ColumnarDataset is the exported type alias of the internal [columnar.Dataset].

func MakeColumnarDataset

func MakeColumnarDataset(section *Section) (*ColumnarDataset, error)

MakeColumnarDataset is the exported entry point for sortmerge, the only caller outside this package; internal callers use makeDataset.

type DatasetSequence

type DatasetSequence struct {
	// contains filtered or unexported fields
}

func NewDatasetSequence

func NewDatasetSequence(r *dataset.RowReader, bufferSize int) DatasetSequence

func (*DatasetSequence) At

func (*DatasetSequence) Close

func (seq *DatasetSequence) Close()

func (*DatasetSequence) Next

func (seq *DatasetSequence) Next() bool

type LogMessageFilterRowPredicate

type LogMessageFilterRowPredicate struct {
	// Keep receives a nil line when the entry has no message value. The slice
	// belongs to the reader and is valid only during the call, so Keep must not
	// retain, modify, or append to it. Reads of one query can call the same Keep
	// at the same time, so it must be safe for concurrent use.
	Keep func(line []byte) bool

	// KeepBatch is optional. It tests a batch of lines at once. This saves a Keep call for each
	// row and a copy of each line that fails. The reader calls Keep when it cannot use KeepBatch,
	// so Keep must always be set.
	//
	// Line i is data[offsets[i]:offsets[i+1]]. An entry with no message is an empty line, and
	// Keep gets a nil line for it, so Keep must treat a nil and an empty line alike. Lines have
	// no separator, offsets can start after 0, and data can hold bytes that belong to no line.
	//
	// KeepBatch appends to dstIndexes the index of every line that Keep keeps, in ascending
	// order with no repeat, and returns the result. It must give the same result as Keep on
	// every line.
	//
	// The arguments belong to the reader and are valid only during the call. KeepBatch must
	// not retain data, offsets or dstIndexes, and must not modify data or offsets. Like Keep,
	// it must be safe for concurrent use.
	KeepBatch func(data []byte, offsets []int32, dstIndexes []int32) []int32
}

A LogMessageFilterRowPredicate is a RowPredicate that requires the log message of the entry to pass a Keep function.

type MetadataFilterRowPredicate

type MetadataFilterRowPredicate struct {
	Key  string
	Keep func(key, value string) bool
}

A MetadataFilterRowPredicate is a RowPredicate that requires that metadata with the provided key pass a Keep function.

The key is provided to the keep function to allow the same function to be used for multiple filter predicates.

Uses of MetadataFilterRowPredicate are not eligible for page filtering and should only be used when a condition cannot be expressed by other basic predicates.

type MetadataMatcherRowPredicate

type MetadataMatcherRowPredicate struct{ Key, Value string }

A MetadataMatcherRowPredicate is a RowPredicate that requires a metadata key named Key to exist with a value of Value.

type Metrics

type Metrics struct {
	// contains filtered or unexported fields
}

Metrics instruments the logs section.

func NewMetrics

func NewMetrics() *Metrics

NewMetrics creates a new set of metrics for the logs section.

func (*Metrics) Observe

func (m *Metrics) Observe(ctx context.Context, section *Section) error

Observe observes section statistics for a given section.

func (*Metrics) Register

func (m *Metrics) Register(reg prometheus.Registerer) error

Register registers metrics to report to reg.

func (*Metrics) Unregister

func (m *Metrics) Unregister(reg prometheus.Registerer)

Unregister unregisters metrics from the provided Registerer.

type NotRowPredicate

type NotRowPredicate struct{ Inner RowPredicate }

A NotRowPredicate is a RowPredicate which requires its Inner predicate to be false.

type OrRowPredicate

type OrRowPredicate struct{ Left, Right RowPredicate }

An OrRowPredicate is a RowPredicate which requires either its Left or Right predicate to be true.

type PageStats

type PageStats struct {
	UncompressedSize uint64
	CompressedSize   uint64
	CRC32            uint32
	RowsCount        uint64
	Encoding         string
	DataOffset       uint64
	DataSize         uint64
	ValuesCount      uint64
}

PageStats provides statistics about a page in a column.

type Record

type Record struct {
	StreamID  int64
	Timestamp time.Time
	Metadata  labels.Labels
	Line      []byte

	// SchemaKey is a pre-computed schema sort key. It is not encoded into the section;
	// it only guides the in-memory sort during building.
	SchemaKey string

	// ShardBucket is the physical shard bucket for the record's stream. It is
	// not encoded into LOG sections; it guides schema-layout sorting.
	ShardBucket uint32

	// StreamHash is labels.StableHash of the record's stream labels. It is not
	// encoded into LOG sections; it guides schema-layout sorting.
	StreamHash uint64
}

A Record is an individual log record within the logs section.

func (*Record) Copy

func (r *Record) Copy() Record

func (*Record) Reset

func (r *Record) Reset()

Reset clears r so it can be reused for another row.

The line is truncated rather than dropped, so it keeps its capacity and a reused Record does not reallocate its buffer for every row.

func (*Record) UncompressedSize

func (r *Record) UncompressedSize() int64

UncompressedSize returns the line bytes plus the structured metadata value bytes of r. It matches streams.Stream.UncompressedSize and the index statistics, so every producer reports the same quantity.

Metadata names are not counted because the section stores each name once per column, not once per record.

type RowPredicate

type RowPredicate interface {
	// contains filtered or unexported methods
}

RowPredicate is an expression used to filter rows in a data object.

type RowReader

type RowReader struct {
	// contains filtered or unexported fields
}

RowReader reads the set of logs from an [Object].

func NewRowReader

func NewRowReader(sec *Section) *RowReader

NewRowReader creates a new RowReader that reads from the provided Section.

Call RowReader.Open before calling RowReader.Read.

func (*RowReader) Close

func (r *RowReader) Close() error

Close closes the RowReader and releases any resources it holds. Closed RowReaders can be reused by calling RowReader.Reset.

func (*RowReader) MatchStreams

func (r *RowReader) MatchStreams(ids iter.Seq[int64]) error

MatchStreams provides a sequence of stream IDs for the logs reader to match. RowReader.Read will only return logs for the provided stream IDs.

MatchStreams may be called multiple times to match multiple sets of streams.

MatchStreams may only be called before reading begins or after a call to RowReader.Reset.

func (*RowReader) Open

func (r *RowReader) Open(ctx context.Context) error

Open initializes RowReader resources.

Open must be called before RowReader.Read. Open is safe to call multiple times. Open is a no-op when the reader has no section.

func (*RowReader) Read

func (r *RowReader) Read(ctx context.Context, s []Record) (int, error)

Read reads up to the next len(s) records from the reader and stores them into s. It returns the number of records read and any error encountered. At the end of the logs section, Read returns 0, io.EOF.

func (*RowReader) Reset

func (r *RowReader) Reset(sec *Section)

Reset resets the RowReader with a new Section to read from. Reset allows reusing a RowReader without allocating a new one.

Any set predicate, stream match and column projection are cleared when Reset is called.

Reset may be called with a nil section to clear the RowReader without needing a new one.

func (*RowReader) SetPredicates

func (r *RowReader) SetPredicates(p []RowPredicate) error

SetPredicate sets the predicates to use for filtering logs. RowReader.Read will only return logs for which the predicate passes.

Predicates may only be set before reading begins or after a call to RowReader.Reset.

func (*RowReader) SetProjectedColumns

func (r *RowReader) SetProjectedColumns(types []ColumnType, metadataNames []string) error

SetProjectedColumns restricts the Record to the given column types plus the metadata columns named in metadataNames. A ColumnTypeMetadata entry in types selects all metadata columns instead.

The field of a column that is not projected comes back as its zero value.

The reader reads the columns a stream match or a predicate needs, even when the caller does not list them. It does not decode them into the Record.

When the section holds no such column, a metadata or message predicate evaluates against the empty value.

When the section holds none of the projected columns, Open returns an error.

When SetProjectedColumns is not called, every recognized column is projected.

SetProjectedColumns may only be called before reading begins or after a call to RowReader.Reset.

type Section

type Section struct {
	// contains filtered or unexported fields
}

Section represents an opened logs section.

func Open

func Open(ctx context.Context, section *dataobj.Section) (*Section, error)

Open opens a Section from an underlying dataobj.Section. Open returns an error if the section metadata could not be read or if the provided ctx is canceled.

func (*Section) Columns

func (s *Section) Columns() []*Column

Columns returns the set of Columns in the section. The slice of returned sections must not be mutated.

Unrecognized columns (e.g., when running older code against newer sterams sections) are skipped.

func (*Section) LegacyLayout

func (s *Section) LegacyLayout() (layout streams.Layout, ok bool)

LegacyLayout returns the stream layout that the section recorded in its own sort info. Writers recorded it there before the streams section did. ok is false if the section records no such layout, or one that is not stable hash ordered with the default shard layout, because no writer produced another.

Readers use LegacyLayout only for objects whose streams section records no layout.

func (*Section) PrimarySortOrder

func (s *Section) PrimarySortOrder() (ColumnType, SortDirection, error)

PrimarySortOrder returns the primary sort order information of the section as a tuple of ColumnType and SortDirection.

type SortDirection

type SortDirection int

SortDirection represents sort direction of a column.

const (
	SortDirectionUnspecified SortDirection = 0 // Sort direction is unspecified.
	SortDirectionAscending   SortDirection = 1 // SortDirectionAscending represents ascending sort order (smallest values first).
	SortDirectionDescending  SortDirection = 2 // SortDirectionDescending represents descending sort order (largest values first).
)

type SortOrder

type SortOrder int
const (
	SortStreamASC SortOrder = iota
	SortTimestampDESC
	SortSchemaASC
)

type Stats

type Stats struct {
	UncompressedSize uint64
	CompressedSize   uint64

	Columns []ColumnStats
}

Stats provides statistics about a streams section.

func ReadStats

func ReadStats(ctx context.Context, section *Section) (Stats, error)

ReadStats returns statistics about the logs section. ReadStats returns an error if the streams section couldn't be inspected or if the provided ctx is canceled.

type TimeRangeRowPredicate

type TimeRangeRowPredicate struct {
	StartTime, EndTime time.Time
	IncludeStart       bool // Whether StartTime is inclusive.
	IncludeEnd         bool // Whether EndTime is inclusive.
}

A TimeRangeRowPredicate is a RowPredicate which requires the timestamp of the entry to be within the range of StartTime and EndTime.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL