Documentation
¶
Overview ¶
Package logs defines types used for the data object logs section. The logs section holds a list of log records across multiple streams.
Index ¶
- Constants
- Variables
- func CheckSection(section *dataobj.Section) bool
- func CompareByStreamSchema(sortKeys []streams.SortKey) func(result.Result[dataset.Row], result.Result[dataset.Row]) bool
- func CompareForSortOrder(sort SortOrder) func(result.Result[dataset.Row], result.Result[dataset.Row]) bool
- func DecodeRow(columns []*Column, row dataset.Row, record *Record, sym *symbolizer.Symbolizer) error
- func Iter(ctx context.Context, obj *dataobj.Object) result.Seq[Record]
- func IterSection(ctx context.Context, section *Section) result.Seq[Record]
- func LegacyLayoutOf(ctx context.Context, sections []*dataobj.Section) (streams.Layout, error)
- type AndRowPredicate
- type AppendStrategy
- type Builder
- func (b *Builder) Append(entry Record)
- func (b *Builder) EstimatedSize() int
- func (b *Builder) Flush(w dataobj.SectionWriter) (n int64, err error)
- func (b *Builder) Reset()
- func (b *Builder) SetTenant(tenant string)
- func (b *Builder) Tenant() string
- func (b *Builder) Type() dataobj.SectionType
- func (b *Builder) UncompressedSize() int
- type BuilderOptions
- type Column
- type ColumnStats
- type ColumnType
- type ColumnarDataset
- type DatasetSequence
- type LogMessageFilterRowPredicate
- type MetadataFilterRowPredicate
- type MetadataMatcherRowPredicate
- type Metrics
- type NotRowPredicate
- type OrRowPredicate
- type PageStats
- type Record
- type RowPredicate
- type RowReader
- func (r *RowReader) Close() error
- func (r *RowReader) MatchStreams(ids iter.Seq[int64]) error
- func (r *RowReader) Open(ctx context.Context) error
- func (r *RowReader) Read(ctx context.Context, s []Record) (int, error)
- func (r *RowReader) Reset(sec *Section)
- func (r *RowReader) SetPredicates(p []RowPredicate) error
- func (r *RowReader) SetProjectedColumns(types []ColumnType, metadataNames []string) error
- type Section
- type SortDirection
- type SortOrder
- type Stats
- type TimeRangeRowPredicate
Constants ¶
const ( AppendUnordered = iota AppendOrdered )
Variables ¶
var RegionRead = "logs.Reader.Read"
RegionRead is the xcap region name that holds the statistics of reading rows from a logs section.
Functions ¶
func CheckSection ¶
CheckSection returns true if section is a logs section.
func CompareByStreamSchema ¶
func CompareByStreamSchema(sortKeys []streams.SortKey) func(result.Result[dataset.Row], result.Result[dataset.Row]) bool
CompareByStreamSchema returns a comparison function for k-way merge of log lines using schema key ordering: [shard bucket ASC, schemaKey ASC, stream hash ASC, streamID ASC, timestamp DESC]. sortKeys map stream IDs to the corresponding schema information (0th element is unused). math.MaxInt64 is treated as a sentinel (loser-tree maxValue) and always compares greater.
func CompareForSortOrder ¶
func CompareForSortOrder(sort SortOrder) func(result.Result[dataset.Row], result.Result[dataset.Row]) bool
CompareForSortOrder returns a comparison function for result rows for the given sort order.
func DecodeRow ¶
func DecodeRow(columns []*Column, row dataset.Row, record *Record, sym *symbolizer.Symbolizer) error
DecodeRow decodes a record from a dataset.Row, using the provided columns to determine the column type. The list of columns must match the columns used to create the row.
DecodeRow resets record first, so a field whose column is absent from columns comes back as its zero value rather than keeping what a previous row left there.
The sym argument is used for reusing metadata strings between calls to DecodeRow. If sym is nil, metadata strings are always allocated.
func Iter ¶
Iter iterates over records in the provided decoder. All logs sections are iterated over in order. Results objects returned to yield may be reused and must be copied for further use via DeepCopy().
func LegacyLayoutOf ¶
LegacyLayoutOf returns the layout that every non-nil section of sections records with Section.LegacyLayout. Readers use it for objects whose streams section records no layout. It returns an error if a section records no legacy layout, if the sections disagree, or if no section is non-nil.
Types ¶
type AndRowPredicate ¶
type AndRowPredicate struct{ Left, Right RowPredicate }
An AndRowPredicate is a RowPredicate which requires both its Left and Right predicate to be true.
type AppendStrategy ¶
type AppendStrategy int
type Builder ¶
type Builder struct {
// contains filtered or unexported fields
}
Builder accumulate a set of [Record]s within a data object.
func NewBuilder ¶
func NewBuilder(metrics *Metrics, opts BuilderOptions) *Builder
Nwe creates a new logs section. The pageSize argument specifies how large pages should be.
func (*Builder) EstimatedSize ¶
EstimatedSize returns the estimated size of the Logs section in bytes.
When using AppendOrdered, records are held uncompressed in memory until flush. If BuilderOptions.EstimatedCompressionRatio is set (> 1), the uncompressed record size is divided by that ratio to approximate compressed output size.
func (*Builder) Flush ¶
func (b *Builder) Flush(w dataobj.SectionWriter) (n int64, err error)
Flush flushes b to the provided writer.
After successful encoding, the b is reset and can be reused.
func (*Builder) SetTenant ¶
SetTenant sets the tenant that owns the builder. A builder can be made multi-tenant by passing an empty string.
func (*Builder) Type ¶
func (b *Builder) Type() dataobj.SectionType
Type returns the dataobj.SectionType of the logs builder.
func (*Builder) UncompressedSize ¶
UncompressedSize returns the current uncompressed size of the logs section in bytes.
type BuilderOptions ¶
type BuilderOptions struct {
// PageSizeHint is the size of pages to use when encoding the logs section.
PageSizeHint int
// PageMaxRowCount is the maximum amount of rows of pages to use when encoding the logs section.
PageMaxRowCount int
// BufferSize is the size of the buffer to use when accumulating log records.
BufferSize int
// StripeMergeLimit is the maximum number of stripes to merge at once when
// flushing stripes into a section. StripeMergeLimit must be larger than 1.
//
// Lower values of StripeMergeLimit reduce the memory overhead of merging but
// increase time spent merging. Higher values of StripeMergeLimit increase
// memory overhead but reduce time spent merging.
StripeMergeLimit int
// AppendStrategy is allowed to control how the builder creates the section.
// When appending logs to the section in strict sort order, the [AppendOrdered] can be used to avoid
// creating and sorting of stripes.
AppendStrategy AppendStrategy
// EstimatedCompressionRatio is the expected compression ratio used by
// [EstimatedSize] to approximate compressed output size from uncompressed
// buffered records when using [AppendOrdered]. Only takes effect with
// AppendOrdered; ignored for AppendUnordered where stripes are already
// compressed. A value of either 0 or 1 disables the adjustment.
EstimatedCompressionRatio int
// SortOrder defines the order in which the rows of the logs sections are sorted.
// They can either be sorted by [streamID ASC, timestamp DESC] ([SortStreamASC]) or [timestamp DESC, streamID ASC] ([SortTimestampDESC]).
SortOrder SortOrder
// DropDuplicates indicates whether duplicate records should be dropped or retained during Flush.
// Records are considered duplicates if they have the same stream ID, timestamp, metadata, and line bytes.
DropDuplicates bool
}
BuilderOptions configures the behavior of the logs section.
type Column ¶
type Column struct {
Section *Section // Section that contains the column.
Name string // Optional name of the column.
Type ColumnType // Type of data in the column.
// contains filtered or unexported fields
}
A Column represents one of the columns in the logs section. Valid columns can only be retrieved by calling Section.Columns.
Data in columns can be read by using a RowReader.
type ColumnStats ¶
type ColumnStats struct {
Name string
Type string
ValueType string
RowsCount uint64
Compression string
UncompressedSize uint64
CompressedSize uint64
MetadataOffset uint64
MetadataSize uint64
ValuesCount uint64
Cardinality uint64
ColumnIndex int64
Pages []PageStats
}
ColumnStats provides statistics about a column in a section.
type ColumnType ¶
type ColumnType int
ColumnType represents the kind of information stored in a Column.
const ( ColumnTypeInvalid ColumnType = iota // ColumnTypeInvalid is an invalid column. ColumnTypeStreamID // ColumnTypeStreamID is a column that contains stream IDs. ColumnTypeTimestamp // ColumnTypeTimestamp is a column that contains timestamps per log record. // ColumnTypeMetadata is a column containing a sequence of structured // metadata values per log record. There will be one ColumnTypeMetadata per // structured metadata key; the name of the structured metadata key is stored // as the column name. ColumnTypeMetadata ColumnTypeMessage // ColumnTypeMessage is a column that contains log messages. )
func ParseColumnType ¶
func ParseColumnType(text string) (ColumnType, error)
ParseColumnType parses a ColumnType from a string. The expected string format is the same as the return value of ColumnType.String.
func (ColumnType) String ¶
func (ct ColumnType) String() string
String returns the human-readable name of ct.
type ColumnarDataset ¶
ColumnarDataset is the exported type alias of the internal [columnar.Dataset].
func MakeColumnarDataset ¶
func MakeColumnarDataset(section *Section) (*ColumnarDataset, error)
MakeColumnarDataset is the exported entry point for sortmerge, the only caller outside this package; internal callers use makeDataset.
type DatasetSequence ¶
type DatasetSequence struct {
// contains filtered or unexported fields
}
func NewDatasetSequence ¶
func NewDatasetSequence(r *dataset.RowReader, bufferSize int) DatasetSequence
func (*DatasetSequence) Close ¶
func (seq *DatasetSequence) Close()
func (*DatasetSequence) Next ¶
func (seq *DatasetSequence) Next() bool
type LogMessageFilterRowPredicate ¶
type LogMessageFilterRowPredicate struct {
// Keep receives a nil line when the entry has no message value. The slice
// belongs to the reader and is valid only during the call, so Keep must not
// retain, modify, or append to it. Reads of one query can call the same Keep
// at the same time, so it must be safe for concurrent use.
Keep func(line []byte) bool
// KeepBatch is optional. It tests a batch of lines at once. This saves a Keep call for each
// row and a copy of each line that fails. The reader calls Keep when it cannot use KeepBatch,
// so Keep must always be set.
//
// Line i is data[offsets[i]:offsets[i+1]]. An entry with no message is an empty line, and
// Keep gets a nil line for it, so Keep must treat a nil and an empty line alike. Lines have
// no separator, offsets can start after 0, and data can hold bytes that belong to no line.
//
// KeepBatch appends to dstIndexes the index of every line that Keep keeps, in ascending
// order with no repeat, and returns the result. It must give the same result as Keep on
// every line.
//
// The arguments belong to the reader and are valid only during the call. KeepBatch must
// not retain data, offsets or dstIndexes, and must not modify data or offsets. Like Keep,
// it must be safe for concurrent use.
KeepBatch func(data []byte, offsets []int32, dstIndexes []int32) []int32
}
A LogMessageFilterRowPredicate is a RowPredicate that requires the log message of the entry to pass a Keep function.
type MetadataFilterRowPredicate ¶
A MetadataFilterRowPredicate is a RowPredicate that requires that metadata with the provided key pass a Keep function.
The key is provided to the keep function to allow the same function to be used for multiple filter predicates.
Uses of MetadataFilterRowPredicate are not eligible for page filtering and should only be used when a condition cannot be expressed by other basic predicates.
type MetadataMatcherRowPredicate ¶
type MetadataMatcherRowPredicate struct{ Key, Value string }
A MetadataMatcherRowPredicate is a RowPredicate that requires a metadata key named Key to exist with a value of Value.
type Metrics ¶
type Metrics struct {
// contains filtered or unexported fields
}
Metrics instruments the logs section.
func NewMetrics ¶
func NewMetrics() *Metrics
NewMetrics creates a new set of metrics for the logs section.
func (*Metrics) Register ¶
func (m *Metrics) Register(reg prometheus.Registerer) error
Register registers metrics to report to reg.
func (*Metrics) Unregister ¶
func (m *Metrics) Unregister(reg prometheus.Registerer)
Unregister unregisters metrics from the provided Registerer.
type NotRowPredicate ¶
type NotRowPredicate struct{ Inner RowPredicate }
A NotRowPredicate is a RowPredicate which requires its Inner predicate to be false.
type OrRowPredicate ¶
type OrRowPredicate struct{ Left, Right RowPredicate }
An OrRowPredicate is a RowPredicate which requires either its Left or Right predicate to be true.
type PageStats ¶
type PageStats struct {
UncompressedSize uint64
CompressedSize uint64
CRC32 uint32
RowsCount uint64
Encoding string
DataOffset uint64
DataSize uint64
ValuesCount uint64
}
PageStats provides statistics about a page in a column.
type Record ¶
type Record struct {
StreamID int64
Timestamp time.Time
Metadata labels.Labels
Line []byte
// SchemaKey is a pre-computed schema sort key. It is not encoded into the section;
// it only guides the in-memory sort during building.
SchemaKey string
// ShardBucket is the physical shard bucket for the record's stream. It is
// not encoded into LOG sections; it guides schema-layout sorting.
ShardBucket uint32
// StreamHash is labels.StableHash of the record's stream labels. It is not
// encoded into LOG sections; it guides schema-layout sorting.
StreamHash uint64
}
A Record is an individual log record within the logs section.
func (*Record) Reset ¶
func (r *Record) Reset()
Reset clears r so it can be reused for another row.
The line is truncated rather than dropped, so it keeps its capacity and a reused Record does not reallocate its buffer for every row.
func (*Record) UncompressedSize ¶
UncompressedSize returns the line bytes plus the structured metadata value bytes of r. It matches streams.Stream.UncompressedSize and the index statistics, so every producer reports the same quantity.
Metadata names are not counted because the section stores each name once per column, not once per record.
type RowPredicate ¶
type RowPredicate interface {
// contains filtered or unexported methods
}
RowPredicate is an expression used to filter rows in a data object.
type RowReader ¶
type RowReader struct {
// contains filtered or unexported fields
}
RowReader reads the set of logs from an [Object].
func NewRowReader ¶
NewRowReader creates a new RowReader that reads from the provided Section.
Call RowReader.Open before calling RowReader.Read.
func (*RowReader) Close ¶
Close closes the RowReader and releases any resources it holds. Closed RowReaders can be reused by calling RowReader.Reset.
func (*RowReader) MatchStreams ¶
MatchStreams provides a sequence of stream IDs for the logs reader to match. RowReader.Read will only return logs for the provided stream IDs.
MatchStreams may be called multiple times to match multiple sets of streams.
MatchStreams may only be called before reading begins or after a call to RowReader.Reset.
func (*RowReader) Open ¶
Open initializes RowReader resources.
Open must be called before RowReader.Read. Open is safe to call multiple times. Open is a no-op when the reader has no section.
func (*RowReader) Read ¶
Read reads up to the next len(s) records from the reader and stores them into s. It returns the number of records read and any error encountered. At the end of the logs section, Read returns 0, io.EOF.
func (*RowReader) Reset ¶
Reset resets the RowReader with a new Section to read from. Reset allows reusing a RowReader without allocating a new one.
Any set predicate, stream match and column projection are cleared when Reset is called.
Reset may be called with a nil section to clear the RowReader without needing a new one.
func (*RowReader) SetPredicates ¶
func (r *RowReader) SetPredicates(p []RowPredicate) error
SetPredicate sets the predicates to use for filtering logs. RowReader.Read will only return logs for which the predicate passes.
Predicates may only be set before reading begins or after a call to RowReader.Reset.
func (*RowReader) SetProjectedColumns ¶
func (r *RowReader) SetProjectedColumns(types []ColumnType, metadataNames []string) error
SetProjectedColumns restricts the Record to the given column types plus the metadata columns named in metadataNames. A ColumnTypeMetadata entry in types selects all metadata columns instead.
The field of a column that is not projected comes back as its zero value.
The reader reads the columns a stream match or a predicate needs, even when the caller does not list them. It does not decode them into the Record.
When the section holds no such column, a metadata or message predicate evaluates against the empty value.
When the section holds none of the projected columns, Open returns an error.
When SetProjectedColumns is not called, every recognized column is projected.
SetProjectedColumns may only be called before reading begins or after a call to RowReader.Reset.
type Section ¶
type Section struct {
// contains filtered or unexported fields
}
Section represents an opened logs section.
func Open ¶
Open opens a Section from an underlying dataobj.Section. Open returns an error if the section metadata could not be read or if the provided ctx is canceled.
func (*Section) Columns ¶
Columns returns the set of Columns in the section. The slice of returned sections must not be mutated.
Unrecognized columns (e.g., when running older code against newer sterams sections) are skipped.
func (*Section) LegacyLayout ¶
LegacyLayout returns the stream layout that the section recorded in its own sort info. Writers recorded it there before the streams section did. ok is false if the section records no such layout, or one that is not stable hash ordered with the default shard layout, because no writer produced another.
Readers use LegacyLayout only for objects whose streams section records no layout.
func (*Section) PrimarySortOrder ¶
func (s *Section) PrimarySortOrder() (ColumnType, SortDirection, error)
PrimarySortOrder returns the primary sort order information of the section as a tuple of ColumnType and SortDirection.
type SortDirection ¶
type SortDirection int
SortDirection represents sort direction of a column.
const ( SortDirectionUnspecified SortDirection = 0 // Sort direction is unspecified. SortDirectionAscending SortDirection = 1 // SortDirectionAscending represents ascending sort order (smallest values first). SortDirectionDescending SortDirection = 2 // SortDirectionDescending represents descending sort order (largest values first). )
type Stats ¶
type Stats struct {
UncompressedSize uint64
CompressedSize uint64
Columns []ColumnStats
}
Stats provides statistics about a streams section.
type TimeRangeRowPredicate ¶
type TimeRangeRowPredicate struct {
StartTime, EndTime time.Time
IncludeStart bool // Whether StartTime is inclusive.
IncludeEnd bool // Whether EndTime is inclusive.
}
A TimeRangeRowPredicate is a RowPredicate which requires the timestamp of the entry to be within the range of StartTime and EndTime.