Documentation
¶
Overview ¶
Package dataobj holds utilities for working with data objects.
Data objects are a container format for storing data intended to be retrieved from object storage. Each data object is composed of one or more "sections," each of which contains a specific type of data, such as logs stored in a columnar format.
Sections are further split into two "regions": the section data and the section metadata. Section metadata is intended to be a lightweight payload per section (usually protobuf) which aids in reading smaller portions of the data region.
Each section has a type indicating what kind of data was encoded in that section. A data object may have multiple sections of the same type.
The dataobj package provides a low-level Builder interface for composing sections into a dataobj, and a SectionReader interface for reading encoded sections.
Section implementations are stored in their own packages and provide higher-level utilities for writing and reading those sections. See github.com/grafana/loki/v4/pkg/dataobj/sections/logs for an example.
Creating a new section implementation ¶
To create a new section implementation:
Create a new package for your section.
Create a new SectionType for your section. Pick a combination of namespace and kind that avoids collisions with other sections that may be written to a file.
Create a "Builder" type which implmeents SectionBuilder. Your builder type should have methods for buffering data to be appended. Encode buffered data on a call to Flush.
Create higher-level reading APIs on top of SectionReader, decoding the data encoded from your builder.
Then, callers can create an instance of your Builder and store it in a data object using Builder.Append, and read it back using your higher-level APIs.
While not required, it is typical for section packages to additionally implement:
A package-level CheckSection function to check if a Section was built with your package.
A package-specific Section type that wraps Section to use in your reading APIs.
A function which returns the estimated size of something to be appended to your builder, so callers can flush the section once it gets big enough.
A method on your builder to report the estimated size of the section, both before and after compression.
## Encoding and decoding
There are no requirements on how to encode or decode a section, but there are recommendations:
Use protobuf for representing the metadata of the section.
Write section data so that it can be read in smaller units. For example, columnar data is split into pages, each of which can be read independently.
[SectionReader]s cannot access data outside of their section. Calling DataRange with an offset of 0 reads data from the beginning of that section's data region, not from the start of the dataobj.
Index ¶
- Variables
- func NewInstrumentedBucketReader(bucket objstore.BucketReader) objstore.BucketReader
- type Builder
- type MetadataCache
- type Metrics
- type Object
- type OpenOption
- type Section
- type SectionBuilder
- type SectionReader
- type SectionType
- type SectionWriter
- type Sections
- type TimeRange
- type WriteSectionOptions
Constants ¶
This section is empty.
Variables ¶
var ( // StatDatasetPagesTotal is the total number of pages contained in the // dataset sections opened by a task's [dataset.RowReader], summed across // every read column. This is the pre-pruning page count: it is recorded // before metadata predicates have a chance to eliminate pages from the read // set. StatDatasetPagesTotal = xcap.NewStatisticInt64("dataobj.dataset.pages.total", xcap.AggregationTypeSum) // StatDatasetPagesPruned counts pages that a task's [dataset.RowReader] // eliminated using page-level metadata statistics (min/max) before any page // data was downloaded. // // The counter is incremented per (page, leaf-predicate) rejection, so a // page that is rejected by multiple predicates contributes more than once. // This matches "work avoided by metadata pruning" rather than "distinct // pages skipped". StatDatasetPagesPruned = xcap.NewStatisticInt64("dataobj.dataset.pages.pruned", xcap.AggregationTypeSum) // StatObjectBytesDownloaded is the total number of bytes a task read from // the backing object store. Every request made through a reader from // [NewInstrumentedBucketReader] counts: metadata reads, page downloads and // ad-hoc reads. // // A body records its bytes when the caller closes it. Bytes served from the // metadata prefetch buffer are not counted again, so this is a true // "transferred from storage" figure. StatObjectBytesDownloaded = xcap.NewStatisticInt64("dataobj.object.bytes.downloaded", xcap.AggregationTypeSum) // StatObjectRequests* count every object-store request a task issued through // a reader from [NewInstrumentedBucketReader], successful or not. // // Reads served from memory, such as the prefetch buffer, issue no request and // are not counted. StatObjectRequestsAttributes = xcap.NewStatisticInt64("dataobj.object.requests.attributes", xcap.AggregationTypeSum) StatObjectRequestsGet = xcap.NewStatisticInt64("dataobj.object.requests.get", xcap.AggregationTypeSum) StatObjectRequestsGetRange = xcap.NewStatisticInt64("dataobj.object.requests.get_range", xcap.AggregationTypeSum) // StatObjectRequestFailures* count the requests of the matching // StatObjectRequests* stat that failed, in the call or while reading the // body it returned. A not-found error or a canceled request is not a // failure. A request counts as failed once, even if several reads of its // body fail. StatObjectRequestFailuresAttributes = xcap.NewStatisticInt64("dataobj.object.request_failures.attributes", xcap.AggregationTypeSum) StatObjectRequestFailuresGet = xcap.NewStatisticInt64("dataobj.object.request_failures.get", xcap.AggregationTypeSum) StatObjectRequestFailuresGetRange = xcap.NewStatisticInt64("dataobj.object.request_failures.get_range", xcap.AggregationTypeSum) // Dataset column statistics. StatDatasetPrimaryColumns = xcap.NewStatisticInt64("primary.columns", xcap.AggregationTypeSum, xcap.Local()) StatDatasetSecondaryColumns = xcap.NewStatisticInt64("secondary.columns", xcap.AggregationTypeSum, xcap.Local()) StatDatasetPrimaryColumnPages = xcap.NewStatisticInt64("primary.column.pages", xcap.AggregationTypeSum, xcap.Local()) StatDatasetSecondaryColumnPages = xcap.NewStatisticInt64("secondary.column.pages", xcap.AggregationTypeSum, xcap.Local()) // Dataset row statistics. StatDatasetMaxRows = xcap.NewStatisticInt64("rows.max", xcap.AggregationTypeSum) StatDatasetRowsAfterPruning = xcap.NewStatisticInt64("rows.after.pruning", xcap.AggregationTypeSum, xcap.Local()) StatDatasetPrimaryRowsRead = xcap.NewStatisticInt64("primary.rows.read", xcap.AggregationTypeSum) StatDatasetSecondaryRowsRead = xcap.NewStatisticInt64("secondary.rows.read", xcap.AggregationTypeSum) StatDatasetPrimaryRowBytes = xcap.NewStatisticInt64("primary.row.read.bytes", xcap.AggregationTypeSum) StatDatasetSecondaryRowBytes = xcap.NewStatisticInt64("secondary.row.read.bytes", xcap.AggregationTypeSum) // Dataset page scan statistics. StatDatasetPageDownloadTime = xcap.NewStatisticFloat64("pages.download.duration", xcap.AggregationTypeSum, xcap.Local()) // Dataset page download byte statistics. StatDatasetPrimaryPagesDownloaded = xcap.NewStatisticInt64("primary.pages.downloaded", xcap.AggregationTypeSum) StatDatasetSecondaryPagesDownloaded = xcap.NewStatisticInt64("secondary.pages.downloaded", xcap.AggregationTypeSum) StatDatasetPrimaryColumnBytes = xcap.NewStatisticInt64("primary.pages.compressed.bytes", xcap.AggregationTypeSum) StatDatasetSecondaryColumnBytes = xcap.NewStatisticInt64("secondary.pages.compressed.bytes", xcap.AggregationTypeSum) StatDatasetPrimaryColumnUncompressedBytes = xcap.NewStatisticInt64("primary.column.uncompressed.bytes", xcap.AggregationTypeSum, xcap.Local()) StatDatasetSecondaryColumnUncompressedBytes = xcap.NewStatisticInt64("secondary.column.uncompressed.bytes", xcap.AggregationTypeSum, xcap.Local()) // Dataset read operation statistics. StatDatasetReadCalls = xcap.NewStatisticInt64("dataset.read.calls", xcap.AggregationTypeSum, xcap.Local()) // StatStreamPagesTotal is the total number of pages in the stream-ID column. StatStreamPagesTotal = xcap.NewStatisticInt64("dataobj.stream.pages.total", xcap.AggregationTypeSum) // StatStreamRelevantPages is the number of stream-ID column pages whose // min/max range overlaps the queried stream IDs. StatStreamRelevantPages = xcap.NewStatisticInt64("dataobj.stream.pages.relevant", xcap.AggregationTypeSum) // StatStreamRelevantRows is the number of rows that passed the stream-ID // predicate before other query predicates narrowed the result further. StatStreamRelevantRows = xcap.NewStatisticInt64("dataobj.stream.rows.relevant", xcap.AggregationTypeSum) // StatStreamPageRuns is the number of contiguous runs of relevant // stream-ID pages. A single run means all relevant pages are back-to-back; // a count equal to relevant pages means every relevant page is isolated. StatStreamPageRuns = xcap.NewStatisticInt64("dataobj.stream.pages.runs", xcap.AggregationTypeSum) )
var ErrNotSingleTenant = fmt.Errorf("data object must hold exactly one tenant")
Functions ¶
func NewInstrumentedBucketReader ¶
func NewInstrumentedBucketReader(bucket objstore.BucketReader) objstore.BucketReader
NewInstrumentedBucketReader returns bucket wrapped to record its object-store requests. Every Attributes, Get and GetRange call records one request in the xcap.Region of the call's context. The reader records the request before it issues it, so a request that never returns still counts.
The reader also records one failure when a request fails with an unexpected error. The error can come from the call, or from reading the body that Get or GetRange returned. The reader does not count these errors as failures:
- an error that says the object does not exist, because the object store answered;
- an error the caller causes, which is a context error returned while the context of the call is done, such as a canceled request;
- the end of the body.
When the caller closes a body, the reader also records the bytes it delivered in StatObjectBytesDownloaded.
Other methods, such as Iter and Exists, pass through and record nothing.
Types ¶
type Builder ¶
type Builder struct {
// contains filtered or unexported fields
}
A Builder builds data objects from a set of incoming log data. Log data is appended to a builder by calling Builder.Append. Buffered log data is flushed manually by calling Builder.Flush.
Methods on Builder are not goroutine-safe; callers are responsible for synchronizing calls.
func NewBuilder ¶
A Builder accumulates data from a set of in-progress sections. A Builder can be flushed into a data object by calling Builder.Flush.
If provided, completed sections will be written to sectionScratchPath to reduce the peak memory usage of a builder to the peak memory usage of in-progress sections.
NewBuilder returns an error if sectionScratchPath specifies an invalid path on disk.
func (*Builder) Append ¶
func (b *Builder) Append(sec SectionBuilder) error
Append flushes a SectionBuilder, buffering its data and metadata into b. Append does not enforce ordering; sections may be flushed to the dataobj in any order.
Append returns an error if the section failed to flush.
After successfully calling Append, sec is reset and can be reused.
func (*Builder) Bytes ¶
Bytes returns the current number of bytes buffered in b for all appended sections.
func (*Builder) Flush ¶
Flush constructs a new Object from the accumulated sections. Allocated resources for the Object must be released by calling Close on the returned io.Closer. After closing, the returned Object must no longer be read.
Flush returns an error if the object could not be constructed. No object is handed to the caller in that case, so the sections buffered so far are released from the scratch store instead.
Builder.Reset is called by Flush, whether it succeeds or fails, to discard any pending data and allow new data to be appended.
type MetadataCache ¶
type MetadataCache interface {
// GetOrLoadMetadataRegion returns the metadata region for key. On a miss it calls load, stores the
// result, and returns it; concurrent calls for the same key share a single load. A load error is
// returned unchanged, or wrapped with %w, so a caller can still match a sentinel in it via errors.Is.
//
// The key must uniquely identify the object across whatever scope a single cache instance is shared
// over. A bucket-relative path is enough as long as one cache instance serves one bucket. A caller
// that shares one instance more broadly, across buckets or tenants whose paths can collide, must fold
// that scope into the key.
//
// The returned slice is read-only and may be shared between concurrent callers (a coalesced miss
// hands them the same backing array). Callers may retain it but must not mutate it; clone first to
// modify.
GetOrLoadMetadataRegion(ctx context.Context, key string, load func(ctx context.Context) ([]byte, error)) ([]byte, error)
// MaxItemBytes returns the largest region the underlying backend will store. It must always be
// positive: an implementation with no real backend limit still reports a safe, conservative value.
MaxItemBytes() int64
}
MetadataCache caches the metadata region of data objects, keyed by an opaque key. Data objects are immutable, so a cached entry never goes stale. Implementations must be safe for concurrent use.
type Metrics ¶
type Metrics struct {
// contains filtered or unexported fields
}
Metrics instruments encoded data objects.
func (*Metrics) Register ¶
func (m *Metrics) Register(reg prometheus.Registerer) error
Register registers metrics to report to reg.
func (*Metrics) Unregister ¶
func (m *Metrics) Unregister(reg prometheus.Registerer)
Unregister unregisters metrics from the provided Registerer.
type Object ¶
type Object struct {
// contains filtered or unexported fields
}
An Object is a representation of a data object.
func FromBucket ¶
func FromBucket(ctx context.Context, bucket objstore.BucketReader, path string, prefetchBytes int64, opts ...OpenOption) (*Object, error)
FromBucket opens an Object from the given storage bucket and path. FromBucket returns an error if the metadata of the Object cannot be read or if the provided ctx times out.
Every object-store request the Object makes records its statistics in the xcap.Region of the context passed to the call that makes it. See NewInstrumentedBucketReader.
func FromReaderAt ¶
FromReadSeeker opens an Object from the given ReaderAt. The size argument specifies the size of the data object in bytes. FromReaderAt returns an error if the metadata of the Object cannot be read.
func (*Object) Sections ¶
Sections returns the list of sections available in the Object. The slice of returned sections must not be mutated.
func (*Object) Tenants ¶
Tenants returns the list of tenant that have sections in the Object. The slice of returned tenants must not be mutated.
Data objects are expected to hold a single tenant. Prefer Object.Tenant, which returns an error otherwise.
type OpenOption ¶
type OpenOption func(*openOptions)
OpenOption customizes how an Object is opened.
func WithLogger ¶
func WithLogger(logger log.Logger) OpenOption
WithLogger reports otherwise-invisible degradations of the metadata cache: a cached entry that does not decode, or metadata that cannot be turned into a cacheable region at all. The default is to report nothing.
func WithMetadataCache ¶
func WithMetadataCache(cache MetadataCache) OpenOption
WithMetadataCache serves the object's metadata region through the cache instead of reading it from object storage on every open.
On the cached path, the prefetched window becomes exactly the cached metadata region, not prefetchBytes. A caller that also relies on prefetchBytes to warm overlapping section reads should account for that before enabling the cache.
type Section ¶
type Section struct {
Type SectionType // The type denoting the kind of data held in a section.
Reader SectionReader // The low-level reader for a Section.
// Tenant specifies the tenant that owns this section. Tenant is required
// for sections which wholly contain tenant-specific data.
Tenant string
}
A Section is a subset of an Object that holds a specific type of data. Use section packages for higher-level abstractions around sections.
type SectionBuilder ¶
type SectionBuilder interface {
// Type returns the SectionType representing the section being built.
// Implementations are responsible for guaranteeing that two no
// SectionBuilders return the same SectionType for different encodings.
//
// The returned Type is encoded directly into data objects. Implementations
// that change SectionType values should be careful to continue supporting
// old values for backwards compatibility.
Type() SectionType
// Flush encodes and flushes the section to w. Encodings that rely on byte
// offsets should be relative to the first byte of the section's data.
//
// Flush returns the number of bytes written to w, and any error encountered
// while encoding or flushing.
//
// After Flush is called, the SectionBuilder is reset to a fresh state and
// can be reused.
Flush(w SectionWriter) (n int64, err error)
// Reset resets the SectionBuilder to a fresh state.
Reset()
}
A SectionBuilder accumulates data for a single in-progress section.
Each section package provides an implementation of SectionBuilder that includes utilities to buffer data into that section. Callers should use Bytes or EstimatedSize to determine when enough data has been accumulated into a section.
type SectionReader ¶
type SectionReader interface {
// ExtensionData returns optional encoded information about the section
// stored at the file level, provided through the [SectionWriter]. Sections
// can use this for retrieving critical information that must be known
// without needing to read the metadata first.
//
// ExtensionData will be nil if no extension data is available.
ExtensionData() []byte
// DataRange opens a reader of length bytes from the data region of a
// section. The offset argument determines where in the data region reading
// should start.
//
// DataRange returns an error if the read fails or if offset+length goes
// beyond the readable data region. The returned reader is only valid as long
// as the provided ctx is not canceled.
DataRange(ctx context.Context, offset, length int64) (io.ReadCloser, error)
// MetadataRange opens a reader of length bytes from the metadata region of
// a section. The offset argument determines where in the metadata region
// reading should start.
//
// MetadataRange returns an error if the read fails or if offset+length goes
// beyond the readable metadata region. The returned reader is only valid as long
// as the provided ctx is not canceled.
MetadataRange(ctx context.Context, offset, length int64) (io.ReadCloser, error)
// DataSize returns the total size of the data region of a section. DataSize
// returns 0 for sections with no data region.
DataSize() int64
// MetadataSize returns the total size of the metadata region of a section.
// MetadataSize returns 0 for sections with no metadata region.
MetadataSize() int64
}
SectionReader is a low-level interface to read data ranges and metadata from a section.
Section packages provider higher-level abstractions around Section using this interface.
type SectionType ¶
type SectionType struct {
Namespace string // A namesapce for the section (e.g., "github.com/grafana/loki").
Kind string // The kind of section, scoped to the namespace (e.g., "logs").
// Version is an optional section-specified value denoting an encoding
// version of the section.
Version uint32
}
SectionType uniquely identifies a Section type.
func (SectionType) Equals ¶
func (ty SectionType) Equals(o SectionType) bool
Equals returns true if o has the same namespace and kind as ty. The Version field is not checked.
func (SectionType) String ¶
func (ty SectionType) String() string
type SectionWriter ¶
type SectionWriter interface {
// WriteSection writes a section to the underlying data stream, partitioned
// by section data and section metadata. It returns the sum of bytes written
// from both input slices (0 <= n <= len(data)+len(metadata)) and any error
// encountered that caused the write to stop early.
//
// The opts argument provides additional information about the section being
// written. If opts is nil, the section is written without any additional
// context.
//
// Implementations of WriteSection:
//
// - Must return an error if the write stops early.
// - Must not modify the slices passed to it, even temporarily.
// - Must not retain references to slices after WriteSection returns.
//
// The physical layout of data and metadata is not defined: they may be
// written non-contiguously, interleaved, or in any order.
WriteSection(opts *WriteSectionOptions, data, metadata []byte) (n int64, err error)
}
SectionWriter writes data object sections to an underlying stream, such as a data object.
type Sections ¶
type Sections []*Section
A Sections is a slice of Section.
type WriteSectionOptions ¶
type WriteSectionOptions struct {
// Tenant that owns the written data and metadata. Tenant must be set for
// sections that are wholly owned by a single tenant.
Tenant string
// ExtensionData is an optional field for section information to store at
// the file level. To minimize the cost of opening data objects, sections
// should only use this field for information that's required to start
// reading section metadata and to keep the payload as small as possible.
//
// ExtensionData does not impact the return value of n in
// [SectionWriter.WriteSection].
//
// Implementations of [SectionWriter] must not retain references to this
// slice after WriteSection returns.
ExtensionData []byte
}
WriteSectionOptions provides additional options when writing sections.
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
compaction
|
|
|
v2
Package compactionv2 plans K-way compaction from sections with inclusive ordering-key bounds.
|
Package compactionv2 plans K-way compaction from sections with inclusive ordering-key bounds. |
|
indexobj
Package indexobj provides tooling for creating index-oriented data objects.
|
Package indexobj provides tooling for creating index-oriented data objects. |
|
internal
|
|
|
arrowconv
Package arrowconv provides helper utilities for converting between Arrow and dataset values.
|
Package arrowconv provides helper utilities for converting between Arrow and dataset values. |
|
dataset
Package dataset contains utilities for working with datasets.
|
Package dataset contains utilities for working with datasets. |
|
result
Package result provides utilities for dealing with iterators that can fail during iteration.
|
Package result provides utilities for dealing with iterators that can fail during iteration. |
|
streamio
Package streamio defines interfaces shared by other packages for streaming binary data.
|
Package streamio defines interfaces shared by other packages for streaming binary data. |
|
util/bitmask
Package bitmask provides an API for creating and manipulating bitmasks of arbitrary length.
|
Package bitmask provides an API for creating and manipulating bitmasks of arbitrary length. |
|
util/bufpool
Package bufpool offers a pool of *bytes.Buffer objects that are placed into exponentially sized buckets.
|
Package bufpool offers a pool of *bytes.Buffer objects that are placed into exponentially sized buckets. |
|
util/protocodec
Package protocodec provides utilities for encoding and decoding protobuf messages into files.
|
Package protocodec provides utilities for encoding and decoding protobuf messages into files. |
|
util/rangeset
Package rangeset implements a half-open interval set [start, end).
|
Package rangeset implements a half-open interval set [start, end). |
|
util/sliceclear
Package sliceclear provides a way to clear and truncate the length of a slice.
|
Package sliceclear provides a way to clear and truncate the length of a slice. |
|
util/symbolizer
Package symbolizer provides a string interning mechanism to reduce memory usage by reusing identical strings.
|
Package symbolizer provides a string interning mechanism to reduce memory usage by reusing identical strings. |
|
Package logsobj provides tooling for creating logs-oriented data objects.
|
Package logsobj provides tooling for creating logs-oriented data objects. |
|
Package objtest builds a data-object storage directory for testing: a bucket holding logs objects, the index objects that describe them, and the tables of contents a metastore.Metastore resolves through.
|
Package objtest builds a data-object storage directory for testing: a bucket holding logs objects, the index objects that describe them, and the tables of contents a metastore.Metastore resolves through. |
|
Package sections locates the sections of a data object that belong to one tenant.
|
Package sections locates the sections of a data object that belong to one tenant. |
|
internal/columnar
Package columnar provides a base implementation for sections which store columnar data using github.com/grafana/loki/v4/pkg/dataobj/internal/dataset.
|
Package columnar provides a base implementation for sections which store columnar data using github.com/grafana/loki/v4/pkg/dataobj/internal/dataset. |
|
logs
Package logs defines types used for the data object logs section.
|
Package logs defines types used for the data object logs section. |
|
pointers
Package pointers defines types used for the data object pointers section.
|
Package pointers defines types used for the data object pointers section. |
|
postings
Package postings defines types for the data object postings section.
|
Package postings defines types for the data object postings section. |
|
stats
Package stats defines types for the data object stats section.
|
Package stats defines types for the data object stats section. |
|
streams
Package streams defines types used for the data object streams section.
|
Package streams defines types used for the data object streams section. |
|
Package sortmerge provides a k-way merge iterator over dataobj logs sections.
|
Package sortmerge provides a k-way merge iterator over dataobj logs sections. |