dataobj

package
v4.0.0-rc.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 3, 2026 License: AGPL-3.0 Imports: 24 Imported by: 0

README

DataObj Package Documentation

Overview

The dataobj package provides a container format for storing and retrieving structured data from object storage. It's designed specifically for Loki's log storage needs, enabling efficient columnar storage and retrieval of log data with support for multiple tenants, metadata indexing, and flexible querying.

Table of Contents


Architecture Overview

The dataobj package provides a hierarchical container format:

┌─────────────────────────────────────┐
│         Data Object File            │
├─────────────────────────────────────┤
│  Header                             │
│  - Magic: "DOBJ"                    │
│  - File Metadata Size (4 bytes)     │
│  - File Format Version              │
│  - File Metadata (Protobuf)         │
│    - Dictionary                     │
│    - Section Types                  │
│    - Section Layout Info            │
├─────────────────────────────────────┤
│  Section 1 Metadata                 │
│  Section 2 Metadata                 │
│  ...                                │
│  Section N Metadata                 │
├─────────────────────────────────────┤
│  Section 1 Data                     │
│  Section 2 Data                     │
│  ...                                │
│  Section N Data                     │
├─────────────────────────────────────┤
│  Tailer                             │
│  - Magic: "DOBJ"                    │
└─────────────────────────────────────┘

The file metadata sits in the header, at the start of the file. This lets a reader resolve every section's layout with one read from offset 0.

Core Components
  1. Builder - Constructs data objects by accumulating sections
  2. Encoder - Low-level encoding of sections and metadata
  3. Decoder - Low-level decoding of sections from object storage
  4. SectionReader - Interface for reading data/metadata from sections
  5. Section Implementations - Specific section types (logs, streams, pointers, etc.)

Key Concepts

Data Objects

A Data Object is a self-contained file stored in object storage containing:

  • One or more Sections, each holding a specific type of data
  • File Metadata describing all sections and their layout
  • Dictionary for efficient string storage (namespaces, kinds, tenant IDs)
Sections

Sections are the primary organizational unit within a data object:

  • Each section has a type (namespace, kind, version)
  • Sections contain both data and metadata regions
  • Multiple sections of the same type can exist in one object
  • Sections can be tenant-specific
Section Regions

Each section is split into two regions:

  1. Data Region - The actual encoded data (e.g., columnar log data)
  2. Metadata Region - Lightweight metadata to aid in reading the data region

This separation allows reading section metadata without loading the entire data payload.

Columnar Storage

Most section implementations use columnar storage via the internal/dataset package:

  • Data is organized into columns (e.g., timestamp, stream_id, message)
  • Columns are split into pages for efficient random access
  • Pages can be compressed (typically with zstd)
  • Supports predicates for filtering during reads

File Format Structure

File Layout
Offset  | Content
--------|----------------------------------------------------------
0       | Magic bytes: "DOBJ" (4 bytes)
4       | File Metadata Size (uint32, little-endian)
8       | Format Version (varint)
...     | File Metadata (protobuf-encoded)
...     | [Section Metadata Region - All sections concatenated]
...     | [Section Data Region - All sections concatenated]
-4      | Magic bytes: "DOBJ" (4 bytes)

File Metadata Structure (Protobuf) can be found in the pkg/dataobj/internal/metadata package.


Encoding Process

High-Level Encoding Flow
┌──────────────┐
│ Log Records  │
└──────┬───────┘
       │
       ▼
┌──────────────────┐
│ Section Builder  │  (e.g., logs.Builder)
│ - Buffer records │
│ - Create stripes │
│ - Merge stripes  │
└──────┬───────────┘
       │
       │ Flush
       ▼
┌──────────────────┐
│ Columnar Encoder │
│ - Encode columns │
│ - Compress pages │
└──────┬───────────┘
       │
       │ WriteSection
       ▼
┌──────────────────┐
│ dataobj.Builder  │
│ - Append section │
│ - Build metadata │
└──────┬───────────┘
       │
       │ Flush
       ▼
┌──────────────────┐
│ Snapshot         │
│ - In-memory/disk │
│ - Ready to upload│
└──────┬───────────┘
       │
       ▼
┌──────────────────┐
│ Object Storage   │
└──────────────────┘
Encoding
1. Section builders

A section builder:

  1. Buffers records in memory up to BufferSize
  2. When buffer is full, sorts records and creates a stripe
  3. Stripes are intermediate compressed tables
  4. Multiple stripes are merged when flushing the section
2. Encoding to Columnar Format

When flushing, the builder:

  1. Converts buffered records into columnar format
  2. Creates separate columns for each field (stream_id, timestamp, metadata, message)
  3. Splits columns into pages
  4. Compresses each page independently (zstd)
3. Writing to Data Object

The data object builder:

  1. Collects data and metadata from each section
  2. Builds file metadata with section layout information

Column metadata includes:

  • Page descriptors (offset, size, row count, compression)
  • Column statistics (min/max values)
  • Encoding information
4. Compression Strategy

The encoding uses a multi-level compression strategy:

  • Stripes (intermediate): zstd with SpeedFastest for quick buffering
  • Sections (final): zstd with SpeedDefault for better compression
  • Ordered Appends: When logs arrive pre-sorted, skips stripe creation for better performance

Decoding Process

High-Level Decoding Flow
┌──────────────────┐
│ Object Storage   │
└──────┬───────────┘
       │
       │ Open
       ▼
┌──────────────────┐
│ dataobj.Object   │
│ - Read metadata  │
│ - Parse sections │
└──────┬───────────┘
       │
       │ Open Section
       ▼
┌──────────────────┐
│ Section (e.g.    │
│  logs.Section)   │
│ - Decode metadata│
│ - List columns   │
└──────┬───────────┘
       │
       │ NewRowReader
       ▼
┌──────────────────┐
│ logs.RowReader   │
│ - Read pages     │
│ - Apply filters  │
│ - Decompress     │
└──────┬───────────┘
       │
       │ Read batches
       ▼
┌──────────────────┐
│ Records          │
└──────────────────┘
Detailed Decoding Steps
1. Opening a Data Object

Opening process:

  1. Reads first 16KB of file in one request (optimistic read for metadata)
  2. Parses the header, at the start of the file, to find the metadata size
  3. Reads and decodes file metadata
  4. Constructs Section objects with SectionReaders
2. Decoding File Metadata

The decoder:

  1. Validates magic bytes ("DOBJ")
  2. Reads metadata size from the header, right after the magic bytes
  3. Reads the format version, then the protobuf metadata that follows it
  4. Uses the header size to compute where the section metadata region starts
3. Opening a Section

Section opening:

  1. Validates section type and version
  2. Reads extension data for quick access to section metadata
  3. Decodes section metadata (columnar structure)
4. Reading Data

Reading process:

  1. Resolves the projected columns of the section
  2. Maps predicates to internal dataset predicates
  3. Reads pages in batches (with prefetching)
  4. Applies predicates at page and row level
  5. Decodes the rows into records

Reader optimizations:

  • Range coalescing: Adjacent pages are read in a single request
  • Parallel reads: Multiple pages can be fetched concurrently
  • Prefetching: Pages are fetched ahead of time while processing current batch
  • Predicate pushdown: Filters applied at page level using statistics
5. Value Encoding Types

Values in pages can use different encodings:

  • Plain: Raw values stored directly
  • Delta: Store deltas between consecutive values (efficient for sorted data)
  • Bitmap: For repetition levels (null/non-null indicators)

Section Types

The package provides several built-in section types:

1. Logs Section

Namespace: github.com/grafana/loki Kind: logs Location: pkg/dataobj/sections/logs/

Stores log records in columnar format.

Columns:

  • stream_id (int64): Stream identifier
  • timestamp (int64): Nanosecond timestamp
  • metadata (binary): Structured metadata key-value pairs, one column per key
  • message (binary): Log line content
2. Streams Section

Namespace: github.com/grafana/loki Kind: streams Location: pkg/dataobj/sections/streams/

Stores stream metadata and statistics in logs objects.

Columns:

  • stream_id (int64): Unique stream identifier
  • min_timestamp (int64): Earliest timestamp in stream
  • max_timestamp (int64): Latest timestamp in stream
  • labels (binary): Label key-value pairs for the stream, one column per key
  • rows (uint64): Number of log records
  • uncompressed_size (uint64): Uncompressed data size
3. Pointers Section

Namespace: github.com/grafana/loki Kind: pointers Location: pkg/dataobj/sections/pointers/

Used in older index objects to point to other objects that contain logs, via stream or predicate lookup. The index builder no longer writes this section, and the compactor skips it.

Columns:

  • path (binary): Path to data object in object storage
  • section (int64): The section number within the referenced data object
  • pointer_kind (int64): The type of pointer entry. Determines which fields are set in the section. Either stream or column.

// Fields present for a stream pointer type

  • stream_id (int64): Stream identifier in this index object
  • stream_id_ref (int64): Stream identifier in the target data object's stream section
  • min_timestamp (int64): Min timestamp in the object for this stream
  • max_timestamp (int64): Max timestamp in the object for this stream
  • row_count (int64): Total rows for this stream in the target object
  • uncompressed_size (int64): Total bytes for this stream in the target object

// Fields for a column pointer type

  • column_name (binary): The name of the column in the referenced object
  • column_index (int64): The index number of the column in the referenced object
  • values_bloom_filter (binary): A bloom filter for unique values seen in the referenced column
4. Index Pointers Section

Namespace: github.com/grafana/loki Kind: indexpointers Location: pkg/dataobj/sections/indexpointers/

Used in Table of Contents (toc) objects to point to index objects, via a time range lookup.

Columns:

  • path (binary): Path to the index file
  • min_time (int64): Minimum time covered by the referenced index object
  • max_time (int64): Maximum time covered by the referenced index object

Operational Components

Builder

Location: pkg/dataobj/builder/

The builder (the dataobj-builder target) reads log data from Kafka and builds data objects.

Key Features:

  • Reads from Kafka partitions
  • Accumulates logs into data objects, one per tenant and 12-hour window
  • Flushes based on size or idle timeout
  • Commits offsets after successful upload
  • Builds an index object for each uploaded data object and records it in the Table of Contents.
Metastore

Location: pkg/dataobj/metastore/

Manages an index of data objects and their contents for efficient querying.

The metastore serves queries by the following:

  1. Fetch and scan the querying tenant's Table of Contents (toc) files for every 12-hour window in the query time range to resolve index objects. Each toc file holds a single tenant and is stored at tocs/<window>/<tenant>/toc.toc.
  2. Fetches resolved index objects and reads their postings sections to resolve the log object sections that match the query.
Index

Location: pkg/dataobj/index/

Creates index objects that contain indexes over data objects containing the logs. Used by the builder and by compaction.

Explorer Service

Location: pkg/dataobj/explorer/

HTTP service for inspecting data objects.


Testing and Tools

Inspect Tool
# Inspect data object structure
go run ./pkg/dataobj/tools/inspect.go -path objects/ab/cd123...

# Show statistics
go run ./pkg/dataobj/tools/stats.go -path objects/ab/cd123...
Explorer Service

The explorer provides a web UI for browsing data objects:

# Start explorer
./loki -target=dataobj-explorer

# Access UI
curl http://localhost:3100/dataobj/api/v1/list

Documentation

Overview

Package dataobj holds utilities for working with data objects.

Data objects are a container format for storing data intended to be retrieved from object storage. Each data object is composed of one or more "sections," each of which contains a specific type of data, such as logs stored in a columnar format.

Sections are further split into two "regions": the section data and the section metadata. Section metadata is intended to be a lightweight payload per section (usually protobuf) which aids in reading smaller portions of the data region.

Each section has a type indicating what kind of data was encoded in that section. A data object may have multiple sections of the same type.

The dataobj package provides a low-level Builder interface for composing sections into a dataobj, and a SectionReader interface for reading encoded sections.

Section implementations are stored in their own packages and provide higher-level utilities for writing and reading those sections. See github.com/grafana/loki/v4/pkg/dataobj/sections/logs for an example.

Creating a new section implementation

To create a new section implementation:

  1. Create a new package for your section.

  2. Create a new SectionType for your section. Pick a combination of namespace and kind that avoids collisions with other sections that may be written to a file.

  3. Create a "Builder" type which implmeents SectionBuilder. Your builder type should have methods for buffering data to be appended. Encode buffered data on a call to Flush.

  4. Create higher-level reading APIs on top of SectionReader, decoding the data encoded from your builder.

Then, callers can create an instance of your Builder and store it in a data object using Builder.Append, and read it back using your higher-level APIs.

While not required, it is typical for section packages to additionally implement:

  • A package-level CheckSection function to check if a Section was built with your package.

  • A package-specific Section type that wraps Section to use in your reading APIs.

  • A function which returns the estimated size of something to be appended to your builder, so callers can flush the section once it gets big enough.

  • A method on your builder to report the estimated size of the section, both before and after compression.

## Encoding and decoding

There are no requirements on how to encode or decode a section, but there are recommendations:

  • Use protobuf for representing the metadata of the section.

  • Write section data so that it can be read in smaller units. For example, columnar data is split into pages, each of which can be read independently.

[SectionReader]s cannot access data outside of their section. Calling DataRange with an offset of 0 reads data from the beginning of that section's data region, not from the start of the dataobj.

Index

Constants

This section is empty.

Variables

View Source
var (
	// StatDatasetPagesTotal is the total number of pages contained in the
	// dataset sections opened by a task's [dataset.RowReader], summed across
	// every read column. This is the pre-pruning page count: it is recorded
	// before metadata predicates have a chance to eliminate pages from the read
	// set.
	StatDatasetPagesTotal = xcap.NewStatisticInt64("dataobj.dataset.pages.total", xcap.AggregationTypeSum)

	// StatDatasetPagesPruned counts pages that a task's [dataset.RowReader]
	// eliminated using page-level metadata statistics (min/max) before any page
	// data was downloaded.
	//
	// The counter is incremented per (page, leaf-predicate) rejection, so a
	// page that is rejected by multiple predicates contributes more than once.
	// This matches "work avoided by metadata pruning" rather than "distinct
	// pages skipped".
	StatDatasetPagesPruned = xcap.NewStatisticInt64("dataobj.dataset.pages.pruned", xcap.AggregationTypeSum)

	// StatObjectBytesDownloaded is the total number of bytes a task read from
	// the backing object store. Every request made through a reader from
	// [NewInstrumentedBucketReader] counts: metadata reads, page downloads and
	// ad-hoc reads.
	//
	// A body records its bytes when the caller closes it. Bytes served from the
	// metadata prefetch buffer are not counted again, so this is a true
	// "transferred from storage" figure.
	StatObjectBytesDownloaded = xcap.NewStatisticInt64("dataobj.object.bytes.downloaded", xcap.AggregationTypeSum)

	// StatObjectRequests* count every object-store request a task issued through
	// a reader from [NewInstrumentedBucketReader], successful or not.
	//
	// Reads served from memory, such as the prefetch buffer, issue no request and
	// are not counted.
	StatObjectRequestsAttributes = xcap.NewStatisticInt64("dataobj.object.requests.attributes", xcap.AggregationTypeSum)
	StatObjectRequestsGet        = xcap.NewStatisticInt64("dataobj.object.requests.get", xcap.AggregationTypeSum)
	StatObjectRequestsGetRange   = xcap.NewStatisticInt64("dataobj.object.requests.get_range", xcap.AggregationTypeSum)

	// StatObjectRequestFailures* count the requests of the matching
	// StatObjectRequests* stat that failed, in the call or while reading the
	// body it returned. A not-found error or a canceled request is not a
	// failure. A request counts as failed once, even if several reads of its
	// body fail.
	StatObjectRequestFailuresAttributes = xcap.NewStatisticInt64("dataobj.object.request_failures.attributes", xcap.AggregationTypeSum)
	StatObjectRequestFailuresGet        = xcap.NewStatisticInt64("dataobj.object.request_failures.get", xcap.AggregationTypeSum)
	StatObjectRequestFailuresGetRange   = xcap.NewStatisticInt64("dataobj.object.request_failures.get_range", xcap.AggregationTypeSum)

	// Dataset column statistics.
	StatDatasetPrimaryColumns       = xcap.NewStatisticInt64("primary.columns", xcap.AggregationTypeSum, xcap.Local())
	StatDatasetSecondaryColumns     = xcap.NewStatisticInt64("secondary.columns", xcap.AggregationTypeSum, xcap.Local())
	StatDatasetPrimaryColumnPages   = xcap.NewStatisticInt64("primary.column.pages", xcap.AggregationTypeSum, xcap.Local())
	StatDatasetSecondaryColumnPages = xcap.NewStatisticInt64("secondary.column.pages", xcap.AggregationTypeSum, xcap.Local())

	// Dataset row statistics.
	StatDatasetMaxRows           = xcap.NewStatisticInt64("rows.max", xcap.AggregationTypeSum)
	StatDatasetRowsAfterPruning  = xcap.NewStatisticInt64("rows.after.pruning", xcap.AggregationTypeSum, xcap.Local())
	StatDatasetPrimaryRowsRead   = xcap.NewStatisticInt64("primary.rows.read", xcap.AggregationTypeSum)
	StatDatasetSecondaryRowsRead = xcap.NewStatisticInt64("secondary.rows.read", xcap.AggregationTypeSum)
	StatDatasetPrimaryRowBytes   = xcap.NewStatisticInt64("primary.row.read.bytes", xcap.AggregationTypeSum)
	StatDatasetSecondaryRowBytes = xcap.NewStatisticInt64("secondary.row.read.bytes", xcap.AggregationTypeSum)

	// Dataset page scan statistics.
	StatDatasetPageDownloadTime = xcap.NewStatisticFloat64("pages.download.duration", xcap.AggregationTypeSum, xcap.Local())

	// Dataset page download byte statistics.
	StatDatasetPrimaryPagesDownloaded           = xcap.NewStatisticInt64("primary.pages.downloaded", xcap.AggregationTypeSum)
	StatDatasetSecondaryPagesDownloaded         = xcap.NewStatisticInt64("secondary.pages.downloaded", xcap.AggregationTypeSum)
	StatDatasetPrimaryColumnBytes               = xcap.NewStatisticInt64("primary.pages.compressed.bytes", xcap.AggregationTypeSum)
	StatDatasetSecondaryColumnBytes             = xcap.NewStatisticInt64("secondary.pages.compressed.bytes", xcap.AggregationTypeSum)
	StatDatasetPrimaryColumnUncompressedBytes   = xcap.NewStatisticInt64("primary.column.uncompressed.bytes", xcap.AggregationTypeSum, xcap.Local())
	StatDatasetSecondaryColumnUncompressedBytes = xcap.NewStatisticInt64("secondary.column.uncompressed.bytes", xcap.AggregationTypeSum, xcap.Local())

	// Dataset read operation statistics.
	StatDatasetReadCalls = xcap.NewStatisticInt64("dataset.read.calls", xcap.AggregationTypeSum, xcap.Local())

	// StatStreamPagesTotal is the total number of pages in the stream-ID column.
	StatStreamPagesTotal = xcap.NewStatisticInt64("dataobj.stream.pages.total", xcap.AggregationTypeSum)

	// StatStreamRelevantPages is the number of stream-ID column pages whose
	// min/max range overlaps the queried stream IDs.
	StatStreamRelevantPages = xcap.NewStatisticInt64("dataobj.stream.pages.relevant", xcap.AggregationTypeSum)

	// StatStreamRelevantRows is the number of rows that passed the stream-ID
	// predicate before other query predicates narrowed the result further.
	StatStreamRelevantRows = xcap.NewStatisticInt64("dataobj.stream.rows.relevant", xcap.AggregationTypeSum)

	// StatStreamPageRuns is the number of contiguous runs of relevant
	// stream-ID pages. A single run means all relevant pages are back-to-back;
	// a count equal to relevant pages means every relevant page is isolated.
	StatStreamPageRuns = xcap.NewStatisticInt64("dataobj.stream.pages.runs", xcap.AggregationTypeSum)
)
View Source
var ErrNotSingleTenant = fmt.Errorf("data object must hold exactly one tenant")

Functions

func NewInstrumentedBucketReader

func NewInstrumentedBucketReader(bucket objstore.BucketReader) objstore.BucketReader

NewInstrumentedBucketReader returns bucket wrapped to record its object-store requests. Every Attributes, Get and GetRange call records one request in the xcap.Region of the call's context. The reader records the request before it issues it, so a request that never returns still counts.

The reader also records one failure when a request fails with an unexpected error. The error can come from the call, or from reading the body that Get or GetRange returned. The reader does not count these errors as failures:

  • an error that says the object does not exist, because the object store answered;
  • an error the caller causes, which is a context error returned while the context of the call is done, such as a canceled request;
  • the end of the body.

When the caller closes a body, the reader also records the bytes it delivered in StatObjectBytesDownloaded.

Other methods, such as Iter and Exists, pass through and record nothing.

Types

type Builder

type Builder struct {
	// contains filtered or unexported fields
}

A Builder builds data objects from a set of incoming log data. Log data is appended to a builder by calling Builder.Append. Buffered log data is flushed manually by calling Builder.Flush.

Methods on Builder are not goroutine-safe; callers are responsible for synchronizing calls.

func NewBuilder

func NewBuilder(scratchStore scratch.Store) *Builder

A Builder accumulates data from a set of in-progress sections. A Builder can be flushed into a data object by calling Builder.Flush.

If provided, completed sections will be written to sectionScratchPath to reduce the peak memory usage of a builder to the peak memory usage of in-progress sections.

NewBuilder returns an error if sectionScratchPath specifies an invalid path on disk.

func (*Builder) Append

func (b *Builder) Append(sec SectionBuilder) error

Append flushes a SectionBuilder, buffering its data and metadata into b. Append does not enforce ordering; sections may be flushed to the dataobj in any order.

Append returns an error if the section failed to flush.

After successfully calling Append, sec is reset and can be reused.

func (*Builder) Bytes

func (b *Builder) Bytes() int

Bytes returns the current number of bytes buffered in b for all appended sections.

func (*Builder) Flush

func (b *Builder) Flush() (*Object, io.Closer, error)

Flush constructs a new Object from the accumulated sections. Allocated resources for the Object must be released by calling Close on the returned io.Closer. After closing, the returned Object must no longer be read.

Flush returns an error if the object could not be constructed. No object is handed to the caller in that case, so the sections buffered so far are released from the scratch store instead.

Builder.Reset is called by Flush, whether it succeeds or fails, to discard any pending data and allow new data to be appended.

func (*Builder) Reset

func (b *Builder) Reset()

Reset discards pending data and resets the builder to an empty state, releasing any sections it still holds from the scratch store.

type MetadataCache

type MetadataCache interface {
	// GetOrLoadMetadataRegion returns the metadata region for key. On a miss it calls load, stores the
	// result, and returns it; concurrent calls for the same key share a single load. A load error is
	// returned unchanged, or wrapped with %w, so a caller can still match a sentinel in it via errors.Is.
	//
	// The key must uniquely identify the object across whatever scope a single cache instance is shared
	// over. A bucket-relative path is enough as long as one cache instance serves one bucket. A caller
	// that shares one instance more broadly, across buckets or tenants whose paths can collide, must fold
	// that scope into the key.
	//
	// The returned slice is read-only and may be shared between concurrent callers (a coalesced miss
	// hands them the same backing array). Callers may retain it but must not mutate it; clone first to
	// modify.
	GetOrLoadMetadataRegion(ctx context.Context, key string, load func(ctx context.Context) ([]byte, error)) ([]byte, error)

	// MaxItemBytes returns the largest region the underlying backend will store. It must always be
	// positive: an implementation with no real backend limit still reports a safe, conservative value.
	MaxItemBytes() int64
}

MetadataCache caches the metadata region of data objects, keyed by an opaque key. Data objects are immutable, so a cached entry never goes stale. Implementations must be safe for concurrent use.

type Metrics

type Metrics struct {
	// contains filtered or unexported fields
}

Metrics instruments encoded data objects.

func NewMetrics

func NewMetrics() *Metrics

NewMetrics creates a new set of metrics for encoding.

func (*Metrics) Observe

func (m *Metrics) Observe(obj *Object) error

Observe updates metrics with statistics about the given Object.

func (*Metrics) Register

func (m *Metrics) Register(reg prometheus.Registerer) error

Register registers metrics to report to reg.

func (*Metrics) Unregister

func (m *Metrics) Unregister(reg prometheus.Registerer)

Unregister unregisters metrics from the provided Registerer.

type Object

type Object struct {
	// contains filtered or unexported fields
}

An Object is a representation of a data object.

func FromBucket

func FromBucket(ctx context.Context, bucket objstore.BucketReader, path string, prefetchBytes int64, opts ...OpenOption) (*Object, error)

FromBucket opens an Object from the given storage bucket and path. FromBucket returns an error if the metadata of the Object cannot be read or if the provided ctx times out.

Every object-store request the Object makes records its statistics in the xcap.Region of the context passed to the call that makes it. See NewInstrumentedBucketReader.

func FromReaderAt

func FromReaderAt(r io.ReaderAt, size int64) (*Object, error)

FromReadSeeker opens an Object from the given ReaderAt. The size argument specifies the size of the data object in bytes. FromReaderAt returns an error if the metadata of the Object cannot be read.

func (*Object) Reader

func (o *Object) Reader(ctx context.Context) (io.ReadCloser, error)

Reader returns a reader for the entire raw data object.

func (*Object) Sections

func (o *Object) Sections() Sections

Sections returns the list of sections available in the Object. The slice of returned sections must not be mutated.

func (*Object) Size

func (o *Object) Size() int64

Size returns the size of the data object in bytes.

func (*Object) Tenant

func (o *Object) Tenant() (string, error)

Tenant returns the tenant of the object.

func (*Object) Tenants

func (o *Object) Tenants() []string

Tenants returns the list of tenant that have sections in the Object. The slice of returned tenants must not be mutated.

Data objects are expected to hold a single tenant. Prefer Object.Tenant, which returns an error otherwise.

type OpenOption

type OpenOption func(*openOptions)

OpenOption customizes how an Object is opened.

func WithLogger

func WithLogger(logger log.Logger) OpenOption

WithLogger reports otherwise-invisible degradations of the metadata cache: a cached entry that does not decode, or metadata that cannot be turned into a cacheable region at all. The default is to report nothing.

func WithMetadataCache

func WithMetadataCache(cache MetadataCache) OpenOption

WithMetadataCache serves the object's metadata region through the cache instead of reading it from object storage on every open.

On the cached path, the prefetched window becomes exactly the cached metadata region, not prefetchBytes. A caller that also relies on prefetchBytes to warm overlapping section reads should account for that before enabling the cache.

type Section

type Section struct {
	Type   SectionType   // The type denoting the kind of data held in a section.
	Reader SectionReader // The low-level reader for a Section.

	// Tenant specifies the tenant that owns this section. Tenant is required
	// for sections which wholly contain tenant-specific data.
	Tenant string
}

A Section is a subset of an Object that holds a specific type of data. Use section packages for higher-level abstractions around sections.

type SectionBuilder

type SectionBuilder interface {
	// Type returns the SectionType representing the section being built.
	// Implementations are responsible for guaranteeing that two no
	// SectionBuilders return the same SectionType for different encodings.
	//
	// The returned Type is encoded directly into data objects. Implementations
	// that change SectionType values should be careful to continue supporting
	// old values for backwards compatibility.
	Type() SectionType

	// Flush encodes and flushes the section to w. Encodings that rely on byte
	// offsets should be relative to the first byte of the section's data.
	//
	// Flush returns the number of bytes written to w, and any error encountered
	// while encoding or flushing.
	//
	// After Flush is called, the SectionBuilder is reset to a fresh state and
	// can be reused.
	Flush(w SectionWriter) (n int64, err error)

	// Reset resets the SectionBuilder to a fresh state.
	Reset()
}

A SectionBuilder accumulates data for a single in-progress section.

Each section package provides an implementation of SectionBuilder that includes utilities to buffer data into that section. Callers should use Bytes or EstimatedSize to determine when enough data has been accumulated into a section.

type SectionReader

type SectionReader interface {
	// ExtensionData returns optional encoded information about the section
	// stored at the file level, provided through the [SectionWriter]. Sections
	// can use this for retrieving critical information that must be known
	// without needing to read the metadata first.
	//
	// ExtensionData will be nil if no extension data is available.
	ExtensionData() []byte

	// DataRange opens a reader of length bytes from the data region of a
	// section. The offset argument determines where in the data region reading
	// should start.
	//
	// DataRange returns an error if the read fails or if offset+length goes
	// beyond the readable data region. The returned reader is only valid as long
	// as the provided ctx is not canceled.
	DataRange(ctx context.Context, offset, length int64) (io.ReadCloser, error)

	// MetadataRange opens a reader of length bytes from the metadata region of
	// a section. The offset argument determines where in the metadata region
	// reading should start.
	//
	// MetadataRange returns an error if the read fails or if offset+length goes
	// beyond the readable metadata region. The returned reader is only valid as long
	// as the provided ctx is not canceled.
	MetadataRange(ctx context.Context, offset, length int64) (io.ReadCloser, error)

	// DataSize returns the total size of the data region of a section. DataSize
	// returns 0 for sections with no data region.
	DataSize() int64

	// MetadataSize returns the total size of the metadata region of a section.
	// MetadataSize returns 0 for sections with no metadata region.
	MetadataSize() int64
}

SectionReader is a low-level interface to read data ranges and metadata from a section.

Section packages provider higher-level abstractions around Section using this interface.

type SectionType

type SectionType struct {
	Namespace string // A namesapce for the section (e.g., "github.com/grafana/loki").
	Kind      string // The kind of section, scoped to the namespace (e.g., "logs").

	// Version is an optional section-specified value denoting an encoding
	// version of the section.
	Version uint32
}

SectionType uniquely identifies a Section type.

func (SectionType) Equals

func (ty SectionType) Equals(o SectionType) bool

Equals returns true if o has the same namespace and kind as ty. The Version field is not checked.

func (SectionType) String

func (ty SectionType) String() string

type SectionWriter

type SectionWriter interface {
	// WriteSection writes a section to the underlying data stream, partitioned
	// by section data and section metadata. It returns the sum of bytes written
	// from both input slices (0 <= n <= len(data)+len(metadata)) and any error
	// encountered that caused the write to stop early.
	//
	// The opts argument provides additional information about the section being
	// written. If opts is nil, the section is written without any additional
	// context.
	//
	// Implementations of WriteSection:
	//
	//   - Must return an error if the write stops early.
	//   - Must not modify the slices passed to it, even temporarily.
	//   - Must not retain references to slices after WriteSection returns.
	//
	// The physical layout of data and metadata is not defined: they may be
	// written non-contiguously, interleaved, or in any order.
	WriteSection(opts *WriteSectionOptions, data, metadata []byte) (n int64, err error)
}

SectionWriter writes data object sections to an underlying stream, such as a data object.

type Sections

type Sections []*Section

A Sections is a slice of Section.

func (Sections) Count

func (s Sections) Count(predicate func(*Section) bool) int

Count returns the number of sections that pass some predicate.

func (Sections) Filter

func (s Sections) Filter(predicate func(*Section) bool) iter.Seq2[int, *Section]

Filter returns an iterator over sections that pass some predicate. The index field is the number of the section that passed the predicate.

type TimeRange

type TimeRange struct {
	Tenant  string
	MinTime time.Time
	MaxTime time.Time
}

TimeRange is the time range of a tenant's data in an object.

type WriteSectionOptions

type WriteSectionOptions struct {
	// Tenant that owns the written data and metadata. Tenant must be set for
	// sections that are wholly owned by a single tenant.
	Tenant string

	// ExtensionData is an optional field for section information to store at
	// the file level. To minimize the cost of opening data objects, sections
	// should only use this field for information that's required to start
	// reading section metadata and to keep the payload as small as possible.
	//
	// ExtensionData does not impact the return value of n in
	// [SectionWriter.WriteSection].
	//
	// Implementations of [SectionWriter] must not retain references to this
	// slice after WriteSection returns.
	ExtensionData []byte
}

WriteSectionOptions provides additional options when writing sections.

Directories

Path Synopsis
compaction
v2
Package compactionv2 plans K-way compaction from sections with inclusive ordering-key bounds.
Package compactionv2 plans K-way compaction from sections with inclusive ordering-key bounds.
indexobj
Package indexobj provides tooling for creating index-oriented data objects.
Package indexobj provides tooling for creating index-oriented data objects.
internal
arrowconv
Package arrowconv provides helper utilities for converting between Arrow and dataset values.
Package arrowconv provides helper utilities for converting between Arrow and dataset values.
dataset
Package dataset contains utilities for working with datasets.
Package dataset contains utilities for working with datasets.
result
Package result provides utilities for dealing with iterators that can fail during iteration.
Package result provides utilities for dealing with iterators that can fail during iteration.
streamio
Package streamio defines interfaces shared by other packages for streaming binary data.
Package streamio defines interfaces shared by other packages for streaming binary data.
util/bitmask
Package bitmask provides an API for creating and manipulating bitmasks of arbitrary length.
Package bitmask provides an API for creating and manipulating bitmasks of arbitrary length.
util/bufpool
Package bufpool offers a pool of *bytes.Buffer objects that are placed into exponentially sized buckets.
Package bufpool offers a pool of *bytes.Buffer objects that are placed into exponentially sized buckets.
util/protocodec
Package protocodec provides utilities for encoding and decoding protobuf messages into files.
Package protocodec provides utilities for encoding and decoding protobuf messages into files.
util/rangeset
Package rangeset implements a half-open interval set [start, end).
Package rangeset implements a half-open interval set [start, end).
util/sliceclear
Package sliceclear provides a way to clear and truncate the length of a slice.
Package sliceclear provides a way to clear and truncate the length of a slice.
util/symbolizer
Package symbolizer provides a string interning mechanism to reduce memory usage by reusing identical strings.
Package symbolizer provides a string interning mechanism to reduce memory usage by reusing identical strings.
Package logsobj provides tooling for creating logs-oriented data objects.
Package logsobj provides tooling for creating logs-oriented data objects.
Package objtest builds a data-object storage directory for testing: a bucket holding logs objects, the index objects that describe them, and the tables of contents a metastore.Metastore resolves through.
Package objtest builds a data-object storage directory for testing: a bucket holding logs objects, the index objects that describe them, and the tables of contents a metastore.Metastore resolves through.
Package sections locates the sections of a data object that belong to one tenant.
Package sections locates the sections of a data object that belong to one tenant.
internal/columnar
Package columnar provides a base implementation for sections which store columnar data using github.com/grafana/loki/v4/pkg/dataobj/internal/dataset.
Package columnar provides a base implementation for sections which store columnar data using github.com/grafana/loki/v4/pkg/dataobj/internal/dataset.
logs
Package logs defines types used for the data object logs section.
Package logs defines types used for the data object logs section.
pointers
Package pointers defines types used for the data object pointers section.
Package pointers defines types used for the data object pointers section.
postings
Package postings defines types for the data object postings section.
Package postings defines types for the data object postings section.
stats
Package stats defines types for the data object stats section.
Package stats defines types for the data object stats section.
streams
Package streams defines types used for the data object streams section.
Package streams defines types used for the data object streams section.
Package sortmerge provides a k-way merge iterator over dataobj logs sections.
Package sortmerge provides a k-way merge iterator over dataobj logs sections.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL