indexpointers

package
v3.7.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 26, 2026 License: AGPL-3.0 Imports: 25 Imported by: 1

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func CheckSection 

func CheckSection(section *dataobj.Section) bool

CheckSection returns true if section is a indexpointers section.

func Iter 

func Iter(ctx context.Context, obj *dataobj.Object) result.Seq[TenantIndexPointer]

Iter iterates over indexpointers in the provided decoder. All indexpointers sections are iterated over in order.

func IterSection 

func IterSection(ctx context.Context, section *Section) result.Seq[IndexPointer]

Types

type AndPredicate added in v3.7.0 

type AndPredicate struct{ Left, Right Predicate }

An AndPredicate is a Predicate which asserts that a row may only be included if both the Left and Right Predicate are true.

type Builder 

type Builder struct {
	// contains filtered or unexported fields
}

func NewBuilder 

func NewBuilder(metrics *Metrics, pageSize, pageRowCount int) *Builder

func (*Builder) Append 

func (b *Builder) Append(path string, startTs time.Time, endTs time.Time)

Append adds a new index pointer to the builder.

func (*Builder) EstimatedSize 

func (b *Builder) EstimatedSize() int

EstimatedSize returns the estimated size of the Pointers section in bytes.

func (*Builder) Flush 

func (b *Builder) Flush(w dataobj.SectionWriter) (n int64, err error)

Flush flushes the streams section to the provided writer.

After successful encoding, b is reset to a fresh state and can be reused.

func (*Builder) Reset 

func (b *Builder) Reset()

Reset resets all state, allowing IndexPointers to be reused.

func (*Builder) SetTenant 

func (b *Builder) SetTenant(tenant string)

func (*Builder) Tenant 

func (b *Builder) Tenant() string

func (*Builder) Type 

func (b *Builder) Type() dataobj.SectionType

type Column 

type Column struct {
	Section *Section
	Name    string
	Type    ColumnType
	// contains filtered or unexported fields
}

A Column represents one of the columns in the indexpointers section. Valid columns can only be retrieved by calling Section.Columns.

Data in columns can be read by using a Reader.

type ColumnStats 

type ColumnStats struct {
	Name             string
	Type             string
	ValueType        string
	RowsCount        uint64
	Compression      string
	UncompressedSize uint64
	CompressedSize   uint64
	MetadataOffset   uint64
	MetadataSize     uint64
	ValuesCount      uint64
	Cardinality      uint64

	Pages []PageStats
}

ColumnStats provides statistics about a column in a section.

type ColumnType 

type ColumnType int

ColumnType represents the kind of information stored in a Column. 

const (
	ColumnTypeInvalid      ColumnType = iota // ColumnTypeInvalid is an invalid column.
	ColumnTypePath                           // ColumnTypePath is a column containing the path to the index object.
	ColumnTypeMinTimestamp                   // ColumnTypeMinTimestamp is a column containing the minimum timestamp of the index object.
	ColumnTypeMaxTimestamp                   // ColumnTypeMaxTimestamp is a column containing the maximum timestamp of the index object.
)

func ParseColumnType 

func ParseColumnType(text string) (ColumnType, error)

ParseColumnType parses a ColumnType from a string. The expected string format is same same as the return value of ColumnType.String.

func (ColumnType) String 

func (ct ColumnType) String() string

String returns the human-readable name of ct.

type EqualPredicate added in v3.7.0 

type EqualPredicate struct {
	Column *Column       // Column to check.
	Value  scalar.Scalar // Value to check equality for.
}

An EqualPredicate is a Predicate which asserts that a row may only be included if the Value of the Column is equal to the Value.

type FalsePredicate added in v3.7.0 

type FalsePredicate struct{}

FalsePredicate is a Predicate which always returns false.

type FuncPredicate added in v3.7.0 

type FuncPredicate struct {
	Column *Column // Column to check.

	// Keep is invoked with the column and value pair to check. Keep is given
	// the Column instance to allow for reusing the same function across
	// multiple columns, if necessary.
	//
	// If Keep returns true, the row is kept.
	Keep func(column *Column, value scalar.Scalar) bool
}

FuncPredicate is a Predicate which asserts that a row may only be included if the Value of the Column passes the Keep function.

Instances of FuncPredicate are ineligible for page filtering and should only be used when there isn't a more explicit Predicate implementation.

type GreaterThanPredicate added in v3.7.0 

type GreaterThanPredicate struct {
	Column *Column       // Column to check.
	Value  scalar.Scalar // Value for which rows in Column must be greater than.
}

A GreaterThanPredicate is a Predicate which asserts that a row may only be included if the Value of the Column is greater than the provided Value.

type InPredicate added in v3.7.0 

type InPredicate struct {
	Column *Column         // Column to check.
	Values []scalar.Scalar // Values to check for inclusion.
}

An InPredicate is a Predicate which asserts that a row may only be included if the Value of the Column is present in the provided Values.

type IndexPointer 

type IndexPointer struct {
	Path    string
	StartTs time.Time
	EndTs   time.Time
}

IndexPointer is a pointer to an index object. It is used to lookup the index object by data object path within a time range.

The path is the data object path, the start and end timestamps are the time range of data stored in the index object.

type LessThanPredicate added in v3.7.0 

type LessThanPredicate struct {
	Column *Column       // Column to check.
	Value  scalar.Scalar // Value for which rows in Column must be less than.
}

A LessThanPredicate is a Predicate which asserts that a row may only be included if the Value of the Column is less than the provided Value.

type Metrics 

type Metrics struct {
	// contains filtered or unexported fields
}

func NewMetrics 

func NewMetrics() *Metrics

func (*Metrics) Observe 

func (m *Metrics) Observe(ctx context.Context, section *Section) error

Observe observes section statistics for a given section.

func (*Metrics) Register 

func (m *Metrics) Register(reg prometheus.Registerer) error

func (*Metrics) Unregister 

func (m *Metrics) Unregister(reg prometheus.Registerer)

type NotPredicate added in v3.7.0 

type NotPredicate struct{ Inner Predicate }

A NotePredicate is a Predicate which asserts that a row may only be included if the inner Predicate is false.

type OrPredicate added in v3.7.0 

type OrPredicate struct{ Left, Right Predicate }

An OrPredicate is a Predicate which asserts that a row may only be included if either the Left or Right Predicate are true.

type PageStats 

type PageStats struct {
	UncompressedSize uint64
	CompressedSize   uint64
	CRC32            uint32
	RowsCount        uint64
	Encoding         string
	DataOffset       uint64
	DataSize         uint64
	ValuesCount      uint64
}

PageStats provides statistics about a page in a column.

type Predicate added in v3.7.0 

type Predicate interface {
	// contains filtered or unexported methods
}

Predicate is an expression used to filter column values in a Reader.

func WhereTimeRangeOverlapsWith added in v3.7.0 

func WhereTimeRangeOverlapsWith(
	colMinTimestamp *Column,
	colMaxTimestamp *Column,
	start scalar.Scalar,
	end scalar.Scalar,
) Predicate

type Reader added in v3.7.0 

type Reader struct {
	// contains filtered or unexported fields
}

A Reader reads batches of rows from a Section.

func NewReader added in v3.7.0 

func NewReader(opts ReaderOptions) *Reader

NewReader creates a new Reader from the provided options. Options are not validated until the first call to Reader.Open.

func (*Reader) Close added in v3.7.0 

func (r *Reader) Close() error

Close closes the Reader and releases any resources it holds. Closed Readers can be reused by calling Reader.Reset.

func (*Reader) Open added in v3.7.0 

func (r *Reader) Open(ctx context.Context) error

Open initializes Reader resources.

Open must be called before Reader.Read. Open is safe to call multiple times.

func (*Reader) Read added in v3.7.0 

func (r *Reader) Read(ctx context.Context, batchSize int) (arrow.RecordBatch, error)

Read reads the batch of rows from the section, returning them as an Arrow record.

If ReaderOptions has predicates, only rows that match the predicates are returned. If none of the next batchSize rows matched the predicate, Read returns a nil record with a nil error.

Read will return an error if the next batch of rows could not be read due to invalid options or I/O errors. At the end of the section, Read returns nil, io.EOF.

Read may return a non-nil record with a non-nil error, including if the error is io.EOF. Callers should always process the record before processing the error value.

When a record is returned, it will match the schema specified by Reader.Schema. These records must always be released after use.

func (*Reader) Reset added in v3.7.0 

func (r *Reader) Reset(opts ReaderOptions)

Reset discards any state and resets r with a new set of options. This permits reusing a Reader rather than allocating a new one.

func (*Reader) Schema added in v3.7.0 

func (r *Reader) Schema() *arrow.Schema

Schema returns the arrow.Schema used by the Reader. Fields in the schema match the order of columns listed in ReaderOptions.

Names of fields in the schema are guaranteed to be unique per column but are not guaranteed to be stable.

The returned Schema must not be modified.

type ReaderOptions added in v3.7.0 

type ReaderOptions struct {
	// Columns to read. Each column must belong to the same [Section].
	Columns []*Column

	// Predicates holds a set of predicates to apply when reading the section.
	// Columns referenced in Predicates must be in the set of Columns.
	Predicates []Predicate

	// Allocator to use for allocating Arrow records. If nil,
	// [memory.DefaultAllocator] is used.
	Allocator memory.Allocator
}

ReaderOptions customizes the behavior of a Reader.

func (*ReaderOptions) Validate added in v3.7.0 

func (opts *ReaderOptions) Validate() error

Validate returns an error if the opts is not valid. ReaderOptions are only valid when:

  • Each Column in Columns belongs to the same Section.
  • Each Predicate in Predicates references a Column from Columns.
  • Scalar values used in predicates are of a supported type: an int64, uint64, timestamp, or a byte array.

type RowPredicate 

type RowPredicate interface {
	// contains filtered or unexported methods
}

RowPredicate is an expression used to filter rows in a data object.

type RowReader 

type RowReader struct {
	// contains filtered or unexported fields
}

RowReader is a reader for index pointers in a data object.

func NewRowReader 

func NewRowReader(sec *Section) *RowReader

NewRowReader creates a new RowReader for the given section.

Call RowReader.Open before calling RowReader.Read.

func (*RowReader) Close 

func (r *RowReader) Close() error

Close closes the RowReader and releases any resources it holds. Closed RowReaders can be reused by calling RowReader.Reset.

func (*RowReader) Open added in v3.7.0 

func (r *RowReader) Open(ctx context.Context) error

Open initializes RowReader resources.

Open must be called before RowReader.Read. Open is safe to call multiple times. Open is a no-op when the reader has no section.

func (*RowReader) Read 

func (r *RowReader) Read(ctx context.Context, s []IndexPointer) (int, error)

Read reads up to the next len(s) indexpointers from the reader and stores them into s. It returns the number of indexpointers read and any error encountered. At the end of the indexpointers section, Read returns 0, io.EOF.

func (*RowReader) Reset 

func (r *RowReader) Reset(sec *Section)

Reset resets the RowReader with a new decoder to read from. Reset allows reusing a RowReader without allocating a new one.

Any set predicate is cleared when Reset is called.

Reset may be called with a nil object and a negative section index to clear the RowReader without needing a new object.

func (*RowReader) SetPredicate 

func (r *RowReader) SetPredicate(p RowPredicate) error

SetPredicate sets the predicate to use for filtering indexpointers. RowReader.Read will only return indexpointers for which the predicate passes.

SetPredicate returns an error if the predicate is not supported by RowReader.

A predicate may only be set before reading begins or after a call to RowReader.Reset.

type Section 

type Section struct {
	// contains filtered or unexported fields
}

Section represents an opened indexpointers section.

func Open 

func Open(ctx context.Context, section *dataobj.Section) (*Section, error)

Open opens a Section from an underlying dataobj.Section. Open returns an error if the section metadata could not be read or if the provided ctx is canceled.

func (*Section) Columns 

func (s *Section) Columns() []*Column

Columns returns the set of Columns in the section. The slice of returned sections must not be mutated.

Unrecognized columns (e.g., when running older code against newer indexpointers sections) are skipped.

type Stats 

type Stats struct {
	UncompressedSize uint64
	CompressedSize   uint64

	MinTimestamp          time.Time
	MaxTimestamp          time.Time
	TimestampDistribution []uint64 // Indexpointer count per hour.

	Columns []ColumnStats
}

Stats provides statistics about a indexpointers section.

func ReadStats 

func ReadStats(ctx context.Context, section *Section) (Stats, error)

ReadStats returns statistics about the indexpointers section. ReadStats returns an error if the indexpointers section couldn't be inspected or if the provided ctx is canceled.

type TenantIndexPointer added in v3.7.0 

type TenantIndexPointer struct {
	Tenant string
	IndexPointer
}

TenantIndexPointer is meant to collectively hold IndexPointer with the ID of the tenant it belongs to.

type TimeRangeRowPredicate 

type TimeRangeRowPredicate struct{ Start, End time.Time }

A TimeRangeRowPredicate is a RowPredicate which requires a start and end timestamp column to exist, and for the timestamp to be within the range.

type TruePredicate added in v3.7.0 

type TruePredicate struct{}

TruePredicate is a Predicate which always returns true.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.