Documentation
¶
Overview ¶
Package columnar provides utilities for working with columnar in-memory arrays.
Columnar types are Arrow-compatible. The columnar package exists to provide a Loki-optimized APIs for columnar data that is based on memory.Allocator for memory management rather than arrow-go's reference counting.
Package columnar is EXPERIMENTAL and currently only intended to be used by github.com/grafana/loki/v3/pkg/dataobj.
Index ¶
- type Array
- type Bool
- func (arr *Bool) Get(i int) bool
- func (arr *Bool) IsNull(i int) bool
- func (arr *Bool) Kind() Kind
- func (arr *Bool) Len() int
- func (arr *Bool) Nulls() int
- func (arr *Bool) Size() int
- func (arr *Bool) Slice(i, j int) Array
- func (arr *Bool) Validity() memory.Bitmap
- func (arr *Bool) Values() memory.Bitmap
- type BoolBuilder
- type BoolScalar
- type Builder
- type Column
- type Datum
- type Kind
- type Null
- type NullBuilder
- type NullScalar
- type Number
- func (arr *Number[T]) Get(i int) T
- func (arr *Number[T]) IsNull(i int) bool
- func (arr *Number[T]) Kind() Kind
- func (arr *Number[T]) Len() int
- func (arr *Number[T]) Nulls() int
- func (arr *Number[T]) Size() int
- func (arr *Number[T]) Slice(i, j int) Array
- func (arr *Number[T]) Validity() memory.Bitmap
- func (arr *Number[T]) Values() []T
- type NumberBuilder
- func (b *NumberBuilder[T]) AppendNull()
- func (b *NumberBuilder[T]) AppendNulls(count int)
- func (b *NumberBuilder[T]) AppendValue(v T)
- func (b *NumberBuilder[T]) AppendValueCount(v T, count int)
- func (b *NumberBuilder[T]) Build() *Number[T]
- func (b *NumberBuilder[T]) BuildArray() Array
- func (b *NumberBuilder[T]) Grow(n int)
- type NumberScalar
- type Numeric
- type RecordBatch
- type Scalar
- type Schema
- type Set
- type UTF8
- func (arr *UTF8) Data() []byte
- func (arr *UTF8) DataLen() int
- func (arr *UTF8) Get(i int) []byte
- func (arr *UTF8) IsNull(i int) bool
- func (arr *UTF8) Kind() Kind
- func (arr *UTF8) Len() int
- func (arr *UTF8) Nulls() int
- func (arr *UTF8) Offsets() []int32
- func (arr *UTF8) Size() int
- func (arr *UTF8) Slice(i, j int) Array
- func (arr *UTF8) Validity() memory.Bitmap
- type UTF8Builder
- type UTF8Scalar
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Array ¶
type Array interface {
Datum
// Len returns the total number of elements in the array.
Len() int
// Nulls returns the number of null elements in the array. The number of
// non-null elements can be calculated from Len() - Nulls().
Nulls() int
// IsNull returns true if the element at index i is null.
IsNull(i int) bool
// Size returns the total consumed size of the array's buffers in bytes.
//
// It doesn't account for the size of the Array type itself, or any unused
// bytes (such as padding past the length up to the capacity of buffers).
Size() int
// Slice returns a slice of the Array from index i to j. Slice panics if j <
// i or if the slice is outside the valid range of the Array. The returned
// slice has a length of j-i and shares memory with the original Array.
//
// Slice panics if the following invariant is not met: 0 <= i <= j <= arr.Len()
Slice(i, j int) Array
// Validity returns the validity bitmap of the array. The returned bitmap
// may be of length 0 if there are no nulls.
//
// A value of 1 in the Validity bitmap indicates that the corresponding
// element at that position is valid (not null).
Validity() memory.Bitmap
// contains filtered or unexported methods
}
An Array is a sequence of elements of the same data type.
type Bool ¶
type Bool struct {
// contains filtered or unexported fields
}
Bool is an Array of bit-packed boolean values.
func NewBool ¶
NewBool creates a new Bool array from the given values and optional validity bitmap.
Bool arrays made from memory owned by a memory.Allocator are invalidated when the allocator reclaims memory.
If validity is of length zero, all elements are considered valid. Otherwise, NewBool panics if the number of elements does not match the length of validity.
func (*Bool) Get ¶
Get returns the value at index i. If the element at index i is null, Get returns an undefined value.
Get panics if i is out of range.
func (*Bool) Nulls ¶
Nulls returns the number of null elements in the array. The number of non-null elements can be calculated from Len() - Nulls().
type BoolBuilder ¶
type BoolBuilder struct {
// contains filtered or unexported fields
}
A BoolBuilder assists with constructing a Bool array. A BoolBuilder must be constructed by calling NewBoolBuilder.
func NewBoolBuilder ¶
func NewBoolBuilder(alloc *memory.Allocator) *BoolBuilder
NewBoolBuilder creates a new BoolBuilder for constructing a Bool array.
func (*BoolBuilder) AppendNull ¶
func (b *BoolBuilder) AppendNull()
AppendNull adds a new null element to b.
func (*BoolBuilder) AppendNulls ¶
func (b *BoolBuilder) AppendNulls(count int)
AppendNulls appends the given number of null elements to b.
func (*BoolBuilder) AppendValue ¶
func (b *BoolBuilder) AppendValue(v bool)
AppendValue adds a new element to b.
func (*BoolBuilder) AppendValueCount ¶
func (b *BoolBuilder) AppendValueCount(v bool, count int)
AppendValue adds a new element to b.
func (*BoolBuilder) Build ¶
func (b *BoolBuilder) Build() *Bool
Build returns the constructed Bool array. After calling Build, the builder is reset to an initial state.
func (*BoolBuilder) BuildArray ¶
func (b *BoolBuilder) BuildArray() Array
Build returns the constructed array. After calling Build, the builder is reset to an initial state.
func (*BoolBuilder) Grow ¶
func (b *BoolBuilder) Grow(n int)
Grow increases b's capacity, if necessary, to guarantee space for another n elements. After Grow(n), at least n elements can be appended to b without another allocation. If n is negative or too large to allocate the memory, Grow panics.
type BoolScalar ¶
type BoolScalar struct {
Value bool // Value of the scalar.
Null bool // True if the scalar is null.
}
BoolScalar is a Scalar representing a boolean value.
func (*BoolScalar) IsNull ¶
func (s *BoolScalar) IsNull() bool
IsNull implements Scalar and returns bs.Null.
type Builder ¶
type Builder interface {
// AppendNull adds a new null element to the Builder.
AppendNull()
// AppendNulls appends the given number of null elements to the Builder.
AppendNulls(count int)
// Build returns the constructed Array. After calling Build, the builder is
// reset to an initial state and can be reused.
BuildArray() Array
}
A Builder assists with constructing arrays. Builder implementations typically have more methods.
type Column ¶
type Column struct {
Name string // Name of the column.
}
A Column describes a single column in a RecordBatch.
type Datum ¶
type Datum interface {
// Kind returns the Kind of value represented by the Datum.
Kind() Kind
// contains filtered or unexported methods
}
A Datum is a piece of data which either represents an Array or a Scalar.
type Kind ¶
type Kind uint8
A Kind represents the specific type of an Array. The zero Kind is a null value.
const ( KindNull Kind = iota // KindNull represents a null value. KindBool // KindBool represents a boolean value. KindInt64 // KindInt64 represents a 64-bit integer value. KindUint64 // KindUint64 represents a 64-bit unsigned integer value. KindUTF8 // KindUTF8 represents a UTF-8 encoded string value. )
type Null ¶
type Null struct {
// contains filtered or unexported fields
}
Null is an Array of null values.
func NewNull ¶
NewNull creates a new Null array with the given validity bitmap. NewNull panics if validity contains any bit set to true.
type NullBuilder ¶
type NullBuilder struct {
// contains filtered or unexported fields
}
A NullBuilder assists with constructing a Null array. A NullBuilder must be constructed by calling NewNullBuilder.
func NewNullBuilder ¶
func NewNullBuilder(alloc *memory.Allocator) *NullBuilder
NewNullBuilder creates a new NullBuilder for constructing a Null array.
func (*NullBuilder) AppendNull ¶
func (b *NullBuilder) AppendNull()
AppendNull adds a new null element to b.
func (*NullBuilder) AppendNulls ¶
func (b *NullBuilder) AppendNulls(count int)
AppendNulls appends the given number of null elements to b.
func (*NullBuilder) Build ¶
func (b *NullBuilder) Build() *Null
Build returns the constructed Null array. After calling Build, the builder is reset to an initial state.
func (*NullBuilder) BuildArray ¶
func (b *NullBuilder) BuildArray() Array
Build returns the constructed array. After calling Build, the builder is reset to an initial state.
func (*NullBuilder) Grow ¶
func (b *NullBuilder) Grow(n int)
Grow increases b's capacity, if necessary, to guarantee space for another n elements. After Grow(n), at least n elements can be appended to b without another allocation. If n is negative or too large to allocate the memory, Grow panics.
type NullScalar ¶
type NullScalar struct{}
NullScalar is a Scalar representing an untyped null value.
func (*NullScalar) IsNull ¶
func (s *NullScalar) IsNull() bool
IsNull implements Scalar and returns true.
type Number ¶
type Number[T Numeric] struct { // contains filtered or unexported fields }
Number is an Array of 64-bit unsigned Numeric values.
func NewNumber ¶
NewNumber creates a new Number array from the given values and optional validity bitmap.
Number arrays made from memory owned by a memory.Allocator are invalidated when the allocator reclaims memory.
If validity is of length zero, all elements are considered valid. Otherwise, NewNumber panics if the number of elements does not match the length of validity.
func (*Number[T]) Get ¶
Get returns the value at index i. If the element at index i is null, Get returns an undefined value.
Get panics if i is out of range.
func (*Number[T]) Nulls ¶
Nulls returns the number of null elements in the array. The number of non-null elements can be calculated from Len() - Nulls().
type NumberBuilder ¶
type NumberBuilder[T Numeric] struct { // contains filtered or unexported fields }
A NumberBuilder assists with constructing a Number array. A NumberBuilder must be constructed by calling NewNumberBuilder.
func NewNumberBuilder ¶
func NewNumberBuilder[T Numeric](alloc *memory.Allocator) *NumberBuilder[T]
NewNumberBuilder creates a new NumberBuilder for constructing a Number array.
func (*NumberBuilder[T]) AppendNull ¶
func (b *NumberBuilder[T]) AppendNull()
AppendNull adds a new null element to b.
func (*NumberBuilder[T]) AppendNulls ¶
func (b *NumberBuilder[T]) AppendNulls(count int)
AppendNulls appends the given number of null elements to b.
func (*NumberBuilder[T]) AppendValue ¶
func (b *NumberBuilder[T]) AppendValue(v T)
AppendValue adds a new element to b.
func (*NumberBuilder[T]) AppendValueCount ¶
func (b *NumberBuilder[T]) AppendValueCount(v T, count int)
AppendValue adds a new element to b.
func (*NumberBuilder[T]) Build ¶
func (b *NumberBuilder[T]) Build() *Number[T]
Build returns the constructed Number array. After calling Build, the builder is reset to an initial state.
func (*NumberBuilder[T]) BuildArray ¶
func (b *NumberBuilder[T]) BuildArray() Array
BuildArray returns the constructed array. After calling Build, the builder is reset to an initial state.
func (*NumberBuilder[T]) Grow ¶
func (b *NumberBuilder[T]) Grow(n int)
Grow increases b's capacity, if necessary, to guarantee space for another n elements. After Grow(n), at least n elements can be appended to b without another allocation. If n is negative or too large to allocate the memory, Grow panics.
type NumberScalar ¶
type NumberScalar[T Numeric] struct { Value T // Value of the scalar. Null bool // True if the scalar is null. // contains filtered or unexported fields }
NumberScalar is a Scalar representing a Numeric value.
func (*NumberScalar[T]) IsNull ¶
func (s *NumberScalar[T]) IsNull() bool
IsNull implements Scalar and returns s.Null.
func (*NumberScalar[T]) Kind ¶
func (s *NumberScalar[T]) Kind() Kind
Kind implements Datum and returns the kind matching T.
type RecordBatch ¶
type RecordBatch struct {
// contains filtered or unexported fields
}
RecordBatch is a collection of equal-length arrays. This corresponds to the RecordBatch concept in the Arrow specification.
func NewRecordBatch ¶
func NewRecordBatch(schema *Schema, nrows int64, arrs []Array) *RecordBatch
NewRecordBatch returns a new RecordBatch created from the provided arrays. nrows specifies the total number of rows in the batch.
func (*RecordBatch) Column ¶
func (rb *RecordBatch) Column(i int64) Array
Column returns the column array at index i.
func (*RecordBatch) NumCols ¶
func (rb *RecordBatch) NumCols() int64
NumCols returns the number of columns in the batch.
func (*RecordBatch) NumRows ¶
func (rb *RecordBatch) NumRows() int64
NumRows returns the number of rows in the batch.
func (*RecordBatch) Schema ¶
func (rb *RecordBatch) Schema() *Schema
Schema returns the schema of the record batch.
func (*RecordBatch) Slice ¶
func (rb *RecordBatch) Slice(i, j int) *RecordBatch
Slice returns a slice of rb from index i to j. The returned slice has a length of j-i and shares memory with rb.
Slice panics if the following invariant is not met: 0 <= i <= j <= rb.NumRows()
type Scalar ¶
type Scalar interface {
Datum
// IsNull returns true if the scalar is a null value. IsNull can return true
// for non-null data types.
IsNull() bool
// contains filtered or unexported methods
}
A Scalar is a single element of a specific data type.
type Schema ¶
type Schema struct {
// contains filtered or unexported fields
}
A Schema describes the set of columns in a RecordBatch.
func NewSchema ¶
NewSchema creates a new schema from a list of columns. Column names must be unique. If column names are not unique, NewSchema panics.
func (*Schema) ColumnIndex ¶
ColumnIndex returns the column with the given name, along with its index. ColumnIndex returns the Column{}, -1 if the column doesn't exist.
func (*Schema) NumColumns ¶
NumColumns returns the number of columns in the schema.
type Set ¶
type Set struct {
// contains filtered or unexported fields
}
Set represents a unique set of values of a given kind. A Set must be constructed using the New functions in order to ensure that the set is properly initialized and all values in the set are of the correct Kind.
func NewNumberSet ¶
NewNumberSet creates a new Set of a numeric Kind T initialized with the provided values. The Numeric type T will map to the following Kinds: - int64 -> KindInt64 - uint64 -> KindUint64
func NewUTF8Set ¶
NewUTF8Set creates a new Set of Kind UTF8 initialized with the provided strings.
type UTF8 ¶
type UTF8 struct {
// contains filtered or unexported fields
}
UTF8 is an Array of UTF-8 encoded strings.
func NewUTF8 ¶
NewUTF8 creates a new UTF8 array from the given data, offsets, and optional validity bitmap.
UTF8 arrays made from memory owned by a memory.Allocator are invalidated when the allocator reclaims memory.
Each UTF-8 string goes from data[offsets[i] : offsets[i+1]]. Offsets must be monotonically increasing, even for null values. The offsets slice is not validated for correctness.
If validity is of length zero, all elements are considered valid. Otherwise, NewUTF8 panics if the number of elements does not match the length of validity.
func (*UTF8) Data ¶
Data returns the underlying packed UTF8 bytes. If arr is a slice, Data may include bytes beyond the offset ranges from the original array.
func (*UTF8) Get ¶
Get returns the value at index i. If the element at index i is null, Get returns an empty string.
Get panics if i is out of range.
func (*UTF8) Nulls ¶
Nulls returns the number of null elements in the array. The number of non-null elements can be calculated from Len() - Nulls().
func (*UTF8) Slice ¶
Slice returns a slice of arr from i to j.
A sliced UTF8 array creates a slice of the offsets and validity buffers, but not the data buffer, permitting the offsets to continue to be valid without needing to normalize them to start at 0.
It is recommended to normalize the offsets and remove unused memory in the data buffer before serializing UTF8 for network communication.
type UTF8Builder ¶
type UTF8Builder struct {
// contains filtered or unexported fields
}
A UTF8Builder assists with constructing a UTF8 array. A UTF8Builder must be constructed by calling NewUTF8Builder.
func NewUTF8Builder ¶
func NewUTF8Builder(alloc *memory.Allocator) *UTF8Builder
NewUTF8Builder creates a new UTF8Builder for constructing a UTF8 array.
func (*UTF8Builder) AppendNull ¶
func (b *UTF8Builder) AppendNull()
AppendNull adds a new null element to b.
func (*UTF8Builder) AppendNulls ¶
func (b *UTF8Builder) AppendNulls(count int)
AppendNulls appends the given number of null elements to b.
func (*UTF8Builder) AppendValue ¶
func (b *UTF8Builder) AppendValue(v []byte)
AppendValue adds a new non-null element to b.
func (*UTF8Builder) Build ¶
func (b *UTF8Builder) Build() *UTF8
Build returns the constructed UTF8 array. After calling Build, the builder is reset to an initial state.
func (*UTF8Builder) BuildArray ¶
func (b *UTF8Builder) BuildArray() Array
BuildArray returns the constructed array. After calling Build, the builder is reset to an initial state.
func (*UTF8Builder) Grow ¶
func (b *UTF8Builder) Grow(n int)
Grow increases b's capacity, if necessary, to guarantee space for another n elements. After Grow(n), at least n elements can be appended to b without another allocation. If n is negative or too large to allocate the memory, Grow panics.
func (*UTF8Builder) GrowData ¶
func (b *UTF8Builder) GrowData(n int)
GrowData increases b's bytes capacity, if necessary, to guarantee space for another n bytes of data. After GrowData(n), at least n bytes can be appended to b (across all UTF8 values) without another allocation. If n is negative or too large to allocate the memory, GrowData panics.
GrowData only impacts capacity of string data. Use UTF8Builder.Grow to reserve space for more elements.
Source Files
Directories