Documentation
¶
Overview ¶
Package data holds ursus's runtime column representation.
Column is contiguous, chunking lives above it ¶
A Column is a single flat run of values, not a chunked array. Every kernel therefore takes a plain []T and never has to reason about chunk boundaries — which removes an entire category of off-by-one bug from the layer where such bugs are most expensive.
Chunking is not lost, it is moved up: a query result is a sequence of Batches, so concatenation is still O(1) at the level where concatenation happens. The cost of this choice is one file (the evaluator) if we ever need a chunked column, and the benefit is that ~100 kernels stay simple. That trade is obviously right at this stage and worth revisiting only with a benchmark.
Ownership ¶
Columns are immutable once built and safe to share between goroutines. There is no Release: see internal/arrowx for why Arrow's refcounting is advisory under the Go allocator.
Index ¶
- func MustValues[T Fixed](c *Column) []T
- func Reinterpret[T Fixed](b []byte) []T
- func Values[T Fixed](c *Column) ([]T, error)
- type Batch
- func (b *Batch) All() iter.Seq2[int, *Column]
- func (b *Batch) ByName(name string) (*Column, bool)
- func (b *Batch) Column(i int) *Column
- func (b *Batch) Columns() []*Column
- func (b *Batch) NBytes() int64
- func (b *Batch) NumCols() int
- func (b *Batch) Rows() int
- func (b *Batch) Schema() *dtype.Schema
- func (b *Batch) Slice(offset, length int) *Batch
- type BufferID
- type BufferSet
- type Column
- func NewBool(name string, bits bitmap.View, valid bitmap.View) *Column
- func NewFixed[T Fixed](name string, dt dtype.DataType, vals []T, valid bitmap.View) *Column
- func NewFixedBuffer(name string, dt dtype.DataType, buf *memory.Buffer, n int, valid bitmap.View) *Column
- func NewNull(name string, dt dtype.DataType, n int) *Column
- func NewString(name string, vals []string, valid bitmap.View) *Column
- func NewStringBuffers(name string, offs, chars *memory.Buffer, n int, valid bitmap.View) *Column
- func NewStringParts(name string, offs []int32, chars []byte, valid bitmap.View) *Column
- func (c *Column) Bools() bitmap.View
- func (c *Column) Buffers() iter.Seq2[BufferID, int64]
- func (c *Column) DType() dtype.DataType
- func (c *Column) IsPayloadFree() bool
- func (c *Column) IsValid(i int) bool
- func (c *Column) Len() int
- func (c *Column) NBytes() int64
- func (c *Column) Name() string
- func (c *Column) NullCount() int
- func (c *Column) RawChars() []byte
- func (c *Column) RawFixed() []byte
- func (c *Column) RawOffsets() []byte
- func (c *Column) Rename(name string) *Column
- func (c *Column) Slice(offset, length int) *Column
- func (c *Column) Strings() StringAccessor
- func (c *Column) Validity() bitmap.View
- func (c *Column) WithDType(dt dtype.DataType) *Column
- type Fixed
- type Primitive
- type Series
- func (s *Series[T]) All() iter.Seq2[T, bool]
- func (s *Series[T]) Column() *Column
- func (s *Series[T]) DType() dtype.DataType
- func (s *Series[T]) Fold[A any](init A, fn func(A, T) A) A
- func (s *Series[T]) Get(i int) (T, bool)
- func (s *Series[T]) Len() int
- func (s *Series[T]) Map[U any](fn func(T) U) []U
- func (s *Series[T]) MapErr[U any](fn func(T) (U, error)) ([]U, error)
- func (s *Series[T]) Name() string
- func (s *Series[T]) NullCount() int
- func (s *Series[T]) Values() []T
- type StringAccessor
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func MustValues ¶
MustValues is Values for call sites that have already checked the dtype.
func Reinterpret ¶
Reinterpret exposes a buffer as a typed slice. Used by kernels that allocate an output buffer and write into it directly.
func Values ¶
Values returns the column's fixed-width payload reinterpreted as []T.
Zero-copy: the slice aliases the column's buffer. It must not be mutated — columns are shared.
T must match the column's PHYSICAL type: a Datetime column is read as []int64, a Date column as []int32. Reading through the physical type is what lets one integer kernel serve every type that stores integers.
Types ¶
type Batch ¶
type Batch struct {
// contains filtered or unexported fields
}
Batch is a horizontal slice of a frame: N equal-length columns with a schema.
It is the unit that flows between physical operators. Batch size is a tuning knob rather than a semantic one — the same query must produce the same answer at any batch size, and the test suite checks that by varying it.
func NewBatch ¶
NewBatch builds a batch, checking that the columns match the schema in count, order, name and type.
The check is not paranoia: a mismatch here means the evaluator produced something other than what the plan's schema resolution promised, which is exactly the class of bug that would otherwise surface as corrupted output far downstream. Failing at the boundary makes it a one-line diagnosis.
func NewBatchRows ¶
NewBatchRows builds a zero-column batch that still has a row count. A frame can legitimately have rows but no columns after `Select()` with nothing.
type BufferID ¶
type BufferID = *byte
BufferID identifies one allocation by the address of its first byte.
Why identity rather than a sum ¶
Columns share payloads. Rename shares one ("O(1): the payload is shared"), so does WithDType, so does the character buffer of a sliced String column, and every pipeline breaker retains the source's own batches rather than copies. A naive per-column sum therefore over-reports, sometimes by a large factor — and an over-reporting memory budget fires early and unpredictably, which is worse than having no budget at all.
It is a *byte rather than a uintptr on purpose. A real pointer stays meaningful to the garbage collector, so an identity held in a ledger can never be recycled by a later allocation while the entry is still live.
type BufferSet ¶
type BufferSet struct {
// contains filtered or unexported fields
}
BufferSet accumulates the distinct allocations of any number of columns.
The zero value is ready to use.
type Column ¶
type Column struct {
// contains filtered or unexported fields
}
Column is one named, typed, nullable run of values.
Exactly one payload representation is populated, determined by dt:
fixed-width types → fixed (a flat values buffer) Bool → bits (a packed bitmap, NOT one byte per value) String, Binary → offs + chars (int32 offsets plus a character buffer)
Bool having a distinct representation is not an accident of implementation: a Boolean column has TWO bitmaps, values and validity, and keeping them the same shape is what makes `null > 5` expressible as "value bit 0, validity bit 0".
func NewFixed ¶
NewFixed builds a fixed-width column over vals.
The values slice must alias a buffer obtained from arrowx so that alignment and lifetime are right; NewFixedBuffer is the lower-level entry point.
func NewFixedBuffer ¶
func NewFixedBuffer(name string, dt dtype.DataType, buf *memory.Buffer, n int, valid bitmap.View) *Column
NewFixedBuffer builds a fixed-width column directly over an existing buffer. This is the path a kernel takes: allocate the output buffer, write into it, and wrap it with no copy.
func NewNull ¶
NewNull builds an all-null column of the given type and length. It carries no payload buffer, which makes a typed null literal free.
func NewStringBuffers ¶
NewStringBuffers builds a String column over existing offset and character buffers, without copying.
func NewStringParts ¶
NewStringParts builds a String column from an offsets slice and a character slice that a reader has already accumulated.
It exists so a decoder can build Arrow's layout DIRECTLY instead of going through []string. The Parquet reader used to append one Go string per value — one heap allocation per row, 2.14M of them for a two-string-column scan of a million rows — and then hand the whole slice to NewString, which copied every byte a second time. Both costs vanish if the decoder appends into offs and chars as it goes, which is what this constructor is for.
offs must have n+1 entries with offs[0] == 0 and offs[n] == len(chars), the ordinary Arrow invariant. The int32 write goes through unsafeData rather than encoding/binary because the buffer is native-endian, which is what every reader of it assumes.
func (*Column) Buffers ¶
Buffers yields each distinct allocation the column holds, with its size.
Size is the buffer's CAPACITY, not the bytes in use: the allocation is what is actually held, and an under-filled buffer still occupies all of it.
A column has at most five slots — fixed, offs, chars and the two bitmaps — so the within-column deduplication below is a linear scan over a fixed array rather than a map. Deduplication ACROSS columns is BufferSet's job.
func (*Column) IsPayloadFree ¶
IsPayloadFree reports whether the column carries no payload buffer at all.
Only NewNull produces one, and it is a genuine value — "which makes a typed null literal free" — but it can only ever be an ANSWER, never an OPERAND. Values refuses it, StringAccessor.Get panics on it, and both are reachable from ordinary queries; kernel.NullColumn exists to build the operand-shaped equivalent.
That rule was previously enforced only by remembering it. A kernel handed a column from an unknown source can now ask.
func (*Column) NBytes ¶
NBytes returns the total capacity of the distinct allocations the column holds.
func (*Column) RawChars ¶
RawChars returns a String or Binary column's character buffer.
Note it is the WHOLE buffer: a sliced column keeps the original characters and re-windows the offsets, so offsets[0] is not necessarily zero and the live range is offsets[0]:offsets[len].
func (*Column) RawFixed ¶
RawFixed returns the fixed-width payload as bytes, or nil if there is none. The slice aliases the column's buffer and must not be mutated.
func (*Column) RawOffsets ¶
RawOffsets returns a String or Binary column's offset buffer as bytes. Read it as []int32 with Reinterpret; it holds len+1 entries.
func (*Column) Slice ¶
Slice returns rows [offset, offset+length) as a zero-copy view.
For fixed-width columns this needs a real offset on the values buffer, which the current representation does not carry, so fixed-width slicing copies. Bool and String slicing is genuinely O(1). This asymmetry is deliberate: Slice is not on any hot path in v0.1 (Filter uses Take, which must copy anyway), and adding an offset field to Column would put a `+ c.offset` in every kernel's inner loop to buy nothing today.
func (*Column) Strings ¶
func (c *Column) Strings() StringAccessor
Strings returns an accessor for a String or Binary column.
type Fixed ¶
Fixed is every type that occupies a constant number of bytes in a values buffer — Primitive plus Int128.
The split exists because Int128 is a STRUCT. It is 16 bytes with no pointers, so it stores exactly like any other numeric type and Values/NewFixed/Take work on it unchanged; but `a < b` and `a + b` do not compile for it, so anything using an operator needs Primitive instead and reaches Int128 through i128's methods.
Go's type system enforces this: adding Int128 to a constraint whose functions use operators is a compile error, not a runtime surprise.
type Primitive ¶
type Primitive interface {
~int8 | ~int16 | ~int32 | ~int64 |
~uint8 | ~uint16 | ~uint32 | ~uint64 |
~float32 | ~float64
}
Primitive is the set of Go types that back a fixed-width column AND support Go's comparison and arithmetic operators.
Generic kernels that write `a < b` or `a + b` must constrain to Primitive.
type Series ¶
type Series[T any] struct { // contains filtered or unexported fields }
Series is a typed view over a Column.
Column owns, Series views ¶
This resolves the "two representations, two code paths" problem the design review flagged. A Series holds no storage of its own: it is a Column plus a resolved accessor, constructed on demand and thrown away freely. The engine only ever handles Columns, so kernels, operators and the evaluator have exactly one type to know about. Series exists purely at the boundary where Go values meet columnar data.
Why generic methods matter here ¶
Before Go 1.27, Map and Fold had to be package-level functions taking the receiver as an argument:
data.Map(data.Map(s, parse), normalise) // reads inside-out
As generic methods they chain in the direction the data flows:
s.Map(parse).Map(normalise)
That difference — nothing more — is a large part of why this library targets Go 1.27.
func TypedColumn ¶
TypedColumn returns a typed view over c, or an error if T does not match the column's physical representation.
func (*Series[T]) Get ¶
Get returns the value at i and whether it is non-null.
The two-value form is deliberate. Boxing nulls into a *T or an Option[T] would cost an allocation per row and would make the values buffer non-contiguous, which is exactly what a columnar library must not do. Validity stays a parallel bitmap and the caller is told about it explicitly.
func (*Series[T]) Map ¶
Map applies fn to every value, producing a plain Go slice.
Nulls yield the zero value of U; use MapErr or All if the distinction matters. This is a generic METHOD, which is what lets it chain.
func (*Series[T]) MapErr ¶
MapErr is Map with a fallible function, stopping at the first error and naming the row it failed on.
type StringAccessor ¶
type StringAccessor struct {
// contains filtered or unexported fields
}
StringAccessor reads a variable-width column.
It is a struct rather than an interface so that the layout choice is resolved once per batch instead of once per value. When Arrow's StringView layout is added it becomes a tagged union here, and no kernel call site changes.
func (StringAccessor) Get ¶
func (a StringAccessor) Get(i int) string
Get returns row i without copying: the result aliases the character buffer.
Source Files