Documentation
¶
Overview ¶
Package data holds ursus's runtime column representation.
Column is contiguous, chunking lives above it ¶
A Column is a single flat run of values, not a chunked array. Every kernel therefore takes a plain []T and never has to reason about chunk boundaries — which removes an entire category of off-by-one bug from the layer where such bugs are most expensive.
Chunking is not lost, it is moved up: a query result is a sequence of Batches, so concatenation is still O(1) at the level where concatenation happens. The cost of this choice is one file (the evaluator) if we ever need a chunked column, and the benefit is that ~100 kernels stay simple. That trade is obviously right at this stage and worth revisiting only with a benchmark.
Ownership ¶
Columns are immutable once built and safe to share between goroutines. There is no Release: see internal/arrowx for why Arrow's refcounting is advisory under the Go allocator.
Index ¶
- Variables
- func MustValues[T Fixed](c *Column) []T
- func Reinterpret[T Fixed](b []byte) []T
- func Values[T Fixed](c *Column) ([]T, error)
- type Batch
- func (b *Batch) All() iter.Seq2[int, *Column]
- func (b *Batch) ByName(name string) (*Column, bool)
- func (b *Batch) Column(i int) *Column
- func (b *Batch) Columns() []*Column
- func (b *Batch) NBytes() int64
- func (b *Batch) NumCols() int
- func (b *Batch) Rows() int
- func (b *Batch) Schema() *dtype.Schema
- func (b *Batch) Slice(offset, length int) *Batch
- type BufferID
- type BufferSet
- type Column
- func ConcatStrings(name string, dt dtype.DataType, parts []*Column, valid bitmap.View) *Column
- func NewBool(name string, bits bitmap.View, valid bitmap.View) *Column
- func NewFixed[T Fixed](name string, dt dtype.DataType, vals []T, valid bitmap.View) *Column
- func NewFixedBuffer(name string, dt dtype.DataType, buf *memory.Buffer, n int, valid bitmap.View) *Column
- func NewFixedOwned[T Fixed](name string, dt dtype.DataType, vals []T, valid bitmap.View) *Column
- func NewList(name string, offs []int32, child *Column, valid bitmap.View) *Column
- func NewNull(name string, dt dtype.DataType, n int) *Column
- func NewString(name string, vals []string, valid bitmap.View) *Column
- func NewStringBuffers(name string, offs, chars *memory.Buffer, n int, valid bitmap.View) *Column
- func NewStringBuffers64(name string, offs []int64, chars *memory.Buffer, valid bitmap.View) *Column
- func NewStringOwned(name string, offs []int32, chars []byte, valid bitmap.View) *Column
- func NewStringParts(name string, offs []int32, chars []byte, valid bitmap.View) *Column
- func NewStruct(name string, fields []*Column, valid bitmap.View) *Column
- func TakeStrings(c *Column, sel []int32, valid bitmap.View) *Column
- func TakeStringsChunked(c *Column, sel []int32, valid bitmap.View, chunk int, ...) (*Column, error)
- func (c *Column) Bools() bitmap.View
- func (c *Column) Buffers() iter.Seq2[BufferID, int64]
- func (c *Column) Child() *Column
- func (c *Column) DType() dtype.DataType
- func (c *Column) Field(name string) (*Column, bool)
- func (c *Column) Fields() []*Column
- func (c *Column) FixedSlice() any
- func (c *Column) IsPayloadFree() bool
- func (c *Column) IsValid(i int) bool
- func (c *Column) Len() int
- func (c *Column) Lists() ListAccessor
- func (c *Column) NBytes() int64
- func (c *Column) Name() string
- func (c *Column) NullCount() int
- func (c *Column) RawChars() []byte
- func (c *Column) RawFixed() []byte
- func (c *Column) RawOffsets() []byte
- func (c *Column) RawOffsets64() []byte
- func (c *Column) Rename(name string) *Column
- func (c *Column) Slice(offset, length int) *Column
- func (c *Column) Strings() (a StringAccessor)
- func (c *Column) Validity() bitmap.View
- func (c *Column) WithDType(dt dtype.DataType) *Column
- func (c *Column) WithValidity(valid bitmap.View) *Column
- type Fixed
- type ListAccessor
- type Primitive
- type Series
- func (s *Series[T]) All() iter.Seq2[T, bool]
- func (s *Series[T]) Column() *Column
- func (s *Series[T]) DType() dtype.DataType
- func (s *Series[T]) Fold[A any](init A, fn func(A, T) A) A
- func (s *Series[T]) Get(i int) (T, bool)
- func (s *Series[T]) Len() int
- func (s *Series[T]) Map[U any](fn func(T) U) []U
- func (s *Series[T]) MapErr[U any](fn func(T) (U, error)) ([]U, error)
- func (s *Series[T]) Name() string
- func (s *Series[T]) NullCount() int
- func (s *Series[T]) Values() []T
- type StringAccessor
Constants ¶
This section is empty.
Variables ¶
var CheckNonNullable = true
CheckNonNullable is the fifth boundary check: a column whose field says Nullable: false must actually hold no nulls. It is on.
On in production since step 98 ¶
It was on only in test binaries (audit.md I24), so a null in a column declared non-nullable was ErrInternal in the suite and silent everywhere else — and the optimizer acts on the declaration, below. The reason given was cost: the four checks in NewBatch are O(columns), and this one may cost a popcount per column, because bitmap.Builder.Finish never returns the no-storage all-set form, so a scanned or concatenated column carries a materialised bitmap even when nothing is null and the O(1) fast path misses exactly there.
Measured (BenchmarkNonNullableCheck): 76 ns per non-nullable column per batch of 8192 rows with a materialised bitmap — 865 ns against 103 ns for ten columns — next to microseconds for any kernel that reads the batch. A column with no stored bitmap takes the O(1) path and costs nothing.
It stays a package var so a benchmark can measure it off. Never change it during a query.
What a violation means ¶
Field.Nullable is a permission — "a non-nullable field is one where the engine may skip validity handling entirely" — and the optimizer already acts on it: rule_simplify's boolIdentity folds `x AND lit(false)` to `lit(false)` EXACTLY when x is declared non-nullable. So a lie here is a potentially unsound rewrite, not a cosmetic mismatch, and it will not show up in the final frame of the query that told it.
var CheckTimeRange bool
CheckTimeRange turns on the sixth boundary check: a Time column's ticks must lie in [0, 24h).
The invariant was written down and enforced nowhere ¶
dtype says TypeTime is "time of day since midnight, int64 in the type's unit". That sentence was the whole of the enforcement. Nothing asserted it at construction, in the resolver, in the kernel, or in any cast, writer or spill path — and the consequence was not an error but a contradiction: `23:00 + 2h` rendered as 01:00:00 and sorted AFTER 23:00:00, while comparing unequal to a genuine 01:00:00 that rendered identically to it.
Fixing the two producers — the arithmetic kernel and Cast — is a point fix. This is what makes the invariant total, and it earned its keep immediately: it fired on code already checked in, which is how the Cast relabel was found.
A toggle for the same reason CheckNonNullable is one, and with a worse constant: it is O(rows) per Time column rather than O(columns). Set it once from a test binary's init, never during a query.
var MaxListElements int64 = math.MaxInt32
MaxListElements is the most elements one List column holds. Its offsets are 32-bit, as a String's are, and past this they wrapped silently: two one-row lists of 2^30 elements concatenated to a second row spanning [2^30, -2^31), with no error (step 99). A variable only so a test can lower it.
var MaxStringBytes int64 = math.MaxInt32
MaxStringBytes is the most character data a String column of 32-bit offsets holds. Past it they wrapped silently, reading other rows' bytes or slicing at a negative offset (audit.md S24). Since step 368 a column built whole, by NewString or by the combiners every Collect, join and sort ends in (ConcatStrings, TakeStrings), takes 64-bit offsets past it instead, and a column built from a reader's 32-bit offsets refuses. A variable only so a test can lower it.
Functions ¶
func MustValues ¶
MustValues is Values for call sites that have already checked the dtype.
func Reinterpret ¶
Reinterpret exposes a buffer as a typed slice. Used by kernels that allocate an output buffer and write into it directly.
func Values ¶
Values returns the column's fixed-width payload reinterpreted as []T.
Zero-copy: the slice aliases the column's buffer. It must not be mutated — columns are shared.
T must match the column's PHYSICAL type: a Datetime column is read as []int64, a Date column as []int32. Reading through the physical type is what lets one integer kernel serve every type that stores integers.
Types ¶
type Batch ¶
type Batch struct {
// contains filtered or unexported fields
}
Batch is a horizontal slice of a frame: N equal-length columns with a schema.
It is the unit that flows between physical operators. Batch size is a tuning knob rather than a semantic one — the same query must produce the same answer at any batch size, and the test suite checks that by varying it.
func NewBatch ¶
NewBatch builds a batch, checking that the columns match the schema in count, order, name and type.
The check is not paranoia: a mismatch here means the evaluator produced something other than what the plan's schema resolution promised, which is exactly the class of bug that would otherwise surface as corrupted output far downstream. Failing at the boundary makes it a one-line diagnosis.
func NewBatchRows ¶
NewBatchRows builds a batch with an explicit row count.
It exists for the zero-column case — a frame can legitimately have rows but no columns after `Select()` with nothing — but six operators also use it with populated columns, because it is the constructor that takes a row count: explode, unnest, unpivot, hstack, the nothing-selected filter path and the concat-of-zero- batches path.
It now runs every check NewBatch runs ¶
It used to run NONE of the four structural ones. Step 49 extended the nullability check into it with the argument that a check living only in NewBatch "would be blind to a third of the engine and blind in exactly the newest operators" — and that argument applies verbatim to name, type, column count and row length, which were left behind. So all five run here now.
It PANICS rather than returning an error because this signature has no error to return, and a batch whose columns disagree with its schema is an invariant violation rather than a condition a caller can handle. Every one of these is an Internalf: they are unreachable from any user input.
The explicit row count is still honoured rather than re-derived, because the zero-column case is the whole reason this constructor exists — checkShape reports 0 for it, and a frame with rows and no columns must keep its height.
type BufferID ¶
type BufferID = *byte
BufferID identifies one allocation by the address of its first byte.
Why identity rather than a sum ¶
Columns share payloads. Rename shares one ("O(1): the payload is shared"), so does WithDType, so does the character buffer of a sliced String column, and every pipeline breaker retains the source's own batches rather than copies. A naive per-column sum therefore over-reports, sometimes by a large factor — and an over-reporting memory budget fires early and unpredictably, which is worse than having no budget at all.
It is a *byte rather than a uintptr on purpose. A real pointer stays meaningful to the garbage collector, so an identity held in a ledger can never be recycled by a later allocation while the entry is still live. That holds because every payload is Go heap: Arrow import copies rather than adopting C Data memory, whose addresses the collector knows nothing about.
type BufferSet ¶
type BufferSet struct {
// contains filtered or unexported fields
}
BufferSet accumulates the distinct allocations of any number of columns.
The zero value is ready to use.
type Column ¶
type Column struct {
// contains filtered or unexported fields
}
Column is one named, typed, nullable run of values.
Exactly one payload representation is populated, determined by dt:
fixed-width types → fixed (a flat values buffer)
Bool → bits (a packed bitmap, NOT one byte per value)
String, Binary → offs + chars (int32 offsets plus a character buffer),
or offs64 + chars past MaxStringBytes (step 368)
List → offs + child (the same offsets, into a child COLUMN)
Struct → fields (N COLUMNS, no payload buffer at all)
Bool having a distinct representation is not an accident of implementation: a Boolean column has TWO bitmaps, values and validity, and keeping them the same shape is what makes `null > 5` expressible as "value bit 0, validity bit 0".
List reuses offs rather than inventing a second offsets field, so the n+1 entries with a leading zero — and everything that already knows that shape, Slice included — carry over unchanged. What differs is only what the offsets index INTO: a flat character buffer for String, a whole Column for List, which is what lets the element be any type the child can be, nested lists included.
A Struct's children are a DIFFERENT relationship and get their own field. A list has one child of some other length that its offsets index into; a struct has N children each exactly as long as the struct, addressed by the same row number and sharing one validity bit. Storing both in one slice would mean every use site needed a comment saying which meaning applied.
func ConcatStrings ¶ added in v0.4.0
ConcatStrings builds one String or Binary column of type dt from parts, end to end, and validity valid: each part's live characters copied once, its offsets shifted by the characters before it.
Every Collect ends in one of these per String column, serially. It used to build a Go string per row and hand the slice to NewString, which copied every byte a second time: 0.37 s of h2o j3's 0.58 s at 2M rows (step 101's profile).
A part is a derived column as often as not — Column.Slice keeps the whole character buffer and re-windows the offsets — so what it contributes is the range its own offsets cover, not its buffer.
Past MaxStringBytes its offsets are 64-bit (step 368), whatever its parts' are.
func NewBool ¶
NewBool builds a Boolean column from a packed payload bitmap.
The length comes from the payload, and a disagreeing validity is a panic ¶
It used to be taken from the payload SILENTLY. `NewBool(name, bitmap.Not(c.Bools()), c.Validity())` over a Null column handed this a zero-length payload and an n-bit validity, and got back a Bool column of length 0 carrying n validity bits — a structurally impossible value, returned with no error, from a kernel that had satisfied its type contract. It surfaced three layers away as "expression produced 0 rows for a 3-row batch; this is a bug in ursus".
Two integer compares against the cost of building the bitmap, so there is no case for hiding this behind CheckNonNullable's debug flag. A panic rather than a clamp for the reason NewBatchRows gives for the same shape of violation: this signature has no error to return, and the caller cannot handle it — two buffers of different lengths are not a condition, they are a bug in whoever built them.
func NewFixed ¶
NewFixed builds a fixed-width column over vals.
The values slice must alias a buffer obtained from arrowx so that alignment and lifetime are right; NewFixedBuffer is the lower-level entry point.
T must be dt's physical type, and that used to go unchecked ¶
Values checks on the way OUT — its doc spells out why a width-only check is not enough: "Int64 and Float64 are both 64 bits, so a width-only check lets Values[float64] succeed on an Int64 column". Nothing checked on the way IN, and that asymmetry cost a whole defect: a cumulative sum handed this a []float64 under a Duration dtype, and because the two are the same width, neither the dtype nor the row count could disagree. One hour read back as about 152 years.
A panic rather than an error for the reason NewBool gives for the same shape of violation: this signature has no error to return, and a caller cannot handle it — values of the wrong type are not a condition, they are a bug in whoever built them. The pun this does NOT forbid is the legitimate one: Physical() maps Date to Int32, the temporal types to Int64, Enum to Uint32 and Decimal to Int128, so every column stored as a different type than it declares still passes.
func NewFixedBuffer ¶
func NewFixedBuffer(name string, dt dtype.DataType, buf *memory.Buffer, n int, valid bitmap.View) *Column
NewFixedBuffer builds a fixed-width column directly over an existing buffer. This is the path a kernel takes: allocate the output buffer, write into it, and wrap it with no copy.
func NewFixedOwned ¶ added in v0.6.0
NewFixedOwned is NewFixed taking vals rather than copying it (step 163): the column wraps vals' allocation, all of it, and the caller hands vals over and never writes it again. A reader that decodes a batch's values into a fresh slice uses it, so the values are written once.
func NewList ¶ added in v0.3.0
NewList builds a List column from offsets and a child column holding every element of every row, concatenated.
offs must have n+1 monotonic entries, and a column CONSTRUCTED here starts at zero with offs[n] == child.Len().
That is a property of this constructor and NOT the invariant of a List column, and this doc used to say otherwise — "offs[0] == 0 and offs[n] == child.Len(), the ordinary Arrow invariant". Column.Slice produces Lists violating both clauses by design, and two kernels believed this sentence over that one: listReduce credited every element before a slice to its first row, and concatList concatenated whole children at shifted offsets. Neither is Arrow's rule either — Arrow reads offsets as absolute and a sliced array's first offset is routinely non-zero. The invariant is stated once, on ListAccessor.Window.
An empty list and a null list are not the same row ¶
Both leave the offset unmoved: offs[i] == offs[i+1] either way. Only the validity bit tells them apart, exactly as it does for an empty string against a null string. Getting it wrong is not a crash — it is one row quietly holding the wrong answer, and the failure surfaces wherever the row is finally read rather than where it was built.
The type is DERIVED from the child rather than passed in. A List whose declared element type disagrees with the column actually holding the elements would be a lie no caller could detect, and there is no case where the two should differ.
At most MaxListElements elements ¶
Past it the offsets have wrapped, so it is refused rather than built. The caller computed offs as int32 and they are already wrong; the child's length is what can be counted.
func NewNull ¶
NewNull builds an all-null column of the given type and length. It carries no payload buffer, which makes a typed null literal free.
func NewStringBuffers ¶
NewStringBuffers builds a String column over existing offset and character buffers, without copying.
func NewStringBuffers64 ¶ added in v0.6.8
NewStringBuffers64 is NewStringBuffers for 64-bit offsets, len(offs)-1 rows, which a column past MaxStringBytes holds (step 368): a spill file's, read back. It takes offs rather than copying them.
func NewStringOwned ¶ added in v0.6.0
NewStringOwned is NewStringParts taking offs and chars rather than copying them (step 163); the caller hands both over and never writes them again.
func NewStringParts ¶
NewStringParts builds a String column from an offsets slice and a character slice that a reader has already accumulated.
It exists so a decoder can build Arrow's layout DIRECTLY instead of going through []string. The Parquet reader used to append one Go string per value — one heap allocation per row, 2.14M of them for a two-string-column scan of a million rows — and then hand the whole slice to NewString, which copied every byte a second time. Both costs vanish if the decoder appends into offs and chars as it goes, which is what this constructor is for.
offs must have n+1 monotonic entries, and a column CONSTRUCTED here starts at zero with offs[n] == len(chars). That is a property of this constructor, not of every String column: a derived one — any Column.Slice — keeps its offsets absolute, so offs[0] may be non-zero. See ListAccessor.Window for why that sentence matters. The int32 write goes through unsafeData rather than encoding/binary because the buffer is native-endian, which is what every reader of it assumes.
func NewStruct ¶ added in v0.3.0
NewStruct builds a Struct column from its field columns.
Every field must be as long as the struct itself: a struct row IS row i of each field, and the struct's own validity is the only thing they share. That is not checked here — the constructors do not validate — but it is the invariant every reader below relies on.
A null struct is not a struct of nulls ¶
The two hold identical field values (none) and differ only in this column's validity bit, exactly as an empty list differs from a null list. It matters because the difference cannot be recovered afterwards: a struct whose fields all happen to be null is a REAL struct, and inferring "null struct" from "every field is null" turns it into an absent one, silently and for one row in a fixture that does not deliberately contain it.
The type is DERIVED from the fields, for the reason NewList gives: a declared type that disagreed with the columns actually holding the values would be a lie no caller could detect.
func TakeStrings ¶ added in v0.4.0
TakeStrings gathers a String or Binary column by row: row j of the result is row sel[j] of c, and a negative index is an empty row, which valid marks null. Each row's bytes are copied once, into one buffer counted before it is allocated.
Past MaxStringBytes its offsets are 64-bit (step 368), whatever c's are.
func TakeStringsChunked ¶ added in v0.6.7
func TakeStringsChunked(c *Column, sel []int32, valid bitmap.View, chunk int, run func(n int, fn func(k int) error) error) (*Column, error)
TakeStringsChunked is TakeStrings of a long selection, a chunk of rows at a time, the chunks run by run, which may run them at once: each chunk's bytes are counted first, which places every chunk's in the one answer, and then each chunk writes its own offsets and bytes.
func (*Column) Buffers ¶
Buffers yields each distinct allocation the column holds, with its size.
Size is the buffer's CAPACITY, not the bytes in use: the allocation is what is actually held, and an under-filled buffer still occupies all of it.
A column has five slots of its own — fixed, offs, chars and the two bitmaps — so the within-column deduplication below is a linear scan rather than a map. Deduplication ACROSS columns is BufferSet's job.
The scan is over a SLICE rather than a fixed array because a List column also yields its child's buffers, and a child may itself be a List: the count is bounded by nesting depth, not by five. It was a [5]BufferID, which a single list column would have run off the end of.
func (*Column) Child ¶ added in v0.3.0
Child returns a List column's element column, or nil.
Row i occupies child[offs[i]:offs[i+1]], with ABSOLUTE offsets. The child may hold elements belonging to no row of this column at all — a sliced List shares its parent's whole child — so it is not "every row's elements concatenated", and code that treats it that way is wrong on any derived column. Use Lists to walk it, and ListAccessor.Window for the part that belongs to this column.
func (*Column) Field ¶ added in v0.3.0
Field returns the field column with this name, and whether there is one.
func (*Column) Fields ¶ added in v0.3.0
Fields returns a Struct column's field columns, or nil.
The result aliases the column's own slice; callers must not reorder it.
func (*Column) FixedSlice ¶ added in v0.3.0
FixedSlice returns the fixed-width payload as a typed slice inside an any — []int64, []float64 and so on — or nil when the column has no fixed payload.
It exists for GENERIC callers that cannot name the type. Values[T Fixed] is the zero-copy reader, but a caller constrained to something wider than Fixed — the UDF methods are constrained to Literal, which also admits string and bool — cannot instantiate it at all, so without this it would fall back to Series.Get and pay an interface boxing per row.
The returned slice ALIASES the column's buffer, exactly as Values does. Callers must not write to it.
Logical types are returned as their physical storage, which is the same punning Values documents: a Datetime column comes back as []int64.
func (*Column) IsPayloadFree ¶
IsPayloadFree reports whether the column carries no payload buffer at all.
Only NewNull produces one, and it is a genuine value — "which makes a typed null literal free" — but it can only ever be an ANSWER, never an OPERAND. Values refuses it, StringAccessor.Get panics on it, and both are reachable from ordinary queries; kernel.NullColumn exists to build the operand-shaped equivalent.
That rule was previously enforced only by remembering it. A kernel handed a column from an unknown source can now ask.
func (*Column) Lists ¶ added in v0.3.0
func (c *Column) Lists() ListAccessor
Lists returns an accessor over a List column's rows.
func (*Column) NBytes ¶
NBytes returns the total capacity of the distinct allocations the column holds.
func (*Column) RawChars ¶
RawChars returns a String or Binary column's character buffer.
Note it is the WHOLE buffer: a sliced column keeps the original characters and re-windows the offsets, so offsets[0] is not necessarily zero and the live range is offsets[0]:offsets[len].
func (*Column) RawFixed ¶
RawFixed returns the fixed-width payload as bytes, or nil if there is none. The slice aliases the column's buffer and must not be mutated.
func (*Column) RawOffsets ¶
RawOffsets returns a String or Binary column's offset buffer as bytes. Read it as []int32 with Reinterpret; it holds len+1 entries. A column past MaxStringBytes has none, and RawOffsets64 instead.
func (*Column) RawOffsets64 ¶ added in v0.6.8
RawOffsets64 returns a String or Binary column's 64-bit offset buffer as bytes, which a column past MaxStringBytes has in its 32-bit buffer's place (step 368): []int64 with Reinterpret, len+1 entries. Nil where the offsets are 32-bit.
func (*Column) Slice ¶
Slice returns rows [offset, offset+length) as a zero-copy view.
Every payload shape shares rather than copies: Bool and the validity bitmap re-window their views, String and List re-window the offsets and keep the character buffer or child column whole, a Struct slices its fields, and a fixed-width column takes a SUB-BUFFER of the values.
The fixed-width case used to copy, and step 37 is why it stopped ¶
It copied because Column carries no offset into its values buffer, and adding one would put a `+ c.offset` in every kernel's inner loop. It does not need one: memory.SliceBuffer produces a buffer whose base IS the sliced range, so Values[T] — which reads unsafeData(c.fixed.Bytes())[:c.len] from the start of the buffer — sees the right rows without knowing anything happened.
Sharing is what makes splitting an in-memory frame into batches free. memsrc used to hand back one batch however large, so an in-memory frame had one unit of work and WithThreads did nothing; splitting it while Slice copied would have traded that for a full copy of the frame.
Two things make this safe, and both were checked rather than assumed. Nothing writes into an existing column's payload — the only two data.Reinterpret call sites write into a freshly allocated buffer and read offsets for spilling. And a sub-buffer may be aligned to less than 64 bytes, which internal/arrowx's own doc already requires kernels to tolerate: "Kernels must treat 64-byte alignment as an optimization hint, never as a precondition." In practice the address is base + offset*width over a 64-byte-aligned base, so it is usually still aligned, and arrowx.IsAligned is what a kernel WOULD ask — no kernel asks today; its only caller in the module is arrowx's own package test.
func (*Column) Strings ¶
func (c *Column) Strings() (a StringAccessor)
Strings returns an accessor for a String or Binary column.
func (*Column) WithDType ¶
WithDType returns a copy reinterpreted as a different logical type with the same physical layout — Int64 to Datetime, say. It does NOT convert values.
func (*Column) WithValidity ¶ added in v0.3.0
WithValidity returns a copy with a different validity bitmap and the same payload.
The values are untouched, so a row that becomes null keeps whatever was stored in its slot — which is exactly the contract every constructor here already has, since a null row's payload is never read.
Unnest is the caller: a field of an absent struct is absent, and that has to be applied to the field on its way out rather than assumed of whoever built the struct.
type Fixed ¶
Fixed is every type that occupies a constant number of bytes in a values buffer — Primitive plus Int128.
The split exists because Int128 is a STRUCT. It is 16 bytes with no pointers, so it stores exactly like any other numeric type and Values/NewFixed/Take work on it unchanged; but `a < b` and `a + b` do not compile for it, so anything using an operator needs Primitive instead and reaches Int128 through i128's methods.
Go's type system enforces this: adding Int128 to a constraint whose functions use operators is a compile error, not a runtime surprise.
type ListAccessor ¶ added in v0.3.0
type ListAccessor struct {
// contains filtered or unexported fields
}
ListAccessor reads one row's element range at a time.
It returns a RANGE rather than a sliced column because slicing per row would allocate per row, and every caller so far — rendering, Take, Concat — wants the bounds so it can copy a span in one move.
func (ListAccessor) Child ¶ added in v0.3.0
func (a ListAccessor) Child() *Column
Child is the column the ranges index into.
func (ListAccessor) Get ¶ added in v0.3.0
func (a ListAccessor) Get(i int) (start, end int32, ok bool)
Get returns the half-open element range for row i, and whether the row is non-null.
The range is ALWAYS the real one, including for a null row — where it is empty, because a null list consumes no elements and leaves the offset unmoved. That matters to any caller walking a column end to end: returning a zeroed range for a null row would collapse every offset after it, and Concat did exactly that before this note existed.
So a present-but-empty list and a null list return the SAME range, and `ok` is the only thing that separates them. That is the distinction, not an accident.
func (ListAccessor) Window ¶ added in v0.3.0
func (a ListAccessor) Window() (lo, hi int32)
Window returns the half-open range of child elements that belong to this column: [offs[0], offs[n]).
This is the List invariant, stated once ¶
Offsets are ABSOLUTE indices into the child. A List built by NewList starts at zero and ends at child.Len(), but that is a property of the constructor, not of a List: Column.Slice keeps the offsets absolute and the child whole — "the offsets are not rebased, so the child stays whole and the slice stays O(1)" — so a derived column has offs[0] > 0 and offs[n] < child.Len(), and the child holds elements that belong to rows this column does not have.
Any code that treats the whole child as this column's elements is therefore wrong on every sliced List, and two kernels were: listReduce credited every element before the window to row 0, and concatList concatenated whole children at shifted offsets. It went unnoticed because every List batch-size test read from Parquet, which builds a fresh column per batch and never produces a sliced one. Measured before the fix, through the public API: Tail(2) then list.sum returned 163 where 100 belonged, and re-lazying a six-row frame at batch size 1 put eight elements in an empty list.
Code that reads rows one at a time through Get is unaffected. Code that works on the child in bulk must restrict itself to Window.
type Primitive ¶
type Primitive interface {
~int8 | ~int16 | ~int32 | ~int64 |
~uint8 | ~uint16 | ~uint32 | ~uint64 |
~float32 | ~float64
}
Primitive is the set of Go types that back a fixed-width column AND support Go's comparison and arithmetic operators.
Generic kernels that write `a < b` or `a + b` must constrain to Primitive.
type Series ¶
type Series[T any] struct { // contains filtered or unexported fields }
Series is a typed view over a Column.
Column owns, Series views ¶
This resolves the "two representations, two code paths" problem the design review flagged. A Series holds no storage of its own: it is a Column plus a resolved accessor, constructed on demand and thrown away freely. The engine only ever handles Columns, so kernels, operators and the evaluator have exactly one type to know about. Series exists purely at the boundary where Go values meet columnar data.
Why generic methods matter here ¶
Before Go 1.27, Map and Fold had to be package-level functions taking the receiver as an argument:
data.Map(data.Map(s, parse), normalise) // reads inside-out
As generic methods they chain in the direction the data flows:
s.Map(parse).Map(normalise)
That difference — nothing more — is a large part of why this library targets Go 1.27.
func TypedColumn ¶
TypedColumn returns a typed view over c, or an error if T does not match the column's physical representation.
func (*Series[T]) Get ¶
Get returns the value at i and whether it is non-null.
The two-value form is deliberate. Boxing nulls into a *T or an Option[T] would cost an allocation per row and would make the values buffer non-contiguous, which is exactly what a columnar library must not do. Validity stays a parallel bitmap and the caller is told about it explicitly.
func (*Series[T]) Map ¶
Map applies fn to every value, producing a plain Go slice.
Nulls yield the zero value of U; use MapErr or All if the distinction matters. This is a generic METHOD, which is what lets it chain.
func (*Series[T]) MapErr ¶
MapErr is Map with a fallible function, stopping at the first error and naming the row it failed on.
type StringAccessor ¶
type StringAccessor struct {
// contains filtered or unexported fields
}
StringAccessor reads a variable-width column.
It is a struct rather than an interface so that the layout choice is resolved once per batch instead of once per value. When Arrow's StringView layout is added it becomes a tagged union here, and no kernel call site changes.
A column past MaxStringBytes has 64-bit offsets (step 368), and Get and Bytes take one branch on which, which the processor predicts for a whole column.
func (StringAccessor) Bytes ¶ added in v0.6.2
func (a StringAccessor) Bytes(i int) []byte
Bytes returns row i as bytes without copying: the result aliases the character buffer, and must not be written.
func (StringAccessor) Get ¶
func (a StringAccessor) Get(i int) string
Get returns row i without copying: the result aliases the character buffer.
Source Files