batch

package
v0.18.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 23, 2026 License: AGPL-3.0 Imports: 16 Imported by: 0

Documentation

Overview

Package batch provides the core columnar data structures for the execution engine.

Index

Constants

View Source 
const (
	TypeBool      = parquet.TypeBool
	TypeInt32     = parquet.TypeInt32
	TypeInt64     = parquet.TypeInt64
	TypeFloat32   = parquet.TypeFloat32
	TypeFloat64   = parquet.TypeFloat64
	TypeString    = parquet.TypeString
	TypeBytes     = parquet.TypeBytes
	TypeTimestamp = parquet.TypeTimestamp
	TypeIPv4      = parquet.TypeIPv4
	TypeIPv6      = parquet.TypeIPv6
	TypeCIDR      = parquet.TypeCIDR
	TypeMAC       = parquet.TypeMAC
	TypePort      = parquet.TypePort
	TypeProtocol  = parquet.TypeProtocol
	TypeDuration  = parquet.TypeDuration
	TypeUUID      = parquet.TypeUUID
	TypeDate      = parquet.TypeDate
	TypeDecimal   = parquet.TypeDecimal
	TypeArray     = parquet.TypeArray
	TypeRow       = parquet.TypeRow
	TypeMap       = parquet.TypeMap
	TypeVector    = parquet.TypeVector
)
View Source 
const DefaultBatchSize = 2048

DefaultBatchSize is the number of rows per batch (2048 for cache-friendly vectorized processing).

View Source 
const ReservoirOwner uint64 = 1

ReservoirOwner is the sentinel ownerID stamped onto every batch minted by a BatchPool (Get, GetForSize, PreWarm). The zero value (ownerID == 0) means the batch is not pool-owned — e.g. the over-size escape hatch in GetForSize or a Detach'd long-lived batch. A non-zero sentinel keeps the zero value unambiguous, matching the Sel==nil / pool==nil "absent" conventions.

Variables

This section is empty.

Functions

func DecodeContainerColumn 

func DecodeContainerColumn(payload []byte, v *Vector, n int) error

DecodeContainerColumn reads a payload written by EncodeContainerColumn into v, which must already have v.Type set (the WSHF schema's type byte) but need not carry any nested structure: child types, ROW field names and the VECTOR dimension all ride in the payload. The payload must be consumed exactly — trailing bytes are a corruption, not slack.

func EncodeContainerColumn 

func EncodeContainerColumn(dst []byte, v *Vector, n int) ([]byte, error)

EncodeContainerColumn appends the payload for rows [0, n) of v to dst and returns the grown slice. v must be a canonical vector: no view indirection, storage exactly n rows wide, ARRAY/MAP offsets starting at 0. (*Vector).NewVectorLike + AppendFrom produces exactly that from any source, which is how the WSHF writer feeds this.

func FormatDate 

func FormatDate(days int32) string

FormatDate formats days-since-epoch as "2006-01-02".

func FormatTimestamp 

func FormatTimestamp(ms int64) string

FormatTimestamp renders epoch MILLISECONDS — the engine's one timestamp unit — the way PostgreSQL renders a `timestamp` (OID 1114): UTC, "2006-01-02 15:04:05", with a fractional part only when the millisecond component is non-zero.

This is the display half of a deliberate split. The COMPUTE half boxes a TIMESTAMP column as a bare int64 (ColRef.Eval, pinned by TestTemporalColumnBoxingUnchanged) because comparison, arithmetic, GROUP BY key serialization, spill codecs and the UPDATE read-modify-write path all read it as a number; Vector.GetValue keeps that same int64 for exactly those consumers. The two halves agree because they are the same value in the same unit — one rendered, one raw — and the conversion between them lives here, applied by renderers that still hold the column's declared type. Formatting inside GetValue instead would push the rendered form into every compute path that shares that boxing.

func FormatValue 

func FormatValue(v any) string

FormatValue formats any value for display, producing SQL-like text for nested types (arrays, rows, maps).

func IsContainerType 

func IsContainerType(t TypeID) bool

IsContainerType reports whether t's WSHF payload is encoded by this codec rather than by a flat per-type arm.

func NewMintOwner 

func NewMintOwner() uint64

NewMintOwner returns a process-unique producer id for MintStamp.Owner.

func PoisonOnRelease 

func PoisonOnRelease() bool

PoisonOnRelease reports whether poison-on-release is armed.

func PoisonedBatches 

func PoisonedBatches() uint64

PoisonedBatches returns the running count of batches poisoned on release.

func SetPoisonOnRelease 

func SetPoisonOnRelease(on bool) bool

SetPoisonOnRelease turns poison-on-release on or off and returns the previous setting, so a caller can restore it with a defer.

It is process-global and affects every pool in the process. Callers that flip it must not run concurrently with unrelated queries whose answers they care about — a gate opens it around one query at a time.

func SyncContainerSchema 

func SyncContainerSchema(b *RecordBatch)

SyncContainerSchema copies the shape a container column's PAYLOAD just revealed back into the batch's schema — VECTOR dimension, ARRAY/MAP element type, ROW field list.

The WSHF schema header records a name and a type byte and nothing else, so a decoded batch's schema says "VECTOR" with dimension 0 and "ROW" with no fields. The vectors themselves come out right (the payload carries the shape), but downstream operators size their OWN output from the schema of the batch they were handed: exec.Sort takes s.schema = b.Schema on its first Consume and then gathers into a vector built from it, which for a dimension-0 VECTOR is a nil Float32Data and a slice-bounds panic in gatherSortVector. That is #397's second face — the receiving operator is handed a shapeless container — and the payload is the authority to fix it with, since it is derived from the data rather than from a plan's guess.

The batch gets its OWN schema slice when there is anything to patch: the decode-ahead reader decodes chunks CONCURRENTLY off one shared schema, so patching in place would be a data race even though every writer would store the same value.

Types

type BatchPool 

type BatchPool struct {
	// contains filtered or unexported fields
}

BatchPool manages reusable RecordBatch allocations with size-class bucketing. Batches are pooled by their schema and row count to avoid allocation on the hot path. Thread-safe for concurrent operator use.

func NewBatchPool 

func NewBatchPool(schema []parquet.Column, batchSize int) *BatchPool

NewBatchPool creates a pool for batches of the given schema and size.

func (*BatchPool) BatchSize 

func (p *BatchPool) BatchSize() int

BatchSize returns the row count this pool is configured for.

func (*BatchPool) Get 

func (p *BatchPool) Get() *RecordBatch

Get returns a batch from the pool, or allocates a new one.

func (*BatchPool) GetForSize 

func (p *BatchPool) GetForSize(numRows int) *RecordBatch

GetForSize returns a batch from the pool reset for the given numRows. If numRows exceeds the pool's batch size, allocates a fresh batch.

func (*BatchPool) PreWarm 

func (p *BatchPool) PreWarm(n int)

PreWarm pre-allocates n batches into the pool. Call before parallel workers start to avoid allocation contention during the scan hot path.

func (*BatchPool) Put 

func (p *BatchPool) Put(b *RecordBatch)

Put returns a batch to the pool for reuse.

type Bitmap 

type Bitmap struct {
	// contains filtered or unexported fields
}

Bitmap is a compact null bitmap using 1 bit per row.

func NewBitmap 

func NewBitmap(n int) Bitmap

NewBitmap creates a new bitmap with the given capacity, all bits set to 1 (non-null).

func NewBitmapAllNull 

func NewBitmapAllNull(n int) Bitmap

NewBitmapAllNull creates a bitmap with all bits cleared (all null).

func (*Bitmap) CopyFrom 

func (b *Bitmap) CopyFrom(src *Bitmap, n int)

CopyFrom copies the first n bits from src into b. Both bitmaps must have capacity for n bits. Uses word-level copy for the bulk and masks the final word.

func (*Bitmap) EnsureLen 

func (b *Bitmap) EnsureLen(n int)

EnsureLen grows the bitmap to at least n bits, defaulting any newly added bits to non-null (1) to match NewBitmap. Existing bits are preserved. Used by append-style builders that grow a column across many source batches instead of pre-sizing to a worst-case capacity.

func (Bitmap) Grow 

func (b Bitmap) Grow(newLen int) Bitmap

Grow returns a bitmap that can hold at least newLen bits, preserving existing data. If the current bitmap is already large enough, it is returned as-is.

All bits in the newly-exposed range [b.len, newLen) are set to 1 (valid). This includes the previously-excess bits of the OLD last word: NewBitmap zeros those as padding, so once Grow brings them into the valid range we must explicitly mark them valid — otherwise they read back as null and silently drop rows from downstream consumers (e.g., HashAggregate routes "null GROUP BY key" rows to strGroupStates while int-keyed Next() emits only intGroupStates, dropping the rows from output).

func (*Bitmap) HasNulls 

func (b *Bitmap) HasNulls() bool

HasNulls returns true if any bit is 0 (null). Short-circuits on the first non-full word, making it O(1) in the common all-valid case. Result is cached — repeated calls are free.

func (*Bitmap) InvalidateCache 

func (b *Bitmap) InvalidateCache()

InvalidateCache forces the next HasNulls() call to rescan the bitmap data. Must be called after modifying bitmap words directly via Words().

func (*Bitmap) IsNull 

func (b *Bitmap) IsNull(i int) bool

IsNull returns true if the bit at position i is 0 (null). Includes bounds checking for safety at API boundaries.

func (*Bitmap) IsNullFast 

func (b *Bitmap) IsNullFast(i int) bool

IsNullFast returns true if the bit at position i is 0 (null). No bounds checking — caller must ensure 0 <= i < b.len. Use in hot loops where the index is known to be valid.

func (*Bitmap) Len 

func (b *Bitmap) Len() int

Len returns the number of bits.

func (*Bitmap) NullCount 

func (b *Bitmap) NullCount() int

NullCount returns the number of null (0) bits.

func (*Bitmap) ResetNonNull 

func (b *Bitmap) ResetNonNull(n int)

ResetNonNull resets the bitmap to all non-null (all bits 1) for n elements, reusing the existing backing slice when capacity allows. This avoids allocation on the batch pool hot path.

func (*Bitmap) SetNull 

func (b *Bitmap) SetNull(i int)

SetNull sets the bit at position i to 0 (null).

func (*Bitmap) SetNullRange 

func (b *Bitmap) SetNullRange(start, count int)

SetNullRange sets bits [start, start+count) to 0 (null) using word-level operations. For runs spanning full 64-bit words, entire words are zeroed in a single assignment instead of 64 individual bit clears.

func (*Bitmap) SetValid 

func (b *Bitmap) SetValid(i int)

SetValid sets the bit at position i to 1 (non-null).

func (*Bitmap) Words 

func (b *Bitmap) Words() []uint64

Words returns the raw uint64 bitmap data for word-level operations.

type BytesColumn 

type BytesColumn struct {
	Offsets []uint32 // len = num_rows + 1
	Data    []byte   // contiguous buffer

	// ShapeOnly marks a column decoded for its SHAPE only: Offsets carry
	// the real per-row byte lengths but Data was never written (the
	// lengths-only scan decode, internal/engine/scan/lengths_decode.go).
	// LENGTH()/octet_length(), IS [NOT] NULL and the empty-string
	// comparisons answer off Offsets and the null mask alone; any attempt
	// to read a VALUE is a planner-analysis bug and panics immediately
	// rather than returning a wrong answer. Copy paths propagate the flag
	// instead of moving bytes that do not exist.
	ShapeOnly bool
}

BytesColumn stores variable-length byte data (strings, binary) with zero per-row allocations using an offset/data layout.

func NewBytesColumn 

func NewBytesColumn(capacity int) BytesColumn

NewBytesColumn creates a new BytesColumn with the given capacity. Pre-allocates offsets for positional access (all offsets start at 0 = empty strings). Data arena is lazily allocated: starts empty and grows on first use. Hot paths (scan BulkSet, gather PreAllocBytes) know the exact size they need, so eager pre-allocation just wastes memclr on unused capacity. For pooled batches, the grown capacity is retained across Reset cycles.

func (*BytesColumn) BulkCopy 

func (dst *BytesColumn) BulkCopy(dstOff int, src *BytesColumn, srcOff, count int)

BulkCopy copies a contiguous range [srcOff, srcOff+count) from src into dst at [dstOff, dstOff+count). Uses a single Data append + offset arithmetic instead of per-element Set calls, reducing memmove overhead for batch merging.

func (*BytesColumn) BulkSet 

func (bc *BytesColumn) BulkSet(dstOffset int, srcData []byte, srcOffsets []uint32, n int)

BulkSet copies a contiguous block of byte array data into the column, computing offsets from the source offset array. This replaces n individual Set calls with a single bulk append + offset arithmetic, reducing memmove overhead for Parquet page loading.

func (*BytesColumn) Len 

func (bc *BytesColumn) Len() int

Len returns the number of values.

func (*BytesColumn) LengthAt 

func (bc *BytesColumn) LengthAt(i int) int

LengthAt returns the byte length of row i without reading the value. It is the only value-shaped accessor valid on a shape-only column, and it mirrors Value's defensive handling of the descending-offset hazard.

func (*BytesColumn) MemBytes 

func (bc *BytesColumn) MemBytes() int64

MemBytes returns the heap bytes consumed by the offset and data slices.

Offsets are sized by len (the logical rows+1), but the data arena is sized by cap: a pooled BytesColumn retains its grown arena capacity across Reset cycles (see NewBytesColumn), so cap(Data) is the true resident footprint. This is the honest byte count that replaces the b.Len*48 estimate in EstimateBatchBytes.

func (*BytesColumn) PreAllocBytes 

func (bc *BytesColumn) PreAllocBytes(n int)

PreAllocBytes ensures the Data arena has at least n bytes of capacity. Use this when the expected total byte size is known (e.g., from Parquet metadata) to avoid reallocations during sequential Set calls.

func (*BytesColumn) Reset 

func (bc *BytesColumn) Reset()

Reset clears the bytes column for reuse.

func (*BytesColumn) ResetForWrite 

func (bc *BytesColumn) ResetForWrite(n int)

ResetForWrite resizes the column to hold exactly n values and clears it for a fresh sequential write, retaining the data arena's capacity. Callers that know the total byte size still call PreAllocBytes afterwards; once the arena has reached its high-water mark that call becomes a no-op instead of a fresh multi-hundred-KB span. See (*Vector).ResetForWrite.

func (*BytesColumn) Set 

func (bc *BytesColumn) Set(i int, val []byte)

Set writes a value at positional index i. The BytesColumn must have been created with NewBytesColumn(capacity >= i+1). Values must be set in order (i = 0, 1, 2, ...) because later offsets depend on prior data length.

func (*BytesColumn) SetFrom 

func (dst *BytesColumn) SetFrom(di int, src *BytesColumn, si int)

SetFrom copies a single value from src at position si into dst at position di. Combines Value + Set into one call, avoiding the intermediate slice creation and reducing function call overhead in gather loops. Values must be set in order (di = 0, 1, 2, ...) because later offsets depend on prior data length.

func (*BytesColumn) SetString 

func (bc *BytesColumn) SetString(i int, val string)

SetString writes a string value at positional index i. Same contract as Set (sequential i), but takes a string: `append(dst, s...)` copies straight out of the string, where Set's callers had to materialize a []byte(s) conversion first — one heap allocation per row on the string-producing projection paths.

func (*BytesColumn) StringValue 

func (bc *BytesColumn) StringValue(i int) string

StringValue returns the string at position i.

func (*BytesColumn) UnsafeStringValue 

func (bc *BytesColumn) UnsafeStringValue(i int) string

UnsafeStringValue returns a zero-copy string view of the value at position i. The returned string shares the BytesColumn's backing buffer and is only valid while the BytesColumn is not modified or recycled. Use for transient comparisons in filter/sort kernels where the string is consumed immediately.

func (*BytesColumn) Value 

func (bc *BytesColumn) Value(i int) []byte

Value returns the byte slice at position i.

Defensive against a gather-output hazard: when HashJoin's gatherBuildVector skips unmatched rows without calling BytesData.SetFrom, the destination Offsets may end up with Offsets[i+1] == 0 while Offsets[i] > 0, producing a malformed descending pair. Treat this as empty rather than panicking; the null bitmap alongside the column records the "no value" state authoritatively, and downstream filter / projection kernels already consult it.

type DecimalColumn 

type DecimalColumn struct {
	Data  []Int128
	Scale int // number of decimal places
}

DecimalColumn stores an array of Int128 values for DECIMAL vectors.

func NewDecimalColumn 

func NewDecimalColumn(capacity, scale int) DecimalColumn

NewDecimalColumn creates a new decimal column with the given capacity and scale.

type GlobalPool 

type GlobalPool struct {
	// contains filtered or unexported fields
}

GlobalPool provides shared batch pooling across operators with the same schema. This avoids each operator maintaining its own pool and improves reuse when multiple operators in a pipeline share a schema.

func NewGlobalPool 

func NewGlobalPool() *GlobalPool

NewGlobalPool creates a new global pool.

func (*GlobalPool) ForSchema 

func (gp *GlobalPool) ForSchema(schema []parquet.Column, batchSize int) *BatchPool

ForSchema returns the pool for the given schema and batch size. Creates one if it doesn't exist yet.

type Int128 

type Int128 struct {
	Hi int64  // upper 64 bits (signed)
	Lo uint64 // lower 64 bits (unsigned)
}

Int128 is a 128-bit signed integer used for DECIMAL storage. Values are stored as scaled integers: DECIMAL(10,2) value 123.45 → 12345.

func Int128From 

func Int128From(v int64) Int128

Int128From creates an Int128 from an int64 value.

func Int128FromFloat64 

func Int128FromFloat64(f float64, scale int) Int128

Int128FromFloat64 converts a float64 to Int128 with the given scale. For example, Int128FromFloat64(123.45, 2) → Int128 representing 12345.

func ParseDecimalString 

func ParseDecimalString(s string, scale int) Int128

ParseDecimalString parses a decimal string like "123.45" into an Int128 with the given scale.

func (Int128) Add 

func (d Int128) Add(other Int128) Int128

Add returns d + other.

func (Int128) BigInt 

func (d Int128) BigInt() *big.Int

BigInt returns the value as a big.Int: Hi x 2^64 + Lo, exactly. Used on the paths where an Int128 is too narrow to hold an intermediate result.

func (Int128) Equal 

func (d Int128) Equal(other Int128) bool

Equal returns true if d == other.

func (Int128) FormatDecimal 

func (d Int128) FormatDecimal(scale int) string

FormatDecimal formats the Int128 as a decimal string with the given scale.

func (Int128) IsNegative 

func (d Int128) IsNegative() bool

IsNegative returns true if the value is negative.

func (Int128) IsZero 

func (d Int128) IsZero() bool

IsZero returns true if the value is zero.

func (Int128) Less 

func (d Int128) Less(other Int128) bool

Less returns true if d < other (signed comparison).

func (Int128) MulPow10 

func (d Int128) MulPow10(n int) (Int128, bool)

MulPow10 returns d x 10^n and reports whether the EXACT product fits in Int128. It never returns an approximation: a false second result means the caller must take a wider path, not that the first result is close.

Rescaling one operand up to the other's scale is how two DECIMALs of different scale are compared exactly (kernel.CompareDecimalAt), which matters at a sort-merge join key where the comparator decides EQUALITY: float64 rescaling makes 9007199254740993 and 9007199254740992.0 the same number and emits a join row for a pair that does not match.

func (Int128) Neg 

func (d Int128) Neg() Int128

Neg returns the negation of the value.

func (Int128) Sub 

func (d Int128) Sub(other Int128) Int128

Sub returns d - other.

func (Int128) ToFloat64 

func (d Int128) ToFloat64(scale int) float64

ToFloat64 converts an Int128 decimal value to float64 using the given scale.

func (Int128) ToInt64 

func (d Int128) ToInt64() int64

ToInt64 returns the unscaled int64 value. Only valid if the value fits.

type MintStamp 

type MintStamp struct {
	Owner uint64
	Seq   uint64
}

MintStamp records WHICH producer minted a batch's storage and WHICH issue of that storage this is. It exists so a producer that hands storage out and takes it back — today the scan row-group backing pool, docs/design/scan-output-backing-reuse.md — can recognize its own batch at the release edge WITHOUT keeping a reference to it.

A registry of outstanding batches keyed by pointer is the obvious implementation and the wrong one: it is a strong reference the GC cannot collect and the memory ledger cannot see. A consumer with no release edge, or a batch the pipeline simply drops, pins whole decoded row groups (~280 MB each at SF100) for the producer's lifetime, and any bound on the registry's SIZE silently turns reuse off instead. The stamp inverts the direction: the batch points at nothing, the producer holds nothing, and identity survives in a pair of integers the batch carries.

Owner is a process-unique producer id from NewMintOwner. Zero means unstamped — what a WSHF shuffle chunk, a row-based fallback batch or any batch from a different producer carries — and a release edge must treat it as foreign, because adopting it would create a second owner for storage somebody else recycles.

Seq is bumped on every re-issue of the SAME storage. A release names the Seq it was handed, so a stale release from a previous generation (a retire that fired twice around a re-mint) names an older Seq and is ignored: re-admitting a LIVE backing to the free list would give two decoders one buffer, the one failure this design must not have.

The stamp is written by the producer while it owns the batch exclusively and read at the release edge; the producer's own publication edge (the decode ring's channel, the dispenser's channel send) and the pool mutex order every access.

func (MintStamp) Valid 

func (m MintStamp) Valid() bool

Valid reports whether the stamp names a producer.

type RecordBatch 

type RecordBatch struct {
	Columns []*Vector
	Schema  []parquet.Column
	Len     int
	Sel     []uint32 // selection vector: indices of active rows (nil = all rows active)
	// contains filtered or unexported fields
}

RecordBatch is the unit of data flowing between operators.

func FromRows 

func FromRows(schema []parquet.Column, rows []map[string]any) *RecordBatch

FromRows creates a RecordBatch from row-oriented data.

This is the ROW→BATCH boundary, and the one place where the two shapes of a MAP meet: the parquet row reader produces a Go map (and the writer consumes one) while the vector stores MAP as ARRAY(ROW("key","value")). Nothing converted between them, so every scan that fell back to the row reader — every query on a table carrying a nested column — handed Vector.SetValue a map it rejects and died on the scan worker (#393).

The conversion belongs HERE rather than in SetValue's MAP arm. SetValue cannot tell a MAP's row shape from a ROW's box (both are map[string]any), so accepting one there would silently reshape a ROW written into a mis-derived MAP vector — a live defect on the stage DAG (#397) that the #361 guard is currently the only thing reporting. At this boundary the context is unambiguous: the value came from the row reader and the catalog says the column is a MAP.

func NewRecordBatch 

func NewRecordBatch(schema []parquet.Column, numRows int) *RecordBatch

NewRecordBatch creates a new record batch with the given schema and row count.

func (*RecordBatch) ActiveLen 

func (b *RecordBatch) ActiveLen() int

ActiveLen returns the number of active rows (respecting selection vector).

func (*RecordBatch) ColumnByName 

func (b *RecordBatch) ColumnByName(name string) *Vector

ColumnByName returns the vector for the named column, or nil if not found.

func (*RecordBatch) ColumnIndex 

func (b *RecordBatch) ColumnIndex(name string) int

ColumnIndex returns the index of the named column, or -1.

func (*RecordBatch) Compact 

func (b *RecordBatch) Compact() *RecordBatch

Compact materializes the selection vector into a contiguous batch using the typed nested-aware value copier. Returns the batch unchanged when no selection vector is set.

Was previously a hand-rolled per-type switch whose ROW case wrote null child rows via SetNull alone — never advancing a string child's offset slot — so every later row in that child read back as concatenated garbage (same bug class as the windowCopyVectorRange nullable-BYTES fix; regression test TestCompact_RowChildNullableString).

func (*RecordBatch) Detach 

func (b *RecordBatch) Detach()

Detach claims ownership of the batch: Release() becomes a no-op so no pool can recycle it, and the claim is recorded on the batch AND on every column vector so a producer that reuses vector backing across calls surrenders it (see (*Vector).Claim). Call it from anything that keeps a batch — or anything pointing into its column storage — past the call that handed it over: the hash-join build, Sort, Window, the collect sinks, the spillable collector, partitioned aggregation's per-partition views.

The per-column claim is what makes the contract hold through a derived batch: ColumnPrune and the set-op emitter mint a NEW RecordBatch over the same *Vector pointers, so a consumer that detaches the derived batch would otherwise leave the producer of the original believing nobody kept it.

func (*RecordBatch) DetachPool 

func (b *RecordBatch) DetachPool()

DetachPool severs only the pool link, WITHOUT claiming the batch or its columns. It is for the one caller whose reference is transitive rather than independent: the hash-join late-materialization emitter, whose output views read the input's vectors, so the input must not be recycled underneath them by a concurrently-running source — but whose views die with its own output batch, whose consumer's Detach (if any) propagates the claim through Vector.Base anyway. Anything that genuinely KEEPS a batch calls Detach.

func (*RecordBatch) EnsureCapacity 

func (b *RecordBatch) EnsureCapacity(n int)

EnsureCapacity grows every column so positions [0, n) are addressable for in-place writes, preserving existing data, and sets Len to n. Used by append-style builders (e.g. the hash-join per-partition accumulator) that grow a batch across many source batches rather than pre-sizing to a worst-case capacity. See Vector.EnsureLen for per-type behavior and the nested-type caveat.

func (*RecordBatch) FlattenColumn 

func (b *RecordBatch) FlattenColumn(i int)

FlattenColumn materializes a single column if it is a view. Use for column-granular consumers (a filter touching one column) so the remaining columns stay lazy.

func (*RecordBatch) FlattenViews 

func (b *RecordBatch) FlattenViews()

FlattenViews materializes every view column in place. Call before retaining a batch past the pipeline's per-batch cycle (Sort/Window/sink Consume) or handing it to code that reads typed storage directly.

func (*RecordBatch) HasViews 

func (b *RecordBatch) HasViews() bool

HasViews reports whether any column of the batch is a view vector.

func (*RecordBatch) MemBytes 

func (b *RecordBatch) MemBytes() int64

MemBytes returns the in-memory byte footprint of the batch's column data, summing each Vector's MemBytes(). It deliberately omits operator-specific overhead (e.g. the HashJoin hash-index charge) — that stays at the call site (see hashBuildBytes in package exec). Replaces the per-type estimate that lived in exec.EstimateBatchBytes.

func (*RecordBatch) Mint 

func (b *RecordBatch) Mint() MintStamp

Mint returns the producer stamp on this batch (zero Owner = unstamped).

func (*RecordBatch) Release 

func (b *RecordBatch) Release()

Release returns the batch to its pool if applicable.

Once released, the batch's storage is undefined: the pool may hand it to another operator, which resets it and writes over the same arenas. Anything keeping a value out of the batch past this point must own it — see Detach. Poison mode (see poison.go) makes that undefinedness observable by scribbling the arenas here, which is what the batch-reuse gate compares against a clean run.

func (*RecordBatch) Reset 

func (b *RecordBatch) Reset(numRows int)

Reset clears the batch for reuse, keeping allocated memory.

func (*RecordBatch) Retained 

func (b *RecordBatch) Retained() bool

Retained reports whether a consumer claimed ownership of this batch with Detach.

func (*RecordBatch) RowAt 

func (b *RecordBatch) RowAt(i int) map[string]any

RowAt boxes a single physical row (Sel is not consulted) as a map. For callers that need a few rows out of a large batch — boxing the whole batch via ToRows for a low-selectivity pick is the documented multi-GB heap pattern.

func (*RecordBatch) SetMint 

func (b *RecordBatch) SetMint(m MintStamp)

SetMint stamps the batch for its producing free list. Only the producer calls it, and only while it owns the batch exclusively — at mint, and again to clear the stamp when the storage is taken back. The stamp travels with a VALUE copy of the RecordBatch (e.g. `nb := *b`), so any such copy taken over a scan batch must zero the stamp (`nb.SetMint(batch.MintStamp{})`) or it would alias the parent's pool identity; today the only value copy (internal/engine/exec/partitioned_agg.go's selView) Detaches immediately, which claims the shared columns and trips the release veto regardless.

func (*RecordBatch) ToRows 

func (b *RecordBatch) ToRows() []map[string]any

ToRows converts a RecordBatch to row-oriented data.

type TypeID 

type TypeID = parquet.TypeID

TypeID is an alias for the parquet TypeID used throughout the engine.

type TypeMismatchError 

type TypeMismatchError struct {
	Dst TypeID // the vector's type
	Val any    // the value that had nowhere to go
}

TypeMismatchError reports a write of a value a vector has nowhere to put: SetValue was handed a Go value whose type has no conversion into the vector's storage.

Until #361 such a write VANISHED — the slot kept its zero value and was marked valid — which is the mechanism behind an entire bug family (#310, #327, #331, #333, #345, #353, #361, #371, #372): some declaration upstream picks the wrong vector type, and instead of an error the query answers 0 on every row. The write site is the one seam every one of those defects must cross, so it now panics with this typed value.

The panic carries a query ERROR, not a crash: it implements the exec.FatalEvalPanic contract (Error + FatalEvalError), the same route the expression evaluator uses for a condition with no error return (#347). The pipeline drivers, the worker's task-level recover, the coordinator's and the embedded API's query entries all convert it back into an error — "a wrong type may cost a wrong answer, never the server" (#310) still holds, with the improvement that it now costs an ERROR instead of a wrong answer.

The deliberate non-panics: a nil value is a NULL (WriteNullAt); STRING and BYTES destinations coerce any value through its string form, which is a documented rendering (group keys rely on it); and a PARSE failure of a value-level string (an unparseable IPv4, MAC, UUID) keeps its historical null-ish result — the type was right, the value was not.

func (*TypeMismatchError) Error 

func (e *TypeMismatchError) Error() string

func (*TypeMismatchError) FatalEvalError 

func (e *TypeMismatchError) FatalEvalError() error

FatalEvalError implements the exec.FatalEvalPanic contract, so pipeline drivers convert the panic into a query error instead of a process exit.

type Vector 

type Vector struct {
	Type        TypeID
	Len         int
	Nulls       Bitmap
	BoolData    []bool
	Int32Data   []int32
	Int64Data   []int64
	Float32Data []float32
	Float64Data []float64
	BytesData   BytesColumn
	DecimalData DecimalColumn // for TypeDecimal

	// VECTOR type: fixed-dimension float32 embeddings
	// Row i's vector: Float32Data[i*VectorDim : (i+1)*VectorDim]
	VectorDim int // VECTOR: dimensionality (number of float32 elements per row)

	// Nested type fields (ARRAY, ROW, MAP)
	Offsets    []int32   // ARRAY/MAP: offsets[i]..offsets[i+1] delimit child elements for row i
	Child      *Vector   // ARRAY: flat vector of all element values
	Children   []*Vector // ROW: one vector per field (same length as parent)
	FieldNames []string  // ROW: names of child fields

	// View (dictionary) form: when Base != nil this vector owns no typed
	// storage — logical row i is Base row Indices[i]. Nulls is the view's OWN
	// override bitmap (a null bit marks the row null regardless of Base; a
	// valid bit defers to Base's nullness through Indices). Base is always an
	// owned vector: NewViewVector composes indices when handed a view base, so
	// views never chain. Views are read-only and understood only by the
	// view-aware accessors (GetValue, CopyValueFrom-as-source, Flatten,
	// MemBytes); typed hot-path accessors (GetInt64, Int64Data[i], ...) fail
	// loud on a view because the typed slices are nil. See view.go.
	Base    *Vector
	Indices []uint32
	// contains filtered or unexported fields
}

Vector holds a single column of data. Uses typed slices instead of interface{}.

func NewArrayVector 

func NewArrayVector(length int, elemType TypeID) *Vector

NewArrayVector creates a new ARRAY vector with the given length and element type. The child vector starts with capacity 0; callers append elements and update offsets.

func NewColumnVector 

func NewColumnVector(col parquet.Column, numRows int) *Vector

NewColumnVector creates a single Vector from a Column definition with numRows pre-allocated rows, recursively initializing nested type children — the per-column equivalent of NewRecordBatch for callers that materialize some columns of a batch while emitting others as views.

func NewMapVector 

func NewMapVector(length int, keyType, valueType TypeID) *Vector

NewMapVector creates a new MAP vector. Internally stored as ARRAY(ROW("key","value")).

func NewRowVector 

func NewRowVector(length int, fieldNames []string, fieldTypes []TypeID) *Vector

NewRowVector creates a new ROW/STRUCT vector with named child fields.

func NewVector 

func NewVector(typ TypeID, length int) *Vector

NewVector creates a new vector of the given type and length.

func NewVectorLike 

func NewVectorLike(src *Vector) *Vector

NewVectorLike returns an empty (zero-row) vector with src's type and nested structure: child element types, ROW field names, VECTOR dim and DECIMAL scale. Element storage is appended by AppendFrom.

func NewVectorVector 

func NewVectorVector(length, dim int) *Vector

NewVectorVector creates a new VECTOR column with fixed dimensionality. Storage: Float32Data of length * dim, where row i occupies [i*dim, (i+1)*dim).

func NewVectorWithScale 

func NewVectorWithScale(typ TypeID, length int, scale int) *Vector

NewVectorWithScale creates a new vector with scale metadata (used for DECIMAL).

func NewViewVector 

func NewViewVector(base *Vector, indices []uint32) *Vector

NewViewVector creates a view over base addressing rows through indices. The indices slice is adopted, not copied — callers must not mutate it afterwards. If base is itself a view, the indirection is composed away (newIndices[i] = base.Indices[indices[i]], own-nulls folded), so Base on the returned vector is always an owned vector.

The view starts all-valid: row i's nullness defers to base through indices. Callers that need null-injection (outer-join fill) mark rows with v.Nulls.SetNull(i); those rows' index values are ignored.

func NewViewVectorReuse 

func NewViewVectorReuse(base *Vector, indices, composeBuf []uint32) (*Vector, []uint32)

NewViewVectorReuse is NewViewVector with a caller-owned composition buffer.

When base is itself a view its indirection has to be folded into a NEW index array, and at join-emit widths that array is a large-object allocation per column per output batch — one of the spans the Go heap lock serializes on. Passing composeBuf lets a caller that owns the resulting view's lifetime keep that array across batches. The returned slice is the array the view ADOPTED: it stays live for as long as the view does, so only a caller that knows the view is dead may re-pass it (see probeEmitBuf's ownership rule in package exec). nil comes back when no composition was needed and `indices` was adopted directly.

func (*Vector) AppendFrom 

func (v *Vector) AppendFrom(src *Vector, si int)

AppendFrom appends src[si] to dst, growing dst by one row. Typed copy — no boxing, no string round-trips — recursive for nested types. dst's nested structure must match src's (build it with NewVectorLike).

func (*Vector) Claim 

func (v *Vector) Claim()

Claim marks this vector's storage as retained by a consumer, recursively through the view base it reads from and its nested children. Once claimed a vector is never reused by its producer: the claim is sticky because nothing tracks when the retaining consumer is finished (a Sort holds its input until Finalize), and a wrong answer here is silent data corruption.

func (*Vector) Claimed 

func (v *Vector) Claimed() bool

Claimed reports whether a consumer has claimed this vector's storage.

func (*Vector) CopyValueFrom 

func (v *Vector) CopyValueFrom(di int, src *Vector, si int)

CopyValueFrom writes src[si] into position di of dst using typed access — no boxing, no string round-trips — for every column type including nested ARRAY/MAP/ROW. Fixed-width slots are indexed; variable-length storage (bytes data, array child elements, lazily-created row children) is appended, so writes must be SEQUENTIAL per column (di = 0, 1, 2, ...) — the same contract BytesColumn.Set has always had. Null source rows still advance offsets and children; skipping them would shift every later row.

Destination shape is flexible per level: parent slots may be pre-allocated (NewRecordBatch with full nested schema) or append-built (NewVectorLike); ROW children handle both — indexed writes when pre-allocated, appends when built lazily.

func (*Vector) EnsureLen 

func (v *Vector) EnsureLen(n int)

EnsureLen grows the vector's backing storage so positions [0, n) are addressable for in-place writes (Set / index-assign), preserving existing values, defaulting new fixed-width slots to zero and new null bits to non-null. Backing arrays grow geometrically (via append) for amortized O(1) appends, so an append-style builder can grow a column across many source batches instead of pre-sizing to a worst-case capacity. Sets Len to n.

Scalar, bytes, decimal and fixed-dim VECTOR columns are fully supported. Nested ARRAY/MAP element storage and ROW children are NOT grown here (their element storage is appended by SetValue); callers that build nested columns should use pre-sized batches. The hash-join accumulator path that relies on EnsureLen guards nested schemas to a pre-sized path for this reason.

func (*Vector) Flatten 

func (v *Vector) Flatten()

Flatten materializes a view in place: owned storage is allocated, values are gathered from Base through Indices (own-null rows become nulls), and the view fields are cleared. Aliases of the *Vector see the flattened form. No-op on owned vectors.

func (*Vector) GetBool 

func (v *Vector) GetBool(i int) (bool, bool)

GetBool returns the bool value at position i. Returns (false, false) if null.

func (*Vector) GetFloat32 

func (v *Vector) GetFloat32(i int) (float32, bool)

GetFloat32 returns the float32 value at position i. Returns (0, false) if null.

func (*Vector) GetFloat64 

func (v *Vector) GetFloat64(i int) (float64, bool)

GetFloat64 returns the float64 value at position i. Returns (0, false) if null.

func (*Vector) GetInt32 

func (v *Vector) GetInt32(i int) (int32, bool)

GetInt32 returns the int32 value at position i. Returns (0, false) if null.

func (*Vector) GetInt64 

func (v *Vector) GetInt64(i int) (int64, bool)

GetInt64 returns the int64 value at position i. Returns (0, false) if null.

func (*Vector) GetNumericFloat64 

func (v *Vector) GetNumericFloat64(i int) (float64, bool)

GetNumericFloat64 returns any numeric column value as float64 without boxing. Handles Int32, Int64, Float32, Float64, Timestamp types.

func (*Vector) GetString 

func (v *Vector) GetString(i int) (string, bool)

GetString returns the string value at position i. Returns ("", false) if null.

func (*Vector) GetValue 

func (v *Vector) GetValue(i int) any

GetValue returns the value at position i as an interface{}. Note: returns boxed values for numeric types (unavoidable with any return type). Prefer typed accessors (GetInt64, GetFloat64, etc.) in hot paths.

func (*Vector) IsView 

func (v *Vector) IsView() bool

IsView reports whether the vector is a view (owns no typed storage).

func (*Vector) MemBytes 

func (v *Vector) MemBytes() int64

MemBytes returns the heap bytes resident in this vector's backing storage: the null bitmap plus the typed data slice, recursing into nested children for ARRAY/MAP/ROW. It is the byte-true accounting primitive for the memory tracker, replacing the per-type b.Len*48 estimate in EstimateBatchBytes. It deliberately omits any operator-specific overhead (e.g. the HashJoin hash index charge) — that stays at the call site.

func (*Vector) ResetForWrite 

func (v *Vector) ResetForWrite(n int)

ResetForWrite resizes an OWNED vector to exactly n rows and clears its per-row state — null bits back to non-null, fixed-width slots to zero, the bytes arena emptied — while RETAINING every backing allocation's capacity. A producer that reuses one vector across output batches therefore allocates only when n passes its high-water mark, where a fresh NewColumnVector allocates (and the runtime zeroes) a new span every single batch.

Slots are cleared, not merely resized: the gather loops skip writing null and unmatched rows, so a stale value under a null bit would be a reuse-visible difference from the freshly-zeroed path for any reader that looks at a null slot's value. The clear costs the same memclr `make` was already paying; what is saved is the allocation, i.e. the Go heap lock.

Nested ARRAY/MAP/ROW element storage is append-built and is NOT reset here; callers must not reuse vectors of those types (the join emit path guards them out and mints fresh).

func (*Vector) SetValue 

func (v *Vector) SetValue(i int, val any)

SetValue sets the value at position i from an interface{}. For string/bytes types, values must be set in sequential order (i = 0, 1, 2, ...).

A non-nil value whose type has no conversion into this vector's storage PANICS with *TypeMismatchError (#361) instead of silently keeping the zero value — see that type's doc for the contract and the seams that convert the panic into a query error. nil is a NULL; STRING/BYTES coerce everything through its string form; a parseable-type string that fails to parse (IPv4, MAC, UUID) keeps its historical value-level behavior.

func (*Vector) SetVector 

func (v *Vector) SetVector(i int, vals []float32)

SetVector sets the float32 values for row i of a VECTOR column.

func (*Vector) String 

func (v *Vector) String() string

String returns a debug representation of the vector.

func (*Vector) VectorAt 

func (v *Vector) VectorAt(i int) []float32

VectorAt returns a slice of float32 values for row i of a VECTOR column.

func (*Vector) WriteNullAt 

func (v *Vector) WriteNullAt(di int)

WriteNullAt writes a null into position di of a pre-allocated vector, advancing variable-length bookkeeping (bytes offsets, array offsets, row children) so later sequential writes stay aligned. This is THE null-write primitive for indexed sequential writers — any writer that sets the null bit without advancing these slots corrupts every later row in the column.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.