batch

package
v0.18.52 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 6, 2026 License: AGPL-3.0 Imports: 17 Imported by: 0

Documentation

Overview

Package batch provides the core columnar data structures for the execution engine.

Index

Constants

View Source 
const (
	MaxDecimalPrecision = 38
	MaxDecimalScale     = 38
)

MaxDecimalScale and MaxDecimalPrecision are the widest DECIMAL an Int128 carrier can hold: 10^38 < 2^127, 10^39 is not.

View Source 
const (
	DecimalNegInf = parquet.DecimalNegInf
	DecimalFinite = parquet.DecimalFinite
	DecimalPosInf = parquet.DecimalPosInf
	DecimalNaN    = parquet.DecimalNaN
)
View Source 
const (
	Int32DecimalDigits = 10
	Int64DecimalDigits = 19
)

Int32DecimalDigits and Int64DecimalDigits are how many decimal digits an integer type's whole range needs, which is what an integer operand contributes to a common DECIMAL type: INT32 spans 10 digits, INT64 spans 19 (ADR-0024 item 2 — "an integer is DECIMAL(10,0) / (19,0)").

View Source 
const (
	TypeBool      = parquet.TypeBool
	TypeInt32     = parquet.TypeInt32
	TypeInt64     = parquet.TypeInt64
	TypeFloat32   = parquet.TypeFloat32
	TypeFloat64   = parquet.TypeFloat64
	TypeString    = parquet.TypeString
	TypeBytes     = parquet.TypeBytes
	TypeTimestamp = parquet.TypeTimestamp
	TypeIPv4      = parquet.TypeIPv4
	TypeIPv6      = parquet.TypeIPv6
	TypeCIDR      = parquet.TypeCIDR
	TypeMAC       = parquet.TypeMAC
	TypePort      = parquet.TypePort
	TypeProtocol  = parquet.TypeProtocol
	TypeDuration  = parquet.TypeDuration
	TypeUUID      = parquet.TypeUUID
	TypeDate      = parquet.TypeDate
	TypeDecimal   = parquet.TypeDecimal
	TypeArray     = parquet.TypeArray
	TypeRow       = parquet.TypeRow
	TypeMap       = parquet.TypeMap
	TypeVector    = parquet.TypeVector
)
View Source 
const AvgScaleIncrement = 4

AvgScaleIncrement is how many fractional digits AVG(DECIMAL) adds to its input's scale.

The contract (ADR-0012 item 9, the AVG bullet): AVG over a DECIMAL is EXACT numeric division rounded half-away-from-zero at scale+4, the rule Spark and SQL Server use. PostgreSQL instead picks a scale giving at least 16 significant digits (and never below the dividend's own scale), so the two agree to at least min(both scales) and differ only in how many digits past that they keep. A fixed increment is the honest choice for an engine whose numeric carrier is 128 bits wide: the digits kept do not depend on the magnitude of the answer, so the same query over more rows cannot silently change the scale of its own output column.

View Source 
const DefaultBatchSize = 2048

DefaultBatchSize is the number of rows per batch (2048 for cache-friendly vectorized processing).

View Source 
const MaxDecimalKeyLen = 18

MaxDecimalKeyLen is the widest AppendDecimalKey output: a scale byte, a sign/length byte, and up to 16 magnitude bytes.

View Source 
const ReservoirOwner uint64 = 1

ReservoirOwner is the sentinel ownerID stamped onto every batch minted by a BatchPool (Get, GetForSize, PreWarm). The zero value (ownerID == 0) means the batch is not pool-owned — e.g. the over-size escape hatch in GetForSize or a Detach'd long-lived batch. A non-zero sentinel keeps the zero value unambiguous, matching the Sel==nil / pool==nil "absent" conventions.

Variables

View Source 
var (
	Int128Max = Int128{Hi: math.MaxInt64, Lo: math.MaxUint64}
	Int128Min = Int128{Hi: math.MinInt64, Lo: 0}
)

Int128Max and Int128Min are the widest values the carrier holds. A DECIMAL column never reaches them — 10^38 < 2^127 — but a LITERAL in a predicate is under no such limit, and the two are what a literal outside the range saturates to.

Functions

func AdjustDecimalPrecisionScale added in v0.18.5 

func AdjustDecimalPrecisionScale(p, s int) (int, int)

AdjustDecimalPrecisionScale brings a computed (p,s) back inside the carrier, ADR-0024 item 3's clause: 

when p > 38: intDigits = p - s; s = max(38 - intDigits, min(s, 6)); p = 38

Fraction digits are what the rule spends: the integer part is kept whole for as long as the fraction floor min(s,6) allows, and only yields once keeping it would push the scale below that floor — which is Spark's adjustPrecisionScale exactly, `max(38-intDigits, min(s,6))` being the same function as its `if intDigits + minScale > 38 then minScale else 38 - intDigits`. That is a documented divergence from PostgreSQL in the NUMBER OF DIGITS KEPT, not in the digits themselves: both engines are exact to the digits they keep and agree to min(scale).

Reducing the precision while leaving the scale alone would be the other thing entirely — a range reduction that shrinks the INTEGER part with no floor at all, which is what #552 is about on the set-operation path.

func AppendDecimalKey added in v0.18.1 

func AppendDecimalKey(buf []byte, d Int128, scale int) []byte

AppendDecimalKey appends the canonical, scale-normalized binary key for the DECIMAL value `d` held at `scale`, and returns the extended buffer.

The encoding is 

[normalized scale : 1 byte][sign | magnitude length : 1 byte][magnitude : N bytes]

with the magnitude in minimal-width BIG-endian bytes (no leading zero byte) and the sign in the high bit of the length byte. It is self-delimiting — the length byte says how many follow — which is what lets it sit inside a multi-column key or a nested container element without a separator, exactly like the fixed-width arms it joins.

Minimal width rather than a flat 16 bytes because these keys are stored per group and per build row: a DECIMAL(9,2) price keys in 4 bytes, so the whole key of a single-DECIMAL GROUP BY still fits the 8-byte compact path.

func AvgScale added in v0.18.1 

func AvgScale(inScale int) int

AvgScale returns the scale AVG emits over a DECIMAL input of this scale.

func CanonicalDecimalText added in v0.18.5 

func CanonicalDecimalText(s string) (string, bool)

CanonicalDecimalText is AppendDecimalKey's rule for a value that is already TEXT: the minimal-scale spelling of the number it names, so two renderings of one value produce one key.

A DECIMAL boxes as its rendered text at ITS OWN declared scale, so "12.75" from a DECIMAL(9,2) and "12.7500" from a DECIMAL(18,4) are the same number under two keys — which is how `d IN (SELECT ...)` missed every row whose two sides were declared at different scales, the boxed twin of #474. AppendDecimalKey already answers this for a stored Int128 and a scale (ADR-0012 item 8); this is the same normalization where the carrier is the text itself, which is what a row-at-a-time membership set holds.

ok=false for text that names no number — the caller must then fall back to the raw text, which is still injective for the values it can key.

func CompareDecimalTexts added in v0.18.1 

func CompareDecimalTexts(a, b string) (int, bool)

CompareDecimalTexts orders two numeric TEXTS as the exact numbers they name, returning -1, 0 or +1, and ok=false when either is not a number.

No scale, no carrier, no float: this is the comparison for two values whose only lossless form is their text — a DECIMAL column rendered by FormatDecimal against a literal too wide for the float64 box the compiler built for it (ADR-0012 item 6). It compares the power of ten of the leading digit first and the digit strings after, so it is exact at any width and allocates nothing.

func DecimalAdd added in v0.18.5 

func DecimalAdd(a Int128, aScale int, b Int128, bScale int, outScale int) (Int128, DecimalStatus)

DecimalAdd returns a (at aScale) + b (at bScale) as an unscaled value at outScale, exactly, rounded half away from zero if outScale is narrower than the sum's own scale. DecimalOverflow when the exact result has no Int128.

func DecimalAddAt added in v0.18.5 

func DecimalAddAt(a Int128, aScale int, b Int128, bScale, p, s int) (Int128, DecimalStatus)

DecimalAddAt returns a + b at the declared DECIMAL(p, s).

func DecimalDiv added in v0.18.5 

func DecimalDiv(a Int128, aScale int, b Int128, bScale int, outScale int) (Int128, DecimalStatus)

DecimalDiv returns a (at aScale) / b (at bScale) at outScale, rounded half away from zero, exactly once.

The rounding decision is made from the REMAINDER of a single exact integer division, never from a quotient computed at some wider scale and rounded a second time: 0.1249 rounded to scale 3 is 0.125 and rounded again to scale 2 is 0.13, where the one correct answer is 0.12. One division, one rounding.

DecimalDivByZero and DecimalOverflow are separate answers here, and they are separate SQLSTATEs — 22012 and 22003 — so a caller must branch on the status rather than on "did it produce a value".

func DecimalDivAt added in v0.18.5 

func DecimalDivAt(a Int128, aScale int, b Int128, bScale, p, s int) (Int128, DecimalStatus)

DecimalDivAt returns a / b at the declared DECIMAL(p, s).

func DecimalFitsLimit added in v0.18.5 

func DecimalFitsLimit(v, limit Int128) bool

DecimalFitsLimit reports whether an unscaled value's MAGNITUDE is below the limit DecimalPrecisionLimit returned. A zero limit means the caller had none to give, which admits every value.

With DecimalPrecisionLimit it is the limit-carrying twin of DecimalFitsPrecision(v, p): exec.coerceDecimalVector and the single-process set operation both call it, so the two paths cannot come to different conclusions about the same value (ADR-0024's consequence that the grouped, DAG and local rules become one function).

func DecimalFitsPrecision added in v0.18.5 

func DecimalFitsPrecision(v Int128, p int) bool

DecimalFitsPrecision reports whether the unscaled value v is inside the bound DECIMAL(p, s) declares: |v| < 10^p.

This is the exported home of what exec.decimalFitsPrecision and physical.setOpDecimalFitsPrecision each rebuilt on their own side of a package boundary; both should call this so the single-process and stage-DAG overflow decisions cannot drift. Its two edge conventions are theirs, unchanged: p <= 0 is the codebase's "unconstrained" sentinel and p past 38 is a bound the carrier cannot even express, so both mean "no bound to check" — true, not a rejection of every row.

func DecimalMod added in v0.18.5 

func DecimalMod(a Int128, aScale int, b Int128, bScale int, outScale int) (Int128, DecimalStatus)

DecimalMod returns the remainder of a (at aScale) / b (at bScale) at outScale. The remainder takes the sign of the DIVIDEND, PostgreSQL's rule and Go's; its natural scale is max(aScale,bScale).

A zero divisor is DecimalDivByZero: PostgreSQL raises 22012 for `%` exactly as it does for `/`, so the two ops report it the same way here.

func DecimalModAt added in v0.18.5 

func DecimalModAt(a Int128, aScale int, b Int128, bScale, p, s int) (Int128, DecimalStatus)

DecimalModAt returns a % b at the declared DECIMAL(p, s).

func DecimalMul added in v0.18.5 

func DecimalMul(a Int128, aScale int, b Int128, bScale int, outScale int) (Int128, DecimalStatus)

DecimalMul returns a (at aScale) * b (at bScale) at outScale. The product's natural scale is aScale+bScale; outScale below that rounds half away from zero, exactly once. DecimalOverflow when the exact result has no Int128.

func DecimalMulAt added in v0.18.5 

func DecimalMulAt(a Int128, aScale int, b Int128, bScale, p, s int) (Int128, DecimalStatus)

DecimalMulAt returns a * b at the declared DECIMAL(p, s).

func DecimalResultType added in v0.18.5 

func DecimalResultType(op string, p1, s1, p2, s2 int) (int, int, bool)

DecimalResultType returns the (precision, scale) of a DECIMAL arithmetic result, per ADR-0024 item 3 — the rule SQL Server, Spark and Hive converged on for a finite 38-digit carrier, adopted verbatim so the choice is not wadjet's own: 

e1 + e2, e1 - e2 : p = max(s1,s2) + max(p1-s1, p2-s2) + 1 ; s = max(s1,s2)
e1 * e2          : p = p1 + p2 + 1                        ; s = s1 + s2
e1 / e2          : s = max(6, s1 + p2 + 1) ; p = p1 - s1 + s2 + s
e1 % e2          : p = min(p1-s1, p2-s2) + max(s1,s2)     ; s = max(s1,s2)

op is the SQL operator text, matched case-insensitively: "+", "-", "*", "/", "%" (and "mod" for the function spelling). ok=false for an op with no DECIMAL rule, and then (p,s) is (0,0) — which the caller must NOT use as a type, because 0 is this codebase's "unconstrained" precision and a scale of 0 would silently truncate every fraction digit. It is a "there is no rule here" answer, not a type.

The result always comes back through AdjustDecimalPrecisionScale, so it is a type the carrier can declare. An INTEGER operand is DECIMAL(10,0) or (19,0) — the caller decides which and passes it in.

func DecimalScalar added in v0.18.5 

func DecimalScalar(op DecimalScalarOp, v Int128, inScale, digits, p, s int) (Int128, DecimalStatus)

DecimalScalar executes op over one unscaled carrier at inScale and returns the exact result at the declared DECIMAL(p, s).

digits is round/trunc's second argument and is ignored by the other ops. PostgreSQL's one-argument round and trunc are these at digits = 0.

A NEGATIVE digits rounds or truncates to a power of ten ABOVE the point: `round(1234.56, -2)` is 1200 and `round(1250, -2)` is 1300. It is done in ONE rescale, by reading the value at the wider scale inScale-digits so that rescaling to 0 lands exactly on the power of ten being rounded to. Rounding to scale 0 first and adjusting afterwards would round TWICE and turn 1249 into 1300.

The status is DecimalOverflow when the answer has no Int128 or no place in DECIMAL(p,s) — ADR-0024 item 4's bound, the same one the ...At arithmetic wrappers apply — and never a saturated or wrapped value.

func DecimalSpecialValueError added in v0.18.5 

func DecimalSpecialValueError(s string) error

DecimalSpecialValueError is the refusal a NaN/±Infinity spelling earns when it reaches a caller producing a stored VALUE, and nil for every other text. It reads through to parquet's copy so this package's two value-producing refusal sites and the file writer's cannot answer different SQLSTATEs for the same text (ADR-0024 item 6).

func DecimalSub added in v0.18.5 

func DecimalSub(a Int128, aScale int, b Int128, bScale int, outScale int) (Int128, DecimalStatus)

DecimalSub returns a (at aScale) - b (at bScale) at outScale, under the same contract as DecimalAdd.

func DecimalSubAt added in v0.18.5 

func DecimalSubAt(a Int128, aScale int, b Int128, bScale, p, s int) (Int128, DecimalStatus)

DecimalSubAt returns a - b at the declared DECIMAL(p, s).

func DecodeContainerColumn 

func DecodeContainerColumn(payload []byte, v *Vector, n int) error

DecodeContainerColumn reads a payload written by EncodeContainerColumn into v, which must already have v.Type set (the WSHF schema's type byte) but need not carry any nested structure: child types, ROW field names and the VECTOR dimension all ride in the payload. The payload must be consumed exactly — trailing bytes are a corruption, not slack.

func EncodeContainerColumn 

func EncodeContainerColumn(dst []byte, v *Vector, n int) ([]byte, error)

EncodeContainerColumn appends the payload for rows [0, n) of v to dst and returns the grown slice. v must be a canonical vector: no view indirection, storage exactly n rows wide, ARRAY/MAP offsets starting at 0. (*Vector).NewVectorLike + AppendFrom produces exactly that from any source, which is how the WSHF writer feeds this.

func EqualFoldIdent added in v0.18.30 

func EqualFoldIdent(a, b string) bool

EqualFoldIdent reports whether two identifiers are the same name under the identifier fold. Exported for the resolvers that match a SUFFIX rather than a whole name.

func FoldIdent added in v0.18.30 

func FoldIdent(s string) string

FoldIdent is the identifier fold, ASCII A-Z only — the same rule `plansql.FoldIdent` applies at the lexer, restated here because the engine cannot import the planner. Anything that has to decide whether two column names are ONE name uses it, so the resolver and the join's duplicate detector cannot disagree about that (#731).

func FormatDate 

func FormatDate(days int32) string

FormatDate formats days-since-epoch as "2006-01-02".

func FormatIPv6 added in v0.18.44 

func FormatIPv6(raw []byte) string

FormatIPv6 renders a 16-byte IPv6 address the way PostgreSQL's inet output function does, which is NOT what net.IP.String() does for two families: 

::ffff:10.0.0.1   Go collapses a v4-MAPPED address to its bare dotted quad
                  (`10.0.0.1`), a value the engine itself says the column
                  does not equal — `a = '10.0.0.1'` is false and
                  `a = '::ffff:10.0.0.1'` is true, both correctly (#580).
::1.2.3.4         Go renders a v4-COMPATIBLE address in hex (`::102:304`)
                  where the server prints the embedded quad.

PostgreSQL's rule, measured on 17.11 rather than remembered: take the FIRST longest run of zero 16-bit words of length >= 2 and write it `::`; render the trailing four bytes as a dotted quad when that run starts at word 0 and is either six words long (`::a.b.c.d`) or five words long with word 5 equal to 0xffff (`::ffff:a.b.c.d`). Everything else is lower-case hex groups. The zero-run choice is Go's too, so only the dotted-quad rule differs.

Measured cells (`SELECT '<lit>'::inet` on 17.11): `::ffff:10.0.0.1`, `::ffff:0.0.0.0`, `::ffff:255.255.255.255`, `::1.2.3.4`, `::2`, `::1`, `::`, `0:1::`, `1::`, `2001:db8::1:0:0:1`, `::ffff:0:102:304`, `64:ff9b::102:304`.

The comparison and ordering value is untouched: this is the printed form only, and `net.ParseIP` reads the text back to the same sixteen bytes, which is what kernel.IPv6RowKey and exec.boxedIPv6Compare rely on.

func FormatTimestamp 

func FormatTimestamp(ms int64) string

FormatTimestamp renders epoch MILLISECONDS — the engine's one timestamp unit — the way PostgreSQL renders a `timestamp` (OID 1114): UTC, "2006-01-02 15:04:05", with a fractional part only when the millisecond component is non-zero.

This is the display half of a deliberate split. The COMPUTE half boxes a TIMESTAMP column as a bare int64 (ColRef.Eval, pinned by TestTemporalColumnBoxingUnchanged) because comparison, arithmetic, GROUP BY key serialization, spill codecs and the UPDATE read-modify-write path all read it as a number; Vector.GetValue keeps that same int64 for exactly those consumers. The two halves agree because they are the same value in the same unit — one rendered, one raw — and the conversion between them lives here, applied by renderers that still hold the column's declared type. Formatting inside GetValue instead would push the rendered form into every compute path that shares that boxing.

func FormatValue 

func FormatValue(v any) string

FormatValue formats any value for display, producing SQL-like text for nested types (arrays, rows, maps).

func IntStorageType added in v0.18.12 

func IntStorageType(t TypeID) bool

IntStorageType reports whether a column of type t stores its values in an integer slice (Int32Data or Int64Data), so that a boxed value of that column has an exact int64 STORAGE form.

It is the type set writeIntKeyToColumn and appendTypedIntKey already enumerate in exec, stated once here beside the two boxings it reconciles.

func IsContainerType 

func IsContainerType(t TypeID) bool

IsContainerType reports whether t's WSHF payload is encoded by this codec rather than by a flat per-type arm.

func IsFoldedIdent added in v0.18.30 

func IsFoldedIdent(s string) bool

IsFoldedIdent reports whether s carries no ASCII upper-case letter, i.e. whether it is already in the form the lexer's identifier fold produces — which is how a resolver tells an unquoted reference from a delimited one with nothing but the name.

func KeyStorageInt added in v0.18.12 

func KeyStorageInt(v any, t TypeID) (int64, bool)

KeyStorageInt is the inverse of GetValue's boxing for the types IntStorageType names: it answers the int64 a column of type t STORES for the boxed value v, whichever of that type's legal boxes v happens to be.

It exists because one value of such a type has more than one box in this engine and they are not interchangeable as bytes. GetValue FORMATS the three whose storage is not their text — DATE to "2006-01-02", IPv4 to its dotted quad, MAC to its colon form — while the aggregate's int-keyed SoA path, its migration to the generic map (migrateToGenericMap) and the packed-key unpacking all hand back the raw integer. A group key that serializes whichever box it is given therefore has TWO identities for one value, which is #788: a k-way merge compares bytes, so "14610" and "\n2010-01-01" never combined and every DATE group came out twice with the right total and the wrong grouping.

The answer is a function of (t, value) and of nothing else, so it lives beside GetValue and SetValue — the two boxings it has to agree with — and its parses are literally theirs (parseDateString, net.ParseIP, net.ParseMAC).

ok is false for a box that no column of type t can produce (a string that is not a date/address, a float where an integer is stored). The caller decides what an impossible box means; this never guesses an integer for it, because a wrong integer is a wrong GROUP.

func NameSetNames added in v0.18.47 

func NameSetNames(set map[string]bool, schemaName string) bool

NameSetNames reports whether a set of column REFERENCES names the schema column spelled schemaName — the resolver's rule read in the other direction, for the consumers that hold a SET of wanted names and walk the schema rather than the reverse (a read-set projection, a keep-set prune).

The rule is the same one `ResolveSchemaIndex` applies: byte-exact, or a reference that is ITSELF folded matching case-insensitively. Ambiguity cannot arise within one schema — `catalog.checkDistinctColumnNames` refuses a schema whose columns collide under the fold — so a folded reference names at most one column of it.

func NewMintOwner 

func NewMintOwner() uint64

NewMintOwner returns a process-unique producer id for MintStamp.Owner.

func PoisonOnRelease 

func PoisonOnRelease() bool

PoisonOnRelease reports whether poison-on-release is armed.

func PoisonedBatches 

func PoisonedBatches() uint64

PoisonedBatches returns the running count of batches poisoned on release.

func PoolRetentionVetoes added in v0.18.49 

func PoolRetentionVetoes() uint64

PoolRetentionVetoes returns the running count of pool admissions refused because a consumer had claimed storage the batch aliases.

func ResolveSchemaIndex added in v0.18.30 

func ResolveSchemaIndex(schema []parquet.Column, name string) int

ResolveSchemaIndex is ResolveColumnIndex over a bare schema, for the resolvers that hold one without a batch.

func SetPoisonOnRelease 

func SetPoisonOnRelease(on bool) bool

SetPoisonOnRelease turns poison-on-release on or off and returns the previous setting, so a caller can restore it with a defer.

It is process-global and affects every pool in the process. Callers that flip it must not run concurrently with unrelated queries whose answers they care about — a gate opens it around one query at a time.

func SyncContainerSchema 

func SyncContainerSchema(b *RecordBatch)

SyncContainerSchema copies the shape a container column's PAYLOAD just revealed back into the batch's schema — VECTOR dimension, ARRAY/MAP element type, ROW field list.

The WSHF schema header records a name and a type byte and nothing else, so a decoded batch's schema says "VECTOR" with dimension 0 and "ROW" with no fields. The vectors themselves come out right (the payload carries the shape), but downstream operators size their OWN output from the schema of the batch they were handed: exec.Sort takes s.schema = b.Schema on its first Consume and then gathers into a vector built from it, which for a dimension-0 VECTOR is a nil Float32Data and a slice-bounds panic in gatherSortVector. That is #397's second face — the receiving operator is handed a shapeless container — and the payload is the authority to fix it with, since it is derived from the data rather than from a plan's guess.

The batch gets its OWN schema slice when there is anything to patch: the decode-ahead reader decodes chunks CONCURRENTLY off one shared schema, so patching in place would be a data race even though every writer would store the same value.

func VectorAcceptsText added in v0.18.36 

func VectorAcceptsText(t TypeID) bool

VectorAcceptsText reports whether a vector of this type can STORE a Go string — that is, whether SetValue/SetValueChecked has a string arm for it rather than raising the #361 guard.

It is not a style question: a caller that hands a typed vector the text form of its value gets a stored value for the types listed here and a TypeMismatchError panic for the rest, and the two are far apart. The set operation's UNKNOWN-typed literal arm is the caller that has to know (ADR-0012 item 12): PostgreSQL gives a quoted literal the type the other arms resolve to and parses it with THAT type's input function, so `SELECT c_ipv4 … UNION ALL SELECT '10.0.0.9'` is inet — and the arm's pipeline produces the literal as a STRING box, which reaches the result column's vector unchanged. Where this returns false the planner refuses the shape at PLAN time with 0A000 rather than letting a deterministic parse failure surface mid-execution (on the stage DAG, after three retries of it).

The list is held to what the code actually does by TestVectorAcceptsTextIsWhatSetValueDoes, which tries a string into a vector of EVERY type and compares. Adding a string arm to SetValue without updating this function fails that test, which is the point: this is a statement about SetValue, not a second opinion about it.

Types

type BatchPool 

type BatchPool struct {
	// contains filtered or unexported fields
}

BatchPool manages reusable RecordBatch allocations with size-class bucketing. Batches are pooled by their schema and row count to avoid allocation on the hot path. Thread-safe for concurrent operator use.

func NewBatchPool 

func NewBatchPool(schema []parquet.Column, batchSize int) *BatchPool

NewBatchPool creates a pool for batches of the given schema and size.

func (*BatchPool) BatchSize 

func (p *BatchPool) BatchSize() int

BatchSize returns the row count this pool is configured for.

func (*BatchPool) Get 

func (p *BatchPool) Get() *RecordBatch

Get returns a batch from the pool, or allocates a new one.

func (*BatchPool) GetForSize 

func (p *BatchPool) GetForSize(numRows int) *RecordBatch

GetForSize returns a batch from the pool reset for the given numRows. If numRows exceeds the pool's batch size, allocates a fresh batch.

func (*BatchPool) PreWarm 

func (p *BatchPool) PreWarm(n int)

PreWarm pre-allocates n batches into the pool. Call before parallel workers start to avoid allocation contention during the scan hot path.

func (*BatchPool) Put 

func (p *BatchPool) Put(b *RecordBatch)

Put returns a batch to the pool for reuse, unless the batch still aliases storage a consumer claimed — see retainsClaimedStorage.

type Bitmap 

type Bitmap struct {
	// contains filtered or unexported fields
}

Bitmap is a compact null bitmap using 1 bit per row.

func NewBitmap 

func NewBitmap(n int) Bitmap

NewBitmap creates a new bitmap with the given capacity, all bits set to 1 (non-null).

func NewBitmapAllNull 

func NewBitmapAllNull(n int) Bitmap

NewBitmapAllNull creates a bitmap with all bits cleared (all null).

func (*Bitmap) CopyFrom 

func (b *Bitmap) CopyFrom(src *Bitmap, n int)

CopyFrom copies the first n bits from src into b. Both bitmaps must have capacity for n bits. Uses word-level copy for the bulk and masks the final word.

func (*Bitmap) EnsureLen 

func (b *Bitmap) EnsureLen(n int)

EnsureLen grows the bitmap to at least n bits, defaulting any newly added bits to non-null (1) to match NewBitmap. Existing bits are preserved. Used by append-style builders that grow a column across many source batches instead of pre-sizing to a worst-case capacity.

func (Bitmap) Grow 

func (b Bitmap) Grow(newLen int) Bitmap

Grow returns a bitmap that can hold at least newLen bits, preserving existing data. If the current bitmap is already large enough, it is returned as-is.

All bits in the newly-exposed range [b.len, newLen) are set to 1 (valid). This includes the previously-excess bits of the OLD last word: NewBitmap zeros those as padding, so once Grow brings them into the valid range we must explicitly mark them valid — otherwise they read back as null and silently drop rows from downstream consumers (e.g., HashAggregate routes "null GROUP BY key" rows to strGroupStates while int-keyed Next() emits only intGroupStates, dropping the rows from output).

func (*Bitmap) HasNulls 

func (b *Bitmap) HasNulls() bool

HasNulls returns true if any bit is 0 (null). Short-circuits on the first non-full word, making it O(1) in the common all-valid case. Result is cached — repeated calls are free.

func (*Bitmap) InvalidateCache 

func (b *Bitmap) InvalidateCache()

InvalidateCache forces the next HasNulls() call to rescan the bitmap data. Must be called after modifying bitmap words directly via Words().

func (*Bitmap) IsNull 

func (b *Bitmap) IsNull(i int) bool

IsNull returns true if the bit at position i is 0 (null). Includes bounds checking for safety at API boundaries.

func (*Bitmap) IsNullFast 

func (b *Bitmap) IsNullFast(i int) bool

IsNullFast returns true if the bit at position i is 0 (null). No bounds checking — caller must ensure 0 <= i < b.len. Use in hot loops where the index is known to be valid.

func (*Bitmap) Len 

func (b *Bitmap) Len() int

Len returns the number of bits.

func (*Bitmap) NullCount 

func (b *Bitmap) NullCount() int

NullCount returns the number of null (0) bits.

func (*Bitmap) ResetNonNull 

func (b *Bitmap) ResetNonNull(n int)

ResetNonNull resets the bitmap to all non-null (all bits 1) for n elements, reusing the existing backing slice when capacity allows. This avoids allocation on the batch pool hot path.

func (*Bitmap) SetNull 

func (b *Bitmap) SetNull(i int)

SetNull sets the bit at position i to 0 (null).

func (*Bitmap) SetNullRange 

func (b *Bitmap) SetNullRange(start, count int)

SetNullRange sets bits [start, start+count) to 0 (null) using word-level operations. For runs spanning full 64-bit words, entire words are zeroed in a single assignment instead of 64 individual bit clears.

func (*Bitmap) SetValid 

func (b *Bitmap) SetValid(i int)

SetValid sets the bit at position i to 1 (non-null).

func (*Bitmap) Words 

func (b *Bitmap) Words() []uint64

Words returns the raw uint64 bitmap data for word-level operations.

type BytesColumn 

type BytesColumn struct {
	Offsets []uint32 // len = num_rows + 1
	Data    []byte   // contiguous buffer

	// ShapeOnly marks a column decoded for its SHAPE only: Offsets carry
	// the real per-row byte lengths but Data was never written (the
	// lengths-only scan decode, internal/engine/scan/lengths_decode.go).
	// LENGTH()/octet_length(), IS [NOT] NULL and the empty-string
	// comparisons answer off Offsets and the null mask alone; any attempt
	// to read a VALUE is a planner-analysis bug and panics immediately
	// rather than returning a wrong answer. Copy paths propagate the flag
	// instead of moving bytes that do not exist.
	ShapeOnly bool
}

BytesColumn stores variable-length byte data (strings, binary) with zero per-row allocations using an offset/data layout.

func NewBytesColumn 

func NewBytesColumn(capacity int) BytesColumn

NewBytesColumn creates a new BytesColumn with the given capacity. Pre-allocates offsets for positional access (all offsets start at 0 = empty strings). Data arena is lazily allocated: starts empty and grows on first use. Hot paths (scan BulkSet, gather PreAllocBytes) know the exact size they need, so eager pre-allocation just wastes memclr on unused capacity. For pooled batches, the grown capacity is retained across Reset cycles.

func (*BytesColumn) BulkCopy 

func (dst *BytesColumn) BulkCopy(dstOff int, src *BytesColumn, srcOff, count int)

BulkCopy copies a contiguous range [srcOff, srcOff+count) from src into dst at [dstOff, dstOff+count). Uses a single Data append + offset arithmetic instead of per-element Set calls, reducing memmove overhead for batch merging.

func (*BytesColumn) BulkSet 

func (bc *BytesColumn) BulkSet(dstOffset int, srcData []byte, srcOffsets []uint32, n int)

BulkSet copies a contiguous block of byte array data into the column, computing offsets from the source offset array. This replaces n individual Set calls with a single bulk append + offset arithmetic, reducing memmove overhead for Parquet page loading.

func (*BytesColumn) Len 

func (bc *BytesColumn) Len() int

Len returns the number of values.

func (*BytesColumn) LengthAt 

func (bc *BytesColumn) LengthAt(i int) int

LengthAt returns the byte length of row i without reading the value. It is the only value-shaped accessor valid on a shape-only column, and it mirrors Value's defensive handling of the descending-offset hazard.

func (*BytesColumn) MemBytes 

func (bc *BytesColumn) MemBytes() int64

MemBytes returns the heap bytes consumed by the offset and data slices.

Offsets are sized by len (the logical rows+1), but the data arena is sized by cap: a pooled BytesColumn retains its grown arena capacity across Reset cycles (see NewBytesColumn), so cap(Data) is the true resident footprint. This is the honest byte count that replaces the b.Len*48 estimate in EstimateBatchBytes.

func (*BytesColumn) PreAllocBytes 

func (bc *BytesColumn) PreAllocBytes(n int)

PreAllocBytes ensures the Data arena has at least n bytes of capacity. Use this when the expected total byte size is known (e.g., from Parquet metadata) to avoid reallocations during sequential Set calls.

func (*BytesColumn) Reset 

func (bc *BytesColumn) Reset()

Reset clears the bytes column for reuse.

func (*BytesColumn) ResetForWrite 

func (bc *BytesColumn) ResetForWrite(n int)

ResetForWrite resizes the column to hold exactly n values and clears it for a fresh sequential write, retaining the data arena's capacity. Callers that know the total byte size still call PreAllocBytes afterwards; once the arena has reached its high-water mark that call becomes a no-op instead of a fresh multi-hundred-KB span. See (*Vector).ResetForWrite.

func (*BytesColumn) Set 

func (bc *BytesColumn) Set(i int, val []byte)

Set writes a value at positional index i. The BytesColumn must have been created with NewBytesColumn(capacity >= i+1). Values must be set in order (i = 0, 1, 2, ...) because later offsets depend on prior data length.

func (*BytesColumn) SetFrom 

func (dst *BytesColumn) SetFrom(di int, src *BytesColumn, si int)

SetFrom copies a single value from src at position si into dst at position di. Combines Value + Set into one call, avoiding the intermediate slice creation and reducing function call overhead in gather loops. Values must be set in order (di = 0, 1, 2, ...) because later offsets depend on prior data length.

func (*BytesColumn) SetString 

func (bc *BytesColumn) SetString(i int, val string)

SetString writes a string value at positional index i. Same contract as Set (sequential i), but takes a string: `append(dst, s...)` copies straight out of the string, where Set's callers had to materialize a []byte(s) conversion first — one heap allocation per row on the string-producing projection paths.

func (*BytesColumn) StringValue 

func (bc *BytesColumn) StringValue(i int) string

StringValue returns the string at position i.

func (*BytesColumn) UnsafeStringValue 

func (bc *BytesColumn) UnsafeStringValue(i int) string

UnsafeStringValue returns a zero-copy string view of the value at position i. The returned string shares the BytesColumn's backing buffer and is only valid while the BytesColumn is not modified or recycled. Use for transient comparisons in filter/sort kernels where the string is consumed immediately.

func (*BytesColumn) Value 

func (bc *BytesColumn) Value(i int) []byte

Value returns the byte slice at position i.

Defensive against a gather-output hazard: when HashJoin's gatherBuildVector skips unmatched rows without calling BytesData.SetFrom, the destination Offsets may end up with Offsets[i+1] == 0 while Offsets[i] > 0, producing a malformed descending pair. Treat this as empty rather than panicking; the null bitmap alongside the column records the "no value" state authoritatively, and downstream filter / projection kernels already consult it.

type DecimalColumn 

type DecimalColumn struct {
	Data  []Int128
	Scale int // number of decimal places
}

DecimalColumn stores an array of Int128 values for DECIMAL vectors.

func NewDecimalColumn 

func NewDecimalColumn(capacity, scale int) DecimalColumn

NewDecimalColumn creates a new decimal column with the given capacity and scale.

type DecimalScalarOp added in v0.18.5 

type DecimalScalarOp uint8

DecimalScalarOp names a scalar math function whose DECIMAL result type this package decides. They are the functions that answer a number IN THE SAME DOMAIN as their argument — PostgreSQL's abs/ceil/floor/round/trunc/sign over a numeric all return numeric — as opposed to the transcendental ones (sqrt/exp/ln/power/log), which PostgreSQL also answers in numeric and which wadjet deliberately keeps in float64: an exact fixed-point tower is what those need, and ADR-0012 item 9 already records that class of divergence for STDDEV and friends. 

const (
	// DecimalScalarAbs keeps the argument's type exactly: |v| is a value the
	// same column holds.
	DecimalScalarAbs DecimalScalarOp = iota
	// DecimalScalarCeil and DecimalScalarFloor drop the fraction and may
	// CARRY into a new integer digit — ceil(9.9) is 10 — so the integer part
	// grows by one.
	DecimalScalarCeil
	DecimalScalarFloor
	// DecimalScalarRound rounds half away from zero to `digits` fraction
	// digits and can carry, like ceil.
	DecimalScalarRound
	// DecimalScalarTrunc cuts at `digits` fraction digits and cannot carry.
	DecimalScalarTrunc
	// DecimalScalarSign answers -1, 0 or 1 whatever the argument's width.
	DecimalScalarSign
)

type DecimalSpecialKind added in v0.18.5 

type DecimalSpecialKind = parquet.DecimalSpecialKind

DecimalSpecialKind names one of the three values PostgreSQL's `numeric` has and this carrier does not: NaN and, since PostgreSQL 14, ±Infinity. An Int128 at a fixed scale has no bit pattern for any of them and the parquet DECIMAL annotation has none either (ADR-0024 items 1 and 6).

The constants ARE their rank in PostgreSQL's numeric order, which is a total order rather than IEEE754's: -Infinity below every finite value, Infinity above every finite value, and NaN above Infinity and equal only to itself. So an int comparison of two kinds orders them, and the sign of a non-finite kind says which end of a column's range it sits past.

DecimalSpecialKind, DecimalSpecialText and the numeric-text grammar below live in internal/storage/parquet and are read through from here.

This package IMPORTS that one (batch.Vector is built from parquet.Column), so the lower package is the only place a SINGLE accept-set can sit — the same reason ParseDateDays lives there. The file writer has to classify the text it is about to store exactly as the comparison path classifies the text it is about to compare, or 'NaN' is 22003 on one path and 22P02 on the other and a client branching on the code cannot see past the difference (ADR-0024 items 4 and 6, #647).

func DecimalSpecialText added in v0.18.5 

func DecimalSpecialText(text string) DecimalSpecialKind

DecimalSpecialText reads PostgreSQL's numeric input grammar for NaN and the infinities, and returns DecimalFinite for everything else — including text that names no number at all, which is DecimalTextAt's question, not this one's. See parquet.DecimalSpecialText for the accept-set.

type DecimalStatus added in v0.18.5 

type DecimalStatus uint8

DecimalStatus is why a fixed-point operation did or did not produce a value. It exists because "no answer" has two causes that PostgreSQL reports as two different conditions, and a single bool made them one: a caller writing `if !ok { raise 22003 }` reports a numeric overflow for `x / 0`. 

const (
	// DecimalOK: the first return is the exact value at the requested scale.
	// It fits the Int128 carrier; see the package note above on why that is
	// not the same as fitting a declared (p,s).
	DecimalOK DecimalStatus = iota
	// DecimalOverflow: the exact value has no Int128 (or, from the ...At
	// wrappers, no value at the declared precision). SQLSTATE 22003,
	// numeric_value_out_of_range.
	DecimalOverflow
	// DecimalDivByZero: the divisor is zero, for both / and %. SQLSTATE
	// 22012, division_by_zero.
	DecimalDivByZero
	// DecimalInvalidScale: a negative scale was asked for. Not a user-visible
	// condition — a column's scale is non-negative by DDL — so a caller that
	// sees this has a planner defect to report as an internal error, not a
	// numeric one.
	DecimalInvalidScale
)

func (DecimalStatus) String added in v0.18.5 

func (s DecimalStatus) String() string

String names the status, for the error messages the wiring sites build.

type DecimalType added in v0.18.5 

type DecimalType struct {
	Precision int
	Scale     int
}

DecimalType is a DECIMAL's declared (precision, scale) — the two facts a bare TypeID cannot express. It is the batch-level twin of logical.DecimalMeta and parquet.Column's Precision/Scale pair; the planner converts at its own boundary rather than this package importing either.

func DecimalCommon added in v0.18.5 

func DecimalCommon(in []DecimalType) (DecimalType, bool)

DecimalCommon is the COMMON DECIMAL type of a set of operands (ADR-0024 item 2): the type every one of them can be moved into without dropping a digit it holds. 

scale     = max over the operands
precision = max over the operands of (precision - scale), plus that scale

The scale is the maximum because that is the only choice that moves no value: a narrower one would DROP digits a wider operand holds, which is the truncating half of #533. Precision is then reconstructed from the widest INTEGER part rather than taken as max(precision), because max(precision) is not a bound on the widened values — DECIMAL(18,2) alongside DECIMAL(9,4) needs 16 integer digits at scale 4, i.e. 20, where max(precision) would declare 18 and hand the parquet writer a leaf too small for the value (ADR-0018 §4's encoding rule keys off precision).

This is the rule for every construct that CHOOSES BETWEEN its operands rather than computing a new number from them: a set operation's arms, CASE's branches, COALESCE/NULLIF/IFNULL/IF/GREATEST/LEAST's arguments.

Item 3's p>38 ADJUSTMENT is deliberately NOT applied here, and the reason is item 7's: a choice's result IS one of its operands' stored values, so giving up fraction digits would DROP digits a row actually holds — over `GREATEST(numeric(38,0), numeric(11,10))` the adjustment reduces the scale from 10 to 6 and the second column's 0.0000000001 becomes 0.000000, silently. Arithmetic is where the adjustment belongs (DecimalResultType): a computed scale is derived rather than carried, so there are no stored digits to lose. The precision cap alone is therefore the whole rule here, and a value with no carrier at the resulting type is a per-value 22003 at the store rather than a plan-time refusal of the query — which is what lets `GREATEST(numeric(38,30), bigint)` answer for every value that fits.

The result is capped at the carrier's full width — 38 digits is what an Int128 holds. The cap reduces the PRECISION and leaves the scale, so it is a RANGE reduction, which is why a value with no Int128 at the output type is then an ERROR at the moment of coercion rather than a wrapped number (ADR-0024 items 4 and 7; #552 records the cost).

ok=false means an operand contributed nothing this rule can use — a computed DECIMAL whose (p,s) nobody resolved, or a non-numeric type. The caller must then decline to declare a DECIMAL at all rather than guess one.

func DecimalScalarType added in v0.18.5 

func DecimalScalarType(op DecimalScalarOp, in DecimalType, digits int) (DecimalType, bool)

DecimalScalarType returns the (precision, scale) of a scalar math function's DECIMAL result, per ADR-0024 items 2 and 3.

digits is the SECOND argument of round(x, n) / trunc(x, n) and is ignored by the one-argument ops. PostgreSQL's one-argument round and trunc are the two-argument ones at n = 0, so a caller with no second argument passes 0 and gets the same answer PostgreSQL gives (`round(12.75::numeric)` is 13).

A NEGATIVE n rounds to a power of ten ABOVE the point — PostgreSQL's `round(1234.56, -2)` is 1200 and `round(1250, -2)` is 1300, half away from zero like every other numeric rounding here — and the result has no fraction at all, so the scale is 0. It is not a range reduction: 1200 still needs its four integer digits.

ok=false when the input carries no usable declaration (precision 0 is the codebase's "unconstrained" sentinel, #458), and then the caller must decline to declare a DECIMAL rather than guess one — the same clause DecimalCommon has, for the same reason.

func DecimalTextType added in v0.18.5 

func DecimalTextType(s string) (DecimalType, bool)

DecimalTextType reports the DECIMAL type a numeric LITERAL names — ADR-0024 item 3's "a numeric literal's (p,s) is its spelling".

PostgreSQL types an unadorned `12.75` as numeric, and the (p,s) a finite carrier needs for it is read off the digits the user actually WROTE: 12.75 is DECIMAL(4,2), 2 is DECIMAL(1,0), 0.5 is DECIMAL(1,1). That is what makes `d * 2` a multiply by DECIMAL(1,0) — result scale 2 — rather than by the INT32 range's DECIMAL(10,0), which would declare eight integer digits nobody wrote. An integer COLUMN is the other rule and keeps its whole range (DecimalTypeOf), because a column's values are not one spelling.

TRAILING ZEROS ARE KEPT, and that is the whole reason this reads the text itself rather than going through decimalParts: `100.0` is DECIMAL(4,1), not (3,0). PostgreSQL's numeric carries a per-value dscale that the zeros are part of — `12.75 * 100.0` renders 1275.000, three fraction digits, because the literal contributed one — and folding them away made the product's declared scale 2 where PostgreSQL's is 3. They cost nothing and they are what the user wrote.

The exponent form is normalized first: `1.5e3` is 1500, DECIMAL(4,0), and `1.5e-3` is 0.0015, DECIMAL(4,4) — the same value written two ways gets the same type, which is what keeps `d + 1.5e3` and `d + 1500` from declaring different columns.

ok=false for text that names no number, and for one whose scale or digit count is past what a DECIMAL can declare — a literal with 40 fraction digits has no fixed-point type here, and the caller must fall back rather than truncate it.

func DecimalTypeOf added in v0.18.5 

func DecimalTypeOf(t TypeID, dec DecimalType) (DecimalType, bool)

DecimalTypeOf returns the DecimalType an operand of type t contributes to a result-type computation, and whether t participates at all. DECIMAL brings its own declaration (handed in by the caller, which is the only holder of it); the integer types bring their whole range at scale 0.

type GlobalPool 

type GlobalPool struct {
	// contains filtered or unexported fields
}

GlobalPool provides shared batch pooling across operators with the same schema. This avoids each operator maintaining its own pool and improves reuse when multiple operators in a pipeline share a schema.

func NewGlobalPool 

func NewGlobalPool() *GlobalPool

NewGlobalPool creates a new global pool.

func (*GlobalPool) ForSchema 

func (gp *GlobalPool) ForSchema(schema []parquet.Column, batchSize int) *BatchPool

ForSchema returns the pool for the given schema and batch size. Creates one if it doesn't exist yet.

type Int128 

type Int128 struct {
	Hi int64  // upper 64 bits (signed)
	Lo uint64 // lower 64 bits (unsigned)
}

Int128 is a 128-bit signed integer used for DECIMAL storage. Values are stored as scaled integers: DECIMAL(10,2) value 123.45 → 12345.

func DecimalAvg added in v0.18.1 

func DecimalAvg(sum Int128, count int64, addScale int) (Int128, bool)

DecimalAvg divides an exact DECIMAL sum by a row count, returning the unscaled quotient at scale+addScale, rounded half away from zero (which is what PostgreSQL's numeric rounding does, and what DECIMAL casts here do).

ok=false means the exact quotient has no Int128 — the caller must report an error rather than an approximation. That is reachable even when the SUM itself fit: scaling by 10^addScale is a multiplication, so a sum near the top of the range with a small count has no representable average.

func DecimalPow10 added in v0.18.5 

func DecimalPow10(n int) (Int128, bool)

DecimalPow10 returns 10^n as an Int128, ok=false past 10^38 where the carrier has no such value.

func DecimalPrecisionLimit added in v0.18.5 

func DecimalPrecisionLimit(precision int) (Int128, bool)

DecimalPrecisionLimit is 10^precision, the EXCLUSIVE bound on the unscaled magnitude a DECIMAL(precision, s) column may hold. ok=false means the declaration named no bound at all — precision 0 is #458's "unconstrained" sentinel.

A precision past the carrier's own width is clamped to it rather than treated as "no bound to check". Skipping the check there was the older behaviour on both sides of the package boundary, and it is wrong in the one direction that matters: a DECIMAL(50,2) declaration cannot make an Int128 hold 10^50, so the values admitted by the skip are exactly the ones with no carrier. 10^38 is the widest bound an Int128 can express and is the honest one to enforce.

func Int128From 

func Int128From(v int64) Int128

Int128From creates an Int128 from an int64 value.

func Int128FromFloat64 

func Int128FromFloat64(f float64, scale int) Int128

Int128FromFloat64 converts a float64 to Int128 with the given scale. For example, Int128FromFloat64(123.45, 2) → Int128 representing 12345.

func ParseDecimalString 

func ParseDecimalString(s string, scale int) Int128

ParseDecimalString parses a decimal string like "123.45" into an Int128 at the given scale, truncating toward zero. Text that is not a number reads as zero here, and text with no Int128 at this scale reads as the SATURATED end of the carrier's range.

Both answers are for a COMPARISON, which is what this function is for: a literal outside a column's range is a BOUND, and orders above (or below) every value the column holds (#462). A caller producing a stored VALUE must take ParseDecimalStringChecked instead — saturating there replaced 10^30 with 2^127-1 and reported nothing (#553), and reading unparseable text as zero made a constant nobody can read compare EQUAL to every stored zero (#463). ADR-0024 item 4 is the rule: a value with no exact carrier is a 22003 error, never a saturated, wrapped, narrowed or zeroed number.

NaN and the infinities read as ZERO here for the same reason 'abc' does — this reader answers through DecimalTextAt, which does not know them. That is the unchecked writer's contract; the COMPARISON reader that does know them is DecimalBoundTextAt, and the value-producing one that reports them is ParseDecimalStringChecked (#534).

func ParseDecimalStringChecked added in v0.18.5 

func ParseDecimalStringChecked(s string, scale int) (Int128, error)

ParseDecimalStringChecked is ParseDecimalString for a caller that is producing a VALUE: the two answers ParseDecimalString gives silently — the saturated carrier end for a magnitude with no Int128 at this scale, and zero for text that is not a number — become the errors PostgreSQL raises for them, with its SQLSTATEs (ADR-0024 item 4).

  • 22003 numeric_value_out_of_range: the number is real but has no exact 128-bit carrier at `scale`. Handing back Int128Max here is what turned 10^30 into 17014118346046923173168730371.5884105727 in a UNION, with no error and no warning (#553).
  • 22P02 invalid_text_representation: the text names no number at all.

NaN and the infinities are a THIRD case and take the first SQLSTATE, not the second: PostgreSQL reads all three as `numeric` values, so the text is not an input-syntax error — it names a value this carrier has no bit pattern for (ADR-0024 item 6). PostgreSQL raises exactly this SQLSTATE for the infinities against a constrained column ("a field with precision 18, scale 4 cannot hold an infinite value", 22003, verified live); NaN it stores, and wadjet refusing it is the divergence item 6 records.

func Rescale added in v0.18.5 

func Rescale(v Int128, fromScale, toScale int) (Int128, bool)

Rescale moves an unscaled value from one scale to another: exactly when the scale rises, rounded HALF AWAY FROM ZERO when it falls, and ok=false when the exact result has no Int128.

Rounding away from zero is PostgreSQL's numeric rounding and ADR-0024's, so 1.5 and -1.5 at scale 0 are 2 and -2, not the 2 and -2 of banker's rounding. The upward half is MulPow10, unchanged, so a rescale that only widens is the same exact shift every DECIMAL comparison already uses.

func (Int128) Add 

func (d Int128) Add(other Int128) Int128

Add returns d + other.

func (Int128) AddChecked added in v0.18.1 

func (d Int128) AddChecked(other Int128) (Int128, bool)

AddChecked returns d + other and reports whether the EXACT sum fits in an Int128. A false second result means the first is the WRAPPED value and must not be shown to anyone: it is a different number, silently.

Two's-complement signed addition overflows exactly when the operands share a sign and the result does not — the standard rule, and the only one needed here because Add is a plain 128-bit add with carry.

SUM over a DECIMAL column accumulates through this (kernel.Accumulator, agg_scatter's flat arrays): the aggregate's carrier is 128 bits wide, so a DECIMAL(38) column holding values near 10^38 overflows after two rows, and the wrapped answer looks like an ordinary number. See docs/adr/0012, item 9 (exact numeric aggregates).

func (Int128) BigInt 

func (d Int128) BigInt() *big.Int

BigInt returns the value as a big.Int: Hi x 2^64 + Lo, exactly. Used on the paths where an Int128 is too narrow to hold an intermediate result.

func (Int128) Cmp added in v0.18.1 

func (d Int128) Cmp(other Int128) int

Cmp orders two values: -1, 0 or +1 as d is less than, equal to or greater than other. Every DECIMAL comparison in the engine bottoms out here.

func (Int128) Equal 

func (d Int128) Equal(other Int128) bool

Equal returns true if d == other.

func (Int128) FitsInt64 added in v0.18.1 

func (d Int128) FitsInt64() bool

FitsInt64 reports whether the value is exactly an int64, i.e. whether Hi is nothing but ToInt64's sign extension.

func (Int128) FormatDecimal 

func (d Int128) FormatDecimal(scale int) string

FormatDecimal renders the unscaled value as a decimal string at the given scale — the text form of a DECIMAL column, and so what GetValue hands the row map, ToRows, the JSON encoder and the pgwire text protocol.

The fraction is EXACTLY scale digits, never fewer. It used to be trimmed of trailing zeros, so a numeric(9,2) holding -24.50 reached a client as "-24.5" and a numeric(38,10) zero as "0.0" (#453). PostgreSQL renders a numeric(p,s) at its DECLARED scale always, and ADR-0012 makes PostgreSQL the authority — but the deeper reason is that a DECIMAL column exists BECAUSE its scale is part of the value's meaning. A currency column that spells itself "-24.5" is one a BI tool displays wrong, and any client that string-compares or formats from the text gets a different answer than it gets from PostgreSQL. The trim also reached the wire's binary form, whose dscale header pgNumericDigits counts off this very string.

scale <= 0 renders no point at all — "12345", not "12345." — which is PostgreSQL's numeric(p,0) too.

It formats the whole 128 bits. It used to read only Lo, through `v := int64(abs.Lo)` and an int64 divmod, which was wrong twice over: a value past 64 bits rendered as its low half (Int128{Hi:5, Lo:0x112210f4- 7de98115} at scale 10 came out 123456789.0123456789 instead of 9346828825.8671214869), and any magnitude with Lo >= 2^63 made that int64 negative, so the sign leaked into both halves and produced text that is not a number at all — "--922337203.-6854775808" for unscaled Int64Min (#434).

Splitting the digit STRING at the scale is also what makes the result exact for every scale: math.Pow10 is a float64, and 10^23 has no exact one.

func (Int128) IsNegative 

func (d Int128) IsNegative() bool

IsNegative returns true if the value is negative.

func (Int128) IsZero 

func (d Int128) IsZero() bool

IsZero returns true if the value is zero.

func (Int128) Less 

func (d Int128) Less(other Int128) bool

Less returns true if d < other (signed comparison).

func (Int128) Mul added in v0.18.5 

func (d Int128) Mul(other Int128) (Int128, bool)

Mul returns d * other and reports whether the EXACT product fits an Int128.

The intermediate is the full 256-bit product of the two magnitudes, built from four bits.Mul64 partial products — never big.Int, because this is the kernel a DECIMAL multiply runs per row. A false second result means the first is zero and carries no information: the product is outside the range, full stop.

func (Int128) MulPow10 

func (d Int128) MulPow10(n int) (Int128, bool)

MulPow10 returns d x 10^n and reports whether the EXACT product fits in Int128. It never returns an approximation: a false second result means the caller must take a wider path, not that the first result is close.

Rescaling one operand up to the other's scale is how two DECIMALs of different scale are compared exactly (kernel.CompareDecimalAt), which matters at a sort-merge join key where the comparator decides EQUALITY: float64 rescaling makes 9007199254740993 and 9007199254740992.0 the same number and emits a join row for a pair that does not match.

func (Int128) Neg 

func (d Int128) Neg() Int128

Neg returns the negation of the value.

func (Int128) QuoRem added in v0.18.5 

func (d Int128) QuoRem(other Int128) (q, r Int128, ok bool)

QuoRem returns the truncated quotient and the remainder of d / other, with Go's (and PostgreSQL's, and C's) sign rule: the quotient truncates TOWARD ZERO and the remainder takes the sign of the dividend, so d == q*other + r always.

ok=false for the two divisions that have no answer in the carrier: a zero divisor, and -2^127 / -1 whose quotient is 2^127.

func (Int128) String added in v0.18.1 

func (d Int128) String() string

String renders the unscaled value in base 10, exactly, at any width.

func (Int128) Sub 

func (d Int128) Sub(other Int128) Int128

Sub returns d - other.

func (Int128) SubChecked added in v0.18.5 

func (d Int128) SubChecked(other Int128) (Int128, bool)

SubChecked returns d - other and reports whether the EXACT difference fits an Int128. A false second result means the first is the WRAPPED value.

The twin of AddChecked, and needed for the same reason: a sliding window frame retracts a row from a running DECIMAL sum by subtracting it, and a wrapped difference there is a plausible-looking number that is not the answer. Sub alone cannot report it, and negating `other` first does not work — -2^127 negates to itself.

func (Int128) ToFloat64 

func (d Int128) ToFloat64(scale int) float64

ToFloat64 converts an Int128 decimal value to float64 using the given scale.

func (Int128) ToInt64 

func (d Int128) ToInt64() int64

ToInt64 returns the low 64 bits of the unscaled value as an int64. It is only the value when the value FITS: Hi is not consulted, so a wider Int128 comes back truncated and, past 2^63, with the wrong sign. Callers that may see a wide value want String, FormatDecimal or BigInt instead — dropping Hi here is what made every DECIMAL past 64 bits render as its low half (#434).

type IntegerRangeError added in v0.18.27 

type IntegerRangeError struct {
	Dst TypeID // the vector's type
	Val any    // the value with no room in it, for diagnosis
}

IntegerRangeError reports a write of an integer VALUE that the vector's narrower integer storage has no room for — the sibling of the guard above, and the one #361's check cannot see: the Go type converts fine, so nothing is "mismatched"; it is the NUMBER that has nowhere to go.

The mechanism this closes is not a typing mistake anywhere. ColRef.Eval widens an INT32 column to an int64 box on purpose (ADR-0012's recorded "every integer spelling is INT64" superset), so a per-row kernel over an int4 column computes in int64 and is RIGHT to: |−2147483648| is 2147483648, exactly what a bigint answer would be. The planner then declares the projection int4, because PostgreSQL's `abs(int4)` IS int4 — and the store narrowed 2147483648 back into an int32 and WRAPPED it to -2147483648. A different number wearing the right type: ADR-0012 item 9, and the class ADR-0024 forbids on every integer path.

So the refusal belongs at the STORE and not inside ABS. Every kernel that computes an int4 result in int64 crosses this one seam, and a check here covers all of them at once, where a check inside ABS would leave the next such kernel for the next census to find. PostgreSQL's own SQLSTATE and its own wording, so a client sees the message it would see there.

func (*IntegerRangeError) Error added in v0.18.27 

func (e *IntegerRangeError) Error() string

func (*IntegerRangeError) FatalEvalError added in v0.18.27 

func (e *IntegerRangeError) FatalEvalError() error

FatalEvalError implements the exec.FatalEvalPanic contract, the same route TypeMismatchError takes: a query error, never a process exit.

func (*IntegerRangeError) SQLState added in v0.18.27 

func (e *IntegerRangeError) SQLState() string

SQLState is PostgreSQL's numeric_value_out_of_range.

type MintStamp 

type MintStamp struct {
	Owner uint64
	Seq   uint64
}

MintStamp records WHICH producer minted a batch's storage and WHICH issue of that storage this is. It exists so a producer that hands storage out and takes it back — today the scan row-group backing pool, docs/design/scan-output-backing-reuse.md — can recognize its own batch at the release edge WITHOUT keeping a reference to it.

A registry of outstanding batches keyed by pointer is the obvious implementation and the wrong one: it is a strong reference the GC cannot collect and the memory ledger cannot see. A consumer with no release edge, or a batch the pipeline simply drops, pins whole decoded row groups (~280 MB each at SF100) for the producer's lifetime, and any bound on the registry's SIZE silently turns reuse off instead. The stamp inverts the direction: the batch points at nothing, the producer holds nothing, and identity survives in a pair of integers the batch carries.

Owner is a process-unique producer id from NewMintOwner. Zero means unstamped — what a WSHF shuffle chunk, a row-based fallback batch or any batch from a different producer carries — and a release edge must treat it as foreign, because adopting it would create a second owner for storage somebody else recycles.

Seq is bumped on every re-issue of the SAME storage. A release names the Seq it was handed, so a stale release from a previous generation (a retire that fired twice around a re-mint) names an older Seq and is ignored: re-admitting a LIVE backing to the free list would give two decoders one buffer, the one failure this design must not have.

The stamp is written by the producer while it owns the batch exclusively and read at the release edge; the producer's own publication edge (the decode ring's channel, the dispenser's channel send) and the pool mutex order every access.

func (MintStamp) Valid 

func (m MintStamp) Valid() bool

Valid reports whether the stamp names a producer.

type RecordBatch 

type RecordBatch struct {
	Columns []*Vector
	Schema  []parquet.Column
	Len     int
	Sel     []uint32 // selection vector: indices of active rows (nil = all rows active)
	// contains filtered or unexported fields
}

RecordBatch is the unit of data flowing between operators.

func FromRows 

func FromRows(schema []parquet.Column, rows []map[string]any) *RecordBatch

FromRows creates a RecordBatch from row-oriented data.

This is the ROW→BATCH boundary, and the one place where the two shapes of a MAP meet: the parquet row reader produces a Go map (and the writer consumes one) while the vector stores MAP as ARRAY(ROW("key","value")). Nothing converted between them, so every scan that fell back to the row reader — every query on a table carrying a nested column — handed Vector.SetValue a map it rejects and died on the scan worker (#393).

The conversion belongs HERE rather than in SetValue's MAP arm. SetValue cannot tell a MAP's row shape from a ROW's box (both are map[string]any), so accepting one there would silently reshape a ROW written into a mis-derived MAP vector — a live defect on the stage DAG (#397) that the #361 guard is currently the only thing reporting. At this boundary the context is unambiguous: the value came from the row reader and the catalog says the column is a MAP.

func FromRowsChecked added in v0.18.5 

func FromRowsChecked(schema []parquet.Column, rows []map[string]any) (*RecordBatch, error)

FromRowsChecked is FromRows for a caller materializing a stored VALUE: it writes through Vector.SetValueChecked, so a DECIMAL conversion that cannot be made exactly is an ERROR carrying PostgreSQL's SQLSTATE instead of the nearest number the carrier can hold (ADR-0024 item 4).

The single-process set-operation adapter is why it exists. It boxes both arms' rows and rebuilds a batch under the unified schema, so every value the operation returns is written HERE — and the saturating writer turned a DECIMAL(38,0) arm's 10^30 into 2^127-1 rendered at scale 10, with no error anywhere (#553).

A partially-filled batch comes back alongside the error rather than nil, so a caller that wants to name the failing row can; callers that do not simply discard it.

func NewRecordBatch 

func NewRecordBatch(schema []parquet.Column, numRows int) *RecordBatch

NewRecordBatch creates a new record batch with the given schema and row count.

The batch and every vector under it share ONE claimState, which is what lets the pool boundary answer "does a consumer still hold anything here" with a single atomic load — see claimState and retainsClaimedStorage. A pooled batch OWNS its columns: nothing may replace b.Columns[i] with a vector minted elsewhere and then release the batch to a pool, an invariant resetVectorForReuse has always relied on and TestAPooledBatchOwnsItsColumns now asserts.

func (*RecordBatch) ActiveLen 

func (b *RecordBatch) ActiveLen() int

ActiveLen returns the number of active rows (respecting selection vector).

func (*RecordBatch) ColumnByName 

func (b *RecordBatch) ColumnByName(name string) *Vector

ColumnByName returns the vector for the named column, or nil if not found.

func (*RecordBatch) ColumnIndex 

func (b *RecordBatch) ColumnIndex(name string) int

ColumnIndex returns the index of the named column, or -1.

func (*RecordBatch) Compact 

func (b *RecordBatch) Compact() *RecordBatch

Compact materializes the selection vector into a contiguous batch using the typed nested-aware value copier. Returns the batch unchanged when no selection vector is set.

Was previously a hand-rolled per-type switch whose ROW case wrote null child rows via SetNull alone — never advancing a string child's offset slot — so every later row in that child read back as concatenated garbage (same bug class as the windowCopyVectorRange nullable-BYTES fix; regression test TestCompact_RowChildNullableString).

func (*RecordBatch) Detach 

func (b *RecordBatch) Detach()

Detach claims ownership of the batch: Release() becomes a no-op so no pool can recycle it, and the claim is recorded on the batch AND on every column vector so a producer that reuses vector backing across calls surrenders it (see (*Vector).Claim). Call it from anything that keeps a batch — or anything pointing into its column storage — past the call that handed it over: the hash-join build, Sort, Window, the collect sinks, the spillable collector, partitioned aggregation's per-partition views.

The per-column claim is what makes the contract hold through a derived batch: ColumnPrune and the set-op emitter mint a NEW RecordBatch over the same *Vector pointers, so a consumer that detaches the derived batch would otherwise leave the producer of the original believing nobody kept it. ORDER MATTERS and always has: Detach BEFORE the producer releases the batch. A claim taken afterwards is a claim on storage the pool has already taken back, and neither the per-column walk nor the claim flag can help — the next Get resets it and writes over the value. The O(1) flag makes the check look more authoritative than the contract is; the contract is unchanged.

func (*RecordBatch) DetachPool 

func (b *RecordBatch) DetachPool()

DetachPool severs only the pool link, WITHOUT claiming the batch or its columns. It is for the one caller whose reference is transitive rather than independent: the hash-join late-materialization emitter, whose output views read the input's vectors, so the input must not be recycled underneath them by a concurrently-running source — but whose views die with its own output batch, whose consumer's Detach (if any) propagates the claim through Vector.Base anyway. Anything that genuinely KEEPS a batch calls Detach.

func (*RecordBatch) EnsureCapacity 

func (b *RecordBatch) EnsureCapacity(n int)

EnsureCapacity grows every column so positions [0, n) are addressable for in-place writes, preserving existing data, and sets Len to n. Used by append-style builders (e.g. the hash-join per-partition accumulator) that grow a batch across many source batches rather than pre-sizing to a worst-case capacity. See Vector.EnsureLen for per-type behavior and the nested-type caveat.

func (*RecordBatch) FlattenColumn 

func (b *RecordBatch) FlattenColumn(i int)

FlattenColumn materializes a single column if it is a view. Use for column-granular consumers (a filter touching one column) so the remaining columns stay lazy.

func (*RecordBatch) FlattenViews 

func (b *RecordBatch) FlattenViews()

FlattenViews materializes every view column in place. Call before retaining a batch past the pipeline's per-batch cycle (Sort/Window/sink Consume) or handing it to code that reads typed storage directly.

func (*RecordBatch) HasViews 

func (b *RecordBatch) HasViews() bool

HasViews reports whether any column of the batch is a view vector.

func (*RecordBatch) MemBytes 

func (b *RecordBatch) MemBytes() int64

MemBytes returns the in-memory byte footprint of the batch's column data, summing each Vector's MemBytes(). It deliberately omits operator-specific overhead (e.g. the HashJoin hash-index charge) — that stays at the call site (see hashBuildBytes in package exec). Replaces the per-type estimate that lived in exec.EstimateBatchBytes.

func (*RecordBatch) Mint 

func (b *RecordBatch) Mint() MintStamp

Mint returns the producer stamp on this batch (zero Owner = unstamped).

func (*RecordBatch) OwnsItsColumns added in v0.18.49 

func (b *RecordBatch) OwnsItsColumns() bool

OwnsItsColumns reports whether every vector under this batch carries the batch's own claim state — the invariant the O(1) pool check rests on.

It exists for the gate that asserts it (TestAPooledBatchOwnsItsColumns) and for anyone debugging a claim that did not veto: a false answer means some column was assigned rather than SetColumn'd, and a claim on it will be invisible to Release. It walks the whole tree and is not for a hot path.

func (*RecordBatch) Release 

func (b *RecordBatch) Release()

Release returns the batch to its pool if applicable.

Once released, the batch's storage is undefined: the pool may hand it to another operator, which resets it and writes over the same arenas. Anything keeping a value out of the batch past this point must own it — see Detach. Poison mode (see poison.go) makes that undefinedness observable by scribbling the arenas here, which is what the batch-reuse gate compares against a clean run.

"Undefined" stops at a CLAIM. A batch carrying storage a consumer claimed is vetoed here — neither poisoned nor pooled — because the alias the consumer holds is the same memory the next Get would reset and write over. See retainsClaimedStorage.

func (*RecordBatch) Reset 

func (b *RecordBatch) Reset(numRows int)

Reset clears the batch for reuse, keeping allocated memory.

func (*RecordBatch) ResolveColumnByName added in v0.18.30 

func (b *RecordBatch) ResolveColumnByName(name string) *Vector

ResolveColumnByName is ResolveColumnIndex's vector-returning form.

func (*RecordBatch) ResolveColumnIndex added in v0.18.30 

func (b *RecordBatch) ResolveColumnIndex(name string) int

ResolveColumnIndex resolves a column REFERENCE to an index in b's schema, or -1. See the rule at the top of this file: byte-exact first, then a unique ASCII-case-insensitive match when the reference is itself folded.

Callers that hold a NAME rather than a reference — a producer writing its own output schema, a stage matching the name it just emitted — keep using ColumnIndex.

func (*RecordBatch) Retained 

func (b *RecordBatch) Retained() bool

Retained reports whether a consumer claimed ownership of this batch with Detach.

func (*RecordBatch) RowAt 

func (b *RecordBatch) RowAt(i int) map[string]any

RowAt boxes a single physical row (Sel is not consulted) as a map. For callers that need a few rows out of a large batch — boxing the whole batch via ToRows for a low-selectivity pick is the documented multi-GB heap pattern.

func (*RecordBatch) RowFieldPath added in v0.18.35 

func (b *RecordBatch) RowFieldPath(name string) (parent, field int, ok bool)

RowFieldPath answers ADR-0022's question for one dotted reference, and it is the ONE place the question is asked: does the reference's QUALIFIER name a ROW column of this batch that DECLARES the field? It returns the parent column's index and the field's position among that container's children.

Every consumer of a column reference asks this BEFORE stripping the qualifier, because stripping first answers with whatever OTHER relation in the stream publishes a column of the FIELD's name: 

SELECT n.id, c_row.b FROM typemx_nested n JOIN decpair d ON n.id = d.id
-- PostgreSQL 17 (spelled `(n.c_row).b`) answers the field: 11, NULL,
--   NULL, 44, 55, 66, 77, 88, NULL. wadjet answered decpair.b's DECIMALs
--   on all four arms, in silence (#769).

Four resolvers had to agree about which value `c_row.b` denotes — the single-process evaluator (expr.ColRef), the stage DAG's projection (exec.lazyFieldIdx), the DECLARATION half that types it (exec.fieldPathColumn) and the vectorized filters' ROW delegation — and each spelled the order for itself. That is the shape ADR-0022 was written about, one level down: a field path LOOKS like a qualified column reference, so every site invents the same three-way order and one of them gets it wrong.

The container must DECLARE the field. Without that test the reorder would capture an ordinary qualified reference whose qualifier happens to name a ROW column of the stream, and a field path naming NO field would stop answering the way it does today (#604).

The parent is looked up the way every other reference is: byte-exact under the fold, then the ONE column spelled `<qualifier>.<name>` — a join qualifies a colliding container, so `c_row.b` has to find `x.c_row`. Two arms spelling it decline HERE, and this function declining is not by itself the refusal: the caller's later branches would still bind one of them. What makes the ambiguity loud is the BINDER, which raises PostgreSQL's own `column reference "c_row" is ambiguous` (42702) at plan time when two of a block's sources publish the container (physical.colScope.check).

func (*RecordBatch) SetColumn added in v0.18.49 

func (b *RecordBatch) SetColumn(i int, v *Vector)

SetColumn replaces column i, adopting the vector into this batch's claim state so the pool boundary can still see a claim on it.

Assigning `b.Columns[i] = v` directly is what every operator that swaps a column does today, and it is correct for all of them because none of those batches is pooled — they are hand-built shells or NewRecordBatch batches with no pool, which Release never admits anywhere. On a POOLED batch it is not correct: the claim flag hangs off the vectors (see claimState), so a vector minted elsewhere carries a different flag — or none — and a consumer claiming it would set somebody else's while THIS batch is the one Reset recycles. Round-2 review P1 measured that hole for all three ways a foreign vector is minted (NewVectorLike, another pooled batch's column, NewColumnVector).

So the swap has a supported form. It stamps the incoming vector and its children, and if the vector is ALREADY claimed it trips the flag at once, because a claim taken before the vector joined the batch is still a claim on storage this batch would otherwise recycle.

func (*RecordBatch) SetMint 

func (b *RecordBatch) SetMint(m MintStamp)

SetMint stamps the batch for its producing free list. Only the producer calls it, and only while it owns the batch exclusively — at mint, and again to clear the stamp when the storage is taken back. The stamp travels with a VALUE copy of the RecordBatch (e.g. `nb := *b`), so any such copy taken over a scan batch must zero the stamp (`nb.SetMint(batch.MintStamp{})`) or it would alias the parent's pool identity; today the only value copy (internal/engine/exec/partitioned_agg.go's selView) Detaches immediately, which claims the shared columns and trips the release veto regardless.

func (*RecordBatch) ToRowValues added in v0.18.3 

func (b *RecordBatch) ToRowValues() [][]any

ToRows converts a RecordBatch to row-oriented data. ToRowValues boxes the batch POSITIONALLY: one []any per active row, its cells aligned index-for-index with b.Schema.

This is ToRows without the lossy step. A result may legally carry two columns of the SAME NAME — PostgreSQL answers `SELECT abs(a), abs(b)` with two columns both called `abs`, and #513 made this engine agree — and a map keyed by name cannot hold both, so the second silently overwrites the first and a client reads column 0's value under column 1's name. A wrong VALUE is strictly worse than a wrong name, so every transport that has the batch boxes it this way and only convenience APIs take the map.

func (*RecordBatch) ToRows 

func (b *RecordBatch) ToRows() []map[string]any

type ScaledDecimal added in v0.18.1 

type ScaledDecimal struct {
	// Unscaled is the value at the column's scale, truncated toward zero.
	Unscaled Int128
	// Residual is the SIGN of what the truncation dropped: +1 when the true
	// value is strictly above Unscaled, -1 when strictly below, 0 when the
	// conversion was exact. It is what keeps a literal finer than the
	// column's scale in its rational place in the order (ADR-0012 item 6).
	Residual int
	// Sat is 0 when the true value HAS an Int128 at this scale, and +1 / -1
	// when it lies above Int128Max / below Int128Min.
	//
	// A value outside the range orders above (or below) every value the
	// column can hold, which is what it actually is. Narrowing it by
	// two's-complement wraparound instead landed it back INSIDE the ordinary
	// range, so `WHERE d < 1e39` — true of every row — selected none of them
	// (#462).
	Sat int
}

ScaledDecimal is a numeric value resolved into a DECIMAL column's domain: the unscaled integer at that column's scale, plus everything the domain could not hold. It is the single carrier every DECIMAL comparison converts a constant through — the vectorized kernel, the row-at-a-time expression and the row-group prune — so that one predicate cannot be read three ways.

func DecimalBoundTextAt added in v0.18.5 

func DecimalBoundTextAt(text string, scale int) (ScaledDecimal, bool)

DecimalBoundTextAt is DecimalTextAt widened to the NaN/±Infinity spellings, for the callers resolving a COMPARISON literal rather than a value.

A DECIMAL column holds no NaN and no infinity, so each of the three is a BOUND lying past one end of everything the column can hold — which is exactly what ScaledDecimal.Sat already expresses for a finite literal too wide for the carrier (#462). NaN and Infinity both sit above every stored value and -Infinity below every one, so `d = 'NaN'` finds nothing, `d < 'NaN'` and `d <= 'Infinity'` and `d > '-Infinity'` find every non-NULL row, and PostgreSQL's NaN > Infinity is invisible over a column that can hold neither (ADR-0024 item 6).

A value-producing caller must NOT take this reader: ParseDecimalStringChecked turns the same three spellings into the 22003 that ADR-0024 item 6 requires, because saturating a stored value is a lie the same way #553's was.

func DecimalTextAt added in v0.18.1 

func DecimalTextAt(text string, scale int) (ScaledDecimal, bool)

DecimalTextAt resolves numeric TEXT into a DECIMAL column's domain at `scale`, exactly and without ever going through a float64.

NaN and the infinities are deliberately NOT read here: this reader answers for a stored value as well as a comparison, and they have no stored value. DecimalBoundTextAt is the comparison-only reader that accepts them.

ok=false means the text is not a number. It is deliberately NOT reported as the value zero: a constant nobody can read used to compare EQUAL to every stored zero (#463), which is the most dangerous shape a parse failure can take because it neither errors nor returns nothing.

func (ScaledDecimal) Order added in v0.18.1 

func (s ScaledDecimal) Order(cell Int128) int

Order returns -1, 0 or +1 as an unscaled column value at the SAME scale is less than, equal to, or greater than this value.

type ShapeOnlyLen added in v0.18.24 

type ShapeOnlyLen int

ShapeOnlyLen is what the boxing boundary hands back for a row of a SHAPE-ONLY column: the value is NOT AVAILABLE, and this is its byte length.

It exists because a shape-only column has to survive a ROW-SHAPED detour. The scan decodes lengths and no bytes when the planner proves every use of the column reads its shape (COUNT, LENGTH, IS NULL, empty-string), and the vector paths carry that faithfully — copyShapeRange propagates the mark rather than moving bytes that do not exist. The row paths could not: a grouped aggregate under memory pressure buffers its input through RecordBatch.ToRows, whose per-row box comes from Vector.GetValue, and the only thing GetValue could produce for such a row was the panic that says a value was read (#791).

So the box is neither the value nor a rendering of it. It is a REFUSAL that carries the length: LengthAt's answer, and nothing else. Written back through SetValue it reconstructs a shape-only column with the same per-row lengths, so what comes out of the detour is what went in — and a consumer that then wants the bytes raises the same guard it always did, at the same place, saying the same thing.

A type of its own rather than an int: an int would be indistinguishable from a value at every `switch v := x.(type)` in the tree, which is exactly how a lossy encoding gets written by accident (#632, ADR-0023 item 6 — an encoder must never write bytes its own reader refuses, and a renderer is not a value).

type TypeID 

type TypeID = parquet.TypeID

TypeID is an alias for the parquet TypeID used throughout the engine.

type TypeMismatchError 

type TypeMismatchError struct {
	Dst TypeID // the vector's type
	Val any    // the value that had nowhere to go
}

TypeMismatchError reports a write of a value a vector has nowhere to put: SetValue was handed a Go value whose type has no conversion into the vector's storage.

Until #361 such a write VANISHED — the slot kept its zero value and was marked valid — which is the mechanism behind an entire bug family (#310, #327, #331, #333, #345, #353, #361, #371, #372): some declaration upstream picks the wrong vector type, and instead of an error the query answers 0 on every row. The write site is the one seam every one of those defects must cross, so it now panics with this typed value.

The panic carries a query ERROR, not a crash: it implements the exec.FatalEvalPanic contract (Error + FatalEvalError), the same route the expression evaluator uses for a condition with no error return (#347). The pipeline drivers, the worker's task-level recover, the coordinator's and the embedded API's query entries all convert it back into an error — "a wrong type may cost a wrong answer, never the server" (#310) still holds, with the improvement that it now costs an ERROR instead of a wrong answer.

The deliberate non-panics: a nil value is a NULL (WriteNullAt); STRING and BYTES destinations coerce any value through its string form, which is a documented rendering (group keys rely on it); and a PARSE failure of a value-level string (an unparseable IPv4, MAC, UUID) keeps its historical null-ish result — the type was right, the value was not.

func (*TypeMismatchError) Error 

func (e *TypeMismatchError) Error() string

func (*TypeMismatchError) FatalEvalError 

func (e *TypeMismatchError) FatalEvalError() error

FatalEvalError implements the exec.FatalEvalPanic contract, so pipeline drivers convert the panic into a query error instead of a process exit.

type Vector 

type Vector struct {
	Type        TypeID
	Len         int
	Nulls       Bitmap
	BoolData    []bool
	Int32Data   []int32
	Int64Data   []int64
	Float32Data []float32
	Float64Data []float64
	BytesData   BytesColumn
	DecimalData DecimalColumn // for TypeDecimal

	// VECTOR type: fixed-dimension float32 embeddings
	// Row i's vector: Float32Data[i*VectorDim : (i+1)*VectorDim]
	VectorDim int // VECTOR: dimensionality (number of float32 elements per row)

	// Nested type fields (ARRAY, ROW, MAP)
	Offsets    []int32   // ARRAY/MAP: offsets[i]..offsets[i+1] delimit child elements for row i
	Child      *Vector   // ARRAY: flat vector of all element values
	Children   []*Vector // ROW: one vector per field (same length as parent)
	FieldNames []string  // ROW: names of child fields

	// View (dictionary) form: when Base != nil this vector owns no typed
	// storage — logical row i is Base row Indices[i]. Nulls is the view's OWN
	// override bitmap (a null bit marks the row null regardless of Base; a
	// valid bit defers to Base's nullness through Indices). Base is always an
	// owned vector: NewViewVector composes indices when handed a view base, so
	// views never chain. Views are read-only and understood only by the
	// view-aware accessors (GetValue, CopyValueFrom-as-source, Flatten,
	// MemBytes); typed hot-path accessors (GetInt64, Int64Data[i], ...) fail
	// loud on a view because the typed slices are nil. See view.go.
	Base    *Vector
	Indices []uint32
	// contains filtered or unexported fields
}

Vector holds a single column of data. Uses typed slices instead of interface{}.

func NewArrayVector 

func NewArrayVector(length int, elemType TypeID) *Vector

NewArrayVector creates a new ARRAY vector with the given length and element type. The child vector starts with capacity 0; callers append elements and update offsets.

func NewColumnVector 

func NewColumnVector(col parquet.Column, numRows int) *Vector

NewColumnVector creates a single Vector from a Column definition with numRows pre-allocated rows, recursively initializing nested type children — the per-column equivalent of NewRecordBatch for callers that materialize some columns of a batch while emitting others as views.

func NewMapVector 

func NewMapVector(length int, keyType, valueType TypeID) *Vector

NewMapVector creates a new MAP vector. Internally stored as ARRAY(ROW("key","value")).

func NewRowVector 

func NewRowVector(length int, fieldNames []string, fieldTypes []TypeID) *Vector

NewRowVector creates a new ROW/STRUCT vector with named child fields.

func NewVector 

func NewVector(typ TypeID, length int) *Vector

NewVector creates a new vector of the given type and length.

func NewVectorLike 

func NewVectorLike(src *Vector) *Vector

NewVectorLike returns an empty (zero-row) vector with src's type and nested structure: child element types, ROW field names, VECTOR dim and DECIMAL scale. Element storage is appended by AppendFrom.

func NewVectorVector 

func NewVectorVector(length, dim int) *Vector

NewVectorVector creates a new VECTOR column with fixed dimensionality. Storage: Float32Data of length * dim, where row i occupies [i*dim, (i+1)*dim).

func NewVectorWithScale 

func NewVectorWithScale(typ TypeID, length int, scale int) *Vector

NewVectorWithScale creates a new vector with scale metadata (used for DECIMAL).

func NewViewVector 

func NewViewVector(base *Vector, indices []uint32) *Vector

NewViewVector creates a view over base addressing rows through indices. The indices slice is adopted, not copied — callers must not mutate it afterwards. If base is itself a view, the indirection is composed away (newIndices[i] = base.Indices[indices[i]], own-nulls folded), so Base on the returned vector is always an owned vector.

The view starts all-valid: row i's nullness defers to base through indices. Callers that need null-injection (outer-join fill) mark rows with v.Nulls.SetNull(i); those rows' index values are ignored.

func NewViewVectorReuse 

func NewViewVectorReuse(base *Vector, indices, composeBuf []uint32) (*Vector, []uint32)

NewViewVectorReuse is NewViewVector with a caller-owned composition buffer.

When base is itself a view its indirection has to be folded into a NEW index array, and at join-emit widths that array is a large-object allocation per column per output batch — one of the spans the Go heap lock serializes on. Passing composeBuf lets a caller that owns the resulting view's lifetime keep that array across batches. The returned slice is the array the view ADOPTED: it stays live for as long as the view does, so only a caller that knows the view is dead may re-pass it (see probeEmitBuf's ownership rule in package exec). nil comes back when no composition was needed and `indices` was adopted directly.

func (*Vector) AppendFrom 

func (v *Vector) AppendFrom(src *Vector, si int)

AppendFrom appends src[si] to dst, growing dst by one row. Typed copy — no boxing, no string round-trips — recursive for nested types. dst's nested structure must match src's (build it with NewVectorLike).

func (*Vector) Claim 

func (v *Vector) Claim()

Claim marks this vector's storage as retained by a consumer, recursively through the view base it reads from and its nested children. Once claimed a vector is never reused by its producer: the claim is sticky because nothing tracks when the retaining consumer is finished (a Sort holds its input until Finalize), and a wrong answer here is silent data corruption.

func (*Vector) Claimed 

func (v *Vector) Claimed() bool

Claimed reports whether a consumer has claimed this vector's storage.

func (*Vector) CopyValueFrom 

func (v *Vector) CopyValueFrom(di int, src *Vector, si int)

CopyValueFrom writes src[si] into position di of dst using typed access — no boxing, no string round-trips — for every column type including nested ARRAY/MAP/ROW. Fixed-width slots are indexed; variable-length storage (bytes data, array child elements, lazily-created row children) is appended, so writes must be SEQUENTIAL per column (di = 0, 1, 2, ...) — the same contract BytesColumn.Set has always had. Null source rows still advance offsets and children; skipping them would shift every later row.

Destination shape is flexible per level: parent slots may be pre-allocated (NewRecordBatch with full nested schema) or append-built (NewVectorLike); ROW children handle both — indexed writes when pre-allocated, appends when built lazily.

func (*Vector) EnsureLen 

func (v *Vector) EnsureLen(n int)

EnsureLen grows the vector's backing storage so positions [0, n) are addressable for in-place writes (Set / index-assign), preserving existing values, defaulting new fixed-width slots to zero and new null bits to non-null. Backing arrays grow geometrically (via append) for amortized O(1) appends, so an append-style builder can grow a column across many source batches instead of pre-sizing to a worst-case capacity. Sets Len to n.

Scalar, bytes, decimal and fixed-dim VECTOR columns are fully supported. Nested ARRAY/MAP element storage and ROW children are NOT grown here (their element storage is appended by SetValue); callers that build nested columns should use pre-sized batches. The hash-join accumulator path that relies on EnsureLen guards nested schemas to a pre-sized path for this reason.

func (*Vector) Flatten 

func (v *Vector) Flatten()

Flatten materializes a view in place: owned storage is allocated, values are gathered from Base through Indices (own-null rows become nulls), and the view fields are cleared. Aliases of the *Vector see the flattened form. No-op on owned vectors.

func (*Vector) GetBool 

func (v *Vector) GetBool(i int) (bool, bool)

GetBool returns the bool value at position i. Returns (false, false) if null.

func (*Vector) GetFloat32 

func (v *Vector) GetFloat32(i int) (float32, bool)

GetFloat32 returns the float32 value at position i. Returns (0, false) if null.

func (*Vector) GetFloat64 

func (v *Vector) GetFloat64(i int) (float64, bool)

GetFloat64 returns the float64 value at position i. Returns (0, false) if null.

func (*Vector) GetInt32 

func (v *Vector) GetInt32(i int) (int32, bool)

GetInt32 returns the int32 value at position i. Returns (0, false) if null.

func (*Vector) GetInt64 

func (v *Vector) GetInt64(i int) (int64, bool)

GetInt64 returns the int64 value at position i. Returns (0, false) if null.

func (*Vector) GetNumericFloat64 

func (v *Vector) GetNumericFloat64(i int) (float64, bool)

GetNumericFloat64 returns any numeric column value as float64 without boxing. Handles Int32, Int64, Float32, Float64, Timestamp types.

func (*Vector) GetString 

func (v *Vector) GetString(i int) (string, bool)

GetString returns the string value at position i. Returns ("", false) if null.

func (*Vector) GetValue 

func (v *Vector) GetValue(i int) any

GetValue returns the value at position i as an interface{}. Note: returns boxed values for numeric types (unavoidable with any return type). Prefer typed accessors (GetInt64, GetFloat64, etc.) in hot paths.

func (*Vector) IsShapeOnly added in v0.18.24 

func (v *Vector) IsShapeOnly() bool

IsShapeOnly reports whether this vector's bytes were never decoded — the lengths-only scan decode, or a copy that propagated the mark. A VIEW is shape-only when the vector it looks through is.

func (*Vector) IsView 

func (v *Vector) IsView() bool

IsView reports whether the vector is a view (owns no typed storage).

func (*Vector) MemBytes 

func (v *Vector) MemBytes() int64

MemBytes returns the heap bytes resident in this vector's backing storage: the null bitmap plus the typed data slice, recursing into nested children for ARRAY/MAP/ROW. It is the byte-true accounting primitive for the memory tracker, replacing the per-type b.Len*48 estimate in EstimateBatchBytes. It deliberately omits any operator-specific overhead (e.g. the HashJoin hash index charge) — that stays at the call site.

func (*Vector) ResetForWrite 

func (v *Vector) ResetForWrite(n int)

ResetForWrite resizes an OWNED vector to exactly n rows and clears its per-row state — null bits back to non-null, fixed-width slots to zero, the bytes arena emptied — while RETAINING every backing allocation's capacity. A producer that reuses one vector across output batches therefore allocates only when n passes its high-water mark, where a fresh NewColumnVector allocates (and the runtime zeroes) a new span every single batch.

Slots are cleared, not merely resized: the gather loops skip writing null and unmatched rows, so a stale value under a null bit would be a reuse-visible difference from the freshly-zeroed path for any reader that looks at a null slot's value. The clear costs the same memclr `make` was already paying; what is saved is the allocation, i.e. the Go heap lock.

Nested ARRAY/MAP/ROW element storage is append-built and is NOT reset here; callers must not reuse vectors of those types (the join emit path guards them out and mints fresh).

func (*Vector) SetComputedChecked added in v0.18.5 

func (v *Vector) SetComputedChecked(i int, val any) error

SetComputedChecked is SetValueChecked for a caller whose value came out of an EXPRESSION rather than off a wire or a file.

The two differ over exactly one box: an INTEGER. SetValueChecked refuses one into a DECIMAL column because its callers are row→batch adapters, where an integer box is the ALREADY-SCALED carrier of ADR-0018 §4 and storing it as a value would divide it by 10^scale (#547/#541). An expression has no such spelling: `expr.ColRef` over a DECIMAL column boxes the value's rendered TEXT, exact arithmetic boxes text, and the only way an integer reaches a DECIMAL output vector is as a genuine value at scale 0 — the integer branch of a choice construct PostgreSQL types numeric (#695).

So this sibling exists rather than a widening of SetValueChecked: the row adapter keeps its refusal, and the expression sites (exec.Project, physical.aggPreProject and expr.EvalDecimalInto) take this one. It is also what makes the box rule DRIFT-PROOF. expr.decimalChoiceArm classifies arms by node kind to compute the result TYPE, and a kind it has not learned yet makes the fold decline — which used to mean the integer box met the DECIMAL vector the PLAN had already allocated and the query died with a 22003 for a value PostgreSQL answers (`CASE WHEN … THEN d ELSE CAST(i AS BIGINT) END`). The store no longer depends on that classification being complete.

The scaling is checked: an integer too large to carry at this scale is 22003, never a wrapped number.

Two limits it shares with SetValueChecked, both recorded rather than fixed because no SQL surface reaches either today:

  • A box of a type a DECIMAL column cannot take at all — a bool, a []byte — falls through to SetValue, whose mismatch() PANICS. The query boundary recovers it, so a client sees an internal error rather than PostgreSQL's 42804 datatype_mismatch. Nothing in the SQL layer produces such a box for a DECIMAL output: the type fold declines for every non-numeric arm, so the vector would not be a DECIMAL one.
  • Neither writer enforces the DECLARED PRECISION, only the scale, because batch.DecimalColumn carries Scale and no precision. So a value inside the Int128 but past the type's own 10^p band is stored: `GREATEST(numeric(38,30), 100000000::bigint)` writes 39 digits under a type capped at 38. ADR-0024 item 4 makes the declared precision the bound that matters, and the set-operation coercion is the only door that currently enforces it (physical.setOpCheckedDecimalText).

func (*Vector) SetValue 

func (v *Vector) SetValue(i int, val any)

SetValue sets the value at position i from an interface{}. For string/bytes types, values must be set in sequential order (i = 0, 1, 2, ...).

A non-nil value whose type has no conversion into this vector's storage PANICS with *TypeMismatchError (#361) instead of silently keeping the zero value — see that type's doc for the contract and the seams that convert the panic into a query error. nil is a NULL; STRING/BYTES coerce everything through its string form; a parseable-type string that fails to parse (IPv4, MAC, UUID) keeps its historical value-level behavior.

func (*Vector) SetValueChecked added in v0.18.5 

func (v *Vector) SetValueChecked(i int, val any) error

SetValueChecked is SetValue for a caller producing a stored VALUE rather than ingesting an already-encoded one.

SetValue's DECIMAL arms answer a conversion they cannot make exactly with the nearest thing they can store — the saturated end of the Int128 range for text too wide at this scale, zero for text that is not a number, the raw carrier for an integer box, and a float64 round trip for a float box. Each is right for the caller it was built for (a comparison bound, #462; ingest's already-scaled carrier, ADR-0018 §4) and each is a silently wrong ROW anywhere else: a 10^30 union arm came back as 17014118346046923173168730371.5884105727 (#553) and an integer arm came back divided by 10^scale (#547/#541).

So this sibling exists rather than a signature change on SetValue: the unchecked writer keeps its callers and its cost, and every value-producing row→batch path (FromRowsChecked, and through it the single-process set-operation adapter) takes this one. Every DECIMAL box is exact-or-error here — text through the checked parser, a float through its shortest round-trip spelling, an integer refused outright — and the errors carry PostgreSQL's SQLSTATEs: 22003 for a value with no carrier, 22P02 for text that names no number (ADR-0024 item 4).

Every other type, and every other box, delegates to SetValue unchanged.

"Every other type" once included the CONTAINERS, and that was #898: a DECIMAL leaf inside a ROW, an ARRAY or a MAP was written by SetValue's recursion (child.SetValue, appendToVector) and got the unchecked contract back — `not-a-number` stored 0.00, an integer 42 stored 0.42, a 53-digit decimal stored a saturated Int128 — with no error, in the same call whose scalar form refuses all three. So the walk descends: a container is traversed HERE and every leaf takes the checked writer, with the field and element path carried into the message so the refusal names WHICH leaf.

func (*Vector) SetVector 

func (v *Vector) SetVector(i int, vals []float32)

SetVector sets the float32 values for row i of a VECTOR column.

A VECTOR(N) value has exactly N components. A shorter or longer write is refused (see VectorWidthError) rather than padded: it used to copy however many components fit and leave the rest holding whatever was in the slots, so on a pooled batch `SetVector(0, []float32{1})` into a VECTOR(2) read back as [1 8] — the 8 belonging to a previous, unrelated batch — while the same write into a fresh batch read [1 0]. Identical input, two answers, decided by pool history (#900). Padding a short value would be the other way to make the two agree and the wrong one: the caller declared the column's width, and PostgreSQL's vector extension refuses `'[1]'::vector(2)` for the same reason.

func (*Vector) String 

func (v *Vector) String() string

String returns a debug representation of the vector.

func (*Vector) VectorAt 

func (v *Vector) VectorAt(i int) []float32

VectorAt returns a slice of float32 values for row i of a VECTOR column.

func (*Vector) WriteNullAt 

func (v *Vector) WriteNullAt(di int)

WriteNullAt writes a null into position di of a pre-allocated vector, advancing variable-length bookkeeping (bytes offsets, array offsets, row children) so later sequential writes stay aligned. This is THE null-write primitive for indexed sequential writers — any writer that sets the null bit without advancing these slots corrupts every later row in the column.

type VectorWidthError added in v0.18.49 

type VectorWidthError struct {
	Dim int // the column's declared dimension
	Got int // components the writer supplied
}

VectorWidthError reports a write of a VECTOR value whose component count is not the column's declared dimension — the third member of this file's family and the one that is neither a wrong Go type nor a number out of range: the box is right and every component is storable, there is just the wrong NUMBER of them.

Until #900 a short write copied what fit and left the remaining slots holding whatever the storage carried, which on a pooled batch is a PREVIOUS batch's components: `SetVector(0, []float32{1})` into a VECTOR(2) answered [1 0] on a fresh batch and [1 8] on a reused one. Same input, two answers, decided by pool history — and no error either way.

Refusing rather than padding is the choice PostgreSQL's vector extension makes ('[1]'::vector(2) is an error), and it is the only one that keeps a declared width meaningful: a padded value is a DIFFERENT vector, and every distance function would then answer about a value nobody wrote.

func (*VectorWidthError) Error added in v0.18.49 

func (e *VectorWidthError) Error() string

Error is pgvector's wording, so a client sees the message it would see there.

func (*VectorWidthError) FatalEvalError added in v0.18.49 

func (e *VectorWidthError) FatalEvalError() error

FatalEvalError implements the exec.FatalEvalPanic contract: a query error, never a process exit.

func (*VectorWidthError) SQLState added in v0.18.49 

func (e *VectorWidthError) SQLState() string

SQLState is PostgreSQL's data_exception, what pgvector raises for this.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.