kernel

func Accelerated ¶

func Accelerated() bool

Accelerated reports whether hardware SIMD kernels are in use.

func AppendFloatText ¶ added in v0.6.7

func AppendFloatText(dst []byte, v float64, bits int) []byte

AppendFloatText appends FloatText's text of v to dst, one allocation a value at most where FloatText adds one for its string.

func ArgSort ¶

func ArgSort(n int, cmp Comparator) []int32

ArgSort returns the permutation that orders n rows under cmp.

The sort is STABLE, and the stability is load-bearing rather than a nicety:

• It makes a multi-pass sort (`Sort(a)` then `Sort(b)`) mean what people expect — ties on b keep their a order.
• It makes a query's output reproducible when keys contain ties, which is what `dataframe-features.md`'s determinism commitment requires.
• It is what lets ArgTopK be checked against ArgSort at all: without a defined tie order there is no single right answer to compare against.

It uses slices.SortStableFunc rather than sort.SliceStable, which is a typed sort over []int32 rather than a reflective one: sort.SliceStable swaps through reflectlite.Swapper, which the profile put at 5.4% of the engine's hot-path CPU on its own. Prefer ArgSortColumns where the key COLUMNS are available — it avoids comparisons entirely.

func ArgSortColumns ¶

func ArgSortColumns(cols []*data.Column, specs []SortSpec, n int) ([]int32, error)

ArgSortColumns is ArgSort given the key COLUMNS rather than a comparator, and is what every sort in the engine should use when it has them.

Having the columns is what lets the sort key be computed ONCE PER ROW instead of once per comparison, which turns an O(n log n) problem with a four-closure inner loop into a handful of linear passes. See radix.go for the argument; the short version is that OrderKeyF64 was already producing exactly the key a radix sort wants, O(n log n) times.

Falls back to the comparator for key types with no order-preserving encoding — Int128 and so Decimal — and for any mixed set containing one. Text is sorted seven bytes a round since step 363 (textsort.go). The result is identical either way, including tie order, which is what TestRadixMatchesComparatorSort and TestTextSortMatchesTheComparator exist to hold true.

func ArgSortColumnsN ¶ added in v0.6.7

func ArgSortColumnsN(cols []*data.Column, specs []SortSpec, n, workers int) ([]int32, error)

ArgSortColumnsN is ArgSortColumns on up to workers goroutines: the same permutation, ties included, the radix passes and the comparator sort each split.

func ArgSortN ¶ added in v0.6.7

func ArgSortN(n int, cmp Comparator, workers int) []int32

ArgSortN is ArgSort on up to workers goroutines: each worker stably sorts a run of the rows, and runs are merged in pairs, each merge split across workers by merge path, a tie taken from the left run, whose rows come first. The permutation is ArgSort's exactly. cmp must be safe for concurrent use, as a comparator over columns, which only reads them, is.

func ArgTopK ¶

func ArgTopK(n, k int, cmp Comparator) []int32

ArgTopK returns the indices of the k smallest rows under cmp, in order.

The contract ¶

ArgTopK(n, k, cmp)  ==  ArgSort(n, cmp)[:min(k, n)]

EXACTLY — the same indices, in the same order, including for ties. That equivalence is the whole point: it is what makes limit-pushdown (`Sort(...).Head(k)` → top-k) a legal rewrite rather than a silent change of which rows come back.

Why ties need explicit handling ¶

A bounded heap is not naturally stable, and getting this wrong is invisible on data without ties. Two independent bugs had to be fixed here:

1. EVICTION. Among equal elements the heap root is whichever element the heap happened to sift there, not the one that should lose. With values [5,5,1,5] and k=2 the naive version evicts index 0 and returns {2,1}, where ArgSort[:2] is {2,0} — a different row entirely.
2. THE FINAL SORT. Sorting the surviving indices with a stable sort is meaningless, because they are in heap order rather than original order; "stable" preserves an arbitrary permutation. [7,7,7,3] with k=3 returned {3,1,2} against ArgSort[:3] = {3,0,1}.

Both are fixed by making the comparator TOTAL: ties break by original index, in the heap (so the largest index sits at the root and is evicted first) and in the final sort (so the output order is the original order among equals).

Cost is O(n log k) time and O(k) memory rather than O(n log n) and O(n). For `Sort(...).Head(10)` over ten million rows that is the difference between a sort and a scan.

func ArrCall ¶ added in v0.6.7

func ArrCall(fn expr.CallFn, name string, out dtype.DataType, c *data.Column, args []any) (*data.Column, error)

ArrCall evaluates an `.arr` call (steps 341 and 342). An Array is held as a List is, each of its rows its size: to_list is the same column under the List's type, and reverse, sort and to_struct are the List's functions of it, reverse and sort keeping its Array.

func BinCall ¶ added in v0.6.6

func BinCall(fn expr.CallFn, name string, out dtype.DataType, c *data.Column, args []any) (*data.Column, error)

BinCall applies a binary function of one operand.

func Binary ¶

func Binary(op expr.BinaryOp, name string, out dtype.DataType, l, r *data.Column) (*data.Column, error)

Binary applies op to two columns, producing a new column named name with type out (as already determined by expression type resolution).

This function owns null propagation ¶

Kernels compute values; the dispatcher decides which of those values are real. Centralising it here means the rule is written once and every kernel inherits it:

comparison / arithmetic  → valid_out = valid_a AND valid_b
missing comparison       → always valid
Kleene boolean           → computed by the kernel, because it is NOT va AND vb
partial (integer div)    → (va AND vb) AND the kernel's own ok mask

Both operands must already be cast to the binding's operand type. Doing the casting in the evaluator rather than here keeps the dispatcher a pure function of physical types.

func Cast ¶

func Cast(name string, to dtype.DataType, strict bool, c *data.Column) (*data.Column, error)

Cast converts a column to another type.

Non-strict casts turn an unrepresentable value into a null, which is why the type checker marks every non-strict cast nullable regardless of its input. Strict casts fail the query and name the offending row.

A Time target is normalised, and that is a fix rather than tidiness ¶

Cast is one of the two producers of a Time value, and it relabels whenever the two sides carry the same tick length — so `Datetime(s) -> Time(s)` reinterpreted SECONDS SINCE THE EPOCH as a time of day. It printed correctly by accident, because ToTime rebuilds the real instant and the "15:04:05" layout throws the date away, while the stored tick stayed a full epoch offset. Sorting such a column therefore ordered by DATE, and equality across two days never matched.

Normalising into [0, 24h) does not merely restore an invariant here; it makes `Cast(Datetime -> Time)` mean what the user asked for. The wrapping is done once, after every arm, rather than in each of the four that can produce a Time.

Only from an instant ¶

Folding is the time of day of an INSTANT. A bare tick count, or a Duration, has no day to take the time of, and folding it made 90000 seconds 01:00 (audit.md S18). From anything but a Datetime, a value outside the day is refused under a strict cast and null under a lossy one, as Polars does.

func CompilePattern ¶

func CompilePattern(fn expr.CallFn, args []any) (*regexp.Regexp, error)

func Concat ¶

func Concat(schema *dtype.Schema, batches []*data.Batch) (*data.Batch, error)

Concat stacks batches vertically. Used by Collect to assemble the final frame.

func ConcatColumns ¶ added in v0.3.0

func ConcatColumns(name string, parts []*data.Column) (*data.Column, error)

ConcatColumns stacks columns of the same type end to end.

Concat's per-COLUMN half, exported because unpivot needs exactly this and nothing else: it has k columns to lay end to end and no batches to build them from. The name is the caller's, because the parts' names are the ones being melted away.

func ConcatOwned ¶ added in v0.4.0

func ConcatOwned(schema *dtype.Schema, batches []*data.Batch) (*data.Batch, error)

ConcatOwned is Concat for a caller that gives its batches up: it clears the slice it is handed, and lets go of each column's pieces as soon as that column is copied.

Concat holds every input until its last column is done, so the input and the answer are both alive in full, at what is usually the operator's largest moment: twice a join's build side when it freezes, twice a query's result when Collect assembles it, and none of it counted against the budget. Here the two overlap by one column (step 128).

func CutIntervals ¶ added in v0.6.7

func CutIntervals(breaks []float64, leftClosed bool) []string

CutIntervals names the k+1 bins of k breaks by their intervals, each break in decimal, as Polars writes one: 0.0000001, not 1e-07 (step 350).

func DenseIndexFits ¶ added in v0.6.2

func DenseIndexFits(lo, hi int64, n int) bool

DenseIndexFits reports whether keys spanning lo to hi, n of them, are dense enough for a direct index, by Index's rule.

func DtCall ¶

func DtCall(fn expr.CallFn, name string, out dtype.DataType,
	c *data.Column, args []any) (*data.Column, error)

DtCall applies a temporal function to a column.

func EncodeOne ¶

func EncodeOne(c *data.Column) ([]byte, error)

EncodeOne returns the set key for the single row of a length-1 column.

It exists so the probe set and the probed column are encoded by the SAME function. Building the set with any other encoding — even one that looks equivalent — would reintroduce exactly the NaN and -0.0 disagreements the shared encoder exists to prevent.

func EnumPhysical ¶ added in v0.6.6

func EnumPhysical(name string, out dtype.DataType, c *data.Column) (*data.Column, error)

func EwmNext ¶ added in v0.6.3

func EwmNext(fn expr.WinFnOp, params expr.WinParams, name string, out dtype.DataType,
	col *data.Column, st *EwmState) (*data.Column, error)

EwmNext answers col, the next rows of a partition in its order, continuing st.

func Extend ¶ added in v0.4.0

func Extend[T any](s []T, n int, fill T) []T

Extend returns s at length n, the new elements set to fill, doubling its capacity when it must grow.

Every accumulator's Reserve grows its per-group arrays here. They appended a group at a time, so past 256 groups each grew by append's quarter steps, and the allocations on the way to a final size summed to about five times it: in h2o gb10 the sum and the count were 12% of everything the query allocated (step 132). Doubling sums to about twice. It can leave half an array unused, which is why each NBytes counts capacity, not length (step 134).

func FilterBatch ¶

func FilterBatch(b *data.Batch, mask *data.Column) (*data.Batch, error)

FilterBatch applies a Boolean mask to every column of a batch.

func FirstSeenIDs ¶ added in v0.6.7

func FirstSeenIDs(op string, c *data.Column) (ids, firsts []int32, _ error)

FirstSeenIDs numbers a column's distinct values in the order each first appears: ids[i] is row i's value's number, and firsts[k] the row value k first appears at. A null is a value, once. Values are equal as group keys are, so NaN is one value and -0.0 is +0.0 (step 320, for to_dummies).

Each row is a key of one of a group-by's tables, given to it a chunk at a time: an integer column's values themselves, a float column's bits, made one for every NaN and for -0.0 and +0.0, a String or Binary column's own bytes, any other column's encoded keys. A column that cannot be a key is refused, op naming the operation.

A long column is numbered in parts, in parallel, each in a table of its own. The parts' values are then numbered again in one table, part by part in order, each by the row it first appears at in its part, so a value's number is where it first appears in the whole column; and each part's ids are renumbered, in parallel.

func FirstSeenKeys ¶ added in v0.6.7

func FirstSeenKeys(op string, cols []*data.Column) (ids, firsts []int32, _ error)

FirstSeenKeys is FirstSeenIDs of the keys columns make together, a row's values in each, as a group-by keys by several columns: encoded, so that they are equal as one column's values are (step 321, for PartitionBy). One column is FirstSeenIDs'.

func FloatText ¶ added in v0.6.7

func FloatText(v float64, bits int) string

FloatText is a float as Polars casts one to text (step 325; it was Go's shortest text, 2 and -0 and 1e-07): the shortest digits that read back as it at its width, plain from 1e-5 to below 1e16, a whole number with ".0", and otherwise in scientific notation with a signed exponent of no padding, "1e+16" and "1.5e-7"; NaN, inf and -inf.

func Hash ¶ added in v0.6.6

func Hash(name string, c *data.Column, seed uint64) (*data.Column, error)

Hash is each value's 64-bit hash, seeded: FNV-1a over the bytes the group-by keys it by, so values a group-by takes as one hash alike, -0 with 0 and NaN with NaN, and a null hashes too; then mixed by splitmix64's finalizer. It is its own algorithm, not Polars' (v0.6.6-scope.md §6.3).

func HashKey ¶

func HashKey(b []byte) uint64

HashKey hashes an encoded group key.

It must be the ENCODED key, never the raw column values ¶

NewGroupKeyEncoder is the single authority on grouping equality: floats go through OrderKey first, so NaN groups with NaN and -0.0 with +0.0. A hash over raw bits would send the two NaNs of one column to different radix partitions, they would be aggregated by separate sinks, and the output would carry TWO rows where one belongs — each with a plausible sub-total and no error anywhere. nuniqueAcc reuses this encoder for exactly the same reason.

Seedless and deterministic, deliberately ¶

hash/maphash is faster and its seed can only come from MakeSeed(), so the group ORDER of a spilled aggregation would differ between processes — a flake that reproduces only under a memory limit and only sometimes. FNV-1a costs a multiply-xor per byte on a path that is already doing file I/O per batch.

The splitmix64 finaliser is not decoration: FNV avalanches poorly in its low bits, and the low bits are exactly what a radix partitioner reads.

func HorizontalCall ¶ added in v0.5.0

func HorizontalCall(fn expr.CallFn, name string, out dtype.DataType,
	ops []*data.Column) (*data.Column, error)

HorizontalCall applies a horizontal function (step 146) to its operands, each a column of the batch's height or a literal of length 1, typed out by expr.ResolveHorizontal.

func InInts ¶ added in v0.6.0

func InInts(name string, c *data.Column, members []int64, lookup map[int64]struct{}) *data.Column

InInts is InSet for an integer column, against the set's members as IntMembers recovers them once per query: a row's value is read from the column and compared, with no key built for it. As InStrings, a few members are compared one by one, and more through a map. lookup is the members as a map, or nil when there are few.

func InSet ¶

func InSet(name string, c *data.Column, set map[string]struct{}) (*data.Column, error)

InSet tests each row for membership in a precomputed set of encoded values.

The set uses GROUPING equality, not IEEE equality ¶

Keys are produced by the same GroupKeyEncoder that n_unique, distinct and the join all use, which canonicalises floats through OrderKey: NaN equals NaN and -0.0 equals +0.0. An is_in built on `==` would disagree with Distinct and GroupBy about the same values, which is the trap nuniqueAcc's doc records for n_unique.

A null is never a member ¶

Output validity is the input's, so `null.IsIn(...)` is NULL rather than false — the rule every non-total unary follows, and the one that keeps null distinct from "absent from the set". The set itself cannot contain a null, because the public signature takes Go values and Go has no null.

func InStrings ¶ added in v0.4.0

func InStrings(name string, c *data.Column, members []string, set map[string]struct{}) *data.Column

InStrings is InSet for a String or Binary column, against the set's members as plain strings, which StringMembers recovers once per query: a row is compared as it is stored, with no key built for it. A few members are compared one by one; a map costs more than that until there are about eight.

PDS-H q12 and q19 test `IsIn` on string columns over millions of rows, and InSet built every row's grouping key to look it up: a fifth of q12's CPU.

func IntHash ¶ added in v0.6.0

func IntHash(k int64) uint64

IntHash is the hash IntKeyTable gives k.

func IntKeys ¶ added in v0.6.0

func IntKeys(c *data.Column, scratch *[]int64) []int64

IntKeys is c's values as IntKeyTable's keys, widened to int64 by their bits: an Int64 column's own values, without a copy, and any other's widened into *scratch, which is grown as needed and kept for the next call. The result is read only.

A null row's value is whatever its slot holds, and a payload-free column, all null, reads as zeros: a caller skips null rows by the column's validity. c must be IsIntKey.

func IntMembers ¶ added in v0.6.0

func IntMembers(set map[string]struct{}, dt dtype.DataType) ([]int64, bool)

IntMembers recovers the integers of a set of keys one integer column of type dt encoded, widened as IntKeys widens a column (step 164): keyWriter's 0x01 and the value's bytes, big-endian, a signed type's sign bit flipped. It reports false for a set of any other shape, or a dt that is not IsIntKey.

func InterpolateByFrom ¶ added in v0.6.7

func InterpolateByFrom(name string, out dtype.DataType, rows int64,
	prevVal, prevAt, own, nextVal, nextBack, prevBy, nextBy, ownBy *data.Column) (*data.Column, error)

InterpolateByFrom is interpolate_by's answer in a sorted pass (step 329), from its steps' columns in by's order: the last value, its place, each row's own place, the next value and its place counted from the end, as InterpolateFrom reads them; and the by of the last value, of the next one, and of the row. Its arithmetic is fillGaps'.

func InterpolateFrom ¶ added in v0.6.4

func InterpolateFrom(nearest bool, name string, out dtype.DataType, rows int64,
	prevVal, prevAt, own, nextVal, nextBack *data.Column) (*data.Column, error)

InterpolateFrom answers interpolation for rows of a partition from what a walk each way over it in its order found (step 238), as winInterpolate and winInterpolateNearest answer them over the partition whole:

• prevVal and prevAt are the value of the last row at or before each row that holds one, and that row's position, both null before the first;
• own is each row's 1-based position, cum_count's;
• nextVal and nextBack are the value of the first row at or after it that holds one, and that row's position counted from the end, both null after the last;
• rows is the partition's.

A row holding a value is its value. A row with no value on one side stays null. Linear takes the point on the line between the two, by position, with the kernel's arithmetic; nearest the nearer of the two, the later on a tie, of the column's own type.

func IsIntKey ¶ added in v0.6.0

func IsIntKey(dt dtype.DataType) bool

IsIntKey reports whether a key of type dt is IntKeys's: stored as an integer of at most 64 bits.

func JSONEncode ¶ added in v0.6.3

func JSONEncode(name string, c *data.Column) (*data.Column, error)

JSONEncode encodes each row of a Struct column.

func KeyHash ¶ added in v0.5.0

func KeyHash(key []byte) uint64

KeyHash is the hash a KeyTable gives key, for a caller that routes keys to tables before inserting them (GetOrInsertHashed).

func ListCall ¶ added in v0.3.0

func ListCall(fn expr.CallFn, name string, out dtype.DataType,
	c *data.Column, args []any, needle []byte) (*data.Column, error)

ListCall applies a list function to a column.

The signature mirrors StrCall, and out comes from expr.ResolveCall so the kernel and the plan cannot disagree about the result type.

func MathCall ¶

func MathCall(fn expr.CallFn, name string, out dtype.DataType,
	c *data.Column, args []any) (*data.Column, error)

MathCall evaluates the parameterised maths family.

Round, two helpers the window sugar uses, and the bit counts. The family exists because a parameter belongs in a Call's argument list rather than in a field on expr.Unary — see the note on FnMathRound for the three hand-written Unary rebuilds that would have dropped it.

func Mix64 ¶

func Mix64(x uint64) uint64

Mix64 is splitmix64's finaliser: a bijection that spreads every input bit over the whole output.

func NullColumn ¶

func NullColumn(name string, dt dtype.DataType, n int) (*data.Column, error)

NullColumn builds an all-null column of dt with n rows and a REAL payload buffer.

Why this is not data.NewNull ¶

data.NewNull carries no payload at all — "which makes a typed null literal free". That is right for a value nobody reads, and wrong for anything a kernel then gathers from or concatenates. Take and Concat both route fixed-width columns through data.Values, which refuses a column with no buffer, and the string path builds a zero-length StringAccessor whose Get PANICS on any index.

Both are reachable from ordinary queries. Min over a group whose values are all null produced an unread single-row null that assembleRows then concatenated — so `GroupBy(g).Agg(Col("v").Min())` failed with "column has no fixed-width payload" whenever one group was entirely null. An outer join has the same shape: the unmatched side is gathered from padding on every row.

The rule this establishes: data.NewNull is for a column that is the ANSWER; NullColumn is for a column that is an OPERAND.

func OrderKeyF32 ¶

func OrderKeyF32(f float32) uint32

OrderKeyF32 is OrderKeyF64 for 32-bit floats.

func OrderKeyF64 ¶

func OrderKeyF64(f float64) uint64

OrderKeyF64 maps a float64 onto a uint64 whose UNSIGNED ordering is ursus's documented total order:

-Inf < … < -0.0 == +0.0 < … < +Inf < NaN

with all NaNs equal to each other.

Why this deliberately is not IEEE ¶

IEEE says NaN compares false against everything, including itself. That is correct for `Expr.Lt` — asking whether a missing measurement is less than five should not invent an answer — but it is fatal for sorting and grouping:

• A comparison-based sort needs a TOTAL order. With IEEE semantics NaN is unordered, sort becomes non-deterministic, and some algorithms fail to terminate.
• Hash grouping needs an equivalence relation. With IEEE, NaN != NaN, so every NaN would form its own group — you could not count them.

So ursus uses IEEE for comparison expressions and this total order for sort, top-k and group-by. The two must never be confused; that is why this lives in its own file with its own name rather than as a flag on the comparison kernels.

The bit trick: for a positive float, flipping the sign bit gives a uint64 that orders correctly. For a negative float, inverting every bit does. Both are exact and branch-light.

func OrderSign ¶ added in v0.6.6

func OrderSign(name string, c *data.Column) (*data.Column, error)

OrderSign is -1, 0 or 1 as each value sorts before, with or after its type's zero: a number's sign, NaN's 1 and -0's 0; an instant's beside the epoch; a Boolean's 1 for true. Null is null.

func PartitionOf ¶ added in v0.5.0

func PartitionOf(h uint64, n int) int

PartitionOf is the partition, of n, a key of hash h belongs to.

func RankNext ¶ added in v0.6.4

func RankNext(method expr.RankMethod, name string, col *data.Column, st *RankState) (*data.Column, error)

RankNext ranks col, the next rows of a partition in rank's order, continuing st: by position, by peer group, or by the group's first position, as winRank's ordinal, dense and min. Two values tie when their order does; a null has no rank and takes no position, as winRank's. A group's last position, which max and average need, is not known until the group ends; the caller has it from min over the order reversed.

func RollingByNext ¶ added in v0.6.4

func RollingByNext(fn expr.WinFnOp, params expr.WinParams, name string, out dtype.DataType,
	pair *data.Column, st *RollingByState) (*data.Column, error)

RollingByNext answers fn by time over pair, the next rows of a partition in their by order, continuing st. See RollingByState.

func Select ¶

func Select(name string, mask, a, b *data.Column) (*data.Column, error)

Select is the masked merge behind When/Then/Otherwise: out[i] is a[i] where the mask is true and b[i] where it is false.

A null mask takes NEITHER branch ¶

This is the same rule SelectionFromMask applies for Filter — "a row survives iff its predicate is VALID AND TRUE" — and it is the one decision here that a reader might expect to go the other way. `When(x.Gt(5)).Then(a).Otherwise(b)` on a row where x is null produces NULL, not b. Treating a null predicate as false would make Otherwise silently absorb missing data, which is exactly the conflation of "absent" with "false" that Filter(p) and Filter(Not(p)) not partitioning exists to prevent.

The mask's payload bit is not trustworthy on its own ¶

Arrow does not guarantee the value bit of a null slot is zero, so every read of the mask is ANDed with its validity before use. dispatch.go's kleene kernel canonicalises for the same reason and says so.

Broadcasting is this kernel's job ¶

Eval returns whatever length a sub-expression produced, and litColumn produces length-1 columns; the operator-boundary broadcast in evalColumn runs afterwards, too late to help. So `When(c).Then(Lit(1)).Otherwise(Lit(0))` arrives here as a length-n mask and two length-1 branches, and this kernel repeats them — the same accommodation kernel.Binary makes via broadcastLen.

a and b must already share a type: the evaluator casts them to the promoted type first, obeying the resolved binding rather than re-deriving it.

func SelectionFromMask ¶

func SelectionFromMask(mask *data.Column) ([]int32, error)

SelectionFromMask turns a Boolean column into the list of row indices to keep.

Filter's null semantics, in one line of code ¶

A row survives iff its predicate is VALID AND TRUE. A null predicate drops the row, matching SQL's WHERE. This is worth being explicit about because the alternative — treating null as true — is a plausible-looking choice that silently changes results, and because it means Filter(p) and Filter(Not(p)) do NOT partition the input: a row where p is null is dropped by both.

func SparseRemap ¶ added in v0.5.0

func SparseRemap(src, dst []int32) []int32

SparseRemap is a remap that merges only src[j], into dst[j] (Accumulator.Merge). A group-by's partitioned fold merges a worker's groups into each partition this way, at a cost of the groups that go there, where a dense remap of the others' -1s costs every group of the worker for every partition (step 150).

func StrCall ¶

func StrCall(fn expr.CallFn, name string, out dtype.DataType,
	c *data.Column, args []any, re *regexp.Regexp) (*data.Column, error)

StrCall applies a string function to a column.

re is the compiled pattern for the regex forms, or nil for literal matching. It is compiled ONCE per expression node by the evaluator, never per batch and never per row — RE2 compilation is orders of magnitude more expensive than the match.

func StringMembers ¶ added in v0.4.0

func StringMembers(set map[string]struct{}) ([]string, map[string]struct{}, bool)

StringMembers recovers the plain strings of a set of keys a single String or Binary column encoded, as keyWriter writes them: 0x01, the length as four big-endian bytes, then the bytes. It reports false for any key not of that shape.

func StructCall ¶ added in v0.3.0

func StructCall(fn expr.CallFn, name string, c *data.Column, args []any) (*data.Column, error)

StructCall applies a struct function to a column.

The output type is not passed in, unlike ListCall's: a field's type comes from the field itself, and expr.ResolveCall has already refused a name the struct does not have. Taking it as an argument would be a second copy of that decision.

func Take ¶

func Take(c *data.Column, sel []int32) (*data.Column, error)

Take gathers rows by index, producing a compacted column.

This kernel is named in none of the design documents and is needed by every operator that changes row count — filter, join, sort, top-k, explode. It is the single most reused primitive in the engine.

A NullIndex entry produces a null, which is what outer joins and out-of-bounds gathers need.

func TakeChunked ¶ added in v0.6.7

func TakeChunked(c *data.Column, sel []int32, chunk int, run func(n int, fn func(k int) error) error) (*data.Column, error)

TakeChunked is Take of a long selection, a chunk of rows at a time, the chunks run by run, which may run them at once: a fixed-width or a string column's chunks each write their own rows of one answer, so no part is joined after, and the answer is Take's. Any other column is taken whole, by Take.

func ToPhysical ¶ added in v0.6.7

func ToPhysical(name string, out dtype.DataType, c *data.Column) (*data.Column, error)

EnumPhysical is an Enum's indices at out's width, Polars' cat.physical: each index is below the category count, which out holds (step 273). ToPhysical is c as stored, under its physical type out, Polars' to_physical (step 324): a relabel of the values, but for an Enum, whose categories are narrowed to out as cat.physical narrows them.

func Unary ¶

func Unary(op expr.UnaryOp, name string, out dtype.DataType, c *data.Column) (*data.Column, error)

Unary applies a one-operand operation.

func ValidUTF8 ¶ added in v0.6.7

func ValidUTF8(b []byte) bool

ValidUTF8 reports whether b is UTF-8, as utf8.Valid does, thirty-two bytes a step where the CPU has AVX2 (step 357). utf8.Valid reads eight bytes of ASCII a step until the first character past it, and a byte a step from there, so that text of a non-ASCII character a word, 東京 or Zürich, was checked at 1.6 GB/s.

func VectorBits ¶

func VectorBits() int

VectorBits returns the active SIMD vector width in bits, or 0 when scalar.

func WinCall ¶

func WinCall(fn expr.WinFnOp, params expr.WinParams, name string, out dtype.DataType,
	col *data.Column, fill *data.Column, seg Segments) (*data.Column, error)

WinCall applies an ordered window function.

col is the function's input in ORIGINAL row order — Perm indexes into it. fill is Shift's replacement for vacated rows, or nil for null.