Help us shape the future of Go! Take the 2026 Go Developer Survey

expr

package
v0.6.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 12, 2026 License: MIT Imports: 12 Imported by: 0

Documentation

Overview

Package expr is ursus's expression IR.

What an expression is

An expression is a lazy description of a column transformation. `Col("a").Add(1)` builds a tree; it computes nothing. The tree is placed in a *context* — Select, WithColumns, Filter, GroupBy.Agg — and the context decides what the expression means and what shape its output must be.

The one method that matters

Field resolves an expression's output name, dtype and nullability against an input schema, WITHOUT TOUCHING DATA. Everything the lazy engine promises rests on it:

  • LazyFrame.CollectSchema tells you the result schema before reading a byte.
  • A typo or type error is reported at plan time, not after a 40-minute scan.
  • Projection pushdown can ask "which columns does this expression need".
  • The evaluator has a contract to check itself against: for every node n and batch b, Eval(n, b).DType() must equal n.Field(b.Schema()).Type.

The design documents named this method and never defined it. It is the reason the walking skeleton exists.

Sealed

Node is sealed by the unexported node() method. Every implementation lives in this package, so a type switch over Node is exhaustive by construction and the compiler tells you when a new node type needs handling somewhere.

Index

Constants

View Source 
const (
	PairFirst  = "0"
	PairSecond = "1"
)

The field names PairOf gives its two inputs. Only their positions are read; the names are fixed so the two can never collide, whatever the inputs are called.

View Source 
const (
	ZoneRaise    = "raise"    // refuse the query, naming the wall clock; the default
	ZoneEarliest = "earliest" // a fold's first reading, at the offset before it
	ZoneLatest   = "latest"   // a fold's second reading
	ZoneNull     = "null"     // a null
)

The policies replace_time_zone takes for a wall clock its new zone reads twice, in a fall-back fold, or never, in a spring-forward gap: Polars' own words, which the plan carries as its third and fourth args.

Variables

View Source 
var ErrUnexpanded = errors.New("expression contains an unexpanded multi-column selection")

ErrUnexpanded is returned when a tree still containing a *Match reaches Field. It always means a caller skipped Expand, i.e. it is an ursus bug rather than a user error.

Functions

func BareColumns 

func BareColumns(n Node) []string

BareColumns returns the columns an expression reads OUTSIDE any aggregate.

These are the columns that make an expression non-aggregating, and naming them is what turns "invalid aggregate expression" into a message that says which column is the problem and what to do about it.

func CallArgs 

func CallArgs(c *Call) ([]any, error)

CallArgs extracts the literal arguments of a Call, requiring each to be a literal rather than a computed expression.

Column-valued arguments are expressible in the IR and rejected here, so the error names the limitation rather than the evaluator failing on a shape it did not expect.

func CutBreaks added in v0.5.0 

func CutBreaks(ops []dtype.Field) int

CutBreaks is how many of a cut's operands after the first are breaks: those before the first label, which is a String.

func FirstErr 

func FirstErr(n Node) error

FirstErr returns the first deferred error carried anywhere in the tree.

Expression construction cannot return an error without destroying chaining (`Col("a").Gt(5)` has nowhere to put one), so a failure such as a bad regex is parked in an *Err node and surfaces here — the expression-level half of the sticky-error model that LazyFrame implements at the frame level.

func FitsExactly 

func FitsExactly(v any, to dtype.DataType) bool

FitsExactly reports whether the Go value v is representable in to with no loss.

Why this is a SECOND range check

The kernel's cast rules answer a nearby question, and reusing them is impossible: internal/expr is L20 and internal/kernel is L30, so the import is a levels violation. Two copies of a rule is the shape this codebase has been bitten by before, so the two are pinned together by a differential test rather than by hope. See TestWeakFitAgreesWithCast.

Nearby, not the same, since step 73: a cast to a float ROUNDS to the nearest value of its width, where this still asks whether the value is EXACT. So a strict cast no longer refuses what this should have refused — 0.1 to Float32 converts — and a mistake here would round silently where it used to fail loudly. The round-trip half of TestWeakFitAgreesWithCast and its "never" cases are what stand in for that backstop now.

It is deliberately CONSERVATIVE at the edges — above 2^53 for a float target, and for every type it does not enumerate. Being conservative costs nothing: each such case falls back to the type ordinary promotion would have chosen anyway.

The switch is on to.ID() and exhaustive, never on BitWidth(): Decimal reports 128 bits and Enum reports 32, so a width-based rule would silently admit both.

func GridRefusal added in v0.5.0 

func GridRefusal(fn CallFn, iv dtype.Interval, in dtype.DataType) error

GridRefusal is dt.truncate's and dt.round's refusal of their interval, for a receiver of type in. truncateOut asks it at plan time and the kernel at run time, which kernel.DtCall's callers reach without resolving, so the two give one answer (TestTruncateRefusalsAgree).

func HasAgg 

func HasAgg(n Node) bool

HasAgg reports whether the tree contains an aggregate.

func HasMatch 

func HasMatch(n Node) bool

HasMatch reports whether the tree contains an unexpanded multi-column selector.

func HasUDF added in v0.3.0 

func HasUDF(n Node) bool

HasUDF reports whether the tree contains a user-defined function.

The optimizer consults this before DUPLICATING an expression. Substituting a column's definition into a predicate is how predicate pushdown rewrites a filter into its child's namespace, and for ordinary expressions that trade is fine: the definition is cheap and the pushdown saves downstream work.

For a UDF it is a pure loss, and arithmetically so. Pushing `Filter(w > 1)` below the WithColumns that defines w runs the function on every input row IN THE FILTER, and the WithColumns still has to run it on every survivor to produce the column — n + survivors calls where not pushing costs n. There is no input for which the rewrite wins, and the function may be arbitrarily expensive.

func HasUnboundAgg 

func HasUnboundAgg(n Node) bool

HasUnboundAgg reports whether the tree contains an aggregate that is NOT enclosed by a window.

Why this is separate from HasAgg

`Col("x").Mean().Over(Col("g"))` contains an *Agg, so HasAgg says true — and the row-preserving contexts use that answer to refuse aggregates. But a window preserves height: the mean is computed per partition and written back to every row, so the expression is exactly as legal in Select as `x + 1` is. An Agg beneath a Window is BOUND by it, the way a bound variable is bound by its quantifier.

HasAgg is left alone rather than redefined, because its other caller wants the literal question: IsAggregation uses it to reject `sum(a).mean()`, and an aggregate of a windowed aggregate is just as meaningless as an aggregate of an aggregate.

Partition and order keys are still descended into, so `Sum(x).Over(Sum(g))` is refused — a partition key must be a per-row value.

func HasWindow 

func HasWindow(n Node) bool

HasWindow reports whether the tree contains a window or an ordered window function.

A bare WinFn counts: `Col("x").CumSum()` with no .Over() is a window over the whole frame, and it is normalised into one rather than being a second shape the planner has to know about.

func Identity added in v0.3.0 

func Identity(n Node) string

Identity is what two expressions must share to be one computation.

String is a rendering, not an identity

String is for Explain, the golden plans and error messages, and it leaves two things out:

  • A strong literal renders its value and not its type. Lit(int8(100)) and Lit(int64(100)) both render "lit(100)", as do Lit(1) and Lit(1.0), and float32(0.1) and 0.1. They compute different things: i8 + int8(100) wraps in Int8, and i8 + int64(100) does not.
  • A UDF renders its name, and two different functions can share one.

Four map instances merged computations on String — the window temporaries, the aggregate specs of a group-by and of a temporal group, and the window partitionings — so the second of two literals that render alike got the first one's answer, or the first one's type under its own schema (audit.md, O3).

Identity is String followed by what the rendering leaves out: for every literal, its Go type and its DataType; for every UDF, the id NewUDF minted. Every part is length-prefixed, so no two different sequences of parts concatenate alike — a DataType's own rendering can hold anything, an Enum's categories for one. It is never shown to anyone, which is what lets it carry an id that would make every golden plan depend on allocation order.

Why a rendering, and not the values compared with ==

== merges −0 and +0, and x/−0.0 and x/0.0 are −Inf and +Inf. It never merges NaN with NaN, which compute alike. And a []byte literal in a map key panics. The rendering has none of those problems: fmt prints the shortest string that reads back as the same float, so −0 prints "-0", and every NaN prints "NaN".

Why a string, and not a rewritten tree

A tree with the types written into it would need a Node type of its own, in a package whose type switches promise to be exhaustive over Node.

func IdentityAll added in v0.3.0 

func IdentityAll(ns []Node) string

IdentityAll is Identity for a list of expressions, where the list is the unit — a window's partition keys.

func IsAggregation 

func IsAggregation(n Node) bool

IsAggregation reports whether the expression is a well-formed aggregate: it reduces its input to one row.

An expression is an aggregation if every path from the root to a Col passes through an Agg. `sum(a) + 1` and `sum(a) / count()` qualify; `sum(a) + b` does not, because b is still a per-row value and there is no defined way to combine the two.

func IsWeakLit 

func IsWeakLit(n Node) bool

IsWeakLit reports whether n is a literal whose type should adapt to its context.

func MembershipType added in v0.3.0 

func MembershipType(c *Call, against dtype.DataType) (dtype.DataType, error)

MembershipType is the type is_in and list.contains compare at: the type Eq would compare the receiver — for list.contains, its element — and each value at. So `x.IsIn(v)` refuses exactly the pairs `x.Eq(v)` refuses, and answers the same rows wherever both are defined.

The values used to be cast to the receiver's type, which answered a different question. A value the column cannot hold — 5000 against an Int8, −1 against a Uint64 — was an error where Eq says false; a string, a bool or an instant was parsed or converted, so an Int64 matched "1", a Bool matched 1 and a Date matched 13:00 on that day, all pairs Eq refuses; and a value was rounded to the column's precision, so a Datetime(s) matched 00:00:01.5 and a Float32 matched 0.1.

Equality is still grouping equality, so NaN matches NaN — the one place the two differ, and on purpose.

func OffsetRefusal added in v0.5.0 

func OffsetRefusal(iv dtype.Interval, in dtype.DataType) error

OffsetRefusal is dt.offset_by's refusal of its interval, for a Date or Datetime of type in, asked at plan time and by the kernel as GridRefusal is.

The answer has the receiver's type, so an offset it cannot hold is refused rather than rounded: an hour is no number of days, and a nanosecond no number of microseconds.

func OrderedAggRefusal added in v0.4.0 

func OrderedAggRefusal(op AggOp) *uerr.Error

OrderedAggRefusal refuses an ordering over an aggregate, here and in the physical planner's backstop.

The hint depends on the aggregate. It used to call every one of them order-independent "except for first, last, arg_min, arg_max and implode", which said both things of first (audit A12).

func OutputName 

func OutputName(n Node) string

OutputName returns the name the expression's result column will take.

The rule is LEFTMOST COLUMN WINS: `Col("a").Add(Col("b"))` is named "a", matching Polars. An explicit Alias anywhere above overrides it. An expression with no column reference at all is named "literal".

Naming has to be defined structurally like this because Select produces a schema, and a schema needs names before any data exists. Naming nodes are only consulted at the ROOT of the tree. They do not descend through operators: `Col("a").Alias("x").Add(1)` is named "a", not "x", because the alias applies to the operand rather than to the sum.

func Resolve 

func Resolve(n Node, in *dtype.Schema) (dtype.Field, error)

Resolve is the entry point for expression type resolution.

It validates the tree once — deferred errors, leftover matchers — and then delegates to Field. Field implementations assume a resolvable tree, so the validation is not repeated at every level; doing it inside Field would make resolution quadratic in tree depth.

func ResolveAgg 

func ResolveAgg(op AggOp, in dtype.DataType) (dtype.DataType, error)

ResolveAgg returns just the output type, for callers that do not run the accumulator.

func ResolveCall 

func ResolveCall(c *Call, in dtype.DataType) (dtype.DataType, error)

ResolveCall gives the output type of a call applied to a receiver of type in.

It is the single authority, consulted by Field for the plan's schema and by the evaluator for the kernel's output type. Two copies would drift, which is the Binding lesson.

It takes the whole Call rather than just the function because `.struct.field` is the first call whose output type depends on an ARGUMENT — `field("age")` is Int64 and `field("city")` is String, from the same receiver.

It used to add "Every other family answers from the receiver's type alone", and that was false. `.dt.truncate` is the second, and the divergence lived exactly in the sentence: the kernel routes on the interval — `truncate(Time, 1 month)` has no answer while `truncate(Time, 1 hour)` does — so a resolver that saw only the receiver promised both. The argument is a CONSTANT of the expression, not data, so there was never a reason the planner could not see it.

func ResolveCond 

func ResolveCond(thenN, elseN Node, then, els dtype.DataType) (dtype.DataType, error)

ResolveCond gives a conditional's output type from its two branches.

It is the single authority, consulted by Field for the plan's schema and by the evaluator for the type it casts both branches to before the kernel. Two copies would drift, which is the Binding lesson.

Why it takes NODES as well as types

Weak literal typing needs to know which branch is a literal whose type was a default rather than a choice, and a dtype.DataType cannot carry that. The evaluator in particular has only the evaluated COLUMNS by the time it asks, and a column has no weakness — so the nodes have to come from the call site. Both callers hold them, so this is a signature change rather than a redesign; the alternative, a second resolver that only the planner consults, is exactly the drift this function exists to prevent.

The temporal refusal is worth a specific hint. Promote deliberately declines two DIFFERENT temporal types — it returns early unless both sides are numeric — even though CanCast permits temporal-to-temporal and kernel.Cast rescales such a pair correctly. Widening Promote would edit a rule every binary operator shares, so the conditional points at the explicit cast instead of quietly guessing.

func ResolveHorizontal added in v0.5.0 

func ResolveHorizontal(fn CallFn, ops []dtype.Field) (dtype.DataType, error)

ResolveHorizontal gives the output type of a horizontal call over operands of these fields. Like ResolveCall, it is the one authority, for the plan's schema and the kernel's output alike.

func ResolveUnary 

func ResolveUnary(op UnaryOp, in dtype.DataType) (dtype.DataType, error)

ResolveUnary computes the result type of op applied to in.

func ResolveWinFn 

func ResolveWinFn(fn WinFnOp, p WinParams, in dtype.DataType) (dtype.DataType, error)

ResolveWinFn gives an ordered window function's output type.

The single authority, consulted by Field for the schema and by the evaluator for the kernel's output type — the Binding lesson, applied a fourth time.

func RootNames 

func RootNames(n Node) []string

RootNames returns the distinct column names the expression reads, in first-seen order.

This is the input to projection pushdown: the union of RootNames over every expression above a scan is exactly the set of columns that scan must produce. It is deliberately conservative — an unexpanded *Match contributes nothing here, which is why Expand must run first.

func StringAll 

func StringAll(ns []Node) string

StringAll renders a slice of nodes for Explain output.

func Walk 

func Walk(n Node, fn func(Node) bool)

Walk calls fn for every node in the tree, parents before children (pre-order). Returning false from fn stops the traversal of that subtree.

func WinFnType added in v0.3.0 

func WinFnType(f *WinFn, child dtype.DataType, in *dtype.Schema) (dtype.DataType, error)

WinFnType is an ordered window function's output type, its fill included: the one authority Field and the physical planner both ask.

A shift's fill used to be ignored here, so the output was always the column's type, the fill kept its own, and the kernel's select between the two refused them at run time — ShiftFill(1, 1.5) on an Int64 column was ErrInternal. The fill now meets the column as a When's branches do, through ResolveCond: a weak literal that fits takes the column's type, and otherwise the two promote, so 1.5 on an Int64 column is a Float64, which is the type Polars collects too.

Types

type Agg 

type Agg struct {
	Op     AggOp
	Child  Node
	Params AggParams
}

Agg applies an aggregate function.

Only inside an aggregation context: GroupBy().Agg(...), or (later) a window via .Over(). Its defining property is that it consumes many rows and produces one, so it cannot appear in Select or WithColumns, where every expression must preserve the frame height. plan.Resolve enforces that — an Agg reaching a Project is rejected with an explanation rather than silently producing a one-row column that then fails to line up.

func (*Agg) Children 

func (a *Agg) Children() []Node

func (*Agg) Field 

func (a *Agg) Field(in *dtype.Schema) (dtype.Field, error)

func (*Agg) String 

func (a *Agg) String() string

String renders the aggregate INCLUDING its parameters, and that is a correctness requirement rather than a display choice.

physical.extractAggs deduplicates inner aggregates by keying a map on this string, so two aggregates that render identically become ONE temporary. Before the parameters were rendered, Agg(x.Quantile(0.5), x.Quantile(0.99)) produced "x.quantile()" twice, collapsed into a single computation, and returned the median under both names with no error anywhere.

type AggBinding 

type AggBinding struct {
	Acc dtype.DataType
	Out dtype.DataType
}

AggBinding is the resolved form of an aggregate: the type it ACCUMULATES in and the type it OUTPUTS.

The two differ, and the difference is load-bearing:

  • Sum of Int8 accumulates in Int128, because a thousand Int8 values overflow an Int8 immediately and a wrapped sum is a plausible wrong number.
  • Sum of Float32 accumulates in Float64 and outputs Float32, because naive float32 accumulation loses everything past ~2^24 elements. Polars has a live bug from exactly this.
  • Mean accumulates a Float64 sum and a count, and outputs a float.

Returning only the output type — as an earlier draft did — leaves the accumulator type implicit, so the first kernel written re-derives it. That is the same promotion-drift failure mode expr.Binding exists to prevent for binary operators, and it is exactly how a Float32 mean ends up silently wrong.

func ResolveAggBinding 

func ResolveAggBinding(op AggOp, in dtype.DataType) (AggBinding, error)

ResolveAggBinding computes an aggregate's accumulator and output types.

type AggOp 

type AggOp uint8

AggOp is an aggregate function. 

const (
	AggSum AggOp = iota
	AggMean
	AggMin
	AggMax
	AggCount   // non-null values
	AggLen     // rows, including nulls
	AggNUnique // distinct non-null values
	AggFirst
	AggLast

	// APPEND ONLY below this line. IsCounting and the classifiers are written
	// against declaration order, and an op inserted mid-enum silently changes its
	// neighbours' family — the same rule CallFn and UnaryOp document.
	AggNullCount // nulls, the complement of Count
	AggAny       // Boolean OR over non-null values
	AggAllTrue   // Boolean AND over non-null values
	AggVar
	AggStd
	AggProduct
	AggArgMin // index of the minimum WITHIN the group
	AggArgMax
	AggMedian
	AggQuantile
	AggImplode // every value of the group, as a List
	AggTopK    // the k largest values of the group, largest first, as a List
	AggBottomK // the k smallest values of the group, smallest first, as a List

	// Two inputs (step 175), carried as a two-field Struct, PairOf's: the value and
	// what orders it, or the two variables.
	AggCorr  // Pearson's correlation of the pairs where both are non-null
	AggCov   // their covariance, with Params.DDof
	AggMinBy // the value at the row whose second field is the least
	AggMaxBy // and the greatest

)

func (AggOp) IsBy added in v0.6.0 

func (o AggOp) IsBy() bool

IsBy reports whether the aggregate is MinBy or MaxBy. They are order-dependent as ArgMin and ArgMax are: a tie goes to the earliest row, which a worker holding rows dealt round-robin cannot know.

func (AggOp) IsCounting 

func (o AggOp) IsCounting() bool

IsCounting reports whether the aggregate returns a count, and is therefore never null and independent of the input type.

ArgMin and ArgMax return an index rather than a count and are deliberately NOT here: a group with no non-null value has no minimum, so there is no position to report and the answer is null.

func (AggOp) IsOrderDependent 

func (o AggOp) IsOrderDependent() bool

IsOrderDependent reports whether the aggregate's ANSWER depends on the order its input arrived in.

Every other aggregate is commutative: a sum, a count, a minimum and a quantile are the same however the rows are shuffled, which is what lets a group-by run on several workers and fold the partials together. These five are not. 

First, Last     name a position outright
ArgMin, ArgMax  report an INDEX, and kernel.argExtremumAcc.Merge shifts the
                other accumulator's positions by this one's row count — which
                is only meaningful if `other` saw a strictly later contiguous
                portion
Implode         a list's element ORDER is its data, so a shuffled input is a
                different answer rather than the same one computed differently

The parallel aggregation driver dispatches round-robin, so no worker holds a contiguous portion and none of those four can be honoured. It declines the whole aggregate rather than the individual expression, because one query's Agg list is one hash table.

Not the same question as IsCounting: Median and Quantile buffer every value and are expensive, but they sort in Finish and are perfectly order-insensitive.

func (AggOp) IsPaired added in v0.6.0 

func (o AggOp) IsPaired() bool

IsPaired reports whether the aggregate reads two inputs, as PairOf's struct.

func (AggOp) String 

func (o AggOp) String() string

type AggParams 

type AggParams struct {
	DDof   uint8
	Q      float64
	Interp Interpolation
	K      int // TopK and BottomK
}

AggParams carries an aggregate's configuration — ddof for Var and Std, the quantile and its interpolation for Quantile. The zero value is correct for every aggregate that takes no parameters.

Why not Args []Node, the way Call does it

Call holds its arguments as ordinary child nodes, and its doc explains that a column-valued argument "is representable in this IR and simply not implemented yet, which is the right way round for a constraint that may lift later". That is true of a regex pattern.

It is not true of these. A ddof or a quantile is CONFIGURATION, not an operand: there is no meaning to a per-row ddof, and there never will be. Making them children would put nodes into Children() that RootNames, expansion and projection pushdown then walk for nothing.

type Alias 

type Alias struct {
	Child Node
	Name  string
}

Alias renames the output column.

func (*Alias) Children 

func (a *Alias) Children() []Node

func (*Alias) Field 

func (a *Alias) Field(in *dtype.Schema) (dtype.Field, error)

func (*Alias) String 

func (a *Alias) String() string

type AllMatcher 

type AllMatcher struct{}

AllMatcher selects every column.

func (*AllMatcher) Match 

func (m *AllMatcher) Match(in *dtype.Schema) ([]string, error)

func (*AllMatcher) String 

func (m *AllMatcher) String() string

type Binary 

type Binary struct {
	Op   BinaryOp
	L, R Node
}

Binary applies a two-operand operation.

The resolved cast targets are NOT stored here: Binary is immutable and shared, and caching a resolution would tie a node to one input schema. The evaluator calls ResolveBinary itself, which is cheap and keeps exactly one copy of the promotion rules in the codebase.

func (*Binary) Children 

func (b *Binary) Children() []Node

func (*Binary) Field 

func (b *Binary) Field(in *dtype.Schema) (dtype.Field, error)

func (*Binary) String 

func (b *Binary) String() string

type BinaryOp 

type BinaryOp uint8

BinaryOp is a two-operand operation. 

const (
	// Arithmetic.
	OpAdd BinaryOp = iota
	OpSub
	OpMul
	OpDiv      // true division: always produces a float
	OpFloorDiv // floored division: −7 // 2 is −4
	OpMod

	// Ordering and equality. Three-valued: if either operand is null the result
	// is null, NOT false.
	OpEq
	OpNe
	OpLt
	OpLe
	OpGt
	OpGe

	// Equality where nulls are DATA rather than absence: null == null is true and
	// the result is never null. Polars calls these eq_missing / ne_missing. They
	// need a separate kernel family, not a flag on the ordinary comparison.
	OpEqMissing
	OpNeMissing

	// Kleene three-valued logic. Note false AND null == false and true OR null ==
	// true — the output validity is NOT simply `va AND vb`.
	OpAnd
	OpOr
	OpXor

	// Exponentiation. APPENDED rather than placed beside the other arithmetic,
	// which reads oddly and is deliberate: both dispatchers — ResolveBinary and
	// kernel's arithmetic — route anything the classifiers below do not claim to
	// the arithmetic path through a `default:` arm, so position carries no meaning
	// for it, and renumbering the enum to make it look tidy would move every
	// constant for no gain.
	OpPow
)

func (BinaryOp) IsComparison 

func (o BinaryOp) IsComparison() bool

IsComparison reports whether the op yields Bool via three-valued comparison.

func (BinaryOp) IsLogical 

func (o BinaryOp) IsLogical() bool

IsLogical reports whether the op is Kleene boolean.

func (BinaryOp) IsMissingComparison 

func (o BinaryOp) IsMissingComparison() bool

IsMissingComparison reports whether the op treats nulls as data.

func (BinaryOp) IsOrdering 

func (o BinaryOp) IsOrdering() bool

IsOrdering reports whether the op uses < / > semantics, as opposed to equality. Ordering comparisons are undefined for types that have no order.

func (BinaryOp) MayProduceNull added in v0.3.0 

func (o BinaryOp) MayProduceNull(out dtype.DataType) bool

MayProduceNull reports whether the kernel for this op, at this output type, can manufacture a null its operands did not contain.

Why this exists

internal/kernel splits kernels into classes by how they relate to nulls, and a PARTIAL kernel "may itself produce nulls (integer division by zero, a narrowing cast, sqrt of a negative)". Binary.Field propagated only — `lf.Nullable || rf.Nullable` — which is conservative about PROPAGATION and says nothing about MANUFACTURE, the axis the kernel layer actually splits on. So `a % b` over two non-nullable integer columns was declared non-nullable and held a null wherever b was zero.

It takes the OUTPUT type, and that is not a detail

kernel.arithmetic dispatches on `Binding.Out.Physical()`, so the output type is what decides which kernel runs, and the operands' logical types can be misleading in both directions: 

Int64 % Int64        Out Int64      -> arithNum   -> PARTIAL
Float64 % Float64    Out Float64    -> arithFloat -> total (math.Mod, NaN)
Int64 / Int64        Out Float64    -> arithFloat -> total (true division)
Duration // Int64    Out Duration   -> arithNum   -> PARTIAL, though the left
                                       operand is not an integer type at all

A predicate written against the operands would get the last two backwards.

Explicit membership, not a range

OpDiv sits immediately above OpFloorDiv, so `o >= OpFloorDiv && o <= OpMod` would be one constant away from wrong — and this file already carries the post-mortem of that exact mistake under "There is deliberately no IsArithmetic".

func (BinaryOp) String 

func (o BinaryOp) String() string

type Binding 

type Binding struct {
	CastL dtype.DataType // cast the left operand to this before the kernel
	CastR dtype.DataType
	Out   dtype.DataType // the kernel's output type
}

Binding is the resolved form of a binary operation: the type each operand must be cast to, and the type the result will have.

The evaluator MUST use this rather than re-deriving types from the operands. Two independent copies of the promotion rules would drift, and the drift would show up as a mismatch between the schema CollectSchema promised and the column Eval actually produced — which is precisely the invariant the evaluator contract test checks.

func ResolveBinary 

func ResolveBinary(op BinaryOp, l, r dtype.DataType) (Binding, error)

ResolveBinary computes the operand and result types of op applied to l and r.

func (Binding) NeedsCastL 

func (b Binding) NeedsCastL(l dtype.DataType) bool

NeedsCastL reports whether the left operand requires a cast.

func (Binding) NeedsCastR 

func (b Binding) NeedsCastR(r dtype.DataType) bool

type Call 

type Call struct {
	Fn   CallFn
	Args []Node
}

Call applies a namespaced function.

Args[0] is the RECEIVER — the column the method was called on. Args[1:] are the function's parameters and must resolve to literals; a column-valued pattern is representable in this IR and simply not implemented yet, which is the right way round for a constraint that may lift later.

Args are ordinary children, so RootNames, expansion and projection pushdown all see through a Call with no special case. That is the property the design docs predicted for window nodes and it holds here for the same reason.

func (*Call) Children 

func (c *Call) Children() []Node

func (*Call) Field 

func (c *Call) Field(in *dtype.Schema) (dtype.Field, error)

func (*Call) String 

func (c *Call) String() string

type CallFn 

type CallFn uint16

CallFn names a namespaced function — the `.str` and `.dt` families.

Why these are not UnaryOps

`Contains(pattern, literal)` and `Truncate(every)` carry ARGUMENTS, and Unary{Op, Child} has nowhere to put them. Three shapes were possible: one op per (function, argument) pair, which does not terminate; arguments smuggled into the op enum, which is the same thing wearing a hat; or one node that holds a function and its operands.

This is the third. It is also the shape every remaining namespace will need — `.list`, `.struct`, `.arr` — so paying for it once here is cheaper than three more bespoke node types later.

Appending only

Like UnaryOp, the classifiers below are range comparisons over declaration order. Insert a constant mid-enum and a neighbouring function silently changes family. Append, and add a name. 

const (
	// --- string ---
	FnStrContains CallFn = iota
	FnStrStartsWith
	FnStrEndsWith
	FnStrFind
	FnStrCountMatches
	FnStrExtract
	FnStrReplace
	FnStrReplaceAll
	FnStrToLower
	FnStrToUpper
	FnStrLenBytes
	FnStrLenChars
	FnStrSlice
	FnStrStripChars
	FnStrStripPrefix
	FnStrStripSuffix
	FnStrReverse
	FnStrPadStart
	FnStrPadEnd
	FnStrZFill
	FnStrStripCharsStart
	FnStrStripCharsEnd
	FnStrEscapeRegex
	FnStrSplit
	FnStrSplitN
	FnStrExtractAll
	// The three parse a string with a strftime-style format (step 113): args are
	// the format, strict, and for a Datetime the unit and zone, for a Time the
	// unit. One function per target, because a type is not a literal.
	FnStrToDateFmt
	FnStrToDatetimeFmt
	FnStrToTimeFmt

	// --- temporal ---
	FnDtYear CallFn = iota + 100
	FnDtMonth
	FnDtDay
	FnDtHour
	FnDtMinute
	FnDtSecond
	FnDtMillisecond
	FnDtMicrosecond
	FnDtNanosecond
	FnDtWeekday
	FnDtOrdinalDay
	FnDtQuarter
	FnDtWeek
	FnDtEpoch
	FnDtTruncate
	FnDtTotalDays
	FnDtTotalHours
	FnDtTotalMinutes
	FnDtTotalSeconds
	FnDtStrftime // formats with a strftime-style format; the one arg is the format
	// The calendar (step 145). offset_by's args are an interval's months, days and
	// nanos; round's are truncate's; convert_time_zone's one arg is the zone.
	FnDtOffsetBy
	FnDtRound
	FnDtMonthStart
	FnDtMonthEnd
	FnDtIsLeapYear
	FnDtConvertTimeZone
	// Step 174: replace_time_zone's args are the zone, "" for none, then the
	// ambiguous and non_existent policies, in Polars' words (ZonePolicies).
	FnDtReplaceTimeZone

	// --- type-agnostic ---
	//
	// A third family, and the first that is not keyed to a receiver type. The two
	// above dispatch on what the receiver IS — a String, an instant — but IsIn
	// applies to anything hashable, so it needs its own classifier rather than
	// widening either of theirs.
	FnIsIn CallFn = iota + 200

	// --- maths ---
	//
	// A fourth family, for the maths operations that carry a PARAMETER. The
	// parameterless ones are UnaryOps; Round needs a decimal count, and a Call is
	// where a parameter belongs.
	//
	// Not a field on expr.Unary, which was the obvious alternative: three places
	// rebuild a Unary by hand — physical/agg.go's extractAggs and both predicate
	// pushdown walkers — as `&Unary{Op: t.Op, Child: ...}`, so a new field would be
	// silently DROPPED the moment a predicate moved, and Round(2) would quietly
	// become Round(0). That is step 7's Agg.Params bug, three sites over. Call.String()
	// renders its arguments structurally, so the same class is impossible here.
	FnMathRound CallFn = iota + 300
	// FnMathAsFloat takes a number, or a Duration's ticks, to a float without
	// changing its value: an integer or a Decimal to the nearest Float64, a float as
	// itself. It is how PctChange subtracts in float, and is not public.
	FnMathAsFloat
	// FnMathDiffWiden takes an unsigned integer to the signed type that holds the
	// difference of any two of its values, and leaves every other type as it is. It
	// is how Diff subtracts without wrapping, and is not public.
	FnMathDiffWiden
	// The bit counts of an integer (step 146), Polars' bitwise_count_ones and its
	// five siblings, each a Uint32.
	FnMathCountOnes
	FnMathCountZeros
	FnMathLeadingOnes
	FnMathLeadingZeros
	FnMathTrailingOnes
	FnMathTrailingZeros

	// --- list ---
	//
	// A fifth family, keyed to a List receiver. These all reduce a list to a
	// SCALAR: nothing here builds a list, which is the seam the step was cut on —
	// sort, unique, reverse, slice and the set operations construct a new List
	// column and are a different problem.
	FnListLen CallFn = iota + 400
	FnListGet
	FnListContains
	FnListMin
	FnListMax
	FnListSum
	FnListMean
	FnListJoin // the elements of a List of String, joined; the one arg is the separator

	// list -> LIST. These reshape the list and hand back a list, which is the
	// distinction that split the namespace across two steps: everything above
	// answers a question ABOUT a list, everything here returns one.
	FnListReverse
	FnListHead
	FnListTail
	FnListSlice
	FnListSort
	FnListUnique
	FnListDropNulls
)
const (
	FnConcatStr CallFn = iota + 600
	FnStructOf
	FnCut
	FnCutLeftClosed
	FnStructWithFields
)

The horizontal family (step 146): every argument is an OPERAND, a column read row by row, where each family above takes a receiver and literal parameters.

concat_str joins its operands' text, and a separator is an operand like any other, a literal between each two; struct builds a Struct whose fields are its operands, named as they are. Neither has a parameter, so Call's rule — Args[1:] literal — does not apply to them, and they are typed from every operand's type (ResolveHorizontal) rather than from a receiver's.

cut (step 147) puts its first operand in a bin: the operands after it are the breaks, numbers, then none or one label more than breaks, strings. Its breaks are operands because QCut's are quantiles of the column, known only when it runs. cut_left_closed is the same with each bin closed on the left. 

const (
	FnStructField CallFn = iota + 500
)

The `.struct` family.

func (CallFn) IsGeneral 

func (f CallFn) IsGeneral() bool

func (CallFn) IsHorizontal added in v0.5.0 

func (f CallFn) IsHorizontal() bool

func (CallFn) IsList added in v0.3.0 

func (f CallFn) IsList() bool

func (CallFn) IsMath 

func (f CallFn) IsMath() bool

func (CallFn) IsString 

func (f CallFn) IsString() bool

IsString, IsTemporal and IsGeneral classify a function by family.

These are range comparisons over declaration order, which is why the enum is append-only within each block and the blocks are spaced 100 apart.

func (CallFn) IsStruct added in v0.3.0 

func (f CallFn) IsStruct() bool

func (CallFn) IsTemporal 

func (f CallFn) IsTemporal() bool

func (CallFn) String 

func (f CallFn) String() string

type Cast 

type Cast struct {
	Child  Node
	To     dtype.DataType
	Strict bool
}

Cast converts to another type.

Strict casts fail the query on the first unrepresentable value. Non-strict casts turn it into a null — which is why a non-strict cast is ALWAYS nullable regardless of its input, even when casting a non-nullable column.

func (*Cast) Children 

func (c *Cast) Children() []Node

func (*Cast) Field 

func (c *Cast) Field(in *dtype.Schema) (dtype.Field, error)

func (*Cast) String 

func (c *Cast) String() string

type Closed 

type Closed uint8

Closed says which endpoints of an interval belong to it.

It lives here, at the level of the expression IR, for the same reason Interpolation does: the plan node, the physical operator and the root package all need it and none of them may import another's caller.

Why four values rather than a pair of bools

Two bools would admit the same four states and read worse at every call site — `Closed: ClosedBoth` against `LeftClosed: true, RightClosed: true`.

And why the zero value is none of them

The three callers want three different defaults, and every one of them is the obviously right answer for its own question: 

IsBetween         both  — "between lo and hi" includes the endpoints
GroupByDynamic    left  — the only convention under which a grid TILES
Rolling           right — a row's window ends at its own timestamp, so closing
                          the left end would drop the row from its own window

A zero value that meant one of them would silently give the wrong default to the other two. So the zero means "ask the operator", each caller resolves it with Or, and no call site has to know which default it inherited. 

const (
	// ClosedDefault defers to whatever the operation's own convention is. It is the
	// zero value precisely so that leaving the field alone cannot mean the wrong
	// thing somewhere else.
	ClosedDefault Closed = iota

	// ClosedLeft is [start, end) — the only value under which a grid of windows
	// partitions its input exactly.
	ClosedLeft

	// ClosedRight is (start, end].
	ClosedRight

	// ClosedBoth is [start, end]. A row landing exactly on a shared boundary is in
	// BOTH neighbouring windows, so the windows no longer partition and a count over
	// them exceeds the row count. That is the documented meaning, not an accident.
	ClosedBoth

	// ClosedNone is (start, end). The mirror: a row on a shared boundary is in
	// neither window and is dropped from the output entirely.
	ClosedNone
)

func (Closed) LowerOpen 

func (c Closed) LowerOpen() bool

LowerOpen and UpperClosed are the two questions every range test asks, phrased so the caller never re-derives the four cases. Both treat ClosedDefault as ClosedLeft, which is only reachable if a caller forgot to call Or. 

contains(x)  ==  (LowerOpen() ? x > lo : x >= lo) && (UpperClosed() ? x <= hi : x < hi)

func (Closed) Or 

func (c Closed) Or(def Closed) Closed

Or resolves ClosedDefault to the caller's own convention and leaves anything else alone. Every consumer calls this exactly once, at the edge, so the rest of the code never has to wonder whether it is holding a resolved value.

func (Closed) String 

func (c Closed) String() string

func (Closed) UpperClosed 

func (c Closed) UpperClosed() bool

func (Closed) Valid added in v0.3.0 

func (c Closed) Valid() bool

Valid reports whether c is one of the declared values. Closed is a public integer type, so Closed(99) compiles; it used to answer as ClosedLeft, because LowerOpen and UpperClosed both say no to it. Every consumer refuses it instead.

type Col 

type Col struct{ Name string }

Col reads a column by name.

func (*Col) Children 

func (c *Col) Children() []Node

func (*Col) Field 

func (c *Col) Field(in *dtype.Schema) (dtype.Field, error)

func (*Col) String 

func (c *Col) String() string

type Cond 

type Cond struct {
	Pred Node
	Then Node
	Else Node

	// Sugar names the method a conditional was desugared from — fill_null,
	// fill_nan — so its refusal says what the caller wrote rather than what it
	// became (audit.md S16: "is_not_nan() requires a floating-point operand", to
	// someone who had called FillNan). "" for a When the caller built. A rewrite
	// that rebuilds the node may drop it: types are resolved before any rule runs,
	// and that is where these refusals arise.
	Sugar string
}

Cond is a conditional: the value of Then where Pred is true, of Else otherwise.

Why this is a node and IsBetween is not

IsBetween is two lines of sugar — `e.Ge(lo).And(e.Le(hi))` — and earns Field, evaluation, expansion, pushdown and Parquet row-group pruning for free by being built out of nodes that already exist. That is the cheapest way to add anything, and it was the first thing tried here.

It does not work for a conditional, for reasons that are properties of this codebase rather than of conditionals in general:

  • There was no masked-select kernel. Sugar has to desugar INTO something, and nothing in kernel took (mask, a, b). kernel.Select was written for this node.
  • The arithmetic dodge does not type-check. `cond*a + !cond*b` needs Bool to be numeric, and dtype.Promote refuses on purpose: "true + true is rejected".
  • There is no CSE pass, so any desugaring that names an operand twice evaluates it twice.

Three children, not parallel slices

A chain — when/then/when/then/otherwise — is built by the root package into a right-nested tree of these, rather than represented as []Pred and []Then. The reason is type unification: dtype.Promote is strictly binary, there is no n-ary helper, and a left fold is NOT safe to invent because Promote's mixed-sign rule (the smallest signed type holding both) makes the result depend on association order for operand sets like (Uint8, Int8, Uint32). Nesting makes the association explicit and identical to the order the user wrote.

All three are ordinary children, so RootNames, Walk, expansion and projection pushdown see through a Cond with no special case.

func (*Cond) Children 

func (c *Cond) Children() []Node

func (*Cond) Field 

func (c *Cond) Field(in *dtype.Schema) (dtype.Field, error)

func (*Cond) String 

func (c *Cond) String() string

type DTypeMatcher 

type DTypeMatcher struct{ Types []dtype.DataType }

DTypeMatcher selects columns by type. The number of columns it expands to is determined at plan time from the schema, so the same expression adapts as the schema changes.

func (*DTypeMatcher) Match 

func (m *DTypeMatcher) Match(in *dtype.Schema) ([]string, error)

func (*DTypeMatcher) String 

func (m *DTypeMatcher) String() string

type Err 

type Err struct{ E error }

Err carries a construction-time error to the point where an error can be returned. Expression builders cannot return errors without destroying method chaining, so `ColRegex("(")` produces an *Err instead of panicking.

func (*Err) Children 

func (e *Err) Children() []Node

func (*Err) Field 

func (e *Err) Field(*dtype.Schema) (dtype.Field, error)

func (*Err) String 

func (e *Err) String() string

type ExcludeMatcher 

type ExcludeMatcher struct {
	Inner  Matcher
	Names  []string
	Re     *regexp.Regexp
	ReSrc  string
	DTypes []dtype.DataType
}

ExcludeMatcher subtracts columns from an inner matcher's result.

func (*ExcludeMatcher) Match 

func (m *ExcludeMatcher) Match(in *dtype.Schema) ([]string, error)

func (*ExcludeMatcher) String 

func (m *ExcludeMatcher) String() string

type FieldMatcher 

type FieldMatcher struct {
	Label string
	Pred  func(dtype.Field) bool
}

FieldMatcher selects columns whose field satisfies a predicate.

It is groundwork for the selector package's dtype families (numeric, temporal, string, ...), which want a predicate rather than an enumeration of concrete types. NOTHING CONSTRUCTS ONE TODAY — that package does not exist yet, and this doc used to claim in the present tense that it did.

func (*FieldMatcher) Match 

func (m *FieldMatcher) Match(in *dtype.Schema) ([]string, error)

func (*FieldMatcher) String 

func (m *FieldMatcher) String() string

type Interpolation 

type Interpolation uint8

Interpolation selects how Quantile resolves a rank that falls between two values.

It lives here, at the level of the expression IR, rather than in the root package, because both Agg's type resolution and the kernel need it and neither may import the other's caller. The root package re-exports it as an alias, the same way it does for JoinKind. 

const (
	// InterpLinear interpolates proportionally between the two neighbours. The
	// default, and the only option whose result need not be an input value.
	InterpLinear Interpolation = iota

	// InterpLower and InterpHigher take the neighbour below or above.
	InterpLower
	InterpHigher

	// InterpNearest takes the closer neighbour, rounding half away from the lower
	// one.
	InterpNearest

	// InterpMidpoint averages the two neighbours regardless of where the rank fell
	// between them.
	InterpMidpoint
)

func (Interpolation) String 

func (i Interpolation) String() string

func (Interpolation) Valid added in v0.3.0 

func (i Interpolation) Valid() bool

Valid reports whether i is one of the declared interpolations. Interpolation is a public integer type, so Interpolation(99) compiles; Agg.Field refuses it. It was silently linear before step 72.

type Lit 

type Lit struct {
	Value any
	DT    dtype.DataType

	// Weak marks a literal whose type is a DEFAULT rather than a choice.
	//
	// ursus.Lit(0) infers Go int and fixes DT = Int64 at BUILD time. For arithmetic
	// that is right — Promote(Int32, Int64) is Int64 because addition can overflow.
	// For a null fill it is wrong, and worse, INVISIBLE: Col("i32").FillNull(FillZero)
	// contains no literal in the source at all, yet the same promotion would widen
	// the column to Int64 with nothing in the query to explain it.
	//
	// A weak literal asks the CONTEXT to adopt the other operand's type when the
	// value fits it exactly. Only ResolveCond consults it; Field still reports DT,
	// because a weak literal standing alone has no other operand to defer to.
	//
	// The zero value is STRONG, which is what keeps this change small: every
	// existing construction site keeps today's meaning untouched, and arithmetic
	// stays weak-free by construction because nothing on that path sets it.
	Weak bool
}

Lit is a scalar constant, broadcast to the length of the context.

A typed null is Value == nil with DT set; an untyped null is Value == nil with DT == Null, which Promote lets adopt the other operand's type.

func (*Lit) Children 

func (l *Lit) Children() []Node

func (*Lit) Field 

func (l *Lit) Field(*dtype.Schema) (dtype.Field, error)

func (*Lit) String 

func (l *Lit) String() string

String renders the literal.

A weak literal MUST NOT render like a strong one

Three maps key on a rendered String() — window temporaries, aggregate temporaries and partition-key sharing — so two expressions that render alike become ONE computation and both names get the first one's answer.

Weak versus strong is a live instance of that: Coalesce(col_i32, Lit(0)) resolves to Int64 and col_i32.FillNull(FillZero) resolves to Int32, and without the marker below they render identically. Put both under .Max() in one Agg and the aggregate dedup collapses them, after which the batch's column type and the plan's promised schema disagree.

The TYPE is in there for a second collision. Weak literals normally converge on the column's type and so agree — but when the value does not fit they fall back to ordinary promotion, and a weak int64 5000 and a weak float64 5000 fall back to Int64 and Float64 respectively while printing the same digits.

type MappingStrategy 

type MappingStrategy uint8

MappingStrategy decides how a window's per-partition result is placed back into the frame. 

const (
	// MapGroupsToRows returns each row's own partition value, in the original row
	// order. The default, and the only one that composes with other columns.
	MapGroupsToRows MappingStrategy = iota

	// MapExplode leaves the output in partition-sorted order instead of permuting
	// it back. Cheaper — it skips the inverse permutation — and it REORDERS the
	// frame, so it is only safe when every other column is windowed the same way.
	MapExplode

	// MapJoin aggregates each partition into a List repeated across its rows.
	// Refused, and from step 46 the refusal names the thing that does it:
	// Col("v").Implode().Over(k) IS this, under the default mapping.
	MapJoin
)

func (MappingStrategy) String 

func (m MappingStrategy) String() string

type Match 

type Match struct{ M Matcher }

Match is a multi-column selection: Col("a","b"), ColRegex, ColDType, All, Exclude. It is a placeholder that Expand replaces with one concrete tree per matched column.

It exists as a node rather than as a separate expression kind so that `Col(pl.Float64).Mul(1.1).Alias(...)` is an ordinary tree that expansion rewrites, rather than a special case threaded through every builder.

func (*Match) Children 

func (m *Match) Children() []Node

func (*Match) Field 

func (m *Match) Field(*dtype.Schema) (dtype.Field, error)

func (*Match) String 

func (m *Match) String() string

type Matcher 

type Matcher interface {
	// Match returns the selected column names, in schema order.
	Match(in *dtype.Schema) ([]string, error)
	// String renders the matcher for Explain and error messages.
	String() string
}

Matcher selects zero or more columns from a schema.

Matchers are what make one written expression apply to many columns: 

Col("weight", "height").Mean()      → two independent means
ColDType(Float64).Mul(1.1)          → every float column, count known only at plan time
All().Exclude("id")                 → everything but id

Matching is by NAME and resolved against a schema, never against data.

type NameMatcher 

type NameMatcher struct{ Names []string }

NameMatcher selects an explicit list of columns, in the order written.

Unlike the other matchers, a name that is not present is an ERROR rather than an empty match: the user named a specific column, so its absence is a typo.

func (*NameMatcher) Match 

func (m *NameMatcher) Match(in *dtype.Schema) ([]string, error)

func (*NameMatcher) String 

func (m *NameMatcher) String() string

type Node 

type Node interface {
	// Field resolves this node's single output field against an input schema.
	// It never reads data.
	//
	// Defined only for RESOLVED trees: a tree containing a *Match must be run
	// through Expand first, and returns ErrUnexpanded otherwise.
	Field(in *dtype.Schema) (dtype.Field, error)

	// Children returns the direct sub-expressions, in evaluation order.
	//
	// A nested SCOPE body is deliberately not a child. When List().Eval() arrives,
	// its body is evaluated against a different schema (the list element type),
	// so a generic walker must not rewrite across it or it would resolve
	// Element() against the outer frame.
	Children() []Node

	// String renders the node canonically. Load-bearing: it appears in Explain
	// output, in golden test files, and in error messages.
	String() string
	// contains filtered or unexported methods
}

Node is one node of the expression IR.

A Node is IMMUTABLE. Rewrites allocate new nodes; nothing ever mutates a node or a slice reachable from one. That is what makes it safe for two LazyFrames to share a subtree, which in turn is what makes `base.Filter(x)` and `base.Filter(y)` independent.

func Expand 

func Expand(n Node, in *dtype.Schema) ([]Node, error)

Expand rewrites one written expression into the concrete expressions it denotes. 

Col("a")                          → [col("a")]                       (unchanged)
Col("a","b").Mul(2)               → [(col("a") * 2), (col("b") * 2)]
ColDType(Float64).Round(2)        → one node per Float64 column
All().Exclude("id")               → one node per remaining column

Where this runs, and why there

Expansion is a PLAN-TIME, SCHEMA-DRIVEN rewrite. It runs in the resolve phase: after a node's input schema is known, before type checking, and before any optimizer rule sees the plan.

It cannot run earlier — `ColDType(Float64)` has no meaning without a schema. It must not run later — projection pushdown asks each expression which columns it reads, and an unexpanded matcher would answer "none", so the optimizer would happily prune away the very columns the expression is about to select. Running before type checking also means the type checker only ever sees single-output trees, which is what lets Field return one Field instead of a slice.

One matcher per expression

A tree may contain at most one DISTINCT matcher. The same matcher repeated is fine and expands in lockstep — `All().Sub(All().Mean())` gives each column minus its own mean. Two different matchers are rejected, because the intended pairing is genuinely ambiguous: is it a zip, or a cross product?

func ExpandAll 

func ExpandAll(ns []Node, in *dtype.Schema) ([]Node, error)

ExpandAll expands a slice of expressions, concatenating the results in order.

func PairOf added in v0.6.0 

func PairOf(a, b Node) Node

PairOf is the child of a paired aggregate: a two-field Struct of a and b, each aliased to its position. The aggregate is still named after a's column, since naming reads an Alias only at the root (OutputName).

func Rebuild 

func Rebuild(n Node, kids []Node) Node

Rebuild returns a copy of n with its children replaced, in the order Children returns them.

Why this exists

Rewriting an expression tree means copying every node with new children, and before this there were two hand-written copies of that walk — substitute, and the physical planner's extractAggs — with a third about to be added for windows. Each copy enumerates the node types and carries their non-child fields across by hand, which is exactly the shape of the bug step 7 hit twice: an Agg rebuilt without its Params silently became a different aggregate, and a Window rebuilt without its Mapping would silently become a different window.

Copying the struct and replacing only the child fields makes that class impossible: a new field on any node is carried by default, and the only way to get it wrong is to add a child field without updating Children, which the length check catches.

It PANICS on an unknown node type, for the same reason substitute did: Node is sealed, so reaching the default means a node type was added without teaching the rewriter about it, and silently dropping its children is worse than a crash.

type OrderKey 

type OrderKey struct {
	Expr       Node
	Descending bool

	// NullsLast places nulls at the END OF THE OUTPUT, independent of direction.
	//
	// The zero value is therefore NULLS FIRST, which is ursus's default and matches
	// Polars (nulls_last=False). Per-key `.NullsLast()` overrides it.
	//
	// Tying null placement to direction — as SQL does, where nulls sort as the
	// largest value and so land last ascending and first descending — was
	// considered and rejected: it means `.Desc()` silently relocates the nulls,
	// which surprises people every time.
	NullsLast bool
}

OrderKey is one component of an ordering: an expression and how to sort by it.

It lives here rather than in the plan package because a window carries its own ordering and internal/expr may not import internal/plan. plan.SortKey is an alias for this type, so there is one definition rather than two that must agree.

func (OrderKey) String 

func (k OrderKey) String() string

type RankMethod 

type RankMethod uint8

RankMethod decides how Rank breaks ties. 

const (
	// RankOrdinal gives every row a distinct rank, ties broken by input order.
	RankOrdinal RankMethod = iota
	// RankDense gives tied rows the same rank and does not skip the next value.
	RankDense
	// RankMin and RankMax give tied rows the lowest or highest of their positions.
	RankMin
	RankMax
	// RankAverage gives tied rows the mean of their positions, so it returns a
	// float where every other method returns an integer.
	RankAverage
)

func (RankMethod) String 

func (m RankMethod) String() string

func (RankMethod) Valid added in v0.3.0 

func (m RankMethod) Valid() bool

Valid reports whether m is one of the declared methods. RankMethod is a public integer type, so RankMethod(99) compiles; ResolveWinFn refuses it, and the kernel never sees one.

type RegexMatcher 

type RegexMatcher struct {
	Pattern string
	Re      *regexp.Regexp
}

RegexMatcher selects columns whose name matches a pattern.

ursus requires the pattern to be passed to ColRegex explicitly rather than inferring "this string looks like a regex" from leading ^ and trailing $ the way Polars does. That inference makes a column literally named "^total$" unselectable and makes every plain Col() call pay a scan for anchors.

func NewRegexMatcher 

func NewRegexMatcher(pattern string) (*RegexMatcher, error)

NewRegexMatcher compiles pattern, returning an error the caller parks in an *Err node.

func (*RegexMatcher) Match 

func (m *RegexMatcher) Match(in *dtype.Schema) ([]string, error)

Match returns every matching column. Matching nothing is not an error — it is the natural answer for a pattern over a schema that happens not to contain it, and erroring would make schema-driven code fragile.

func (*RegexMatcher) String 

func (m *RegexMatcher) String() string

type Rename 

type Rename struct {
	Child Node
	Fn    func(string) string
	Label string // rendering only, e.g. `name.prefix("avg_")`
}

Rename applies a function to the output name.

This is the expansion-safe counterpart to Alias: where Alias imposes one fixed name (and so cannot be applied to a multi-column expression), Rename derives a name per output, which is what Name().Prefix / .Suffix / .Map need.

func (*Rename) Children 

func (r *Rename) Children() []Node

func (*Rename) Field 

func (r *Rename) Field(in *dtype.Schema) (dtype.Field, error)

func (*Rename) String 

func (r *Rename) String() string

type UDF added in v0.3.0 

type UDF struct {
	Child Node

	// Out is the output type as DECLARED by the caller. It is not derived from
	// anything and cannot be — the function is opaque. The evaluator refuses a
	// column whose type disagrees with it.
	Out dtype.DataType

	// Name distinguishes this UDF from every other one in the query. Required.
	Name string

	// Kind is "map_elements" or "map_batches", for rendering and errors only.
	Kind string

	Impl UDFImpl
	// contains filtered or unexported fields
}

UDF applies a user-supplied Go function to one column.

It is the escape hatch, and it is OPAQUE: nothing in the planner can inspect it, fold it, prove it pure, or predict what it returns. Everything the engine knows about it, the user declared — which is why the evaluator CHECKS those declarations rather than trusting them. See physical.evalUDF.

Name is required, and that is a correctness argument rather than a style one

Node.String is load-bearing: "it appears in Explain output, in golden test files, and in error messages". Worse, until step 72 FOUR maps deduplicated expressions on their rendering, over three code sites, because extractAggs is instantiated twice — plan.resolveWindow's temporaries, physical.extractAggs' byKey for a hash aggregate AND for a temporal group, and physical's partition sharing — and names_test.go states the consequence: "two expressions that RENDER THE SAME become one computation, and both names get the first one's answer".

A Go closure has no printable identity. Without a name, two DIFFERENT user functions over the same column render alike, collapse into one computation, and both results come from whichever was seen first. That is the shape of two bugs this project has already shipped and fixed: an unparameterised Quantile that collapsed p50 and p99, and weak-versus-strong literals.

A monotonic counter would deduplicate correctly too, and would make every golden plan file depend on allocation order. So: a name, supplied by the caller.

That objection is about a counter that RENDERS. The id field below is a monotonic counter which never renders — String is unchanged and the golden plans with it. It is what lets plan.CheckUDFNames refuse a collision instead of merging one, and since step 72 it is part of what the four maps key on, through Identity, so two names that look alike no longer merge even where the check has not run.

func NewUDF added in v0.3.0 

func NewUDF(child Node, out dtype.DataType, name, kind string, impl UDFImpl) *UDF

NewUDF builds a UDF node with a fresh identity, and is the only way to mint one.

The fields stay exported because plan and physical read them and copy the struct wholesale. The id does not, so it cannot be forged onto a different Impl.

func (*UDF) Children added in v0.3.0 

func (u *UDF) Children() []Node

func (*UDF) Field added in v0.3.0 

func (u *UDF) Field(in *dtype.Schema) (dtype.Field, error)

Field takes the child's NAME and the caller's declared type.

Naming follows the receiver, like Call.Field: leftmost-column-wins means Col("c").MapElements(...) is named "c", and an Alias above overrides it.

Resolving the child is not a formality — it is what makes a UDF over an unknown column a schema error at plan time rather than at the first batch.

func (*UDF) ID added in v0.3.0 

func (u *UDF) ID() uint64

ID is the node's identity: distinct for every UDF the constructor ever built, and EQUAL for every copy of one. Zero means the node did not come from NewUDF.

func (*UDF) String added in v0.3.0 

func (u *UDF) String() string

String renders the name, because the closure cannot be rendered and the name is the only thing that makes two UDFs distinguishable. Quoted so that a name containing punctuation cannot forge a different expression's rendering.

type UDFImpl added in v0.3.0 

type UDFImpl interface{ UDFKernel() }

UDFImpl is a user-supplied kernel, erased.

Why an opaque interface rather than a func field

The obvious field is `func(*data.Column) (*data.Column, error)`, and it does not compile. internal/gen/levels puts internal/data and internal/expr at the SAME level (20), and the rule is strictly-lower, so this package cannot import internal/data. Nor can that be fixed by renumbering: internal/plan imports this package, and TestPlanPackageNeedsNoArrow asserts that plan has zero transitive dependency on data, bitmap, arrowx or anything named arrow.

So the erasure goes one step further than it first appears. A generic method on the public Expr captures its type parameters into a closure — which is required anyway, because Go 1.27 permits generic methods but forbids one from implementing an interface method, and Node is an interface. That closure is then held here behind a marker interface and implemented in a package above internal/data. internal/kernel (30) declares the concrete type; internal/physical (50) asserts it back out.

It is NOT sealed, and cannot be

Node is sealed with an unexported node() because every Node implementation lives in this package. That technique is unavailable here for the very reason this interface exists: an unexported method can only be implemented from inside the package that declares it, and the implementation MUST live outside — above internal/data, which this package may not import. So the marker is exported.

The consequence is that a type outside internal/kernel could satisfy this and reach expr.UDF.Impl. physical.evalUDF asserts for the one concrete type it knows and reports anything else as an internal error rather than ignoring it, which is the containment the seal would otherwise have provided.

type Unary 

type Unary struct {
	Op    UnaryOp
	Child Node
}

func (*Unary) Children 

func (u *Unary) Children() []Node

func (*Unary) Field 

func (u *Unary) Field(in *dtype.Schema) (dtype.Field, error)

func (*Unary) String 

func (u *Unary) String() string

type UnaryOp 

type UnaryOp uint8

UnaryOp is a one-operand operation. 

const (
	OpNeg UnaryOp = iota
	OpAbs
	OpNot // Kleene NOT: NOT null is null

	// Null predicates. These are the only operations whose result is never null,
	// because the null-ness IS the answer.
	OpIsNull
	OpIsNotNull

	// NaN predicates. Distinct from null predicates: null.is_nan() is NULL, not
	// false. Null and NaN are different things all the way down.
	OpIsNan
	OpIsNotNan
	OpIsFinite
	OpIsInfinite

	// Transcendental and rounding maths. These are the ops whose OUTPUT TYPE
	// DIFFERS FROM THEIR INPUT — Sqrt(Int64) is Float64 — which is why they cannot
	// share unaryArith, whose whole shape assumes one type throughout.
	//
	// Contiguous, because IsMath is a range test over them.
	OpSqrt
	OpCbrt
	OpExp
	OpLn
	OpLog10
	OpLog1p

	// Sign, Floor and Ceil are NOT in that block: each keeps its operand's type, so
	// they travel with Neg and Abs through unaryArith. Grouping them with the
	// transcendentals would have made sign(Int64) and floor(Int64) into Float64s —
	// and floor on an integer is the identity, so widening it would be a schema
	// change in exchange for nothing.
	OpSign
	OpFloor
	OpCeil

	// Trigonometry, appended in step 114 rather than inserted after OpLog1p, so no
	// existing constant moves. Widening like sqrt, which is why IsMath tests a
	// second range for them. Radians in, radians out; Degrees and Radians convert.
	OpSin
	OpCos
	OpTan
	OpArcSin
	OpArcCos
	OpArcTan
	OpSinh
	OpCosh
	OpTanh
	OpArcSinh
	OpArcCosh
	OpArcTanh
	OpDegrees
	OpRadians
)

func (UnaryOp) IsMath 

func (o UnaryOp) IsMath() bool

IsMath reports whether the op widens its operand to a float and returns a float. A range comparison over declaration order, like the classifiers above, which is why that block is append-only.

func (UnaryOp) IsNanPredicate 

func (o UnaryOp) IsNanPredicate() bool

IsNanPredicate reports whether the op tests a float property.

func (UnaryOp) IsNullPredicate 

func (o UnaryOp) IsNullPredicate() bool

IsNullPredicate reports whether the op's result is never null.

func (UnaryOp) String 

func (o UnaryOp) String() string

type WinFn 

type WinFn struct {
	Fn     WinFnOp
	Child  Node
	Fill   Node
	Params WinParams
}

WinFn is an ordered window function — rank, a running total, a shift.

Why not a Call

Call{Fn, Args} already carries a function plus literal arguments, and this is the same shape. But Call is contractually ELEMENTWISE: evalCall runs inside a BatchOp, which sees one batch and may run on any worker. These functions need the whole partition in order, so they are pipeline breakers. Same shape, opposite contract — and a separate node type is what makes "only legal inside a Window" checkable.

Why not an AggOp

They do not reduce. kernel.Accumulator's Finish(name, nGroups) returns one row per group; these return one row per INPUT row. They need a different kernel family.

Fill is Shift's optional replacement for the rows shifted in from outside the partition. nil means null, which is the usual want.

func (*WinFn) Children 

func (f *WinFn) Children() []Node

func (*WinFn) Field 

func (f *WinFn) Field(in *dtype.Schema) (dtype.Field, error)

func (*WinFn) String 

func (f *WinFn) String() string

type WinFnOp 

type WinFnOp uint8

WinFnOp is a function that is only meaningful within an ordered window. 

const (
	WinRank WinFnOp = iota
	WinCumCount
	WinCumSum
	WinCumProd
	WinCumMin
	WinCumMax
	WinShift

	// Forward and backward fill. TWO ops rather than one plus a Reverse flag, on
	// purpose: WinFn.String() renders the op name, so the direction can never
	// collide even if the parameter rendering is imperfect. With one op the
	// direction would live ONLY in WinParams.args, which is exactly the field that
	// has been the source of every collision bug so far.
	WinForwardFill
	WinBackwardFill

	// The rolling functions (step 149): each row's answer over the Params.N rows of
	// its partition up to and including it, in the window's order, or null where
	// fewer than Params.MinSamples of them hold a value.
	WinRollingSum
	WinRollingMean
	WinRollingMin
	WinRollingMax
	WinRollingVar
	WinRollingStd

	// The exponentially weighted functions (step 153), Polars' and pandas' ewm_*:
	// each row's answer over every row of its partition up to it, weighted by
	// Params.Alpha's decay.
	WinEwmMean
	WinEwmStd
	WinEwmVar

	// WinInterpolate fills each run of nulls between two values on the straight line
	// between them, by position in the window's order.
	WinInterpolate
)

func (WinFnOp) IsCumulative 

func (o WinFnOp) IsCumulative() bool

IsCumulative reports whether the op is a running scan over the ordered partition.

func (WinFnOp) IsEwm added in v0.5.0 

func (o WinFnOp) IsEwm() bool

IsEwm reports whether the op is exponentially weighted.

func (WinFnOp) IsRolling added in v0.5.0 

func (o WinFnOp) IsRolling() bool

IsRolling reports whether the op is over a fixed number of rows up to each.

func (WinFnOp) String 

func (o WinFnOp) String() string

type WinParams 

type WinParams struct {
	Method     RankMethod
	Descending bool
	Reverse    bool
	N          int64

	// MinSamples is a rolling function's least number of values in a window for it
	// to answer; zero means N, as Polars' min_samples=None does. Ddof is rolling
	// var's and std's delta degrees of freedom.
	MinSamples int64
	Ddof       int64

	// Center labels a rolling window at its middle row rather than its last, as
	// Polars' center=True; Nearest is Interpolate's method="nearest" (step 172).
	Center  bool
	Nearest bool

	// The ewm functions' (step 153): Alpha is the decay; NoAdjust is Polars'
	// adjust=False, so the zero value is its default; IgnoreNulls weighs by
	// position among the values rather than among the rows; Bias is ewm_var's and
	// ewm_std's bias=True.
	Alpha       float64
	NoAdjust    bool
	IgnoreNulls bool
	Bias        bool

	// A rolling function by time (step 177), Polars' rolling_*_by: each row's window
	// is the rows whose by value lies within Period of its own, closed as Closed says,
	// ClosedRight by default. The function's child is then PairOf(value, by), and N is
	// unused.
	By     bool
	Period dtype.Interval
	Closed Closed
}

WinParams carries an ordered function's configuration.

A typed struct rather than child nodes, for the reason AggParams gives: these are configuration, not operands. There is no meaning to a per-row rank method.

type Window 

type Window struct {
	Child       Node
	PartitionBy []Node
	OrderBy     []OrderKey
	Mapping     MappingStrategy

	// Inherits marks a window the library built for a sugar, not one the user
	// wrote: FillNull(FillMean)'s mean over the frame. Inside an enclosing Over it
	// takes that Over's partition, order and mapping (plan's extractWindows, step
	// 148), so FillNull(FillMean).Over(g) fills with each group's mean, as Polars
	// does. A window the user wrote inside another is still refused.
	Inherits bool
}

Window computes its child within partitions and puts the answer back on every row — the `.Over()` of the API, and SQL's OVER clause.

Its operands are ordinary children, deliberately

Node.Children's doc offers an escape hatch for exactly this situation: "A nested SCOPE body is deliberately not a child … so a generic walker must not rewrite across it." That mechanism is WRONG here, and the reason is worth stating because the alternative looks so plausible.

It exists because a list body resolves against a DIFFERENT schema — the element type. A window's child resolves against the same schema as everything around it: `Col("x").Mean().Over(Col("g"))` resolves x and g against one input schema. There is no schema change to protect against.

And hiding the child would break four things that presently work for free: RootNames would report {g} and not {x}, so projection pushdown would prune away the very column being aggregated; Expand would not reach a matcher inside the body, making All().Mean().Over(g) inexpressible; FirstErr would miss a deferred error; and OutputName would name the result after the partition key.

So the operands stay ordinary children and the scoping lives in HasUnboundAgg instead. This is what design/logical.md predicted: "Spec.PartitionBy and OrderBy are ordinary children, so RootNames and expansion pick them up automatically."

In Select, WithColumns, Filter and Sort — the row-preserving contexts, because a window preserves height. NOT inside GroupBy().Agg(), which is the nested-window case and is refused.

func (*Window) Children 

func (w *Window) Children() []Node

Children returns the child first, then the partition keys, then the order keys.

Child first matters: OutputName's generic fallback takes the first name from RootNames, so any other order would name a window after its partition key. There is an explicit OutputName case as well, and this ordering means the two agree rather than one masking the other.

func (*Window) Field 

func (w *Window) Field(in *dtype.Schema) (dtype.Field, error)

func (*Window) String 

func (w *Window) String() string

String renders the child, the partition keys, the order keys AND the mapping.

Every part is load-bearing rather than decorative, for the reason step 7 recorded when it fixed the same class of bug for Agg: the physical planner deduplicates extracted sub-expressions by keying a map on this string. Two windows that render identically become ONE computation. So `x.mean().over(g)` and `x.mean().over(h)` rendering the same would give every row the wrong partition's mean, with no error and no length mismatch anywhere to catch it.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.