Help us shape the future of Go! Take the 2026 Go Developer Survey

expr

package
v0.6.7 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 26, 2026 License: MIT Imports: 12 Imported by: 0

Documentation

Overview

Package expr is ursus's expression IR.

What an expression is

An expression is a lazy description of a column transformation. `Col("a").Add(1)` builds a tree; it computes nothing. The tree is placed in a *context* — Select, WithColumns, Filter, GroupBy.Agg — and the context decides what the expression means and what shape its output must be.

The one method that matters

Field resolves an expression's output name, dtype and nullability against an input schema, WITHOUT TOUCHING DATA. Everything the lazy engine promises rests on it:

  • LazyFrame.CollectSchema tells you the result schema before reading a byte.
  • A typo or type error is reported at plan time, not after a 40-minute scan.
  • Projection pushdown can ask "which columns does this expression need".
  • The evaluator has a contract to check itself against: for every node n and batch b, Eval(n, b).DType() must equal n.Field(b.Schema()).Type.

The design documents named this method and never defined it. It is the reason the walking skeleton exists.

Sealed

Node is sealed by the unexported node() method. Every implementation lives in this package, so a type switch over Node is exhaustive by construction and the compiler tells you when a new node type needs handling somewhere.

Index

Constants

View Source
const (
	PairFirst  = "0"
	PairSecond = "1"
)

The field names PairOf gives its two inputs. Only their positions are read; the names are fixed so the two can never collide, whatever the inputs are called.

View Source
const (
	ZoneRaise    = "raise"    // refuse the query, naming the wall clock; the default
	ZoneEarliest = "earliest" // a fold's first reading, at the offset before it
	ZoneLatest   = "latest"   // a fold's second reading
	ZoneNull     = "null"     // a null
)

The policies replace_time_zone takes for a wall clock its new zone reads twice, in a fall-back fold, or never, in a spring-forward gap: Polars' own words, which the plan carries as its third and fourth args.

View Source
const (
	RollRaise    = "raise"
	RollForward  = "forward"
	RollBackward = "backward"
)

Rolls are add_business_days' answers to a value not on a business day, in Polars' words (step 272).

View Source
const (
	AppendStrict int64 = iota
	AppendUpcast
	AppendCast
)

How Append meets its two sides' types (step 282): the same type, as Polars' upcast=False; their common one, as its default; or the other side cast to this one's, a float truncated toward zero, as extend_constant casts its value.

View Source
const CumulativeName = "\x00cumulative"

CumulativeName is the column a lowered cumulative_eval body reads its prefixes from, a name no frame's column can have.

View Source
const ElementName = "\x00element"

ElementName and OuterName are the columns a ListMap's body reads its elements and its values a list from, names no frame's column can have.

View Source
const EpochDay = 86_400e9

EpochDay is a day's nanoseconds, epoch's unit for days.

Variables

View Source
var (
	TotalUnits = []int64{86_400e9, 3_600e9, 60e9, 1e9, 1e6, 1e3, 1}
	EpochUnits = []int64{1, 1e3, 1e6, 1e9, EpochDay}
)

TotalUnits are the units a fractional total counts in, as their nanoseconds: days to nanoseconds. EpochUnits are epoch's: nanoseconds to seconds, and days.

View Source
var DateTimeParts = []string{"year", "month", "day", "hour", "minute", "second", "microsecond"}

DateTimeParts are dt.replace's components, in their operands' order.

View Source
var ErrUnexpanded = errors.New("expression contains an unexpanded multi-column selection")

ErrUnexpanded is returned when a tree still containing a *Match reaches Field. It always means a caller skipped Expand, i.e. it is an ursus bug rather than a user error.

View Source
var Generators = map[CallFn]string{
	FnIntRanges: "int_range", FnLinearSpaces: "linear_space", FnRepeatBy: "repeat",
	FnDateRanges: "date_range", FnDatetimeRanges: "datetime_range", FnTimeRanges: "time_range",
}

Generators are the per-row functions whose list a scalar generator explodes, and the name the generator reads as (step 263).

Functions

func AppendType added in v0.6.6

func AppendType(mode int64, a, b dtype.DataType) (dtype.DataType, error)

AppendType is the type Append's answer is of, its sides of types a and b met by mode.

func ArrDotType added in v0.6.7

func ArrDotType(a, b dtype.DataType) (dtype.DataType, error)

ArrDotType is arr.dot's type over Arrays a and b: the sum of their elements' products, as list.sum sums them. Each is an Array of numbers, both of one size.

func ArrFieldNames added in v0.6.7

func ArrFieldNames(c *Call, in dtype.DataType) ([]string, error)

ArrFieldNames is arr.to_struct's fields' names over an Array of in: those given, as many as its size, or field_0 on, as Polars names them.

func BareColumns

func BareColumns(n Node) []string

BareColumns returns the columns an expression reads OUTSIDE any aggregate.

These are the columns that make an expression non-aggregating, and naming them is what turns "invalid aggregate expression" into a message that says which column is the problem and what to do about it.

func CallArgs

func CallArgs(c *Call) ([]any, error)

CallArgs extracts the literal arguments of a Call, requiring each to be a literal rather than a computed expression.

Column-valued arguments are expressible in the IR and rejected here, so the error names the limitation rather than the evaluator failing on a shape it did not expect.

func CallOutputName added in v0.6.6

func CallOutputName(c *Call, recv string) string

CallOutputName is a call's name for a receiver named recv: the receiver's, but a struct's field's, which is the field's own.

func ConcatArrType added in v0.6.7

func ConcatArrType(ops []dtype.Field) (dtype.DataType, error)

ConcatArrType is concat_arr's type over ops (step 343): an Array of their one element type, an Array operand's own or another's type, as wide as an Array operand's elements and another's one value come to. Polars refuses operands of two element types, an integer beside a float too; so does this.

func CutBreaks added in v0.5.0

func CutBreaks(ops []dtype.Field) int

CutBreaks is how many of a cut's operands after the first are breaks: those before the first label, which is a String.

func EnumPhysical added in v0.6.6

func EnumPhysical(t dtype.DataType) dtype.DataType

EnumPhysical is an Enum's indices' type as Polars gives it: UInt8 up to 255 categories, UInt16 up to 65,535, and UInt32 past them. ursus stores UInt32.

func FilterChain added in v0.6.5

func FilterChain(n Node) (Node, []Node)

FilterChain is an aggregate's input stripped of its filters, and their predicates, outermost first; none when it has none.

func FirstErr

func FirstErr(n Node) error

FirstErr returns the first deferred error carried anywhere in the tree.

Expression construction cannot return an error without destroying chaining (`Col("a").Gt(5)` has nowhere to put one), so a failure such as a bad regex is parked in an *Err node and surfaces here — the expression-level half of the sticky-error model that LazyFrame implements at the frame level.

func FitsExactly

func FitsExactly(v any, to dtype.DataType) bool

FitsExactly reports whether the Go value v is representable in to with no loss.

Why this is a SECOND range check

The kernel's cast rules answer a nearby question, and reusing them is impossible: internal/expr is L20 and internal/kernel is L30, so the import is a levels violation. Two copies of a rule is the shape this codebase has been bitten by before, so the two are pinned together by a differential test rather than by hope. See TestWeakFitAgreesWithCast.

Nearby, not the same, since step 73: a cast to a float ROUNDS to the nearest value of its width, where this still asks whether the value is EXACT. So a strict cast no longer refuses what this should have refused — 0.1 to Float32 converts — and a mistake here would round silently where it used to fail loudly. The round-trip half of TestWeakFitAgreesWithCast and its "never" cases are what stand in for that backstop now.

It is deliberately CONSERVATIVE at the edges — above 2^53 for a float target, and for every type it does not enumerate. Being conservative costs nothing: each such case falls back to the type ordinary promotion would have chosen anyway.

The switch is on to.ID() and exhaustive, never on BitWidth(): Decimal reports 128 bits and Enum reports 32, so a width-based rule would silently admit both.

func GridRefusal added in v0.5.0

func GridRefusal(fn CallFn, iv dtype.Interval, in dtype.DataType) error

GridRefusal is dt.truncate's and dt.round's refusal of their interval, for a receiver of type in. truncateOut asks it at plan time and the kernel at run time, which kernel.DtCall's callers reach without resolving, so the two give one answer (TestTruncateRefusalsAgree).

func GroupNames added in v0.6.6

func GroupNames(re *regexp.Regexp) []string

GroupNames are a pattern's capture groups' field names, as Polars names them: each group's name, or its number where it has none.

func HasAgg

func HasAgg(n Node) bool

HasAgg reports whether the tree contains an aggregate.

func HasCategories added in v0.6.6

func HasCategories(n Node) bool

HasCategories reports whether n holds a get_categories.

func HasMatch

func HasMatch(n Node) bool

HasMatch reports whether the tree contains an unexpanded multi-column selector.

func HasUDF added in v0.3.0

func HasUDF(n Node) bool

HasUDF reports whether the tree contains a user-defined function.

The optimizer consults this before DUPLICATING an expression. Substituting a column's definition into a predicate is how predicate pushdown rewrites a filter into its child's namespace, and for ordinary expressions that trade is fine: the definition is cheap and the pushdown saves downstream work.

For a UDF it is a pure loss, and arithmetically so. Pushing `Filter(w > 1)` below the WithColumns that defines w runs the function on every input row IN THE FILTER, and the WithColumns still has to run it on every survivor to produce the column — n + survivors calls where not pushing costs n. There is no input for which the rewrite wins, and the function may be arbitrarily expensive.

func HasUnboundAgg

func HasUnboundAgg(n Node) bool

HasUnboundAgg reports whether the tree contains an aggregate that is NOT enclosed by a window.

Why this is separate from HasAgg

`Col("x").Mean().Over(Col("g"))` contains an *Agg, so HasAgg says true — and the row-preserving contexts use that answer to refuse aggregates. But a window preserves height: the mean is computed per partition and written back to every row, so the expression is exactly as legal in Select as `x + 1` is. An Agg beneath a Window is BOUND by it, the way a bound variable is bound by its quantifier.

HasAgg is left alone rather than redefined, because its other caller wants the literal question: IsAggregation uses it to reject `sum(a).mean()`, and an aggregate of a windowed aggregate is just as meaningless as an aggregate of an aggregate.

Partition and order keys are still descended into, so `Sum(x).Over(Sum(g))` is refused — a partition key must be a per-row value.

func HasWindow

func HasWindow(n Node) bool

HasWindow reports whether the tree contains a window or an ordered window function.

A bare WinFn counts: `Col("x").CumSum()` with no .Over() is a window over the whole frame, and it is normalised into one rather than being a second shape the planner has to know about.

func HistType added in v0.6.6

func HistType(in dtype.DataType, category, breakpoint bool) (dtype.DataType, error)

HistType is hist's answer a bin over in (step 286): its count, a UInt32, or a Struct of the bin's right edge, its interval as text and its count, as Polars' include_breakpoint and include_category ask. Polars' category is a Categorical; here it is text, as Cut's labels are. in is a number other than a Decimal.

func HorizontalInput added in v0.6.3

func HorizontalInput(fn CallFn, ops []dtype.Field) (dtype.DataType, error)

HorizontalInput is the type a horizontal reduction's operands meet at, as a conditional's branches meet: Promote, so an Int32 and a Float64 are a Float64. The aggregate's binding then types the answer, as for a column.

func Identity added in v0.3.0

func Identity(n Node) string

Identity is what two expressions must share to be one computation.

String is a rendering, not an identity

String is for Explain, the golden plans and error messages, and it leaves two things out:

  • A strong literal renders its value and not its type. Lit(int8(100)) and Lit(int64(100)) both render "lit(100)", as do Lit(1) and Lit(1.0), and float32(0.1) and 0.1. They compute different things: i8 + int8(100) wraps in Int8, and i8 + int64(100) does not.
  • A UDF renders its name, and two different functions can share one.

Four map instances merged computations on String — the window temporaries, the aggregate specs of a group-by and of a temporal group, and the window partitionings — so the second of two literals that render alike got the first one's answer, or the first one's type under its own schema (audit.md, O3).

Identity is String followed by what the rendering leaves out: for every literal, its Go type and its DataType; for every UDF, the id NewUDF minted. Every part is length-prefixed, so no two different sequences of parts concatenate alike — a DataType's own rendering can hold anything, an Enum's categories for one. It is never shown to anyone, which is what lets it carry an id that would make every golden plan depend on allocation order.

Why a rendering, and not the values compared with ==

== merges −0 and +0, and x/−0.0 and x/0.0 are −Inf and +Inf. It never merges NaN with NaN, which compute alike. And a []byte literal in a map key panics. The rendering has none of those problems: fmt prints the shortest string that reads back as the same float, so −0 prints "-0", and every NaN prints "NaN".

Why a string, and not a rewritten tree

A tree with the types written into it would need a Node type of its own, in a package whose type switches promise to be exhaustive over Node.

func IdentityAll added in v0.3.0

func IdentityAll(ns []Node) string

IdentityAll is Identity for a list of expressions, where the list is the unit — a window's partition keys.

func IndexInterval added in v0.6.7

func IndexInterval(op, name string, iv dtype.Interval, column dtype.DataType) error

IndexInterval refuses an interval of the wrong kind for its column (step 312): an integer column counts by Polars' integer intervals, "3i", and nothing else; a temporal one takes time, and no "i". Polars refuses both, the first "must be a parsed integer", the second "may not be a parsed integer". A zero interval, which some options leave unset, passes either.

func IsAggregation

func IsAggregation(n Node) bool

IsAggregation reports whether the expression is a well-formed aggregate: it reduces its input to one row.

An expression is an aggregation if every path from the root to a Col passes through an Agg. `sum(a) + 1` and `sum(a) / count()` qualify; `sum(a) + b` does not, because b is still a per-row value and there is no defined way to combine the two.

func IsConstant added in v0.6.5

func IsConstant(n Node) bool

IsConstant reports whether n reads no row and aggregates nothing: a literal, or an expression of literals.

func IsFieldPattern added in v0.6.6

func IsFieldPattern(name string) bool

IsFieldPattern reports whether a name of struct.field is a regex, between ^ and $, or "*", every field, as Polars reads it.

func IsScalar added in v0.6.5

func IsScalar(n Node) bool

IsScalar reports whether n is one value for the whole frame: it aggregates, and reads no column, and no window's answer, outside an aggregate.

func IsWeakLit

func IsWeakLit(n Node) bool

IsWeakLit reports whether n is a literal whose type should adapt to its context.

func MembershipType added in v0.3.0

func MembershipType(c *Call, against dtype.DataType) (dtype.DataType, error)

MembershipType is the type is_in and list.contains compare at: the type Eq would compare the receiver — for list.contains, its element — and each value at. So `x.IsIn(v)` refuses exactly the pairs `x.Eq(v)` refuses, and answers the same rows wherever both are defined.

The values used to be cast to the receiver's type, which answered a different question. A value the column cannot hold — 5000 against an Int8, −1 against a Uint64 — was an error where Eq says false; a string, a bool or an instant was parsed or converted, so an Int64 matched "1", a Bool matched 1 and a Date matched 13:00 on that day, all pairs Eq refuses; and a value was rounded to the column's precision, so a Datetime(s) matched 00:00:01.5 and a Float32 matched 0.1.

Equality is still grouping equality, so NaN matches NaN — the one place the two differ, and on purpose.

func OffsetRefusal added in v0.5.0

func OffsetRefusal(iv dtype.Interval, in dtype.DataType) error

OffsetRefusal is dt.offset_by's refusal of its interval, for a Date or Datetime of type in, asked at plan time and by the kernel as GridRefusal is.

The answer has the receiver's type, so an offset it cannot hold is refused rather than rounded: an hour is no number of days, and a nanosecond no number of microseconds.

func OrderCmpType added in v0.6.6

func OrderCmpType(a, b dtype.DataType) (dtype.DataType, error)

OrderCmpType is the type order_cmp compares a and b in: their common one, which must have an order.

func OrderedAggRefusal added in v0.4.0

func OrderedAggRefusal(op AggOp) *uerr.Error

OrderedAggRefusal refuses an ordering over an aggregate, here and in the physical planner's backstop.

The hint depends on the aggregate. It used to call every one of them order-independent "except for first, last, arg_min, arg_max and implode", which said both things of first (audit A12).

func OuterName added in v0.6.6

func OuterName(i int) string

func OutputName

func OutputName(n Node) string

OutputName returns the name the expression's result column will take.

The rule is LEFTMOST COLUMN WINS: `Col("a").Add(Col("b"))` is named "a", matching Polars. An explicit Alias anywhere above overrides it. An expression with no column reference at all is named "literal".

Naming has to be defined structurally like this because Select produces a schema, and a schema needs names before any data exists. Naming nodes are only consulted at the ROOT of the tree. They do not descend through operators: `Col("a").Alias("x").Add(1)` is named "a", not "x", because the alias applies to the operand rather than to the sum.

func PhysicalOf added in v0.6.7

func PhysicalOf(in dtype.DataType) (dtype.DataType, error)

PhysicalOf is the type to_physical answers in, Polars' (step 324): a Date's Int32 of days, a Datetime's, a Duration's and a Time's Int64 of ticks, a Decimal's Int128 of unscaled units, an Enum's smallest unsigned integer that holds its categories, as cat.physical's; any other scalar type itself. A nested type is refused.

func ReinterpretOf added in v0.6.7

func ReinterpretOf(in dtype.DataType, signed bool) (dtype.DataType, error)

ReinterpretOf is the type reinterpret answers in: the integer of in's width, signed or not, its bits unchanged (step 324). Polars' reinterpret, of integers only.

func ReinterpretWidth added in v0.6.6

func ReinterpretWidth(dt dtype.DataType) int

ReinterpretWidth is the bytes of dt reinterpret reads, 0 for a type it does not: an integer, a float, a Date, a Datetime, a Duration or a Time, as Polars reads them.

func ReplaceParts added in v0.6.3

func ReplaceParts(c *Call) (old, new []*Lit, def *Lit, err error)

ReplaceParts reads a replace call's literals: its old values, its new ones, and its default, nil but for FnReplaceOr.

func ReshapeType added in v0.6.7

func ReshapeType(t dtype.DataType, shape []int64) (dtype.DataType, error)

ReshapeType is the type of t's values in Arrays of shape (step 343): its leaves, what a List's or an Array's elements hold, nested in an Array for each dimension past the first, the last innermost; of one dimension, the leaves themselves. The first dimension is the rows, -1 to infer it, as Polars' reshape.

func Resolve

func Resolve(n Node, in *dtype.Schema) (dtype.Field, error)

Resolve is the entry point for expression type resolution.

It validates the tree once — deferred errors, leftover matchers — and then delegates to Field. Field implementations assume a resolvable tree, so the validation is not repeated at every level; doing it inside Field would make resolution quadratic in tree depth.

func ResolveAgg

func ResolveAgg(op AggOp, in dtype.DataType) (dtype.DataType, error)

ResolveAgg returns just the output type, for callers that do not run the accumulator.

func ResolveCall

func ResolveCall(c *Call, in dtype.DataType) (dtype.DataType, error)

ResolveCall gives the output type of a call applied to a receiver of type in.

It is the single authority, consulted by Field for the plan's schema and by the evaluator for the kernel's output type. Two copies would drift, which is the Binding lesson.

It takes the whole Call rather than just the function because `.struct.field` is the first call whose output type depends on an ARGUMENT — `field("age")` is Int64 and `field("city")` is String, from the same receiver.

It used to add "Every other family answers from the receiver's type alone", and that was false. `.dt.truncate` is the second, and the divergence lived exactly in the sentence: the kernel routes on the interval — `truncate(Time, 1 month)` has no answer while `truncate(Time, 1 hour)` does — so a resolver that saw only the receiver promised both. The argument is a CONSTANT of the expression, not data, so there was never a reason the planner could not see it.

func ResolveCond

func ResolveCond(thenN, elseN Node, then, els dtype.DataType) (dtype.DataType, error)

ResolveCond gives a conditional's output type from its two branches.

It is the single authority, consulted by Field for the plan's schema and by the evaluator for the type it casts both branches to before the kernel. Two copies would drift, which is the Binding lesson.

Why it takes NODES as well as types

Weak literal typing needs to know which branch is a literal whose type was a default rather than a choice, and a dtype.DataType cannot carry that. The evaluator in particular has only the evaluated COLUMNS by the time it asks, and a column has no weakness — so the nodes have to come from the call site. Both callers hold them, so this is a signature change rather than a redesign; the alternative, a second resolver that only the planner consults, is exactly the drift this function exists to prevent.

The temporal refusal is worth a specific hint. Promote deliberately declines two DIFFERENT temporal types — it returns early unless both sides are numeric — even though CanCast permits temporal-to-temporal and kernel.Cast rescales such a pair correctly. Widening Promote would edit a rule every binary operator shares, so the conditional points at the explicit cast instead of quietly guessing.

func ResolveHorizontal added in v0.5.0

func ResolveHorizontal(fn CallFn, ops []dtype.Field) (dtype.DataType, error)

ResolveHorizontal gives the output type of a horizontal call over operands of these fields. Like ResolveCall, it is the one authority, for the plan's schema and the kernel's output alike.

func ResolveUnary

func ResolveUnary(op UnaryOp, in dtype.DataType) (dtype.DataType, error)

ResolveUnary computes the result type of op applied to in.

func ResolveWinFn

func ResolveWinFn(fn WinFnOp, p WinParams, in dtype.DataType) (dtype.DataType, error)

ResolveWinFn gives an ordered window function's output type.

The single authority, consulted by Field for the schema and by the evaluator for the kernel's output type — the Binding lesson, applied a fourth time.

func RleType added in v0.6.6

func RleType(value dtype.DataType) dtype.DataType

RleType is rle's Struct: each run's length, a UInt32, and its value, as Polars'.

func RootNames

func RootNames(n Node) []string

RootNames returns the distinct column names the expression reads, in first-seen order.

This is the input to projection pushdown: the union of RootNames over every expression above a scan is exactly the set of columns that scan must produce. It is deliberately conservative — an unexpanded *Match contributes nothing here, which is why Expand must run first.

func StringAll

func StringAll(ns []Node) string

StringAll renders a slice of nodes for Explain output.

func ValueCountsType added in v0.6.6

func ValueCountsType(value dtype.Field, label string, normalize bool) dtype.DataType

ValueCountsType is value_counts' Struct: the value, named as its column, and its count, a UInt32, or with normalize its share, a Float64, named label.

func Walk

func Walk(n Node, fn func(Node) bool)

Walk calls fn for every node in the tree, parents before children (pre-order). Returning false from fn stops the traversal of that subtree.

func WinFnType added in v0.3.0

func WinFnType(f *WinFn, child dtype.DataType, in *dtype.Schema) (dtype.DataType, error)

WinFnType is an ordered window function's output type, its fill included: the one authority Field and the physical planner both ask.

A shift's fill used to be ignored here, so the output was always the column's type, the fill kept its own, and the kernel's select between the two refused them at run time — ShiftFill(1, 1.5) on an Int64 column was ErrInternal. The fill now meets the column as a When's branches do, through ResolveCond: a weak literal that fits takes the column's type, and otherwise the two promote, so 1.5 on an Int64 column is a Float64, which is the type Polars collects too.

Types

type Agg

type Agg struct {
	Op     AggOp
	Child  Node
	Params AggParams
}

Agg applies an aggregate function.

Only inside an aggregation context: GroupBy().Agg(...), or (later) a window via .Over(). Its defining property is that it consumes many rows and produces one, so it cannot appear in Select or WithColumns, where every expression must preserve the frame height. plan.Resolve enforces that — an Agg reaching a Project is rejected with an explanation rather than silently producing a one-row column that then fails to line up.

func (*Agg) Children

func (a *Agg) Children() []Node

func (*Agg) Field

func (a *Agg) Field(in *dtype.Schema) (dtype.Field, error)

func (*Agg) String

func (a *Agg) String() string

String renders the aggregate INCLUDING its parameters, and that is a correctness requirement rather than a display choice.

physical.extractAggs deduplicates inner aggregates by keying a map on this string, so two aggregates that render identically become ONE temporary. Before the parameters were rendered, Agg(x.Quantile(0.5), x.Quantile(0.99)) produced "x.quantile()" twice, collapsed into a single computation, and returned the median under both names with no error anywhere.

type AggBinding

type AggBinding struct {
	Acc dtype.DataType
	Out dtype.DataType
}

AggBinding is the resolved form of an aggregate: the type it ACCUMULATES in and the type it OUTPUTS.

The two differ, and the difference is load-bearing:

  • Sum of Int8 accumulates in Int128, because a thousand Int8 values overflow an Int8 immediately and a wrapped sum is a plausible wrong number.
  • Sum of Float32 accumulates in Float64 and outputs Float32, because naive float32 accumulation loses everything past ~2^24 elements. Polars has a live bug from exactly this.
  • Mean accumulates a Float64 sum and a count, and outputs a float.

Returning only the output type — as an earlier draft did — leaves the accumulator type implicit, so the first kernel written re-derives it. That is the same promotion-drift failure mode expr.Binding exists to prevent for binary operators, and it is exactly how a Float32 mean ends up silently wrong.

func ResolveAggBinding

func ResolveAggBinding(op AggOp, in dtype.DataType) (AggBinding, error)

ResolveAggBinding computes an aggregate's accumulator and output types.

type AggOp

type AggOp uint8

AggOp is an aggregate function.

const (
	AggSum AggOp = iota
	AggMean
	AggMin
	AggMax
	AggCount   // non-null values
	AggLen     // rows, including nulls
	AggNUnique // distinct non-null values
	AggFirst
	AggLast

	// APPEND ONLY below this line. IsCounting and the classifiers are written
	// against declaration order, and an op inserted mid-enum silently changes its
	// neighbours' family — the same rule CallFn and UnaryOp document.
	AggNullCount // nulls, the complement of Count
	AggAny       // Boolean OR over non-null values
	AggAllTrue   // Boolean AND over non-null values
	AggVar
	AggStd
	AggProduct
	AggArgMin // index of the minimum WITHIN the group
	AggArgMax
	AggMedian
	AggQuantile
	AggImplode // every value of the group, as a List
	AggTopK    // the k largest values of the group, largest first, as a List
	AggBottomK // the k smallest values of the group, smallest first, as a List

	// Two inputs (step 175), carried as a two-field Struct, PairOf's: the value and
	// what orders it, or the two variables.
	AggCorr  // Pearson's correlation of the pairs where both are non-null
	AggCov   // their covariance, with Params.DDof
	AggMinBy // the value at the row whose second field is the least
	AggMaxBy // and the greatest

	// Step 276: the third and fourth standardized moments, with Params.Bias and
	// Params.Fisher; and the bitwise reductions of integers or Booleans.
	AggSkew
	AggKurtosis
	AggBitAnd
	AggBitOr
	AggBitXor
	AggNanMin // the minimum, NaN where there is one: Polars' nan_min

	// Step 277: entropy, with Params.Base and Params.Normalize; and Spearman's
	// correlation of a pair, PairOf's, as AggCorr.
	AggEntropy
	AggSpearman
	// Step 324: item, the group's one value, with Params.AllowEmpty; lower_bound and
	// upper_bound, its type's least and greatest; and is_sorted, with
	// Params.Descending and Params.NullsLast.
	AggItem
	AggLowerBound
	AggUpperBound
	AggIsSorted
	// Step 330: top_k_by and bottom_k_by, a pair, PairOf's, as AggMaxBy, with Params.K.
	AggTopKBy
	AggBottomKBy
	// Step 340: mode, the group's most frequent values, a List, with
	// Params.MaintainOrder: Polars' mode in an aggregation, where it was the list
	// kernel over the values imploded.
	AggMode
)

func HorizontalAgg added in v0.6.3

func HorizontalAgg(fn CallFn) AggOp

HorizontalAgg is the aggregate a horizontal reduction runs over each row's values.

func (AggOp) IsBy added in v0.6.0

func (o AggOp) IsBy() bool

IsBy reports whether the aggregate is MinBy or MaxBy. They are order-dependent as ArgMin and ArgMax are: a tie goes to the earliest row, which a worker holding rows dealt round-robin cannot know.

func (AggOp) IsCounting

func (o AggOp) IsCounting() bool

IsCounting reports whether the aggregate returns a count, and is therefore never null and independent of the input type.

ArgMin and ArgMax return an index rather than a count and are deliberately NOT here: a group with no non-null value has no minimum, so there is no position to report and the answer is null.

func (AggOp) IsOrderDependent

func (o AggOp) IsOrderDependent() bool

IsOrderDependent reports whether the aggregate's ANSWER depends on the order its input arrived in.

Every other aggregate is commutative: a sum, a count, a minimum and a quantile are the same however the rows are shuffled, which is what lets a group-by run on several workers and fold the partials together. These five are not.

First, Last     name a position outright
ArgMin, ArgMax  report an INDEX, and kernel.argExtremumAcc.Merge shifts the
                other accumulator's positions by this one's row count — which
                is only meaningful if `other` saw a strictly later contiguous
                portion
Implode         a list's element ORDER is its data, so a shuffled input is a
                different answer rather than the same one computed differently
Mode            its values come in the order they first appear, with
                maintain_order; and without, each is its first row's value, so
                a −0 is −0 where it came before a +0, as Polars answers

The parallel aggregation driver dispatches round-robin, so no worker holds a contiguous portion and none of those four can be honoured. It declines the whole aggregate rather than the individual expression, because one query's Agg list is one hash table.

Not the same question as IsCounting: Median and Quantile buffer every value and are expensive, but they sort in Finish and are perfectly order-insensitive.

func (AggOp) IsPaired added in v0.6.0

func (o AggOp) IsPaired() bool

IsPaired reports whether the aggregate reads two inputs, as PairOf's struct.

func (AggOp) IsTopKBy added in v0.6.7

func (o AggOp) IsTopKBy() bool

IsTopKBy reports whether the aggregate is TopKBy or BottomKBy. Unlike MinBy and MaxBy they are not order-dependent: rows of one by are ordered by their values.

func (AggOp) String

func (o AggOp) String() string

type AggParams

type AggParams struct {
	DDof   uint8
	Q      float64
	Interp Interpolation
	K      int  // TopK and BottomK
	Bias   bool // Skew and Kurtosis: the biased estimator, Polars' default
	Fisher bool // Kurtosis: the excess, Fisher's, less 3, Polars' default

	Base      float64 // Entropy: the logarithm's base
	Normalize bool    // Entropy: of the values as a distribution, each over their sum

	AllowEmpty bool // Item: a group of no value is null, not refused
	Descending bool // IsSorted: in descending order
	NullsLast  bool // IsSorted: its nulls after its values

	MaintainOrder bool // Mode: its values in the order they first appear
}

AggParams carries an aggregate's configuration — ddof for Var and Std, the quantile and its interpolation for Quantile. The zero value is correct for every aggregate that takes no parameters.

Why not Args []Node, the way Call does it

Call holds its arguments as ordinary child nodes, and its doc explains that a column-valued argument "is representable in this IR and simply not implemented yet, which is the right way round for a constraint that may lift later". That is true of a regex pattern.

It is not true of these. A ddof or a quantile is CONFIGURATION, not an operand: there is no meaning to a per-row ddof, and there never will be. Making them children would put nodes into Children() that RootNames, expansion and projection pushdown then walk for nothing.

type Alias

type Alias struct {
	Child Node
	Name  string
}

Alias renames the output column.

func (*Alias) Children

func (a *Alias) Children() []Node

func (*Alias) Field

func (a *Alias) Field(in *dtype.Schema) (dtype.Field, error)

func (*Alias) String

func (a *Alias) String() string

type AllMatcher

type AllMatcher struct{}

AllMatcher selects every column.

func (*AllMatcher) Match

func (m *AllMatcher) Match(in *dtype.Schema) ([]string, error)

func (*AllMatcher) String

func (m *AllMatcher) String() string

type Binary

type Binary struct {
	Op   BinaryOp
	L, R Node
}

Binary applies a two-operand operation.

The resolved cast targets are NOT stored here: Binary is immutable and shared, and caching a resolution would tie a node to one input schema. The evaluator calls ResolveBinary itself, which is cheap and keeps exactly one copy of the promotion rules in the codebase.

func (*Binary) Children

func (b *Binary) Children() []Node

func (*Binary) Field

func (b *Binary) Field(in *dtype.Schema) (dtype.Field, error)

func (*Binary) String

func (b *Binary) String() string

type BinaryOp

type BinaryOp uint8

BinaryOp is a two-operand operation.

const (
	// Arithmetic.
	OpAdd BinaryOp = iota
	OpSub
	OpMul
	OpDiv      // true division: always produces a float
	OpFloorDiv // floored division: −7 // 2 is −4
	OpMod

	// Ordering and equality. Three-valued: if either operand is null the result
	// is null, NOT false.
	OpEq
	OpNe
	OpLt
	OpLe
	OpGt
	OpGe

	// Equality where nulls are DATA rather than absence: null == null is true and
	// the result is never null. Polars calls these eq_missing / ne_missing. They
	// need a separate kernel family, not a flag on the ordinary comparison.
	OpEqMissing
	OpNeMissing

	// Kleene three-valued logic. Note false AND null == false and true OR null ==
	// true — the output validity is NOT simply `va AND vb`.
	OpAnd
	OpOr
	OpXor

	// Exponentiation. APPENDED rather than placed beside the other arithmetic,
	// which reads oddly and is deliberate: both dispatchers — ResolveBinary and
	// kernel's arithmetic — route anything the classifiers below do not claim to
	// the arithmetic path through a `default:` arm, so position carries no meaning
	// for it, and renumbering the enum to make it look tidy would move every
	// constant for no gain.
	OpPow

	// Polars' arctan2(y, x), y the left (step 324): the angle of (x, y), a float, as
	// OpPow's binding casts both sides to one.
	OpArctan2
)

func (BinaryOp) IsComparison

func (o BinaryOp) IsComparison() bool

IsComparison reports whether the op yields Bool via three-valued comparison.

func (BinaryOp) IsLogical

func (o BinaryOp) IsLogical() bool

IsLogical reports whether the op is Kleene boolean.

func (BinaryOp) IsMissingComparison

func (o BinaryOp) IsMissingComparison() bool

IsMissingComparison reports whether the op treats nulls as data.

func (BinaryOp) IsOrdering

func (o BinaryOp) IsOrdering() bool

IsOrdering reports whether the op uses < / > semantics, as opposed to equality. Ordering comparisons are undefined for types that have no order.

func (BinaryOp) MayProduceNull added in v0.3.0

func (o BinaryOp) MayProduceNull(out dtype.DataType) bool

MayProduceNull reports whether the kernel for this op, at this output type, can manufacture a null its operands did not contain.

Why this exists

internal/kernel splits kernels into classes by how they relate to nulls, and a PARTIAL kernel "may itself produce nulls (integer division by zero, a narrowing cast, sqrt of a negative)". Binary.Field propagated only — `lf.Nullable || rf.Nullable` — which is conservative about PROPAGATION and says nothing about MANUFACTURE, the axis the kernel layer actually splits on. So `a % b` over two non-nullable integer columns was declared non-nullable and held a null wherever b was zero.

It takes the OUTPUT type, and that is not a detail

kernel.arithmetic dispatches on `Binding.Out.Physical()`, so the output type is what decides which kernel runs, and the operands' logical types can be misleading in both directions:

Int64 % Int64        Out Int64      -> arithNum   -> PARTIAL
Float64 % Float64    Out Float64    -> arithFloat -> total (math.Mod, NaN)
Int64 / Int64        Out Float64    -> arithFloat -> total (true division)
Duration // Int64    Out Duration   -> arithNum   -> PARTIAL, though the left
                                       operand is not an integer type at all

A predicate written against the operands would get the last two backwards.

Explicit membership, not a range

OpDiv sits immediately above OpFloorDiv, so `o >= OpFloorDiv && o <= OpMod` would be one constant away from wrong — and this file already carries the post-mortem of that exact mistake under "There is deliberately no IsArithmetic".

func (BinaryOp) String

func (o BinaryOp) String() string

type Binding

type Binding struct {
	CastL dtype.DataType // cast the left operand to this before the kernel
	CastR dtype.DataType
	Out   dtype.DataType // the kernel's output type
}

Binding is the resolved form of a binary operation: the type each operand must be cast to, and the type the result will have.

The evaluator MUST use this rather than re-deriving types from the operands. Two independent copies of the promotion rules would drift, and the drift would show up as a mismatch between the schema CollectSchema promised and the column Eval actually produced — which is precisely the invariant the evaluator contract test checks.

func ResolveBinary

func ResolveBinary(op BinaryOp, l, r dtype.DataType) (Binding, error)

ResolveBinary computes the operand and result types of op applied to l and r.

func ResolveBinaryNode added in v0.6.6

func ResolveBinaryNode(b *Binary, l, r dtype.DataType) (Binding, error)

ResolveBinaryNode is ResolveBinary for the node b, whose operands are of types l and r: an integer's bits beside a weak literal that fits its type exactly are of that type, as Polars keeps Col("u8") & 6 a UInt8 (step 278). A bitwise result is never wider than its operands, so the narrowing never overflows, where arithmetic's would. Binary.Field and the evaluator both ask it, so the two agree.

func (Binding) NeedsCastL

func (b Binding) NeedsCastL(l dtype.DataType) bool

NeedsCastL reports whether the left operand requires a cast.

func (Binding) NeedsCastR

func (b Binding) NeedsCastR(r dtype.DataType) bool

type Call

type Call struct {
	Fn   CallFn
	Args []Node
}

Call applies a namespaced function.

Args[0] is the RECEIVER — the column the method was called on. Args[1:] are the function's parameters and must resolve to literals; a column-valued pattern is representable in this IR and simply not implemented yet, which is the right way round for a constraint that may lift later.

Args are ordinary children, so RootNames, expansion and projection pushdown all see through a Call with no special case. That is the property the design docs predicted for window nodes and it holds here for the same reason.

func (*Call) Children

func (c *Call) Children() []Node

func (*Call) Field

func (c *Call) Field(in *dtype.Schema) (dtype.Field, error)

func (*Call) String

func (c *Call) String() string

type CallFn

type CallFn uint16

CallFn names a namespaced function — the `.str` and `.dt` families.

Why these are not UnaryOps

`Contains(pattern, literal)` and `Truncate(every)` carry ARGUMENTS, and Unary{Op, Child} has nowhere to put them. Three shapes were possible: one op per (function, argument) pair, which does not terminate; arguments smuggled into the op enum, which is the same thing wearing a hat; or one node that holds a function and its operands.

This is the third. It is also the shape every remaining namespace will need — `.list`, `.struct`, `.arr` — so paying for it once here is cheaper than three more bespoke node types later.

Appending only

Like UnaryOp, the classifiers below are range comparisons over declaration order. Insert a constant mid-enum and a neighbouring function silently changes family. Append, and add a name.

const (
	// --- string ---
	FnStrContains CallFn = iota
	FnStrStartsWith
	FnStrEndsWith
	FnStrFind
	FnStrCountMatches
	FnStrExtract
	FnStrReplace
	FnStrReplaceAll
	FnStrToLower
	FnStrToUpper
	FnStrLenBytes
	FnStrLenChars
	FnStrSlice
	FnStrStripChars
	FnStrStripPrefix
	FnStrStripSuffix
	FnStrReverse
	FnStrPadStart
	FnStrPadEnd
	FnStrZFill
	FnStrStripCharsStart
	FnStrStripCharsEnd
	FnStrEscapeRegex
	FnStrSplit
	FnStrSplitN
	FnStrExtractAll
	// The three parse a string with a strftime-style format (step 113): args are
	// the format, strict, and for a Datetime the unit and zone, for a Time the
	// unit. One function per target, because a type is not a literal.
	FnStrToDateFmt
	FnStrToDatetimeFmt
	FnStrToTimeFmt

	// --- temporal ---
	FnDtYear CallFn = iota + 100
	FnDtMonth
	FnDtDay
	FnDtHour
	FnDtMinute
	FnDtSecond
	FnDtMillisecond
	FnDtMicrosecond
	FnDtNanosecond
	FnDtWeekday
	FnDtOrdinalDay
	FnDtQuarter
	FnDtWeek
	FnDtEpoch
	FnDtTruncate
	FnDtTotalDays
	FnDtTotalHours
	FnDtTotalMinutes
	FnDtTotalSeconds
	FnDtStrftime // formats with a strftime-style format; the one arg is the format
	// The calendar (step 145). offset_by's args are an interval's months, days and
	// nanos; round's are truncate's; convert_time_zone's one arg is the zone.
	FnDtOffsetBy
	FnDtRound
	FnDtMonthStart
	FnDtMonthEnd
	FnDtIsLeapYear
	FnDtConvertTimeZone
	// Step 174: replace_time_zone's args are the zone, "" for none, then the
	// ambiguous and non_existent policies, in Polars' words (ZonePolicies).
	FnDtReplaceTimeZone
	FnDtDaysInMonth // the month's length, read in the column's zone (step 230)
	// The local date and time of day of an instant, Polars' dt.date and dt.time
	// (step 249).
	FnDtDate
	FnDtTime

	// --- type-agnostic ---
	//
	// A third family, and the first that is not keyed to a receiver type. The two
	// above dispatch on what the receiver IS — a String, an instant — but IsIn
	// applies to anything hashable, so it needs its own classifier rather than
	// widening either of theirs.
	FnIsIn CallFn = iota + 200

	// A value map (step 226). Args[0] is the receiver, Args[1] the count k of old
	// values as an Int64 literal, then the k old values, the k new ones, and for
	// FnReplaceOr the default. ReplaceParts reads them.
	FnReplace       // a value not among old keeps itself
	FnReplaceStrict // and here is refused
	FnReplaceOr     // and here, a null too, is the default

	// --- maths ---
	//
	// A fourth family, for the maths operations that carry a PARAMETER. The
	// parameterless ones are UnaryOps; Round needs a decimal count, and a Call is
	// where a parameter belongs.
	//
	// Not a field on expr.Unary, which was the obvious alternative: three places
	// rebuild a Unary by hand — physical/agg.go's extractAggs and both predicate
	// pushdown walkers — as `&Unary{Op: t.Op, Child: ...}`, so a new field would be
	// silently DROPPED the moment a predicate moved, and Round(2) would quietly
	// become Round(0). That is step 7's Agg.Params bug, three sites over. Call.String()
	// renders its arguments structurally, so the same class is impossible here.
	FnMathRound CallFn = iota + 300
	// FnMathAsFloat takes a number, or a Duration's ticks, to a float without
	// changing its value: an integer or a Decimal to the nearest Float64, a float as
	// itself. It is how PctChange subtracts in float, and is not public.
	FnMathAsFloat
	// FnMathDiffWiden takes an unsigned integer to the signed type that holds the
	// difference of any two of its values, and leaves every other type as it is. It
	// is how Diff subtracts without wrapping, and is not public.
	FnMathDiffWiden
	// The bit counts of an integer (step 146), Polars' bitwise_count_ones and its
	// five siblings, each a Uint32.
	FnMathCountOnes
	FnMathCountZeros
	FnMathLeadingOnes
	FnMathLeadingZeros
	FnMathTrailingOnes
	FnMathTrailingZeros

	// --- list ---
	//
	// A fifth family, keyed to a List receiver. These all reduce a list to a
	// SCALAR: nothing here builds a list, which is the seam the step was cut on —
	// sort, unique, reverse, slice and the set operations construct a new List
	// column and are a different problem.
	FnListLen CallFn = iota + 400
	FnListGet
	FnListContains
	FnListMin
	FnListMax
	FnListSum
	FnListMean
	FnListJoin // the elements of a List of String, joined; the one arg is the separator

	// list -> LIST. These reshape the list and hand back a list, which is the
	// distinction that split the namespace across two steps: everything above
	// answers a question ABOUT a list, everything here returns one.
	FnListReverse
	FnListHead
	FnListTail
	FnListSlice
	FnListSort
	FnListUnique
	FnListDropNulls

	// list -> a SCALAR again, each a column aggregate over a list's elements (step
	// 222), after the reshaping half because that is where the family ends.
	FnListNUnique
	FnListArgMin
	FnListArgMax

	// list -> STRUCT (step 224): element i as field i, its names the arguments.
	FnListToStruct
)
const (
	FnStructField        CallFn = iota + 500
	FnStructRenameFields        // every field renamed, in order (step 224)
	FnStructJSONEncode          // each struct as a JSON object (step 231)
	FnStructSelectFields        // the named fields, in that order, the struct's validity kept (step 299)
	FnStructDropFields          // the fields not named, Polars' struct.drop (step 323)

)

The `.struct` family.

const (
	FnConcatStr CallFn = iota + 600
	FnStructOf
	FnCut
	FnCutLeftClosed
	FnStructWithFields

	// The list set operations and gather (step 223): their second List is an
	// operand, a column, not a parameter, so they take this family's shape. The
	// answer is named after the first.
	FnListSetUnion
	FnListSetIntersection
	FnListSetDifference
	FnListSetSymmetricDifference
	FnListGather       // an index out of range is refused
	FnListGatherOrNull // and here gives null

	// The horizontal reductions (step 225): one value a row from every operand,
	// max, min, sum and mean each the column aggregate over the row's values, and
	// any and all Kleene's.
	FnMaxHorizontal
	FnMinHorizontal
	FnSumHorizontal
	FnMeanHorizontal
	FnAnyHorizontal
	FnAllHorizontal

	// A Date's or Datetime's day with a Time of day (step 230): its second operand
	// is a column, so it takes this family's shape.
	FnDtCombine

	// A list sorted by another of its length, element for element (step 253),
	// ascending or descending, nulls first: its second list is an operand.
	FnListSortBy
	FnListSortByDesc

	// Membership and indexing whose argument is a column (step 254): is_in against
	// each row's own list, and list.get and list.contains with a per-row index and
	// value. The constant forms are their families' own; these take this one's
	// shape, every operand a column.
	FnIsInExpr
	FnListGetExpr
	FnListContainsExpr

	// String functions whose pattern is a column, and replace's whose value is (step
	// 255). Each takes the constant form's arguments after the receiver, any of the
	// pattern or value a column; the flags stay literals.
	FnStrContainsExpr   // the receiver, the pattern, literal
	FnStrStartsWithExpr // the receiver, the prefix
	FnStrEndsWithExpr
	FnStrStripPrefixExpr
	FnStrStripSuffixExpr
	FnStrSplitExpr
	FnStrReplaceExpr // the receiver, the pattern (a literal), the value, literal
	FnStrReplaceAllExpr

	// dt.offset_by with an offset a row, a duration string as Every reads it (step
	// 256).
	FnDtOffsetByExpr

	// Generators, a list a row (step 263): int_ranges(start, end, step),
	// linear_spaces(start, end, n, closed) with closed a literal Closed, and
	// value.repeat_by(n). The scalar forms are these over one row, exploded.
	FnIntRanges
	FnLinearSpaces
	FnRepeatBy

	// Temporal ranges, a list a row (step 264): start, end, the interval's months,
	// days and nanoseconds, and closed, the last four literals.
	FnDateRanges
	FnDatetimeRanges
	FnTimeRanges

	// Values from their components (step 265): date(y, m, d), datetime(y, m, d, h,
	// mi, s, us), time(h, mi, s, us); a duration's and an epoch count's ticks, Int64,
	// which the root package casts; and business_day_count(start, end, mask,
	// holidays), the mask an Int64 literal and the holidays a List(Int64) literal of
	// days.
	FnMakeDate
	FnMakeDatetime
	FnMakeTime
	FnDurationTicks
	FnEpochTicks
	FnBusinessDayCount

	// concat_list(exprs...): each row's operands in one list, a List operand's
	// elements and another's value (step 266).
	FnConcatList

	// list.shift(n) and list.count_matches(v), n and v a value a row (step 269).
	FnListShift
	FnListCountMatches

	// to_integer in a base a row, signed or unsigned, strict a literal; and
	// concat_str past nulls, the separator last (step 271).
	FnStrParseInt
	FnStrParseUint
	FnConcatStrIgnoreNulls

	// Step 272: is_business_day(value, mask, holidays), the calendar as
	// business_day_count's; add_business_days(value, n, mask, holidays, roll); and
	// replace(value, year, month, day, hour, minute, second, microsecond, ambiguous,
	// non_existent), a component a null literal to keep the value's own.
	FnDtIsBusinessDay
	FnDtAddBusinessDays
	FnDtReplace

	// Step 274's binary functions of a row's operands: contains, starts_with and
	// ends_with(value, pattern); get(value, index, null_on_oob); and slice(value, a,
	// b, mode), mode "slice" with a the offset and b the length, a null for the rest,
	// or "head" or "tail" with a their n and b a null.
	FnBinContains
	FnBinStartsWith
	FnBinEndsWith
	FnBinGet
	FnBinSlice

	// order_cmp(a, b): -1, 0 or 1 as a sorts before, with or after b in the total
	// order Sort uses, NaN above every number and -0 with 0 (step 279), which
	// search_sorted, index_of and the peaks are written in.
	FnOrderCmp

	// list.append(a, b, mode): each row's two lists one after the other, their
	// elements met as Append's mode says (step 282), which Append in an aggregation
	// lowers to.
	FnListAppend

	// hash_rows(operands..., seed): each row's hash over its operands, the row's
	// group-by key hashed as hash hashes one value's, the seed an Int64 literal last
	// (step 302).
	FnHashRows

	// arr.shift(n) and arr.dot(other), n and other a value a row (step 342): an
	// Array's list.shift, the Array kept, and the sum of two Arrays' products.
	FnArrShift
	FnArrDot
	// concat_arr(exprs...): each row's operands in one Array, an Array operand's
	// elements and another's value, of one element type (step 343).
	FnConcatArr
)

The horizontal family (step 146): every argument is an OPERAND, a column read row by row, where each family above takes a receiver and literal parameters.

concat_str joins its operands' text, and a separator is an operand like any other, a literal between each two; struct builds a Struct whose fields are its operands, named as they are. Neither has a parameter, so Call's rule — Args[1:] literal — does not apply to them, and they are typed from every operand's type (ResolveHorizontal) rather than from a receiver's.

cut (step 147) puts its first operand in a bin: the operands after it are the breaks, numbers, then none or one label more than breaks, strings. Its breaks are operands because QCut's are quantiles of the column, known only when it runs. cut_left_closed is the same with each bin closed on the left.

const (
	FnListMedian CallFn = iota + 700
	FnListStd
	FnListVar
	FnListQuantile
	FnListDropNans
	FnListGatherEvery
	// A List of Lists flattened a level (step 259), which Explode in an aggregation
	// lowers to: an empty or null inner list one null, as explode makes it a row.
	FnListExplode
	// Each list's indices (step 267): in its elements' sorted order, descending and
	// nulls last two literal Booleans; of its true elements; of each value's first.
	FnListArgSort
	FnListArgTrue
	FnListArgUnique
	// list.any and list.all, ignore_nulls a literal Boolean; list.item, allow_empty
	// one; list.sample, n (negative for a fraction), the fraction, with_replacement,
	// shuffle and the seed (step 269).
	FnListAny
	FnListAll
	FnListItem
	FnListSample
	// list.mode, maintain_order a literal Boolean (step 277), which Mode in an
	// aggregation lowers to.
	FnListMode
	// list.unique_counts; list.value_counts, sort, normalize, the count's name and the
	// value's (step 280).
	FnListUniqueCounts
	FnListValueCounts
	FnListRle // list.rle, each list's runs, a Struct each (step 281)
	// list.hist, the bin count or -1, include_category, include_breakpoint and the
	// edges (step 286), which Hist in an aggregation lowers to.
	FnListHist
	FnListToArray // list.to_array, the width: an Array of it (step 341)
	FnListReshape // list.reshape, the shape: each list's values in Arrays of it, as a list (step 343)

)

The list family, continued (step 253): the aggregates a list did not have, which a reshape's aggregate lowers to, and the reshapes of E3's second half. std and var take ddof, quantile q and an interpolation; gather_every n and an offset.

A block of its own: the first families share one const block, whose iota runs on, so the list family's first block already ends at the struct family's base, and six more there collided with FnStructField. IsList covers both blocks.

const (
	FnStrContainsAny CallFn = iota + 800
	FnStrReplaceMany
	FnStrExtractMany
	FnStrFindMany
	FnStrExtractGroups
	FnStrSplitExact
	FnStrSplitInclusive
	FnStrToTitlecase
	FnStrNormalize
	// Step 271: encode and decode, hex or base64; json_path_match, the path;
	// json_decode, a null literal of the type it decodes to.
	FnStrEncode
	FnStrDecode
	FnStrJsonPathMatch
	FnStrJsonDecode
)

The string family, continued (step 270): many literal patterns at once (contains_any, replace_many, extract_many, find_many), a List literal of them first; extract_groups and split_exact, a Struct; split keeping its separator; to_titlecase; normalize, the form's name.

A block of its own, as the list family's at 700 is: nine more in the first block pushed the list family's end onto the struct family's base, 500, and the build caught the duplicate keys. IsString covers both blocks.

const (
	FnDtIsoYear CallFn = iota + 900
	FnDtCentury
	FnDtMillennium
	FnDtBaseUtcOffset
	FnDtDstOffset
	FnDtWithTimeUnit
	FnDtTotalMilliseconds
	FnDtTotalMicroseconds
	FnDtTotalNanoseconds
	FnDtTotalFractional
	FnDtSecondFractional
	FnDtEpochIn
	FnDtCastTimeUnit // a Datetime's or a Duration's values in another unit, Polars' cast_time_unit (step 325)

)

The temporal family, continued (step 272): iso_year, century and millennium; a zone's base and daylight-saving offsets; with_time_unit, the TimeUnit's code; the totals in the smaller units, and fractional in any, the unit's nanoseconds; second with its fraction; and epoch in a unit, its nanoseconds, a day's being days.

A block of its own, as the string family's at 800 is. IsTemporal covers both.

const (
	FnCatGetCategories CallFn = iota + 1000
	FnCatPhysical
)

The categorical family (step 273): get_categories, which Expand writes out as its Enum's categories once the receiver's type is known, so no kernel sees it; and physical, an Enum's indices at Polars' width. The text functions of .cat are the string family's, which read an Enum as its text.

A block of its own, as the others after 600 are. IsGeneral covers it.

const (
	FnHash CallFn = iota + 1200
	FnOrderSign
	FnToPhysical  // the values as stored, Polars' to_physical (step 324)
	FnReinterpret // an integer's bits at the other signedness; Args[1] signed

)

The binary family (step 274): encode and decode, the encoding's name and decode's strict; size, the unit's bytes, 1 for bytes; and reinterpret, a null literal of the type and the endianness. Its functions of a row's operands are horizontal. The general family, continued (step 279): hash, the seed; and order_sign, a value's sign in the total order, beside its type's zero. IsGeneral covers it.

const (
	FnMathRoundSigFigs CallFn = iota + 1300
	FnMathTruncate
)

The maths family, continued (step 283): round_sig_figs, the digits; truncate, the decimals, round's kernel toward zero. round's mode is its own second argument. IsMath covers it.

const (
	FnBinEncode CallFn = iota + 1100
	FnBinDecode
	FnBinSize
	FnBinReinterpret
)
const (
	FnArrToList CallFn = iota + 1400
	FnArrReverse
	FnArrSort
	FnArrToStruct
)

The array family (step 341): to_list, an Array's rows as a List's, which holds them as an Array does. A block of its own at 1400. Step 342's reverse and sort, a List's kept an Array, sort with list.sort's arguments; and to_struct, the fields' names, field_0 on unless given.

func (CallFn) IsArr added in v0.6.7

func (f CallFn) IsArr() bool

func (CallFn) IsBinary added in v0.6.6

func (f CallFn) IsBinary() bool

func (CallFn) IsGeneral

func (f CallFn) IsGeneral() bool

func (CallFn) IsHorizontal added in v0.5.0

func (f CallFn) IsHorizontal() bool

func (CallFn) IsList added in v0.3.0

func (f CallFn) IsList() bool

func (CallFn) IsMath

func (f CallFn) IsMath() bool

func (CallFn) IsString

func (f CallFn) IsString() bool

IsString, IsTemporal and IsGeneral classify a function by family.

These are range comparisons over declaration order, which is why the enum is append-only within each block and the blocks are spaced 100 apart.

func (CallFn) IsStruct added in v0.3.0

func (f CallFn) IsStruct() bool

func (CallFn) IsTemporal

func (f CallFn) IsTemporal() bool

func (CallFn) String

func (f CallFn) String() string

type Cast

type Cast struct {
	Child  Node
	To     dtype.DataType
	Strict bool
}

Cast converts to another type.

Strict casts fail the query on the first unrepresentable value. Non-strict casts turn it into a null — which is why a non-strict cast is ALWAYS nullable regardless of its input, even when casting a non-nullable column.

func (*Cast) Children

func (c *Cast) Children() []Node

func (*Cast) Field

func (c *Cast) Field(in *dtype.Schema) (dtype.Field, error)

func (*Cast) String

func (c *Cast) String() string

type Closed

type Closed uint8

Closed says which endpoints of an interval belong to it.

It lives here, at the level of the expression IR, for the same reason Interpolation does: the plan node, the physical operator and the root package all need it and none of them may import another's caller.

Why four values rather than a pair of bools

Two bools would admit the same four states and read worse at every call site — `Closed: ClosedBoth` against `LeftClosed: true, RightClosed: true`.

And why the zero value is none of them

The three callers want three different defaults, and every one of them is the obviously right answer for its own question:

IsBetween         both  — "between lo and hi" includes the endpoints
GroupByDynamic    left  — the only convention under which a grid TILES
Rolling           right — a row's window ends at its own timestamp, so closing
                          the left end would drop the row from its own window

A zero value that meant one of them would silently give the wrong default to the other two. So the zero means "ask the operator", each caller resolves it with Or, and no call site has to know which default it inherited.

const (
	// ClosedDefault defers to whatever the operation's own convention is. It is the
	// zero value precisely so that leaving the field alone cannot mean the wrong
	// thing somewhere else.
	ClosedDefault Closed = iota

	// ClosedLeft is [start, end) — the only value under which a grid of windows
	// partitions its input exactly.
	ClosedLeft

	// ClosedRight is (start, end].
	ClosedRight

	// ClosedBoth is [start, end]. A row landing exactly on a shared boundary is in
	// BOTH neighbouring windows, so the windows no longer partition and a count over
	// them exceeds the row count. That is the documented meaning, not an accident.
	ClosedBoth

	// ClosedNone is (start, end). The mirror: a row on a shared boundary is in
	// neither window and is dropped from the output entirely.
	ClosedNone
)

func (Closed) LowerOpen

func (c Closed) LowerOpen() bool

LowerOpen and UpperClosed are the two questions every range test asks, phrased so the caller never re-derives the four cases. Both treat ClosedDefault as ClosedLeft, which is only reachable if a caller forgot to call Or.

contains(x)  ==  (LowerOpen() ? x > lo : x >= lo) && (UpperClosed() ? x <= hi : x < hi)

func (Closed) Or

func (c Closed) Or(def Closed) Closed

Or resolves ClosedDefault to the caller's own convention and leaves anything else alone. Every consumer calls this exactly once, at the edge, so the rest of the code never has to wonder whether it is holding a resolved value.

func (Closed) String

func (c Closed) String() string

func (Closed) UpperClosed

func (c Closed) UpperClosed() bool

func (Closed) Valid added in v0.3.0

func (c Closed) Valid() bool

Valid reports whether c is one of the declared values. Closed is a public integer type, so Closed(99) compiles; it used to answer as ClosedLeft, because LowerOpen and UpperClosed both say no to it. Every consumer refuses it instead.

type Col

type Col struct{ Name string }

Col reads a column by name.

func (*Col) Children

func (c *Col) Children() []Node

func (*Col) Field

func (c *Col) Field(in *dtype.Schema) (dtype.Field, error)

func (*Col) String

func (c *Col) String() string

type Cond

type Cond struct {
	Pred Node
	Then Node
	Else Node

	// Sugar names the method a conditional was desugared from — fill_null,
	// fill_nan — so its refusal says what the caller wrote rather than what it
	// became (audit.md S16: "is_not_nan() requires a floating-point operand", to
	// someone who had called FillNan). "" for a When the caller built. A rewrite
	// that rebuilds the node may drop it: types are resolved before any rule runs,
	// and that is where these refusals arise.
	Sugar string
}

Cond is a conditional: the value of Then where Pred is true, of Else otherwise.

Why this is a node and IsBetween is not

IsBetween is two lines of sugar — `e.Ge(lo).And(e.Le(hi))` — and earns Field, evaluation, expansion, pushdown and Parquet row-group pruning for free by being built out of nodes that already exist. That is the cheapest way to add anything, and it was the first thing tried here.

It does not work for a conditional, for reasons that are properties of this codebase rather than of conditionals in general:

  • There was no masked-select kernel. Sugar has to desugar INTO something, and nothing in kernel took (mask, a, b). kernel.Select was written for this node.
  • The arithmetic dodge does not type-check. `cond*a + !cond*b` needs Bool to be numeric, and dtype.Promote refuses on purpose: "true + true is rejected".
  • There is no CSE pass, so any desugaring that names an operand twice evaluates it twice.

Three children, not parallel slices

A chain — when/then/when/then/otherwise — is built by the root package into a right-nested tree of these, rather than represented as []Pred and []Then. The reason is type unification: dtype.Promote is strictly binary, there is no n-ary helper, and a left fold is NOT safe to invent because Promote's mixed-sign rule (the smallest signed type holding both) makes the result depend on association order for operand sets like (Uint8, Int8, Uint32). Nesting makes the association explicit and identical to the order the user wrote.

All three are ordinary children, so RootNames, Walk, expansion and projection pushdown see through a Cond with no special case.

func (*Cond) Children

func (c *Cond) Children() []Node

func (*Cond) Field

func (c *Cond) Field(in *dtype.Schema) (dtype.Field, error)

func (*Cond) String

func (c *Cond) String() string

type DTypeMatcher

type DTypeMatcher struct{ Types []dtype.DataType }

DTypeMatcher selects columns by type. The number of columns it expands to is determined at plan time from the schema, so the same expression adapts as the schema changes.

func (*DTypeMatcher) Match

func (m *DTypeMatcher) Match(in *dtype.Schema) ([]string, error)

func (*DTypeMatcher) String

func (m *DTypeMatcher) String() string

type Element added in v0.6.6

type Element struct{}

Element is a list's elements, in List().Eval's body: Polars' pl.element().

func (*Element) Children added in v0.6.6

func (e *Element) Children() []Node

func (*Element) Field added in v0.6.6

func (e *Element) Field(in *dtype.Schema) (dtype.Field, error)

func (*Element) String added in v0.6.6

func (e *Element) String() string

type Err

type Err struct{ E error }

Err carries a construction-time error to the point where an error can be returned. Expression builders cannot return errors without destroying method chaining, so `ColRegex("(")` produces an *Err instead of panicking.

func (*Err) Children

func (e *Err) Children() []Node

func (*Err) Field

func (e *Err) Field(*dtype.Schema) (dtype.Field, error)

func (*Err) String

func (e *Err) String() string

type ExcludeMatcher

type ExcludeMatcher struct {
	Inner  Matcher
	Names  []string
	Re     *regexp.Regexp
	ReSrc  string
	DTypes []dtype.DataType
}

ExcludeMatcher subtracts columns from an inner matcher's result.

func (*ExcludeMatcher) Match

func (m *ExcludeMatcher) Match(in *dtype.Schema) ([]string, error)

func (*ExcludeMatcher) String

func (m *ExcludeMatcher) String() string

type FieldMatcher

type FieldMatcher struct {
	Label string
	Pred  func(dtype.Field) bool
}

FieldMatcher selects columns whose field satisfies a predicate.

It is groundwork for the selector package's dtype families (numeric, temporal, string, ...), which want a predicate rather than an enumeration of concrete types. NOTHING CONSTRUCTS ONE TODAY — that package does not exist yet, and this doc used to claim in the present tense that it did.

func (*FieldMatcher) Match

func (m *FieldMatcher) Match(in *dtype.Schema) ([]string, error)

func (*FieldMatcher) String

func (m *FieldMatcher) String() string

type Filter added in v0.6.5

type Filter struct {
	Child Node
	Pred  Node
}

Filter is a column's values where a predicate holds, Polars' Expr.filter (v0.6.5 step 251, E2 of v0.6.5-scope.md).

Its length is not the frame's, so it stands beneath an aggregate: a conditional aggregate, SQL's FILTER (WHERE ...). col("a").filter(col("b") > 0).sum() sums the a's whose b is positive, in a group-by, a window, a temporal group, and as a scalar in Select. The aggregate's sink feeds its accumulator only the rows the predicate is true on, so len, first and last count and pick among them. In a Select it is a column of its own length, the root package's plan of its own (step 258); anywhere else it is refused (LooseFilter).

func LooseFilter added in v0.6.5

func LooseFilter(n Node) *Filter

LooseFilter is the first Filter in n that is not an aggregate's input, or nil. An aggregate's input may be a chain of filters, whose predicates are themselves checked.

func (*Filter) Children added in v0.6.5

func (f *Filter) Children() []Node

func (*Filter) Field added in v0.6.5

func (f *Filter) Field(in *dtype.Schema) (dtype.Field, error)

func (*Filter) String added in v0.6.5

func (f *Filter) String() string

type Fold added in v0.6.6

type Fold struct {
	Op    string // fold, reduce, cum_fold, cum_reduce or cum_sum
	Acc   Node   // nil for a reduce
	Exprs []Node
	Fn    func(acc, x Node) Node
	// Promote casts the running values to the last one's type, as Polars' fields of
	// cum_sum_horizontal are of one type.
	Promote bool
}

Fold is Polars' fold, reduce, cum_fold, cum_reduce and cum_sum_horizontal (step 266, F1 of v0.6.6-scope.md): Fn applied from an accumulator across Exprs. Polars runs its function over Series per batch; here Fn builds the expression once, when the plan is resolved, so the optimizer sees an ordinary expression and no UDF runs.

Expand writes it out after the input's schema has expanded its Exprs, so a fold over All() folds every column, as Polars' does:

  • fold and reduce are the last accumulator, named after the first one: Acc's name, or a reduce's, whose accumulator is its first expression;
  • the cumulative forms are a Struct of the running values, a field each named after the expression folded in, the Struct named Op.

func (*Fold) Children added in v0.6.6

func (f *Fold) Children() []Node

func (*Fold) Field added in v0.6.6

func (f *Fold) Field(*dtype.Schema) (dtype.Field, error)

Field refuses: Expand writes a Fold out before anything types it.

func (*Fold) String added in v0.6.6

func (f *Fold) String() string

type IndexMatcher added in v0.6.6

type IndexMatcher struct{ Indices []int }

IndexMatcher selects columns by position, in the order written: Polars' nth (step 267). A negative index counts from the end, and one past either end is an error, as Polars refuses it.

func (*IndexMatcher) Match added in v0.6.6

func (m *IndexMatcher) Match(in *dtype.Schema) ([]string, error)

func (*IndexMatcher) String added in v0.6.6

func (m *IndexMatcher) String() string

type Interpolation

type Interpolation uint8

Interpolation selects how Quantile resolves a rank that falls between two values.

It lives here, at the level of the expression IR, rather than in the root package, because both Agg's type resolution and the kernel need it and neither may import the other's caller. The root package re-exports it as an alias, the same way it does for JoinKind.

const (
	// InterpLinear interpolates proportionally between the two neighbours. The
	// default, and the only option whose result need not be an input value.
	InterpLinear Interpolation = iota

	// InterpLower and InterpHigher take the neighbour below or above.
	InterpLower
	InterpHigher

	// InterpNearest takes the closer neighbour, rounding half away from the lower
	// one.
	InterpNearest

	// InterpMidpoint averages the two neighbours regardless of where the rank fell
	// between them.
	InterpMidpoint
)

func (Interpolation) String

func (i Interpolation) String() string

func (Interpolation) Valid added in v0.3.0

func (i Interpolation) Valid() bool

Valid reports whether i is one of the declared interpolations. Interpolation is a public integer type, so Interpolation(99) compiles; Agg.Field refuses it. It was silently linear before step 72.

type JSONPathStep added in v0.6.6

type JSONPathStep struct {
	Key     string
	Index   int
	IsIndex bool
	Deep    bool
}

JSONPathStep is one step of a JSONPath: a key, an index, or a key looked for at any depth.

func ParseJSONPath added in v0.6.6

func ParseJSONPath(p string) ([]JSONPathStep, error)

ParseJSONPath reads the subset of JSONPath Polars' json_path_match is used with: $, then .key, ['key'], [n] (from the end where negative) and ..key, which finds the first key of that name at any depth.

type ListMap added in v0.6.6

type ListMap struct {
	List  Node
	Body  Node
	Outer []Node
}

ListMap is Body evaluated over List's elements: a List of the answers with the list's offsets and nulls. Its children are List and Outer, the values a list its body reads; the body is its own, so no pass over the frame's expressions reaches it, and a window function in it is the list's, not the frame's.

func (*ListMap) Children added in v0.6.6

func (m *ListMap) Children() []Node

func (*ListMap) Field added in v0.6.6

func (m *ListMap) Field(in *dtype.Schema) (dtype.Field, error)

func (*ListMap) Scope added in v0.6.6

func (m *ListMap) Scope(in *dtype.Schema) (*dtype.Schema, error)

Scope is the schema the body is typed against: the list's elements and its values a list.

func (*ListMap) String added in v0.6.6

func (m *ListMap) String() string

type Lit

type Lit struct {
	Value any
	DT    dtype.DataType

	// Weak marks a literal whose type is a DEFAULT rather than a choice.
	//
	// ursus.Lit(0) infers Go int and fixes DT = Int64 at BUILD time. For arithmetic
	// that is right — Promote(Int32, Int64) is Int64 because addition can overflow.
	// For a null fill it is wrong, and worse, INVISIBLE: Col("i32").FillNull(FillZero)
	// contains no literal in the source at all, yet the same promotion would widen
	// the column to Int64 with nothing in the query to explain it.
	//
	// A weak literal asks the CONTEXT to adopt the other operand's type when the
	// value fits it exactly. Only ResolveCond consults it; Field still reports DT,
	// because a weak literal standing alone has no other operand to defer to.
	//
	// The zero value is STRONG, which is what keeps this change small: every
	// existing construction site keeps today's meaning untouched, and arithmetic
	// stays weak-free by construction because nothing on that path sets it.
	Weak bool
}

Lit is a scalar constant, broadcast to the length of the context.

A typed null is Value == nil with DT set; an untyped null is Value == nil with DT == Null, which Promote lets adopt the other operand's type.

func (*Lit) Children

func (l *Lit) Children() []Node

func (*Lit) Field

func (l *Lit) Field(*dtype.Schema) (dtype.Field, error)

func (*Lit) String

func (l *Lit) String() string

String renders the literal.

A weak literal MUST NOT render like a strong one

Three maps key on a rendered String() — window temporaries, aggregate temporaries and partition-key sharing — so two expressions that render alike become ONE computation and both names get the first one's answer.

Weak versus strong is a live instance of that: Coalesce(col_i32, Lit(0)) resolves to Int64 and col_i32.FillNull(FillZero) resolves to Int32, and without the marker below they render identically. Put both under .Max() in one Agg and the aggregate dedup collapses them, after which the batch's column type and the plan's promised schema disagree.

The TYPE is in there for a second collision. Weak literals normally converge on the column's type and so agree — but when the value does not fit they fall back to ordinary promotion, and a weak int64 5000 and a weak float64 5000 fall back to Int64 and Float64 respectively while printing the same digits.

type MappingStrategy

type MappingStrategy uint8

MappingStrategy decides how a window's per-partition result is placed back into the frame.

const (
	// MapGroupsToRows returns each row's own partition value, in the original row
	// order. The default, and the only one that composes with other columns.
	MapGroupsToRows MappingStrategy = iota

	// MapExplode leaves the output in partition-sorted order instead of permuting
	// it back. Cheaper — it skips the inverse permutation — and it REORDERS the
	// frame, so it is only safe when every other column is windowed the same way.
	MapExplode

	// MapJoin aggregates each partition into a List repeated across its rows.
	// Refused, and from step 46 the refusal names the thing that does it:
	// Col("v").Implode().Over(k) IS this, under the default mapping.
	MapJoin
)

func (MappingStrategy) String

func (m MappingStrategy) String() string

type Match

type Match struct{ M Matcher }

Match is a multi-column selection: Col("a","b"), ColRegex, ColDType, All, Exclude. It is a placeholder that Expand replaces with one concrete tree per matched column.

It exists as a node rather than as a separate expression kind so that `Col(pl.Float64).Mul(1.1).Alias(...)` is an ordinary tree that expansion rewrites, rather than a special case threaded through every builder.

func (*Match) Children

func (m *Match) Children() []Node

func (*Match) Field

func (m *Match) Field(*dtype.Schema) (dtype.Field, error)

func (*Match) String

func (m *Match) String() string

type Matcher

type Matcher interface {
	// Match returns the selected column names, in schema order.
	Match(in *dtype.Schema) ([]string, error)
	// String renders the matcher for Explain and error messages.
	String() string
}

Matcher selects zero or more columns from a schema.

Matchers are what make one written expression apply to many columns:

Col("weight", "height").Mean()      → two independent means
ColDType(Float64).Mul(1.1)          → every float column, count known only at plan time
All().Exclude("id")                 → everything but id

Matching is by NAME and resolved against a schema, never against data.

type NameMatcher

type NameMatcher struct{ Names []string }

NameMatcher selects an explicit list of columns, in the order written.

Unlike the other matchers, a name that is not present is an ERROR rather than an empty match: the user named a specific column, so its absence is a typo.

func (*NameMatcher) Match

func (m *NameMatcher) Match(in *dtype.Schema) ([]string, error)

func (*NameMatcher) String

func (m *NameMatcher) String() string

type Node

type Node interface {
	// Field resolves this node's single output field against an input schema.
	// It never reads data.
	//
	// Defined only for RESOLVED trees: a tree containing a *Match must be run
	// through Expand first, and returns ErrUnexpanded otherwise.
	Field(in *dtype.Schema) (dtype.Field, error)

	// Children returns the direct sub-expressions, in evaluation order.
	//
	// A nested SCOPE body is deliberately not a child. When List().Eval() arrives,
	// its body is evaluated against a different schema (the list element type),
	// so a generic walker must not rewrite across it or it would resolve
	// Element() against the outer frame.
	Children() []Node

	// String renders the node canonically. Load-bearing: it appears in Explain
	// output, in golden test files, and in error messages.
	String() string
	// contains filtered or unexported methods
}

Node is one node of the expression IR.

A Node is IMMUTABLE. Rewrites allocate new nodes; nothing ever mutates a node or a slice reachable from one. That is what makes it safe for two LazyFrames to share a subtree, which in turn is what makes `base.Filter(x)` and `base.Filter(y)` independent.

func BindScalars added in v0.6.5

func BindScalars(n Node) Node

BindScalars wraps each largest scalar subexpression of n in a window with no partition, so that a row context takes it as Polars does. A name at the root, an alias or a prefix, stays at the root: naming reads through a window either way, but Explain then shows the window where a user would write it.

func BuildCategories added in v0.6.6

func BuildCategories(n Node, in *dtype.Schema) (Node, error)

BuildCategories writes each get_categories in n out. Where n is one, the explode is named as well, as a selection names its expressions by their outermost alias; below an aggregate an alias there would hide the explode from its lowering.

func Expand

func Expand(n Node, in *dtype.Schema) ([]Node, error)

Expand rewrites one written expression into the concrete expressions it denotes.

Col("a")                          → [col("a")]                       (unchanged)
Col("a","b").Mul(2)               → [(col("a") * 2), (col("b") * 2)]
ColDType(Float64).Round(2)        → one node per Float64 column
All().Exclude("id")               → one node per remaining column

Where this runs, and why there

Expansion is a PLAN-TIME, SCHEMA-DRIVEN rewrite. It runs in the resolve phase: after a node's input schema is known, before type checking, and before any optimizer rule sees the plan.

It cannot run earlier — `ColDType(Float64)` has no meaning without a schema. It must not run later — projection pushdown asks each expression which columns it reads, and an unexpanded matcher would answer "none", so the optimizer would happily prune away the very columns the expression is about to select. Running before type checking also means the type checker only ever sees single-output trees, which is what lets Field return one Field instead of a slice.

One matcher per expression

A tree may contain at most one DISTINCT matcher. The same matcher repeated is fine and expands in lockstep — `All().Sub(All().Mean())` gives each column minus its own mean. Two different matchers are rejected, because the intended pairing is genuinely ambiguous: is it a zip, or a cross product?

func ExpandAll

func ExpandAll(ns []Node, in *dtype.Schema) ([]Node, error)

ExpandAll expands a slice of expressions, concatenating the results in order.

func HistCall added in v0.6.6

func HistCall(list Node, args []int64) Node

HistCall is list.hist of list by a ReshapeHist's arguments: the bin count, the two includes, and the edges as floats.

func LowerCumulativeEval added in v0.6.6

func LowerCumulativeEval(body Node) (Node, error)

LowerCumulativeEval is cumulative_eval's body (step 286), lowered as List().Eval lowers an aggregate, to one value a list of CumulativeName, a row's prefix. A body that is not one value a prefix is refused, as Polars refuses it, and so is one reading another column.

func LowerListEval added in v0.6.6

func LowerListEval(list, body Node, agg bool) Node

LowerListEval is Polars' list.eval(body) of list, or with agg list.agg(body): an aggregate is then the value itself, where eval makes it a list of one.

func LowerReshapes added in v0.6.5

func LowerReshapes(n Node, agg bool) Node

LowerReshapes rewrites n's reshapes into implodes and list kernels: an aggregate over a reshape anywhere, and in an aggregation (agg) every reshape, which is a list a group there. What it cannot lower is left as an error node naming why.

func ModeAggs added in v0.6.7

func ModeAggs(n Node) Node

ModeAggs rewrites each mode of a group's values, the list kernel over the values imploded as LowerReshapes lowers it, into the aggregate that counts them, AggMode (step 340): its state is an entry a distinct value, which spills past a limit, where the implode's was every value. A mode over another reshape keeps the kernel, which reads that reshape's list; so does one in a window, whose partitions implode as they did.

func PairArgs added in v0.6.4

func PairArgs(child Node) (a, b Node, ok bool)

PairArgs is PairOf's two inputs as written, or false if child is not its shape.

func PairOf added in v0.6.0

func PairOf(a, b Node) Node

PairOf is the child of a paired aggregate: a two-field Struct of a and b, each aliased to its position. The aggregate is still named after a's column, since naming reads an Alias only at the root (OutputName).

func Rebuild

func Rebuild(n Node, kids []Node) Node

Rebuild returns a copy of n with its children replaced, in the order Children returns them.

Why this exists

Rewriting an expression tree means copying every node with new children, and before this there were two hand-written copies of that walk — substitute, and the physical planner's extractAggs — with a third about to be added for windows. Each copy enumerates the node types and carries their non-child fields across by hand, which is exactly the shape of the bug step 7 hit twice: an Agg rebuilt without its Params silently became a different aggregate, and a Window rebuilt without its Mapping would silently become a different window.

Copying the struct and replacing only the child fields makes that class impossible: a new field on any node is carried by default, and the only way to get it wrong is to add a child field without updating Children, which the length check catches.

It PANICS on an unknown node type, for the same reason substitute did: Node is sealed, so reaching the default means a node type was added without teaching the rewriter about it, and silently dropping its children is worse than a crash.

type NotMatcher added in v0.6.7

type NotMatcher struct{ Inner Matcher }

NotMatcher is every column its inner matcher does not select.

func (*NotMatcher) Match added in v0.6.7

func (m *NotMatcher) Match(in *dtype.Schema) ([]string, error)

func (*NotMatcher) String added in v0.6.7

func (m *NotMatcher) String() string

type OrderKey

type OrderKey struct {
	Expr       Node
	Descending bool

	// NullsLast places nulls at the END OF THE OUTPUT, independent of direction.
	//
	// The zero value is therefore NULLS FIRST, which is ursus's default and matches
	// Polars (nulls_last=False). Per-key `.NullsLast()` overrides it.
	//
	// Tying null placement to direction — as SQL does, where nulls sort as the
	// largest value and so land last ascending and first descending — was
	// considered and rejected: it means `.Desc()` silently relocates the nulls,
	// which surprises people every time.
	NullsLast bool
}

OrderKey is one component of an ordering: an expression and how to sort by it.

It lives here rather than in the plan package because a window carries its own ordering and internal/expr may not import internal/plan. plan.SortKey is an alias for this type, so there is one definition rather than two that must agree.

func (OrderKey) String

func (k OrderKey) String() string

type OuterRef added in v0.6.6

type OuterRef struct {
	I    int
	Text string // the value's expression, for the body's text
}

OuterRef is a ListMap body's I-th value a list, computed over the frame and read by each of the list's elements.

func (*OuterRef) Children added in v0.6.6

func (o *OuterRef) Children() []Node

func (*OuterRef) Field added in v0.6.6

func (o *OuterRef) Field(in *dtype.Schema) (dtype.Field, error)

func (*OuterRef) String added in v0.6.6

func (o *OuterRef) String() string

type RankMethod

type RankMethod uint8

RankMethod decides how Rank breaks ties.

const (
	// RankOrdinal gives every row a distinct rank, ties broken by input order.
	RankOrdinal RankMethod = iota
	// RankDense gives tied rows the same rank and does not skip the next value.
	RankDense
	// RankMin and RankMax give tied rows the lowest or highest of their positions.
	RankMin
	RankMax
	// RankAverage gives tied rows the mean of their positions, so it returns a
	// float where every other method returns an integer.
	RankAverage
)

func (RankMethod) String

func (m RankMethod) String() string

func (RankMethod) Valid added in v0.3.0

func (m RankMethod) Valid() bool

Valid reports whether m is one of the declared methods. RankMethod is a public integer type, so RankMethod(99) compiles; ResolveWinFn refuses it, and the kernel never sees one.

type RegexMatcher

type RegexMatcher struct {
	Pattern string
	Re      *regexp.Regexp
}

RegexMatcher selects columns whose name matches a pattern.

ursus requires the pattern to be passed to ColRegex explicitly rather than inferring "this string looks like a regex" from leading ^ and trailing $ the way Polars does. That inference makes a column literally named "^total$" unselectable and makes every plain Col() call pay a scan for anchors.

func NewRegexMatcher

func NewRegexMatcher(pattern string) (*RegexMatcher, error)

NewRegexMatcher compiles pattern, returning an error the caller parks in an *Err node.

func (*RegexMatcher) Match

func (m *RegexMatcher) Match(in *dtype.Schema) ([]string, error)

Match returns every matching column. Matching nothing is not an error — it is the natural answer for a pattern over a schema that happens not to contain it, and erroring would make schema-driven code fragile.

func (*RegexMatcher) String

func (m *RegexMatcher) String() string

type Rename

type Rename struct {
	Child Node
	// Fn derives the output name from the child's; nil leaves it.
	Fn func(string) string
	// Keep names the output after the first column the child reads; Fn is unread.
	Keep bool
	// Fields renames each field of the struct the child answers; nil leaves them.
	Fields func(string) string
	Label  string // rendering only, e.g. `name.prefix("avg_")`
	// contains filtered or unexported fields
}

Rename applies a function to the output name.

This is the expansion-safe counterpart to Alias: where Alias imposes one fixed name (and so cannot be applied to a multi-column expression), Rename derives a name per output, which is what Name().Prefix / .Suffix / .Map need.

It carries the name namespace's other functions too (step 323): Keep names the output after the first column the child reads, Polars' name.keep, and Fields renames a struct's fields, its prefix_fields, suffix_fields and map_fields. A Rename with Fields changes the value's type, so it is a computation, not only a name, and stripNaming keeps it.

func NewRename added in v0.6.7

func NewRename(r Rename) *Rename

NewRename is r given an id of its own, for a function its label cannot render: two of them over one child are two computations to Identity.

func (*Rename) Children

func (r *Rename) Children() []Node

func (*Rename) Field

func (r *Rename) Field(in *dtype.Schema) (dtype.Field, error)

func (*Rename) Name added in v0.6.7

func (r *Rename) Name(child string) string

Name is the output name, the child's being child.

func (*Rename) String

func (r *Rename) String() string

type Reshape added in v0.6.5

type Reshape struct {
	Op    ReshapeOp
	Child Node
	Args  []int64
	By    Node
	Label string    // ValueCounts: its count's field name
	Roll  *RollSpec // Rolling: its window
}

Reshape is a length-changing operation on Child. By is SortBy's keys and Gather's indices, nil for every other.

func LooseReshape added in v0.6.5

func LooseReshape(n Node) *Reshape

LooseReshape is the first Reshape in n that LowerReshapes left, or nil: one outside an aggregation.

func (*Reshape) Children added in v0.6.5

func (r *Reshape) Children() []Node

func (*Reshape) Field added in v0.6.5

func (r *Reshape) Field(in *dtype.Schema) (dtype.Field, error)

Field is the child's: a reshape keeps its values' type, but Explode, whose values are its List's elements.

func (*Reshape) IsGenerator added in v0.6.6

func (r *Reshape) IsGenerator() bool

IsGenerator reports whether r is a scalar generator: an explode in which an empty list is no row, Polars' int_range and the rest.

func (*Reshape) String added in v0.6.5

func (r *Reshape) String() string

type ReshapeOp added in v0.6.5

type ReshapeOp uint8

ReshapeOp is a length-changing operation on a column.

const (
	ReshapeSort         ReshapeOp = iota // Args: descending, 0 or 1
	ReshapeHead                          // Args: n
	ReshapeTail                          // Args: n
	ReshapeSlice                         // Args: offset, length
	ReshapeReverse                       // no args
	ReshapeUnique                        // no args: first-seen order, null a value
	ReshapeDropNulls                     // no args
	ReshapeGet                           // Args: index; one value, not a column
	ReshapeDropNans                      // no args (step 253)
	ReshapeGatherEvery                   // Args: n, offset (step 253)
	ReshapeSortBy                        // Args: descending, 0 or 1; By the keys (step 253)
	ReshapeGather                        // no args; By the indices (step 257)
	ReshapeExplode                       // a List's elements (step 259); Args [1]: a generator's, an empty list no row (step 263)
	ReshapeArgSort                       // Args: descending, nulls last (step 267)
	ReshapeArgTrue                       // no args: the true rows' indices (step 267)
	ReshapeArgUnique                     // no args: each value's first index (step 267)
	ReshapeArgSortBy                     // Child: struct_of the keys; Args: each key's descending and nulls last (step 267)
	ReshapeMode                          // Args: maintain_order, 0 or 1: the most frequent values (step 277)
	ReshapeUniqueCounts                  // no args: each value's count, in first-seen order (step 280)
	ReshapeValueCounts                   // Args: sort, normalize; Label: the count's name; a Struct a value (step 280)
	ReshapeRle                           // no args: a Struct of each run's length and value (step 281)
	ReshapeAppend                        // By: the values appended; Args: their meeting, AppendStrict, AppendUpcast or AppendCast (step 282)
	ReshapeSample                        // Args: n, or -1 for a fraction; the fraction's bits; with_replacement; shuffle; the seed (step 282)
	ReshapeHist                          // Args: the bin count, or -1 for edges; include_category; include_breakpoint; the edges' bits (step 286)
	ReshapeRolling                       // By: the index; Roll: the window; Child aggregated over each row's (step 337)
	ReshapeArr                           // Args: the shape, -1 first to infer it: the values in Arrays of it (step 343)
)

func (ReshapeOp) String added in v0.6.5

func (o ReshapeOp) String() string

type RollSpec added in v0.6.7

type RollSpec struct {
	Period, Offset dtype.Interval
	Closed         Closed
}

RollSpec is a rolling reshape's window (step 337): a row's window is its index's instant plus Offset, Period long, Closed at the ends it says, as a frame's rolling group's.

type RollingMapFn added in v0.6.7

type RollingMapFn struct {
	Name string
	Out  dtype.DataType
	Impl UDFImpl
	// contains filtered or unexported fields
}

RollingMapFn is rolling_map's Go function (step 338), held by a window function's parameters: named, its answers declared Out, and Impl the kernel's, which expr cannot name, as a UDF's. Its identity is minted as a UDF's, by NewRollingMap, so CheckUDFNames refuses two of one name as it refuses two UDFs.

func NewRollingMap added in v0.6.7

func NewRollingMap(name string, out dtype.DataType, impl UDFImpl) *RollingMapFn

NewRollingMap builds a rolling_map's function with a fresh identity.

func (*RollingMapFn) ID added in v0.6.7

func (m *RollingMapFn) ID() uint64

ID is the function's identity, a UDF's: distinct for every one built, shared by every copy of the window that holds it.

type SetMatcher added in v0.6.7

type SetMatcher struct {
	Op   SetOp
	A, B Matcher
}

SetMatcher is two matchers' columns combined by a set operation.

func (*SetMatcher) Match added in v0.6.7

func (m *SetMatcher) Match(in *dtype.Schema) ([]string, error)

func (*SetMatcher) String added in v0.6.7

func (m *SetMatcher) String() string

type SetOp added in v0.6.7

type SetOp uint8

SetOp is how a SetMatcher combines its two matchers' columns.

const (
	SetUnion      SetOp = iota // in either
	SetIntersect               // in both
	SetDifference              // in the first and not the second
	SetXor                     // in one and not the other
)

type UDF added in v0.3.0

type UDF struct {
	Child Node

	// Out is the output type as DECLARED by the caller. It is not derived from
	// anything and cannot be — the function is opaque. The evaluator refuses a
	// column whose type disagrees with it.
	Out dtype.DataType

	// Name distinguishes this UDF from every other one in the query. Required.
	Name string

	// Kind is "map_elements" or "map_batches", for rendering and errors only.
	Kind string

	Impl UDFImpl
	// contains filtered or unexported fields
}

UDF applies a user-supplied Go function to one column.

It is the escape hatch, and it is OPAQUE: nothing in the planner can inspect it, fold it, prove it pure, or predict what it returns. Everything the engine knows about it, the user declared — which is why the evaluator CHECKS those declarations rather than trusting them. See physical.evalUDF.

Name is required, and that is a correctness argument rather than a style one

Node.String is load-bearing: "it appears in Explain output, in golden test files, and in error messages". Worse, until step 72 FOUR maps deduplicated expressions on their rendering, over three code sites, because extractAggs is instantiated twice — plan.resolveWindow's temporaries, physical.extractAggs' byKey for a hash aggregate AND for a temporal group, and physical's partition sharing — and names_test.go states the consequence: "two expressions that RENDER THE SAME become one computation, and both names get the first one's answer".

A Go closure has no printable identity. Without a name, two DIFFERENT user functions over the same column render alike, collapse into one computation, and both results come from whichever was seen first. That is the shape of two bugs this project has already shipped and fixed: an unparameterised Quantile that collapsed p50 and p99, and weak-versus-strong literals.

A monotonic counter would deduplicate correctly too, and would make every golden plan file depend on allocation order. So: a name, supplied by the caller.

That objection is about a counter that RENDERS. The id field below is a monotonic counter which never renders — String is unchanged and the golden plans with it. It is what lets plan.CheckUDFNames refuse a collision instead of merging one, and since step 72 it is part of what the four maps key on, through Identity, so two names that look alike no longer merge even where the check has not run.

func NewUDF added in v0.3.0

func NewUDF(child Node, out dtype.DataType, name, kind string, impl UDFImpl) *UDF

NewUDF builds a UDF node with a fresh identity, and is the only way to mint one.

The fields stay exported because plan and physical read them and copy the struct wholesale. The id does not, so it cannot be forged onto a different Impl.

func (*UDF) Children added in v0.3.0

func (u *UDF) Children() []Node

func (*UDF) Field added in v0.3.0

func (u *UDF) Field(in *dtype.Schema) (dtype.Field, error)

Field takes the child's NAME and the caller's declared type.

Naming follows the receiver, like Call.Field: leftmost-column-wins means Col("c").MapElements(...) is named "c", and an Alias above overrides it.

Resolving the child is not a formality — it is what makes a UDF over an unknown column a schema error at plan time rather than at the first batch.

func (*UDF) ID added in v0.3.0

func (u *UDF) ID() uint64

ID is the node's identity: distinct for every UDF the constructor ever built, and EQUAL for every copy of one. Zero means the node did not come from NewUDF.

func (*UDF) String added in v0.3.0

func (u *UDF) String() string

String renders the name, because the closure cannot be rendered and the name is the only thing that makes two UDFs distinguishable. Quoted so that a name containing punctuation cannot forge a different expression's rendering.

type UDFImpl added in v0.3.0

type UDFImpl interface{ UDFKernel() }

UDFImpl is a user-supplied kernel, erased.

Why an opaque interface rather than a func field

The obvious field is `func(*data.Column) (*data.Column, error)`, and it does not compile. internal/gen/levels puts internal/data and internal/expr at the SAME level (20), and the rule is strictly-lower, so this package cannot import internal/data. Nor can that be fixed by renumbering: internal/plan imports this package, and TestPlanPackageNeedsNoArrow asserts that plan has zero transitive dependency on data, bitmap, arrowx or anything named arrow.

So the erasure goes one step further than it first appears. A generic method on the public Expr captures its type parameters into a closure — which is required anyway, because Go 1.27 permits generic methods but forbids one from implementing an interface method, and Node is an interface. That closure is then held here behind a marker interface and implemented in a package above internal/data. internal/kernel (30) declares the concrete type; internal/physical (50) asserts it back out.

It is NOT sealed, and cannot be

Node is sealed with an unexported node() because every Node implementation lives in this package. That technique is unavailable here for the very reason this interface exists: an unexported method can only be implemented from inside the package that declares it, and the implementation MUST live outside — above internal/data, which this package may not import. So the marker is exported.

The consequence is that a type outside internal/kernel could satisfy this and reach expr.UDF.Impl. physical.evalUDF asserts for the one concrete type it knows and reports anything else as an internal error rather than ignoring it, which is the containment the seal would otherwise have provided.

type Unary

type Unary struct {
	Op    UnaryOp
	Child Node
}

func (*Unary) Children

func (u *Unary) Children() []Node

func (*Unary) Field

func (u *Unary) Field(in *dtype.Schema) (dtype.Field, error)

func (*Unary) String

func (u *Unary) String() string

type UnaryOp

type UnaryOp uint8

UnaryOp is a one-operand operation.

const (
	OpNeg UnaryOp = iota
	OpAbs
	OpNot // Kleene NOT: NOT null is null

	// Null predicates. These are the only operations whose result is never null,
	// because the null-ness IS the answer.
	OpIsNull
	OpIsNotNull

	// NaN predicates. Distinct from null predicates: null.is_nan() is NULL, not
	// false. Null and NaN are different things all the way down.
	OpIsNan
	OpIsNotNan
	OpIsFinite
	OpIsInfinite

	// Transcendental and rounding maths. These are the ops whose OUTPUT TYPE
	// DIFFERS FROM THEIR INPUT — Sqrt(Int64) is Float64 — which is why they cannot
	// share unaryArith, whose whole shape assumes one type throughout.
	//
	// Contiguous, because IsMath is a range test over them.
	OpSqrt
	OpCbrt
	OpExp
	OpLn
	OpLog10
	OpLog1p

	// Sign, Floor and Ceil are NOT in that block: each keeps its operand's type, so
	// they travel with Neg and Abs through unaryArith. Grouping them with the
	// transcendentals would have made sign(Int64) and floor(Int64) into Float64s —
	// and floor on an integer is the identity, so widening it would be a schema
	// change in exchange for nothing.
	OpSign
	OpFloor
	OpCeil

	// Trigonometry, appended in step 114 rather than inserted after OpLog1p, so no
	// existing constant moves. Widening like sqrt, which is why IsMath tests a
	// second range for them. Radians in, radians out; Degrees and Radians convert.
	OpSin
	OpCos
	OpTan
	OpArcSin
	OpArcCos
	OpArcTan
	OpSinh
	OpCosh
	OpTanh
	OpArcSinh
	OpArcCosh
	OpArcTanh
	OpDegrees
	OpRadians
	// OpCot, the cotangent, 1/tan (step 283), appended after the block it joins.
	OpCot
)

func (UnaryOp) IsMath

func (o UnaryOp) IsMath() bool

IsMath reports whether the op widens its operand to a float and returns a float. A range comparison over declaration order, like the classifiers above, which is why that block is append-only.

func (UnaryOp) IsNanPredicate

func (o UnaryOp) IsNanPredicate() bool

IsNanPredicate reports whether the op tests a float property.

func (UnaryOp) IsNullPredicate

func (o UnaryOp) IsNullPredicate() bool

IsNullPredicate reports whether the op's result is never null.

func (UnaryOp) String

func (o UnaryOp) String() string

type Unnest added in v0.6.5

type Unnest struct {
	Child  Node
	Fields []string
}

Unnest is a Struct column's fields, a column each (step 260, E4 of v0.6.5-scope.md): Polars' struct.unnest(). Like a selection of several columns it is one expression that denotes several, and Expand writes it out: a copy of the expression for each field, the Unnest replaced by that field and the copy named after it. So Col("s").Unnest() is s's fields, in Select beside other columns and in WithColumns after the frame's, and Col("s").Unnest().Add(1) each field plus one.

Fields selects the fields written out, Polars' struct.field of several names (step 275): each a field's name, a regex between ^ and $, or "*" for every field. None is every field, unnest's.

func (*Unnest) Children added in v0.6.5

func (u *Unnest) Children() []Node

func (*Unnest) Field added in v0.6.5

func (u *Unnest) Field(*dtype.Schema) (dtype.Field, error)

Field refuses: Expand writes an Unnest out before anything types it, and one that reaches here stands where one column is wanted.

func (*Unnest) String added in v0.6.5

func (u *Unnest) String() string

type Variadic added in v0.6.7

type Variadic struct {
	Op    string
	Exprs []Node
	Build func(exprs []Node) (Node, error)
}

Variadic is a function whose operands are a list, which Polars writes out before it calls the function (step 302): sum_horizontal(pl.all()) sums every column into one, concat_str(pl.all()) joins them, struct(pl.all()) packs them, coalesce(...) takes the first value, and hash_rows hashes them. Expand writes out each of Exprs over the input's schema, a selection into its columns in order, and Build makes the node over them, as buildFolds builds a Fold.

Before step 302 these were built at once, and a selection among their operands expanded the whole function once per column, as it expands any other expression: MaxHorizontal(All()) gave each column its own row's maximum, itself. A list's body, lowered when it is written, builds one over its operands as written (buildAsWritten); OutputName names one after its first operand, as the function built is named.

func (*Variadic) Children added in v0.6.7

func (v *Variadic) Children() []Node

func (*Variadic) Field added in v0.6.7

func (v *Variadic) Field(*dtype.Schema) (dtype.Field, error)

Field refuses: Expand builds a Variadic before anything types it.

func (*Variadic) String added in v0.6.7

func (v *Variadic) String() string

type WinFn

type WinFn struct {
	Fn     WinFnOp
	Child  Node
	Fill   Node
	Params WinParams
}

WinFn is an ordered window function — rank, a running total, a shift.

Why not a Call

Call{Fn, Args} already carries a function plus literal arguments, and this is the same shape. But Call is contractually ELEMENTWISE: evalCall runs inside a BatchOp, which sees one batch and may run on any worker. These functions need the whole partition in order, so they are pipeline breakers. Same shape, opposite contract — and a separate node type is what makes "only legal inside a Window" checkable.

Why not an AggOp

They do not reduce. kernel.Accumulator's Finish(name, nGroups) returns one row per group; these return one row per INPUT row. They need a different kernel family.

Fill is Shift's optional replacement for the rows shifted in from outside the partition. nil means null, which is the usual want.

func (*WinFn) Children

func (f *WinFn) Children() []Node

func (*WinFn) Field

func (f *WinFn) Field(in *dtype.Schema) (dtype.Field, error)

func (*WinFn) String

func (f *WinFn) String() string

type WinFnOp

type WinFnOp uint8

WinFnOp is a function that is only meaningful within an ordered window.

const (
	WinRank WinFnOp = iota
	WinCumCount
	WinCumSum
	WinCumProd
	WinCumMin
	WinCumMax
	WinShift

	// Forward and backward fill. TWO ops rather than one plus a Reverse flag, on
	// purpose: WinFn.String() renders the op name, so the direction can never
	// collide even if the parameter rendering is imperfect. With one op the
	// direction would live ONLY in WinParams.args, which is exactly the field that
	// has been the source of every collision bug so far.
	WinForwardFill
	WinBackwardFill

	// The rolling functions (step 149): each row's answer over the Params.N rows of
	// its partition up to and including it, in the window's order, or null where
	// fewer than Params.MinSamples of them hold a value.
	WinRollingSum
	WinRollingMean
	WinRollingMin
	WinRollingMax
	WinRollingVar
	WinRollingStd
	// A median and a quantile of the window's values (step 227): Params.Q and
	// Params.Interp, a median's 0.5 and linear.
	WinRollingMedian
	WinRollingQuantile

	// The exponentially weighted functions (step 153), Polars' and pandas' ewm_*:
	// each row's answer over every row of its partition up to it, weighted by
	// Params.Alpha's decay.
	WinEwmMean
	WinEwmStd
	WinEwmVar

	// WinInterpolate fills each run of nulls between two values on the straight line
	// between them, by position in the window's order.
	WinInterpolate

	// WinRleID numbers each row's run of equal values, from 0, by the group-by's keys
	// (step 281): Polars' rle_id.
	WinRleID

	// More rolling functions (step 284), appended so no op moves: the skew and
	// kurtosis of each row's window, Params.Bias and Params.Fisher as Polars' bias and
	// fisher; each row's rank among its window's values, by Params.Method; and the
	// covariance and correlation of a pair, the child PairOf(a, b), Params.Ddof the
	// covariance's. IsRolling takes both blocks.
	WinRollingSkew
	WinRollingKurtosis
	WinRollingRank
	WinRollingCov
	WinRollingCorr

	// WinEwmSum is Polars' ewm_sum (step 285): each value plus the running sum
	// decayed by Params.Alpha, appended so no op moves; IsEwm takes it. With
	// Params.By it and WinEwmMean are ewm_sum_by and ewm_mean_by, their child
	// PairOf(value, by), decayed by Params.Period, a half-life, in by's order.
	WinEwmSum

	// WinCumulativeEval is Polars' cumulative_eval (step 286): Params.Body, lowered by
	// LowerCumulativeEval, over each row's prefix of its partition. The physical
	// window evaluates it, as no kernel can evaluate an expression.
	WinCumulativeEval

	// WinRollingMap is Polars' rolling_map (step 338): Params.Map's Go function over
	// each row's window of Params.N rows, as the rolling functions' are, appended so
	// no op moves; IsRolling takes it.
	WinRollingMap

	// WinCatCodes is each row's value's code (step 351): the order, from 0, its
	// value first appears in among the window's non-null values, by the group-by's
	// keys; a null's null. Over the frame it is a Categorical's codes, as Polars
	// numbers them in a fresh process, which the root writes ToPhysical out as.
	WinCatCodes
)

func (WinFnOp) IsCumulative

func (o WinFnOp) IsCumulative() bool

IsCumulative reports whether the op is a running scan over the ordered partition.

func (WinFnOp) IsEwm added in v0.5.0

func (o WinFnOp) IsEwm() bool

IsEwm reports whether the op is exponentially weighted.

func (WinFnOp) IsPaired added in v0.6.6

func (o WinFnOp) IsPaired() bool

IsPaired reports whether the op is of two series, its child PairOf(a, b): rolling_cov and rolling_corr (step 284).

func (WinFnOp) IsRolling added in v0.5.0

func (o WinFnOp) IsRolling() bool

IsRolling reports whether the op is over a fixed number of rows up to each.

func (WinFnOp) String

func (o WinFnOp) String() string

type WinParams

type WinParams struct {
	Method     RankMethod
	Descending bool
	Reverse    bool
	N          int64

	// MinSamples is a rolling function's least number of values in a window for it
	// to answer; zero means N, as Polars' min_samples=None does. Ddof is rolling
	// var's and std's delta degrees of freedom.
	MinSamples int64
	Ddof       int64

	// Q and Interp are rolling_quantile's quantile and interpolation (step 227).
	Q      float64
	Interp Interpolation

	// Center labels a rolling window at its middle row rather than its last, as
	// Polars' center=True; Nearest is Interpolate's method="nearest" (step 172).
	Center  bool
	Nearest bool

	// The ewm functions' (step 153): Alpha is the decay; NoAdjust is Polars'
	// adjust=False, so the zero value is its default; IgnoreNulls weighs by
	// position among the values rather than among the rows; Bias is ewm_var's and
	// ewm_std's bias=True.
	Alpha       float64
	NoAdjust    bool
	IgnoreNulls bool
	Bias        bool

	// Fisher is rolling_kurtosis's fisher, the excess kurtosis; Bias is also the
	// rolling skew's and kurtosis's bias (step 284).
	Fisher bool

	// Body is cumulative_eval's, lowered over CumulativeName (step 286).
	Body Node

	// Map is rolling_map's function (step 338), WinRollingMap's.
	Map *RollingMapFn

	// A rolling function by time (step 177), Polars' rolling_*_by: each row's window
	// is the rows whose by value lies within Period of its own, closed as Closed says,
	// ClosedRight by default. The function's child is then PairOf(value, by), and N is
	// unused.
	By     bool
	Period dtype.Interval
	Closed Closed
}

WinParams carries an ordered function's configuration.

A typed struct rather than child nodes, for the reason AggParams gives: these are configuration, not operands. There is no meaning to a per-row rank method.

type Window

type Window struct {
	Child       Node
	PartitionBy []Node
	OrderBy     []OrderKey
	Mapping     MappingStrategy

	// Inherits marks a window the library built for a sugar, not one the user
	// wrote: FillNull(FillMean)'s mean over the frame. Inside an enclosing Over it
	// takes that Over's partition, order and mapping (plan's extractWindows, step
	// 148), so FillNull(FillMean).Over(g) fills with each group's mean, as Polars
	// does. A window the user wrote inside another is still refused.
	Inherits bool
}

Window computes its child within partitions and puts the answer back on every row — the `.Over()` of the API, and SQL's OVER clause.

Its operands are ordinary children, deliberately

Node.Children's doc offers an escape hatch for exactly this situation: "A nested SCOPE body is deliberately not a child … so a generic walker must not rewrite across it." That mechanism is WRONG here, and the reason is worth stating because the alternative looks so plausible.

It exists because a list body resolves against a DIFFERENT schema — the element type. A window's child resolves against the same schema as everything around it: `Col("x").Mean().Over(Col("g"))` resolves x and g against one input schema. There is no schema change to protect against.

And hiding the child would break four things that presently work for free: RootNames would report {g} and not {x}, so projection pushdown would prune away the very column being aggregated; Expand would not reach a matcher inside the body, making All().Mean().Over(g) inexpressible; FirstErr would miss a deferred error; and OutputName would name the result after the partition key.

So the operands stay ordinary children and the scoping lives in HasUnboundAgg instead. This is what design/logical.md predicted: "Spec.PartitionBy and OrderBy are ordinary children, so RootNames and expansion pick them up automatically."

In Select, WithColumns, Filter and Sort — the row-preserving contexts, because a window preserves height. NOT inside GroupBy().Agg(), which is the nested-window case and is refused.

func (*Window) Children

func (w *Window) Children() []Node

Children returns the child first, then the partition keys, then the order keys.

Child first matters: OutputName's generic fallback takes the first name from RootNames, so any other order would name a window after its partition key. There is an explicit OutputName case as well, and this ordering means the two agree rather than one masking the other.

func (*Window) Field

func (w *Window) Field(in *dtype.Schema) (dtype.Field, error)

func (*Window) String

func (w *Window) String() string

String renders the child, the partition keys, the order keys AND the mapping.

Every part is load-bearing rather than decorative, for the reason step 7 recorded when it fixed the same class of bug for Agg: the physical planner deduplicates extracted sub-expressions by keying a map on this string. Two windows that render identically become ONE computation. So `x.mean().over(g)` and `x.mean().over(h)` rendering the same would give every row the wrong partition's mean, with no error and no length mismatch anywhere to catch it.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL