Documentation
¶
Overview ¶
Package expr is ursus's expression IR.
What an expression is ¶
An expression is a lazy description of a column transformation. `Col("a").Add(1)` builds a tree; it computes nothing. The tree is placed in a *context* — Select, WithColumns, Filter, GroupBy.Agg — and the context decides what the expression means and what shape its output must be.
The one method that matters ¶
Field resolves an expression's output name, dtype and nullability against an input schema, WITHOUT TOUCHING DATA. Everything the lazy engine promises rests on it:
- LazyFrame.CollectSchema tells you the result schema before reading a byte.
- A typo or type error is reported at plan time, not after a 40-minute scan.
- Projection pushdown can ask "which columns does this expression need".
- The evaluator has a contract to check itself against: for every node n and batch b, Eval(n, b).DType() must equal n.Field(b.Schema()).Type.
The design documents named this method and never defined it. It is the reason the walking skeleton exists.
Sealed ¶
Node is sealed by the unexported node() method. Every implementation lives in this package, so a type switch over Node is exhaustive by construction and the compiler tells you when a new node type needs handling somewhere.
Index ¶
- Constants
- Variables
- func AppendType(mode int64, a, b dtype.DataType) (dtype.DataType, error)
- func ArrDotType(a, b dtype.DataType) (dtype.DataType, error)
- func ArrFieldNames(c *Call, in dtype.DataType) ([]string, error)
- func BareColumns(n Node) []string
- func CallArgs(c *Call) ([]any, error)
- func CallOutputName(c *Call, recv string) string
- func ConcatArrType(ops []dtype.Field) (dtype.DataType, error)
- func CutBreaks(ops []dtype.Field) int
- func EnumPhysical(t dtype.DataType) dtype.DataType
- func FilterChain(n Node) (Node, []Node)
- func FirstErr(n Node) error
- func FitsExactly(v any, to dtype.DataType) bool
- func GridRefusal(fn CallFn, iv dtype.Interval, in dtype.DataType) error
- func GroupNames(re *regexp.Regexp) []string
- func HasAgg(n Node) bool
- func HasCategories(n Node) bool
- func HasMatch(n Node) bool
- func HasUDF(n Node) bool
- func HasUnboundAgg(n Node) bool
- func HasWindow(n Node) bool
- func HistType(in dtype.DataType, category, breakpoint bool) (dtype.DataType, error)
- func HorizontalInput(fn CallFn, ops []dtype.Field) (dtype.DataType, error)
- func Identity(n Node) string
- func IdentityAll(ns []Node) string
- func IndexInterval(op, name string, iv dtype.Interval, column dtype.DataType) error
- func IsAggregation(n Node) bool
- func IsConstant(n Node) bool
- func IsFieldPattern(name string) bool
- func IsScalar(n Node) bool
- func IsWeakLit(n Node) bool
- func MembershipType(c *Call, against dtype.DataType) (dtype.DataType, error)
- func OffsetRefusal(iv dtype.Interval, in dtype.DataType) error
- func OrderCmpType(a, b dtype.DataType) (dtype.DataType, error)
- func OrderedAggRefusal(op AggOp) *uerr.Error
- func OuterName(i int) string
- func OutputName(n Node) string
- func PhysicalOf(in dtype.DataType) (dtype.DataType, error)
- func ReinterpretOf(in dtype.DataType, signed bool) (dtype.DataType, error)
- func ReinterpretWidth(dt dtype.DataType) int
- func ReplaceParts(c *Call) (old, new []*Lit, def *Lit, err error)
- func ReshapeType(t dtype.DataType, shape []int64) (dtype.DataType, error)
- func Resolve(n Node, in *dtype.Schema) (dtype.Field, error)
- func ResolveAgg(op AggOp, in dtype.DataType) (dtype.DataType, error)
- func ResolveCall(c *Call, in dtype.DataType) (dtype.DataType, error)
- func ResolveCond(thenN, elseN Node, then, els dtype.DataType) (dtype.DataType, error)
- func ResolveHorizontal(fn CallFn, ops []dtype.Field) (dtype.DataType, error)
- func ResolveUnary(op UnaryOp, in dtype.DataType) (dtype.DataType, error)
- func ResolveWinFn(fn WinFnOp, p WinParams, in dtype.DataType) (dtype.DataType, error)
- func RleType(value dtype.DataType) dtype.DataType
- func RootNames(n Node) []string
- func StringAll(ns []Node) string
- func ValueCountsType(value dtype.Field, label string, normalize bool) dtype.DataType
- func Walk(n Node, fn func(Node) bool)
- func WinFnType(f *WinFn, child dtype.DataType, in *dtype.Schema) (dtype.DataType, error)
- type Agg
- type AggBinding
- type AggOp
- type AggParams
- type Alias
- type AllMatcher
- type Binary
- type BinaryOp
- type Binding
- type Call
- type CallFn
- func (f CallFn) IsArr() bool
- func (f CallFn) IsBinary() bool
- func (f CallFn) IsGeneral() bool
- func (f CallFn) IsHorizontal() bool
- func (f CallFn) IsList() bool
- func (f CallFn) IsMath() bool
- func (f CallFn) IsString() bool
- func (f CallFn) IsStruct() bool
- func (f CallFn) IsTemporal() bool
- func (f CallFn) String() string
- type Cast
- type Closed
- type Col
- type Cond
- type DTypeMatcher
- type Element
- type Err
- type ExcludeMatcher
- type FieldMatcher
- type Filter
- type Fold
- type IndexMatcher
- type Interpolation
- type JSONPathStep
- type ListMap
- type Lit
- type MappingStrategy
- type Match
- type Matcher
- type NameMatcher
- type Node
- func BindScalars(n Node) Node
- func BuildCategories(n Node, in *dtype.Schema) (Node, error)
- func Expand(n Node, in *dtype.Schema) ([]Node, error)
- func ExpandAll(ns []Node, in *dtype.Schema) ([]Node, error)
- func HistCall(list Node, args []int64) Node
- func LowerCumulativeEval(body Node) (Node, error)
- func LowerListEval(list, body Node, agg bool) Node
- func LowerReshapes(n Node, agg bool) Node
- func ModeAggs(n Node) Node
- func PairArgs(child Node) (a, b Node, ok bool)
- func PairOf(a, b Node) Node
- func Rebuild(n Node, kids []Node) Node
- type NotMatcher
- type OrderKey
- type OuterRef
- type RankMethod
- type RegexMatcher
- type Rename
- type Reshape
- type ReshapeOp
- type RollSpec
- type RollingMapFn
- type SetMatcher
- type SetOp
- type UDF
- type UDFImpl
- type Unary
- type UnaryOp
- type Unnest
- type Variadic
- type WinFn
- type WinFnOp
- type WinParams
- type Window
Constants ¶
const ( PairFirst = "0" PairSecond = "1" )
The field names PairOf gives its two inputs. Only their positions are read; the names are fixed so the two can never collide, whatever the inputs are called.
const ( ZoneRaise = "raise" // refuse the query, naming the wall clock; the default ZoneEarliest = "earliest" // a fold's first reading, at the offset before it ZoneLatest = "latest" // a fold's second reading ZoneNull = "null" // a null )
The policies replace_time_zone takes for a wall clock its new zone reads twice, in a fall-back fold, or never, in a spring-forward gap: Polars' own words, which the plan carries as its third and fourth args.
const ( RollRaise = "raise" RollForward = "forward" RollBackward = "backward" )
Rolls are add_business_days' answers to a value not on a business day, in Polars' words (step 272).
const ( AppendStrict int64 = iota AppendUpcast AppendCast )
How Append meets its two sides' types (step 282): the same type, as Polars' upcast=False; their common one, as its default; or the other side cast to this one's, a float truncated toward zero, as extend_constant casts its value.
const CumulativeName = "\x00cumulative"
CumulativeName is the column a lowered cumulative_eval body reads its prefixes from, a name no frame's column can have.
const ElementName = "\x00element"
ElementName and OuterName are the columns a ListMap's body reads its elements and its values a list from, names no frame's column can have.
const EpochDay = 86_400e9
EpochDay is a day's nanoseconds, epoch's unit for days.
Variables ¶
var ( TotalUnits = []int64{86_400e9, 3_600e9, 60e9, 1e9, 1e6, 1e3, 1} EpochUnits = []int64{1, 1e3, 1e6, 1e9, EpochDay} )
TotalUnits are the units a fractional total counts in, as their nanoseconds: days to nanoseconds. EpochUnits are epoch's: nanoseconds to seconds, and days.
var DateTimeParts = []string{"year", "month", "day", "hour", "minute", "second", "microsecond"}
DateTimeParts are dt.replace's components, in their operands' order.
var ErrUnexpanded = errors.New("expression contains an unexpanded multi-column selection")
ErrUnexpanded is returned when a tree still containing a *Match reaches Field. It always means a caller skipped Expand, i.e. it is an ursus bug rather than a user error.
var Generators = map[CallFn]string{
FnIntRanges: "int_range", FnLinearSpaces: "linear_space", FnRepeatBy: "repeat",
FnDateRanges: "date_range", FnDatetimeRanges: "datetime_range", FnTimeRanges: "time_range",
}
Generators are the per-row functions whose list a scalar generator explodes, and the name the generator reads as (step 263).
Functions ¶
func AppendType ¶ added in v0.6.6
AppendType is the type Append's answer is of, its sides of types a and b met by mode.
func ArrDotType ¶ added in v0.6.7
ArrDotType is arr.dot's type over Arrays a and b: the sum of their elements' products, as list.sum sums them. Each is an Array of numbers, both of one size.
func ArrFieldNames ¶ added in v0.6.7
ArrFieldNames is arr.to_struct's fields' names over an Array of in: those given, as many as its size, or field_0 on, as Polars names them.
func BareColumns ¶
BareColumns returns the columns an expression reads OUTSIDE any aggregate.
These are the columns that make an expression non-aggregating, and naming them is what turns "invalid aggregate expression" into a message that says which column is the problem and what to do about it.
func CallArgs ¶
CallArgs extracts the literal arguments of a Call, requiring each to be a literal rather than a computed expression.
Column-valued arguments are expressible in the IR and rejected here, so the error names the limitation rather than the evaluator failing on a shape it did not expect.
func CallOutputName ¶ added in v0.6.6
CallOutputName is a call's name for a receiver named recv: the receiver's, but a struct's field's, which is the field's own.
func ConcatArrType ¶ added in v0.6.7
ConcatArrType is concat_arr's type over ops (step 343): an Array of their one element type, an Array operand's own or another's type, as wide as an Array operand's elements and another's one value come to. Polars refuses operands of two element types, an integer beside a float too; so does this.
func CutBreaks ¶ added in v0.5.0
CutBreaks is how many of a cut's operands after the first are breaks: those before the first label, which is a String.
func EnumPhysical ¶ added in v0.6.6
EnumPhysical is an Enum's indices' type as Polars gives it: UInt8 up to 255 categories, UInt16 up to 65,535, and UInt32 past them. ursus stores UInt32.
func FilterChain ¶ added in v0.6.5
FilterChain is an aggregate's input stripped of its filters, and their predicates, outermost first; none when it has none.
func FirstErr ¶
FirstErr returns the first deferred error carried anywhere in the tree.
Expression construction cannot return an error without destroying chaining (`Col("a").Gt(5)` has nowhere to put one), so a failure such as a bad regex is parked in an *Err node and surfaces here — the expression-level half of the sticky-error model that LazyFrame implements at the frame level.
func FitsExactly ¶
FitsExactly reports whether the Go value v is representable in to with no loss.
Why this is a SECOND range check ¶
The kernel's cast rules answer a nearby question, and reusing them is impossible: internal/expr is L20 and internal/kernel is L30, so the import is a levels violation. Two copies of a rule is the shape this codebase has been bitten by before, so the two are pinned together by a differential test rather than by hope. See TestWeakFitAgreesWithCast.
Nearby, not the same, since step 73: a cast to a float ROUNDS to the nearest value of its width, where this still asks whether the value is EXACT. So a strict cast no longer refuses what this should have refused — 0.1 to Float32 converts — and a mistake here would round silently where it used to fail loudly. The round-trip half of TestWeakFitAgreesWithCast and its "never" cases are what stand in for that backstop now.
It is deliberately CONSERVATIVE at the edges — above 2^53 for a float target, and for every type it does not enumerate. Being conservative costs nothing: each such case falls back to the type ordinary promotion would have chosen anyway.
The switch is on to.ID() and exhaustive, never on BitWidth(): Decimal reports 128 bits and Enum reports 32, so a width-based rule would silently admit both.
func GridRefusal ¶ added in v0.5.0
GridRefusal is dt.truncate's and dt.round's refusal of their interval, for a receiver of type in. truncateOut asks it at plan time and the kernel at run time, which kernel.DtCall's callers reach without resolving, so the two give one answer (TestTruncateRefusalsAgree).
func GroupNames ¶ added in v0.6.6
GroupNames are a pattern's capture groups' field names, as Polars names them: each group's name, or its number where it has none.
func HasCategories ¶ added in v0.6.6
HasCategories reports whether n holds a get_categories.
func HasUDF ¶ added in v0.3.0
HasUDF reports whether the tree contains a user-defined function.
The optimizer consults this before DUPLICATING an expression. Substituting a column's definition into a predicate is how predicate pushdown rewrites a filter into its child's namespace, and for ordinary expressions that trade is fine: the definition is cheap and the pushdown saves downstream work.
For a UDF it is a pure loss, and arithmetically so. Pushing `Filter(w > 1)` below the WithColumns that defines w runs the function on every input row IN THE FILTER, and the WithColumns still has to run it on every survivor to produce the column — n + survivors calls where not pushing costs n. There is no input for which the rewrite wins, and the function may be arbitrarily expensive.
func HasUnboundAgg ¶
HasUnboundAgg reports whether the tree contains an aggregate that is NOT enclosed by a window.
Why this is separate from HasAgg ¶
`Col("x").Mean().Over(Col("g"))` contains an *Agg, so HasAgg says true — and the row-preserving contexts use that answer to refuse aggregates. But a window preserves height: the mean is computed per partition and written back to every row, so the expression is exactly as legal in Select as `x + 1` is. An Agg beneath a Window is BOUND by it, the way a bound variable is bound by its quantifier.
HasAgg is left alone rather than redefined, because its other caller wants the literal question: IsAggregation uses it to reject `sum(a).mean()`, and an aggregate of a windowed aggregate is just as meaningless as an aggregate of an aggregate.
Partition and order keys are still descended into, so `Sum(x).Over(Sum(g))` is refused — a partition key must be a per-row value.
func HasWindow ¶
HasWindow reports whether the tree contains a window or an ordered window function.
A bare WinFn counts: `Col("x").CumSum()` with no .Over() is a window over the whole frame, and it is normalised into one rather than being a second shape the planner has to know about.
func HistType ¶ added in v0.6.6
HistType is hist's answer a bin over in (step 286): its count, a UInt32, or a Struct of the bin's right edge, its interval as text and its count, as Polars' include_breakpoint and include_category ask. Polars' category is a Categorical; here it is text, as Cut's labels are. in is a number other than a Decimal.
func HorizontalInput ¶ added in v0.6.3
HorizontalInput is the type a horizontal reduction's operands meet at, as a conditional's branches meet: Promote, so an Int32 and a Float64 are a Float64. The aggregate's binding then types the answer, as for a column.
func Identity ¶ added in v0.3.0
Identity is what two expressions must share to be one computation.
String is a rendering, not an identity ¶
String is for Explain, the golden plans and error messages, and it leaves two things out:
- A strong literal renders its value and not its type. Lit(int8(100)) and Lit(int64(100)) both render "lit(100)", as do Lit(1) and Lit(1.0), and float32(0.1) and 0.1. They compute different things: i8 + int8(100) wraps in Int8, and i8 + int64(100) does not.
- A UDF renders its name, and two different functions can share one.
Four map instances merged computations on String — the window temporaries, the aggregate specs of a group-by and of a temporal group, and the window partitionings — so the second of two literals that render alike got the first one's answer, or the first one's type under its own schema (audit.md, O3).
Identity is String followed by what the rendering leaves out: for every literal, its Go type and its DataType; for every UDF, the id NewUDF minted. Every part is length-prefixed, so no two different sequences of parts concatenate alike — a DataType's own rendering can hold anything, an Enum's categories for one. It is never shown to anyone, which is what lets it carry an id that would make every golden plan depend on allocation order.
Why a rendering, and not the values compared with == ¶
== merges −0 and +0, and x/−0.0 and x/0.0 are −Inf and +Inf. It never merges NaN with NaN, which compute alike. And a []byte literal in a map key panics. The rendering has none of those problems: fmt prints the shortest string that reads back as the same float, so −0 prints "-0", and every NaN prints "NaN".
Why a string, and not a rewritten tree ¶
A tree with the types written into it would need a Node type of its own, in a package whose type switches promise to be exhaustive over Node.
func IdentityAll ¶ added in v0.3.0
IdentityAll is Identity for a list of expressions, where the list is the unit — a window's partition keys.
func IndexInterval ¶ added in v0.6.7
IndexInterval refuses an interval of the wrong kind for its column (step 312): an integer column counts by Polars' integer intervals, "3i", and nothing else; a temporal one takes time, and no "i". Polars refuses both, the first "must be a parsed integer", the second "may not be a parsed integer". A zero interval, which some options leave unset, passes either.
func IsAggregation ¶
IsAggregation reports whether the expression is a well-formed aggregate: it reduces its input to one row.
An expression is an aggregation if every path from the root to a Col passes through an Agg. `sum(a) + 1` and `sum(a) / count()` qualify; `sum(a) + b` does not, because b is still a per-row value and there is no defined way to combine the two.
func IsConstant ¶ added in v0.6.5
IsConstant reports whether n reads no row and aggregates nothing: a literal, or an expression of literals.
func IsFieldPattern ¶ added in v0.6.6
IsFieldPattern reports whether a name of struct.field is a regex, between ^ and $, or "*", every field, as Polars reads it.
func IsScalar ¶ added in v0.6.5
IsScalar reports whether n is one value for the whole frame: it aggregates, and reads no column, and no window's answer, outside an aggregate.
func MembershipType ¶ added in v0.3.0
MembershipType is the type is_in and list.contains compare at: the type Eq would compare the receiver — for list.contains, its element — and each value at. So `x.IsIn(v)` refuses exactly the pairs `x.Eq(v)` refuses, and answers the same rows wherever both are defined.
The values used to be cast to the receiver's type, which answered a different question. A value the column cannot hold — 5000 against an Int8, −1 against a Uint64 — was an error where Eq says false; a string, a bool or an instant was parsed or converted, so an Int64 matched "1", a Bool matched 1 and a Date matched 13:00 on that day, all pairs Eq refuses; and a value was rounded to the column's precision, so a Datetime(s) matched 00:00:01.5 and a Float32 matched 0.1.
Equality is still grouping equality, so NaN matches NaN — the one place the two differ, and on purpose.
func OffsetRefusal ¶ added in v0.5.0
OffsetRefusal is dt.offset_by's refusal of its interval, for a Date or Datetime of type in, asked at plan time and by the kernel as GridRefusal is.
The answer has the receiver's type, so an offset it cannot hold is refused rather than rounded: an hour is no number of days, and a nanosecond no number of microseconds.
func OrderCmpType ¶ added in v0.6.6
OrderCmpType is the type order_cmp compares a and b in: their common one, which must have an order.
func OrderedAggRefusal ¶ added in v0.4.0
OrderedAggRefusal refuses an ordering over an aggregate, here and in the physical planner's backstop.
The hint depends on the aggregate. It used to call every one of them order-independent "except for first, last, arg_min, arg_max and implode", which said both things of first (audit A12).
func OutputName ¶
OutputName returns the name the expression's result column will take.
The rule is LEFTMOST COLUMN WINS: `Col("a").Add(Col("b"))` is named "a", matching Polars. An explicit Alias anywhere above overrides it. An expression with no column reference at all is named "literal".
Naming has to be defined structurally like this because Select produces a schema, and a schema needs names before any data exists. Naming nodes are only consulted at the ROOT of the tree. They do not descend through operators: `Col("a").Alias("x").Add(1)` is named "a", not "x", because the alias applies to the operand rather than to the sum.
func PhysicalOf ¶ added in v0.6.7
PhysicalOf is the type to_physical answers in, Polars' (step 324): a Date's Int32 of days, a Datetime's, a Duration's and a Time's Int64 of ticks, a Decimal's Int128 of unscaled units, an Enum's smallest unsigned integer that holds its categories, as cat.physical's; any other scalar type itself. A nested type is refused.
func ReinterpretOf ¶ added in v0.6.7
ReinterpretOf is the type reinterpret answers in: the integer of in's width, signed or not, its bits unchanged (step 324). Polars' reinterpret, of integers only.
func ReinterpretWidth ¶ added in v0.6.6
ReinterpretWidth is the bytes of dt reinterpret reads, 0 for a type it does not: an integer, a float, a Date, a Datetime, a Duration or a Time, as Polars reads them.
func ReplaceParts ¶ added in v0.6.3
ReplaceParts reads a replace call's literals: its old values, its new ones, and its default, nil but for FnReplaceOr.
func ReshapeType ¶ added in v0.6.7
ReshapeType is the type of t's values in Arrays of shape (step 343): its leaves, what a List's or an Array's elements hold, nested in an Array for each dimension past the first, the last innermost; of one dimension, the leaves themselves. The first dimension is the rows, -1 to infer it, as Polars' reshape.
func Resolve ¶
Resolve is the entry point for expression type resolution.
It validates the tree once — deferred errors, leftover matchers — and then delegates to Field. Field implementations assume a resolvable tree, so the validation is not repeated at every level; doing it inside Field would make resolution quadratic in tree depth.
func ResolveAgg ¶
ResolveAgg returns just the output type, for callers that do not run the accumulator.
func ResolveCall ¶
ResolveCall gives the output type of a call applied to a receiver of type in.
It is the single authority, consulted by Field for the plan's schema and by the evaluator for the kernel's output type. Two copies would drift, which is the Binding lesson.
It takes the whole Call rather than just the function because `.struct.field` is the first call whose output type depends on an ARGUMENT — `field("age")` is Int64 and `field("city")` is String, from the same receiver.
It used to add "Every other family answers from the receiver's type alone", and that was false. `.dt.truncate` is the second, and the divergence lived exactly in the sentence: the kernel routes on the interval — `truncate(Time, 1 month)` has no answer while `truncate(Time, 1 hour)` does — so a resolver that saw only the receiver promised both. The argument is a CONSTANT of the expression, not data, so there was never a reason the planner could not see it.
func ResolveCond ¶
ResolveCond gives a conditional's output type from its two branches.
It is the single authority, consulted by Field for the plan's schema and by the evaluator for the type it casts both branches to before the kernel. Two copies would drift, which is the Binding lesson.
Why it takes NODES as well as types ¶
Weak literal typing needs to know which branch is a literal whose type was a default rather than a choice, and a dtype.DataType cannot carry that. The evaluator in particular has only the evaluated COLUMNS by the time it asks, and a column has no weakness — so the nodes have to come from the call site. Both callers hold them, so this is a signature change rather than a redesign; the alternative, a second resolver that only the planner consults, is exactly the drift this function exists to prevent.
The temporal refusal is worth a specific hint. Promote deliberately declines two DIFFERENT temporal types — it returns early unless both sides are numeric — even though CanCast permits temporal-to-temporal and kernel.Cast rescales such a pair correctly. Widening Promote would edit a rule every binary operator shares, so the conditional points at the explicit cast instead of quietly guessing.
func ResolveHorizontal ¶ added in v0.5.0
ResolveHorizontal gives the output type of a horizontal call over operands of these fields. Like ResolveCall, it is the one authority, for the plan's schema and the kernel's output alike.
func ResolveUnary ¶
ResolveUnary computes the result type of op applied to in.
func ResolveWinFn ¶
ResolveWinFn gives an ordered window function's output type.
The single authority, consulted by Field for the schema and by the evaluator for the kernel's output type — the Binding lesson, applied a fourth time.
func RleType ¶ added in v0.6.6
RleType is rle's Struct: each run's length, a UInt32, and its value, as Polars'.
func RootNames ¶
RootNames returns the distinct column names the expression reads, in first-seen order.
This is the input to projection pushdown: the union of RootNames over every expression above a scan is exactly the set of columns that scan must produce. It is deliberately conservative — an unexpanded *Match contributes nothing here, which is why Expand must run first.
func ValueCountsType ¶ added in v0.6.6
ValueCountsType is value_counts' Struct: the value, named as its column, and its count, a UInt32, or with normalize its share, a Float64, named label.
func Walk ¶
Walk calls fn for every node in the tree, parents before children (pre-order). Returning false from fn stops the traversal of that subtree.
func WinFnType ¶ added in v0.3.0
WinFnType is an ordered window function's output type, its fill included: the one authority Field and the physical planner both ask.
A shift's fill used to be ignored here, so the output was always the column's type, the fill kept its own, and the kernel's select between the two refused them at run time — ShiftFill(1, 1.5) on an Int64 column was ErrInternal. The fill now meets the column as a When's branches do, through ResolveCond: a weak literal that fits takes the column's type, and otherwise the two promote, so 1.5 on an Int64 column is a Float64, which is the type Polars collects too.
Types ¶
type Agg ¶
Agg applies an aggregate function.
Where an Agg is legal ¶
Only inside an aggregation context: GroupBy().Agg(...), or (later) a window via .Over(). Its defining property is that it consumes many rows and produces one, so it cannot appear in Select or WithColumns, where every expression must preserve the frame height. plan.Resolve enforces that — an Agg reaching a Project is rejected with an explanation rather than silently producing a one-row column that then fails to line up.
func (*Agg) String ¶
String renders the aggregate INCLUDING its parameters, and that is a correctness requirement rather than a display choice.
physical.extractAggs deduplicates inner aggregates by keying a map on this string, so two aggregates that render identically become ONE temporary. Before the parameters were rendered, Agg(x.Quantile(0.5), x.Quantile(0.99)) produced "x.quantile()" twice, collapsed into a single computation, and returned the median under both names with no error anywhere.
type AggBinding ¶
AggBinding is the resolved form of an aggregate: the type it ACCUMULATES in and the type it OUTPUTS.
The two differ, and the difference is load-bearing:
- Sum of Int8 accumulates in Int128, because a thousand Int8 values overflow an Int8 immediately and a wrapped sum is a plausible wrong number.
- Sum of Float32 accumulates in Float64 and outputs Float32, because naive float32 accumulation loses everything past ~2^24 elements. Polars has a live bug from exactly this.
- Mean accumulates a Float64 sum and a count, and outputs a float.
Returning only the output type — as an earlier draft did — leaves the accumulator type implicit, so the first kernel written re-derives it. That is the same promotion-drift failure mode expr.Binding exists to prevent for binary operators, and it is exactly how a Float32 mean ends up silently wrong.
func ResolveAggBinding ¶
func ResolveAggBinding(op AggOp, in dtype.DataType) (AggBinding, error)
ResolveAggBinding computes an aggregate's accumulator and output types.
type AggOp ¶
type AggOp uint8
AggOp is an aggregate function.
const ( AggSum AggOp = iota AggMean AggMin AggMax AggCount // non-null values AggLen // rows, including nulls AggNUnique // distinct non-null values AggFirst AggLast // APPEND ONLY below this line. IsCounting and the classifiers are written // against declaration order, and an op inserted mid-enum silently changes its // neighbours' family — the same rule CallFn and UnaryOp document. AggNullCount // nulls, the complement of Count AggAny // Boolean OR over non-null values AggAllTrue // Boolean AND over non-null values AggVar AggStd AggProduct AggArgMin // index of the minimum WITHIN the group AggArgMax AggMedian AggQuantile AggImplode // every value of the group, as a List AggTopK // the k largest values of the group, largest first, as a List AggBottomK // the k smallest values of the group, smallest first, as a List // Two inputs (step 175), carried as a two-field Struct, PairOf's: the value and // what orders it, or the two variables. AggCorr // Pearson's correlation of the pairs where both are non-null AggCov // their covariance, with Params.DDof AggMinBy // the value at the row whose second field is the least AggMaxBy // and the greatest // Step 276: the third and fourth standardized moments, with Params.Bias and // Params.Fisher; and the bitwise reductions of integers or Booleans. AggSkew AggKurtosis AggBitAnd AggBitOr AggBitXor AggNanMin // the minimum, NaN where there is one: Polars' nan_min // Step 277: entropy, with Params.Base and Params.Normalize; and Spearman's // correlation of a pair, PairOf's, as AggCorr. AggEntropy AggSpearman // Step 324: item, the group's one value, with Params.AllowEmpty; lower_bound and // upper_bound, its type's least and greatest; and is_sorted, with // Params.Descending and Params.NullsLast. AggItem AggLowerBound AggUpperBound AggIsSorted // Step 330: top_k_by and bottom_k_by, a pair, PairOf's, as AggMaxBy, with Params.K. AggTopKBy AggBottomKBy // Step 340: mode, the group's most frequent values, a List, with // Params.MaintainOrder: Polars' mode in an aggregation, where it was the list // kernel over the values imploded. AggMode )
func HorizontalAgg ¶ added in v0.6.3
HorizontalAgg is the aggregate a horizontal reduction runs over each row's values.
func (AggOp) IsBy ¶ added in v0.6.0
IsBy reports whether the aggregate is MinBy or MaxBy. They are order-dependent as ArgMin and ArgMax are: a tie goes to the earliest row, which a worker holding rows dealt round-robin cannot know.
func (AggOp) IsCounting ¶
IsCounting reports whether the aggregate returns a count, and is therefore never null and independent of the input type.
ArgMin and ArgMax return an index rather than a count and are deliberately NOT here: a group with no non-null value has no minimum, so there is no position to report and the answer is null.
func (AggOp) IsOrderDependent ¶
IsOrderDependent reports whether the aggregate's ANSWER depends on the order its input arrived in.
Every other aggregate is commutative: a sum, a count, a minimum and a quantile are the same however the rows are shuffled, which is what lets a group-by run on several workers and fold the partials together. These five are not.
First, Last name a position outright
ArgMin, ArgMax report an INDEX, and kernel.argExtremumAcc.Merge shifts the
other accumulator's positions by this one's row count — which
is only meaningful if `other` saw a strictly later contiguous
portion
Implode a list's element ORDER is its data, so a shuffled input is a
different answer rather than the same one computed differently
Mode its values come in the order they first appear, with
maintain_order; and without, each is its first row's value, so
a −0 is −0 where it came before a +0, as Polars answers
The parallel aggregation driver dispatches round-robin, so no worker holds a contiguous portion and none of those four can be honoured. It declines the whole aggregate rather than the individual expression, because one query's Agg list is one hash table.
Not the same question as IsCounting: Median and Quantile buffer every value and are expensive, but they sort in Finish and are perfectly order-insensitive.
func (AggOp) IsPaired ¶ added in v0.6.0
IsPaired reports whether the aggregate reads two inputs, as PairOf's struct.
type AggParams ¶
type AggParams struct {
DDof uint8
Q float64
Interp Interpolation
K int // TopK and BottomK
Bias bool // Skew and Kurtosis: the biased estimator, Polars' default
Fisher bool // Kurtosis: the excess, Fisher's, less 3, Polars' default
Base float64 // Entropy: the logarithm's base
Normalize bool // Entropy: of the values as a distribution, each over their sum
AllowEmpty bool // Item: a group of no value is null, not refused
Descending bool // IsSorted: in descending order
NullsLast bool // IsSorted: its nulls after its values
MaintainOrder bool // Mode: its values in the order they first appear
}
AggParams carries an aggregate's configuration — ddof for Var and Std, the quantile and its interpolation for Quantile. The zero value is correct for every aggregate that takes no parameters.
Why not Args []Node, the way Call does it ¶
Call holds its arguments as ordinary child nodes, and its doc explains that a column-valued argument "is representable in this IR and simply not implemented yet, which is the right way round for a constraint that may lift later". That is true of a regex pattern.
It is not true of these. A ddof or a quantile is CONFIGURATION, not an operand: there is no meaning to a per-row ddof, and there never will be. Making them children would put nodes into Children() that RootNames, expansion and projection pushdown then walk for nothing.
type AllMatcher ¶
type AllMatcher struct{}
AllMatcher selects every column.
func (*AllMatcher) String ¶
func (m *AllMatcher) String() string
type Binary ¶
Binary applies a two-operand operation.
The resolved cast targets are NOT stored here: Binary is immutable and shared, and caching a resolution would tie a node to one input schema. The evaluator calls ResolveBinary itself, which is cheap and keeps exactly one copy of the promotion rules in the codebase.
type BinaryOp ¶
type BinaryOp uint8
BinaryOp is a two-operand operation.
const ( // Arithmetic. OpAdd BinaryOp = iota OpSub OpMul OpDiv // true division: always produces a float OpFloorDiv // floored division: −7 // 2 is −4 OpMod // Ordering and equality. Three-valued: if either operand is null the result // is null, NOT false. OpEq OpNe OpLt OpLe OpGt OpGe // Equality where nulls are DATA rather than absence: null == null is true and // the result is never null. Polars calls these eq_missing / ne_missing. They // need a separate kernel family, not a flag on the ordinary comparison. OpEqMissing OpNeMissing // Kleene three-valued logic. Note false AND null == false and true OR null == // true — the output validity is NOT simply `va AND vb`. OpAnd OpOr OpXor // Exponentiation. APPENDED rather than placed beside the other arithmetic, // which reads oddly and is deliberate: both dispatchers — ResolveBinary and // kernel's arithmetic — route anything the classifiers below do not claim to // the arithmetic path through a `default:` arm, so position carries no meaning // for it, and renumbering the enum to make it look tidy would move every // constant for no gain. OpPow // Polars' arctan2(y, x), y the left (step 324): the angle of (x, y), a float, as // OpPow's binding casts both sides to one. OpArctan2 )
func (BinaryOp) IsComparison ¶
IsComparison reports whether the op yields Bool via three-valued comparison.
func (BinaryOp) IsMissingComparison ¶
IsMissingComparison reports whether the op treats nulls as data.
func (BinaryOp) IsOrdering ¶
IsOrdering reports whether the op uses < / > semantics, as opposed to equality. Ordering comparisons are undefined for types that have no order.
func (BinaryOp) MayProduceNull ¶ added in v0.3.0
MayProduceNull reports whether the kernel for this op, at this output type, can manufacture a null its operands did not contain.
Why this exists ¶
internal/kernel splits kernels into classes by how they relate to nulls, and a PARTIAL kernel "may itself produce nulls (integer division by zero, a narrowing cast, sqrt of a negative)". Binary.Field propagated only — `lf.Nullable || rf.Nullable` — which is conservative about PROPAGATION and says nothing about MANUFACTURE, the axis the kernel layer actually splits on. So `a % b` over two non-nullable integer columns was declared non-nullable and held a null wherever b was zero.
It takes the OUTPUT type, and that is not a detail ¶
kernel.arithmetic dispatches on `Binding.Out.Physical()`, so the output type is what decides which kernel runs, and the operands' logical types can be misleading in both directions:
Int64 % Int64 Out Int64 -> arithNum -> PARTIAL
Float64 % Float64 Out Float64 -> arithFloat -> total (math.Mod, NaN)
Int64 / Int64 Out Float64 -> arithFloat -> total (true division)
Duration // Int64 Out Duration -> arithNum -> PARTIAL, though the left
operand is not an integer type at all
A predicate written against the operands would get the last two backwards.
Explicit membership, not a range ¶
OpDiv sits immediately above OpFloorDiv, so `o >= OpFloorDiv && o <= OpMod` would be one constant away from wrong — and this file already carries the post-mortem of that exact mistake under "There is deliberately no IsArithmetic".
type Binding ¶
type Binding struct {
CastL dtype.DataType // cast the left operand to this before the kernel
CastR dtype.DataType
Out dtype.DataType // the kernel's output type
}
Binding is the resolved form of a binary operation: the type each operand must be cast to, and the type the result will have.
The evaluator MUST use this rather than re-deriving types from the operands. Two independent copies of the promotion rules would drift, and the drift would show up as a mismatch between the schema CollectSchema promised and the column Eval actually produced — which is precisely the invariant the evaluator contract test checks.
func ResolveBinary ¶
ResolveBinary computes the operand and result types of op applied to l and r.
func ResolveBinaryNode ¶ added in v0.6.6
ResolveBinaryNode is ResolveBinary for the node b, whose operands are of types l and r: an integer's bits beside a weak literal that fits its type exactly are of that type, as Polars keeps Col("u8") & 6 a UInt8 (step 278). A bitwise result is never wider than its operands, so the narrowing never overflows, where arithmetic's would. Binary.Field and the evaluator both ask it, so the two agree.
func (Binding) NeedsCastL ¶
NeedsCastL reports whether the left operand requires a cast.
type Call ¶
Call applies a namespaced function.
Args[0] is the RECEIVER — the column the method was called on. Args[1:] are the function's parameters and must resolve to literals; a column-valued pattern is representable in this IR and simply not implemented yet, which is the right way round for a constraint that may lift later.
Args are ordinary children, so RootNames, expansion and projection pushdown all see through a Call with no special case. That is the property the design docs predicted for window nodes and it holds here for the same reason.
type CallFn ¶
type CallFn uint16
CallFn names a namespaced function — the `.str` and `.dt` families.
Why these are not UnaryOps ¶
`Contains(pattern, literal)` and `Truncate(every)` carry ARGUMENTS, and Unary{Op, Child} has nowhere to put them. Three shapes were possible: one op per (function, argument) pair, which does not terminate; arguments smuggled into the op enum, which is the same thing wearing a hat; or one node that holds a function and its operands.
This is the third. It is also the shape every remaining namespace will need — `.list`, `.struct`, `.arr` — so paying for it once here is cheaper than three more bespoke node types later.
Appending only ¶
Like UnaryOp, the classifiers below are range comparisons over declaration order. Insert a constant mid-enum and a neighbouring function silently changes family. Append, and add a name.
const ( // --- string --- FnStrContains CallFn = iota FnStrStartsWith FnStrEndsWith FnStrFind FnStrCountMatches FnStrExtract FnStrReplace FnStrReplaceAll FnStrToLower FnStrToUpper FnStrLenBytes FnStrLenChars FnStrSlice FnStrStripChars FnStrStripPrefix FnStrStripSuffix FnStrReverse FnStrPadStart FnStrPadEnd FnStrZFill FnStrStripCharsStart FnStrStripCharsEnd FnStrEscapeRegex FnStrSplit FnStrSplitN FnStrExtractAll // The three parse a string with a strftime-style format (step 113): args are // the format, strict, and for a Datetime the unit and zone, for a Time the // unit. One function per target, because a type is not a literal. FnStrToDateFmt FnStrToDatetimeFmt FnStrToTimeFmt // --- temporal --- FnDtYear CallFn = iota + 100 FnDtMonth FnDtDay FnDtHour FnDtMinute FnDtSecond FnDtMillisecond FnDtMicrosecond FnDtNanosecond FnDtWeekday FnDtOrdinalDay FnDtQuarter FnDtWeek FnDtEpoch FnDtTruncate FnDtTotalDays FnDtTotalHours FnDtTotalMinutes FnDtTotalSeconds FnDtStrftime // formats with a strftime-style format; the one arg is the format // The calendar (step 145). offset_by's args are an interval's months, days and // nanos; round's are truncate's; convert_time_zone's one arg is the zone. FnDtOffsetBy FnDtRound FnDtMonthStart FnDtMonthEnd FnDtIsLeapYear FnDtConvertTimeZone // Step 174: replace_time_zone's args are the zone, "" for none, then the // ambiguous and non_existent policies, in Polars' words (ZonePolicies). FnDtReplaceTimeZone FnDtDaysInMonth // the month's length, read in the column's zone (step 230) // The local date and time of day of an instant, Polars' dt.date and dt.time // (step 249). FnDtDate FnDtTime // --- type-agnostic --- // // A third family, and the first that is not keyed to a receiver type. The two // above dispatch on what the receiver IS — a String, an instant — but IsIn // applies to anything hashable, so it needs its own classifier rather than // widening either of theirs. FnIsIn CallFn = iota + 200 // A value map (step 226). Args[0] is the receiver, Args[1] the count k of old // values as an Int64 literal, then the k old values, the k new ones, and for // FnReplaceOr the default. ReplaceParts reads them. FnReplace // a value not among old keeps itself FnReplaceStrict // and here is refused FnReplaceOr // and here, a null too, is the default // --- maths --- // // A fourth family, for the maths operations that carry a PARAMETER. The // parameterless ones are UnaryOps; Round needs a decimal count, and a Call is // where a parameter belongs. // // Not a field on expr.Unary, which was the obvious alternative: three places // rebuild a Unary by hand — physical/agg.go's extractAggs and both predicate // pushdown walkers — as `&Unary{Op: t.Op, Child: ...}`, so a new field would be // silently DROPPED the moment a predicate moved, and Round(2) would quietly // become Round(0). That is step 7's Agg.Params bug, three sites over. Call.String() // renders its arguments structurally, so the same class is impossible here. FnMathRound CallFn = iota + 300 // FnMathAsFloat takes a number, or a Duration's ticks, to a float without // changing its value: an integer or a Decimal to the nearest Float64, a float as // itself. It is how PctChange subtracts in float, and is not public. FnMathAsFloat // FnMathDiffWiden takes an unsigned integer to the signed type that holds the // difference of any two of its values, and leaves every other type as it is. It // is how Diff subtracts without wrapping, and is not public. FnMathDiffWiden // The bit counts of an integer (step 146), Polars' bitwise_count_ones and its // five siblings, each a Uint32. FnMathCountOnes FnMathCountZeros FnMathLeadingOnes FnMathLeadingZeros FnMathTrailingOnes FnMathTrailingZeros // --- list --- // // A fifth family, keyed to a List receiver. These all reduce a list to a // SCALAR: nothing here builds a list, which is the seam the step was cut on — // sort, unique, reverse, slice and the set operations construct a new List // column and are a different problem. FnListLen CallFn = iota + 400 FnListGet FnListContains FnListMin FnListMax FnListSum FnListMean FnListJoin // the elements of a List of String, joined; the one arg is the separator // list -> LIST. These reshape the list and hand back a list, which is the // distinction that split the namespace across two steps: everything above // answers a question ABOUT a list, everything here returns one. FnListReverse FnListHead FnListTail FnListSlice FnListSort FnListUnique FnListDropNulls // list -> a SCALAR again, each a column aggregate over a list's elements (step // 222), after the reshaping half because that is where the family ends. FnListNUnique FnListArgMin FnListArgMax // list -> STRUCT (step 224): element i as field i, its names the arguments. FnListToStruct )
const ( FnStructField CallFn = iota + 500 FnStructRenameFields // every field renamed, in order (step 224) FnStructJSONEncode // each struct as a JSON object (step 231) FnStructSelectFields // the named fields, in that order, the struct's validity kept (step 299) FnStructDropFields // the fields not named, Polars' struct.drop (step 323) )
The `.struct` family.
const ( FnConcatStr CallFn = iota + 600 FnStructOf FnCut FnCutLeftClosed FnStructWithFields // The list set operations and gather (step 223): their second List is an // operand, a column, not a parameter, so they take this family's shape. The // answer is named after the first. FnListSetUnion FnListSetIntersection FnListSetDifference FnListSetSymmetricDifference FnListGather // an index out of range is refused FnListGatherOrNull // and here gives null // The horizontal reductions (step 225): one value a row from every operand, // max, min, sum and mean each the column aggregate over the row's values, and // any and all Kleene's. FnMaxHorizontal FnMinHorizontal FnSumHorizontal FnMeanHorizontal FnAnyHorizontal FnAllHorizontal // A Date's or Datetime's day with a Time of day (step 230): its second operand // is a column, so it takes this family's shape. FnDtCombine // A list sorted by another of its length, element for element (step 253), // ascending or descending, nulls first: its second list is an operand. FnListSortBy FnListSortByDesc // Membership and indexing whose argument is a column (step 254): is_in against // each row's own list, and list.get and list.contains with a per-row index and // value. The constant forms are their families' own; these take this one's // shape, every operand a column. FnIsInExpr FnListGetExpr FnListContainsExpr // String functions whose pattern is a column, and replace's whose value is (step // 255). Each takes the constant form's arguments after the receiver, any of the // pattern or value a column; the flags stay literals. FnStrContainsExpr // the receiver, the pattern, literal FnStrStartsWithExpr // the receiver, the prefix FnStrEndsWithExpr FnStrStripPrefixExpr FnStrStripSuffixExpr FnStrSplitExpr FnStrReplaceExpr // the receiver, the pattern (a literal), the value, literal FnStrReplaceAllExpr // dt.offset_by with an offset a row, a duration string as Every reads it (step // 256). FnDtOffsetByExpr // Generators, a list a row (step 263): int_ranges(start, end, step), // linear_spaces(start, end, n, closed) with closed a literal Closed, and // value.repeat_by(n). The scalar forms are these over one row, exploded. FnIntRanges FnLinearSpaces FnRepeatBy // Temporal ranges, a list a row (step 264): start, end, the interval's months, // days and nanoseconds, and closed, the last four literals. FnDateRanges FnDatetimeRanges FnTimeRanges // Values from their components (step 265): date(y, m, d), datetime(y, m, d, h, // mi, s, us), time(h, mi, s, us); a duration's and an epoch count's ticks, Int64, // which the root package casts; and business_day_count(start, end, mask, // holidays), the mask an Int64 literal and the holidays a List(Int64) literal of // days. FnMakeDate FnMakeDatetime FnMakeTime FnDurationTicks FnEpochTicks FnBusinessDayCount // concat_list(exprs...): each row's operands in one list, a List operand's // elements and another's value (step 266). FnConcatList // list.shift(n) and list.count_matches(v), n and v a value a row (step 269). FnListShift FnListCountMatches // to_integer in a base a row, signed or unsigned, strict a literal; and // concat_str past nulls, the separator last (step 271). FnStrParseInt FnStrParseUint FnConcatStrIgnoreNulls // Step 272: is_business_day(value, mask, holidays), the calendar as // business_day_count's; add_business_days(value, n, mask, holidays, roll); and // replace(value, year, month, day, hour, minute, second, microsecond, ambiguous, // non_existent), a component a null literal to keep the value's own. FnDtIsBusinessDay FnDtAddBusinessDays FnDtReplace // Step 274's binary functions of a row's operands: contains, starts_with and // ends_with(value, pattern); get(value, index, null_on_oob); and slice(value, a, // b, mode), mode "slice" with a the offset and b the length, a null for the rest, // or "head" or "tail" with a their n and b a null. FnBinContains FnBinStartsWith FnBinEndsWith FnBinGet FnBinSlice // order_cmp(a, b): -1, 0 or 1 as a sorts before, with or after b in the total // order Sort uses, NaN above every number and -0 with 0 (step 279), which // search_sorted, index_of and the peaks are written in. FnOrderCmp // list.append(a, b, mode): each row's two lists one after the other, their // elements met as Append's mode says (step 282), which Append in an aggregation // lowers to. FnListAppend // hash_rows(operands..., seed): each row's hash over its operands, the row's // group-by key hashed as hash hashes one value's, the seed an Int64 literal last // (step 302). FnHashRows // arr.shift(n) and arr.dot(other), n and other a value a row (step 342): an // Array's list.shift, the Array kept, and the sum of two Arrays' products. FnArrShift FnArrDot // concat_arr(exprs...): each row's operands in one Array, an Array operand's // elements and another's value, of one element type (step 343). FnConcatArr )
The horizontal family (step 146): every argument is an OPERAND, a column read row by row, where each family above takes a receiver and literal parameters.
concat_str joins its operands' text, and a separator is an operand like any other, a literal between each two; struct builds a Struct whose fields are its operands, named as they are. Neither has a parameter, so Call's rule — Args[1:] literal — does not apply to them, and they are typed from every operand's type (ResolveHorizontal) rather than from a receiver's.
cut (step 147) puts its first operand in a bin: the operands after it are the breaks, numbers, then none or one label more than breaks, strings. Its breaks are operands because QCut's are quantiles of the column, known only when it runs. cut_left_closed is the same with each bin closed on the left.
const ( FnListMedian CallFn = iota + 700 FnListStd FnListVar FnListQuantile FnListDropNans FnListGatherEvery // A List of Lists flattened a level (step 259), which Explode in an aggregation // lowers to: an empty or null inner list one null, as explode makes it a row. FnListExplode // Each list's indices (step 267): in its elements' sorted order, descending and // nulls last two literal Booleans; of its true elements; of each value's first. FnListArgSort FnListArgTrue FnListArgUnique // list.any and list.all, ignore_nulls a literal Boolean; list.item, allow_empty // one; list.sample, n (negative for a fraction), the fraction, with_replacement, // shuffle and the seed (step 269). FnListAny FnListAll FnListItem FnListSample // list.mode, maintain_order a literal Boolean (step 277), which Mode in an // aggregation lowers to. FnListMode // list.unique_counts; list.value_counts, sort, normalize, the count's name and the // value's (step 280). FnListUniqueCounts FnListValueCounts FnListRle // list.rle, each list's runs, a Struct each (step 281) // list.hist, the bin count or -1, include_category, include_breakpoint and the // edges (step 286), which Hist in an aggregation lowers to. FnListHist FnListToArray // list.to_array, the width: an Array of it (step 341) FnListReshape // list.reshape, the shape: each list's values in Arrays of it, as a list (step 343) )
The list family, continued (step 253): the aggregates a list did not have, which a reshape's aggregate lowers to, and the reshapes of E3's second half. std and var take ddof, quantile q and an interpolation; gather_every n and an offset.
A block of its own: the first families share one const block, whose iota runs on, so the list family's first block already ends at the struct family's base, and six more there collided with FnStructField. IsList covers both blocks.
const ( FnStrContainsAny CallFn = iota + 800 FnStrReplaceMany FnStrExtractMany FnStrFindMany FnStrExtractGroups FnStrSplitExact FnStrSplitInclusive FnStrToTitlecase FnStrNormalize // Step 271: encode and decode, hex or base64; json_path_match, the path; // json_decode, a null literal of the type it decodes to. FnStrEncode FnStrDecode FnStrJsonPathMatch FnStrJsonDecode )
The string family, continued (step 270): many literal patterns at once (contains_any, replace_many, extract_many, find_many), a List literal of them first; extract_groups and split_exact, a Struct; split keeping its separator; to_titlecase; normalize, the form's name.
A block of its own, as the list family's at 700 is: nine more in the first block pushed the list family's end onto the struct family's base, 500, and the build caught the duplicate keys. IsString covers both blocks.
const ( FnDtIsoYear CallFn = iota + 900 FnDtCentury FnDtMillennium FnDtBaseUtcOffset FnDtDstOffset FnDtWithTimeUnit FnDtTotalMilliseconds FnDtTotalMicroseconds FnDtTotalNanoseconds FnDtTotalFractional FnDtSecondFractional FnDtEpochIn FnDtCastTimeUnit // a Datetime's or a Duration's values in another unit, Polars' cast_time_unit (step 325) )
The temporal family, continued (step 272): iso_year, century and millennium; a zone's base and daylight-saving offsets; with_time_unit, the TimeUnit's code; the totals in the smaller units, and fractional in any, the unit's nanoseconds; second with its fraction; and epoch in a unit, its nanoseconds, a day's being days.
A block of its own, as the string family's at 800 is. IsTemporal covers both.
The categorical family (step 273): get_categories, which Expand writes out as its Enum's categories once the receiver's type is known, so no kernel sees it; and physical, an Enum's indices at Polars' width. The text functions of .cat are the string family's, which read an Enum as its text.
A block of its own, as the others after 600 are. IsGeneral covers it.
const ( FnHash CallFn = iota + 1200 FnOrderSign FnToPhysical // the values as stored, Polars' to_physical (step 324) FnReinterpret // an integer's bits at the other signedness; Args[1] signed )
The binary family (step 274): encode and decode, the encoding's name and decode's strict; size, the unit's bytes, 1 for bytes; and reinterpret, a null literal of the type and the endianness. Its functions of a row's operands are horizontal. The general family, continued (step 279): hash, the seed; and order_sign, a value's sign in the total order, beside its type's zero. IsGeneral covers it.
The maths family, continued (step 283): round_sig_figs, the digits; truncate, the decimals, round's kernel toward zero. round's mode is its own second argument. IsMath covers it.
The array family (step 341): to_list, an Array's rows as a List's, which holds them as an Array does. A block of its own at 1400. Step 342's reverse and sort, a List's kept an Array, sort with list.sort's arguments; and to_struct, the fields' names, field_0 on unless given.
func (CallFn) IsHorizontal ¶ added in v0.5.0
func (CallFn) IsString ¶
IsString, IsTemporal and IsGeneral classify a function by family.
These are range comparisons over declaration order, which is why the enum is append-only within each block and the blocks are spaced 100 apart.
func (CallFn) IsTemporal ¶
type Cast ¶
Cast converts to another type.
Strict casts fail the query on the first unrepresentable value. Non-strict casts turn it into a null — which is why a non-strict cast is ALWAYS nullable regardless of its input, even when casting a non-nullable column.
type Closed ¶
type Closed uint8
Closed says which endpoints of an interval belong to it.
It lives here, at the level of the expression IR, for the same reason Interpolation does: the plan node, the physical operator and the root package all need it and none of them may import another's caller.
Why four values rather than a pair of bools ¶
Two bools would admit the same four states and read worse at every call site — `Closed: ClosedBoth` against `LeftClosed: true, RightClosed: true`.
And why the zero value is none of them ¶
The three callers want three different defaults, and every one of them is the obviously right answer for its own question:
IsBetween both — "between lo and hi" includes the endpoints
GroupByDynamic left — the only convention under which a grid TILES
Rolling right — a row's window ends at its own timestamp, so closing
the left end would drop the row from its own window
A zero value that meant one of them would silently give the wrong default to the other two. So the zero means "ask the operator", each caller resolves it with Or, and no call site has to know which default it inherited.
const ( // ClosedDefault defers to whatever the operation's own convention is. It is the // zero value precisely so that leaving the field alone cannot mean the wrong // thing somewhere else. ClosedDefault Closed = iota // ClosedLeft is [start, end) — the only value under which a grid of windows // partitions its input exactly. ClosedLeft // ClosedRight is (start, end]. ClosedRight // ClosedBoth is [start, end]. A row landing exactly on a shared boundary is in // BOTH neighbouring windows, so the windows no longer partition and a count over // them exceeds the row count. That is the documented meaning, not an accident. ClosedBoth // ClosedNone is (start, end). The mirror: a row on a shared boundary is in // neither window and is dropped from the output entirely. ClosedNone )
func (Closed) LowerOpen ¶
LowerOpen and UpperClosed are the two questions every range test asks, phrased so the caller never re-derives the four cases. Both treat ClosedDefault as ClosedLeft, which is only reachable if a caller forgot to call Or.
contains(x) == (LowerOpen() ? x > lo : x >= lo) && (UpperClosed() ? x <= hi : x < hi)
func (Closed) Or ¶
Or resolves ClosedDefault to the caller's own convention and leaves anything else alone. Every consumer calls this exactly once, at the edge, so the rest of the code never has to wonder whether it is holding a resolved value.
func (Closed) UpperClosed ¶
type Cond ¶
type Cond struct {
Pred Node
Then Node
Else Node
// Sugar names the method a conditional was desugared from — fill_null,
// fill_nan — so its refusal says what the caller wrote rather than what it
// became (audit.md S16: "is_not_nan() requires a floating-point operand", to
// someone who had called FillNan). "" for a When the caller built. A rewrite
// that rebuilds the node may drop it: types are resolved before any rule runs,
// and that is where these refusals arise.
Sugar string
}
Cond is a conditional: the value of Then where Pred is true, of Else otherwise.
Why this is a node and IsBetween is not ¶
IsBetween is two lines of sugar — `e.Ge(lo).And(e.Le(hi))` — and earns Field, evaluation, expansion, pushdown and Parquet row-group pruning for free by being built out of nodes that already exist. That is the cheapest way to add anything, and it was the first thing tried here.
It does not work for a conditional, for reasons that are properties of this codebase rather than of conditionals in general:
- There was no masked-select kernel. Sugar has to desugar INTO something, and nothing in kernel took (mask, a, b). kernel.Select was written for this node.
- The arithmetic dodge does not type-check. `cond*a + !cond*b` needs Bool to be numeric, and dtype.Promote refuses on purpose: "true + true is rejected".
- There is no CSE pass, so any desugaring that names an operand twice evaluates it twice.
Three children, not parallel slices ¶
A chain — when/then/when/then/otherwise — is built by the root package into a right-nested tree of these, rather than represented as []Pred and []Then. The reason is type unification: dtype.Promote is strictly binary, there is no n-ary helper, and a left fold is NOT safe to invent because Promote's mixed-sign rule (the smallest signed type holding both) makes the result depend on association order for operand sets like (Uint8, Int8, Uint32). Nesting makes the association explicit and identical to the order the user wrote.
All three are ordinary children, so RootNames, Walk, expansion and projection pushdown see through a Cond with no special case.
type DTypeMatcher ¶
DTypeMatcher selects columns by type. The number of columns it expands to is determined at plan time from the schema, so the same expression adapts as the schema changes.
func (*DTypeMatcher) String ¶
func (m *DTypeMatcher) String() string
type Element ¶ added in v0.6.6
type Element struct{}
Element is a list's elements, in List().Eval's body: Polars' pl.element().
type Err ¶
type Err struct{ E error }
Err carries a construction-time error to the point where an error can be returned. Expression builders cannot return errors without destroying method chaining, so `ColRegex("(")` produces an *Err instead of panicking.
type ExcludeMatcher ¶
type ExcludeMatcher struct {
Inner Matcher
Names []string
Re *regexp.Regexp
ReSrc string
DTypes []dtype.DataType
}
ExcludeMatcher subtracts columns from an inner matcher's result.
func (*ExcludeMatcher) String ¶
func (m *ExcludeMatcher) String() string
type FieldMatcher ¶
FieldMatcher selects columns whose field satisfies a predicate.
It is groundwork for the selector package's dtype families (numeric, temporal, string, ...), which want a predicate rather than an enumeration of concrete types. NOTHING CONSTRUCTS ONE TODAY — that package does not exist yet, and this doc used to claim in the present tense that it did.
func (*FieldMatcher) String ¶
func (m *FieldMatcher) String() string
type Filter ¶ added in v0.6.5
Filter is a column's values where a predicate holds, Polars' Expr.filter (v0.6.5 step 251, E2 of v0.6.5-scope.md).
Its length is not the frame's, so it stands beneath an aggregate: a conditional aggregate, SQL's FILTER (WHERE ...). col("a").filter(col("b") > 0).sum() sums the a's whose b is positive, in a group-by, a window, a temporal group, and as a scalar in Select. The aggregate's sink feeds its accumulator only the rows the predicate is true on, so len, first and last count and pick among them. In a Select it is a column of its own length, the root package's plan of its own (step 258); anywhere else it is refused (LooseFilter).
func LooseFilter ¶ added in v0.6.5
LooseFilter is the first Filter in n that is not an aggregate's input, or nil. An aggregate's input may be a chain of filters, whose predicates are themselves checked.
type Fold ¶ added in v0.6.6
type Fold struct {
Op string // fold, reduce, cum_fold, cum_reduce or cum_sum
Acc Node // nil for a reduce
Exprs []Node
Fn func(acc, x Node) Node
// Promote casts the running values to the last one's type, as Polars' fields of
// cum_sum_horizontal are of one type.
Promote bool
}
Fold is Polars' fold, reduce, cum_fold, cum_reduce and cum_sum_horizontal (step 266, F1 of v0.6.6-scope.md): Fn applied from an accumulator across Exprs. Polars runs its function over Series per batch; here Fn builds the expression once, when the plan is resolved, so the optimizer sees an ordinary expression and no UDF runs.
Expand writes it out after the input's schema has expanded its Exprs, so a fold over All() folds every column, as Polars' does:
- fold and reduce are the last accumulator, named after the first one: Acc's name, or a reduce's, whose accumulator is its first expression;
- the cumulative forms are a Struct of the running values, a field each named after the expression folded in, the Struct named Op.
type IndexMatcher ¶ added in v0.6.6
type IndexMatcher struct{ Indices []int }
IndexMatcher selects columns by position, in the order written: Polars' nth (step 267). A negative index counts from the end, and one past either end is an error, as Polars refuses it.
func (*IndexMatcher) Match ¶ added in v0.6.6
func (m *IndexMatcher) Match(in *dtype.Schema) ([]string, error)
func (*IndexMatcher) String ¶ added in v0.6.6
func (m *IndexMatcher) String() string
type Interpolation ¶
type Interpolation uint8
Interpolation selects how Quantile resolves a rank that falls between two values.
It lives here, at the level of the expression IR, rather than in the root package, because both Agg's type resolution and the kernel need it and neither may import the other's caller. The root package re-exports it as an alias, the same way it does for JoinKind.
const ( // InterpLinear interpolates proportionally between the two neighbours. The // default, and the only option whose result need not be an input value. InterpLinear Interpolation = iota // InterpLower and InterpHigher take the neighbour below or above. InterpLower InterpHigher // InterpNearest takes the closer neighbour, rounding half away from the lower // one. InterpNearest // InterpMidpoint averages the two neighbours regardless of where the rank fell // between them. InterpMidpoint )
func (Interpolation) String ¶
func (i Interpolation) String() string
func (Interpolation) Valid ¶ added in v0.3.0
func (i Interpolation) Valid() bool
Valid reports whether i is one of the declared interpolations. Interpolation is a public integer type, so Interpolation(99) compiles; Agg.Field refuses it. It was silently linear before step 72.
type JSONPathStep ¶ added in v0.6.6
JSONPathStep is one step of a JSONPath: a key, an index, or a key looked for at any depth.
func ParseJSONPath ¶ added in v0.6.6
func ParseJSONPath(p string) ([]JSONPathStep, error)
ParseJSONPath reads the subset of JSONPath Polars' json_path_match is used with: $, then .key, ['key'], [n] (from the end where negative) and ..key, which finds the first key of that name at any depth.
type ListMap ¶ added in v0.6.6
ListMap is Body evaluated over List's elements: a List of the answers with the list's offsets and nulls. Its children are List and Outer, the values a list its body reads; the body is its own, so no pass over the frame's expressions reaches it, and a window function in it is the list's, not the frame's.
type Lit ¶
type Lit struct {
Value any
DT dtype.DataType
// Weak marks a literal whose type is a DEFAULT rather than a choice.
//
// ursus.Lit(0) infers Go int and fixes DT = Int64 at BUILD time. For arithmetic
// that is right — Promote(Int32, Int64) is Int64 because addition can overflow.
// For a null fill it is wrong, and worse, INVISIBLE: Col("i32").FillNull(FillZero)
// contains no literal in the source at all, yet the same promotion would widen
// the column to Int64 with nothing in the query to explain it.
//
// A weak literal asks the CONTEXT to adopt the other operand's type when the
// value fits it exactly. Only ResolveCond consults it; Field still reports DT,
// because a weak literal standing alone has no other operand to defer to.
//
// The zero value is STRONG, which is what keeps this change small: every
// existing construction site keeps today's meaning untouched, and arithmetic
// stays weak-free by construction because nothing on that path sets it.
Weak bool
}
Lit is a scalar constant, broadcast to the length of the context.
A typed null is Value == nil with DT set; an untyped null is Value == nil with DT == Null, which Promote lets adopt the other operand's type.
func (*Lit) String ¶
String renders the literal.
A weak literal MUST NOT render like a strong one ¶
Three maps key on a rendered String() — window temporaries, aggregate temporaries and partition-key sharing — so two expressions that render alike become ONE computation and both names get the first one's answer.
Weak versus strong is a live instance of that: Coalesce(col_i32, Lit(0)) resolves to Int64 and col_i32.FillNull(FillZero) resolves to Int32, and without the marker below they render identically. Put both under .Max() in one Agg and the aggregate dedup collapses them, after which the batch's column type and the plan's promised schema disagree.
The TYPE is in there for a second collision. Weak literals normally converge on the column's type and so agree — but when the value does not fit they fall back to ordinary promotion, and a weak int64 5000 and a weak float64 5000 fall back to Int64 and Float64 respectively while printing the same digits.
type MappingStrategy ¶
type MappingStrategy uint8
MappingStrategy decides how a window's per-partition result is placed back into the frame.
const ( // MapGroupsToRows returns each row's own partition value, in the original row // order. The default, and the only one that composes with other columns. MapGroupsToRows MappingStrategy = iota // MapExplode leaves the output in partition-sorted order instead of permuting // it back. Cheaper — it skips the inverse permutation — and it REORDERS the // frame, so it is only safe when every other column is windowed the same way. MapExplode // MapJoin aggregates each partition into a List repeated across its rows. // Refused, and from step 46 the refusal names the thing that does it: // Col("v").Implode().Over(k) IS this, under the default mapping. MapJoin )
func (MappingStrategy) String ¶
func (m MappingStrategy) String() string
type Match ¶
type Match struct{ M Matcher }
Match is a multi-column selection: Col("a","b"), ColRegex, ColDType, All, Exclude. It is a placeholder that Expand replaces with one concrete tree per matched column.
It exists as a node rather than as a separate expression kind so that `Col(pl.Float64).Mul(1.1).Alias(...)` is an ordinary tree that expansion rewrites, rather than a special case threaded through every builder.
type Matcher ¶
type Matcher interface {
// Match returns the selected column names, in schema order.
Match(in *dtype.Schema) ([]string, error)
// String renders the matcher for Explain and error messages.
String() string
}
Matcher selects zero or more columns from a schema.
Matchers are what make one written expression apply to many columns:
Col("weight", "height").Mean() → two independent means
ColDType(Float64).Mul(1.1) → every float column, count known only at plan time
All().Exclude("id") → everything but id
Matching is by NAME and resolved against a schema, never against data.
type NameMatcher ¶
type NameMatcher struct{ Names []string }
NameMatcher selects an explicit list of columns, in the order written.
Unlike the other matchers, a name that is not present is an ERROR rather than an empty match: the user named a specific column, so its absence is a typo.
func (*NameMatcher) String ¶
func (m *NameMatcher) String() string
type Node ¶
type Node interface {
// Field resolves this node's single output field against an input schema.
// It never reads data.
//
// Defined only for RESOLVED trees: a tree containing a *Match must be run
// through Expand first, and returns ErrUnexpanded otherwise.
Field(in *dtype.Schema) (dtype.Field, error)
// Children returns the direct sub-expressions, in evaluation order.
//
// A nested SCOPE body is deliberately not a child. When List().Eval() arrives,
// its body is evaluated against a different schema (the list element type),
// so a generic walker must not rewrite across it or it would resolve
// Element() against the outer frame.
Children() []Node
// String renders the node canonically. Load-bearing: it appears in Explain
// output, in golden test files, and in error messages.
String() string
// contains filtered or unexported methods
}
Node is one node of the expression IR.
A Node is IMMUTABLE. Rewrites allocate new nodes; nothing ever mutates a node or a slice reachable from one. That is what makes it safe for two LazyFrames to share a subtree, which in turn is what makes `base.Filter(x)` and `base.Filter(y)` independent.
func BindScalars ¶ added in v0.6.5
BindScalars wraps each largest scalar subexpression of n in a window with no partition, so that a row context takes it as Polars does. A name at the root, an alias or a prefix, stays at the root: naming reads through a window either way, but Explain then shows the window where a user would write it.
func BuildCategories ¶ added in v0.6.6
BuildCategories writes each get_categories in n out. Where n is one, the explode is named as well, as a selection names its expressions by their outermost alias; below an aggregate an alias there would hide the explode from its lowering.
func Expand ¶
Expand rewrites one written expression into the concrete expressions it denotes.
Col("a") → [col("a")] (unchanged)
Col("a","b").Mul(2) → [(col("a") * 2), (col("b") * 2)]
ColDType(Float64).Round(2) → one node per Float64 column
All().Exclude("id") → one node per remaining column
Where this runs, and why there ¶
Expansion is a PLAN-TIME, SCHEMA-DRIVEN rewrite. It runs in the resolve phase: after a node's input schema is known, before type checking, and before any optimizer rule sees the plan.
It cannot run earlier — `ColDType(Float64)` has no meaning without a schema. It must not run later — projection pushdown asks each expression which columns it reads, and an unexpanded matcher would answer "none", so the optimizer would happily prune away the very columns the expression is about to select. Running before type checking also means the type checker only ever sees single-output trees, which is what lets Field return one Field instead of a slice.
One matcher per expression ¶
A tree may contain at most one DISTINCT matcher. The same matcher repeated is fine and expands in lockstep — `All().Sub(All().Mean())` gives each column minus its own mean. Two different matchers are rejected, because the intended pairing is genuinely ambiguous: is it a zip, or a cross product?
func HistCall ¶ added in v0.6.6
HistCall is list.hist of list by a ReshapeHist's arguments: the bin count, the two includes, and the edges as floats.
func LowerCumulativeEval ¶ added in v0.6.6
LowerCumulativeEval is cumulative_eval's body (step 286), lowered as List().Eval lowers an aggregate, to one value a list of CumulativeName, a row's prefix. A body that is not one value a prefix is refused, as Polars refuses it, and so is one reading another column.
func LowerListEval ¶ added in v0.6.6
LowerListEval is Polars' list.eval(body) of list, or with agg list.agg(body): an aggregate is then the value itself, where eval makes it a list of one.
func LowerReshapes ¶ added in v0.6.5
LowerReshapes rewrites n's reshapes into implodes and list kernels: an aggregate over a reshape anywhere, and in an aggregation (agg) every reshape, which is a list a group there. What it cannot lower is left as an error node naming why.
func ModeAggs ¶ added in v0.6.7
ModeAggs rewrites each mode of a group's values, the list kernel over the values imploded as LowerReshapes lowers it, into the aggregate that counts them, AggMode (step 340): its state is an entry a distinct value, which spills past a limit, where the implode's was every value. A mode over another reshape keeps the kernel, which reads that reshape's list; so does one in a window, whose partitions implode as they did.
func PairArgs ¶ added in v0.6.4
PairArgs is PairOf's two inputs as written, or false if child is not its shape.
func PairOf ¶ added in v0.6.0
PairOf is the child of a paired aggregate: a two-field Struct of a and b, each aliased to its position. The aggregate is still named after a's column, since naming reads an Alias only at the root (OutputName).
func Rebuild ¶
Rebuild returns a copy of n with its children replaced, in the order Children returns them.
Why this exists ¶
Rewriting an expression tree means copying every node with new children, and before this there were two hand-written copies of that walk — substitute, and the physical planner's extractAggs — with a third about to be added for windows. Each copy enumerates the node types and carries their non-child fields across by hand, which is exactly the shape of the bug step 7 hit twice: an Agg rebuilt without its Params silently became a different aggregate, and a Window rebuilt without its Mapping would silently become a different window.
Copying the struct and replacing only the child fields makes that class impossible: a new field on any node is carried by default, and the only way to get it wrong is to add a child field without updating Children, which the length check catches.
It PANICS on an unknown node type, for the same reason substitute did: Node is sealed, so reaching the default means a node type was added without teaching the rewriter about it, and silently dropping its children is worse than a crash.
type NotMatcher ¶ added in v0.6.7
type NotMatcher struct{ Inner Matcher }
NotMatcher is every column its inner matcher does not select.
func (*NotMatcher) Match ¶ added in v0.6.7
func (m *NotMatcher) Match(in *dtype.Schema) ([]string, error)
func (*NotMatcher) String ¶ added in v0.6.7
func (m *NotMatcher) String() string
type OrderKey ¶
type OrderKey struct {
Expr Node
Descending bool
// NullsLast places nulls at the END OF THE OUTPUT, independent of direction.
//
// The zero value is therefore NULLS FIRST, which is ursus's default and matches
// Polars (nulls_last=False). Per-key `.NullsLast()` overrides it.
//
// Tying null placement to direction — as SQL does, where nulls sort as the
// largest value and so land last ascending and first descending — was
// considered and rejected: it means `.Desc()` silently relocates the nulls,
// which surprises people every time.
NullsLast bool
}
OrderKey is one component of an ordering: an expression and how to sort by it.
It lives here rather than in the plan package because a window carries its own ordering and internal/expr may not import internal/plan. plan.SortKey is an alias for this type, so there is one definition rather than two that must agree.
type OuterRef ¶ added in v0.6.6
OuterRef is a ListMap body's I-th value a list, computed over the frame and read by each of the list's elements.
type RankMethod ¶
type RankMethod uint8
RankMethod decides how Rank breaks ties.
const ( // RankOrdinal gives every row a distinct rank, ties broken by input order. RankOrdinal RankMethod = iota // RankDense gives tied rows the same rank and does not skip the next value. RankDense // RankMin and RankMax give tied rows the lowest or highest of their positions. RankMin RankMax // RankAverage gives tied rows the mean of their positions, so it returns a // float where every other method returns an integer. RankAverage )
func (RankMethod) String ¶
func (m RankMethod) String() string
func (RankMethod) Valid ¶ added in v0.3.0
func (m RankMethod) Valid() bool
Valid reports whether m is one of the declared methods. RankMethod is a public integer type, so RankMethod(99) compiles; ResolveWinFn refuses it, and the kernel never sees one.
type RegexMatcher ¶
RegexMatcher selects columns whose name matches a pattern.
ursus requires the pattern to be passed to ColRegex explicitly rather than inferring "this string looks like a regex" from leading ^ and trailing $ the way Polars does. That inference makes a column literally named "^total$" unselectable and makes every plain Col() call pay a scan for anchors.
func NewRegexMatcher ¶
func NewRegexMatcher(pattern string) (*RegexMatcher, error)
NewRegexMatcher compiles pattern, returning an error the caller parks in an *Err node.
func (*RegexMatcher) Match ¶
func (m *RegexMatcher) Match(in *dtype.Schema) ([]string, error)
Match returns every matching column. Matching nothing is not an error — it is the natural answer for a pattern over a schema that happens not to contain it, and erroring would make schema-driven code fragile.
func (*RegexMatcher) String ¶
func (m *RegexMatcher) String() string
type Rename ¶
type Rename struct {
Child Node
// Fn derives the output name from the child's; nil leaves it.
Fn func(string) string
// Keep names the output after the first column the child reads; Fn is unread.
Keep bool
// Fields renames each field of the struct the child answers; nil leaves them.
Fields func(string) string
Label string // rendering only, e.g. `name.prefix("avg_")`
// contains filtered or unexported fields
}
Rename applies a function to the output name.
This is the expansion-safe counterpart to Alias: where Alias imposes one fixed name (and so cannot be applied to a multi-column expression), Rename derives a name per output, which is what Name().Prefix / .Suffix / .Map need.
It carries the name namespace's other functions too (step 323): Keep names the output after the first column the child reads, Polars' name.keep, and Fields renames a struct's fields, its prefix_fields, suffix_fields and map_fields. A Rename with Fields changes the value's type, so it is a computation, not only a name, and stripNaming keeps it.
func NewRename ¶ added in v0.6.7
NewRename is r given an id of its own, for a function its label cannot render: two of them over one child are two computations to Identity.
type Reshape ¶ added in v0.6.5
type Reshape struct {
Op ReshapeOp
Child Node
Args []int64
By Node
Label string // ValueCounts: its count's field name
Roll *RollSpec // Rolling: its window
}
Reshape is a length-changing operation on Child. By is SortBy's keys and Gather's indices, nil for every other.
func LooseReshape ¶ added in v0.6.5
LooseReshape is the first Reshape in n that LowerReshapes left, or nil: one outside an aggregation.
func (*Reshape) Field ¶ added in v0.6.5
Field is the child's: a reshape keeps its values' type, but Explode, whose values are its List's elements.
func (*Reshape) IsGenerator ¶ added in v0.6.6
IsGenerator reports whether r is a scalar generator: an explode in which an empty list is no row, Polars' int_range and the rest.
type ReshapeOp ¶ added in v0.6.5
type ReshapeOp uint8
ReshapeOp is a length-changing operation on a column.
const ( ReshapeSort ReshapeOp = iota // Args: descending, 0 or 1 ReshapeHead // Args: n ReshapeTail // Args: n ReshapeSlice // Args: offset, length ReshapeReverse // no args ReshapeUnique // no args: first-seen order, null a value ReshapeDropNulls // no args ReshapeGet // Args: index; one value, not a column ReshapeDropNans // no args (step 253) ReshapeGatherEvery // Args: n, offset (step 253) ReshapeSortBy // Args: descending, 0 or 1; By the keys (step 253) ReshapeGather // no args; By the indices (step 257) ReshapeExplode // a List's elements (step 259); Args [1]: a generator's, an empty list no row (step 263) ReshapeArgSort // Args: descending, nulls last (step 267) ReshapeArgTrue // no args: the true rows' indices (step 267) ReshapeArgUnique // no args: each value's first index (step 267) ReshapeArgSortBy // Child: struct_of the keys; Args: each key's descending and nulls last (step 267) ReshapeMode // Args: maintain_order, 0 or 1: the most frequent values (step 277) ReshapeUniqueCounts // no args: each value's count, in first-seen order (step 280) ReshapeValueCounts // Args: sort, normalize; Label: the count's name; a Struct a value (step 280) ReshapeRle // no args: a Struct of each run's length and value (step 281) ReshapeAppend // By: the values appended; Args: their meeting, AppendStrict, AppendUpcast or AppendCast (step 282) ReshapeSample // Args: n, or -1 for a fraction; the fraction's bits; with_replacement; shuffle; the seed (step 282) ReshapeHist // Args: the bin count, or -1 for edges; include_category; include_breakpoint; the edges' bits (step 286) ReshapeRolling // By: the index; Roll: the window; Child aggregated over each row's (step 337) ReshapeArr // Args: the shape, -1 first to infer it: the values in Arrays of it (step 343) )
type RollSpec ¶ added in v0.6.7
RollSpec is a rolling reshape's window (step 337): a row's window is its index's instant plus Offset, Period long, Closed at the ends it says, as a frame's rolling group's.
type RollingMapFn ¶ added in v0.6.7
type RollingMapFn struct {
Name string
Out dtype.DataType
Impl UDFImpl
// contains filtered or unexported fields
}
RollingMapFn is rolling_map's Go function (step 338), held by a window function's parameters: named, its answers declared Out, and Impl the kernel's, which expr cannot name, as a UDF's. Its identity is minted as a UDF's, by NewRollingMap, so CheckUDFNames refuses two of one name as it refuses two UDFs.
func NewRollingMap ¶ added in v0.6.7
func NewRollingMap(name string, out dtype.DataType, impl UDFImpl) *RollingMapFn
NewRollingMap builds a rolling_map's function with a fresh identity.
func (*RollingMapFn) ID ¶ added in v0.6.7
func (m *RollingMapFn) ID() uint64
ID is the function's identity, a UDF's: distinct for every one built, shared by every copy of the window that holds it.
type SetMatcher ¶ added in v0.6.7
SetMatcher is two matchers' columns combined by a set operation.
func (*SetMatcher) Match ¶ added in v0.6.7
func (m *SetMatcher) Match(in *dtype.Schema) ([]string, error)
func (*SetMatcher) String ¶ added in v0.6.7
func (m *SetMatcher) String() string
type SetOp ¶ added in v0.6.7
type SetOp uint8
SetOp is how a SetMatcher combines its two matchers' columns.
type UDF ¶ added in v0.3.0
type UDF struct {
Child Node
// Out is the output type as DECLARED by the caller. It is not derived from
// anything and cannot be — the function is opaque. The evaluator refuses a
// column whose type disagrees with it.
Out dtype.DataType
// Name distinguishes this UDF from every other one in the query. Required.
Name string
// Kind is "map_elements" or "map_batches", for rendering and errors only.
Kind string
Impl UDFImpl
// contains filtered or unexported fields
}
UDF applies a user-supplied Go function to one column.
It is the escape hatch, and it is OPAQUE: nothing in the planner can inspect it, fold it, prove it pure, or predict what it returns. Everything the engine knows about it, the user declared — which is why the evaluator CHECKS those declarations rather than trusting them. See physical.evalUDF.
Name is required, and that is a correctness argument rather than a style one ¶
Node.String is load-bearing: "it appears in Explain output, in golden test files, and in error messages". Worse, until step 72 FOUR maps deduplicated expressions on their rendering, over three code sites, because extractAggs is instantiated twice — plan.resolveWindow's temporaries, physical.extractAggs' byKey for a hash aggregate AND for a temporal group, and physical's partition sharing — and names_test.go states the consequence: "two expressions that RENDER THE SAME become one computation, and both names get the first one's answer".
A Go closure has no printable identity. Without a name, two DIFFERENT user functions over the same column render alike, collapse into one computation, and both results come from whichever was seen first. That is the shape of two bugs this project has already shipped and fixed: an unparameterised Quantile that collapsed p50 and p99, and weak-versus-strong literals.
A monotonic counter would deduplicate correctly too, and would make every golden plan file depend on allocation order. So: a name, supplied by the caller.
That objection is about a counter that RENDERS. The id field below is a monotonic counter which never renders — String is unchanged and the golden plans with it. It is what lets plan.CheckUDFNames refuse a collision instead of merging one, and since step 72 it is part of what the four maps key on, through Identity, so two names that look alike no longer merge even where the check has not run.
func NewUDF ¶ added in v0.3.0
NewUDF builds a UDF node with a fresh identity, and is the only way to mint one.
The fields stay exported because plan and physical read them and copy the struct wholesale. The id does not, so it cannot be forged onto a different Impl.
func (*UDF) Field ¶ added in v0.3.0
Field takes the child's NAME and the caller's declared type.
Naming follows the receiver, like Call.Field: leftmost-column-wins means Col("c").MapElements(...) is named "c", and an Alias above overrides it.
Resolving the child is not a formality — it is what makes a UDF over an unknown column a schema error at plan time rather than at the first batch.
type UDFImpl ¶ added in v0.3.0
type UDFImpl interface{ UDFKernel() }
UDFImpl is a user-supplied kernel, erased.
Why an opaque interface rather than a func field ¶
The obvious field is `func(*data.Column) (*data.Column, error)`, and it does not compile. internal/gen/levels puts internal/data and internal/expr at the SAME level (20), and the rule is strictly-lower, so this package cannot import internal/data. Nor can that be fixed by renumbering: internal/plan imports this package, and TestPlanPackageNeedsNoArrow asserts that plan has zero transitive dependency on data, bitmap, arrowx or anything named arrow.
So the erasure goes one step further than it first appears. A generic method on the public Expr captures its type parameters into a closure — which is required anyway, because Go 1.27 permits generic methods but forbids one from implementing an interface method, and Node is an interface. That closure is then held here behind a marker interface and implemented in a package above internal/data. internal/kernel (30) declares the concrete type; internal/physical (50) asserts it back out.
It is NOT sealed, and cannot be ¶
Node is sealed with an unexported node() because every Node implementation lives in this package. That technique is unavailable here for the very reason this interface exists: an unexported method can only be implemented from inside the package that declares it, and the implementation MUST live outside — above internal/data, which this package may not import. So the marker is exported.
The consequence is that a type outside internal/kernel could satisfy this and reach expr.UDF.Impl. physical.evalUDF asserts for the one concrete type it knows and reports anything else as an internal error rather than ignoring it, which is the containment the seal would otherwise have provided.
type UnaryOp ¶
type UnaryOp uint8
UnaryOp is a one-operand operation.
const ( OpNeg UnaryOp = iota OpAbs OpNot // Kleene NOT: NOT null is null // Null predicates. These are the only operations whose result is never null, // because the null-ness IS the answer. OpIsNull OpIsNotNull // NaN predicates. Distinct from null predicates: null.is_nan() is NULL, not // false. Null and NaN are different things all the way down. OpIsNan OpIsNotNan OpIsFinite OpIsInfinite // Transcendental and rounding maths. These are the ops whose OUTPUT TYPE // DIFFERS FROM THEIR INPUT — Sqrt(Int64) is Float64 — which is why they cannot // share unaryArith, whose whole shape assumes one type throughout. // // Contiguous, because IsMath is a range test over them. OpSqrt OpCbrt OpExp OpLn OpLog10 OpLog1p // Sign, Floor and Ceil are NOT in that block: each keeps its operand's type, so // they travel with Neg and Abs through unaryArith. Grouping them with the // transcendentals would have made sign(Int64) and floor(Int64) into Float64s — // and floor on an integer is the identity, so widening it would be a schema // change in exchange for nothing. OpSign OpFloor OpCeil // Trigonometry, appended in step 114 rather than inserted after OpLog1p, so no // existing constant moves. Widening like sqrt, which is why IsMath tests a // second range for them. Radians in, radians out; Degrees and Radians convert. OpSin OpCos OpTan OpArcSin OpArcCos OpArcTan OpSinh OpCosh OpTanh OpArcSinh OpArcCosh OpArcTanh OpDegrees OpRadians // OpCot, the cotangent, 1/tan (step 283), appended after the block it joins. OpCot )
func (UnaryOp) IsMath ¶
IsMath reports whether the op widens its operand to a float and returns a float. A range comparison over declaration order, like the classifiers above, which is why that block is append-only.
func (UnaryOp) IsNanPredicate ¶
IsNanPredicate reports whether the op tests a float property.
func (UnaryOp) IsNullPredicate ¶
IsNullPredicate reports whether the op's result is never null.
type Unnest ¶ added in v0.6.5
Unnest is a Struct column's fields, a column each (step 260, E4 of v0.6.5-scope.md): Polars' struct.unnest(). Like a selection of several columns it is one expression that denotes several, and Expand writes it out: a copy of the expression for each field, the Unnest replaced by that field and the copy named after it. So Col("s").Unnest() is s's fields, in Select beside other columns and in WithColumns after the frame's, and Col("s").Unnest().Add(1) each field plus one.
Fields selects the fields written out, Polars' struct.field of several names (step 275): each a field's name, a regex between ^ and $, or "*" for every field. None is every field, unnest's.
type Variadic ¶ added in v0.6.7
Variadic is a function whose operands are a list, which Polars writes out before it calls the function (step 302): sum_horizontal(pl.all()) sums every column into one, concat_str(pl.all()) joins them, struct(pl.all()) packs them, coalesce(...) takes the first value, and hash_rows hashes them. Expand writes out each of Exprs over the input's schema, a selection into its columns in order, and Build makes the node over them, as buildFolds builds a Fold.
Before step 302 these were built at once, and a selection among their operands expanded the whole function once per column, as it expands any other expression: MaxHorizontal(All()) gave each column its own row's maximum, itself. A list's body, lowered when it is written, builds one over its operands as written (buildAsWritten); OutputName names one after its first operand, as the function built is named.
type WinFn ¶
WinFn is an ordered window function — rank, a running total, a shift.
Why not a Call ¶
Call{Fn, Args} already carries a function plus literal arguments, and this is the same shape. But Call is contractually ELEMENTWISE: evalCall runs inside a BatchOp, which sees one batch and may run on any worker. These functions need the whole partition in order, so they are pipeline breakers. Same shape, opposite contract — and a separate node type is what makes "only legal inside a Window" checkable.
Why not an AggOp ¶
They do not reduce. kernel.Accumulator's Finish(name, nGroups) returns one row per group; these return one row per INPUT row. They need a different kernel family.
Fill is Shift's optional replacement for the rows shifted in from outside the partition. nil means null, which is the usual want.
type WinFnOp ¶
type WinFnOp uint8
WinFnOp is a function that is only meaningful within an ordered window.
const ( WinRank WinFnOp = iota WinCumCount WinCumSum WinCumProd WinCumMin WinCumMax WinShift // Forward and backward fill. TWO ops rather than one plus a Reverse flag, on // purpose: WinFn.String() renders the op name, so the direction can never // collide even if the parameter rendering is imperfect. With one op the // direction would live ONLY in WinParams.args, which is exactly the field that // has been the source of every collision bug so far. WinForwardFill WinBackwardFill // The rolling functions (step 149): each row's answer over the Params.N rows of // its partition up to and including it, in the window's order, or null where // fewer than Params.MinSamples of them hold a value. WinRollingSum WinRollingMean WinRollingMin WinRollingMax WinRollingVar WinRollingStd // A median and a quantile of the window's values (step 227): Params.Q and // Params.Interp, a median's 0.5 and linear. WinRollingMedian WinRollingQuantile // The exponentially weighted functions (step 153), Polars' and pandas' ewm_*: // each row's answer over every row of its partition up to it, weighted by // Params.Alpha's decay. WinEwmMean WinEwmStd WinEwmVar // WinInterpolate fills each run of nulls between two values on the straight line // between them, by position in the window's order. WinInterpolate // WinRleID numbers each row's run of equal values, from 0, by the group-by's keys // (step 281): Polars' rle_id. WinRleID // More rolling functions (step 284), appended so no op moves: the skew and // kurtosis of each row's window, Params.Bias and Params.Fisher as Polars' bias and // fisher; each row's rank among its window's values, by Params.Method; and the // covariance and correlation of a pair, the child PairOf(a, b), Params.Ddof the // covariance's. IsRolling takes both blocks. WinRollingSkew WinRollingKurtosis WinRollingRank WinRollingCov WinRollingCorr // WinEwmSum is Polars' ewm_sum (step 285): each value plus the running sum // decayed by Params.Alpha, appended so no op moves; IsEwm takes it. With // Params.By it and WinEwmMean are ewm_sum_by and ewm_mean_by, their child // PairOf(value, by), decayed by Params.Period, a half-life, in by's order. WinEwmSum // WinCumulativeEval is Polars' cumulative_eval (step 286): Params.Body, lowered by // LowerCumulativeEval, over each row's prefix of its partition. The physical // window evaluates it, as no kernel can evaluate an expression. WinCumulativeEval // WinRollingMap is Polars' rolling_map (step 338): Params.Map's Go function over // each row's window of Params.N rows, as the rolling functions' are, appended so // no op moves; IsRolling takes it. WinRollingMap // WinCatCodes is each row's value's code (step 351): the order, from 0, its // value first appears in among the window's non-null values, by the group-by's // keys; a null's null. Over the frame it is a Categorical's codes, as Polars // numbers them in a fresh process, which the root writes ToPhysical out as. WinCatCodes )
func (WinFnOp) IsCumulative ¶
IsCumulative reports whether the op is a running scan over the ordered partition.
func (WinFnOp) IsPaired ¶ added in v0.6.6
IsPaired reports whether the op is of two series, its child PairOf(a, b): rolling_cov and rolling_corr (step 284).
type WinParams ¶
type WinParams struct {
Method RankMethod
Descending bool
Reverse bool
N int64
// MinSamples is a rolling function's least number of values in a window for it
// to answer; zero means N, as Polars' min_samples=None does. Ddof is rolling
// var's and std's delta degrees of freedom.
MinSamples int64
Ddof int64
// Q and Interp are rolling_quantile's quantile and interpolation (step 227).
Q float64
Interp Interpolation
// Center labels a rolling window at its middle row rather than its last, as
// Polars' center=True; Nearest is Interpolate's method="nearest" (step 172).
Center bool
Nearest bool
// The ewm functions' (step 153): Alpha is the decay; NoAdjust is Polars'
// adjust=False, so the zero value is its default; IgnoreNulls weighs by
// position among the values rather than among the rows; Bias is ewm_var's and
// ewm_std's bias=True.
Alpha float64
NoAdjust bool
IgnoreNulls bool
Bias bool
// Fisher is rolling_kurtosis's fisher, the excess kurtosis; Bias is also the
// rolling skew's and kurtosis's bias (step 284).
Fisher bool
// Body is cumulative_eval's, lowered over CumulativeName (step 286).
Body Node
// Map is rolling_map's function (step 338), WinRollingMap's.
Map *RollingMapFn
// A rolling function by time (step 177), Polars' rolling_*_by: each row's window
// is the rows whose by value lies within Period of its own, closed as Closed says,
// ClosedRight by default. The function's child is then PairOf(value, by), and N is
// unused.
By bool
Period dtype.Interval
Closed Closed
}
WinParams carries an ordered function's configuration.
A typed struct rather than child nodes, for the reason AggParams gives: these are configuration, not operands. There is no meaning to a per-row rank method.
type Window ¶
type Window struct {
Child Node
PartitionBy []Node
OrderBy []OrderKey
Mapping MappingStrategy
// Inherits marks a window the library built for a sugar, not one the user
// wrote: FillNull(FillMean)'s mean over the frame. Inside an enclosing Over it
// takes that Over's partition, order and mapping (plan's extractWindows, step
// 148), so FillNull(FillMean).Over(g) fills with each group's mean, as Polars
// does. A window the user wrote inside another is still refused.
Inherits bool
}
Window computes its child within partitions and puts the answer back on every row — the `.Over()` of the API, and SQL's OVER clause.
Its operands are ordinary children, deliberately ¶
Node.Children's doc offers an escape hatch for exactly this situation: "A nested SCOPE body is deliberately not a child … so a generic walker must not rewrite across it." That mechanism is WRONG here, and the reason is worth stating because the alternative looks so plausible.
It exists because a list body resolves against a DIFFERENT schema — the element type. A window's child resolves against the same schema as everything around it: `Col("x").Mean().Over(Col("g"))` resolves x and g against one input schema. There is no schema change to protect against.
And hiding the child would break four things that presently work for free: RootNames would report {g} and not {x}, so projection pushdown would prune away the very column being aggregated; Expand would not reach a matcher inside the body, making All().Mean().Over(g) inexpressible; FirstErr would miss a deferred error; and OutputName would name the result after the partition key.
So the operands stay ordinary children and the scoping lives in HasUnboundAgg instead. This is what design/logical.md predicted: "Spec.PartitionBy and OrderBy are ordinary children, so RootNames and expansion pick them up automatically."
Where a Window is legal ¶
In Select, WithColumns, Filter and Sort — the row-preserving contexts, because a window preserves height. NOT inside GroupBy().Agg(), which is the nested-window case and is refused.
func (*Window) Children ¶
Children returns the child first, then the partition keys, then the order keys.
Child first matters: OutputName's generic fallback takes the first name from RootNames, so any other order would name a window after its partition key. There is an explicit OutputName case as well, and this ordering means the two agree rather than one masking the other.
func (*Window) String ¶
String renders the child, the partition keys, the order keys AND the mapping.
Every part is load-bearing rather than decorative, for the reason step 7 recorded when it fixed the same class of bug for Agg: the physical planner deduplicates extracted sub-expressions by keying a map on this string. Two windows that render identically become ONE computation. So `x.mean().over(g)` and `x.mean().over(h)` rendering the same would give every row the wrong partition's mean, with no error and no length mismatch anywhere to catch it.