Help us shape the future of Go! Take the 2026 Go Developer Survey

ursus

package module
v0.6.8 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 26, 2026 License: MIT Imports: 56 Imported by: 0

README

ursus

Go Reference CI Docs

ursus is a dataframe library for Go (Golang), modelled on Polars — lazy execution with a query optimizer, Apache Arrow memory, SIMD kernels, and streaming execution that spills to disk rather than falling over. Pure Go, no cgo; experimental, at 0.6.

Documentation: advenn.github.io/ursus — a quick start, the concepts, a guide for Polars users, versioned by release, with every example run by the test suite.

Sort, group-by, join, unique, every window function and the temporal groups spill or stream — by default once a query holds half the memory its container or machine allows, on Linux, macOS and Windows — and an operator that cannot spill fails with an error naming itself. Under that default budget ursus also sets the Go runtime's soft memory limit near the same ceiling, so the heap's own slack stays under it too; GOMEMLIMIT, or ursus.LeaveProcessMemoryLimit(), keeps a program's own choice.

df, err := ursus.ScanParquet("events.parquet").
    Filter(ursus.Col("price").Gt(5)).
    GroupBy(ursus.Col("region")).
    Agg(
        ursus.Col("price").Sum().Alias("revenue"),
        ursus.Col("qty").Mean().Alias("avg_qty"),
    ).
    Sort(ursus.Desc(ursus.Col("revenue"))).
    Collect(ctx)

The projection reaches the Parquet reader, the filter becomes a row-group predicate, and the group-by runs on every core. None of that is visible in the query.

Is this for you?

It is pure Go. No cgo, no C++ toolchain, no Python runtime, no sidecar process. It cross-compiles and links into a static binary like any other Go dependency, and go get is the whole install.

That is the reason to pick it, and the cost should be just as plain: ursus is slower than the serious analytical engines. On the h2o.ai benchmark at ten million rows it is about 1.3x Polars over Parquet and 1.6x over CSV; on TPC-H at scale factor 1 it is about 2x. It is not trying to beat Polars or DuckDB, and on current evidence it is not going to.

Where that trade is a good one

  • Data where the speed difference is small in absolute terms — up to tens of millions of rows, where it is half a second against a tenth of one and nobody is waiting. Larger data still runs, spilling to disk past the memory limit, but there the same ratio is seconds or minutes, and a spilled query pays for its disk too.
  • A Go service that needs real dataframe work — joins, group-bys, window functions — and cannot take on cgo, a Python sidecar, or a large C++ dependency to get it.
  • Anywhere the deployment story is worth more than the last multiple of speed.

Where it is not

  • Interactive analytics over hundreds of millions of rows. Use DuckDB or Polars.
  • Anywhere you can link cgo freely: duckdb-go is around 2x faster than this on TPC-H and is a binding to a mature engine.
  • Anything production-critical today. This is v0.6, the API still moves, and nothing here is promised.

The honest summary is that Go has not had a dataframe library of this shape, and one that is correct and a few times slower is more useful than none — for the sizes most services actually handle.

Docs

The guide is at advenn.github.io/ursus: a quick start, the concepts, a page for Polars users, and benchmarks, versioned by release. Every code block on it is a Go example in docs/examples that go test runs, its output the one shown.

The API reference is on pkg.go.dev, generated from the source — nothing to host and nothing to keep in sync.

Start with the runnable examples: filter, group-by, join, computed columns, whole-frame aggregation, and reading typed values back out — and, new in 0.3, reading Parquet through any io.ReaderAt, a Go UDF, lists with Split and Explode, JoinWhere, exact Decimal arithmetic, an Enum, a spilling Unique, and a List column written to Parquet. They render beside the methods they document, and go test checks each one's output, so an example that stops being true breaks the build rather than misleading someone quietly.

Beyond that, the doc comments are the documentation. They are unusually long on purpose: each explains why a thing is the way it is, not only what it does.

Install

go get github.com/advenn/ursus

Go 1.27 or newer is required. Two of its changes are load-bearing: generic methods are why Series[T].Map[U] and df.Column[T](name) exist at all, and because a generic method still cannot satisfy an interface, every public type is a concrete struct with polymorphism kept in unexported interfaces.

GOEXPERIMENT=simd is optional. It switches on the SIMD kernels:

GOEXPERIMENT=simd go build ./...

Without it every kernel falls back to its scalar twin, behind //go:build !(goexperiment.simd && amd64). That is not a claim — CI runs the entire suite with the experiment off on every push, and make test-all includes an experiment-off leg locally. The flag buys speed, not correctness.

Status

v0.6 — what changed since v0.5 is in CHANGELOG.md, and what 0.6 set out to do, and did, in v0.6-scope.md. What works today:
Sources Parquet, CSV (gzip, zlib or Zstandard too), JSON, NDJSON and Arrow IPC, read and written, from paths, memory, or any io.ReaderAt / stream you open; directories, ** globs and hive partitions, read and written, with Parquet schema evolution; databases read and written through database/sql; in-memory frames and Go values (FromRows, Rows, ToMaps); Arrow (records and streams in; records out, zero-copy)
Types Bool, Int8–64, Uint8–64, Float32/64, String and Binary (past 2 GiB too), Date, Time, Datetime (unit + zone), Duration, Decimal (128-bit; exact + - * and sum, refused past 38 digits; / to the nearest Float64; casts to and from every numeric type exact or refused, except to a float, which rounds to the nearest), Enum (built from String, ordered by its categories), Categorical, List, Array (fixed width, the Arr() namespace) and Struct, each of the last three a key too
Expressions Polars' contexts: an aggregate is one value in Select, WithColumns and Filter; Expr.Filter is an aggregate's input; Sort, SortBy, Head, Tail, Slice, Reverse, Unique, DropNulls, DropNans, Gather, GatherEvery, Get, Explode, ArgSort, Mode, ValueCounts, Rle, Append, SampleN and Hist are a list a group in an aggregation and a column of their own length in Select and WithColumns; Unnest and Struct().Field of several names are a column a field; arguments take expressions, and a Go slice is a List literal. Generators (IntRange, LinearSpace, Repeat, the date, datetime and time ranges, MakeDate and its kin), folds (Fold, Reduce, CumFold, ConcatList, Format), selection by position (Nth, ArgWhere), and List().Eval over Element(). Then arithmetic, comparison, Kleene three-valued logic and the bits of integers, conditionals, casts, null repair, value maps (Replace, ReplaceStrict, ReplaceOr), rounding with a mode, RoundSigFigs and Truncate, trigonometric and hyperbolic functions, bit counts, Hash, SearchSorted and IndexOf, ConcatStr, Struct, Max/Min/Sum/Mean/Any/AllHorizontal, Cut/QCut, the .str (incl. Split, SplitExact, ContainsAny and ReplaceMany over many patterns, Normalize, Encode, JsonDecode and Strptime with strftime formats), .dt (incl. Date, Time, Strftime, OffsetBy, Round, ConvertTimeZone, Replace, IsBusinessDay, AddBusinessDays and the fractional totals), .list (incl. Join, NUnique, ArgMin/ArgMax, the set operations, Gather, ToStruct, Shift, Diff, SampleN), .cat (on an Enum), .bin and .struct namespaces, the aggregates (incl. Implode, TopK, BottomK, Corr, Cov, CorrSpearman, Skew, Kurtosis, Entropy, MinBy, MaxBy, the bitwise reductions and ApproxNUnique), and window functions (incl. Rolling* with RollingMedian, RollingQuantile, RollingSkew, RollingKurtosis, RollingRank, RollingCov and RollingCorr, Rolling*By by time, Ewm* with EwmSum and EwmMeanBy, RleID, Interpolate, CumulativeEval, and any body around an aggregate inside .Over)
Frame ops filter, select, with-columns, sort, top-k, distinct (UniqueKeep, duplicates over columns), concat in its modes/vstack/hstack, slice/tail/reverse/row-index, drop/rename/drop-nulls, a frame's cast, fills, aggregates and sample, pivot, unpivot, transpose, PartitionBy, Upsample, ToDummies, selectors, ValueCounts, Describe, a Series with operations
Joins all seven equi-join kinds with Validate and maintain_order, Update, JoinWhere and WhereExists/WhereNotExists (non-equi), as-of join with a tolerance, by keys and coalesce, merge-sorted
Grouping group-by with Having, a group's head and tail, and its groups as frames; GroupByDynamic, Rolling, calendar-aware intervals
Optimizer predicate pushdown (including through joins), projection pushdown (a count reads no column), limit/top-k pushdown through what keeps rows, cross-join collapse, constant folding and simplification, a shared subtree run once
Execution order-preserving pipeline parallelism, Parquet row groups decoded and CSV parsed in parallel, parallel hash aggregation with a partitioned fold and a frame's aggregate a column at a time, Profile of each node, a partitioned parallel join build, a sort and a unique on several threads, text sorted by radix, inner joins that hash their smaller input, and spilling for sort and top-k, hash aggregation (a median's, quantile's and n_unique's values too), hash join, unique, every window function over one huge partition too, keyed Rolling and GroupByDynamic, tail and reverse (WithMemoryLimit names the rest, which fail rather than spill)
UDFs MapElements (per value) and MapBatches (per column) — generic methods, so the Go types are inferred from your function

Nested types are partly there: List and Struct read from Parquet and Arrow and write to Parquet, with Explode, Unnest, a .list namespace and .struct (Field, WithFields, RenameFields, JsonEncode). An Arrow Map arrives as a list of key/value structs. Array is not there, and a List of Lists or of Structs is refused by name on both sides.

Arrow goes both ways, in pure Go. df.Record() and lf.CollectRecords(ctx) export without copying; ScanArrow reads any array.RecordReader — an IPC stream, Flight, the C Data Interface — and ScanArrowRecords reads records already in memory. Import copies, so nothing you release afterwards can reach a frame:

lf := ursus.ScanArrow(func() (array.RecordReader, error) {
    return ipc.NewReader(bytes.NewReader(stream)) // a new reader every call
})

The function is called once to plan the query and once per scan, because a reader cannot be rewound. 38 of Arrow's 45 types map; Decimal256, intervals, unions and run-end encoding are refused by name.

Object storage has no client built in: each store wants its vendor's SDK, and ursus is pure Go with one dependency. ScanParquetFrom reads Parquet through any io.ReaderAt instead, so a ReadAt over ranged GETs reaches S3, GCS or an HTTP server — and ursus reads the footer and then only the column chunks the query needs, after statistics have pruned the row groups it does not:

lf := ursus.ScanParquetFrom([]ursus.ParquetFile{{
    Name: "s3://logs/2026/10/part-0.parquet",
    Open: func(ctx context.Context) (io.ReaderAt, int64, error) {
        return openRanged(ctx, client, "logs", "2026/10/part-0.parquet") // your ReadAt over GetObject with a Range
    },
}})

ScanCSVFrom takes a stream the same way, and WriteParquet and WriteCSV write to any io.Writer, an upload included. Built-in stores — s3:// paths, listing for globs, retries — are out by decision; the seam is the way in.

The escape hatch is real: a per-element UDF in Go is a function call, not a Python interpreter round trip, which is the one place this library can beat Polars outright rather than merely keep up.

lf.Select(ursus.Col("celsius").MapElements("to_fahrenheit", ursus.Float64,
    func(c float64) (float64, error) { return c*9/5 + 32, nil }))

Nulls pass through untouched, so your function never receives a zero value it cannot tell from a real one; use MapBatches when the null is the point. The name is required, and Explain shows it. Your function is called from several goroutines at once, so it must be safe for that.

Not done: built-in object stores and SQL, both out by decision (the seam above reaches a store today); MapGroups and RollingMap; a lazy Pivot, whose output columns are the distinct values of a column, so its schema would depend on data and no plan node here does (Unpivot ships, and DataFrame.Pivot and Transpose eagerly); Upsample; Sample and List().Eval; a rolling rank, and a rolling window by an integer by; Spearman's correlation; Arrow IPC files and NDJSON. v0.6-scope.md says why each waits, and feature-gaps.md lists everything Polars, pandas and DuckDB have that ursus does not, which the rest of 0.6 works through.

ursus is on ZeroVer: 0.x, and a stable 1.0 is not expected yet. Version numbers follow Go's rule for v0, so a minor version may rename, remove or reshape an API when the result is better, with no deprecation period. The changelog lists each release's breaks with their replacements.

Correctness

make test-all   # four SIMD widths (512/256/128/0) plus the experiment off
make race       # the whole suite under -race
make levels     # import-level invariants

3240 test cases, and the matrix is not decoration. Vector width is a runtime property, so a single-width run proves very little: 512-bit gives 8 float64 lanes, which happens to be exactly one bitmap byte — a coincidence that hides an entire class of sub-byte bitmap bug. The 128-bit leg is where those surface.

Correctness is also checked against other engines. The benchmark suite validates every result against a duckdb reference, and ursus currently passes 22/22 PDS-H (TPC-H) queries and 15/15 h2o.ai queries — PDS-H at scale factor 0.1, re-run at every step, and both suites at the published sizes in the report below; every answer ursus gave there validates.

Benchmarks

Everything that measures ursus lives in bench/ — three tiers behind one Makefile, competing against duckdb, polars, pandas, DataFusion, chDB, Arrow-Go, Gota and QFrame.

cd bench
make preflight    # refuses to run on a machine without the free memory; that is the feature
make setup
make bench        # gen -> reference answers -> run -> validate -> report

See bench/README.md for what is timed and why.

bench/results/REPORT.md is checked in. Every result in it is validated against a duckdb reference, and a disagreement is struck through rather than quietly reported as a fast number.

How current it is, precisely. ursus and Polars were re-run together, every query of all four suites, on step 246's code, 659ec39, on 2026-10-14 (step 247). v0.6.5 to v0.6.8 changed no plan the suites run: the optimized plans of all 37 queries are byte-identical to v0.6.4's, v0.6.5's, v0.6.6's and then v0.6.7's, and every answer still matches DuckDB's (steps 261, 287, 364 and 374). v0.6.7 changed the sort beneath them, and PDS-H at SF=0.1 took 0.95 to 0.98 of v0.6.6's time; v0.6.8 the strings, the scans and the aggregates, and it took 1.00 of v0.6.7's, so the figures stand. v0.6.8's CSV reader is faster than they say: h2o's gb1 over CSV took 0.78 of the time before it. The other engines' numbers are from an earlier session, every engine together on 5a721e0 on 2026-10-08 (step 127): their code has not changed, and re-running them would have taken the laptop for an hour and more. So ursus is compared with Polars within one session, and with the rest across sessions.

The caveats that apply to this run:

  • No cgroup was set, where v0.5.0's report ran every query in an 8 GB one. It ran on 8 threads, on a laptop with 4 to 7 GB of memory available and 4 GB of swap in use by IDEs and the rest. Nothing else of this project's ran alongside it.
  • The session, not the code, moved the ratios. Alternated against v0.6.3's code, both built alike, 0.6.4 is level: PDS-H at SF=1 1.4% less wall time, at SF=0.1 2% less, h2o within noise. 0.6.4 changed what happens past a memory limit, and these run without one. Both engines ran h2o faster than in v0.6.3's session, which was short of memory: over Parquet ursus 928 to 697 ms and Polars 774 to 531, over CSV 2,535 to 1,784 and 1,434 to 1,088.
  • PDS-H at SF=0.1 ran twice: the first run, straight after SF=1, was slower on every query than the same code a minute later, and was run again alone.
  • ursus passed every query of all four suites, and every answer validated against duckdb's.
  • ursus's highest peak is h2o gb10 over Parquet, 7.72 GB, with no limit set, and 6.44 GB over CSV; Polars' highest is 5.03 GB. With no limit a peak is where the garbage collector lets the heap grow, and it moves with the memory free: v0.6.3's report had 6.76 and 8.17 GB.
  • Out of core, each path v0.6.4 made spill or stream, under a tenth of its input, gives the answer it gives without a limit, peaking at 1.01 to 2.11 times the limit; step 247 has the table.
  • Other engines: chDB timed out on two group-bys and failed gb6 in both h2o suites; gota ran out of memory on two group-bys; DataFusion, and chDB at SF=0.1, answered PDS-H q15 wrongly, and are struck through.

Timings come from a laptop under real conditions, so treat small differences as noise and the ordering as the signal.

Where it stands, in one place. Against Polars, on the current report, with v0.6.3's, v0.6.2's, v0.6.1's, v0.6.0's, v0.5.0's, v0.4.0's and v0.3.0's for comparison:

ursus Polars at v0.6.3 at v0.6.2 at v0.6.1 at v0.6.0 at v0.5.0 at v0.4.0 at v0.3.0
h2o.ai, 10M rows, Parquet 697 ms 531 ms 1.3x 1.2x 1.5x 1.5x 1.6x 1.7x 1.9x 3.3x
h2o.ai, 10M rows, CSV 1,784 ms 1,088 ms 1.6x 1.8x 1.65x 1.7x 1.8x 2.0x 3.3x 3.6x
TPC-H SF=1 158 ms 79 ms 2.0x 1.9x 2.1x 2.2x 2.3x 3.6x 4.4x 10.8x
TPC-H SF=0.1 20 ms 9 ms 2.3x 1.8x 2.8x 2.4x 2.6x 3.1x 4.0x 7.5x
TPC-H SF=1 peak memory 0.83 GB 0.92 GB 0.9x 1.0x 0.9x 1.0x 1.0x 1.3x 1.1x 2.0x

Geomeans over each engine's passed queries; ursus and Polars pass every one. v0.6.4's code is level with v0.6.3's when the two are alternated, so the moves against Polars are the session's; see the caveats above. v0.6.3's h2o figures came from a session short of memory that slowed both engines.

Where ursus is behind and why is recorded rather than glossed — every step of construction has an as-built document in context_files/, including the optimisations that were tried and reverted for being slower.

How it is built

dtype/          types, schemas, the null contract
i128/           128-bit integers, for Decimal
internal/
  data/         Column and Batch — Arrow layout, three payload shapes
  bitmap/       validity and boolean bitmaps, offset-correct by construction
  kernel/       the compute kernels, SIMD with scalar twins
  expr/         the expression IR
  plan/         logical plan, resolution, the optimizer rules
  physical/     operators, the sink/breaker protocol, spilling
  exec/         the driver
  source/       parquet, csv, memory
ursustest/      assertions for testing code that uses ursus

Packages are arranged in strict import levels — internal/gen/levels fails the build if a package imports one at or above its own level, which is what keeps the dependency graph a DAG rather than a suggestion.

Design notes

context_files/ holds the design documents and an as-built record for every step of construction, each written to be authoritative over the ones before it. They are unusually candid: they record the defects found, the measurements taken, the tests that turned out to be vacuous, and the claims that did not survive contact with the code. Start with the newest.

Licence

MIT.

Documentation

Overview

Package ursus is a dataframe library for Go modelled on Polars: an expression DSL, lazy execution with a query optimizer, Arrow memory, and SIMD kernels.

Runnable examples for the common tasks — filter, group-by, join, reading values back out — are in example_test.go and render beside each method on pkg.go.dev.

A tour 

df, err := ursus.ScanParquet("sales.parquet").
	Filter(ursus.Col("price").Gt(5)).
	Select(ursus.Col("id"), ursus.Col("price")).
	Collect(ctx)

Nothing runs until Collect. In between, ursus resolves the schema, expands any multi-column selections, type-checks every expression, and pushes the projection down into the scan so only the columns actually used are ever read.

The design in ten lines

  1. Lazy is the real API. Eager helpers are thin wrappers over it.
  2. Public types are concrete structs — Go forbids generic methods on interfaces, and generic methods are the point.
  3. Errors are deferred and sticky, surfaced at Collect, so chaining stays clean.
  4. context.Context appears at execution boundaries only, never in builders.
  5. Named methods, never operator tricks: a.Gt(5), not a > 5.
  6. Functional options where the parameter surface is wide.
  7. Generics at the boundary where Go values meet columns, not in the engine.
  8. Iterators for streaming output.
  9. No row index: row order is a property, not a label space.
  10. Null is not NaN, and both are handled explicitly everywhere.

Building

ursus requires Go 1.27. GOEXPERIMENT=simd is optional: it switches on the SIMD kernels, and without it each kernel falls back to its scalar twin, which the CI runs the whole suite against.

Example

The shortest useful thing: build a frame and look at it. 

package main

import (
	"context"
	"fmt"
	"log"

	"github.com/advenn/ursus"
)

// exampleSales is the little frame the examples below work on.
func exampleSales() *ursus.LazyFrame {
	return ursus.Frame(
		ursus.Values("region", []string{"north", "south", "north", "east", "south"}),
		ursus.Values("product", []string{"apple", "apple", "pear", "pear", "fig"}),
		ursus.Values("qty", []int64{10, 4, 7, 2, 9}),
		ursus.Values("price", []float64{1.50, 1.50, 2.25, 2.25, 3.00}),
	)
}

func main() {
	df, err := exampleSales().Collect(context.Background())
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(df)
}

Output:
shape: (5, 4)
┌─────────┬─────────┬───────┬─────────┐
│ region  │ product │ qty   │ price   │
│ String  │ String  │ Int64 │ Float64 │
├─────────┼─────────┼───────┼─────────┤
│ "north" │ "apple" │ 10    │ 1.5     │
│ "south" │ "apple" │ 4     │ 1.5     │
│ "north" │ "pear"  │ 7     │ 2.25    │
│ "east"  │ "pear"  │ 2     │ 2.25    │
│ "south" │ "fig"   │ 9     │ 3       │
└─────────┴─────────┴───────┴─────────┘

Index

Examples

Constants

View Source 
const (
	// ClosedDefault is the zero value and means "this operation's own convention":
	// both for IsBetween, left for GroupByDynamic, right for Rolling.
	ClosedDefault = expr.ClosedDefault
	ClosedLeft    = expr.ClosedLeft
	ClosedRight   = expr.ClosedRight
	ClosedBoth    = expr.ClosedBoth
	ClosedNone    = expr.ClosedNone
)

The four window boundary conventions. ClosedLeft is the default for a window grid — an instant exactly on a boundary starts the new window — because it is the only one under which consecutive windows tile the line exactly.

View Source 
const (
	// InterpLinear interpolates proportionally between the neighbours. It is the
	// conventional default and the only option whose result need not be an input
	// value.
	InterpLinear = expr.InterpLinear

	// InterpLower and InterpHigher take the neighbour below or above the rank.
	InterpLower  = expr.InterpLower
	InterpHigher = expr.InterpHigher

	// InterpNearest takes the closer neighbour, rounding halves upward.
	InterpNearest = expr.InterpNearest

	// InterpMidpoint averages the two neighbours regardless of the rank's position
	// between them.
	InterpMidpoint = expr.InterpMidpoint
)
View Source 
const (
	// AsOfBackward is the default: the last right key at or before the left key.
	// "Attach the most recent quote to each trade."
	AsOfBackward = plan.AsOfBackward
	// AsOfForward is the first right key at or after the left key.
	AsOfForward = plan.AsOfForward
	// AsOfNearest is whichever is closer, ties going backward.
	AsOfNearest = plan.AsOfNearest
)
View Source 
const (
	// QuoteNecessary quotes a field holding the separator, the quote or a line
	// break, the empty string, and one beginning with a space: the default.
	QuoteNecessary = csv.QuoteNecessary
	// QuoteAlways quotes every field, the header's and a null's too.
	QuoteAlways = csv.QuoteAlways
	// QuoteNever quotes no field.
	QuoteNever = csv.QuoteNever
	// QuoteNonNumeric quotes every field but a number and a null.
	QuoteNonNumeric = csv.QuoteNonNumeric
)
View Source 
const (
	CSVUncompressed = csv.Uncompressed // the default
	CSVGzip         = csv.Gzip         // gzip, a path's name ending .gz
	CSVZstd         = csv.Zstd         // Zstandard, .zst
)
View Source 
const (
	// JoinInner keeps only matching pairs. The default.
	JoinInner = plan.JoinInner
	// JoinLeft keeps every left row, with right columns null where unmatched.
	JoinLeft = plan.JoinLeft
	// JoinRight is the mirror of JoinLeft.
	JoinRight = plan.JoinRight
	// JoinFull keeps every row from both sides.
	JoinFull = plan.JoinFull
	// JoinSemi keeps left rows that have at least one match. A filter: no right
	// column is added.
	JoinSemi = plan.JoinSemi
	// JoinAnti keeps left rows with no match. The inverse of JoinSemi.
	JoinAnti = plan.JoinAnti
	// JoinCross is the cartesian product. No keys.
	JoinCross = plan.JoinCross
)
View Source 
const (
	ValidateNone = plan.ValidateNone
	// ValidateOneToOne requires unique keys on both sides.
	ValidateOneToOne = plan.ValidateOneToOne
	// ValidateOneToMany requires unique keys on the LEFT.
	ValidateOneToMany = plan.ValidateOneToMany
	// ValidateManyToOne requires unique keys on the RIGHT. This is the one that
	// catches accidental fan-out, which is the most common analytics bug there is.
	ValidateManyToOne = plan.ValidateManyToOne
	// ValidateManyToMany imposes no constraint.
	ValidateManyToMany = plan.ValidateManyToMany
)
View Source 
const (
	// JoinOrderNone promises no order, Polars' default.
	JoinOrderNone = plan.JoinOrderNone
	// JoinOrderLeft keeps the left frame's order: each left row's matches, in the
	// right frame's order, and then a right or full join's unmatched right rows, in
	// theirs.
	JoinOrderLeft = plan.JoinOrderLeft
	// JoinOrderRight keeps the right frame's order: each right row's matches, in the
	// left frame's order, and then a left or full join's unmatched left rows.
	JoinOrderRight = plan.JoinOrderRight
	// JoinOrderLeftRight is JoinOrderLeft. Polars' left promises less, the order of
	// a row's matches; here both keep it.
	JoinOrderLeftRight = plan.JoinOrderLeftRight
	// JoinOrderRightLeft is JoinOrderRight, as JoinOrderLeftRight is JoinOrderLeft.
	JoinOrderRightLeft = plan.JoinOrderRightLeft
)
View Source 
const (
	Second = dtype.Second
	Milli  = dtype.Milli
	Micro  = dtype.Micro
	Nano   = dtype.Nano
)

Time resolutions.

View Source 
const (
	// MapGroupsToRows gives every row its own partition's value, in the original
	// row order. The default, and the only one that composes with other columns.
	MapGroupsToRows = expr.MapGroupsToRows

	// MapExplode would leave the output in partition order instead of permuting it
	// back, Polars' mapping_strategy="explode". Refused while the plan is built: it
	// reorders the frame to partition-major order, which no operator produces yet.
	MapExplode = expr.MapExplode

	// MapJoin would aggregate each partition into a List repeated across its rows.
	// Refused, because it is a synonym: Col(...).Implode().Over(...) is exactly
	// this under the default mapping.
	MapJoin = expr.MapJoin
)
View Source 
const (
	// RankOrdinal gives every row a distinct rank, ties broken by input order.
	RankOrdinal = expr.RankOrdinal
	// RankDense gives tied rows the same rank and does not skip the next value:
	// [10,20,20,30] ranks as 1,2,2,3.
	RankDense = expr.RankDense
	// RankMin and RankMax give tied rows the lowest or highest of their positions:
	// [10,20,20,30] ranks as 1,2,2,4 and 1,3,3,4.
	RankMin = expr.RankMin
	RankMax = expr.RankMax
	// RankAverage gives tied rows the mean of their positions, so it returns
	// Float64 where every other method returns Uint32.
	RankAverage = expr.RankAverage
)

Variables

View Source 
var (
	Bool    = dtype.Bool
	Int8    = dtype.Int8
	Int16   = dtype.Int16
	Int32   = dtype.Int32
	Int64   = dtype.Int64
	Uint8   = dtype.Uint8
	Uint16  = dtype.Uint16
	Uint32  = dtype.Uint32
	Uint64  = dtype.Uint64
	Float32 = dtype.Float32
	Float64 = dtype.Float64
	// Int128 is the accumulator and output type of integer Sum. Widening to 128
	// bits is what makes an integer sum incapable of silently overflowing.
	Int128 = dtype.Int128
	String = dtype.String
	Binary = dtype.Binary
	Date   = dtype.Date
	NullT  = dtype.Null
)

Simple types.

View Source 
var (
	// ErrSchema is an unknown, duplicate or ambiguous column — or several files,
	// read as one, whose columns differ in anything but their order.
	ErrSchema = uerr.ErrSchema
	// ErrType is an operation not defined for the given types.
	ErrType = uerr.ErrType
	// ErrValue is a bad literal, an out-of-range cast, an unparseable pattern.
	ErrValue = uerr.ErrValue
	// ErrUnsupported is a well-formed request ursus cannot yet serve.
	ErrUnsupported = uerr.ErrUnsupported
	// ErrIO is a failure reading or writing a data source.
	ErrIO = uerr.ErrIO
	// ErrResource is a limit ursus refused to exceed: the query would have worked
	// with a larger WithMemoryLimit or a writable WithSpillDir.
	ErrResource = uerr.ErrResource
	// ErrInternal is a bug in ursus. Users should never legitimately see one.
	ErrInternal = uerr.ErrInternal
)

Error kinds, for branching on a failure without matching on its message. 

if errors.Is(err, ursus.ErrResource) { retry with a bigger memory limit }

These are re-exported because the sentinels themselves live under internal/, so no package outside this module could reach them — which made "a caller should be able to detect that without matching on a message" true only for callers inside the module.

View Source 
var (
	Datetime = dtype.Datetime
	Duration = dtype.Duration
	TimeOf   = dtype.Time
	Decimal  = dtype.Decimal
	List     = dtype.List
	Array    = dtype.Array
	Enum     = dtype.Enum
)

Parameterised type constructors.

View Source 
var (
	Of      = dtype.Of
	NotNull = dtype.NotNull
)

Of builds a nullable field; NotNull builds a non-nullable one.

View Source 
var Categorical = dtype.Categorical

Categorical is text with a type of its own, Polars' Categorical: ordered as its text, its codes the order its values first appear in (ToPhysical, Cat().Physical()). New in 0.6.7.

Functions

func LeaveProcessMemoryLimit added in v0.4.0 

func LeaveProcessMemoryLimit()

LeaveProcessMemoryLimit keeps ursus from setting the Go runtime's soft memory limit when a query first runs under the default budget. Call it before the first query, in a program that manages the limit itself, or wants none and cannot set GOMEMLIMIT=off.

func SetProcessMemoryLimit added in v0.4.0 

func SetProcessMemoryLimit() int64

SetProcessMemoryLimit sets the Go runtime's soft memory limit — the one GOMEMLIMIT sets — to nine tenths of the memory this process may use: the smaller of its cgroup's limit and the machine's RAM, as the default query budget reads them. It returns the limit in effect afterwards, in bytes; math.MaxInt64 means none.

ursus calls it itself, once, the first time a query runs under the default budget (step 128). Calling it at start-up does the same thing earlier.

Why

The default budget is half the ceiling (WithMemoryLimit says why): ursus's buffers live on the Go heap, and with GOGC's default the heap grows to about twice what is live before the collector runs. Half leaves room for that, and it is still a race — a query holding its whole budget can take the heap to the ceiling. A soft limit near the ceiling makes the collector run harder as the heap approaches it, instead of letting the kernel kill the process. Measured (step 97): a 300 MB live set, with batches built and dropped around it, peaked at about 650 MB of runtime memory under the defaults and at about 440 MB under a 450 MB soft limit.

Why it is automatic

The limit is the whole process's, not ursus's: it changes how the program around ursus collects garbage too, which is why step 97 left it to the caller. Measured since, that was the wrong trade. The budget counts what the operators hold, and the heap ran to about twice it: under an 8 GB cgroup, h2o's six-key group-by peaked at 7.6 GB against a 4 GB budget, and under a 4 GB cgroup a join of two ten-million- row tables was killed at the cap. A program that never called this, which is most of them, had a default budget that did not hold.

So it is set where ursus already decides how much memory the process may use: when a query runs under the default budget, which ursus derives from that same ceiling. Not on import, and not under a limit the caller gave WithMemoryLimit.

To keep ursus from setting it, set GOMEMLIMIT, "off" included, or call LeaveProcessMemoryLimit before the first query.

What it leaves alone

A limit already chosen — GOMEMLIMIT in the environment, "off" included, or an earlier debug.SetMemoryLimit — is kept, and returned. Where no ceiling can be read (anywhere but Linux, macOS and Windows) nothing is set.

Types

type Ambiguous added in v0.6.0 

type Ambiguous string

Ambiguous answers a wall clock that ReplaceTimeZone's new zone reads twice, in the hour its clocks fall back: Polars' ambiguous. 

const (
	AmbiguousRaise    Ambiguous = expr.ZoneRaise    // refuse the query; the default
	AmbiguousEarliest Ambiguous = expr.ZoneEarliest // the first reading, before the clocks fell back
	AmbiguousLatest   Ambiguous = expr.ZoneLatest   // the second reading, after
	AmbiguousNull     Ambiguous = expr.ZoneNull     // a null
)

type ArrExpr added in v0.6.7 

type ArrExpr struct {
	// contains filtered or unexported fields
}

ArrExpr is the array namespace, Polars' arr: `Col("xyz").Arr().Sum()`.

An Array is a fixed-size list: Array(Int64, 3) holds three Int64s in each row, or is null, and casts to and from a List of its elements, a List's only where each of its rows holds that many (List().ToArray, Cast). It is read from and written to Arrow IPC as a FixedSizeList, and to Parquet as a list whose size the footer's Arrow schema keeps, as Polars writes it. It is a key, as a List is (step 344).

Its functions are a List's of the same elements, and answer as List()'s do, nulls and all: a function of each array's values gives a value, and one that reshapes it gives a List, but Reverse, Sort, Shift and Eval of an element's value, which keep each array's length, keep its Array. ToStruct and Dot read its size.

New in 0.6.7.

func (ArrExpr) Agg added in v0.6.7 

func (a ArrExpr) Agg(body Expr) Expr

Agg runs body over each array's elements, an aggregate's value its answer, as List().Agg.

func (ArrExpr) All added in v0.6.7 

func (a ArrExpr) All(ignoreNulls bool) Expr

func (ArrExpr) Any added in v0.6.7 

func (a ArrExpr) Any(ignoreNulls bool) Expr

Any and All are each Boolean array's or and and, nulls passed over with ignoreNulls, as List()'s.

func (ArrExpr) ArgMax added in v0.6.7 

func (a ArrExpr) ArgMax() Expr

func (ArrExpr) ArgMin added in v0.6.7 

func (a ArrExpr) ArgMin() Expr

ArgMin and ArgMax are the position in each array of its least or greatest value.

func (ArrExpr) Contains added in v0.6.7 

func (a ArrExpr) Contains(v any) Expr

Contains reports whether each array holds v, CountMatches how many times; v may be an expression, one a row. See List().Contains.

func (ArrExpr) CountMatches added in v0.6.7 

func (a ArrExpr) CountMatches(v any) Expr

func (ArrExpr) Dot added in v0.6.7 

func (a ArrExpr) Dot(other Expr) Expr

Dot is each row's sum of this array's and other's products, element by element, of two Arrays of numbers of one size: Polars' arr.dot. A null element's product is passed over, and a row is null where either array is. A sum of integers is an Int128, as Sum's.

func (ArrExpr) Eval added in v0.6.7 

func (a ArrExpr) Eval(body Expr, asList bool) Expr

Eval runs body over each array's elements, the array its group, as List().Eval: a body of a value an element, Element().Mul(2), keeps the array's length, and so its Array. With asList the answer is a List, any body's, as Polars' eval(as_list=True); without, a body that changes the length is refused, as Polars refuses it.

func (ArrExpr) Explode added in v0.6.7 

func (a ArrExpr) Explode() Expr

Explode is each array's elements, a row each, a null array one null row.

func (ArrExpr) First added in v0.6.7 

func (a ArrExpr) First() Expr

First and Last are each array's first and last element, Get the i-th, from the end where i is negative, null past either end. i may be an expression, one a row.

func (ArrExpr) Get added in v0.6.7 

func (a ArrExpr) Get[I int | Expr](i I) Expr

func (ArrExpr) Head added in v0.6.7 

func (a ArrExpr) Head(n int) Expr

Head, Tail and Slice are each array's first n, last n, or length from offset, as a List, Polars' default: List().ToArray(n) makes one an Array again.

func (ArrExpr) Join added in v0.6.7 

func (a ArrExpr) Join(sep string) Expr

Join joins each array's strings with sep, passing nulls over, as List().Join.

func (ArrExpr) Last added in v0.6.7 

func (a ArrExpr) Last() Expr

func (ArrExpr) Len added in v0.6.7 

func (a ArrExpr) Len() Expr

Len is each array's size, null where it is null.

func (ArrExpr) Max added in v0.6.7 

func (a ArrExpr) Max() Expr

func (ArrExpr) Mean added in v0.6.7 

func (a ArrExpr) Mean() Expr

func (ArrExpr) Median added in v0.6.7 

func (a ArrExpr) Median() Expr

func (ArrExpr) Min added in v0.6.7 

func (a ArrExpr) Min() Expr

Min, Max, Sum, Mean and Median reduce each array's values as List()'s do: a null element is passed over, and a sum of integers is an Int128.

func (ArrExpr) NUnique added in v0.6.7 

func (a ArrExpr) NUnique() Expr

NUnique is each array's count of distinct values, a null among them.

func (ArrExpr) Reverse added in v0.6.7 

func (a ArrExpr) Reverse() Expr

Reverse is each array's elements last first, Sort ascending and SortDesc descending, nulls first, and SortWith either way with nulls first or last: an Array of its size, as Polars keeps it.

func (ArrExpr) Shift added in v0.6.7 

func (a ArrExpr) Shift[N Operand](n N) Expr

Shift moves each array's elements n places later, nulls in the places they leave, earlier where n is negative: an Array of its size. n may be an expression, one a row; a null n is a null array.

func (ArrExpr) Slice added in v0.6.7 

func (a ArrExpr) Slice(offset, length int) Expr

func (ArrExpr) Sort added in v0.6.7 

func (a ArrExpr) Sort() Expr

func (ArrExpr) SortDesc added in v0.6.7 

func (a ArrExpr) SortDesc() Expr

func (ArrExpr) SortWith added in v0.6.7 

func (a ArrExpr) SortWith(descending, nullsLast bool) Expr

func (ArrExpr) Std added in v0.6.7 

func (a ArrExpr) Std(ddof int) Expr

Std and Var are each array's standard deviation and variance with ddof, Polars' arr.std and arr.var, whose ddof is 1 unless given.

func (ArrExpr) Sum added in v0.6.7 

func (a ArrExpr) Sum() Expr

func (ArrExpr) Tail added in v0.6.7 

func (a ArrExpr) Tail(n int) Expr

func (ArrExpr) ToList added in v0.6.7 

func (a ArrExpr) ToList() Expr

ToList is each array as a List of its elements, Polars' arr.to_list.

func (ArrExpr) ToStruct added in v0.6.7 

func (a ArrExpr) ToStruct(names ...string) Expr

ToStruct makes each array a Struct, a field an element, named by names, as many as its size, or field_0 on without them, as Polars names them. A null array is a null struct, where Polars gives a struct of nulls, as List().ToStruct.

func (ArrExpr) Unique added in v0.6.7 

func (a ArrExpr) Unique() Expr

Unique is each array's distinct values, each first where it first appears, as a List.

func (ArrExpr) Var added in v0.6.7 

func (a ArrExpr) Var(ddof int) Expr

type AsOfOption 

type AsOfOption func(*asOfCfg)

AsOfOption configures JoinAsOf.

func AsOfAllowExactMatches 

func AsOfAllowExactMatches(b bool) AsOfOption

AsOfAllowExactMatches decides whether a right key EQUAL to the left key may match. Default true.

func AsOfBy 

func AsOfBy(exprs ...Expr) AsOfOption

AsOfBy adds EXACT-match keys applied before the nearest-key search, so a trade matches only quotes for its own symbol.

Without it every left row searches one global run, which on any real fixture means matching the nearest row of the wrong instrument — a plausible number and the wrong one.

A null by-key matches nothing: a left row with one gets nulls, and a right row with one is never found — as in Polars and DuckDB, and as a null key does in every other ursus join unless NullsEqual says otherwise. By-keys of different types are compared at the type that holds both exactly, as a join's keys are.

func AsOfCoalesce added in v0.6.7 

func AsOfCoalesce(b bool) AsOfOption

AsOfCoalesce says whether the ordering key is merged into one column, as Polars' coalesce: true, the default, merges it where both sides name it alike; false keeps the right one too, suffixed where the names collide, `ts` and `ts_right`. Keys named apart are both kept either way, and the by-keys are always merged, as in Polars (step 315).

func AsOfLeftBy 

func AsOfLeftBy(exprs ...Expr) AsOfOption

AsOfLeftBy and AsOfRightBy are AsOfBy for differently-named columns.

func AsOfLeftOn 

func AsOfLeftOn(e Expr) AsOfOption

AsOfLeftOn and AsOfRightOn name differently-named ordering keys. Used together.

func AsOfOn 

func AsOfOn(e Expr) AsOfOption

AsOfOn names the ordering key, present under the same name on both sides.

func AsOfRightBy 

func AsOfRightBy(exprs ...Expr) AsOfOption

func AsOfRightOn 

func AsOfRightOn(e Expr) AsOfOption

func AsOfStrategyOpt 

func AsOfStrategyOpt(s AsOfStrategy) AsOfOption

AsOfStrategyOpt picks backward (the default), forward or nearest.

func AsOfSuffix 

func AsOfSuffix(s string) AsOfOption

AsOfSuffix renames a colliding right column. Defaults to Join's suffix.

func AsOfTolerance 

func AsOfTolerance[T AsOfToleranceValue](v T) AsOfOption

AsOfTolerance bounds how far the search may reach. A candidate further than this from the left key does not match, and the left row comes back null-padded.

A temporal key takes an interval or a time.Duration, and a numeric key a number, its distance, as Polars' tolerance: AsOfTolerance(5) on an integer key admits a key five away, AsOfTolerance(0.5) on a float key half a unit (step 315). A tolerance of zero admits only an exact match, as Polars' does; before step 315 an interval of zero admitted any distance.

The bound is recomputed per row for a CALENDAR interval, so Every("1mo") is a different distance in February than in March — which is the whole reason Interval is not a duration.

type AsOfStrategy 

type AsOfStrategy = plan.AsOfStrategy

AsOfStrategy picks which neighbouring right row a left row matches.

type AsOfToleranceValue added in v0.6.7 

type AsOfToleranceValue interface {
	Interval | time.Duration | int | int64 | float64
}

AsOfToleranceValue is what AsOfTolerance takes: an Interval or a time.Duration for a temporal key, a number for a numeric one.

type BinExpr added in v0.6.6 

type BinExpr struct {
	// contains filtered or unexported fields
}

BinExpr is the binary namespace, Polars' .bin, on a Binary column: `Col("payload").Bin().Head(4)`. Every function counts bytes. A String is refused, as Polars refuses one; .Cast(ursus.Binary) is its text's bytes.

func (BinExpr) Contains added in v0.6.6 

func (b BinExpr) Contains[P []byte | string | Expr](pattern P) Expr

Contains, StartsWith and EndsWith are whether each value holds, starts or ends with a pattern: bytes, text, or an expression of either, a row's own; a null pattern is null. Polars' bin.contains, starts_with and ends_with.

func (BinExpr) Decode added in v0.6.6 

func (b BinExpr) Decode(enc Encoding, strict bool) Expr

func (BinExpr) Encode added in v0.6.6 

func (b BinExpr) Encode(enc Encoding) Expr

Encode is each value's bytes as lowercase hex or padded base64, a String; Decode is each value's hex or base64 text as its bytes, a Binary, refused where it is not of the encoding, or with strict false null. Polars' bin.encode and bin.decode.

func (BinExpr) EndsWith added in v0.6.6 

func (b BinExpr) EndsWith[P []byte | string | Expr](suffix P) Expr

func (BinExpr) Get added in v0.6.6 

func (b BinExpr) Get[I int | Expr](index I, nullOnOOB bool) Expr

Get is each value's byte at index, a constant or an expression, from the end where negative, a UInt8: Polars' bin.get. An index past a value is refused, or with nullOnOOB null.

func (BinExpr) Head added in v0.6.6 

func (b BinExpr) Head[N int | Expr](n N) Expr

Head is each value's first n bytes and Tail its last, where n is negative all but |n| from the other end: Polars' bin.head and bin.tail. n is a constant or an expression, whose null is null.

func (BinExpr) Reinterpret added in v0.6.6 

func (b BinExpr) Reinterpret(dt DataType, endianness Endianness) Expr

Reinterpret reads each value's bytes as one value of dt: an integer, a float, a Date, a Datetime, a Duration or a Time, in the byte order given. A value of other than dt's width is null. Polars' bin.reinterpret.

func (BinExpr) Size added in v0.6.6 

func (b BinExpr) Size(unit SizeUnit) Expr

Size is each value's length: in bytes, a UInt32, or in kilobytes to terabytes of 1024, a Float64. Polars' bin.size.

func (BinExpr) Slice added in v0.6.6 

func (b BinExpr) Slice[O int | Expr](offset O, length ...any) Expr

Slice is length bytes of each value from offset, which counts from the end where negative; with no length, or a null one, the rest: Polars' bin.slice. Each is an int or an Expr. An offset before the start shortens the length by as much, as Polars has it: Slice(-6, 3) of "hello" is "he".

func (BinExpr) StartsWith added in v0.6.6 

func (b BinExpr) StartsWith[P []byte | string | Expr](prefix P) Expr

func (BinExpr) Tail added in v0.6.6 

func (b BinExpr) Tail[N int | Expr](n N) Expr

type BusinessCalendar added in v0.6.6 

type BusinessCalendar struct {
	// WeekMask holds the days of the week that are business days, Monday first. All
	// false means Monday to Friday, Polars' default.
	WeekMask [7]bool
	// Holidays are dates that are not business days, read as their UTC date.
	Holidays []time.Time
}

BusinessCalendar is what a business day is for BusinessDayCount, and for the business-day functions of .dt: Polars' week_mask and holidays.

type CSVCompression added in v0.6.7 

type CSVCompression = csv.Compression

CSVCompression is the codec a CSV file is written in, as Polars' write_csv compression. New in 0.6.7.

type CSVEncoding added in v0.6.7 

type CSVEncoding int

CSVEncoding is how a CSV file's text is read, as Polars' read_csv encoding. 

const (
	// UTF8 refuses a String's text that is not UTF-8, as Polars does: the default. A
	// column's name is read as UTF8Lossy reads it either way, as Polars reads one.
	UTF8 CSVEncoding = iota
	// UTF8Lossy reads each invalid sequence as U+FFFD, as Polars' "utf8-lossy".
	UTF8Lossy
)

type CSVFile added in v0.3.0 

type CSVFile struct {
	// Name identifies the file in errors and in Explain.
	Name string

	// Open returns the file's bytes from the start.
	//
	// It is called more than once per query — once to infer the schema, once for
	// each execution — and must return a fresh stream each time. ctx is the query's
	// and outlives the stream, which ursus closes when it is done with it.
	Open func(ctx context.Context) (io.ReadCloser, error)
}

CSVFile is one CSV file, read as a stream from wherever it lives — an object store, an HTTP server, an archive — by code that knows how to reach it.

type CSVOption 

type CSVOption func(*csv.Options)

CSVOption configures ScanCSV.

func WithCSVIncludeFilePaths added in v0.6.7 

func WithCSVIncludeFilePaths(name string) CSVOption

WithCSVIncludeFilePaths adds a String column named name of each row's file path, after the files' columns, as Polars' include_file_paths: the path as it was given, or a glob's or a directory's as it found it; a reader's, its name. New in 0.6.7.

func WithColumnNames 

func WithColumnNames(names ...string) CSVOption

WithColumnNames overrides the column names positionally.

func WithComment 

func WithComment(prefix string) CSVOption

WithComment makes lines starting with prefix comments. Multi-byte prefixes such as "//" are supported.

func WithDecimalComma added in v0.6.7 

func WithDecimalComma(b bool) CSVOption

WithDecimalComma reads a float's and a Decimal's point as a comma, 1,5, as Polars' read_csv(decimal_comma=True), inferred so too: a value with a point is then text, and under a comma separator a number with a comma is quoted. New in 0.6.7.

func WithEncoding added in v0.6.7 

func WithEncoding(e CSVEncoding) CSVOption

WithEncoding sets how text is read: UTF8, the default, or UTF8Lossy. New in 0.6.7.

func WithHasHeader 

func WithHasHeader(b bool) CSVOption

WithHasHeader says whether the first record holds column names. Default true.

func WithIgnoreErrors added in v0.6.7 

func WithIgnoreErrors(b bool) CSVOption

WithIgnoreErrors reads a value its column's type cannot take as a null, as Polars' ignore_errors. A record of too many fields is refused still, and text that is not UTF-8, as Polars refuses them. New in 0.6.7.

func WithInferRows 

func WithInferRows(n int) CSVOption

WithInferRows bounds how many records inference reads. 0 reads the whole file, which is exact and, on a large file, expensive. Default 100.

func WithInferSchema added in v0.6.7 

func WithInferSchema(b bool) CSVOption

WithInferSchema(false) reads every column as a String, but those a schema or an override types, as Polars' infer_schema=False. New in 0.6.7.

func WithMaxRecordSize 

func WithMaxRecordSize(n int) CSVOption

WithMaxRecordSize bounds a single record. Default 16 MiB. The bound exists so an unterminated quote is an error rather than an out-of-memory kill.

func WithNullValues 

func WithNullValues(vals ...string) CSVOption

WithNullValues adds texts that mean NULL. An empty field already means null for every type; in a String column a quoted "" is the empty string, as Polars reads it, so the two stay apart.

func WithQuote 

func WithQuote(c byte) CSVOption

WithQuote sets the quote character. Default '"'.

func WithQuoting added in v0.6.7 

func WithQuoting(b bool) CSVOption

WithQuoting(false) reads a quote as any byte, as Polars' quote_char=None: no field is quoted, so a separator or a line end always ends one. New in 0.6.7.

func WithRaiseIfEmpty added in v0.6.7 

func WithRaiseIfEmpty(b bool) CSVOption

WithRaiseIfEmpty(false) reads an empty file as a frame of no columns and no rows, as Polars' raise_if_empty=False, where by default it is refused. New in 0.6.7.

func WithSchema 

func WithSchema(s *Schema) CSVOption

WithSchema supplies the schema and skips inference entirely. On a large file this is the option that matters: inference costs a read of the sample, and an explicit schema also removes the risk of a column being typed from rows that do not represent it.

func WithSchemaOverrides 

func WithSchemaOverrides(m map[string]dtype.DataType) CSVOption

WithSchemaOverrides fixes individual columns while inferring the rest. The usual case is an identity column that looks numeric and must not be.

func WithSeparator 

func WithSeparator(c byte) CSVOption

WithSeparator sets the field delimiter. Default ','.

func WithSkipLines added in v0.6.7 

func WithSkipLines(n int) CSVOption

WithSkipLines discards n lines before the header, by their line ends alone, a quote in them any byte, as Polars' skip_lines; WithSkipRows skips records. The two are not both set, as Polars refuses them. New in 0.6.7.

func WithSkipRows 

func WithSkipRows(n int) CSVOption

WithSkipRows discards n records before the header.

func WithSkipRowsAfterHeader added in v0.6.7 

func WithSkipRowsAfterHeader(n int) CSVOption

WithSkipRowsAfterHeader discards n records after the header, or the first n where there is none, a blank line one, as Polars' skip_rows_after_header. New in 0.6.7.

func WithTruncateRaggedLines 

func WithTruncateRaggedLines(b bool) CSVOption

WithTruncateRaggedLines accepts a record of too many fields, the extra ones dropped, as Polars' truncate_ragged_lines; by default it is refused. A record of too few is padded with nulls either way, as Polars pads one.

func WithTryParseDates added in v0.6.7 

func WithTryParseDates(b bool) CSVOption

WithTryParseDates infers a column of dates, instants or times of day as a Date, a Datetime in microseconds, or a Time, as Polars' try_parse_dates: 2024-01-02 and 02/01/2024 dates, the year first or the day, split by '-', '/' or '.'; 2024-01-02 03:04:05.123 instants, a 'T' or a space between, and in UTC where they have an offset, Z, +09:00 or +0900; and 03:04:05 times. A month first is not read: 01/02/2024 is the first of February. A date beside an instant, or an instant in a zone beside one in none, is text. Off by default, as Polars'. A Date, Datetime or Time column a schema gives is read by the same patterns. New in 0.6.7.

type CSVSinkOption 

type CSVSinkOption interface {
	// contains filtered or unexported methods
}

CSVSinkOption is anything SinkCSV and WriteCSV accept: a writer option or an execution option. See ParquetSinkOption for why it is an interface.

type CSVWriteOption 

type CSVWriteOption func(*csv.WriteOptions)

CSVWriteOption configures the CSV writer.

func WithCSVCheckExtension added in v0.6.7 

func WithCSVCheckExtension(b bool) CSVWriteOption

WithCSVCheckExtension says whether SinkCSV checks its path's extension against the compression, as Polars' check_extension. Default true. New in 0.6.7.

func WithCSVCompression added in v0.6.7 

func WithCSVCompression(c CSVCompression) CSVWriteOption

WithCSVCompression writes the text compressed, at the codec's default level, as Polars' compression; ScanCSV reads it back by its first bytes. SinkCSV's path must end as the codec's files do, .gz or .zst, and an uncompressed one must not, as Polars checks unless WithCSVCheckExtension(false) says otherwise. New in 0.6.7.

func WithCSVCompressionLevel added in v0.6.7 

func WithCSVCompressionLevel(level int) CSVWriteOption

WithCSVCompressionLevel sets the codec's level, as Polars' compression_level: gzip's 0 to 9, where the default is 6, and Zstandard's 1 to 22, where it is 3. New in 0.6.7.

func WithDateFormat added in v0.6.7 

func WithDateFormat(f string) CSVWriteOption

WithDateFormat writes a Date in a strftime format, "%d/%m/%Y", as Polars' date_format, where the default is 2024-01-02. A format of a time of day is refused. New in 0.6.7.

func WithDatetimeFormat added in v0.6.7 

func WithDatetimeFormat(f string) CSVWriteOption

WithDatetimeFormat writes a Datetime in a strftime format, "%Y-%m-%d %H:%M", as Polars' datetime_format, a zoned one on its zone's wall clock. The default is 2024-01-02T03:04:05.123456, to the instant's unit, with its offset, +0900, where it has a zone. A naive instant's format of a zone, %z or %Z, is refused. New in 0.6.7.

func WithFloatPrecision added in v0.6.7 

func WithFloatPrecision(n int) CSVWriteOption

WithFloatPrecision writes every float with n digits after its point, as Polars' float_precision: 2 writes 1.5 as 1.50. New in 0.6.7.

func WithFloatScientific added in v0.6.7 

func WithFloatScientific(b bool) CSVWriteOption

WithFloatScientific writes every float in scientific notation, 1.5e0 and 1e-7, or with false none, 100000000000000000000, as Polars' float_scientific; by default a float is scientific past 1e16 and below 1e-5. New in 0.6.7.

func WithIncludeBOM added in v0.6.7 

func WithIncludeBOM(b bool) CSVWriteOption

WithIncludeBOM begins the file with a UTF-8 byte-order mark, as Polars' include_bom, which ScanCSV skips. New in 0.6.7.

func WithLineTerminator 

func WithLineTerminator(s string) CSVWriteOption

WithLineTerminator sets the record separator. Default "\n".

func WithNullValue 

func WithNullValue(s string) CSVWriteOption

WithNullValue sets the text written for a null. Default "".

The default reads back as null for every type. An empty String is written as a quoted "", which reads back as the empty string, so null and empty strings both round-trip — as Polars writes and reads them.

func WithQuoteStyle added in v0.6.7 

func WithQuoteStyle(s QuoteStyle) CSVWriteOption

WithQuoteStyle sets when a field is quoted, as Polars' quote_style. New in 0.6.7.

func WithTimeFormat added in v0.6.7 

func WithTimeFormat(f string) CSVWriteOption

WithTimeFormat writes a Time in a strftime format, "%H:%M", as Polars' time_format, where the default is 03:04:05.000000000. New in 0.6.7.

func WithWriteDecimalComma added in v0.6.7 

func WithWriteDecimalComma(b bool) CSVWriteOption

WithWriteDecimalComma writes a float's and a Decimal's point as a comma, 1,5, as Polars' write_csv(decimal_comma=True); under a comma separator such a field is quoted. New in 0.6.7.

func WithWriteHeader 

func WithWriteHeader(b bool) CSVWriteOption

WithWriteHeader says whether to write column names first. Default true.

func WithWriteQuote added in v0.6.7 

func WithWriteQuote(c byte) CSVWriteOption

WithWriteQuote sets the quote, as Polars' quote_char. Default '"'. New in 0.6.7.

func WithWriteSeparator 

func WithWriteSeparator(c byte) CSVWriteOption

WithWriteSeparator sets the delimiter. Default ','.

type CastOption added in v0.6.7 

type CastOption func(*castCfg)

CastOption configures a frame's Cast, CastTypes and CastAll.

func CastStrict added in v0.6.7 

func CastStrict(strict bool) CastOption

CastStrict says what a value that does not convert becomes: an error naming it, by default and with true, as Expr.Cast answers; a null with false, as Expr.CastLossy does. Polars' strict.

type CatExpr added in v0.6.6 

type CatExpr struct {
	// contains filtered or unexported fields
}

CatExpr is the categorical namespace, Polars' .cat, on an Enum, the categorical type ursus has: `Col("fruit").Cat().LenChars()`.

The text functions read each value as its category's text, as Str() reads an Enum, and each takes a String too; Polars' take only an Enum or a Categorical.

func (CatExpr) EndsWith added in v0.6.6 

func (c CatExpr) EndsWith(suffix string) Expr

func (CatExpr) GetCategories added in v0.6.6 

func (c CatExpr) GetCategories() Expr

GetCategories is an Enum's categories, in their order, a String column named as the expression: a column of its own length in a selection and a list a group in an aggregation, as Polars' get_categories is.

func (CatExpr) LenBytes added in v0.6.6 

func (c CatExpr) LenBytes() Expr

LenBytes and LenChars are each value's text's length in bytes and in characters, a UInt32.

func (CatExpr) LenChars added in v0.6.6 

func (c CatExpr) LenChars() Expr

func (CatExpr) Physical added in v0.6.6 

func (c CatExpr) Physical() Expr

Physical is each value's index in its Enum's categories, at Polars' width: UInt8 up to 255 categories, UInt16 up to 65,535, and UInt32 past them.

func (CatExpr) Slice added in v0.6.6 

func (c CatExpr) Slice(offset int, length ...int) Expr

Slice is length characters of each value's text from offset, which counts from the end where negative; with no length, the rest: Polars' cat.slice, a String.

func (CatExpr) StartsWith added in v0.6.6 

func (c CatExpr) StartsWith(prefix string) Expr

StartsWith and EndsWith are whether each value's text starts or ends with a constant: Polars' cat.starts_with and cat.ends_with, which take no expression.

type Closed 

type Closed = expr.Closed

Closed says which endpoints of an interval belong to it: [lo, hi), (lo, hi], [lo, hi] or (lo, hi). Used by IsBetween and by the temporal group-by's windows.

type CollectOption 

type CollectOption func(*collectCfg)

CollectOption configures execution.

func WithBatchSize 

func WithBatchSize(n int) CollectOption

WithBatchSize sets the rows per batch. Results must not depend on it; the test suite varies it precisely to check that.

func WithMemoryLimit 

func WithMemoryLimit(bytes int64) CollectOption

WithMemoryLimit caps what buffering operators may hold, in bytes. 

df, err := ursus.ScanParquet("120gb/*.parquet").
    Sort(ursus.Desc(ursus.Col("ts"))).
    SinkParquet(ctx, "sorted.parquet",
        ursus.WithMemoryLimit(512<<20),
        ursus.WithSpillDir("/tmp/ursus"))

What it bounds

Sort spills past the limit and merges the runs back, so a sort over more data than memory works. Group-by spills too, by radix-partitioning the keys it cannot hold: past the limit new keys are routed to one of sixteen files per level and re-aggregated afterwards. Join partitions BOTH sides on the join key and replays the pairs, so an equi-join over more data than memory works as well. Unique routes the rows whose keys it cannot hold the same way, and a window with a partition_by routes whole partitions; both stamp each row with its input position and merge back on it, so their answers keep their order exactly.

Reverse writes what it holds to a file past the limit, and gives the files back last to first, each read whole, so it holds one at a time. Tail writes the last rows it holds to files too, drops the oldest as later rows cover them, and gives them back in order.

Hstack pairs its frames a batch at a time, holding one of each.

A window with no partition_by that computes a rank, an ewm function or an interpolation cannot spill, since those need the whole frame, nor can one whose single partition is larger than the limit: past the limit it FAILS, with an error naming the operator, which is a better outcome than being killed by the OS with no explanation. With only aggregates, cumulative functions, shifts, limited fills and rolling windows of rows, such a frame or partition is written to one file and read back. JoinAsOf and MergeSorted stream both sides, holding a few batches; a forward or nearest JoinAsOf also holds the right rows it reads ahead, which for a left by-key the right side lacks is the rest of that side. Rolling and GroupByDynamic stream their input; with no key they hand their windows out as they close, a batch's worth at a time, and with a key they hold their answer, which they give per bucket.

A bounded sort — one under a Head or a TopK — never spills at all: past the limit it discards everything outside the current best k, which is exact.

The one thing the limit CAN change

An unordered group-by's ROW ORDER depends on the limit, because the point at which it stops admitting new keys does. Past that point a new key goes to a partition file and comes back after every resident group, so the output is resident-first then partition-major. A spilling JOIN is the same, for the same reason: rows whose key stayed in memory come out in probe order, and the rest follow bucket by bucket.

The CONTENT is invariant: same rows, same values, at every limit and at none. Only the sequence moves. A group-by can opt out with GroupBy.MaintainOrder, which restores first-appearance order exactly, spilled or not.

A JOIN CANNOT, and that is a decision rather than an omission. Restoring probe order means holding the whole output to permute it, and a join's output can be larger than both of its inputs — so the flag would only work in the cases that did not need it. TestMemoryLimitIsNotASemanticKnob pins the content half for both, and TestJoinOrderIsUnspecifiedUnderALimit pins that the order really does move.

What a group-by's limit does NOT bound

Partitioning divides the KEY SPACE, so it bounds a group-by whose problem is cardinality. It cannot divide a single key. What a key holds per VALUE rather than per group, a quantile's, a median's, n_unique's or a Spearman's, goes to disk of its own (steps 220, 241 and 339); implode, whose answer is every value, does not — so a group-by imploding a few very large groups still fails, with an error that says which of the two situations it is in.

The join has the same wall in two places: a join with no key and a residual, a WhereExists of a predicate alone, has nothing to divide, and one join key with more build rows than the limit cannot be split however deep the recursion goes. Both refuse with a message naming which it is. A plain CROSS join has no key either, and needs none: past the limit its build side goes to disk whole, and each probe row reads it back. A window has the wall too: one partition with more rows than the limit refuses, saying so.

Without it

A query with no WithMemoryLimit is budgeted at half the memory it can have: the smaller of its cgroup's limit — a container's, or a systemd slice's — and the machine's RAM on Linux, and the machine's RAM on macOS and Windows. Half, because ursus's buffers are on the Go heap, which with GOGC=100 grows to about twice what is live before it collects. So a query spills by default rather than growing until the kernel kills it, and an operator that cannot spill fails at that point with an error naming itself. Collect also refuses a result too large to hold: see Collect. WithMemoryLimit(0) turns the budget off; on another system, where no memory can be read, there is no default and it is off already.

The heap's slack is the Go runtime's to bound, not the budget's. Under the default budget ursus sets the Go soft memory limit too, to nine tenths of the same ceiling, unless GOMEMLIMIT or an earlier debug.SetMemoryLimit chose one: see SetProcessMemoryLimit. Under a limit given here, it does not, and a program running under a container's limit should call SetProcessMemoryLimit itself.

A group-by under a limit you set is serial

So that a group-by that spills gives its rows in the same order on every run.

Under the DEFAULT budget it runs on several workers until the query holds half the budget, then folds them into one and finishes serially, spilling as the serial path does. Half, because the fold builds the merged table while the workers' still exist. The batches the workers already had queued are aggregated first, so the switch can come a few megabytes late at the default batch size. Where it switches depends on scheduling, so an unordered group-by that gets that far can give its rows in a different order from one run to the next; their contents do not change. MaintainOrder, or a limit of your own, makes the order reproducible.

What it does not bound

Collect materialises the whole result by definition, and under a limit you give, asking for a frame in memory is a request to hold it: Collect holds whatever it is asked to. The default budget checks it, which a limit of yours turns off. The larger-than-RAM story runs through SinkParquet and CollectBatches, both of which stream.

Example

Under a memory limit, sort, group-by, join, unique and partitioned windows spill to disk and give the answer they give in memory; an operator that cannot spill fails with an error naming itself. 

package main

import (
	"context"
	"fmt"
	"log"
	"os"

	"github.com/advenn/ursus"
)

func main() {
	keys := make([]int64, 40000)
	for i := range keys {
		keys[i] = int64(i * 7919 % 20000) // each of 20000 keys, twice
	}
	var stats ursus.MemoryStats
	n, err := ursus.Frame(ursus.Values("k", keys)).Unique("k").
		Count(context.Background(), ursus.WithBatchSize(1024),
			ursus.WithMemoryLimit(16<<10), ursus.WithSpillDir(os.TempDir()),
			ursus.WithMemoryStats(&stats))
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(n, "distinct keys; spilled:", stats.Spills > 0)
}

Output:
20000 distinct keys; spilled: true

func WithMemoryStats 

func WithMemoryStats(out *MemoryStats) CollectOption

WithMemoryStats stores the query's memory accounting into *out when the query finishes.

It is written at the end of Collect, Count, CollectInto, SinkParquet, SinkCSV, WriteParquet and WriteCSV, and after the last batch of CollectBatches — so reading it early gives a partial figure rather than a wrong one.

The sinks matter most and were the ones this list used to omit: they are the consumers that stream, so they are where a memory limit is worth setting.

func WithOptFlags 

func WithOptFlags(f OptFlags) CollectOption

WithOptFlags overrides which optimizer rules run. Use it to benchmark a rule's value or to bisect a wrong answer down to the rule that caused it. It took the internal plan.Flags until step 186, which no caller outside the module could build.

func WithSpillDir 

func WithSpillDir(dir string) CollectOption

WithSpillDir chooses where spill files are written. The default is the system temporary directory. Files are removed when the query finishes, including on error and on an early break out of CollectBatches.

func WithThreads 

func WithThreads(n int) CollectOption

WithThreads sets how many workers a query runs on. Default runtime.NumCPU().

Results do not depend on it. Parallelism over the PIPELINE — the scan and the stateless per-batch operators above it — is ORDER-PRESERVING: workers process batches concurrently and the results are read back in input order, so the same query returns byte-identical output at any thread count. That is worth stating because it is not what every engine does, and because five things in ursus quietly depend on batch order — stable sort, First and Last, distinct's first-row rule, limit, and first-appearance group ids.

WithThreads(1) reproduces the serial operator tree exactly, which is the knob to reach for when bisecting.

Group-by also runs N-ways, and only where that cannot change an answer

Since step 17 a hash aggregation is drained by several sinks whose partial tables are merged. It is NOT order-preserving in the same structural way — the dispatcher is round-robin, so no worker holds a contiguous portion of the input — so it is used only where the result cannot depend on order at all: not under MaintainOrder, not with First, Last, ArgMin or ArgMax, and not under a memory limit, because a sink that has spilled cannot be merged. Every one of those falls back to the serial path silently and exactly.

A join's build and probe run in parallel too (parjoin.go, joinbuild.go). A sort remains serial; it still benefits, because its input arrives faster.

The trade, stated: a parallel group-by holds roughly N times the group table, because each worker builds its own. That is why the memory limit disables it rather than dividing the budget.

func WithVerify 

func WithVerify() CollectOption

WithVerify makes the optimizer assert schema preservation after every rule. On in tests, off by default because it costs a schema resolution per rule.

type Column 

type Column = data.Column

Column is a named, typed, nullable run of values.

func Values 

func Values[T Literal](name string, vals []T) *Column

Values builds a column from Go values, with no nulls.

The element type determines the column type: []int64 gives Int64, []string gives String, []time.Time gives Datetime(ns, UTC).

func ValuesNullable 

func ValuesNullable[T Literal](name string, vals []T, valid []bool) *Column

ValuesNullable builds a column with an explicit validity mask, where valid[i] false means row i is null.

Nulls are a parallel bitmap rather than a *T or an Option[T] because boxing would cost an allocation per row and make the values buffer non-contiguous — exactly what a columnar layout must not do.

type ColumnMatch added in v0.6.7 

type ColumnMatch struct {
	// Missing is MissingColumns for this column.
	Missing MatchAction
	// Fill computes the column when the frame lacks it, from the frame's other
	// columns: Polars' missing_columns={"c": expr}. Its answer meets the schema's
	// type as a column of the frame would, widened only as IntegerCast and FloatCast
	// allow; Polars keeps the expression's own type instead.
	Fill Expr
	// The rest are MatchOptions' for this column.
	MissingStructFields, ExtraStructFields, IntegerCast, FloatCast MatchAction
}

ColumnMatch is MatchOptions for one column.

type ConcatMode 

type ConcatMode uint8

ConcatMode says how Concat puts frames together: Polars' concat(how=...), its eleven modes, "align" and "align_full" being one. 

const (
	// ConcatStrict stacks frames of the same columns in the same order — the
	// default, Polars' "vertical".
	//
	// Types still promote and nullability still widens. Neither is a relaxation:
	// refusing to stack two frames that differ only in a nullability flag would be
	// absurd, because that is the ordinary result of filtering one of them. Types
	// meet exactly, an Int32 beside an Int64 at Int64, where Polars' vertical
	// refuses them; an Int64 beside a Float64 is refused, which no type holds.
	ConcatStrict ConcatMode = iota

	// ConcatDiagonal takes the union of the columns, filling each frame's missing
	// ones with nulls: Polars' "diagonal". This is what makes stacking
	// heterogeneous files work.
	ConcatDiagonal

	// ConcatRelaxed and ConcatDiagonalRelaxed meet a column's types at Polars'
	// supertype, as its "vertical_relaxed" and "diagonal_relaxed": an Int64 beside a
	// Float64 at Float64, which may round; text beside a number, a Boolean or an
	// instant at text; a Boolean beside a number at the number; a Date beside a
	// Datetime at the Datetime.
	ConcatRelaxed
	ConcatDiagonalRelaxed

	// ConcatHorizontal places frames side by side, of one height, as HStack does:
	// Polars' "horizontal". ConcatHorizontalExtend pads a shorter frame's columns
	// with nulls: its "horizontal_extend".
	ConcatHorizontal
	ConcatHorizontalExtend

	// ConcatAlign joins frames on the columns every frame has, a full join, sorted
	// by them, as Polars' "align" and "align_full"; ConcatAlignInner,
	// ConcatAlignLeft and ConcatAlignRight join so. A column in some frames but not
	// every one is refused, as Polars' docs have it; Polars 1.44.1 keeps the first
	// frame's.
	ConcatAlign
	ConcatAlignInner
	ConcatAlignLeft
	ConcatAlignRight
)

func (ConcatMode) String added in v0.6.7 

func (m ConcatMode) String() string

String is the mode's name in Polars' concat, "vertical" for ConcatStrict.

type ConcatOption 

type ConcatOption func(*concatCfg)

ConcatOption configures Concat.

func WithConcatMode 

func WithConcatMode(m ConcatMode) ConcatOption

WithConcatMode selects how Concat puts its frames together.

type CutOption added in v0.5.0 

type CutOption func(*cutOptions)

CutOption configures Cut and QCut.

func CutLabels added in v0.5.0 

func CutLabels(labels ...string) CutOption

CutLabels names the bins, lowest first: one label more than there are breaks. Without it, a bin is named by its interval, "(-inf, 1]", "(1, 2.5]", "(2.5, inf]".

func CutLeftClosed added in v0.5.0 

func CutLeftClosed() CutOption

CutLeftClosed closes each bin on the left, [1, 2.5), where the default, Polars' too, closes it on the right, (1, 2.5].

type DataFrame 

type DataFrame struct {
	// contains filtered or unexported fields
}

DataFrame is a materialised result: named, equal-length, typed columns.

There is no row index. Row order is a property of the data, not an addressable label space — the single most consequential thing Polars got right and pandas did not.

func ConcatFrames added in v0.6.7 

func ConcatFrames(ctx context.Context, frames []*DataFrame, opts ...ConcatOption) (*DataFrame, error)

ConcatFrames is Concat of DataFrames, collected.

func ReadDatabase added in v0.6.7 

func ReadDatabase(ctx context.Context, db Queryer, query string, args ...any) (*DataFrame, error)

ReadDatabase runs query, with args, on db and reads its rows as a frame: Polars' read_database.

A column's type is its values', as the driver gives them: integers an Int64, floats a Float64, Booleans, text a String, bytes a Binary, and instants a Datetime in microseconds, of no zone where every one is in UTC, as a driver gives a timestamp without one, and in UTC where one is not. Values of two kinds meet as Polars meets them: an integer and a float at Float64, a Boolean and a number at the number, text and a number, a Boolean or an instant at text, and text and bytes at bytes; two that do not meet are an error. Every row is read for it, where Polars reads 100 and truncates or refuses a later one.

Then a column is of the type its driver names, where every value fits it: an integer of the width the driver scans it to, a Float32, a Boolean; a DECIMAL of its precision and scale, or of 38 digits where it gives none and the values are text, as Polars reads a Python Decimal; a DATE, a TIME, or a TIMESTAMP in a zone, from an instant's wall clock or parsed from text; and text where the driver gives bytes for a type with a name that is not a binary one. A column of no value is of its driver's type, or a Null. NOT NULL, where the driver says it of a column, makes its field non-nullable.

The frame is read whole, as Polars' is. ReadDatabaseRows reads a query already run, and takes options. New in 0.6.7.

func ReadDatabaseRows added in v0.6.7 

func ReadDatabaseRows(rows *sql.Rows, opts ...DatabaseOption) (*DataFrame, error)

ReadDatabaseRows reads every row of rows as a frame, as ReadDatabase reads its query's, and closes it: a statement's result, or one a caller ran itself. New in 0.6.7.

func (*DataFrame) ArrowSchema added in v0.3.0 

func (df *DataFrame) ArrowSchema() (s *arrow.Schema, err error)

ArrowSchema returns the frame's schema as an Arrow schema.

It can fail, because a few ursus types have no Arrow equivalent yet — see Record. Asking for the schema is how to find that out without running anything.

func (*DataFrame) At 

func (df *DataFrame) At[T any](row int, name string) (T, bool, error)

At returns one value and whether it is non-null.

func (*DataFrame) BottomK added in v0.6.3 

func (df *DataFrame) BottomK(ctx context.Context, k int, by ...SortKey) (*DataFrame, error)

func (*DataFrame) Cast added in v0.6.7 

func (df *DataFrame) Cast(ctx context.Context, types map[string]DataType, opts ...CastOption) (*DataFrame, error)

Cast, CastTypes and CastAll cast a DataFrame's columns: LazyFrame's.

func (*DataFrame) CastAll added in v0.6.7 

func (df *DataFrame) CastAll(ctx context.Context, to DataType, opts ...CastOption) (*DataFrame, error)

func (*DataFrame) CastTypes added in v0.6.7 

func (df *DataFrame) CastTypes(ctx context.Context, types map[DataType]DataType,
	opts ...CastOption) (*DataFrame, error)

func (*DataFrame) Clear added in v0.6.7 

func (df *DataFrame) Clear(ctx context.Context, n int) (*DataFrame, error)

func (*DataFrame) Column 

func (df *DataFrame) Column[T any](name string) (*Series[T], error)

Column returns a typed view of a column.

A generic METHOD — the thing Go 1.27 unlocked. Previously this had to be `ursus.Column[int64](df, "x")`, with the receiver demoted to an argument.

Example

Reading values back out. Column gives a typed series; At gives one cell and whether it was null. 

package main

import (
	"context"
	"fmt"
	"log"

	"github.com/advenn/ursus"
)

// exampleSales is the little frame the examples below work on.
func exampleSales() *ursus.LazyFrame {
	return ursus.Frame(
		ursus.Values("region", []string{"north", "south", "north", "east", "south"}),
		ursus.Values("product", []string{"apple", "apple", "pear", "pear", "fig"}),
		ursus.Values("qty", []int64{10, 4, 7, 2, 9}),
		ursus.Values("price", []float64{1.50, 1.50, 2.25, 2.25, 3.00}),
	)
}

func main() {
	df, err := exampleSales().Collect(context.Background())
	if err != nil {
		log.Fatal(err)
	}

	// Column and At are GENERIC METHODS — Go 1.27's headline feature, and the
	// reason ursus can hand back typed data without a cast at the call site.
	qty, err := df.Column[int64]("qty")
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println("values:", qty.Values())

	v, ok, err := df.At[string](0, "region")
	if err != nil {
		log.Fatal(err)
	}
	fmt.Printf("row 0 region: %q present=%v\n", v, ok)
}

Output:
values: [10 4 7 2 9]
row 0 region: "north" present=true

func (*DataFrame) ColumnAt added in v0.6.0 

func (df *DataFrame) ColumnAt(i int) *Column

ColumnAt returns the i-th column, in order, untyped: what DataFrame.Batch gave before step 186, a column at a time, through the public Column rather than the internal batch.

func (*DataFrame) Columns 

func (df *DataFrame) Columns() []string

Columns returns the column names in order.

func (*DataFrame) Concat 

func (df *DataFrame) Concat(ctx context.Context, others ...*DataFrame) (*DataFrame, error)

Concat stacks frames vertically.

func (*DataFrame) Corr added in v0.6.7 

func (df *DataFrame) Corr(ctx context.Context) (*DataFrame, error)

Corr is the Pearson correlation of every pair of columns, as Polars' DataFrame.corr: a Float64 column a column, its row i the correlation with column i. Every column must be a number or a Bool. Polars computes it through NumPy's corrcoef, and so does this, to within rounding: a column holding a null is NaN throughout, its own correlation too, as is one that does not vary, and each value is clipped to [-1, 1]. Corr of two expressions differs: it skips the rows where either is null.

func (*DataFrame) Describe added in v0.5.0 

func (df *DataFrame) Describe(ctx context.Context, percentiles ...float64) (*DataFrame, error)

Describe is LazyFrame.Describe over the frame.

func (*DataFrame) Drop 

func (df *DataFrame) Drop(ctx context.Context, names ...string) (*DataFrame, error)

Drop removes columns; Rename changes their names.

func (*DataFrame) DropInPlace added in v0.6.7 

func (df *DataFrame) DropInPlace(name string) (*Column, *DataFrame, error)

DropInPlace is column name, and the frame without it, as Polars' drop_in_place, which ursus answers with a new frame where Polars changes its own.

func (*DataFrame) DropNans added in v0.6.7 

func (df *DataFrame) DropNans(ctx context.Context, subset ...string) (*DataFrame, error)

DropNans and Interpolate are a DataFrame's: LazyFrame's.

func (*DataFrame) DropNulls added in v0.6.3 

func (df *DataFrame) DropNulls(ctx context.Context, subset ...string) (*DataFrame, error)

DropNulls removes rows with a null in subset, or in any column.

func (*DataFrame) Equals added in v0.6.7 

func (df *DataFrame) Equals(ctx context.Context, other *DataFrame, nullEqual bool) (bool, error)

Equals reports whether other holds the same columns, by name and in order, and the same values row for row: Polars' DataFrame.equals(other, null_equal). Values compare as Polars compares them: NaN equals NaN, -0 equals 0, and a List or a Struct element by element. Two numbers of different types are compared at their common type, as Polars compares an Int64 with an Int32 or a Float64; other differing types are unequal. A null equals a null when nullEqual, and with it false a frame holding any null equals no frame, as Polars' does.

func (*DataFrame) EstimatedSize added in v0.6.7 

func (df *DataFrame) EstimatedSize() int64

EstimatedSize is how many bytes the frame's buffers hold, as Polars' estimated_size: shared buffers counted once.

func (*DataFrame) Explode added in v0.6.3 

func (df *DataFrame) Explode(ctx context.Context, names ...string) (*DataFrame, error)

Explode gives each element of the named List columns a row of its own; Unnest gives each field of the named Struct columns a column of its own.

func (*DataFrame) Extend added in v0.6.7 

func (df *DataFrame) Extend(ctx context.Context, other *DataFrame) (*DataFrame, error)

Extend is the frame with other's rows after its own, as Polars' extend, which ursus answers with a new frame where Polars changes its own: ConcatFrames of the two, strict, the columns' names and types the same.

func (*DataFrame) FillNan added in v0.6.3 

func (df *DataFrame) FillNan[T Operand](ctx context.Context, v T, subset ...string) (*DataFrame, error)

FillNan fills the NaNs of subset's float columns, or every float column, with v.

func (*DataFrame) FillNull added in v0.6.3 

func (df *DataFrame) FillNull[T Operand](ctx context.Context, v T, subset ...string) (*DataFrame, error)

FillNull fills the nulls of subset's columns, or every column v fits, with v.

func (*DataFrame) Filter 

func (df *DataFrame) Filter(ctx context.Context, preds ...Expr) (*DataFrame, error)

Filter keeps rows where every predicate is true.

func (*DataFrame) Fold added in v0.6.7 

func (df *DataFrame) Fold(ctx context.Context, fn func(acc, x Expr) Expr) (*Column, error)

Fold is fn folded across the columns, row by row, as Polars' frame fold: the first column the accumulator, then fn of it and each column after, in their order, named after the first; Reduce over every column. fn builds an expression, which runs over the frame once.

func (*DataFrame) Gather added in v0.6.7 

func (df *DataFrame) Gather(ctx context.Context, indices []int, nullOnOOB bool) (_ *DataFrame, err error)

Gather is the rows at indices, in their order, as Polars' DataFrame.gather: a negative index counts from the end, and one past either end is refused, or with nullOnOOB is a row of nulls.

func (*DataFrame) GatherEvery added in v0.6.7 

func (df *DataFrame) GatherEvery(ctx context.Context, n, offset int) (*DataFrame, error)

func (*DataFrame) GetColumnIndex added in v0.6.7 

func (df *DataFrame) GetColumnIndex(name string) (int, error)

GetColumnIndex is column name's place, as Polars' get_column_index; a name the frame lacks is refused.

func (*DataFrame) Glimpse added in v0.6.7 

func (df *DataFrame) Glimpse() string

Glimpse is the frame on its side, a line a column, as Polars' glimpse prints it: its height and width, then each column's name, type and first ten values, as the frame prints them, a null "null". A name past 50 characters is cut to 49 and "…": 

Rows: 3
Columns: 2
$ price <Float64> 9.5, 3, null
$ sku    <String> "a", "b", "c"

func (*DataFrame) GroupBy added in v0.6.3 

func (df *DataFrame) GroupBy(keys ...Expr) *GroupBy

GroupBy, GroupByDynamic and Rolling start a grouping of the frame. Each gives the lazy builder, whose Agg, or Sum, Mean and the rest, is then collected: 

out, err := df.GroupBy(Col("k")).Agg(Col("v").Sum()).Collect(ctx)

An eager answer would need a second builder with every aggregate again, each taking a context; the context belongs to Collect.

func (*DataFrame) GroupByDynamic added in v0.6.3 

func (df *DataFrame) GroupByDynamic(index Expr, o DynamicOptions) *GroupBy

func (*DataFrame) HStack added in v0.6.3 

func (df *DataFrame) HStack(ctx context.Context, others ...*DataFrame) (*DataFrame, error)

HStack places frames side by side; VStack stacks one under this one.

func (*DataFrame) HashRows added in v0.6.7 

func (df *DataFrame) HashRows(ctx context.Context, seed uint64) (*Series[uint64], error)

HashRows is each row's hash over every column, seeded, as Polars' DataFrame.hash_rows answers it: a Series of UInt64, named "" as Polars names it.

func (*DataFrame) Head 

func (df *DataFrame) Head(ctx context.Context, n int) (*DataFrame, error)

Head keeps the first n rows; Tail the last n.

func (*DataFrame) Height 

func (df *DataFrame) Height() int

Height returns the number of rows; Width the number of columns.

func (*DataFrame) InsertColumn added in v0.6.7 

func (df *DataFrame) InsertColumn[C Expr | *Column](ctx context.Context, i int, c C) (*DataFrame, error)

InsertColumn puts a column in a DataFrame: LazyFrame's.

func (*DataFrame) Interpolate added in v0.6.7 

func (df *DataFrame) Interpolate(ctx context.Context) (*DataFrame, error)

func (*DataFrame) IsDuplicated added in v0.6.7 

func (df *DataFrame) IsDuplicated(ctx context.Context) (*Series[bool], error)

IsDuplicated is whether each row is another row too, over every column, and IsUnique whether no other row is: Polars' DataFrame.is_duplicated and is_unique (step 290), compared as Unique compares rows.

func (*DataFrame) IsEmpty added in v0.6.7 

func (df *DataFrame) IsEmpty() bool

IsEmpty reports whether the frame has no rows: Polars' is_empty. A frame of no columns has none.

func (*DataFrame) IsUnique added in v0.6.7 

func (df *DataFrame) IsUnique(ctx context.Context) (*Series[bool], error)

func (*DataFrame) Item added in v0.6.7 

func (df *DataFrame) Item() (any, error)

Item is the frame's one value, as Polars' DataFrame.item(): a frame of one row and one column.

func (*DataFrame) ItemAt added in v0.6.7 

func (df *DataFrame) ItemAt(row int, column string) (any, error)

ItemAt is the value at row of column, as Polars' item(row, column): a negative row counts from the end.

func (*DataFrame) Join 

func (df *DataFrame) Join(ctx context.Context, other *DataFrame, opts ...JoinOption) (*DataFrame, error)

Join combines two frames on keys.

func (*DataFrame) JoinAsOf added in v0.6.3 

func (df *DataFrame) JoinAsOf(ctx context.Context, other *DataFrame, opts ...AsOfOption) (*DataFrame, error)

JoinAsOf joins each row to the nearest row of other by an ordered key.

func (*DataFrame) JoinWhere added in v0.3.0 

func (df *DataFrame) JoinWhere(ctx context.Context, other *DataFrame, preds ...Expr) (*DataFrame, error)

JoinWhere joins every pair of rows that satisfies the predicates.

func (*DataFrame) Lazy 

func (df *DataFrame) Lazy() *LazyFrame

Lazy turns a materialised frame back into a query.

The obvious spelling is wrong in three specific ways

`ursus.Frame(df.batch.Columns()...)` compiles and runs, because Column is a type alias — and it is wrong:

  • It would re-derive nullability from the data. Field.Nullable is a static property of the schema, not a count of nulls actually present, so a nullable column that happens to hold none — the ordinary result of a filter — would come back non-nullable and every downstream join and cast would reason from the wrong schema.
  • It would lose the row count of a frame with rows but no columns.
  • It would alias the batch's own column slice, against Columns()' contract.

So this goes through the batch, which already carries the declared schema and the row count.

It is free: no copy, no re-derivation. The batch is immutable and shared.

func (*DataFrame) Limit added in v0.6.3 

func (df *DataFrame) Limit(ctx context.Context, n int) (*DataFrame, error)

Limit keeps the first n rows, as Head does.

func (*DataFrame) MapColumns added in v0.6.7 

func (df *DataFrame) MapColumns(ctx context.Context, names []string, fn func(*Column) (*Column, error)) (*DataFrame, error)

MapColumns is the frame with each column named replaced by fn of it, as Polars' map_columns, for what only Go can say of a whole column: the answer keeps the column's name and place, and is as long as the frame, or one value long, which is repeated. A selector's names are its Expand of the frame's schema.

func (*DataFrame) MapRows added in v0.6.7 

func (df *DataFrame) MapRows[T, R any](ctx context.Context, fn func(T) (R, error)) (*DataFrame, error)

MapRows is fn of each row, as Polars' map_rows: the frame's rows each a T, and fn's answers a frame. A T that is a struct is read as Rows reads one, a field a column by its name or tag; a T that is []any is the row's values in the columns' order, as Row gives them. An R that is a struct is a column a field, as FromRows builds them; any other R is one column named "map", as Polars names a function's one answer: a pointer nil a null, a slice a List. fn runs on one goroutine, in the rows' order, and an error from it ends the walk, naming the row. 

type in struct{ Qty int64; Price float64 }
revenue, err := df.MapRows(ctx, func(r in) (float64, error) { return float64(r.Qty) * r.Price, nil })

An expression is faster: MapRows decodes every row and calls fn once a row, as Polars' map_rows calls Python.

func (*DataFrame) MatchToSchema added in v0.6.7 

func (df *DataFrame) MatchToSchema(ctx context.Context, schema *Schema, o MatchOptions) (*DataFrame, error)

MatchToSchema makes a DataFrame's columns the schema's: LazyFrame's.

func (*DataFrame) Max added in v0.6.7 

func (df *DataFrame) Max(ctx context.Context) (*DataFrame, error)

func (*DataFrame) MaxHorizontal added in v0.6.7 

func (df *DataFrame) MaxHorizontal(ctx context.Context) (*Column, error)

func (*DataFrame) Mean added in v0.6.7 

func (df *DataFrame) Mean(ctx context.Context) (*DataFrame, error)

func (*DataFrame) MeanHorizontal added in v0.6.7 

func (df *DataFrame) MeanHorizontal(ctx context.Context, ignoreNulls bool) (*Column, error)

func (*DataFrame) Median added in v0.6.7 

func (df *DataFrame) Median(ctx context.Context) (*DataFrame, error)

func (*DataFrame) MergeSorted added in v0.6.3 

func (df *DataFrame) MergeSorted(ctx context.Context, other *DataFrame, key string) (*DataFrame, error)

MergeSorted merges two frames sorted by key into one sorted by it.

func (*DataFrame) Min added in v0.6.7 

func (df *DataFrame) Min(ctx context.Context) (*DataFrame, error)

func (*DataFrame) MinHorizontal added in v0.6.7 

func (df *DataFrame) MinHorizontal(ctx context.Context) (*Column, error)

MinHorizontal and MaxHorizontal are each row's least and greatest value across the columns, as Polars' frame min_horizontal and max_horizontal, named "min" and "max": the expressions MinHorizontal and MaxHorizontal over every column.

func (*DataFrame) NUnique added in v0.6.7 

func (df *DataFrame) NUnique(ctx context.Context, subset ...string) (int, error)

NUnique is the number of distinct rows over subset, every column when it is empty: Polars' DataFrame.n_unique (step 290). Rows are compared as Unique compares them.

func (*DataFrame) NullCount added in v0.6.7 

func (df *DataFrame) NullCount(ctx context.Context) (*DataFrame, error)

func (*DataFrame) PartitionBy added in v0.6.7 

func (df *DataFrame) PartitionBy(ctx context.Context, o PartitionOptions) ([]Partition, error)

PartitionBy splits the frame into a part for each distinct key of By, in the order each key first appears, a null key among them, as Polars' DataFrame.partition_by does. Polars' as_dict maps key tuples to frames, and Go has no map by a tuple, so here each part's Key stands beside its Frame. Keys group as a group-by groups them, -0 with 0 and NaN with NaN; a List or a Struct column is no key here, where Polars partitions by one. 

parts, err := df.PartitionBy(ctx, ursus.PartitionOptions{By: []string{"region"}})
for _, p := range parts {
	region, _, _ := p.Key.At[string](0, "region")
	fmt.Println(region, p.Frame.Height())
}

The parts are taken from the frame it is given, and together hold as much as it does.

func (*DataFrame) Pivot added in v0.6.3 

func (df *DataFrame) Pivot(ctx context.Context, o PivotOptions) (*DataFrame, error)

Pivot spreads a frame wide, as Polars' DataFrame.pivot does: LazyFrame's, with OnColumns, when nil, every combination of On's values the frame holds, in the order each first appears, or sorted with SortColumns, a null first. 

wide, err := df.Pivot(ctx, ursus.PivotOptions{On: []string{"quarter"}, Index: []string{"region"},
	Values: []string{"sales"}, Agg: ursus.Expr.Sum})

func (*DataFrame) Product added in v0.6.7 

func (df *DataFrame) Product(ctx context.Context) (*DataFrame, error)

func (*DataFrame) Quantile added in v0.6.7 

func (df *DataFrame) Quantile(ctx context.Context, q float64, interp Interpolation) (*DataFrame, error)

func (*DataFrame) Record added in v0.3.0 

func (df *DataFrame) Record() (rec arrow.Record, err error)

Record exports the frame as an Arrow record, sharing its memory.

This is the zero-copy boundary to the rest of the Arrow ecosystem: hand the result to arrow-go's IPC writer, to Flight, to ADBC, or to any library that speaks arrow.Record, and no column data is copied on the way. 

rec, err := df.Record()
if err != nil { return err }
defer rec.Release()

w := ipc.NewWriter(f, ipc.WithSchema(rec.Schema()))
w.Write(rec)

Release is optional here, and that is deliberate

The record's buffers are re-wrapped so that Release on them cannot reach ursus's own memory. Releasing is still the right habit — every other arrow-go producer expects it, and the example above does it — but forgetting leaks nothing, and releasing twice cannot empty a frame you still hold. internal/arrowout's doc explains why that second guarantee needed arranging.

It can fail

Array, Enum, Categorical and Uint128 are declared in ursus's type enum and are not exportable yet; a frame containing one is refused by name rather than exported wrongly. Everything else maps, including Decimal and the Int128 that every integer Sum produces.

func (*DataFrame) Remove added in v0.6.3 

func (df *DataFrame) Remove(ctx context.Context, preds ...Expr) (*DataFrame, error)

Remove drops the rows where any predicate is true, or null.

func (*DataFrame) Rename 

func (df *DataFrame) Rename(ctx context.Context, names map[string]string) (*DataFrame, error)

func (*DataFrame) ReplaceColumn added in v0.6.7 

func (df *DataFrame) ReplaceColumn(i int, c *Column) (*DataFrame, error)

ReplaceColumn is the frame with c in place of the column at place i, under c's name, as Polars' replace_column, which ursus answers with a new frame where Polars changes its own. A negative i counts from the end; c must be as long as the frame, and its name another column's is refused, where Polars' takes it.

func (*DataFrame) Reverse 

func (df *DataFrame) Reverse(ctx context.Context) (*DataFrame, error)

Reverse emits rows in the opposite order.

func (*DataFrame) Rolling added in v0.6.3 

func (df *DataFrame) Rolling(index Expr, o RollingOptions) *GroupBy

func (*DataFrame) Row added in v0.6.7 

func (df *DataFrame) Row(i int) ([]any, error)

Row is row i's values, as Polars' DataFrame.row: each a Go value as ToMaps reads it, a null nil; a negative i counts from the end.

func (*DataFrame) RowNamed added in v0.6.7 

func (df *DataFrame) RowNamed(i int) (map[string]any, error)

RowNamed is Row as a map from each column's name to its value: Polars' row(named=True).

func (*DataFrame) RowWhere added in v0.6.7 

func (df *DataFrame) RowWhere(ctx context.Context, pred Expr) ([]any, error)

RowWhere is the one row pred holds of, as Polars' row(by_predicate=): none, or several, is refused.

func (*DataFrame) Rows 

func (df *DataFrame) Rows[T any]() (rows []T, err error)

Rows decodes every row into a T.

func (*DataFrame) RowsByKey added in v0.6.7 

func (df *DataFrame) RowsByKey[K comparable, T any](key string) (map[K][]T, error)

RowsByKey is the frame's rows, each a T as Rows decodes it, grouped by the value of the column key, a K, as Polars' rows_by_key(key, named=True) groups them: a key's rows in the frame's order. K is read from the column as a field of its type would be, so an Int64 key reads into an int or an int64. A null key's rows are under K's zero value; T's fields name the columns it reads, the key's among them or not. 

byRegion, err := df.RowsByKey[string, sale]("region")

func (*DataFrame) SampleFraction added in v0.6.7 

func (df *DataFrame) SampleFraction(ctx context.Context, fraction float64, withReplacement, shuffle bool,
	seed uint64) (*DataFrame, error)

func (*DataFrame) SampleN added in v0.6.7 

func (df *DataFrame) SampleN(ctx context.Context, n int, withReplacement, shuffle bool, seed uint64) (*DataFrame, error)

SampleN, SampleFraction and Shuffle draw the frame's rows: LazyFrame's.

func (*DataFrame) Schema 

func (df *DataFrame) Schema() *Schema

Schema returns the column names and types.

func (*DataFrame) Select 

func (df *DataFrame) Select(ctx context.Context, exprs ...Expr) (*DataFrame, error)

Select evaluates expressions and returns the result.

func (*DataFrame) Shape 

func (df *DataFrame) Shape() (int, int)

Shape returns (rows, columns).

func (*DataFrame) Shift added in v0.6.7 

func (df *DataFrame) Shift(ctx context.Context, n int) (*DataFrame, error)

func (*DataFrame) ShiftFill added in v0.6.7 

func (df *DataFrame) ShiftFill[T Operand](ctx context.Context, n int, fill T) (*DataFrame, error)

func (*DataFrame) Show added in v0.6.7 

func (df *DataFrame) Show(limit int) string

Show is the frame's first limit rows as a table, every one of them drawn, where String draws ten, as Polars' show prints them; a negative limit leaves the last -limit rows out, and Show(df.Height()) draws them all. It is the text to print, as Glimpse's is.

func (*DataFrame) Shuffle added in v0.6.7 

func (df *DataFrame) Shuffle(ctx context.Context, seed uint64) (*DataFrame, error)

func (*DataFrame) Slice added in v0.6.3 

func (df *DataFrame) Slice(ctx context.Context, offset, length int) (*DataFrame, error)

Slice keeps length rows from offset; a negative length keeps to the end.

func (*DataFrame) Sort 

func (df *DataFrame) Sort(ctx context.Context, keys ...SortKey) (*DataFrame, error)

Sort orders rows.

func (*DataFrame) Std added in v0.6.7 

func (df *DataFrame) Std(ctx context.Context, ddof int) (*DataFrame, error)

func (*DataFrame) String 

func (df *DataFrame) String() string

String renders the frame as an aligned table.

Display quality is not cosmetic: this is what people look at all day, and a column of unaligned numbers with no visible types is the difference between a library that feels finished and one that does not.

func (*DataFrame) Sum added in v0.6.7 

func (df *DataFrame) Sum(ctx context.Context) (*DataFrame, error)

Sum, Mean, Median, Min, Max, Std, Var, Quantile, Product and NullCount are a DataFrame's aggregates: LazyFrame's.

func (*DataFrame) SumHorizontal added in v0.6.7 

func (df *DataFrame) SumHorizontal(ctx context.Context, ignoreNulls bool) (*Column, error)

SumHorizontal and MeanHorizontal are each row's sum and mean across the columns, as Polars' frame sum_horizontal and mean_horizontal, named "sum" and "mean". They are the expressions SumHorizontal and MeanHorizontal over every column, nulls skipped; with ignoreNulls false, a row holding a null is null.

func (*DataFrame) Tail 

func (df *DataFrame) Tail(ctx context.Context, n int) (*DataFrame, error)

func (*DataFrame) ToDummies added in v0.6.7 

func (df *DataFrame) ToDummies(ctx context.Context, o DummiesOptions) (*DataFrame, error)

ToDummies replaces each of Columns with a Uint8 column for each of its distinct values, 1 where a row holds the value and 0 where it does not: Polars' to_dummies. A null is a value, NaN is one, and -0.0 is 0.0.

A dummy is named the column, the separator and the value as Polars displays it: text as itself, 1.0 as "1.0", a date "2024-01-02", an instant "2024-01-02 03:04:05.120" and a zoned one with its zone's abbreviation after, a duration "1m 30s", a null "null". A column's dummies stand in its place, ordered by name, as Polars orders them, so 10 comes before 2. A column of lists or structs is refused, as a group key of one is. 

df.ToDummies(ctx, ursus.DummiesOptions{Columns: []string{"color"}}) // color_blue, color_red

func (*DataFrame) ToMaps added in v0.6.7 

func (df *DataFrame) ToMaps() ([]map[string]any, error)

ToMaps is the frame's rows, each a map from its columns' names to their values, as Polars' to_dicts: an integer or a float its Go type, an Int64 an int64; text and an Enum a string; a Binary a []byte; a Boolean a bool; a Date or a Datetime a time.Time in its zone; a Duration a time.Duration; a Time the time.Duration since midnight; a Decimal its text; a List a []any; a Struct a map[string]any; a null nil.

func (*DataFrame) ToSeries added in v0.6.7 

func (df *DataFrame) ToSeries(i int) (*Column, error)

ToSeries is the column at place i, as Polars' to_series: a negative i counts from the end.

func (*DataFrame) ToStruct added in v0.6.7 

func (df *DataFrame) ToStruct(name string) *Column

ToStruct is the frame as one Struct column named name, a field a column, as Polars' to_struct; no row of it null.

func (*DataFrame) TopK added in v0.6.3 

func (df *DataFrame) TopK(ctx context.Context, k int, by ...SortKey) (*DataFrame, error)

TopK keeps the k rows with the largest keys and BottomK the k with the smallest, a null key last in both.

func (*DataFrame) Transpose added in v0.6.3 

func (df *DataFrame) Transpose(ctx context.Context, o TransposeOptions) (*DataFrame, error)

Transpose turns rows into columns, as Polars' DataFrame.transpose does: column j of the answer holds row j of the frame, a value for each of its columns, in their order. Every column is cast to the type all of them meet at, Polars' supertype, so an Int64 and a Float64 make Float64 columns and text beside a number makes text (step 320; it was refused). A List or a Struct meets only its own type.

func (*DataFrame) Unique 

func (df *DataFrame) Unique(ctx context.Context, subset ...string) (*DataFrame, error)

Unique removes duplicate rows.

func (*DataFrame) UniqueKeep added in v0.6.7 

func (df *DataFrame) UniqueKeep(ctx context.Context, keep KeepStrategy, subset ...string) (*DataFrame, error)

UniqueKeep removes duplicate rows, keeping the row keep says of each key.

func (*DataFrame) Unnest added in v0.6.3 

func (df *DataFrame) Unnest(ctx context.Context, names ...string) (*DataFrame, error)

func (*DataFrame) Unpivot added in v0.3.0 

func (df *DataFrame) Unpivot(ctx context.Context, opts UnpivotOptions) (*DataFrame, error)

Unpivot turns several value columns into a variable/value pair.

func (*DataFrame) Unstack added in v0.6.7 

func (df *DataFrame) Unstack(ctx context.Context, o UnstackOptions) (*DataFrame, error)

Unstack lays each of Columns out as a grid without keys, Polars' unstack, cheaper than a pivot. Vertically, a column x's values go a Step at a time down new columns x_0, x_1 and on, Step rows high; horizontally, a Step at a time across Step new columns, so x_0 holds the first of each Step. The slots after the last value are null, or Fill. Each suffix is zero-padded to the digits of the count of new columns, as Polars pads it: x_00 to x_12 for 13.

A vertical column is a slice of the old, shared, but for a last one with slots to fill, and a horizontal one a gather.

func (*DataFrame) Update added in v0.6.7 

func (df *DataFrame) Update(ctx context.Context, other *DataFrame, o UpdateOptions) (*DataFrame, error)

Update is a DataFrame's: LazyFrame's.

func (*DataFrame) Upsample added in v0.6.7 

func (df *DataFrame) Upsample[T Span](ctx context.Context, time string, every T, groupBy ...string) (*DataFrame, error)

Upsample is a DataFrame's: LazyFrame's.

func (*DataFrame) VStack added in v0.6.3 

func (df *DataFrame) VStack(ctx context.Context, other *DataFrame) (*DataFrame, error)

func (*DataFrame) ValueCounts added in v0.5.0 

func (df *DataFrame) ValueCounts(ctx context.Context, by ...Expr) (*DataFrame, error)

ValueCounts is LazyFrame.ValueCounts, collected.

func (*DataFrame) Var added in v0.6.7 

func (df *DataFrame) Var(ctx context.Context, ddof int) (*DataFrame, error)

func (*DataFrame) WhereExists added in v0.3.0 

func (df *DataFrame) WhereExists(ctx context.Context, other *DataFrame, preds ...Expr) (*DataFrame, error)

WhereExists keeps rows with at least one satisfying partner in other.

func (*DataFrame) WhereNotExists added in v0.3.0 

func (df *DataFrame) WhereNotExists(ctx context.Context, other *DataFrame, preds ...Expr) (*DataFrame, error)

WhereNotExists keeps rows with no satisfying partner in other.

func (*DataFrame) Width 

func (df *DataFrame) Width() int

func (*DataFrame) WithColumns 

func (df *DataFrame) WithColumns(ctx context.Context, exprs ...Expr) (*DataFrame, error)

WithColumns adds or replaces columns.

func (*DataFrame) WithRowIndex added in v0.6.3 

func (df *DataFrame) WithRowIndex(ctx context.Context, name string, offset uint32) (*DataFrame, error)

WithRowIndex adds a column numbering the rows from offset.

func (*DataFrame) WriteDatabase added in v0.6.8 

func (df *DataFrame) WriteDatabase(ctx context.Context, db DatabaseWriter, table string,
	opts ...DatabaseWriteOption) (int64, error)

WriteDatabase writes the frame to table, as WriteDatabase of its LazyFrame does. New in 0.6.8.

type DataType 

type DataType = dtype.DataType

DataType is a logical column type. Comparable, usable as a map key.

func StructOf 

func StructOf(fields ...Field) DataType

StructOf builds a struct type. Named StructOf rather than Struct to leave the bare name available and to match DateOf / TimeOf.

type DatabaseOption added in v0.6.7 

type DatabaseOption func(*sqlrows.Options)

DatabaseOption configures ReadDatabaseRows.

func WithDatabaseSchemaOverrides added in v0.6.7 

func WithDatabaseSchemaOverrides(types map[string]DataType) DatabaseOption

WithDatabaseSchemaOverrides gives the named columns their types, as Polars' schema_overrides does: each value is cast to its column's type, and one that does not fit it is an error. A name the result has no column of is an error too. New in 0.6.7.

type DatabaseWriteOption added in v0.6.8 

type DatabaseWriteOption interface {
	// contains filtered or unexported methods
}

DatabaseWriteOption configures WriteDatabase: WithIfTableExists, WithDialect, or any CollectOption, which the query runs with.

func WithDialect added in v0.6.8 

func WithDialect(d SQLDialect) DatabaseWriteOption

WithDialect names the database's dialect, which the driver otherwise names. New in 0.6.8.

func WithIfTableExists added in v0.6.8 

func WithIfTableExists(mode IfTableExists) DatabaseWriteOption

WithIfTableExists says what to do with a table already there. Default IfExistsFail. New in 0.6.8.

type DatabaseWriter added in v0.6.8 

type DatabaseWriter interface {
	ExecContext(ctx context.Context, query string, args ...any) (sql.Result, error)
	QueryContext(ctx context.Context, query string, args ...any) (*sql.Rows, error)
	PrepareContext(ctx context.Context, query string) (*sql.Stmt, error)
}

DatabaseWriter is what WriteDatabase writes through: a *sql.DB or a *sql.Conn, whose write is one transaction of its own, or a *sql.Tx, inside which it writes.

type DateTimeParts added in v0.6.6 

type DateTimeParts struct {
	Year, Month, Day, Hour, Minute, Second, Microsecond any
}

DateTimeParts are the components Replace sets, Polars' replace keywords. Each is nil, to keep the value's own, a Go integer, or an Expr of integers, whose null keeps the value's own too.

type DtExpr 

type DtExpr struct {
	// contains filtered or unexported fields
}

DtExpr is the temporal namespace: `Col("ts").Dt().Year()`.

Components are read in the column's own timezone

2024-01-01T00:00:00+09:00 is hour 0 in Tokyo and hour 15 in UTC. A Datetime carries its zone, and every component below is read in it — otherwise `hour` would be wrong for every row of any non-UTC column, silently.

Durations have no components

`.Year()` on a Duration is refused at plan time. A duration is a length, not a point in time; "the year of 90 minutes" has no answer. Use the Total* family.

func (DtExpr) AddBusinessDays added in v0.6.6 

func (d DtExpr) AddBusinessDays[N Operand](n N, roll Roll, cal ...BusinessCalendar) Expr

AddBusinessDays moves each value n business days of cal, backward where n is negative, keeping its time of day: Polars' add_business_days. n is a constant or an expression of integers. A value not on a business day is refused, or rolled first. A zoned value moves on its wall clock, and a wall clock its zone reads twice or never is refused, as Polars refuses it.

func (DtExpr) BaseUtcOffset added in v0.6.6 

func (d DtExpr) BaseUtcOffset() Expr

BaseUtcOffset is a zoned Datetime's zone's offset from UTC without daylight-saving time, and DstOffset that time's own offset, the two summing to the offset in force, each a Duration(ms): Polars' base_utc_offset and dst_offset.

They are read from the system's zone database. Its main form, which Polars bundles, has Dublin's and Casablanca's daylight-saving time negative, Dublin's in winter; its rearguard form, which some systems install (Ubuntu's), has it positive, Dublin's in summer. The two forms differ in no other zone of today, and in no offset in force.

func (DtExpr) CastTimeUnit added in v0.6.7 

func (d DtExpr) CastTimeUnit(unit TimeUnit) Expr

CastTimeUnit is a Datetime's or a Duration's values in another unit, as Polars' cast_time_unit (step 325): an instant floored to the coarser unit, a length truncated toward zero, as a Cast gives them. A Datetime keeps its zone; any other type is refused. WithTimeUnit keeps the integers instead.

func (DtExpr) Century added in v0.6.6 

func (d DtExpr) Century() Expr

func (DtExpr) Combine added in v0.6.3 

func (d DtExpr) Combine(t Expr) Expr

Combine puts each row's day together with the Time of day in t, a column or a literal: Polars' dt.combine. A Date makes a Datetime(us); a Datetime keeps its unit and zone, its day read on its own wall clock, where Polars answers microseconds unless asked, and a Cast gives another unit. In a zone, a wall clock the zone skips or repeats fails the query, as Polars' does.

func (DtExpr) ConvertTimeZone added in v0.5.0 

func (d DtExpr) ConvertTimeZone(tz string) Expr

ConvertTimeZone reads a zoned Datetime's instants in another zone: the same instants, whose components, Strftime and rendering now follow tz. Polars' dt.convert_time_zone. 

Col("ts").Dt().ConvertTimeZone("Asia/Tokyo")   // 09:00 UTC reads as 18:00

No value changes, so it costs nothing per row. A naive Datetime names no instant to convert and is refused; if its wall clocks are UTC, cast it to Datetime(unit, "UTC") first. A zone the system does not know is refused while the query is planned.

func (DtExpr) Date added in v0.6.5 

func (d DtExpr) Date() Expr

Date is each instant's date on its own zone's wall clock, Polars' dt.date: 23:30 UTC on the 1st, read in a zone five hours ahead, is the 2nd. A Date is itself.

Where it differs from Cast(ursus.Date): a cast reads the UTC wall clock, as Polars' cast does, so the two agree only for a naive Datetime or one in UTC.

func (DtExpr) Day 

func (d DtExpr) Day() Expr

func (DtExpr) DaysInMonth added in v0.6.3 

func (d DtExpr) DaysInMonth() Expr

DaysInMonth is the number of days in a Date's or Datetime's month, 28 to 31, the month read in the column's own zone: Polars' dt.days_in_month, an Int8.

func (DtExpr) DstOffset added in v0.6.6 

func (d DtExpr) DstOffset() Expr

func (DtExpr) Epoch 

func (d DtExpr) Epoch() Expr

Epoch is whole seconds since 1970-01-01T00:00:00Z, floored: the second an instant lies in, so half a second before the epoch is -1.

func (DtExpr) EpochDays added in v0.6.6 

func (d DtExpr) EpochDays() Expr

func (DtExpr) EpochIn added in v0.6.6 

func (d DtExpr) EpochIn(unit TimeUnit) Expr

EpochIn is the time since 1970-01-01T00:00:00Z in unit, floored, so a microsecond before it is millisecond -1: Polars' epoch(time_unit). EpochDays is it in days, an Int32. A Date or a Datetime; a zoned one counts from its instant.

func (DtExpr) Hour 

func (d DtExpr) Hour() Expr

func (DtExpr) IsBusinessDay added in v0.6.6 

func (d DtExpr) IsBusinessDay(cal ...BusinessCalendar) Expr

IsBusinessDay is whether each value's date, on its zone's wall clock, is a business day of cal, Monday to Friday by default: Polars' is_business_day.

func (DtExpr) IsLeapYear added in v0.5.0 

func (d DtExpr) IsLeapYear() Expr

IsLeapYear reports whether a Date or Datetime falls in a year of 366 days, the year read in the column's own zone.

func (DtExpr) IsoYear added in v0.6.6 

func (d DtExpr) IsoYear() Expr

IsoYear is the ISO 8601 year, the year of the ISO week a day is in, so 2024-12-30 is in 2025's first week; Century and Millennium count from year 1, so 2000 is in the 20th century and 2001 the 21st, and year 0 is in neither, as Polars counts. Each is an Int32 of a Date or a Datetime, read on its zone's wall clock.

func (DtExpr) Microsecond 

func (d DtExpr) Microsecond() Expr

func (DtExpr) Millennium added in v0.6.6 

func (d DtExpr) Millennium() Expr

func (DtExpr) Millisecond 

func (d DtExpr) Millisecond() Expr

func (DtExpr) Minute 

func (d DtExpr) Minute() Expr

func (DtExpr) Month 

func (d DtExpr) Month() Expr

func (DtExpr) MonthEnd added in v0.5.0 

func (d DtExpr) MonthEnd() Expr

MonthEnd moves a Date or Datetime to the last day of its month, keeping the time of day, as MonthStart does: Polars' dt.month_end. February 2024 ends on the 29th.

func (DtExpr) MonthStart added in v0.5.0 

func (d DtExpr) MonthStart() Expr

MonthStart moves a Date or Datetime to the first day of its month, keeping the time of day on its wall clock: Polars' dt.month_start. A Datetime's month is the one in its own zone.

func (DtExpr) Nanosecond 

func (d DtExpr) Nanosecond() Expr

func (DtExpr) OffsetBy added in v0.5.0 

func (d DtExpr) OffsetBy[T Span | Expr](by T) Expr

OffsetBy moves each Date or Datetime by an interval, keeping its type: Polars' dt.offset_by. 

Col("due").Dt().OffsetBy(Every("1mo"))      // the same day next month, clamped
Col("ts").Dt().OffsetBy(-90 * time.Minute)  // ninety minutes earlier

Months and days move the WALL CLOCK in the column's own zone: one day after 09:00 is 09:00 the next day across a daylight-saving change, and January 31st plus a month is the last day of February. Nanoseconds, and a time.Duration, move elapsed time. A wall clock the zone skips gives the instant the gap ends; one it repeats, the reading at the input's own offset when that offset holds there.

An offset the type cannot hold is refused while the query is planned: an hour on a Date, a nanosecond on Datetime(us). An answer past the type's range is refused when it is computed. A Time has no date to move: add a Duration to it instead.

by may be an expression, an offset a row, each a String that Every reads, such as "1mo2d" (step 256): Polars' offset_by(expr). A null offset gives null. An offset that does not parse, or that the type cannot hold, is refused when it is read, naming its row.

func (DtExpr) OrdinalDay 

func (d DtExpr) OrdinalDay() Expr

OrdinalDay is the day of the year, 1-366.

func (DtExpr) Quarter 

func (d DtExpr) Quarter() Expr

Quarter is 1-4.

func (DtExpr) Replace added in v0.6.6 

func (d DtExpr) Replace(p DateTimeParts, opts ...ZoneOption) Expr

Replace sets the components p gives on each value's wall clock: Polars' dt.replace. 

Col("ts").Dt().Replace(DateTimeParts{Day: 1, Hour: 0})

A Date's time components are not read. A Microsecond sets the whole fraction of the second, a Datetime(ns)'s nanoseconds too. Components that are no date or time, the 30th of February or an hour of 24, are refused. A zoned value is read again in its zone, and a wall clock it reads twice or never is refused, unless opts answer it as ReplaceTimeZone's do; Polars takes ambiguous, and refuses a gap always.

func (DtExpr) ReplaceTimeZone added in v0.6.0 

func (d DtExpr) ReplaceTimeZone(tz string, opts ...ZoneOption) Expr

ReplaceTimeZone keeps each value's wall clock and reads it in the zone tz, so the instant moves: Polars' dt.replace_time_zone. A naive Datetime gains a zone, a zoned one changes it, and tz "" makes it naive, keeping the clock as it read. 

Col("ts").Dt().ReplaceTimeZone("Asia/Tokyo")   // 09:00 naive reads 09:00 in Tokyo, 00:00 UTC

The new zone may read a wall clock twice, in the hour its clocks fall back, or never, in the hour they spring forward. By default either refuses the query, naming the clock; AmbiguousEarliest, AmbiguousLatest or AmbiguousNull, and NonExistentNull, answer them instead. A value already in tz is kept as it is, as Polars keeps it. A zone the system does not know is refused while the query is planned.

Where Polars differs: it also takes a column of policies, one per row. Here a policy is the whole column's.

func (DtExpr) Round added in v0.5.0 

func (d DtExpr) Round[T Span](every T) Expr

Round rounds a Date or Datetime to the nearer boundary of Truncate's grid, and a value exactly halfway up: Polars' dt.round. 

Col("ts").Dt().Round(15 * time.Minute)   // 10:07:30 → 10:15, 10:07:29 → 10:00
Col("ts").Dt().Round(Every("1mo"))       // the nearer first of a month

The grid is Truncate's, in both spellings, and nearer is measured in elapsed time: the middle of February 2024 is 2024-02-15T12:00, and the middle of a 23-hour day 11:30 after its start. A Time is refused, because it can round up to 24:00, which it cannot hold.

func (DtExpr) Second 

func (d DtExpr) Second() Expr

func (DtExpr) SecondFractional added in v0.6.6 

func (d DtExpr) SecondFractional() Expr

SecondFractional is the second with its fraction, 0 up to 60, a Float64: Polars' second(fractional=True). A Datetime or a Time; a Date has no seconds.

func (DtExpr) Strftime added in v0.4.0 

func (d DtExpr) Strftime(format string) Expr

Strftime formats a Date, Datetime or Time with a strftime-style format, in a zoned Datetime's own wall clock — Polars' dt.strftime. 

Col("day").Dt().Strftime("%A, %e %B %Y")   // "Friday,  9 February 2024"

The directives are Strptime's. A format that asks for what the value does not have is refused while the query is planned: a date from a Time, or an offset or zone name (%z, %Z) from anything but a zoned Datetime.

func (DtExpr) Time added in v0.6.5 

func (d DtExpr) Time() Expr

Time is each instant's time of day on its own zone's wall clock, Polars' dt.time, in the Datetime's unit. A Time is itself; a Date has none and is refused. Cast reads the UTC wall clock instead, as Date's doc says.

func (DtExpr) Timestamp added in v0.6.6 

func (d DtExpr) Timestamp(unit TimeUnit) Expr

Timestamp is EpochIn in milliseconds, microseconds or nanoseconds: Polars' timestamp(time_unit), which takes no seconds.

func (DtExpr) ToString 

func (d DtExpr) ToString() Expr

ToString formats using the same ISO 8601 layouts the CSV writer emits, so a frame printed to a terminal and a frame written to a file agree.

func (DtExpr) TotalDays 

func (d DtExpr) TotalDays() Expr

The Total family converts a Duration to a whole number of units, truncating. They are the only temporal functions defined on a Duration.

func (DtExpr) TotalDaysFractional added in v0.6.6 

func (d DtExpr) TotalDaysFractional() Expr

The Fractional totals are the Total family with the unit's fraction, a Float64: Polars' total_*(fractional=True). Each is the tick count times the unit's reciprocal, as Polars divides by a constant, so each agrees with Polars' to the bit.

func (DtExpr) TotalHours 

func (d DtExpr) TotalHours() Expr

func (DtExpr) TotalHoursFractional added in v0.6.6 

func (d DtExpr) TotalHoursFractional() Expr

func (DtExpr) TotalMicroseconds added in v0.6.6 

func (d DtExpr) TotalMicroseconds() Expr

func (DtExpr) TotalMicrosecondsFractional added in v0.6.6 

func (d DtExpr) TotalMicrosecondsFractional() Expr

func (DtExpr) TotalMilliseconds added in v0.6.6 

func (d DtExpr) TotalMilliseconds() Expr

TotalMilliseconds, TotalMicroseconds and TotalNanoseconds are the Total family in the smaller units, truncating toward zero. A Duration whose total in nanoseconds is past an Int64 is refused.

func (DtExpr) TotalMillisecondsFractional added in v0.6.6 

func (d DtExpr) TotalMillisecondsFractional() Expr

func (DtExpr) TotalMinutes 

func (d DtExpr) TotalMinutes() Expr

func (DtExpr) TotalMinutesFractional added in v0.6.6 

func (d DtExpr) TotalMinutesFractional() Expr

func (DtExpr) TotalNanoseconds added in v0.6.6 

func (d DtExpr) TotalNanoseconds() Expr

func (DtExpr) TotalNanosecondsFractional added in v0.6.6 

func (d DtExpr) TotalNanosecondsFractional() Expr

func (DtExpr) TotalSeconds 

func (d DtExpr) TotalSeconds() Expr

func (DtExpr) TotalSecondsFractional added in v0.6.6 

func (d DtExpr) TotalSecondsFractional() Expr

func (DtExpr) Truncate 

func (d DtExpr) Truncate[T Span](every T) Expr

Truncate floors an instant to a multiple of every, toward negative infinity so instants before the epoch move backwards like every other instant.

A Duration floors ABSOLUTELY; an Interval floors on the CALENDAR

The two spellings answer different questions and neither is a special case of the other: 

Truncate(time.Hour)     the instant, floored to a whole hour since the epoch
Truncate(Every("1d"))   the start of the LOCAL day, in the column's own zone
Truncate(Every("1h"))   the start of the LOCAL hour — 10:00 for 10:47 at +05:30,
                        where time.Hour gives 10:30, the hour since the epoch

An Interval floors the wall clock at every size. Where the floored wall clock falls into a spring-forward gap, the answer is the instant the gap ends; in a fall-back fold, the input keeps its own offset, so the second 01:30 floors to the second 01:00. Both are Polars' answers.

A Duration is a fixed span of elapsed time, so flooring by one is zone-independent by definition — `Truncate(24*time.Hour)` on a New York column lands on 00:00 UTC, which is 19:00 or 20:00 local, and that is the correct answer to the question a Duration asks. It is almost never the question a user grouping by day is asking, which is why Every("1d") exists and reads the column's timezone.

func (DtExpr) Week 

func (d DtExpr) Week() Expr

Week is the ISO 8601 week number, which is not "day of year / 7" — the first week of a year can begin in the previous one.

func (DtExpr) Weekday 

func (d DtExpr) Weekday() Expr

Weekday is ISO: Monday is 1 and Sunday is 7. Go's time.Weekday puts Sunday at 0, and passing that through is the classic off-by-one in date libraries.

func (DtExpr) WithTimeUnit added in v0.6.6 

func (d DtExpr) WithTimeUnit(unit TimeUnit) Expr

WithTimeUnit reads a Datetime's or a Duration's integers in unit, so the values change by the ratio of the units: Polars' with_time_unit. A Datetime keeps its zone. Cast converts instead, keeping the values.

func (DtExpr) Year 

func (d DtExpr) Year() Expr

type DummiesOptions added in v0.6.7 

type DummiesOptions struct {
	// Columns are the columns to encode, every column when empty. A Series encodes
	// itself and does not read them.
	Columns []string

	// Separator joins a column's name to a value's, "_" when empty.
	Separator string

	// DropFirst leaves out the first value to appear that is not null, which the
	// others imply, and DropNulls the null.
	DropFirst, DropNulls bool
}

DummiesOptions configures ToDummies.

type DurationParts added in v0.6.6 

type DurationParts struct {
	Weeks, Days, Hours, Minutes, Seconds, Milliseconds, Microseconds, Nanoseconds any
}

DurationParts are a duration's parts for MakeDuration, Polars' pl.duration keywords. Each is nil, for none, a Go number, or an Expr of numbers.

type DynamicOptions 

type DynamicOptions struct {
	// Every is where windows START. Consecutive windows are Every apart.
	Every Interval

	// Period is a window's WIDTH. Zero means Every, which gives windows that tile
	// the input exactly. A Period larger than Every gives OVERLAPPING windows, and a
	// row then belongs to several of them — so a Len() over the result exceeds the
	// input's row count, which is the correct answer and surprises everyone once.
	Period Interval

	// Offset shifts the whole grid. A day grid with Offset 9h starts its windows at
	// 09:00 rather than at midnight.
	Offset Interval

	// Closed says which end of a window belongs to it. The zero value is
	// ClosedLeft — [start, end) — which is the only convention under which a grid
	// partitions its input exactly.
	Closed Closed

	// GroupBy adds CATEGORICAL keys. Windows are cut independently inside each
	// distinct combination, so an hourly rollup per service is one call.
	GroupBy []Expr

	// Label is what each window's index column holds: "left", its start, the default;
	// "right", its end, the start plus Period; or "datapoint", the instant of its
	// first row, or its start for a window with no row. Polars' label (step 187).
	Label string

	// StartBy is where the grid begins: "window", Every's own grid, the default;
	// "datapoint", the first row's instant, with Offset ignored, as Polars ignores it;
	// or a weekday, "monday" to "sunday", that a grid of weeks starts on: the one at
	// or before the grid's own first Monday, plus Offset. A weekday on a grid not in
	// weeks is refused: Polars documents it for weeks alone. Polars' start_by.
	StartBy string

	// IncludeBoundaries adds _lower_boundary and _upper_boundary, each window's start
	// and end, before the index column: Polars' include_boundaries.
	IncludeBoundaries bool
}

DynamicOptions configures GroupByDynamic.

Every field is implemented

Every, Period, Offset, Closed and GroupBy since step 40; Label, StartBy and IncludeBoundaries since step 187. Until then those three produced an error saying so rather than being accepted and dropped, which is the rule a new field still follows: plan.Aggregate.MaintainOrder once sat assigned, rendered and unread for four steps, and a field that is accepted and does nothing is worse than one that does not exist, because the next reader believes it.

type Encoding added in v0.6.6 

type Encoding string

Encoding is a transfer encoding for Encode and Decode. 

const (
	Hex    Encoding = "hex"
	Base64 Encoding = "base64"
)

type Endianness added in v0.6.6 

type Endianness string

Endianness is the byte order Reinterpret reads. 

const (
	LittleEndian Endianness = "little"
	BigEndian    Endianness = "big"
)

type EwmDecay added in v0.5.0 

type EwmDecay struct {
	// contains filtered or unexported fields
}

EwmDecay is how fast an exponentially weighted function forgets: the weight a row loses with each row after it. Build it from whichever quantity is to hand, as Polars' com, span, half_life and alpha are.

func EwmAlpha added in v0.5.0 

func EwmAlpha(alpha float64) EwmDecay

EwmAlpha is the decay itself: each row after keeps 1-alpha of a row's weight. 0 < alpha <= 1.

func EwmCom added in v0.5.0 

func EwmCom(com float64) EwmDecay

EwmCom is a decay by a centre of mass: alpha = 1/(1+com), for com >= 0.

func EwmHalfLife added in v0.5.0 

func EwmHalfLife(halfLife float64) EwmDecay

EwmHalfLife is a decay by the rows over which a weight halves: alpha = 1 - exp(-ln 2 / halfLife), for halfLife > 0.

func EwmSpan added in v0.5.0 

func EwmSpan(span float64) EwmDecay

EwmSpan is a decay by an N-row span: alpha = 2/(span+1), for span >= 1.

type ExplainOption 

type ExplainOption func(*explainCfg)

ExplainOption configures Explain.

func ExplainOptFlags added in v0.6.8 

func ExplainOptFlags(f OptFlags) ExplainOption

ExplainOptFlags renders the plan as WithOptFlags(f) runs it: the rules f turns off left out, where Explain would run them all.

func Optimized 

func Optimized(b bool) ExplainOption

Optimized selects the optimized plan (the default) or the plan as written.

func WithSchemas 

func WithSchemas() ExplainOption

WithSchemas annotates every plan node with its output schema.

type Expr 

type Expr struct {
	// contains filtered or unexported fields
}

Expr is a lazy description of a column transformation.

It is a CONCRETE STRUCT rather than an interface, and that is forced by Go 1.27: interface methods may not declare type parameters, and a generic method can never satisfy an interface method. Since Gt[T Operand] — the thing that makes `Col("age").Gt(30)` work without wrapping 30 in Lit — must be a generic method, Expr cannot be an interface. Polymorphism lives in the unexported expr.Node instead.

func All 

func All() Expr

All selects every column.

func AllHorizontal added in v0.6.3 

func AllHorizontal(exprs ...Expr) Expr

func AnyHorizontal added in v0.6.3 

func AnyHorizontal(exprs ...Expr) Expr

AnyHorizontal and AllHorizontal are each row's Or and And across exprs, Kleene's, as Polars' any_horizontal and all_horizontal: a true decides Any and a false decides All, and a row that nothing decides is null if it holds a null. The expressions must be Boolean. The answer is named after the first.

func Arctan2 added in v0.6.7 

func Arctan2(y, x Expr) Expr

Arctan2 is the angle of the point (x, y) from the positive x axis, in radians from -π to π, as Polars' arctan2(y, x), y first (step 324). Integers and Decimals give a Float64, two Float32s a Float32; a null either side is null.

func ArgSortBy added in v0.6.6 

func ArgSortBy(keys ...SortKey) Expr

ArgSortBy is the indices that sort the frame by keys, as Sort(keys...) sorts it, UInt32: Polars' arg_sort_by. Each key is Asc or Desc, NullsLast or not. In an aggregation it takes one key; Polars 1.44.1 answers several there with indices past the group's end.

func ArgWhere added in v0.6.6 

func ArgWhere(pred Expr) Expr

ArgWhere is the indices of the rows pred is true on: Polars' arg_where, pred.ArgTrue().

func BusinessDayCount added in v0.6.6 

func BusinessDayCount[S, E Operand](start S, end E, cal ...BusinessCalendar) Expr

BusinessDayCount is the business days from start up to end, end excluded, an Int32: Polars' business_day_count. Where end is before start it is minus the business days from end up to start. The ends are Dates.

func Coalesce 

func Coalesce(exprs ...Expr) Expr

Coalesce returns the first non-null value across the given expressions.

Built out of conditionals rather than its own kernel

Coalesce(a, b, c) becomes when(a.is_not_null()).then(a).otherwise(coalesce(b, c)), which is correct and costs one thing worth knowing: each operand is MENTIONED twice, and there is no common-subexpression pass, so each is EVALUATED twice. For the ordinary Coalesce(Col("a"), Col("b")) that is two extra column reads; for an operand that is itself an expensive expression it is not free.

A dedicated n-ary null-merge kernel would remove that and change no semantics. It is deliberately not built yet: the desugaring needs no new IR node, no new kernel and no new arms in the five type switches a node type must reach.

func Col 

func Col(names ...string) Expr

Col selects one or more columns by name.

With several names it becomes a multi-column selection that expands at plan time: `Col("a","b").Mul(2)` is two independent expressions.

func ColDType 

func ColDType(types ...dtype.DataType) Expr

ColDType selects every column of one of the given types. How many columns that is is determined at plan time from the schema, so the same expression adapts as the schema changes.

func ColRegex 

func ColRegex(pattern string) Expr

ColRegex selects every column whose name matches pattern.

The pattern is explicit rather than inferred from leading ^ and trailing $ the way Polars does. That inference makes a column literally named "^total$" unselectable and taxes every plain Col call with an anchor scan.

func ConcatArr added in v0.6.7 

func ConcatArr(exprs ...Expr) Expr

ConcatArr is each row's operands in one Array, Polars' concat_arr: an Array operand's elements and another's value, in order, so two Arrays of 2 and of 3 and a value make an Array of 6. The operands are of one element type, as Polars wants them: cast one first. A null Array makes the row null, and a null value is a null element.

New in 0.6.7.

func ConcatList added in v0.6.6 

func ConcatList(exprs ...Expr) Expr

ConcatList is each row's operands in one List: a List operand's elements and another's value, Polars' concat_list. The elements meet at the type a conditional's branches meet at; a null value is a null element, and a null List makes the row's list null, as Polars makes it. Where Polars casts numbers beside strings to text, this refuses them: cast first.

func ConcatStr added in v0.5.0 

func ConcatStr(sep string, exprs ...Expr) Expr

func ConcatStrIgnoreNulls added in v0.6.6 

func ConcatStrIgnoreNulls(sep string, exprs ...Expr) Expr

ConcatStr joins its expressions' values, row by row, with sep between each two: Polars' concat_str. 

ConcatStr(" ", Col("first"), Col("last"))           // "Ada Lovelace"
ConcatStr("", Col("code"), Lit("-"), Col("n"))       // "AB-7", n formatted

A value that is not a String is formatted as a Cast to String formats it. A row is null where any of its values is, as Polars' default is; to skip a null instead, fill it first, e.g. Col("middle").FillNull(""). The answer is named after the first expression. ConcatStrIgnoreNulls is ConcatStr past nulls, Polars' concat_str(ignore_nulls=True): each row's values that are not null joined by sep, and a row of nulls alone "", not null (step 271).

func Corr added in v0.6.0 

func Corr(a, b Expr) Expr

Corr is Pearson's correlation of a and b over the rows where both are non-null: Polars' pl.corr. It is a Float64, named after a, and NaN, never null, where it is undefined: no such rows, one, or a variable that does not vary. A NaN in either makes it NaN. 

GroupBy(Col("g")).Agg(Corr(Col("x"), Col("y")))

It runs on every worker and merges by Chan's formulas, as Var does. Spearman's rank correlation, Polars' method="spearman", is not built.

func CorrSpearman added in v0.6.6 

func CorrSpearman(a, b Expr) Expr

CorrSpearman is Spearman's rank correlation of a and b over the rows where both are non-null: Pearson's of their ranks, a tie the mean of its ranks and NaN above every number, Polars' pl.corr(method="spearman"). Fewer than two such rows, or values all alike, are NaN, as Polars answers. It keeps every pair of a group, as Median keeps its values. Polars ranks text too; this takes numbers.

func Cov added in v0.6.0 

func Cov(a, b Expr, ddof int) Expr

Cov is the covariance of a and b over the rows where both are non-null: Polars' pl.cov. ddof is Var's: 1 for the sample covariance, Polars' default, and 0 for the population's. A group of ddof or fewer such rows is null, as Var is.

Where Polars differs: over a single row with ddof 1, it answers 0.0 when neither input has a null, and null when nulls left the one row; here both are null.

func CumFold added in v0.6.6 

func CumFold[A Operand](acc A, fn func(acc, x Expr) Expr, exprs ...Expr) Expr

CumFold is Fold's running values, a Struct named "cum_fold" with a field for each expression, named after it: Polars' cum_fold.

func CumReduce added in v0.6.6 

func CumReduce(fn func(acc, x Expr) Expr, exprs ...Expr) Expr

CumReduce is Reduce's running values, a Struct named "cum_reduce" whose first field is the first expression: Polars' cum_reduce.

func CumSumHorizontal added in v0.6.6 

func CumSumHorizontal(exprs ...Expr) Expr

CumSumHorizontal is the running sums across exprs, a Struct named "cum_sum", every field of the total's type: Polars' cum_sum_horizontal. A null makes the sums from it on null.

func DateRange added in v0.6.6 

func DateRange[S, E Operand, T Span](start S, end E, every T, closed ...Closed) Expr

DateRange is the dates from start to end every interval, a Date column: Polars' date_range. A Datetime end is read as its UTC date, as a cast reads it. The interval is whole days, weeks, months or years. 

lf.Select(DateRange(Lit(jan31).Cast(Date), Lit(may1).Cast(Date), Every("1mo")))

func DateRanges added in v0.6.6 

func DateRanges[S, E Operand, T Span](start S, end E, every T, closed ...Closed) Expr

DateRanges is DateRange a row, a List(Date): Polars' date_ranges.

func DatetimeRange added in v0.6.6 

func DatetimeRange[S, E Operand, T Span](start S, end E, every T, closed ...Closed) Expr

DatetimeRange is the instants from start to end every interval: Polars' datetime_range. Its type is the ends', two Dates Datetime(us); cast the ends for another unit or zone, e.g. Lit(t).Cast(Datetime(Micro, "")).Dt().ReplaceTimeZone(z), and the range steps on z's wall clock.

func DatetimeRanges added in v0.6.6 

func DatetimeRanges[S, E Operand, T Span](start S, end E, every T, closed ...Closed) Expr

DatetimeRanges is DatetimeRange a row, a List: Polars' datetime_ranges.

func Element added in v0.6.6 

func Element() Expr

Element is a list's elements, inside List().Eval, Agg and Filter: Polars' pl.element(). Anywhere else it is refused.

func Exclude 

func Exclude(names ...string) Expr

Exclude selects every column except the named ones.

func ExcludeRegex 

func ExcludeRegex(pattern string) Expr

ExcludeRegex selects every column whose name does not match pattern.

func First added in v0.6.6 

func First(names ...string) Expr

First is the frame's first column, Nth(0), or with names their first values, Col(names...).First(): Polars' first.

func Fold added in v0.6.6 

func Fold[A Operand](acc A, fn func(acc, x Expr) Expr, exprs ...Expr) Expr

Fold is fn applied from acc across exprs, acc = fn(acc, x) for each: Polars' fold, named after acc. 

Fold(0, func(acc, x Expr) Expr { return acc.Add(x) }, All())   // every column's sum

fn builds an expression, once, when the query is planned, and the expressions are written out first, so a selection of several columns folds over each. Polars runs its function over the columns per batch; built ahead, the fold is an expression the optimizer sees through, and no function of yours runs per row.

func Format added in v0.6.6 

func Format(f string, args ...Expr) Expr

Format is f with each {} replaced by the next of args as text, {{ and }} a brace: Polars' format, a ConcatStr of its pieces. A null argument makes the row null. The column is named after the first argument, "literal" with none, as Polars names it. As many arguments as placeholders are wanted.

func FromEpoch added in v0.6.6 

func FromEpoch(e Expr, unit TimeUnit) Expr

FromEpoch is the naive instant e units after 1970-01-01T00:00:00: Polars' from_epoch. Seconds, milliseconds and microseconds give a Datetime(us), nanoseconds a Datetime(ns). An integer count past the type's range is refused, where Polars wraps; a float one is truncated toward zero at the answer's unit, as Polars truncates it. Days since the epoch are e.Cast(Date).

func HashRows added in v0.6.7 

func HashRows(seed uint64, exprs ...Expr) Expr

HashRows is each row's 64-bit hash over the given columns, every column when none is given, seeded, a UInt64: Polars' DataFrame.hash_rows, here an expression, as Polars' pl.struct(...).hash() is one.

Rows a group-by takes as one hash alike: -0 with 0, NaN with NaN, and a row of nulls has a hash of its own. The columns' order counts. The algorithm is Hash's, over the row's group-by key, so the numbers are not Polars' (v0.6.6-scope.md §6.3); one seed takes four in Polars. A List or a Struct column is refused, being no group-by key here, where Polars hashes one. The answer is named after its first column; alias it.

func IntRange added in v0.6.6 

func IntRange[S, E Operand](start S, end E, step int) Expr

IntRange is the integers from start up to end, end excluded, by step, as Int64: Polars' int_range. A negative step counts down, a range that step points away from is empty, and a step of zero is refused. Cast the column for another integer type.

func IntRanges added in v0.6.6 

func IntRanges[S, E, T Operand](start S, end E, step T) Expr

IntRanges is IntRange a row, a List(Int64): Polars' int_ranges. Each argument is a constant or an expression, and a null among a row's makes its list null.

func IsDuplicated added in v0.6.7 

func IsDuplicated(exprs ...Expr) Expr

IsDuplicated and IsUnique are Expr.IsDuplicated and IsUnique over several columns at once: whether each row's values of exprs together are another row's too, or no other row's (step 290). Polars' pl.struct(...).is_duplicated() and DataFrame.is_duplicated(). They are named after their first expression, as Polars names a struct after its first field. A key's equality is grouping's, so nulls, NaNs and -0 with 0 are each equal, as Unique compares them.

The columns are named: All() here would expand the whole expression once a column, as Polars' over(pl.all()) does, each partitioned by one column. Whole rows are DataFrame.IsDuplicated and IsUnique.

func IsUnique added in v0.6.7 

func IsUnique(exprs ...Expr) Expr

func Last added in v0.6.6 

func Last(names ...string) Expr

Last is the frame's last column, Nth(-1), or with names their last values: Polars' last.

func Len 

func Len() Expr

Len counts rows in the current group — SQL's COUNT(*) with no column argument.

It aggregates over a literal rather than a column, because counting rows must not depend on any column existing: `GroupBy(k).Agg(Len())` has to work on a frame whose only column is the key. It is named "len", as Polars' pl.len() is; the literal's name, "literal", showed through in some plans until step 273. The name is OutputName's (step 287): an Alias node here rendered inside a cast, so h2o's gb9 planned other than at v0.6.5.

func LinearSpace added in v0.6.6 

func LinearSpace[S, E Operand](start S, end E, n int, closed ...Closed) Expr

LinearSpace is n values evenly spaced from start to end: Polars' linear_space. Both ends belong unless closed says otherwise, ClosedLeft, ClosedRight or ClosedNone.

Two numbers give Float64, two Float32 ends Float32. Two values of one temporal type give that type, spaced over its ticks and truncated toward zero, and two Dates a Datetime(us), as Polars gives them. The values are Polars' to the bit: each but the last is the first plus i steps, and the last is the far end exactly.

func LinearSpaces added in v0.6.6 

func LinearSpaces[S, E, N Operand](start S, end E, n N, closed ...Closed) Expr

LinearSpaces is LinearSpace a row, a List: Polars' linear_spaces. n may be an expression too.

func Lit 

func Lit[T Literal | ListLiteral](v T) Expr

Lit builds a literal: a value, or a Go slice, which is one List value (step 257).

A List literal is one row: alone in Select it is a frame of one row, beside a column it is that list in every row, and against a column a List argument, the same for every row: Col("x").IsIn(Lit([]int64{1, 2})), whose list goes into a hash set once.

func MakeDate added in v0.6.6 

func MakeDate[Y, M, D Operand](year Y, month M, day D) Expr

MakeDate is the date of year, month and day, integers, a column named "date": Polars' pl.date. 

lf.Select(MakeDate(Col("y"), Col("m"), 1))   // the first of each row's month

func MakeDatetime added in v0.6.6 

func MakeDatetime[Y, M, D, H, Mi, S, U Operand](year Y, month M, day D, hour H, minute Mi, second S,
	microsecond U) Expr

MakeDatetime is the naive Datetime(us) of its components, a column named "datetime": Polars' pl.datetime, every component given. For another unit cast it; for a zone, Dt().ReplaceTimeZone(zone), whose options say what an ambiguous or skipped wall clock becomes, as Polars' ambiguous does.

func MakeDuration added in v0.6.6 

func MakeDuration(p DurationParts, unit ...TimeUnit) Expr

MakeDuration is the Duration of its parts, a column named "duration": Polars' pl.duration. The unit is unit if given, else microseconds, or nanoseconds when Nanoseconds is set, as Polars chooses it. 

lf.Select(MakeDuration(DurationParts{Days: Col("d"), Hours: 12}))

Integer parts sum exactly, and a total past the type's range is refused, where Polars wraps. With a float part the total is summed as a float and truncated toward zero, as Polars sums it. Polars names a duration of columns "literal"; this is "duration" either way.

func MakeTime added in v0.6.6 

func MakeTime[H, Mi, S, U Operand](hour H, minute Mi, second S, microsecond U) Expr

MakeTime is the time of day of its components, a Time(ns) column named "time": Polars' pl.time, every component given.

func MaxHorizontal added in v0.6.3 

func MaxHorizontal(exprs ...Expr) Expr

MaxHorizontal and MinHorizontal are each row's largest and smallest value across exprs, as Polars' max_horizontal and min_horizontal. The values meet at one type as a conditional's branches do, an Int32 and a Float64 at Float64, and are compared as Max and Min compare a column's: nulls skipped, null when every value is, and NaN the largest, where Polars skips it. The answer is named after the first expression.

func MeanHorizontal added in v0.6.3 

func MeanHorizontal(exprs ...Expr) Expr

func MinHorizontal added in v0.6.3 

func MinHorizontal(exprs ...Expr) Expr

func Nth added in v0.6.6 

func Nth(indices ...int) Expr

Nth selects columns by position, in the order given: Polars' nth. A negative index counts from the end, and one past either end is refused when the query is planned. Like Col("a", "b"), it expands: Nth(0, 1).Mul(2) is two columns.

func Null 

func Null(dt dtype.DataType) Expr

Null builds a typed null literal.

func Ones added in v0.6.6 

func Ones[N Operand](n N) Expr

Ones and Zeros are 1.0 and 0.0 n times, Float64 columns named "ones" and "zeros": Polars' ones and zeros. Cast them for another type.

func Reduce added in v0.6.6 

func Reduce(fn func(acc, x Expr) Expr, exprs ...Expr) Expr

Reduce is Fold whose accumulator is the first expression, named after it: Polars' reduce.

func Repeat added in v0.6.6 

func Repeat[V, N Operand](value V, n N) Expr

Repeat is value n times, a column named "repeat": Polars' repeat. Both are constants or aggregates: Repeat(Col("x").Sum(), Len()) is the total on every row. A literal integer is Int64 here, where Polars makes it Int32.

func RollingCorr added in v0.6.6 

func RollingCorr(a, b Expr, size int, opts ...WindowOption) Expr

func RollingCov added in v0.6.6 

func RollingCov(a, b Expr, size int, opts ...WindowOption) Expr

RollingCov is the covariance of a and b over each row's window of size rows, and RollingCorr their Pearson correlation: Polars' rolling_cov and rolling_corr (step 284). They are built as Polars builds them, from rolling means and variances, pandas' way, over the window's rows where both hold a value, n of them: 

cov  = (mean(a·b) − mean(a)·mean(b)) · n/(n − ddof)
corr = the cov at ddof 1 / sqrt(var(a)·var(b))

Ddof sets the covariance's, 1 by default; a correlation does not depend on it, as Polars says of its own. Each is a Float32 for two Float32s and a Float64 otherwise. A window needs size such rows, or MinSamples, and a correlation two.

func Struct added in v0.5.0 

func Struct(exprs ...Expr) Expr

Struct builds a Struct column whose fields are its expressions, each named as its expression is: Polars' pl.struct. Struct().Field reads a field back. 

Struct(Col("lat"), Col("lon")).Alias("point")   // {lat: 51.5, lon: -0.1}

Two expressions of one name would be two fields Field could not tell apart, so the query is refused; alias one. The struct itself is never null: a row whose values are all null is a struct of nulls, which StructExpr's doc tells apart from a null struct. The answer is named after the first expression.

func SumHorizontal added in v0.6.3 

func SumHorizontal(exprs ...Expr) Expr

SumHorizontal and MeanHorizontal are each row's sum and mean across exprs, as Sum and Mean give a column's: nulls skipped, an integer sum widened to Int128, a mean the nearest double to the exact one, and either null when every value is null. Polars' sum_horizontal keeps an integer operand's type, so wraps, and sums a row of nulls to 0. The answer is named after the first expression.

func TimeRange added in v0.6.6 

func TimeRange[S, E Operand, T Span](start S, end E, every T, closed ...Closed) Expr

TimeRange is the times of day from start to end every interval, a Time column: Polars' time_range. Polars' defaults, midnight to 23:59:59.999999 by the hour, are spelled out here.

func TimeRanges added in v0.6.6 

func TimeRanges[S, E Operand, T Span](start S, end E, every T, closed ...Closed) Expr

TimeRanges is TimeRange a row, a List(Time): Polars' time_ranges.

func Zeros added in v0.6.6 

func Zeros[N Operand](n N) Expr

func (Expr) Abs 

func (e Expr) Abs() Expr

func (Expr) Add 

func (e Expr) Add[T Operand](v T) Expr

func (Expr) Alias 

func (e Expr) Alias(name string) Expr

Alias renames the output column.

It sets ONE fixed name, so applying it to a multi-column selection is an error. Use Prefix or Suffix to rename an expansion.

func (Expr) AllTrue 

func (e Expr) AllTrue() Expr

func (Expr) And 

func (e Expr) And[T Operand](v T) Expr

And, Or and Xor of two Booleans are Kleene logic. Of two integers they are the integers' bits, at their common type as arithmetic's; a Go integer beside one is of its type where it fits, as Polars keeps Col("u8").And(6) a UInt8 (step 278). A Boolean beside an integer is refused.

func (Expr) Any 

func (e Expr) Any() Expr

Any and AllTrue reduce a Boolean column.

AllTrue rather than All, because All is already the top-level selector for every column and one name cannot mean both.

Nulls are SKIPPED, so a group with no non-null value gives NULL rather than false (for Any) or true (for AllTrue). Polars returns the vacuous answer; ursus returns null, because every other aggregate here does and an inconsistent one is a permanent trap. FillNull makes either convention available.

func (Expr) Append added in v0.6.6 

func (e Expr) Append(other Expr, upcast bool) Expr

Append is the values, then other's: Polars' append, a column of its own length in a Select and a list a group in an aggregation. With upcast the two meet at their common type, as Polars' default does; without, they must be of one type. Polars meets text with anything as text; a common type here is a numeric one.

func (Expr) ApproxNUnique added in v0.6.6 

func (e Expr) ApproxNUnique() Expr

ApproxNUnique is the distinct values, a null among them, as Polars' approx_n_unique counts it. It is exact: Polars estimates, and an exact count is an estimate's best case (v0.6.6-scope.md §6.3). NUnique does not count a null.

func (Expr) ArcCos added in v0.4.0 

func (e Expr) ArcCos() Expr

func (Expr) ArcCosh added in v0.4.0 

func (e Expr) ArcCosh() Expr

func (Expr) ArcSin added in v0.4.0 

func (e Expr) ArcSin() Expr

func (Expr) ArcSinh added in v0.4.0 

func (e Expr) ArcSinh() Expr

func (Expr) ArcTan added in v0.4.0 

func (e Expr) ArcTan() Expr

func (Expr) ArcTanh added in v0.4.0 

func (e Expr) ArcTanh() Expr

func (Expr) ArgMax 

func (e Expr) ArgMax() Expr

func (Expr) ArgMin 

func (e Expr) ArgMin() Expr

ArgMin and ArgMax give the POSITION of the smallest or largest value within the group, as a Uint64.

The index counts every row of the group, nulls included, so it lines up with anything else gathered from the same group — but nulls can never win, and a group with no non-null value has no position and returns NULL.

Ties go to the earliest row, which is what makes the answer independent of batch size and thread count.

func (Expr) ArgSort added in v0.6.6 

func (e Expr) ArgSort(descending, nullsLast bool) Expr

ArgSort is the indices that sort the column, a UInt32 column of its length: Polars' arg_sort. Ties keep their order, in either direction, and nulls come first unless nullsLast. In an aggregation it is a list a group, of positions within it.

func (Expr) ArgTrue added in v0.6.6 

func (e Expr) ArgTrue() Expr

ArgTrue is the indices of the true values of a Boolean column, UInt32: Polars' arg_true. A null is not true.

func (Expr) ArgUnique added in v0.6.6 

func (e Expr) ArgUnique() Expr

ArgUnique is the index of each distinct value's first row, in the order they first appear, a null among them: Polars' arg_unique.

func (Expr) Arr added in v0.6.7 

func (e Expr) Arr() ArrExpr

Arr is the array namespace: functions of an Array column's rows.

func (Expr) BackwardFill 

func (e Expr) BackwardFill(limit int) Expr

BackwardFill replaces each null with the next non-null value after it.

It is ForwardFill walked in the other direction, which costs nothing: the ordered walk flips its direction rather than building a second permutation.

func (Expr) Bin added in v0.6.6 

func (e Expr) Bin() BinExpr

Bin opens the binary namespace.

func (Expr) BitwiseAnd added in v0.6.6 

func (e Expr) BitwiseAnd() Expr

BitwiseAnd, BitwiseOr and BitwiseXor fold the non-null values by their bits, an integer or a Boolean, of the input's type: Polars' bitwise_and and its kin. A group of no value is null.

func (Expr) BitwiseCountOnes added in v0.5.0 

func (e Expr) BitwiseCountOnes() Expr

The bit counts of an integer of 8 to 64 bits, each a Uint32, in the type's own width: Polars' bitwise_count_ones and its siblings. An Int8 -1 has eight ones and no zeros, and an Int8 1 has seven leading zeros where an Int64 1 has 63. Zero's trailing zeros are its width.

func (Expr) BitwiseCountZeros added in v0.5.0 

func (e Expr) BitwiseCountZeros() Expr

func (Expr) BitwiseLeadingOnes added in v0.5.0 

func (e Expr) BitwiseLeadingOnes() Expr

func (Expr) BitwiseLeadingZeros added in v0.5.0 

func (e Expr) BitwiseLeadingZeros() Expr

func (Expr) BitwiseOr added in v0.6.6 

func (e Expr) BitwiseOr() Expr

func (Expr) BitwiseTrailingOnes added in v0.5.0 

func (e Expr) BitwiseTrailingOnes() Expr

func (Expr) BitwiseTrailingZeros added in v0.5.0 

func (e Expr) BitwiseTrailingZeros() Expr

func (Expr) BitwiseXor added in v0.6.6 

func (e Expr) BitwiseXor() Expr

func (Expr) BottomK added in v0.4.0 

func (e Expr) BottomK(k int) Expr

BottomK is the k smallest values of the group, smallest first, as a List. See TopK, whose rules it follows.

func (Expr) BottomKBy added in v0.6.7 

func (e Expr) BottomKBy(by Expr, k int) Expr

BottomKBy is the k values whose by is the least, least first, as a List: Polars' bottom_k_by. A null by still comes last, and rows of one by are ordered by their values, the lesser first. See TopKBy, whose rules it follows.

func (Expr) Cast 

func (e Expr) Cast(to dtype.DataType) Expr

Cast converts to another type, failing the query on the first unrepresentable value.

What is representable depends on the target:

  • An integer, a Decimal: exactly the value, in range. 3.5 is not an Int64, and 0.123 is not a Decimal(10, 2).
  • A float: the nearest value of its width, so 0.1 cast to Float32 is the float32 nearest 0.1, and an Int64 above 2^53 is the float64 nearest it. Only a finite value past the type's range — one that would round to an infinity — is unrepresentable.
  • From a String: the text parsed at the target's own width, by the same rules. "256" is not a Uint8, and text that is no number cannot be parsed at all.

func (Expr) CastLossy 

func (e Expr) CastLossy(to dtype.DataType) Expr

CastLossy converts to another type, turning unrepresentable values into nulls. The result is always nullable as a consequence.

func (Expr) Cat added in v0.6.6 

func (e Expr) Cat() CatExpr

Cat opens the categorical namespace.

func (Expr) Cbrt 

func (e Expr) Cbrt() Expr

func (Expr) Ceil 

func (e Expr) Ceil() Expr

func (Expr) Clip 

func (e Expr) Clip[L, H Operand](lo L, hi H) Expr

Clip bounds each value to [lo, hi].

Sugar, and that is what lets the bounds be expressions 

When(e.Lt(lo)).Then(lo).When(e.Gt(hi)).Then(hi).Otherwise(e)

A parameterised Call could not do this: CallArgs requires every argument to be a literal and rejects a column-valued one by name. As a conditional, Clip takes Expr bounds naturally — Col("x").Clip(Col("floor"), Col("cap")) works.

The bounds are WEAK literals, like a fill value

So Col("i32").Clip(0, 100) stays Int32. It used to become Int64, because the When/Then builder lifts strongly — and Col("u64").Clip(0, 100) became Int128, which is a startling type for clipping an unsigned column to a small range. That is exactly the inconsistency weak literals were built to remove: both spellings look like "a Go scalar bound on a column", and only one of them preserved the type. A bound that does not fit falls back to ordinary promotion, the same as a fill value that does not fit.

An Expr bound is used as written, so Clip(Lit(0), Lit(100)) still widens.

Edge cases, as Polars answers them 

e is null       → null
e is NaN        → NaN < lo and NaN > hi are both false → e, unchanged
a bound is null → no bound on that side, for that row (step 283)
lo > hi         → lo wins, because the chain is right-nested in written order

A null bound was a null result until step 283; it is no bound now, as Polars' None is. Each comparison's null is taken as false, so a row whose bound is null passes that side unchanged. ClipLower and ClipUpper bound one side alone.

func (Expr) ClipLower added in v0.6.6 

func (e Expr) ClipLower[L Operand](lo L) Expr

ClipLower bounds each value below by lo, and ClipUpper above by hi: Polars' clip with one bound. Each is Clip's sugar, its bound weak as Clip's are.

func (Expr) ClipUpper added in v0.6.6 

func (e Expr) ClipUpper[H Operand](hi H) Expr

func (Expr) Cos added in v0.4.0 

func (e Expr) Cos() Expr

func (Expr) Cosh added in v0.4.0 

func (e Expr) Cosh() Expr

func (Expr) Cot added in v0.6.6 

func (e Expr) Cot() Expr

Cot is the cotangent, 1/tan, as Polars computes it: Cot(0) is +Inf and Cot(-0) -Inf. It widens as Tan does. Its tangent is Go's, within 3 ulps of Polars' glibc one over ordinary values, as each trigonometric function here is within a few; near a multiple of pi/2 its relative error grows, its absolute error below 1e-30.

func (Expr) Count 

func (e Expr) Count() Expr

Count counts NON-NULL values — SQL's COUNT(col).

func (Expr) CumCount 

func (e Expr) CumCount(reverse bool) Expr

CumCount numbers rows within the window, starting at 1.

With reverse it counts from the end, so the last row is 1. That is what makes IsLastDistinct expressible as a window rather than a second pass.

func (Expr) CumMax 

func (e Expr) CumMax(reverse bool) Expr

func (Expr) CumMin 

func (e Expr) CumMin(reverse bool) Expr

func (Expr) CumProd 

func (e Expr) CumProd(reverse bool) Expr

func (Expr) CumSum 

func (e Expr) CumSum(reverse bool) Expr

CumSum, CumProd, CumMin and CumMax are running reductions over the window.

Nulls are SKIPPED by the running value and PRESERVED in place: cum_sum of [1, null, 3] is [1, null, 4], not [1, 1, 4] and not [1, null, null]. The running total ignores what is not there; the output still says the value was missing.

CumSum uses the same accumulator width as Sum, so the last row of `cum_sum(x)` equals `sum(x)` rather than differing by an overflow. CumProd returns Float64 for the reason Product does: i128 has addition but no multiplication.

func (Expr) CumulativeEval added in v0.6.6 

func (e Expr) CumulativeEval(body Expr, opts ...WindowOption) Expr

CumulativeEval answers each row with body over the values up to and including it, in the window's order: Polars' cumulative_eval (step 286). body reads Element(), as List().Eval's does, and is one value a prefix, an aggregate such as Element().Sum() or Element().First().Sub(Element().Last().Pow(2)); another column is refused, as Polars refuses it. A row whose prefix holds fewer than MinSamples values, 1 by default, is null.

It is a window: over the frame in its order, or .Over(key) a partition's own. Each row evaluates its whole prefix, so its time grows with the square of a partition's rows, as Polars' does; the prefixes are built a run of rows at a time. A window over the whole frame needs the frame in memory.

func (Expr) Cut added in v0.5.0 

func (e Expr) Cut(breaks []float64, opts ...CutOption) Expr

Cut puts each value in a bin between breaks, and answers the bin's label, an Enum of the labels in their order, as Polars' cut does (step 350; a String before). 

Col("age").Cut([]float64{18, 65})                                 // "(-inf, 18]", "(18, 65]", "(65, inf]"
Col("age").Cut([]float64{18, 65}, CutLabels("minor", "adult", "senior"))

The breaks must be finite and increase, and the labels, the Enum's categories, must differ, as Polars refuses them otherwise. A value is compared as a Float64; a null or a NaN has no bin and is null. The answer is named after the value.

func (Expr) Degrees added in v0.4.0 

func (e Expr) Degrees() Expr

Degrees converts radians to degrees, and Radians degrees to radians.

func (Expr) Diff 

func (e Expr) Diff(n int) Expr

Diff is the difference from the value n rows earlier.

It costs ONE shift, not two

`e.Sub(e.Shift(n))` mentions the shift twice, and window resolution dedups on the rendered expression — so both mentions resolve to the same temporary and the shift is computed once. That is real common-subexpression elimination, unlike Coalesce, which has none.

The type is whatever subtraction gives, which is the operand's own for a signed number and a DURATION for an instant — the temporal algebra's instant-minus-instant rule. Polars' diff carries a null_behavior parameter this composition cannot express; the first n rows are null here.

An unsigned column is widened first

Subtracting at its own width wrapped: a UInt8 that went from 3 to 1 changed by 254. It is taken one signed width up — UInt8 to Int16, UInt16 to Int32, UInt32 to Int64, UInt64 to Int128 — which holds every difference exactly, by the same plan-time call PctChange uses to reach a float. A signed integer keeps its width and wraps, as all integer arithmetic here does, and as Polars' diff does.

func (Expr) Div 

func (e Expr) Div[T Operand](v T) Expr

Div is TRUE division: it always produces a float, so 7/2 is 3.5 even for two integers. Integer division is FloorDiv, so the surprising behaviour has to be asked for by name.

func (Expr) Dot added in v0.6.6 

func (e Expr) Dot(other Expr) Expr

Dot is the sum of the products of e and other over the rows where both are non-null: Polars' dot. It is Sum's, so an integer dot product is an Int128, and one of no such row null, where Polars' is 0, as feature-gaps.md §9 records of Sum.

func (Expr) DropNans 

func (e Expr) DropNans() Expr

DropNans is the values that are not NaN; a null stays, as Polars' drop_nans keeps it. A column of floats (step 253): another is refused, where Polars returns it as it was.

func (Expr) DropNulls 

func (e Expr) DropNulls() Expr

DropNulls is the values that are not null.

The frame-level lf.DropNulls drops rows; this is the column's values, a shorter column, in an aggregation or a Select.

func (Expr) Dt 

func (e Expr) Dt() DtExpr

Dt opens the temporal namespace.

func (Expr) Entropy added in v0.6.6 

func (e Expr) Entropy(base float64, normalize bool) Expr

Entropy is the entropy of the non-null values over the logarithm of base: Polars' entropy. With normalize the values are a distribution, each over their sum, and it is −Σ p·ln p; without, −Σ x·ln x. A zero or negative value makes it NaN, as Polars' 0·ln 0 is. A group of no value is null, as Sum's is, where Polars' is −0. A Float32's is a Float32, any other number's a Float64. It merges, as ln S − (Σ x·ln x)/S, so it agrees with Polars' to the last few bits.

func (Expr) Eq 

func (e Expr) Eq[T Operand](v T) Expr

func (Expr) EqMissing 

func (e Expr) EqMissing[T Operand](v T) Expr

EqMissing is equality where nulls are DATA: null equals null, and the result is never null. NeMissing is its negation.

func (Expr) Err 

func (e Expr) Err() error

Err returns any error deferred during construction — a bad regex, say. Expression builders cannot return errors without destroying chaining, so the error rides in the tree and surfaces here or at Collect.

func (Expr) EwmMean added in v0.5.0 

func (e Expr) EwmMean(decay EwmDecay, opts ...WindowOption) Expr

EwmMean, EwmStd and EwmVar are exponentially weighted: each row's answer is over every row of its window up to it, each weighted by decay's power of how far behind it lies — Polars' ewm_mean, ewm_std and ewm_var, whose recurrences are pandas'. 

Col("price").EwmMean(EwmSpan(20))                         // a 20-row moving average
Col("price").EwmStd(EwmHalfLife(5)).OverWith(byDayPerTicker)

A null row is no observation, and answers null, as Polars' does, where pandas answers the running value; MinSamples sets how many values must have been seen before a row answers, one by default. Each is a Float64, or a Float32 for one. A NaN is a value, and makes the answer NaN from there on, where pandas would skip it.

func (Expr) EwmMeanBy added in v0.6.6 

func (e Expr) EwmMeanBy(by Expr, halfLife Interval) Expr

EwmMeanBy and EwmSumBy are exponentially weighted by time: Polars' ewm_mean_by and ewm_sum_by (step 285). In by's order, each row decays the last answer by d = 0.5^(Δt/halfLife), Δt the time since the last row holding a value: 

mean: d·last + (1−d)·x
sum:  d·last + x

The first such row is its value. by is a Date or a Datetime, or an integer of 32 or 64 bits whose half-life is in Polars' integer unit, Every("2i") (step 312); it need not be sorted, and the answers go back to the rows' order. A row whose value or by is null is null and skipped. halfLife is a constant duration: no months, and no days over a zoned by, as Polars requires. A Float32 answers a Float32 and any other number a Float64. Rows of one instant are taken in the window's order, where Polars' unstable sort takes them as it finds them.

func (Expr) EwmStd added in v0.5.0 

func (e Expr) EwmStd(decay EwmDecay, opts ...WindowOption) Expr

func (Expr) EwmSum added in v0.6.6 

func (e Expr) EwmSum(decay EwmDecay, opts ...WindowOption) Expr

EwmSum is each value plus the sum before it decayed by decay, s = x + (1−alpha)·s, the first value its own: Polars' ewm_sum (step 285). A null decays the sum, as a row ages the weights, but with IgnoreNulls; a null row is null, and a row answers once MinSamples values have been seen. EwmAdjust and Biased do nothing. A Float32 is summed in float32, as Polars sums it, and any other number answers a Float64.

func (Expr) EwmSumBy added in v0.6.6 

func (e Expr) EwmSumBy(by Expr, halfLife Interval) Expr

func (Expr) EwmVar added in v0.5.0 

func (e Expr) EwmVar(decay EwmDecay, opts ...WindowOption) Expr

func (Expr) Exp 

func (e Expr) Exp() Expr

func (Expr) Explode added in v0.3.0 

func (e Expr) Explode() Expr

Explode is a List's elements, a column of their number (step 259): Polars' explode. An empty or null list is one null, as the frame-level lf.Explode makes it a null row.

In Select it is the frame-level Explode of its one column; in an aggregation each group's lists end to end, and Col("tags").Explode().Sum() their sum. A column that is not a List is refused, where Polars returns it as it was.

func (Expr) ExtendConstant added in v0.6.6 

func (e Expr) ExtendConstant[N int | Expr](value any, n N) Expr

ExtendConstant is the values, then value n times: Polars' extend_constant. value is a Go value or nil, cast to the column's type, a float truncated toward zero to an integer and one out of its range null, as Polars casts it; n is an int or an Expr.

func (Expr) FillNan 

func (e Expr) FillNan[T Operand](v T) Expr

FillNan replaces NaN with a value, leaving nulls alone.

Nulls survive, and the reason is worth knowing

`e.IsNotNan()` on a null row is NULL, not true — null and NaN are different things all the way down — and since step 311 a null condition takes Otherwise, as Polars' when does. So the condition is IsNotNan or IsNull, true on a null row, which then keeps its null; the fill value never lands on it. Until step 311 a null condition took neither branch, and IsNotNan alone did this, which is why TestFillNanLeavesNullsAlone exists.

Why the predicate is IsNotNan rather than IsNan

Both spell the same thing, and only one names the result correctly. OutputName takes the LEFTMOST column of an expression, so with `IsNan → Then: value, Else: e` the leftmost column reference is the literal and every filled column came back called "literal" — which for the frame-level form meant a new column instead of a replaced one.

func (Expr) FillNull 

func (e Expr) FillNull(s FillStrategy) Expr

FillNull replaces nulls according to a named strategy. 

Col("temp").FillNull(ursus.FillForward)
Col("qty").FillNull(ursus.FillZero)

A forward or backward fill takes a limit, FillForward.Limit(1), as ForwardFill(1) does.

Three of these buffer the whole frame

FillMin, FillMax and FillMean compute an aggregate over every row before they can fill anything, so they turn a streaming query into one that holds its input. Inside .Over(g) the aggregate is each group's, as Polars computes it. FillForward and FillBackward are ordered windows and buffer for the same reason. Only FillZero and FillOne stream.

func (Expr) FillNullWith 

func (e Expr) FillNullWith[T Operand](v T) Expr

FillNullWith replaces nulls with a value.

It does not widen the column 

Col("i32").FillNullWith(0)     // stays Int32
Coalesce(Col("i32"), Lit(0))   // becomes Int64

A Go scalar lifts to a WEAK literal, which adopts the column's own type when the value fits it exactly. That is deliberately not what arithmetic does: Col("i32").Add(1) still widens, because addition can overflow and filling a null cannot. An Expr operand is used as written, so FillNullWith(Lit(0)) widens.

If the value does not fit — FillNullWith(int64(5000)) on an Int8 column — the type falls back to ordinary promotion rather than truncating, so the result is an Int64 column holding 5000 rather than an Int8 column holding -120.

Cost

This desugars to a conditional, so the receiver is MENTIONED twice and, with no common-subexpression pass, EVALUATED twice — exactly as Coalesce is.

func (Expr) Filter added in v0.6.5 

func (e Expr) Filter(preds ...Expr) Expr

Filter is the values where every predicate holds, Polars' Expr.filter. Its length is not the frame's, so it is an aggregate's input (step 251): a conditional aggregate, SQL's FILTER (WHERE ...). 

Col("amount").Filter(Col("qty").Gt(1)).Sum()             // in Agg, Over, Rolling, or Select
Col("amount").Filter(Col("region").Eq("eu")).Len()       // the rows that pass

The aggregate reads only the rows where the predicates are true, so Len counts them and First and Last pick among them, as Polars' do. A null predicate drops the row, as the frame's Filter does. A group none of whose rows pass aggregates no rows: its sum is null, as an all-null group's is here, where Polars gives 0.

In Select it is a column of its own length, the values that pass (step 258): Select(Col("a").Filter(Col("b").Gt(0))). Anywhere else it is refused at planning, until a later step gives a column of another length a place; the frame's Filter keeps rows.

func (Expr) First 

func (e Expr) First() Expr

First and Last are POSITIONAL and may return null.

They are a selection rather than a reduction: skipping nulls would mean First(a) and First(b) could come from different rows, which is exactly what people use them together for.

func (Expr) FirstNonNull added in v0.6.7 

func (e Expr) FirstNonNull() Expr

FirstNonNull and LastNonNull are the first and last value that is not null, null when there is none: Polars' first(ignore_nulls=True) and last(ignore_nulls=True) (step 310). NaN is a value. Unlike First and Last, two of them may come from different rows.

Each is First or Last of the values a filter keeps, a conditional aggregate, so it streams in a group-by, a window and a temporal group as First does.

func (Expr) Floor 

func (e Expr) Floor() Expr

Floor and Ceil round towards -Inf and +Inf. Both are the identity on an integer column and both are refused for Decimal, whose stored value is unscaled — so flooring it would floor 1234 rather than 12.34.

func (Expr) FloorDiv 

func (e Expr) FloorDiv[T Operand](v T) Expr

FloorDiv FLOORS, toward negative infinity, for integers, floats and a Duration alike: −7 // 2 is −4, not the −3 Go's / gives. Mod is its remainder, so a == b*FloorDiv(a, b) + Mod(a, b) and Mod takes the divisor's sign: −7 % 2 is 1. Both are Polars' and Python's rule. An integer divisor of zero is null; a float one follows IEEE.

func (Expr) ForwardFill 

func (e Expr) ForwardFill(limit int) Expr

ForwardFill replaces each null with the last non-null value before it.

limit caps how many consecutive nulls one value may fill; 0 means unlimited. A leading run of nulls has nothing to carry and stays null.

Like every ordered function it is a window: with no .Over() it runs over the whole frame in scan order, which makes it a pipeline breaker.

func (Expr) Gather added in v0.6.5 

func (e Expr) Gather(indices Expr) Expr

Gather is the values at indices, in their order, Polars' gather (step 257). indices is a List literal, the same for every group, Gather(Lit([]int64{0, 2})), or a column of integers, each group's own. A negative index counts from the end, and one out of range is refused, as Polars refuses it. Col("a").Sort().Gather(...) gathers from the sorted values.

func (Expr) GatherEvery added in v0.6.5 

func (e Expr) GatherEvery(n, offset int) Expr

GatherEvery is every n-th value, from offset (step 253).

func (Expr) Ge 

func (e Expr) Ge[T Operand](v T) Expr

func (Expr) Get added in v0.6.5 

func (e Expr) Get(i int) Expr

Get is the value at index i, one value a group; a negative index counts from the end, and an index past the end is null.

func (Expr) Gt 

func (e Expr) Gt[T Operand](v T) Expr

func (Expr) HasNulls added in v0.6.6 

func (e Expr) HasNulls() Expr

HasNulls is whether a value is null, and IsEmpty whether there are no rows, nulls counted: Polars' has_nulls and is_empty, a Boolean each.

func (Expr) Hash added in v0.6.6 

func (e Expr) Hash(seed uint64) Expr

Hash is each value's 64-bit hash, seeded, a UInt64: Polars' hash. Values a group-by takes as one hash alike, -0 with 0 and NaN with NaN, and a null has a hash of its own. The algorithm is ursus's, so the numbers are not Polars' (v0.6.6-scope.md §6.3).

func (Expr) Head added in v0.6.5 

func (e Expr) Head(n int) Expr

Head is the first n values.

func (Expr) Hist added in v0.6.6 

func (e Expr) Hist(opts ...HistOption) Expr

Hist counts the values in bins: Polars' hist (step 286), each bin's count a UInt32, a column of its own length in a Select and a list a group in an aggregation. By default the bins are 10 of equal width over the values' least to greatest, NaN aside, the last edge the greatest itself; one value, or values all alike, are the bins over the value ± 0.5, and none the bins over [0, 1]. A null or NaN is in no bin. It is Polars' algorithm, edges and counts alike. It counts numbers.

func (Expr) Implode added in v0.3.0 

func (e Expr) Implode() Expr

Implode collects every value of the group into a List, in input order. 

GroupBy(Col("k")).Agg(
    Col("v").Implode().Alias("all"),
    Col("v").Implode().List().Sort().List().Head(3).Alias("smallest3"),
)

It is what makes the .list namespace available inside a group-by: everything list.go can do to a list, this can now do to a group.

Nulls are kept

Every other aggregate skips them — the sum of an all-null group is null, not zero. Implode is a selection rather than a reduction, so a null is a value that takes a slot, and an all-null group implodes to a list OF nulls rather than to an empty list or a null one.

It makes the group-by single-threaded

A list's element order is its data, and the parallel driver dispatches batches round-robin, so a shuffled input would be a different answer rather than the same one computed differently. The whole Agg list runs on one worker as a result — including the aggregates beside this one, because a query's aggregates share one hash table.

Its memory is the data

Implode holds every value of every group, so a group-by containing one is O(rows) rather than O(distinct keys), and under WithMemoryLimit it refuses once a single key grows past the budget. Partitioning divides the key space and cannot divide one key.

func (Expr) IndexOf added in v0.6.6 

func (e Expr) IndexOf(value any) Expr

IndexOf is the index of the first value equal to value, a UInt32, null where there is none: Polars' index_of. value is a Go value, or a null, which finds the first null; an Expr holding an aggregate cannot be, as SearchSorted's cannot. NaN finds NaN and 0 finds -0, as Polars finds them.

func (Expr) Interpolate added in v0.5.0 

func (e Expr) Interpolate(opts ...WindowOption) Expr

Interpolate fills each run of nulls between two values with the points on the straight line between them, by position in the window's order: Polars' interpolate. [1, null, null, 7] is [1, 3, 5, 7]. A run before the first value or after the last stays null. An integer or a Decimal answers a Float64; a float keeps its width. An instant, a duration or a time keeps its type, interpolated on its ticks in integers as Polars' is: a + step·(b−a)/span, the quotient truncated toward zero, so halfway from January 1 to January 4 is January 2 (step 301). InterpolateNearest() fills it with the nearer value instead: [1, 1, 7, 7].

func (Expr) InterpolateBy added in v0.6.7 

func (e Expr) InterpolateBy(by Expr) Expr

InterpolateBy fills each run of nulls between two values on the straight line between them, at each row's by: Polars' interpolate_by (step 329). A null at x between (x0, y0) and (x1, y1) is y0 + (y1−y0)/(x1−x0)·(x−x0), Polars' arithmetic to the bit. In by's order: it need not be sorted, and the answers go back to the rows' order, rows of one by taken in the window's order. A run before the first value or after the last stays null, and a null in by is refused, as Polars refuses it. The value is a number, a Float32 answering a Float32 and any other a Float64; by is a Date, counted in days, a Datetime, an integer of 32 or 64 bits, or a float.

func (Expr) IsBetween 

func (e Expr) IsBetween[L, H Operand](lo L, hi H, closed ...Closed) Expr

IsBetween tests that e falls between lo and hi. Both bounds lift, so IsBetween(0, 100) works.

The default is CLOSED-BOTH — lo <= e <= hi — which is what this method has always meant and what reads naturally from the name. Pass a Closed to say otherwise: 

Col("ts").IsBetween(start, end, ursus.ClosedLeft)   // start <= ts < end

Variadic rather than a required third argument, so every existing two-argument call keeps compiling and the common case stays short. More than one is a mistake and is refused rather than ignored.

func (Expr) IsClose 

func (e Expr) IsClose(other Expr, relTol, absTol float64) Expr

IsClose reports whether each value is within a relative or absolute tolerance of other. 

|a - b| <= max(relTol * max(|a|, |b|), absTol)

Both guards are load-bearing, and neither is obvious

The EQUALITY disjunct is what makes IsClose(Inf, Inf) true: the tolerance term alone would compute |Inf - Inf|, which is NaN, and NaN <= x is false.

The FINITENESS conjunct is what makes IsClose(Inf, -Inf) false. Without it the relative tolerance scales by max(|a|, |b|) = Inf, the limit becomes Inf, and Inf <= Inf says "close" — which it plainly is not. Python's math.isclose has the same special case for the same reason.

Together they give: equal infinities are close, opposite ones are not, and IsClose(NaN, NaN) is false because IEEE says NaN equals nothing including itself. With a null operand every term is null and Kleene OR gives null, matching every other comparison.

It computes in Float64

Both operands are cast first, so an integer column works — integers are always finite — and the finiteness test, which is float-only, has something to test.

Cost

The receiver is mentioned five times and other four, with no CSE. For two bare columns that is nine column reads; for expensive operands it is not free.

func (Expr) IsDuplicated 

func (e Expr) IsDuplicated() Expr

func (Expr) IsEmpty added in v0.6.6 

func (e Expr) IsEmpty() Expr

func (Expr) IsFinite 

func (e Expr) IsFinite() Expr

func (Expr) IsFirstDistinct 

func (e Expr) IsFirstDistinct() Expr

IsFirstDistinct and IsLastDistinct mark the first and last occurrence of each value, in input order.

Note the asymmetry with IsUnique: a value occurring three times has one first occurrence, one last, and is not unique anywhere.

func (Expr) IsIn 

func (e Expr) IsIn[T Operand](vs ...T) Expr

IsIn tests membership in a set of values, Col("region").IsIn("eu", "us"), or in each row's own list, Col("x").IsIn(Col("tags")).

Values, or one expression

Values are one set for every row. They share one type parameter, so the set is homogeneous by construction.

One expression is a List column, and each row's value is looked for in the row's own list, as Polars' is_in(list) does (step 254). A null value or a null list is null; a value its list does not hold is false, a null element too, as Polars' is_in with nulls_equal=False. A column of the value's own type is refused, as Polars deprecates it for being ambiguous: membership among a column's values is IsIn(Col("b").Implode()), which is not there yet.

Membership in another frame's column is a semi join wearing a predicate's clothes, and belongs with the join machinery.

IsIn(v) answers what Eq(v) answers

The column and the values meet at the type == would compare them at, so IsIn refuses exactly the pairs Eq refuses, at plan time — a String against an Int64, an int against a Bool, an instant against a Date — and is false wherever Eq is: 5000 is not in an Int8 column, 0.1 is not in a Float32 one (they compare at Float64, as Polars does), and 00:00:01.5 is not in a Datetime(s) one. To match the float32 nearest 0.1, say so: IsIn(float32(0.1)).

Equality is GROUPING equality

Membership uses the same encoding GroupBy and Distinct use, so NaN matches NaN and -0.0 matches +0.0. An is_in built on IEEE equality would disagree with Distinct about the same data.

Nulls

`null.IsIn(...)` is NULL, not false — the same rule as IsNan, and the reason Filter(p) and Filter(Not(p)) do not partition. The set itself can never contain a null, because these are Go values.

An empty set makes every non-null row false, matching SQL's `x IN ()`.

func (Expr) IsInfinite 

func (e Expr) IsInfinite() Expr

func (Expr) IsLastDistinct 

func (e Expr) IsLastDistinct() Expr

func (Expr) IsNan 

func (e Expr) IsNan() Expr

IsNan and friends ask about a float VALUE, so `null.IsNan()` is null rather than false. Null and NaN are different things; use IsNull to test for missing.

func (Expr) IsNotNan 

func (e Expr) IsNotNan() Expr

func (Expr) IsNotNull 

func (e Expr) IsNotNull() Expr

func (Expr) IsNull 

func (e Expr) IsNull() Expr

IsNull and IsNotNull ask about presence, so their results are never null.

func (Expr) IsSorted added in v0.6.7 

func (e Expr) IsSorted(descending, nullsLast bool) Expr

IsSorted reports whether the values come in order, as Polars' is_sorted (step 324): ascending, or descending, equal values allowed, NaN the greatest; and every null before the values, or after them with nullsLast. A group's in an aggregation, where it reads the rows in their order, so a group-by holding it runs on one worker.

func (Expr) IsUnique 

func (e Expr) IsUnique() Expr

IsUnique reports whether each row's value occurs exactly once in the column. IsDuplicated is its negation for non-null rows.

These are windows, and that is why they were not built earlier

The answer for row 0 depends on the last row, so neither is elementwise. Step 7 deferred them precisely here, and they need no new machinery: a value is unique iff the partition keyed by that value has exactly one row.

Equality is GROUPING equality, the same the partitioning uses — so NaN equals NaN and -0.0 equals +0.0, and IsUnique agrees with Unique() about the same data. Nulls likewise group together, so two nulls are duplicates of each other.

func (Expr) Item added in v0.6.7 

func (e Expr) Item(allowEmpty bool) Expr

Item is the one value, as Polars' item (step 324): a group's in an aggregation, and in a Select the column's, beside the other columns. Several values are refused, and none too unless allowEmpty, which makes it null.

func (Expr) Kurtosis added in v0.6.6 

func (e Expr) Kurtosis(fisher, bias bool) Expr

func (Expr) Last 

func (e Expr) Last() Expr

func (Expr) LastNonNull added in v0.6.7 

func (e Expr) LastNonNull() Expr

func (Expr) Le 

func (e Expr) Le[T Operand](v T) Expr

func (Expr) Len 

func (e Expr) Len() Expr

Len counts ROWS, including nulls — SQL's COUNT(*).

Count and Len are deliberately separate and are never aliased: over a column with nulls they give different answers, and which one was meant is not recoverable after the fact.

func (Expr) Limit added in v0.6.7 

func (e Expr) Limit(n int) Expr

Limit is Head, as Polars' limit is (step 324).

func (Expr) List added in v0.3.0 

func (e Expr) List() ListExpr

List opens the list namespace.

func (Expr) Ln 

func (e Expr) Ln() Expr

Ln is the natural logarithm; Log10 is base 10; Log1p is ln(1+x), which keeps its precision for x near zero where ln(1+x) would lose it.

ln(0) is -Inf and ln(-1) is NaN. Both are values.

func (Expr) Log 

func (e Expr) Log(base float64) Expr

Log is the logarithm to an arbitrary base.

It is SUGAR — ln(x)/ln(base) — rather than a parameterised kernel, which is the same definition Polars uses and costs no new machinery. Two consequences worth knowing, both from the division:

  • Log(10) and Log10() can differ in the last ULP. Log10 is the exact form.
  • On a Float32 column Log10() stays Float32 and Log(10) becomes Float64, because dividing by a float64 promotes. Cast the result back if the narrow width was a deliberate choice.

func (Expr) Log1p 

func (e Expr) Log1p() Expr

func (Expr) Log10 

func (e Expr) Log10() Expr

func (Expr) LowerBound added in v0.6.7 

func (e Expr) LowerBound() Expr

LowerBound and UpperBound are the least and the greatest value of the expression's type, as Polars' lower_bound and upper_bound (step 324): a number's own, -Inf and +Inf for a float; and a type stored as integers its storage's, as Polars gives them, so a Date's are an Int32's and a Decimal's an Int128's. A Boolean, text and nested types have none.

func (Expr) Lt 

func (e Expr) Lt[T Operand](v T) Expr

func (Expr) MapBatches added in v0.3.0 

func (e Expr) MapBatches[In, Out Literal](name string, out DataType,
	fn func([]In, []bool) ([]Out, []bool, error)) Expr

MapBatches applies a Go function to a whole column at a time.

The vectorized escape hatch, and the one to reach for first: one call per batch rather than one per row, and full control of the validity mask. 

Col("v").MapBatches[float64, float64](
    "normalise", ursus.Float64,
    func(vals []float64, ok []bool) ([]float64, []bool, error) { ... })

The two slices are positional and the same length

vals[i] is row i and ok[i] says whether it is present. A null row's vals entry is the zero value and must not be read. The returned pair must be the same length as the input; returning a nil mask means every output row is valid.

vals ALIASES the engine's own buffer when the layout permits, so fn must not write to it. Build a new slice for the result.

The notes on name, goroutine safety and out under MapElements apply here identically.

func (Expr) MapElements added in v0.3.0 

func (e Expr) MapElements[In, Out Literal](name string, out DataType,
	fn func(In) (Out, error)) Expr

MapElements applies a Go function to every non-null value.

The per-element escape hatch. In Polars this is the slow one, because each call crosses into the Python interpreter; in Go it is a function call, which is what makes it worth offering rather than merely tolerating. 

Col("celsius").MapElements[float64, float64](
    "to_fahrenheit", ursus.Float64,
    func(c float64) (float64, error) { return c*9/5 + 32, nil })

Nulls pass through

fn never sees a null: a null input row is a null output row, and fn is not called for it. That is deliberate — the alternative hands fn a zero value it cannot distinguish from a real one. Use MapBatches when the null matters.

name must be unique within the query, and this is CHECKED

A Go closure has no printable form, so the name is all that Explain and every error message can show of a udf. Two different functions sharing one could not be told apart there — and until the engine stopped deduplicating expressions by their rendering, they were one computation, with both results coming from whichever it saw first.

plan.CheckUDFNames refuses that at plan time, before any data is read, so the sentence above is a guarantee rather than advice. Two udfs with one name are an error; ONE udf used twice is not, so if two calls are meant to be the same function, hold the Expr in a variable and use it twice rather than calling a helper that builds a fresh closure each time.

A panic in fn is an error

It is returned like an error fn returns — KindValue, naming the udf, the row and the column — whichever goroutine fn ran on. Before step 72 it killed the process.

fn must be safe to call from several goroutines

ursus parallelises a Select or Filter across runtime.NumCPU() workers by default, and there is no way to opt a single expression out. A closure over a shared map is a data race. fn must also not depend on the ORDER it is called in: output order is preserved, invocation order is not.

out is a declaration, and it is checked

The engine reports out as the column's type from CollectSchema and Explain, before fn has ever run. If fn then produces something else, the evaluator fails the query rather than letting a wrong schema propagate.

Example

A Go function over each value, taking part in the plan like any expression. Nulls pass through without calling it. 

package main

import (
	"context"
	"fmt"
	"log"
	"strings"

	"github.com/advenn/ursus"
)

// exampleSales is the little frame the examples below work on.
func exampleSales() *ursus.LazyFrame {
	return ursus.Frame(
		ursus.Values("region", []string{"north", "south", "north", "east", "south"}),
		ursus.Values("product", []string{"apple", "apple", "pear", "pear", "fig"}),
		ursus.Values("qty", []int64{10, 4, 7, 2, 9}),
		ursus.Values("price", []float64{1.50, 1.50, 2.25, 2.25, 3.00}),
	)
}

func main() {
	df, err := exampleSales().Select(
		ursus.Col("product"),
		ursus.Col("product").MapElements("shout", ursus.String,
			func(s string) (string, error) { return strings.ToUpper(s) + "!", nil }).Alias("loud"),
	).Head(3).Collect(context.Background())
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(df)
}

Output:
shape: (3, 2)
┌─────────┬──────────┐
│ product │ loud     │
│ String  │ String   │
├─────────┼──────────┤
│ "apple" │ "APPLE!" │
│ "apple" │ "APPLE!" │
│ "pear"  │ "PEAR!"  │
└─────────┴──────────┘

func (Expr) MapName 

func (e Expr) MapName(fn func(string) string) Expr

MapName derives the output name from the input name.

fn is called while the query is planned. A panic in it is the caller's error, of kind ErrValue, naming the input name it panicked on. Two MapNames over one expression are two computations, whatever their functions (step 323).

func (Expr) Max 

func (e Expr) Max() Expr

func (Expr) MaxBy added in v0.6.0 

func (e Expr) MaxBy(by Expr) Expr

func (Expr) Mean 

func (e Expr) Mean() Expr

Mean is the sum of non-null values divided by the COUNT of non-null values — never by the group size, which would silently bias the result downward whenever nulls are present.

Mean of a group with no non-null values is NULL, not NaN: there was nothing to compute, as opposed to a computation that came out undefined.

The mean of an instant or a duration is one, exact and truncated toward zero, as Polars' (step 173): a Datetime, Time or Duration keeps its type, and a Date's mean is a Datetime(us), keeping the fraction of a day.

func (Expr) Median 

func (e Expr) Median() Expr

Median is the 0.5 quantile with linear interpolation, so an even-sized group gives the mean of the two middle values.

It returns Float64 for every numeric input, and skips nulls. The median of instants or durations is one, truncated toward zero, as Polars' (step 173): its type kept, but a Date's, which is a Datetime(us). Quantile still refuses a temporal column; Cast(Int64) gives its ticks.

func (Expr) Meta added in v0.6.7 

func (e Expr) Meta() MetaExpr

Meta is the expression's meta functions.

func (Expr) Min 

func (e Expr) Min() Expr

Min and Max use ursus's TOTAL order, the same one Sort uses: NaN sorts above everything and -0.0 equals +0.0. They therefore always agree with Sort(...).First() / .Last(), which IEEE comparison would not.

func (Expr) MinBy added in v0.6.0 

func (e Expr) MinBy(by Expr) Expr

MinBy is the value of e at the row where by is the least, and MaxBy where it is the greatest: Polars' min_by and max_by. The answer is e's type, and by may be of any ordered type. 

GroupBy(Col("store")).Agg(Col("product").MaxBy(Col("sales")))

A row whose by is null is skipped, and a group where every by is null answers null; a null value at the chosen row is null too. A tie goes to the earliest row.

Where Polars differs: by is ordered as Max and ArgMax order it, so NaN is the greatest value and MaxBy chooses a NaN's row, where Polars skips the NaN. Then v.MaxBy(by) is always v at by.ArgMax().

func (Expr) Mod 

func (e Expr) Mod[T Operand](v T) Expr

func (Expr) Mode added in v0.6.6 

func (e Expr) Mode(maintainOrder bool) Expr

Mode is the most frequent values, each once, a null among them where nulls are: Polars' mode, a column of its own length in a Select and a list a group in an aggregation. With maintainOrder they come in the order they first appear; without, ascending, nulls first, where Polars' order is its hash table's. Each value is its first row's, so a −0 is −0 where it came before a +0, as Polars answers.

In an aggregation it counts each distinct value of a group, and past a memory limit its counts spill to disk (step 340).

func (Expr) Mul 

func (e Expr) Mul[T Operand](v T) Expr

func (Expr) NUnique 

func (e Expr) NUnique() Expr

NUnique counts distinct non-null values.

Nulls are SKIPPED, so NUnique over an all-null group is 0. Polars counts null as a distinct value; ursus follows SQL and, more importantly, follows its own other aggregates.

func (Expr) Name added in v0.6.7 

func (e Expr) Name() NameExpr

Name is the expression's naming functions.

func (Expr) NanMax added in v0.6.6 

func (e Expr) NanMax() Expr

func (Expr) NanMin added in v0.6.6 

func (e Expr) NanMin() Expr

NanMin is the minimum, NaN where a value is NaN: Polars' nan_min. NanMax is Max, whose order puts NaN above every number, so it is NaN where a value is: Polars' nan_max. Min skips a NaN as Polars' min does; Max does not, as feature-gaps.md §9 records.

func (Expr) Ne 

func (e Expr) Ne[T Operand](v T) Expr

func (Expr) NeMissing 

func (e Expr) NeMissing[T Operand](v T) Expr

func (Expr) Neg 

func (e Expr) Neg() Expr

Neg negates. Abs takes the absolute value.

Both are defined for every signed numeric type, Int128 and Decimal included — which matters more than it sounds, because every integer Sum outputs Int128, so `Col("x").Sum().Abs()` is an ordinary query. Abs of an unsigned integer is itself. Neg of one is refused, because most of its negations do not fit: cast it to a signed type first.

func (Expr) Not 

func (e Expr) Not() Expr

Not is Kleene negation: NOT null is null. Of an integer it is its bits flipped, ~x, of its type (step 278).

func (Expr) NullCount 

func (e Expr) NullCount() Expr

NullCount counts the NULLS in each group — the complement of Count, and like the other counters it is never itself null.

func (Expr) Or 

func (e Expr) Or[T Operand](v T) Expr

func (Expr) Over 

func (e Expr) Over(partitionBy ...Expr) Expr

Over computes the expression within partitions and writes the answer back onto every row. 

// each row's deviation from its group's mean
ursus.Col("x").Sub(ursus.Col("x").Mean().Over(ursus.Col("g")))

// dense rank of speed within each type
ursus.Col("speed").Rank(ursus.RankDense, true).Over(ursus.Col("type"))

The distinction from GroupBy

GroupBy REDUCES: N rows in, one row per group out. Over PRESERVES: N rows in, N rows out, each carrying its own partition's answer. That is why an aggregate is refused in Select but the same aggregate under Over is not.

With no partition keys the whole frame is one partition, so `Col("x").Sum().Over()` puts the grand total on every row.

Nulls form their own partition

A null partition key groups with other nulls, the same rule GroupBy uses — as opposed to joins, where null keys match nothing. The two differ on purpose and each is documented where it applies.

func (Expr) OverWith 

func (e Expr) OverWith(spec WindowSpec) Expr

OverWith is Over with an ordering and a mapping strategy.

An ordering is what makes Rank, the cumulative functions and Shift meaningful: without one they run in input order, which is well-defined but rarely what a ranking wants.

func (Expr) PctChange 

func (e Expr) PctChange(n int) Expr

PctChange is the fractional change from the value n rows earlier.

Always a float on a numeric column, because it divides — and a zero previous value gives ±Inf rather than a null, since float division is total. A Float32 stays Float32. On an instant it is a plan-time error.

The value is taken to a float BEFORE the subtraction

It used to subtract at the column's own width, and integer subtraction wraps: a UInt8 that went from 3 to 1 changed by 254, so its pct_change was 84.67 where the answer is −0.67. The public Expr has no type in hand to choose a cast with, so an internal call, typed at plan time, takes an integer or a Decimal to Float64 and a Duration's ticks to Float64, keeps a float as it is, and refuses an instant.

func (Expr) PeakMax added in v0.6.6 

func (e Expr) PeakMax() Expr

PeakMax is whether each value is greater than both its neighbours, and PeakMin whether it is less, a Boolean: Polars' peak_max and peak_min. A value at an end is compared with its type's zero there, as Polars fills its shifts with 0, so the first of [3, 1] is a peak and of [-1, -2] is not; a null neighbour makes it null. Text is refused, as Polars refuses it. A group's own peaks are .Over(key), where Polars' group_by gives each group's as a list; an aggregation takes aggregates.

func (Expr) PeakMin added in v0.6.6 

func (e Expr) PeakMin() Expr

func (Expr) Pipe added in v0.6.7 

func (e Expr) Pipe(fn func(Expr) Expr) Expr

Pipe is fn of the expression, as Polars' pipe: a function of an expression applied in a chain of methods (step 324).

func (Expr) Pow 

func (e Expr) Pow[T Operand](v T) Expr

Pow raises the receiver to the power v.

The result is ALWAYS a float, for the same reason Div's is: 2**-1 is 0.5, and a version that truncated negative powers to zero would be surprising in the direction that loses data silently. Cast the result if an integer is wanted.

func (Expr) Prefix 

func (e Expr) Prefix(p string) Expr

Prefix prepends to the output name. Unlike Alias it is expansion-safe, deriving a distinct name per output column.

func (Expr) Product 

func (e Expr) Product() Expr

Product multiplies the non-null values.

Unlike Sum it returns FLOAT64 for every numeric input, including integers. Sum widened to Int128 so that overflow became unreachable; product cannot, because i128 has no multiplication — and an Int64 product wraps silently after roughly twenty ordinary factors. Float64 is exact to 2^53 and approximate above it, which is the weaker guarantee, stated rather than hidden.

Product of a group with no non-null values is NULL, not 1, following Sum.

func (Expr) QCut added in v0.5.0 

func (e Expr) QCut(quantiles []float64, opts ...CutOption) Expr

QCut puts each value in a bin between quantiles of its column, and answers the bin's label: Polars' qcut. 

Col("income").QCut([]float64{0.25, 0.5, 0.75})   // quartiles

Each break is the column's Quantile at q, interpolated linearly, over the whole frame (Over()), so QCut is a window: it belongs in Select or WithColumns. Labels, the closed side, and a null or NaN value are as Cut's. A column of few distinct values can give two equal quantiles, and the bin between them is then empty, where Polars refuses it unless allow_duplicates is set.

func (Expr) QCutN added in v0.5.0 

func (e Expr) QCutN(n int, opts ...CutOption) Expr

QCutN is QCut into n bins of equal count: QCutN(4) is QCut at 0.25, 0.5 and 0.75.

func (Expr) Quantile 

func (e Expr) Quantile(q float64, interp Interpolation) Expr

Quantile returns the value at rank q, where q runs from 0 (the minimum) to 1 (the maximum). interp decides what happens when the rank falls between two values.

The rank is q·(n−1) over the non-null values — the convention numpy, Polars and DuckDB share — and the result is Float64 even for InterpLower and InterpHigher, which do return an actual input value.

Values are ordered by ursus's TOTAL order, so NaN sorts above everything and a high quantile of a column containing NaN is NaN, exactly as Sort would place it.

A quantile of instants or durations is one of them, truncated toward zero, as its median is and as Polars' is: a Date's a Datetime(us), keeping a fraction of a day (step 300; refused before).

func (Expr) Radians added in v0.4.0 

func (e Expr) Radians() Expr

func (Expr) Rank 

func (e Expr) Rank(method RankMethod, descending bool) Expr

Rank numbers rows by value within the window, smallest first unless descending.

Ties are resolved by method; see RankMethod. A null's rank is null, and the values are ranked among themselves, as Polars ranks them. It returns Uint32, or Float64 for RankAverage.

func (Expr) Reinterpret added in v0.6.7 

func (e Expr) Reinterpret(signed bool) Expr

Reinterpret is an integer's bits as the integer of its width and the other signedness, as Polars' reinterpret (step 324): signed, a UInt64's maximum is an Int64's -1, and unsigned the reverse. Any other type is refused.

func (Expr) RepeatBy added in v0.6.6 

func (e Expr) RepeatBy[N Operand](n N) Expr

RepeatBy is each row's value n times, a List: Polars' repeat_by. A null value is n nulls, a null n a null list, and a negative n is refused.

func (Expr) Replace added in v0.6.3 

func (e Expr) Replace[K, V Literal](old []K, new []V) Expr

Replace maps values: a row equal to old[j] becomes new[j], and every other row keeps its value, a null too, as Polars' replace does.

The old values meet the column as IsIn's do, so Replace(old, new) replaces exactly the rows IsIn(old...) holds, NaN matching NaN. The answer is the type the conditional When(x.IsIn(old_j)).Then(new_j)….Otherwise(x) would have: a new value that fits the column's type exactly keeps it, so 20 into an Int32 is an Int32, and one that does not widens it, so 2.5 into an Int64 makes a Float64, where Polars truncates 2.5 to 2. old and new are as long as each other, and old holds no value twice. To replace a null, use FillNull.

func (Expr) ReplaceOr added in v0.6.3 

func (e Expr) ReplaceOr[K, V Literal](old []K, new []V, def V) Expr

ReplaceOr maps values with a default: a row equal to old[j] becomes new[j], and every other row, a null too, becomes def, as Polars' replace_strict with a default does. The answer is the type the new values and def meet at.

func (Expr) ReplaceStrict added in v0.6.3 

func (e Expr) ReplaceStrict[K, V Literal](old []K, new []V) Expr

ReplaceStrict maps every value: a row equal to old[j] becomes new[j], a null stays null, and any other value fails the query, naming it, as Polars' replace_strict does. The answer is the type the new values meet at.

func (Expr) Reshape added in v0.6.7 

func (e Expr) Reshape(shape ...int) Expr

Reshape is the values in Arrays of shape, Polars' reshape: shape's first dimension is the rows, -1 to infer them, and each one after it an Array of the next, the last innermost, so Reshape(-1, 3) of six values is two rows of an Array(T, 3), and Reshape(-1, 1, 3) two of an Array(Array(T, 3), 1). Of one dimension, Reshape(-1), it is the values themselves, a List's or an Array's elements flattened. The values are an expression's leaves: a List's or an Array's elements, an Array's null row its size of nulls, as Polars lays one out, and a List's null or empty one none.

It is a column of its own length, as Explode is; in an aggregation, a list of the Arrays a group makes. Values the shape does not divide are refused, as Polars refuses them.

New in 0.6.7.

func (Expr) Reverse added in v0.6.5 

func (e Expr) Reverse() Expr

Reverse is the values in reverse order.

func (Expr) Rle added in v0.6.6 

func (e Expr) Rle() Expr

Rle is each run of equal values' length and value, a Struct of len, a UInt32, and value: Polars' rle, a column of its own length in a Select and a list a group in an aggregation. Its runs are RleID's.

func (Expr) RleID added in v0.6.6 

func (e Expr) RleID() Expr

RleID is each row's run's number, from 0, a run being rows of equal values, a UInt32: Polars' rle_id. NaN runs with NaN, -0 with 0 and a null with a null, as Polars runs them. It is a window function, over the column or .Over(key) a partition's own; Polars' group_by gives each group's as a list. A Struct cannot be a key here and is refused, where Polars runs it.

func (Expr) Rolling added in v0.6.7 

func (e Expr) Rolling(index Expr, o RollingOptions) Expr

Rolling is e over each row's window of a rolling group, as Polars' Expr.rolling (step 337): index, which must be sorted, places each row's window as the frame's Rolling does under o, and e, an aggregate, is its value for each row. So 

ursus.Col("x").Sum().Rolling(ursus.Col("t"), ursus.RollingOptions{Period: ursus.Every("2d")})

is each row's sum of the rows of the two days ending at it, and any aggregate expression of them may be rolled, Col("x").Max().Sub(Col("x").Min()) as well. A row's value that is not aggregated is the list of its window's. Under Over it rolls within each partition, the answer in the frame's rows' order. Over gives its partitions, not RollingOptions.GroupBy, as Polars' rolling takes none.

Where every aggregate in e is a Sum, a Mean, a Min, a Max, a Var, a Std, a Median or a Quantile, and o gives no Offset, each is rolled by its own window by time, RollingSumBy's and its kin's, a window found by value: the index need not be sorted, and in an aggregation it answers a list a group. Anything else is rolled by the frame's rolling group, whose index is checked sorted, within each partition under Over, where it holds every window until the input ends; in an aggregation it is refused.

func (Expr) RollingKurtosis added in v0.6.6 

func (e Expr) RollingKurtosis(size int, fisher, bias bool, opts ...WindowOption) Expr

func (Expr) RollingMap added in v0.6.7 

func (e Expr) RollingMap[In, Out Literal](name string, out DataType, size int,
	fn func(window []In, ok []bool) (Out, error), opts ...WindowOption) Expr

RollingMap applies a Go function to each row's window of size rows: Polars' rolling_map (step 338). fn is given the window's values in the window's order and whether each holds one, a null's value the zero value and not to be read, as MapBatches gives a batch; its answer is the row's. The window is the row and the size-1 rows before it, fewer at a partition's start, or with RollingCenter the size rows centred on it. One holding fewer values than MinSamples, size unless set, as Polars', is null, and fn is not called for it. 

Col("price").RollingMap[float64, float64]("range", ursus.Float64, 5,
    func(w []float64, ok []bool) (float64, error) { ... })

It is a rolling function: it runs within each partition under Over, in an order under OverWith, and streams past a memory limit as they do. name, goroutine safety and out are as MapElements has them, two functions of one name refused, and a panic or an error in fn fails the query, naming the row. Polars' weights are not taken: fn weighs its window itself.

func (Expr) RollingMax added in v0.5.0 

func (e Expr) RollingMax(size int, opts ...WindowOption) Expr

func (Expr) RollingMaxBy added in v0.6.0 

func (e Expr) RollingMaxBy(by Expr, period Interval, opts ...WindowOption) Expr

func (Expr) RollingMean added in v0.5.0 

func (e Expr) RollingMean(size int, opts ...WindowOption) Expr

func (Expr) RollingMeanBy added in v0.6.0 

func (e Expr) RollingMeanBy(by Expr, period Interval, opts ...WindowOption) Expr

func (Expr) RollingMedian added in v0.6.3 

func (e Expr) RollingMedian(size int, opts ...WindowOption) Expr

RollingMedian and RollingQuantile answer each row over the size rows up to and including it, as the rolling functions above do: the median, and the q-quantile interpolated as interp says, of the window's values (step 227). Polars' rolling_median and rolling_quantile; Polars' rolling_quantile defaults to InterpNearest. Each answers as Median and Quantile do over the window: NaN is the largest value, as in Polars, and the answer is a Float64, where Polars keeps a Float32 window a Float32.

func (Expr) RollingMedianBy added in v0.6.3 

func (e Expr) RollingMedianBy(by Expr, period Interval, opts ...WindowOption) Expr

RollingMedianBy and RollingQuantileBy are RollingMedian and RollingQuantile over the windows of RollingSumBy and its kin (step 227): Polars' rolling_median_by and rolling_quantile_by.

func (Expr) RollingMin added in v0.5.0 

func (e Expr) RollingMin(size int, opts ...WindowOption) Expr

func (Expr) RollingMinBy added in v0.6.0 

func (e Expr) RollingMinBy(by Expr, period Interval, opts ...WindowOption) Expr

func (Expr) RollingQuantile added in v0.6.3 

func (e Expr) RollingQuantile(q float64, interp Interpolation, size int, opts ...WindowOption) Expr

func (Expr) RollingQuantileBy added in v0.6.3 

func (e Expr) RollingQuantileBy(q float64, interp Interpolation, by Expr, period Interval,
	opts ...WindowOption) Expr

func (Expr) RollingRank added in v0.6.6 

func (e Expr) RollingRank(size int, method RankMethod, opts ...WindowOption) Expr

RollingRank answers each row with its own value's rank among its window's values: Polars' rolling_rank (step 284). RankMin, RankMax and RankDense are a UInt32 and RankAverage, Polars' default, a Float64; RankOrdinal is refused, as Polars has none, and Polars' random is not here. Values compare as Sort orders them, NaN above every number and -0 with 0. It ranks numbers, instants and Booleans; a row whose value is null is null.

With RollingCenter a row is ranked in the window centred on it. Polars then answers null where the window's last row is null, though the row's own is not; here such a row is ranked.

func (Expr) RollingRankBy added in v0.6.7 

func (e Expr) RollingRankBy(by Expr, period Interval, method RankMethod, opts ...WindowOption) Expr

RollingRankBy is RollingRank over the windows of RollingSumBy and its kin, each row's own value ranked among its window's values: Polars' rolling_rank_by (step 338). RankMin, RankMax and RankDense are a UInt32 and RankAverage a Float64; RankOrdinal is refused, as Polars has none, and Polars' random is not here. A row whose value is null is null, and a window answers from one value unless MinSamples asks for more.

func (Expr) RollingSkew added in v0.6.6 

func (e Expr) RollingSkew(size int, bias bool, opts ...WindowOption) Expr

RollingSkew and RollingKurtosis answer each row with the skew or kurtosis of its window's values, as the rolling functions above do: Polars' rolling_skew and rolling_kurtosis, bias and fisher as Skew's and Kurtosis's, Polars' defaults both true (step 284). They are Polars' algorithm, a queue of mergeable moments, so a kurtosis is Polars' to the bit, and a skew is but for about one in a thousand, an ulp apart: its m2^1.5 is rounded once, where glibc's pow, Polars', can round the other way near a midpoint. A Float32 answers a Float32 and any other number a Float64. A window of values all alike is NaN, as is one holding a NaN or an infinity; without bias, a skew needs three values and a kurtosis four.

func (Expr) RollingStd added in v0.5.0 

func (e Expr) RollingStd(size int, opts ...WindowOption) Expr

func (Expr) RollingStdBy added in v0.6.0 

func (e Expr) RollingStdBy(by Expr, period Interval, opts ...WindowOption) Expr

func (Expr) RollingSum added in v0.5.0 

func (e Expr) RollingSum(size int, opts ...WindowOption) Expr

RollingSum, RollingMean, RollingMin, RollingMax, RollingVar and RollingStd answer each row over the size rows up to and including it: Polars' rolling_* with a window_size, center=False. RollingCenter() labels each window at its middle row instead, as center=True does. 

Col("price").RollingMean(7)                           // the week's average, from the 7th row
Col("price").RollingMean(7).Over(Col("ticker"))       // per ticker
Col("price").RollingMax(3, MinSamples(1)).OverWith(spec)  // in the spec's order

The rows are the window's: a partition's, in its order, with .Over or .OverWith, or else the frame's, in scan order. A row whose window holds fewer than MinSamples values — by default, size, so any null in a full window — is null.

Each answers what the aggregate of the same name would over that window, with the same type: a sum of integers is an Int128, a mean, var and std of numbers floats, a min and max the column's own type, of any type with an order. A NaN in a float window makes its sum, mean, var and std NaN, and its max NaN, as the aggregates'.

func (Expr) RollingSumBy added in v0.6.0 

func (e Expr) RollingSumBy(by Expr, period Interval, opts ...WindowOption) Expr

RollingSumBy, RollingMeanBy, RollingMinBy, RollingMaxBy, RollingVarBy and RollingStdBy answer each row over the rows whose by value lies within period of its own, (t − period, t]: Polars' rolling_*_by, and the expression form of the frame-level Rolling. 

Col("amount").RollingSumBy(Col("at"), Every("1h"))               // the hour up to each row
Col("amount").RollingMeanBy(Col("at"), Every("7d")).Over(Col("account"))

The window is found by value, not position: by need not be sorted, the rows that tie with a row's by are in its window, the later ones too, and a row whose by is null is in no window and is null. By default a window answers from one value, as Polars' min_samples=1; MinSamples asks for more. WindowClosed chooses which ends of the window are in it.

by is a Date, Datetime or Time, or an integer of 32 or 64 bits, whose period counts it in Polars' integer unit, Every("3i") (step 312). A calendar period counts months and days on the wall clock of by's zone, so Every("1d") across a change of offset is a calendar day, and FromDuration(24*time.Hour) twenty-four hours. The answers' types are the rolling functions' of the same names.

Where Polars differs: it answers the sum of a window holding no value, one empty under closed="left" or "none" or one whose rows are all null, as 0.0, although min_samples is 1. Here that window is null.

func (Expr) RollingVar added in v0.5.0 

func (e Expr) RollingVar(size int, opts ...WindowOption) Expr

func (Expr) RollingVarBy added in v0.6.0 

func (e Expr) RollingVarBy(by Expr, period Interval, opts ...WindowOption) Expr

func (Expr) Round 

func (e Expr) Round(decimals int, mode ...RoundMode) Expr

Round rounds to decimals decimal places, HALF AWAY FROM ZERO, or by mode.

Round(0.5) is 1 and Round(2.5) is 3, as a spreadsheet rounds, rather than IEEE's round-half-to-even. That is Polars' `mode="half_away_from_zero"`, and NOT Polars' default, which rounds half to even: Polars' `round(0)` gives 2 for 2.5, as Round(0, RoundHalfToEven) does here. It preserves the operand's type: rounding an integer column is the identity, not a widening.

It rounds twice — the value is scaled by 10^decimals first, and that multiply rounds too. Round(2.675, 2) is 2.68, because 2.675 * 100 lands on exactly 267.5 even though 2.675 itself is stored as 2.67499999999999982… The scaling is Polars', so each answer is Polars' to the bit.

A Float32 is rounded as a Float64 and the result rounded once more to a float32, so Round(0.1, 1) of a Float32 is the float32 nearest 0.1.

func (Expr) RoundSigFigs added in v0.6.6 

func (e Expr) RoundSigFigs(digits int) Expr

RoundSigFigs rounds to digits significant figures, half away from zero, in the operand's own type: Polars' round_sig_figs. 1234.5678 to two figures is 1200 and -0.0012345 is -0.0012. An integer is rounded through a float64 and brought back saturating, as Polars brings it, so an Int8 127 to two figures stays 127, its 130 past the type. A Decimal is refused, as Round refuses it.

func (Expr) SampleFraction added in v0.6.6 

func (e Expr) SampleFraction(fraction float64, withReplacement, shuffle bool, seed uint64) Expr

func (Expr) SampleN added in v0.6.6 

func (e Expr) SampleN(n int, withReplacement, shuffle bool, seed uint64) Expr

SampleN is n of the values drawn, and SampleFraction a fraction of them, as Polars' sample: with replacement or without, a sample of more than there are refused then; in their order, or shuffled. Each draw is seeded, by the seed and the values, so a group's sample does not depend on how the rows came in batches. The generator is ursus's, so the values are not Polars' (v0.6.6-scope.md §6.3). A column of its own length in a Select and a list a group in an aggregation.

func (Expr) SearchSorted added in v0.6.6 

func (e Expr) SearchSorted(value any, side SearchSide, descending bool) Expr

SearchSorted is the index at which value would be inserted into the column, sorted ascending, or descending, to keep it sorted, a UInt32: Polars' search_sorted, its value a Go value or a null. It is an aggregate, so an Expr holding one, such as Max(), cannot be the value, as an aggregate is not taken inside another's input; a column of values is not one value either, where Polars searches for each. SideLeft is before its equals and SideRight after them; SideAny is SideLeft.

Its nulls are where Polars' are: first, where the first value is null, else last. A null value goes before them, or after them with SideRight; where there is no null it is 0. Over an unsorted column it is the count of values that sort before value, where Polars' binary search lands where it lands.

func (Expr) Shift 

func (e Expr) Shift(n int) Expr

Shift moves values n places later within the window, filling the vacated rows with null. A negative n shifts earlier — Polars' lag and lead, one function.

Rows shifted in from outside the partition are null, never a neighbouring partition's value.

func (Expr) ShiftFill 

func (e Expr) ShiftFill[T Operand](n int, fill T) Expr

ShiftFill is Shift with a value instead of null in the vacated rows.

The fill meets the column as FillNullWith's does: a Go value that fits the column's type takes it, so ShiftFill(1, 0) on an Int32 column is an Int32, and otherwise the two promote, so ShiftFill(1, 1.5) on an Int64 column is a Float64.

func (Expr) Shuffle added in v0.6.6 

func (e Expr) Shuffle(seed uint64) Expr

Shuffle is the values in an order drawn by seed: Polars' shuffle.

func (Expr) Sign 

func (e Expr) Sign() Expr

Sign is -1, 0 or 1, keeping the operand's type.

For floats it returns the operand itself in the zero case, so sign(-0.0) is -0.0 and sign(NaN) is NaN. That matches NumPy and falls out of the ordering test rather than being special-cased.

func (Expr) Sin added in v0.4.0 

func (e Expr) Sin() Expr

The trigonometric and hyperbolic functions, in radians, with Polars' names. Like Sqrt they widen to a float, keeping Float32, and a value outside the domain — ArcSin(2), ArcCosh(0) — is NaN rather than an error or a null.

func (Expr) Sinh added in v0.4.0 

func (e Expr) Sinh() Expr

func (Expr) Skew added in v0.6.6 

func (e Expr) Skew(bias bool) Expr

Skew is the skewness of the non-null values, a Float64: Polars' skew. bias true is the biased estimator, m3/m2^1.5, Polars' default; false corrects it, and is null below 3 values. Kurtosis is m4/m2², less 3 with fisher, Fisher's excess and Polars' default; bias false corrects it, and is null below 4 values. Over values all alike either is NaN, as Polars'.

The moments are kept online and merged pairwise, so a group-by runs N-ways; Polars computes them in two passes, so the two agree to the last few bits.

func (Expr) Slice added in v0.6.5 

func (e Expr) Slice(offset, length int) Expr

Slice is length values from offset; a negative offset counts from the end.

func (Expr) Sort added in v0.6.5 

func (e Expr) Sort() Expr

Sort is the values in ascending order, nulls first, as Polars' sort() puts them.

func (Expr) SortBy added in v0.6.5 

func (e Expr) SortBy(by Expr) Expr

SortBy is the values in the order of by's, ascending, nulls first, stably: in a group, each value sorted by the by of its own row, as Polars' sort_by sorts (step 253). by is read on the same rows, so Col("a").SortBy(Col("b")).Last() is the a of the group's largest b.

func (Expr) SortByDesc added in v0.6.5 

func (e Expr) SortByDesc(by Expr) Expr

SortByDesc is SortBy in descending order of by, nulls first.

func (Expr) SortDesc added in v0.6.5 

func (e Expr) SortDesc() Expr

SortDesc is the values in descending order, nulls first.

func (Expr) Sqrt 

func (e Expr) Sqrt() Expr

Sqrt, Cbrt and Exp. Sqrt of a negative is NaN, not an error and not a null.

func (Expr) Std 

func (e Expr) Std(ddof int) Expr

func (Expr) Str 

func (e Expr) Str() StrExpr

Str opens the string namespace.

func (Expr) String 

func (e Expr) String() string

String renders the expression canonically.

func (Expr) Struct added in v0.3.0 

func (e Expr) Struct() StructExpr

Struct opens the struct namespace.

func (Expr) Sub 

func (e Expr) Sub[T Operand](v T) Expr

func (Expr) Suffix 

func (e Expr) Suffix(s string) Expr

Suffix appends to the output name. Expansion-safe, like Prefix.

func (Expr) Sum 

func (e Expr) Sum() Expr

Sum adds the non-null values.

An INTEGER sum accumulates and returns Int128, following DuckDB. Summing a thousand Int8 values overflows an Int8 immediately, and a wrapped sum is a plausible-looking wrong number that nothing downstream can detect. At 128 bits overflow needs 2^63 maximum-magnitude rows, i.e. it cannot happen.

A FLOAT sum accumulates in Float64 even when it returns Float32, because naive float32 accumulation stops making progress past ~2^24 elements.

A BOOL sums as 0 and 1, so its sum counts the trues, as an Int128. Count counts the non-null rows, falses included.

Sum of a group with no non-null values is NULL, not 0 — ursus follows SQL here rather than Polars. `0` cannot be distinguished afterwards from a genuine zero; NULL can be turned into 0 with FillNull if that is what you want.

func (Expr) Tail added in v0.6.5 

func (e Expr) Tail(n int) Expr

Tail is the last n values.

func (Expr) Tan added in v0.4.0 

func (e Expr) Tan() Expr

func (Expr) Tanh added in v0.4.0 

func (e Expr) Tanh() Expr

func (Expr) ToPhysical added in v0.6.7 

func (e Expr) ToPhysical() Expr

ToPhysical is the values as they are stored, as Polars' to_physical (step 324): a Date's days as an Int32, a Datetime's, a Duration's and a Time's ticks as an Int64, a Decimal's unscaled units as an Int128, and an Enum's category as the smallest unsigned integer that holds every one; any other type as it is. Only an Enum's are copied. A nested type is refused.

func (Expr) TopK added in v0.4.0 

func (e Expr) TopK(k int) Expr

TopK is the k largest values of the group, largest first, as a List. 

GroupBy(Col("id6")).Agg(Col("v3").TopK(2).Alias("largest2")).Explode("largest2")

It holds only k values a group, so unlike Implode it does not grow with the data, and it keeps a parallel group-by parallel. Values are in ursus's total order — NaN is the largest float, as Sort places it. Nulls are skipped: a group with fewer than k values gives a shorter list, and one with none gives null. Defined for numeric, temporal, decimal and string columns; k must be at least 1.

func (Expr) TopKBy added in v0.6.7 

func (e Expr) TopKBy(by Expr, k int) Expr

TopKBy is the k values whose by is the greatest, greatest first, as a List: Polars' top_k_by (step 330). by is any ordered type, NaN the greatest float, and a null by comes after every value. A value's null is kept, as Polars keeps it, so a group's list is min(k, its rows) long. 

GroupBy(Col("region")).Agg(Col("product").TopKBy(Col("sales"), 3))

Rows of one by are ordered by their values, the greater first, a null last, where Polars' sort orders ties as it finds them: so the answer depends on no row order, and a group-by keeps its workers, as with TopK. Like TopK, it holds only k rows a group. The value is numeric, temporal, decimal, string or Boolean; k is at least 1.

func (Expr) TrueDiv added in v0.6.7 

func (e Expr) TrueDiv[T Operand](v T) Expr

TrueDiv is Div, Polars' truediv: true division, a float, as Div always is (step 324).

func (Expr) Truncate added in v0.6.6 

func (e Expr) Truncate(decimals int) Expr

Truncate drops each value's digits past decimals decimal places, toward zero: Polars' truncate, Round's scaling with math.Trunc for its rounding. Truncate(1) of 0.29 is 0.2 and of -0.99 -0.9; an integer is its own truncation.

func (Expr) Unique added in v0.6.5 

func (e Expr) Unique() Expr

Unique is the distinct values in the order they first appear, a null among them. Polars' unique() orders them as it likes unless asked to keep their order.

func (Expr) UniqueCounts added in v0.6.6 

func (e Expr) UniqueCounts() Expr

UniqueCounts is each distinct value's count, in the order the values first appear, a null among them, a UInt32: Polars' unique_counts, a column of its own length in a Select and a list a group in an aggregation.

func (Expr) Unnest added in v0.3.0 

func (e Expr) Unnest() Expr

Unnest is a Struct column's fields, a column each named after its field: Polars' struct.unnest() (step 260). It denotes several columns, as Col("a", "b") does, and is written out when the query is planned: 

lf.Select(Col("person").Unnest(), Col("id"))     // age, city, id
lf.WithColumns(Col("person").Unnest())            // the frame, then age and city
lf.Select(Col("point").Unnest().Mul(2))           // each field doubled

The frame-level lf.Unnest replaces the struct in place; this keeps it where WithColumns keeps the frame's columns. A null struct's fields are null. An Alias names every field alike, and so is refused for a struct of two fields or more.

The single-field form is `Col("person").Struct().Field("age")`.

func (Expr) UpperBound added in v0.6.7 

func (e Expr) UpperBound() Expr

func (Expr) ValueCounts added in v0.6.6 

func (e Expr) ValueCounts(sort, normalize bool, name ...string) Expr

ValueCounts is each distinct value beside its count, a Struct of the value, named as the column, and the count, a UInt32: Polars' value_counts, a column of its own length in a Select and a list a group in an aggregation. With sort the most frequent come first; without, the values come in the order they first appear, where Polars' order is its hash table's. With normalize the count is its share of the rows, nulls among them, a Float64. The count is named name if given, else "count", or "proportion" with normalize, as Polars names it.

func (Expr) Var 

func (e Expr) Var(ddof int) Expr

Var and Std are the variance and standard deviation of the non-null values.

ddof is the delta degrees of freedom: 0 for the population statistic, 1 for the sample one. It has no default because there is no defensible default — Polars and pandas use 1, numpy uses 0, and silently picking either produces a number that is wrong for half its readers.

A group with ddof or fewer values returns NULL: the sample variance of a single observation is undefined, and 0 would claim the data has no spread rather than that the question has no answer.

Computed with Welford's algorithm, so a column of large near-equal values gives the right answer rather than the zero (or negative) that E[x²]−E[x]² produces.

A temporal column, a Duration included, is refused for now; Cast(Int64) gives its ticks.

func (Expr) Xor 

func (e Expr) Xor[T Operand](v T) Expr

type Field 

type Field = dtype.Field

Field is a named, typed, nullable column slot.

type FillStrategy 

type FillStrategy uint64

FillStrategy names a fill rule that needs no value at the call site.

It is a root-package enum and never reaches the IR, so it is not a dedup key and carries none of the collision hazard the expression enums do.

Its low byte is the rule; the bits above carry a limit, which Limit sets (step 301). So compare a strategy through kind, never with ==: FillForward.Limit(1) is not FillForward. 

const (
	// FillZero and FillOne fill with a constant. The literal is WEAK, so the
	// column's type is preserved rather than widened — which matters more here than
	// anywhere else, because the call site contains no literal at all to explain a
	// widening.
	FillZero FillStrategy = iota
	FillOne

	// FillForward and FillBackward carry the neighbouring value. Both are ordered
	// window functions, so both are pipeline breakers.
	FillForward
	FillBackward

	// FillMin, FillMax and FillMean fill with the column's own aggregate over the
	// WHOLE FRAME. That makes them pipeline breakers too: the window sink holds
	// every row, and past a memory limit writes them to one file and reads them
	// back beside the aggregate (step 197).
	//
	// FillMean returns a FLOAT even for an integer column, because a mean is a
	// float. FillMin and FillMax preserve the type.
	FillMin
	FillMax
	FillMean
)

func (FillStrategy) Limit added in v0.6.7 

func (s FillStrategy) Limit(n int) FillStrategy

Limit caps how many consecutive nulls one value fills: FillForward.Limit(1) fills the null after a value and no more, as Polars' fill_null(strategy="forward", limit=1). It is for FillForward and FillBackward, as Polars' limit is; another strategy's is refused when the fill is used, as Polars refuses it.

n is at least 1. Polars' limit=0 fills nothing; here it is refused, because ForwardFill(0) and BackwardFill(0) mean no limit at all, and a strategy whose 0 meant the opposite would be a trap. Leave the limit off for none.

func (FillStrategy) String 

func (s FillStrategy) String() string

type GroupBy 

type GroupBy struct {
	// contains filtered or unexported fields
}

GroupBy is the intermediate handle between GroupBy and Agg.

func (*GroupBy) Agg 

func (g *GroupBy) Agg(exprs ...Expr) *LazyFrame

Agg reduces each group.

Every expression must be an aggregation: every column it reads has to pass through an aggregate function. `Col("a").Sum().Add(1)` is fine; `Col("a").Sum().Add(Col("b"))` is not, because b is still a per-row value and there is no defined way to combine the two shapes.

func (*GroupBy) All added in v0.6.7 

func (g *GroupBy) All() *LazyFrame

All is each non-key column's values a group, a List in the frame's order, nulls among them: Polars' group_by().all() (step 310). It is Implode of each, and its memory is the data, as Implode's is: under WithMemoryLimit the values of the groups held must fit, since partitioning sends new keys to disk but not the values of those it already holds.

func (*GroupBy) Count 

func (g *GroupBy) Count() *LazyFrame

Count is shorthand for Agg(Len().Alias("count")).

func (*GroupBy) First 

func (g *GroupBy) First() *LazyFrame

func (*GroupBy) FirstNonNull added in v0.6.7 

func (g *GroupBy) FirstNonNull() *LazyFrame

FirstNonNull and LastNonNull are each non-key column's first or last value that is not null: Polars' first(ignore_nulls=True) and last(ignore_nulls=True) (step 310).

func (*GroupBy) Groups added in v0.6.7 

func (g *GroupBy) Groups(ctx context.Context, opts ...CollectOption) iter.Seq2[Partition, error]

Groups is each group's rows as a frame, beside its key, as Polars' `for key, df in df.group_by(...)` gives them: the groups in the order each first appears, each group's rows in the frame's order, a key computed from a column its own column. It runs the query once and holds its answer, as PartitionBy does, which it is. 

for g, err := range lf.GroupBy(ursus.Col("region")).Groups(ctx) { ... g.Key, g.Frame ... }

func (*GroupBy) Having added in v0.6.7 

func (g *GroupBy) Having(preds ...Expr) *GroupBy

Having keeps the groups whose predicates all hold, as Polars' having does: each predicate an aggregate of its group, Boolean, and a group whose predicate is null left out. Called again, or given several, every one must hold. It filters what Agg gives, the shorthands Sum, Len and the rest, and Head and Tail; a dynamic or a rolling group's windows too. 

lf.GroupBy(ursus.Col("region")).
	Having(ursus.Len().Gt(1), ursus.Col("qty").Sum().Gt(10)).
	Agg(ursus.Col("price").Mean())

The predicates are aggregated beside Agg's expressions, the groups filtered by them, and the predicates dropped: a filter after the aggregation, as Polars' doc says it is.

func (*GroupBy) Head added in v0.6.7 

func (g *GroupBy) Head(n int) *LazyFrame

Head is each group's first n rows, as Polars' GroupBy.head: the key columns first, then the frame's others, the groups in the order each first appears, and each group's rows in the frame's order. A group of n rows or fewer gives them all, and a key computed from a column replaces a column of its name, as in Polars. 

lf.GroupBy(ursus.Col("region")).Head(2)   // each region's first two orders

It is a window over the keys numbering each group's rows, a filter and a sort, so it spills past a memory limit as each does, and its order holds there too. Polars builds it as an aggregation of every column's head, exploded; here an aggregation would hold every value of a group to keep n of them. Head(0) is no rows, where Polars' is a row of nulls for each group.

func (*GroupBy) Last 

func (g *GroupBy) Last() *LazyFrame

func (*GroupBy) LastNonNull added in v0.6.7 

func (g *GroupBy) LastNonNull() *LazyFrame

func (*GroupBy) Len 

func (g *GroupBy) Len(name string) *LazyFrame

Len counts the rows in each group under the given name.

Distinct from Count, which uses the fixed name "count": Len takes one because a frame that already has a column called "count" is not unusual, and silently colliding would be worse than asking.

func (*GroupBy) MaintainOrder 

func (g *GroupBy) MaintainOrder() *GroupBy

MaintainOrder emits groups in first-appearance order.

Without it the group order is unspecified — which is standard (SQL uses set semantics and DuckDB exploits it for parallelism) but means a test that assumes an order is flaky.

The flag is no longer free, and that is the point of it

It USED to document a guarantee the implementation happened to give anyway. Since step 12 a group-by over more distinct keys than the memory limit holds partitions the overflow to disk, and the unordered path then emits partition-major: resident groups first, then partition 0's groups, then partition 1's. The order depends on where the freeze fell, so it depends on the limit. Since step 150 a group-by of tens of thousands of groups or more, on several threads, folds its workers' tables partitioned, each partition on its own goroutine, and emits partition-major as well, in an order that depends on the thread count.

The CONTENT never does. Every group, every value, identical at every limit and thread count. See WithMemoryLimit, which states the same contract from the other side. MaintainOrder runs serially.

Costs nothing when the aggregation fits in memory. When it spills it adds an int64 ordinal to every routed row and materialises the whole result to permute it — plan.Aggregate.MaintainOrder has the detail.

func (*GroupBy) MapGroups added in v0.6.7 

func (g *GroupBy) MapGroups(ctx context.Context, fn func(*DataFrame) (*DataFrame, error),
	opts ...CollectOption) (*DataFrame, error)

MapGroups is fn of each group's rows, the answers stacked in the groups' order, as Polars' map_groups: a Go function over a frame where an aggregation will not do. Each answer must have the first's columns and types. 

top, err := lf.GroupBy(ursus.Col("region")).MapGroups(ctx, func(g *ursus.DataFrame) (*ursus.DataFrame, error) {
	return g.Lazy().Sort(ursus.Desc(ursus.Col("amount"))).Head(3).Collect(ctx)
})

func (*GroupBy) Max 

func (g *GroupBy) Max() *LazyFrame

func (*GroupBy) Mean 

func (g *GroupBy) Mean() *LazyFrame

func (*GroupBy) Median 

func (g *GroupBy) Median() *LazyFrame

func (*GroupBy) Min 

func (g *GroupBy) Min() *LazyFrame

Min, Max, First, Last and NUnique apply to EVERY non-key column, because they select or count rather than compute.

func (*GroupBy) NUnique 

func (g *GroupBy) NUnique() *LazyFrame

func (*GroupBy) Quantile 

func (g *GroupBy) Quantile(q float64, interp Interpolation) *LazyFrame

Quantile applies one quantile to every numeric non-key column.

func (*GroupBy) Sum 

func (g *GroupBy) Sum() *LazyFrame

Sum, Mean, Min, Max, Median, NUnique, First and Last apply that aggregate to every non-key column. Sum, Mean, Median and Quantile apply to the NUMERIC non-key columns.

func (*GroupBy) Tail added in v0.6.7 

func (g *GroupBy) Tail(n int) *LazyFrame

Tail is each group's last n rows, in the frame's order: Polars' GroupBy.tail, and Head's in all else.

type HistOption added in v0.6.6 

type HistOption func(*histOptions)

HistOption configures Hist.

func HistBinCount added in v0.6.6 

func HistBinCount(n int) HistOption

HistBinCount counts in n bins of equal width, from the least value to the greatest: Polars' bin_count, 10 by default.

func HistBins added in v0.6.6 

func HistBins(edges ...float64) HistOption

HistBins counts in the bins between edges, which must increase: Polars' bins. The first bin holds its left edge too, and a value outside the edges is in none; fewer than two edges make no bin.

func HistBreakpoint added in v0.6.6 

func HistBreakpoint() HistOption

HistBreakpoint and HistCategory make each bin a Struct of its right edge, a Float64, and its interval as text, "[1.0, 3.333333]" then "(3.333333, 5.666667]", beside its count: Polars' include_breakpoint and include_category, the category a Categorical, as Polars' is (step 352).

func HistCategory added in v0.6.6 

func HistCategory() HistOption

type IPCCompression added in v0.6.7 

type IPCCompression int

IPCCompression is the codec an Arrow IPC file's buffers are written in. 

const (
	IPCUncompressed IPCCompression = iota // Polars' default
	IPCLZ4                                // LZ4 frames
	IPCZstd                               // Zstandard
)

type IPCScanOption added in v0.6.7 

type IPCScanOption func(*ipcScan)

IPCScanOption configures ScanIPC, ScanIPCFiles and ScanIPCGlob. New in 0.6.7.

func WithIPCHiveDates added in v0.6.7 

func WithIPCHiveDates(b bool) IPCScanOption

WithIPCHiveDates reads a partition key of dates or of instants as a Date or a Datetime, as Polars' try_parse_hive_dates; on by default. New in 0.6.7.

func WithIPCHivePartitioning added in v0.6.7 

func WithIPCHivePartitioning(b bool) IPCScanOption

WithIPCHivePartitioning reads the key=value directories of each file's path as columns, as Polars' hive_partitioning: on by default for a directory, off for a file, a list and a glob. New in 0.6.7.

func WithIPCHiveSchema added in v0.6.7 

func WithIPCHiveSchema(s *Schema) IPCScanOption

WithIPCHiveSchema gives the types of the partition keys it names, as Polars' hive_schema; the others are inferred. New in 0.6.7.

func WithIPCIncludeFilePaths added in v0.6.7 

func WithIPCIncludeFilePaths(name string) IPCScanOption

WithIPCIncludeFilePaths adds a String column named name of each row's file path, last, as Polars' include_file_paths. New in 0.6.7.

type IPCSinkOption added in v0.6.7 

type IPCSinkOption interface {
	// contains filtered or unexported methods
}

IPCSinkOption is an option to WriteIPC, WriteIPCStream and SinkIPC: an IPCWriteOption or a CollectOption.

type IPCWriteOption added in v0.6.7 

type IPCWriteOption func(*ipcWriteCfg)

IPCWriteOption configures an IPC writer.

func WithIPCCompression added in v0.6.7 

func WithIPCCompression(c IPCCompression) IPCWriteOption

WithIPCCompression sets the codec, as Polars' write_ipc(compression=...): none by default, as Polars'.

type IfTableExists added in v0.6.8 

type IfTableExists int

IfTableExists is what WriteDatabase does with a table already there: Polars' if_table_exists. 

const (
	// IfExistsFail refuses a table already there, and creates one that is not: the
	// default.
	IfExistsFail IfTableExists = iota
	// IfExistsReplace drops a table already there and creates it again.
	IfExistsReplace
	// IfExistsAppend inserts into a table already there, and creates one that is not.
	IfExistsAppend
)

type Int128Value 

type Int128Value = i128.Int128

Int128Value is a signed 128-bit integer, the Go representation of an Int128 column. Read one with df.Column[ursus.Int128Value]("total").

type Interpolation 

type Interpolation = expr.Interpolation

Interpolation selects how Quantile resolves a rank falling between two values.

Aliased from the expression IR rather than redeclared, so the kernel and the public API cannot drift — the same arrangement JoinKind uses.

type Interval 

type Interval = dtype.Interval

Interval is a calendar-aware span: months, days and nanoseconds, kept apart because each obeys a different rule. See dtype.Interval for why a time.Duration cannot express "1 month" or, in a daylight-saving zone, "1 day".

func Every 

func Every(s string) Interval

Every parses an interval: "1y", "3mo", "1w", "2d", "6h", "15m", "30s", "1ns". Components may be concatenated ("1h30m") and a leading "-" negates the whole thing.

A grid of weeks starts on a Monday, as Polars': Truncate(Every("1w")) and a GroupByDynamic every "1w" give Mondays, where "7d" counts days from the epoch, a Thursday. Weeks mixed with any other unit count from the epoch too.

"3i" is Polars' integer interval, three steps of an integer index: a GroupByDynamic, a Rolling or an Upsample over an Int32 or an Int64 index steps by it, as do the windows by such a column. It mixes with no unit of time, and is refused where a date is moved: by Truncate, Round, OffsetBy, a range or an as-of tolerance.

A parse failure is carried in the value rather than returned, and surfaces when the query is built — the deferred-error rule the rest of the library follows. Note "m" is minutes and "mo" is months.

func FromDuration 

func FromDuration(d time.Duration) Interval

FromDuration builds an ABSOLUTE interval. Deliberately not the same as the calendar spelling: FromDuration(24*time.Hour) is twenty-four hours, Every("1d") is one day, and on the two days a year a zone changes its offset those are different instants.

func NoOffset added in v0.6.7 

func NoOffset() Interval

NoOffset is a rolling group's offset of none, Polars' offset of "0d": a window that starts at its row's own instant, (t, t + Period]. RollingOptions' zero Offset is Polars' default instead, minus the period (step 313).

type JoinKind 

type JoinKind = plan.JoinKind

JoinKind is the shape of a join: which rows survive.

type JoinOption 

type JoinOption func(*joinCfg)

JoinOption configures a join.

func JoinCoalesce 

func JoinCoalesce(b bool) JoinOption

JoinCoalesce forces key merging on or off.

Unset is not the same as either: by default the key appears once for inner, left, right, semi and anti joins, and twice for a full join — which can leave the key null on either side, so there is no single side to take it from. JoinCoalesce(true) merges a full join's keys too: the left key where the row has a left side, and the right key where it does not.

The default only merges keys that are named the SAME on both sides. Merging JoinLeftOn(Col("cust_id")) with JoinRightOn(Col("id")) would silently delete a column the caller named explicitly, so it takes an explicit JoinCoalesce(true).

func JoinHow 

func JoinHow(k JoinKind) JoinOption

JoinHow sets the join kind. Default JoinInner.

func JoinLeftOn 

func JoinLeftOn(keys ...Expr) JoinOption

JoinLeftOn and JoinRightOn name the key columns separately, for frames where they are called different things. They are used together and must agree in count; keys are paired positionally.

func JoinMaintainOrder added in v0.4.0 

func JoinMaintainOrder(o JoinOrder) JoinOption

JoinMaintainOrder keeps a join's rows in an order, as Polars' maintain_order does (step 314): 

orders.Join(customers, ursus.JoinOn(ursus.Col("cid")), ursus.JoinHow(ursus.JoinLeft),
	ursus.JoinMaintainOrder(ursus.JoinOrderLeft))

JoinOrderLeft is the left frame's rows in their order, each row's matches in the right frame's, and then a right or full join's unmatched right rows, in theirs. JoinOrderRight is its mirror: the right frame's rows, each row's matches in the left frame's, then a left or full join's unmatched left rows. A semi or anti join keeps the left frame's order, either way, and a cross join is the left's rows each beside every right row, or the right's beside every left.

The order holds under a memory limit too: a join that spills sorts its rows back into it, by their places in the inputs, which a join that fits never does. A right order hashes the left frame rather than the right.

Without it, as in Polars, the order is not promised: ursus may hash either input of an inner join, the smaller estimated from the sources' row counts, and the rows come in the other's order; and a join that spills gives its rows in no order. The other kinds keep the left frame's order while they fit.

func JoinNullsEqual 

func JoinNullsEqual(b bool) JoinOption

JoinNullsEqual makes null keys match each other. Default false, which is SQL's rule and Polars': a null key matches nothing, including another null.

Note this is deliberately the OPPOSITE of GroupBy, where null forms its own group. Grouping asks "are these the same value?" and joining asks "is this the same entity?", and a missing identifier is not evidence of sameness.

func JoinOn 

func JoinOn(keys ...Expr) JoinOption

JoinOn names key columns present under the same name on both sides.

func JoinRightOn 

func JoinRightOn(keys ...Expr) JoinOption

func JoinSuffix 

func JoinSuffix(s string) JoinOption

JoinSuffix sets what is appended to a right-side column whose name collides with a left-side one. Default "_right".

func JoinValidate 

func JoinValidate(v JoinValidation) JoinOption

JoinValidate asserts a key cardinality, failing the query when the data violates it. 

orders.Join(customers, JoinOn(Col("customer_id")),
    JoinValidate(ursus.ValidateManyToOne))

Worth reaching for: an unexpected duplicate on the right multiplies every matching left row, and the result looks entirely plausible.

type JoinOrder added in v0.6.7 

type JoinOrder = plan.JoinOrder

JoinOrder is the row order a join promises, Polars' maintain_order: see JoinMaintainOrder.

type JoinValidation 

type JoinValidation = plan.JoinValidation

JoinValidation asserts a cardinality and errors when the data violates it.

The first term names the LEFT frame: ManyToOne on orders.Join(customers) asserts that the customers are unique.

type KeepStrategy added in v0.6.7 

type KeepStrategy uint8

KeepStrategy is which of a key's rows UniqueKeep keeps: Polars' unique(keep=). 

const (
	// KeepFirst keeps each key's first row; it is what Unique does.
	KeepFirst KeepStrategy = iota
	// KeepLast keeps each key's last row.
	KeepLast
	// KeepAny keeps one of each key's rows. Here it is the first, as Unique keeps
	// it; Polars' is whichever its hash table meets first.
	KeepAny
	// KeepNone keeps only the rows whose key no other row has.
	KeepNone
)

type LazyFrame 

type LazyFrame struct {
	// contains filtered or unexported fields
}

LazyFrame is an unexecuted query.

Immutable

Every builder method returns a NEW LazyFrame sharing an immutable plan. So 

base := ursus.ScanParquet("events.parquet")
a := base.Filter(ursus.Col("x").Gt(5))
b := base.Filter(ursus.Col("x").Lt(0))

gives two independent queries, and `base` is unchanged. That idiom is the most common thing anyone does with a lazy frame, and it only works because nothing mutates in place. It also makes a LazyFrame safe to share between goroutines.

Sticky errors

A builder cannot return an error without destroying chaining, so errors are carried and surfaced at Collect (or read early via Err). An unknown column in Select does not panic and does not fail silently — it fails at Collect, with the available columns and a spelling suggestion.

func Concat 

func Concat(frames []*LazyFrame, opts ...ConcatOption) *LazyFrame

Concat stacks frames vertically, in the order given, or, in the horizontal and align modes, puts them side by side: Polars' concat. 

ursus.Concat([]*ursus.LazyFrame{jan, feb, mar})
ursus.Concat(fs, ursus.WithConcatMode(ursus.ConcatDiagonal))
ursus.Concat([]*ursus.LazyFrame{prices, volumes}, ursus.WithConcatMode(ursus.ConcatAlign))

Why a slice rather than a variadic

It was `Concat(frames ...any)` for one release, which read better and let the mode be passed inline. It also made `ursus.Concat(a, "oops")` compile and fail at run time — in a library that spends generic methods and the Operand constraint precisely so that `Col("x").Gt(struct{}{})` does not. Ergonomics is not worth the one property the type system was being paid to provide.

For the common case the method form keeps both: LazyFrame.Concat is variadic and type-safe, so `jan.Concat(feb, mar)` needs no slice literal.

Order is preserved

Every row of the first frame precedes every row of the second. That is not free — it is why the operator drains children serially rather than racing them — and it is what lets Concat compose with Head, Tail and anything else that cares which rows come first.

Reconciliation happens in the plan

A frame missing a column, or holding one at a narrower type, is wrapped in a projection that supplies the null and performs the cast, so Explain shows why a column is null and projection pushdown still narrows each input independently.

func Frame 

func Frame(cols ...*Column) *LazyFrame

Frame builds an in-memory LazyFrame from columns.

This is the entry point for data that is already in Go — the counterpart to ScanParquet and friends, which arrive in step 2.

func FrameOf 

func FrameOf(cols ...*Column) (*LazyFrame, error)

FrameOf is Frame for callers who want the error separately.

func FromRows added in v0.6.7 

func FromRows[T any](rows []T) *LazyFrame

FromRows builds a frame from Go structs, a column a field, in the fields' order: 

type sale struct {
	ID    int64   `ursus:"id"`
	Price float64 `ursus:"unit_price"`
	Note  *string // nil is null
}
lf := ursus.FromRows([]sale{{1, 9.5, nil}})

A field is named by its `ursus:"name"` tag, as Rows reads it, or by its own name; an unexported field, and one tagged `ursus:"-"`, is left out. A field takes the column type Values gives its Go type: the integers, floats, bool, string, []byte a Binary, time.Time a Datetime(ns, UTC) and time.Duration a Duration(ns). A pointer field is nullable, nil a null; a slice field, other than []byte, a List of its elements, nil a null list; and a struct field a Struct of its own fields, named as these are. Any other field type is refused, naming the field.

Each field is read once a row by its offset, so a frame of a million rows costs about what copying their fields does.

func ScanArrow added in v0.3.0 

func ScanArrow(open func() (array.RecordReader, error)) *LazyFrame

ScanArrow reads a stream of Arrow records.

open must return a NEW, independent reader every time it is called: ursus opens one to learn the schema when the query is planned, and one per scan each time it runs — a self-join scans twice in one query, and a LazyFrame can be collected again. A reader cannot be rewound, so handing over one reader would be read once and then be empty. 

stream, err := os.ReadFile("data.arrows") // an Arrow IPC stream
if err != nil { return err }
lf := ursus.ScanArrow(func() (array.RecordReader, error) {
    return ipc.NewReader(bytes.NewReader(stream))
})

Returning the same reader twice is refused by name rather than read as an empty stream.

Records are copied a batch at a time, and the reader is not advanced until its current record has been copied in full, so nothing ursus returns refers to memory the reader reuses or frees. Explain and CollectSchema open a reader and release it without reading a record. Head(n) stops asking for records once it has n rows. A reader blocked inside Next cannot be interrupted by the query's context.

Every column is nullable, and types are mapped as ScanArrowRecords describes.

func ScanArrowRecords added in v0.3.0 

func ScanArrowRecords(recs ...arrow.RecordBatch) *LazyFrame

ScanArrowRecords reads Arrow records already in memory.

The records are COPIED before this returns, so the caller may release them, or let the reader they came from move on, as soon as it likes: 

var recs []arrow.RecordBatch
for rdr.Next() {
    rec := rdr.RecordBatch()
    rec.Retain() // an IPC reader reuses the record on its next Next
    recs = append(recs, rec)
}
lf := ursus.ScanArrowRecords(recs...)
for _, rec := range recs {
    rec.Release()
}

Every record must convert to the same schema. Every column is nullable whatever the Arrow schema declares, because arrow-go never checks that flag and producers routinely leave it false while writing nulls. A type ursus cannot represent — a 256-bit decimal, an interval, a union — is refused by name.

With no records there is no schema, so that is an error too; hand over an empty record instead. For a stream too large to copy up front, use ScanArrow.

Example

Arrow both ways. Record shares a frame's memory with arrow-go; ScanArrowRecords copies records in, so the record can be released straight away. 

package main

import (
	"context"
	"fmt"
	"log"

	"github.com/advenn/ursus"
)

// exampleSales is the little frame the examples below work on.
func exampleSales() *ursus.LazyFrame {
	return ursus.Frame(
		ursus.Values("region", []string{"north", "south", "north", "east", "south"}),
		ursus.Values("product", []string{"apple", "apple", "pear", "pear", "fig"}),
		ursus.Values("qty", []int64{10, 4, 7, 2, 9}),
		ursus.Values("price", []float64{1.50, 1.50, 2.25, 2.25, 3.00}),
	)
}

func main() {
	ctx := context.Background()
	big, err := exampleSales().Filter(ursus.Col("qty").Gt(5)).Collect(ctx)
	if err != nil {
		log.Fatal(err)
	}
	rec, err := big.Record()
	if err != nil {
		log.Fatal(err)
	}

	lf := ursus.ScanArrowRecords(rec)
	rec.Release()

	df, err := lf.Select(ursus.Col("region"), ursus.Col("qty")).Collect(ctx)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(df)
}

Output:
shape: (3, 2)
┌─────────┬───────┐
│ region  │ qty   │
│ String  │ Int64 │
├─────────┼───────┤
│ "north" │ 10    │
│ "north" │ 7     │
│ "south" │ 9     │
└─────────┴───────┘

func ScanCSV 

func ScanCSV(path string, opts ...CSVOption) *LazyFrame

ScanCSV reads a CSV file lazily, or every file below a directory, as Polars reads one: as ScanCSVFiles of them, sorted by path, a name beginning '.' or '_' passed over and files of two extensions refused (the directory new in 0.6.7).

Nothing is read until Collect, except the sample inference needs — and not even that if WithSchema is given. Projection reaches the reader, so selecting two columns of a hundred parses two.

A file compressed by gzip, zlib or Zstandard is read as itself, known by its first bytes whatever its name, as Polars reads one, and decompressed as it is read; so is every CSV, NDJSON and JSON scan's stream. New in 0.6.7. 

df, err := ursus.ScanCSV("sales.csv").
    Filter(ursus.Col("qty").Gt(10)).
    Select(ursus.Col("region"), ursus.Col("qty")).
    Collect(ctx)

func ScanCSVFiles 

func ScanCSVFiles(paths []string, opts ...CSVOption) *LazyFrame

ScanCSVFiles reads several files as one frame, in the order given.

Columns are matched by the NAMES in each file's header: every file must name the same columns, in any order. A file that differs — a missing, extra, renamed or repeated column, or no header at all — is refused with ErrSchema, naming both files, when the scan reaches it. The types are inferred from the first file (or given by WithSchema). WithColumnNames renames the first file's columns by position, and later files are still matched to it by what the files say. With WithHasHeader(false) there is nothing to match, and files are read by position. A UTF-8 byte-order mark at the start of a file is dropped.

func ScanCSVFrom added in v0.3.0 

func ScanCSVFrom(files []CSVFile, opts ...CSVOption) *LazyFrame

ScanCSVFrom reads CSV files the caller opens, as one frame in the order given, under ScanCSVFiles' rules for several files.

It is how ursus reaches an object store without depending on one: give each file an Open that returns the body of a GET.

func ScanCSVGlob 

func ScanCSVGlob(pattern string, opts ...CSVOption) *LazyFrame

ScanCSVGlob reads every file matching a shell pattern as one frame.

The files must share their columns; the types are inferred from the first match, and every other file's header is matched to the first's by NAME — see ScanCSVFiles. Matches are sorted, so `part-*.csv` reads in the order the names imply rather than whatever the filesystem returns — row order is a property of the data here, and leaving it to readdir would make the same query return different orders on different machines. An element of ** matches any number of directories, none among them, as Polars' globs do: `logs/**/*.csv`. New in 0.6.7.

func ScanCSVReader 

func ScanCSVReader(b []byte, name string, opts ...CSVOption) *LazyFrame

ScanCSVReader reads CSV from memory.

It takes the bytes rather than an io.Reader because a source is opened more than once — once for inference, once per execution — and a Reader cannot be rewound.

func ScanIPC added in v0.6.7 

func ScanIPC(path string, opts ...IPCScanOption) *LazyFrame

ScanIPC reads an Arrow IPC file, Feather version 2, the format of Polars' write_ipc and read_ipc: its record batches a batch at a time, each column as ScanArrow converts one, string views among them, which Polars writes. The file is opened each time the query is planned or run, and closed after.

New in 0.6.7: a directory is read whole, every file below it in the order of their paths, as a hive dataset, each directory below it named key=value giving its files' rows a column key after the files' own, as Polars' scan_ipc reads one; see WithIPCHivePartitioning, WithIPCHiveSchema, WithIPCHiveDates and WithIPCIncludeFilePaths. Several files are read as the first one's columns, by name, as ScanIPCFiles reads them.

func ScanIPCFiles added in v0.6.7 

func ScanIPCFiles(paths []string, opts ...IPCScanOption) *LazyFrame

ScanIPCFiles reads IPC files as one frame, in the order given, as the first one's columns: every other file's are read by name, and one it lacks, one more or one of another type is refused, as Polars refuses them. New in 0.6.7.

func ScanIPCGlob added in v0.6.7 

func ScanIPCGlob(pattern string, opts ...IPCScanOption) *LazyFrame

ScanIPCGlob reads every IPC file a shell pattern matches as one frame, sorted by name, an element of ** matching any number of directories, as ScanIPCFiles reads them. New in 0.6.7.

func ScanIPCStream added in v0.6.7 

func ScanIPCStream(path string) *LazyFrame

ScanIPCStream reads an Arrow IPC stream, the format of Polars' write_ipc_stream and read_ipc_stream, as ScanIPC reads a file.

func ScanJSON added in v0.6.7 

func ScanJSON(path string, opts ...NDJSONOption) *LazyFrame

ScanJSON reads a JSON file of an array of objects, a row an object, or of one object, one row, as Polars' read_json: each object is read as ScanNDJSON reads a line, under its options and rules, the schema inferred from the first 100 objects unless given. As Polars, it refuses an element that is not an object, a null among them, a comma before the array's end, text after it, and a key the inferred schema has not, which a given schema's reading skips. An error names its object, counted from 1. New in 0.6.7.

The file is read as a stream, each object made a line of NDJSON in the buffer it is read into and parsed as one, in blocks a part a thread: the document is never held whole, nor parsed into a tree.

func ScanJSONBytes added in v0.6.7 

func ScanJSONBytes(b []byte, name string, opts ...NDJSONOption) *LazyFrame

ScanJSONBytes is ScanJSON of a JSON document in memory, named name in Explain. New in 0.6.7.

func ScanNDJSON added in v0.6.7 

func ScanNDJSON(path string, opts ...NDJSONOption) *LazyFrame

ScanNDJSON reads a file of newline-delimited JSON, a JSON object a line, as Polars' scan_ndjson and read_ndjson: a column a key, in the order the keys are first seen. The types are inferred from the first 100 lines, as Polars infers them:

  • an integer is an Int64, or an Int128 past it; a number with a fraction or an exponent a Float64; true and false a Boolean; text a String; only nulls Null;
  • an array a List and an object a Struct, of their values' types;
  • a key holding integers and floats Float64, and text beside anything String, the other value as Polars writes it: a number as Rust prints it, 1.5 for 1.50, and an array or an object as JSON;
  • an array's elements by Polars' rules for them: a Boolean and an integer meet in Int64, and two types no rule meets in String, so [1.0, {"b": 2}] is a List(String).

A Boolean beside a number, or a List or a Struct beside a number or a Boolean, has no type in common and is refused, as Polars refuses it, as is a later line whose value does not fit its column, unless WithNDJSONIgnoreErrors. A key missing from a line is a null, a key given twice its first value, a key the schema does not have skipped, and a blank line too. An error names its line and its key.

The file is read in blocks, each cut at its lines into a part a thread, the parts parsed at once; only the columns the query uses are built, and a query that wants only the first rows reads little more than them. A directory is read as ScanNDJSONFiles of every file below it, as Polars reads one, sorted by path, a name beginning '.' or '_' passed over and files of two extensions refused (new in 0.6.7).

func ScanNDJSONBytes added in v0.6.7 

func ScanNDJSONBytes(b []byte, name string, opts ...NDJSONOption) *LazyFrame

ScanNDJSONBytes is ScanNDJSON of NDJSON in memory, named name in Explain.

func ScanNDJSONFiles added in v0.6.7 

func ScanNDJSONFiles(paths []string, opts ...NDJSONOption) *LazyFrame

ScanNDJSONFiles reads several NDJSON files as one frame, in the order given, as Polars' scan_ndjson of a list: the schema is inferred from the first file's first lines, and a key a later file adds is skipped.

func ScanNDJSONFrom added in v0.6.7 

func ScanNDJSONFrom(files []NDJSONFile, opts ...NDJSONOption) *LazyFrame

ScanNDJSONFrom reads NDJSON files the caller opens, as one frame in the order given, under ScanNDJSONFiles' rules: an object store's, say, each Open the body of a GET.

func ScanNDJSONGlob added in v0.6.7 

func ScanNDJSONGlob(pattern string, opts ...NDJSONOption) *LazyFrame

ScanNDJSONGlob reads every file matching a shell pattern as one frame, as ScanNDJSONFiles of the matches sorted by name. An element of ** matches any number of directories, none among them, as Polars' globs do. New in 0.6.7.

func ScanParquet 

func ScanParquet(path string, opts ...ParquetOption) *LazyFrame

ScanParquet reads a Parquet file lazily.

Only the footer is read until Collect. Projection reaches the reader, so a query selecting two columns of a hundred never decompresses the other ninety-eight, and a predicate on a column with statistics skips whole row groups. 

df, err := ursus.ScanParquet("events.parquet").
    Filter(ursus.Col("ts").Gt(cutoff)).
    Select(ursus.Col("user"), ursus.Col("ts")).
    Collect(ctx)

Lists and structs are read; a column ursus cannot read — an encrypted one, a map, a list of lists — is refused with an error naming it when a query reads it, rather than dropped.

A directory is read whole, every file below it in the order of their paths, as Polars reads one, and as a hive dataset: each directory below it named key=value gives its files' rows a column key, after the files' own columns, of the type the values infer; a filter on the keys skips the files it rules out before any is opened. A name beginning '.' or '_' is passed over, and files of two extensions are refused, as Polars refuses them. See WithHivePartitioning, WithHiveSchema, WithHiveDates and WithIncludeFilePaths. New in 0.6.7.

func ScanParquetBytes 

func ScanParquetBytes(b []byte, name string, opts ...ParquetOption) *LazyFrame

ScanParquetBytes reads a Parquet file from memory.

func ScanParquetFiles 

func ScanParquetFiles(paths []string, opts ...ParquetOption) *LazyFrame

ScanParquetFiles reads several files as one frame, in the order given.

Columns are matched by NAME: every file must have the same columns, of the same types, in any order. A file that differs is refused with ErrSchema, naming both files and the difference, when the query is planned — every file's footer is read then, before any row is produced. A column is nullable if it is nullable in any file. To combine files that differ, scan them separately and Concat them.

func ScanParquetFrom added in v0.3.0 

func ScanParquetFrom(files []ParquetFile, opts ...ParquetOption) *LazyFrame

ScanParquetFrom reads Parquet files the caller opens, as one frame in the order given, under ScanParquetFiles' rules for several files.

It is how ursus reaches an object store without depending on one: give each file an Open that returns a ReaderAt over ranged GETs. 

files := []ursus.ParquetFile{{
    Name: "s3://logs/2026/10/part-0.parquet",
    Open: func(ctx context.Context) (io.ReaderAt, int64, error) {
        return newRangedReader(ctx, client, "logs", "2026/10/part-0.parquet")
    },
}}
df, err := ursus.ScanParquetFrom(files).Filter(ursus.Col("status").Eq(500)).Collect(ctx)
Example

Read Parquet from wherever it lives. Open returns an io.ReaderAt and its size — here over bytes in memory, standing in for a ranged GET against an object store. ursus reads the footer, then only the column chunks the query needs. 

package main

import (
	"bytes"
	"context"
	"fmt"
	"io"
	"log"

	"github.com/advenn/ursus"
)

// exampleSales is the little frame the examples below work on.
func exampleSales() *ursus.LazyFrame {
	return ursus.Frame(
		ursus.Values("region", []string{"north", "south", "north", "east", "south"}),
		ursus.Values("product", []string{"apple", "apple", "pear", "pear", "fig"}),
		ursus.Values("qty", []int64{10, 4, 7, 2, 9}),
		ursus.Values("price", []float64{1.50, 1.50, 2.25, 2.25, 3.00}),
	)
}

func main() {
	ctx := context.Background()
	var file bytes.Buffer
	if err := exampleSales().WriteParquet(ctx, &file); err != nil {
		log.Fatal(err)
	}

	lf := ursus.ScanParquetFrom([]ursus.ParquetFile{{
		Name: "s3://shop/sales.parquet",
		Open: func(ctx context.Context) (io.ReaderAt, int64, error) {
			return bytes.NewReader(file.Bytes()), int64(file.Len()), nil
		},
	}})
	n, err := lf.Filter(ursus.Col("product").Eq("pear")).Count(ctx)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(n, "pear sales")
}

Output:
2 pear sales

func ScanParquetGlob 

func ScanParquetGlob(pattern string, opts ...ParquetOption) *LazyFrame

ScanParquetGlob reads every file matching a shell pattern as one frame, sorted by name so `part-*.parquet` reads in the order the names imply. An element of ** matches any number of directories, none among them, as Polars' globs do. New in 0.6.7.

func (*LazyFrame) BottomK 

func (lf *LazyFrame) BottomK(k int, by ...SortKey) *LazyFrame

BottomK keeps the k smallest rows by the given keys. See TopK — including that nulls are placed last and so never appear in the result.

func (*LazyFrame) Cache added in v0.6.7 

func (lf *LazyFrame) Cache() *LazyFrame

Cache marks the query to run once where it is used at several places, as Polars' cache: a query reused, joined with itself or in two arms of a Concat, computes its answer once and every use reads it. ursus does so unasked for a reused query that joins, aggregates or sorts; Cache asks it for any other, a scan read twice or a Go function's answer. A query used once is not cached, as Polars drops its cache. A cached query is a fence: a filter or a projection above one use stops at it.

func (*LazyFrame) Cast added in v0.6.7 

func (lf *LazyFrame) Cast(types map[string]DataType, opts ...CastOption) *LazyFrame

Cast casts each named column to its type, and leaves the others as they are, in their places: Polars' cast({"a": pl.Int32}). A name the frame does not have is refused, as Polars refuses it, and a column already of its type is left out of the plan. 

lf.Cast(map[string]ursus.DataType{"qty": ursus.Int32, "price": ursus.Float32})

func (*LazyFrame) CastAll added in v0.6.7 

func (lf *LazyFrame) CastAll(to DataType, opts ...CastOption) *LazyFrame

CastAll casts every column to one type: Polars' cast(pl.String).

func (*LazyFrame) CastTypes added in v0.6.7 

func (lf *LazyFrame) CastTypes(types map[DataType]DataType, opts ...CastOption) *LazyFrame

CastTypes casts every column of each type to another: Polars' cast({pl.Int64: pl.Float32}). A type the frame has no column of changes nothing.

A key matches a type exactly: a Datetime key its unit and its zone, a List key its element. Polars' pl.Datetime, with no unit, matches every Datetime; here each one is its own key. 

lf.CastTypes(map[ursus.DataType]ursus.DataType{ursus.Int64: ursus.Int32})

func (*LazyFrame) Clear added in v0.6.7 

func (lf *LazyFrame) Clear(n int) *LazyFrame

Clear is the frame's schema with n rows of nulls, none by default: Polars' clear(n).

func (*LazyFrame) Collect 

func (lf *LazyFrame) Collect(ctx context.Context, opts ...CollectOption) (df *DataFrame, err error)

Collect runs the query and returns the whole result.

A result too large to hold is an error

Under the default budget, Collect counts its result as it arrives, and refuses once the query holds more, the result and what its operators hold together, than three quarters of the memory it may use: the default budget and half as much again. The error is a resource error naming collect.

Three quarters, because ursus sets the Go soft limit at nine tenths of the same ceiling, and a query holds up to about a fifth more than it counts: three quarters and a fifth is nine tenths. Past that the collector cannot keep the heap under the soft limit, and under a container's limit the kernel kills the process instead. CollectBatches, SinkParquet and SinkCSV hold one batch at a time and are not checked.

The result is counted apart from what the operators hold, so they spill exactly as they would if it were streamed, and MemoryStats.Peak leaves it out.

Under a limit you give with WithMemoryLimit, or none, Collect is not checked and holds whatever it is asked to, as it always has.

A panic is an error

Every entry point that runs a query recovers a panic in it and returns it as ErrInternal — "this is a bug in ursus" — and every goroutine the engine starts does the same for its own work, so no query can take down the process that asked it. A panic in a udf is the caller's own, and is a KindValue error naming the udf.

func (*LazyFrame) CollectBatches 

func (lf *LazyFrame) CollectBatches(ctx context.Context, opts ...CollectOption) iter.Seq2[*DataFrame, error]

CollectBatches streams the result without materialising the whole frame.

Breaking out of the loop tears the operator tree down, so an early exit is safe.

A panic while planning, or in any operator, is an error yielded like any other. A panic in the loop BODY is the caller's own and propagates as it would from any loop: recovering around yield is something Go forbids a range function to do.

func (*LazyFrame) CollectInto 

func (lf *LazyFrame) CollectInto[T any](ctx context.Context, opts ...CollectOption) (rows []T, err error)

CollectInto runs the query and decodes each row into a T.

A generic METHOD, which is the Go 1.27 feature this library was waiting for: before, this had to be a package-level function taking the frame as an argument.

func (*LazyFrame) CollectRecords added in v0.3.0 

func (lf *LazyFrame) CollectRecords(ctx context.Context, opts ...CollectOption) iter.Seq2[arrow.Record, error]

CollectRecords streams the result as Arrow records without materialising the whole frame, which is what makes a larger-than-RAM result exportable.

It is CollectBatches with the conversion applied, and it inherits every property: breaking out of the loop tears the operator tree down, so an early exit is safe. 

for rec, err := range lf.CollectRecords(ctx) {
    if err != nil { return err }
    if err := w.Write(rec); err != nil { rec.Release(); return err }
    rec.Release()
}

Each record shares the memory of the batch it came from, and the engine reuses nothing across batches, so a record stays valid after the next one arrives.

func (*LazyFrame) CollectSchema 

func (lf *LazyFrame) CollectSchema(ctx context.Context) (s *Schema, err error)

CollectSchema returns the result schema without reading any data.

It takes a context because resolving a source's schema is real I/O — a Parquet footer, a CSV sample. A schema request that could not be cancelled would be a hole in the cancellation story, which is why this is not the no-argument Schema() the design documents originally specified.

func (*LazyFrame) Columns added in v0.6.7 

func (lf *LazyFrame) Columns(ctx context.Context) ([]string, error)

Columns, DTypes and Width are the lazy frame's column names, their types, and how many there are, as Polars' LazyFrame properties: read from its schema, as CollectSchema resolves it, with no data read.

func (*LazyFrame) Concat 

func (lf *LazyFrame) Concat(others ...*LazyFrame) *LazyFrame

Concat stacks other frames beneath this one.

func (*LazyFrame) Count 

func (lf *LazyFrame) Count(ctx context.Context, opts ...CollectOption) (n int64, err error)

Count runs the query and returns its row count, retaining no data. It is not Polars' count(), which counts each column's non-null values; that is GroupBy().Agg(All().Count()).

func (*LazyFrame) DTypes added in v0.6.7 

func (lf *LazyFrame) DTypes(ctx context.Context) ([]DataType, error)

func (*LazyFrame) Describe added in v0.5.0 

func (lf *LazyFrame) Describe(ctx context.Context, percentiles ...float64) (*DataFrame, error)

Describe summarises each column, one statistic a row: Polars' describe. 

statistic   count, null_count, mean, std, min, 25%, 50%, 75%, max

A number's column is a Float64: its mean, its standard deviation (ddof 1), and its percentiles, each the nearest value, as Polars takes them. A Bool's is a Float64 too, with true as 1: its mean is the share of trues, and its min and max say whether there is a false and a true. Any other column is a String, holding its counts, and for a String or a temporal column its min and max, formatted as a Cast to String formats them. A statistic a column has no answer for is null.

percentiles replace 0.25, 0.5 and 0.75, each between 0 and 1. It runs the query once, through one aggregate per statistic and column, and is therefore eager.

func (*LazyFrame) Drop 

func (lf *LazyFrame) Drop(names ...string) *LazyFrame

Drop removes columns by name.

Sugar over `Select(Exclude(...))`, which is the whole implementation: Exclude is already an expansion-time selector, so Drop inherits its error messages, its interaction with projection pushdown, and its behaviour on a name that is not there — the selector ignores it rather than failing, matching Polars.

func (*LazyFrame) DropNans added in v0.6.7 

func (lf *LazyFrame) DropNans(subset ...string) *LazyFrame

DropNans drops the rows holding a NaN in any of the named columns, or in any float column when none is named: Polars' drop_nans. A null is not a NaN, and its row is kept.

A named column of integers holds no NaN, and drops nothing, as Polars'; a named column of another type, text, a Boolean, a date or a Decimal, is refused, as Polars refuses is_nan of one. It is a Filter, so it streams.

func (*LazyFrame) DropNulls 

func (lf *LazyFrame) DropNulls(subset ...string) *LazyFrame

DropNulls removes rows that are null in any of the named columns, or in any column at all when no names are given.

It is `Filter(Col(c).IsNotNull(), ...)`, and being a filter rather than a dedicated node is what earns it Parquet row-group pruning: the pruner claims IsNotNull over a bare column, so a row group whose statistics say a column is entirely null is skipped without being read.

Note the asymmetry with a hypothetical DropNans: IsNotNull is TOTAL, so the predicate is never itself null and the filter's own null rule never comes into play. `Col(c).IsNotNan()` is null on a null row, and Filter drops null predicates, so the obvious spelling of DropNans would silently drop nulls too. That is why it is not here.

func (*LazyFrame) Err 

func (lf *LazyFrame) Err() error

Err returns the first deferred error, or nil.

func (*LazyFrame) Explain 

func (lf *LazyFrame) Explain(ctx context.Context, opts ...ExplainOption) (out string, err error)

Explain renders the query plan.

The output is stable and diff-friendly on purpose: it is what golden tests snapshot, and a plan diff is the only thing that notices a query which still returns the right answer while reading forty columns instead of two.

func (*LazyFrame) Explode added in v0.3.0 

func (lf *LazyFrame) Explode(names ...string) *LazyFrame

Explode turns each row's list into one row per element, repeating every other column. 

id | tags        ->   id | tags
 1 | [10, 11]          1 | 10
 2 | []                1 | 11
 3 | null              2 | null
 4 | [12]              3 | null
                       4 | 12

An empty list and a null list each produce ONE row holding a null, rather than vanishing. Dropping them would silently lower the height — which is only visible to someone who checks it — and would make two columns disagree about how many rows a pair of empty lists gives. It is also what polars does.

The consequence is worth knowing: **explode erases the difference between an empty list and a null list.** A List column keeps those apart deliberately; after this they are the same row. The distinction belongs to the list, and the list is what explode consumes.

Several columns explode TOGETHER — zipped, not multiplied. Exploding twice in sequence would give the cross product instead, so the lists must be the same length in every row, and a row where they disagree is an error rather than a truncation.

Example

Split builds a List column; the .list namespace asks questions of each list, and Explode gives every element a row of its own. 

package main

import (
	"context"
	"fmt"
	"log"

	"github.com/advenn/ursus"
)

func main() {
	lf := ursus.Frame(
		ursus.Values("order", []int64{1, 2}),
		ursus.Values("items", []string{"apple,pear", "fig"}),
	).WithColumns(ursus.Col("items").Str().Split(","))

	df, err := lf.WithColumns(ursus.Col("items").List().Len().Alias("n")).
		Explode("items").Collect(context.Background())
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(df)
}

Output:
shape: (3, 3)
┌───────┬─────────┬────────┐
│ order │ items   │ n      │
│ Int64 │ String  │ Uint32 │
├───────┼─────────┼────────┤
│ 1     │ "apple" │ 2      │
│ 1     │ "pear"  │ 2      │
│ 2     │ "fig"   │ 1      │
└───────┴─────────┴────────┘

func (*LazyFrame) FillNan 

func (lf *LazyFrame) FillNan[T Operand](v T, subset ...string) *LazyFrame

FillNan replaces NaN in the named columns, or in every FLOAT column when no subset is given. Nulls are left alone — see Expr.FillNan.

func (*LazyFrame) FillNull 

func (lf *LazyFrame) FillNull[T Operand](v T, subset ...string) *LazyFrame

FillNull replaces nulls in the named columns, or in every column the value can fill when no subset is given. A FillStrategy fills by the strategy, as Polars' fill_null(strategy=) does, rather than with a value. 

lf.FillNull(0)                    // every numeric column
lf.FillNull(0, "qty", "amount")   // exactly these
lf.FillNull(ursus.FillForward)    // every column, from the value before

A strategy's columns

With no subset, FillForward and FillBackward fill every column; FillMin and FillMax every column of an ordered type; FillZero, FillOne and FillMean every numeric one, as the value 0 would. Polars applies each to every column, filling a String with "" and a Bool with false for FillZero, and refusing a String for FillOne and FillMean; here they leave those columns alone, as `lf.FillNull(0)` does.

A forward or backward fill takes a limit, Polars' fill_null(strategy="forward", limit=1): `lf.FillNull(ursus.FillForward.Limit(1))` fills at most one null after each value, in every column (step 301).

Until step 249 a strategy was taken for its value: FillStrategy is a uint8, so it satisfies Operand, and `lf.FillNull(ursus.FillForward)` filled every null with 2.

Why the no-subset form restricts rather than refuses

Almost every real frame has a string column, and `lf.FillNull(0)` erroring on it would make the shorthand useless exactly where it is most wanted. So the value's own type chooses the columns — the same argument GroupBy(k).Sum()'s doc makes for restricting to numeric columns, and with the same consequence: it is not a silent skip, because the selector is part of the expression and Explain shows precisely which columns were filled.

A named column whose type the value cannot fill is a plan-time error, because the user asked for it by name.

The value is a WEAK literal, so filling an Int32 column with 0 leaves it Int32.

func (*LazyFrame) Filter 

func (lf *LazyFrame) Filter(preds ...Expr) *LazyFrame

Filter keeps rows where every predicate is true.

Predicates are AND-ed. A row whose predicate is NULL is DROPPED, matching SQL's WHERE. One consequence worth knowing: Filter(p) and Filter(p.Not()) do not partition the input, because a row where p is null is dropped by both.

An aggregate in a predicate is over the whole frame, as in Polars: Filter(Col("x").Gt(Col("x").Mean())) keeps the rows above the mean (step 250).

Example

Filter keeps rows; Select keeps columns. Neither runs until Collect. 

package main

import (
	"context"
	"fmt"
	"log"

	"github.com/advenn/ursus"
)

// exampleSales is the little frame the examples below work on.
func exampleSales() *ursus.LazyFrame {
	return ursus.Frame(
		ursus.Values("region", []string{"north", "south", "north", "east", "south"}),
		ursus.Values("product", []string{"apple", "apple", "pear", "pear", "fig"}),
		ursus.Values("qty", []int64{10, 4, 7, 2, 9}),
		ursus.Values("price", []float64{1.50, 1.50, 2.25, 2.25, 3.00}),
	)
}

func main() {
	df, err := exampleSales().
		Filter(ursus.Col("qty").Gt(int64(5))).
		Select(ursus.Col("region"), ursus.Col("qty")).
		Collect(context.Background())
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(df)
}

Output:
shape: (3, 2)
┌─────────┬───────┐
│ region  │ qty   │
│ String  │ Int64 │
├─────────┼───────┤
│ "north" │ 10    │
│ "north" │ 7     │
│ "south" │ 9     │
└─────────┴───────┘

func (*LazyFrame) First added in v0.6.7 

func (lf *LazyFrame) First() *LazyFrame

First and Last are the frame's first and last row, none for no rows: Polars' LazyFrame.first() and last().

func (*LazyFrame) GatherEvery added in v0.6.7 

func (lf *LazyFrame) GatherEvery(n, offset int) *LazyFrame

GatherEvery is every nth row from offset: Polars' gather_every(n, offset). n must be positive. It is a filter by the row's index, so it streams.

func (*LazyFrame) GroupBy 

func (lf *LazyFrame) GroupBy(keys ...Expr) *GroupBy

GroupBy begins an aggregation. 

lf.GroupBy(ursus.Col("region")).
    Agg(ursus.Col("revenue").Sum(), ursus.Len().Alias("n"))

With no keys it is a GLOBAL aggregate producing exactly one row — including over an empty input, where `count(*)` is 0 rather than no rows at all.

Example

GroupBy takes the keys, Agg takes what to compute per group. 

package main

import (
	"context"
	"fmt"
	"log"

	"github.com/advenn/ursus"
)

// exampleSales is the little frame the examples below work on.
func exampleSales() *ursus.LazyFrame {
	return ursus.Frame(
		ursus.Values("region", []string{"north", "south", "north", "east", "south"}),
		ursus.Values("product", []string{"apple", "apple", "pear", "pear", "fig"}),
		ursus.Values("qty", []int64{10, 4, 7, 2, 9}),
		ursus.Values("price", []float64{1.50, 1.50, 2.25, 2.25, 3.00}),
	)
}

func main() {
	df, err := exampleSales().
		GroupBy(ursus.Col("region")).
		Agg(
			ursus.Col("qty").Sum().Alias("total_qty"),
			ursus.Col("price").Mean().Alias("avg_price"),
			ursus.Len().Alias("orders"),
		).
		Sort(ursus.Asc(ursus.Col("region"))).
		Collect(context.Background())
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(df)
}

Output:
shape: (3, 4)
┌─────────┬───────────┬───────────┬────────┐
│ region  │ total_qty │ avg_price │ orders │
│ String  │ Int128    │ Float64   │ Uint64 │
├─────────┼───────────┼───────────┼────────┤
│ "east"  │ 2         │ 2.25      │ 1      │
│ "north" │ 17        │ 1.875     │ 2      │
│ "south" │ 13        │ 2.25      │ 2      │
└─────────┴───────────┴───────────┴────────┘
Example (WholeFrame)

GroupBy with NO keys aggregates the whole frame as one group. polars infers this from a bare select; ursus asks for it, so it cannot be confused with a row-wise select that happens to return one row. 

package main

import (
	"context"
	"fmt"
	"log"

	"github.com/advenn/ursus"
)

// exampleSales is the little frame the examples below work on.
func exampleSales() *ursus.LazyFrame {
	return ursus.Frame(
		ursus.Values("region", []string{"north", "south", "north", "east", "south"}),
		ursus.Values("product", []string{"apple", "apple", "pear", "pear", "fig"}),
		ursus.Values("qty", []int64{10, 4, 7, 2, 9}),
		ursus.Values("price", []float64{1.50, 1.50, 2.25, 2.25, 3.00}),
	)
}

func main() {
	df, err := exampleSales().
		GroupBy().
		Agg(
			ursus.Len().Alias("rows"),
			ursus.Col("qty").Sum().Alias("total_qty"),
		).
		Collect(context.Background())
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(df)
}

Output:
shape: (1, 2)
┌────────┬───────────┐
│ rows   │ total_qty │
│ Uint64 │ Int128    │
├────────┼───────────┤
│ 5      │ 32        │
└────────┴───────────┘

func (*LazyFrame) GroupByDynamic 

func (lf *LazyFrame) GroupByDynamic(index Expr, o DynamicOptions) *GroupBy

GroupByDynamic groups rows into fixed temporal windows cut from a sorted index. 

lf.GroupByDynamic(ursus.Col("ts"), ursus.DynamicOptions{
        Every:   ursus.Every("1h"),
        GroupBy: []ursus.Expr{ursus.Col("service")},
    }).
    Agg(ursus.Len().Alias("n"))

It is not GroupBy(truncate(ts, every)), and the difference is empty windows

A hash group-by creates a group when a row arrives, so an hour with no rows simply is not in the output and a gap in a time series is invisible. A dynamic group generates the whole grid between the first and last instant, so the gap is a row with a zero in it. That is the entire reason this operator exists.

The index must be sorted, and it is CHECKED

Windows are cut as contiguous ranges of a sorted index, so an unsorted one gives wrong groups. ursus verifies rather than trusting an assertion: an unchecked SetSorted-style hint is a user claim that deletes a correctness check, and its failure mode is wrong rows with no error. Sort first if the check refuses.

An integer index counts by "i"

The index is a Date or a Datetime, or an Int32 or an Int64 whose Every, Period and Offset count it in Polars' integer unit: Every("3i") cuts windows of three index values, 0, 3, 6, and the index keeps its type (step 312). An interval of time on an integer, or "i" on a date, is refused, as Polars refuses both.

func (*LazyFrame) HStack 

func (lf *LazyFrame) HStack(others ...*LazyFrame) *LazyFrame

HStack places frames side by side: the same rows, with the columns concatenated.

It buffers, and Concat does not

Nothing in the engine aligns batch boundaries across independent pipelines — one frame may deliver 8192 rows at a time while another delivers 100 — and pairing row i of each means having row i of each in hand. So every input is materialised. Concat, which appends rows rather than pairing them, streams.

Every frame must have the same height, and column names must not collide. There is no automatic suffixing: a join suffixes because it has a principled left and right to name the suffix after, and these inputs are peers.

func (*LazyFrame) Head 

func (lf *LazyFrame) Head(n int) *LazyFrame

Head keeps at most n rows.

func (*LazyFrame) InsertColumn added in v0.6.7 

func (lf *LazyFrame) InsertColumn[C Expr | *Column](i int, c C) *LazyFrame

InsertColumn puts a column at index i, an expression computed over the frame or Go values, as Polars' DataFrame.insert_column takes an Expr or a Series. The columns from i on move one place right: i of the frame's width appends, and a negative i counts from the width, -1 the place before the last column, as Polars counts it. 

lf.InsertColumn(0, ursus.Col("qty").Mul(ursus.Col("price")).Alias("revenue"))
lf.InsertColumn(-1, ursus.Values("note", notes))

An index past either end is refused, as Polars refuses it, and so is a name the frame already has. An expression is one column, and an aggregate or a literal is broadcast to every row, as in WithColumns. Go values are as many as the frame's rows, or the query fails, as HStack's does; their frame is held, as HStack holds it.

func (*LazyFrame) Interpolate added in v0.6.7 

func (lf *LazyFrame) Interpolate() *LazyFrame

Interpolate fills each run of nulls in a column between two values with the points on the straight line between them, by row, every column it can: Polars' interpolate. A run before a column's first value or after its last stays null, and NaN is a value, so a line to one is NaN.

Numbers and instants are interpolated, as Polars interpolates them: an integer, an Int128 too, answers a Float64, a float keeps its width, and a date, a datetime, a duration or a time keeps its type, a point between two days truncated to the earlier. A Decimal answers a Float64, as Expr.Interpolate's does, where Polars' keeps the Decimal. Text, Booleans, lists and structs are left as they are.

Each column is Expr.Interpolate, an ordered window over the whole frame, so it holds its input, and past a memory limit spills it.

func (*LazyFrame) IterRows added in v0.6.7 

func (lf *LazyFrame) IterRows[T any](ctx context.Context, opts ...CollectOption) iter.Seq2[T, error]

IterRows streams the query's rows, each a T as Rows decodes it, a batch at a time, as Polars' iter_rows(named=True) walks a frame's: the query runs as CollectBatches runs it, holding a batch, so a result larger than memory can be read row by row. An error ends the walk, yielded with T's zero value. 

for row, err := range lf.IterRows[sale](ctx) { ... }

func (*LazyFrame) Join 

func (lf *LazyFrame) Join(other *LazyFrame, opts ...JoinOption) *LazyFrame

Join combines this frame with another. 

enriched := orders.Join(customers,
    ursus.JoinOn(ursus.Col("customer_id")),
    ursus.JoinHow(ursus.JoinLeft),
)

Column names that appear on both sides get the right one suffixed; the join key appears once unless the kind is a full join. See JoinCoalesce and JoinSuffix.

An inner join's row order is not promised: ursus hashes whichever input it estimates is smaller, and the rows follow the other. JoinMaintainOrder keeps an order, Polars' maintain_order.

Example

Joins take the key columns and, optionally, the kind. Inner is the default. 

package main

import (
	"context"
	"fmt"
	"log"

	"github.com/advenn/ursus"
)

// exampleSales is the little frame the examples below work on.
func exampleSales() *ursus.LazyFrame {
	return ursus.Frame(
		ursus.Values("region", []string{"north", "south", "north", "east", "south"}),
		ursus.Values("product", []string{"apple", "apple", "pear", "pear", "fig"}),
		ursus.Values("qty", []int64{10, 4, 7, 2, 9}),
		ursus.Values("price", []float64{1.50, 1.50, 2.25, 2.25, 3.00}),
	)
}

func main() {
	regions := ursus.Frame(
		ursus.Values("region", []string{"north", "south", "east"}),
		ursus.Values("manager", []string{"ada", "bob", "cy"}),
	)
	df, err := exampleSales().
		Join(regions, ursus.JoinOn(ursus.Col("region"))).
		Select(ursus.Col("product"), ursus.Col("manager")).
		Collect(context.Background())
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(df)
}

Output:
shape: (5, 2)
┌─────────┬─────────┐
│ product │ manager │
│ String  │ String  │
├─────────┼─────────┤
│ "apple" │ "ada"   │
│ "apple" │ "bob"   │
│ "pear"  │ "ada"   │
│ "pear"  │ "cy"    │
│ "fig"   │ "bob"   │
└─────────┴─────────┘

func (*LazyFrame) JoinAsOf 

func (lf *LazyFrame) JoinAsOf(other *LazyFrame, opts ...AsOfOption) *LazyFrame

JoinAsOf matches each left row to the NEAREST right row rather than an equal one. 

trades.JoinAsOf(quotes,
    ursus.AsOfOn(ursus.Col("ts")),
    ursus.AsOfBy(ursus.Col("symbol")),
    ursus.AsOfTolerance(ursus.Every("1m")),
)

It is a LEFT join

Every left row survives; the right columns are null when nothing is near enough. So the output height always equals the left height, tolerance and strategy only decide which rows come back populated.

Both sides must be sorted on the as-of key, and it is CHECKED

The search is a binary search over a sorted run. ursus verifies rather than trusting an assertion — an unchecked sortedness hint is a user claim that deletes a correctness check, and its failure mode is wrong matches with no error. Sort first if the check refuses.

Memory

Every strategy streams both sides. The default, backward, holds a row a by-key and a few right batches, however long the right side is (step 183). Forward and nearest hold the right rows read ahead to each left key's next candidate (step 194): the gap to a by-key's next row, or the whole right side for a left by-key it never holds.

func (*LazyFrame) JoinWhere added in v0.3.0 

func (lf *LazyFrame) JoinWhere(other *LazyFrame, preds ...Expr) *LazyFrame

JoinWhere joins every pair of rows that satisfies the predicates. 

sessions.JoinWhere(events,
    ursus.Col("ts").Ge(ursus.Col("start")),
    ursus.Col("ts").Lt(ursus.Col("end")),
)

This is the non-equi join: the predicates may be any boolean expression over both sides, and are AND-ed. Several predicates are the usual case, because an interval match takes two.

Which names the predicates use

The join's OUTPUT names, which is to say left columns by their own name and a colliding right column with the suffix — `k` and `k_right`. There is no separate left/right namespace to qualify with, and the same rule applies here as to a Filter written after any other join. JoinWhereSuffix takes another suffix.

What it costs

A pair of rows for every pair that satisfies the predicates, which can be very large — that is the nature of the operation and not a defect. When at least one predicate is an EQUALITY between the two sides, the optimizer turns it into a hash join key and evaluates the rest per matching pair; without one, every pair is tested. Explain shows which happened.

At least one predicate is required. A JoinWhere with none is a cartesian product written as though it were a filtered one, and the cardinality difference is |L| vs |L|x|R| — so it is an error rather than a silent cross join, for the same reason a keyless equi-join is.

Example

A join on a condition rather than equal keys: each sale with the price band it falls in. 

package main

import (
	"context"
	"fmt"
	"log"

	"github.com/advenn/ursus"
)

// exampleSales is the little frame the examples below work on.
func exampleSales() *ursus.LazyFrame {
	return ursus.Frame(
		ursus.Values("region", []string{"north", "south", "north", "east", "south"}),
		ursus.Values("product", []string{"apple", "apple", "pear", "pear", "fig"}),
		ursus.Values("qty", []int64{10, 4, 7, 2, 9}),
		ursus.Values("price", []float64{1.50, 1.50, 2.25, 2.25, 3.00}),
	)
}

func main() {
	bands := ursus.Frame(
		ursus.Values("band", []string{"cheap", "dear"}),
		ursus.Values("lo", []float64{0, 2}),
		ursus.Values("hi", []float64{2, 10}),
	)
	df, err := exampleSales().Select(ursus.Col("product"), ursus.Col("price")).
		JoinWhere(bands,
			ursus.Col("price").Ge(ursus.Col("lo")),
			ursus.Col("price").Lt(ursus.Col("hi"))).
		Select(ursus.Col("product"), ursus.Col("price"), ursus.Col("band")).
		Collect(context.Background())
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(df)
}

Output:
shape: (5, 3)
┌─────────┬─────────┬─────────┐
│ product │ price   │ band    │
│ String  │ Float64 │ String  │
├─────────┼─────────┼─────────┤
│ "apple" │ 1.5     │ "cheap" │
│ "apple" │ 1.5     │ "cheap" │
│ "pear"  │ 2.25    │ "dear"  │
│ "pear"  │ 2.25    │ "dear"  │
│ "fig"   │ 3       │ "dear"  │
└─────────┴─────────┴─────────┘

func (*LazyFrame) JoinWhereSuffix added in v0.6.7 

func (lf *LazyFrame) JoinWhereSuffix(other *LazyFrame, suffix string, preds ...Expr) *LazyFrame

JoinWhereSuffix is JoinWhere with suffix appended to a right column whose name the left frame has, where JoinWhere appends "_right": Polars' join_where(suffix=) (step 315). The predicates name such a column by it. 

a.JoinWhereSuffix(b, "_b", ursus.Col("x").Lt(ursus.Col("x_b")))

func (*LazyFrame) Last added in v0.6.7 

func (lf *LazyFrame) Last() *LazyFrame

func (*LazyFrame) Limit 

func (lf *LazyFrame) Limit(n int) *LazyFrame

Limit is an alias for Head.

func (*LazyFrame) MatchToSchema added in v0.6.7 

func (lf *LazyFrame) MatchToSchema(schema *Schema, o MatchOptions) *LazyFrame

MatchToSchema makes the frame's columns the schema's, in its order and of its types: Polars' match_to_schema. 

lf.MatchToSchema(ursus.MustSchema(
	ursus.Of("id", ursus.Int64),
	ursus.Of("amount", ursus.Float64),
), ursus.MatchOptions{IntegerCast: ursus.MatchUpcast, ExtraColumns: ursus.MatchIgnore})

A column of the frame's own type is read as it is, and a frame that already is the schema is returned unchanged. A Null column meets any type. Only types are met: a schema field's nullability is not checked.

func (*LazyFrame) Max added in v0.6.7 

func (lf *LazyFrame) Max() *LazyFrame

Max is each column's greatest value, one row: Polars' DataFrame.max.

func (*LazyFrame) Mean added in v0.6.7 

func (lf *LazyFrame) Mean() *LazyFrame

Mean is each column's mean, one row: Polars' DataFrame.mean. A Boolean's is the share of its trues, and a Date's a Datetime(us), as Polars'.

func (*LazyFrame) Median added in v0.6.7 

func (lf *LazyFrame) Median() *LazyFrame

Median is each column's median, one row: Polars' DataFrame.median.

func (*LazyFrame) MergeSorted 

func (lf *LazyFrame) MergeSorted(other *LazyFrame, key string) *LazyFrame

MergeSorted interleaves this frame with another that is sorted on the same key, keeping the result sorted.

Not a concat: a concat appends, so the result is ordered only if the second frame begins after the first ends. Not a join either — nothing is matched and no row is dropped, so the output height is always the sum of the two.

Both schemas must match EXACTLY and both inputs must be sorted on key, which is checked rather than assumed, as the rows arrive: it streams, a batch of each side at a time, so its memory does not grow with the inputs (step 182).

func (*LazyFrame) Min added in v0.6.7 

func (lf *LazyFrame) Min() *LazyFrame

Min is each column's least value, one row: Polars' DataFrame.min. A List or a Struct is null.

func (*LazyFrame) NullCount added in v0.6.7 

func (lf *LazyFrame) NullCount() *LazyFrame

NullCount is each column's count of nulls, one row: Polars' DataFrame.null_count.

func (*LazyFrame) Pipe 

func (lf *LazyFrame) Pipe(fn func(*LazyFrame) *LazyFrame) *LazyFrame

Pipe applies fn, for composing reusable query fragments.

func (*LazyFrame) Pivot added in v0.6.7 

func (lf *LazyFrame) Pivot(o PivotOptions) *LazyFrame

Pivot spreads a frame wide, as Polars' LazyFrame.pivot does: a row for each distinct Index, in the order each first appears, a null among them; and, for each Values column, a column for each row of OnColumns, holding Agg of that column's values where the row's Index and the On values meet. A new column is named after its On value, "null" for a null; after several, {"x","p"}, text quoted; and, when Values is several, after the value column, Separator and that. 

wide := lf.Pivot(ursus.PivotOptions{On: []string{"quarter"}, Index: []string{"region"},
	Values: []string{"sales"}, Agg: ursus.Expr.Sum,
	OnColumns: []*ursus.Column{ursus.Values("quarter", []string{"Q1", "Q2"})}})

The new columns are given, as Polars' on_columns are: a plan's schema is known before it runs. DataFrame.Pivot finds them. A group-by of Index and On reduces the rows, streaming the input and spilling past a memory limit; the pivot then holds a row for each index and On values that meet.

func (*LazyFrame) Product added in v0.6.7 

func (lf *LazyFrame) Product() *LazyFrame

Product is each column's product, one row: Polars' DataFrame.product. A Boolean's is of its 0s and 1s. A column that is not a number or a Boolean is a null of the Null type, as Polars answers it.

func (*LazyFrame) Profile added in v0.6.8 

func (lf *LazyFrame) Profile(ctx context.Context, opts ...CollectOption) (df, profile *DataFrame, err error)

Profile runs the query as Collect does, and answers its result and a frame of how each node of its plan ran: Polars' LazyFrame.profile (step 369).

The profile has a row a node, in the order Explain prints the plan, after a first row, "optimization", for the time the plan took to resolve and optimize:

  • node: the node as Explain names it, indented two spaces a level;
  • start, end: microseconds since the query began, of the node's first work and its last, as Polars' start and end; null for a node that did none;
  • rows, batches: what the node handed on;
  • own: microseconds of the node's own work, summed over the threads that did it. A batch transform's, a filter's or a projection's, is its time; an operator's, a scan's, a sort's or a join's, its time less what it waited on its inputs. Where an operator works on threads beside its inputs, as a parallel aggregation does, only what it did beyond that wait is its own;
  • peak_memory: the most its operators held at once, in bytes, as the memory budget counts them; 0 for a node that holds nothing across batches.

A profiled query runs each operator in a wrapper that reads the clock, so it takes a little longer; Collect builds none.

func (*LazyFrame) Quantile added in v0.6.7 

func (lf *LazyFrame) Quantile(q float64, interp Interpolation) *LazyFrame

Quantile is each column's quantile q, one row: Polars' DataFrame.quantile(q, interpolation). An instant's, a duration's and a time's is one of them; a Boolean's is null, as Polars'.

func (*LazyFrame) Remove 

func (lf *LazyFrame) Remove(preds ...Expr) *LazyFrame

Remove is the inverse of Filter: it drops rows where ANY predicate is true.

Spelled out because the alternative reading is equally plausible: this is NOT(p1 OR p2 OR ...), i.e. a row survives only if every predicate is false. Rows where a predicate is null are dropped, same as Filter.

func (*LazyFrame) Rename 

func (lf *LazyFrame) Rename(names map[string]string) *LazyFrame

Rename changes column names, leaving everything else in place.

Sugar over `Select(All().MapName(...))`. MapName is the expansion-safe renamer — Alias sets ONE fixed name and so cannot apply to a multi-column selection, which is exactly why the two are different methods.

Names not present in the frame are ignored rather than an error, so a rename map written against a wider schema still works. Renaming two columns onto the same name IS an error, caught by Select's output-name uniqueness check with both offending expressions named.

func (*LazyFrame) Reverse 

func (lf *LazyFrame) Reverse() *LazyFrame

Reverse emits rows in the opposite order.

A full pipeline breaker — the last input row is the first output row — so it holds the whole frame. It does no comparison, which is what makes it cheaper than sorting by a synthesised descending index.

func (*LazyFrame) Rolling 

func (lf *LazyFrame) Rolling(index Expr, o RollingOptions) *GroupBy

Rolling gives every row its own window, reaching back Period from that row's own instant. The output has one row per input row.

Where GroupByDynamic answers "how many per hour", Rolling answers "how many in the hour before each event"; with an Offset, the hour after it, or any span the options place. The index must be sorted, and is checked. An integer index counts by "i", as GroupByDynamic's does, and may be a UInt32 or a UInt64 too, as Polars' may.

func (*LazyFrame) SampleFraction added in v0.6.7 

func (lf *LazyFrame) SampleFraction(fraction float64, withReplacement, shuffle bool, seed uint64) *LazyFrame

SampleFraction is SampleN of a fraction of the rows, truncated, as Polars counts it: 0.4 of six rows is two.

func (*LazyFrame) SampleN added in v0.6.7 

func (lf *LazyFrame) SampleN(n int, withReplacement, shuffle bool, seed uint64) *LazyFrame

SampleN is n of the frame's rows drawn, every column's values of the same rows: Polars' DataFrame.sample(n, with_replacement, shuffle, seed). Without replacement more rows than the frame has are refused, as Polars refuses them. Unshuffled, the rows keep their order, where Polars' unshuffled draw comes in its generator's order. The draw is ursus's generator's, seeded by seed and the frame's height, so the rows are not Polars' (v0.6.6-scope.md §6.3), and one seed draws the same rows of the same frame each time.

func (*LazyFrame) Select 

func (lf *LazyFrame) Select(exprs ...Expr) *LazyFrame

Select computes a new set of columns, replacing the frame's columns entirely.

Expressions may expand: Select(All()) keeps everything, and Select(ColDType(Float64).Suffix("_f")) selects and renames every float column.

An aggregate is a scalar

As in Polars, an aggregate here is one value broadcast to every row: Select(Col("x"), Col("x").Sum()) gives each row the total. A Select whose every expression aggregates is one row, GroupBy().Agg(...) with no keys: Select(Col("x").Sum(), Col("x").Mean()). The broadcast is the aggregate over a window with no partition, .Over(), which streams past a memory limit (step 250).

A column of another length

A reshape or an Expr.Filter is a column of its own length, as in Polars (step 258): Select(Col("a").Sort(), Col("b").Reverse()) sorts a and reverses b each alone, and Select(Col("a").Head(3), Col("b").Sum()) is three rows. Columns of two lengths are refused when the query runs. Each is a plan of its own over the frame, so it spills where the frame-level Sort, Unique and the rest do (zipselect.go).

func (*LazyFrame) SelectSeq added in v0.6.7 

func (lf *LazyFrame) SelectSeq(exprs ...Expr) *LazyFrame

func (*LazyFrame) Shift added in v0.6.7 

func (lf *LazyFrame) Shift(n int) *LazyFrame

Shift moves every column's values n rows later, a negative n earlier, nulls where the rows were vacated: Polars' DataFrame.shift(n). It is Expr.Shift of each column, a window function, so it streams.

func (*LazyFrame) ShiftFill added in v0.6.7 

func (lf *LazyFrame) ShiftFill[T Operand](n int, fill T) *LazyFrame

ShiftFill is Shift with fill in the vacated rows: Polars' shift(n, fill_value). The fill meets each column as Expr.ShiftFill's does, a Go value of a type that fits a column's taking the column's type, so 0 into an Int32 column is an Int32 and 1.5 into an Int64 one makes it a Float64. A text column takes the fill's text, "0", as Polars' does. A column the fill cannot meet, a number column with a text fill, is refused naming it, where Polars makes the column text.

func (*LazyFrame) Show added in v0.6.7 

func (lf *LazyFrame) Show(ctx context.Context, limit int) (string, error)

Show is the query's first limit rows as a table, as DataFrame's Show draws them, Polars' LazyFrame.show: only those rows are computed, unless limit is negative.

func (*LazyFrame) Shuffle added in v0.6.7 

func (lf *LazyFrame) Shuffle(seed uint64) *LazyFrame

Shuffle is every row in an order drawn by seed: Polars' sample(fraction=1, shuffle=True).

func (*LazyFrame) SinkCSV 

func (lf *LazyFrame) SinkCSV(ctx context.Context, path string, opts ...CSVSinkOption) error

SinkCSV runs the query and writes the result to path, streaming.

This is the half that makes "larger than RAM" true end to end. Collect holds the whole result; this holds one batch, so a query that reads more data than fits in memory and writes more data than fits in memory works.

The file is written to a temporary name and renamed on success, so a failed query leaves no half-written file where a complete one is expected.

With WithPartitionBy or WithMaxRowsPerFile, path is a directory, and the result a hive dataset below it, of files named .csv, as Polars names them, compressed or not.

func (*LazyFrame) SinkIPC added in v0.6.7 

func (lf *LazyFrame) SinkIPC(ctx context.Context, path string, opts ...IPCSinkOption) error

SinkIPC runs the query and writes the answer to path as an Arrow IPC file, streaming, as Polars' sink_ipc: to a temporary name, renamed on success, since a file without its footer cannot be read.

With WithPartitionBy or WithMaxRowsPerFile, path is a directory, and the result a hive dataset below it, of files named .ipc, as Polars names them.

func (*LazyFrame) SinkNDJSON added in v0.6.7 

func (lf *LazyFrame) SinkNDJSON(ctx context.Context, path string, opts ...NDJSONSinkOption) error

SinkNDJSON runs the query and writes it to path as WriteNDJSON writes, streaming, as Polars' sink_ndjson: to a temporary name, renamed on success, so a failed query leaves no file half written.

With WithPartitionBy or WithMaxRowsPerFile, path is a directory, and the result a hive dataset below it, of files named .jsonl, as Polars names them.

func (*LazyFrame) SinkParquet 

func (lf *LazyFrame) SinkParquet(ctx context.Context, path string, opts ...ParquetSinkOption) error

SinkParquet runs the query and writes the result to path, streaming.

Memory is one row group rather than the whole result, so this is the write half of "larger than RAM". The file is written to a temporary name and renamed on success: a Parquet file without its footer is not merely truncated, it is unreadable, so leaving a partial one where a complete one is expected would be worse than leaving nothing.

With WithPartitionBy or WithMaxRowsPerFile, path is a directory, and the result a hive dataset below it, each file written to a temporary name and renamed as it ends.

func (*LazyFrame) Slice 

func (lf *LazyFrame) Slice(offset, length int) *LazyFrame

Slice keeps length rows starting at offset. A negative length means "to the end", which is how you drop a prefix without knowing the height.

A negative offset counts from the end, as Polars' does (step 291): the rows [height+offset, height+offset+length), clipped to the frame. Slice(-3, 2) of six rows is rows 3 and 4; Slice(-7, 2) is row 0 alone, the window starting a row before the frame; Slice(-10, 2) is no row.

Streaming: from the start it counts rows past and stops, so it never holds more than one batch; from the end it holds the last -offset rows, as Tail does.

func (*LazyFrame) Sort 

func (lf *LazyFrame) Sort(keys ...SortKey) *LazyFrame

func (*LazyFrame) Std added in v0.6.7 

func (lf *LazyFrame) Std(ddof int) *LazyFrame

Std is each column's standard deviation with ddof, one row: Polars' DataFrame.std(ddof). A column that is not a number or a Boolean is null, an instant's too, as Polars'.

func (*LazyFrame) Sum added in v0.6.7 

func (lf *LazyFrame) Sum() *LazyFrame

Sum is each column's sum, one row: Polars' DataFrame.sum. A column that is not a number, a Boolean or a duration is null.

func (*LazyFrame) Tail 

func (lf *LazyFrame) Tail(n int) *LazyFrame

Tail keeps the last n rows.

Unlike Head it cannot stream to completion: which rows are the last n is unknown until the input ends, so it holds a ring of the most recent n. Bounded — O(n), not O(input) — but not free.

func (*LazyFrame) TopK 

func (lf *LazyFrame) TopK(k int, by ...SortKey) *LazyFrame

TopK keeps the k LARGEST rows by the given keys, and BottomK the k smallest.

Both are sugar, and that is the point

Each is `Sort(...).Head(k)`, which the limit-pushdown rule turns into a bounded top-k: kernel.ArgTopK is O(n log k) time and O(k) memory against a full sort's O(n log n) and O(n), and its contract is that it returns EXACTLY the indices ArgSort would, ties included. So there is no separate operator to keep in step with Sort, and `TopK(k)` and `Sort(...).Head(k)` cannot drift apart because they are the same plan.

Both place nulls LAST, unlike Sort

Sort keeps direction and null placement orthogonal on purpose — "`.Desc()` silently relocating the nulls surprises people every time" — and that argument does NOT transfer here. In a sort, placement is cosmetic: every row comes back either way. In a top-k it is SELECTION, and a null has no rank, so leaving the default (nulls first) would make `TopK(2)` over [3,1,4,1,5,null] return the null and the 5 — a row with no value in the very column being ranked.

So both force nulls last, and both say so. `Sort(...).Head(k)` remains available for a different placement, and it is one call away.

func (*LazyFrame) Unique 

func (lf *LazyFrame) Unique(subset ...string) *LazyFrame

Unique removes duplicate rows.

With no arguments it compares whole rows. With column names it compares only those, keeping the FIRST row for each distinct combination — so the other columns come from that row rather than being chosen arbitrarily. UniqueKeep keeps the last, or only the rows whose key is not repeated.

func (*LazyFrame) UniqueKeep added in v0.6.7 

func (lf *LazyFrame) UniqueKeep(keep KeepStrategy, subset ...string) *LazyFrame

UniqueKeep removes duplicate rows as Unique does, keeping the row keep says of each key: Polars' unique(subset, keep, maintain_order=True) (step 289). The kept rows are in their own order, so KeepLast of a key whose rows are 0 and 2 keeps row 2 where row 2 was. With no subset a key is a whole row.

Keys are compared as Unique and GroupBy compare them: a null with a null, NaN with NaN, and -0 with 0, as Polars does.

KeepFirst and KeepAny are Unique, which streams, emitting a key's first row as it meets it. KeepLast and KeepNone cannot know a row is kept until its key's rows have all been read, so they are a filter by a window over the keys: a key's last row is the one a reversed CumCount numbers 1, and a key seen once the one whose partition's Len is 1. The window spills its partitions past a memory limit, as Over does.

func (*LazyFrame) Unnest added in v0.3.0 

func (lf *LazyFrame) Unnest(names ...string) *LazyFrame

Unnest replaces a struct column with its fields, in place. 

id | person                  ->   id | age  | city
 1 | {age: 30, city: "NY"}         1 | 30   | "NY"
 2 | {age: null, city: null}       2 | null | null
 3 | null                          3 | null | null

It is to a struct what Explode is to a list: afterwards the nested column is gone and the fields are ordinary columns, so every filter, aggregate, join and sort applies to them directly rather than through `.Struct().Field(...)`.

The fields land where the struct was and keep their OWN names. A name that would collide with a column already in the frame is refused, naming both — renaming a field the caller cannot see yet is not something this can guess at.

**Unnest erases the difference between a null struct and a struct of nulls**, the way Explode erases the difference between a null list and an empty one (rows 2 and 3 above). The distinction belongs to the struct, and this is what consumes it — `Col("person").IsNull()` is how to keep it, before unnesting.

At least one column must be named. "Every struct column" is well defined here, unlike for Explode, but it would silently widen the frame the day a file grows a struct.

func (*LazyFrame) Unpivot added in v0.3.0 

func (lf *LazyFrame) Unpivot(opts UnpivotOptions) *LazyFrame

Unpivot turns several value columns into two: which column, and what it held. 

sales.Unpivot(ursus.UnpivotOptions{On: []string{"jan", "feb"}})

id  jan  feb            id  variable  value
1   10   20     ->      1   jan       10
2   30   40             1   feb       20
                        2   jan       30
                        2   feb       40

Known elsewhere as melt. Every melted column shares one value column, so their types must promote to a common one — an Int32 beside an Int64 gives an Int64, and two types with no common one is an error naming both.

The row order is not Polars'

Polars emits every row's first variable, then every row's second. This emits every variable for one row, then the next row.

The reason is that ursus streams: unpivot transforms one batch at a time, so Polars' order would interleave differently at different batch sizes — the row order would depend on how the query ran rather than on what it asked for. Sort afterwards if a particular order matters.

Each of On and Index defaults to the other's rest

Leaving On empty melts every column that is not in Index, as Polars' on=None does (step 304): with Index empty too, every column. Leaving Index empty keeps every column that is not melted, so adding a column to the source keeps it, where Polars keeps none. Index naming every column melts nothing: no rows, and a value column of the Null type, as Polars answers.

On is written out when Unpivot is called, against the frame's schema then, so the plan names what it melts. Which columns those are is then the source's rather than the query's, the reason Unnest, which consumes its columns, still refuses to default them.

func (*LazyFrame) Update added in v0.6.7 

func (lf *LazyFrame) Update(other *LazyFrame, o UpdateOptions) *LazyFrame

Update writes other's values over this frame's, in the columns both have, matched by keys, as Polars' DataFrame.update does: 

prices.Update(corrections, ursus.UpdateOptions{On: []string{"sku"}})

A null in other leaves the value unless IncludeNulls. A column of other the frame does not have is ignored, and the frame's columns keep their places. A column whose two types differ takes the type that holds both, Polars' supertype: an Int64 updated with a Float64 is a Float64, and with text, text. A key other holds twice gives its frame row twice, as a join does.

It is a join, ordered, and a column each: it streams and spills as a join does, and keeps its order under a memory limit.

func (*LazyFrame) Upsample added in v0.6.7 

func (lf *LazyFrame) Upsample[T Span](time string, every T, groupBy ...string) *LazyFrame

Upsample puts a row at every step of every from each group's first instant to its last, as Polars' upsample does. A row of the frame at an instant of that grid is kept, and where there is none, a row of nulls but the time and the group's keys. The time column is a Date or a Datetime, and every an interval as Every counts it: "1mo" steps by months from the first instant, Jan 31, Feb 29, Mar 31, as Polars does; a day in a zone is a day of its wall clock. An integer index, an Int32, an Int64, a UInt32 or a UInt64, steps by Polars' integer interval, "2i". 

daily := lf.Upsample("time", ursus.Every("1d"), "station").
	WithColumns(ursus.Col("reading").FillNull(ursus.FillForward))

A row whose instant falls between two steps is left out, as Polars leaves it, and two rows at one instant are both kept. The groups come in the order each first appears, as Polars' with maintain_order, and each group's rows in the grid's order, two at one instant in the frame's: with a memory limit or without, so that a fill forward after it reads the rows it should.

It is a join: a grid made by a group-by of each group's first and last instant, joined with the frame and sorted back into the grid's order, each part streaming and spilling as it does. The frame need not be sorted, which Polars' must be. A row of a null instant has no place on the grid and is left out, and a frame of no rows upsamples to none, where Polars refuses either.

func (*LazyFrame) VStack 

func (lf *LazyFrame) VStack(other *LazyFrame) *LazyFrame

VStack is Concat for exactly two frames, named for the operation people look for.

It is not cheaper than Concat. Polars documents vstack as "cheap (adds a chunk)", which relies on a chunked column layout ursus does not have — a Column here is one contiguous run, so stacking copies. Naming it the same and pretending otherwise would be the misleading part.

func (*LazyFrame) ValueCounts added in v0.5.0 

func (lf *LazyFrame) ValueCounts(by ...Expr) *LazyFrame

ValueCounts counts the rows of each distinct value of by, most frequent first, in a column named "count": Polars' value_counts, sorted. 

lf.ValueCounts(Col("city"))                  // city, count
lf.ValueCounts(Col("city"), Col("year"))     // each combination

Ties keep the order in which their values first appear, and a null is a value like any other, as a group-by's key. It is GroupBy(by...).MaintainOrder(), Len, and a stable Sort by count, descending. A key named "count" is refused, as Polars refuses it; alias it first.

func (*LazyFrame) Var added in v0.6.7 

func (lf *LazyFrame) Var(ddof int) *LazyFrame

Var is each column's variance with ddof, one row: Polars' DataFrame.var(ddof).

func (*LazyFrame) WhereExists added in v0.3.0 

func (lf *LazyFrame) WhereExists(other *LazyFrame, preds ...Expr) *LazyFrame

WhereExists keeps the rows of this frame that have at least one row in other satisfying every predicate. 

orders.WhereExists(lineitem,
    ursus.Col("o_orderkey").Eq(ursus.Col("l_orderkey")),
    ursus.Col("l_quantity").Gt(40.0),
)

It is SQL's `EXISTS (SELECT ... WHERE ...)`, and the output is this frame: same columns, same order, fewer rows. No column of other appears in the result, which is why this is not a join you can write as one — see below.

Why it is not JoinWhere plus a filter

JoinWhere pairs rows, so filtering its output filters pairs. This asks whether a satisfying partner EXISTS, which is a question about the left row, and a left row with three partners must come back once rather than three times. The predicate is therefore evaluated before the verdict, inside the join, and no combination of JoinWhere and Filter expresses that without a deduplication.

Which names the predicates use

The names a JOIN of the two frames would produce: this frame's columns by their own name, and a colliding column of other with the suffix — `k` and `k_right`. The output has no such column, but the predicate is evaluated on pairs, so it can name both.

What it costs

An equality between the two sides becomes a hash join key and the rest is tested per matching pair; with no equality among the predicates, every pair is tested. Explain shows which. A partner is found by scanning the candidates, so this stops at the first satisfying one — WhereNotExists cannot.

func (*LazyFrame) WhereNotExists added in v0.3.0 

func (lf *LazyFrame) WhereNotExists(other *LazyFrame, preds ...Expr) *LazyFrame

WhereNotExists keeps the rows of this frame that have NO row in other satisfying every predicate. 

// customers who never ordered anything large
customers.WhereNotExists(orders,
    ursus.Col("c_custkey").Eq(ursus.Col("o_custkey")),
    ursus.Col("o_totalprice").Gt(1000.0),
)

SQL's `NOT EXISTS`, and the exact complement of WhereExists: for any frame and any predicates, the two partition this frame's rows.

It is strictly more work than WhereExists on the same inputs. Finding one satisfying partner ends the search; proving there is none means examining every candidate.

func (*LazyFrame) Width added in v0.6.7 

func (lf *LazyFrame) Width(ctx context.Context) (int, error)

func (*LazyFrame) WithColumns 

func (lf *LazyFrame) WithColumns(exprs ...Expr) *LazyFrame

WithColumns adds columns, replacing any that already exist BY NAME and keeping their original position.

Later expressions can reference columns that earlier ones added, so `WithColumns(a.Alias("x"), Col("x").Mul(2))` works.

An aggregate is the whole frame's, broadcast to every row, as in Polars: WithColumns(Col("x").Sub(Col("x").Mean())) centres x (step 250).

Example

Expressions compose: a computed column is just another expression. 

package main

import (
	"context"
	"fmt"
	"log"

	"github.com/advenn/ursus"
)

// exampleSales is the little frame the examples below work on.
func exampleSales() *ursus.LazyFrame {
	return ursus.Frame(
		ursus.Values("region", []string{"north", "south", "north", "east", "south"}),
		ursus.Values("product", []string{"apple", "apple", "pear", "pear", "fig"}),
		ursus.Values("qty", []int64{10, 4, 7, 2, 9}),
		ursus.Values("price", []float64{1.50, 1.50, 2.25, 2.25, 3.00}),
	)
}

func main() {
	df, err := exampleSales().
		WithColumns(ursus.Col("qty").Mul(ursus.Col("price")).Alias("revenue")).
		Select(ursus.Col("product"), ursus.Col("revenue")).
		Collect(context.Background())
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(df)
}

Output:
shape: (5, 2)
┌─────────┬─────────┐
│ product │ revenue │
│ String  │ Float64 │
├─────────┼─────────┤
│ "apple" │ 15      │
│ "apple" │ 6       │
│ "pear"  │ 15.75   │
│ "pear"  │ 4.5     │
│ "fig"   │ 27      │
└─────────┴─────────┘

func (*LazyFrame) WithColumnsSeq added in v0.6.7 

func (lf *LazyFrame) WithColumnsSeq(exprs ...Expr) *LazyFrame

WithColumnsSeq and SelectSeq are WithColumns and Select, as Polars' with_columns_seq and select_seq, which evaluate their expressions one after another rather than in parallel; ursus's evaluate them alike either way.

func (*LazyFrame) WithRowIndex 

func (lf *LazyFrame) WithRowIndex(name string, offset uint32) *LazyFrame

WithRowIndex prepends a Uint32 column numbering the rows from offset.

Row order is a property rather than an addressable label space, which is why there is no implicit index — this is how you ask for one when you want it.

It is a counter that crosses batch boundaries, so it depends on batches arriving in input order, and it is deliberately not the same thing as `CumCount(false).Over()`: that computes identical numbers through the window sink, which buffers the entire frame to do it.

func (*LazyFrame) WriteCSV 

func (lf *LazyFrame) WriteCSV(ctx context.Context, w io.Writer, opts ...CSVSinkOption) (err error)

WriteCSV runs the query and writes the result to w, streaming.

It does not close w. Whoever opened it closes it — the only rule that works when the destination might be os.Stdout.

func (*LazyFrame) WriteDatabase added in v0.6.8 

func (lf *LazyFrame) WriteDatabase(ctx context.Context, db DatabaseWriter, table string,
	opts ...DatabaseWriteOption) (n int64, err error)

WriteDatabase runs the query and inserts its rows into table, and answers how many it wrote: Polars' write_database. A name of dots is a schema's table, "s.t", each part quoted.

A table already there is refused, replaced or appended to, as WithIfTableExists says; one that is not is created, its columns of the types ReadDatabase reads back as theirs: an Int64 a BIGINT, a Float64 a DOUBLE PRECISION, text a TEXT, an instant a TIMESTAMP, with a zone a TIMESTAMPTZ, in the dialect's own spelling. A List, an Array, a Struct and a Duration have no column type, and are refused before anything is written.

The rows go as they are read, in INSERTs of as many rows as the dialect's parameters allow, up to a thousand, all in one transaction: a *sql.DB's or a *sql.Conn's own, committed at the end and rolled back on an error, or the caller's *sql.Tx. A value is what ToMaps gives, but a time of day, which is its text, and an unsigned integer past 2^63-1, which is its text, or refused by SQLite, which holds none. New in 0.6.8.

func (*LazyFrame) WriteIPC added in v0.6.7 

func (lf *LazyFrame) WriteIPC(ctx context.Context, w io.Writer, opts ...IPCSinkOption) error

WriteIPC runs the query and writes the answer to w as an Arrow IPC file, Feather version 2, a record batch a batch, streaming, as Polars' write_ipc writes one. It does not close w.

func (*LazyFrame) WriteIPCStream added in v0.6.7 

func (lf *LazyFrame) WriteIPCStream(ctx context.Context, w io.Writer, opts ...IPCSinkOption) error

WriteIPCStream runs the query and writes the answer to w as an Arrow IPC stream, as Polars' write_ipc_stream.

func (*LazyFrame) WriteJSON added in v0.6.7 

func (lf *LazyFrame) WriteJSON(ctx context.Context, w io.Writer, opts ...CollectOption) error

WriteJSON runs the query and writes it to w as one JSON array of the rows' objects, as Polars' write_json: "[]" for no rows. See WriteNDJSON for each value's text.

func (*LazyFrame) WriteNDJSON added in v0.6.7 

func (lf *LazyFrame) WriteNDJSON(ctx context.Context, w io.Writer, opts ...CollectOption) error

WriteNDJSON runs the query and writes each row to w as a JSON object on a line of its own, streaming, as Polars' write_ndjson: a key a column, in the columns' order. It does not close w. A value is written as Polars writes one:

  • a number as one, a float in Polars' text, "1.0", "1e+20"; NaN and the infinities null, which JSON has no word for;
  • text quoted and escaped, its UTF-8 kept;
  • a Decimal quoted, "1.25"; a Date "2024-01-02", a Time "03:04:05.000006", a Datetime "2024-01-02 03:04:05.123456", a zoned one "2024-01-02T03:04:05+05:00"; a Duration in ISO 8601's seconds, "PT90S", "P0D" for none;
  • a List an array and a Struct an object; a null null.

A Binary column is refused, as Polars cannot write one either.

func (*LazyFrame) WriteParquet 

func (lf *LazyFrame) WriteParquet(ctx context.Context, w io.Writer, opts ...ParquetSinkOption) (err error)

WriteParquet runs the query and writes the result to w, streaming. It does not close w.

Example

A List column writes to Parquet and reads back, in the standard encoding PyArrow and Polars read too. 

package main

import (
	"bytes"
	"context"
	"fmt"
	"log"

	"github.com/advenn/ursus"
)

func main() {
	ctx := context.Background()
	orders := ursus.Frame(
		ursus.Values("order", []int64{1, 2}),
		ursus.Values("items", []string{"apple,pear", "fig"}),
	).WithColumns(ursus.Col("items").Str().Split(","))

	var file bytes.Buffer
	if err := orders.WriteParquet(ctx, &file); err != nil {
		log.Fatal(err)
	}
	df, err := ursus.ScanParquetBytes(file.Bytes(), "orders.parquet").Collect(ctx)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(df)
}

Output:
shape: (2, 2)
┌───────┬───────────────────┐
│ order │ items             │
│ Int64 │ List(String)      │
├───────┼───────────────────┤
│ 1     │ ["apple", "pear"] │
│ 2     │ ["fig"]           │
└───────┴───────────────────┘

type ListExpr added in v0.3.0 

type ListExpr struct {
	// contains filtered or unexported fields
}

ListExpr is the list namespace: `Col("tags").List().Len()`.

The same wrapper shape `.str` and `.dt` use, for the reason ursus-api.md §4.5 gives: methods return Expr, so chaining continues naturally through it.

Two halves, told apart by what they return

Len, Get, Contains, Min, Max, Sum, Mean, NUnique, ArgMin and ArgMax answer a question ABOUT a list and hand back an ordinary column — a length, an element, a total, a position.

Reverse, Head, Tail, Slice, Sort, Unique and DropNulls RESHAPE the list and hand back a list, so they chain into each other and into the first half: `Sort().Head(3).Get(0)`. The element type is unchanged by all seven — reshaping a list cannot change what it is a list OF.

The set operations and Gather take a SECOND list, a column like the first, and hand back a list (step 223).

A null list is not an empty list

It is the distinction the whole List type is careful about, and it survives into every method: 

                Len()  Sum()  Min()  Get(0)  Contains(x)
[1, 2]            2      3      1      1      as asked
[]  (empty)       0     null   null   null      false
null (list)      null   null   null   null      null

An empty list HAS a length, and it is zero. A null list has no length at all. An empty list contains nothing, which is a false answer to a real question; a null list cannot answer it. Working from "both are empty" gives three wrong cells here, none of which errors.

The aggregates are where the two agree, and that is a choice: polars gives 0 for the sum of an empty list, ursus gives null, because ursus's Sum gives null for a group with no values. Agreeing with the aggregate beside it beats agreeing with polars.

func (ListExpr) Agg added in v0.6.6 

func (l ListExpr) Agg(body Expr) Expr

Agg is Eval whose aggregate is its value, not a list of one: Polars' list.agg. 

Col("l").List().Agg(Element().Filter(Element().Gt(0)).Sum())   // each list's positive sum

func (ListExpr) All added in v0.6.6 

func (l ListExpr) All(ignoreNulls bool) Expr

func (ListExpr) Any added in v0.6.6 

func (l ListExpr) Any(ignoreNulls bool) Expr

Any and All are whether some, or every, element of a List of Booleans is true: Polars' list.any and list.all. With ignoreNulls, Polars' default, a null element is skipped, so an empty list is false for Any and true for All; without it they are Kleene's, all of a true and a null null.

func (ListExpr) ArgMax added in v0.6.3 

func (l ListExpr) ArgMax() Expr

func (ListExpr) ArgMin added in v0.6.3 

func (l ListExpr) ArgMin() Expr

ArgMin and ArgMax are the position in each list of its smallest or largest element, as a Uint64, as the column-wide ArgMin and ArgMax give a group's: the first of equal elements, counting null elements as positions but never choosing one. An empty list, a list of nulls, and a null list are null. The element type must have an order.

NaN is the largest value, as it is for Max: [2.0, NaN, 1.0] has its ArgMax at 1. Polars skips NaN there, answering 0, and its positions are UInt32s.

func (ListExpr) Concat added in v0.6.6 

func (l ListExpr) Concat(others ...Expr) Expr

Concat is each list followed by others' elements, or their values: Polars' list.concat, ConcatList(l, others...). A null list among them makes the row's null.

func (ListExpr) Contains added in v0.3.0 

func (l ListExpr) Contains(v any) Expr

Contains reports whether the list holds v.

The comparison is ursus's, not Go's: values go through the same group-key encoder `IsIn` uses, so NaN matches NaN and -0.0 matches +0.0, consistently with how GroupBy and Sort treat them.

The element and v meet as == would compare them, as IsIn's values do: v of a type == refuses is refused, and `Contains(5000)` against Int8 elements is false.

v may be an expression, one value a row (step 254). A null value there is found where the list holds a null, as Polars' list.contains finds it.

func (ListExpr) CountMatches added in v0.6.6 

func (l ListExpr) CountMatches(v any) Expr

CountMatches is how many of each list's elements equal v, a UInt32: Polars' list.count_matches. v is a value, nil for null, which counts the null elements as Polars counts them, or an expression, a value a row; they meet as == compares.

func (ListExpr) Diff added in v0.6.6 

func (l ListExpr) Diff(n int, dropNulls bool) Expr

Diff is each element less the one n places before it, null for the first n: Polars' list.diff, its null_behavior "ignore"; with dropNulls, "drop", those first n are dropped. It is Eval(Element().Diff(n)).

func (ListExpr) DropNans added in v0.6.5 

func (l ListExpr) DropNans() Expr

DropNans drops each list's NaN elements; a null stays (step 253).

func (ListExpr) DropNulls added in v0.3.0 

func (l ListExpr) DropNulls() Expr

DropNulls removes the null ELEMENTS from each list. A null LIST stays null — there is nothing to drop from, which is not the same as having dropped everything.

func (ListExpr) Eval added in v0.6.6 

func (l ListExpr) Eval(body Expr) Expr

Eval runs body over each list's elements, the list its group: Polars' list.eval (step 268). Element() is the elements; the body reads nothing else. 

Col("l").List().Eval(Element().Mul(2))                  // each element doubled
Col("l").List().Eval(Element().Sub(Element().Mean()))   // each list centred
Col("l").List().Eval(Element().Rank(RankAverage, false)) // ranked within its list

An expression of a value an element keeps each list's length; a constant is repeated along it. An aggregate is a list of one, its value; a reshape or a filter is the list it makes. A window function, rank or cum_sum, is computed a list at a time. A null list stays null. As the list functions here, the sum of an empty list is null, where Polars' is 0.

func (ListExpr) Filter added in v0.6.6 

func (l ListExpr) Filter(pred Expr) Expr

Filter keeps each list's elements pred is true on: Polars' list.filter, Eval of Element().Filter(pred). A null predicate drops the element, as Filter drops a row.

func (ListExpr) First added in v0.3.0 

func (l ListExpr) First() Expr

First is Get(0) and Last is Get(-1).

func (ListExpr) Gather added in v0.6.3 

func (l ListExpr) Gather(indices Expr, nullOnOOB bool) Expr

Gather picks each list's elements at the positions in indices, a List of integers a row, as Polars' list.gather does: a negative index counts from the end, and a null index gives a null element. An index out of range fails the query, or with nullOnOOB gives a null element. A null list, or null indices, give a null list.

func (ListExpr) GatherEvery added in v0.6.5 

func (l ListExpr) GatherEvery(n, offset int) Expr

GatherEvery is every n-th element of each list, from offset (step 253).

func (ListExpr) Get added in v0.3.0 

func (l ListExpr) Get[I int | Expr](i I) Expr

Get returns the element at i, or null if there is none there.

A negative index counts from the END, the convention every dataframe library uses and the one `.str.slice` already follows here.

Out of range is null rather than an error. Lists are ragged by nature, so asking for element 3 of a column whose rows mostly hold two is an ordinary question — erroring would make `Get` unusable on exactly the data it is for.

i is a constant, or an expression of an integer type, one index a row (step 254), null where it is null. Polars' list.get(expr) errs out of range unless null_on_oob; here it is null either way, as for a constant.

func (ListExpr) Head added in v0.3.0 

func (l ListExpr) Head(n int) Expr

Head keeps the first n elements of each list, Tail the last n.

Asking for more than a row holds returns the whole row, not an error and not a padded list. Lists are ragged by nature, so asking for three tags from a column whose rows mostly have two is an ordinary question — the same reasoning Get uses for an index past the end.

A negative n is refused: "the first -2 elements" has no reading worth guessing at. Slice is where a negative number means something.

func (ListExpr) Item added in v0.6.6 

func (l ListExpr) Item(allowEmpty bool) Expr

Item is each list's one element: Polars' list.item. A list of another length is refused, but an empty one with allowEmpty, which is null.

func (ListExpr) Join added in v0.5.0 

func (l ListExpr) Join(sep string) Expr

Join joins each list's strings with sep, skipping null elements, as Polars' list.join does by default: ["a", null, "b"] joins with "-" to "a-b". An empty list joins to "", and a null list is null. The list must be of String; cast it first otherwise, e.g. .Cast(List(String)).

func (ListExpr) Last added in v0.3.0 

func (l ListExpr) Last() Expr

func (ListExpr) Len added in v0.3.0 

func (l ListExpr) Len() Expr

Len is the number of elements, as a Uint32 — matching `.str.len_bytes`, since a length is never negative and never needs 64 bits.

Null ELEMENTS are counted: they occupy a slot, so `[1, null]` has length 2. A null LIST has no length and gives null.

func (ListExpr) Max added in v0.3.0 

func (l ListExpr) Max() Expr

func (ListExpr) Mean added in v0.3.0 

func (l ListExpr) Mean() Expr

Mean averages each list, as a Float64. Null elements are skipped, so the divisor is the count of PRESENT elements.

func (ListExpr) Median added in v0.6.5 

func (l ListExpr) Median() Expr

Median is each list's median, the column aggregate over its elements (step 253).

func (ListExpr) Min added in v0.3.0 

func (l ListExpr) Min() Expr

Min and Max reduce each list to its smallest or largest element, keeping the element type. Null elements are skipped, exactly as the column-wide aggregates skip null rows.

func (ListExpr) NUnique added in v0.6.3 

func (l ListExpr) NUnique() Expr

NUnique counts each list's distinct elements, as a Uint64, as the column-wide NUnique counts a group's: null elements are skipped, NaN is one value, and −0 and +0 are one. An empty list counts 0, a null list is null.

Polars counts a null element as one more value, and answers a UInt32: [3, null, 1, 1, null] is 3 there and 2 here. Agreeing with the aggregate beside it is the rule Sum states.

func (ListExpr) Quantile added in v0.6.5 

func (l ListExpr) Quantile(q float64, interp Interpolation) Expr

Quantile is each list's q-quantile, interpolated as the aggregate's (step 253).

func (ListExpr) Reverse added in v0.3.0 

func (l ListExpr) Reverse() Expr

Reverse puts each list's elements in the opposite order. Null elements move with everything else — they occupy a slot.

func (ListExpr) SampleFraction added in v0.6.6 

func (l ListExpr) SampleFraction(fraction float64, withReplacement, shuffle bool, seed uint64) Expr

func (ListExpr) SampleN added in v0.6.6 

func (l ListExpr) SampleN(n int, withReplacement, shuffle bool, seed uint64) Expr

SampleN is n of each list's elements at random, and SampleFraction the fraction of its length, rounded down: Polars' list.sample. Without replacement a list shorter than the sample is refused, as Polars refuses it, and the sample keeps the list's order unless shuffle.

The draw is seeded by seed and the list's own elements, so it does not depend on how the rows are batched and two equal lists draw alike. Polars draws a seed when none is given; here it is always given. The values are not Polars', whose generator is its own.

func (ListExpr) SetDifference added in v0.6.3 

func (l ListExpr) SetDifference(other Expr) Expr

func (ListExpr) SetIntersection added in v0.6.3 

func (l ListExpr) SetIntersection(other Expr) Expr

func (ListExpr) SetSymmetricDifference added in v0.6.3 

func (l ListExpr) SetSymmetricDifference(other Expr) Expr

func (ListExpr) SetUnion added in v0.6.3 

func (l ListExpr) SetUnion(other Expr) Expr

SetUnion, SetIntersection, SetDifference and SetSymmetricDifference treat each row's two lists as sets, as Polars' set_union and its siblings do:

  • SetUnion is this list's distinct elements in the order they first appear, then other's that are not among them;
  • SetIntersection is this list's distinct elements that other holds, in this list's order, and SetDifference those it does not;
  • SetSymmetricDifference is SetDifference, then other's distinct elements this list does not hold, in other's order.

Elements are equal as group_by keys are: a null element is a value, equal to another null; NaN equals NaN; −0 equals +0. A null list on either side gives a null list. The two element types meet as a join's keys do: an Int64 and a Float64 are refused, where Polars meets them at Float64 and compares rounded integers.

func (ListExpr) Shift added in v0.6.6 

func (l ListExpr) Shift[N Operand](n N) Expr

Shift moves each list's elements n places later, nulls in the places they leave, a negative n earlier: Polars' list.shift. n is a constant or an expression, a count a row; a null count gives a null list.

func (ListExpr) Slice added in v0.3.0 

func (l ListExpr) Slice(offset, length int) Expr

Slice takes length elements from offset.

A negative OFFSET counts from the end, matching Get and `.str.slice`, and the window is clipped to the list as Polars clips it: a start before the first element cuts the length by as much, so (-7, 2) of six elements is the first alone. A length running off the end stops there. A negative length is refused.

func (ListExpr) Sort added in v0.3.0 

func (l ListExpr) Sort() Expr

Sort orders each list's elements ascending; SortDesc orders them descending.

The ordering is ursus's own — the same comparator a column-wide Sort uses — so a list of values and a column of the same values order identically. That includes null placement: nulls come FIRST, matching the default of `Asc()`. Both placements are defensible and this is the one ursus picked.

Equal elements keep their input order.

func (ListExpr) SortBy added in v0.6.5 

func (l ListExpr) SortBy(by Expr) Expr

SortBy sorts each list by the list beside it, element for element, ascending, nulls first, stably (step 253). Two lists of a row must be of one length.

func (ListExpr) SortByDesc added in v0.6.5 

func (l ListExpr) SortByDesc(by Expr) Expr

SortByDesc is SortBy in descending order, nulls first.

func (ListExpr) SortDesc added in v0.3.0 

func (l ListExpr) SortDesc() Expr

func (ListExpr) SortWith added in v0.6.6 

func (l ListExpr) SortWith(descending, nullsLast bool) Expr

SortWith sorts each list in either direction with its nulls first or last: Polars' list.sort(descending, nulls_last) (step 269).

func (ListExpr) Std added in v0.6.5 

func (l ListExpr) Std(ddof int) Expr

Std is each list's standard deviation with ddof, as the aggregate's (step 253).

func (ListExpr) Sum added in v0.3.0 

func (l ListExpr) Sum() Expr

Sum totals each list.

It widens an integer element to Int128, because the column-wide Sum does and a per-row sum that overflowed where that one does not would be the worse surprise. The rule is not restated here — the type comes from the aggregate's own binding, so the two cannot drift.

An empty list sums to NULL, not zero — the same answer ursus's Sum gives for a group with no values. polars differs here.

func (ListExpr) Tail added in v0.3.0 

func (l ListExpr) Tail(n int) Expr

Tail keeps the last n elements. See Head.

func (ListExpr) ToArray added in v0.6.7 

func (l ListExpr) ToArray(width int) Expr

ToArray is each list as an Array of width elements, Polars' list.to_array: every list but a null one must hold width elements, or the query is refused, as Polars refuses it ("not all elements have the specified width"). A null list is a null Array. Arr().ToList() is the way back.

New in 0.6.7.

func (ListExpr) ToStruct added in v0.6.3 

func (l ListExpr) ToStruct(names ...string) Expr

ToStruct makes each list a Struct whose fields are its first elements, named by names: field i is element i, null where the list is shorter, and the elements past the last field are dropped. A null list is a null struct, where Polars gives a struct of nulls. At least one name, each different. Polars' to_struct with its fields named; the names are needed because a plan's schema cannot wait for the data to say how long the lists are.

func (ListExpr) Unique added in v0.3.0 

func (l ListExpr) Unique() Expr

Unique drops repeated elements, keeping the FIRST occurrence and leaving the order alone — sorting as a side effect would make `Unique().Head(2)` mean something different from `Head(2)` on the same data.

Equality is ursus's, as it is for Contains: NaN dedupes against NaN. A null element counts as a value here, so a list of three nulls keeps one.

func (ListExpr) Var added in v0.6.5 

func (l ListExpr) Var(ddof int) Expr

Var is each list's variance with ddof, as the aggregate's (step 253).

type ListLiteral added in v0.6.5 

type ListLiteral interface {
	[]bool | []int | []int8 | []int16 | []int32 | []int64 |
		[]uint | []uint16 | []uint32 | []uint64 |
		[]float32 | []float64 | []string | []time.Time | []time.Duration
}

ListLiteral is a Go slice that Lit makes a List literal (step 257): one value, a list of the elements' type, Lit([]int64{1, 2}) a List(Int64). Every element is a value; a list with a null element is built by an expression, not a literal.

[]byte is absent: it is a Binary literal, as it always was, and a List of Uint8 is Lit([]uint16{...}).Cast(List(Uint8)). The terms are exact, so a defined slice type must be converted to its slice first.

type Literal 

type Literal interface {
	~bool |
		~int | ~int8 | ~int16 | ~int32 | ~int64 |
		~uint | ~uint8 | ~uint16 | ~uint32 | ~uint64 |
		~float32 | ~float64 |
		~string | ~[]byte |
		time.Time
}

Literal is the set of Go types that lift to a column literal.

Every accepted width is enumerated explicitly because Go infers a type parameter from the argument's type with NO implicit widening: `a.Gt(int32(5))` is a compile error unless ~int32 appears here.

time.Duration is DELIBERATELY ABSENT. Its underlying type is int64, so listing it alongside ~int64 gives overlapping type sets and the union does not compile — this is the defect the design review found. ~int64 already admits it, and a type switch recovers the named type exactly, so nothing is lost.

time.Time is a bare (non-~) term: it is a defined struct type, disjoint from every other term. A union term may be a type with methods; only interface terms with methods are forbidden.

type MappingStrategy 

type MappingStrategy = expr.MappingStrategy

MappingStrategy decides how a window's per-partition result returns to the frame.

type MatchAction added in v0.6.7 

type MatchAction uint8

MatchAction is how MatchToSchema answers one kind of difference between a frame and a schema. Each field of MatchOptions takes two of them; the zero value is the first, Polars' default. 

const (

	// MatchRaise refuses the difference: a missing column or field, or an extra one.
	MatchRaise MatchAction
	// MatchInsert adds a missing column or struct field, as nulls of its type.
	MatchInsert
	// MatchIgnore drops an extra column or struct field.
	MatchIgnore
	// MatchForbid refuses a column of a narrower type than the schema's.
	MatchForbid
	// MatchUpcast widens it, when the schema's type holds every value of the
	// column's: an Int32 to an Int64, a UInt32 to an Int64, a Float32 to a Float64.
	MatchUpcast
)

func (MatchAction) String added in v0.6.7 

func (a MatchAction) String() string

type MatchOptions added in v0.6.7 

type MatchOptions struct {
	// MissingColumns is a schema column the frame lacks: MatchRaise, or MatchInsert
	// as nulls of its type. Polars' missing_columns.
	MissingColumns MatchAction
	// ExtraColumns is a frame column the schema lacks: MatchRaise, or MatchIgnore to
	// drop it. Polars' extra_columns.
	ExtraColumns MatchAction
	// MissingStructFields and ExtraStructFields are the same for a struct's fields,
	// at any depth: MatchRaise, or MatchInsert and MatchIgnore.
	MissingStructFields, ExtraStructFields MatchAction
	// IntegerCast and FloatCast are a column narrower than the schema's type:
	// MatchForbid, or MatchUpcast to widen it. An integer never becomes a float, as
	// in Polars, and a type that cannot hold every value, an Int64 as an Int32 or a
	// UInt64 as an Int64, is refused either way. A List's element and a struct's
	// field are widened as a column is.
	IntegerCast, FloatCast MatchAction
	// Columns sets any of these for one column of the schema, by its name; a zero
	// field keeps the setting above. Polars' per-column mappings.
	Columns map[string]ColumnMatch
}

MatchOptions says how MatchToSchema meets the schema, as Polars' match_to_schema's keyword arguments do. The zero value refuses every difference, as Polars' defaults do.

type MemoryStats 

type MemoryStats struct {
	// Peak is the largest total the query retained at one moment, in bytes.
	//
	// It counts what buffering operators HOLD ACROSS BATCHES, deduplicated by
	// allocation so a payload two operators share is counted once. It does not
	// count a kernel's transient output, because ursus's allocator reports
	// nothing and there is no hook at the allocation site.
	//
	// Nor the result Collect assembles. Under the default budget Collect counts
	// that apart, against a limit of its own: see Collect.
	Peak int64

	// Limit is the ceiling that was in force, or 0 if there was none.
	Limit int64

	// Spills is how many files a spilling operator wrote: sorted runs for a sort,
	// radix partitions for a group-by, and for a join BOTH sides' partitions —
	// counted at every level of the recursion.
	//
	// It is a count of FILES, not of bytes and not of distinct spill events, and a
	// group-by that recursed reports more than one level's fan-out — which is the
	// only public evidence that a partition did not fit on the first try.
	Spills int64
}

MemoryStats reports what a query actually held.

Peak is the number that demonstrates bounded memory: that a query FINISHED proves only that it did not run out, while a peak far below the input size proves it never held the input at all.

type MetaExpr added in v0.6.7 

type MetaExpr struct {
	// contains filtered or unexported fields
}

MetaExpr is an expression's meta functions, as Polars' meta namespace has them.

func (MetaExpr) Eq added in v0.6.7 

func (m MetaExpr) Eq(other Expr) bool

Eq reports whether two expressions are one computation, as Polars' meta.eq: alike in their operations, their names, and their literals' values and types, so Lit(1) is not Lit(1.0).

func (MetaExpr) HasMultipleOutputs added in v0.6.7 

func (m MetaExpr) HasMultipleOutputs() bool

HasMultipleOutputs reports whether the expression may make several columns: a selector in it, Col of several names, a pattern, a type, All or Exclude, as Polars' has_multiple_outputs.

func (MetaExpr) IsColumn added in v0.6.7 

func (m MetaExpr) IsColumn() bool

IsColumn reports whether the expression is one column read by name, unnamed, as Polars' is_column.

func (MetaExpr) IsColumnSelection added in v0.6.7 

func (m MetaExpr) IsColumnSelection(allowAliasing bool) bool

IsColumnSelection reports whether the expression only selects columns, a column by name or a selector, as Polars' is_column_selection; with allowAliasing, under a name too.

func (MetaExpr) IsLiteral added in v0.6.7 

func (m MetaExpr) IsLiteral(allowAliasing bool) bool

IsLiteral reports whether the expression is a literal, as Polars' is_literal; with allowAliasing, under a name too.

func (MetaExpr) IsRegexProjection added in v0.6.7 

func (m MetaExpr) IsRegexProjection() bool

IsRegexProjection reports whether the expression selects columns by a regular expression, ColRegex("^a.*$"), as Polars' is_regex_projection. Col reads every name as a name, an anchored one too, where Polars' col("^a.*$") is a pattern.

func (MetaExpr) OutputName added in v0.6.7 

func (m MetaExpr) OutputName() (string, error)

OutputName is the name the expression's column takes, as Polars' output_name: an alias's or a name function's, else its first column's, a literal's "literal". An expression that may make several columns has no one name, and is an error.

func (MetaExpr) Pop added in v0.6.7 

func (m MetaExpr) Pop() []Expr

Pop is the inputs of the expression's outermost operation, in the order Polars' pop gives them, the last first: of a + b, b then a; of a column, none.

func (MetaExpr) RootNames added in v0.6.7 

func (m MetaExpr) RootNames() []string

RootNames is the columns the expression reads by name, a name each time it is read, in the order Polars' root_names lists them: an operation's inputs left to right, a conditional's branches before its condition. A selector, Col of several names, a pattern or All, names none, as Polars' does.

func (MetaExpr) TreeFormat added in v0.6.7 

func (m MetaExpr) TreeFormat() string

TreeFormat is the expression as a tree, an operation a line and its inputs under it, as Polars' tree_format draws one; ursus draws its own.

func (MetaExpr) UndoAliases added in v0.6.7 

func (m MetaExpr) UndoAliases() Expr

UndoAliases is the expression without its names, every Alias and every name function in it, as Polars' undo_aliases.

type NDJSONFile added in v0.6.7 

type NDJSONFile struct {
	// Name identifies the file in errors and in Explain.
	Name string

	// Open returns the file's bytes from the start. It is called more than once
	// a query, once to infer the schema and once a run, and must return a fresh
	// stream each time. ctx is the query's and outlives the stream, which ursus
	// closes when it is done with it.
	Open func(ctx context.Context) (io.ReadCloser, error)
}

NDJSONFile is one NDJSON file, read as a stream from wherever it lives by code that knows how to reach it, as CSVFile.

type NDJSONOption added in v0.6.7 

type NDJSONOption func(*ndjsonCfg)

NDJSONOption configures an NDJSON scan.

func WithInferSchemaLines added in v0.6.7 

func WithInferSchemaLines(n int) NDJSONOption

WithInferSchemaLines infers the schema from the first n lines, 100 unless set, as Polars' infer_schema_length; 0 reads every line first. A JSON document's are its objects.

func WithNDJSONIgnoreErrors added in v0.6.7 

func WithNDJSONIgnoreErrors(b bool) NDJSONOption

WithNDJSONIgnoreErrors reads a value its column cannot take as a null, as Polars' ignore_errors: text in an Int64 column, an element of a list that does not fit its type, a Date column's "not a date". A line that is not JSON is still refused.

func WithNDJSONIncludeFilePaths added in v0.6.7 

func WithNDJSONIncludeFilePaths(name string) NDJSONOption

WithNDJSONIncludeFilePaths adds a String column named name of each row's file path, after the files' columns, as Polars' include_file_paths: the path as it was given, or a glob's as it found it; bytes', their name. New in 0.6.7.

func WithNDJSONSchema added in v0.6.7 

func WithNDJSONSchema(s *Schema) NDJSONOption

WithNDJSONSchema reads the lines as schema, its keys as its columns in its order and no other key, as Polars' schema=: no line is read to infer one. A value is read as JSON gives it and cast to its column's type, so a Date column reads "2024-01-02".

func WithNDJSONSchemaOverrides added in v0.6.7 

func WithNDJSONSchemaOverrides(m map[string]dtype.DataType) NDJSONOption

WithNDJSONSchemaOverrides reads the named columns as these types, the others' inferred or given, as Polars' schema_overrides: a column of numbers read as text, or of text as dates. Each name must be one of the schema's. New in 0.6.7.

type NDJSONSinkOption added in v0.6.8 

type NDJSONSinkOption interface {
	// contains filtered or unexported methods
}

NDJSONSinkOption is an option to SinkNDJSON: a PartitionOption or a CollectOption. New in 0.6.8, where SinkNDJSON took CollectOptions alone.

type NameExpr added in v0.6.7 

type NameExpr struct {
	// contains filtered or unexported fields
}

NameExpr is an expression's naming functions, as Polars' name namespace has them: Col("a").Name().Suffix("_x"). Each names every column a multi-column expression makes, as Alias, one name, cannot.

func (NameExpr) Keep added in v0.6.7 

func (n NameExpr) Keep() Expr

Keep names the output after the first column the expression reads, through any alias or rename, as Polars' name.keep: Col("a").Mul(2).Alias("x").Name().Keep() is a. An expression that reads no column is refused.

func (NameExpr) Map added in v0.6.7 

func (n NameExpr) Map(fn func(string) string) Expr

Map derives the output name from the input name, as MapName does.

func (NameExpr) MapFields added in v0.6.7 

func (n NameExpr) MapFields(fn func(string) string) Expr

MapFields is fn of each field's name. fn is called while the query is planned and as it runs; a panic in it is the caller's error, of kind ErrValue.

func (NameExpr) Prefix added in v0.6.7 

func (n NameExpr) Prefix(p string) Expr

Prefix prepends to the output name, and Suffix appends to it, as Expr's do.

func (NameExpr) PrefixFields added in v0.6.7 

func (n NameExpr) PrefixFields(p string) Expr

PrefixFields prepends to the name of each field of a struct, and SuffixFields appends to it, as Polars' name.prefix_fields and suffix_fields; MapFields derives each from the old. The struct's own fields only, not a nested struct's, and the column keeps its name. The expression must be a Struct, and two fields of one new name are refused.

func (NameExpr) Replace added in v0.6.7 

func (n NameExpr) Replace(pattern, value string, literal bool) Expr

Replace replaces every match of pattern in the output name with value, as Polars' name.replace: a regular expression whose groups value names as $1 or ${name}, or with literal the text itself. A pattern that does not compile is refused when the query is planned.

func (NameExpr) Suffix added in v0.6.7 

func (n NameExpr) Suffix(s string) Expr

func (NameExpr) SuffixFields added in v0.6.7 

func (n NameExpr) SuffixFields(s string) Expr

func (NameExpr) ToLower added in v0.6.7 

func (n NameExpr) ToLower() Expr

ToLower and ToUpper change the output name's case, every script's, as Polars' name.to_lowercase and to_uppercase.

func (NameExpr) ToUpper added in v0.6.7 

func (n NameExpr) ToUpper() Expr

type NonExistent added in v0.6.0 

type NonExistent string

NonExistent answers a wall clock that ReplaceTimeZone's new zone never reads, in the hour its clocks spring forward: Polars' non_existent. 

const (
	NonExistentRaise NonExistent = expr.ZoneRaise // refuse the query; the default
	NonExistentNull  NonExistent = expr.ZoneNull  // a null
)

type Operand 

type Operand interface {
	Expr | Literal
}

Operand is either another Expr or a Go scalar that lifts to a literal.

This union — a struct type alongside an embedded constraint interface — is what lets every binary operator accept both forms with no wrapper at the call site.

type OptFlags added in v0.6.0 

type OptFlags struct {
	ProjectionPushdown    bool // read only the columns a query uses
	PredicatePushdown     bool // filter as early as the rows allow
	LimitPushdown         bool // stop reading once a Head has its rows
	SimplifyExprs         bool // fold constants and simplify expressions
	JoinPredicatePushdown bool // predicate pushdown's arm through joins
	CollapseCrossJoin     bool // a filter over a cross join becomes a join
	BuildSide             bool // an inner join hashes its smaller input
	RuntimeFilters        bool // a join's keys filter its probe side's source
	ShareSubplans         bool // a subtree used twice runs once
}

OptFlags chooses which optimizer rules run, for WithOptFlags: each is on in DefaultOptFlags, the optimizer every query gets, and off in NoOptFlags. 

q.Collect(ctx, ursus.WithOptFlags(ursus.NoOptFlags()))   // the plan as written

f := ursus.DefaultOptFlags()
f.BuildSide = false                                     // all but one rule
q.Collect(ctx, ursus.WithOptFlags(f))

A rule changes how a query runs, never its answer; turning one off is how a wrong answer is bisected to the rule that caused it, or a rule's worth measured.

func DefaultOptFlags added in v0.6.0 

func DefaultOptFlags() OptFlags

DefaultOptFlags is every rule on, as a query runs by default.

func NoOptFlags added in v0.6.0 

func NoOptFlags() OptFlags

NoOptFlags is every rule off: the plan runs as it was written.

type ParquetFile added in v0.3.0 

type ParquetFile struct {
	// Name identifies the file in errors and in Explain.
	Name string

	// Open returns the file's bytes and their size.
	//
	// It is called more than once per query — once to read the schema when the
	// query is planned, once for each execution — and must return a fresh reader
	// each time. ctx is the query's and outlives the reader, so a reader over a
	// network may keep it for its reads. If the reader is also an io.Closer, ursus
	// closes it when it is done with it.
	Open func(ctx context.Context) (r io.ReaderAt, size int64, err error)
}

ParquetFile is one Parquet file, read through ReadAt from wherever it lives — an object store, an HTTP server, an archive — by code that knows how to reach it.

ursus reads a file the way an object store wants to be read: the footer from the end, then one ranged read for each column chunk the query needs, after the statistics have pruned the row groups it does not. A ReadAt over a ranged GET therefore fetches what the query reads and nothing else.

type ParquetOption 

type ParquetOption func(*parquet.Options)

ParquetOption configures ScanParquet.

func WithHiveDates added in v0.6.7 

func WithHiveDates(b bool) ParquetOption

WithHiveDates reads a partition key of dates or of instants as a Date or a Datetime, as Polars' try_parse_hive_dates; on by default. New in 0.6.7.

func WithHivePartitioning added in v0.6.7 

func WithHivePartitioning(b bool) ParquetOption

WithHivePartitioning reads a hive dataset's partition columns from its files' paths, as Polars' hive_partitioning: each directory named key=value gives the rows of the files beneath it a column key of value, its type inferred from every file's value, as Polars infers it. On by default for a directory, where the directories below it count, and off otherwise; on for a file, a list or a glob, every directory of each path counts, as in Polars. New in 0.6.7.

func WithHiveSchema added in v0.6.7 

func WithHiveSchema(s *Schema) ParquetOption

WithHiveSchema gives the types of the partition keys it names, as Polars' hive_schema; the others are inferred, where Polars refuses a key it does not name. New in 0.6.7.

func WithIgnoreExtraColumns added in v0.6.7 

func WithIgnoreExtraColumns(b bool) ParquetOption

WithIgnoreExtraColumns passes over a column a file has beyond the schema's, as Polars' extra_columns="ignore", where it is refused by default. New in 0.6.7.

func WithIncludeFilePaths added in v0.6.7 

func WithIncludeFilePaths(name string) ParquetOption

WithIncludeFilePaths adds a column named name of each row's file path, last, as Polars' include_file_paths. New in 0.6.7.

func WithInsertMissingColumns added in v0.6.7 

func WithInsertMissingColumns(b bool) ParquetOption

WithInsertMissingColumns reads a column a file lacks as nulls in its rows, as Polars' missing_columns="insert", where it is refused by default. New in 0.6.7.

func WithParquetSchema added in v0.6.7 

func WithParquetSchema(s *Schema) ParquetOption

WithParquetSchema reads the files as schema's columns, by name, as Polars' scan_parquet schema: a file's other columns refused unless WithIgnoreExtraColumns, one it lacks refused unless WithInsertMissingColumns, and one of a narrower type refused unless widened. Without it, the files are read as the first file's columns. New in 0.6.7.

func WithPruning 

func WithPruning(b bool) ParquetOption

WithPruning enables or disables row-group skipping from column statistics.

It is on by default. The switch exists because pruning is the one part of the Parquet reader that can change which rows come back, so a wrong answer must be bisectable to it — the same reason plan.Flags can disable each optimizer rule.

func WithUpcastFloats added in v0.6.7 

func WithUpcastFloats(b bool) ParquetOption

WithUpcastFloats reads a Float32 column as the schema's Float64, as Polars' ScanCastOptions(float_cast="upcast"), where it is refused by default. New in 0.6.7.

func WithUpcastIntegers added in v0.6.7 

func WithUpcastIntegers(b bool) ParquetOption

WithUpcastIntegers reads an integer column a file holds in a narrower type than the schema's as the schema's, an Int32 as an Int64, as Polars' ScanCastOptions(integer_cast="upcast"), where it is refused by default. New in 0.6.7.

type ParquetSinkOption 

type ParquetSinkOption interface {
	// contains filtered or unexported methods
}

ParquetSinkOption is anything SinkParquet and WriteParquet accept: a writer option (WithCompression, WithRowGroupRows, WithStatistics) or an execution option (WithMemoryLimit, WithSpillDir, WithBatchSize, WithThreads).

A sink is where a memory limit matters most — it is the consumer that streams, so it is the one a larger-than-RAM query ends in — and before this the two option families could not meet.

It is an interface rather than a variadic `...any` because the union has to be checked at compile time. A library that uses generic methods and an Operand constraint to stop `Col("x").Gt(struct{}{})` from compiling should not then accept SinkParquet(ctx, path, "oops").

type ParquetWriteOption 

type ParquetWriteOption func(*parquet.WriteOptions)

ParquetWriteOption configures the Parquet writer.

func WithCompression 

func WithCompression(c compress.Compression) ParquetWriteOption

WithCompression sets the codec applied to every column. Default Snappy.

func WithRowGroupRows 

func WithRowGroupRows(n int) ParquetWriteOption

WithRowGroupRows sets the target row-group size. It is the unit of both pruning granularity and writer memory: smaller groups prune better and buffer less.

func WithStatistics 

func WithStatistics(b bool) ParquetWriteOption

WithStatistics enables or disables column statistics. On by default — a file without them cannot be pruned, which gives up the main reason to use Parquet.

type Partition added in v0.6.7 

type Partition struct {
	// Key is the key's values: one row, a column for each of By, typed as the frame's,
	// a null key's null. Key.At[string](0, "region") reads one.
	Key *DataFrame
	// Frame is the rows holding the key, in the frame's order: every column, or every
	// column but By with DropKey.
	Frame *DataFrame
}

Partition is one of PartitionBy's parts: the key its rows share, and the rows.

type PartitionOption added in v0.6.8 

type PartitionOption func(*partitionCfg)

PartitionOption makes SinkParquet, SinkCSV, SinkIPC or SinkNDJSON write a hive dataset to its path, a directory, as Polars' PartitionBy does. New in 0.6.8.

func WithMaxRowsPerFile added in v0.6.8 

func WithMaxRowsPerFile(n int) PartitionOption

WithMaxRowsPerFile ends a key's file at n rows and begins its next, numbered after it, as Polars' max_rows_per_file. Default none. New in 0.6.8.

func WithPartitionBy added in v0.6.8 

func WithPartitionBy(keys ...string) PartitionOption

WithPartitionBy writes a hive dataset partitioned by the columns keys: each key's rows to the directory k=v below the path, a directory a key, in the keys' order, in files numbered from 00000000, as Polars writes one. A value is percent-encoded where it holds `/`, `=`, a space, `%`, `:`, a control byte or a byte past ASCII, a null is __HIVE_DEFAULT_PARTITION__, and an instant is its wall clock, 2024-01-02 03:04:05.123456, with its offset where it has a zone; ScanParquet and its kin read the dataset back. New in 0.6.8.

func WithPartitionIncludeKey added in v0.6.8 

func WithPartitionIncludeKey(include bool) PartitionOption

WithPartitionIncludeKey says whether the key columns are written in each file, as well as in its path. Default true, as Polars' include_key. New in 0.6.8.

type PartitionOptions added in v0.6.7 

type PartitionOptions struct {
	// By is the columns whose values key the parts: one or more.
	By []string
	// DropKey leaves By out of each part's Frame, as Polars' include_key=False does;
	// its Key holds them still.
	DropKey bool
}

PartitionOptions says how PartitionBy splits a frame.

type PivotOptions added in v0.6.3 

type PivotOptions struct {
	// On is the columns whose values name the new columns: one, or several, whose
	// values together name one, {"x","p"}, as Polars names it.
	On []string
	// OnColumns are the new columns' On values: a column for each of On, named as
	// it, and a new column for each row, in their order. A row the frame never holds
	// is a column of cells over no values; a value the frame holds and OnColumns
	// does not is left out, its index still a row. LazyFrame.Pivot needs them, its
	// columns known before it runs, as Polars' LazyFrame.pivot needs on_columns;
	// DataFrame.Pivot finds them when nil: every combination the frame holds, in the
	// order each first appears, or sorted with SortColumns.
	OnColumns []*Column
	// Index is the columns whose values make the rows, and Values those spread into
	// the new columns. Either may be left empty: it is then every column that is
	// neither On nor in the other. Both empty is refused.
	Index, Values []string
	// Agg reduces the values that fall in one cell, e.g. Expr.Sum or Expr.First, and
	// reads nothing else, as Polars' reads only pl.element(). A cell none fall in is
	// Agg of none: a count's 0, a sum's null, as every empty sum is here, where
	// Polars' is 0. Without Agg, a cell of two values or more is refused, and one of
	// none is null, as Polars' item has it.
	Agg func(Expr) Expr
	// Separator joins a value column's name and the On values' name, when Values is
	// more than one column: "v_x". "_" when empty.
	Separator string
	// SortColumns orders the columns DataFrame.Pivot finds by their On values,
	// rather than by where each first appears. Given OnColumns keep their order.
	SortColumns bool
}

PivotOptions says how Pivot spreads a frame.

type Queryer added in v0.6.7 

type Queryer interface {
	QueryContext(ctx context.Context, query string, args ...any) (*sql.Rows, error)
}

Queryer is what ReadDatabase runs a query on: a *sql.DB, a *sql.Tx or a *sql.Conn.

type QuoteStyle added in v0.6.7 

type QuoteStyle = csv.QuoteStyle

QuoteStyle is when the CSV writer quotes a field, as Polars' quote_style. New in 0.6.7.

type RankMethod 

type RankMethod = expr.RankMethod

RankMethod decides how Rank breaks ties.

type Roll added in v0.6.6 

type Roll string

Roll answers a value that AddBusinessDays is given on a day that is not a business day: Polars' roll. 

const (
	RollRaise    Roll = expr.RollRaise    // refuse the query, naming the value; the default
	RollForward  Roll = expr.RollForward  // move it to the next business day first
	RollBackward Roll = expr.RollBackward // or to the one before
)

type RollingOptions 

type RollingOptions struct {
	// Period is each row's window's length.
	Period Interval

	// Offset is where a row's window starts, from the row's own instant: the window
	// is (t + Offset, t + Offset + Period], as Polars' rolling offset places it. The
	// zero value is Polars' default, minus Period, (t − Period, t]; NoOffset() is an
	// offset of none, a window wholly after its row, (t, t + Period].
	//
	// A window whose Offset is as long as Period, behind its row, ends at the row
	// itself, as Polars' does, measuring a month as 28 days: so a period of "1mo"
	// with an offset of "-1mo" ends at March 31 for March 31, where February 29 plus
	// a month is March 29. Before step 313, Offset moved the window's end.
	Offset Interval

	// Closed says which end belongs to the window. The zero value is ClosedRight —
	// (end-period, end] — because a rolling window ends AT the row, so closing the
	// left end instead would drop every row from its own window.
	Closed Closed

	// GroupBy adds categorical keys, as in DynamicOptions.
	GroupBy []Expr
}

RollingOptions configures Rolling.

type RoundMode added in v0.6.6 

type RoundMode string

RoundMode is how Round breaks a tie, a value halfway between its two roundings. 

const (
	// RoundHalfAwayFromZero takes a tie away from zero, 2.5 to 3 and -2.5 to -3, as a
	// spreadsheet rounds: ursus's default, and Polars' mode="half_away_from_zero".
	RoundHalfAwayFromZero RoundMode = "half_away_from_zero"
	// RoundHalfToEven takes a tie to its even neighbour, 2.5 to 2 and 3.5 to 4:
	// banker's rounding, IEEE's, and Polars' default.
	RoundHalfToEven RoundMode = "half_to_even"
)

type SQLDialect added in v0.6.8 

type SQLDialect int

SQLDialect names the database WriteDatabase writes to, which database/sql cannot tell: its placeholders, its quoting and its columns' types. 

const (
	// DialectAuto names the dialect by the driver's package: modernc's or mattn's
	// SQLite, pgx or lib/pq, the MySQL driver, or Microsoft's SQL Server driver. A
	// *sql.Tx shows no driver, so writing into one names its dialect.
	DialectAuto SQLDialect = iota
	DialectSQLite
	DialectPostgres
	DialectMySQL
	DialectSQLServer
)

type Schema 

type Schema = dtype.Schema

Schema is an ordered, name-unique sequence of Fields.

func MustSchema 

func MustSchema(fields ...Field) *Schema

MustSchema is NewSchema for tests and package-level vars.

func NewSchema 

func NewSchema(fields ...Field) (*Schema, error)

NewSchema builds a schema, rejecting duplicate column names.

func ReadIPCSchema added in v0.6.7 

func ReadIPCSchema(path string) (*Schema, error)

ReadIPCSchema is an Arrow IPC file's schema, read from its footer, as Polars' read_ipc_schema; no record is read.

func ReadParquetSchema added in v0.6.7 

func ReadParquetSchema(ctx context.Context, path string) (*Schema, error)

ReadParquetSchema is a Parquet file's schema, read from its footer, as Polars' read_parquet_schema.

type SearchSide added in v0.6.6 

type SearchSide string

SearchSide is where SearchSorted places a value among those equal to it. 

const (
	SideAny   SearchSide = "any" // anywhere among its equals: the first, here
	SideLeft  SearchSide = "left"
	SideRight SearchSide = "right"
)

type Series 

type Series[T any] struct {
	*data.Series[T]
}

Series is a typed view over a Column, with the operations a user reaches for on one column, as Polars' Series has them: Sort, Unique, Reverse, FillNull, DropNulls, Cast and the aggregates. It holds no storage of its own: an operation runs the column through the engine as a one-column frame, and gives a Series, or a value, of the answer. Head, Tail, Slice and Rename share the column.

func TypedColumn 

func TypedColumn[T any](c *Column) (*Series[T], error)

TypedColumn returns a typed view over a column.

func (*Series[T]) Cast added in v0.6.7 

func (s *Series[T]) Cast[U any](ctx context.Context, dt DataType) (*Series[U], error)

Cast is the values as the type dt, as Expr.Cast casts them, read as a U: 

f, err := s.Cast[float64](ctx, ursus.Float64)

func (*Series[T]) Count added in v0.6.7 

func (s *Series[T]) Count(ctx context.Context) (int, error)

Count is how many values are not null, and NUnique how many are distinct, a null not counted, as ursus's NUnique counts them.

func (*Series[T]) DropNulls added in v0.6.7 

func (s *Series[T]) DropNulls(ctx context.Context) (*Series[T], error)

DropNulls is the values that are not null.

func (*Series[T]) FillNull added in v0.6.7 

func (s *Series[T]) FillNull(ctx context.Context, strategy FillStrategy) (*Series[T], error)

FillNull is the values with each null filled by a strategy, as Expr.FillNull fills them: FillForward, FillBackward, FillMin and the others.

func (*Series[T]) Head added in v0.6.7 

func (s *Series[T]) Head(n int) *Series[T]

Head is the first n values, Tail the last n, and Slice the length values from offset, a negative offset counting from the end, each sharing the column.

func (*Series[T]) Max added in v0.6.7 

func (s *Series[T]) Max(ctx context.Context) (T, bool, error)

func (*Series[T]) Mean added in v0.6.7 

func (s *Series[T]) Mean(ctx context.Context) (float64, bool, error)

Mean and Median are the values' mean and median, and Std and Var their standard deviation and variance with ddof degrees of freedom, Polars' default 1, as Expr.Std takes them; false when there are too few values.

func (*Series[T]) Median added in v0.6.7 

func (s *Series[T]) Median(ctx context.Context) (float64, bool, error)

func (*Series[T]) Min added in v0.6.7 

func (s *Series[T]) Min(ctx context.Context) (T, bool, error)

Min and Max are the least and the greatest value, false when there is none.

func (*Series[T]) NUnique added in v0.6.7 

func (s *Series[T]) NUnique(ctx context.Context) (int, error)

func (*Series[T]) Rename added in v0.6.7 

func (s *Series[T]) Rename(name string) *Series[T]

Rename is the series under another name.

func (*Series[T]) Reverse added in v0.6.7 

func (s *Series[T]) Reverse(ctx context.Context) (*Series[T], error)

Reverse is the values in the opposite order.

func (*Series[T]) Slice added in v0.6.7 

func (s *Series[T]) Slice(offset, length int) *Series[T]

func (*Series[T]) Sort added in v0.6.7 

func (s *Series[T]) Sort(ctx context.Context) (*Series[T], error)

Sort is the values in ascending order, nulls first, as Polars' Series.sort; SortDesc descending.

func (*Series[T]) SortDesc added in v0.6.7 

func (s *Series[T]) SortDesc(ctx context.Context) (*Series[T], error)

func (*Series[T]) Std added in v0.6.7 

func (s *Series[T]) Std(ctx context.Context, ddof int) (float64, bool, error)

func (*Series[T]) Sum added in v0.6.7 

func (s *Series[T]) Sum(ctx context.Context) (T, error)

Sum is the values' sum, in the series' own type, as Polars' Series.sum gives it: an integer sum past the type is an error, where Polars' wraps.

func (*Series[T]) Tail added in v0.6.7 

func (s *Series[T]) Tail(n int) *Series[T]

func (*Series[T]) ToDummies added in v0.6.7 

func (s *Series[T]) ToDummies(ctx context.Context, o DummiesOptions) (*DataFrame, error)

ToDummies is the series' dummy columns, as DataFrame.ToDummies makes a column's: Polars' Series.to_dummies.

func (*Series[T]) Unique added in v0.6.7 

func (s *Series[T]) Unique(ctx context.Context) (*Series[T], error)

Unique is the distinct values, in the order each first appears; a null is one of them, once.

func (*Series[T]) Var added in v0.6.7 

func (s *Series[T]) Var(ctx context.Context, ddof int) (float64, bool, error)

type SizeUnit added in v0.6.6 

type SizeUnit int64

SizeUnit is a unit for Size. 

const (
	SizeB  SizeUnit = 1
	SizeKB SizeUnit = 1 << 10
	SizeMB SizeUnit = 1 << 20
	SizeGB SizeUnit = 1 << 30
	SizeTB SizeUnit = 1 << 40
)

type SortKey 

type SortKey struct {
	// contains filtered or unexported fields
}

SortKey is one component of an ordering. Build one with Asc or Desc.

func Asc 

func Asc(e Expr) SortKey

Asc orders ascending. Nulls come FIRST by default; call NullsLast to move them.

func Desc 

func Desc(e Expr) SortKey

Desc orders descending, with nulls still FIRST by default.

Null placement is deliberately independent of direction. SQL ties them together — nulls sort as the largest value, so they land last ascending and first descending — which means switching to Desc silently relocates them. Keeping the two orthogonal is more predictable.

func (SortKey) NullsFirst 

func (k SortKey) NullsFirst() SortKey

NullsFirst places nulls at the start of the output.

func (SortKey) NullsLast 

func (k SortKey) NullsLast() SortKey

NullsLast places nulls at the END of the output, whichever direction the key sorts in.

type Span 

type Span interface {
	Interval | time.Duration
}

Span is an interval in either spelling — the calendar-aware Interval, or a plain time.Duration for the absolute case.

The same union shape as Operand, and for the same reason: it lets one method accept both forms with no wrapper at the call site, so `Truncate(time.Hour)` keeps compiling now that `Truncate(Every("1mo"))` is also legal.

type StrExpr 

type StrExpr struct {
	// contains filtered or unexported fields
}

StrExpr is the string namespace: `Col("name").Str().ToLower()`.

A small wrapper struct whose methods return Expr, so chaining continues naturally through it. This is the shape ursus-api.md §4.5 specifies for every namespace, and `.dt` follows it identically.

The literal flag

Contains, Find, CountMatches, Replace and ReplaceAll take `literal bool`, and literal is what you want unless you know otherwise. Go's regexp is RE2: no backtracking, so no catastrophic blowup, but also no JIT — a plain substring test through it costs far more than strings.Contains. Passing literal=true keeps the common case on the fast path, which is why the flag is explicit rather than inferred from whether the pattern looks like a regex.

func (StrExpr) Contains 

func (s StrExpr) Contains[P string | Expr](pattern P, literal bool) Expr

Contains reports whether each value contains pattern.

pattern is a string, or an expression, a pattern a row: a regex compiled once for each distinct pattern in a batch, and an invalid one an error naming its row. A null pattern gives null. The same holds for StartsWith, EndsWith, StripPrefix, StripSuffix and Split, and for Replace's and ReplaceAll's value (step 255).

func (StrExpr) ContainsAny added in v0.6.6 

func (s StrExpr) ContainsAny(patterns []string, asciiCaseInsensitive bool) Expr

ContainsAny reports whether each value contains one of patterns, literally: Polars' contains_any. With asciiCaseInsensitive an ASCII letter matches its other case; other letters keep theirs, as Polars' do.

func (StrExpr) CountMatches 

func (s StrExpr) CountMatches(pattern string, literal bool) Expr

CountMatches counts non-overlapping occurrences. An empty pattern, literal or regex, matches at every rune boundary: len_chars + 1 times.

func (StrExpr) Decode added in v0.6.6 

func (s StrExpr) Decode(enc Encoding, strict bool) Expr

Decode is each value's hex or base64 as Binary: Polars' decode. Text that is not of the encoding is refused with strict, naming its row, and null without it; base64 must be padded, as Polars has it.

func (StrExpr) Encode added in v0.6.6 

func (s StrExpr) Encode(enc Encoding) Expr

Encode is each value's bytes as lowercase hex or padded base64: Polars' encode (step 271).

func (StrExpr) EndsWith 

func (s StrExpr) EndsWith[P string | Expr](suffix P) Expr

func (StrExpr) EscapeRegex added in v0.3.0 

func (s StrExpr) EscapeRegex() Expr

EscapeRegex quotes every regex metacharacter, so the result matches itself literally when used as a pattern.

func (StrExpr) Extract 

func (s StrExpr) Extract(pattern string, group int) Expr

Extract returns capture group n of the first match, or NULL if there is none. Group 0 is the whole match. Regex only: extracting a literal is just Find.

func (StrExpr) ExtractAll added in v0.3.0 

func (s StrExpr) ExtractAll(pattern string) Expr

ExtractAll returns every non-overlapping match of a regex as a List(String).

Regex only, for the reason Extract is: extracting every literal occurrence of a pattern returns that many copies of the pattern.

func (StrExpr) ExtractGroups added in v0.6.6 

func (s StrExpr) ExtractGroups(pattern string) Expr

ExtractGroups is the first match of a regex's capture groups, a Struct of String fields, each named after its group or numbered where it has none, all null where nothing matches: Polars' extract_groups.

func (StrExpr) ExtractMany added in v0.6.6 

func (s StrExpr) ExtractMany(patterns []string, asciiCaseInsensitive, overlapping, leftmost bool) Expr

ExtractMany is each value's matches of patterns, a List(String), and FindMany where they start, in bytes, a List(UInt32): Polars' extract_many and find_many. overlapping finds every match, ordered by where it ends; it takes the default matching, not leftmost, as Polars' does.

func (StrExpr) Find 

func (s StrExpr) Find(pattern string, literal bool) Expr

Find returns the byte offset of the first match, or NULL when there is none.

Null rather than -1: a sentinel index would compare and do arithmetic like a real position, so `find(x) < 5` would be true for "not found".

func (StrExpr) FindMany added in v0.6.6 

func (s StrExpr) FindMany(patterns []string, asciiCaseInsensitive, overlapping, leftmost bool) Expr

func (StrExpr) Head 

func (s StrExpr) Head(n int) Expr

Head and Tail are Slice's common cases.

func (StrExpr) Join added in v0.5.0 

func (s StrExpr) Join(sep string) Expr

Join joins a column's strings into one, with sep between each two and nulls skipped: Polars' str.join. It is an aggregate, so GroupBy().Agg gives one row for the frame, and GroupBy(keys).Agg one per group: 

df.GroupBy(Col("team")).Agg(Col("name").Str().Join(", "))

It is Implode, then List().Join, and its order is the rows': a group-by's rows arrive in the input's order within each group.

func (StrExpr) JsonDecode added in v0.6.6 

func (s StrExpr) JsonDecode(dt DataType) Expr

JsonDecode reads each value's JSON as dt, of Booleans, numbers, Strings, and Lists and Structs of them: Polars' json_decode with its dtype, which ursus does not infer. A Struct's missing fields are null and its extra ones skipped; a float read as an integer is truncated, as Polars casts it. Text that is not JSON, or JSON of another shape than dt, is refused, naming its row.

func (StrExpr) JsonPathMatch added in v0.6.6 

func (s StrExpr) JsonPathMatch(path string) Expr

JsonPathMatch is each value's JSON at path, as text: Polars' json_path_match. The path is $ then .key, ['key'], [n] (from the end where negative) and ..key, the first key of that name at any depth. A string is unquoted and any other value its compact JSON, its keys in their order; JSON null, a path that matches nothing and text that is not JSON are null.

func (StrExpr) LenBytes 

func (s StrExpr) LenBytes() Expr

LenBytes counts bytes; LenChars counts runes. They differ on any non-ASCII input, and which one is wanted is not guessable — so there is no `Len`.

func (StrExpr) LenChars 

func (s StrExpr) LenChars() Expr

func (StrExpr) Normalize added in v0.6.6 

func (s StrExpr) Normalize(form UnicodeForm) Expr

Normalize puts each value in a Unicode normal form: Polars' normalize, NFC its default.

func (StrExpr) PadEnd added in v0.3.0 

func (s StrExpr) PadEnd(width int, fill string) Expr

func (StrExpr) PadStart added in v0.3.0 

func (s StrExpr) PadStart(width int, fill string) Expr

PadStart and PadEnd pad each value to width, counted in RUNES, using the first rune of fill. A value already that wide is returned unchanged — padding never truncates.

func (StrExpr) Replace 

func (s StrExpr) Replace[V string | Expr](pattern string, value V, literal bool) Expr

Replace substitutes the FIRST match; ReplaceAll substitutes every match.

For a regex, the replacement expands $1, ${1} and ${name} in both, as Go's regexp.Expand does — so $1x names a group called "1x", and ${1}x is group 1 then an x. A literal pattern's replacement is literal, $ and all.

value may be an expression, a value a row, which is inserted as it is written: $1 is not expanded, as Polars does not expand it. A null value gives null, where Polars keeps the string as it was. The pattern stays a constant, as Polars' does.

func (StrExpr) ReplaceAll 

func (s StrExpr) ReplaceAll[V string | Expr](pattern string, value V, literal bool) Expr

func (StrExpr) ReplaceMany added in v0.6.6 

func (s StrExpr) ReplaceMany(patterns, replaceWith []string, asciiCaseInsensitive, leftmost bool) Expr

ReplaceMany replaces each of patterns with the replacement at its place, or with one replacement for all, literally: Polars' replace_many. Matches do not overlap. By default the match that ends first is taken, and of two ending together the one that starts first, as Polars' default is; leftmost takes the match that starts first, and of two starting together the pattern listed first.

func (StrExpr) ReplaceN added in v0.6.6 

func (s StrExpr) ReplaceN(pattern, value string, literal bool, n int) Expr

ReplaceN replaces the first n matches of pattern, all where n is negative: Polars' replace(n=). Polars takes n > 1 for a literal pattern only; here a regex takes it too, each replacement expanding $1 as Replace's does.

func (StrExpr) Reverse 

func (s StrExpr) Reverse() Expr

Reverse reverses by RUNE, not by byte.

func (StrExpr) Slice 

func (s StrExpr) Slice(offset, length int) Expr

Slice takes length runes from offset, which may be negative to count from the end. Runes rather than bytes, so it can never split a character in half. The window is clipped as Polars clips it: a start before the first rune cuts the length by as much, so (-7, 2) of "012345" is "0" (step 291).

func (StrExpr) Split added in v0.3.0 

func (s StrExpr) Split[P string | Expr](by P) Expr

Split splits each value on a literal separator, producing a List(String). 

lf.WithColumns(ursus.Col("tags").Str().Split(",").Alias("tag")).
    Explode("tag")

That two-stage form is the canonical use, and it is two stages because Explode changes the frame's height, which an expression may not.

Three cases that are not the same 

"a,b"   ->  ["a", "b"]
""      ->  [""]      one empty element, matching strings.Split
NULL    ->  NULL      a null list, not an empty one

The separator is LITERAL. A regex split would be a different function, and splitting on "." meaning "any character" is the mistake it would invite.

func (StrExpr) SplitExact added in v0.6.6 

func (s StrExpr) SplitExact(by string, n int, inclusive bool) Expr

SplitExact is the first n+1 parts of each value split on by, a Struct of fields field_0 to field_n, what follows them dropped: Polars' split_exact. With inclusive each part keeps the separator after it.

func (StrExpr) SplitInclusive added in v0.6.6 

func (s StrExpr) SplitInclusive(by string) Expr

SplitInclusive is Split whose parts keep the separator after them, Polars' split(inclusive=True): "a,b" is ["a,", "b"], and an empty value no part.

func (StrExpr) SplitN added in v0.3.0 

func (s StrExpr) SplitN(by string, n int) Expr

SplitN is the first n parts of each value split on by, a Struct of fields field_0 to field_{n-1}, the last holding the rest, separators and all: Polars' splitn. A field is null where there are fewer parts, and a null value's fields are null. Before step 270 it was a List; Split(by) is that, and SplitExact drops the rest.

func (StrExpr) StartsWith 

func (s StrExpr) StartsWith[P string | Expr](prefix P) Expr

StartsWith and EndsWith are always literal — a prefix match against a regex is not a well-defined thing to ask for.

func (StrExpr) StripChars 

func (s StrExpr) StripChars(chars string) Expr

StripChars trims any of the given characters from both ends. An empty set trims whitespace, matching Polars.

func (StrExpr) StripCharsEnd added in v0.3.0 

func (s StrExpr) StripCharsEnd(chars string) Expr

func (StrExpr) StripCharsStart added in v0.3.0 

func (s StrExpr) StripCharsStart(chars string) Expr

StripCharsStart and StripCharsEnd are the one-sided forms of StripChars: they remove any leading (or trailing) character in the cutset. An empty cutset means Unicode whitespace, as it does for StripChars.

func (StrExpr) StripPrefix 

func (s StrExpr) StripPrefix[P string | Expr](prefix P) Expr

func (StrExpr) StripSuffix 

func (s StrExpr) StripSuffix[P string | Expr](suffix P) Expr

func (StrExpr) Strptime added in v0.4.0 

func (s StrExpr) Strptime(dt dtype.DataType, format string, strict bool) Expr

Strptime parses each string with a strftime-style format into dt, a Date, Datetime or Time — Polars' str.strptime. 

Col("when").Str().Strptime(ursus.Date, "%d/%m/%Y", true)
Col("ts").Str().Strptime(ursus.Datetime(ursus.Micro, "UTC"), "%Y-%m-%d %H:%M:%S%.f %z", true)

The directives are strftime's: %Y %m %d %H %M %S, %.f for a fraction, %z for an offset, %b and %B for month names, %j, %p, and the rest (internal/strftime lists them). A format that cannot produce dt is refused while the query is planned: a Date needs a date, a Time has none, and only a Datetime holds an offset.

Strict, a value that does not match — or names a day that does not exist, such as 30 February — is an error naming it. Otherwise it becomes null, as ToDate's do.

An empty format is Polars' format=None: each value is read by Polars' patterns, as ToDate, ToDatetime and ToTime read it (step 356).

A Datetime takes its instant from the offset when the format reads one; otherwise the wall clock is read in dt's zone, or kept as it is when dt is naive. Digits finer than the unit are dropped.

func (StrExpr) Tail 

func (s StrExpr) Tail(n int) Expr

func (StrExpr) ToDate 

func (s StrExpr) ToDate() Expr

ToDate parses a date as Polars' str.to_date() with no format reads one (step 356): by Polars' patterns, 2024-01-02, 2024/1/2, 02.01.2024 and 02/01/2024, the year first or the day, never the month; a value that does not parse is null. Each value is read by its own pattern, where Polars infers one from the first. Strptime takes a format, and strictness.

func (StrExpr) ToDatetime 

func (s StrExpr) ToDatetime(unit dtype.TimeUnit, tz string) Expr

ToDatetime parses an instant as Polars' str.to_datetime() with no format reads one (step 356): by Polars' patterns, a date as ToDate's and a time of day after a T or a space, or its midnight. An offset, Z, +09:00 or +0900, makes it the instant; one without is read on tz's wall clock, as Polars reads it, or kept naive where tz is empty. It was a cast, which took such a value as UTC's. A value that does not parse, or a wall clock tz skips or reads twice, is null; Strptime takes a format, and strictness.

func (StrExpr) ToDecimal added in v0.6.7 

func (s StrExpr) ToDecimal(scale int) Expr

ToDecimal parses a number into a Decimal(38, scale), as Polars' str.to_decimal (step 325): rounded half away from zero to the scale, an exponent read, 1e3 a thousand; text that is no number, or past 38 digits, is null.

func (StrExpr) ToInteger 

func (s StrExpr) ToInteger() Expr

ToInteger parses an integer, an ordinary cast: a value that does not parse is null, as is an integer past Int64's range. It parses exactly at 64 bits: "9007199254740993" is itself, not the float64 beside it.

func (StrExpr) ToIntegerWith added in v0.6.6 

func (s StrExpr) ToIntegerWith[B int | Expr](base B, dt DataType, strict bool) Expr

ToIntegerWith reads each value as an integer in base, a constant or an expression, 2 to 36, as dt: Polars' to_integer(base, dtype, strict). A sign is read and nothing else but digits: no space, 0x or underscore. A value that does not read, or does not fit dt, is refused with strict and null without it.

func (StrExpr) ToLower 

func (s StrExpr) ToLower() Expr

func (StrExpr) ToTime added in v0.6.7 

func (s StrExpr) ToTime(format string, strict bool) Expr

ToTime parses a time of day, a Time of nanoseconds, as Polars' str.to_time (step 325): by format, as Strptime reads one, or, with none, by Polars' patterns, "10:30", "3:04:05" and "10:30:15.123456789" alike (step 356), each value by its own where Polars infers one format from the first. Strict, a value that does not parse is an error; otherwise it is null.

func (StrExpr) ToTitlecase added in v0.6.6 

func (s StrExpr) ToTitlecase() Expr

ToTitlecase upper-cases each letter that follows a character that is not a letter, and lower-cases the rest: Polars' to_titlecase, "3rd place" "3Rd Place".

func (StrExpr) ToUpper 

func (s StrExpr) ToUpper() Expr

func (StrExpr) ZFill added in v0.3.0 

func (s StrExpr) ZFill(width int) Expr

ZFill left-pads with zeros to width, keeping any leading sign in front of them: -5 at width 4 is "-005", not "0-05". That is what makes it different from PadStart with a '0', and it is the reason it is a separate function.

type StructExpr added in v0.3.0 

type StructExpr struct {
	// contains filtered or unexported fields
}

StructExpr is the struct namespace: `Col("person").Struct().Field("age")`.

The same wrapper shape `.str`, `.dt` and `.list` use, for the reason ursus-api.md §4.5 gives: methods return Expr, so chaining continues naturally through it.

A null struct is not a struct of nulls

Both hold no field values, and only the struct's own validity separates them: 

                          Field("age")   renders as
{age: 30, city: "NY"}          30        {age: 30, city: "NY"}
{age: null, city: null}       null       {age: null, city: null}
null (the struct)             null       null

The last two rows answer identically through Field — there is no age either way — and differ where it matters, on the struct itself. Working from "a struct with nothing in it is absent" gives the wrong answer for a real struct that happens to hold nulls, and nothing errors.

func (StructExpr) Drop added in v0.6.7 

func (s StructExpr) Drop(names ...string) Expr

Drop is the struct without the named fields, the others in their order, as Polars' struct.drop (step 323): a name that is no field is refused, as Polars' default, strict, refuses it. A null struct stays null. Dropping every field is refused, as a struct of no fields has no length of its own; Polars makes one.

func (StructExpr) Field added in v0.3.0 

func (s StructExpr) Field(names ...string) Expr

Field reads one field out of each struct.

The result is an ordinary column of the field's own type, so everything else in the language applies to it: filter on it, group by it, join on it.

A name the struct does not have is refused when the query is PLANNED, with the available names listed — the type of the result depends on which field was asked for, so the name has to be resolved against the schema anyway.

Of several names, or of a regex between ^ and $, or "*", it is several columns, a field each, as Polars' struct.field of several names is (step 275): written out as Unnest is, in the struct's order, each field once, each named after its field. Like a selection of several columns it may be the whole of an expression or a part, Field("a", "b").Mul(2) each field doubled. A regex may match no field.

func (StructExpr) JsonEncode added in v0.6.3 

func (s StructExpr) JsonEncode() Expr

JsonEncode writes each struct as a JSON object, its fields in order, as Polars' struct.json_encode: a null is null, a Bool and a number as JSON has them, a float the shortest decimal that reads back to it with NaN and the infinities null, a String quoted and escaped, a List an array and a Struct an object. A Decimal, a Date, a Time, a Datetime, a Duration and an Enum are strings of the text a Cast to String gives them, the one the CSV writer writes; Polars writes a Datetime with no zone as "2024-01-02 03:04:05" and a Duration in ISO 8601, which this does not. A null struct is a null String, where Polars writes "null". A Binary is refused.

func (StructExpr) RenameFields added in v0.6.3 

func (s StructExpr) RenameFields(names ...string) Expr

RenameFields gives each field of the struct a new name, in order, as Polars' struct.rename_fields does. Every field must be named, and each differently: Polars renames as many as it is given and drops the rest, and warns that 2.0 will refuse it, so this refuses it now. A null struct stays null.

func (StructExpr) Unnest added in v0.6.5 

func (s StructExpr) Unnest() Expr

Unnest is the struct's fields, a column each: Expr.Unnest, spelled as Polars spells it, struct.unnest() (step 260).

func (StructExpr) WithFields added in v0.6.0 

func (s StructExpr) WithFields(exprs ...Expr) Expr

WithFields sets fields of each struct: an expression named as one of its fields replaces that field in place, and one of a new name is added after the others, each named as its expression is. Polars' struct.with_fields. 

Col("person").Struct().WithFields(
	Col("person").Struct().Field("age").Add(1).Alias("age"),  // replaces age
	Col("city"),                                              // adds city
)

Each expression is read from the frame, so a field of the struct is read as it is anywhere, through Field, and named as any expression is: a Field read is named after its struct, so alias it to set a field of that name. The struct's own validity is kept: a null struct stays null. Two expressions of one name would set one field twice, and are refused.

type ThenBuilder 

type ThenBuilder struct {
	// contains filtered or unexported fields
}

ThenBuilder is a conditional that may take another When or be closed by Otherwise.

func (ThenBuilder) Otherwise 

func (t ThenBuilder) Otherwise[T Operand](v T) Expr

Otherwise supplies the fall-through value and closes the chain.

The chain becomes a right-nested tree of conditionals, in the order written: when(c1).then(v1).when(c2).then(v2).otherwise(d) is Cond{c1, v1, Cond{c2, v2, d}}. Nesting rather than a flat list is what keeps type unification well-defined — promotion is pairwise, and folding a flat list would make the result depend on an association order the user never chose.

func (ThenBuilder) When 

func (t ThenBuilder) When(pred Expr) WhenBuilder

When adds another condition, the else-if of the chain.

type TimeUnit 

type TimeUnit = dtype.TimeUnit

TimeUnit is the resolution of a Time, Datetime or Duration.

type TransposeOptions added in v0.6.3 

type TransposeOptions struct {
	// IncludeHeader adds a first column of the frame's column names, named
	// HeaderName, "column" when empty.
	IncludeHeader bool
	HeaderName    string
	// ColumnNames names the new columns, one for each row; "column_0" and on when
	// empty.
	ColumnNames []string

	// ColumnNamesFrom names the new columns by a text column's values, which must
	// not be null, and leaves that column out: Polars' column_names given a
	// column's name (step 320). It is not given with ColumnNames.
	ColumnNamesFrom string
}

TransposeOptions says how Transpose names what it makes.

type UnicodeForm added in v0.6.6 

type UnicodeForm string

UnicodeForm is a Unicode normal form for Normalize. 

const (
	NFC  UnicodeForm = "NFC"
	NFD  UnicodeForm = "NFD"
	NFKC UnicodeForm = "NFKC"
	NFKD UnicodeForm = "NFKD"
)

type UnpivotOptions added in v0.3.0 

type UnpivotOptions struct {
	// On names the value columns to melt into the variable/value pair. Empty means
	// every column that is not in Index, as Polars' on=None: every column, when
	// Index is empty too.
	On []string

	// Index names the columns carried through unchanged. Empty means every column
	// that is not in On, where Polars' keeps none.
	Index []string

	// VariableName and ValueName name the two invented columns. Empty means
	// "variable" and "value", which are Polars' names.
	VariableName string
	ValueName    string
}

UnpivotOptions configures Unpivot.

Every field has a working default, so the common calls are UnpivotOptions{On: []string{...}} and UnpivotOptions{Index: []string{...}}.

type UnstackHow added in v0.6.7 

type UnstackHow uint8

UnstackHow is the direction Unstack lays a column's values out in. 

const (
	// UnstackVertical fills each new column with Step values in turn, Polars'
	// default.
	UnstackVertical UnstackHow = iota
	// UnstackHorizontal fills each new row with Step values in turn, across Step new
	// columns.
	UnstackHorizontal
)

type UnstackOptions added in v0.6.7 

type UnstackOptions struct {
	// Step is how many values a new column holds, vertically, or a new row,
	// horizontally.
	Step int
	How  UnstackHow

	// Columns are the columns to unstack, every column when empty; the others are
	// left out, as in Polars.
	Columns []string

	// Fill fills the slots after the last value: one value for every column, or
	// one for each of Columns, cast to its type as ExtendConstant casts. They are
	// null when it is empty.
	Fill []any
}

UnstackOptions configures Unstack.

type UpdateHow added in v0.6.7 

type UpdateHow uint8

UpdateHow is which rows an Update keeps. 

const (
	// UpdateLeft keeps every row of the frame, Polars' default.
	UpdateLeft UpdateHow = iota
	// UpdateInner keeps only the rows other matches.
	UpdateInner
	// UpdateFull keeps every row of the frame, and adds other's unmatched rows after
	// them, as new rows: null in the columns other does not have.
	UpdateFull
)

type UpdateOptions added in v0.6.7 

type UpdateOptions struct {
	// On names the key columns, alike on both frames. With On, LeftOn and RightOn all
	// empty, rows are matched by their places: the frame's first row with other's
	// first, as Polars' row count is.
	On []string

	// LeftOn and RightOn name keys named apart, paired by position.
	LeftOn, RightOn []string

	// How is which rows are kept: UpdateLeft, the zero value, UpdateInner or
	// UpdateFull.
	How UpdateHow

	// IncludeNulls writes other's nulls over the frame's values; by default a null in
	// other leaves the value.
	IncludeNulls bool

	// MaintainOrder is the rows' order. JoinOrderNone, the zero value, is the
	// frame's, as JoinOrderLeft is: Polars' default for update. JoinOrderRight is
	// other's, each of its rows' matches in the frame's order, and then the frame's
	// unmatched rows.
	MaintainOrder JoinOrder
}

UpdateOptions configures Update.

type WhenBuilder 

type WhenBuilder struct {
	// contains filtered or unexported fields
}

WhenBuilder is a conditional awaiting its Then.

func When 

func When(pred Expr) WhenBuilder

When begins a conditional. 

ursus.When(ursus.Col("score").Ge(90)).Then("A").
    When(ursus.Col("score").Ge(80)).Then("B").
    Otherwise("F").Alias("grade")

Why a builder and not a function

The alternative was a variadic Case(cond1, val1, cond2, val2, …, default). That reads compactly and pairs its arguments POSITIONALLY, so a miscounted list is a runtime error rather than a type error — and the compiler cannot tell a condition from a value when both are Expr. The builder makes each pair a method call, so there is nothing to miscount.

An unterminated chain cannot be used, by construction

Only Otherwise returns an Expr; WhenBuilder and ThenBuilder are not expressions and satisfy nothing that takes one. So a chain that forgets its else branch fails to compile rather than silently defaulting. Where a null default IS wanted, say so: Otherwise(ursus.Null(ursus.NullT)).

A null condition is false

A row whose condition is null takes the next branch, the next When or Otherwise, as Polars' when does and SQL's CASE WHEN: When(Col("b")).Then(1).Otherwise(0) of a null b is 0. Until step 311 it was null.

Both branches are evaluated

This is a columnar engine, so Then and Otherwise are each computed for every row and then merged. That is unobservable — expressions have no side effects, and an arithmetic fault such as division by zero yields NULL rather than trapping — but it does mean a conditional does not make an expensive branch cheaper.

func (WhenBuilder) Then 

func (w WhenBuilder) Then[T Operand](v T) ThenBuilder

Then supplies the value for the pending condition. The value lifts, so Then("A") needs no Lit.

type WindowOption added in v0.5.0 

type WindowOption func(*expr.WinParams)

WindowOption configures a rolling or an exponentially weighted function. An option a function has no use for, Ddof on an EwmMean, does nothing.

func Biased added in v0.5.0 

func Biased() WindowOption

Biased makes EwmVar and EwmStd the biased estimate, Polars' bias=True. By default they are corrected for the weights, and a first value's is null.

func Ddof added in v0.5.0 

func Ddof(d int) WindowOption

Ddof sets RollingVar's, RollingStd's and RollingCov's delta degrees of freedom; the default, 1, is the sample variance, as Var(1) and Polars' rolling_var.

func EwmAdjust added in v0.5.0 

func EwmAdjust(adjust bool) WindowOption

EwmAdjust chooses how the first rows are weighed; true is the default, as Polars' adjust=True. Adjusted, each row's answer divides by the weights its rows had; unadjusted, the newest row weighs alpha and the running answer 1-alpha.

func IgnoreNulls added in v0.5.0 

func IgnoreNulls() WindowOption

IgnoreNulls weighs an ewm by position among the values, so a null does not age the rows before it. By default it does, as a row would: Polars' ignore_nulls=False.

func InterpolateNearest added in v0.6.0 

func InterpolateNearest() WindowOption

InterpolateNearest makes Interpolate fill a run of nulls with the nearer of the two values around it, the later on a tie, as Polars' method="nearest" (step 172). The column's own type is kept, a number's or an instant's.

func MinSamples added in v0.5.0 

func MinSamples(k int) WindowOption

MinSamples lets a rolling window answer from k values, where by default it needs as many as it has rows: RollingMean(7, MinSamples(1)) averages what the first six rows have. k is between 1 and the window's size.

func RollingCenter added in v0.6.0 

func RollingCenter() WindowOption

RollingCenter labels a rolling window at its middle row rather than its last, as Polars' center=True (step 172): a window of size rows covers size/2 rows before the row and (size-1)/2 after it. A window that runs past either end of the rows holds fewer values, and is null unless MinSamples allows it.

func WindowClosed added in v0.6.0 

func WindowClosed(c Closed) WindowOption

WindowClosed chooses which ends of a rolling window by time hold their instant: ClosedRight, the default, is (t − period, t], as Polars' closed="right"; ClosedLeft is [t − period, t), ClosedBoth both ends and ClosedNone neither. A rolling window of N rows has no use for it.

type WindowSpec 

type WindowSpec struct {
	PartitionBy []Expr
	OrderBy     []SortKey
	Mapping     MappingStrategy
}

WindowSpec is the full form of a window, for OverWith.

type ZoneOption added in v0.6.0 

type ZoneOption interface {
	// contains filtered or unexported methods
}

ZoneOption is a policy for ReplaceTimeZone: an Ambiguous or a NonExistent.

Source Files

View all Source files

Directories

Path Synopsis
docs
examples
Package examples is the documentation site's code.
 Package examples is the documentation site's code.
Package dtype is ursus's type system: DataType, Field and Schema.
 Package dtype is ursus's type system: DataType, Field and Schema.
Package i128 provides a signed 128-bit integer.
 Package i128 provides a signed 128-bit integer.
internal
arrowin
Package arrowin imports Arrow data into ursus, by copying it.
 Package arrowin imports Arrow data into ursus, by copying it.
arrowout
Package arrowout exports an ursus batch as Arrow, which is how anything outside this module reads ursus's output without a copy.
 Package arrowout exports an ursus batch as Arrow, which is how anything outside this module reads ursus's output without a copy.
arrowx
Package arrowx owns ursus's allocator and the few Arrow facts shared across packages.
 Package arrowx owns ursus's allocator and the few Arrow facts shared across packages.
bitmap
Package bitmap is ursus's validity-bitmap layer.
 Package bitmap is ursus's validity-bitmap layer.
data
Package data holds ursus's runtime column representation.
 Package data holds ursus's runtime column representation.
exec
Package exec drives a physical operator tree to completion.
 Package exec drives a physical operator tree to completion.
execopt
Package execopt carries the execution-time knobs that are not planning decisions: how much memory one query may hold, and where it is allowed to spill.
 Package execopt carries the execution-time knobs that are not planning decisions: how much memory one query may hold, and where it is allowed to spill.
expr
Package expr is ursus's expression IR.
 Package expr is ursus's expression IR.
exprbridge
Package exprbridge lets a package beside the root one build the root's expressions, package selector (step 322): the root sets Wrap when it initialises, so the root's API names no internal type.
 Package exprbridge lets a package beside the root one build the root's expressions, package selector (step 322): the root sets Wrap when it initialises, so the root's API names no internal type.
gen/docsite command
Command docsite writes the fragments the documentation site includes (context_files/docs-site-scope.md §2).
 Command docsite writes the fragments the documentation site includes (context_files/docs-site-scope.md §2).
gen/levels command
Command levels enforces ursus's import-level invariant:
 Command levels enforces ursus's import-level invariant:
kernel
Package kernel is ursus's compute layer.
 Package kernel is ursus's compute layer.
memcheck
Package memcheck holds tests, and nothing else, that measure a query's live heap against what its memory budget counts.
 Package memcheck holds tests, and nothing else, that measure a query's live heap against what its memory budget counts.
physical
Package physical turns a logical plan into runnable operators, and expressions into columns.
 Package physical turns a logical plan into runnable operators, and expressions into columns.
plan
Package plan is ursus's logical plan IR.
 Package plan is ursus's logical plan IR.
source
Package source is the runtime half of ursus's scan contract.
 Package source is the runtime half of ursus's scan contract.
source/arrowsrc
Package arrowsrc scans a stream of Arrow records: an IPC stream, a Flight response, a C Data Interface stream — anything that is an array.RecordReader.
 Package arrowsrc scans a stream of Arrow records: an IPC stream, a Flight response, a C Data Interface stream — anything that is an array.RecordReader.
source/csv
Package csv reads and writes delimited text files.
 Package csv reads and writes delimited text files.
source/files
Package files reads several files of one format as one scan, in order, each file by a source of its own, with columns every row of a file shares after the files' own: a hive dataset's partition keys, and the file's path (step 360).
 Package files reads several files of one format as one scan, in order, each file by a source of its own, with columns every row of a file shares after the files' own: a hive dataset's partition keys, and the file's path (step 360).
source/hive
Package hive reads a dataset's partition columns from its files' paths, as Hive lays a dataset out and Polars reads one: each directory below the dataset's root named key=value gives every file beneath it a column key of value (step 347).
 Package hive reads a dataset's partition columns from its files' paths, as Hive lays a dataset out and Polars reads one: each directory below the dataset's root named key=value gives every file beneath it a column key of value (step 347).
source/memsrc
Package memsrc is an in-memory scan source.
 Package memsrc is an in-memory scan source.
source/ndjson
Package ndjson reads newline-delimited JSON (step 334): Polars' read_ndjson and scan_ndjson, a JSON object a line, a column a key.
 Package ndjson reads newline-delimited JSON (step 334): Polars' read_ndjson and scan_ndjson, a JSON object a line, a column a key.
source/parquet
Package parquet reads and writes Apache Parquet files.
 Package parquet reads and writes Apache Parquet files.
source/sqlrows
Package sqlrows reads a database/sql result into a batch: Polars' read_database (step 353).
 Package sqlrows reads a database/sql result into a batch: Polars' read_database (step 353).
source/testsrc
Package testsrc is a scan source that lies about its capabilities on purpose.
 Package testsrc is a scan source that lies about its capabilities on purpose.
spill
Package spill writes batches to a file and reads them back exactly.
 Package spill writes batches to a file and reads them back exactly.
strftime
Package strftime parses and formats instants with strftime-style formats, the ones Polars, chrono, Python and C share: "%Y-%m-%d %H:%M:%S".
 Package strftime parses and formats instants with strftime-style formats, the ones Polars, chrono, Python and C share: "%Y-%m-%d %H:%M:%S".
uerr
Package uerr is ursus's error type.
 Package uerr is ursus's error type.
Package selector picks a frame's columns by what they are, as Polars' selectors do: by type, by name, by place, and the unions, intersections, differences and complements of those (step 322).
 Package selector picks a frame's columns by what they are, as Polars' selectors do: by type, by name, by place, and the unions, intersections, differences and complements of those (step 322).
Package ursustest provides assertions for testing code that uses ursus.
 Package ursustest provides assertions for testing code that uses ursus.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.