Help us shape the future of Go! Take the 2026 Go Developer Survey

ursus

package module
v0.3.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 6, 2026 License: MIT Imports: 37 Imported by: 0

README

ursus

A (experimental) dataframe library for Go 1.27, modelled on Polars — lazy execution with a query optimizer, Arrow memory layout, SIMD kernels, and streaming execution that spills to disk rather than falling over. Sort, group-by, join, unique and partitioned windows spill — by default once a query holds half the memory its container or machine allows — and an operator that cannot spill fails with an error naming itself.

df, err := ursus.ScanParquet("events.parquet").
    Filter(ursus.Col("price").Gt(5)).
    GroupBy(ursus.Col("region")).
    Agg(
        ursus.Col("price").Sum().Alias("revenue"),
        ursus.Col("qty").Mean().Alias("avg_qty"),
    ).
    Sort(ursus.Desc(ursus.Col("revenue"))).
    Collect(ctx)

The projection reaches the Parquet reader, the filter becomes a row-group predicate, and the group-by runs on every core. None of that is visible in the query.

Is this for you?

It is pure Go. No cgo, no C++ toolchain, no Python runtime, no sidecar process. It cross-compiles and links into a static binary like any other Go dependency, and go get is the whole install.

That is the reason to pick it, and the cost should be just as plain: ursus is slower than the serious analytical engines. On the h2o.ai benchmark at ten million rows it is roughly 3x Polars; on TPC-H at scale factor 1 it is about 10x. CSV parsing is worse than that. It is not trying to beat Polars or DuckDB, and on current evidence it is not going to.

Where that trade is a good one

  • Data that fits comfortably in memory — up to tens of millions of rows, where the difference is half a second against a tenth of a second and nobody is waiting.
  • A Go service that needs real dataframe work — joins, group-bys, window functions — and cannot take on cgo, a Python sidecar, or a large C++ dependency to get it.
  • Anywhere the deployment story is worth more than the last multiple of speed.

Where it is not

  • Interactive analytics over hundreds of millions of rows. Use DuckDB or Polars.
  • Anywhere you can link cgo freely: duckdb-go is around 10x faster than this and is a binding to a mature engine.
  • Anything production-critical today. This is v0.3, the API still moves, and nothing here is promised.

The honest summary is that Go has not had a dataframe library of this shape, and one that is correct and a few times slower is more useful than none — for the sizes most services actually handle.

Docs

The API reference is on pkg.go.dev, generated from the source — nothing to host and nothing to keep in sync.

Start with the runnable examples: filter, group-by, join, computed columns, whole-frame aggregation, and reading typed values back out — and, new in 0.3, reading Parquet through any io.ReaderAt, a Go UDF, lists with Split and Explode, JoinWhere, exact Decimal arithmetic, an Enum, a spilling Unique, and a List column written to Parquet. They render beside the methods they document, and go test checks each one's output, so an example that stops being true breaks the build rather than misleading someone quietly.

Beyond that, the doc comments are the documentation. They are unusually long on purpose: each explains why a thing is the way it is, not only what it does.

Install

go get github.com/advenn/ursus

Go 1.27 or newer is required. Two of its changes are load-bearing: generic methods are why Series[T].Map[U] and df.Column[T](name) exist at all, and because a generic method still cannot satisfy an interface, every public type is a concrete struct with polymorphism kept in unexported interfaces.

GOEXPERIMENT=simd is optional. It switches on the SIMD kernels:

GOEXPERIMENT=simd go build ./...

Without it every kernel falls back to its scalar twin, behind //go:build !(goexperiment.simd && amd64). That is not a claim — CI runs the entire suite with the experiment off on every push, and make test-all includes an experiment-off leg locally. The flag buys speed, not correctness.

Status

v0.3 — what changed since v0.2 is in CHANGELOG.md. What works today:
Sources Parquet and CSV (read and write, List and Struct included for Parquet), from paths, memory, or any io.ReaderAt / stream you open (ScanParquetFrom, ScanCSVFrom); in-memory frames; Arrow (records and streams in; records out, zero-copy)
Types Bool, Int8–64, Uint8–64, Float32/64, String, Binary, Date, Time, Datetime (unit + zone), Duration, Decimal (128-bit; exact + - * and sum, refused past 38 digits; / to the nearest Float64; casts to and from every numeric type exact or refused, except to a float, which rounds to the nearest), Enum (built from String, ordered by its categories)
Expressions arithmetic, comparison, Kleene three-valued logic, conditionals, casts, null repair, .str (incl. Split → List) and .dt namespaces, 20 aggregates including Implode, window functions
Frame ops filter, select, with-columns, sort, top-k, distinct, concat/vstack/hstack, slice/tail/reverse/row-index, drop/rename/drop-nulls, unpivot
Joins all seven equi-join kinds with Validate, JoinWhere and WhereExists/WhereNotExists (non-equi), as-of join with tolerance and by keys, merge-sorted
Grouping group-by, GroupByDynamic, Rolling, calendar-aware intervals
Optimizer predicate pushdown (including through joins), projection pushdown, limit/top-k pushdown, cross-join collapse, constant folding and simplification
Execution order-preserving pipeline parallelism, parallel hash aggregation, and spilling for sort, hash aggregation, hash join, unique and partitioned windows (WithMemoryLimit names the rest, which fail rather than spill)
UDFs MapElements (per value) and MapBatches (per column) — generic methods, so the Go types are inferred from your function

Nested types are partly there: List and Struct read from Parquet and Arrow and write to Parquet, with Explode, Unnest, a .list namespace and .struct.field(). An Arrow Map arrives as a list of key/value structs. Array is not there, and a List of Lists or of Structs is refused by name on both sides.

Arrow goes both ways, in pure Go. df.Record() and lf.CollectRecords(ctx) export without copying; ScanArrow reads any array.RecordReader — an IPC stream, Flight, the C Data Interface — and ScanArrowRecords reads records already in memory. Import copies, so nothing you release afterwards can reach a frame:

lf := ursus.ScanArrow(func() (array.RecordReader, error) {
    return ipc.NewReader(bytes.NewReader(stream)) // a new reader every call
})

The function is called once to plan the query and once per scan, because a reader cannot be rewound. 38 of Arrow's 45 types map; Decimal256, intervals, unions and run-end encoding are refused by name.

Object storage has no client built in: each store wants its vendor's SDK, and ursus is pure Go with one dependency. ScanParquetFrom reads Parquet through any io.ReaderAt instead, so a ReadAt over ranged GETs reaches S3, GCS or an HTTP server — and ursus reads the footer and then only the column chunks the query needs, after statistics have pruned the row groups it does not:

lf := ursus.ScanParquetFrom([]ursus.ParquetFile{{
    Name: "s3://logs/2026/10/part-0.parquet",
    Open: func(ctx context.Context) (io.ReaderAt, int64, error) {
        return openRanged(ctx, client, "logs", "2026/10/part-0.parquet") // your ReadAt over GetObject with a Range
    },
}})

ScanCSVFrom takes a stream the same way, and WriteParquet and WriteCSV write to any io.Writer, an upload included. Built-in stores — s3:// paths, listing for globs, retries — are planned for 0.4.

The escape hatch is real: a per-element UDF in Go is a function call, not a Python interpreter round trip, which is the one place this library can beat Polars outright rather than merely keep up.

lf.Select(ursus.Col("celsius").MapElements("to_fahrenheit", ursus.Float64,
    func(c float64) (float64, error) { return c*9/5 + 32, nil }))

Nulls pass through untouched, so your function never receives a zero value it cannot tell from a real one; use MapBatches when the null is the point. The name is required, and Explain shows it. Your function is called from several goroutines at once, so it must be safe for that.

Not done: built-in object stores (planned for 0.4; the seam above reaches one today), MapGroups and RollingMap, common subexpression elimination, SQL, Pivot — whose output columns are the distinct values of a column, so its schema would depend on data and no plan node here does; Unpivot (melt) ships — and the long tail of Expr.Rolling*, Upsample, Interpolate and the trigonometric block.

Version numbers follow Go's own rule for v0: nothing is promised. The API is still moving, and the preamble above says why.

Correctness

make test-all   # four SIMD widths (512/256/128/0) plus the experiment off
make race       # the whole suite under -race
make levels     # import-level invariants

2967 test cases, and the matrix is not decoration. Vector width is a runtime property, so a single-width run proves very little: 512-bit gives 8 float64 lanes, which happens to be exactly one bitmap byte — a coincidence that hides an entire class of sub-byte bitmap bug. The 128-bit leg is where those surface.

Correctness is also checked against other engines. The benchmark suite validates every result against a duckdb reference, and ursus currently passes 22/22 PDS-H (TPC-H) queries and 15/15 h2o.ai queries — PDS-H at scale factor 0.1, re-run at every step, and both suites at the published sizes in the v0.3.0 report; every answer ursus gave there validates.

Benchmarks

Everything that measures ursus lives in bench/ — three tiers behind one Makefile, competing against duckdb, polars, pandas, DataFusion, chDB, Arrow-Go, Gota and QFrame.

cd bench
make preflight    # refuses to run on a machine without the free memory; that is the feature
make setup
make bench        # gen -> reference answers -> run -> validate -> report

See bench/README.md for what is timed and why.

bench/results/REPORT.md is checked in. Every result in it is validated against a duckdb reference, and a disagreement is struck through rather than quietly reported as a fast number.

How current it is, precisely. Every table was measured in one session, on one commit, with every engine re-run together — so the numbers are comparable across engines rather than stitched from different days. That commit is v0.3.0's, measured on 2026-10-06.

The caveats that apply to this run:

  • Every query ran in an 8 GB cgroup, on a laptop with 6–9 GiB of swap already in use.
  • ursus's h2o gb10 over CSV was killed at that cap. It needs about 9 GB, and v0.3.0 spilled only under an explicit WithMemoryLimit, so it never saw the cgroup.
  • PDS-H q7 regressed to 3.9 s and 1.68 GB, from 2.4 s and 0.60 GB at step 40. Its date filter stopped being pushed below its joins.
  • Both are fixed on master, in CHANGELOG.md's Unreleased section. Re-run under the same cap, ursus passes 15/15 on h2o over CSV, q7 takes 1.7 s and 0.63 GB, and the h2o Parquet geomean is 1,950 ms.
  • Other engines: chDB timed out on two group-bys and gota ran out of memory on two, under the same cap.

Timings come from a laptop under real conditions, so treat small differences as noise and the ordering as the signal.

Where it stands, in one place. Against Polars, on the current report:

ursus Polars
h2o.ai, 10M rows 2,426 ms 731 ms 3.3x
TPC-H SF=1 918 ms 85 ms 10.8x
TPC-H SF=0.1 82 ms 11 ms 7.5x
TPC-H SF=1 peak memory 1.68 GB 0.86 GB 2.0x

Geomeans over queries every engine passed. The gap narrows as the data gets smaller, which is the shape of the trade described at the top of this file.

Where ursus is behind and why is recorded rather than glossed — every step of construction has an as-built document in context_files/, including the optimisations that were tried and reverted for being slower.

How it is built

dtype/          types, schemas, the null contract
i128/           128-bit integers, for Decimal
internal/
  data/         Column and Batch — Arrow layout, three payload shapes
  bitmap/       validity and boolean bitmaps, offset-correct by construction
  kernel/       the compute kernels, SIMD with scalar twins
  expr/         the expression IR
  plan/         logical plan, resolution, the optimizer rules
  physical/     operators, the sink/breaker protocol, spilling
  exec/         the driver
  source/       parquet, csv, memory
ursustest/      assertions for testing code that uses ursus

Packages are arranged in strict import levels — internal/gen/levels fails the build if a package imports one at or above its own level, which is what keeps the dependency graph a DAG rather than a suggestion.

Design notes

context_files/ holds the design documents and an as-built record for every step of construction, each written to be authoritative over the ones before it. They are unusually candid: they record the defects found, the measurements taken, the tests that turned out to be vacuous, and the claims that did not survive contact with the code. Start with the newest.

Licence

MIT.

Documentation

Overview

Package ursus is a dataframe library for Go modelled on Polars: an expression DSL, lazy execution with a query optimizer, Arrow memory, and SIMD kernels.

Runnable examples for the common tasks — filter, group-by, join, reading values back out — are in example_test.go and render beside each method on pkg.go.dev.

A tour 

df, err := ursus.Scan(src).
	Filter(ursus.Col("price").Gt(5)).
	Select(ursus.Col("id"), ursus.Col("price")).
	Collect(ctx)

Nothing runs until Collect. In between, ursus resolves the schema, expands any multi-column selections, type-checks every expression, and pushes the projection down into the scan so only the columns actually used are ever read.

The design in ten lines

  1. Lazy is the real API. Eager helpers are thin wrappers over it.
  2. Public types are concrete structs — Go forbids generic methods on interfaces, and generic methods are the point.
  3. Errors are deferred and sticky, surfaced at Collect, so chaining stays clean.
  4. context.Context appears at execution boundaries only, never in builders.
  5. Named methods, never operator tricks: a.Gt(5), not a > 5.
  6. Functional options where the parameter surface is wide.
  7. Generics at the boundary where Go values meet columns, not in the engine.
  8. Iterators for streaming output.
  9. No row index: row order is a property, not a label space.
  10. Null is not NaN, and both are handled explicitly everywhere.

Building

ursus requires Go 1.27 and GOEXPERIMENT=simd. Use the Makefile, or export the variable yourself — package simd does not compile without it.

Example

The shortest useful thing: build a frame and look at it. 

package main

import (
	"context"
	"fmt"
	"log"

	"github.com/advenn/ursus"
)

// exampleSales is the little frame the examples below work on.
func exampleSales() *ursus.LazyFrame {
	return ursus.Frame(
		ursus.Values("region", []string{"north", "south", "north", "east", "south"}),
		ursus.Values("product", []string{"apple", "apple", "pear", "pear", "fig"}),
		ursus.Values("qty", []int64{10, 4, 7, 2, 9}),
		ursus.Values("price", []float64{1.50, 1.50, 2.25, 2.25, 3.00}),
	)
}

func main() {
	df, err := exampleSales().Collect(context.Background())
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(df)
}

Output:
shape: (5, 4)
┌─────────┬─────────┬───────┬─────────┐
│ region  │ product │ qty   │ price   │
│ String  │ String  │ Int64 │ Float64 │
├─────────┼─────────┼───────┼─────────┤
│ "north" │ "apple" │ 10    │ 1.5     │
│ "south" │ "apple" │ 4     │ 1.5     │
│ "north" │ "pear"  │ 7     │ 2.25    │
│ "east"  │ "pear"  │ 2     │ 2.25    │
│ "south" │ "fig"   │ 9     │ 3       │
└─────────┴─────────┴───────┴─────────┘

Index

Examples

Constants

View Source 
const (
	// ClosedDefault is the zero value and means "this operation's own convention":
	// both for IsBetween, left for GroupByDynamic, right for Rolling.
	ClosedDefault = expr.ClosedDefault
	ClosedLeft    = expr.ClosedLeft
	ClosedRight   = expr.ClosedRight
	ClosedBoth    = expr.ClosedBoth
	ClosedNone    = expr.ClosedNone
)

The four window boundary conventions. ClosedLeft is the default for a window grid — an instant exactly on a boundary starts the new window — because it is the only one under which consecutive windows tile the line exactly.

View Source 
const (
	// InterpLinear interpolates proportionally between the neighbours. It is the
	// conventional default and the only option whose result need not be an input
	// value.
	InterpLinear = expr.InterpLinear

	// InterpLower and InterpHigher take the neighbour below or above the rank.
	InterpLower  = expr.InterpLower
	InterpHigher = expr.InterpHigher

	// InterpNearest takes the closer neighbour, rounding halves upward.
	InterpNearest = expr.InterpNearest

	// InterpMidpoint averages the two neighbours regardless of the rank's position
	// between them.
	InterpMidpoint = expr.InterpMidpoint
)
View Source 
const (
	// AsOfBackward is the default: the last right key at or before the left key.
	// "Attach the most recent quote to each trade."
	AsOfBackward = plan.AsOfBackward
	// AsOfForward is the first right key at or after the left key.
	AsOfForward = plan.AsOfForward
	// AsOfNearest is whichever is closer, ties going backward.
	AsOfNearest = plan.AsOfNearest
)
View Source 
const (
	// ConcatStrict requires the same columns in the same order — the default.
	//
	// Types still promote and nullability still widens. Neither is a relaxation:
	// refusing to stack two frames that differ only in a nullability flag would be
	// absurd, because that is the ordinary result of filtering one of them.
	ConcatStrict = plan.ConcatStrict

	// ConcatDiagonal takes the union of the columns, filling each frame's missing
	// ones with nulls. This is what makes stacking heterogeneous files work.
	ConcatDiagonal = plan.ConcatDiagonal
)
View Source 
const (
	// JoinInner keeps only matching pairs. The default.
	JoinInner = plan.JoinInner
	// JoinLeft keeps every left row, with right columns null where unmatched.
	JoinLeft = plan.JoinLeft
	// JoinRight is the mirror of JoinLeft.
	JoinRight = plan.JoinRight
	// JoinFull keeps every row from both sides.
	JoinFull = plan.JoinFull
	// JoinSemi keeps left rows that have at least one match. A filter: no right
	// column is added.
	JoinSemi = plan.JoinSemi
	// JoinAnti keeps left rows with no match. The inverse of JoinSemi.
	JoinAnti = plan.JoinAnti
	// JoinCross is the cartesian product. No keys.
	JoinCross = plan.JoinCross
)
View Source 
const (
	ValidateNone = plan.ValidateNone
	// ValidateOneToOne requires unique keys on both sides.
	ValidateOneToOne = plan.ValidateOneToOne
	// ValidateOneToMany requires unique keys on the LEFT.
	ValidateOneToMany = plan.ValidateOneToMany
	// ValidateManyToOne requires unique keys on the RIGHT. This is the one that
	// catches accidental fan-out, which is the most common analytics bug there is.
	ValidateManyToOne = plan.ValidateManyToOne
	// ValidateManyToMany imposes no constraint.
	ValidateManyToMany = plan.ValidateManyToMany
)
View Source 
const (
	Second = dtype.Second
	Milli  = dtype.Milli
	Micro  = dtype.Micro
	Nano   = dtype.Nano
)

Time resolutions.

View Source 
const (
	// MapGroupsToRows gives every row its own partition's value, in the original
	// row order. The default, and the only one that composes with other columns.
	MapGroupsToRows = expr.MapGroupsToRows

	// MapExplode leaves the output in partition order instead of permuting it back.
	// Cheaper, because it skips the inverse permutation — and it REORDERS the frame,
	// so it is only meaningful when every column is windowed the same way.
	MapExplode = expr.MapExplode

	// MapJoin would aggregate each partition into a List repeated across its rows.
	// Refused, because it is a synonym: Col(...).Implode().Over(...) is exactly
	// this under the default mapping.
	MapJoin = expr.MapJoin
)
View Source 
const (
	// RankOrdinal gives every row a distinct rank, ties broken by input order.
	RankOrdinal = expr.RankOrdinal
	// RankDense gives tied rows the same rank and does not skip the next value:
	// [10,20,20,30] ranks as 1,2,2,3.
	RankDense = expr.RankDense
	// RankMin and RankMax give tied rows the lowest or highest of their positions:
	// [10,20,20,30] ranks as 1,2,2,4 and 1,3,3,4.
	RankMin = expr.RankMin
	RankMax = expr.RankMax
	// RankAverage gives tied rows the mean of their positions, so it returns
	// Float64 where every other method returns Uint32.
	RankAverage = expr.RankAverage
)

Variables

View Source 
var (
	Bool    = dtype.Bool
	Int8    = dtype.Int8
	Int16   = dtype.Int16
	Int32   = dtype.Int32
	Int64   = dtype.Int64
	Uint8   = dtype.Uint8
	Uint16  = dtype.Uint16
	Uint32  = dtype.Uint32
	Uint64  = dtype.Uint64
	Float32 = dtype.Float32
	Float64 = dtype.Float64
	// Int128 is the accumulator and output type of integer Sum. Widening to 128
	// bits is what makes an integer sum incapable of silently overflowing.
	Int128 = dtype.Int128
	String = dtype.String
	Binary = dtype.Binary
	Date   = dtype.Date
	NullT  = dtype.Null
)

Simple types.

View Source 
var (
	// ErrSchema is an unknown, duplicate or ambiguous column — or several files,
	// read as one, whose columns differ in anything but their order.
	ErrSchema = uerr.ErrSchema
	// ErrType is an operation not defined for the given types.
	ErrType = uerr.ErrType
	// ErrValue is a bad literal, an out-of-range cast, an unparseable pattern.
	ErrValue = uerr.ErrValue
	// ErrUnsupported is a well-formed request ursus cannot yet serve.
	ErrUnsupported = uerr.ErrUnsupported
	// ErrIO is a failure reading or writing a data source.
	ErrIO = uerr.ErrIO
	// ErrResource is a limit ursus refused to exceed: the query would have worked
	// with a larger WithMemoryLimit or a writable WithSpillDir.
	ErrResource = uerr.ErrResource
	// ErrInternal is a bug in ursus. Users should never legitimately see one.
	ErrInternal = uerr.ErrInternal
)

Error kinds, for branching on a failure without matching on its message. 

if errors.Is(err, ursus.ErrResource) { retry with a bigger memory limit }

These are re-exported because the sentinels themselves live under internal/, so no package outside this module could reach them — which made "a caller should be able to detect that without matching on a message" true only for callers inside the module.

View Source 
var (
	Datetime = dtype.Datetime
	Duration = dtype.Duration
	TimeOf   = dtype.Time
	Decimal  = dtype.Decimal
	List     = dtype.List
	Array    = dtype.Array
	Enum     = dtype.Enum
)

Parameterised type constructors.

View Source 
var (
	Of      = dtype.Of
	NotNull = dtype.NotNull
)

Of builds a nullable field; NotNull builds a non-nullable one.

Functions

This section is empty.

Types

type AsOfOption 

type AsOfOption func(*asOfCfg)

AsOfOption configures JoinAsOf.

func AsOfAllowExactMatches 

func AsOfAllowExactMatches(b bool) AsOfOption

AsOfAllowExactMatches decides whether a right key EQUAL to the left key may match. Default true.

func AsOfBy 

func AsOfBy(exprs ...Expr) AsOfOption

AsOfBy adds EXACT-match keys applied before the nearest-key search, so a trade matches only quotes for its own symbol.

Without it every left row searches one global run, which on any real fixture means matching the nearest row of the wrong instrument — a plausible number and the wrong one.

A null by-key matches nothing: a left row with one gets nulls, and a right row with one is never found — as in Polars and DuckDB, and as a null key does in every other ursus join unless NullsEqual says otherwise. By-keys of different types are compared at the type that holds both exactly, as a join's keys are.

func AsOfLeftBy 

func AsOfLeftBy(exprs ...Expr) AsOfOption

AsOfLeftBy and AsOfRightBy are AsOfBy for differently-named columns.

func AsOfLeftOn 

func AsOfLeftOn(e Expr) AsOfOption

AsOfLeftOn and AsOfRightOn name differently-named ordering keys. Used together.

func AsOfOn 

func AsOfOn(e Expr) AsOfOption

AsOfOn names the ordering key, present under the same name on both sides.

func AsOfRightBy 

func AsOfRightBy(exprs ...Expr) AsOfOption

func AsOfRightOn 

func AsOfRightOn(e Expr) AsOfOption

func AsOfStrategyOpt 

func AsOfStrategyOpt(s AsOfStrategy) AsOfOption

AsOfStrategyOpt picks backward (the default), forward or nearest.

func AsOfSuffix 

func AsOfSuffix(s string) AsOfOption

AsOfSuffix renames a colliding right column. Defaults to Join's suffix.

func AsOfTolerance 

func AsOfTolerance(i Interval) AsOfOption

AsOfTolerance bounds how far the search may reach. A candidate further than this from the left key does not match, and the left row comes back null-padded.

The bound is recomputed per row for a CALENDAR interval, so Every("1mo") is a different distance in February than in March — which is the whole reason Interval is not a duration.

type AsOfStrategy 

type AsOfStrategy = plan.AsOfStrategy

AsOfStrategy picks which neighbouring right row a left row matches.

type CSVFile added in v0.3.0 

type CSVFile struct {
	// Name identifies the file in errors and in Explain.
	Name string

	// Open returns the file's bytes from the start.
	//
	// It is called more than once per query — once to infer the schema, once for
	// each execution — and must return a fresh stream each time. ctx is the query's
	// and outlives the stream, which ursus closes when it is done with it.
	Open func(ctx context.Context) (io.ReadCloser, error)
}

CSVFile is one CSV file, read as a stream from wherever it lives — an object store, an HTTP server, an archive — by code that knows how to reach it.

type CSVOption 

type CSVOption func(*csv.Options)

CSVOption configures ScanCSV.

func WithColumnNames 

func WithColumnNames(names ...string) CSVOption

WithColumnNames overrides the column names positionally.

func WithComment 

func WithComment(prefix string) CSVOption

WithComment makes lines starting with prefix comments. Multi-byte prefixes such as "//" are supported.

func WithHasHeader 

func WithHasHeader(b bool) CSVOption

WithHasHeader says whether the first record holds column names. Default true.

func WithInferRows 

func WithInferRows(n int) CSVOption

WithInferRows bounds how many records inference reads. 0 reads the whole file, which is exact and, on a large file, expensive. Default 100.

func WithMaxRecordSize 

func WithMaxRecordSize(n int) CSVOption

WithMaxRecordSize bounds a single record. Default 16 MiB. The bound exists so an unterminated quote is an error rather than an out-of-memory kill.

func WithNullValues 

func WithNullValues(vals ...string) CSVOption

WithNullValues adds texts that mean NULL. An empty field already means null for every type; in a String column a quoted "" is the empty string, as Polars reads it, so the two stay apart.

func WithQuote 

func WithQuote(c byte) CSVOption

WithQuote sets the quote character. Default '"'.

func WithSchema 

func WithSchema(s *Schema) CSVOption

WithSchema supplies the schema and skips inference entirely. On a large file this is the option that matters: inference costs a read of the sample, and an explicit schema also removes the risk of a column being typed from rows that do not represent it.

func WithSchemaOverrides 

func WithSchemaOverrides(m map[string]dtype.DataType) CSVOption

WithSchemaOverrides fixes individual columns while inferring the rest. The usual case is an identity column that looks numeric and must not be.

func WithSeparator 

func WithSeparator(c byte) CSVOption

WithSeparator sets the field delimiter. Default ','.

func WithSkipRows 

func WithSkipRows(n int) CSVOption

WithSkipRows discards n records before the header.

func WithTruncateRaggedLines 

func WithTruncateRaggedLines(b bool) CSVOption

WithTruncateRaggedLines accepts records with the wrong field count: extra fields are dropped and missing ones become null. Off by default, because a changed field count usually means the file is not what the reader thinks it is.

type CSVSinkOption 

type CSVSinkOption interface {
	// contains filtered or unexported methods
}

CSVSinkOption is anything SinkCSV and WriteCSV accept: a writer option or an execution option. See ParquetSinkOption for why it is an interface.

type CSVWriteOption 

type CSVWriteOption func(*csv.WriteOptions)

CSVWriteOption configures the CSV writer.

func WithLineTerminator 

func WithLineTerminator(s string) CSVWriteOption

WithLineTerminator sets the record separator. Default "\n".

func WithNullValue 

func WithNullValue(s string) CSVWriteOption

WithNullValue sets the text written for a null. Default "".

The default reads back as null for every type. An empty String is written as a quoted "", which reads back as the empty string, so null and empty strings both round-trip — as Polars writes and reads them.

func WithWriteHeader 

func WithWriteHeader(b bool) CSVWriteOption

WithWriteHeader says whether to write column names first. Default true.

func WithWriteSeparator 

func WithWriteSeparator(c byte) CSVWriteOption

WithWriteSeparator sets the delimiter. Default ','.

type Closed 

type Closed = expr.Closed

Closed says which endpoints of an interval belong to it: [lo, hi), (lo, hi], [lo, hi] or (lo, hi). Used by IsBetween and by the temporal group-by's windows.

type CollectOption 

type CollectOption func(*collectCfg)

CollectOption configures execution.

func WithBatchSize 

func WithBatchSize(n int) CollectOption

WithBatchSize sets the rows per batch. Results must not depend on it; the test suite varies it precisely to check that.

func WithMemoryLimit 

func WithMemoryLimit(bytes int64) CollectOption

WithMemoryLimit caps what buffering operators may hold, in bytes. 

df, err := ursus.ScanParquet("120gb/*.parquet").
    Sort(ursus.Desc(ursus.Col("ts"))).
    SinkParquet(ctx, "sorted.parquet",
        ursus.WithMemoryLimit(512<<20),
        ursus.WithSpillDir("/tmp/ursus"))

What it bounds

Sort spills past the limit and merges the runs back, so a sort over more data than memory works. Group-by spills too, by radix-partitioning the keys it cannot hold: past the limit new keys are routed to one of sixteen files per level and re-aggregated afterwards. Join partitions BOTH sides on the join key and replays the pairs, so an equi-join over more data than memory works as well. Unique routes the rows whose keys it cannot hold the same way, and a window with a partition_by routes whole partitions; both stamp each row with its input position and merge back on it, so their answers keep their order exactly.

Reverse, hstack, tail, join_asof, merge_sorted, rolling, group_by_dynamic and a window with no partition_by cannot spill: past the limit they FAIL, with an error naming the operator, which is a better outcome than being killed by the OS with no explanation.

A bounded sort — one under a Head or a TopK — never spills at all: past the limit it discards everything outside the current best k, which is exact.

The one thing the limit CAN change

An unordered group-by's ROW ORDER depends on the limit, because the point at which it stops admitting new keys does. Past that point a new key goes to a partition file and comes back after every resident group, so the output is resident-first then partition-major. A spilling JOIN is the same, for the same reason: rows whose key stayed in memory come out in probe order, and the rest follow bucket by bucket.

The CONTENT is invariant: same rows, same values, at every limit and at none. Only the sequence moves. A group-by can opt out with GroupBy.MaintainOrder, which restores first-appearance order exactly, spilled or not.

A JOIN CANNOT, and that is a decision rather than an omission. Restoring probe order means holding the whole output to permute it, and a join's output can be larger than both of its inputs — so the flag would only work in the cases that did not need it. TestMemoryLimitIsNotASemanticKnob pins the content half for both, and TestJoinOrderIsUnspecifiedUnderALimit pins that the order really does move.

What a group-by's limit does NOT bound

Partitioning divides the KEY SPACE, so it bounds a group-by whose problem is cardinality. It cannot divide a single key, and quantile, median and n_unique hold state per VALUE rather than per group — so a group-by over a few very large groups still fails, with an error that says which of the two situations it is in.

The join has the same wall in two places: a CROSS join has no key at all, and one join key with more build rows than the limit cannot be split however deep the recursion goes. Both refuse with a message naming which it is. A window has it too: one partition with more rows than the limit refuses, saying so.

Without it

A query with no WithMemoryLimit is budgeted at half the memory it can have: the smaller of its cgroup's limit — a container's, or a systemd slice's — and the machine's RAM, on Linux. Half, because ursus's buffers are on the Go heap, which with GOGC=100 grows to about twice what is live before it collects. So a query spills by default rather than growing until the kernel kills it, and an operator that cannot spill fails at that point with an error naming itself. WithMemoryLimit(0) turns the budget off; elsewhere than Linux there is no default and it is off already.

A parallel group-by overshoots a little

A group-by runs on several workers until the query holds half its budget, then folds them into one and finishes serially, spilling as the serial path does. Half, because the fold builds the merged table while the workers' still exist. The batches the workers already had queued are aggregated first, so the switch can come that much late — a few megabytes at the default batch size.

What it does not bound

Collect materialises the whole result by definition, and asking for a frame in memory is a request to hold it. The larger-than-RAM story runs through SinkParquet and CollectBatches, both of which stream.

Example

Under a memory limit, sort, group-by, join, unique and partitioned windows spill to disk and give the answer they give in memory; an operator that cannot spill fails with an error naming itself. 

package main

import (
	"context"
	"fmt"
	"log"
	"os"

	"github.com/advenn/ursus"
)

func main() {
	keys := make([]int64, 40000)
	for i := range keys {
		keys[i] = int64(i * 7919 % 20000) // each of 20000 keys, twice
	}
	var stats ursus.MemoryStats
	n, err := ursus.Frame(ursus.Values("k", keys)).Unique("k").
		Count(context.Background(), ursus.WithBatchSize(1024),
			ursus.WithMemoryLimit(16<<10), ursus.WithSpillDir(os.TempDir()),
			ursus.WithMemoryStats(&stats))
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(n, "distinct keys; spilled:", stats.Spills > 0)
}

Output:
20000 distinct keys; spilled: true

func WithMemoryStats 

func WithMemoryStats(out *MemoryStats) CollectOption

WithMemoryStats stores the query's memory accounting into *out when the query finishes.

It is written at the end of Collect, Count, CollectInto, SinkParquet, SinkCSV, WriteParquet and WriteCSV, and after the last batch of CollectBatches — so reading it early gives a partial figure rather than a wrong one.

The sinks matter most and were the ones this list used to omit: they are the consumers that stream, so they are where a memory limit is worth setting.

func WithOptFlags 

func WithOptFlags(f plan.Flags) CollectOption

WithOptFlags overrides which optimizer rules run. Use it to benchmark a rule's value or to bisect a wrong answer down to the rule that caused it.

func WithSpillDir 

func WithSpillDir(dir string) CollectOption

WithSpillDir chooses where spill files are written. The default is the system temporary directory. Files are removed when the query finishes, including on error and on an early break out of CollectBatches.

func WithThreads 

func WithThreads(n int) CollectOption

WithThreads sets how many workers a query runs on. Default runtime.NumCPU().

Results do not depend on it. Parallelism over the PIPELINE — the scan and the stateless per-batch operators above it — is ORDER-PRESERVING: workers process batches concurrently and the results are read back in input order, so the same query returns byte-identical output at any thread count. That is worth stating because it is not what every engine does, and because five things in ursus quietly depend on batch order — stable sort, First and Last, distinct's first-row rule, limit, and first-appearance group ids.

WithThreads(1) reproduces the serial operator tree exactly, which is the knob to reach for when bisecting.

Group-by also runs N-ways, and only where that cannot change an answer

Since step 17 a hash aggregation is drained by several sinks whose partial tables are merged. It is NOT order-preserving in the same structural way — the dispatcher is round-robin, so no worker holds a contiguous portion of the input — so it is used only where the result cannot depend on order at all: not under MaintainOrder, not with First, Last, ArgMin or ArgMax, and not under a memory limit, because a sink that has spilled cannot be merged. Every one of those falls back to the serial path silently and exactly.

Sort and join breakers remain serial. They still benefit, because their input arrives faster.

The trade, stated: a parallel group-by holds roughly N times the group table, because each worker builds its own. That is why the memory limit disables it rather than dividing the budget.

func WithVerify 

func WithVerify() CollectOption

WithVerify makes the optimizer assert schema preservation after every rule. On in tests, off by default because it costs a schema resolution per rule.

type Column 

type Column = data.Column

Column is a named, typed, nullable run of values.

func Values 

func Values[T Literal](name string, vals []T) *Column

Values builds a column from Go values, with no nulls.

The element type determines the column type: []int64 gives Int64, []string gives String, []time.Time gives Datetime(ns, UTC).

func ValuesNullable 

func ValuesNullable[T Literal](name string, vals []T, valid []bool) *Column

ValuesNullable builds a column with an explicit validity mask, where valid[i] false means row i is null.

Nulls are a parallel bitmap rather than a *T or an Option[T] because boxing would cost an allocation per row and make the values buffer non-contiguous — exactly what a columnar layout must not do.

type ConcatMode 

type ConcatMode = plan.ConcatMode

ConcatMode says how much Concat will reconcile differing schemas.

type ConcatOption 

type ConcatOption func(*concatCfg)

ConcatOption configures Concat.

func WithConcatMode 

func WithConcatMode(m ConcatMode) ConcatOption

WithConcatMode selects strict or diagonal reconciliation.

type DataFrame 

type DataFrame struct {
	// contains filtered or unexported fields
}

DataFrame is a materialised result: named, equal-length, typed columns.

There is no row index. Row order is a property of the data, not an addressable label space — the single most consequential thing Polars got right and pandas did not.

func (*DataFrame) ArrowSchema added in v0.3.0 

func (df *DataFrame) ArrowSchema() (s *arrow.Schema, err error)

ArrowSchema returns the frame's schema as an Arrow schema.

It can fail, because a few ursus types have no Arrow equivalent yet — see Record. Asking for the schema is how to find that out without running anything.

func (*DataFrame) At 

func (df *DataFrame) At[T any](row int, name string) (T, bool, error)

At returns one value and whether it is non-null.

func (*DataFrame) Batch 

func (df *DataFrame) Batch() *data.Batch

Batch exposes the underlying batch for internal use and tests.

func (*DataFrame) Column 

func (df *DataFrame) Column[T any](name string) (*data.Series[T], error)

Column returns a typed view of a column.

A generic METHOD — the thing Go 1.27 unlocked. Previously this had to be `ursus.Column[int64](df, "x")`, with the receiver demoted to an argument.

Example

Reading values back out. Column gives a typed series; At gives one cell and whether it was null. 

package main

import (
	"context"
	"fmt"
	"log"

	"github.com/advenn/ursus"
)

// exampleSales is the little frame the examples below work on.
func exampleSales() *ursus.LazyFrame {
	return ursus.Frame(
		ursus.Values("region", []string{"north", "south", "north", "east", "south"}),
		ursus.Values("product", []string{"apple", "apple", "pear", "pear", "fig"}),
		ursus.Values("qty", []int64{10, 4, 7, 2, 9}),
		ursus.Values("price", []float64{1.50, 1.50, 2.25, 2.25, 3.00}),
	)
}

func main() {
	df, err := exampleSales().Collect(context.Background())
	if err != nil {
		log.Fatal(err)
	}

	// Column and At are GENERIC METHODS — Go 1.27's headline feature, and the
	// reason ursus can hand back typed data without a cast at the call site.
	qty, err := df.Column[int64]("qty")
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println("values:", qty.Values())

	v, ok, err := df.At[string](0, "region")
	if err != nil {
		log.Fatal(err)
	}
	fmt.Printf("row 0 region: %q present=%v\n", v, ok)
}

Output:
values: [10 4 7 2 9]
row 0 region: "north" present=true

func (*DataFrame) Columns 

func (df *DataFrame) Columns() []string

Columns returns the column names in order.

func (*DataFrame) Concat 

func (df *DataFrame) Concat(ctx context.Context, others ...*DataFrame) (*DataFrame, error)

Concat stacks frames vertically.

func (*DataFrame) Drop 

func (df *DataFrame) Drop(ctx context.Context, names ...string) (*DataFrame, error)

Drop removes columns; Rename changes their names.

func (*DataFrame) Filter 

func (df *DataFrame) Filter(ctx context.Context, preds ...Expr) (*DataFrame, error)

Filter keeps rows where every predicate is true.

func (*DataFrame) Head 

func (df *DataFrame) Head(ctx context.Context, n int) (*DataFrame, error)

Head keeps the first n rows; Tail the last n.

func (*DataFrame) Height 

func (df *DataFrame) Height() int

Height returns the number of rows; Width the number of columns.

func (*DataFrame) Join 

func (df *DataFrame) Join(ctx context.Context, other *DataFrame, opts ...JoinOption) (*DataFrame, error)

Join combines two frames on keys.

func (*DataFrame) JoinWhere added in v0.3.0 

func (df *DataFrame) JoinWhere(ctx context.Context, other *DataFrame, preds ...Expr) (*DataFrame, error)

JoinWhere joins every pair of rows that satisfies the predicates.

func (*DataFrame) Lazy 

func (df *DataFrame) Lazy() *LazyFrame

Lazy turns a materialised frame back into a query.

The obvious spelling is wrong in three specific ways

`ursus.Frame(df.Batch().Columns()...)` compiles and runs, because Column is a type alias — and it is wrong:

  • It would re-derive nullability from the data. Field.Nullable is a static property of the schema, not a count of nulls actually present, so a nullable column that happens to hold none — the ordinary result of a filter — would come back non-nullable and every downstream join and cast would reason from the wrong schema.
  • It would lose the row count of a frame with rows but no columns.
  • It would alias the batch's own column slice, against Columns()' contract.

So this goes through the batch, which already carries the declared schema and the row count.

It is free: no copy, no re-derivation. The batch is immutable and shared.

func (*DataFrame) Record added in v0.3.0 

func (df *DataFrame) Record() (rec arrow.Record, err error)

Record exports the frame as an Arrow record, sharing its memory.

This is the zero-copy boundary to the rest of the Arrow ecosystem: hand the result to arrow-go's IPC writer, to Flight, to ADBC, or to any library that speaks arrow.Record, and no column data is copied on the way. 

rec, err := df.Record()
if err != nil { return err }
defer rec.Release()

w := ipc.NewWriter(f, ipc.WithSchema(rec.Schema()))
w.Write(rec)

Release is optional here, and that is deliberate

The record's buffers are re-wrapped so that Release on them cannot reach ursus's own memory. Releasing is still the right habit — every other arrow-go producer expects it, and the example above does it — but forgetting leaks nothing, and releasing twice cannot empty a frame you still hold. internal/arrowout's doc explains why that second guarantee needed arranging.

It can fail

Array, Enum, Categorical and Uint128 are declared in ursus's type enum and are not exportable yet; a frame containing one is refused by name rather than exported wrongly. Everything else maps, including Decimal and the Int128 that every integer Sum produces.

func (*DataFrame) Rename 

func (df *DataFrame) Rename(ctx context.Context, names map[string]string) (*DataFrame, error)

func (*DataFrame) Reverse 

func (df *DataFrame) Reverse(ctx context.Context) (*DataFrame, error)

Reverse emits rows in the opposite order.

func (*DataFrame) Rows 

func (df *DataFrame) Rows[T any]() (rows []T, err error)

Rows decodes every row into a T.

func (*DataFrame) Schema 

func (df *DataFrame) Schema() *Schema

Schema returns the column names and types.

func (*DataFrame) Select 

func (df *DataFrame) Select(ctx context.Context, exprs ...Expr) (*DataFrame, error)

Select evaluates expressions and returns the result.

func (*DataFrame) Shape 

func (df *DataFrame) Shape() (int, int)

Shape returns (rows, columns).

func (*DataFrame) Sort 

func (df *DataFrame) Sort(ctx context.Context, keys ...SortKey) (*DataFrame, error)

Sort orders rows.

func (*DataFrame) String 

func (df *DataFrame) String() string

String renders the frame as an aligned table.

Display quality is not cosmetic: this is what people look at all day, and a column of unaligned numbers with no visible types is the difference between a library that feels finished and one that does not.

func (*DataFrame) Tail 

func (df *DataFrame) Tail(ctx context.Context, n int) (*DataFrame, error)

func (*DataFrame) Unique 

func (df *DataFrame) Unique(ctx context.Context, subset ...string) (*DataFrame, error)

Unique removes duplicate rows.

func (*DataFrame) Unpivot added in v0.3.0 

func (df *DataFrame) Unpivot(ctx context.Context, opts UnpivotOptions) (*DataFrame, error)

Unpivot turns several value columns into a variable/value pair.

func (*DataFrame) WhereExists added in v0.3.0 

func (df *DataFrame) WhereExists(ctx context.Context, other *DataFrame, preds ...Expr) (*DataFrame, error)

WhereExists keeps rows with at least one satisfying partner in other.

func (*DataFrame) WhereNotExists added in v0.3.0 

func (df *DataFrame) WhereNotExists(ctx context.Context, other *DataFrame, preds ...Expr) (*DataFrame, error)

WhereNotExists keeps rows with no satisfying partner in other.

func (*DataFrame) Width 

func (df *DataFrame) Width() int

func (*DataFrame) WithColumns 

func (df *DataFrame) WithColumns(ctx context.Context, exprs ...Expr) (*DataFrame, error)

WithColumns adds or replaces columns.

type DataType 

type DataType = dtype.DataType

DataType is a logical column type. Comparable, usable as a map key.

func StructOf 

func StructOf(fields ...Field) DataType

StructOf builds a struct type. Named StructOf rather than Struct to leave the bare name available and to match DateOf / TimeOf.

type DtExpr 

type DtExpr struct {
	// contains filtered or unexported fields
}

DtExpr is the temporal namespace: `Col("ts").Dt().Year()`.

Components are read in the column's own timezone

2024-01-01T00:00:00+09:00 is hour 0 in Tokyo and hour 15 in UTC. A Datetime carries its zone, and every component below is read in it — otherwise `hour` would be wrong for every row of any non-UTC column, silently.

Durations have no components

`.Year()` on a Duration is refused at plan time. A duration is a length, not a point in time; "the year of 90 minutes" has no answer. Use the Total* family.

func (DtExpr) Day 

func (d DtExpr) Day() Expr

func (DtExpr) Epoch 

func (d DtExpr) Epoch() Expr

Epoch is whole seconds since 1970-01-01T00:00:00Z, floored: the second an instant lies in, so half a second before the epoch is -1.

func (DtExpr) Hour 

func (d DtExpr) Hour() Expr

func (DtExpr) Microsecond 

func (d DtExpr) Microsecond() Expr

func (DtExpr) Millisecond 

func (d DtExpr) Millisecond() Expr

func (DtExpr) Minute 

func (d DtExpr) Minute() Expr

func (DtExpr) Month 

func (d DtExpr) Month() Expr

func (DtExpr) Nanosecond 

func (d DtExpr) Nanosecond() Expr

func (DtExpr) OrdinalDay 

func (d DtExpr) OrdinalDay() Expr

OrdinalDay is the day of the year, 1-366.

func (DtExpr) Quarter 

func (d DtExpr) Quarter() Expr

Quarter is 1-4.

func (DtExpr) Second 

func (d DtExpr) Second() Expr

func (DtExpr) ToString 

func (d DtExpr) ToString() Expr

ToString formats using the same ISO 8601 layouts the CSV writer emits, so a frame printed to a terminal and a frame written to a file agree.

func (DtExpr) TotalDays 

func (d DtExpr) TotalDays() Expr

The Total family converts a Duration to a whole number of units, truncating. They are the only temporal functions defined on a Duration.

func (DtExpr) TotalHours 

func (d DtExpr) TotalHours() Expr

func (DtExpr) TotalMinutes 

func (d DtExpr) TotalMinutes() Expr

func (DtExpr) TotalSeconds 

func (d DtExpr) TotalSeconds() Expr

func (DtExpr) Truncate 

func (d DtExpr) Truncate[T Span](every T) Expr

Truncate floors an instant to a multiple of every, toward negative infinity so instants before the epoch move backwards like every other instant.

A Duration floors ABSOLUTELY; an Interval floors on the CALENDAR

The two spellings answer different questions and neither is a special case of the other: 

Truncate(time.Hour)     the instant, floored to a whole hour since the epoch
Truncate(Every("1d"))   the start of the LOCAL day, in the column's own zone
Truncate(Every("1h"))   the start of the LOCAL hour — 10:00 for 10:47 at +05:30,
                        where time.Hour gives 10:30, the hour since the epoch

An Interval floors the wall clock at every size. Where the floored wall clock falls into a spring-forward gap, the answer is the instant the gap ends; in a fall-back fold, the input keeps its own offset, so the second 01:30 floors to the second 01:00. Both are Polars' answers.

A Duration is a fixed span of elapsed time, so flooring by one is zone-independent by definition — `Truncate(24*time.Hour)` on a New York column lands on 00:00 UTC, which is 19:00 or 20:00 local, and that is the correct answer to the question a Duration asks. It is almost never the question a user grouping by day is asking, which is why Every("1d") exists and reads the column's timezone.

func (DtExpr) Week 

func (d DtExpr) Week() Expr

Week is the ISO 8601 week number, which is not "day of year / 7" — the first week of a year can begin in the previous one.

func (DtExpr) Weekday 

func (d DtExpr) Weekday() Expr

Weekday is ISO: Monday is 1 and Sunday is 7. Go's time.Weekday puts Sunday at 0, and passing that through is the classic off-by-one in date libraries.

func (DtExpr) Year 

func (d DtExpr) Year() Expr

type DynamicOptions 

type DynamicOptions struct {
	// Every is where windows START. Consecutive windows are Every apart.
	Every Interval

	// Period is a window's WIDTH. Zero means Every, which gives windows that tile
	// the input exactly. A Period larger than Every gives OVERLAPPING windows, and a
	// row then belongs to several of them — so a Len() over the result exceeds the
	// input's row count, which is the correct answer and surprises everyone once.
	Period Interval

	// Offset shifts the whole grid. A day grid with Offset 9h starts its windows at
	// 09:00 rather than at midnight.
	Offset Interval

	// Closed says which end of a window belongs to it. The zero value is
	// ClosedLeft — [start, end) — which is the only convention under which a grid
	// partitions its input exactly.
	Closed Closed

	// GroupBy adds CATEGORICAL keys. Windows are cut independently inside each
	// distinct combination, so an hourly rollup per service is one call.
	GroupBy []Expr

	// Label, StartBy and IncludeBoundaries are not implemented. They are declared so
	// that setting one is an error naming it rather than silence.
	Label             string
	StartBy           string
	IncludeBoundaries bool
}

DynamicOptions configures GroupByDynamic.

Five fields, and three that are refused rather than ignored

Every, Period, Offset, Closed and GroupBy are implemented. Label, StartBy and IncludeBoundaries are named in the design docs and are NOT — and they produce an error saying so rather than being accepted and dropped.

That is deliberate and it is a lesson this library has already paid for once: plan.Aggregate.MaintainOrder sat assigned, rendered and unread for four steps, and its own doc had to say "the flag documents the guarantee rather than changing behaviour". A field that is accepted and does nothing is worse than one that does not exist, because the next reader believes it.

type ExplainOption 

type ExplainOption func(*explainCfg)

ExplainOption configures Explain.

func Optimized 

func Optimized(b bool) ExplainOption

Optimized selects the optimized plan (the default) or the plan as written.

func WithSchemas 

func WithSchemas() ExplainOption

WithSchemas annotates every plan node with its output schema.

type Expr 

type Expr struct {
	// contains filtered or unexported fields
}

Expr is a lazy description of a column transformation.

It is a CONCRETE STRUCT rather than an interface, and that is forced by Go 1.27: interface methods may not declare type parameters, and a generic method can never satisfy an interface method. Since Gt[T Operand] — the thing that makes `Col("age").Gt(30)` work without wrapping 30 in Lit — must be a generic method, Expr cannot be an interface. Polymorphism lives in the unexported expr.Node instead.

func All 

func All() Expr

All selects every column.

func Coalesce 

func Coalesce(exprs ...Expr) Expr

Coalesce returns the first non-null value across the given expressions.

Built out of conditionals rather than its own kernel

Coalesce(a, b, c) becomes when(a.is_not_null()).then(a).otherwise(coalesce(b, c)), which is correct and costs one thing worth knowing: each operand is MENTIONED twice, and there is no common-subexpression pass, so each is EVALUATED twice. For the ordinary Coalesce(Col("a"), Col("b")) that is two extra column reads; for an operand that is itself an expensive expression it is not free.

A dedicated n-ary null-merge kernel would remove that and change no semantics. It is deliberately not built yet: the desugaring needs no new IR node, no new kernel and no new arms in the five type switches a node type must reach.

func Col 

func Col(names ...string) Expr

Col selects one or more columns by name.

With several names it becomes a multi-column selection that expands at plan time: `Col("a","b").Mul(2)` is two independent expressions.

func ColDType 

func ColDType(types ...dtype.DataType) Expr

ColDType selects every column of one of the given types. How many columns that is is determined at plan time from the schema, so the same expression adapts as the schema changes.

func ColRegex 

func ColRegex(pattern string) Expr

ColRegex selects every column whose name matches pattern.

The pattern is explicit rather than inferred from leading ^ and trailing $ the way Polars does. That inference makes a column literally named "^total$" unselectable and taxes every plain Col call with an anchor scan.

func Exclude 

func Exclude(names ...string) Expr

Exclude selects every column except the named ones.

func ExcludeRegex 

func ExcludeRegex(pattern string) Expr

ExcludeRegex selects every column whose name does not match pattern.

func Len 

func Len() Expr

Len counts rows in the current group — SQL's COUNT(*) with no column argument.

It aggregates over a literal rather than a column, because counting rows must not depend on any column existing: `GroupBy(k).Agg(Len())` has to work on a frame whose only column is the key.

func Lit 

func Lit[T Literal](v T) Expr

Lit builds a literal.

func Null 

func Null(dt dtype.DataType) Expr

Null builds a typed null literal.

func (Expr) Abs 

func (e Expr) Abs() Expr

func (Expr) Add 

func (e Expr) Add[T Operand](v T) Expr

func (Expr) Alias 

func (e Expr) Alias(name string) Expr

Alias renames the output column.

It sets ONE fixed name, so applying it to a multi-column selection is an error. Use Prefix or Suffix to rename an expansion.

func (Expr) AllTrue 

func (e Expr) AllTrue() Expr

func (Expr) And 

func (e Expr) And[T Operand](v T) Expr

func (Expr) Any 

func (e Expr) Any() Expr

Any and AllTrue reduce a Boolean column.

AllTrue rather than All, because All is already the top-level selector for every column and one name cannot mean both.

Nulls are SKIPPED, so a group with no non-null value gives NULL rather than false (for Any) or true (for AllTrue). Polars returns the vacuous answer; ursus returns null, because every other aggregate here does and an inconsistent one is a permanent trap. FillNull makes either convention available.

func (Expr) ArgMax 

func (e Expr) ArgMax() Expr

func (Expr) ArgMin 

func (e Expr) ArgMin() Expr

ArgMin and ArgMax give the POSITION of the smallest or largest value within the group, as a Uint64.

The index counts every row of the group, nulls included, so it lines up with anything else gathered from the same group — but nulls can never win, and a group with no non-null value has no position and returns NULL.

Ties go to the earliest row, which is what makes the answer independent of batch size and thread count.

func (Expr) BackwardFill 

func (e Expr) BackwardFill(limit int) Expr

BackwardFill replaces each null with the next non-null value after it.

It is ForwardFill walked in the other direction, which costs nothing: the ordered walk flips its direction rather than building a second permutation.

func (Expr) Cast 

func (e Expr) Cast(to dtype.DataType) Expr

Cast converts to another type, failing the query on the first unrepresentable value.

What is representable depends on the target:

  • An integer, a Decimal: exactly the value, in range. 3.5 is not an Int64, and 0.123 is not a Decimal(10, 2).
  • A float: the nearest value of its width, so 0.1 cast to Float32 is the float32 nearest 0.1, and an Int64 above 2^53 is the float64 nearest it. Only a finite value past the type's range — one that would round to an infinity — is unrepresentable.
  • From a String: the text parsed at the target's own width, by the same rules. "256" is not a Uint8, and text that is no number cannot be parsed at all.

func (Expr) CastLossy 

func (e Expr) CastLossy(to dtype.DataType) Expr

CastLossy converts to another type, turning unrepresentable values into nulls. The result is always nullable as a consequence.

func (Expr) Cbrt 

func (e Expr) Cbrt() Expr

func (Expr) Ceil 

func (e Expr) Ceil() Expr

func (Expr) Clip 

func (e Expr) Clip[L, H Operand](lo L, hi H) Expr

Clip bounds each value to [lo, hi].

Sugar, and that is what lets the bounds be expressions 

When(e.Lt(lo)).Then(lo).When(e.Gt(hi)).Then(hi).Otherwise(e)

A parameterised Call could not do this: CallArgs requires every argument to be a literal and rejects a column-valued one by name. As a conditional, Clip takes Expr bounds naturally — Col("x").Clip(Col("floor"), Col("cap")) works.

The bounds are WEAK literals, like a fill value

So Col("i32").Clip(0, 100) stays Int32. It used to become Int64, because the When/Then builder lifts strongly — and Col("u64").Clip(0, 100) became Int128, which is a startling type for clipping an unsigned column to a small range. That is exactly the inconsistency weak literals were built to remove: both spellings look like "a Go scalar bound on a column", and only one of them preserved the type. A bound that does not fit falls back to ordinary promotion, the same as a fill value that does not fit.

An Expr bound is used as written, so Clip(Lit(0), Lit(100)) still widens.

Edge cases, each of which falls out rather than being coded 

e is null      → the comparisons are null → neither branch → null
e is NaN       → NaN < lo and NaN > hi are both false → e, unchanged
a bound is null → every row's mask is null → the whole column is null
lo > hi        → lo wins, because the chain is right-nested in written order

The third is the one to watch: a null bound is not "no bound", it is a null result. Use two separate clips if only one side should apply.

func (Expr) Count 

func (e Expr) Count() Expr

Count counts NON-NULL values — SQL's COUNT(col).

func (Expr) CumCount 

func (e Expr) CumCount(reverse bool) Expr

CumCount numbers rows within the window, starting at 1.

With reverse it counts from the end, so the last row is 1. That is what makes IsLastDistinct expressible as a window rather than a second pass.

func (Expr) CumMax 

func (e Expr) CumMax(reverse bool) Expr

func (Expr) CumMin 

func (e Expr) CumMin(reverse bool) Expr

func (Expr) CumProd 

func (e Expr) CumProd(reverse bool) Expr

func (Expr) CumSum 

func (e Expr) CumSum(reverse bool) Expr

CumSum, CumProd, CumMin and CumMax are running reductions over the window.

Nulls are SKIPPED by the running value and PRESERVED in place: cum_sum of [1, null, 3] is [1, null, 4], not [1, 1, 4] and not [1, null, null]. The running total ignores what is not there; the output still says the value was missing.

CumSum uses the same accumulator width as Sum, so the last row of `cum_sum(x)` equals `sum(x)` rather than differing by an overflow. CumProd returns Float64 for the reason Product does: i128 has addition but no multiplication.

func (Expr) Diff 

func (e Expr) Diff(n int) Expr

Diff is the difference from the value n rows earlier.

It costs ONE shift, not two

`e.Sub(e.Shift(n))` mentions the shift twice, and window resolution dedups on the rendered expression — so both mentions resolve to the same temporary and the shift is computed once. That is real common-subexpression elimination, unlike Coalesce, which has none.

The type is whatever subtraction gives, which is the operand's own for a signed number and a DURATION for an instant — the temporal algebra's instant-minus-instant rule. Polars' diff carries a null_behavior parameter this composition cannot express; the first n rows are null here.

An unsigned column is widened first

Subtracting at its own width wrapped: a UInt8 that went from 3 to 1 changed by 254. It is taken one signed width up — UInt8 to Int16, UInt16 to Int32, UInt32 to Int64, UInt64 to Int128 — which holds every difference exactly, by the same plan-time call PctChange uses to reach a float. A signed integer keeps its width and wraps, as all integer arithmetic here does, and as Polars' diff does.

func (Expr) Div 

func (e Expr) Div[T Operand](v T) Expr

Div is TRUE division: it always produces a float, so 7/2 is 3.5 even for two integers. Integer division is FloorDiv, so the surprising behaviour has to be asked for by name.

func (Expr) DropNans 

func (e Expr) DropNans() Expr

DropNans is refused for the same reason as DropNulls, and its hint carries the asymmetry that keeps a frame-level DropNans from being written the obvious way.

func (Expr) DropNulls 

func (e Expr) DropNulls() Expr

DropNulls is not an expression. It is refused here so the mistake is caught at build time with a message naming the operation that does exist.

Why there is no expression form

Every expression must produce exactly one value per input row — the evaluator enforces it — and dropping rows changes the frame's height. There is no Expr.Filter, Expr.Slice or Expr.Gather family to hang a height-changing expression on, so this is not a missing case but a different shape of thing.

Refused at BUILD time rather than at plan time on purpose: the deferred error is the first thing expression resolution looks at, so CollectSchema and Explain both fail too. A refusal that only Collect notices — which is what happened to one of the window mapping strategies — lets Explain print a plan that cannot run.

func (Expr) Dt 

func (e Expr) Dt() DtExpr

Dt opens the temporal namespace.

func (Expr) Eq 

func (e Expr) Eq[T Operand](v T) Expr

func (Expr) EqMissing 

func (e Expr) EqMissing[T Operand](v T) Expr

EqMissing is equality where nulls are DATA: null equals null, and the result is never null. NeMissing is its negation.

func (Expr) Err 

func (e Expr) Err() error

Err returns any error deferred during construction — a bad regex, say. Expression builders cannot return errors without destroying chaining, so the error rides in the tree and surfaces here or at Collect.

func (Expr) Exp 

func (e Expr) Exp() Expr

func (Expr) Explode added in v0.3.0 

func (e Expr) Explode() Expr

Explode is refused as an expression for the reason DropNulls is: it changes the frame's height, and an expression produces one value per input row.

It is the same refusal rather than a new one on purpose. A caller who reaches for `Col("tags").Explode()` has made the height mistake, not a list mistake, and the answer is the same shape as it is for DropNulls.

func (Expr) FillNan 

func (e Expr) FillNan[T Operand](v T) Expr

FillNan replaces NaN with a value, leaving nulls alone.

Nulls survive, and the reason is worth knowing

`e.IsNotNan()` on a null row is NULL, not true — null and NaN are different things all the way down — and kernel.Select makes a null mask take NEITHER branch. So a null row keeps its null and the fill value never lands on it. That is exactly right, and it looks accidental, which is why TestFillNanLeavesNullsAlone exists.

Why the predicate is IsNotNan rather than IsNan

Both spell the same thing, and only one names the result correctly. OutputName takes the LEFTMOST column of an expression, so with `IsNan → Then: value, Else: e` the leftmost column reference is the literal and every filled column came back called "literal" — which for the frame-level form meant a new column instead of a replaced one.

func (Expr) FillNull 

func (e Expr) FillNull(s FillStrategy) Expr

FillNull replaces nulls according to a named strategy. 

Col("temp").FillNull(ursus.FillForward)
Col("qty").FillNull(ursus.FillZero)

For a forward or backward fill with a limit, use ForwardFill(n) or BackwardFill(n) directly — the strategy form is unlimited.

Three of these buffer the whole frame

FillMin, FillMax and FillMean compute an aggregate over every row before they can fill anything, so they turn a streaming query into one that holds its input. FillForward and FillBackward are ordered windows and buffer for the same reason. Only FillZero and FillOne stream.

func (Expr) FillNullWith 

func (e Expr) FillNullWith[T Operand](v T) Expr

FillNullWith replaces nulls with a value.

It does not widen the column 

Col("i32").FillNullWith(0)     // stays Int32
Coalesce(Col("i32"), Lit(0))   // becomes Int64

A Go scalar lifts to a WEAK literal, which adopts the column's own type when the value fits it exactly. That is deliberately not what arithmetic does: Col("i32").Add(1) still widens, because addition can overflow and filling a null cannot. An Expr operand is used as written, so FillNullWith(Lit(0)) widens.

If the value does not fit — FillNullWith(int64(5000)) on an Int8 column — the type falls back to ordinary promotion rather than truncating, so the result is an Int64 column holding 5000 rather than an Int8 column holding -120.

Cost

This desugars to a conditional, so the receiver is MENTIONED twice and, with no common-subexpression pass, EVALUATED twice — exactly as Coalesce is.

func (Expr) First 

func (e Expr) First() Expr

First and Last are POSITIONAL and may return null.

They are a selection rather than a reduction: skipping nulls would mean First(a) and First(b) could come from different rows, which is exactly what people use them together for.

func (Expr) Floor 

func (e Expr) Floor() Expr

Floor and Ceil round towards -Inf and +Inf. Both are the identity on an integer column and both are refused for Decimal, whose stored value is unscaled — so flooring it would floor 1234 rather than 12.34.

func (Expr) FloorDiv 

func (e Expr) FloorDiv[T Operand](v T) Expr

FloorDiv FLOORS, toward negative infinity, for integers, floats and a Duration alike: −7 // 2 is −4, not the −3 Go's / gives. Mod is its remainder, so a == b*FloorDiv(a, b) + Mod(a, b) and Mod takes the divisor's sign: −7 % 2 is 1. Both are Polars' and Python's rule. An integer divisor of zero is null; a float one follows IEEE.

func (Expr) ForwardFill 

func (e Expr) ForwardFill(limit int) Expr

ForwardFill replaces each null with the last non-null value before it.

limit caps how many consecutive nulls one value may fill; 0 means unlimited. A leading run of nulls has nothing to carry and stays null.

Like every ordered function it is a window: with no .Over() it runs over the whole frame in scan order, which makes it a pipeline breaker.

func (Expr) Ge 

func (e Expr) Ge[T Operand](v T) Expr

func (Expr) Gt 

func (e Expr) Gt[T Operand](v T) Expr

func (Expr) Implode added in v0.3.0 

func (e Expr) Implode() Expr

Implode collects every value of the group into a List, in input order. 

GroupBy(Col("k")).Agg(
    Col("v").Implode().Alias("all"),
    Col("v").Implode().List().Sort().List().Head(3).Alias("smallest3"),
)

It is what makes the .list namespace available inside a group-by: everything list.go can do to a list, this can now do to a group.

Nulls are kept

Every other aggregate skips them — the sum of an all-null group is null, not zero. Implode is a selection rather than a reduction, so a null is a value that takes a slot, and an all-null group implodes to a list OF nulls rather than to an empty list or a null one.

It makes the group-by single-threaded

A list's element order is its data, and the parallel driver dispatches batches round-robin, so a shuffled input would be a different answer rather than the same one computed differently. The whole Agg list runs on one worker as a result — including the aggregates beside this one, because a query's aggregates share one hash table.

Its memory is the data

Implode holds every value of every group, so a group-by containing one is O(rows) rather than O(distinct keys), and under WithMemoryLimit it refuses once a single key grows past the budget. Partitioning divides the key space and cannot divide one key.

func (Expr) IsBetween 

func (e Expr) IsBetween[L, H Operand](lo L, hi H, closed ...Closed) Expr

IsBetween tests that e falls between lo and hi. Both bounds lift, so IsBetween(0, 100) works.

The default is CLOSED-BOTH — lo <= e <= hi — which is what this method has always meant and what reads naturally from the name. Pass a Closed to say otherwise: 

Col("ts").IsBetween(start, end, ursus.ClosedLeft)   // start <= ts < end

Variadic rather than a required third argument, so every existing two-argument call keeps compiling and the common case stays short. More than one is a mistake and is refused rather than ignored.

func (Expr) IsClose 

func (e Expr) IsClose(other Expr, relTol, absTol float64) Expr

IsClose reports whether each value is within a relative or absolute tolerance of other. 

|a - b| <= max(relTol * max(|a|, |b|), absTol)

Both guards are load-bearing, and neither is obvious

The EQUALITY disjunct is what makes IsClose(Inf, Inf) true: the tolerance term alone would compute |Inf - Inf|, which is NaN, and NaN <= x is false.

The FINITENESS conjunct is what makes IsClose(Inf, -Inf) false. Without it the relative tolerance scales by max(|a|, |b|) = Inf, the limit becomes Inf, and Inf <= Inf says "close" — which it plainly is not. Python's math.isclose has the same special case for the same reason.

Together they give: equal infinities are close, opposite ones are not, and IsClose(NaN, NaN) is false because IEEE says NaN equals nothing including itself. With a null operand every term is null and Kleene OR gives null, matching every other comparison.

It computes in Float64

Both operands are cast first, so an integer column works — integers are always finite — and the finiteness test, which is float-only, has something to test.

Cost

The receiver is mentioned five times and other four, with no CSE. For two bare columns that is nine column reads; for expensive operands it is not free.

func (Expr) IsDuplicated 

func (e Expr) IsDuplicated() Expr

func (Expr) IsFinite 

func (e Expr) IsFinite() Expr

func (Expr) IsFirstDistinct 

func (e Expr) IsFirstDistinct() Expr

IsFirstDistinct and IsLastDistinct mark the first and last occurrence of each value, in input order.

Note the asymmetry with IsUnique: a value occurring three times has one first occurrence, one last, and is not unique anywhere.

func (Expr) IsIn 

func (e Expr) IsIn[T Literal](vs ...T) Expr

IsIn tests membership in a set of values: Col("region").IsIn("eu", "us").

Values, not an expression

The published sketch was IsIn(other Expr), and it cannot be written: Literal's only slice term is ~[]byte, so there is no way to spell a list-valued literal. The Expr-valued form — testing membership in another frame's column — is a semi join wearing a predicate's clothes, and it belongs with the join machinery.

All values share one type parameter, so the set is homogeneous by construction.

IsIn(v) answers what Eq(v) answers

The column and the values meet at the type == would compare them at, so IsIn refuses exactly the pairs Eq refuses, at plan time — a String against an Int64, an int against a Bool, an instant against a Date — and is false wherever Eq is: 5000 is not in an Int8 column, 0.1 is not in a Float32 one (they compare at Float64, as Polars does), and 00:00:01.5 is not in a Datetime(s) one. To match the float32 nearest 0.1, say so: IsIn(float32(0.1)).

Equality is GROUPING equality

Membership uses the same encoding GroupBy and Distinct use, so NaN matches NaN and -0.0 matches +0.0. An is_in built on IEEE equality would disagree with Distinct about the same data.

Nulls

`null.IsIn(...)` is NULL, not false — the same rule as IsNan, and the reason Filter(p) and Filter(Not(p)) do not partition. The set itself can never contain a null, because these are Go values.

An empty set makes every non-null row false, matching SQL's `x IN ()`.

func (Expr) IsInfinite 

func (e Expr) IsInfinite() Expr

func (Expr) IsLastDistinct 

func (e Expr) IsLastDistinct() Expr

func (Expr) IsNan 

func (e Expr) IsNan() Expr

IsNan and friends ask about a float VALUE, so `null.IsNan()` is null rather than false. Null and NaN are different things; use IsNull to test for missing.

func (Expr) IsNotNan 

func (e Expr) IsNotNan() Expr

func (Expr) IsNotNull 

func (e Expr) IsNotNull() Expr

func (Expr) IsNull 

func (e Expr) IsNull() Expr

IsNull and IsNotNull ask about presence, so their results are never null.

func (Expr) IsUnique 

func (e Expr) IsUnique() Expr

IsUnique reports whether each row's value occurs exactly once in the column. IsDuplicated is its negation for non-null rows.

These are windows, and that is why they were not built earlier

The answer for row 0 depends on the last row, so neither is elementwise. Step 7 deferred them precisely here, and they need no new machinery: a value is unique iff the partition keyed by that value has exactly one row.

Equality is GROUPING equality, the same the partitioning uses — so NaN equals NaN and -0.0 equals +0.0, and IsUnique agrees with Unique() about the same data. Nulls likewise group together, so two nulls are duplicates of each other.

func (Expr) Last 

func (e Expr) Last() Expr

func (Expr) Le 

func (e Expr) Le[T Operand](v T) Expr

func (Expr) Len 

func (e Expr) Len() Expr

Len counts ROWS, including nulls — SQL's COUNT(*).

Count and Len are deliberately separate and are never aliased: over a column with nulls they give different answers, and which one was meant is not recoverable after the fact.

func (Expr) List added in v0.3.0 

func (e Expr) List() ListExpr

List opens the list namespace.

func (Expr) Ln 

func (e Expr) Ln() Expr

Ln is the natural logarithm; Log10 is base 10; Log1p is ln(1+x), which keeps its precision for x near zero where ln(1+x) would lose it.

ln(0) is -Inf and ln(-1) is NaN. Both are values.

func (Expr) Log 

func (e Expr) Log(base float64) Expr

Log is the logarithm to an arbitrary base.

It is SUGAR — ln(x)/ln(base) — rather than a parameterised kernel, which is the same definition Polars uses and costs no new machinery. Two consequences worth knowing, both from the division:

  • Log(10) and Log10() can differ in the last ULP. Log10 is the exact form.
  • On a Float32 column Log10() stays Float32 and Log(10) becomes Float64, because dividing by a float64 promotes. Cast the result back if the narrow width was a deliberate choice.

func (Expr) Log1p 

func (e Expr) Log1p() Expr

func (Expr) Log10 

func (e Expr) Log10() Expr

func (Expr) Lt 

func (e Expr) Lt[T Operand](v T) Expr

func (Expr) MapBatches added in v0.3.0 

func (e Expr) MapBatches[In, Out Literal](name string, out DataType,
	fn func([]In, []bool) ([]Out, []bool, error)) Expr

MapBatches applies a Go function to a whole column at a time.

The vectorized escape hatch, and the one to reach for first: one call per batch rather than one per row, and full control of the validity mask. 

Col("v").MapBatches[float64, float64](
    "normalise", ursus.Float64,
    func(vals []float64, ok []bool) ([]float64, []bool, error) { ... })

The two slices are positional and the same length

vals[i] is row i and ok[i] says whether it is present. A null row's vals entry is the zero value and must not be read. The returned pair must be the same length as the input; returning a nil mask means every output row is valid.

vals ALIASES the engine's own buffer when the layout permits, so fn must not write to it. Build a new slice for the result.

The notes on name, goroutine safety and out under MapElements apply here identically.

func (Expr) MapElements added in v0.3.0 

func (e Expr) MapElements[In, Out Literal](name string, out DataType,
	fn func(In) (Out, error)) Expr

MapElements applies a Go function to every non-null value.

The per-element escape hatch. In Polars this is the slow one, because each call crosses into the Python interpreter; in Go it is a function call, which is what makes it worth offering rather than merely tolerating. 

Col("celsius").MapElements[float64, float64](
    "to_fahrenheit", ursus.Float64,
    func(c float64) (float64, error) { return c*9/5 + 32, nil })

Nulls pass through

fn never sees a null: a null input row is a null output row, and fn is not called for it. That is deliberate — the alternative hands fn a zero value it cannot distinguish from a real one. Use MapBatches when the null matters.

name must be unique within the query, and this is CHECKED

A Go closure has no printable form, so the name is all that Explain and every error message can show of a udf. Two different functions sharing one could not be told apart there — and until the engine stopped deduplicating expressions by their rendering, they were one computation, with both results coming from whichever it saw first.

plan.CheckUDFNames refuses that at plan time, before any data is read, so the sentence above is a guarantee rather than advice. Two udfs with one name are an error; ONE udf used twice is not, so if two calls are meant to be the same function, hold the Expr in a variable and use it twice rather than calling a helper that builds a fresh closure each time.

A panic in fn is an error

It is returned like an error fn returns — KindValue, naming the udf, the row and the column — whichever goroutine fn ran on. Before step 72 it killed the process.

fn must be safe to call from several goroutines

ursus parallelises a Select or Filter across runtime.NumCPU() workers by default, and there is no way to opt a single expression out. A closure over a shared map is a data race. fn must also not depend on the ORDER it is called in: output order is preserved, invocation order is not.

out is a declaration, and it is checked

The engine reports out as the column's type from CollectSchema and Explain, before fn has ever run. If fn then produces something else, the evaluator fails the query rather than letting a wrong schema propagate.

Example

A Go function over each value, taking part in the plan like any expression. Nulls pass through without calling it. 

package main

import (
	"context"
	"fmt"
	"log"
	"strings"

	"github.com/advenn/ursus"
)

// exampleSales is the little frame the examples below work on.
func exampleSales() *ursus.LazyFrame {
	return ursus.Frame(
		ursus.Values("region", []string{"north", "south", "north", "east", "south"}),
		ursus.Values("product", []string{"apple", "apple", "pear", "pear", "fig"}),
		ursus.Values("qty", []int64{10, 4, 7, 2, 9}),
		ursus.Values("price", []float64{1.50, 1.50, 2.25, 2.25, 3.00}),
	)
}

func main() {
	df, err := exampleSales().Select(
		ursus.Col("product"),
		ursus.Col("product").MapElements("shout", ursus.String,
			func(s string) (string, error) { return strings.ToUpper(s) + "!", nil }).Alias("loud"),
	).Head(3).Collect(context.Background())
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(df)
}

Output:
shape: (3, 2)
┌─────────┬──────────┐
│ product │ loud     │
│ String  │ String   │
├─────────┼──────────┤
│ "apple" │ "APPLE!" │
│ "apple" │ "APPLE!" │
│ "pear"  │ "PEAR!"  │
└─────────┴──────────┘

func (Expr) MapName 

func (e Expr) MapName(fn func(string) string) Expr

MapName derives the output name from the input name.

func (Expr) Max 

func (e Expr) Max() Expr

func (Expr) Mean 

func (e Expr) Mean() Expr

Mean is the sum of non-null values divided by the COUNT of non-null values — never by the group size, which would silently bias the result downward whenever nulls are present.

Mean of a group with no non-null values is NULL, not NaN: there was nothing to compute, as opposed to a computation that came out undefined.

func (Expr) Median 

func (e Expr) Median() Expr

Median is the 0.5 quantile with linear interpolation, so an even-sized group gives the mean of the two middle values.

It returns Float64 for every numeric input, and skips nulls.

func (Expr) Min 

func (e Expr) Min() Expr

Min and Max use ursus's TOTAL order, the same one Sort uses: NaN sorts above everything and -0.0 equals +0.0. They therefore always agree with Sort(...).First() / .Last(), which IEEE comparison would not.

func (Expr) Mod 

func (e Expr) Mod[T Operand](v T) Expr

func (Expr) Mul 

func (e Expr) Mul[T Operand](v T) Expr

func (Expr) NUnique 

func (e Expr) NUnique() Expr

NUnique counts distinct non-null values.

Nulls are SKIPPED, so NUnique over an all-null group is 0. Polars counts null as a distinct value; ursus follows SQL and, more importantly, follows its own other aggregates.

func (Expr) Ne 

func (e Expr) Ne[T Operand](v T) Expr

func (Expr) NeMissing 

func (e Expr) NeMissing[T Operand](v T) Expr

func (Expr) Neg 

func (e Expr) Neg() Expr

Neg negates. Abs takes the absolute value.

Both are defined for every numeric type, including the unsigned integers, Int128 and Decimal — which matters more than it sounds, because every integer Sum outputs Int128, so `Col("x").Sum().Abs()` is an ordinary query.

func (Expr) Not 

func (e Expr) Not() Expr

Not is Kleene negation: NOT null is null.

func (Expr) NullCount 

func (e Expr) NullCount() Expr

NullCount counts the NULLS in each group — the complement of Count, and like the other counters it is never itself null.

func (Expr) Or 

func (e Expr) Or[T Operand](v T) Expr

func (Expr) Over 

func (e Expr) Over(partitionBy ...Expr) Expr

Over computes the expression within partitions and writes the answer back onto every row. 

// each row's deviation from its group's mean
ursus.Col("x").Sub(ursus.Col("x").Mean().Over(ursus.Col("g")))

// dense rank of speed within each type
ursus.Col("speed").Rank(ursus.RankDense, true).Over(ursus.Col("type"))

The distinction from GroupBy

GroupBy REDUCES: N rows in, one row per group out. Over PRESERVES: N rows in, N rows out, each carrying its own partition's answer. That is why an aggregate is refused in Select but the same aggregate under Over is not.

With no partition keys the whole frame is one partition, so `Col("x").Sum().Over()` puts the grand total on every row.

Nulls form their own partition

A null partition key groups with other nulls, the same rule GroupBy uses — as opposed to joins, where null keys match nothing. The two differ on purpose and each is documented where it applies.

func (Expr) OverWith 

func (e Expr) OverWith(spec WindowSpec) Expr

OverWith is Over with an ordering and a mapping strategy.

An ordering is what makes Rank, the cumulative functions and Shift meaningful: without one they run in input order, which is well-defined but rarely what a ranking wants.

func (Expr) PctChange 

func (e Expr) PctChange(n int) Expr

PctChange is the fractional change from the value n rows earlier.

Always a float on a numeric column, because it divides — and a zero previous value gives ±Inf rather than a null, since float division is total. A Float32 stays Float32. On an instant it is a plan-time error.

The value is taken to a float BEFORE the subtraction

It used to subtract at the column's own width, and integer subtraction wraps: a UInt8 that went from 3 to 1 changed by 254, so its pct_change was 84.67 where the answer is −0.67. The public Expr has no type in hand to choose a cast with, so an internal call, typed at plan time, takes an integer or a Decimal to Float64 and a Duration's ticks to Float64, keeps a float as it is, and refuses an instant.

func (Expr) Pow 

func (e Expr) Pow[T Operand](v T) Expr

Pow raises the receiver to the power v.

The result is ALWAYS a float, for the same reason Div's is: 2**-1 is 0.5, and a version that truncated negative powers to zero would be surprising in the direction that loses data silently. Cast the result if an integer is wanted.

func (Expr) Prefix 

func (e Expr) Prefix(p string) Expr

Prefix prepends to the output name. Unlike Alias it is expansion-safe, deriving a distinct name per output column.

func (Expr) Product 

func (e Expr) Product() Expr

Product multiplies the non-null values.

Unlike Sum it returns FLOAT64 for every numeric input, including integers. Sum widened to Int128 so that overflow became unreachable; product cannot, because i128 has no multiplication — and an Int64 product wraps silently after roughly twenty ordinary factors. Float64 is exact to 2^53 and approximate above it, which is the weaker guarantee, stated rather than hidden.

Product of a group with no non-null values is NULL, not 1, following Sum.

func (Expr) Quantile 

func (e Expr) Quantile(q float64, interp Interpolation) Expr

Quantile returns the value at rank q, where q runs from 0 (the minimum) to 1 (the maximum). interp decides what happens when the rank falls between two values.

The rank is q·(n−1) over the non-null values — the convention numpy, Polars and DuckDB share — and the result is Float64 even for InterpLower and InterpHigher, which do return an actual input value.

Values are ordered by ursus's TOTAL order, so NaN sorts above everything and a high quantile of a column containing NaN is NaN, exactly as Sort would place it.

func (Expr) Rank 

func (e Expr) Rank(method RankMethod, descending bool) Expr

Rank numbers rows by value within the window, smallest first unless descending.

Ties are resolved by method; see RankMethod. Rank is never null — a null value still occupies a position — and returns Uint32, or Float64 for RankAverage.

func (Expr) Round 

func (e Expr) Round(decimals int) Expr

Round rounds to decimals decimal places, HALF AWAY FROM ZERO.

Round(0.5) is 1 and Round(2.5) is 3, as a spreadsheet rounds, rather than IEEE's round-half-to-even. That is Polars' `mode="half_away_from_zero"`, and NOT Polars' default, which rounds half to even: Polars' `round(0)` gives 2 for 2.5. It preserves the operand's type: rounding an integer column is the identity, not a widening.

It rounds twice — the value is scaled by 10^decimals first, and that multiply rounds too. Round(2.675, 2) is 2.68, because 2.675 * 100 lands on exactly 267.5 even though 2.675 itself is stored as 2.67499999999999982…

A Float32 is rounded as a Float64 and the result rounded once more to a float32, so Round(0.1, 1) of a Float32 is the float32 nearest 0.1.

func (Expr) Shift 

func (e Expr) Shift(n int) Expr

Shift moves values n places later within the window, filling the vacated rows with null. A negative n shifts earlier — Polars' lag and lead, one function.

Rows shifted in from outside the partition are null, never a neighbouring partition's value.

func (Expr) ShiftFill 

func (e Expr) ShiftFill[T Operand](n int, fill T) Expr

ShiftFill is Shift with a value instead of null in the vacated rows.

The fill meets the column as FillNullWith's does: a Go value that fits the column's type takes it, so ShiftFill(1, 0) on an Int32 column is an Int32, and otherwise the two promote, so ShiftFill(1, 1.5) on an Int64 column is a Float64.

func (Expr) Sign 

func (e Expr) Sign() Expr

Sign is -1, 0 or 1, keeping the operand's type.

For floats it returns the operand itself in the zero case, so sign(-0.0) is -0.0 and sign(NaN) is NaN. That matches NumPy and falls out of the ordering test rather than being special-cased.

func (Expr) Sqrt 

func (e Expr) Sqrt() Expr

Sqrt, Cbrt and Exp. Sqrt of a negative is NaN, not an error and not a null.

func (Expr) Std 

func (e Expr) Std(ddof int) Expr

func (Expr) Str 

func (e Expr) Str() StrExpr

Str opens the string namespace.

func (Expr) String 

func (e Expr) String() string

String renders the expression canonically.

func (Expr) Struct added in v0.3.0 

func (e Expr) Struct() StructExpr

Struct opens the struct namespace.

func (Expr) Sub 

func (e Expr) Sub[T Operand](v T) Expr

func (Expr) Suffix 

func (e Expr) Suffix(s string) Expr

Suffix appends to the output name. Expansion-safe, like Prefix.

func (Expr) Sum 

func (e Expr) Sum() Expr

Sum adds the non-null values.

An INTEGER sum accumulates and returns Int128, following DuckDB. Summing a thousand Int8 values overflows an Int8 immediately, and a wrapped sum is a plausible-looking wrong number that nothing downstream can detect. At 128 bits overflow needs 2^63 maximum-magnitude rows, i.e. it cannot happen.

A FLOAT sum accumulates in Float64 even when it returns Float32, because naive float32 accumulation stops making progress past ~2^24 elements.

A BOOL sums as 0 and 1, so its sum counts the trues, as an Int128. Count counts the non-null rows, falses included.

Sum of a group with no non-null values is NULL, not 0 — ursus follows SQL here rather than Polars. `0` cannot be distinguished afterwards from a genuine zero; NULL can be turned into 0 with FillNull if that is what you want.

func (Expr) Unnest added in v0.3.0 

func (e Expr) Unnest() Expr

Unnest is refused as an expression, and for a DIFFERENT reason than Explode.

Explode is not an expression because it changes the frame's HEIGHT. Unnest leaves the height alone and changes the WIDTH: an expression produces exactly one column, and this produces one per field. Saying which of the two rules was broken is the difference between a message that teaches and one that just refuses.

The single-column form exists and is spelled `Col("person").Struct().Field("age")`.

func (Expr) Var 

func (e Expr) Var(ddof int) Expr

Var and Std are the variance and standard deviation of the non-null values.

ddof is the delta degrees of freedom: 0 for the population statistic, 1 for the sample one. It has no default because there is no defensible default — Polars and pandas use 1, numpy uses 0, and silently picking either produces a number that is wrong for half its readers.

A group with ddof or fewer values returns NULL: the sample variance of a single observation is undefined, and 0 would claim the data has no spread rather than that the question has no answer.

Computed with Welford's algorithm, so a column of large near-equal values gives the right answer rather than the zero (or negative) that E[x²]−E[x]² produces.

func (Expr) Xor 

func (e Expr) Xor[T Operand](v T) Expr

type Field 

type Field = dtype.Field

Field is a named, typed, nullable column slot.

type FillStrategy 

type FillStrategy uint8

FillStrategy names a fill rule that needs no value at the call site.

It is a root-package enum and never reaches the IR, so it is not a dedup key and carries none of the collision hazard the expression enums do. 

const (
	// FillZero and FillOne fill with a constant. The literal is WEAK, so the
	// column's type is preserved rather than widened — which matters more here than
	// anywhere else, because the call site contains no literal at all to explain a
	// widening.
	FillZero FillStrategy = iota
	FillOne

	// FillForward and FillBackward carry the neighbouring value. Both are ordered
	// window functions, so both are pipeline breakers.
	FillForward
	FillBackward

	// FillMin, FillMax and FillMean fill with the column's own aggregate over the
	// WHOLE FRAME. That makes them pipeline breakers too, and worse ones: the
	// window sink holds every row, and it is one of the operators that is accounted
	// and refused under a memory limit rather than spilled.
	//
	// FillMean returns a FLOAT even for an integer column, because a mean is a
	// float. FillMin and FillMax preserve the type.
	FillMin
	FillMax
	FillMean
)

func (FillStrategy) String 

func (s FillStrategy) String() string

type GroupBy 

type GroupBy struct {
	// contains filtered or unexported fields
}

GroupBy is the intermediate handle between GroupBy and Agg.

func (*GroupBy) Agg 

func (g *GroupBy) Agg(exprs ...Expr) *LazyFrame

Agg reduces each group.

Every expression must be an aggregation: every column it reads has to pass through an aggregate function. `Col("a").Sum().Add(1)` is fine; `Col("a").Sum().Add(Col("b"))` is not, because b is still a per-row value and there is no defined way to combine the two shapes.

func (*GroupBy) Count 

func (g *GroupBy) Count() *LazyFrame

Count is shorthand for Agg(Len().Alias("count")).

func (*GroupBy) First 

func (g *GroupBy) First() *LazyFrame

func (*GroupBy) Last 

func (g *GroupBy) Last() *LazyFrame

func (*GroupBy) Len 

func (g *GroupBy) Len(name string) *LazyFrame

Len counts the rows in each group under the given name.

Distinct from Count, which uses the fixed name "count": Len takes one because a frame that already has a column called "count" is not unusual, and silently colliding would be worse than asking.

func (*GroupBy) MaintainOrder 

func (g *GroupBy) MaintainOrder() *GroupBy

MaintainOrder emits groups in first-appearance order.

Without it the group order is unspecified — which is standard (SQL uses set semantics and DuckDB exploits it for parallelism) but means a test that assumes an order is flaky.

The flag is no longer free, and that is the point of it

It USED to document a guarantee the implementation happened to give anyway. Since step 12 a group-by over more distinct keys than the memory limit holds partitions the overflow to disk, and the unordered path then emits partition-major: resident groups first, then partition 0's groups, then partition 1's. The order depends on where the freeze fell, so it depends on the limit.

The CONTENT never does. Every group, every value, identical at every limit. See WithMemoryLimit, which states the same contract from the other side.

Costs nothing when the aggregation fits in memory. When it spills it adds an int64 ordinal to every routed row and materialises the whole result to permute it — plan.Aggregate.MaintainOrder has the detail.

func (*GroupBy) Max 

func (g *GroupBy) Max() *LazyFrame

func (*GroupBy) Mean 

func (g *GroupBy) Mean() *LazyFrame

func (*GroupBy) Median 

func (g *GroupBy) Median() *LazyFrame

func (*GroupBy) Min 

func (g *GroupBy) Min() *LazyFrame

Min, Max, First, Last and NUnique apply to EVERY non-key column, because they select or count rather than compute.

func (*GroupBy) NUnique 

func (g *GroupBy) NUnique() *LazyFrame

func (*GroupBy) Quantile 

func (g *GroupBy) Quantile(q float64, interp Interpolation) *LazyFrame

Quantile applies one quantile to every numeric non-key column.

func (*GroupBy) Sum 

func (g *GroupBy) Sum() *LazyFrame

Sum, Mean, Min, Max, Median, NUnique, First and Last apply that aggregate to every non-key column. Sum, Mean, Median and Quantile apply to the NUMERIC non-key columns.

type Int128Value 

type Int128Value = i128.Int128

Int128Value is a signed 128-bit integer, the Go representation of an Int128 column. Read one with df.Column[ursus.Int128Value]("total").

type Interpolation 

type Interpolation = expr.Interpolation

Interpolation selects how Quantile resolves a rank falling between two values.

Aliased from the expression IR rather than redeclared, so the kernel and the public API cannot drift — the same arrangement JoinKind uses.

type Interval 

type Interval = dtype.Interval

Interval is a calendar-aware span: months, days and nanoseconds, kept apart because each obeys a different rule. See dtype.Interval for why a time.Duration cannot express "1 month" or, in a daylight-saving zone, "1 day".

func Every 

func Every(s string) Interval

Every parses an interval: "1y", "3mo", "1w", "2d", "6h", "15m", "30s", "1ns". Components may be concatenated ("1h30m") and a leading "-" negates the whole thing.

A parse failure is carried in the value rather than returned, and surfaces when the query is built — the deferred-error rule the rest of the library follows. Note "m" is minutes and "mo" is months.

func FromDuration 

func FromDuration(d time.Duration) Interval

FromDuration builds an ABSOLUTE interval. Deliberately not the same as the calendar spelling: FromDuration(24*time.Hour) is twenty-four hours, Every("1d") is one day, and on the two days a year a zone changes its offset those are different instants.

type JoinKind 

type JoinKind = plan.JoinKind

JoinKind is the shape of a join: which rows survive.

type JoinOption 

type JoinOption func(*joinCfg)

JoinOption configures a join.

func JoinCoalesce 

func JoinCoalesce(b bool) JoinOption

JoinCoalesce forces key merging on or off.

Unset is not the same as either: by default the key appears once for inner, left, right, semi and anti joins, and twice for a full join — which can leave the key null on either side, so there is no single side to take it from.

The default only merges keys that are named the SAME on both sides. Merging JoinLeftOn(Col("cust_id")) with JoinRightOn(Col("id")) would silently delete a column the caller named explicitly, so it takes an explicit JoinCoalesce(true).

func JoinHow 

func JoinHow(k JoinKind) JoinOption

JoinHow sets the join kind. Default JoinInner.

func JoinLeftOn 

func JoinLeftOn(keys ...Expr) JoinOption

JoinLeftOn and JoinRightOn name the key columns separately, for frames where they are called different things. They are used together and must agree in count; keys are paired positionally.

func JoinNullsEqual 

func JoinNullsEqual(b bool) JoinOption

JoinNullsEqual makes null keys match each other. Default false, which is SQL's rule and Polars': a null key matches nothing, including another null.

Note this is deliberately the OPPOSITE of GroupBy, where null forms its own group. Grouping asks "are these the same value?" and joining asks "is this the same entity?", and a missing identifier is not evidence of sameness.

func JoinOn 

func JoinOn(keys ...Expr) JoinOption

JoinOn names key columns present under the same name on both sides.

func JoinRightOn 

func JoinRightOn(keys ...Expr) JoinOption

func JoinSuffix 

func JoinSuffix(s string) JoinOption

JoinSuffix sets what is appended to a right-side column whose name collides with a left-side one. Default "_right".

func JoinValidate 

func JoinValidate(v JoinValidation) JoinOption

JoinValidate asserts a key cardinality, failing the query when the data violates it. 

orders.Join(customers, JoinOn(Col("customer_id")),
    JoinValidate(ursus.ValidateManyToOne))

Worth reaching for: an unexpected duplicate on the right multiplies every matching left row, and the result looks entirely plausible.

type JoinValidation 

type JoinValidation = plan.JoinValidation

JoinValidation asserts a cardinality and errors when the data violates it.

The first term names the LEFT frame: ManyToOne on orders.Join(customers) asserts that the customers are unique.

type LazyFrame 

type LazyFrame struct {
	// contains filtered or unexported fields
}

LazyFrame is an unexecuted query.

Immutable

Every builder method returns a NEW LazyFrame sharing an immutable plan. So 

base := ursus.Scan(src)
a := base.Filter(ursus.Col("x").Gt(5))
b := base.Filter(ursus.Col("x").Lt(0))

gives two independent queries, and `base` is unchanged. That idiom is the most common thing anyone does with a lazy frame, and it only works because nothing mutates in place. It also makes a LazyFrame safe to share between goroutines.

Sticky errors

A builder cannot return an error without destroying chaining, so errors are carried and surfaced at Collect (or read early via Err). An unknown column in Select does not panic and does not fail silently — it fails at Collect, with the available columns and a spelling suggestion.

func Concat 

func Concat(frames []*LazyFrame, opts ...ConcatOption) *LazyFrame

Concat stacks frames vertically, in the order given. 

ursus.Concat([]*ursus.LazyFrame{jan, feb, mar})
ursus.Concat(fs, ursus.WithConcatMode(ursus.ConcatDiagonal))

Why a slice rather than a variadic

It was `Concat(frames ...any)` for one release, which read better and let the mode be passed inline. It also made `ursus.Concat(a, "oops")` compile and fail at run time — in a library that spends generic methods and the Operand constraint precisely so that `Col("x").Gt(struct{}{})` does not. Ergonomics is not worth the one property the type system was being paid to provide.

For the common case the method form keeps both: LazyFrame.Concat is variadic and type-safe, so `jan.Concat(feb, mar)` needs no slice literal.

Order is preserved

Every row of the first frame precedes every row of the second. That is not free — it is why the operator drains children serially rather than racing them — and it is what lets Concat compose with Head, Tail and anything else that cares which rows come first.

Reconciliation happens in the plan

A frame missing a column, or holding one at a narrower type, is wrapped in a projection that supplies the null and performs the cast, so Explain shows why a column is null and projection pushdown still narrows each input independently.

func Frame 

func Frame(cols ...*Column) *LazyFrame

Frame builds an in-memory LazyFrame from columns.

This is the entry point for data that is already in Go — the counterpart to ScanParquet and friends, which arrive in step 2.

func FrameOf 

func FrameOf(cols ...*Column) (*LazyFrame, error)

FrameOf is Frame for callers who want the error separately.

func FromPlan 

func FromPlan(n plan.Node) *LazyFrame

FromPlan builds a LazyFrame over an existing logical plan. It exists so that a future SQL frontend can lower into the same IR and hand back a frame, without the root package having to import it — which would be a cycle.

func Scan 

func Scan(src plan.Source) *LazyFrame

Scan starts a query against a source.

func ScanArrow added in v0.3.0 

func ScanArrow(open func() (array.RecordReader, error)) *LazyFrame

ScanArrow reads a stream of Arrow records.

open must return a NEW, independent reader every time it is called: ursus opens one to learn the schema when the query is planned, and one per scan each time it runs — a self-join scans twice in one query, and a LazyFrame can be collected again. A reader cannot be rewound, so handing over one reader would be read once and then be empty. 

stream, err := os.ReadFile("data.arrows") // an Arrow IPC stream
if err != nil { return err }
lf := ursus.ScanArrow(func() (array.RecordReader, error) {
    return ipc.NewReader(bytes.NewReader(stream))
})

Returning the same reader twice is refused by name rather than read as an empty stream.

Records are copied a batch at a time, and the reader is not advanced until its current record has been copied in full, so nothing ursus returns refers to memory the reader reuses or frees. Explain and CollectSchema open a reader and release it without reading a record. Head(n) stops asking for records once it has n rows. A reader blocked inside Next cannot be interrupted by the query's context.

Every column is nullable, and types are mapped as ScanArrowRecords describes.

func ScanArrowRecords added in v0.3.0 

func ScanArrowRecords(recs ...arrow.RecordBatch) *LazyFrame

ScanArrowRecords reads Arrow records already in memory.

The records are COPIED before this returns, so the caller may release them, or let the reader they came from move on, as soon as it likes: 

var recs []arrow.RecordBatch
for rdr.Next() {
    rec := rdr.RecordBatch()
    rec.Retain() // an IPC reader reuses the record on its next Next
    recs = append(recs, rec)
}
lf := ursus.ScanArrowRecords(recs...)
for _, rec := range recs {
    rec.Release()
}

Every record must convert to the same schema. Every column is nullable whatever the Arrow schema declares, because arrow-go never checks that flag and producers routinely leave it false while writing nulls. A type ursus cannot represent — a 256-bit decimal, an interval, a union — is refused by name.

With no records there is no schema, so that is an error too; hand over an empty record instead. For a stream too large to copy up front, use ScanArrow.

Example

Arrow both ways. Record shares a frame's memory with arrow-go; ScanArrowRecords copies records in, so the record can be released straight away. 

package main

import (
	"context"
	"fmt"
	"log"

	"github.com/advenn/ursus"
)

// exampleSales is the little frame the examples below work on.
func exampleSales() *ursus.LazyFrame {
	return ursus.Frame(
		ursus.Values("region", []string{"north", "south", "north", "east", "south"}),
		ursus.Values("product", []string{"apple", "apple", "pear", "pear", "fig"}),
		ursus.Values("qty", []int64{10, 4, 7, 2, 9}),
		ursus.Values("price", []float64{1.50, 1.50, 2.25, 2.25, 3.00}),
	)
}

func main() {
	ctx := context.Background()
	big, err := exampleSales().Filter(ursus.Col("qty").Gt(5)).Collect(ctx)
	if err != nil {
		log.Fatal(err)
	}
	rec, err := big.Record()
	if err != nil {
		log.Fatal(err)
	}

	lf := ursus.ScanArrowRecords(rec)
	rec.Release()

	df, err := lf.Select(ursus.Col("region"), ursus.Col("qty")).Collect(ctx)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(df)
}

Output:
shape: (3, 2)
┌─────────┬───────┐
│ region  │ qty   │
│ String  │ Int64 │
├─────────┼───────┤
│ "north" │ 10    │
│ "north" │ 7     │
│ "south" │ 9     │
└─────────┴───────┘

func ScanCSV 

func ScanCSV(path string, opts ...CSVOption) *LazyFrame

ScanCSV reads a CSV file lazily.

Nothing is read until Collect, except the sample inference needs — and not even that if WithSchema is given. Projection reaches the reader, so selecting two columns of a hundred parses two. 

df, err := ursus.ScanCSV("sales.csv").
    Filter(ursus.Col("qty").Gt(10)).
    Select(ursus.Col("region"), ursus.Col("qty")).
    Collect(ctx)

func ScanCSVFiles 

func ScanCSVFiles(paths []string, opts ...CSVOption) *LazyFrame

ScanCSVFiles reads several files as one frame, in the order given.

Columns are matched by the NAMES in each file's header: every file must name the same columns, in any order. A file that differs — a missing, extra, renamed or repeated column, or no header at all — is refused with ErrSchema, naming both files, when the scan reaches it. The types are inferred from the first file (or given by WithSchema). WithColumnNames renames the first file's columns by position, and later files are still matched to it by what the files say. With WithHasHeader(false) there is nothing to match, and files are read by position. A UTF-8 byte-order mark at the start of a file is dropped.

func ScanCSVFrom added in v0.3.0 

func ScanCSVFrom(files []CSVFile, opts ...CSVOption) *LazyFrame

ScanCSVFrom reads CSV files the caller opens, as one frame in the order given, under ScanCSVFiles' rules for several files.

It is how ursus reaches an object store without depending on one: give each file an Open that returns the body of a GET.

func ScanCSVGlob 

func ScanCSVGlob(pattern string, opts ...CSVOption) *LazyFrame

ScanCSVGlob reads every file matching a shell pattern as one frame.

The files must share their columns; the types are inferred from the first match, and every other file's header is matched to the first's by NAME — see ScanCSVFiles. Matches are sorted, so `part-*.csv` reads in the order the names imply rather than whatever the filesystem returns — row order is a property of the data here, and leaving it to readdir would make the same query return different orders on different machines.

func ScanCSVReader 

func ScanCSVReader(b []byte, name string, opts ...CSVOption) *LazyFrame

ScanCSVReader reads CSV from memory.

It takes the bytes rather than an io.Reader because a source is opened more than once — once for inference, once per execution — and a Reader cannot be rewound.

func ScanParquet 

func ScanParquet(path string, opts ...ParquetOption) *LazyFrame

ScanParquet reads a Parquet file lazily.

Only the footer is read until Collect. Projection reaches the reader, so a query selecting two columns of a hundred never decompresses the other ninety-eight, and a predicate on a column with statistics skips whole row groups. 

df, err := ursus.ScanParquet("events.parquet").
    Filter(ursus.Col("ts").Gt(cutoff)).
    Select(ursus.Col("user"), ursus.Col("ts")).
    Collect(ctx)

Lists and structs are read; a column ursus cannot read — an encrypted one, a map, a list of lists — is refused with an error naming it when a query reads it, rather than dropped.

func ScanParquetBytes 

func ScanParquetBytes(b []byte, name string, opts ...ParquetOption) *LazyFrame

ScanParquetBytes reads a Parquet file from memory.

func ScanParquetFiles 

func ScanParquetFiles(paths []string, opts ...ParquetOption) *LazyFrame

ScanParquetFiles reads several files as one frame, in the order given.

Columns are matched by NAME: every file must have the same columns, of the same types, in any order. A file that differs is refused with ErrSchema, naming both files and the difference, when the query is planned — every file's footer is read then, before any row is produced. A column is nullable if it is nullable in any file. To combine files that differ, scan them separately and Concat them.

func ScanParquetFrom added in v0.3.0 

func ScanParquetFrom(files []ParquetFile, opts ...ParquetOption) *LazyFrame

ScanParquetFrom reads Parquet files the caller opens, as one frame in the order given, under ScanParquetFiles' rules for several files.

It is how ursus reaches an object store without depending on one: give each file an Open that returns a ReaderAt over ranged GETs. 

files := []ursus.ParquetFile{{
    Name: "s3://logs/2026/10/part-0.parquet",
    Open: func(ctx context.Context) (io.ReaderAt, int64, error) {
        return newRangedReader(ctx, client, "logs", "2026/10/part-0.parquet")
    },
}}
df, err := ursus.ScanParquetFrom(files).Filter(ursus.Col("status").Eq(500)).Collect(ctx)
Example

Read Parquet from wherever it lives. Open returns an io.ReaderAt and its size — here over bytes in memory, standing in for a ranged GET against an object store. ursus reads the footer, then only the column chunks the query needs. 

package main

import (
	"bytes"
	"context"
	"fmt"
	"io"
	"log"

	"github.com/advenn/ursus"
)

// exampleSales is the little frame the examples below work on.
func exampleSales() *ursus.LazyFrame {
	return ursus.Frame(
		ursus.Values("region", []string{"north", "south", "north", "east", "south"}),
		ursus.Values("product", []string{"apple", "apple", "pear", "pear", "fig"}),
		ursus.Values("qty", []int64{10, 4, 7, 2, 9}),
		ursus.Values("price", []float64{1.50, 1.50, 2.25, 2.25, 3.00}),
	)
}

func main() {
	ctx := context.Background()
	var file bytes.Buffer
	if err := exampleSales().WriteParquet(ctx, &file); err != nil {
		log.Fatal(err)
	}

	lf := ursus.ScanParquetFrom([]ursus.ParquetFile{{
		Name: "s3://shop/sales.parquet",
		Open: func(ctx context.Context) (io.ReaderAt, int64, error) {
			return bytes.NewReader(file.Bytes()), int64(file.Len()), nil
		},
	}})
	n, err := lf.Filter(ursus.Col("product").Eq("pear")).Count(ctx)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(n, "pear sales")
}

Output:
2 pear sales

func ScanParquetGlob 

func ScanParquetGlob(pattern string, opts ...ParquetOption) *LazyFrame

ScanParquetGlob reads every file matching a shell pattern as one frame, sorted by name so `part-*.parquet` reads in the order the names imply.

func (*LazyFrame) BottomK 

func (lf *LazyFrame) BottomK(k int, by ...SortKey) *LazyFrame

BottomK keeps the k smallest rows by the given keys. See TopK — including that nulls are placed last and so never appear in the result.

func (*LazyFrame) Collect 

func (lf *LazyFrame) Collect(ctx context.Context, opts ...CollectOption) (df *DataFrame, err error)

Collect runs the query and returns the whole result.

A panic is an error

Every entry point that runs a query recovers a panic in it and returns it as ErrInternal — "this is a bug in ursus" — and every goroutine the engine starts does the same for its own work, so no query can take down the process that asked it. A panic in a udf is the caller's own, and is a KindValue error naming the udf.

func (*LazyFrame) CollectBatches 

func (lf *LazyFrame) CollectBatches(ctx context.Context, opts ...CollectOption) iter.Seq2[*DataFrame, error]

CollectBatches streams the result without materialising the whole frame.

Breaking out of the loop tears the operator tree down, so an early exit is safe.

A panic while planning, or in any operator, is an error yielded like any other. A panic in the loop BODY is the caller's own and propagates as it would from any loop: recovering around yield is something Go forbids a range function to do.

func (*LazyFrame) CollectInto 

func (lf *LazyFrame) CollectInto[T any](ctx context.Context, opts ...CollectOption) (rows []T, err error)

CollectInto runs the query and decodes each row into a T.

A generic METHOD, which is the Go 1.27 feature this library was waiting for: before, this had to be a package-level function taking the frame as an argument.

func (*LazyFrame) CollectRecords added in v0.3.0 

func (lf *LazyFrame) CollectRecords(ctx context.Context, opts ...CollectOption) iter.Seq2[arrow.Record, error]

CollectRecords streams the result as Arrow records without materialising the whole frame, which is what makes a larger-than-RAM result exportable.

It is CollectBatches with the conversion applied, and it inherits every property: breaking out of the loop tears the operator tree down, so an early exit is safe. 

for rec, err := range lf.CollectRecords(ctx) {
    if err != nil { return err }
    if err := w.Write(rec); err != nil { rec.Release(); return err }
    rec.Release()
}

Each record shares the memory of the batch it came from, and the engine reuses nothing across batches, so a record stays valid after the next one arrives.

func (*LazyFrame) CollectSchema 

func (lf *LazyFrame) CollectSchema(ctx context.Context) (s *Schema, err error)

CollectSchema returns the result schema without reading any data.

It takes a context because resolving a source's schema is real I/O — a Parquet footer, a CSV sample. A schema request that could not be cancelled would be a hole in the cancellation story, which is why this is not the no-argument Schema() the design documents originally specified.

func (*LazyFrame) Concat 

func (lf *LazyFrame) Concat(others ...*LazyFrame) *LazyFrame

Concat stacks other frames beneath this one.

func (*LazyFrame) Count 

func (lf *LazyFrame) Count(ctx context.Context, opts ...CollectOption) (n int64, err error)

Count runs the query and returns the row count, retaining no data.

func (*LazyFrame) Drop 

func (lf *LazyFrame) Drop(names ...string) *LazyFrame

Drop removes columns by name.

Sugar over `Select(Exclude(...))`, which is the whole implementation: Exclude is already an expansion-time selector, so Drop inherits its error messages, its interaction with projection pushdown, and its behaviour on a name that is not there — the selector ignores it rather than failing, matching Polars.

func (*LazyFrame) DropNulls 

func (lf *LazyFrame) DropNulls(subset ...string) *LazyFrame

DropNulls removes rows that are null in any of the named columns, or in any column at all when no names are given.

It is `Filter(Col(c).IsNotNull(), ...)`, and being a filter rather than a dedicated node is what earns it Parquet row-group pruning: the pruner claims IsNotNull over a bare column, so a row group whose statistics say a column is entirely null is skipped without being read.

Note the asymmetry with a hypothetical DropNans: IsNotNull is TOTAL, so the predicate is never itself null and the filter's own null rule never comes into play. `Col(c).IsNotNan()` is null on a null row, and Filter drops null predicates, so the obvious spelling of DropNans would silently drop nulls too. That is why it is not here.

func (*LazyFrame) Err 

func (lf *LazyFrame) Err() error

Err returns the first deferred error, or nil.

func (*LazyFrame) Explain 

func (lf *LazyFrame) Explain(ctx context.Context, opts ...ExplainOption) (out string, err error)

Explain renders the query plan.

The output is stable and diff-friendly on purpose: it is what golden tests snapshot, and a plan diff is the only thing that notices a query which still returns the right answer while reading forty columns instead of two.

func (*LazyFrame) Explode added in v0.3.0 

func (lf *LazyFrame) Explode(names ...string) *LazyFrame

Explode turns each row's list into one row per element, repeating every other column. 

id | tags        ->   id | tags
 1 | [10, 11]          1 | 10
 2 | []                1 | 11
 3 | null              2 | null
 4 | [12]              3 | null
                       4 | 12

An empty list and a null list each produce ONE row holding a null, rather than vanishing. Dropping them would silently lower the height — which is only visible to someone who checks it — and would make two columns disagree about how many rows a pair of empty lists gives. It is also what polars does.

The consequence is worth knowing: **explode erases the difference between an empty list and a null list.** A List column keeps those apart deliberately; after this they are the same row. The distinction belongs to the list, and the list is what explode consumes.

Several columns explode TOGETHER — zipped, not multiplied. Exploding twice in sequence would give the cross product instead, so the lists must be the same length in every row, and a row where they disagree is an error rather than a truncation.

Example

Split builds a List column; the .list namespace asks questions of each list, and Explode gives every element a row of its own. 

package main

import (
	"context"
	"fmt"
	"log"

	"github.com/advenn/ursus"
)

func main() {
	lf := ursus.Frame(
		ursus.Values("order", []int64{1, 2}),
		ursus.Values("items", []string{"apple,pear", "fig"}),
	).WithColumns(ursus.Col("items").Str().Split(","))

	df, err := lf.WithColumns(ursus.Col("items").List().Len().Alias("n")).
		Explode("items").Collect(context.Background())
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(df)
}

Output:
shape: (3, 3)
┌───────┬─────────┬────────┐
│ order │ items   │ n      │
│ Int64 │ String  │ Uint32 │
├───────┼─────────┼────────┤
│ 1     │ "apple" │ 2      │
│ 1     │ "pear"  │ 2      │
│ 2     │ "fig"   │ 1      │
└───────┴─────────┴────────┘

func (*LazyFrame) FillNan 

func (lf *LazyFrame) FillNan[T Operand](v T, subset ...string) *LazyFrame

FillNan replaces NaN in the named columns, or in every FLOAT column when no subset is given. Nulls are left alone — see Expr.FillNan.

func (*LazyFrame) FillNull 

func (lf *LazyFrame) FillNull[T Operand](v T, subset ...string) *LazyFrame

FillNull replaces nulls in the named columns, or in every column the value can fill when no subset is given. 

lf.FillNull(0)                  // every numeric column
lf.FillNull(0, "qty", "amount") // exactly these

Why the no-subset form restricts rather than refuses

Almost every real frame has a string column, and `lf.FillNull(0)` erroring on it would make the shorthand useless exactly where it is most wanted. So the value's own type chooses the columns — the same argument GroupBy(k).Sum()'s doc makes for restricting to numeric columns, and with the same consequence: it is not a silent skip, because the selector is part of the expression and Explain shows precisely which columns were filled.

A named column whose type the value cannot fill is a plan-time error, because the user asked for it by name.

The value is a WEAK literal, so filling an Int32 column with 0 leaves it Int32.

func (*LazyFrame) Filter 

func (lf *LazyFrame) Filter(preds ...Expr) *LazyFrame

Filter keeps rows where every predicate is true.

Predicates are AND-ed. A row whose predicate is NULL is DROPPED, matching SQL's WHERE. One consequence worth knowing: Filter(p) and Filter(p.Not()) do not partition the input, because a row where p is null is dropped by both.

Example

Filter keeps rows; Select keeps columns. Neither runs until Collect. 

package main

import (
	"context"
	"fmt"
	"log"

	"github.com/advenn/ursus"
)

// exampleSales is the little frame the examples below work on.
func exampleSales() *ursus.LazyFrame {
	return ursus.Frame(
		ursus.Values("region", []string{"north", "south", "north", "east", "south"}),
		ursus.Values("product", []string{"apple", "apple", "pear", "pear", "fig"}),
		ursus.Values("qty", []int64{10, 4, 7, 2, 9}),
		ursus.Values("price", []float64{1.50, 1.50, 2.25, 2.25, 3.00}),
	)
}

func main() {
	df, err := exampleSales().
		Filter(ursus.Col("qty").Gt(int64(5))).
		Select(ursus.Col("region"), ursus.Col("qty")).
		Collect(context.Background())
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(df)
}

Output:
shape: (3, 2)
┌─────────┬───────┐
│ region  │ qty   │
│ String  │ Int64 │
├─────────┼───────┤
│ "north" │ 10    │
│ "north" │ 7     │
│ "south" │ 9     │
└─────────┴───────┘

func (*LazyFrame) GroupBy 

func (lf *LazyFrame) GroupBy(keys ...Expr) *GroupBy

GroupBy begins an aggregation. 

lf.GroupBy(ursus.Col("region")).
    Agg(ursus.Col("revenue").Sum(), ursus.Len().Alias("n"))

With no keys it is a GLOBAL aggregate producing exactly one row — including over an empty input, where `count(*)` is 0 rather than no rows at all.

Example

GroupBy takes the keys, Agg takes what to compute per group. 

package main

import (
	"context"
	"fmt"
	"log"

	"github.com/advenn/ursus"
)

// exampleSales is the little frame the examples below work on.
func exampleSales() *ursus.LazyFrame {
	return ursus.Frame(
		ursus.Values("region", []string{"north", "south", "north", "east", "south"}),
		ursus.Values("product", []string{"apple", "apple", "pear", "pear", "fig"}),
		ursus.Values("qty", []int64{10, 4, 7, 2, 9}),
		ursus.Values("price", []float64{1.50, 1.50, 2.25, 2.25, 3.00}),
	)
}

func main() {
	df, err := exampleSales().
		GroupBy(ursus.Col("region")).
		Agg(
			ursus.Col("qty").Sum().Alias("total_qty"),
			ursus.Col("price").Mean().Alias("avg_price"),
			ursus.Len().Alias("orders"),
		).
		Sort(ursus.Asc(ursus.Col("region"))).
		Collect(context.Background())
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(df)
}

Output:
shape: (3, 4)
┌─────────┬───────────┬───────────┬────────┐
│ region  │ total_qty │ avg_price │ orders │
│ String  │ Int128    │ Float64   │ Uint64 │
├─────────┼───────────┼───────────┼────────┤
│ "east"  │ 2         │ 2.25      │ 1      │
│ "north" │ 17        │ 1.875     │ 2      │
│ "south" │ 13        │ 2.25      │ 2      │
└─────────┴───────────┴───────────┴────────┘
Example (WholeFrame)

GroupBy with NO keys aggregates the whole frame as one group. polars infers this from a bare select; ursus asks for it, so it cannot be confused with a row-wise select that happens to return one row. 

package main

import (
	"context"
	"fmt"
	"log"

	"github.com/advenn/ursus"
)

// exampleSales is the little frame the examples below work on.
func exampleSales() *ursus.LazyFrame {
	return ursus.Frame(
		ursus.Values("region", []string{"north", "south", "north", "east", "south"}),
		ursus.Values("product", []string{"apple", "apple", "pear", "pear", "fig"}),
		ursus.Values("qty", []int64{10, 4, 7, 2, 9}),
		ursus.Values("price", []float64{1.50, 1.50, 2.25, 2.25, 3.00}),
	)
}

func main() {
	df, err := exampleSales().
		GroupBy().
		Agg(
			ursus.Len().Alias("rows"),
			ursus.Col("qty").Sum().Alias("total_qty"),
		).
		Collect(context.Background())
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(df)
}

Output:
shape: (1, 2)
┌────────┬───────────┐
│ rows   │ total_qty │
│ Uint64 │ Int128    │
├────────┼───────────┤
│ 5      │ 32        │
└────────┴───────────┘

func (*LazyFrame) GroupByDynamic 

func (lf *LazyFrame) GroupByDynamic(index Expr, o DynamicOptions) *GroupBy

GroupByDynamic groups rows into fixed temporal windows cut from a sorted index. 

lf.GroupByDynamic(ursus.Col("ts"), ursus.DynamicOptions{
        Every:   ursus.Every("1h"),
        GroupBy: []ursus.Expr{ursus.Col("service")},
    }).
    Agg(ursus.Len().Alias("n"))

It is not GroupBy(truncate(ts, every)), and the difference is empty windows

A hash group-by creates a group when a row arrives, so an hour with no rows simply is not in the output and a gap in a time series is invisible. A dynamic group generates the whole grid between the first and last instant, so the gap is a row with a zero in it. That is the entire reason this operator exists.

The index must be sorted, and it is CHECKED

Windows are cut as contiguous ranges of a sorted index, so an unsorted one gives wrong groups. ursus verifies rather than trusting an assertion: an unchecked SetSorted-style hint is a user claim that deletes a correctness check, and its failure mode is wrong rows with no error. Sort first if the check refuses.

func (*LazyFrame) HStack 

func (lf *LazyFrame) HStack(others ...*LazyFrame) *LazyFrame

HStack places frames side by side: the same rows, with the columns concatenated.

It buffers, and Concat does not

Nothing in the engine aligns batch boundaries across independent pipelines — one frame may deliver 8192 rows at a time while another delivers 100 — and pairing row i of each means having row i of each in hand. So every input is materialised. Concat, which appends rows rather than pairing them, streams.

Every frame must have the same height, and column names must not collide. There is no automatic suffixing: a join suffixes because it has a principled left and right to name the suffix after, and these inputs are peers.

func (*LazyFrame) Head 

func (lf *LazyFrame) Head(n int) *LazyFrame

Head keeps at most n rows.

func (*LazyFrame) Join 

func (lf *LazyFrame) Join(other *LazyFrame, opts ...JoinOption) *LazyFrame

Join combines this frame with another. 

enriched := orders.Join(customers,
    ursus.JoinOn(ursus.Col("customer_id")),
    ursus.JoinHow(ursus.JoinLeft),
)

Column names that appear on both sides get the right one suffixed; the join key appears once unless the kind is a full join. See JoinCoalesce and JoinSuffix.

Example

Joins take the key columns and, optionally, the kind. Inner is the default. 

package main

import (
	"context"
	"fmt"
	"log"

	"github.com/advenn/ursus"
)

// exampleSales is the little frame the examples below work on.
func exampleSales() *ursus.LazyFrame {
	return ursus.Frame(
		ursus.Values("region", []string{"north", "south", "north", "east", "south"}),
		ursus.Values("product", []string{"apple", "apple", "pear", "pear", "fig"}),
		ursus.Values("qty", []int64{10, 4, 7, 2, 9}),
		ursus.Values("price", []float64{1.50, 1.50, 2.25, 2.25, 3.00}),
	)
}

func main() {
	regions := ursus.Frame(
		ursus.Values("region", []string{"north", "south", "east"}),
		ursus.Values("manager", []string{"ada", "bob", "cy"}),
	)
	df, err := exampleSales().
		Join(regions, ursus.JoinOn(ursus.Col("region"))).
		Select(ursus.Col("product"), ursus.Col("manager")).
		Collect(context.Background())
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(df)
}

Output:
shape: (5, 2)
┌─────────┬─────────┐
│ product │ manager │
│ String  │ String  │
├─────────┼─────────┤
│ "apple" │ "ada"   │
│ "apple" │ "bob"   │
│ "pear"  │ "ada"   │
│ "pear"  │ "cy"    │
│ "fig"   │ "bob"   │
└─────────┴─────────┘

func (*LazyFrame) JoinAsOf 

func (lf *LazyFrame) JoinAsOf(other *LazyFrame, opts ...AsOfOption) *LazyFrame

JoinAsOf matches each left row to the NEAREST right row rather than an equal one. 

trades.JoinAsOf(quotes,
    ursus.AsOfOn(ursus.Col("ts")),
    ursus.AsOfBy(ursus.Col("symbol")),
    ursus.AsOfTolerance(ursus.Every("1m")),
)

It is a LEFT join

Every left row survives; the right columns are null when nothing is near enough. So the output height always equals the left height, tolerance and strategy only decide which rows come back populated.

Both sides must be sorted on the as-of key, and it is CHECKED

The search is a binary search over a sorted run. ursus verifies rather than trusting an assertion — an unchecked sortedness hint is a user claim that deletes a correctness check, and its failure mode is wrong matches with no error. Sort first if the check refuses.

func (*LazyFrame) JoinWhere added in v0.3.0 

func (lf *LazyFrame) JoinWhere(other *LazyFrame, preds ...Expr) *LazyFrame

JoinWhere joins every pair of rows that satisfies the predicates. 

sessions.JoinWhere(events,
    ursus.Col("ts").Ge(ursus.Col("start")),
    ursus.Col("ts").Lt(ursus.Col("end")),
)

This is the non-equi join: the predicates may be any boolean expression over both sides, and are AND-ed. Several predicates are the usual case, because an interval match takes two.

Which names the predicates use

The join's OUTPUT names, which is to say left columns by their own name and a colliding right column with the suffix — `k` and `k_right` by default. There is no separate left/right namespace to qualify with, and the same rule applies here as to a Filter written after any other join. JoinSuffix changes the suffix.

What it costs

A pair of rows for every pair that satisfies the predicates, which can be very large — that is the nature of the operation and not a defect. When at least one predicate is an EQUALITY between the two sides, the optimizer turns it into a hash join key and evaluates the rest per matching pair; without one, every pair is tested. Explain shows which happened.

At least one predicate is required. A JoinWhere with none is a cartesian product written as though it were a filtered one, and the cardinality difference is |L| vs |L|x|R| — so it is an error rather than a silent cross join, for the same reason a keyless equi-join is.

Example

A join on a condition rather than equal keys: each sale with the price band it falls in. 

package main

import (
	"context"
	"fmt"
	"log"

	"github.com/advenn/ursus"
)

// exampleSales is the little frame the examples below work on.
func exampleSales() *ursus.LazyFrame {
	return ursus.Frame(
		ursus.Values("region", []string{"north", "south", "north", "east", "south"}),
		ursus.Values("product", []string{"apple", "apple", "pear", "pear", "fig"}),
		ursus.Values("qty", []int64{10, 4, 7, 2, 9}),
		ursus.Values("price", []float64{1.50, 1.50, 2.25, 2.25, 3.00}),
	)
}

func main() {
	bands := ursus.Frame(
		ursus.Values("band", []string{"cheap", "dear"}),
		ursus.Values("lo", []float64{0, 2}),
		ursus.Values("hi", []float64{2, 10}),
	)
	df, err := exampleSales().Select(ursus.Col("product"), ursus.Col("price")).
		JoinWhere(bands,
			ursus.Col("price").Ge(ursus.Col("lo")),
			ursus.Col("price").Lt(ursus.Col("hi"))).
		Select(ursus.Col("product"), ursus.Col("price"), ursus.Col("band")).
		Collect(context.Background())
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(df)
}

Output:
shape: (5, 3)
┌─────────┬─────────┬─────────┐
│ product │ price   │ band    │
│ String  │ Float64 │ String  │
├─────────┼─────────┼─────────┤
│ "apple" │ 1.5     │ "cheap" │
│ "apple" │ 1.5     │ "cheap" │
│ "pear"  │ 2.25    │ "dear"  │
│ "pear"  │ 2.25    │ "dear"  │
│ "fig"   │ 3       │ "dear"  │
└─────────┴─────────┴─────────┘

func (*LazyFrame) Limit 

func (lf *LazyFrame) Limit(n int) *LazyFrame

Limit is an alias for Head.

func (*LazyFrame) MergeSorted 

func (lf *LazyFrame) MergeSorted(other *LazyFrame, key string) *LazyFrame

MergeSorted interleaves this frame with another that is sorted on the same key, keeping the result sorted.

Not a concat: a concat appends, so the result is ordered only if the second frame begins after the first ends. Not a join either — nothing is matched and no row is dropped, so the output height is always the sum of the two.

Both schemas must match EXACTLY and both inputs must be sorted on key, which is checked rather than assumed.

func (*LazyFrame) Pipe 

func (lf *LazyFrame) Pipe(fn func(*LazyFrame) *LazyFrame) *LazyFrame

Pipe applies fn, for composing reusable query fragments.

func (*LazyFrame) Plan 

func (lf *LazyFrame) Plan() plan.Node

Plan returns the underlying logical plan.

func (*LazyFrame) Remove 

func (lf *LazyFrame) Remove(preds ...Expr) *LazyFrame

Remove is the inverse of Filter: it drops rows where ANY predicate is true.

Spelled out because the alternative reading is equally plausible: this is NOT(p1 OR p2 OR ...), i.e. a row survives only if every predicate is false. Rows where a predicate is null are dropped, same as Filter.

func (*LazyFrame) Rename 

func (lf *LazyFrame) Rename(names map[string]string) *LazyFrame

Rename changes column names, leaving everything else in place.

Sugar over `Select(All().MapName(...))`. MapName is the expansion-safe renamer — Alias sets ONE fixed name and so cannot apply to a multi-column selection, which is exactly why the two are different methods.

Names not present in the frame are ignored rather than an error, so a rename map written against a wider schema still works. Renaming two columns onto the same name IS an error, caught by Select's output-name uniqueness check with both offending expressions named.

func (*LazyFrame) Reverse 

func (lf *LazyFrame) Reverse() *LazyFrame

Reverse emits rows in the opposite order.

A full pipeline breaker — the last input row is the first output row — so it holds the whole frame. It does no comparison, which is what makes it cheaper than sorting by a synthesised descending index.

func (*LazyFrame) Rolling 

func (lf *LazyFrame) Rolling(index Expr, o RollingOptions) *GroupBy

Rolling gives every row its own window, reaching back Period from that row's own instant. The output has one row per input row.

Where GroupByDynamic answers "how many per hour", Rolling answers "how many in the hour before each event". The index must be sorted, and is checked.

func (*LazyFrame) Select 

func (lf *LazyFrame) Select(exprs ...Expr) *LazyFrame

Select computes a new set of columns, replacing the frame's columns entirely.

Expressions may expand: Select(All()) keeps everything, and Select(ColDType(Float64).Suffix("_f")) selects and renames every float column.

func (*LazyFrame) SinkCSV 

func (lf *LazyFrame) SinkCSV(ctx context.Context, path string, opts ...CSVSinkOption) error

SinkCSV runs the query and writes the result to path, streaming.

This is the half that makes "larger than RAM" true end to end. Collect holds the whole result; this holds one batch, so a query that reads more data than fits in memory and writes more data than fits in memory works.

The file is written to a temporary name and renamed on success, so a failed query leaves no half-written file where a complete one is expected.

func (*LazyFrame) SinkParquet 

func (lf *LazyFrame) SinkParquet(ctx context.Context, path string, opts ...ParquetSinkOption) error

SinkParquet runs the query and writes the result to path, streaming.

Memory is one row group rather than the whole result, so this is the write half of "larger than RAM". The file is written to a temporary name and renamed on success: a Parquet file without its footer is not merely truncated, it is unreadable, so leaving a partial one where a complete one is expected would be worse than leaving nothing.

func (*LazyFrame) Slice 

func (lf *LazyFrame) Slice(offset, length int) *LazyFrame

Slice keeps length rows starting at offset. A negative length means "to the end", which is how you drop a prefix without knowing the height.

Streaming: it counts rows past and stops, so it never holds more than one batch.

func (*LazyFrame) Sort 

func (lf *LazyFrame) Sort(keys ...SortKey) *LazyFrame

func (*LazyFrame) Tail 

func (lf *LazyFrame) Tail(n int) *LazyFrame

Tail keeps the last n rows.

Unlike Head it cannot stream to completion: which rows are the last n is unknown until the input ends, so it holds a ring of the most recent n. Bounded — O(n), not O(input) — but not free.

func (*LazyFrame) TopK 

func (lf *LazyFrame) TopK(k int, by ...SortKey) *LazyFrame

TopK keeps the k LARGEST rows by the given keys, and BottomK the k smallest.

Both are sugar, and that is the point

Each is `Sort(...).Head(k)`, which the limit-pushdown rule turns into a bounded top-k: kernel.ArgTopK is O(n log k) time and O(k) memory against a full sort's O(n log n) and O(n), and its contract is that it returns EXACTLY the indices ArgSort would, ties included. So there is no separate operator to keep in step with Sort, and `TopK(k)` and `Sort(...).Head(k)` cannot drift apart because they are the same plan.

Both place nulls LAST, unlike Sort

Sort keeps direction and null placement orthogonal on purpose — "`.Desc()` silently relocating the nulls surprises people every time" — and that argument does NOT transfer here. In a sort, placement is cosmetic: every row comes back either way. In a top-k it is SELECTION, and a null has no rank, so leaving the default (nulls first) would make `TopK(2)` over [3,1,4,1,5,null] return the null and the 5 — a row with no value in the very column being ranked.

So both force nulls last, and both say so. `Sort(...).Head(k)` remains available for a different placement, and it is one call away.

func (*LazyFrame) Unique 

func (lf *LazyFrame) Unique(subset ...string) *LazyFrame

Unique removes duplicate rows.

With no arguments it compares whole rows. With column names it compares only those, keeping the FIRST row for each distinct combination — so the other columns come from that row rather than being chosen arbitrarily.

func (*LazyFrame) Unnest added in v0.3.0 

func (lf *LazyFrame) Unnest(names ...string) *LazyFrame

Unnest replaces a struct column with its fields, in place. 

id | person                  ->   id | age  | city
 1 | {age: 30, city: "NY"}         1 | 30   | "NY"
 2 | {age: null, city: null}       2 | null | null
 3 | null                          3 | null | null

It is to a struct what Explode is to a list: afterwards the nested column is gone and the fields are ordinary columns, so every filter, aggregate, join and sort applies to them directly rather than through `.Struct().Field(...)`.

The fields land where the struct was and keep their OWN names. A name that would collide with a column already in the frame is refused, naming both — renaming a field the caller cannot see yet is not something this can guess at.

**Unnest erases the difference between a null struct and a struct of nulls**, the way Explode erases the difference between a null list and an empty one (rows 2 and 3 above). The distinction belongs to the struct, and this is what consumes it — `Col("person").IsNull()` is how to keep it, before unnesting.

At least one column must be named. "Every struct column" is well defined here, unlike for Explode, but it would silently widen the frame the day a file grows a struct.

func (*LazyFrame) Unpivot added in v0.3.0 

func (lf *LazyFrame) Unpivot(opts UnpivotOptions) *LazyFrame

Unpivot turns several value columns into two: which column, and what it held. 

sales.Unpivot(ursus.UnpivotOptions{On: []string{"jan", "feb"}})

id  jan  feb            id  variable  value
1   10   20     ->      1   jan       10
2   30   40             1   feb       20
                        2   jan       30
                        2   feb       40

Known elsewhere as melt. Every melted column shares one value column, so their types must promote to a common one — an Int32 beside an Int64 gives an Int64, and two types with no common one is an error naming both.

The row order is not Polars'

Polars emits every row's first variable, then every row's second. This emits every variable for one row, then the next row.

The reason is that ursus streams: unpivot transforms one batch at a time, so Polars' order would interleave differently at different batch sizes — the row order would depend on how the query ran rather than on what it asked for. Sort afterwards if a particular order matters.

Index defaults; On does not

Leaving Index empty keeps every column that is not melted, so adding a column to the source keeps it. Unnest refuses the equivalent default for the opposite reason: there, a new column would silently start being CONSUMED.

func (*LazyFrame) VStack 

func (lf *LazyFrame) VStack(other *LazyFrame) *LazyFrame

VStack is Concat for exactly two frames, named for the operation people look for.

It is not cheaper than Concat. Polars documents vstack as "cheap (adds a chunk)", which relies on a chunked column layout ursus does not have — a Column here is one contiguous run, so stacking copies. Naming it the same and pretending otherwise would be the misleading part.

func (*LazyFrame) WhereExists added in v0.3.0 

func (lf *LazyFrame) WhereExists(other *LazyFrame, preds ...Expr) *LazyFrame

WhereExists keeps the rows of this frame that have at least one row in other satisfying every predicate. 

orders.WhereExists(lineitem,
    ursus.Col("o_orderkey").Eq(ursus.Col("l_orderkey")),
    ursus.Col("l_quantity").Gt(40.0),
)

It is SQL's `EXISTS (SELECT ... WHERE ...)`, and the output is this frame: same columns, same order, fewer rows. No column of other appears in the result, which is why this is not a join you can write as one — see below.

Why it is not JoinWhere plus a filter

JoinWhere pairs rows, so filtering its output filters pairs. This asks whether a satisfying partner EXISTS, which is a question about the left row, and a left row with three partners must come back once rather than three times. The predicate is therefore evaluated before the verdict, inside the join, and no combination of JoinWhere and Filter expresses that without a deduplication.

Which names the predicates use

The names a JOIN of the two frames would produce: this frame's columns by their own name, and a colliding column of other with the suffix — `k` and `k_right`. The output has no such column, but the predicate is evaluated on pairs, so it can name both.

What it costs

An equality between the two sides becomes a hash join key and the rest is tested per matching pair; with no equality among the predicates, every pair is tested. Explain shows which. A partner is found by scanning the candidates, so this stops at the first satisfying one — WhereNotExists cannot.

func (*LazyFrame) WhereNotExists added in v0.3.0 

func (lf *LazyFrame) WhereNotExists(other *LazyFrame, preds ...Expr) *LazyFrame

WhereNotExists keeps the rows of this frame that have NO row in other satisfying every predicate. 

// customers who never ordered anything large
customers.WhereNotExists(orders,
    ursus.Col("c_custkey").Eq(ursus.Col("o_custkey")),
    ursus.Col("o_totalprice").Gt(1000.0),
)

SQL's `NOT EXISTS`, and the exact complement of WhereExists: for any frame and any predicates, the two partition this frame's rows.

It is strictly more work than WhereExists on the same inputs. Finding one satisfying partner ends the search; proving there is none means examining every candidate.

func (*LazyFrame) WithColumns 

func (lf *LazyFrame) WithColumns(exprs ...Expr) *LazyFrame

WithColumns adds columns, replacing any that already exist BY NAME and keeping their original position.

Later expressions can reference columns that earlier ones added, so `WithColumns(a.Alias("x"), Col("x").Mul(2))` works.

Example

Expressions compose: a computed column is just another expression. 

package main

import (
	"context"
	"fmt"
	"log"

	"github.com/advenn/ursus"
)

// exampleSales is the little frame the examples below work on.
func exampleSales() *ursus.LazyFrame {
	return ursus.Frame(
		ursus.Values("region", []string{"north", "south", "north", "east", "south"}),
		ursus.Values("product", []string{"apple", "apple", "pear", "pear", "fig"}),
		ursus.Values("qty", []int64{10, 4, 7, 2, 9}),
		ursus.Values("price", []float64{1.50, 1.50, 2.25, 2.25, 3.00}),
	)
}

func main() {
	df, err := exampleSales().
		WithColumns(ursus.Col("qty").Mul(ursus.Col("price")).Alias("revenue")).
		Select(ursus.Col("product"), ursus.Col("revenue")).
		Collect(context.Background())
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(df)
}

Output:
shape: (5, 2)
┌─────────┬─────────┐
│ product │ revenue │
│ String  │ Float64 │
├─────────┼─────────┤
│ "apple" │ 15      │
│ "apple" │ 6       │
│ "pear"  │ 15.75   │
│ "pear"  │ 4.5     │
│ "fig"   │ 27      │
└─────────┴─────────┘

func (*LazyFrame) WithRowIndex 

func (lf *LazyFrame) WithRowIndex(name string, offset uint32) *LazyFrame

WithRowIndex prepends a Uint32 column numbering the rows from offset.

Row order is a property rather than an addressable label space, which is why there is no implicit index — this is how you ask for one when you want it.

It is a counter that crosses batch boundaries, so it depends on batches arriving in input order, and it is deliberately not the same thing as `CumCount(false).Over()`: that computes identical numbers through the window sink, which buffers the entire frame to do it.

func (*LazyFrame) WriteCSV 

func (lf *LazyFrame) WriteCSV(ctx context.Context, w io.Writer, opts ...CSVSinkOption) (err error)

WriteCSV runs the query and writes the result to w, streaming.

It does not close w. Whoever opened it closes it — the only rule that works when the destination might be os.Stdout.

func (*LazyFrame) WriteParquet 

func (lf *LazyFrame) WriteParquet(ctx context.Context, w io.Writer, opts ...ParquetSinkOption) (err error)

WriteParquet runs the query and writes the result to w, streaming. It does not close w.

Example

A List column writes to Parquet and reads back, in the standard encoding PyArrow and Polars read too. 

package main

import (
	"bytes"
	"context"
	"fmt"
	"log"

	"github.com/advenn/ursus"
)

func main() {
	ctx := context.Background()
	orders := ursus.Frame(
		ursus.Values("order", []int64{1, 2}),
		ursus.Values("items", []string{"apple,pear", "fig"}),
	).WithColumns(ursus.Col("items").Str().Split(","))

	var file bytes.Buffer
	if err := orders.WriteParquet(ctx, &file); err != nil {
		log.Fatal(err)
	}
	df, err := ursus.ScanParquetBytes(file.Bytes(), "orders.parquet").Collect(ctx)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(df)
}

Output:
shape: (2, 2)
┌───────┬───────────────────┐
│ order │ items             │
│ Int64 │ List(String)      │
├───────┼───────────────────┤
│ 1     │ ["apple", "pear"] │
│ 2     │ ["fig"]           │
└───────┴───────────────────┘

type ListExpr added in v0.3.0 

type ListExpr struct {
	// contains filtered or unexported fields
}

ListExpr is the list namespace: `Col("tags").List().Len()`.

The same wrapper shape `.str` and `.dt` use, for the reason ursus-api.md §4.5 gives: methods return Expr, so chaining continues naturally through it.

Two halves, told apart by what they return

Len, Get, Contains, Min, Max, Sum and Mean answer a question ABOUT a list and hand back an ordinary column — a length, an element, a total.

Reverse, Head, Tail, Slice, Sort, Unique and DropNulls RESHAPE the list and hand back a list, so they chain into each other and into the first half: `Sort().Head(3).Get(0)`. The element type is unchanged by all seven — reshaping a list cannot change what it is a list OF.

The set operations (`set_union` and friends) and `gather` take a SECOND list and are not here yet.

A null list is not an empty list

It is the distinction the whole List type is careful about, and it survives into every method: 

                Len()  Sum()  Min()  Get(0)  Contains(x)
[1, 2]            2      3      1      1      as asked
[]  (empty)       0     null   null   null      false
null (list)      null   null   null   null      null

An empty list HAS a length, and it is zero. A null list has no length at all. An empty list contains nothing, which is a false answer to a real question; a null list cannot answer it. Working from "both are empty" gives three wrong cells here, none of which errors.

The aggregates are where the two agree, and that is a choice: polars gives 0 for the sum of an empty list, ursus gives null, because ursus's Sum gives null for a group with no values. Agreeing with the aggregate beside it beats agreeing with polars.

func (ListExpr) Contains added in v0.3.0 

func (l ListExpr) Contains(v any) Expr

Contains reports whether the list holds v.

The comparison is ursus's, not Go's: values go through the same group-key encoder `IsIn` uses, so NaN matches NaN and -0.0 matches +0.0, consistently with how GroupBy and Sort treat them.

The element and v meet as == would compare them, as IsIn's values do: v of a type == refuses is refused, and `Contains(5000)` against Int8 elements is false.

func (ListExpr) DropNulls added in v0.3.0 

func (l ListExpr) DropNulls() Expr

DropNulls removes the null ELEMENTS from each list. A null LIST stays null — there is nothing to drop from, which is not the same as having dropped everything.

func (ListExpr) First added in v0.3.0 

func (l ListExpr) First() Expr

First is Get(0) and Last is Get(-1).

func (ListExpr) Get added in v0.3.0 

func (l ListExpr) Get(i int) Expr

Get returns the element at i, or null if there is none there.

A negative index counts from the END, the convention every dataframe library uses and the one `.str.slice` already follows here.

Out of range is null rather than an error. Lists are ragged by nature, so asking for element 3 of a column whose rows mostly hold two is an ordinary question — erroring would make `Get` unusable on exactly the data it is for.

func (ListExpr) Head added in v0.3.0 

func (l ListExpr) Head(n int) Expr

Head keeps the first n elements of each list, Tail the last n.

Asking for more than a row holds returns the whole row, not an error and not a padded list. Lists are ragged by nature, so asking for three tags from a column whose rows mostly have two is an ordinary question — the same reasoning Get uses for an index past the end.

A negative n is refused: "the first -2 elements" has no reading worth guessing at. Slice is where a negative number means something.

func (ListExpr) Last added in v0.3.0 

func (l ListExpr) Last() Expr

func (ListExpr) Len added in v0.3.0 

func (l ListExpr) Len() Expr

Len is the number of elements, as a Uint32 — matching `.str.len_bytes`, since a length is never negative and never needs 64 bits.

Null ELEMENTS are counted: they occupy a slot, so `[1, null]` has length 2. A null LIST has no length and gives null.

func (ListExpr) Max added in v0.3.0 

func (l ListExpr) Max() Expr

func (ListExpr) Mean added in v0.3.0 

func (l ListExpr) Mean() Expr

Mean averages each list, as a Float64. Null elements are skipped, so the divisor is the count of PRESENT elements.

func (ListExpr) Min added in v0.3.0 

func (l ListExpr) Min() Expr

Min and Max reduce each list to its smallest or largest element, keeping the element type. Null elements are skipped, exactly as the column-wide aggregates skip null rows.

func (ListExpr) Reverse added in v0.3.0 

func (l ListExpr) Reverse() Expr

Reverse puts each list's elements in the opposite order. Null elements move with everything else — they occupy a slot.

func (ListExpr) Slice added in v0.3.0 

func (l ListExpr) Slice(offset, length int) Expr

Slice takes length elements from offset.

A negative OFFSET counts from the end, matching Get and `.str.slice`; the clamping is `.str.slice`'s too, so an offset past either edge lands in range and a length running off the end stops there. A negative length is refused.

func (ListExpr) Sort added in v0.3.0 

func (l ListExpr) Sort() Expr

Sort orders each list's elements ascending; SortDesc orders them descending.

The ordering is ursus's own — the same comparator a column-wide Sort uses — so a list of values and a column of the same values order identically. That includes null placement: nulls come FIRST, matching the default of `Asc()`. Both placements are defensible and this is the one ursus picked.

Equal elements keep their input order.

func (ListExpr) SortDesc added in v0.3.0 

func (l ListExpr) SortDesc() Expr

func (ListExpr) Sum added in v0.3.0 

func (l ListExpr) Sum() Expr

Sum totals each list.

It widens an integer element to Int128, because the column-wide Sum does and a per-row sum that overflowed where that one does not would be the worse surprise. The rule is not restated here — the type comes from the aggregate's own binding, so the two cannot drift.

An empty list sums to NULL, not zero — the same answer ursus's Sum gives for a group with no values. polars differs here.

func (ListExpr) Tail added in v0.3.0 

func (l ListExpr) Tail(n int) Expr

Tail keeps the last n elements. See Head.

func (ListExpr) Unique added in v0.3.0 

func (l ListExpr) Unique() Expr

Unique drops repeated elements, keeping the FIRST occurrence and leaving the order alone — sorting as a side effect would make `Unique().Head(2)` mean something different from `Head(2)` on the same data.

Equality is ursus's, as it is for Contains: NaN dedupes against NaN. A null element counts as a value here, so a list of three nulls keeps one.

type Literal 

type Literal interface {
	~bool |
		~int | ~int8 | ~int16 | ~int32 | ~int64 |
		~uint | ~uint8 | ~uint16 | ~uint32 | ~uint64 |
		~float32 | ~float64 |
		~string | ~[]byte |
		time.Time
}

Literal is the set of Go types that lift to a column literal.

Every accepted width is enumerated explicitly because Go infers a type parameter from the argument's type with NO implicit widening: `a.Gt(int32(5))` is a compile error unless ~int32 appears here.

time.Duration is DELIBERATELY ABSENT. Its underlying type is int64, so listing it alongside ~int64 gives overlapping type sets and the union does not compile — this is the defect the design review found. ~int64 already admits it, and a type switch recovers the named type exactly, so nothing is lost.

time.Time is a bare (non-~) term: it is a defined struct type, disjoint from every other term. A union term may be a type with methods; only interface terms with methods are forbidden.

type MappingStrategy 

type MappingStrategy = expr.MappingStrategy

MappingStrategy decides how a window's per-partition result returns to the frame.

type MemoryStats 

type MemoryStats struct {
	// Peak is the largest total the query retained at one moment, in bytes.
	//
	// It counts what buffering operators HOLD ACROSS BATCHES, deduplicated by
	// allocation so a payload two operators share is counted once. It does not
	// count a kernel's transient output, because ursus's allocator reports
	// nothing and there is no hook at the allocation site.
	Peak int64

	// Limit is the ceiling that was in force, or 0 if there was none.
	Limit int64

	// Spills is how many files a spilling operator wrote: sorted runs for a sort,
	// radix partitions for a group-by, and for a join BOTH sides' partitions —
	// counted at every level of the recursion.
	//
	// It is a count of FILES, not of bytes and not of distinct spill events, and a
	// group-by that recursed reports more than one level's fan-out — which is the
	// only public evidence that a partition did not fit on the first try.
	Spills int64
}

MemoryStats reports what a query actually held.

Peak is the number that demonstrates bounded memory: that a query FINISHED proves only that it did not run out, while a peak far below the input size proves it never held the input at all.

type Operand 

type Operand interface {
	Expr | Literal
}

Operand is either another Expr or a Go scalar that lifts to a literal.

This union — a struct type alongside an embedded constraint interface — is what lets every binary operator accept both forms with no wrapper at the call site.

type ParquetFile added in v0.3.0 

type ParquetFile struct {
	// Name identifies the file in errors and in Explain.
	Name string

	// Open returns the file's bytes and their size.
	//
	// It is called more than once per query — once to read the schema when the
	// query is planned, once for each execution — and must return a fresh reader
	// each time. ctx is the query's and outlives the reader, so a reader over a
	// network may keep it for its reads. If the reader is also an io.Closer, ursus
	// closes it when it is done with it.
	Open func(ctx context.Context) (r io.ReaderAt, size int64, err error)
}

ParquetFile is one Parquet file, read through ReadAt from wherever it lives — an object store, an HTTP server, an archive — by code that knows how to reach it.

ursus reads a file the way an object store wants to be read: the footer from the end, then one ranged read for each column chunk the query needs, after the statistics have pruned the row groups it does not. A ReadAt over a ranged GET therefore fetches what the query reads and nothing else.

type ParquetOption 

type ParquetOption func(*parquet.Options)

ParquetOption configures ScanParquet.

func WithPruning 

func WithPruning(b bool) ParquetOption

WithPruning enables or disables row-group skipping from column statistics.

It is on by default. The switch exists because pruning is the one part of the Parquet reader that can change which rows come back, so a wrong answer must be bisectable to it — the same reason plan.Flags can disable each optimizer rule.

type ParquetSinkOption 

type ParquetSinkOption interface {
	// contains filtered or unexported methods
}

ParquetSinkOption is anything SinkParquet and WriteParquet accept: a writer option (WithCompression, WithRowGroupRows, WithStatistics) or an execution option (WithMemoryLimit, WithSpillDir, WithBatchSize, WithThreads).

A sink is where a memory limit matters most — it is the consumer that streams, so it is the one a larger-than-RAM query ends in — and before this the two option families could not meet.

It is an interface rather than a variadic `...any` because the union has to be checked at compile time. A library that uses generic methods and an Operand constraint to stop `Col("x").Gt(struct{}{})` from compiling should not then accept SinkParquet(ctx, path, "oops").

type ParquetWriteOption 

type ParquetWriteOption func(*parquet.WriteOptions)

ParquetWriteOption configures the Parquet writer.

func WithCompression 

func WithCompression(c compress.Compression) ParquetWriteOption

WithCompression sets the codec applied to every column. Default Snappy.

func WithRowGroupRows 

func WithRowGroupRows(n int) ParquetWriteOption

WithRowGroupRows sets the target row-group size. It is the unit of both pruning granularity and writer memory: smaller groups prune better and buffer less.

func WithStatistics 

func WithStatistics(b bool) ParquetWriteOption

WithStatistics enables or disables column statistics. On by default — a file without them cannot be pruned, which gives up the main reason to use Parquet.

type RankMethod 

type RankMethod = expr.RankMethod

RankMethod decides how Rank breaks ties.

type RollingOptions 

type RollingOptions struct {
	// Period is how far back each row's window reaches from its own instant.
	Period Interval

	// Offset shifts the window's end away from the row's own instant.
	Offset Interval

	// Closed says which end belongs to the window. The zero value is ClosedRight —
	// (end-period, end] — because a rolling window ends AT the row, so closing the
	// left end instead would drop every row from its own window.
	Closed Closed

	// GroupBy adds categorical keys, as in DynamicOptions.
	GroupBy []Expr
}

RollingOptions configures Rolling.

type Schema 

type Schema = dtype.Schema

Schema is an ordered, name-unique sequence of Fields.

func MustSchema 

func MustSchema(fields ...Field) *Schema

MustSchema is NewSchema for tests and package-level vars.

func NewSchema 

func NewSchema(fields ...Field) (*Schema, error)

NewSchema builds a schema, rejecting duplicate column names.

type Series 

type Series[T any] = data.Series[T]

Series is a typed view over a Column.

func TypedColumn 

func TypedColumn[T any](c *Column) (*Series[T], error)

TypedColumn returns a typed view over a column.

type SortKey 

type SortKey struct {
	// contains filtered or unexported fields
}

SortKey is one component of an ordering. Build one with Asc or Desc.

func Asc 

func Asc(e Expr) SortKey

Asc orders ascending. Nulls come FIRST by default; call NullsLast to move them.

func Desc 

func Desc(e Expr) SortKey

Desc orders descending, with nulls still FIRST by default.

Null placement is deliberately independent of direction. SQL ties them together — nulls sort as the largest value, so they land last ascending and first descending — which means switching to Desc silently relocates them. Keeping the two orthogonal is more predictable.

func (SortKey) NullsFirst 

func (k SortKey) NullsFirst() SortKey

NullsFirst places nulls at the start of the output.

func (SortKey) NullsLast 

func (k SortKey) NullsLast() SortKey

NullsLast places nulls at the END of the output, whichever direction the key sorts in.

type Span 

type Span interface {
	Interval | time.Duration
}

Span is an interval in either spelling — the calendar-aware Interval, or a plain time.Duration for the absolute case.

The same union shape as Operand, and for the same reason: it lets one method accept both forms with no wrapper at the call site, so `Truncate(time.Hour)` keeps compiling now that `Truncate(Every("1mo"))` is also legal.

type StrExpr 

type StrExpr struct {
	// contains filtered or unexported fields
}

StrExpr is the string namespace: `Col("name").Str().ToLower()`.

A small wrapper struct whose methods return Expr, so chaining continues naturally through it. This is the shape ursus-api.md §4.5 specifies for every namespace, and `.dt` follows it identically.

The literal flag

Contains, Find, CountMatches, Replace and ReplaceAll take `literal bool`, and literal is what you want unless you know otherwise. Go's regexp is RE2: no backtracking, so no catastrophic blowup, but also no JIT — a plain substring test through it costs far more than strings.Contains. Passing literal=true keeps the common case on the fast path, which is why the flag is explicit rather than inferred from whether the pattern looks like a regex.

func (StrExpr) Contains 

func (s StrExpr) Contains(pattern string, literal bool) Expr

Contains reports whether each value contains pattern.

func (StrExpr) CountMatches 

func (s StrExpr) CountMatches(pattern string, literal bool) Expr

CountMatches counts non-overlapping occurrences. An empty pattern, literal or regex, matches at every rune boundary: len_chars + 1 times.

func (StrExpr) EndsWith 

func (s StrExpr) EndsWith(suffix string) Expr

func (StrExpr) EscapeRegex added in v0.3.0 

func (s StrExpr) EscapeRegex() Expr

EscapeRegex quotes every regex metacharacter, so the result matches itself literally when used as a pattern.

func (StrExpr) Extract 

func (s StrExpr) Extract(pattern string, group int) Expr

Extract returns capture group n of the first match, or NULL if there is none. Group 0 is the whole match. Regex only: extracting a literal is just Find.

func (StrExpr) ExtractAll added in v0.3.0 

func (s StrExpr) ExtractAll(pattern string) Expr

ExtractAll returns every non-overlapping match of a regex as a List(String).

Regex only, for the reason Extract is: extracting every literal occurrence of a pattern returns that many copies of the pattern.

func (StrExpr) Find 

func (s StrExpr) Find(pattern string, literal bool) Expr

Find returns the byte offset of the first match, or NULL when there is none.

Null rather than -1: a sentinel index would compare and do arithmetic like a real position, so `find(x) < 5` would be true for "not found".

func (StrExpr) Head 

func (s StrExpr) Head(n int) Expr

Head and Tail are Slice's common cases.

func (StrExpr) LenBytes 

func (s StrExpr) LenBytes() Expr

LenBytes counts bytes; LenChars counts runes. They differ on any non-ASCII input, and which one is wanted is not guessable — so there is no `Len`.

func (StrExpr) LenChars 

func (s StrExpr) LenChars() Expr

func (StrExpr) PadEnd added in v0.3.0 

func (s StrExpr) PadEnd(width int, fill string) Expr

func (StrExpr) PadStart added in v0.3.0 

func (s StrExpr) PadStart(width int, fill string) Expr

PadStart and PadEnd pad each value to width, counted in RUNES, using the first rune of fill. A value already that wide is returned unchanged — padding never truncates.

func (StrExpr) Replace 

func (s StrExpr) Replace(pattern, value string, literal bool) Expr

Replace substitutes the FIRST match; ReplaceAll substitutes every match.

For a regex, the replacement expands $1, ${1} and ${name} in both, as Go's regexp.Expand does — so $1x names a group called "1x", and ${1}x is group 1 then an x. A literal pattern's replacement is literal, $ and all.

func (StrExpr) ReplaceAll 

func (s StrExpr) ReplaceAll(pattern, value string, literal bool) Expr

func (StrExpr) Reverse 

func (s StrExpr) Reverse() Expr

Reverse reverses by RUNE, not by byte.

func (StrExpr) Slice 

func (s StrExpr) Slice(offset, length int) Expr

Slice takes length runes from offset, which may be negative to count from the end. Runes rather than bytes, so it can never split a character in half.

func (StrExpr) Split added in v0.3.0 

func (s StrExpr) Split(by string) Expr

Split splits each value on a literal separator, producing a List(String). 

lf.WithColumns(ursus.Col("tags").Str().Split(",").Alias("tag")).
    Explode("tag")

That two-stage form is the canonical use, and it is two stages because Explode changes the frame's height, which an expression may not.

Three cases that are not the same 

"a,b"   ->  ["a", "b"]
""      ->  [""]      one empty element, matching strings.Split
NULL    ->  NULL      a null list, not an empty one

The separator is LITERAL. A regex split would be a different function, and splitting on "." meaning "any character" is the mistake it would invite.

func (StrExpr) SplitN added in v0.3.0 

func (s StrExpr) SplitN(by string, n int) Expr

SplitN is Split with at most n parts; the last part keeps the rest, separators and all. n <= 0 means unlimited — unlike strings.SplitN, whose 0 means none.

func (StrExpr) StartsWith 

func (s StrExpr) StartsWith(prefix string) Expr

StartsWith and EndsWith are always literal — a prefix match against a regex is not a well-defined thing to ask for.

func (StrExpr) StripChars 

func (s StrExpr) StripChars(chars string) Expr

StripChars trims any of the given characters from both ends. An empty set trims whitespace, matching Polars.

func (StrExpr) StripCharsEnd added in v0.3.0 

func (s StrExpr) StripCharsEnd(chars string) Expr

func (StrExpr) StripCharsStart added in v0.3.0 

func (s StrExpr) StripCharsStart(chars string) Expr

StripCharsStart and StripCharsEnd are the one-sided forms of StripChars: they remove any leading (or trailing) character in the cutset. An empty cutset means Unicode whitespace, as it does for StripChars.

func (StrExpr) StripPrefix 

func (s StrExpr) StripPrefix(prefix string) Expr

func (StrExpr) StripSuffix 

func (s StrExpr) StripSuffix(suffix string) Expr

func (StrExpr) Tail 

func (s StrExpr) Tail(n int) Expr

func (StrExpr) ToDate 

func (s StrExpr) ToDate() Expr

func (StrExpr) ToDatetime 

func (s StrExpr) ToDatetime(unit dtype.TimeUnit, tz string) Expr

func (StrExpr) ToInteger 

func (s StrExpr) ToInteger() Expr

ToInteger, ToDate and ToDatetime parse. They are ordinary casts, which is why they are spelled as casts rather than as new kernels — and unparseable values become NULL rather than failing the query, as do integers past Int64's range. ToInteger parses exactly at 64 bits: "9007199254740993" is itself, not the float64 beside it.

func (StrExpr) ToLower 

func (s StrExpr) ToLower() Expr

func (StrExpr) ToUpper 

func (s StrExpr) ToUpper() Expr

func (StrExpr) ZFill added in v0.3.0 

func (s StrExpr) ZFill(width int) Expr

ZFill left-pads with zeros to width, keeping any leading sign in front of them: -5 at width 4 is "-005", not "0-05". That is what makes it different from PadStart with a '0', and it is the reason it is a separate function.

type StructExpr added in v0.3.0 

type StructExpr struct {
	// contains filtered or unexported fields
}

StructExpr is the struct namespace: `Col("person").Struct().Field("age")`.

The same wrapper shape `.str`, `.dt` and `.list` use, for the reason ursus-api.md §4.5 gives: methods return Expr, so chaining continues naturally through it.

A null struct is not a struct of nulls

Both hold no field values, and only the struct's own validity separates them: 

                          Field("age")   renders as
{age: 30, city: "NY"}          30        {age: 30, city: "NY"}
{age: null, city: null}       null       {age: null, city: null}
null (the struct)             null       null

The last two rows answer identically through Field — there is no age either way — and differ where it matters, on the struct itself. Working from "a struct with nothing in it is absent" gives the wrong answer for a real struct that happens to hold nulls, and nothing errors.

func (StructExpr) Field added in v0.3.0 

func (s StructExpr) Field(name string) Expr

Field reads one field out of each struct.

The result is an ordinary column of the field's own type, so everything else in the language applies to it: filter on it, group by it, join on it.

A name the struct does not have is refused when the query is PLANNED, with the available names listed — the type of the result depends on which field was asked for, so the name has to be resolved against the schema anyway.

type ThenBuilder 

type ThenBuilder struct {
	// contains filtered or unexported fields
}

ThenBuilder is a conditional that may take another When or be closed by Otherwise.

func (ThenBuilder) Otherwise 

func (t ThenBuilder) Otherwise[T Operand](v T) Expr

Otherwise supplies the fall-through value and closes the chain.

The chain becomes a right-nested tree of conditionals, in the order written: when(c1).then(v1).when(c2).then(v2).otherwise(d) is Cond{c1, v1, Cond{c2, v2, d}}. Nesting rather than a flat list is what keeps type unification well-defined — promotion is pairwise, and folding a flat list would make the result depend on an association order the user never chose.

func (ThenBuilder) When 

func (t ThenBuilder) When(pred Expr) WhenBuilder

When adds another condition, the else-if of the chain.

type TimeUnit 

type TimeUnit = dtype.TimeUnit

TimeUnit is the resolution of a Time, Datetime or Duration.

type UnpivotOptions added in v0.3.0 

type UnpivotOptions struct {
	// On names the value columns to melt into the variable/value pair.
	On []string

	// Index names the columns carried through unchanged. Empty means every column
	// that is not in On.
	Index []string

	// VariableName and ValueName name the two invented columns. Empty means
	// "variable" and "value", which are Polars' names.
	VariableName string
	ValueName    string
}

UnpivotOptions configures Unpivot.

On is required; everything else has a working default, so the common call is UnpivotOptions{On: []string{...}}.

type WhenBuilder 

type WhenBuilder struct {
	// contains filtered or unexported fields
}

WhenBuilder is a conditional awaiting its Then.

func When 

func When(pred Expr) WhenBuilder

When begins a conditional. 

ursus.When(ursus.Col("score").Ge(90)).Then("A").
    When(ursus.Col("score").Ge(80)).Then("B").
    Otherwise("F").Alias("grade")

Why a builder and not a function

The alternative was a variadic Case(cond1, val1, cond2, val2, …, default). That reads compactly and pairs its arguments POSITIONALLY, so a miscounted list is a runtime error rather than a type error — and the compiler cannot tell a condition from a value when both are Expr. The builder makes each pair a method call, so there is nothing to miscount.

An unterminated chain cannot be used, by construction

Only Otherwise returns an Expr; WhenBuilder and ThenBuilder are not expressions and satisfy nothing that takes one. So a chain that forgets its else branch fails to compile rather than silently defaulting. Where a null default IS wanted, say so: Otherwise(ursus.Null(ursus.NullT)).

Both branches are evaluated

This is a columnar engine, so Then and Otherwise are each computed for every row and then merged. That is unobservable — expressions have no side effects, and an arithmetic fault such as division by zero yields NULL rather than trapping — but it does mean a conditional does not make an expensive branch cheaper.

func (WhenBuilder) Then 

func (w WhenBuilder) Then[T Operand](v T) ThenBuilder

Then supplies the value for the pending condition. The value lifts, so Then("A") needs no Lit.

type WindowSpec 

type WindowSpec struct {
	PartitionBy []Expr
	OrderBy     []SortKey
	Mapping     MappingStrategy
}

WindowSpec is the full form of a window, for OverWith.

Source Files

View all Source files

Directories

Path Synopsis
Package dtype is ursus's type system: DataType, Field and Schema.
 Package dtype is ursus's type system: DataType, Field and Schema.
Package i128 provides a signed 128-bit integer.
 Package i128 provides a signed 128-bit integer.
internal
arrowin
Package arrowin imports Arrow data into ursus, by copying it.
 Package arrowin imports Arrow data into ursus, by copying it.
arrowout
Package arrowout exports an ursus batch as Arrow, which is how anything outside this module reads ursus's output without a copy.
 Package arrowout exports an ursus batch as Arrow, which is how anything outside this module reads ursus's output without a copy.
arrowx
Package arrowx owns ursus's allocator and the few Arrow facts shared across packages.
 Package arrowx owns ursus's allocator and the few Arrow facts shared across packages.
bitmap
Package bitmap is ursus's validity-bitmap layer.
 Package bitmap is ursus's validity-bitmap layer.
data
Package data holds ursus's runtime column representation.
 Package data holds ursus's runtime column representation.
exec
Package exec drives a physical operator tree to completion.
 Package exec drives a physical operator tree to completion.
execopt
Package execopt carries the execution-time knobs that are not planning decisions: how much memory one query may hold, and where it is allowed to spill.
 Package execopt carries the execution-time knobs that are not planning decisions: how much memory one query may hold, and where it is allowed to spill.
expr
Package expr is ursus's expression IR.
 Package expr is ursus's expression IR.
gen/levels command
Command levels enforces ursus's import-level invariant:
 Command levels enforces ursus's import-level invariant:
kernel
Package kernel is ursus's compute layer.
 Package kernel is ursus's compute layer.
physical
Package physical turns a logical plan into runnable operators, and expressions into columns.
 Package physical turns a logical plan into runnable operators, and expressions into columns.
plan
Package plan is ursus's logical plan IR.
 Package plan is ursus's logical plan IR.
source
Package source is the runtime half of ursus's scan contract.
 Package source is the runtime half of ursus's scan contract.
source/arrowsrc
Package arrowsrc scans a stream of Arrow records: an IPC stream, a Flight response, a C Data Interface stream — anything that is an array.RecordReader.
 Package arrowsrc scans a stream of Arrow records: an IPC stream, a Flight response, a C Data Interface stream — anything that is an array.RecordReader.
source/csv
Package csv reads and writes delimited text files.
 Package csv reads and writes delimited text files.
source/memsrc
Package memsrc is an in-memory scan source.
 Package memsrc is an in-memory scan source.
source/parquet
Package parquet reads and writes Apache Parquet files.
 Package parquet reads and writes Apache Parquet files.
source/testsrc
Package testsrc is a scan source that lies about its capabilities on purpose.
 Package testsrc is a scan source that lies about its capabilities on purpose.
spill
Package spill writes batches to a file and reads them back exactly.
 Package spill writes batches to a file and reads them back exactly.
uerr
Package uerr is ursus's error type.
 Package uerr is ursus's error type.
Package ursustest provides assertions for testing code that uses ursus.
 Package ursustest provides assertions for testing code that uses ursus.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.