gobi

package module
v0.4.4 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 10, 2026 License: MIT Imports: 27 Imported by: 0

README

gobi

gobi is a geospatial dataframe library for Go, built on top of Apache Arrow. Think of it as a GeoPandas-shaped API with Polars-shaped internals: columnar, chunk-slice fast paths, and built around a strongly-typed schema.

Status: early. The API is settled enough to build a small pipeline on; semver stability begins with v1.0. GeoParquet v1.1 output has been verified against GeoPandas v1.1.1 and QGIS v4.0.2.

Highlights

  • Arrow-native. A Frame is a set of Arrow Columns. Head, Tail, and row selection are zero-copy where possible.
  • Full 2D + optional XYZ geometry. Point, LineString, Polygon (with holes), MultiPoint, MultiLineString, MultiPolygon, GeometryCollection. WKB and WKT round-trip per OGC/ISO SFA 1.2 (type codes 1..7 for 2D, 1001..1007 for XYZ).
  • Real spatial operations. Area, length, centroid (every geometry type), convex hull, containment, Simplify (Douglas-Peucker), Buffer with rounded OR square joins/caps, EstimateUTMCRS on every type, full topological predicates (Intersects / Contains / Within / Touches / Overlaps / Crosses / Disjoint).
  • Polygon boolean ops. Pure-Go Martinez-Rueda sweepline: Clip / Union / Difference / SymDifference and a Dissolve(geoms) collection reducer using spatially-sorted divide-and-conquer merge (like shapely's unary_union). Bit-exact parity with geopandas verified on a 500-polygon benchmark; a Sutherland-Hodgman fast path takes over when both operands are convex (4× faster than the general sweep). Linestring-vs-polygon clipping has its own Cyrus-Beck primitive: LineString.Clip(p) and SplitBy(p) (same on MultiLineString), targeting the Overture road-vs-h3-cell partitioning workload — 100-vertex hex clip in ~890 ns on M3 Pro.
  • Antimeridian handling. CrossesAntimeridian(g) detects ±180°-crossing geographic-CRS input; SplitAtAntimeridian(g) splits it into per-side components via linear-lon interpolation. EstimateUTMCRS refuses crossing input with a clear error rather than silently returning the wrong zone (a Fiji-shaped bbox [178, -178] used to pick UTM 31N over Africa).
  • Geometry constructors from columns. PointsFromXY(x, y, crs) and PointsFromXYZ(x, y, z, crs) build a WKB geometry Series directly from numeric coordinate columns — modeled on geopandas.points_from_xy. Mixed numeric types (Float64, Float32, Int64, Int32) auto-promote; nulls in either input yield a null geometry.
  • Reprojection engine. WGS84 ↔ Web Mercator ↔ all 120 UTM zones, using the ellipsoidal Redfearn/Snyder formulas. Sub-cm round-trip accuracy verified against reference cities worldwide.
  • Spatial index and join. Static Sort-Tile-Recursive R-tree with bounding-box and k-nearest queries. Internal storage is struct-of- arrays (parallel []float64 bbox arrays) — measured −27% wall time on bbox-intersect Search (100k-item tree, 1k queries) compared to the AoS shape it replaces. Public API unchanged. Frame.SJoin(right, ..., pred) with SPIntersects / SPContains / SPWithin predicates, multi-threaded across left rows, tunable via Workers(n).
  • SoA geometry kernels (v0.4.1). For hot-path callers that touch a geometry's coordinates without needing the full []Point / Polygon object graph — a common shape on the parquetio write path, spatial-sort pipelines, and per-polygon many-candidate refine loops. PointsView{Xs, Ys []float64, …} materializes parallel coordinate slabs via LineString.View(), Polygon.RingViews(), and friends. Zero-allocation WKB byte-stream scanners (BoundsFromWKB, CentroidFromWKB, CentroidAndBoundsFromWKB, PIPFromWKB, PlanarLengthFromWKB, PlanarAreaFromWKB) walk WKB bytes tracking running accumulators — semantics match ParseWKB(data).<op>() exactly on projected CRSes. PIPRingFromXY / PIPPolygonFromRings deliver a measured 3× held-view speedup vs. Polygon.Contains(pt) on the SJoin-refine shape (64-vertex polygon × 100 candidates). Wired into geoparquet.computeBboxColumns (−31% wall time, −47% memory on 100k-row GeoParquet writes), both Hilbert sort paths (−24% wall time, essentially zero-alloc WKB pass on 100k-row sorts), and Series.GeomArea / Series.GeomLength on projected CRSes (−48% / −67% wall time on 1M-vertex WKB blobs, zero allocations per row). Iterative Douglas-Peucker on parallel slabs (SimplifyDPFromXY, PointsView.SimplifyDP) replaces the recursive AoS []Point splice+append pattern — wired into every Simplify entry point transparently. −81% wall time and 24k× fewer allocations on a 1M-vertex polyline (3.85 s → 0.75 s, 260k allocs → 11). WKB → PointsView direct-parse (PolygonRingViewsFromWKB, MultiPolygonRingViewsFromWKB, LineStringViewFromWKB, PrepareFromWKB) skips the []Point intermediate for polygon-family types — 2.3× faster and 5× less memory than ParseWKB(data).RingViews() on a 1M-vertex polygon. SoA min-distance kernels (PointToSegmentDistanceSqXY, PointToPolylineMinDistanceSq) power an internal rewrite of planarMinDistance — every GeomDistance / WithinDistance caller gets −80% wall time on a 64-vertex Polygon×Polygon and −82% at 1024 vertices, from deferring math.Sqrt to a single call at the end instead of paying one per (vertex, segment) pair. Andrew's monotone-chain convex hull on slabs (ConvexHullFromXY, PointsView.ConvexHull) replaces the Graham-scan polar-angle sort — every Polygon.ConvexHull / geometry.ConvexHull(g) caller sees −30 to −54% wall time at n≥1024 and 3-4× lower memory (index sort on []int instead of []Point structs). WKB-direct Series.GeomDistance fast path (PlanarMinDistanceFromWKB) skips the AoS ParseWKB on bbox-disjoint rows — −73% memory on 10k rows of 64-vertex polygons vs a fixed target (65 MB → 17.6 MB per call, −12% wall time). Slice 14 extends the same bbox-reject pattern to Series.GeomIntersects / GeomContains / GeomWithin and the LazyFrame Col('geom').GeomIntersects(x) filter expression — −74% wall time and 98.5% less memory on a 5k-row × fixed-AOI benchmark (802 μs → 205 μs, 2.37 MB → 36 KB). Series.GeomBounds and Series.GeomCentroid also wired to the byte-stream scanners. Slice 15 sweeps the last non-clip Series entry points: Series.GeomDWithin (bbox-min-distance reject: −66% wall, −93% memory) and Series.GeomType (WKB-header peek: −88% wall, 358× fewer allocs — was pure ParseWKB waste). Slice 16 adds boundary-inclusive PIPInclusiveFromWKB and wires an SJoin Points × Polygons fast path that skips both decodeGeometryColumn passes — −62% wall, −70% memory, 33× fewer allocs on the 10k-points × 100-large-polygons bench. Slice 17 adds Series-level bbox reject for GeomClip (empty on disjoint intersection) and GeomDifference (row unchanged on disjoint difference) — −81% / −93% wall time and 334× / 530× fewer allocs on 5k rows × small mask. Slice 18 adds convex-containment fast paths inside geometry.Boolean() itself — when either operand is a convex polygon that fully contains the other, intersection returns the contained operand unchanged. −50% wall and −96% memory on the isolated containment case; −26% wall, −30% memory on the M-cell loop (100 cells × user disc). Slice 19 extends the Sutherland-Hodgman fast path from convex×convex to convex clipper × any single-ring subject when the intersection is guaranteed simply connected (transition-count gate on vertex containment) — −83% wall time on the concave-L-shape-clipped-by-AOI shape. Slice 20 extends the SJoin fast path from Points-only to LineString × Polygon and single-ring Polygon × single-ring Polygon (bilateral vertex-inside + AoS fallback for edge crossings) — −36% wall and −72% memory on 1k lines × 100 large polygons. Plus Union/Difference containment fast paths in geometry.Boolean() (union with a convex containing operand → operand unchanged; a ⊆ convex b difference → empty without the sweep). Slice 21 finishes clip coverage: Series.GeomUnion and Series.GeomSymDifference now build the combined MultiPolygon inline for bbox-disjoint rows (−89% wall, 22× fewer allocs on 5k rows × small mask), and SJoin gains SPWithin / SPContains fast paths for Polygon×Polygon and LineString×Polygon (convex-container gate; convexity precomputed during extract). Slice 22 pushes the same shape into the LazyFrame executor: Series scalar-comparison ops (GtScalar / LtScalar / etc.) now dispatch to compute.CmpF64* SIMD kernels, and the expression executor recognizes AND-chained range and bbox filters, dispatching to fused kernels (compute.AndChainF64Range / AndChainF64BBox) that skip intermediate boolean-column materialization — −49% wall and 71% fewer allocs on a 100k-row 4-comparison bbox filter. Slice 23 adds Int64 comparison kernels (compute.CmpI64Ge/Le/Gt/Lt) with matching Series wire-in, a foundational compute.CountTrue bool-reduce + Series.CountTrue() wrap, and a lane-count gate on the SIMD compare kernels that fixes a pre-existing Apple 2-lane NEON regression the Slice 22a dispatch had exposed (SIMD build now matches scalar build on Apple, wins expected on amd64 AVX2 / AVX-512).
  • Prepared-geometry predicate API. PreparedGeometry + Prepare(g) + TestPrepared(pred, a, b) amortize the AoS→SoA materialization tax across many predicate calls — the shape spatial-index-free refine loops need. Fast paths for Point×Polygon and Point×MultiPolygon; other pair shapes fall through to Test() transparently. Not used by gobi's built-in SJoin (measurement showed R-tree pre-filter drives candidate count too low for amortization to pay off); designed for advanced callers with dense overlapping polygons or many candidates per prepared geometry.
  • DataFrame ops. Filter, Take, Head, Tail, SortBy (multi-key stable, nulls-last), WithColumn, DropColumn, SelectCols, Rename, Explode (also as a LazyFrame streaming step), Join (inner / left / right / full / semi / anti with coalesced keys). GroupBy(...).Agg(...) with built-in kinds: Count, Sum, Mean, Min, Max, First, Last, NUnique, Std, Var, Median, Mode. Aggregations can carry a per-aggregation Filter Expr for SUM(x) FILTER (WHERE …)-style reductions. Series arithmetic, comparisons, aggregations — all with single-chunk bulk fast paths and Int64-preserving scalar arithmetic.
  • Built-in collect-set aggregators. NewStringSetAggregator(), NewInt64SetAggregator(), NewUint64SetAggregator(), NewInt32SetAggregator(), NewUint32SetAggregator() — distinct- value roll-ups that emit List<T> per group and stream through the aggregate executor via IncrementalAggregator.
  • User-defined aggregations. type Aggregator interface { ... } plugs directly into GroupBy.Agg alongside the built-ins. Opt into the streaming aggregate executor by additionally implementing IncrementalAggregator (Clone / Update / Finalize) — per- batch state updates instead of a materialize-then-reduce pass.
  • Expression IR. gobi.Col("price").Mul(gobi.Lit(1.08)).Gt(gobi.Lit(100)) builds a data tree, not a chain of already-executed calls. Frame.FilterExpr and Frame.WithColumnExpr evaluate it. The built-in vocabulary covers arithmetic (Add/Sub/Mul/Div), bitwise (BitAnd/BitOr/BitXor), comparisons, logical (And/Or/Not), IsNull/IsNotNull, Cast(dtype) (numeric- to-numeric + Timestamp source), If/Coalesce, LitNull(dtype), LitEmptyList(elem), ListLen, ListUnion, Shift(n), window functions (.Sum()/.Mean()/.Min()/.Max()/.Count()/.Median()/ .Mode().Over(cols...) for scalar-agg-and-broadcast; shape-preserving inners like Shift(1).Over(K) for prev-row-within-partition patterns), UnixNano() (Timestamp → Int64 ns), and HaversineExpr(lat1, lon1, lat2, lon2, unit) for great-circle distance between two point columns. A Custom(node ExprNode) escape hatch lets sibling packages (H3, hashes, ML inference) plug in their own expression types alongside the built-ins.
  • Alignment-aware fast paths. PartitionMetadata claims (attached via LazyFrame.WithPartitionAssertion or produced by contrib/athenaio on Iceberg CTAS output) let GroupBy, Over, and Join skip the hash-shuffle and linear-scan partition boundaries directly. Runs 30-70% faster than the general path when applicable; falls through automatically otherwise.
  • List and Struct columns. First-class support end-to-end: Explode expands list rows, ListUnion merges per-row, aggregations can emit List<T> (see set aggregators above), and Struct-typed builders round-trip through Frame → LazyFrame → Frame.
  • LazyFrame + rule-based optimizer. df.Lazy() and parquetio.ScanFile(path) build plan trees that don't execute until .Collect(). Nine rewrite rules run to a fixed point: constant folding, dead-filter removal, adjacent-filter combining, push-filter-below-project, push-filter-below-sort, column projection pushdown, predicate pushdown (into row-group stats), and cascade-empty (short-circuits Lit(false)-derived subtrees). Projection pushdown routes into parquetio.ReadOptions.Columns — 2.4× faster reads on partial-column queries, matching the eager baseline. Optimizer overhead is ~8 µs on a five-node plan; always-on.
  • Parallel streaming executor. LazyFrame.Collect() compiles the optimized plan to a tree of ExecOperators that pull one record batch at a time — bounded memory regardless of source size. Filter, Project, WithColumn, Drop, Rename, Select, Explode, Limit, ScanFrame, and ScanFile all stream natively. Adjacent batch-transform ops are fused into a single per-batch pass by the compiler (~22% fewer allocations on typical Filter → Project → WithColumn chains). Aggregate (all built-in kinds + custom IncrementalAggregators) and hash-join (Inner/Left/Semi/Anti) run as native streaming operators — no materialization step. Parquet scan parallelizes across row-groups; the streaming hash aggregate partitions rows across workers by key hash. Both scale to GOMAXPROCS out of the box. LazyFrame.ExplainPhysical() prints what strategy each node compiles to (worker counts included).
  • Datetime + timezone-aware ops. Timestamp[ns] columns with optional IANA tz label. Component extractors, AddDuration / DiffDuration, comparisons, sub-day + calendar truncation. ResampleEvery(timeCol, interval) for downsampling and RollingBy(timeCol, period) for trailing time windows; plus fixed-size Series.RollingSum / Mean / Min / Max / Count.
  • Multi-format I/O. Every format has ReadFile / WriteFile at Frame level plus a ScanFile LazyFrame entry point where the underlying source supports it, and every format also has struct-direct ReadStructs[T] / WriteStructs[T] wrappers with a per-format struct-tag namespace — parquet: / csv: / geojson: / gpkg: / shp: / kml: / pgio: — so the same Go type can carry different column names per format (shp's 10-char DBF alias, parquet:"-" to omit from parquet only, etc.). Formats: CSV (with .gz / .zst / .bz2 auto-detect), Parquet with proper GeoParquet 1.1 metadata (snappy / gzip / brotli / zstd / lz4, canonical PROJJSON for EPSG:3857 + all 120 UTM zones), full RFC 7946 GeoJSON (every geometry type + XYZ, .geojsonl streaming), OGC GeoPackage 1.3 (SQLite, RTree spatial index, spec-compliant metadata), PostgreSQL / PostGIS (via pgx/v5, native CopyFrom bulk load), KML + KMZ read/write (zipped KML auto-detected by .kmz extension), and Shapefile read/write (.shp + .shx + .dbf + optional .prj).
  • Streaming readers. csvio.ReadFileChunksFunc and parquetio.ReadFileChunksFunc yield one Frame per record batch (~64k rows), releasing arrow buffers after each callback. Peak memory is bounded regardless of source-file size — good for ETL over multi-GB inputs.
  • Column projection. parquetio.ReadOptions{Columns: ...} skips fetch, decompress, and arrow materialization for the columns you don't need. Composes with streaming.
  • Parquet write tuning. parquetio.WriteOptions exposes RowGroupRows (predicate-pushdown-friendly small groups vs. compression-friendly large ones), BloomFilterColumns + BloomFilterFPP (equality-filter skipping in DuckDB / Spark / Polars / pyarrow / gobi readers today).
  • Parallelism controls. Package-level SetMaxParallelism(n) or per-op Workers(n).
  • Pure Go, no cgo. No GDAL, no GEOS, no libproj. Cross-compiles cleanly to every architecture Go supports.

Install

go get github.com/zoobst/gobi

Requires Go 1.26 or newer.

Docs

Full API reference is on pkg.go.dev — auto-generated from source doc comments. Every subpackage (parquetio, csvio, geometry, geojsonio, gpkgio, pgio, kmlio, shpio) has its own page; use the same base URL with the package path appended.

Quick start

Read a CSV, write a GeoParquet
package main

import (
    "github.com/zoobst/gobi/csvio"
    "github.com/zoobst/gobi/parquetio"
)

type city struct {
    Name       string `csv:"name"`
    Population int64  `csv:"population"`
    Geom       string `csv:"geometry" geom:"true"`
}

func main() {
    // .gz / .zst / .bz2 are auto-detected from the filename; explicit
    // ReadOptions.Compression overrides.
    df, err := csvio.ReadFile[city]("cities.csv.gz", &csvio.ReadOptions{CRSHint: 4326})
    if err != nil { panic(err) }
    defer df.Release()

    // The output file carries a spec-compliant GeoParquet 1.1 metadata
    // blob and reads cleanly in GeoPandas / QGIS. nil = Snappy + arrow's
    // default row-group sizing.
    _ = parquetio.WriteFile(df, "cities.parquet", nil)
}
Build a geometry column from lat/lng
// Turn two numeric columns into a WKB geometry column. Argument
// order is x, y — i.e. longitude, latitude — matching WKB / GeoJSON /
// geopandas.points_from_xy. Nulls on either side yield a null point.
lng, _   := df.Column("lng")
lat, _   := df.Column("lat")
points, _ := gobi.PointsFromXY(lng, lat, 4326)
df, _     = df.WithColumn("geometry", points)
// df["geometry"] is now a proper WKB column, ready for SJoin,
// GeoParquet write, etc.
Spatial join
cities, _ := parquetio.ReadFile("cities.parquet", nil)   // 1M points
regions, _ := parquetio.ReadFile("regions.parquet", nil) // 5k polygons

// Which region contains each city?
joined, err := cities.SJoin(regions, "geometry", "geometry", gobi.SPWithin)

// Cap parallelism per-op (see "Parallelism" below):
joined, err = cities.SJoin(regions, "geometry", "geometry", gobi.SPWithin, gobi.Workers(4))
GroupBy + aggregate
gb, _ := df.GroupBy("region")
totals, _ := gb.Agg(
    gobi.Aggregation{Column: "population", Kind: gobi.AggSum},
    gobi.Aggregation{Column: "population", Kind: gobi.AggMean, Alias: "avg_pop"},
)
Sort
// Multi-key, stable, nulls-last. Earlier keys have priority;
// later keys break ties.
sorted, _ := df.SortBy(
    gobi.SortKey{Column: "date"},                        // ascending
    gobi.SortKey{Column: "revenue", Descending: true},
)
Tuned parquet write
// Small row groups + bloom filters on high-cardinality equality columns.
// DuckDB / Spark / Polars / pyarrow readers all use the bloom filters
// for predicate pushdown on equality filters.
err := parquetio.WriteFile(df, "events.parquet", &parquetio.WriteOptions{
    Codec:              parquetio.CodecZstd,
    RowGroupRows:       128_000,                     // 0 = arrow default (~1M)
    BloomFilterColumns: []string{"user_id", "session_id"},
    BloomFilterFPP:     0.01,                        // 0 = arrow default (0.05)
})
Streaming ETL
// Reads a 5 GB parquet file at ~15 MB peak memory. Only two columns are
// fetched off disk; the rest are never decompressed.
err := parquetio.ReadFileChunksFunc(
    "events.parquet",
    &parquetio.ReadOptions{Columns: []string{"user_id", "ts"}, ChunkRows: 64_000},
    func(batch *gobi.Frame) error {
        // Process ~64k rows at a time. The batch is released after
        // return; call batch.Retain() to keep it past this callback.
        return sink.Write(batch)
    },
)

CSV has the same shape: csvio.ReadFileChunksFunc[Row](path, opts, fn).

Derived columns

Two shapes. WithColumn accepts any Series the caller built by hand:

// A user-space helper produces the derived Series any way it likes
// (external library call, vectorized loop, whatever). WithColumn wires
// it back into the Frame — appending, or replacing an existing column
// of the same name.
lat, _   := df.Column("lat")
lng, _   := df.Column("lng")
cells, _ := h3x.Encode(lat, lng, 9)
df, _     = df.WithColumn("h3", cells)

df, _ = df.DropColumn("raw_geometry")

WithColumnExpr accepts an expression tree, so pipelines composed from built-in ops read left-to-right:

// Same shape via the expression IR — no intermediate Series to name.
df, _ = df.WithColumnExpr("usd_price",
    gobi.Col("eur_price").Mul(gobi.Lit(1.08)),
)
df, _ = df.WithColumnExpr("margin",
    gobi.Col("revenue").Sub(gobi.Col("cost")),
)
User-defined aggregation
// Compute the 95th percentile of a numeric column per group.
type P95 struct{}

func (P95) Aggregate(s gobi.Series, rows []int) (any, error) {
    arr := s.Column().Data().Chunks()[0].(*array.Float64)
    vals := make([]float64, 0, len(rows))
    for _, r := range rows {
        if !arr.IsNull(r) { vals = append(vals, arr.Value(r)) }
    }
    if len(vals) == 0 { return nil, nil }
    sort.Float64s(vals)
    return vals[int(float64(len(vals)-1)*0.95)], nil
}
func (P95) Type() arrow.DataType { return arrow.PrimitiveTypes.Float64 }
func (P95) Name() string          { return "p95" }

// Mix custom + built-in aggregations in one call.
gb, _ := df.GroupBy("h3")   // Uint64 keys are hashable
out, _ := gb.Agg(
    gobi.Aggregation{Column: "latency_ms", Kind: gobi.AggMean},
    gobi.Aggregation{Column: "latency_ms", Fn: P95{}},
)
Filter

Two shapes. Filter takes an already-computed Boolean Series (mask):

pops, _ := df.Column("population")
mask, _ := pops.GtScalar(1_000_000)
big,  _ := df.Filter(mask)

FilterExpr takes an expression tree — no intermediate Series to name:

big, _ := df.FilterExpr(
    gobi.Col("population").Gt(gobi.Lit(1_000_000)).
        And(gobi.Col("country").Eq(gobi.Lit("US"))),
)
Window functions

Over(partitionCols...) runs either an aggregate (broadcast to every row in the partition) or a shape-preserving inner like Shift(1) (prev-row-within-partition). Both compose with the alignment fast paths when PartitionMetadata proves the frame is partition-contiguous.

// Per-region total (broadcast). Same shape works for Mean / Min /
// Max / Count / Median / Mode.
df, _ := df.WithColumnExpr("region_total",
    gobi.Col("sales").Sum().Over("region"),
)

// Previous row's timestamp within (eid) partition. First ping of
// each entity emits null.
df, _ := df.WithColumnExpr("prev_ts",
    gobi.Col("ts").Shift(1).Over("eid"),
)

// Great-circle distance between successive pings, entirely in the
// expression tree — no Custom ExprNode needed.
df, _ := df.WithColumnExpr("step_km", gobi.HaversineExpr(
    gobi.Col("lat"),
    gobi.Col("lon"),
    gobi.Col("lat").Shift(1).Over("eid"),
    gobi.Col("lon").Shift(1).Over("eid"),
    geometry.UnitKilometers,
))

For flag-unpacking a packed Int64 into per-bit indicator columns:

const FlagBogonIP = 1 << 3
df, _ := df.WithColumnExpr("has_bogon_ip",
    gobi.Col("flags").
        BitAnd(gobi.Lit(int64(FlagBogonIP))).
        Ne(gobi.Lit(int64(0))),
)
Reproject
p := geometry.Point{X: -73.9857, Y: 40.7484, CRSValue: geometry.WGS84}
utm, _ := p.EstimateUTMCRS()   // EPSG:32618 (WGS 84 / UTM zone 18N)
proj,  _ := p.ToCRS(utm)       // coordinates now in meters
Buffer + simplify
poly := geometry.SimplePolygon(points, geometry.PseudoMercator)

// Rounded (semicircle caps, rounded joins) — the default.
rounded, _ := geometry.Buffer(poly, 100, geometry.BufferOptions{Segments: 32})

// Square-style buffer: flat/extended caps, mitre joins. Faster and
// produces fewer vertices (5 instead of 33 for a Point).
square, _ := geometry.Buffer(poly, 100, geometry.BufferOptions{
    Style: geometry.BufferSquare,
})

simpler := poly.Simplify(5.0) // Douglas-Peucker at 5-unit tolerance
Polygon boolean ops (Clip / Union / Dissolve)
subject := geometry.SimplePolygon(subjectPts, geometry.PseudoMercator)
mask    := geometry.SimplePolygon(maskPts, geometry.PseudoMercator)

// Row-scalar ops on a whole geometry Series.
gs, _ := df.Column("geom")
clipped,    _ := gs.GeomClip(mask)         // intersection per row
unioned,    _ := gs.GeomUnion(mask)
diffed,     _ := gs.GeomDifference(mask)

// Dissolve every row's polygon into a single geometry (like
// shapely's unary_union — divide-and-conquer merge, bit-exact
// against geopandas on the bundled 500-polygon bench).
merged, _ := gs.GeomDissolve()

// Or reach the raw ops directly on a Geometry.
inter, _ := geometry.Clip(subject, mask)
uni,   _ := geometry.Union(subject, mask)
Linestring clipping (LineString / MultiLineString vs Polygon)

Boolean ops handle Polygon-vs-Polygon; linestring-vs-polygon has its own primitive that skips the general polygon sweep entirely.

line := geometry.NewLineString([]geometry.Point{
    {X: -5, Y: 5}, {X: 15, Y: 5}, {X: 15, Y: -5}, {X: -5, Y: -5},
}, geometry.PseudoMercator)
cell := geometry.SimplePolygon(hexPts, geometry.PseudoMercator)

inside            := line.Clip(cell)          // fragments inside cell
inside, outside   := line.SplitBy(cell)       // both sides, one pass

// MultiLineString version — flat []LineString result, walk order
// preserved across components.
ml := geometry.NewMultiLineString([]geometry.LineString{line, other}, geometry.PseudoMercator)
frags := ml.Clip(cell)

Convex polygons (including h3 cells) take a Cyrus-Beck fast path with per-segment AABB reject; concave / multi-ring polygons fall back to sort-and-march. Coordinate system is planar — densify first if you need geodesic accuracy, and use SplitAtAntimeridian before clipping if your linestring crosses ±180°.

Antimeridian split
// A geographic-CRS polygon spanning ±180° confuses every naive
// cartesian primitive. Detect + split it into per-side components
// before feeding it downstream.
poly := geometry.SimplePolygon([]geometry.Point{
    {X: 170, Y: -10}, {X: -170, Y: -10},
    {X: -170, Y: 10}, {X: 170, Y: 10}, {X: 170, Y: -10},
}, geometry.WGS84)

if geometry.CrossesAntimeridian(poly) {
    split, _ := geometry.SplitAtAntimeridian(poly)
    // split is a MultiPolygon with one component in [170,180] and
    // another in [-180,-170]; each has valid, non-crossing bounds.
}
Struct-direct io

Every io package has a struct-direct wrapper that uses per-format struct tags — parquet: for parquet, shp: for shapefile, geojson: for GeoJSON, etc. Resolution falls back to a universal gobi: tag when the format-specific one is absent, then to csv: (legacy), then to the Go field name. format:"-" skips a field.

type Feature struct {
    ID         int64   `parquet:"id" shp:"OBJECTID"` // per-format aliases
    Name       string  `gobi:"name"`                 // shared across formats
    Population float64 `shp:"POP10"`                 // 10-char DBF alias
    Notes      string  `parquet:"-"`                 // omit from parquet
    Geometry   []byte  `geom:"true"`                 // geometry marker
}

// Read a slice of structs directly — no Frame boilerplate.
rows, err := parquetio.ReadStructs[Feature]("in.parquet", nil)

// Write the same slice out as a shapefile — column names come from
// the shp: tags automatically.
err = shpio.WriteStructs(rows, "out", nil)

The same shape works for csvio.ReadStructs, geojsonio.ReadStructs / WriteStructs, gpkgio.ReadStructs / WriteStructs, kmlio.ReadStructs / WriteStructs, and pgio.ReadStructsQuery / ReadStructsTable / WriteStructsTable.

R-tree
tree := geometry.NewRTree(bboxes)
hits    := tree.Search(query)      // IDs whose bounds intersect query
nearest := tree.Nearest(x, y, k)   // k closest bounds, sorted

Internal storage is struct-of-arrays: parallel []float64 bbox arrays for both item and node bboxes. Search sees a measured 27% wall-time reduction on 100k-item / 1k-query workloads vs. the AoS shape it replaces. Public API is unchanged from earlier releases.

Datetime + timezone
type event struct {
    Name string    `csv:"name"`
    When time.Time `csv:"when" time:"2006-01-02 15:04:05"`
}

df, _ := csvio.ReadFile[event]("events.csv", nil)
when, _ := df.Column("when")

// Render the same instants in New York local time.
nyWhen, _ := when.WithTimezone("America/New_York")

// Component extractors honor the tz.
hourNY, _ := nyWhen.Hour()   // Int64 series with local hours

// Truncate to the top of each local day.
dayStart, _ := nyWhen.TruncateToCalendar(gobi.CalendarDay)
Resample + rolling
// Downsample to hourly buckets (Unix-epoch aligned).
r, _ := df.ResampleEvery("when", time.Hour)
hourly, _ := r.Agg(
    gobi.Aggregation{Column: "value", Kind: gobi.AggSum},
    gobi.Aggregation{Column: "value", Kind: gobi.AggMean, Alias: "avg"},
)

// Trailing 5-minute rolling sum keyed by timestamp.
tr, _ := df.RollingBy("when", 5*time.Minute)
rollSum, _ := tr.Agg("value", gobi.AggSum)

// Fixed-window rolling on a plain Series.
val, _ := df.Column("value")
m7, _ := val.RollingMean(7) // 7-row moving average
KML / KMZ / Shapefile
// KML → Frame (auto-parses ExtendedData into columns)
places, _ := kmlio.ReadFile("places.kml", nil)
_ = kmlio.WriteFile(places, "out.kml", nil)

// KMZ works the same way — extension picks the format automatically.
// Set kmlio.WriteOptions{Format: FormatKMZ} explicitly for Writer flows.
_ = kmlio.WriteFile(places, "out.kmz", nil)         // writes zip with doc.kml

// Shapefile → Frame (reads .shp + .shx + .dbf + optional .prj)
counties, _ := shpio.ReadFile("counties", nil)      // no .shp suffix needed
_ = shpio.WriteFile(counties, "counties_out", nil)  // writes all four files

Packages

Package What it does
github.com/zoobst/gobi Frame, Series, GroupBy, Join, SJoin, Explode, datetime + rolling + resample, options
.../gobi/geometry 2D + XYZ primitives, WKB / WKT, CRS + reprojection, predicates, R-tree, Buffer / Simplify / Centroid
.../gobi/csvio Typed CSV read + streaming (ReadFileChunksFunc), gzip / zstd / bzip2 auto-detect
.../gobi/parquetio Parquet read/write + streaming + column projection + row-group + bloom-filter tuning; snappy/gzip/brotli/zstd/lz4 + GeoParquet 1.1
.../gobi/geojsonio Full RFC 7946 GeoJSON (all geometry types + XYZ) — Frame-level ReadFile/WriteFile/ScanFile, .geojsonl streaming
.../gobi/gpkgio Read / write OGC GeoPackage 1.3 (SQLite) with RTree spatial index + LazyFrame ScanFile + SQL predicate pushdown
.../gobi/pgio Beta. PostgreSQL / PostGIS via pgx/v5 — ReadQuery/ReadTable/ScanTable + WriteTable with CopyFrom bulk load. Integration tests are //go:build integration-gated; set PGIO_TEST_DSN and run against a live PostGIS to exercise them.
.../gobi/kmlio Read / write KML (OGC 12-007r2) + KMZ (zipped KML). Placemarks + ExtendedData. .kmz extension auto-detected.
.../gobi/shpio Read / write ESRI Shapefile (.shp + .shx + .dbf + optional .prj)
.../gobi/contrib/athenaio AWS Athena CTAS integration — UnloadAndRead, UnloadAndReadBuckets, RawCTAS, RawCTASBuckets return LazyFrames with PartitionMetadata claims that flow into gobi's aligned fast paths. Own go.mod; versions independently.

Geometry columns

gobi intentionally does not use an Arrow custom-extension type for geometry. Geometries are Arrow Binary columns holding WKB, with the column marked in schema metadata:

"gobi:geometry_type" = "WKB"
"gobi:crs_epsg"      = "4326"

When writing Parquet, gobi additionally emits a proper GeoParquet 1.1 geo blob at the file level with primary_column, geometry_types, crs, and bbox. That is what makes gobi-produced files interoperate with GeoPandas and QGIS out of the box.

Use gobi.GeometryField(name, epsg) to construct a tagged field manually.

Extending gobi

Three extension points cover most add-on work today, without forking:

1. Derived columns via a helper package + Frame.WithColumn. Write a sibling package (e.g. h3x, hashcol) whose functions take one or more gobi.Series and return a gobi.Series. Users compose:

lat, _   := df.Column("lat")
lng, _   := df.Column("lng")
cells, _ := h3x.Encode(lat, lng, 9)
df, _     = df.WithColumn("h3", cells)

Because the helper controls the whole loop, it can dispatch to native libraries (H3, MurmurHash, whatever) once per row without the DataFrame having to know anything about the operation. WithColumn appends or replaces, DropColumn removes.

2. Custom aggregations via the Aggregator interface.

type Aggregator interface {
    // Reduce s[rows...] to a single scalar. Return nil for a null.
    Aggregate(s Series, rows []int) (any, error)
    // Declares the arrow type of Aggregate's return values. Supports
    // Float32/64, Int32/64, Uint32/64, Bool, String, Binary, Timestamp.
    Type() arrow.DataType
    // Suffix for the default output column name.
    Name() string
}

Set Aggregation{Column: "col", Fn: myAgg} and call GroupBy.Agg as usual. Mix custom + built-in aggregations in a single call. If the returned dynamic type doesn't match the declared Type(), Agg returns an error naming the offending aggregation rather than panicking.

Opt into the streaming aggregate executor by additionally implementing IncrementalAggregator:

type IncrementalAggregator interface {
    Aggregator
    // Clone returns a fresh instance with empty per-group state.
    // Called once per group at first-touch; each group runs on its
    // own clone, no cross-group state sharing.
    Clone() IncrementalAggregator
    // Update adds col[rows] to this instance's state. Called at
    // most once per input batch per group. Additive — do not reset.
    Update(col Series, rows []int) error
    // Finalize returns the group's aggregated value. Called once
    // per group after all Updates.
    Finalize() any
}

Plain Aggregators still work — they route through the materializing fallback exactly as before. IncrementalAggregators skip the materialize-then-reduce and process per batch, matching the built-in aggregators' streaming shape. The bundled set aggregators (NewStringSetAggregator, etc.) are the reference implementations.

3. Custom expression nodes via the ExprNode interface.

type ExprNode interface {
    // Evaluate against a Frame. Return a Series with input.NumRows() rows.
    Eval(input *Frame) (Series, error)
    // Declared output arrow type given the input schema. Used by
    // FilterExpr / WithColumnExpr for validation.
    Type(schema *arrow.Schema) (arrow.DataType, error)
    // Sub-expressions, for tree walkers.
    Children() []Expr
    // Pretty-printer for logs and debug output.
    String() string
}

Wrap your node with gobi.Custom(node) and it composes with the built-in Col, Lit, and operator methods:

// h3x.Encode returns a gobi.Expr backed by a custom node.
cellExpr := h3x.Encode(gobi.Col("lat"), gobi.Col("lng"), 9)

df, _ = df.WithColumnExpr("h3", cellExpr)
df, _ = df.FilterExpr(cellExpr.Eq(gobi.Lit(uint64(0xdead))))

Because expressions are data, not function calls, the tree can be inspected before evaluation (e.String(), e.Node().Children()), type-checked without touching the buffers (e.Node().Type(schema)), and — in a future release — rewritten by an optimizer that pushes predicates into scans and prunes unused columns. Extension points that implement ExprNode will benefit from those passes automatically.

Group-by key types. Hashable key columns: String, Bool, Int32, Int64, Uint32, Uint64, Float64, Timestamp. Uint64 is what makes H3-cell grouping ergonomic.

Parallelism

Two layers of parallel execution work together in a LazyFrame collect: the parquet scan splits row-groups across workers, and the streaming aggregate partitions rows by key hash across workers.

  • Parallel scan. parquetio.ScanFile(path, &parquetio.ReadOptions{ScanWorkers: N}) splits row-groups across N goroutines. Each worker reads a disjoint subset of row-groups; batches fan-in through a bounded channel. ScanWorkers: 0 (the default) auto-picks GOMAXPROCS, capped at the file's row-group count. ScanWorkers: 1 forces serial for reproducibility. Files with a single row-group skip parallel scan automatically (no benefit possible).
  • Parallel aggregate. The streaming hash aggregate (GroupBy(...).Agg(...) on a LazyFrame with built-in Kinds) partitions rows across GOMAXPROCS workers by key hash — no cross-worker key overlap, no locks, no value-level combine at merge. Kicks in for any aggregate where every Aggregation uses a built-in Kind; custom Fn aggregators still route through the materializing fallback.

Both layers respect the package-level SetMaxParallelism(n) / per-op Workers(n) overrides, in this priority:

  1. Per-op gobi.Workers(n) option
  2. Package default via gobi.SetMaxParallelism(n)
  3. GOMAXPROCS
gobi.SetMaxParallelism(4)                 // process-wide default
df.SJoin(..., gobi.Workers(8))            // override for one call
df.SJoin(..., gobi.Workers(1))            // force sequential

LazyFrame.ExplainPhysical() shows the resolved worker count for each parallel node — useful when debugging why a query didn't get the parallelism you expected.

Design constraints

  • Pure Go, no cgo. GDAL, GEOS, libproj, and other C libraries are intentionally off the table. This keeps go build clean across every platform Go targets and avoids the LGPL/toolchain overhead. The trade: no File Geodatabase support and no PROJ-grade reprojection beyond WGS84 / Web Mercator / UTM. Polygon boolean ops (Clip / Union / Difference / SymDifference / Dissolve) DO ship — as a pure-Go Martinez-Rueda sweepline — but complex-projection pairs like Albers ↔ Lambert still require an external tool.

Performance

All numbers Apple M3 Pro, warm cache, 10–20 iterations per op. Fixtures and scripts live under benchmarks/ — regenerate with go run generate_fixture.go, go run generate_csv_fixture.go, and go run generate_spatial_fixture.go.

Compute ops (1M-row Parquet, non-spatial)
Op gobi (default) gobi (SIMD) pandas 2.3 Polars 1T Polars all
Sum(value_a) 1.04 ms 0.43 ms 0.31 ms 0.10 ms 0.09 ms
value_a + value_b 1.79 ms 1.55 ms 1.03 ms 0.82 ms 0.90 ms
Filter(value_a > 500k) 15.9 ms 14.9 ms 6.83 ms 1.96 ms 1.60 ms
GroupBy(key).Agg(Sum,Mean) 45.4 ms 42.6 ms 19.73 ms 9.89 ms 2.42 ms

Polars 1T = POLARS_MAX_THREADS=1; Polars all = default (all cores). gobi (SIMD) = compiled with GOEXPERIMENT=simd on Go 1.27+ arm64/amd64; gobi (default) = normal build with the scalar compute path. Sum gained a 2.4× speedup under SIMD (compute.SumF64 via simd.Float64s.Add lane-parallel reduce). The other ops changed less because SIMD helps compute-bound loops most and pandas/Polars' remaining lead here is architectural (NumPy's arena allocator, Polars' cache-blocked kernels) rather than kernel-level SIMD. See compute/ for the full per-op arrow-go vs gobi kernel positioning.

Cross-arch SIMD note (v0.4.1): the SIMD wins in this table are on Apple M-series (deep out-of-order execution engines). Measured on AWS Graviton 2 / Ampere Altra (Neoverse-N-class server ARM64), the compute.BoundsF64 kernel actually regresses ~48% because Neoverse's NEON Float64.Min/Max instructions cost more per lane than scalar compare-branch. Other kernels (SumF64, and the new-in-v0.4.1 PolygonCentroidShoelace) win on both architectures — shoelace measures −28% wall time on Ampere at every size ≥64, while flat on Apple silicon. The rule: GOEXPERIMENT=simd is safe to enable on server ARM64, but call compute.BoundsF64 explicitly only when targeting Apple silicon. geometry.BoundsFromXY stays portable scalar so it's fast everywhere. See the "Slice 6" section in .vscode/CLAUDE.md for the arch-vs-kernel decision table + measured numbers.

CSV read (38.6 MB / 1M rows)
Reader per-read notes
Polars 1.42, all threads 11.5 ms multi-threaded Rust tokenizer + SIMD (typed schema)
Polars 1.42, 1 thread 13.5 ms SIMD numeric parse, single core
pandas 2.3, engine="pyarrow" 43.2 ms pyarrow C++ tokenizer
pandas 2.3, default (C engine) 149.4 ms pandas' native C tokenizer
gobi csvio.Read 224.3 ms arrow-go's CSV wraps stdlib encoding/csv

The gap is entirely in stdlib encoding/csv allocating a []string per row + per-cell strconv — 99.5% of gobi's CSV allocations show up there in a pprof run. Closing it means replacing that layer with a byte-level tokenizer that writes straight into Arrow buffers; not on the roadmap yet. Maybe the arrow-go folks will pick that up.

Spatial ops (100k points × 100 polygons)
Op gobi geopandas 1.1 result
Read points.parquet (100k) 4.04 ms 37.74 ms 9.3× faster
Read polygons.parquet (100) 0.26 ms 0.83 ms 3.2× faster
Area(polygons) 0.02 ms 0.13 ms 6.5× faster
Centroid(polygons) 0.02 ms 0.16 ms 8× faster
SJoin(100k pts, 100 polys) 3.49 ms 2.62 ms 1.3× slower

Gobi wins on read and per-row bulk ops because it doesn't have to construct Shapely Python objects per row on load. The one gap is sjoin: geopandas uses Shapely 2's GEOS-backed STRtree in C++; gobi's Sort-Tile-Recursive R-tree is pure Go. Landing within 40% of a GEOS-C++ index while staying cgo-free is the intended trade.

LazyFrame + optimizer (projection pushdown)

Same 1M-row parquet fixture, Select(id, value_a) — reading 2 of 4 columns. Measures the projection-pushdown rule's effect on I/O and decode cost.

Path per-op vs. baseline
ReadFile(path, Options{Columns:[id,value_a]}) (eager) 6.21 ms 1.0× (baseline)
ScanFile(path).Select(id,value_a).Collect() (optimized) 8.12 ms ~1.31× — close to eager
ScanFile(path).Select(id,value_a).CollectRaw() (no rules) 14.58 ms 2.3× slower — reads all 4 cols

The optimizer's ProjectionPushdown rule turns the lazy pipeline into the equivalent of an eager Options.Columns — 2.0× faster than the same pipeline with optimization disabled. Optimizer overhead itself is 8 µs per plan (measured on a 5-node pipeline), or 0.14% of the collect time. Always-on optimization is effectively free.

1 Billion Row Challenge

The 1BRC fixture is 1 billion weather-station rows in Snappy-compressed parquet (~4 GB on disk). Query: min / mean / max of temperature grouped by station. Apple M3 Pro, 11 GOMAXPROCS.

Engine wall user CPU peak RSS
gobi streaming (parallel scan + agg) 12.7s ~88s 611 MB
Polars 1.42 streaming (reference) 3.0 s ~15s 4.42 GB
Polars 1.42 eager (reference) 12.0 s ~120s 20.96 GB

Streaming end-to-end — no LazyFrame.CollectRaw() materialization, no disk spill. Getting here took six complementary changes: partitioning the parquet scan across row-groups; sharding the streaming hash aggregate by key hash across workers; eliminating per-row key allocations (reusable scratch buffers + single-string-key fast path that reads the arrow value zero-copy); eliminating the per-row map[*aggGroup][]int bucket lookup that sat between groups[key] and the accumulator Update calls (v0.3.9); pooling the parallel dispatcher's per-worker []int row-index payloads via sync.Pool (v0.3.9); and threading the pooled payloads as *[]int through the channel path so sync.Pool.Put doesn't box the slice header into an interface per call (v0.3.9). gobi's own allocations across a full 1BRC run are ~540 MB total — down from ~13 GB pre-pool.

Peak RSS is 7.2× lower than Polars streaming and 34× lower than Polars eager because gobi keeps at most one batch per worker in memory + recycles per-batch scratch through sync.Pool — Polars buffers larger working sets by design.

Alignment fast paths (1M rows, 1000 groups)

When the caller can prove input partitioning + sort order via WithPartitionMeta(...), Over and GroupBy skip the general hash-map path and linear-scan group boundaries directly. Same bench_alignment.parquet fixture (1M rows, 10 regions × 100 ids, pre-sorted by (region, id)) across all three rows; Polars doesn't expose alignment metadata, so it picks its own path.

Op gobi (unaligned) gobi (aligned) Polars all
Over(region, id).Sum(v) 47.8 ms 29.9 ms (37% faster) 7.11 ms
GroupBy(region, id).Sum(v) 94.6 ms 22.1 ms (77% faster) 4.65 ms

The aligned GroupBy number improved from 32.4 ms → 22.1 ms in v0.3.9 (vs the v0.2.0 landing) via two complementary changes: the streaming- aggregate refactor (buckets-map elimination + sync.Pool for per-worker row-index payloads, -32% overall) and the aligned-path SIMD reduce wire-in (compute.SumF64 on contiguous per-group slices under GOEXPERIMENT=simd).

Sort-merge Inner join (also alignment-gated) is measured in-tree rather than in this fixture bench — a fixture-scale self-join is dominated by 100M-row cross-product construction rather than the join algorithm itself. BenchmarkJoin_MergeAligned on 10k×10k Int64 keys shows the aligned sort-merge path is 31% faster than the hash join it replaces, and BenchmarkJoin_HashMultiProbeBatch shows the streaming-hash-join build-side cache fix that landed with the alignment work is 49% faster on multi-batch Inner joins.

Polars still wins in absolute terms — the aligned fast paths close a 1.6× gap on Over and cut a 3.3× gap on GroupBy to 4.8× overall. If the workload can carry alignment metadata, taking it is nearly free.

GeoParquet write + Hilbert sort (v0.4.1 SoA push)

Two write-path measurements on a 100k-row 9-vertex-polygon Frame, before/after the Slice 2 + Slice 3 wire-ins (BoundsFromWKB, CentroidAndBoundsFromWKB). No config change required — the scanners are the default path on every build.

Path before (v0.4.0) after (v0.4.1) delta
WithBboxCoveringColumns (write) 31.7 ms / 154 MB / 600k allocs 21.9 ms / 81 MB / 300k allocs −31% wall, −47% mem
SortByHilbert 33.6 ms / 115.6 MB / 300k allocs 25.5 ms / 42.9 MB / 66 allocs −24% wall, −63% mem, essentially zero-alloc pass
HilbertSortWithCovering 51.1 ms / 203.5 MB / 600k allocs 41.8 ms / 131 MB / 300k allocs −18% wall, −36% mem

The wins come from skipping the ParseWKB full-geometry allocation on every row when the caller immediately discards the geometry after .Bounds() / .Centroid(). Byte-stream scanners walk the WKB blob tracking running accumulators — zero allocation per row on well-formed input.

Slice 7 extends the same shape to Series.GeomArea / Series.GeomLength on projected CRSes via PlanarAreaFromWKB / PlanarLengthFromWKB. Microbench on Apple M3 Pro:

Path n ParseWKB + AoS Scanner delta wall allocs/op
PlanarLengthFromWKB 1M 12.2 ms 4.0 ms −67% 2 → 0
PlanarLengthFromWKB 1K 14.5 μs 3.4 μs −76% 2 → 0
PlanarAreaFromWKB 1M 6.6 ms 3.5 ms −48% 3 → 0
PlanarAreaFromWKB 1K 8.9 μs 3.5 μs −61% 3 → 0

Geographic CRSes still take the AoS haversine / spherical-excess path (the scanner has no CRS context from the WKB blob alone).

Slice 9 replaces the recursive AoS douglasPeucker with an iterative SoA form (SimplifyDPFromXY / PointsView.SimplifyDP). Every Simplify entry point routes through the shared internal helper, so LineString.Simplify / Polygon.Simplify / MultiLineString.Simplify / MultiPolygon.Simplify / series.GeomSimplify all benefit transparently. Wiggly-sine polyline benchmark, Apple M3 Pro:

Path (Simplify) n AoS recursive SoA iterative delta wall allocs/op
SimplifyDP 64 1.80 μs / 17 allocs 302 ns / 3 allocs −83% 17 → 3
SimplifyDP 1K 126 μs / 259 allocs 25 μs / 4 allocs −80% 259 → 4
SimplifyDP 64K 81.5 ms / 17k / 122 MB 16.7 ms / 8 / 242 KB −80% 17k → 8
SimplifyDP 1M 3.85 s / 260k / 5.75 GB 0.75 s / 11 / 3.2 MB −81% 260k → 11

The 24k× allocation reduction on 1M-vertex input is the killer metric — the AoS recursion appends O(log n) intermediate []Point slices at every split, which the SoA form skips entirely with a retain-bitmap + argmax-of-cross-squared inner loop that defers the per-split sqrt to a single compare.

Slice 10 closes the remaining AoS gap in the PreparedGeometry build path. Before: Prepare(ParseWKB(data)) walked the WKB bytes to []Point, then walked []Point to []float64 slabs — two full O(n) passes and two allocations per ring. After: PolygonRingViewsFromWKB / PrepareFromWKB walk the WKB bytes once into slabs. On a 1M-vertex polygon (Apple M3 Pro):

Path ParseWKB + RingViews direct parse delta wall delta mem
PolygonRingViewsFromWKB 8.4 ms / 80 MB / 6a 3.7 ms / 16 MB / 3a −56% −80%
PrepareFromWKB (retains AoS G) 9.5 ms / 80 MB / 6a 7.4 ms / 80 MB / 6a −22% flat

The PrepareFromWKB gap is smaller because it still materializes the AoS Polygon for TestPrepared's non-fast-path fall-through contract — but that materialization is a cheap slab-to-Point copy, not a second WKB byte walk. Advanced callers wanting the full slab-only win can build PreparedGeometry directly with PolygonRingViewsFromWKB.

Slice 11 targets a different hot loop: planarMinDistance, the internal driver behind GeomDistance and WithinDistance. The pre-Slice-11 shape walked geometries via forEachVertex / forEachSegment closures and called math.Hypot per (vertex, segment) pair — for a Polygon×Polygon distance at n=1024 that's ~1M Hypot calls (each doing a sqrt) per row. The rewrite extracts each input's polylines into slab form once, then tracks running-min squared distance on flat coordinate arrays; the final sqrt runs exactly once at the end.

n Legacy AoS (closure + per-pair Hypot) Slab-form SoA delta wall
16 6.9 μs 1.7 μs −76%
64 106 μs 21 μs −80%
256 1.60 ms 272 μs −83%
1024 28.8 ms 5.3 ms −82%
h3-agg (bbox filter + groupby-h3 + aggregate, 2M rows)

A geospatial analytics workload: bbox-filter 2M (lat, lon, eid, dt, h3_res8) rows down to ~500K survivors, group by H3 cell (~39K groups), and aggregate count, n_unique(eid), min(dt), max(dt). The h3-agg bench harness lives at benchmarks/gobi/gobi_h3_agg_lazy_bench.go

Engine per-iter wall peak RSS
Polars 1.42 streaming 15 ms 341 MB
gobi (v0.3.9) 86 ms 501 MB
pandas 2.3 86 ms 792 MB
gobi (v0.3.8, pre-session) 152 ms ~500 MB

gobi (v0.3.9) matches pandas within measurement noise on wall time and uses 36% less peak RSS than pandas. Getting here took five stacked changes that each earn their entry in CHANGELOG.md:

  1. Filter fusion — Col(a) OP lit AND Col(b) OP lit AND … evaluates in a single row-loop with short-circuit on first false; no intermediate Boolean Series per comparison.
  2. Scalar-cmp fast path for Ne / Le / Ge — the pre-existing binOpNode.Eval shortcut only fired on Eq / Lt / Gt.
  3. Null-check hoisting in the fused filter — when every leaf's column has NullN() == 0 (typical for numeric parquet), the per-row × per-leaf IsNull() calls vanish.
  4. aggFast (fast-path GroupBy) covers AggNUnique + Timestamp Min/Max — previously "one bad agg disabled the whole call."
  5. Streaming nUniqueAcc type-specialization — LazyFrame groupby-with-nunique dispatches to native map[string]/[int64]/ [float64] instead of byte-encoded keys.

Polars is still 5.7× faster wall-time on this shape. That gap is architectural (cache-blocked kernels, arena allocator, compiled expression graph) rather than kernel-level and won't close without a rewrite of the same magnitude. gobi's differentiator here is matching pandas on throughput while using less memory than pandas — on this specific workload Polars beats gobi on both wall and RSS, because Polars streams the parquet decode straight into its grouper without our intermediate LazyFrame batches. The RSS story inverts at the 1BRC scale below.

Why the remaining compute-op gap will shrink

Sum / Add are already memory-bandwidth-bound. The remaining gap on Sum is SIMD reduction (Polars and numpy both use parallel-lane accumulators). Go 1.27 (August 2026) shipped the portable simd stdlib package with arm64 NEON + amd64 AVX2/AVX-512 backends under GOEXPERIMENT=simd. The compute/ subpackage plus series_ops_simd.go (v0.4.0) ship SIMD kernels behind //go:build goexperiment.simd && (arm64 || amd64) — comparisons, reductions (SumF64 / MinF64 / MaxF64), and elementwise arithmetic all landed on portable simd (previously amd64-only via simd/archsimd).

The remaining ~4.5× 1BRC gap vs Polars streaming breaks down (from CPU profile, post-v0.3.9):

  • ~22% arrow-go parquet decode (not our code; a custom pooled memory.Allocator wrapping arrow-go's decoder path is a scoped-out future win — see Future Work in CLAUDE.md)
  • ~12% Go runtime map[string]*aggGroup probe (the fundamental hash-by-string cost; possibly addressable with a specialized Robin-Hood / Swiss-table impl)
  • ~30% runtime scheduling / idle wait between workers
  • ~14% GC background work (down proportionally with v0.3.9's alloc churn cuts)
  • ~11% gobi's aggregate consume path (already tight per-batch typed-slice loops; the fast-path type-switch already dispatches once per chunk, not per row)
  • remainder: misc

Wall-time gains from here need either (a) SIMD reduction kernels wired into the streaming aggregate's Update methods (Go 1.27), (b) a custom decoder-buffer allocator to cut arrow-go's contribution, or (c) a specialized hash-map for the string-keyed groupby probe. None are on a specific timeline; the v0.3.9 wins have already put gobi solidly ahead of pandas (matched exactly on h3-agg, substantially ahead on memory) and closed the wall-time gap to Polars from 6× to 4.5×.

Development

  • go test -race ./... should pass before you push.
  • Please keep dependencies minimal — Arrow is the one big one on purpose.

License

MIT. See LICENSE.

Documentation

Overview

Package gobi is a geospatial dataframe library for Go, built on Apache Arrow.

A DataFrame is a set of named, equal-length Series backed by Arrow columns. Geometry columns are stored as Well-Known Binary (WKB) in Arrow Binary arrays and marked as geometries in the schema metadata (following the GeoParquet convention). This keeps gobi interoperable with other Arrow- aware tools without a custom extension type registry.

Reading data 

df, err := gobi.ReadCSV("cities.csv", nil)
df, err := gobi.ReadParquet("cities.parquet")

Selecting columns and rows 

col, _ := df.Column("name")
head, _ := df.Head(10)

Working with geometries 

geom, _ := df.Geometry("geometry", 0) // decode the WKB at row 0

See the subpackages for more:

  • geometry: 2D primitives, WKB/WKT, CRS, area/distance/hull
  • csvio: typed CSV read/write
  • parquetio: Parquet + GeoParquet read/write, LazyFrame scan
  • geojsonio: GeoJSON encoding/decoding for individual features
  • gpkgio: OGC GeoPackage read/write with RTree spatial index
  • kmlio: KML read/write with ExtendedData attributes
  • shpio: ESRI Shapefile read/write

Index

Constants

View Source 
const (
	GeoParquetVersion = "1.1.0"
	// GeoParquetMetadataKey is the file-level metadata key used by GeoParquet.
	GeoParquetMetadataKey = "geo"
)

GeoParquetVersion is the GeoParquet spec version this package emits.

View Source 
const (
	MetaGeometryType = "gobi:geometry_type"
	MetaGeometryCRS  = "gobi:crs_epsg"
)

Schema metadata keys used by gobi to tag geometry columns. This follows the GeoParquet convention closely enough that files written by gobi should be readable by other GeoParquet-aware tools for primitive geometry types.

View Source 
const SJoinMinParallelRows = 1024

SJoinMinParallelRows is the smallest left-frame size at which SJoin will spawn goroutines. Below this, the sequential path is used to avoid scheduling overhead swamping the useful work.

View Source 
const STRDefaultLeafSize = 5000

STRDefaultLeafSize is the target row-group size when SortBySTR is called with leafSize <= 0. Matches typical GeoParquet row-group defaults (5000).

Variables

View Source 
var (
	ErrColumnNotFound     = errors.New("gobi: column not found")
	ErrColumnLenMismatch  = errors.New("gobi: column length mismatch")
	ErrColumnTypeMismatch = errors.New("gobi: column type mismatch")
	ErrRowOutOfRange      = errors.New("gobi: row index out of range")
	ErrEmptyFrame         = errors.New("gobi: empty dataframe")
	ErrNotGeometry        = errors.New("gobi: column is not a geometry column")
	ErrExprTypeMismatch   = errors.New("gobi: expression type mismatch")
	ErrUnsupportedLiteral = errors.New("gobi: unsupported literal type")
)
View Source 
var ErrMaskNotBoolean = fmt.Errorf("gobi: filter mask must be a boolean series")

ErrMaskNotBoolean is returned when Filter receives a non-boolean Series.

View Source 
var ErrNotDateTime = fmt.Errorf("gobi: series is not a datetime column")

ErrNotDateTime is returned when a time-only op is attempted on a Series that is not a Timestamp / Date32 / Date64 column.

View Source 
var ErrNotMonotonic = fmt.Errorf("gobi: time column must be monotonically non-decreasing")

ErrNotMonotonic is returned by ops that require a time column sorted in non-decreasing order (currently: Frame.RollingBy).

View Source 
var ErrNotNumeric = fmt.Errorf("gobi: series is not numeric")

ErrNotNumeric is returned when arithmetic is attempted on a non-numeric Series. Only int64/int32/float64/float32 are currently supported.

View Source 
var ErrUnsupportedStructField = errors.New("gobi: unsupported struct field type")

ErrUnsupportedStructField is returned when FromStructs / ToStructs encounters a struct field type it can't map to an arrow column. The error is wrap-friendly: errors.Is(err, ErrUnsupportedStructField) is true for every type-mapping failure.

Functions

func Aligned added in v0.2.0 

func Aligned(meta *PartitionMetadata, columns []string) bool

Aligned reports whether meta's partition claim colocates rows by the given columns — the shape the optimizer needs to prove that .Over(K) / GroupBy(K) / hash-partition-aware Aggregate can skip the cross-worker shuffle.

First-cut rule (v1): exact match only. meta must be non-nil, meta.Columns must equal columns in order, and meta's HashFn is otherwise unconstrained (any hash — including "" for value partitioning — is fine for a single-source alignment check; two-source alignment for partition-wise join uses AlignedWith).

Refuses aliasing (via FDs), subset matching (Over("A") on a hash(A, B) partition), and column-order permutations. Users who need looser matches go through LazyFrame.WithPartitionAssertion and own correctness.

func AlignedWith added in v0.2.0 

func AlignedWith(l, r *PartitionMetadata) bool

AlignedWith reports whether two partition claims describe the same physical partitioning — same ordered Columns AND same HashFn. Consumed by the partition-wise Join rule (both sides must be partitioned on the join key by a byte-equal hash for a shuffle-free join to be safe).

Both metas must be non-nil with non-empty Columns; empty-Columns claims explicitly represent "no partitioning" and never align with anything.

func BboxColumnNames added in v0.3.4 

func BboxColumnNames(geomName string) (xmin, ymin, xmax, ymax string)

BboxColumnNames returns the four flat column names gobi uses to hold per-row bbox coordinates for a geometry column named geomName. Kept in one place so writer, reader, and predicate pushdown agree on the naming convention.

func CanPossiblyMatch 

func CanPossiblyMatch(pred Expr, stats Stats) bool

CanPossiblyMatch reports whether pred could be satisfied by any row in the range described by stats. Returns true when uncertain — false positives are safe (over-read); false negatives break correctness.

Supported predicate shapes:

  • AND / OR of shapes below
  • col == literal, col != literal
  • col <, <=, >, >= literal
  • literal on either side (auto-normalized)

Anything else (NOT, arithmetic, custom nodes) is treated as "possibly matches." Used by parquetio for row-group skipping; callable by any source package that can produce column stats.

func ExprToSQL 

func ExprToSQL(e Expr) (string, []any, bool)

ExprToSQL translates a gobi.Expr into a SQL fragment + a slice of bind arguments suitable for parameterized SQL queries (SQLite, PostgreSQL / PostGIS, DuckDB, etc.).

Returns ok=false when the expression contains a node the translator can't represent: custom nodes (Custom), aliased sub-expressions inside a predicate, or any binary op that doesn't map cleanly to SQL. Callers that get ok=false should fall back to evaluating the expression in the executor rather than pushing it into SQL.

The emitted SQL uses positional `?` placeholders for every literal, matching the SQLite driver contract that gpkgio uses. Drivers that expect `$1`/`$2` (pgx) or `:name` (Oracle) can rewrite the placeholders trivially since the args slice is 1:1 with `?` order.

Column names are double-quoted per SQL:2016 identifier rules; any embedded double-quote in a name is escaped by doubling. NULL comparisons (`x = NULL`) are rewritten to `IS NULL` / `IS NOT NULL` to match SQL's null-safe semantics.

Example: 

e := gobi.Col("price").Mul(gobi.Lit(1.08)).Gt(gobi.Lit(100.0))
sql, args, ok := gobi.ExprToSQL(e)
// sql  = `(("price" * ?) > ?)`
// args = []any{1.08, 100.0}
// ok   = true

func GeoParquetSchemaWithMetadata 

func GeoParquetSchemaWithMetadata(schema *arrow.Schema, meta *GeoParquetMetadata) (*arrow.Schema, error)

GeoParquetSchemaWithMetadata returns a copy of schema with the given GeoParquet metadata injected under the "geo" key at the file level.

func GeometryField 

func GeometryField(name string, epsg int32) arrow.Field

GeometryField returns a schema field tagged as a WKB geometry column. Pass epsg=0 to leave the CRS unset.

func HilbertSortWithCovering added in v0.3.5 

func HilbertSortWithCovering(f *Frame, geomCol string) (*Frame, *GeoParquetMetadata, error)

HilbertSortWithCovering is the fused single-pass equivalent of `f.SortByHilbert(geomCol)` followed by `WithBboxCoveringColumns`. Parses each row's WKB exactly once (vs twice for the two-step form) — the sort's centroid computation and the covering columns' per-row bbox computation share a single scan pass.

On the parquetio.WriteFile HilbertSort=true path this halves the dominant O(N·V) parse cost for large row counts.

Only the specified `geomCol` gets its covering columns emitted from the fused pass. If the frame carries additional geometry columns, their bbox covering is computed via the standard (second-pass) route — negligible relative to the primary column's cost in typical single-geom-column frames.

Semantics match the two-step form: null-last stable sort by centroid Hilbert index over bounds derived from the column's own centroids, covering columns declared under the geo metadata's `columns[geomCol].covering.bbox`.

func MarshalGeoParquetMetadata 

func MarshalGeoParquetMetadata(meta *GeoParquetMetadata) (string, error)

MarshalGeoParquetMetadata serializes meta to JSON.

func MaxParallelism 

func MaxParallelism() int

MaxParallelism returns the current package-level default worker count. Returns 0 if unset (in which case operations use GOMAXPROCS).

func ResolveFieldName added in v0.3.0 

func ResolveFieldName(sf reflect.StructField, format string) (name string, skip bool)

ResolveFieldName returns the column name to use for sf under the given io-format tag namespace, with a fallback chain to `gobi:`, `csv:` (legacy), and finally the field's Go name. skip is true when the tag value is exactly "-", meaning the field should be omitted from the output entirely.

io-package convenience wrappers (parquetio.WriteStructs, csvio, etc.) use this to keep tag resolution consistent with FromStructs / ToStructs. Pass "" as format to skip the per-io namespace lookup.

func SetMaxParallelism 

func SetMaxParallelism(n int)

SetMaxParallelism sets the default max worker count for every parallel gobi operation across the process. Pass 0 (the default) to defer to GOMAXPROCS. Individual calls can override with the Workers option.

Negative values are treated as zero.

func ToStructs added in v0.1.1 

func ToStructs[T any](f *Frame, opts ...StructOption) ([]T, error)

ToStructs converts a Frame back to a slice of Go structs, using the same struct-tag conventions as FromStructs.

Columns are matched to fields by name (resolved via the `csv:"..."` tag or field name). A struct field with no matching column stays at its zero value. A frame column with no matching struct field is ignored.

Null cells populate the zero value for non-pointer fields, or nil for pointer fields. Type mismatches between column and field return an error.

Geometry columns (Binary with the geometry metadata) can be written back to a string field tagged `geom:"true"` (emits WKT via geometry.WKT()) or a []byte field tagged `geom:"true"` (raw WKB pass-through).

func WithBboxCoveringColumns added in v0.3.4 

func WithBboxCoveringColumns(f *Frame) (*Frame, *GeoParquetMetadata, error)

WithBboxCoveringColumns returns a copy of f augmented with four Float64 columns per geometry column (xmin/ymin/xmax/ymax) plus a GeoParquetMetadata whose per-column Covering references them. This is the write-side half of predicate pushdown: readers use the covering columns' parquet-native min/max row-group statistics to skip whole row groups whose bboxes are disjoint from a spatial-predicate constant.

Empty/null geometry rows produce NaN bbox values, which parquet stats treat as "outside the min/max range" — safe (we err on keeping the row group) rather than incorrectly proving disjointness.

The returned frame retains its own reference to every column (original + new); callers own the Release, symmetric with NewFrame.

Types

type AggKind 

type AggKind uint8

AggKind identifies an aggregation operation. 

const (
	AggCount AggKind = iota
	AggSum
	AggMean
	AggMin
	AggMax
	// AggFirst / AggLast: first (or last) non-null value in the
	// group, in the group's row order. Output type matches the source
	// column (preserved via the Frame's schema).
	AggFirst
	AggLast
	// AggStd / AggVar: sample standard deviation / variance
	// (Bessel-corrected, n-1 denominator). Streaming path uses
	// Welford's online algorithm. Empty or single-row groups emit
	// null.
	AggStd
	AggVar
	// AggNUnique: count of distinct non-null values per group. Output
	// is Int64, always non-null (empty group counts as 0).
	AggNUnique
	// AggMedian: 50th-percentile value. Output is Float64 (Bessel-
	// style interpolation between the two middle values on even
	// group sizes). Empty groups emit null. Buffers all non-null
	// values per group, so memory is proportional to input size.
	AggMedian
	// AggMode: most frequent non-null value. Output preserves the
	// source column's arrow type. Ties broken by first-seen order.
	// Empty groups (or all-null groups) emit null.
	AggMode
)

func (AggKind) String 

func (k AggKind) String() string

type Aggregation 

type Aggregation struct {
	Column string
	Kind   AggKind
	Alias  string
	Fn     Aggregator
	// Filter is an optional predicate expression. When non-zero
	// (Filter.node != nil), only rows where Filter evaluates to
	// TRUE (non-null) participate in this aggregation. Different
	// aggregations in the same Agg call may carry different
	// filters — each is applied independently to that agg's row
	// set. Polars-parity `pl.col("x").sum().filter(cond)`.
	//
	// Filtered aggregations currently route through the eager
	// GroupBy.Agg path (the streaming aggregate rejects them via
	// allBuiltInAggs, forcing the materializing fallback). This
	// keeps the streaming hot path unchanged for filter-free
	// workloads.
	Filter Expr
}

Aggregation names a column to aggregate and how to aggregate it. The resulting column in the aggregated frame is named Alias (or, if empty, "<column>_<kind>" for built-in kinds, "<column>_<Fn.Name()>" for custom aggregators).

When Fn is non-nil, it takes precedence over Kind: the aggregation is user-defined and Fn is called once per group. Otherwise Kind selects a built-in aggregation.

type Aggregator 

type Aggregator interface {
	Aggregate(s Series, rows []int) (any, error)
	Merge(other Aggregator) error
	Type() arrow.DataType
	Name() string
}

Aggregator is a user-defined aggregation function called once per group during GroupBy.Agg. Implementations reduce the rows of a Series to a single scalar value.

Typical implementations: 

weighted mean, mode, percentile / quantile, log-sum-exp, first / last,
geospatial reductions (H3 cell of the centroid, dominant hex),
string aggregations (concat, longest common prefix).

Return nil from Aggregate to emit an Arrow null for the group. The declared Type must match the concrete Go type Aggregate returns: 

Float64 → float64        Uint64  → uint64      String    → string
Float32 → float32        Uint32  → uint32      Binary    → []byte
Int64   → int64          Boolean → bool        Timestamp → arrow.Timestamp
Int32   → int32

If the returned dynamic type doesn't match Type, Agg reports an error naming the offending Aggregation.

Aggregate is called sequentially per group; the same Aggregator instance is reused across groups. Implementations that need per-group scratch space should allocate it inside Aggregate rather than as receiver fields.

Merge combines the state of a peer Aggregator (one that has been fed a disjoint subset of the same group's rows via Aggregate) into the receiver. It exists so future parallel modes — rolling windows, pipeline-parallel executors, task parallelism — can partition a group's rows across workers and still assemble a correct result. gobi's v0.2 hash-partitioned executor never splits a group across workers, so Merge is not called on the current parallel path; it is still required so users don't have to guess whether they'll need it later. Implementations that don't carry state across Aggregate calls (e.g. stateless "return first non-null") can implement Merge as a no-op returning nil.

State lifecycle: the eager engine reuses a single Aggregator instance across every group, so implementations must reset internal state at the start of Aggregate — the receiver is fresh-per-group from the caller's perspective. Merge is called by parallel/window executors that deliberately hand the receiver a peer instance whose state should be combined; the framework never mixes state across groups.

func NewInt32SetAggregator added in v0.2.4 

func NewInt32SetAggregator() Aggregator

NewInt32SetAggregator returns an Aggregator that collects distinct non-null int32 values per group into a `List<Int32>` column. Input column must be `arrow.INT32`.

func NewInt64SetAggregator added in v0.2.4 

func NewInt64SetAggregator() Aggregator

NewInt64SetAggregator returns an Aggregator that collects distinct non-null int64 values per group into a `List<Int64>` column. Input column must be `arrow.INT64`.

func NewStringSetAggregator added in v0.2.4 

func NewStringSetAggregator() Aggregator

NewStringSetAggregator returns an Aggregator that collects distinct non-null string values per group into a `List<String>` column. Input column must be `arrow.STRING`.

func NewUint32SetAggregator added in v0.2.4 

func NewUint32SetAggregator() Aggregator

NewUint32SetAggregator returns an Aggregator that collects distinct non-null uint32 values per group into a `List<Uint32>` column. Input column must be `arrow.UINT32`.

func NewUint64SetAggregator added in v0.2.4 

func NewUint64SetAggregator() Aggregator

NewUint64SetAggregator returns an Aggregator that collects distinct non-null uint64 values per group into a `List<Uint64>` column. Input column must be `arrow.UINT64`. Common h3 cell-id shape.

type CalendarUnit 

type CalendarUnit uint8

CalendarUnit selects a calendar-aware truncation granularity for Series.TruncateToCalendar. 

const (
	CalendarWeek  CalendarUnit = iota // ISO week: truncate to the previous Monday 00:00
	CalendarMonth                     // truncate to the first day of the month at 00:00
	CalendarYear                      // truncate to Jan 1 at 00:00
)

type ExecOperator 

type ExecOperator interface {
	// Next returns the next batch. When no more batches are available
	// it returns (nil, io.EOF). Callers must Release the returned
	// batch when done.
	Next(ctx context.Context) (arrow.RecordBatch, error)

	// Schema is the operator's output schema. Stable across all
	// batches; safe to call before Next has been called.
	Schema() *arrow.Schema

	// Close releases resources held by the operator AND its upstream
	// operators. Idempotent — safe to call multiple times. Called by
	// Execute at the end of a plan, so hand-written test drivers
	// usually don't need to call it explicitly.
	Close() error
}

ExecOperator is a pull-based iterator over arrow record batches. Each call to Next produces the next batch of rows (or io.EOF when the operator has exhausted its input); the returned batch is owned by the caller and must be Released when no longer needed.

Operators can be composed into a tree: `filterExec` wraps another ExecOperator as its input and yields filtered batches; `scanFrameExec` is a leaf that produces batches from a source Frame. The plan compiler (Compile) turns a LogicalPlan into such a tree.

Layer 6 (vectorized execution) is the migration from the whole- Frame-per-node dispatch in lazy.go's collectPlan to batch-streaming through operators. Slice 1 lands the interface + streaming filter / project / limit / with-column / drop and a materializing fallback for blocking ops (Sort, Aggregate, Join, Tail); those still route to the eager engine internally but see one input Frame instead of a per-step re-materialization.

func Compile 

func Compile(p LogicalPlan) (ExecOperator, error)

Compile translates a LogicalPlan tree into a tree of ExecOperators ready for streaming execution via Execute.

The mapping is one-to-one for streaming operators (Filter → filterExec, Project → projectExec, and so on). Blocking operators (Sort, Aggregate, Join, Tail) compile to a materializeExec that pulls its upstream to a Frame and delegates the actual computation to the existing eager engine. This keeps Layer 6 slice 1 focused on the execution model itself; later slices can replace each materializeExec with a native streaming implementation.

Compile itself does no I/O and starts no goroutines. Scan operators that use a background producer (scanFileExec) start their goroutine on construction, so callers should always follow Compile with Execute or the operator's Close to avoid leaks.

type Expr 

type Expr struct {
	// contains filtered or unexported fields
}

Expr is an expression tree — a value that describes a computation without performing it. Build expressions with the package-level constructors (Col, Lit, Custom) and combine them with the fluent methods on Expr (Add, Mul, Gt, And, etc.).

Expressions evaluate lazily: nothing runs until Frame.FilterExpr or Frame.WithColumnExpr consumes them. Because the tree is data, it can be inspected, printed, and (in a future release) rewritten by an optimizer before evaluation.

Example: 

// (price * 1.08) > 100
e := gobi.Col("price").Mul(gobi.Lit(1.08)).Gt(gobi.Lit(100.0))
filtered, err := df.FilterExpr(e)

func Coalesce added in v0.2.5 

func Coalesce(exprs ...Expr) Expr

Coalesce returns an expression that evaluates to the first non-null value among exprs, per row. Matches SQL COALESCE / polars `coalesce()`. All operands must produce the same arrow type — no automatic widening; cast explicitly or match Lit types.

If every operand is null at a given row, the output at that row is null too. Zero operands or a single operand each error at construction time (zero) or degenerate to a passthrough (single).

Common use — fill nulls on either side of a Full outer join before running ListUnion: 

segSafe  := gobi.Coalesce(gobi.Col("seg"),  gobi.LitEmptyList(arrow.BinaryTypes.String))
pingSafe := gobi.Coalesce(gobi.Col("ping"), gobi.LitEmptyList(arrow.BinaryTypes.String))
merged   := segSafe.ListUnion(pingSafe)

Not short-circuit: every operand is evaluated in full. If short- circuit matters (e.g. an expensive right-hand operand you only want on rows where the left is null), split into filtered pipelines.

func Col 

func Col(name string) Expr

Col returns an expression that references the column named name in the input Frame.

func Custom 

func Custom(node ExprNode) Expr

Custom wraps a user-defined ExprNode into an Expr. This is the entry point for extending gobi with your own expression types.

func HaversineExpr added in v0.2.10 

func HaversineExpr(from, to PointExpr, u geometry.Unit) Expr

HaversineExpr returns an expression that computes the great-circle distance between two lat/lon point-column pairs, in the requested unit. Composes with Shift(1).Over(K) for prev-row coordinate lookups, so per-segment ground-track distance is expressible entirely in LazyFrame land: 

speedKMH := HaversineExpr(
    PointExpr{Lat: Col("lat"), Lon: Col("lon")},
    PointExpr{
        Lat: Col("lat").Shift(1).Over("eid"),
        Lon: Col("lon").Shift(1).Over("eid"),
    },
    geometry.UnitKilometers,
).Div(Col("delta_hours"))

Coordinate components are named-field to eliminate the "did I pass lat first or lon first?" bug class. All four underlying Expr operands must be Float64 columns; types are checked at Compile. Nulls in any of the four inputs propagate to a null output row.

Perf shape: zero-copy `Float64Values()` fast path when the four operand columns are single-chunk (the common shape post-scan or post-materialize); multi-chunk fallback via `Float64s()`. Inner loop uses hoisted scale constant + inline trig — no per-row function-call boundary into the scalar `geometry.Haversine`.

Breaking change in v0.2.16: previously took four positional Expr args (lat1, lon1, lat2, lon2). Migration: 

// Before
HaversineExpr(Col("lat"), Col("lon"), Col("lat2"), Col("lon2"), u)
// After
HaversineExpr(
    PointExpr{Lat: Col("lat"),  Lon: Col("lon")},
    PointExpr{Lat: Col("lat2"), Lon: Col("lon2")},
    u,
)

func If added in v0.2.4 

func If(cond, a, b Expr) Expr

If returns an expression that evaluates to a where cond is true, to b where cond is false, and to null where cond is null (SQL CASE-WHEN semantics). All three arguments are evaluated in full — this is not short-circuit. If short-circuit evaluation matters (e.g. one branch would error on rows the other branch handles), split into two filtered pipelines.

cond must produce a Boolean column. a and b must have the same arrow output type — no automatic numeric widening in v1. To combine mixed numeric types write `gobi.If(cond, gobi.Lit(1.0), gobi.Col("x"))` where "x" is Float64, or add a Cast on one side.

Example — mean-fill a nullable column: 

gobi.If(gobi.Col("x").IsNull(), gobi.Col("x_mean"), gobi.Col("x"))

Nested else-if via chained If is supported: 

gobi.If(
    gobi.Col("region").Eq(gobi.Lit("EU")), gobi.Lit(1.08),
    gobi.If(
        gobi.Col("region").Eq(gobi.Lit("NA")), gobi.Lit(1.10),
        gobi.Lit(1.0), // default
    ),
)

func Lit 

func Lit(v any) Expr

Lit returns an expression that evaluates to a constant value. The value's arrow type is inferred from its Go type; the following are supported: 

bool                → Boolean
int, int32, int64   → Int64
float32, float64    → Float64
string              → String
time.Time           → Timestamp[ns] (stored as UnixNano; matches
                      the type from_structs.go produces for
                      time.Time struct fields)

Values implementing geometry.Geometry (Point, LineString, Polygon, etc.) route to LitGeom, so `Lit(aoi).GeomIntersects(...)` composes with the spatial-predicate builders without a separate constructor.

Other Go types return an Expr whose Eval reports a type-inference error. For a typed-null literal (e.g. a null-of-type-String column), use LitNull.

func LitEmptyList added in v0.2.5 

func LitEmptyList(elemType arrow.DataType) Expr

LitEmptyList returns an expression that broadcasts a non-null empty list of the given element type to every input row. Companion to LitNull for the "coalesce null-list to empty-list" pattern that makes ListUnion usable across a full outer join: 

segSafe  := gobi.Coalesce(gobi.Col("seg_providers"),  gobi.LitEmptyList(arrow.BinaryTypes.String))
pingSafe := gobi.Coalesce(gobi.Col("ping_providers"), gobi.LitEmptyList(arrow.BinaryTypes.String))
merged   := segSafe.ListUnion(pingSafe)

LitNull(ListOf(t)) produces all-NULL lists per row; LitEmptyList(t) produces all-non-null empty lists per row. Different — the latter gets ListUnion past its null-propagation gate.

func LitGeom added in v0.3.4 

func LitGeom(g geometry.Geometry) Expr

LitGeom returns an expression that broadcasts a constant geometry value to every input row. Companion to Lit for the spatial-predicate builders (Expr.GeomIntersects and friends): the executor detects a literalGeomNode on the right side of a predicate and takes the constant-right fast path, which is the same code path Series-level GeomIntersects(g) already uses.

The plain `Lit(v)` constructor also accepts any value implementing geometry.Geometry and routes it here — so `Lit(aoi)` and `LitGeom(aoi)` are interchangeable in a Filter expression. Callers building predicates over a geometry column typically write the former for symmetry with `Lit(1.0)` etc.

The CRS of the constant is inherited from the left-hand geometry column at evaluation time, matching Series.GeomIntersects's behavior — the constant is re-tagged with the column's CRS so predicates don't cross-reference CRSes silently.

func LitNull added in v0.2.4 

func LitNull(dtype arrow.DataType) Expr

LitNull returns an expression that broadcasts a null value of the given arrow type to every input row. Useful for filling a column with typed nulls (e.g. when unioning two branches where one side legitimately lacks a value): 

pings.WithColumn("provider", gobi.LitNull(arrow.BinaryTypes.String))

The type must be one that builderForType supports (every primitive gobi handles today plus List / Struct). Passing an unsupported type surfaces the error at Eval, not at construction.

func SplitConjuncts 

func SplitConjuncts(e Expr) []Expr

SplitConjuncts breaks an expression at top-level AND boundaries so callers can push the pieces that translate to SQL individually and keep the rest for post-scan filtering.

Example: given `(a > 5 AND custom_fn(b) AND c = "x")`, SplitConjuncts yields `[a > 5, custom_fn(b), c = "x"]`. The custom_fn part won't translate; the other two will, and the caller can push `a > 5 AND c = "x"` into SQL while leaving `custom_fn(b)` in the executor.

Non-AND expressions come back as a single-element slice — a top-level OR isn't safe to split (both sides must match), so we don't try.

func (Expr) Add 

func (e Expr) Add(o Expr) Expr

Add returns e + o.

func (Expr) AddDuration added in v0.3.6 

func (e Expr) AddDuration(d time.Duration) Expr

AddDuration returns a Timestamp Expr with `d` added to each value. Negative durations subtract.

func (Expr) Alias 

func (e Expr) Alias(name string) Expr

Alias renames the output column produced by e. Only used by Frame.WithColumnExpr and future SelectExpr — has no effect inside FilterExpr.

func (Expr) And 

func (e Expr) And(o Expr) Expr

And returns e AND o. Both e and o must produce Boolean values.

func (Expr) BitAnd added in v0.2.10 

func (e Expr) BitAnd(o Expr) Expr

BitAnd returns the bitwise AND of two integer columns (or an integer column and an integer literal). Both operands must be integer-typed (Int32/Int64/Uint32/Uint64); errors at Type-check time otherwise. Common shape: unpacking a packed-flags Int64 column into per-bit indicator columns, e.g. 

Col("flags").BitAnd(Lit(int64(1 << 3))).Ne(Lit(int64(0)))

func (Expr) BitOr added in v0.2.10 

func (e Expr) BitOr(o Expr) Expr

BitOr returns the bitwise OR of two integer columns / literals. Same type rules as BitAnd.

func (Expr) BitXor added in v0.2.10 

func (e Expr) BitXor(o Expr) Expr

BitXor returns the bitwise XOR of two integer columns / literals. Same type rules as BitAnd.

func (Expr) Cast added in v0.2.9 

func (e Expr) Cast(target arrow.DataType) Expr

Cast returns an expression that converts the inner column to the given arrow data type. Numeric-to-numeric conversions are supported today; string / boolean / date-time conversions are follow-ups.

Supported source→target combinations (v1, numeric-to-numeric only):

  • Float64 ← {Float32, Int64, Int32, Uint64, Uint32}
  • Int64 ← {Float64, Float32, Int32, Uint32}
  • Float32 ← {Float64, Int64, Int32}
  • Int32 ← {Int64, Float64, Float32}
  • Uint64 ← {Int64, Uint32}
  • Uint32 ← {Uint64, Int64}

Same-type is a no-op (returns the input). Unsupported combinations error at Eval with a clear message.

Numeric widening carries values exactly. Narrowing (Int64→Int32, Float64→Int64, etc.) truncates without overflow-checking — mirrors Go's numeric conversion semantics. Nulls propagate: a null row in the source produces a null row in the output.

Primary motivating use case is unlocking numeric widening in If / Coalesce, which require exact type match otherwise: 

gobi.If(cond, gobi.Col("int_col").Cast(arrow.PrimitiveTypes.Float64),
              gobi.Lit(1.5))

Implementation

Since v0.3.9 the numeric-cast kernel is arrow-go's `arrow.compute.CastArray`, which on arm64 dispatches to hand-written NEON SIMD (`internal/kernels/cast_numeric_neon_arm64.s`) and on amd64 to hand-written AVX2/SSE4 SIMD. Measured 13.9× faster than the previous hand-rolled scalar builder loops on a 10M-row Float64→Int64 cast (arm64, Apple M3 Pro). See the `compute` package docstring for the arrow-go vs gobi kernel overlap map and the benchmark that produced the number.

gobi wraps the arrow-go call to preserve the previous surface: multi-chunk columns cast per chunk and reassemble; overflow modes match Go's built-in conversion (silent wrap for int narrowing, silent truncate for float→int); the whitelist of accepted target types stays limited to numeric so unrelated arrow-go cast capabilities (String↔Int, Date↔Timestamp) don't silently activate — those are opt-in follow-ups.

func (Expr) Count added in v0.2.0 

func (e Expr) Count() Expr

Count returns an expression that evaluates to the count of e's non-null values, broadcast to every input row. Output is Int64.

func (Expr) DateFormat added in v0.3.6 

func (e Expr) DateFormat(layout string) Expr

DateFormat returns a String Expr formatting each value with the given Go time layout (RFC3339 default when layout is empty).

func (Expr) DateTruncate added in v0.3.6 

func (e Expr) DateTruncate(unit string) Expr

DateTruncate returns a Timestamp Expr with each value truncated down to the start of `unit`. Supported: "year", "month", "day", "hour", "minute", "second". Truncation is calendar-aware for year/month/day and wall-clock aligned for hour/minute/second.

func (Expr) Day added in v0.3.6 

func (e Expr) Day() Expr

Day returns the day-of-month (1..31) as Int64.

func (Expr) DayOfYear added in v0.3.6 

func (e Expr) DayOfYear() Expr

DayOfYear returns the day-of-year (1..366) as Int64.

func (Expr) Div 

func (e Expr) Div(o Expr) Expr

Div returns e / o.

func (Expr) Eq 

func (e Expr) Eq(o Expr) Expr

Eq returns e == o (Boolean result).

func (Expr) Ge 

func (e Expr) Ge(o Expr) Expr

Ge returns e >= o (Boolean result).

func (Expr) GeomContains added in v0.3.4 

func (e Expr) GeomContains(other Expr) Expr

GeomContains returns e ⊇ other (every point of other lies in e) as a Boolean expression. Same composition rules as GeomIntersects. Matches GeoPandas' GeoSeries.contains and Series.GeomContains.

func (Expr) GeomCrosses added in v0.3.4 

func (e Expr) GeomCrosses(other Expr) Expr

GeomCrosses returns "e crosses other" (share some but not all interior points, with the intersection lower-dimensional than the higher operand) as a Boolean expression. Typical shape: LineString × Polygon. Matches GeoPandas' GeoSeries.crosses. Same composition rules as GeomIntersects.

func (Expr) GeomDWithin added in v0.3.5 

func (e Expr) GeomDWithin(other Expr, distance float64) Expr

GeomDWithin returns "e is within distance of other" as a Boolean expression. distance is in the column's coordinate units — meters for projected CRSes, degrees for lon/lat. For lon/lat data, apply GeomToCRS first so the distance comparison stays meaningful.

distance = 0 degenerates to GeomIntersects. Negative or NaN distance yields all-false. Nulls propagate.

The killer spatial-join operator: "AOIs within 5km of a road", "polygons within 100m of the shoreline", "points within 50m of a track". Composes with Filter like any other Expr predicate; constant-right shape gets the bbox-expanded row-group pushdown (see canMatchGeomDWithin).

func (Expr) GeomDisjoint added in v0.3.4 

func (e Expr) GeomDisjoint(other Expr) Expr

GeomDisjoint returns e ∩ other == ∅ as a Boolean expression — exactly the negation of GeomIntersects (nulls still propagate; a null row is null, NOT true). Same composition rules as GeomIntersects.

func (Expr) GeomIntersects added in v0.3.4 

func (e Expr) GeomIntersects(other Expr) Expr

GeomIntersects returns e ∩ other ≠ ∅ as a Boolean expression. Compose it into LazyFrame.Filter alongside scalar predicates: 

lf.Filter(
    gobi.Col("level").Eq(gobi.Lit(1.0)).
        And(gobi.Col("geometry").GeomIntersects(gobi.Lit(aoi))),
).Collect()

e must reference a geometry column. `other` may be a LitGeom constant (fast path — bbox short-circuit per row, matches Series.GeomIntersects semantics) or another geometry-column expression (pair-wise per-row test — new capability not exposed on Series today).

Null propagates from either side: if the left row is null, or the right row is null (column-right case), the output row is null. Matches Polars' null-propagating comparison contract.

func (Expr) GeomOverlaps added in v0.3.4 

func (e Expr) GeomOverlaps(other Expr) Expr

GeomOverlaps returns "e and other share interior points, neither contains the other, and both are of the same dimension" as a Boolean expression. Matches GeoPandas' GeoSeries.overlaps. Same composition rules as GeomIntersects.

func (Expr) GeomTouches added in v0.3.4 

func (e Expr) GeomTouches(other Expr) Expr

GeomTouches returns "e and other share boundary points but no interior points" as a Boolean expression. Matches GeoPandas' GeoSeries.touches. Same composition rules as GeomIntersects.

func (Expr) GeomWithin added in v0.3.4 

func (e Expr) GeomWithin(other Expr) Expr

GeomWithin returns e ⊆ other (every point of e lies in other) as a Boolean expression. Equivalent to `other.GeomContains(e)`; kept as its own method for symmetry with GeoPandas' .within(). Same composition rules as GeomIntersects.

func (Expr) Gt 

func (e Expr) Gt(o Expr) Expr

Gt returns e > o (Boolean result).

func (Expr) Hour added in v0.3.6 

func (e Expr) Hour() Expr

Hour returns the hour-of-day (0..23) as Int64.

func (Expr) IsNotNull added in v0.2.3 

func (e Expr) IsNotNull() Expr

IsNotNull returns a Boolean expression that is true where e evaluates to a non-null value. Semantic complement of IsNull.

func (Expr) IsNull added in v0.2.3 

func (e Expr) IsNull() Expr

IsNull returns a Boolean expression that is true where e evaluates to null, false everywhere else. The returned column itself is never null.

Composes with Filter / WithColumn / Select and with the boolean combinators (And, Or, Not): 

// Rows where price is missing.
f.FilterExpr(gobi.Col("price").IsNull())

// Rows where price is set AND qty is missing.
f.FilterExpr(gobi.Col("price").IsNotNull().And(gobi.Col("qty").IsNull()))

func (Expr) Le 

func (e Expr) Le(o Expr) Expr

Le returns e <= o (Boolean result).

func (Expr) ListContains added in v0.2.0 

func (e Expr) ListContains(elem any) Expr

ListContains returns an expression that reports whether each row's list contains elem. Null lists → null; otherwise Boolean. elem is a Go scalar (bool, int/int64, float32/float64, string) matching the list's element type.

func (Expr) ListFirst added in v0.2.0 

func (e Expr) ListFirst() Expr

ListFirst returns the first element of each row's list. Alias for ListGet(0) — kept as a distinct constructor to match polars/pandas naming conventions.

func (Expr) ListGet added in v0.2.0 

func (e Expr) ListGet(i int) Expr

ListGet returns an expression that reads the element at index i from each row's list. Negative i counts from the end (-1 = last). Out-of-range indices, null lists, and null elements all evaluate to null. The result type is the list's element type.

func (Expr) ListLast added in v0.2.0 

func (e Expr) ListLast() Expr

ListLast returns the last element of each row's list. Alias for ListGet(-1).

func (Expr) ListLen added in v0.2.0 

func (e Expr) ListLen() Expr

ListLen returns an expression that evaluates each row's list length. e must produce a List column. Null lists → null; empty lists → 0.

func (Expr) ListMax added in v0.2.0 

func (e Expr) ListMax() Expr

ListMax returns the maximum non-null element of each row's list. Output widens like ListMin.

func (Expr) ListMean added in v0.2.0 

func (e Expr) ListMean() Expr

ListMean returns the arithmetic mean of each row's non-null list elements. Always Float64. Null / empty lists produce null.

func (Expr) ListMin added in v0.2.0 

func (e Expr) ListMin() Expr

ListMin returns the minimum non-null element of each row's list. Output widens to Int64 / Uint64 / Float64 per element category. Null / empty lists produce null.

func (Expr) ListSlice added in v0.2.0 

func (e Expr) ListSlice(start, stop int) Expr

ListSlice returns an expression that slices each row's list to [start, stop). Python-style semantics: negative indices count from the end; out-of-range endpoints clamp to the list bounds. The result is still a List column.

func (Expr) ListSum added in v0.2.0 

func (e Expr) ListSum() Expr

ListSum returns an expression that sums each row's list elements. Null lists and empty lists produce null. Null elements are skipped (polars-parity). Signed integer element types widen to Int64; unsigned to Uint64; floats to Float64.

func (Expr) ListUnion added in v0.2.4 

func (e Expr) ListUnion(other Expr) Expr

ListUnion returns an expression that unions each row's list with the corresponding row of other's list, deduplicating elements. Both e and other must be List<T> columns with the same element type.

Order semantics: first-seen order preserved. Elements from e come first (in their order), then any new elements from other in their order.

Null semantics: null elements inside a list are skipped. A null list itself (either side) makes the whole row null — polars / Spark array_union parity. Users who want "treat null as empty set" should wrap with a coalesce-to-empty step upstream.

Polars parity: pl.col("a").list.set_union(pl.col("b")). Spark parity: array_union(a, b).

func (Expr) Lt 

func (e Expr) Lt(o Expr) Expr

Lt returns e < o (Boolean result).

func (Expr) MaxAgg added in v0.2.0 

func (e Expr) MaxAgg() Expr

MaxAgg returns an expression that evaluates to the maximum of e's non-null values, broadcast to every input row.

func (Expr) Mean added in v0.2.0 

func (e Expr) Mean() Expr

Mean returns an expression that evaluates to the arithmetic mean of e's non-null values, broadcast to every input row.

func (Expr) Median added in v0.2.9 

func (e Expr) Median() Expr

Median returns an expression that evaluates to the sample median of e's non-null numeric values, broadcast to every input row. Output is Float64. Even-sized groups interpolate between the two middle values.

func (Expr) MinAgg added in v0.2.0 

func (e Expr) MinAgg() Expr

MinAgg returns an expression that evaluates to the minimum of e's non-null values, broadcast to every input row. Named MinAgg to avoid clashing with Series.Min-shaped methods that already exist.

func (Expr) Minute added in v0.3.6 

func (e Expr) Minute() Expr

Minute returns the minute-of-hour (0..59) as Int64.

func (Expr) Mode added in v0.2.9 

func (e Expr) Mode() Expr

Mode returns an expression that evaluates to the most-frequent non-null value of e, broadcast to every input row. Ties are broken by first-seen order. Output type matches the source column.

func (Expr) Month added in v0.3.6 

func (e Expr) Month() Expr

Month returns the calendar month (1..12) as Int64.

func (Expr) Mul 

func (e Expr) Mul(o Expr) Expr

Mul returns e * o.

func (Expr) Nanosecond added in v0.3.6 

func (e Expr) Nanosecond() Expr

Nanosecond returns the sub-second nanosecond component (0..999_999_999) as Int64.

func (Expr) Ne 

func (e Expr) Ne(o Expr) Expr

Ne returns e != o (Boolean result).

func (Expr) Node 

func (e Expr) Node() ExprNode

Node returns the underlying ExprNode. Rarely needed unless you're implementing your own tree walker or optimizer.

func (Expr) Not 

func (e Expr) Not() Expr

Not returns the logical negation of e. e must produce Boolean values.

func (Expr) Or 

func (e Expr) Or(o Expr) Expr

Or returns e OR o. Both e and o must produce Boolean values.

func (Expr) Over added in v0.2.0 

func (e Expr) Over(partitionCols ...string) Expr

Over wraps an expression with partition keys.

Two shapes are supported depending on the inner:

  1. **Scalar aggregate inner** (Sum/Mean/Min/Max/Count on a column): the aggregate is computed per unique partition-key combination and broadcast to every row in that group. This is the reduce- and-scatter shape gobi has had since v0.2.0.

  2. **Shape-preserving inner** (Shift, arithmetic chains like `Col("v").Add(Lit(1.0))`, and other row-order-preserving ExprNodes): the inner is evaluated separately on each partition's rows and the per-row output is scattered back to the original row positions. Input row order is preserved within each partition (polars-parity default).

If the shape-preserving transform is row-order sensitive (e.g. Shift) and you need a specific within-partition order, use OverOrdered — this variant does not sort within partitions.

func (Expr) OverOrdered added in v0.2.2 

func (e Expr) OverOrdered(partitionCols []string, orderBy ...SortKey) Expr

OverOrdered is Over with an explicit within-partition sort order. Rows in each partition are sorted by orderBy (multi-key, stable, nulls-last, per-key Descending) before the shape-preserving inner runs on them. Output row order matches input row order — the sort only affects what "previous row" / "next row" mean inside each partition.

For scalar aggregate inners the orderBy is ignored (Sum, Mean, Min, Max, Count are order-invariant).

Example: previous v within each K, ordered by t. 

Col("v").Shift(1).OverOrdered([]string{"K"}, gobi.SortKey{Column: "t"})

func (Expr) Second added in v0.3.6 

func (e Expr) Second() Expr

Second returns the second-of-minute (0..59) as Int64.

func (Expr) Shift added in v0.2.1 

func (e Expr) Shift(n int) Expr

Shift returns an expression that shifts the values of e by n positions. Positive n shifts forward (the first n rows become null, each output row i takes the source value from row i-n). Negative n shifts backward. Matches Series.Shift semantics.

Composes naturally with WithColumn / Select for lag / lead computations: 

// prior period's price
lf.WithColumn("prev_price", Col("price").Shift(1))

// running delta
lf.WithColumn("delta", Col("price").Sub(Col("price").Shift(1)))

Per-group shifts via .Over(K) are on the roadmap; today Over requires a scalar aggregate as its immediate inner, so shift-then-partition isn't wired up (see CLAUDE.md's Track 3 note on ordered-partition windows).

func (Expr) StrConcat added in v0.3.6 

func (e Expr) StrConcat(suffix string) Expr

StrConcat returns each value followed by `suffix`.

func (Expr) StrContains added in v0.3.6 

func (e Expr) StrContains(substr string) Expr

StrContains returns a Boolean Expr: true when the row's value contains `substr`.

func (Expr) StrEndsWith added in v0.3.6 

func (e Expr) StrEndsWith(suffix string) Expr

StrEndsWith returns a Boolean Expr: true when the row's value ends with `suffix`.

func (Expr) StrLen added in v0.3.6 

func (e Expr) StrLen() Expr

StrLen returns an Int64 Expr with each value's Unicode-codepoint count.

func (Expr) StrLower added in v0.3.6 

func (e Expr) StrLower() Expr

StrLower returns e.StrLower() as an Expr.

func (Expr) StrRegexMatch added in v0.3.6 

func (e Expr) StrRegexMatch(pattern string) Expr

StrRegexMatch returns a Boolean Expr: true when the row's value matches `pattern` (RE2 semantics, MatchString shape).

Pattern is compiled ONCE at Expr-build time and the compiled *regexp.Regexp is cached on the executor node — a Filter chain evaluated across N batches pays the compile cost once, not N times. Invalid patterns don't panic; the compile error is stored on the node and surfaced at Eval (same pattern as Lit's unsupported-type handling).

func (Expr) StrRegexReplace added in v0.3.6 

func (e Expr) StrRegexReplace(pattern, replacement string) Expr

StrRegexReplace returns e with every regex match of `pattern` replaced by `replacement` (ReplaceAllString semantics). Same compile-once caching as StrRegexMatch.

func (Expr) StrReplace added in v0.3.6 

func (e Expr) StrReplace(find, replacement string) Expr

StrReplace returns e with every literal occurrence of `find` replaced by `replacement`.

func (Expr) StrSlice added in v0.3.6 

func (e Expr) StrSlice(start, end int) Expr

StrSlice returns each value sliced by codepoint index [start, end). See Series.StrSlice for the negative-index / end-zero convention.

func (Expr) StrStartsWith added in v0.3.6 

func (e Expr) StrStartsWith(prefix string) Expr

StrStartsWith returns a Boolean Expr: true when the row's value begins with `prefix`.

func (Expr) StrTrim added in v0.3.6 

func (e Expr) StrTrim() Expr

StrTrim returns e.StrTrim() as an Expr (strip whitespace both ends).

func (Expr) StrTrimLeft added in v0.3.6 

func (e Expr) StrTrimLeft(cutset string) Expr

StrTrimLeft returns e with each value's leading `cutset` codepoints removed.

func (Expr) StrTrimRight added in v0.3.6 

func (e Expr) StrTrimRight(cutset string) Expr

StrTrimRight returns e with each value's trailing `cutset` codepoints removed.

func (Expr) StrUpper added in v0.3.6 

func (e Expr) StrUpper() Expr

StrUpper returns e.StrUpper() as an Expr.

func (Expr) String 

func (e Expr) String() string

String returns a human-readable representation of the expression.

func (Expr) Sub 

func (e Expr) Sub(o Expr) Expr

Sub returns e - o.

func (Expr) SubDuration added in v0.3.6 

func (e Expr) SubDuration(d time.Duration) Expr

SubDuration returns a Timestamp Expr with `d` subtracted from each value. Equivalent to AddDuration(-d).

func (Expr) Sum added in v0.2.0 

func (e Expr) Sum() Expr

Sum returns an expression that evaluates to the sum of e's non-null values, broadcast to every input row. Combine with Over to get per-partition sums.

func (Expr) UnixNano added in v0.2.10 

func (e Expr) UnixNano() Expr

UnixNano returns an expression that converts a Timestamp column to Int64 nanoseconds since the Unix epoch, normalizing across the source column's TimeUnit (Second / Millisecond / Microsecond / Nanosecond). Distinct from `Cast(Int64)` on a Timestamp — that returns the raw underlying value in the source unit; UnixNano always scales to nanoseconds.

Combines with arithmetic to derive time-delta expressions inline without leaving LazyFrame land: 

// ts_hours = float64 hours-since-epoch
Col("ts").UnixNano().Cast(arrow.PrimitiveTypes.Float64).
    Div(Lit(float64(time.Hour)))

Nulls propagate. Errors at Eval time if the source isn't a Timestamp column.

func (Expr) Weekday added in v0.3.6 

func (e Expr) Weekday() Expr

Weekday returns the day-of-week as Int64 (0=Sunday..6=Saturday).

func (Expr) Year added in v0.3.6 

func (e Expr) Year() Expr

Year returns the calendar year of each row's timestamp as Int64.

type ExprNode 

type ExprNode interface {
	// Eval evaluates the expression against input, returning a Series
	// whose values are the result. The returned Series should have
	// length equal to input.NumRows(). Scalar broadcasts are handled
	// by the caller for built-in nodes; custom nodes may either
	// broadcast internally or emit a length-1 Series.
	Eval(input *Frame) (Series, error)

	// Type returns the arrow data type of the result given the input
	// schema. Called by type-inference passes before evaluation.
	Type(schema *arrow.Schema) (arrow.DataType, error)

	// Children returns the sub-expressions of this node in
	// deterministic order (empty for leaves). Used by tree walkers.
	Children() []Expr

	// String returns a human-readable representation of the node.
	String() string
}

ExprNode is the interface every expression node implements. Users extending gobi with custom expression types (H3 encoding, hashes, ML inference, etc.) implement this and wrap the result with Custom.

type Frame 

type Frame struct {
	// contains filtered or unexported fields
}

Frame is a columnar dataset: an Arrow schema plus a set of named Series.

The API mirrors GeoPandas / Polars in shape: Shape, Head, Tail, Row, Column, and geometry helpers.

func Concat 

func Concat(frames ...*Frame) (*Frame, error)

Concat is the package-level counterpart to Frame.Concat: stacks rows from every frame in-order, no dedupe. Useful when the caller holds a []*Frame and doesn't want the `frames[0].Concat(frames[1:]...)` dance the method form requires.

Errors when frames is empty (there's nothing to return a schema from) or when any two frames have incompatible schemas — same rules as Frame.Concat.

func Execute 

func Execute(ctx context.Context, op ExecOperator) (*Frame, error)

Execute drives op to EOF, gathers every batch, and assembles them into a single-chunk *Frame. Closes op unconditionally.

This is the terminal entry point for the streaming executor: LazyFrame.Collect calls Compile + Execute. Callers that want to consume batches directly (streaming ETL) should walk op.Next in a loop themselves — but that's a lower-level API not yet exposed.

func FromStructs added in v0.1.1 

func FromStructs[T any](rows []T, opts ...StructOption) (*Frame, error)

FromStructs builds a Frame from a slice of Go structs. Column order follows struct-field declaration order; unexported fields are ignored.

Struct-tag conventions: 

<format>:"col"      override the column name for the io format
                    passed via StructTagFormat (e.g. parquet:"col");
                    highest-priority name source when opts include
                    StructTagFormat("<format>").
gobi:"col"          universal name override, applied for every
                    io format when no format-specific tag matches.
csv:"col"           legacy fallback name override.
<tag>:"-"           skip this field entirely.
geom:"true"         geometry column — see below for value handling
time:"2006-01-02"   parse string field as time.Time using the layout;
                    ignored for time.Time-typed fields

Geometry handling. A field tagged `geom:"true"` becomes a Binary arrow column tagged with GeometryField metadata:

  • string field: value is parsed as WKT via geometry.ParseWKT and emitted as WKB bytes.
  • []byte field: value is emitted as-is (assumed already WKB).
  • other field types: error.

Nulls. Pointer-typed fields (*string, *int64, ...) emit null when the pointer is nil. Non-pointer fields never emit null — their zero value goes in.

Supported non-tagged field types: 

string, bool
int, int8, int16, int32, int64
uint, uint8, uint16, uint32, uint64
float32, float64
[]byte           (arrow Binary)
time.Time        (arrow Timestamp[ns])
*T of any above  (nullable)

func NewFrame 

func NewFrame(schema *arrow.Schema, cols []arrow.Column) (*Frame, error)

NewFrame builds a Frame from a schema and a set of arrow columns. The number of columns must match the schema.

func NewFrameFromTable 

func NewFrameFromTable(t arrow.Table) *Frame

NewFrameFromTable wraps an arrow.Table as a Frame. Each column's underlying Chunked is Retained so the Frame owns its own refs independent of t — callers are expected to Release t after this returns (arrow.Table's usual ownership contract). Without the Retain here, the Frame would share t's Chunked pointers without an ownership increment, and either the Frame's buffers would be use-after-freed when the caller Released t, or the Table's refs would leak if the caller didn't.

func (*Frame) Column 

func (f *Frame) Column(name string) (Series, error)

Column returns the Series named name.

func (*Frame) ColumnAt 

func (f *Frame) ColumnAt(i int) (Series, error)

ColumnAt returns the Series at position i.

func (*Frame) ColumnNames 

func (f *Frame) ColumnNames() []string

ColumnNames returns the column names in order.

func (*Frame) CompactChunks added in v0.4.4 

func (f *Frame) CompactChunks() (*Frame, error)

CompactChunks returns f with every column re-chunked as a single contiguous Arrow array. Concatenates each column's chunk list via `array.Concatenate` and rebuilds the Frame with the same schema.

Required before SortBy, Timestamp column-vs-column comparisons, and ListUnion on Concat output — those code paths currently error on multi-chunk input pending a chunk-walking rework. Also useful before handing a Frame to code that expects the RecordBatch shape (which is single-array-per-column by construction).

Cost

One `array.Concatenate` per column: allocates a fresh contiguous buffer sized to the total row count, then memcpies each chunk's bytes in. Caller pays the compaction tax explicitly at the call site — the Concat output stays cheap (chunk-list stitch, no memcpy) for callers who don't need single-chunk downstream.

Idempotency

When every column is already single-chunk (`f.IsSingleChunk()` returns true), returns f itself with a matched Retain — no new allocation. Callers can defensively `f.CompactChunks()` on every Frame that might have come from Concat without paying for frames that don't need it.

Errors

Propagates errors from `array.Concatenate` (chunk-type mismatches, unsupported types). In practice these only surface on malformed Frames — the schema-alignment invariants that `NewFrame` enforces prevent them on any Frame built through the public API.

func (*Frame) Concat 

func (f *Frame) Concat(others ...*Frame) (*Frame, error)

Concat returns a new Frame with rows from f followed by the rows of each frame in others, in the order given. No deduplication — this is a pure row stack (SQL UNION ALL / polars vstack / pandas `concat(axis=0)`).

All frames must share the exact same schema: matching column count, order, names, and arrow types. Fingerprint mismatches return an error naming the offending column with both types so the caller can decide how to cast.

Each frame's Arrow buffers are kept alive as separate chunks of the output — no memcpy of the underlying data. The resulting Frame's columns are multi-chunk; downstream ops that assume single-chunk (SortBy, Timestamp col-vs-col comparisons, ListUnion, RecordBatch views) will error until compacted. Call `.CompactChunks()` to materialize a single-chunk copy — cheap no-op when the frame is already single-chunk, so callers can compact defensively without checking. See also `Frame.IsSingleChunk()` / `Frame.NumChunks()`.

func (*Frame) Difference 

func (f *Frame) Difference(other *Frame, cols ...string) (*Frame, error)

Difference returns rows in f that do not appear in other, deduplicated over cols. cols nil/empty → all columns. Nulls are treated as equal.

Equivalent to SQL EXCEPT / polars' `df.filter(~col.is_in(other))` generalized to whole-row membership.

Both frames must share the exact same schema.

func (*Frame) DropColumn 

func (f *Frame) DropColumn(name string) (*Frame, error)

DropColumn returns a new Frame with the named column removed. Returns ErrColumnNotFound if no column matches. Every retained column is independently ref-counted so the caller can Release either Frame without affecting the other's buffers.

func (*Frame) Explode 

func (f *Frame) Explode(col string) (*Frame, error)

Explode returns a new Frame where each row of col has been split into one row per component.

Two column shapes are accepted:

  • Geometry column: multi-part geometries (MultiPoint, MultiLineString, MultiPolygon, GeometryCollection) expand to one row per contained geometry. Single geometries pass through unchanged. Null rows are kept as a single null row.
  • List<T> column: each element becomes its own row. The exploded column's arrow type becomes the list's element type. Null lists and empty lists both produce a single output row with a null value in the exploded column (polars-parity).

All other columns are duplicated across the exploded rows so per-row attributes propagate to every component.

func (*Frame) Filter 

func (f *Frame) Filter(mask Series) (*Frame, error)

Filter returns a new Frame containing only the rows where mask is true. Null mask entries are treated as false.

The mask length must equal the frame's row count.

func (*Frame) FilterExpr 

func (f *Frame) FilterExpr(e Expr) (*Frame, error)

FilterExpr returns a new Frame containing the rows for which e evaluates to true. e must produce a Boolean Series; null entries in the mask are treated as false (SQL-style — matches Frame.Filter).

Example: 

// Keep rows where (price * 1.08) > 100.
e := gobi.Col("price").Mul(gobi.Lit(1.08)).Gt(gobi.Lit(100.0))
out, err := df.FilterExpr(e)

func (*Frame) Geometry 

func (f *Frame) Geometry(colName string, row int) (any, error)

Geometry decodes the geometry at (row, colName). Returns ErrNotGeometry if the column is not a geometry column.

func (*Frame) GroupBy 

func (f *Frame) GroupBy(keys ...string) (*GroupBy, error)

GroupBy returns a GroupBy over the given key column names.

func (*Frame) Head 

func (f *Frame) Head(n int) *Frame

Head returns a Frame with the first n rows (default 5).

func (*Frame) Intersect 

func (f *Frame) Intersect(other *Frame, cols ...string) (*Frame, error)

Intersect returns rows that exist in both f and other, deduplicated over cols. cols nil/empty → all columns. Nulls are treated as equal (see Union for the semantics rationale).

The result draws rows + column values from f (not from other) — so if other has a row with the same key columns but different non-key values, you get f's version. This matches SQL INTERSECT and polars behavior.

Both frames must share the exact same schema.

func (*Frame) IsSingleChunk added in v0.4.4 

func (f *Frame) IsSingleChunk() bool

IsSingleChunk reports whether every column is backed by exactly one Arrow chunk. Equivalent to `f.NumChunks() == 1` but reads better at call sites that gate on the invariant (e.g. code paths that require a RecordBatch view via frameToBatch).

func (*Frame) Join 

func (f *Frame) Join(right *Frame, leftKey, rightKey string, kind JoinType) (*Frame, error)

Join returns a new Frame formed by combining rows of f (the left frame) and right, where the left column named leftKey equals the right column named rightKey. The join key must be a hashable type (String, Int64, Int32, Bool, Uint32, Uint64, Float64, Timestamp).

The result contains all columns from the left frame followed by all columns from the right frame except the join key. Right-side columns whose names collide with left-side columns are renamed with a "_right" suffix. For JoinSemi and JoinAnti, only left-side columns appear in the output.

Null keys never match (SQL semantics). In JoinLeft, JoinFull, and JoinAnti a null-keyed left row still appears in the output; in JoinRight and JoinFull a null-keyed right row still appears; in JoinSemi and JoinInner it is filtered out.

func (*Frame) Lazy 

func (f *Frame) Lazy() *LazyFrame

Lazy wraps f in a LazyFrame anchored at a Scan[frame] leaf. Fluent operations on the returned LazyFrame append nodes to the plan; Collect() replays them against f.

func (*Frame) NumChunks added in v0.4.4 

func (f *Frame) NumChunks() int

NumChunks returns the maximum Arrow chunk count across all columns. A well-formed Frame typically has the same chunk count on every column (chunk boundaries line up with row boundaries), so this is almost always the per-column count — but taking max stays robust against hand-constructed frames with mismatched chunk shapes.

Returns 0 for an empty (no-column) frame. Returns 1 for the common single-chunk shape produced by Collect, SortBy, Filter, etc. Returns >1 primarily after Concat / append-style operations that stitch existing chunks together without materialization.

Useful for the defensive-compact pattern: 

if f.NumChunks() > 1 {
    f, err = f.CompactChunks()
}

but callers can just always call CompactChunks — it's a no-op on already-single-chunk frames.

func (*Frame) NumCols 

func (f *Frame) NumCols() int

NumCols returns the number of columns.

func (*Frame) NumRows 

func (f *Frame) NumRows() int

NumRows returns the number of rows in the frame (0 if there are no columns).

func (*Frame) PartitionMetadata added in v0.2.0 

func (f *Frame) PartitionMetadata() *PartitionMetadata

PartitionMetadata returns the partition claim carried through by Collect from the source plan node, or nil if no claim was made. Runtime consumers (Over's aligned fast path, future partition-wise join / GroupBy dispatch) check this to decide whether shuffle-skip optimizations apply.

Users constructing Frames via NewFrame get a nil claim by default. The LazyFrame → Collect path attaches whatever the root plan node's PartitionMetadata() reports. Attach explicitly via Frame.WithPartitionMeta for hand-built frames that carry a claim.

func (*Frame) Pivot 

func (f *Frame) Pivot(index, columns, values string, agg AggKind) (*Frame, error)

Pivot reshapes a long-form Frame into a wide-form one.

  • index: column whose distinct values become the rows.
  • columns: column whose distinct values become the new columns.
  • values: column supplying the cell values.
  • agg: how to reduce when multiple input rows map to the same (index, columns) cell. Use AggFirst / AggLast if you're sure there's no collision; use AggSum / AggMean / etc. to reduce. Any built-in AggKind is accepted.

The output schema is: 

<index column> | <col_value_1> | <col_value_2> | ...

Column values are stringified for use as arrow field names — the header row is always a string. The value column's arrow type follows aggOutputType(Aggregation{Kind: agg, ...}); it's Int64 for Count/NUnique and Float64 for the rest, matching what GroupBy.Agg would produce.

Cells with no matching input rows are emitted as null. Output rows are ordered by the sorted index value; output columns are ordered by the sorted distinct columns-column value so the shape is deterministic across runs.

Equivalent to `pandas.DataFrame.pivot_table(index, columns, values, aggfunc)` / polars' `DataFrame.pivot`. Multi-column pivot indices aren't supported yet — pass a single index column.

func (*Frame) Release 

func (f *Frame) Release()

Release decrements the reference count of the underlying Arrow columns. Callers should match every Retain (including the implicit one at construction) with exactly one Release.

func (*Frame) Rename added in v0.2.4 

func (f *Frame) Rename(old, new string) (*Frame, error)

Rename returns a new Frame with the column named old renamed to new. Column order and buffers are preserved (the underlying arrow arrays are shared via ref-count; only the arrow.Field's Name changes).

Returns ErrColumnNotFound if old doesn't exist. If new already exists on the Frame under a different index, the returned Frame has two columns with the same name — usually an error at downstream operators. Callers who want swap-safety should DropColumn(new) first.

func (*Frame) ResampleEvery 

func (f *Frame) ResampleEvery(timeCol string, interval time.Duration) (*Resampler, error)

ResampleEvery returns a Resampler that will group f's rows into interval-wide buckets by the values in timeCol. The interval must be positive.

func (*Frame) Retain 

func (f *Frame) Retain()

Retain increments the reference count of the underlying Arrow columns.

func (*Frame) RollingBy 

func (f *Frame) RollingBy(timeCol string, period time.Duration) (*TimeRolling, error)

RollingBy returns a TimeRolling. The time column must be a Timestamp series in non-decreasing order.

func (*Frame) Row 

func (f *Frame) Row(i int) (*Frame, error)

Row returns a Frame containing the single row at index i.

func (*Frame) SJoin 

func (f *Frame) SJoin(right *Frame, leftGeomCol, rightGeomCol string, pred SpatialPredicate, opts ...Option) (*Frame, error)

SJoin performs a spatial join of f (left) and right by evaluating pred on each pair of geometries from leftGeomCol and rightGeomCol. Only rows where pred holds are emitted. The output frame contains all columns from the left frame, followed by all columns from the right frame except its geometry column (analogous to Frame.Join dropping the right join key). Right-side column names that collide with left-side names get a "_right" suffix.

Under the hood, an R-tree is built over the right frame's geometry bounds so each left row scans only overlapping candidates. Parallelism follows the priority order documented on resolveWorkers: Workers(n) > package SetMaxParallelism > GOMAXPROCS.

CRS handling

SJoin does NOT validate that left and right geometries share a CRS. Both the AoS and SoA fast paths compare coordinates in whatever native unit the WKB carries; joining WGS84 lat/lon against a projected EPSG:3857 column silently produces bogus matches. Callers must reproject upstream (e.g. Series.To(crs)) to ensure the two columns are in the same CRS before calling SJoin. This is a known limitation, tracked as a future correctness hardening pass — surfacing here so tests + callers can plan for it.

func (*Frame) Schema 

func (f *Frame) Schema() *arrow.Schema

Schema returns the underlying Arrow schema.

func (*Frame) SelectCols added in v0.2.4 

func (f *Frame) SelectCols(names ...string) (*Frame, error)

SelectCols returns a Frame containing only the named columns, in the given order. Buffers are shared with the source Frame — the returned Frame owns its own field/schema structure but the underlying arrow arrays are ref-counted.

Also handles reordering: SelectCols("b", "a") returns a Frame whose output has "b" before "a". Zero names produce an empty Frame with the same row count and no columns. Missing columns error with ErrColumnNotFound.

func (*Frame) Shape 

func (f *Frame) Shape() (rows, cols int)

Shape returns (rows, cols) — matching Pandas convention.

func (*Frame) SortBy 

func (f *Frame) SortBy(keys ...SortKey) (*Frame, error)

SortBy returns a new Frame with rows arranged according to keys. The sort is stable: rows that compare equal on every key retain their input order.

Supported key column types: String, Bool, Int32, Int64, Uint32, Uint64, Float64, Float32, Timestamp. Nulls sort last.

Example: 

// Chronologically by date; break ties by value descending.
out, err := df.SortBy(
    gobi.SortKey{Column: "date"},
    gobi.SortKey{Column: "value", Descending: true},
)

func (*Frame) SortByHilbert added in v0.3.5 

func (f *Frame) SortByHilbert(geomCol string) (*Frame, error)

SortByHilbert returns a new Frame with rows reordered by the Hilbert-curve position of each row's geometry centroid. Spatial pre-sorting is what turns GeoParquet 1.1 row-group bbox pushdown from a synthetic-benchmark curiosity into a real-world speedup: after a Hilbert sort, per-row-group bboxes cluster tightly in space, so an AOI-shaped predicate can skip most of a file.

geomCol is the name of the geometry column to sort by. If the column is missing or isn't a geometry column, an error is returned. Empty frames pass through as-is.

The sort is stable — rows whose centroids hash to the same Hilbert cell (order-16 default → 65,536 cells per axis) retain their input order. Null-geometry rows sort last so downstream consumers can drop them with a Head/Limit if desired.

The bounding box used for Hilbert normalization is computed from the column's own centroids, so the sort is self-contained. For multi-file / multi-partition sorts that need a SHARED reference bbox (so outputs merge cleanly), use SortByHilbertWith and pass HilbertSortOptions.Bounds.

**Multi-part geometries:** the sort key is the row's single centroid, which for a scattered MultiPolygon (US with Alaska + Hawaii, France with overseas territories, a MultiLineString of disconnected roads) can land in a "no-man's-land" Hilbert cell far from any actual part. Rows like that can end up ordering worse than their per-part centroids would suggest. If it matters for your access pattern, Explode the multi-part column first (Frame.Explode) so each row carries a single spatial location, sort, then re-aggregate — or accept the coarser ordering and rely on the per-row bbox stats to handle the outliers.

func (*Frame) SortByHilbertWith added in v0.3.5 

func (f *Frame) SortByHilbertWith(geomCol string, opts HilbertSortOptions) (*Frame, error)

SortByHilbertWith is the bounds-and-order-parameterized variant of SortByHilbert. Empty opts.Bounds falls back to the column- derived rectangle; empty opts.Order falls back to geometry.DefaultHilbertOrder.

Multi-file usage: compute a single Bounds covering every partition (e.g. by unioning per-partition bboxes) and pass it to every SortByHilbertWith call so the resulting Hilbert indices are comparable across files. Then a downstream merge preserves global spatial locality.

func (*Frame) SortBySTR added in v0.3.5 

func (f *Frame) SortBySTR(geomCol string, leafSize int) (*Frame, error)

SortBySTR returns a new Frame with rows reordered using the Sort-Tile-Recursive (STR) leaf-level ordering. Alternative to SortByHilbert with different locality tradeoffs.

STR partitions the N centroids into ⌈√(N/leafSize)⌉ vertical strips sorted by x, then sorts each strip's centroids by y — producing groups of leafSize consecutive rows that share both a tight X range (they're in the same strip) AND a tight Y range (they're consecutive-in-y within the strip). Sub-groups of leafSize consecutive rows come out spatially rectangular.

leafSize is the target row-group size (typically the same value you'll pass to WriteOptions.RowGroupRows). A value <= 0 falls back to STRDefaultLeafSize.

**When to prefer STR over Hilbert:**

  • Axis-aligned AOI queries (rectangular bboxes with sides parallel to X/Y): STR row groups are rectangles, so an axis-aligned AOI cleanly overlaps at most a strip's worth of them. Hilbert curves cross tile boundaries at odd angles, so a rectangular AOI can straddle more row groups.
  • Static datasets partitioned into predictable strips (latitude bands, admin regions, time-series windows).

**When to stick with Hilbert:**

  • Point queries or diagonal-aligned AOIs — Hilbert's curve preserves 2D locality symmetrically; STR privileges the X axis over Y (arbitrary but structural).
  • Multi-file / cross-partition sorts — Hilbert indices in a shared reference frame merge cleanly, whereas STR needs a custom merge policy.

Sort is stable within a leaf. Null geometries sort last.

func (*Frame) String 

func (f *Frame) String() string

String returns a debug representation of the frame. Not intended for pretty printing at scale.

func (*Frame) Table 

func (f *Frame) Table() arrow.Table

Table returns an arrow.Table view of the frame. The returned Table carries its own reference on each column's Chunked (arrow.NewTable Retains internally), so the Frame and the Table are independent owners of the underlying buffers — Release each once when done. Callers who pass the Table to a consumer that iterates it (e.g. parquet WriteTable) should Release the Table after the consumer returns; failing to do so leaks the Table's Retained refs and pins the Frame's buffers even after the Frame goes out of scope.

func (*Frame) Tail 

func (f *Frame) Tail(n int) *Frame

Tail returns a Frame with the last n rows (default 5).

func (*Frame) Take 

func (f *Frame) Take(indexes []int) (*Frame, error)

Take returns a new Frame consisting of rows selected by the given indexes, in the order given. Duplicates are allowed. Out-of-range indexes produce an error.

func (*Frame) Union 

func (f *Frame) Union(other *Frame, cols ...string) (*Frame, error)

Union returns rows in either f or other, deduplicated over cols. When cols is nil or empty, uniqueness is determined over every column. Uses the same composite-key encoding as GroupBy so null-vs-null comparisons collide (nulls are treated as equal for set membership — pandas / polars semantics, not SQL semantics).

Both frames must share the exact same schema — see Concat for the compatibility rules and error format.

Equivalent to `f.Concat(other).Unique(cols...)` but bundled for clarity of intent.

func (*Frame) Unique 

func (f *Frame) Unique(cols ...string) (*Frame, error)

Unique returns a Frame with duplicate rows removed. Uniqueness is determined by the columns listed in cols; other columns come along for the ride from the first occurrence of each distinct key.

When cols is empty, distinctness is computed over every column (SQL `DISTINCT *`). Any listed column must be a hashable arrow type (String, LargeString, Int64, Int32, Uint64, Uint32, Float64, Bool, Timestamp) — the same set GroupBy accepts.

Semantics match pandas' `df.drop_duplicates(subset=cols)` and polars' `df.unique(subset=cols)`: the result preserves the full input schema and first-occurrence order. If you only want the distinct values of a single column, use `df.Column(name).Unique()` which returns a Series.

This is the whole-frame `distinct` primitive. For per-group distinct counts, use `GroupBy(...).Agg(Aggregation{Kind: AggNUnique})`.

func (*Frame) ValueCounts 

func (f *Frame) ValueCounts(col string) (*Frame, error)

ValueCounts returns a two-column Frame of (value, count) pairs counting the occurrences of each distinct value of col, sorted descending by count. Ties are broken by the value itself ascending — makes the output stable / deterministic even when several distinct values share a frequency.

The output columns are named `col` (matching the input) and `count` (Int64). Null values in the source column are counted as their own group under the null value slot — the count row for null contains a null in the value column and its count in the count column.

Equivalent to pandas' `Series.value_counts()` / polars' `Series.value_counts()`. For per-group value counts (e.g. count of role per department), you want a nested output that gobi doesn't have a schema for yet — build it via two-level GroupBy today or wait for list-typed columns.

func (*Frame) WithColumn 

func (f *Frame) WithColumn(name string, s Series) (*Frame, error)

WithColumn returns a new Frame with s appended (or replaced, if a column already exists with that name). The returned Frame independently ref-counts every column, so the caller can Release either Frame without affecting the other's buffers.

s.Len() must equal f.NumRows() unless f has zero columns.

The Series' arrow.Field is carried over — its Nullable flag, geometry metadata, and any other field-level attributes — with only the Name replaced by name. This preserves geometry-column identification when swapping in a WKB Binary column derived from a user function.

func (*Frame) WithColumnExpr 

func (f *Frame) WithColumnExpr(name string, e Expr) (*Frame, error)

WithColumnExpr returns a new Frame with e evaluated and attached under name. If a column named name already exists, it is replaced in place; otherwise it is appended.

Example: 

// Add a computed column: usd_price = eur_price * 1.08
out, err := df.WithColumnExpr("usd_price",
    gobi.Col("eur_price").Mul(gobi.Lit(1.08)),
)

func (*Frame) WithPartitionMeta added in v0.2.0 

func (f *Frame) WithPartitionMeta(meta *PartitionMetadata) *Frame

WithPartitionMeta returns f with the given partition claim attached. Used by collectPlan to propagate plan-node metadata onto the emitted Frame; also available for hand-built frames that carry a claim (rare — most callers should compose a LazyFrame + WithPartitionAssertion instead).

Returns the same Frame receiver — this is a mutating helper on a value that's already immutable-by-convention, so callers should only call it once immediately after construction.

type GeoParquetBboxCovering added in v0.3.4 

type GeoParquetBboxCovering struct {
	Xmin []string `json:"xmin"`
	Ymin []string `json:"ymin"`
	Xmax []string `json:"xmax"`
	Ymax []string `json:"ymax"`
}

GeoParquetBboxCovering names the four columns holding per-row bbox coordinates. Each is a path array (typically single-element for flat top-level columns).

type GeoParquetColumnMeta 

type GeoParquetColumnMeta struct {
	Encoding      string         `json:"encoding"`
	GeometryTypes []string       `json:"geometry_types"`
	CRS           map[string]any `json:"crs,omitempty"`
	Bbox          []float64      `json:"bbox,omitempty"`

	// Covering names the per-row bounding-box columns that a reader
	// can use for row-group pruning without decoding WKB. Populated
	// by parquetio.WriteFile when it emits companion bbox columns
	// (see BboxCoveringSuffixes). Follows the GeoParquet 1.1
	// "covering.bbox" spec:
	//
	//	{"xmin": ["<col>"], "ymin": ["<col>"], "xmax": ["<col>"], "ymax": ["<col>"]}
	//
	// A single-element path array means "top-level column with this
	// name" — gobi's flat-column approach — but consumers that
	// support nested struct-covering (geopandas, DuckDB spatial) read
	// it identically.
	Covering *GeoParquetCovering `json:"covering,omitempty"`
}

GeoParquetColumnMeta is the per-column entry inside GeoParquetMetadata.

type GeoParquetCovering added in v0.3.4 

type GeoParquetCovering struct {
	Bbox *GeoParquetBboxCovering `json:"bbox,omitempty"`
}

GeoParquetCovering describes how to compute a row's bounding box from other columns in the same file. Only the bbox variant is used today.

type GeoParquetMetadata 

type GeoParquetMetadata struct {
	Version       string                          `json:"version"`
	PrimaryColumn string                          `json:"primary_column"`
	Columns       map[string]GeoParquetColumnMeta `json:"columns"`
}

GeoParquetMetadata is the JSON payload written under the "geo" key of a GeoParquet file's Arrow-level metadata.

func BuildGeoParquetMetadata 

func BuildGeoParquetMetadata(f *Frame) (*GeoParquetMetadata, error)

BuildGeoParquetMetadata scans f and produces a GeoParquet metadata blob describing its geometry columns. Every column tagged as a geometry column (via GeometryField) is scanned once to compute its bounding box and the set of geometry types it contains.

func ParseGeoParquetMetadata 

func ParseGeoParquetMetadata(raw string) (*GeoParquetMetadata, error)

ParseGeoParquetMetadata decodes the JSON blob under the "geo" key.

type GroupBy 

type GroupBy struct {
	// contains filtered or unexported fields
}

GroupBy partitions a Frame by the values in one or more key columns. The keys must be of a hashable Arrow type (String, Int64, Int32, Bool, Float64).

func (*GroupBy) Agg 

func (g *GroupBy) Agg(aggs ...Aggregation) (*Frame, error)

Agg computes the requested aggregations over each group, returning a Frame whose first N columns are the group keys (in the order passed to GroupBy) and whose remaining columns are the aggregations in order.

When the group-by uses exactly one hashable key column with a single Arrow chunk and every aggregation column is a single-chunk primitive numeric type, the fast path (aggFast) is taken — it avoids the per-row byte-slice hashing and chunk walks that dominate the general path. All other shapes fall back to the multi-key path below.

type HilbertSortOptions added in v0.3.5 

type HilbertSortOptions struct {
	// Bounds is the reference rectangle used to normalize centroid
	// (x, y) into the Hilbert grid. Empty (zero-value) means "derive
	// from the column's own centroids" — matches SortByHilbert's
	// default behavior.
	//
	// Non-empty bounds are the point of this variant: multi-file or
	// multi-partition sorts that need a SHARED reference frame so
	// their outputs merge cleanly. Rows whose centroids fall outside
	// bounds clamp to the grid edge (see geometry.HilbertIndex).
	Bounds geometry.Bounds

	// Order is the Hilbert-curve depth. Zero uses
	// geometry.DefaultHilbertOrder. Higher values give finer
	// discrimination at cost of nothing meaningful — the sort itself
	// is O(N log N) regardless.
	Order int
}

HilbertSortOptions controls the reference frame used by SortByHilbertWith. Zero-value is the same as calling SortByHilbert (bounds computed from the column's own centroids, default order).

type IncrementalAggregator added in v0.2.7 

type IncrementalAggregator interface {
	Aggregator
	Clone() IncrementalAggregator
	Update(col Series, rows []int) error
	Finalize() any
}

IncrementalAggregator is the optional extension that opts a custom aggregator into the streaming aggregate executor. Implementers process a group's rows incrementally across batches instead of receiving the whole row set at once via Aggregate, avoiding the materialize-both-sides concat that plain Aggregator triggers.

Contract:

  • Clone returns a fresh instance with empty per-group state. The streaming executor calls Clone once per group at first-touch; each group's Update / Finalize / Merge run on its own clone. Callers never share a clone across groups.
  • Update adds col[rows] to this instance's state. Called at most once per input batch per group. Must be additive — do not reset internal state at the top of Update.
  • Finalize returns the group's aggregated value. Called once per group after all Updates. Should not mutate state.
  • Aggregate (inherited from Aggregator) is still called by the eager path; implementers should keep it consistent with the Clone → Update → Finalize sequence. A typical implementation of Aggregate is just: reset state, call Update, return Finalize's value.

Custom aggregators that implement only Aggregator (not this interface) continue to work — they route through the materializing fallback exactly as before.

type JoinType 

type JoinType uint8

JoinType selects the join behavior. 

const (
	// JoinInner returns rows where the key exists on both sides.
	JoinInner JoinType = iota
	// JoinLeft returns every row from the left frame, with nulls where the
	// right side has no matching key.
	JoinLeft
	// JoinRight returns every row from the right frame, with nulls where
	// the left side has no matching key. Row order follows the right
	// frame's row order.
	JoinRight
	// JoinFull (a.k.a. FULL OUTER) returns the union of JoinLeft and
	// JoinRight: every row from both frames, matched where possible and
	// null-padded on the missing side otherwise. Left rows come first
	// (in left-row order), followed by unmatched right rows in right-row
	// order.
	JoinFull
	// JoinSemi returns each left row that has at least one match on the
	// right, without duplication when there are multiple matches. Only
	// left-side columns are emitted.
	JoinSemi
	// JoinAnti returns each left row that has NO match on the right.
	// Only left-side columns are emitted. Left rows with a null key are
	// treated as unmatched.
	JoinAnti
)

type LazyFrame 

type LazyFrame struct {
	// contains filtered or unexported fields
}

LazyFrame is a Frame that hasn't been computed yet — a chain of operations expressed as a plan tree. Building a LazyFrame does no work; each fluent method appends a node to the plan. Call Collect() to actually run it.

Today (Slice 1 of Layer 2-3) Collect() dispatches each plan node to the existing eager Frame methods. There is no optimizer yet — a LazyFrame chain runs at exactly the same speed as the equivalent eager code. The value here is API shape:

  • Inspect a plan without running it: Explain() / Schema().
  • Compose complex pipelines without naming intermediate Frames.
  • Ready surface for Layer 4 optimizer passes and Layer 6 vectorized executor.

func NewLazyFrame 

func NewLazyFrame(plan LogicalPlan) *LazyFrame

NewLazyFrame constructs a LazyFrame from an arbitrary LogicalPlan. Reserved for advanced users building their own plan trees (custom scan nodes, plan-tree rewrites). Most callers should use Frame.Lazy() and the fluent builders.

func (*LazyFrame) Collect 

func (lf *LazyFrame) Collect() (*Frame, error)

Collect materializes the LazyFrame into a concrete *Frame. Runs the default optimizer, compiles the plan into a tree of ExecOperators, and drives execution via the streaming executor.

Streaming operators (Filter, Project, WithColumn, Drop, Limit) pull one batch at a time; blocking operators (Sort, Aggregate, Join, Tail) buffer their input to a Frame and delegate to the eager engine. Peak memory is bounded to one batch per streaming node plus the accumulated Frame at each blocking node.

This is where errors surface: bad expressions, type mismatches, unknown columns, scan failures.

func (*LazyFrame) CollectRaw 

func (lf *LazyFrame) CollectRaw() (*Frame, error)

CollectRaw skips both the optimizer AND the streaming executor, executing the plan tree via the bottom-up whole-Frame walker used before Layer 6. Useful for debugging optimizer bugs, for benchmarks that isolate the eager engine's cost, and as a correctness oracle for the executor's tests. Prefer Collect for real use.

func (*LazyFrame) DropColumn 

func (lf *LazyFrame) DropColumn(name string) *LazyFrame

DropColumn appends a Drop node. If the named column is missing at Collect time, the underlying Frame.DropColumn surfaces ErrColumnNotFound.

func (*LazyFrame) Explain 

func (lf *LazyFrame) Explain() string

Explain returns a human-readable representation of the plan tree. Deepest node first (which is how the tree evaluates: bottom-up). Handy for debugging what a chain of fluent methods actually built.

Example: 

Limit(1000)
  Project(col("id"), (col("value") * lit(1.08)) AS "usd")
    Filter((col("value") > lit(100)))
      Scan[frame](500 rows × 3 cols)

func (*LazyFrame) ExplainOptimized 

func (lf *LazyFrame) ExplainOptimized() string

ExplainOptimized returns the Explain output for the plan tree after the default rule set has been applied.

func (*LazyFrame) ExplainPhysical 

func (lf *LazyFrame) ExplainPhysical() string

ExplainPhysical returns a description of how the LazyFrame will be executed — which operators stream, which materialize, which scan strategy each source uses. Predicts the compilation choices Compile would make without actually building the executor tree (no goroutines started, no right-side materialization for joins, no file I/O). Cheap enough to call in tests and inspection loops.

Format: outer-most node first, one node per line, two-space indent per depth. Each label is prefixed with the strategy ("Streaming*" or "Materialize*") so the choice is visible at a glance.

Example (analytical pipeline over parquet): 

StreamingAggregate(keys=[region], aggs=[value_sum])
  StreamingJoin(inner, left.user_id = right.user_id)
    Filter((col("value") > lit(100)))
      ScanFile[stream, parquet]("events.parquet", cols=[user_id value region])
    ScanFrame(100 rows × 2 cols)

func (*LazyFrame) Explode added in v0.2.1 

func (lf *LazyFrame) Explode(col string) *LazyFrame

Explode appends an Explode node — one row per component of the named geometry or list column. Semantics match Frame.Explode: multi-part geometries expand to their constituents; List<T> columns expand to one row per element, with null/empty lists producing a single output row with a null element (polars-parity); non-exploded columns are duplicated across the expansion.

The exploded column's output type is the list element type for List<T> columns, or unchanged for geometry columns (Binary/WKB with the multi-part reduced to a single part). Errors for non-exploded- able column types surface at Collect().

func (*LazyFrame) Filter 

func (lf *LazyFrame) Filter(cond Expr) *LazyFrame

Filter appends a Filter node. cond must produce a Boolean Series when evaluated; the error surfaces at Collect() time, not here.

func (*LazyFrame) GroupBy 

func (lf *LazyFrame) GroupBy(keys ...string) *LazyGroupBy

GroupBy returns a lazy group-by builder. Chain .Agg(aggs...) to get back a LazyFrame whose Collect() runs Frame.GroupBy(keys...).Agg(aggs...).

The two-step shape mirrors the eager API — no ambiguity about what gb, err := lf.GroupBy(...) means (there's no error return here, missing keys surface at Collect).

func (*LazyFrame) Head 

func (lf *LazyFrame) Head(n int) *LazyFrame

Head returns the first n rows. Alias for Limit — the plan node is the same. Kept as a separate method for pandas-shaped ergonomics.

func (*LazyFrame) Join 

func (lf *LazyFrame) Join(right *LazyFrame, leftKey, rightKey string, kind JoinType) *LazyFrame

Join appends a Join node combining lf and right on (leftKey, rightKey). Semantics match Frame.Join for all six JoinType values; see that method's documentation for null-key handling and key- coalescing on Right/Full joins.

func (*LazyFrame) Limit 

func (lf *LazyFrame) Limit(n int) *LazyFrame

Limit appends a Limit node that keeps the first n rows. Negative or zero n produces an empty result at Collect() time.

func (*LazyFrame) Optimize 

func (lf *LazyFrame) Optimize() LogicalPlan

Optimize returns the optimizer-rewritten plan without executing. Inspection aid: pair with String() / Explain() to see what a chain of fluent methods reduces to.

func (*LazyFrame) PartitionMetadata added in v0.2.0 

func (lf *LazyFrame) PartitionMetadata() *PartitionMetadata

PartitionMetadata returns the partition claim on this LazyFrame's output, or nil if no claim has been made. The claim is derived from the root plan node — scan sources populate it at construction time (see NewScanNode's WithPartitionMetadata option), intermediate nodes propagate it via rules landing in step 2 of the v0.3.0 plan.

Consumers of this method today are primarily test code and debugging; the alignment predicate that decides whether .Over(K) / Join(K) / GroupBy(K) can skip shuffle lands in step 3-4 of the v0.3.0 plan. Users composing hand-annotated claims go through LazyFrame.WithPartitionAssertion (step 3).

func (*LazyFrame) Plan 

func (lf *LazyFrame) Plan() LogicalPlan

Plan returns the underlying plan tree. Used by tree walkers and by tests exercising plan structure directly.

func (*LazyFrame) Rename added in v0.2.4 

func (lf *LazyFrame) Rename(old, new string) *LazyFrame

Rename appends a Rename step that changes the column named old to new, preserving column order and buffers. Missing old surfaces ErrColumnNotFound at Collect(). Same-name rename (old == new) is a no-op.

func (*LazyFrame) Schema 

func (lf *LazyFrame) Schema() *arrow.Schema

Schema returns the plan's output schema without executing anything. For plans that involve type inference (Project, WithColumn), fields whose expressions can't be statically typed are reported with a nil Type — Collect() will surface the underlying evaluation error.

func (*LazyFrame) Select 

func (lf *LazyFrame) Select(exprs ...Expr) *LazyFrame

Select appends a Project node — the resulting LazyFrame contains only the columns produced by exprs, in that order. Each expression's output column name comes from Namer.OutputName if the expression has one (Col("id") → "id", something.Alias("x") → "x"); otherwise a positional default ("expr_0", "expr_1", ...) is used.

Zero expressions is a valid but empty Select — the resulting Frame has 0 columns. Add an Alias if you want a non-default output name for computed columns.

func (*LazyFrame) SelectCols added in v0.2.4 

func (lf *LazyFrame) SelectCols(names ...string) *LazyFrame

SelectCols is a name-based shorthand for Select — picks the given columns from the input plan, in the given order. Equivalent to Select(Col(names[0]), Col(names[1]), ...).

Also handles column reordering: SelectCols("b", "a") returns a LazyFrame whose output has "b" before "a". Missing columns surface at Collect() (colRefNode's Eval reports the not-found error).

func (*LazyFrame) SortBy 

func (lf *LazyFrame) SortBy(keys ...SortKey) *LazyFrame

SortBy appends a Sort node. Semantics match Frame.SortBy: multi-key stable, nulls-last, direction per-key.

func (*LazyFrame) Tail 

func (lf *LazyFrame) Tail(n int) *LazyFrame

Tail returns the last n rows. Unlike Head, this cannot be implemented as a plain Limit — offset depends on the total row count, which isn't known until the input has been materialized. Dispatches to Frame.Tail at Collect time.

func (*LazyFrame) WithColumn 

func (lf *LazyFrame) WithColumn(name string, e Expr) *LazyFrame

WithColumn appends a WithColumn node. If a column named name already exists at this point in the plan, it will be replaced; otherwise the new column is appended.

func (*LazyFrame) WithPartitionAssertion added in v0.2.0 

func (lf *LazyFrame) WithPartitionAssertion(assertion *PartitionMetadata) (*LazyFrame, error)

WithPartitionAssertion attaches user-provided PartitionMetadata to the LazyFrame. Enables partition-aware operators (.Over(K), future partition-wise Join / GroupBy) on sources whose partitioning can't be inferred automatically — RawUnload / RawCTAS in athenaio, hand-crafted parquet scans, custom UDFs that write partitioned output.

Only narrowing is allowed. The assertion must:

  • Match the input's Columns exactly (can't change what's partitioned on).
  • Match the input's HashFn exactly (can't change hash function).
  • Provide a SortedBy that's a (possibly empty) prefix of the input's SortedBy. A writer that enforced [a, b, c] as a lex sort also enforced [a, b] as a prefix, so shorter prefixes are narrowings; different columns or reorderings are not.
  • Downgrade SortEnforced from true to false, or leave it. It may not upgrade false → true (that would be a stronger claim than the source made).

Nil input claim → any assertion allowed (opaque source, user owns the whole claim). Nil assertion → always allowed (narrows any claim to "no claim").

User owns correctness. gobi never verifies the assertion against actual data. A wrong assertion produces wrong window results with no visible error.

type LazyGroupBy 

type LazyGroupBy struct {
	// contains filtered or unexported fields
}

LazyGroupBy is the intermediate builder returned by LazyFrame.GroupBy. Its only method, Agg, closes the group-by out into a LazyFrame.

func (*LazyGroupBy) Agg 

func (lg *LazyGroupBy) Agg(aggs ...Aggregation) *LazyFrame

Agg attaches the given aggregations and returns a LazyFrame whose output schema is [group keys...] + [agg outputs...] in the order given. Naming, type mapping, and null semantics match the eager GroupBy.Agg.

type LogicalPlan 

type LogicalPlan interface {
	// Schema returns the output schema of this node. Called during
	// LazyFrame.Schema() and by nodes that need to inspect their
	// input's shape (e.g. Project computing its output columns).
	Schema() *arrow.Schema

	// Children returns the immediate sub-plans, in evaluation order
	// (empty for leaves). Used by tree walkers, optimizers, and
	// Explain().
	Children() []LogicalPlan

	// String returns a single-line description of this node —
	// no children, no formatting. Used as the label in Explain
	// output; the tree walker handles indentation and recursion.
	String() string

	// PartitionMetadata returns the partition claim carried by this
	// node's output rows. nil = no claim; alignment proofs never
	// prove anything against a nil claim. Consumed by the optimizer
	// to prove shuffle-free .Over(K) / Join(K) / GroupBy(K).
	//
	// Scan sources (parquetio's ScanFile, athenaio's UnloadAndRead,
	// hand-annotated via LazyFrame.WithPartitionAssertion) attach
	// metadata at construction time. Intermediate nodes derive
	// their output metadata from their inputs — see the propagation
	// rules in contrib/athenaio/PARTITION-METADATA.md. Every non-
	// scan node returns nil today (step 1 of the v0.3.0 plan);
	// propagation wiring lands in step 2.
	PartitionMetadata() *PartitionMetadata
}

LogicalPlan represents a node in a query plan tree. Plans are immutable data structures — building a plan does not execute anything. LazyFrame consumes a LogicalPlan and executes it via Collect().

A future optimizer will walk plan trees, rewrite them, and translate them to a physical plan for execution. Slice 1 (this file) provides the tree shape and node types; Collect() dispatches each node to the existing eager engine.

func NewScanNode 

func NewScanNode(label string, schema *arrow.Schema, read func() (*Frame, error), opts ...ScanOption) LogicalPlan

NewScanNode constructs a leaf plan node representing a deferred I/O source. label is used in Explain output (e.g. "Scan[parquet](path)"), schema is the eagerly-inferred output schema (or nil if unknown at construction), and read is called by Collect() to materialize the scan's *Frame.

Optional ScanOptions declare capabilities the optimizer can exploit — WithColumnProjection enables projection pushdown from any Project/Filter above the scan.

Exposed for source packages (parquetio, csvio, ...) to build scan leaves without a per-format node type in the core package. Most callers should use the format-specific wrapper (parquetio.ScanFile and friends) instead of building nodes by hand.

func Optimize 

func Optimize(plan LogicalPlan, rules ...Rule) LogicalPlan

Optimize applies rules to plan until either nothing changes or the iteration cap is hit. Returns the (possibly rewritten) root. If rules is empty, DefaultRules() is used.

type Namer 

type Namer interface {
	OutputName() string
}

Namer is implemented by expression nodes that carry a meaningful output column name. Select and WithColumn use it to derive default column names (Col("id") → "id", Col(x).Alias("y") → "y").

Nodes that don't implement Namer, or whose OutputName is empty, fall back to a positional default ("expr_0", "expr_1", ...).

type Option 

type Option interface {
	// contains filtered or unexported methods
}

Option configures a parallel gobi operation. Values are applied to an internal options struct in the order the caller supplies them, so later options override earlier ones.

Every parallel entry point (currently SJoin; more to come) accepts a trailing `...Option`. Callers that don't care can omit them entirely.

func Workers 

func Workers(n int) Option

Workers overrides the maximum number of parallel workers used by the operation it is passed to.

  • Workers(n) with n > 0 caps parallelism at n workers.
  • Workers(0) or Workers(-1) means "use the package default" (see SetMaxParallelism); if that too is unset, GOMAXPROCS is used.
  • Workers(1) forces sequential execution.

type PartitionMetadata added in v0.2.0 

type PartitionMetadata struct {
	// Columns are the partition columns in the order used by the
	// hash function. hash(a, b) ≠ hash(b, a) in every hash function
	// gobi tags, so ordering is significant. Alignment always
	// requires ordered equality on this slice.
	Columns []string

	// HashFn is a versioned, source-namespaced string tag naming
	// the hash function used to partition. Empty string is reserved
	// for value-partitioning (Hive-style directory-per-unique-value)
	// where "same value → same partition" is trivial.
	//
	// Cross-tag comparisons always fail — the optimizer never
	// assumes two sources' hash functions are compatible even when
	// both name the same underlying algorithm. Type-encoding of
	// hash inputs differs across sources (Trino's hash normalizes
	// decimals differently than xxhash64 over UTF-8 bytes, for
	// example). Alignment holds iff HashFn strings are byte-equal.
	//
	// Reserved tags (see PARTITION-METADATA.md for the registry):
	//   ""                                  — value partitioning
	//   "gobi/xxhash64/v1"                  — gobi runtime shuffle
	//   "athenaio/iceberg/murmur3-32/v1"    — athenaio Iceberg CTAS
	//   "athenaio/hive/bucket/v1"           — athenaio Hive CTAS
	//   "pgio/hash/v1"                      — reserved for pgio
	//   "bigqueryio/hash/v1"                — reserved for bigqueryio
	HashFn string

	// SortedBy describes within-partition ordering. Nil = unsorted.
	// Each SortKey is a (column, direction) pair — direction matters
	// for Diff / Shift / fill-forward correctness.
	SortedBy []SortKey

	// SortEnforced distinguishes "writer guaranteed the sort" (e.g.
	// Iceberg's first-class sorted_by table property) from "sort
	// was a hint, verify at read time or don't rely on it for
	// correctness" (e.g. Hive's sorted_by, which is advisory).
	//
	// Operators that use sortedness only for performance (skipping
	// a re-sort) may trust either value. Operators whose correctness
	// depends on order (Diff, Shift with fill_forward) MUST check
	// this flag and either verify at read time or fall back to a
	// sorted path.
	SortEnforced bool
}

PartitionMetadata describes how a LazyFrame's rows are physically grouped by partition key. Populated by scan sources that can prove partitioning (athenaio's CTAS write path, future parquetio per-file partitioning claims, hand-annotated via WithPartitionAssertion). Consumed by the optimizer to prove alignment for Over / Join / GroupBy and eliminate cross-worker shuffles.

Two nil-versus-empty distinctions matter:

  • A nil *PartitionMetadata means "no claim made" — the optimizer assumes worst-case (any row can be anywhere) and inserts the shuffle it would insert today.
  • A non-nil pointer with Columns == nil is an explicit "no partitioning" claim (a source that ran without bucketing) and is distinct from nil. Alignment proofs refuse to prove anything against it. Never conflate the two.

func (*PartitionMetadata) Clone added in v0.2.0 

func (m *PartitionMetadata) Clone() *PartitionMetadata

Clone returns a deep copy of m. Used by plan-tree walkers that need to attach a modified metadata to a downstream node without mutating the upstream one. Returns nil for a nil receiver.

type PointExpr added in v0.2.16 

type PointExpr struct {
	Lat, Lon Expr
}

PointExpr bundles a Lat + Lon Expr pair for use with HaversineExpr. Kept as a named struct so the two coordinate components can't be swapped at the call site — a bug class that plagued the older four-argument positional signature (lat/lon vs lon/lat is a classic geospatial footgun).

PointExpr is a value type, not an ExprNode — it doesn't compose with .Add / .Sub / etc. Use it only where an API explicitly asks for a PointExpr; individual coordinate columns still use plain Expr elsewhere.

type PredicateSink 

type PredicateSink interface {
	LogicalPlan
	// ApplyPredicate returns a plan node identical to the receiver
	// but with pred available for row-group / bloom-filter skipping
	// at read time. The Filter node above the scan is not removed
	// (row-level filtering still runs after coarse skipping), so
	// callbacks may return the receiver unchanged when they decide
	// the hint isn't useful.
	ApplyPredicate(pred Expr) LogicalPlan
}

PredicateSink is implemented by scan nodes that can accept a filter-predicate hint from the optimizer. Rules test via type assertion; scans that don't implement it are left untouched by predicate pushdown.

type ProjectableScan 

type ProjectableScan interface {
	LogicalPlan
	// ProjectColumns returns a plan node identical to the receiver
	// but restricted to the given column names. If projection is
	// not beneficial (e.g. caller already set an explicit column
	// list), implementations should return the receiver unchanged.
	ProjectColumns(cols []string) LogicalPlan
}

ProjectableScan is implemented by scan nodes that can accept a column-projection hint from the optimizer. Rules test for this interface via type assertion; scans that don't implement it are left untouched by projection pushdown.

type Resampler 

type Resampler struct {
	// contains filtered or unexported fields
}

Resampler bins rows of a Frame by fixed time intervals aligned to the Unix epoch. Use Frame.ResampleEvery to construct one, then call Agg to produce a downsampled Frame.

Bucket alignment: bucketStart = floor(unix_ns / interval_ns) * interval_ns. A 1-hour resample produces buckets that start on the hour in UTC — regardless of any timezone label on the input series. If you need calendar-aligned resampling (e.g. daily buckets that start at midnight New York time), truncate to a calendar boundary first and group on that.

func (*Resampler) Agg 

func (r *Resampler) Agg(aggs ...Aggregation) (*Frame, error)

Agg computes the requested aggregations over each non-empty bucket. The output frame has one row per bucket, ordered by bucket start, with the time column named the same as the input's time column followed by one column per aggregation. Empty buckets are omitted (this mirrors GroupBy's behavior).

type Rule 

type Rule interface {
	Name() string
	Apply(plan LogicalPlan) (rewritten LogicalPlan, changed bool)
}

Rule is a single plan-tree rewrite. A rule is a pure function from a plan to an equivalent plan, potentially cheaper to execute. The optimizer applies rules to a fixed point — no rule reports a change on the final pass.

Rules should walk the whole tree themselves; the optimizer does not traverse for them. This lets a rule choose bottom-up or top-down order depending on what it does.

func DefaultRules 

func DefaultRules() []Rule

DefaultRules returns the rule set applied by Optimize when no explicit list is passed. Order is a best-effort ordering that enables downstream folds — FoldConstants can turn a filter's condition into a literal, which RemoveTrivialTrueFilter then eliminates. The optimizer runs to a fixed point, so any correct interleaving eventually converges.

type ScanOption 

type ScanOption func(*scanFileNode)

ScanOption configures optional capabilities on a scan node. Source packages (parquetio, csvio, ...) pass these to NewScanNode to declare features the optimizer can exploit — currently just column-projection pushdown.

func WithColumnProjection 

func WithColumnProjection(fn func(cols []string) LogicalPlan) ScanOption

WithColumnProjection registers a callback that returns a new scan node projected to the given column names. The optimizer's projection-pushdown rule calls this when it determines a scan source is being asked for a subset of its columns.

The callback should:

  • Return nil if projection doesn't apply (e.g. the caller already restricted columns explicitly). Treated as "no change" — the receiver stays in the plan.
  • Otherwise return a fresh scan node with the projection applied — the optimizer replaces the old node with what this callback returns.

cols is guaranteed non-empty and sorted; entries outside the scan's schema are the callback's responsibility to filter out.

func WithParallelStreamReads 

func WithParallelStreamReads(fn func() []func(cb func(*Frame) error) error) ScanOption

WithParallelStreamReads registers a callback that returns N stream-read closures, each processing a disjoint portion of the source. The Layer 6 executor spawns one producer goroutine per closure and fan-ins their batches into the downstream pipeline.

The callback is responsible for deciding N (typically bounded by GOMAXPROCS and by the source's natural partition count — e.g. number of parquet row-groups). Return nil (or a slice of length 0-1) to signal "no parallel version available"; the executor falls back to WithStreamRead in that case.

Semantics: batches from different workers arrive in unspecified order. Downstream operators that require order (Sort, Tail) buffer and reorder internally, so this is safe to enable for any plan shape.

func WithPartitionMetadata added in v0.2.0 

func WithPartitionMetadata(meta *PartitionMetadata) ScanOption

WithPartitionMetadata attaches a partition claim to a scan node. Sources that can prove partitioning (athenaio's UnloadAndRead after read-back verification, parquetio when per-file partition info is available, etc.) use this option to populate metadata the optimizer will consume in step 4 / step 6 of the v0.3.0 plan.

Nil meta is a no-op — same as not calling the option. Callers that want to explicitly claim "no partitioning" (distinct from "no claim made") pass a non-nil pointer with Columns == nil.

func WithPredicatePushdown 

func WithPredicatePushdown(fn func(pred Expr) LogicalPlan) ScanOption

WithPredicatePushdown registers a callback that returns a new scan node with a predicate baked in. The optimizer's PushPredicateToScan rule calls this when a Filter sits directly above the scan; the scan source is expected to use the predicate for row-group / bloom-filter skipping at read time.

The Filter node above the scan is NOT removed after pushdown — row-group skipping is coarse (whole groups only), so the row-level Filter still runs for correctness. Callbacks should treat the predicate as a hint, not a guarantee.

nil return = "no change" (same convention as WithColumnProjection).

func WithStreamRead 

func WithStreamRead(fn func(cb func(*Frame) error) error) ScanOption

WithStreamRead registers a streaming callback API for scan sources that support incremental read (parquetio.ReadFileChunksFunc, csvio.ReadFileChunksFunc). fn should call cb once per record batch and honor the batch-lifetime contract of the source package (typically: batches are Released after cb returns).

When present, the Layer 6 executor uses this callback for true bounded-memory streaming through the plan. When absent, the executor falls back to the `read` closure (whole-file materialize then batch — correct but not memory-bounded).

type Series 

type Series struct {
	// contains filtered or unexported fields
}

Series is a single named, typed Arrow column.

A Series does not own its column: constructors that receive an *arrow.Column treat it as borrowed. Callers wanting shared lifetime should Retain the underlying column; DataFrame.Release does not release Series contents.

func NewSeries 

func NewSeries(col *arrow.Column) Series

NewSeries returns a Series wrapping col. The Series takes the column's name from col.Name(); its field is derived from col.Field().

func NewTimestampSeries 

func NewTimestampSeries(name string, ts []time.Time, validity []bool) Series

NewTimestampSeries builds a Timestamp[ns] Series from the given time.Time values. Passing a nil validity slice means every row is valid; otherwise validity[i]==false marks row i as null (its value is ignored).

func PointsFromXY 

func PointsFromXY(x, y Series, crs int32) (Series, error)

PointsFromXY builds a geometry Series of 2D WKB Points from two coordinate columns. x and y must be numeric (Float64, Float32, Int64, or Int32) and the same length. Mixed-type inputs are promoted to Float64. Null values on either side emit a null geometry for that row.

The returned Series is a WKB Binary column tagged with geometry metadata + the given EPSG code, so it plugs directly into Frame.WithColumn, Frame.SJoin, GeoParquet write paths, and other geometry-aware operations.

Modeled on geopandas.points_from_xy — the intended flow is to build a geometry column from two attribute columns without hand-rolling the WKB encoding: 

lat, _ := df.Column("lat")
lng, _ := df.Column("lng")
geom, _ := gobi.PointsFromXY(lng, lat, 4326)   // x=lng, y=lat
df, _ = df.WithColumn("geometry", geom)

Note the argument order: x first, y second. In geographic coordinates that means longitude first, latitude second — matching GeoJSON / WKB / shapefile conventions (and geopandas).

func PointsFromXYZ 

func PointsFromXYZ(x, y, z Series, crs int32) (Series, error)

PointsFromXYZ is the 3D variant of PointsFromXY. z must be numeric and the same length as x and y; rows with a null z produce null geometries even if x and y are valid.

The resulting Point geometries carry HasZ=true so downstream WKB encoding emits XYZ type codes (1001..) rather than 2D (1..).

func SeriesFromArray added in v0.2.19 

func SeriesFromArray(field arrow.Field, arr arrow.Array) Series

SeriesFromArray wraps a freshly-built arrow.Array in a Series with the given field. Handles the full ref-count dance so callers don't have to hand-roll it:

  • `arr.Release()` is called inline after the wrapping. Callers transfer their ref on arr into this function and must NOT release it themselves. The Arrow Chunked constructed here retained arr internally, so arr's buffer survives on the Chunked-held reference.
  • The intermediate Chunked's constructor reference is also released here (NewColumn retained it separately). After return, only the Column-held reference to the Chunked is alive.

End state: `arr.refCount == 1` (owned by Chunked) and `chunked.refCount == 1` (owned by Column). No leaks, no double-frees.

Motivating shape: sibling packages that build custom column types (geometry columns, hash columns, ML feature columns) previously had to write the Chunked+NewColumn ceremony inline and remember the two `.Release()` calls that pair with NewChunked's and NewColumn's internal retains. This helper folds all of that into one function so downstream code becomes: 

arr := b.NewArray()
series := gobi.SeriesFromArray(
    gobi.GeometryField("points", 4326), arr)

Instead of the 5-line hand-rolled equivalent.

func (Series) Add 

func (s Series) Add(o Series) (Series, error)

Add returns s + o element-wise.

func (Series) AddDuration 

func (s Series) AddDuration(d time.Duration) (Series, error)

AddDuration returns a Timestamp Series with d added to each row. Output inherits the source column's TimeUnit and TimeZone — downstream operators reading the plan-declared schema stay in sync with the runtime column. Nulls propagate.

func (Series) AddScalar 

func (s Series) AddScalar(v float64) (Series, error)

AddScalar returns s + v element-wise (result is float64).

func (Series) Bools added in v0.2.4 

func (s Series) Bools() ([]bool, error)

Bools returns the Series values as []bool. Nulls come through as false.

func (Series) Column 

func (s Series) Column() *arrow.Column

Column returns the underlying Arrow column. Do not mutate the returned value; use Series methods for slicing.

func (Series) Concat 

func (s Series) Concat(others ...Series) (Series, error)

Concat returns a new Series with s's values followed by each other's values, in order. All series must share the same arrow type; type mismatches return an error naming both sides.

The result is a multi-chunk column — each input becomes a chunk. Zero-copy: no memcpy of the underlying arrow buffers.

func (Series) Count 

func (s Series) Count() int

Count returns the number of non-null rows.

func (Series) CountTrue added in v0.4.1 

func (s Series) CountTrue() (int, error)

CountTrue returns the number of true (and non-null) entries in a Boolean Series. The natural aggregation on filter masks — a user running `frame.FilterExpr(...)` and wanting to know how many rows survived can call `mask.CountTrue()` without materializing a full slice or paying the generic-aggregation path.

Non-Boolean input returns ErrNotBoolean. Null entries are counted as false (they aren't "true" — matches shapely / pandas convention where nulls don't satisfy predicates).

Uses compute.CountTrue per chunk; SIMD-friendly on the non-null hot path.

func (Series) DataType 

func (s Series) DataType() arrow.DataType

DataType returns the Arrow data type.

func (Series) DateFormat added in v0.3.6 

func (s Series) DateFormat(layout string) (Series, error)

DateFormat returns a String Series with each non-null value formatted using the given Go time layout (see time.Layout / time.RFC3339 constants). Empty layout defaults to RFC3339. Formatting uses the column's declared timezone.

func (Series) DateTruncate added in v0.3.6 

func (s Series) DateTruncate(unit string) (Series, error)

DateTruncate returns a Timestamp Series with each non-null value truncated down to the start of the named unit. Supported units: "year", "month", "day", "hour", "minute", "second". Calendar- aware for year/month/day (start-of-year at 00:00:00, etc.); wall-clock aligned for hour/minute/second.

The output preserves the source column's TimeUnit and TimeZone — truncation happens in the value's local timezone, so DateTruncate("day") on a US/Eastern-tagged timestamp gives local-midnight, not UTC-midnight.

func (Series) Day 

func (s Series) Day() (Series, error)

Day returns each row's day-of-month [1..31] as Int64.

func (Series) DayOfYear 

func (s Series) DayOfYear() (Series, error)

DayOfYear returns each row's day-of-year [1..366] as Int64.

func (Series) Diff 

func (s Series) Diff(n int) (Series, error)

Diff returns a Series holding the element-wise difference between s and its n-lagged self: `out[i] = s[i] - s[i-n]`. The first n positions are null (no previous value to subtract). n must be positive; n=0 is disallowed (would trivially be all zeros); use Shift(-n) if you want a negative-lag first-difference.

Only numeric types are supported (matching the underlying Series.Sub). Non-numeric callers should compose Shift + a domain-specific comparison instead.

This is the columnar equivalent of pandas / polars `diff(n)` used for period-over-period changes, discrete derivatives, and change-point detection.

func (Series) DiffDuration 

func (s Series) DiffDuration(other Series) (Series, error)

DiffDuration returns an Int64 Series whose row i is (s[i] - other[i]) expressed in nanoseconds. Nulls on either side produce nulls in the output.

func (Series) Difference 

func (s Series) Difference(other Series) (Series, error)

Difference returns the distinct non-null values in s that are absent from other, in first-occurrence order. Types must match.

func (Series) Div 

func (s Series) Div(o Series) (Series, error)

Div returns s / o element-wise, promoting to float64. Division by zero yields ±Inf or NaN per IEEE 754.

func (Series) DivScalar 

func (s Series) DivScalar(v float64) (Series, error)

DivScalar returns s / v element-wise.

func (Series) Eq 

func (s Series) Eq(o Series) (Series, error)

Eq returns a boolean Series that is true where s[i] == o[i].

func (Series) EqScalar 

func (s Series) EqScalar(v float64) (Series, error)

EqScalar returns a boolean Series true where s[i] == v.

func (Series) EqTime 

func (s Series) EqTime(t time.Time) (Series, error)

EqTime returns a Boolean Series true where s[i] == t.

func (Series) Float32Values added in v0.2.9 

func (s Series) Float32Values() ([]float32, bool)

Float32Values returns a zero-copy view of the underlying float32 slice when s is a single-chunk Float32 Series. Returns (nil, false) on multi-chunk or type mismatch.

func (Series) Float32s added in v0.2.4 

func (s Series) Float32s() ([]float32, error)

Float32s returns the Series values as []float32. Nulls come through as 0.

func (Series) Float64Values added in v0.2.9 

func (s Series) Float64Values() ([]float64, bool)

Float64Values returns a zero-copy view of the underlying float64 slice when s is a single-chunk Float64 Series. Returns (nil, false) on multi-chunk or type mismatch. Null rows still occupy positions in the slice — combine with Nulls() to filter them.

func (Series) Float64s added in v0.2.4 

func (s Series) Float64s() ([]float64, error)

Float64s returns the Series values as []float64. Nulls come through as 0.

func (Series) Ge 

func (s Series) Ge(o Series) (Series, error)

Ge returns a boolean Series that is true where s[i] >= o[i].

func (Series) GeScalar added in v0.3.8 

func (s Series) GeScalar(v float64) (Series, error)

GeScalar returns a boolean Series true where s[i] >= v.

func (Series) GeTime 

func (s Series) GeTime(t time.Time) (Series, error)

GeTime returns a Boolean Series true where s[i] >= t.

func (Series) GeomArea 

func (s Series) GeomArea(u geometry.Unit) (Series, error)

GeomArea returns a Float64 Series holding the planar (XY) area of each geometry in s, in u². Non-polygonal geometries contribute 0. Null geometries produce null values.

For projected CRSes with a meter-based linear unit (the geoparquet default), this dispatches to geometry.PlanarAreaFromWKB — a byte-stream shoelace scanner that skips the ParseWKB alloc. Other CRSes (geographic, or projected with a non-meter linear unit) fall through to the AoS path.

func (Series) GeomBounds 

func (s Series) GeomBounds() (*Frame, error)

GeomBounds returns a Frame with four Float64 columns — MinX, MinY, MaxX, MaxY — one row per input geometry. Null geometries produce four nulls.

SoA fast path (Slice 14): dispatches to `geometry.BoundsFromWKB` unconditionally — bbox semantics match `g.Bounds()` exactly for every geometry type, so no per-shape dispatch is needed.

func (Series) GeomBuffer added in v0.3.0 

func (s Series) GeomBuffer(distance float64, opts geometry.BufferOptions) (Series, error)

GeomBuffer returns a geometry Series holding a buffered version of each row. The distance is in the CRS's linear unit (meters for UTM, degrees for WGS84); the opts field controls smoothness (Segments) and shape (Style: BufferRound vs BufferSquare). Null rows pass through as null.

func (Series) GeomCentroid 

func (s Series) GeomCentroid() (Series, error)

GeomCentroid returns a geometry Series holding the centroid of each input geometry as a Point (encoded as WKB). Null inputs produce nulls. The output column inherits s's CRS.

SoA fast path (Slice 14)

For Point / LineString / Polygon / MultiPoint / MultiLineString / GeometryCollection rows this dispatches to `geometry.CentroidFromWKB` — a byte-stream centroid scanner that skips the ParseWKB alloc. MultiPolygon rows fall back to AoS ParseWKB → Centroid because `CentroidFromWKB` uses bbox-center for MultiPolygons vs the AoS's area-weighted definition. Preserving the AoS semantic keeps `GeomCentroid` behavior stable for existing users.

func (Series) GeomCircleContains added in v0.3.1 

func (s Series) GeomCircleContains(c geometry.Circle) (Series, error)

GeomCircleContains returns a Boolean Series where row i is true if the row's geometry is inside c. Points are tested directly; other geometry types are tested via their Centroid. Null rows produce null. Circle units follow c.Center.CRSValue — reproject the input (via GeomToCRS) to the same CRS before calling if they differ.

func (Series) GeomClip added in v0.3.0 

func (s Series) GeomClip(mask geometry.Geometry) (Series, error)

GeomClip returns a geometry Series holding the intersection of each geometry in s with the mask. Null inputs pass through as null. Both s and mask must be in a projected CRS (or unset), and mask must be a Polygon or MultiPolygon; other geometry types return an error.

func (Series) GeomContains added in v0.3.0 

func (s Series) GeomContains(other geometry.Geometry) (Series, error)

GeomContains returns a Boolean Series where row i is true if the geometry at s[i] fully contains other. Null rows produce null output. Matches geopandas's GeoSeries.contains(other) semantics.

func (Series) GeomConvexHull added in v0.3.0 

func (s Series) GeomConvexHull() (Series, error)

GeomConvexHull returns a geometry Series where each row is the convex hull of the input row's vertices (as a Polygon). Rows with fewer than 3 unique vertices produce an empty-ring Polygon. Null rows pass through as null.

func (Series) GeomCrosses added in v0.3.0 

func (s Series) GeomCrosses(other geometry.Geometry) (Series, error)

GeomCrosses returns a Boolean Series where row i is true if row i's geometry crosses other — typically LineString × Polygon or LineString × LineString with mixed dimensions. Matches geopandas's GeoSeries.crosses(other) semantics.

func (Series) GeomCrossesAntimeridian added in v0.3.0 

func (s Series) GeomCrossesAntimeridian() (Series, error)

GeomCrossesAntimeridian returns a Boolean Series where row i is true if row i's geometry has any adjacent-vertex pair with |Δlon| > 180°, i.e. the edge between them wraps around the ±180° meridian. Only meaningful for geographic-CRS inputs; projected-CRS series always return false per row. Null rows produce null.

func (Series) GeomDWithin added in v0.3.5 

func (s Series) GeomDWithin(other geometry.Geometry, distance float64) (Series, error)

GeomDWithin returns a Boolean Series where row i is true if the geometry at s[i] is within `distance` coordinate units of other. The killer spatial-join operator — "rows whose geometry is at most 5km from this line" or "polygons within 100m of this AOI" map straight to this call.

distance is measured in the column's coordinate units — meters for projected CRSes (UTM, PseudoMercator), degrees for lon/lat. For lon/lat data, project to a suitable CRS first (see GeomToCRS) so distance comparisons stay meaningful.

distance = 0 degenerates to GeomIntersects. Negative or NaN distance yields all-false (matches the geometry.WithinDistance contract). Null rows produce null output.

Under the hood, the bbox short-circuit in WithinDistance skips polygons whose bounding rectangle is already farther than distance from other's bbox — no WKB decoding of the far rows' interiors. Combined with the v0.3.4 row-group pushdown, this is what makes a "roads within 100m" query over a million-row parquet finish in tens of milliseconds instead of seconds.

func (Series) GeomDensifyGeodesic added in v0.3.2 

func (s Series) GeomDensifyGeodesic(stepMeters float64) (Series, error)

GeomDensifyGeodesic replaces each row's LineString with its great-circle densification at ≤ stepMeters spacing (see geometry.DensifyGeodesic). Rows carrying non-LineString geometry pass through unchanged. Requires the Series' CRS metadata to be geographic (or unset — treated as WGS84); a projected CRS returns ErrGeodesicRequiresGeographic without inspecting per-row values.

Null rows pass through as null.

func (Series) GeomDifference added in v0.3.0 

func (s Series) GeomDifference(other geometry.Geometry) (Series, error)

GeomDifference returns a geometry Series holding s.geom minus other for each row. See GeomClip for input requirements.

func (Series) GeomDisjoint added in v0.3.0 

func (s Series) GeomDisjoint(other geometry.Geometry) (Series, error)

GeomDisjoint returns a Boolean Series where row i is true if the geometry at s[i] shares no point with other. This is !Intersects. Matches geopandas's GeoSeries.disjoint(other) semantics.

func (Series) GeomDissolve added in v0.3.0 

func (s Series) GeomDissolve() (geometry.Geometry, error)

GeomDissolve returns a single Geometry containing the union of every geometry in s. Rows with null geometry are skipped. Returns an empty Polygon when the series is empty or all rows are null.

func (Series) GeomDistance added in v0.3.0 

func (s Series) GeomDistance(other geometry.Geometry, u geometry.Unit) (Series, error)

GeomDistance returns a Float64 Series where row i is the minimum planar (Euclidean) distance from row i's geometry to other, in the requested unit. Returns 0 for intersecting geometries. Null inputs produce null outputs.

Coordinates are treated as planar meters; for geographic CRSes project via GeomToCRS(WGS84 UTM) first or the result is Euclidean on lon/lat degrees, which is meaningless.

Slice 13 SoA fast path

For projected CRSes with meter-based linear units this dispatches to `geometry.PlanarMinDistanceFromWKB` when the row's bbox is disjoint from `other`'s bbox (the common case — most rows in a distance-scan don't overlap the target). Bbox disjoint means **definitely non-intersecting**, so the SoA min-distance kernel produces the correct answer without a segment-segment intersects check.

Rows whose bboxes DO overlap `other` fall through to the AoS `geometry.GeomDistance` — those need the full intersects check to correctly return 0 on overlapping geometries. Geographic CRSes (haversine required) also fall back to AoS.

func (Series) GeomDistanceToCircle added in v0.3.1 

func (s Series) GeomDistanceToCircle(c geometry.Circle, u geometry.Unit) (Series, error)

GeomDistanceToCircle returns a Float64 Series with the SIGNED distance from each row's geometry (Point directly, otherwise its centroid) to c's boundary, in the requested unit. Negative when the point is inside the circle, positive outside, zero on the boundary. Null rows produce null.

Distance is Euclidean in the coordinate plane's linear unit. For geographic-CRS input this is degrees × <unit conversion> — meaningless in physical distance terms. Project to a projected CRS (GeomToCRS) first for meters.

func (Series) GeomEllipseContains added in v0.3.3 

func (s Series) GeomEllipseContains(e geometry.Ellipse) (Series, error)

GeomEllipseContains returns a Boolean Series where row i is true if the row's geometry is inside e. Points are tested directly; other geometry types are tested via their Centroid. Null rows pass through as null. Ellipse coordinates follow e.Center.CRSValue — reproject the input (GeomToCRS) to the same CRS before calling if they differ.

func (Series) GeomEnvelope added in v0.3.0 

func (s Series) GeomEnvelope() (Series, error)

GeomEnvelope returns a geometry Series where each row is the axis-aligned bounding-box polygon of the input row. Matches geopandas's GeoSeries.envelope. Different from GeomBounds, which returns a 4-column Frame of MinX/MinY/MaxX/MaxY floats.

func (Series) GeomEstimateUTMCRS added in v0.3.0 

func (s Series) GeomEstimateUTMCRS() (geometry.CRS, error)

GeomEstimateUTMCRS returns the WGS 84 / UTM zone CRS covering the total bounds of every non-null geometry in s. Matches geopandas's GeoDataFrame.estimate_utm_crs() by aggregating over the collection rather than per-row. Returns ErrEmptyGeometry if every row is null.

func (Series) GeomFitCircle added in v0.3.1 

func (s Series) GeomFitCircle(opts geometry.CircleFitOptions) (geometry.Circle, error)

GeomFitCircle fits a Circle across every non-null Point row (or centroid of non-Point rows) in s via least squares. Errors if fewer than 3 non-null rows are present or the input is collinear-degenerate. Uses Taubin by default (see geometry.FitCircle).

func (Series) GeomIntersects added in v0.3.0 

func (s Series) GeomIntersects(other geometry.Geometry) (Series, error)

GeomIntersects returns a Boolean Series where row i is true if the geometry at s[i] shares any point with other. Null rows produce null output. Matches geopandas's GeoSeries.intersects(other) semantics.

Uses geometry.Intersects, which short-circuits on disjoint bboxes, so on a Series with many rows disjoint from other the loop is effectively linear in row count with O(1) work per row.

func (Series) GeomIsEmpty added in v0.3.0 

func (s Series) GeomIsEmpty() (Series, error)

GeomIsEmpty returns a Boolean Series where row i is true if the geometry at s[i] has no coordinates (empty ring, empty collection, LineString with fewer than 2 points). Matches shapely's .is_empty.

func (Series) GeomIsValid added in v0.3.0 

func (s Series) GeomIsValid() (Series, error)

GeomIsValid returns a Boolean Series where row i is true if the geometry at s[i] passes gobi's structural validity checks (see geometry.IsValid).

func (Series) GeomLength 

func (s Series) GeomLength(u geometry.Unit) (Series, error)

GeomLength returns a Float64 Series holding the planar (XY) length of each geometry in u. Non-linear geometries contribute 0. Null geometries produce null values.

For projected CRSes this dispatches to geometry.PlanarLengthFromWKB — a byte-stream Euclidean-segment scanner that skips the ParseWKB alloc. Geographic CRSes fall through to the AoS haversine path.

func (Series) GeomOverlaps added in v0.3.0 

func (s Series) GeomOverlaps(other geometry.Geometry) (Series, error)

GeomOverlaps returns a Boolean Series where row i is true if row i's geometry shares interior points with other but neither contains the other, and both are of the same dimension. Matches geopandas's GeoSeries.overlaps(other) semantics.

func (Series) GeomSimplify added in v0.3.0 

func (s Series) GeomSimplify(tolerance float64) (Series, error)

GeomSimplify returns a geometry Series with each row simplified via Douglas-Peucker at the given tolerance. Tolerance is in the CRS's linear unit — vertices within `tolerance` of a straight line between their neighbors are removed. Null rows pass through as null.

func (Series) GeomSplitAtAntimeridian added in v0.3.0 

func (s Series) GeomSplitAtAntimeridian() (Series, error)

GeomSplitAtAntimeridian returns a geometry Series where every antimeridian-crossing row is replaced by its split components (Polygon → MultiPolygon, LineString → MultiLineString). Non-crossing rows pass through unchanged. Points always pass through. Nulls stay null. See geometry.SplitAtAntimeridian for the crossing detection and interpolation semantics.

func (Series) GeomSymDifference added in v0.3.0 

func (s Series) GeomSymDifference(other geometry.Geometry) (Series, error)

GeomSymDifference returns a geometry Series holding the symmetric difference of each row with other. See GeomClip for input requirements.

func (Series) GeomToCRS added in v0.3.0 

func (s Series) GeomToCRS(target geometry.CRS) (Series, error)

GeomToCRS reprojects every geometry in s into target, returning a new Series whose field is tagged with target's EPSG code. Null rows pass through as null. Both this and geometry.Project must know how to bridge the source and target CRSes (currently WGS84 ↔ PseudoMercator ↔ UTM).

func (Series) GeomTouches added in v0.3.0 

func (s Series) GeomTouches(other geometry.Geometry) (Series, error)

GeomTouches returns a Boolean Series where row i is true if row i's geometry shares boundary points with other but no interior points. Matches geopandas's GeoSeries.touches(other) semantics.

func (Series) GeomType added in v0.3.0 

func (s Series) GeomType() (Series, error)

GeomType returns a String Series where row i is the OGC-style type name of row i's geometry ("Point", "MultiPolygon", etc.). Matches shapely's .geom_type.

SoA fast path (Slice 15): peeks the WKB type code via `geometry.WKBTypeCode` (zero-alloc, reads only the 5-byte header) and maps it to a string via `geometry.Type.String()`. No ParseWKB, no `[]Point` materialization.

func (Series) GeomUnion added in v0.3.0 

func (s Series) GeomUnion(other geometry.Geometry) (Series, error)

GeomUnion returns a geometry Series holding the union of each row with other. See GeomClip for input requirements.

func (Series) GeomWithin added in v0.3.0 

func (s Series) GeomWithin(other geometry.Geometry) (Series, error)

GeomWithin returns a Boolean Series where row i is true if the geometry at s[i] lies fully within other. Matches geopandas's GeoSeries.within(other) semantics. Equivalent to other.Contains(s[i]), so we swap the argument order in the per-row call.

func (Series) Geometry 

func (s Series) Geometry(i int) (geometry.Geometry, error)

Geometry decodes the WKB at row i into a Geometry. Returns ErrNotGeometry if the series is not a geometry column.

func (Series) Gt 

func (s Series) Gt(o Series) (Series, error)

Gt returns a boolean Series that is true where s[i] > o[i].

func (Series) GtScalar 

func (s Series) GtScalar(v float64) (Series, error)

GtScalar returns a boolean Series true where s[i] > v.

func (Series) GtTime 

func (s Series) GtTime(t time.Time) (Series, error)

GtTime returns a Boolean Series true where s[i] > t.

func (Series) HasNulls added in v0.2.9 

func (s Series) HasNulls() bool

HasNulls reports whether the Series has any null rows. Cheap — consults arrow's cached NullN metadata per chunk without touching the bitmap.

func (Series) Head 

func (s Series) Head(n int) Series

Head returns a Series with the first n rows. n<=0 means default 5.

func (Series) Hour 

func (s Series) Hour() (Series, error)

Hour returns each row's hour-of-day [0..23] as Int64 (UTC).

func (Series) Int32Values added in v0.2.9 

func (s Series) Int32Values() ([]int32, bool)

Int32Values returns a zero-copy view of the underlying int32 slice when s is a single-chunk Int32 Series. Returns (nil, false) on multi-chunk or type mismatch.

func (Series) Int32s added in v0.2.4 

func (s Series) Int32s() ([]int32, error)

Int32s returns the Series values as []int32. Nulls come through as 0.

func (Series) Int64Values added in v0.2.9 

func (s Series) Int64Values() ([]int64, bool)

Int64Values returns a zero-copy view of the underlying int64 slice when s is a single-chunk Int64 Series. Returns (nil, false) on multi-chunk or type mismatch.

func (Series) Int64s added in v0.2.4 

func (s Series) Int64s() ([]int64, error)

Int64s returns the Series values as []int64. Nulls come through as 0.

func (Series) Intersect 

func (s Series) Intersect(other Series) (Series, error)

Intersect returns the distinct non-null values present in both s and other, in first-occurrence order. Types must match.

func (Series) IsDateTime 

func (s Series) IsDateTime() bool

IsDateTime reports whether s is a Timestamp / Date32 / Date64 column.

func (Series) IsGeometry 

func (s Series) IsGeometry() bool

IsGeometry reports whether the series is tagged as a WKB geometry column.

func (Series) Le 

func (s Series) Le(o Series) (Series, error)

Le returns a boolean Series that is true where s[i] <= o[i].

func (Series) LeScalar added in v0.3.8 

func (s Series) LeScalar(v float64) (Series, error)

LeScalar returns a boolean Series true where s[i] <= v.

func (Series) LeTime 

func (s Series) LeTime(t time.Time) (Series, error)

LeTime returns a Boolean Series true where s[i] <= t.

func (Series) Len 

func (s Series) Len() int

Len returns the number of rows.

func (Series) Lt 

func (s Series) Lt(o Series) (Series, error)

Lt returns a boolean Series that is true where s[i] < o[i].

func (Series) LtScalar 

func (s Series) LtScalar(v float64) (Series, error)

LtScalar returns a boolean Series true where s[i] < v.

func (Series) LtTime 

func (s Series) LtTime(t time.Time) (Series, error)

LtTime returns a Boolean Series true where s[i] < t.

func (Series) Max 

func (s Series) Max() (float64, error)

Max returns the maximum non-null numeric value, or NaN when the series has no non-null rows.

func (Series) Mean 

func (s Series) Mean() (float64, error)

Mean returns the arithmetic mean of non-null numeric values, or NaN when the series has no non-null rows.

func (Series) Min 

func (s Series) Min() (float64, error)

Min returns the minimum non-null numeric value, or NaN when the series has no non-null rows.

func (Series) Minute 

func (s Series) Minute() (Series, error)

Minute returns each row's minute-of-hour [0..59] as Int64.

func (Series) Month 

func (s Series) Month() (Series, error)

Month returns each row's calendar month [1..12] as Int64.

func (Series) Mul 

func (s Series) Mul(o Series) (Series, error)

Mul returns s * o element-wise.

func (Series) MulScalar 

func (s Series) MulScalar(v float64) (Series, error)

MulScalar returns s * v element-wise.

func (Series) NUnique 

func (s Series) NUnique() (int64, error)

NUnique returns the count of distinct non-null values in s.

Equivalent to `s.Unique().Len()` but skips the arrow array construction. Uses the same byte-encoding as GroupBy so numeric bit-equal values collapse identically. O(n) time, O(distinct) memory.

func (Series) Name 

func (s Series) Name() string

Name returns the column name.

func (Series) Nanosecond added in v0.3.6 

func (s Series) Nanosecond() (Series, error)

Nanosecond returns each row's sub-second nanosecond component [0, 999_999_999] as Int64. Distinct from the raw underlying Timestamp value: this pulls the fractional-second part after converting to time.Time in the column's declared timezone.

func (Series) Ne 

func (s Series) Ne(o Series) (Series, error)

Ne returns a boolean Series that is true where s[i] != o[i].

func (Series) NeScalar added in v0.3.8 

func (s Series) NeScalar(v float64) (Series, error)

NeScalar returns a boolean Series true where s[i] != v.

func (Series) NeTime 

func (s Series) NeTime(t time.Time) (Series, error)

NeTime returns a Boolean Series true where s[i] != t.

func (Series) NullCount added in v0.2.9 

func (s Series) NullCount() int

NullCount returns the total number of null rows across all chunks. Cheap — sums each chunk's cached NullN.

func (Series) Nulls added in v0.2.4 

func (s Series) Nulls() []bool

Nulls returns a []bool where true means "row i is null". Zero-length series → empty slice, no error. Use alongside a value extractor to distinguish arrow-null from a zero-valued row. 

vals, _ := s.Int64s()
nulls := s.Nulls()
for i, v := range vals {
    if nulls[i] { continue } // skip nulls
    // ... use v
}

Fast path: chunks with no nulls (per arrow's cached NullN) skip the bitmap walk entirely — leave those output slots at their zero value. Chunks with nulls read the raw validity bitmap once per row rather than the (offset-recomputing) IsNull call. Fully-null Series still allocate the []bool.

func (Series) RollingCount 

func (s Series) RollingCount(window int) (Series, error)

RollingCount returns an Int64 Series holding the count of non-null values in each window. Rows before the first full window are null.

func (Series) RollingMax 

func (s Series) RollingMax(window int) (Series, error)

RollingMax returns a Float64 Series holding the maximum non-null value over each window.

func (Series) RollingMean 

func (s Series) RollingMean(window int) (Series, error)

RollingMean returns a Float64 Series holding the arithmetic mean of s over each window (non-null values only).

func (Series) RollingMin 

func (s Series) RollingMin(window int) (Series, error)

RollingMin returns a Float64 Series holding the minimum non-null value over each window.

func (Series) RollingSum 

func (s Series) RollingSum(window int) (Series, error)

RollingSum returns a Float64 Series where row i is the sum of s over the closed window [i-window+1 .. i]. window must be >= 1.

func (Series) Row 

func (s Series) Row(i int) (Series, error)

Row returns a Series containing the single row at index i.

func (Series) Second 

func (s Series) Second() (Series, error)

Second returns each row's second-of-minute [0..59] as Int64.

func (Series) Shift 

func (s Series) Shift(n int) (Series, error)

Shift returns a new Series with values shifted by n positions. Positive n shifts values forward (down): the first n positions become null, and each output position i gets the source value from position i-n when that index is in bounds.

Negative n shifts backward (up): the last |n| positions become null; output position i gets the source value from position i+|n|.

n=0 is a no-op that still returns a fresh Series (new arrow array, same values). Extreme |n| >= Len() returns an all-null Series of the same type + name — same as pandas / polars.

Nulls in the source at the source index propagate to the output. The output preserves the source's arrow type, name, and field metadata (including the geometry tag, if any).

func (Series) StrConcat added in v0.3.6 

func (s Series) StrConcat(suffix string) (Series, error)

StrConcat returns a copy of s where every value is followed by the same `suffix` string. Pairwise concat against another String column isn't currently exposed — build it explicitly via a Custom ExprNode or a Frame.WithColumn expression until the pairwise variant lands (v0.3.7 candidate).

func (Series) StrContains added in v0.3.6 

func (s Series) StrContains(substr string) (Series, error)

StrContains returns a Boolean column: true when the row's value contains `substr`, false otherwise. Case-sensitive.

func (Series) StrEndsWith added in v0.3.6 

func (s Series) StrEndsWith(suffix string) (Series, error)

StrEndsWith returns a Boolean column: true when the row's value ends with `suffix`. Case-sensitive.

func (Series) StrLen added in v0.3.6 

func (s Series) StrLen() (Series, error)

StrLen returns an Int64 column with the Unicode-codepoint count of every value. Distinct from byte length: a 3-character emoji string returns 3, not the number of bytes. Use `.Cast(Int64)` after a byte-length op if you need bytes.

func (Series) StrLower added in v0.3.6 

func (s Series) StrLower() (Series, error)

StrLower returns a copy of s with every value lowercased. Unicode-aware (delegates to strings.ToLower).

func (Series) StrRegexMatch added in v0.3.6 

func (s Series) StrRegexMatch(pattern string) (Series, error)

StrRegexMatch returns a Boolean column: true when the row's value matches `pattern` (RE2 syntax, matches regexp.MatchString semantics — searches for the pattern anywhere in the value). Anchor with ^ / $ for full-string match. The pattern is compiled once inside this call.

Streaming pipelines that filter with the same regex across many batches should prefer Expr.StrRegexMatch — that path compiles the pattern once at Expr-build time and reuses the compiled *regexp.Regexp across every batch's Eval.

func (Series) StrRegexReplace added in v0.3.6 

func (s Series) StrRegexReplace(pattern, replacement string) (Series, error)

StrRegexReplace returns a copy of s with every match of `pattern` replaced by `replacement` in every value. Uses regexp.ReplaceAllString semantics — `replacement` may reference capture groups via $1 / $2 / ${name}. Pattern compiled once inside this call; use Expr.StrRegexReplace for streaming pipelines that need across-batch caching.

func (Series) StrReplace added in v0.3.6 

func (s Series) StrReplace(find, replacement string) (Series, error)

StrReplace returns a copy of s with every occurrence of `find` replaced by `replacement` in every value. Literal (non-regex) replacement. Empty `find` inserts `replacement` between every pair of characters (matches strings.ReplaceAll semantics).

func (Series) StrSlice added in v0.3.6 

func (s Series) StrSlice(start, end int) (Series, error)

StrSlice returns a substring of each value by Unicode-codepoint index. Negative `start` or `end` counts from the right (Python convention: -1 is the last codepoint). end == 0 means "to the end of the string." Out-of-range indices clamp to the string's extent — no panics.

Example: "hello".StrSlice(1, 4) → "ell".

func (Series) StrStartsWith added in v0.3.6 

func (s Series) StrStartsWith(prefix string) (Series, error)

StrStartsWith returns a Boolean column: true when the row's value begins with `prefix`. Case-sensitive.

func (Series) StrTrim added in v0.3.6 

func (s Series) StrTrim() (Series, error)

StrTrim returns a copy of s with leading and trailing whitespace removed from every value (Unicode-aware, matches strings.TrimSpace).

func (Series) StrTrimLeft added in v0.3.6 

func (s Series) StrTrimLeft(cutset string) (Series, error)

StrTrimLeft returns a copy of s with any leading Unicode code points in `cutset` removed from every value.

func (Series) StrTrimRight added in v0.3.6 

func (s Series) StrTrimRight(cutset string) (Series, error)

StrTrimRight returns a copy of s with any trailing Unicode code points in `cutset` removed from every value.

func (Series) StrUpper added in v0.3.6 

func (s Series) StrUpper() (Series, error)

StrUpper returns a copy of s with every value uppercased. Unicode-aware (delegates to strings.ToUpper).

func (Series) Strings added in v0.2.4 

func (s Series) Strings() ([]string, error)

Strings returns the Series values as []string. Nulls come through as "". Works for both String and LargeString columns.

func (Series) Sub 

func (s Series) Sub(o Series) (Series, error)

Sub returns s - o element-wise.

func (Series) SubDuration 

func (s Series) SubDuration(d time.Duration) (Series, error)

SubDuration returns a Timestamp Series with d subtracted from each row. Same TimeUnit / TimeZone preservation as AddDuration.

func (Series) SubScalar 

func (s Series) SubScalar(v float64) (Series, error)

SubScalar returns s - v element-wise.

func (Series) Sum 

func (s Series) Sum() (float64, error)

Sum returns the sum of non-null numeric values.

func (Series) Tail 

func (s Series) Tail(n int) Series

Tail returns a Series with the last n rows. n<=0 means default 5.

func (Series) TimeAt 

func (s Series) TimeAt(i int) (time.Time, bool, error)

TimeAt returns the value at row i as a time.Time, plus a validity flag (false for null). The returned time.Time carries the series' timezone (UTC if the series is tz-naive). Errors for non-datetime series.

func (Series) Timestamps added in v0.2.4 

func (s Series) Timestamps() ([]arrow.Timestamp, error)

Timestamps returns the Series values as []arrow.Timestamp (int64 under the hood). Nulls come through as 0. The unit and timezone are preserved in the source column's arrow type; use DataType() to recover them if you need to interpret the values.

func (Series) Timezone 

func (s Series) Timezone() string

Timezone returns the IANA timezone label on s (e.g. "America/New_York"), or "" if the series is tz-naive or not a Timestamp column.

func (Series) TruncateTo 

func (s Series) TruncateTo(unit TimeUnit) (Series, error)

TruncateTo returns a new Timestamp[ns] Series with each row rounded down to the nearest boundary of the given unit. UnitDay truncates to 00:00:00 UTC of the same date.

func (Series) TruncateToCalendar 

func (s Series) TruncateToCalendar(unit CalendarUnit) (Series, error)

TruncateToCalendar truncates each row to a calendar boundary (week, month, year). If s carries a timezone, the boundary is computed in that timezone (so "start of month" is midnight local time in the tz, not UTC). Week uses ISO semantics: previous Monday at 00:00.

func (Series) Uint32Values added in v0.2.9 

func (s Series) Uint32Values() ([]uint32, bool)

Uint32Values returns a zero-copy view of the underlying uint32 slice when s is a single-chunk Uint32 Series.

func (Series) Uint32s added in v0.2.4 

func (s Series) Uint32s() ([]uint32, error)

Uint32s returns the Series values as []uint32. Nulls come through as 0.

func (Series) Uint64Values added in v0.2.9 

func (s Series) Uint64Values() ([]uint64, bool)

Uint64Values returns a zero-copy view of the underlying uint64 slice when s is a single-chunk Uint64 Series.

func (Series) Uint64s added in v0.2.4 

func (s Series) Uint64s() ([]uint64, error)

Uint64s returns the Series values as []uint64. Nulls come through as 0.

func (Series) Union 

func (s Series) Union(other Series) (Series, error)

Union returns the distinct non-null values in either s or other, in first-occurrence order. Types must match. Nulls are treated as equal for set membership — matching Frame.Union — but the null value itself is not emitted in the output (parity with Series.Unique which drops nulls).

func (Series) Unique 

func (s Series) Unique() (Series, error)

Unique returns a Series of distinct non-null values in first- occurrence order. The result's arrow type matches the input's.

Nulls are dropped (matches pandas / polars `unique` semantics). Callers that want a "did I see any null?" signal should combine with `Series.NullCount()`.

func (Series) Weekday 

func (s Series) Weekday() (Series, error)

Weekday returns each row's day-of-week as Int64 with Sunday=0..Saturday=6 (matches Go's time.Weekday int values).

func (Series) WithTimezone 

func (s Series) WithTimezone(tz string) (Series, error)

WithTimezone returns a copy of s that carries the given IANA timezone label. The underlying Unix-nanosecond values are NOT shifted — only the display timezone changes. This matches pandas' `tz_convert` when the input is already tz-aware, or "assume the values are absolute instants and now render them in tz" when the input is naive.

Passing tz == "" strips any existing label (result is tz-naive). Passing a name that time.LoadLocation cannot resolve returns an error.

func (Series) Year 

func (s Series) Year() (Series, error)

Year returns each row's calendar year as an Int64 Series (UTC).

type SortKey 

type SortKey struct {
	Column     string
	Descending bool
}

SortKey names a column to sort by and its direction. Compose multiple SortKeys in a single SortBy call for lexicographic (a-then-b-then-c) ordering — earlier keys have priority; later keys break ties.

Nulls sort last regardless of Descending, matching pandas / polars default behavior. NaN floats also sort last (numpy semantics).

type SpatialPredicate 

type SpatialPredicate uint8

SpatialPredicate names a binary spatial predicate for SJoin. 

const (
	// SPIntersects matches when the left and right geometries share any point.
	SPIntersects SpatialPredicate = iota
	// SPContains matches when the left geometry fully contains the right.
	SPContains
	// SPWithin matches when the left geometry lies fully within the right.
	SPWithin
)

func (SpatialPredicate) String 

func (p SpatialPredicate) String() string

type Stats 

type Stats interface {
	// MinMax returns the inclusive bounds for col over the range
	// this Stats describes. ok=false when statistics are missing.
	MinMax(col string) (minV, maxV any, ok bool)
	// NullCount returns the number of null values in col over the
	// range. ok=false when null counts weren't recorded.
	NullCount(col string) (n int64, ok bool)
	// TotalRows is the total row count of the range.
	TotalRows() int64
}

Stats reports column-level bounds used by CanPossiblyMatch to prove predicates unsatisfiable over a data range (typically a parquet row-group). Implementations are supplied by source packages: parquetio, csvio, etc.

The values returned by MinMax must be Go-typed scalars matching the column's arrow type: int64 for INT64, float64 for FLOAT64, string for STRING, bool for BOOL, and so on. Mixed types or unknown columns should signal ok=false so the caller falls back to the conservative "possibly matches" answer.

type StructOption added in v0.3.0 

type StructOption func(*structOpts)

StructOption tunes FromStructs / ToStructs behavior. Passed as variadic arguments to keep the call site clean when no options are needed.

func StructTagFormat added in v0.3.0 

func StructTagFormat(format string) StructOption

StructTagFormat sets the primary struct-tag namespace to consult when resolving column names and per-field options. If a field has no tag under this namespace, resolution falls back through: 

gobi:"..."  — universal namespace, works for every io
csv:"..."   — legacy tag kept working for backward compat
<field name> — final fallback

A tag value of "-" (in whichever namespace matches first) skips the field entirely. Multi-part tag values (`format:"name,opt1,opt2"`) use the first comma-separated element as the column name.

Example usage: 

frame, _ := gobi.FromStructs(rows, gobi.StructTagFormat("parquet"))

Per-io convenience wrappers (parquetio.WriteStructs, etc.) pass their own format automatically.

type TimeRolling 

type TimeRolling struct {
	// contains filtered or unexported fields
}

TimeRolling represents a right-anchored, time-based rolling window over a Frame. Use Frame.RollingBy to construct one.

func (*TimeRolling) Agg 

func (r *TimeRolling) Agg(column string, kind AggKind) (Series, error)

Agg computes a single aggregation over each row's trailing window and returns a Series of the aggregated values. Rows whose window is empty (e.g. all-null time or empty column) produce nulls.

type TimeUnit 

type TimeUnit uint8

TimeUnit selects a truncation granularity for Series.TruncateTo. Values intentionally include only sub-day units; week / month / year truncations use TruncateToCalendar because they require calendar (not clock) arithmetic. 

const (
	UnitNanosecond TimeUnit = iota
	UnitMicrosecond
	UnitMillisecond
	UnitSecond
	UnitMinute
	UnitHour
	UnitDay
)

Source Files

View all Source files

Directories

Path Synopsis
Package compute provides vectorized numeric kernels used by gobi's compute layer.
 Package compute provides vectorized numeric kernels used by gobi's compute layer.
contrib
athenaio module
Package csvio reads and writes CSV data as gobi Frames.
 Package csvio reads and writes CSV data as gobi Frames.
Package geojsonio reads and writes gobi Frames as GeoJSON per RFC 7946.
 Package geojsonio reads and writes gobi Frames as GeoJSON per RFC 7946.
Package geometry provides 2D geometry primitives (Point, LineString, Polygon, MultiPoint) with WKB and WKT encoding, a coordinate reference system model, and common spatial operations (area, distance, centroid, convex hull, intersection tests, polygon boolean ops).
 Package geometry provides 2D geometry primitives (Point, LineString, Polygon, MultiPoint) with WKB and WKT encoding, a coordinate reference system model, and common spatial operations (area, distance, centroid, convex hull, intersection tests, polygon boolean ops).
Package gpkgio reads and writes gobi Frames as OGC GeoPackage (SQLite) files.
 Package gpkgio reads and writes gobi Frames as OGC GeoPackage (SQLite) files.
Package kmlio reads and writes KML (Keyhole Markup Language, OGC 12-007r2) and KMZ (zipped KML) as gobi Frames.
 Package kmlio reads and writes KML (Keyhole Markup Language, OGC 12-007r2) and KMZ (zipped KML) as gobi Frames.
Package parquetio reads and writes gobi Frames as Apache Parquet.
 Package parquetio reads and writes gobi Frames as Apache Parquet.
Package pgio reads and writes gobi Frames against PostgreSQL / PostGIS databases.
 Package pgio reads and writes gobi Frames against PostgreSQL / PostGIS databases.
Package shpio reads and writes ESRI Shapefiles as gobi Frames.
 Package shpio reads and writes ESRI Shapefiles as gobi Frames.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.