golars

Golars

A high-performance Arrow-native, Polars-like DataFrame API for Go.

If you know polars, you'll feel right at home. In fact, golars uses native polars under the hood.

go get github.com/guywaldman/golars

The package bundles native dependencies such that you can install it easily (as you would polars).
It exposes high-level DataFrame APIs, with "zero configuration" support (e.g., for reading from S3/GCS):

import gl "github.com/guywaldman/golars"

// ...
df, _ := gl.ReadParquet(ctx, "events.parquet")
summary, _ := df.GroupBy(gl.Col("country")).Agg(
	gl.Col("amount").Sum().Alias("amount_total"),
	gl.Col("user_id").Count().Alias("users"),
)
_ = summary.WriteCSV(ctx, "out.csv")

Note that you can read from local files as well as cloud storage (S3 with s3:// or Google Cloud Storage with gs://) using their default credential discovery, as Polars does. If you need to configure it (such as for auth), you can use:

store, err := gl.NewGCSStore(context.Background(), option.WithoutAuthentication())
if err != nil {
  panic(err)
}
defer store.Close()

if err := gl.RegisterObjectStore("gs", store); err != nil {
  panic(err)
}

// ...

Featureset

• Arrow-backed DataFrame and Series, with retained Arrow interop.
• Row-oriented constructors: DataFrameFromMaps and generic DataFrameFromStructs, with nullable missing fields and standard json tags.
• Eager and lazy Select/SelectSeq, Filter, WithColumns/WithColumnsSeq, Limit/Head/Tail, Slice, Reverse, Clear, Sort, GroupBy(...).Agg(...), joins, window expressions, Unique, Explode, Unnest, Unpivot/Melt, fixed rolling expressions, as-of joins, Drop, Rename, DropNulls, and DropNaNs.
• Expression trees with arithmetic (including modulo, powers, rounding, and trigonometric kernels), comparisons, boolean logic, null/NaN handling, casts, aggregates, conditional expressions, partitioned windows, descending sort, membership/range predicates, list expressions, and common string/date kernels.
• Native JSON-column decoding into typed structs and lists, with SIMD parsing, row-level nulling for malformed JSON, field-level nulling for bad leaves, coercion controls, and strict required-field validation.
• Struct field expressions such as Col("payload").Struct().Element("user_id"), including nested struct access.
• Arrow type descriptors - primitive, temporal, decimal, dictionary (categorical/enum), nested list/struct/map, fixed-size, view, interval, and run-end encoded types. DataFrame/Series constructors accept Arrow arrays, chunked arrays, tables, and record batches without a copy at the boundary.
• Lazy scans and eager reads for Parquet, CSV, NDJSON, JSON arrays, and Arrow IPC file/stream formats (with ScanArrow/ReadArrow aliases).
• Scan projection and bounded reads, CSV delimiters, headers, comments, null markers, quote handling, and schema-inference controls.
• Lazy and eager sinks for the same formats, with non-overwriting local-file defaults. LazyFrame.Sink selects the writer from the destination extension.
• Row-oriented helpers: Rows materializes detached row maps, while ForEachRow streams them without retaining the complete result. ToDicts remains as a Polars-compatible alias.
• Series value access with Values/ToSlice, zero-copy Slice, Clone, IsEmpty, and Arrow chunk metadata; DataFrames expose Columns, IsEmpty, Clone, and the corresponding row operations.
• Local globbing and Hive-style Parquet partition discovery.
• Hive dataset writes with Create/Append, partition columns, _SUCCESS, and _golars_schema.json.
• Built-in S3 and GCS URI support using standard credential discovery. Custom ObjectStore implementations can override them with RegisterObjectStore or WithObjectStore.
• Engine-wide memory accounting with a 1 GiB default quota and contextual cancellation through context.Context.

Native execution

All relational execution runs through the embedded Rust/Polars backend. The Go layer owns the public API, Arrow conversion, and storage adapters (including custom object stores and formats Polars does not open directly). The target-specific native library is embedded in the Go module and loaded with CGO-free FFI, so consumers only need the normal go get and Go build workflow. For object-store scans, the Go adapter preserves projected columns, simple Parquet predicates, row-group pruning, and bounded reads before handing Arrow batches to Polars.

The checked-in assets currently support Linux amd64/arm64 and macOS arm64. A target without an embedded asset returns an explicit native-backend availability error; there is no portable execution fallback.

Supported platforms

The package can still be compiled on other Go targets, but collection returns an explicit native-backend availability error because no matching Polars library is embedded.

Fast JSON parsing

DecodeJSON turns a string expression into a typed Arrow value. You can provide an Arrow descriptor or derive the schema directly from a Go struct. Exported fields use standard json tags; pointer fields are nullable, and nested structs and lists are supported.

schema := gl.StructOf(
	gl.Field{Name: "id", Type: gl.Int64, Nullable: false},
	gl.Field{Name: "tenant", Type: gl.String, Nullable: true},
	gl.Field{Name: "tags", Type: gl.ListOf(gl.String), Nullable: true},
)

decoded := gl.DecodeJSON(gl.Col("payload_json"), schema)
result, err := lf.Select(decoded.Alias("payload")).Collect(ctx)

The struct form avoids a separate schema declaration:

type Payload struct {
	ID      int64    `json:"id"`
	Tenant  string   `json:"tenant"`
	Tags    []string `json:"tags"`
	Profile *struct {
		Country string `json:"country"`
	} `json:"profile"`
}

decoded := gl.DecodeJSON(gl.Col("payload_json"), Payload{})
// Or use gl.DecodeJSONAs[Payload](gl.Col("payload_json")).

Rows with malformed JSON or incompatible leaves become null by default, while valid sibling fields are retained. Use WithJSONCoerce(false) for strict JSON types, WithJSONErrorMode(JSONErrorOnError) to fail the query, or WithJSONStrictRequiredFields(true) to enforce non-nullable struct fields.

Examples

df, err := gl.ReadCSV(ctx, "events.csv")
if err != nil { return err }
defer df.Close()

summary, err := df.GroupBy(gl.Col("country")).Agg(
	gl.Col("amount").Sum().Alias("amount_total"),
	gl.Col("user_id").Count().Alias("users"),
)
if err != nil { return err }
defer summary.Close()

err = summary.WriteParquet(ctx, "out.parquet")

For an explicitly configured engine:

engine, err := gl.NewEngine(
	gl.WithMemoryLimit(512<<20),
	gl.WithBatchSize(32<<10),
	gl.WithObjectStore("s3", s3Store),
)
if err != nil { return err }
defer engine.Close()

df, err := engine.ScanParquet("s3://bucket/events/*.parquet").Collect(ctx)

The implementation is intentionally a fast-breaking v0 API. The execution kernel and format contracts are stable enough for experimentation, while the public type names and richer Polars parity surface will continue to grow.

Performance and Benchmarks

The benchmarks directory contains equivalent Go and Rust Polars workloads. The following run was captured on 2026-08-03 on a MacBook Pro with an Apple M3 Max (16 cores, 128 GB RAM), using Go 1.26.4, Rust 1.96.0, Polars 0.52.0, and 16 threads for both implementations. Each cell is golars / polars in milliseconds per execution, followed by the golars-to- polars ratio.

Each implementation had one unmeasured warmup and three measured executions. The Go side used -benchtime=3x; the Rust side used --warmup 1 --iterations 3.

For the 1M-row dataset, the complete workload set was:

Go's B/op is the allocation metric reported by go test; the Polars column is the process RSS increase reported by the Rust runner, so those two memory figures are directional rather than directly interchangeable.

To reproduce this exact scale sweep with explicit thread parity:

GOMAXPROCS=16 POLARS_MAX_THREADS=16 \
GOLARS_BENCH_TIME=3x \
GOLARS_BENCH_POLARS_ITERATIONS=3 \
GOLARS_BENCH_WARMUP=1 \
./benchmarks/run.sh

Contributing

Feel very welcome to do so! See CONTRIBUTING.md.