arrow

package
v0.0.0-...-370038a Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 24, 2026 License: MIT Imports: 19 Imported by: 0

Documentation

Overview

Package arrow provides an Apache Arrow-backed compute engine for the dataset package. It implements dataset.ColumnFactory, dataset.BuilderFactory, dataset.Aggregator, and dataset.Caster using Arrow arrays and arrow/math SIMD kernels.

Usage: 

eng := arrow.NewEngine(memory.DefaultAllocator)
f := eng.(dataset.ColumnFactory)
ds, _ := f.FromColumns(
    dataset.NewSchema(dataset.FloatCol("x"), dataset.StringCol("label")),
    f.NewFloat64Column("x", []float64{1, 2, 3}),
    f.NewStringColumn("label", []string{"a", "b", "c"}),
)

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Engine 

type Engine struct {
	// contains filtered or unexported fields
}

Engine is the Arrow compute backend.

func NewEngine 

func NewEngine(alloc memory.Allocator) *Engine

NewEngine creates an Arrow engine with the given memory allocator.

func (*Engine) Abs 

func (e *Engine) Abs(col dataset.AnyColumn) (dataset.AnyColumn, error)

func (*Engine) Acos 

func (e *Engine) Acos(col dataset.AnyColumn) (dataset.AnyColumn, error)

func (*Engine) AddCols 

func (e *Engine) AddCols(a, b dataset.AnyColumn) (dataset.AnyColumn, error)

func (*Engine) AddScalar 

func (e *Engine) AddScalar(col dataset.AnyColumn, val float64) (dataset.AnyColumn, error)

func (*Engine) Alloc 

func (e *Engine) Alloc() memory.Allocator

Alloc returns the engine's memory allocator.

func (*Engine) Asin 

func (e *Engine) Asin(col dataset.AnyColumn) (dataset.AnyColumn, error)

func (*Engine) Atan 

func (e *Engine) Atan(col dataset.AnyColumn) (dataset.AnyColumn, error)

func (*Engine) Atan2 

func (e *Engine) Atan2(y, x dataset.AnyColumn) (dataset.AnyColumn, error)

func (*Engine) BitAnd 

func (e *Engine) BitAnd(a, b dataset.AnyColumn) (dataset.AnyColumn, error)

func (*Engine) BitNot 

func (e *Engine) BitNot(col dataset.AnyColumn) (dataset.AnyColumn, error)

func (*Engine) BitOr 

func (e *Engine) BitOr(a, b dataset.AnyColumn) (dataset.AnyColumn, error)

func (*Engine) BitShiftLeft 

func (e *Engine) BitShiftLeft(col dataset.AnyColumn, n int) (dataset.AnyColumn, error)

func (*Engine) BitShiftRight 

func (e *Engine) BitShiftRight(col dataset.AnyColumn, n int) (dataset.AnyColumn, error)

func (*Engine) BitXor 

func (e *Engine) BitXor(a, b dataset.AnyColumn) (dataset.AnyColumn, error)

func (*Engine) Cast 

func (e *Engine) Cast(col dataset.AnyColumn, target dataset.DType) (dataset.AnyColumn, error)

func (*Engine) Ceil 

func (e *Engine) Ceil(col dataset.AnyColumn) (dataset.AnyColumn, error)

Ceil — Arrow lacks this, use stdlib

func (*Engine) Combine 

func (e *Engine) Combine(datasets ...dataset.Table) (dataset.Table, error)

func (*Engine) Complete 

func (e *Engine) Complete(ds dataset.Table, cols ...string) (dataset.Table, error)

Complete generates all combinations of the specified columns' unique values.

func (*Engine) Concatenate 

func (e *Engine) Concatenate(ds dataset.Table, col string, from []string, sep string) (dataset.Table, error)

Concatenate joins multiple string columns into one with a separator.

func (*Engine) Cos 

func (e *Engine) Cos(col dataset.AnyColumn) (dataset.AnyColumn, error)

func (*Engine) Count 

func (e *Engine) Count(col dataset.AnyColumn) (dataset.AnyColumn, error)

func (*Engine) CumMax 

func (e *Engine) CumMax(col dataset.AnyColumn) (dataset.AnyColumn, error)

func (*Engine) CumMin 

func (e *Engine) CumMin(col dataset.AnyColumn) (dataset.AnyColumn, error)

func (*Engine) CumSum 

func (e *Engine) CumSum(col dataset.AnyColumn) (dataset.AnyColumn, error)

func (*Engine) DenseRank 

func (e *Engine) DenseRank(col dataset.AnyColumn) (dataset.AnyColumn, error)

func (*Engine) DivCols 

func (e *Engine) DivCols(a, b dataset.AnyColumn) (dataset.AnyColumn, error)

func (*Engine) DropNA 

func (e *Engine) DropNA(ds dataset.Table, cols ...string) (dataset.Table, error)

func (*Engine) Erf 

func (e *Engine) Erf(col dataset.AnyColumn) (dataset.AnyColumn, error)

func (*Engine) Exp 

func (e *Engine) Exp(col dataset.AnyColumn) (dataset.AnyColumn, error)

Exp — Arrow lacks this, use highway

func (*Engine) Fill 

func (e *Engine) Fill(col dataset.AnyColumn, dir dataset.FillDirection) (dataset.AnyColumn, error)

func (*Engine) Filter 

func (e *Engine) Filter(ds dataset.Table, mask dataset.Masker) (dataset.Table, error)

func (*Engine) FilterIndices 

func (e *Engine) FilterIndices(mask []bool) []int

func (*Engine) Floor 

func (e *Engine) Floor(col dataset.AnyColumn) (dataset.AnyColumn, error)

Floor — Arrow lacks this, use stdlib

func (*Engine) FromColumns 

func (e *Engine) FromColumns(schema *dataset.Schema, cols ...dataset.AnyColumn) (dataset.Table, error)

func (*Engine) Join 

func (e *Engine) Join(left, right dataset.Table, spec dataset.JoinSpec) (dataset.Table, error)

Join implements the Joiner interface with a hash-join algorithm. It supports Inner, Left, Right, Full, Semi, and Anti joins.

func (*Engine) Lag 

func (e *Engine) Lag(col dataset.AnyColumn, offset int) (dataset.AnyColumn, error)

func (*Engine) Lead 

func (e *Engine) Lead(col dataset.AnyColumn, offset int) (dataset.AnyColumn, error)

func (*Engine) Ln 

func (e *Engine) Ln(col dataset.AnyColumn) (dataset.AnyColumn, error)

Ln — Arrow native

func (*Engine) Log2 

func (e *Engine) Log2(col dataset.AnyColumn) (dataset.AnyColumn, error)

Log2 — Arrow native

func (*Engine) Log10 

func (e *Engine) Log10(col dataset.AnyColumn) (dataset.AnyColumn, error)

Log10 — Arrow native

func (*Engine) Mean 

func (e *Engine) Mean(col dataset.AnyColumn) (dataset.AnyColumn, error)

func (*Engine) Median 

func (e *Engine) Median(col dataset.AnyColumn) (dataset.AnyColumn, error)

func (*Engine) MinMax 

func (e *Engine) MinMax(col dataset.AnyColumn) (dataset.AnyColumn, dataset.AnyColumn, error)

func (*Engine) MulCols 

func (e *Engine) MulCols(a, b dataset.AnyColumn) (dataset.AnyColumn, error)

func (*Engine) MulScalar 

func (e *Engine) MulScalar(col dataset.AnyColumn, val float64) (dataset.AnyColumn, error)

func (*Engine) Name 

func (e *Engine) Name() string

Name returns "arrow".

func (*Engine) Neg 

func (e *Engine) Neg(col dataset.AnyColumn) (dataset.AnyColumn, error)

func (*Engine) NewBoolColumn 

func (e *Engine) NewBoolColumn(name string, data []bool) dataset.AnyColumn

func (*Engine) NewBuilder 

func (e *Engine) NewBuilder(schema *dataset.Schema) dataset.Builder

func (*Engine) NewFloat64Column 

func (e *Engine) NewFloat64Column(name string, data []float64) dataset.AnyColumn

func (*Engine) NewInt64Column 

func (e *Engine) NewInt64Column(name string, data []int64) dataset.AnyColumn

func (*Engine) NewStringColumn 

func (e *Engine) NewStringColumn(name string, data []string) dataset.AnyColumn

func (*Engine) NewTimestampColumn 

func (e *Engine) NewTimestampColumn(name string, data []int64) dataset.AnyColumn

func (*Engine) PercentRank 

func (e *Engine) PercentRank(col dataset.AnyColumn) (dataset.AnyColumn, error)

func (*Engine) PivotLonger 

func (e *Engine) PivotLonger(ds dataset.Table, spec dataset.PivotLongerSpec) (dataset.Table, error)

PivotLonger reshapes a wide dataset to long format.

func (*Engine) PivotWider 

func (e *Engine) PivotWider(ds dataset.Table, spec dataset.PivotWiderSpec) (dataset.Table, error)

PivotWider reshapes a long dataset to wide format.

func (*Engine) Pow 

func (e *Engine) Pow(col dataset.AnyColumn, exp float64) (dataset.AnyColumn, error)

func (*Engine) Rank 

func (e *Engine) Rank(col dataset.AnyColumn) (dataset.AnyColumn, error)

func (*Engine) ReadCSV 

func (e *Engine) ReadCSV(ctx context.Context, r io.Reader, cfg dataset.CSVConfig) (dataset.Table, error)

ReadCSV reads CSV data using arrow/csv.NewInferringReader with chunked streaming. Default chunk is 64K rows per batch to bound memory for large files.

func (*Engine) ReadParquet 

func (e *Engine) ReadParquet(ctx context.Context, r io.ReaderAt, size int64, cfg dataset.ParquetConfig) (dataset.Table, error)

ReadParquet reads Parquet data using pqarrow for zero-copy columnar ingest.

func (*Engine) ReplaceNA 

func (e *Engine) ReplaceNA(col dataset.AnyColumn, defaultVal float64) (dataset.AnyColumn, error)

func (*Engine) Round 

func (e *Engine) Round(col dataset.AnyColumn) (dataset.AnyColumn, error)

Round — use stdlib for predictable half-away-from-zero behavior

func (*Engine) RowNumber 

func (e *Engine) RowNumber(n int) (dataset.AnyColumn, error)

func (*Engine) Select 

func (e *Engine) Select(col dataset.AnyColumn, indices []int) (dataset.AnyColumn, error)

func (*Engine) Separate 

func (e *Engine) Separate(ds dataset.Table, col string, into []string, sep string) (dataset.Table, error)

Separate splits a string column by a delimiter into multiple columns.

func (*Engine) Sigmoid 

func (e *Engine) Sigmoid(col dataset.AnyColumn) (dataset.AnyColumn, error)

func (*Engine) Sign 

func (e *Engine) Sign(col dataset.AnyColumn) (dataset.AnyColumn, error)

func (*Engine) Sin 

func (e *Engine) Sin(col dataset.AnyColumn) (dataset.AnyColumn, error)

func (*Engine) Slice 

func (e *Engine) Slice(col dataset.AnyColumn, start, end int) (dataset.AnyColumn, error)

func (*Engine) SortIndices 

func (e *Engine) SortIndices(col dataset.AnyColumn) ([]int, error)

SortIndices uses Arrow compute's SortIndicesArray kernel. Arrow's implementation handles null placement and type dispatch natively.

func (*Engine) Sqrt 

func (e *Engine) Sqrt(col dataset.AnyColumn) (dataset.AnyColumn, error)

Sqrt — Arrow lacks this, use highway/stdlib

func (*Engine) Stack 

func (e *Engine) Stack(datasets ...dataset.Table) (dataset.Table, error)

func (*Engine) SubCols 

func (e *Engine) SubCols(a, b dataset.AnyColumn) (dataset.AnyColumn, error)

func (*Engine) Sum 

func (e *Engine) Sum(col dataset.AnyColumn) (dataset.AnyColumn, error)

func (*Engine) Tan 

func (e *Engine) Tan(col dataset.AnyColumn) (dataset.AnyColumn, error)

func (*Engine) Tanh 

func (e *Engine) Tanh(col dataset.AnyColumn) (dataset.AnyColumn, error)

func (*Engine) Variance 

func (e *Engine) Variance(col dataset.AnyColumn) (dataset.AnyColumn, error)

func (*Engine) WriteCSV 

func (e *Engine) WriteCSV(ctx context.Context, w io.Writer, ds dataset.Table, cfg dataset.CSVConfig) error

WriteCSV writes a Dataset as CSV using go-simdcsv (generic string-based output).

func (*Engine) WriteParquet 

func (e *Engine) WriteParquet(ctx context.Context, w io.Writer, ds dataset.Table, cfg dataset.ParquetConfig) error

WriteParquet writes a Dataset as Parquet using pqarrow.WriteTable.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.