Documentation
¶
Index ¶
- type Aggregation
- type DataFrame
- func FromMaps(data []map[string]interface{}) (*DataFrame, error)
- func FromMapsEngine(e *Engine, data []map[string]interface{}) (*DataFrame, error)
- func FromStructs(data interface{}) (*DataFrame, error)
- func FromStructsEngine(e *Engine, data interface{}) (*DataFrame, error)
- func ReadCSV(path string) (*DataFrame, error)
- func ReadCSVEngine(e *Engine, path string) (*DataFrame, error)
- func ReadCSVString(csvData string) (*DataFrame, error)
- func ReadCSVStringEngine(e *Engine, csvData string) (*DataFrame, error)
- func ReadJSON(path string) (*DataFrame, error)
- func ReadJSONBytes(data []byte) (*DataFrame, error)
- func ReadJSONEngine(e *Engine, path string) (*DataFrame, error)
- func ReadParquet(path string) (*DataFrame, error)
- func ReadParquetEngine(e *Engine, path string) (*DataFrame, error)
- func ReadSQL(query string) (*DataFrame, error)
- func ReadSQLEngine(e *Engine, query string) (*DataFrame, error)
- func (df *DataFrame) AddSeries(name string, data interface{}) (*DataFrame, error)
- func (df *DataFrame) Agg(aggs ...Aggregation) (*DataFrame, error)
- func (df *DataFrame) AsType(column string, newType DataType) (*DataFrame, error)
- func (df *DataFrame) Collect() ([]Row, error)
- func (df *DataFrame) Columns() ([]string, error)
- func (df *DataFrame) Concat(other *DataFrame) (*DataFrame, error)
- func (df *DataFrame) CumSum() (*DataFrame, error)
- func (df *DataFrame) Describe() (*DataFrame, error)
- func (df *DataFrame) Distinct(columns ...string) (*DataFrame, error)
- func (df *DataFrame) Drop(columns ...string) (*DataFrame, error)
- func (df *DataFrame) DropNA(columns ...string) (*DataFrame, error)
- func (df *DataFrame) Engine() *Engine
- func (df *DataFrame) FillNA(values map[string]interface{}) (*DataFrame, error)
- func (df *DataFrame) Filter(condition string) (*DataFrame, error)
- func (df *DataFrame) FilterBy(preds ...Predicate) (*DataFrame, error)
- func (df *DataFrame) Float() (float64, error)
- func (df *DataFrame) GroupBy(groupColumns []string, aggs ...Aggregation) (*DataFrame, error)
- func (df *DataFrame) Head(n int) *DataFrame
- func (df *DataFrame) Iterate(fn func(Row) (bool, error)) error
- func (df *DataFrame) Join(other *DataFrame, joinType, leftOn, rightOn string) (*DataFrame, error)
- func (df *DataFrame) Limit(offset, n int) *DataFrame
- func (df *DataFrame) Materialize() (*DataFrame, error)
- func (df *DataFrame) NumCols() (int, error)
- func (df *DataFrame) NumRows() (int, error)
- func (df *DataFrame) Print(maxRows ...int) error
- func (df *DataFrame) Release() error
- func (df *DataFrame) Rename(oldName, newName string) (*DataFrame, error)
- func (df *DataFrame) SQL() string
- func (df *DataFrame) Schema() (*Schema, error)
- func (df *DataFrame) Select(columns ...string) (*DataFrame, error)
- func (df *DataFrame) Shape() (rows, cols int, err error)
- func (df *DataFrame) Sort(columns []string, ascending []bool) (*DataFrame, error)
- func (df *DataFrame) Tail(n int) (*DataFrame, error)
- func (df *DataFrame) Where(sqlExpr string) *DataFrame
- func (df *DataFrame) WithColumn(name, sqlExpr string) (*DataFrame, error)
- func (df *DataFrame) WriteCSV(path string) error
- func (df *DataFrame) WriteJSON(path string) error
- func (df *DataFrame) WriteParquet(path string) error
- type DataType
- type Engine
- type Predicate
- type Row
- type Schema
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Aggregation ¶
Aggregation pairs a source column with a function to apply to it in GroupBy. Func is one of: "sum", "avg"/"mean", "min", "max", "count", "count_distinct", "std"/"stddev", "var"/"variance", "median", "first", "last". The result column is named "<column>_<func>" unless As is set.
type DataFrame ¶
type DataFrame struct {
// contains filtered or unexported fields
}
DataFrame is a lazy handle to a tabular relation living inside DuckDB.
Unlike a materialized in-memory frame, a DataFrame mostly holds a SQL query. Transformations (Select, Filter, GroupBy, Sort, Join, ...) don't move any data: they wrap the current relation in a new query and return a new DataFrame. Data only crosses back into Go when you read it (Collect, Print, WriteToCSV, scalar aggregations, ...).
A DataFrame is safe for concurrent reads. Operations never mutate the receiver; they always return a new DataFrame, so there is no shared mutable state to guard beyond the engine's own pool.
func FromMaps ¶
FromMaps builds a DataFrame from a slice of column->value maps, inferring each column's type from the values present. Columns are ordered alphabetically for determinism (Go map iteration is random). Missing or nil entries become NULL.
Type inference per column: if any value is a string that doesn't parse as a timestamp, the column is a string; otherwise time beats float beats int.
func FromMapsEngine ¶
FromMapsEngine is FromMaps against a specific engine.
func FromStructs ¶
FromStructs builds a DataFrame from a slice of structs using reflection. Exported fields become columns, in declaration order. Field types map as: integer kinds -> TypeInt, float kinds -> TypeFloat, time.Time -> TypeTime, everything else -> TypeString (via fmt). A `df:"name"` struct tag overrides the column name; `df:"-"` skips the field.
This replaces the original's hard-coded Person special case with something that works for any struct.
func FromStructsEngine ¶
FromStructsEngine is FromStructs against a specific engine.
func ReadCSVEngine ¶
ReadCSVEngine is ReadCSV against a specific engine.
func ReadCSVString ¶
ReadCSVString loads CSV data from an in-memory string. DuckDB reads files, not Go strings, so we parse the text ourselves into rows and materialize them. For large inputs prefer ReadCSV against a file so DuckDB's vectorized reader does the work.
func ReadCSVStringEngine ¶
ReadCSVStringEngine is ReadCSVString against a specific engine.
func ReadJSON ¶
ReadJSON loads a JSON file (array of objects, or newline-delimited) using DuckDB's read_json_auto.
func ReadJSONBytes ¶
ReadJSONBytes loads JSON from an in-memory byte slice by staging it to a temp file and reading it with DuckDB.
func ReadJSONEngine ¶
ReadJSONEngine is ReadJSON against a specific engine.
func ReadParquet ¶
ReadParquet loads a Parquet file using DuckDB's native read_parquet. This replaces the original's ~200-line reflection-based reader entirely.
func ReadParquetEngine ¶
ReadParquetEngine is ReadParquet against a specific engine.
func ReadSQL ¶
ReadSQL runs a query against an external database via a DuckDB scanner.
DuckDB can attach external systems, but configuring that generically is beyond this helper. Instead, ReadSQL executes the query against the DataFrame engine's own DuckDB instance, which is the right call when the data already lives in DuckDB (tables you've created, attached databases, etc.). For pulling from Postgres/MySQL/SQLite, ATTACH the source first using DuckDB's scanner extensions, then call ReadSQL.
func ReadSQLEngine ¶
ReadSQLEngine is ReadSQL against a specific engine.
func (*DataFrame) AddSeries ¶
AddSeries returns a frame with a new column appended from Go-side data. The data length must equal the frame's current row count. Supported element types are float64 (numeric column) and time.Time (time column).
Because a DataFrame is a lazy relation with no inherent row order, "append this slice positionally" only makes sense against a fixed ordering. We therefore materialize the current frame with a row number, build a small table from the slice with matching row numbers, and join them. If you care which row gets which value, Sort first so the ordering is defined.
func (*DataFrame) Agg ¶
func (df *DataFrame) Agg(aggs ...Aggregation) (*DataFrame, error)
Agg computes aggregations over the whole frame (no grouping), returning a one-row frame. Convenient for summary numbers without a group key.
func (*DataFrame) AsType ¶
AsType returns a frame with one column cast to a new logical type using DuckDB's CAST. Casts that DuckDB can't perform surface as a query error when the frame is read. Unlike the original, this works between any of the four types DuckDB supports converting.
func (*DataFrame) Collect ¶
Collect executes the frame's query and returns every row as a []Row. For large results prefer Iterate to avoid holding the whole set in memory.
func (*DataFrame) Columns ¶
Columns returns the ordered column names. It panics only if the relation is malformed in a way that prevents DuckDB from describing it; for an error-returning variant use Schema.
func (*DataFrame) Concat ¶
Concat stacks another frame's rows beneath this one. By default it aligns columns by name and unions the two schemas: columns missing from one side are filled with NULL on that side (UNION ALL BY NAME). This is more forgiving than the original, which required identical layouts.
Both frames must share an engine.
func (*DataFrame) CumSum ¶
CumSum returns a frame where each numeric column is replaced by its running cumulative sum over the current row order. Non-numeric columns pass through unchanged. NULLs are treated as zero for the running total (they don't reset it), matching the original's skip-NaN behavior.
Row order matters for a cumulative sum; chain Sort beforehand if you need a particular order. We materialize a row number to define "preceding rows" deterministically.
func (*DataFrame) Describe ¶
Describe returns per-column summary statistics for the numeric columns: count, mean, std, min, 25%, 50%, 75%, and max. The output frame has a "stat" label column followed by one column per numeric input column, matching the shape of the original Describe.
The original computed quantiles in Go after sorting; here DuckDB's QUANTILE_CONT does it in-engine.
func (*DataFrame) Distinct ¶
Distinct returns a frame with duplicate rows removed. With no columns, whole rows must match; with columns, DuckDB's DISTINCT ON keeps the first row per distinct combination of those columns.
func (*DataFrame) Drop ¶
Drop returns a frame with the named columns removed. Columns that don't exist are ignored, matching the intuitive "remove if present" semantics.
func (*DataFrame) DropNA ¶
DropNA returns a frame with rows containing any NULL removed. If columns are given, only those are checked; otherwise every column is checked.
func (*DataFrame) Engine ¶
Engine exposes the underlying engine, mainly so related frames can be built against the same DuckDB instance.
func (*DataFrame) FillNA ¶
FillNA returns a frame with NULLs replaced per column. The map keys are column names; values are the fill value for that column, coerced to the column's type. Columns not in the map are unchanged.
func (*DataFrame) Filter ¶
Filter applies a simple "column operator value" condition, parsed from a string for convenience and backward familiarity. Supported operators: =, ==, !=, <>, <, <=, >, >=, and the word forms LIKE / NOT LIKE.
df.Filter("age > 30")
df.Filter("name = Alice")
df.Filter("city LIKE San%")
The value is bound as a typed literal based on the column's type, so it is quoted correctly and is injection-safe. For multi-clause logic use Where.
func (*DataFrame) FilterBy ¶
FilterBy applies one or more Predicates joined by AND. Passing several predicates is the common case ("age > 30 AND city = Paris") without writing SQL by hand.
func (*DataFrame) Float ¶
Float returns a single numeric scalar from a one-row, one-column frame, e.g. the result of Agg(...) with a single aggregation. It's a convenience over Collect for "give me the number" cases. If the frame has more than one column or more than one row, only the first cell is read.
func (*DataFrame) GroupBy ¶
func (df *DataFrame) GroupBy(groupColumns []string, aggs ...Aggregation) (*DataFrame, error)
GroupBy groups by the given columns and computes the given aggregations. Output columns are the group keys followed by each aggregation. This replaces the original's hand-built grouping with DuckDB's GROUP BY, which is both correct for all type combinations and dramatically faster.
df.GroupBy([]string{"city"},
Aggregation{Column: "sales", Func: "sum"},
Aggregation{Column: "sales", Func: "avg"},
)
func (*DataFrame) Head ¶
Head returns the first n rows. Note: without an explicit Sort, "first" is whatever order the engine produces; chain Sort first if order matters.
func (*DataFrame) Iterate ¶
Iterate streams rows to a callback, one at a time, stopping early if the callback returns false or an error. This keeps memory flat for large frames.
func (*DataFrame) Join ¶
Join combines two frames on equality of a left and right key column. joinType is "inner", "left", "right", "outer"/"full", or "cross". The join key from the right frame is dropped from the output (it duplicates the left key); other right columns that collide with left names are suffixed "_right".
Both frames must belong to the same engine. Because every DataFrame is just a relation in the shared DuckDB instance, the join is a single SQL statement rather than the original's manual hash-join over parallel arrays.
left.Join(right, "inner", "user_id", "id")
func (*DataFrame) Limit ¶
Limit returns at most n rows starting at the given offset. It's the general form behind Head/Tail and is handy for pagination.
func (*DataFrame) Materialize ¶
Materialize executes this frame's query once and stores the result in a new base table, returning a frame over that table. This is useful when a frame is the product of an expensive query chain that you will read repeatedly: rather than re-running the whole chain on each read, you pay for it once here.
The returned frame owns its table; call Release on it when done (especially on the long-lived default engine) to free the memory. The original frame is unchanged.
func (*DataFrame) NumRows ¶
NumRows returns the number of rows the frame currently represents by running a COUNT(*). It does not materialize the rows themselves.
func (*DataFrame) Print ¶
Print writes a human-readable table preview to stdout (up to maxRows rows). It mirrors the original's Print but reads through the query engine. A maxRows of 0 or less defaults to 10.
func (*DataFrame) Release ¶
Release drops the base table backing this frame, if it owns one. Frames built by transformations own no storage, so Release is a no-op for them and only the ingestion/Materialize roots need releasing. After Release, this frame and any frames derived from it are invalid.
Because the package's default engine is a single long-lived in-memory DuckDB instance, base tables accumulate until released or until the process exits. Short programs can ignore this; long-running services that create many frames should Release ingestion roots (or use a dedicated Engine they Close) to keep memory bounded.
func (*DataFrame) Rename ¶
Rename returns a frame with one column renamed. Other columns pass through unchanged and in order.
func (*DataFrame) Select ¶
Select returns a frame containing only the named columns, in the given order. Unknown column names are reported as an error (the original silently dropped them, which hid typos).
func (*DataFrame) Sort ¶
Sort returns a frame ordered by the given columns. ascending[i] controls the direction of columns[i]; if ascending is shorter than columns, missing entries default to ascending. NULLs sort last in ascending order (DuckDB's default), which matches typical expectations.
func (*DataFrame) Tail ¶
Tail returns the last n rows by skipping the first (total-n) rows with OFFSET. Like Head, "last" is defined by whatever order the underlying relation produces; for base tables that's insertion order, but if a particular order matters, call Sort first.
func (*DataFrame) Where ¶
Where filters rows by a raw SQL boolean expression evaluated per row, e.g.
df.Where("age > 30 AND city = 'Paris'")
df.Where("score IS NULL OR score < 0")
The expression is trusted caller SQL. This is the most flexible filter and the right tool when the shorthand isn't expressive enough.
func (*DataFrame) WithColumn ¶
WithColumn returns a frame with an added or replaced column defined by a raw SQL expression evaluated over the existing columns. For example:
df.WithColumn("total", "price * quantity")
The expression is trusted SQL (the API surface for computed columns); column references inside it should be quoted by the caller if they contain unusual characters. This is the escape hatch that lets callers express computations the typed helpers don't cover.
func (*DataFrame) WriteCSV ¶
WriteCSV writes the frame to a CSV file using DuckDB's COPY, including a header row. COPY streams directly from the query engine to disk without pulling rows into Go.
func (*DataFrame) WriteJSON ¶
WriteJSON writes the frame to a newline-delimited / array JSON file via COPY.
func (*DataFrame) WriteParquet ¶
WriteParquet writes the frame to a Parquet file via COPY with Snappy compression (DuckDB's default), replacing the original's manual writer.
type DataType ¶
type DataType int
DataType is the coarse logical type of a column. It intentionally mirrors the original package's four-way classification so existing callers keep working. Internally, DuckDB tracks far richer types; we collapse them into these buckets for the Go-facing API.
type Engine ¶
type Engine struct {
// contains filtered or unexported fields
}
engine wraps a single in-process DuckDB instance. All DataFrames created against the same engine share its connection pool, which lets them be joined, concatenated, and combined without copying data across processes.
DuckDB lives in-process, so the *sql.DB handle is the whole database. We keep one default engine for convenience (most callers never need more than one), but callers who want isolation can construct their own with NewEngine.
func NewEngine ¶
NewEngine opens a DuckDB instance at the given DSN. An empty DSN opens a fresh in-memory database. A file path opens (or creates) a persistent one. DuckDB config options may be appended as query parameters, e.g. "/tmp/foo.db?threads=4&access_mode=read_only".
type Predicate ¶
type Predicate struct {
Column string
Op string // one of the operators accepted by normalizeOp
Value interface{}
}
Predicate is a structured, programmatic filter clause. Build one and pass it to FilterBy. Values are bound as typed literals, so Predicates are safe to construct from untrusted input (unlike Where, which takes raw SQL).
type Row ¶
type Row map[string]interface{}
Row is one materialized record: column name -> value. Numeric columns come back as int64 or float64, time columns as time.Time, string columns as string, and SQL NULLs as nil.
type Schema ¶
Schema describes the columns of a DataFrame: their names in order, and the logical type of each.
func (*Schema) NumericColumns ¶
NumericColumns returns the names of columns DuckDB treats as numbers.
Source Files