parquet

package
v0.2.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 28, 2026 License: MIT Imports: 14 Imported by: 0

Documentation

Overview

Package parquet writes and reads the tables a document holds beside its drawing (docs/spec.md §4.6): Apache Parquet files of flat columns.

Write makes the files the converters store: the columns of a table as optional fields of a flat schema, PLAIN values, RLE definition levels, one row group, no compression (the part the file becomes is compressed as a whole). Read takes those files back, and the files of other writers whose columns are flat and whose pages are PLAIN or dictionary encoded and uncompressed, Snappy or gzip (what pandas, Arrow and DuckDB write by default). Nested columns, the delta encodings and the other codecs are left to converter/parquet, which reads every file.

Index

Constants

View Source
const (
	// MaxColumns bounds the columns of a table (docs/spec.md §4.6).
	MaxColumns = 1 << 14
	// DefaultMaxRows is the rows Read returns when Limits.MaxRows is 0.
	DefaultMaxRows = 1 << 20
)

Limits of what is read and written.

Variables

View Source
var (
	// ErrNotParquet is returned for data that is not a Parquet file.
	ErrNotParquet = errors.New("parquet: not a Parquet file")
)

Functions

func Encode

func Encode(t *Table) ([]byte, error)

Encode writes a table as a Parquet file in memory.

func Text

func Text(c Column, v any) string

Text writes a value as text: a date as 2006-01-02, a date and time in RFC 3339, a number as Go prints it, a null as "".

func Write

func Write(w io.Writer, t *Table) error

Write writes a table as a Parquet file: its columns as optional fields of a flat schema, their values PLAIN, their definition levels RLE, in one row group, uncompressed. A value that is not of its column's type (see Table) is an error; a row shorter than the columns is padded with nulls, and a column of type Other cannot be written.

func WriteCSV

func WriteCSV(w io.Writer, t *Table) error

WriteCSV writes a table as CSV: the names of the columns, then the rows, their values as Text writes them.

func WriteJSON

func WriteJSON(w io.Writer, t *Table) error

WriteJSON writes a table as JSON: an array of objects, a value under the name of its column (dates and times as strings, as Text writes them; nulls as null; bytes as hexadecimal).

Types

type Column

type Column struct {
	Name string
	Type Type
}

Column is a column of a table.

func Schema

func Schema(data []byte) ([]Column, int64, error)

Schema reads the columns of a file and its row count from its footer. A column of a type the package does not take (a group, a repeated field, a decimal of bytes) is of type Other.

type Limits

type Limits struct {
	// MaxRows is the most rows returned (DefaultMaxRows when 0; the rows
	// after them are left out).
	MaxRows int
}

Limits bounds what Read returns.

type Table

type Table struct {
	Columns []Column
	Rows    [][]any
}

Table is a table in memory: its columns, and its rows of values. A value is a string (String), a float64 (Number), an int64 (Integer), a bool (Boolean), a time.Time (Date, at 0:00 UTC; DateTime, UTC) or nil (a null); a value of a column of type Other is what Read made of it (a []byte, a string, a number), or nil.

func Read

func Read(data []byte, limits Limits) (*Table, error)

Read reads the rows of a file, up to limits. Columns of type Other hold nil, except binaries, which hold their bytes.

type Type

type Type uint8

Type is the type of a column (docs/spec.md §4.6).

const (
	Other Type = iota
	String
	Number
	Integer
	Boolean
	Date
	DateTime
)

The types of columns. Other is a type of Parquet that is none of the rest (a nested column, a decimal of bytes, …), as the file was read.

func ParseType

func ParseType(s string) (Type, bool)

ParseType reads the name of a type, as the manifest writes it.

func (Type) String

func (t Type) String() string

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL