parquetio

package

v1.8.0 Latest Latest Go to latest Published: Oct 19, 2026 License: MIT Imports: 16 Imported by: 0

Details

Valid go.mod file
Redistributable license
Tagged version
Stable version
Learn more about best practices

Repository

github.com/blackbirdworks/gopherstack

Links

Open Source Insights

Documentation ¶

Overview ¶

Package parquetio reads and writes Parquet files for the SQL engines (Athena, S3 Select, Firehose format conversion).

Index ¶

Variables
func Write(w io.Writer, cols []Column, rows [][]any, compression string) error
type Column
type File
- func Open(data []byte) (*File, error)

Constants ¶

This section is empty.

Variables ¶

View Source

var (
	// ErrUnsupportedType is returned for a column type the writer cannot encode.
	ErrUnsupportedType = errors.New("unsupported parquet column type")
	// ErrBadValue is returned when a cell does not parse as its column type.
	ErrBadValue = errors.New("invalid parquet cell value")
	// ErrBadFile is returned when the bytes are not a readable Parquet file.
	ErrBadFile = errors.New("invalid parquet file")
)

Functions ¶

func Write ¶

func Write(w io.Writer, cols []Column, rows [][]any, compression string) error

Write encodes rows as one Parquet file. Cells are nil, bool, int64, float64 or string; date/timestamp/decimal cells are strings. Nested Hive types (array/map/struct) are stored as UTF-8 strings.

Types ¶

type Column ¶

type Column struct {
	Name string
	Type string
}

Column is one top-level column to write; Type is a Hive/Athena type name.

type File ¶

type File struct {
	// contains filtered or unexported fields
}

File is an opened Parquet file held in memory.

func Open ¶

func Open(data []byte) (*File, error)

Open parses the footer of an in-memory Parquet file.

func (*File) Columns ¶

func (f *File) Columns() []string

Columns lists the top-level column names in file order.

func (*File) ForEach ¶

func (f *File) ForEach(cols []string, fn func(vals []any) error) (err error)

ForEach streams every row, yielding one value per requested column (case-insensitive match; nil when the column is null or absent). Leaf values are bool, int64, float64 or string (date, timestamp, time and decimal are formatted as Athena prints them); nested columns yield JSON-marshalable map[string]any / []any.

func (*File) NumRows ¶

func (f *File) NumRows() int64

NumRows is the total row count from the footer.

func (*File) ScanBytes ¶

func (f *File) ScanBytes(cols []string) int64

ScanBytes is the footer plus the compressed size of every column chunk belonging to the named top-level columns (case-insensitive).

Source Files ¶

View all Source files

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL