Documentation
¶
Overview ¶
Package parquetio reads and writes frames as Parquet files, through github.com/apache/arrow-go's pure-Go Parquet implementation (no cgo).
The frame is laid out as package arrowio lays out an Arrow record: the same type mapping, the same handling of missing values, and the same "pandas" metadata, so that pandas reads the files back with their index and dtypes, and files pandas writes read here with theirs. See the arrowio documentation for the mapping.
Parquet lives in its own package because arrow-go's Parquet bridge links far more code (Thrift, every codec, and, through arrow/flight, gRPC) than the Arrow IPC formats: a program that only needs IPC imports arrowio alone.
Index ¶
- func ReadParquet(r arrowio.ReadAtSeeker, opts ...Option) (*dataframe.DataFrame, error)
- func ReadParquetFile(path string, opts ...Option) (*dataframe.DataFrame, error)
- func WriteParquet(w io.Writer, df *dataframe.DataFrame, opts ...Option) error
- func WriteParquetFile(path string, df *dataframe.DataFrame, opts ...Option) error
- type Compression
- type Option
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func ReadParquet ¶
ReadParquet reads a Parquet file, every row group, decoding the columns in parallel, and restores the index from "pandas" metadata as arrowio.FromRecord does. The only option it takes is WithAllocator.
func ReadParquetFile ¶
ReadParquetFile reads the Parquet file at path as ReadParquet does.
func WriteParquet ¶
WriteParquet writes a frame as a Parquet file, laid out as arrowio.ToRecord lays it out (index columns and "pandas" metadata included), with the Arrow schema stored in the file so that a reader gets the exact Arrow types back (dictionary columns, nanosecond timestamps with their zone). The writer is not closed.
Types ¶
type Compression ¶
type Compression int
Compression selects the codec of a written Parquet file.
const ( // Snappy is the default, as pandas writes it. Snappy Compression = iota Uncompressed Gzip Brotli Zstd // LZ4 is Parquet's LZ4_RAW. LZ4 )
The codecs.
type Option ¶
type Option func(*config)
Option configures a write or a read.
func WithAllocator ¶
WithAllocator sets the Arrow memory allocator used to read and write (default memory.DefaultAllocator).
func WithCompression ¶
func WithCompression(c Compression) Option
WithCompression sets the codec of a written file (default Snappy).
func WithRowGroupLength ¶
WithRowGroupLength sets the most rows a row group holds (default 1<<20, as pyarrow uses).
func WithTimeZone ¶
WithTimeZone sets the time zone written with datetime columns, as arrowio.WithTimeZone does (default "UTC"; "" writes naive timestamps).
func WithoutIndex ¶
func WithoutIndex() Option
WithoutIndex leaves the row labels out of a written file, as arrowio.WithoutIndex does.
Source Files