Documentation
¶
Overview ¶
Package arrowio supplies streamio's Apache Arrow IPC (file format) capabilities: decoding into canonical records and encoding records back into Arrow IPC documents. There is no raw-passthrough route — see NewRawSource's absence and encoder.go's doc for why.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func NewDecoder ¶
NewDecoder opens src as a stream of canonical records. NewDecoder does not take ownership of src.Reader; the caller closes it, if it needs closing, once done with every decoder Split hands out. The caller must still Close the returned decoder, to release any record batch it holds.
func NewEncoder ¶
func NewEncoder(cfg options.Config) (formatio.RecordEncoder, error)
NewEncoder returns an encoder rendering a whole batch of canonical records as one self-contained Arrow IPC file, one document per batch rather than per record.
The schema is derived from the first batch and then fixed: column type comes from each column's first non-null value, an untyped column becomes a string column, and a later record whose shape doesn't match is an error rather than a silent coercion. Every column is nullable, since there's no cheap way to know up front which fields will stay non-null — mirrors parquetio.NewEncoder's identical contract. Unlike parquetio, record.KindMap and record.KindList are supported: Arrow's StructType/ListType map onto them directly, so a nested value round-trips through Arrow the same way it already does through jsonio. SingleFileOutput selects the streaming encoder below instead: see NewStreamingEncoder.
func NewStreamingEncoder ¶
func NewStreamingEncoder(_ options.Config) (formatio.RecordEncoder, error)
NewStreamingEncoder returns a formatio.FinalizableRecordEncoder producing one continuous Arrow IPC file, with one record batch per call to EncodeBatch, across every batch in the run.
Types ¶
This section is empty.
Source Files