Documentation
¶
Overview ¶
Package csvio reads and writes CSV data as gobi Frames.
The reader infers each column's Arrow type from a user-supplied Go struct whose fields carry `csv:"header"`, `geom:"true"`, and optional `time:"…"` tags. Read parsing is delegated to Arrow's typed CSV reader, which parses each column directly into its Arrow buffer — no per-row `[]string` intermediary and no per-cell reflection dispatch. Columns tagged as geometry or time.Time are read as strings and post-transformed into their target types (WKB Binary / Timestamp[ns]) in a single bulk pass per column.
Index ¶
- Constants
- Variables
- func Read[T any](r io.Reader, opts *ReadOptions) (*gobi.Frame, error)
- func ReadChunksFunc[T any](r io.Reader, opts *ReadOptions, fn func(*gobi.Frame) error) error
- func ReadFile[T any](path string, opts *ReadOptions) (*gobi.Frame, error)
- func ReadFileChunksFunc[T any](path string, opts *ReadOptions, fn func(*gobi.Frame) error) error
- func ScanFile[T any](path string, opts *ReadOptions) *gobi.LazyFrame
- type Codec
- type ReadOptions
Constants ¶
const DefaultChunkRows = 64 * 1024
DefaultChunkRows is the row batch size Arrow's CSV reader emits when ReadOptions.ChunkRows is zero. Larger batches amortize per-batch overhead; smaller batches bound peak memory. 64k rows is a reasonable middle ground for typical wide tables.
Variables ¶
var ( ErrUnsupportedFieldType = errors.New("csvio: unsupported field type") ErrHeaderMissing = errors.New("csvio: expected header row not found in file") ErrRowFieldCountMismatch = errors.New("csvio: row field count does not match schema") ErrTimeParse = errors.New("csvio: cannot parse cell as time.Time") )
Errors.
var DefaultTimeLayouts = []string{ time.RFC3339Nano, time.RFC3339, "2006-01-02T15:04:05", "2006-01-02 15:04:05", "2006-01-02", }
DefaultTimeLayouts is the ordered list of time layouts tried by the CSV reader when a time.Time field has no `time:"…"` tag. Callers who need a different format should set the tag explicitly rather than mutating this slice.
var ErrChunksAborted = errors.New("csvio: chunk callback returned error")
ErrChunksAborted is returned by ReadChunksFunc when the user's callback returned a non-nil error. The original callback error is wrapped so callers can `errors.Is` / `errors.As` it out of the chain.
var ErrUnknownCodec = fmt.Errorf("csvio: unknown compression codec")
ErrUnknownCodec is returned when a codec name isn't recognized.
Functions ¶
func Read ¶
Read reads r into a Frame, inferring the schema from T. Streams reaching this function are treated as uncompressed unless opts.Compression is set explicitly (Read has no filename to inspect).
func ReadChunksFunc ¶
ReadChunksFunc reads r and invokes fn once per record batch (~64k rows by default; override via ReadOptions.ChunkRows). The Frame passed to fn has single-chunk columns matching the final output schema — tagged geometry/time columns have already been post-transformed. Peak memory is bounded to roughly one batch plus a working buffer for transforms.
The Frame's Arrow buffers are released after fn returns. To retain a Frame past the callback, call frame.Retain() before returning; the caller is then responsible for the matching Release().
If fn returns an error, iteration stops and the error is wrapped in ErrChunksAborted. If reading fails, the underlying Arrow error is returned directly.
func ReadFile ¶
ReadFile reads path into a Frame, inferring the schema from T. If opts.Compression is CodecAuto (the default), the codec is inferred from the filename's extension (`.gz`, `.zst`, `.bz2`).
func ReadFileChunksFunc ¶
ReadFileChunksFunc is the file variant of ReadChunksFunc: opens path, auto-detects compression from the filename (unless opts.Compression is explicitly set), and streams record-batch-sized Frames to fn.
func ScanFile ¶
func ScanFile[T any](path string, opts *ReadOptions) *gobi.LazyFrame
ScanFile returns a LazyFrame that reads path as CSV when Collect is called. Schema is inferred from the type parameter T (same struct-tag conventions as ReadFile[T]).
The scan participates in gobi's optimizer:
- Streaming: reads flow one batch at a time through ReadFileChunksFunc[T]. Peak memory stays bounded to one batch even on multi-GB inputs.
- Projection: gobi.Frame.Select() above the scan applies at the LazyFrame layer — the Frame drops unused columns before the executor sees them. CSV lacks random-access column projection at the parser level (arrow-csv panics if IncludeColumns is mixed with an explicit schema), so parse work is NOT reduced; memory downstream of the parse is. Adding true parser-level column skipping would require dropping the T-derived schema and using arrow-csv's WithIncludeColumns — a v2 concern.
- Predicate pushdown: not supported. CSV is sequential, so any predicate is evaluated by the executor above the scan.
Compression is auto-detected from the filename extension unless ReadOptions.Compression is set explicitly — same rules as ReadFile[T].
Types ¶
type Codec ¶
type Codec string
Codec selects the stream-compression codec used when reading a CSV.
const ( // CodecAuto is the zero value: infer the codec from the filename in // ReadFile, or treat as uncompressed when the source is a plain // io.Reader (Read). CodecAuto Codec = "" // CodecNone forces no decompression, regardless of filename. CodecNone Codec = "none" // CodecGzip decompresses via compress/gzip (RFC 1952, ".gz" files). CodecGzip Codec = "gzip" // CodecZstd decompresses via klauspost/compress/zstd (".zst" files). CodecZstd Codec = "zstd" // CodecBzip2 decompresses via compress/bzip2 (".bz2" files). Read-only; // the Go stdlib doesn't ship a bzip2 writer. CodecBzip2 Codec = "bzip2" )
type ReadOptions ¶
type ReadOptions struct {
// HasHeader indicates whether the first row is a header. Defaults to true.
HasHeader *bool
// Delimiter overrides the default comma. Any rune is accepted, e.g.
// '\t' for TSV or ';' for European-style CSV.
Delimiter rune
// Comment marks lines that should be skipped when starting with this rune.
Comment rune
// SkipRows drops the first N data rows (after any header). Implemented
// by consuming N line-terminated records from the raw stream before
// the Arrow reader sees them, so quoted embedded newlines inside
// skipped rows will not be counted correctly — reserve SkipRows for
// well-behaved (no quoted newlines in the skip zone) input.
SkipRows int
// CRSHint gives geometry columns a CRS when the CSV does not encode one.
CRSHint int32
// Allocator overrides the Arrow allocator.
Allocator memory.Allocator
// Compression selects the stream-compression codec used to decode the
// input. The zero value (CodecAuto) means "infer from the filename in
// ReadFile, or treat as uncompressed in Read." Set to CodecNone to
// force no decompression even if the filename suggests otherwise.
Compression Codec
// NullTokens is the set of cell values that decode as SQL null in
// addition to the empty string (which is always treated as null).
// Useful for CSVs that write "NA", "NULL", "N/A", etc.
NullTokens []string
// LazyQuotes tolerates broken quoting: unescaped quotes inside a
// quoted field, unquoted fields that begin with a quote, etc. Off
// by default because it can mask genuinely malformed input.
LazyQuotes bool
// UseCRLF signals that record separators are "\r\n" rather than "\n".
// Rare in practice; Arrow's reader auto-handles most cases without
// this hint.
UseCRLF bool
// ChunkRows overrides DefaultChunkRows. Values ≤ 0 use the default.
ChunkRows int
}
ReadOptions controls CSV parsing.
Source Files