Documentation
¶
Overview ¶
Package csvio reads and writes CSV data as gobi Frames.
The reader infers each column's Arrow type from a user-supplied Go struct whose fields carry `csv:"header"`, `geom:"true"`, and optional `time:"…"` tags. Read parsing is delegated to Arrow's typed CSV reader, which parses each column directly into its Arrow buffer — no per-row `[]string` intermediary and no per-cell reflection dispatch. Columns tagged as geometry or time.Time are read as strings and post-transformed into their target types (WKB Binary / Timestamp[ns]) in a single bulk pass per column.
For CSVs whose columns aren't known at compile time, ReadStrings and its file / streaming variants take columns from the header and read every cell as a String, with no type inference.
Write, WriteFile and WriteStructs write a Frame (or structs) as CSV in formats the readers parse back; NewWriter streams frames to one file in pieces.
Index ¶
- Constants
- Variables
- func Read[T any](r io.Reader, opts *ReadOptions) (*gobi.Frame, error)
- func ReadChunksFunc[T any](r io.Reader, opts *ReadOptions, fn func(*gobi.Frame) error) error
- func ReadFile[T any](path string, opts *ReadOptions) (*gobi.Frame, error)
- func ReadFileChunksFunc[T any](path string, opts *ReadOptions, fn func(*gobi.Frame) error) error
- func ReadFileStrings(path string, opts *ReadOptions) (*gobi.Frame, error)
- func ReadFileStringsChunksFunc(path string, opts *ReadOptions, fn func(*gobi.Frame) error) error
- func ReadStrings(r io.Reader, opts *ReadOptions) (*gobi.Frame, error)
- func ReadStringsChunksFunc(r io.Reader, opts *ReadOptions, fn func(*gobi.Frame) error) error
- func ReadStructs[T any](path string, opts *ReadOptions, structOpts ...gobi.StructOption) ([]T, error)
- func ReadStructsReader[T any](r io.Reader, opts *ReadOptions, structOpts ...gobi.StructOption) ([]T, error)
- func ScanFile[T any](path string, opts *ReadOptions) *gobi.LazyFrame
- func Write(f *gobi.Frame, w io.Writer, opts *WriteOptions) error
- func WriteFile(f *gobi.Frame, path string, opts *WriteOptions) (err error)
- func WriteStructs[T any](rows []T, path string, opts *WriteOptions, structOpts ...gobi.StructOption) error
- func WriteStructsWriter[T any](rows []T, w io.Writer, opts *WriteOptions, structOpts ...gobi.StructOption) error
- type Codec
- type ReadOptions
- type WriteOptions
- type Writer
Constants ¶
const DefaultChunkRows = 64 * 1024
DefaultChunkRows is the row batch size Arrow's CSV reader emits when ReadOptions.ChunkRows is zero. Larger batches amortize per-batch overhead; smaller batches bound peak memory. 64k rows is a reasonable middle ground for typical wide tables.
const DefaultDateFormat = "2006-01-02"
DefaultDateFormat is the layout Date32 and Date64 cells are written with when WriteOptions.DateFormat is empty.
Variables ¶
var ( ErrUnsupportedFieldType = errors.New("csvio: unsupported field type") ErrHeaderMissing = errors.New("csvio: expected header row not found in file") ErrRowFieldCountMismatch = errors.New("csvio: row field count does not match schema") ErrTimeParse = errors.New("csvio: cannot parse cell as time.Time") )
Errors.
var DefaultTimeLayouts = []string{ time.RFC3339Nano, time.RFC3339, "2006-01-02T15:04:05", "2006-01-02 15:04:05", "2006-01-02", }
DefaultTimeLayouts is the ordered list of time layouts tried by the CSV reader when a time.Time field has no `time:"…"` tag. Callers who need a different format should set the tag explicitly rather than mutating this slice.
var ErrChunksAborted = errors.New("csvio: chunk callback returned error")
ErrChunksAborted is returned by ReadChunksFunc when the user's callback returned a non-nil error. The original callback error is wrapped so callers can `errors.Is` / `errors.As` it out of the chain.
var ErrDuplicateHeader = errors.New("csvio: duplicate column name in header")
ErrDuplicateHeader is returned by the string readers when two header cells share a name; a Frame's columns are looked up by name.
var ErrInvalidDelimiter = errors.New("csvio: invalid delimiter")
ErrInvalidDelimiter is returned when WriteOptions.Delimiter can't separate CSV fields: a quote, carriage return, newline, or an invalid rune.
var ErrUnknownCodec = fmt.Errorf("csvio: unknown compression codec")
ErrUnknownCodec is returned when a codec name isn't recognized.
Functions ¶
func Read ¶
Read reads r into a Frame, inferring the schema from T. Streams reaching this function are treated as uncompressed unless opts.Compression is set explicitly (Read has no filename to inspect).
func ReadChunksFunc ¶
ReadChunksFunc reads r and invokes fn once per record batch (~64k rows by default; override via ReadOptions.ChunkRows). The Frame passed to fn has single-chunk columns matching the final output schema — tagged geometry/time columns have already been post-transformed. Peak memory is bounded to roughly one batch plus a working buffer for transforms.
The Frame's Arrow buffers are released after fn returns. To retain a Frame past the callback, call frame.Retain() before returning; the caller is then responsible for the matching Release().
If fn returns an error, iteration stops and the error is wrapped in ErrChunksAborted; errors.Is / errors.As match both. If reading fails, the underlying Arrow error is returned directly.
func ReadFile ¶
ReadFile reads path into a Frame, inferring the schema from T. If opts.Compression is CodecAuto (the default), the codec is inferred from the filename's extension (`.gz`, `.zst`, `.bz2`).
func ReadFileChunksFunc ¶
ReadFileChunksFunc is the file variant of ReadChunksFunc: opens path, auto-detects compression from the filename (unless opts.Compression is explicitly set), and streams record-batch-sized Frames to fn.
func ReadFileStrings ¶ added in v0.4.16
func ReadFileStrings(path string, opts *ReadOptions) (*gobi.Frame, error)
ReadFileStrings reads path into a Frame without a struct: columns come from the header and every column is String. No type inference runs, so a cell such as "00123" or "1e5" stays exactly as written. Compression is detected from the filename unless opts.Compression is set. See ReadStrings for the null and header rules.
func ReadFileStringsChunksFunc ¶ added in v0.4.16
ReadFileStringsChunksFunc is the streaming form of ReadFileStrings: fn gets one Frame per ChunkRows rows. Lifetime and error rules match ReadFileChunksFunc.
func ReadStrings ¶ added in v0.4.16
ReadStrings is the io.Reader counterpart to ReadFileStrings. Every column is String, nullable, and a single chunk.
- Columns are named from the header row. A UTF-8 byte-order mark at the start of the input is dropped, and a blank header cell is named f<i> (its zero-based index). Duplicate names return ErrDuplicateHeader. Without a header (HasHeader false), columns are f0, f1, ….
- Empty cells and opts.NullTokens become null, as in Read.
- A row with a different field count from the first returns ErrRowFieldCountMismatch.
Delimiter, Comment, LazyQuotes, SkipRows, Compression, NullTokens and Allocator apply as in Read; CRSHint, UseCRLF (line endings are detected) and ChunkRows don't.
func ReadStringsChunksFunc ¶ added in v0.4.16
ReadStringsChunksFunc is the streaming form of ReadStrings. fn gets one Frame per ChunkRows rows (DefaultChunkRows when unset), released after fn returns; call frame.Retain() to keep one. A header-only input calls fn once with a zero-row Frame so the columns are still visible. An error from fn stops the read and is wrapped in ErrChunksAborted.
func ReadStructs ¶ added in v0.3.0
func ReadStructs[T any](path string, opts *ReadOptions, structOpts ...gobi.StructOption) ([]T, error)
ReadStructs reads a CSV file and decodes each row into a T. Column names are matched against T's fields via gobi's tag resolution under the "csv" namespace: `csv:"col"` tags take priority, with fallback to `gobi:"col"` → field name. `csv:"-"` skips a field.
type Row struct {
ID int64 `csv:"id"`
Name string `csv:"name" gobi:"name"`
}
rows, err := csvio.ReadStructs[Row]("data.csv", nil)
Wraps ReadFile[T] + gobi.ToStructs. WriteStructs is the writing counterpart.
Strings and []byte fields are copied out of the intermediate Frame, which is released before returning, so the rows own their memory and don't pin read buffers. Pass gobi.StructInterner in structOpts to share one copy of each value of `intern`-tagged fields across calls.
func ReadStructsReader ¶ added in v0.3.0
func ReadStructsReader[T any](r io.Reader, opts *ReadOptions, structOpts ...gobi.StructOption) ([]T, error)
ReadStructsReader is the io.Reader-backed variant of ReadStructs.
Strings and []byte fields are copied out of the intermediate Frame, which is released before returning, so the rows own their memory and don't pin read buffers. Pass gobi.StructInterner in structOpts to share one copy of each value of `intern`-tagged fields across calls.
func ScanFile ¶
func ScanFile[T any](path string, opts *ReadOptions) *gobi.LazyFrame
ScanFile returns a LazyFrame that reads path as CSV when Collect is called. Schema is inferred from the type parameter T (same struct-tag conventions as ReadFile[T]).
The scan participates in gobi's optimizer:
- Streaming: reads flow one batch at a time through ReadFileChunksFunc[T]. Peak memory stays bounded to one batch even on multi-GB inputs.
- Projection: gobi.Frame.Select() above the scan applies at the LazyFrame layer — the Frame drops unused columns before the executor sees them. CSV lacks random-access column projection at the parser level (arrow-csv panics if IncludeColumns is mixed with an explicit schema), so parse work is NOT reduced; memory downstream of the parse is. Adding true parser-level column skipping would require dropping the T-derived schema and using arrow-csv's WithIncludeColumns — a v2 concern.
- Predicate pushdown: not supported. CSV is sequential, so any predicate is evaluated by the executor above the scan.
Compression is auto-detected from the filename extension unless ReadOptions.Compression is set explicitly — same rules as ReadFile[T].
func Write ¶ added in v0.4.19
Write writes f to w as CSV: a header row of the column names (unless opts.HasHeader is false), then one record per row. Fields are quoted as encoding/csv quotes them.
Cell formats, chosen so Read and ReadStrings read them back:
- Null cells: opts.NullValue (default: empty). An empty string cell is also written empty, so it reads back as null.
- Strings: as is, except that a CR LF inside a value reads back as LF (the reader normalizes it). Booleans: true / false.
- Integers: decimal. Floats: the shortest form that parses back to the same value, with NaN, +Inf and -Inf spelled that way. Decimals: with the column's scale, e.g. 12.50.
- Timestamp: opts.TimeFormat in the column's zone, UTC when it has none. Date32 / Date64: opts.DateFormat. Time32 / Time64: 15:04:05.999999999. Duration: the integer count in the column's unit.
- Geometry columns: WKT. The CRS isn't written; pass it back as ReadOptions.CRSHint.
- Other binary columns: standard base64.
- Lists, structs and maps: JSON.
- Dictionary columns: the dictionary value.
Any other type is written as Arrow formats it (Array.ValueStr).
func WriteFile ¶ added in v0.4.19
func WriteFile(f *gobi.Frame, path string, opts *WriteOptions) (err error)
WriteFile writes f to path as CSV. With opts.Compression unset, a .gz / .zst / .zstd name gzip- or zstd-compresses the output; a .bz2 name is an error. See Write for how each column type is written.
func WriteStructs ¶ added in v0.4.19
func WriteStructs[T any](rows []T, path string, opts *WriteOptions, structOpts ...gobi.StructOption) error
WriteStructs writes rows to path as CSV, one column per field. Column names and options come from gobi's tag resolution under the "csv" namespace, as in ReadStructs; structOpts are passed to gobi.FromStructs. Wraps gobi.FromStructs + WriteFile, so a `geom:"true"` string field is written as WKT and a zero time.Time as null.
func WriteStructsWriter ¶ added in v0.4.19
func WriteStructsWriter[T any](rows []T, w io.Writer, opts *WriteOptions, structOpts ...gobi.StructOption) error
WriteStructsWriter is the io.Writer counterpart to WriteStructs.
Types ¶
type Codec ¶
type Codec string
Codec selects the stream-compression codec used when reading or writing a CSV.
const ( // CodecAuto is the zero value: infer the codec from the filename in // ReadFile / WriteFile, or treat the stream as uncompressed when it is // a plain io.Reader / io.Writer (Read, Write, NewWriter). CodecAuto Codec = "" // CodecNone forces no compression either way, regardless of filename. CodecNone Codec = "none" // CodecGzip reads and writes via compress/gzip (RFC 1952, ".gz" files). CodecGzip Codec = "gzip" // CodecZstd reads and writes via klauspost/compress/zstd (".zst" files). CodecZstd Codec = "zstd" // CodecBzip2 decompresses via compress/bzip2 (".bz2" files). Read-only; // the Go stdlib doesn't ship a bzip2 writer. CodecBzip2 Codec = "bzip2" )
type ReadOptions ¶
type ReadOptions struct {
// HasHeader indicates whether the first row is a header. Defaults to true.
HasHeader *bool
// Delimiter overrides the default comma. Any rune is accepted, e.g.
// '\t' for TSV or ';' for European-style CSV.
Delimiter rune
// Comment marks lines that should be skipped when starting with this rune.
Comment rune
// SkipRows drops the first N data rows (after any header). Implemented
// by consuming N line-terminated records from the raw stream before
// the Arrow reader sees them, so quoted embedded newlines inside
// skipped rows will not be counted correctly — reserve SkipRows for
// well-behaved (no quoted newlines in the skip zone) input.
SkipRows int
// CRSHint gives geometry columns a CRS when the CSV does not encode one.
CRSHint int32
// Allocator overrides the Arrow allocator.
Allocator memory.Allocator
// Compression selects the stream-compression codec used to decode the
// input. The zero value (CodecAuto) means "infer from the filename in
// ReadFile, or treat as uncompressed in Read." Set to CodecNone to
// force no decompression even if the filename suggests otherwise.
Compression Codec
// NullTokens is the set of cell values that decode as SQL null in
// addition to the empty string (which is always treated as null).
// Useful for CSVs that write "NA", "NULL", "N/A", etc.
NullTokens []string
// LazyQuotes tolerates broken quoting: unescaped quotes inside a
// quoted field, unquoted fields that begin with a quote, etc. Off
// by default because it can mask genuinely malformed input.
LazyQuotes bool
// UseCRLF signals that record separators are "\r\n" rather than "\n".
// Rare in practice; Arrow's reader auto-handles most cases without
// this hint.
UseCRLF bool
// ChunkRows overrides DefaultChunkRows. Values ≤ 0 use the default.
ChunkRows int
}
ReadOptions controls CSV parsing.
type WriteOptions ¶ added in v0.4.19
type WriteOptions struct {
// HasHeader writes the column names as the first row. Defaults to
// true.
HasHeader *bool
// Delimiter overrides the default comma, e.g. '\t' for TSV.
Delimiter rune
// UseCRLF ends records with "\r\n" instead of "\n".
UseCRLF bool
// NullValue is written for null cells. The default, an empty field,
// is what Read and ReadStrings read back as null. A non-empty value
// is ambiguous with a string cell holding the same text.
NullValue string
// TimeFormat is the time.Format layout for Timestamp cells, written
// in the column's time zone (UTC for a column without one). Defaults
// to time.RFC3339Nano, the first layout Read tries.
TimeFormat string
// DateFormat is the layout for Date32 and Date64 cells. Defaults to
// DefaultDateFormat.
DateFormat string
// Compression selects the stream codec. The zero value (CodecAuto)
// infers it from the filename in WriteFile and writes uncompressed
// in Write and NewWriter. CodecBzip2 can't be written: the Go
// standard library has no bzip2 encoder.
Compression Codec
}
WriteOptions controls CSV output. A nil *WriteOptions uses the defaults.
type Writer ¶ added in v0.4.19
type Writer struct {
// contains filtered or unexported fields
}
Writer writes frames to one CSV stream in pieces, e.g. from ReadFileChunksFunc or a LazyFrame's batches, so the whole table never has to be in memory. The output is what Write produces for all the frames concatenated.
Not safe for concurrent use. After any error the Writer is unusable: later calls return the same error.
func NewWriter ¶ added in v0.4.19
NewWriter starts a CSV stream on w for frames with the given schema, writing the header row now (unless opts.HasHeader is false), so a stream closed after no Write is header-only. Invalid options fail here. Close finishes the stream but doesn't close w.
func (*Writer) Close ¶ added in v0.4.19
Close flushes buffered output and finishes the compression stream. It doesn't close the io.Writer passed to NewWriter. Calling it again returns nil.
Source Files