Help us shape the future of Go! Take the 2026 Go Developer Survey

csv

package
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 4, 2026 License: MIT Imports: 15 Imported by: 0

Documentation

Overview

Package csv reads and writes delimited text files.

It is a streaming source: it never holds more than one batch plus one buffer, so a file larger than memory is a normal case rather than a special one.

What it pushes down

Projection, by not PARSING the columns nobody asked for. Field splitting is unavoidable — field 7 cannot be found without walking past 1–6 — but converting bytes to values is skipped for every column outside the projection, which is where essentially all the cost is.

ScanSpec.MaxRows, by stopping. Head(10) on a 10 GB file is the first thing anyone types, and it should take a millisecond.

Predicates: nothing. See Caps.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Opener

type Opener func() (io.ReadCloser, error)

Opener produces a fresh reader over the same bytes.

A source is opened more than once — once to infer the schema, once per execution — and an io.Reader cannot be rewound. Taking a factory instead of a Reader makes that explicit and lets the same type serve a file, a byte slice, or an object store later, without this package knowing which.

func FileOpener

func FileOpener(path string) Opener

FileOpener returns an Opener for a path.

type Options

type Options struct {
	// Separator and Quote are single bytes. Multi-byte separators are not
	// supported: they are rare, and supporting them would put a byte-compare in
	// the innermost loop of the scanner for every file.
	Separator byte
	Quote     byte

	// Comment, if non-empty, makes any line starting with it a comment. It is a
	// byte slice rather than a byte because "//" and "#!" are both real.
	Comment []byte

	// HasHeader treats the first record as column names.
	HasHeader bool

	// ColumnNames overrides the names, whether or not there is a header. With
	// HasHeader false and no names, columns are called column_1, column_2, …
	ColumnNames []string

	// SkipRows discards this many records before the header is read. Files with a
	// banner above the header are common enough that leaving this out means
	// telling people to preprocess with sed.
	SkipRows int

	// Schema, if set, skips inference entirely. This is the option to reach for
	// when the file is big: inference costs a read of the sample, and an explicit
	// schema also removes the risk of a column being inferred from rows that do
	// not represent it.
	Schema *dtype.Schema

	// SchemaOverrides fixes individual columns while inferring the rest. The
	// common case is one identity column that looks numeric and must not be.
	SchemaOverrides map[string]dtype.DataType

	// InferRows bounds how many records inference reads. 0 means the whole file,
	// which is exact and, on a large file, expensive.
	InferRows int

	// NullValues are the texts that mean NULL, in addition to the empty string for
	// non-string columns. Case-sensitive.
	NullValues []string

	// TruncateRaggedLines accepts records with the wrong field count: extra fields
	// are dropped, missing ones become null. Off by default, because a changed
	// field count usually means the file is not what the reader thinks it is, and
	// silently padding with nulls turns that into a data problem discovered much
	// later.
	TruncateRaggedLines bool

	// MaxRecordSize bounds one record. 0 selects the default (16 MiB).
	MaxRecordSize int
}

Options configures a CSV source. The zero value is not usable; call DefaultOptions and adjust.

func DefaultOptions

func DefaultOptions() Options

DefaultOptions returns RFC 4180 defaults with a header.

type Source

type Source struct {
	// contains filtered or unexported fields
}

Source is one or more CSV files read as a single table.

Several files are read as several STREAMS, not as concatenated bytes. Each has its own header to skip and its own leading rows to skip, and gluing the bytes together would turn every header after the first into a data row — which for an inferred Int64 column is a parse error on a file that is not actually malformed.

func FromFile

func FromFile(path string, o Options) *Source

FromFile builds a source over a path.

func FromFiles

func FromFiles(paths []string, desc string, o Options) *Source

FromFiles builds a source over several paths, read in the order given.

func New

func New(open Opener, desc string, o Options) *Source

New builds a source over anything an Opener can produce. desc appears in Explain output.

func NewMulti

func NewMulti(opens []Opener, desc string, o Options) *Source

NewMulti builds a source over several streams read in order. The schema is inferred from the first; the rest must match it.

func (*Source) Caps

func (s *Source) Caps() plan.Caps

Caps reports projection support and NO predicate support.

A CSV file has no statistics: no min/max, no row groups, no page index, no dictionary. Every fact about a value requires reading and parsing the bytes, and once the bytes are parsed the row is already in a batch and the engine's own filterOp will apply the predicate faster and — unlike a second implementation living here — correctly.

There IS one useful thing a CSV reader could do with a predicate: parse only the filter columns, evaluate, and parse the rest for surviving rows only. Polars does this. It is not here because a source at import level 45 cannot import internal/physical at level 50 and therefore cannot call physical.Eval, so doing it would mean a second implementation of the promotion lattice and Kleene logic inside this package. Step 2's audit found the running-schema walk had been written four times and two copies had already diverged; a second evaluator would be that defect with higher stakes.

The seam, when it is wanted, is a compiled predicate closure supplied by the physical planner through ScanSpec. Note that Source does not implement plan.PredicatePushdown at all, so plan.ClassifyPredicates returns Unsupported for every conjunct without this type having to say so.

The one pushdown a CSV reader CAN serve, and does, is ScanSpec.MaxRows: stopping after N rows turns Head(10) on a 10 GB file from minutes into milliseconds.

func (*Source) Describe

func (s *Source) Describe() string

func (*Source) Name

func (s *Source) Name() string

func (*Source) Open

func (s *Source) Open(ctx context.Context, spec source.ScanSpec) (source.BatchSource, error)

func (*Source) Schema

func (s *Source) Schema(ctx context.Context) (*dtype.Schema, error)

Schema infers the schema on first use and caches it. Inference reads the file, so it happens at most once per Source regardless of how many times a plan is resolved.

type WriteOptions

type WriteOptions struct {
	Separator byte
	Quote     byte

	// Header writes the column names as the first record.
	Header bool

	// LineTerminator defaults to "\n". Set it to "\r\n" for consumers that need it.
	LineTerminator string

	// NullValue is the text written for a null. It defaults to the empty string,
	// which reads back as null for every type EXCEPT String — where "" is a real
	// value and the reader cannot tell the two apart.
	//
	// That asymmetry is a property of CSV, not a bug here: the format has no way to
	// distinguish an absent value from an empty one. A round trip that must
	// preserve null strings needs a sentinel on both sides — WithNullValue("\\N")
	// when writing and WithNullValues("\\N") when reading.
	NullValue string
}

WriteOptions configures a CSV writer.

func DefaultWriteOptions

func DefaultWriteOptions() WriteOptions

DefaultWriteOptions returns RFC 4180 defaults with a header.

type Writer

type Writer struct {
	// contains filtered or unexported fields
}

Writer writes batches as CSV.

It is a streaming sink: nothing accumulates, so a query whose result does not fit in memory writes correctly as long as each batch does.

func NewWriter

func NewWriter(w io.Writer, o WriteOptions) *Writer

NewWriter builds a writer over w. Close must be called to flush.

func (*Writer) Close

func (w *Writer) Close() error

Close flushes. It does not close the underlying writer — whoever opened it closes it, which is the only rule that works when the destination might be os.Stdout.

func (*Writer) WriteBatch

func (w *Writer) WriteBatch(b *data.Batch) error

WriteBatch appends a batch. The first call fixes the schema and writes the header; later batches must match it.

func (*Writer) WriteEmpty

func (w *Writer) WriteEmpty(s *dtype.Schema) error

WriteEmpty emits just the header for a result with no rows, so a downstream reader still sees the columns.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL