Help us shape the future of Go! Take the 2026 Go Developer Survey

csv

package
v0.6.8 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 26, 2026 License: MIT Imports: 29 Imported by: 0

Documentation

Overview

Package csv reads and writes delimited text files.

It is a streaming source: it never holds more than one batch plus one buffer, so a file larger than memory is a normal case rather than a special one.

What it pushes down

Projection, by not PARSING the columns nobody asked for. Field splitting is unavoidable — field 7 cannot be found without walking past 1–6 — but converting bytes to values is skipped for every column outside the projection, which is where essentially all the cost is.

ScanSpec.MaxRows, by stopping. Head(10) on a 10 GB file is the first thing anyone types, and it should take a millisecond.

Predicates: nothing. See Caps.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func CheckExtension added in v0.6.7

func CheckExtension(path string, o WriteOptions) error

CheckExtension refuses a path whose name does not end as the codec's files do, .gz for gzip and .zst for Zstandard, or which ends so without the codec, as Polars' check_extension does.

Types

type Compression added in v0.6.7

type Compression uint8

Compression is the codec the text is written in (step 354).

const (
	Uncompressed Compression = iota
	Gzip
	Zstd
)

type Opener

type Opener func(ctx context.Context) (io.ReadCloser, error)

Opener produces a fresh reader over the same bytes.

A source is opened more than once — once to infer the schema, once per execution — and an io.Reader cannot be rewound. Taking a factory instead of a Reader makes that explicit and lets the same type serve a file, a byte slice, or an object store later, without this package knowing which.

ctx is the query's: the one Schema and Open were given, so a stream that lives behind a network can be cancelled through it.

func FileOpener

func FileOpener(path string) Opener

FileOpener returns an Opener for a path.

type Options

type Options struct {
	// Separator and Quote are single bytes. Multi-byte separators are not
	// supported: they are rare, and supporting them would put a byte-compare in
	// the innermost loop of the scanner for every file.
	Separator byte
	Quote     byte

	// Comment, if non-empty, makes any line starting with it a comment. It is a
	// byte slice rather than a byte because "//" and "#!" are both real.
	Comment []byte

	// HasHeader treats the first record as column names.
	HasHeader bool

	// ColumnNames overrides the names, whether or not there is a header. With
	// HasHeader false and no names, columns are called column_1, column_2, …
	ColumnNames []string

	// SkipRows discards this many records before the header is read. Files with a
	// banner above the header are common enough that leaving this out means
	// telling people to preprocess with sed.
	SkipRows int

	// SkipLines discards this many lines before anything is read, by their line
	// ends alone, a quote read as any byte, as Polars' skip_lines (step 358). It and
	// SkipRows are not both set, as Polars refuses them both.
	SkipLines int

	// SkipRowsAfterHeader discards this many records after the header, or at the
	// start where there is none, a blank line one, as Polars' skip_rows_after_header.
	SkipRowsAfterHeader int

	// NoQuote reads a quote as any byte, as Polars' quote_char=None: no field is
	// quoted, and a separator or a line end always ends one.
	NoQuote bool

	// PathColumn, when set, is a String column of each row's file path after the
	// files' columns, as Polars' include_file_paths (step 359).
	PathColumn string

	// Schema, if set, skips inference entirely. This is the option to reach for
	// when the file is big: inference costs a read of the sample, and an explicit
	// schema also removes the risk of a column being inferred from rows that do
	// not represent it.
	Schema *dtype.Schema

	// SchemaOverrides fixes individual columns while inferring the rest. The
	// common case is one identity column that looks numeric and must not be.
	SchemaOverrides map[string]dtype.DataType

	// InferRows bounds how many records inference reads. 0 means the whole file,
	// which is exact and, on a large file, expensive.
	InferRows int

	// NullValues are the texts that mean NULL, in addition to the empty string for
	// non-string columns. Case-sensitive.
	NullValues []string

	// TruncateRaggedLines accepts a record of too many fields, the extra ones
	// dropped, as Polars' truncate_ragged_lines. A record of too few is padded with
	// nulls whatever it says, and a blank line read as a record of nulls, as Polars
	// reads them (step 357).
	TruncateRaggedLines bool

	// MaxRecordSize bounds one record. 0 selects the default (16 MiB).
	MaxRecordSize int

	// TryParseDates infers a column of dates, instants or times of day as a Date, a
	// Datetime in microseconds, in UTC where its values have offsets, or a Time, by
	// Polars' patterns, as Polars' try_parse_dates (step 355).
	TryParseDates bool

	// IgnoreErrors reads a value its column's type cannot take as a null, as Polars'
	// ignore_errors (step 357). A record of too many fields is refused still, and
	// text that is not UTF-8, as Polars refuses them.
	IgnoreErrors bool

	// NoInference reads every column as a String but those a schema or an override
	// types, as Polars' infer_schema=False.
	NoInference bool

	// AllowEmpty reads an empty file as a frame of no columns, as Polars'
	// raise_if_empty=False; otherwise it is refused.
	AllowEmpty bool

	// DecimalComma reads a float's and a Decimal's point as a comma, 1,5, as Polars'
	// decimal_comma: a value with a '.' is then no number.
	DecimalComma bool

	// Lossy reads text that is not UTF-8 with each invalid sequence a U+FFFD, as
	// Polars' encoding="utf8-lossy"; otherwise a String's such text is refused, as
	// Polars refuses it.
	Lossy bool
}

Options configures a CSV source. The zero value is not usable; call DefaultOptions and adjust.

func DefaultOptions

func DefaultOptions() Options

DefaultOptions returns RFC 4180 defaults with a header.

type QuoteStyle added in v0.6.7

type QuoteStyle uint8

QuoteStyle is when a field is quoted: Polars' quote_style (step 354).

const (
	// QuoteNecessary quotes a field holding the separator, the quote or a line
	// break, the empty string, and one beginning with a space: the default.
	QuoteNecessary QuoteStyle = iota
	// QuoteAlways quotes every field, the header's and a null's too.
	QuoteAlways
	// QuoteNever quotes no field.
	QuoteNever
	// QuoteNonNumeric quotes every field but a number and a null.
	QuoteNonNumeric
)

type Source

type Source struct {
	// contains filtered or unexported fields
}

Source is one or more CSV files read as a single table.

Several files are read as several STREAMS, not as concatenated bytes. Each has its own header to skip and its own leading rows to skip, and gluing the bytes together would turn every header after the first into a data row — which for an inferred Int64 column is a parse error on a file that is not actually malformed.

func FromFile

func FromFile(path string, o Options) *Source

FromFile builds a source over a path.

func FromFiles

func FromFiles(paths []string, desc string, o Options) *Source

FromFiles builds a source over several paths, read in the order given.

func New

func New(open Opener, desc string, o Options) *Source

New builds a source over anything an Opener can produce. desc appears in Explain output.

func NewMulti

func NewMulti(opens []Opener, desc string, o Options) *Source

NewMulti builds a source over several streams read in order. The schema is inferred from the first. Every later stream's header must name the same columns, in any order: each is matched to the first by name, and refused if it differs (see matchHeader).

func NewNamed added in v0.3.0

func NewNamed(opens []Opener, names []string, desc string, o Options) *Source

NewNamed is NewMulti with a name per stream, for its errors.

func (*Source) Caps

func (s *Source) Caps() plan.Caps

Caps reports projection support and NO predicate support.

A CSV file has no statistics: no min/max, no row groups, no page index, no dictionary. Every fact about a value requires reading and parsing the bytes, and once the bytes are parsed the row is already in a batch and the engine's own filterOp will apply the predicate faster and — unlike a second implementation living here — correctly.

There IS one useful thing a CSV reader could do with a predicate: parse only the filter columns, evaluate, and parse the rest for surviving rows only. Polars does this. It is not here because a source at import level 45 cannot import internal/physical at level 50 and therefore cannot call physical.Eval, so doing it would mean a second implementation of the promotion lattice and Kleene logic inside this package. Step 2's audit found the running-schema walk had been written four times and two copies had already diverged; a second evaluator would be that defect with higher stakes.

The seam, when it is wanted, is a compiled predicate closure supplied by the physical planner through ScanSpec. Note that Source does not implement plan.PredicatePushdown at all, so plan.ClassifyPredicates returns Unsupported for every conjunct without this type having to say so.

The one pushdown a CSV reader CAN serve, and does, is ScanSpec.MaxRows: stopping after N rows turns Head(10) on a 10 GB file from minutes into milliseconds.

func (*Source) Describe

func (s *Source) Describe() string

func (*Source) Name

func (s *Source) Name() string

func (*Source) Open

func (s *Source) Open(ctx context.Context, spec source.ScanSpec) (source.BatchSource, error)

func (*Source) RowsOnly added in v0.6.8

func (s *Source) RowsOnly(source.ScanSpec) bool

RowsOnly: a scan of no columns is its rows (step 370).

func (*Source) Schema

func (s *Source) Schema(ctx context.Context) (*dtype.Schema, error)

Schema infers the schema on first use and caches it. Inference reads the file, so it happens at most once per Source regardless of how many times a plan is resolved.

type WriteOptions

type WriteOptions struct {
	Separator byte
	Quote     byte

	// Header writes the column names as the first record.
	Header bool

	// LineTerminator defaults to "\n". Set it to "\r\n" for consumers that need it.
	LineTerminator string

	// NullValue is the text written for a null. It defaults to the empty string,
	// which reads back as null for every type. An empty String is written quoted,
	// as "", which reads back as the empty string, so the two stay apart — as
	// Polars writes and reads them. This used to say the format cannot tell them
	// apart; quoting can.
	NullValue string

	// QuoteStyle is when a field is quoted, Polars' quote_style.
	QuoteStyle QuoteStyle

	// DateFormat, DatetimeFormat and TimeFormat are the strftime formats a Date, a
	// Datetime and a Time are written in, Polars' date_format, datetime_format and
	// time_format. Empty, each is Polars' own: 2024-01-02; 2024-01-02T03:04:05.123456
	// to the instant's unit, and its offset where it has a zone, +0900; and
	// 03:04:05.000000000.
	DateFormat, DatetimeFormat, TimeFormat string

	// FloatPrecision is the digits after a float's point, Polars' float_precision;
	// -1 is as many as the float needs.
	FloatPrecision int

	// FloatScientific is 1 for every float in scientific notation, 1.5e0, and -1 for
	// none, Polars' float_scientific; 0 is Polars' text, scientific past 1e16 and
	// below 1e-5.
	FloatScientific int

	// DecimalComma writes a float's and a Decimal's point as a comma.
	DecimalComma bool

	// BOM begins the text with a UTF-8 byte-order mark, Polars' include_bom.
	BOM bool

	// Compression and CompressionLevel are the codec and its level, Polars'
	// compression and compression_level: gzip's 0 to 9, Zstandard's 1 to 22, and -1
	// the codec's default, 6 and 3.
	Compression      Compression
	CompressionLevel int

	// SkipExtensionCheck lets a file's name end as another codec's files do, which
	// CheckExtension refuses, as Polars' check_extension=False.
	SkipExtensionCheck bool
}

WriteOptions configures a CSV writer.

func DefaultWriteOptions

func DefaultWriteOptions() WriteOptions

DefaultWriteOptions returns RFC 4180 defaults with a header.

type Writer

type Writer struct {
	// contains filtered or unexported fields
}

Writer writes batches as CSV.

It is a streaming sink: nothing accumulates, so a query whose result does not fit in memory writes correctly as long as each batch does.

func NewWriter

func NewWriter(w io.Writer, o WriteOptions) (*Writer, error)

NewWriter builds a writer over w, its formats compiled and its codec begun. Close must be called to flush.

func (*Writer) Close

func (w *Writer) Close() error

Close flushes, and ends the codec's stream. It does not close the underlying writer — whoever opened it closes it, which is the only rule that works when the destination might be os.Stdout.

func (*Writer) WriteBatch

func (w *Writer) WriteBatch(b *data.Batch) error

WriteBatch appends a batch. The first call fixes the schema and writes the header; later batches must match it.

func (*Writer) WriteEmpty

func (w *Writer) WriteEmpty(s *dtype.Schema) error

WriteEmpty emits just the header for a result with no rows, so a downstream reader still sees the columns.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL