Help us shape the future of Go! Take the 2026 Go Developer Survey

ndjson

package
v0.6.8 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 26, 2026 License: MIT Imports: 22 Imported by: 0

Documentation

Overview

Package ndjson reads newline-delimited JSON (step 334): Polars' read_ndjson and scan_ndjson, a JSON object a line, a column a key.

Index

Constants

View Source
const DefaultInferLines = 100

DefaultInferLines is how many lines a schema is inferred from, Polars' infer_schema_length.

Variables

This section is empty.

Functions

This section is empty.

Types

type File

type File struct {
	Name string
	Open func(ctx context.Context) (io.ReadCloser, error)
}

File is one NDJSON stream, named in Explain and in an error, and opened afresh for the schema and for each run. ctx is the query's, and outlives the stream, which the reader closes when it is done with it.

type Options

type Options struct {
	Schema       *dtype.Schema // nil to infer one
	InferLines   int           // lines to infer from; 0 every line
	IgnoreErrors bool

	// Overrides are some columns' types, the schema's others given or inferred,
	// Polars' schema_overrides (step 345).
	Overrides map[string]dtype.DataType

	// JSON reads each stream as a JSON document, an array of objects or one object,
	// an object a row, as Polars' read_json (step 345). A key an inferred schema
	// has not is refused, as Polars refuses it, where NDJSON's is skipped.
	JSON bool

	// PathColumn, when set, is a String column of each row's file path after the
	// files' columns, as Polars' include_file_paths (step 359).
	PathColumn string
}

Options are a scan's: its schema, or how many lines to infer one from, the types of some of its columns, and whether a value its column cannot take is a null rather than an error, Polars' ignore_errors.

type Source

type Source struct {
	// contains filtered or unexported fields
}

Source is NDJSON streams as one scan, read in order: of the given schema, or of the one the first stream's first lines infer, as Polars infers several files' from the first.

func New

func New(desc string, files []File, o Options) *Source

New is a source of files, described as desc in Explain. Every column is nullable, whatever a given schema declares: a line may leave any key out.

func (*Source) Caps

func (s *Source) Caps() plan.Caps

func (*Source) Describe

func (s *Source) Describe() string

func (*Source) Name

func (s *Source) Name() string

func (*Source) Open

func (s *Source) Open(ctx context.Context, spec source.ScanSpec) (source.BatchSource, error)

Open reads the streams a batch at a time, the projection's columns only.

func (*Source) RowsOnly added in v0.6.8

func (s *Source) RowsOnly(source.ScanSpec) bool

RowsOnly: a scan of no columns is its rows (step 370).

func (*Source) Schema

func (s *Source) Schema(ctx context.Context) (*dtype.Schema, error)

Schema is the given schema, or the one the first lines infer, read once.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL