Documentation
¶
Overview ¶
Package ndjson reads newline-delimited JSON (step 334): Polars' read_ndjson and scan_ndjson, a JSON object a line, a column a key.
Index ¶
- Constants
- type File
- type Options
- type Source
- func (s *Source) Caps() plan.Caps
- func (s *Source) Describe() string
- func (s *Source) Name() string
- func (s *Source) Open(ctx context.Context, spec source.ScanSpec) (source.BatchSource, error)
- func (s *Source) RowsOnly(source.ScanSpec) bool
- func (s *Source) Schema(ctx context.Context) (*dtype.Schema, error)
Constants ¶
const DefaultInferLines = 100
DefaultInferLines is how many lines a schema is inferred from, Polars' infer_schema_length.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type File ¶
File is one NDJSON stream, named in Explain and in an error, and opened afresh for the schema and for each run. ctx is the query's, and outlives the stream, which the reader closes when it is done with it.
type Options ¶
type Options struct {
Schema *dtype.Schema // nil to infer one
InferLines int // lines to infer from; 0 every line
IgnoreErrors bool
// Overrides are some columns' types, the schema's others given or inferred,
// Polars' schema_overrides (step 345).
Overrides map[string]dtype.DataType
// JSON reads each stream as a JSON document, an array of objects or one object,
// an object a row, as Polars' read_json (step 345). A key an inferred schema
// has not is refused, as Polars refuses it, where NDJSON's is skipped.
JSON bool
// PathColumn, when set, is a String column of each row's file path after the
// files' columns, as Polars' include_file_paths (step 359).
PathColumn string
}
Options are a scan's: its schema, or how many lines to infer one from, the types of some of its columns, and whether a value its column cannot take is a null rather than an error, Polars' ignore_errors.
type Source ¶
type Source struct {
// contains filtered or unexported fields
}
Source is NDJSON streams as one scan, read in order: of the given schema, or of the one the first stream's first lines infer, as Polars infers several files' from the first.
func New ¶
New is a source of files, described as desc in Explain. Every column is nullable, whatever a given schema declares: a line may leave any key out.