Documentation
¶
Overview ¶
Package json reads JSON data (JSONL or JSON array format) and converts it to the columnar RecordBatch format used by Wadjet's execution engine.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type ColumnarReader ¶
type ColumnarReader struct {
// contains filtered or unexported fields
}
ColumnarReader reads JSON directly into columnar vectors using a custom tokenizer that avoids encoding/json's per-token boxing overhead. Schema inference still uses encoding/json's Token API for correctness, but the main parse pass operates on raw bytes.
func NewColumnarReader ¶
func NewColumnarReader(data []byte) (*ColumnarReader, error)
NewColumnarReader creates a columnar reader from raw JSON bytes.
func NewColumnarReaderFromStream ¶
func NewColumnarReaderFromStream(r io.Reader) (*ColumnarReader, error)
NewColumnarReaderFromStream reads all data from r and creates a columnar reader.
func (*ColumnarReader) Next ¶
func (r *ColumnarReader) Next() (*batch.RecordBatch, error)
Next returns the next batch. Returns nil when exhausted.
func (*ColumnarReader) Schema ¶
func (r *ColumnarReader) Schema() []parquet.Column
Schema returns the inferred schema.
type Reader ¶
type Reader struct {
// contains filtered or unexported fields
}
Reader reads JSON data and produces RecordBatches.
func NewReaderFromBytes ¶
NewReaderFromBytes parses JSON data from a byte slice, auto-detecting whether the input is JSONL (newline-delimited) or a JSON array.
func NewReaderFromBytesWithCoercion ¶
NewReaderFromBytesWithCoercion parses JSON data and coerces values to match the inferred schema. This is the recommended entry point when mixed-type columns are expected.
func NewReaderFromStream ¶
NewReaderFromStream reads all data from r and parses it as JSON.
type StreamReader ¶
type StreamReader struct {
// contains filtered or unexported fields
}
StreamReader is the incremental counterpart of ColumnarReader: it parses JSON (top-level array of objects, or JSONL/concatenated objects) from an io.Reader one batch at a time, holding only a bounded byte window instead of the whole file plus a full columnar copy (issue #130 — read_json materialized ~2-3× the input in heap).
Schema semantics match the eager reader exactly: inferred from the first defaultSampleSize complete objects (the eager path samples the same prefix), capped at maxSampleBytes of buffered input. Values are parsed by the same scanObjectInto byte scanner, so output batches are identical to NewColumnarReader's for the same input.
func NewStreamReader ¶
func NewStreamReader(r io.Reader) (*StreamReader, error)
NewStreamReader buffers just enough input to infer the schema, then parses lazily. The caller retains ownership of r (close it after the reader is exhausted or abandoned).
func (*StreamReader) Next ¶
func (sr *StreamReader) Next() (*batch.RecordBatch, error)
Next parses and returns the next batch, or nil when exhausted.
func (*StreamReader) Schema ¶
func (sr *StreamReader) Schema() []parquet.Column
Schema returns the inferred schema (nil for empty input).
Source Files