Documentation
¶
Overview ¶
Package ndjson provides NDJSON (newline-delimited JSON) import and export for the pulse I/O pipeline.
Package ndjson provides NDJSON (newline-delimited JSON) import and export for the pulse I/O pipeline.
Index ¶
Constants ¶
const OverlayTrailerKey = "_overlays"
OverlayTrailerKey is the JSON object key the trailing sidecar line carries. Reserved namespace — pulse cohort schemas never produce a field with this name (the underscore prefix is reserved per the Crosstab long-form `_margin` precedent). Readers detect the trailer by checking whether a line's parsed object contains this key.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Reader ¶
type Reader struct {
// contains filtered or unexported fields
}
Reader reads NDJSON data from a byte source. Each line is one JSON object.
func NewReaderFromBytes ¶
NewReaderFromBytes creates an NDJSON reader from raw bytes.
func (*Reader) ReadHeader ¶
ReadHeader returns column names derived from the keys of the first JSON object.
The object is a record as well as a column declaration, so its raw bytes are buffered and replayed as the first row of the following ReadRows pass. Only the KEYS of this object define the column set — a key appearing for the first time in a later object is not a column.
func (*Reader) ReadOverlays ¶ added in v0.19.0
func (r *Reader) ReadOverlays() ([]*types.OverlayLayer, error)
ReadOverlays walks the NDJSON source and returns the overlay layers carried by the trailing `{"_overlays": [...]}` line, if present. Returns nil (not error) when no trailer is found — matches the Arrow / Excel / Parquet "absent means nil" contract.
The reader scans every line; if multiple trailing-shaped lines are present (a defensive case the writer never produces) the LAST one wins. Lines whose first parsed key is not the trailer key are skipped silently — host record lines remain consumable through ReadRows.
ReadOverlays is independent of ReadHeader / ReadRows — the underlying byte source (data or afero path) is re-read so calling ReadOverlays before, after, or instead of ReadRows is safe.
func (*Reader) ReadRows ¶
ReadRows streams rows to fn. ReadHeader is auto-called if not yet invoked. The object ReadHeader consumed to derive the column names is replayed as the FIRST row here — NDJSON has no header line, so that object is a record too and dropping it is silent data loss.
func (*Reader) Reset ¶
Reset rewinds the reader to the beginning. The buffered first record is discarded with everything else; the next ReadHeader re-reads and re-buffers it, so a second pass yields the same rows as the first.
type Writer ¶
type Writer struct {
// contains filtered or unexported fields
}
Writer writes NDJSON data to a buffer.
func NewWriterToBuffer ¶
func NewWriterToBuffer() *Writer
NewWriterToBuffer creates an NDJSON writer that writes to an internal buffer.
func (*Writer) Close ¶
Close flushes and writes to the target path if configured. When SetOverlays handed the writer a non-empty []*OverlayLayer slice the trailing `{"_overlays": [...]}` block is appended to the internal buffer BEFORE the buffer is flushed to fs — so both Bytes() and the on-disk artefact carry the trailer. Idempotent: a second Close() re-flushes the same buffer without appending the trailer twice.
func (*Writer) SetExplicitNulls ¶ added in v0.38.0
SetExplicitNulls implements pio.NullAwareWriter.
func (*Writer) SetOverlays ¶ added in v0.19.0
func (w *Writer) SetOverlays(layers []*types.OverlayLayer)
SetOverlays records the Response.Overlays layers the export pipeline wants the writer to embed in the NDJSON file. Each layer lands inside a single trailing line `{"_overlays": [...]}` appended after the last host-record line per research/export-embedding-shape.md § 6. nil or empty layers leave the NDJSON output byte-identical to a pre-overlay export (no trailer line lands). Implements pio.OverlayAwareWriter.
Emission happens at Close() time so the host record stream stays untouched until the file is finalised. Must be called BEFORE Close.
func (*Writer) WriteHeader ¶
WriteHeader records the column names for use in subsequent WriteRow calls. NDJSON has no header line; the columns are used as keys in each JSON object.
Source Files