Documentation
¶
Overview ¶
Package arrow provides Arrow IPC (Feather V2) import and export for the pulse I/O pipeline, plus shared Arrow<->Pulse type-mapping helpers used by both this package and io/parquet.
Format scope:
- This package implements the Arrow IPC FILE format (random-access, footer-indexed). The standard file extensions are .arrow and .feather; Feather V2 is identical to the Arrow IPC file format on disk.
- The streaming variant (Arrow IPC stream, .arrows) is intentionally NOT supported. The file format is sufficient for batch import/export and trivially seekable, which lets the reader iterate batches without speculative buffering.
Memory model:
The reader materializes the full table into memory at init time, mirroring the policy in io/parquet. Files larger than available memory are out of scope for v1; users that need bounded streaming should use NDJSON or CSV.
Type policy:
All values written by Writer are encoded as Arrow String. Type-promotion on write is intentionally deferred until a real consumer demands it; this matches the parquet writer policy and keeps the export path simple.
Index ¶
- Constants
- func AppendDecimal128(b array.Builder, f encoding.Field, v any) error
- func AppendOverlayLayerStruct(structBldr *array.StructBuilder, layer *types.OverlayLayer) error
- func FieldFromPulse(f encoding.Field) arrow.Field
- func FormatValue(arr arrow.Array, idx int) string
- func JSONMarshalOverlay(v any) ([]byte, error)
- func OverlaysArrowField() arrow.Field
- func OverlaysFieldType() *arrow.StructType
- func PulseFieldFromArrow(af arrow.Field) (encoding.Field, bool)
- func ReadOverlaysFromArray(listArr arrow.Array) ([]*types.OverlayLayer, error)
- func TypeFromPulse(ft encoding.FieldType) arrow.DataType
- func TypeToPulse(dt arrow.DataType) encoding.FieldType
- type Reader
- func (r *Reader) Close() error
- func (r *Reader) InferPulseSchema() (*encoding.Schema, error)
- func (r *Reader) ReadHeader() ([]string, error)
- func (r *Reader) ReadOverlays() ([]*types.OverlayLayer, error)
- func (r *Reader) ReadRows(ctx context.Context, fn func(row []string) error) error
- func (r *Reader) Reset() error
- type Writer
Constants ¶
const OverlaysFieldName = "overlays"
OverlaysFieldName is the top-level Arrow field name that carries an overlay sidecar column family. Reserved namespace — pulse cohort schemas never produce a field with this name (schema.Field.Name is validated against the source schema, which does not allow synthesised names with this label).
Per research/export-embedding-shape.md § 3.1, the field rides at the SAME nesting level as the host record fields (top-level of the Arrow Schema). It is structurally absent (the field is OMITTED, not present with-empty-list) when IncludeOverlays resolves to false OR when Response.Overlays is empty.
const (
PulseTypeMetadataKey = "pulse:type"
)
Pulse field metadata key carried as an Arrow Field.Metadata pair so a reader can recover the original Pulse type for fields that map to a generic Arrow type. Decimal128 carries its precision/scale natively in arrow.Decimal128Type and does not need an extension marker.
Variables ¶
This section is empty.
Functions ¶
func AppendDecimal128 ¶
AppendDecimal128 appends a decimal value to an Arrow Decimal128 builder, accepting either an encoding.Decimal128 typed value or a canonical decimal string at the field's scale. nil and "" append a null. Used by the Arrow IPC and Parquet writers.
func AppendOverlayLayerStruct ¶ added in v0.19.0
func AppendOverlayLayerStruct(structBldr *array.StructBuilder, layer *types.OverlayLayer) error
AppendOverlayLayerStruct appends one struct entry to the overlay list's value builder. Field order matches OverlaysFieldType.
Exported so io/parquet can populate the same struct schema through its own pqarrow-fed RecordBuilder (research/export-embedding-shape.md § 4.4).
func FieldFromPulse ¶
FieldFromPulse builds an Arrow Field for a Pulse schema entry, including per-type details (decimal128 precision/scale). Arrow's nullable flag is driven by the Pulse field's Nullable attribute.
func FormatValue ¶
FormatValue renders a single element of an Arrow array as a string suitable for emission through the pio.Reader interface. Lifted from the original io/parquet implementation. Null handling is the caller's responsibility: FormatValue assumes idx is a non-null position.
Formatting rules:
- Integers: base-10, unsigned where possible.
- Float32 / Float64: shortest round-trippable decimal at the given precision (strconv 'f', precision -1).
- Boolean: lowercase "true" / "false".
- String: the raw value verbatim.
- Date32: rendered as YYYY-MM-DD using days-since-epoch.
- Date64: rendered as YYYY-MM-DD; the sub-day milliseconds are dropped.
- Dictionary with a string value type: the resolved string label.
- Anything else: falls back to fmt-formatting GetOneForMarshal, which yields the Arrow library's canonical text representation.
func JSONMarshalOverlay ¶ added in v0.19.0
JSONMarshalOverlay centralises encoding/json marshalling of overlay sub-payloads (Ref / Payload / Summary) so the Writer path and the helper builders share one error surface. All overlay JSON in the Arrow embedding goes through encoding/json per CLAUDE.md "Structural defense bans".
Exported so io/parquet (which rides the same Arrow LIST<STRUCT> schema through the pqarrow bridge per research/export-embedding-shape.md § 4.4) can share the marshaller without duplicating it.
func OverlaysArrowField ¶ added in v0.19.0
OverlaysArrowField returns the top-level Arrow Field carrying the overlay column family. Caller appends this to the schema when overlays should be embedded.
Exported so io/parquet can append the same field to the Arrow schema it hands the pqarrow FileWriter (research/export-embedding-shape.md § 4.4).
func OverlaysFieldType ¶ added in v0.19.0
func OverlaysFieldType() *arrow.StructType
OverlaysFieldType returns the fixed Arrow `LIST<STRUCT>` shape used for every overlay column family per research/export-embedding-shape.md § 3.1. The struct schema is FIXED — every Arrow writer emits the SAME nested struct schema regardless of which overlay kinds were produced so downstream Arrow consumers (DuckDB, polars, Spark) can register the schema once and reuse it.
The nested payload arms (scalar / series / matrix) are present in the struct with NULL cells for absent arms — exactly one arm is populated per layer per the OverlayPayload.Shape discriminator. The dispatch is inline on `shape` (UTF8).
Exported so io/parquet can ride the SAME schema through the pqarrow bridge (research/export-embedding-shape.md § 4.4 "Why Parquet shares the Arrow schema").
func PulseFieldFromArrow ¶
PulseFieldFromArrow inverts FieldFromPulse for decimal128 columns. Returns (field, true) if a Pulse decimal type was identified; (zero, false) otherwise — the caller should fall back to TypeToPulse.
func ReadOverlaysFromArray ¶ added in v0.19.0
func ReadOverlaysFromArray(listArr arrow.Array) ([]*types.OverlayLayer, error)
ReadOverlaysFromArray walks the LIST<STRUCT> overlay array and reconstructs []*types.OverlayLayer. The reader scans every non-empty list element across every record-batch and returns the first non- empty list (the canonical writer emits the populated list in row 0 of batch 0; reading any earlier row is sufficient). Returns nil when the array is structurally absent or every list element is empty — nil signals "no overlays present" to callers.
Exported so io/parquet can rebuild []*OverlayLayer from the pqarrow- materialised Arrow array (research/export-embedding-shape.md § 4.3 "Round-trip rule").
func TypeFromPulse ¶
TypeFromPulse maps a Pulse FieldType to an Arrow DataType for use when constructing an Arrow schema for export. Nullability is carried by the arrow.Field's Nullable flag, not the data type.
func TypeToPulse ¶
TypeToPulse maps an Arrow data type to a Pulse FieldType. Nullability is carried separately by encoding.Field.Nullable (driven by the Arrow Field's nullable flag at the call site).
Mapping rules:
- Signed and unsigned integers of the same width collapse to Pulse's unsigned type at that width (Pulse has no signed integer types).
- Both Date32 and Date64 collapse to FieldTypeDate; Pulse stores dates as days-since-epoch and discards the sub-day precision Date64 carries.
- String and binary (in both standard and large variants), and any dictionary-encoded type, collapse to FieldTypeCategoricalU8. The import pipeline upgrades to wider categorical widths when the dictionary overflows.
- Anything unrecognized falls back to FieldTypeF64.
Types ¶
type Reader ¶
type Reader struct {
// contains filtered or unexported fields
}
Reader reads Arrow IPC (Feather V2) data and implements pio.Reader and pio.ResetReader. The full file is materialized in memory at init time — see the package doc for the rationale.
func NewReader ¶
NewReader creates an Arrow IPC reader that loads from the given filesystem path on first read. The file is not opened until ReadHeader, ReadRows, or InferPulseSchema is called.
func NewReaderFromBytes ¶
NewReaderFromBytes creates an Arrow IPC reader over an in-memory byte slice. The slice is not copied; callers must not mutate it after handing it to the reader.
func (*Reader) Close ¶
Close releases all materialized record batches and the underlying file reader. Safe to call multiple times.
func (*Reader) InferPulseSchema ¶
InferPulseSchema builds a Pulse schema from the Arrow file's schema using the shared TypeToPulse mapping.
func (*Reader) ReadHeader ¶
ReadHeader returns column names from the Arrow schema.
A top-level "overlays" field (the overlay column-family sidecar emitted by SetOverlays per research/export-embedding-shape.md § 3.1) is filtered out of the returned header — host-data consumers that were written before overlay embedding existed still see the original column list. Callers that want the embedded overlays read them via ReadOverlays.
func (*Reader) ReadOverlays ¶ added in v0.19.0
func (r *Reader) ReadOverlays() ([]*types.OverlayLayer, error)
ReadOverlays extracts the embedded overlay layers from the top-level "overlays" sidecar column emitted by a SetOverlays-enabled Writer. Returns nil when the Arrow file carries no overlays sidecar (the "overlays" field is absent from the schema) or when every list element is empty.
Per research/export-embedding-shape.md § 3.3 the reader walks the first non-empty list element across the first record-batch; the canonical writer emits the populated list in row 0 of batch 0 (§ 3.2 "emit once" optimisation), so this scan terminates on the first row.
func (*Reader) ReadRows ¶
ReadRows iterates rows across all materialized record batches, calling fn for each row. Null cells produce empty strings.
The top-level "overlays" sidecar column (when present) is filtered from the row tuple — host-data row consumers see only the host columns. Use ReadOverlays to extract the embedded overlay layers.
type Writer ¶
type Writer struct {
// contains filtered or unexported fields
}
Writer writes tabular data as Arrow IPC (Feather V2), accumulating rows into an Arrow RecordBuilder and flushing in batch-sized chunks to bound peak memory.
func NewWriter ¶
NewWriter creates an Arrow IPC writer targeting a filesystem path. The file is materialized in an internal buffer and flushed to fs on Close.
func NewWriterToBuffer ¶
func NewWriterToBuffer() *Writer
NewWriterToBuffer creates an Arrow IPC writer that writes to an internal buffer accessible via Bytes after Close.
func (*Writer) Close ¶
Close flushes any pending batch, closes the underlying Arrow IPC writer, and writes the buffered file to fs if a path was configured. If no header was written, Close is a no-op and Bytes returns an empty slice. If a header was written but no rows, an empty record batch is emitted so the file is structurally valid.
func (*Writer) SetOverlays ¶ added in v0.19.0
func (w *Writer) SetOverlays(layers []*types.OverlayLayer)
SetOverlays records the Response.Overlays layers the export pipeline wants the writer to embed in the Arrow file. The layers ride as a top-level "overlays" LIST<STRUCT> column whose populated value lands in the first record-batch and whose subsequent record-batches carry an empty list — per research/export-embedding-shape.md § 3.1 / § 3.2. nil or empty layers leaves the Arrow output byte-identical to a pre-overlay export (the "overlays" field is OMITTED from the schema rather than emitted with empty lists). Implements pio.OverlayAwareWriter.
Must be called BEFORE WriteHeader so the lazily-built Arrow schema includes the overlay field. Calling after the writer has been initialised has no effect on the schema.
func (*Writer) SetPulseSchema ¶
SetPulseSchema records the source .pulse schema so subsequent initWriter can build native typed Arrow columns. Implements pio.SchemaAwareWriter.
func (*Writer) WriteHeader ¶
WriteHeader records the column names. The Arrow schema and writer are constructed lazily on the first WriteRow so that callers can WriteHeader and immediately Close to produce a schema-only file.
func (*Writer) WriteRow ¶
WriteRow writes a single row of values. With a Pulse schema set the row dispatches per-column to the native typed builder for that column's Pulse type. Without a schema every value is stringified via fmt.Sprintf and appended to a StringBuilder. Buffered rows flush in defaultRecordBatchSize chunks; batch boundaries fall between rows.
When SetOverlays handed the writer non-empty layers, the row also appends to the trailing overlays LIST<STRUCT> column. Row 0 of the first record-batch carries the populated layer list; every subsequent row (including subsequent record-batches) carries an empty list per research/export-embedding-shape.md § 3.2 "emit once" optimisation.
Source Files