arrowsrc

Overview ¶

Package arrowsrc scans a stream of Arrow records: an IPC stream, a Flight response, a C Data Interface stream — anything that is an array.RecordReader.

A factory, not a reader ¶

A source is opened more than once: once to learn the schema when a query is planned, and once per scan per execution — a self-join scans it twice in one query, and a LazyFrame can be collected again. A RecordReader cannot be rewound. So the source holds a function that returns a fresh, independent reader every time, which is the rule ScanCSVReader states for byte streams.

Copied a window at a time ¶

A reader's record is valid only until its next Next, and a C Data record is freed through its producer on release. Each batch is copied out of the current record (see internal/arrowin), and the reader is not advanced until every row of that record has been copied — so nothing handed to the engine ever points into Arrow memory, and the reader's own contract holds without a Retain.

What the engine's calls cost ¶

• Schema opens a reader, reads its schema, and releases it — once per Source. Explain and CollectSchema stop there.
• Open does no I/O at all. The reader is opened on the first Next, so a scan that is planned and then abandoned — the left side of a join whose right side failed to plan — never opens one, and has nothing to leak.
• Close releases the reader. A blocking Next inside the reader cannot be interrupted by a context, so Close waits for it.