Documentation
¶
Overview ¶
Package arrowin imports Arrow data into ursus, by copying it.
Why a copy ¶
ursus stores Arrow's layout, so sharing the bytes looks free. It is not safe with today's types: bitmap.View, StringAccessor and ListAccessor hold raw []byte and cannot keep an owner alive; strings read through Get alias the character buffer and reach user code; and a record from an IPC reader is valid only until that reader's next Next, while C Data memory is freed through the producer's release callback. A column that outlived its record would read freed memory, silently. So every buffer is copied into ursus's own allocator, and nothing imported keeps any Arrow object alive.
Arrow is looser than ursus, in four places, and each would be silent ¶
- A null list row may own a non-empty range of the child. ursus's list kernels assume a null list owns nothing.
- A null struct row may hold valid field values. struct.field hands back the field's own validity.
- A null slot may hold any bytes. Every ursus producer zeroes it.
- A record's columns may be longer than the record.
Import normalises all four: a null row is empty, zeroed, and null in every field beneath it, which is the shape the Parquet reader and kernel.Take already produce. See builder.go.
Index ¶
- func Batch(rec arrow.RecordBatch, out *dtype.Schema, cols []int, lo, hi int) (*data.Batch, error)
- func Column(name string, arr arrow.Array, dt dtype.DataType, lo, hi int) (*data.Column, error)
- func Schema(s *arrow.Schema) (*dtype.Schema, error)
- func Type(at arrow.DataType, meta arrow.Metadata) (dtype.DataType, error)
- type Checker
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func Batch ¶
Batch copies rows [lo, hi) of a record into an ursus batch whose columns are out's, read from the record's columns cols. The record must have passed a Checker whose schema out was selected from.
func Schema ¶
Schema maps an Arrow schema to an ursus one.
Every field is NULLABLE, whatever Arrow declares. arrow.Field.Nullable's zero value is false, Go producers routinely leave it unset while writing nulls, and arrow-go never checks the claim — while ursus's optimizer acts on it, folding `x AND false` to `false` exactly when x is declared non-nullable. Trusting an unchecked false would turn a producer's omission into a wrong answer.
func Type ¶
Type maps an Arrow type to ursus. meta is the metadata of the FIELD holding it, which is where export marks an Int128 (see arrowx.FieldTypeKey).
It is total over arrow.Type: every id either maps or is refused by name, and TestEveryArrowTypeIsMappedOrRefused walks the enum to prove it. An id this version of the switch has never seen — a newer arrow-go — is refused too, never guessed at.
Types ¶
type Checker ¶
type Checker struct {
// contains filtered or unexported fields
}
Checker verifies records against the schema a scan was planned with, before any of their values are read. It is not safe for concurrent use.
Types are checked, not trusted ¶
A builder is chosen from the SCHEMA and then reads an array. If the array's type disagreed — a record whose columns were reordered, an Int32 where an Int64 was planned — a fixed-width builder would read 4n bytes as 8n, or one column's values under another's name. So every record's schema is converted and compared with the planned one, every array is compared with its field, and array.Validate checks that each child agrees with its parent's declared type and that buffers are long enough for the rows they claim.
arrow.Schema.Equal is not the comparison: it compares Nullable and metadata, which ursus deliberately ignores, and never looks at the arrays at all. It is used only to skip re-converting a schema already known to be good.
func NewChecker ¶
NewChecker returns a Checker for records that must convert to want.
Source Files