Documentation
¶
Overview ¶
Package arrowscan converts Arrow array cells to database/sql driver.Values, with nested types (List/Map/Struct, and VARIANT which arrives nested) rendered to a JSON string byte-identical to the Thrift arrow path (internal/rows/arrowbased). It is pure Go (no cgo), so it is shared by the kernel backend and testable in the default CGO_ENABLED=0 build — the tests here are the regression guard for the exact rendering rules (native float32, exact decimals, time.Time formatting, JSON grammar) both backends must agree on.
Rendering to JSON (not a Go map/slice) is deliberate: it is what the Thrift path returns, so a query's result is identical across backends.
- list → [v0,v1,...]
- map → {"k0":v0,"k1":v1,...} (keys stringified)
- struct → {"field0":v0,...}
- nested NULL → null
- time.Time → quoted .String() (matches the Thrift marshal() special-case)
- nested decimal → exact scale-applied JSON number literal (never a lossy float64), matching Thrift's marshalScalar → ValueString
- float32 → native float32 (not widened to float64), so JSON renders 3.14, not 3.140000104904175
Index ¶
- func ScanCell(col arrow.Array, row int, loc *time.Location) (driver.Value, error)
- func ScanCellCached(col arrow.Array, row int, loc *time.Location, keys *StructKeyCache) (driver.Value, error)
- func ScanCellCachedDecimalFloat(col arrow.Array, row int, loc *time.Location, keys *StructKeyCache, ...) (driver.Value, error)
- type ColumnTypeInfo
- type StructKeyCache
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func ScanCell ¶
ScanCell extracts one cell as a driver.Value. Scalars map to their Go value: bool, all int/uint widths, float (native float32/float64), string, binary, date, timestamp, and top-level decimal (as an exact fixed-point string, matching the Thrift path — a float64 would lose precision beyond ~17 digits). Nested types (List/Map/Struct, and VARIANT which arrives nested) render to a JSON string byte-identical to the Thrift path; GEOMETRY arrives as a WKB/WKT string and is handled by the string arm. INTERVAL day-time/year-month arrive as native arrow duration/month-interval and format to the same string the Thrift path receives pre-formatted from the server. NULLs map to nil. A genuinely unhandled type returns an error rather than a silently wrong value. loc renders DATE / TIMESTAMP in the session time zone (nil = UTC, arrow's ToTime default).
func ScanCellCached ¶
func ScanCellCached(col arrow.Array, row int, loc *time.Location, keys *StructKeyCache) (driver.Value, error)
ScanCellCached is ScanCell with a caller-owned StructKeyCache (see StructKeyCache) so struct field-name keys are escaped once per result set rather than once per row. Pass nil for the un-memoized one-shot behavior.
func ScanCellCachedDecimalFloat ¶
func ScanCellCachedDecimalFloat(col arrow.Array, row int, loc *time.Location, keys *StructKeyCache, decimalAsFloat bool) (driver.Value, error)
ScanCellCachedDecimalFloat is ScanCellCached that, when decimalAsFloat is true, scans a TOP-LEVEL Decimal128 to a lossy float64 instead of the exact string (nested decimals still render exactly). Opt in via WithKernelDecimalAsFloat.
Types ¶
type ColumnTypeInfo ¶
type ColumnTypeInfo struct {
// DatabaseTypeName is the Databricks type name (e.g. "BIGINT", "DECIMAL"),
// matching the Thrift path; "" for a type with no Databricks name.
DatabaseTypeName string
// ScanType is the Go type database/sql recommends scanning the column into,
// matching the Thrift path.
ScanType reflect.Type
// Length / HasLength report a variable-length column's unbounded length
// (math.MaxInt64), matching Thrift; fixed-width types report (0, false).
Length int64
HasLength bool
}
ColumnTypeInfo is the per-column metadata database/sql surfaces through sql.ColumnType. The kernel derives it from the result's Arrow schema via ColumnTypeInfoFor — the mapping the value scanner and type reporter share, kept byte-identical to the Thrift backend (guarded by the coltype parity tests).
func ColumnTypeInfoFor ¶
func ColumnTypeInfoFor(dt arrow.DataType) ColumnTypeInfo
ColumnTypeInfoFor maps an Arrow column type to the metadata database/sql exposes, matching the Thrift backend for every Databricks type. The Arrow types here are exactly those ScanCellCached scans, so a column's reported type and its scanned value stay in lockstep.
type StructKeyCache ¶
type StructKeyCache struct {
// contains filtered or unexported fields
}
StructKeyCache memoizes the JSON-escaped `"name":` prefixes for a struct type, so writeStructJSON doesn't re-marshal constant field names on every row. It is caller-owned and must be scoped to a single result set (e.g. one driver.Rows) and discarded with it — NOT a process-global, which would leak.
The Arrow C Data import allocates a fresh *StructType per batch, so a key is only ever hit within the batch that created it: across a multi-batch result the map would otherwise accrue one never-evicted entry per batch for the whole Rows lifetime. Callers should therefore Reset() the cache at each batch boundary — all rows of a batch share one imported Record, so the intra-batch win (escape each field name once per batch, not once per row) is fully preserved while the map stays bounded to a single batch's struct types. A nil cache is valid: rendering just recomputes the keys inline.
func NewStructKeyCache ¶
func NewStructKeyCache() *StructKeyCache
NewStructKeyCache returns a cache ready to pass to ScanCellCached.
func (*StructKeyCache) Reset ¶
func (c *StructKeyCache) Reset()
Reset drops all memoized prefixes. Callers scope the cache to one batch by calling this when a new batch is imported (see StructKeyCache): the prior batch's *StructType keys can never be hit again, so keeping them only grows the map. Safe on a nil receiver.
Source Files