Documentation
¶
Overview ¶
Package arrowscan converts Arrow array cells to database/sql driver.Values, with nested types (List/Map/Struct, and VARIANT which arrives nested) rendered to a JSON string byte-identical to the Thrift arrow path (internal/rows/arrowbased). It is pure Go (no cgo), so it is shared by the kernel backend and testable in the default CGO_ENABLED=0 build — the tests here are the regression guard for the exact rendering rules (native float32, exact decimals, time.Time formatting, JSON grammar) both backends must agree on.
Rendering to JSON (not a Go map/slice) is deliberate: it is what the Thrift path returns, so a query's result is identical across backends.
- list → [v0,v1,...]
- map → {"k0":v0,"k1":v1,...} (keys stringified)
- struct → {"field0":v0,...}
- nested NULL → null
- time.Time → quoted .String() (matches the Thrift marshal() special-case)
- nested decimal → exact scale-applied JSON number literal (never a lossy float64), matching Thrift's marshalScalar → ValueString
- float32 → native float32 (not widened to float64), so JSON renders 3.14, not 3.140000104904175
Index ¶
- func IsBinaryGeospatialField(field arrow.Field) bool
- func LogicalGeospatialType(field arrow.Field) (string, bool)
- func ScanCell(col arrow.Array, row int, loc *time.Location) (driver.Value, error)
- func ScanCellCached(col arrow.Array, row int, loc *time.Location, keys *StructKeyCache) (driver.Value, error)
- func ScanCellCachedDecimalFloat(col arrow.Array, row int, loc *time.Location, keys *StructKeyCache, ...) (driver.Value, error)
- func ScanCellCachedWithOptions(col arrow.Array, row int, loc *time.Location, keys *StructKeyCache, ...) (driver.Value, error)
- type ColumnTypeInfo
- type ScanOptions
- type StructKeyCache
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func IsBinaryGeospatialField ¶ added in v1.17.0
IsBinaryGeospatialField reports whether field is a logical top-level GEOMETRY / GEOGRAPHY using the kernel's struct representation. The scanner performs the detailed child-shape validation when it reads a non-null value.
func LogicalGeospatialType ¶ added in v1.17.0
LogicalGeospatialType returns the top-level Databricks geospatial family recorded by the kernel on an Arrow field. The physical value is UTF-8 in string mode and a struct in binary mode, so DataType alone cannot preserve this database type name.
func ScanCell ¶
ScanCell extracts one cell as a driver.Value. Scalars map to their Go value: bool, all int/uint widths, float (native float32/float64), string, binary, date, timestamp, and top-level decimal (as an exact fixed-point string, matching the Thrift path — a float64 would lose precision beyond ~17 digits). Nested types (List/Map/Struct, and VARIANT which arrives nested) render to a JSON string byte-identical to the Thrift path. Top-level binary GEOMETRY / GEOGRAPHY is selected explicitly through ScanOptions and combines its SRID and WKB children into EWKB []byte; string mode is handled by the regular string arm. INTERVAL day-time/year-month arrive as native arrow duration/month-interval and format to the same string the Thrift path receives pre-formatted from the server. NULLs map to nil. A genuinely unhandled type returns an error rather than a silently wrong value. loc renders DATE / TIMESTAMP in the session time zone (nil = UTC, arrow's ToTime default).
func ScanCellCached ¶
func ScanCellCached(col arrow.Array, row int, loc *time.Location, keys *StructKeyCache) (driver.Value, error)
ScanCellCached is ScanCell with a caller-owned StructKeyCache (see StructKeyCache) so struct field-name keys are escaped once per result set rather than once per row. Pass nil for the un-memoized one-shot behavior.
func ScanCellCachedDecimalFloat ¶
func ScanCellCachedDecimalFloat(col arrow.Array, row int, loc *time.Location, keys *StructKeyCache, decimalAsFloat bool) (driver.Value, error)
ScanCellCachedDecimalFloat is ScanCellCached that, when decimalAsFloat is true, scans a TOP-LEVEL Decimal128 to a lossy float64 instead of the exact string (nested decimals still render exactly). Opt in via WithKernelDecimalAsFloat.
func ScanCellCachedWithOptions ¶ added in v1.17.0
func ScanCellCachedWithOptions(col arrow.Array, row int, loc *time.Location, keys *StructKeyCache, options ScanOptions) (driver.Value, error)
ScanCellCachedWithOptions is the full scanner entry point used by the kernel rows implementation. Callers that do not need result-specific adaptations should continue to use ScanCellCached.
Types ¶
type ColumnTypeInfo ¶
type ColumnTypeInfo struct {
// DatabaseTypeName is the Databricks type name (e.g. "BIGINT", "DECIMAL"),
// matching the Thrift path; "" for a type with no Databricks name.
DatabaseTypeName string
// ScanType is the Go type database/sql recommends scanning the column into,
// matching the Thrift path.
ScanType reflect.Type
// Length / HasLength report a variable-length column's unbounded length
// (math.MaxInt64), matching Thrift; fixed-width types report (0, false).
Length int64
HasLength bool
}
ColumnTypeInfo is the per-column metadata database/sql surfaces through sql.ColumnType. The kernel derives it from the result's Arrow schema via ColumnTypeInfoFor — the mapping the value scanner and type reporter share, kept byte-identical to the Thrift backend (guarded by the coltype parity tests).
func ColumnTypeInfoFor ¶
func ColumnTypeInfoFor(dt arrow.DataType) ColumnTypeInfo
ColumnTypeInfoFor maps an Arrow column type to the metadata database/sql exposes, matching the Thrift backend for every Databricks type. The Arrow types here are exactly those ScanCellCached scans, so a column's reported type and its scanned value stay in lockstep.
func ColumnTypeInfoForField ¶ added in v1.17.0
func ColumnTypeInfoForField(field arrow.Field) ColumnTypeInfo
ColumnTypeInfoForField reports logical GEOMETRY / GEOGRAPHY metadata for the native struct representation. String mode follows the existing driver contract and reports STRING, matching its UTF-8 physical representation. Other fields retain the Arrow-only mapping.
type ScanOptions ¶ added in v1.17.0
ScanOptions controls top-level result adaptations that cannot be inferred from an Arrow array alone. Logical GEOMETRY / GEOGRAPHY identity lives on the enclosing schema Field, so kernelRows sets GeospatialBinary only for a field carrying that metadata and a physical struct value.
type StructKeyCache ¶
type StructKeyCache struct {
// contains filtered or unexported fields
}
StructKeyCache memoizes the JSON-escaped `"name":` prefixes for a struct type, so writeStructJSON doesn't re-marshal constant field names on every row. It is caller-owned and must be scoped to a single result set (e.g. one driver.Rows) and discarded with it — NOT a process-global, which would leak.
The Arrow C Data import allocates a fresh *StructType per batch, so a key is only ever hit within the batch that created it: across a multi-batch result the map would otherwise accrue one never-evicted entry per batch for the whole Rows lifetime. Callers should therefore Reset() the cache at each batch boundary — all rows of a batch share one imported Record, so the intra-batch win (escape each field name once per batch, not once per row) is fully preserved while the map stays bounded to a single batch's struct types. A nil cache is valid: rendering just recomputes the keys inline.
func NewStructKeyCache ¶
func NewStructKeyCache() *StructKeyCache
NewStructKeyCache returns a cache ready to pass to ScanCellCached.
func (*StructKeyCache) Reset ¶
func (c *StructKeyCache) Reset()
Reset drops all memoized prefixes. Callers scope the cache to one batch by calling this when a new batch is imported (see StructKeyCache): the prior batch's *StructType keys can never be hit again, so keeping them only grows the map. Safe on a nil receiver.