Help us shape the future of Go! Take the 2026 Go Developer Survey

arrowscan

package
v1.17.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 20, 2026 License: Apache-2.0 Imports: 12 Imported by: 0

Documentation

Overview

Package arrowscan converts Arrow array cells to database/sql driver.Values, with nested types (List/Map/Struct, and VARIANT which arrives nested) rendered to a JSON string byte-identical to the Thrift arrow path (internal/rows/arrowbased). It is pure Go (no cgo), so it is shared by the kernel backend and testable in the default CGO_ENABLED=0 build — the tests here are the regression guard for the exact rendering rules (native float32, exact decimals, time.Time formatting, JSON grammar) both backends must agree on.

Rendering to JSON (not a Go map/slice) is deliberate: it is what the Thrift path returns, so a query's result is identical across backends.

  • list → [v0,v1,...]
  • map → {"k0":v0,"k1":v1,...} (keys stringified)
  • struct → {"field0":v0,...}
  • nested NULL → null
  • time.Time → quoted .String() (matches the Thrift marshal() special-case)
  • nested decimal → exact scale-applied JSON number literal (never a lossy float64), matching Thrift's marshalScalar → ValueString
  • float32 → native float32 (not widened to float64), so JSON renders 3.14, not 3.140000104904175

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func IsBinaryGeospatialField added in v1.17.0

func IsBinaryGeospatialField(field arrow.Field) bool

IsBinaryGeospatialField reports whether field is a logical top-level GEOMETRY / GEOGRAPHY using the kernel's struct representation. The scanner performs the detailed child-shape validation when it reads a non-null value.

func LogicalGeospatialType added in v1.17.0

func LogicalGeospatialType(field arrow.Field) (string, bool)

LogicalGeospatialType returns the top-level Databricks geospatial family recorded by the kernel on an Arrow field. The physical value is UTF-8 in string mode and a struct in binary mode, so DataType alone cannot preserve this database type name.

func ScanCell

func ScanCell(col arrow.Array, row int, loc *time.Location) (driver.Value, error)

ScanCell extracts one cell as a driver.Value. Scalars map to their Go value: bool, all int/uint widths, float (native float32/float64), string, binary, date, timestamp, and top-level decimal (as an exact fixed-point string, matching the Thrift path — a float64 would lose precision beyond ~17 digits). Nested types (List/Map/Struct, and VARIANT which arrives nested) render to a JSON string byte-identical to the Thrift path. Top-level binary GEOMETRY / GEOGRAPHY is selected explicitly through ScanOptions and combines its SRID and WKB children into EWKB []byte; string mode is handled by the regular string arm. INTERVAL day-time/year-month arrive as native arrow duration/month-interval and format to the same string the Thrift path receives pre-formatted from the server. NULLs map to nil. A genuinely unhandled type returns an error rather than a silently wrong value. loc renders DATE / TIMESTAMP in the session time zone (nil = UTC, arrow's ToTime default).

func ScanCellCached

func ScanCellCached(col arrow.Array, row int, loc *time.Location, keys *StructKeyCache) (driver.Value, error)

ScanCellCached is ScanCell with a caller-owned StructKeyCache (see StructKeyCache) so struct field-name keys are escaped once per result set rather than once per row. Pass nil for the un-memoized one-shot behavior.

func ScanCellCachedDecimalFloat

func ScanCellCachedDecimalFloat(col arrow.Array, row int, loc *time.Location, keys *StructKeyCache, decimalAsFloat bool) (driver.Value, error)

ScanCellCachedDecimalFloat is ScanCellCached that, when decimalAsFloat is true, scans a TOP-LEVEL Decimal128 to a lossy float64 instead of the exact string (nested decimals still render exactly). Opt in via WithKernelDecimalAsFloat.

func ScanCellCachedWithOptions added in v1.17.0

func ScanCellCachedWithOptions(col arrow.Array, row int, loc *time.Location, keys *StructKeyCache, options ScanOptions) (driver.Value, error)

ScanCellCachedWithOptions is the full scanner entry point used by the kernel rows implementation. Callers that do not need result-specific adaptations should continue to use ScanCellCached.

Types

type ColumnTypeInfo

type ColumnTypeInfo struct {
	// DatabaseTypeName is the Databricks type name (e.g. "BIGINT", "DECIMAL"),
	// matching the Thrift path; "" for a type with no Databricks name.
	DatabaseTypeName string
	// ScanType is the Go type database/sql recommends scanning the column into,
	// matching the Thrift path.
	ScanType reflect.Type
	// Length / HasLength report a variable-length column's unbounded length
	// (math.MaxInt64), matching Thrift; fixed-width types report (0, false).
	Length    int64
	HasLength bool
}

ColumnTypeInfo is the per-column metadata database/sql surfaces through sql.ColumnType. The kernel derives it from the result's Arrow schema via ColumnTypeInfoFor — the mapping the value scanner and type reporter share, kept byte-identical to the Thrift backend (guarded by the coltype parity tests).

func ColumnTypeInfoFor

func ColumnTypeInfoFor(dt arrow.DataType) ColumnTypeInfo

ColumnTypeInfoFor maps an Arrow column type to the metadata database/sql exposes, matching the Thrift backend for every Databricks type. The Arrow types here are exactly those ScanCellCached scans, so a column's reported type and its scanned value stay in lockstep.

func ColumnTypeInfoForField added in v1.17.0

func ColumnTypeInfoForField(field arrow.Field) ColumnTypeInfo

ColumnTypeInfoForField reports logical GEOMETRY / GEOGRAPHY metadata for the native struct representation. String mode follows the existing driver contract and reports STRING, matching its UTF-8 physical representation. Other fields retain the Arrow-only mapping.

type ScanOptions added in v1.17.0

type ScanOptions struct {
	DecimalAsFloat   bool
	GeospatialBinary bool
}

ScanOptions controls top-level result adaptations that cannot be inferred from an Arrow array alone. Logical GEOMETRY / GEOGRAPHY identity lives on the enclosing schema Field, so kernelRows sets GeospatialBinary only for a field carrying that metadata and a physical struct value.

type StructKeyCache

type StructKeyCache struct {
	// contains filtered or unexported fields
}

StructKeyCache memoizes the JSON-escaped `"name":` prefixes for a struct type, so writeStructJSON doesn't re-marshal constant field names on every row. It is caller-owned and must be scoped to a single result set (e.g. one driver.Rows) and discarded with it — NOT a process-global, which would leak.

The Arrow C Data import allocates a fresh *StructType per batch, so a key is only ever hit within the batch that created it: across a multi-batch result the map would otherwise accrue one never-evicted entry per batch for the whole Rows lifetime. Callers should therefore Reset() the cache at each batch boundary — all rows of a batch share one imported Record, so the intra-batch win (escape each field name once per batch, not once per row) is fully preserved while the map stays bounded to a single batch's struct types. A nil cache is valid: rendering just recomputes the keys inline.

func NewStructKeyCache

func NewStructKeyCache() *StructKeyCache

NewStructKeyCache returns a cache ready to pass to ScanCellCached.

func (*StructKeyCache) Reset

func (c *StructKeyCache) Reset()

Reset drops all memoized prefixes. Callers scope the cache to one batch by calling this when a new batch is imported (see StructKeyCache): the prior batch's *StructType keys can never be hit again, so keeping them only grows the map. Safe on a nil receiver.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL