Documentation
¶
Index ¶
Constants ¶
const KernelMaxChunksInMemoryConfKey = "cloudfetch_max_chunks_in_memory"
KernelMaxChunksInMemoryConfKey is the CLIENT-only session conf the kernel reads to bound how many decompressed CloudFetch chunks it holds in memory (WithKernelMaxChunksInMemory). Unlike MetricViewMetadataConfKey this is NOT a server SET parameter and is NOT added by EffectiveSessionParams: the kernel backend injects it into its own SessionConf only, and the kernel strips it before the SEA wire. It must match the key the kernel's apply_client_result_overrides looks for.
const MetricViewMetadataConfKey = "spark.sql.thriftserver.metadata.metricview.enabled"
MetricViewMetadataConfKey is the server session conf that enables metric-view metadata (WithEnableMetricViewMetadata). Despite the "thriftserver" in its name — a historical server-side name — it is an ordinary Spark SQL session conf the server honors regardless of client transport, so both the Thrift and kernel backends send the identical key/value. Defined once here so no backend hardcodes the literal (see EffectiveSessionParams).
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type ArrowConfig ¶ added in v1.1.0
type ArrowConfig struct {
UseArrowBatches bool
UseArrowNativeDecimal bool
UseArrowNativeTimestamp bool
// the following are currently not supported
UseArrowNativeComplexTypes bool
UseArrowNativeIntervalTypes bool
}
func (ArrowConfig) DeepCopy ¶ added in v1.1.0
func (arrowConfig ArrowConfig) DeepCopy() ArrowConfig
DeepCopy returns a true deep copy of UserConfig
func (ArrowConfig) WithDefaults ¶ added in v1.1.0
func (ucfg ArrowConfig) WithDefaults() ArrowConfig
type CloudFetchConfig ¶ added in v1.4.0
type CloudFetchConfig struct {
UseCloudFetch bool
MaxDownloadThreads int
MaxFilesInMemory int
MinTimeToExpiry time.Duration
CloudFetchSpeedThresholdMbps float64 // Minimum download speed in MBps before WARN logging (default: 0.1)
HTTPClient *http.Client
}
func (CloudFetchConfig) DeepCopy ¶ added in v1.4.0
func (cfg CloudFetchConfig) DeepCopy() CloudFetchConfig
func (CloudFetchConfig) WithDefaults ¶ added in v1.4.0
func (cfg CloudFetchConfig) WithDefaults() CloudFetchConfig
type Config ¶
type Config struct {
UserConfig
TLSConfig *tls.Config // nil disables TLS
ArrowConfig
PollInterval time.Duration
ClientTimeout time.Duration // max time the http request can last
PingTimeout time.Duration // max time allowed for ping
CanUseMultipleCatalogs bool
DriverName string
DriverVersion string
ThriftProtocol string
ThriftTransport string
ThriftProtocolVersion cli_service.TProtocolVersion
ThriftDebugClientProtocol bool
// ClientQueryTimeout is the connector-only, kernel execution deadline set by
// WithClientQueryTimeout. nil means the option was omitted, which preserves
// the legacy kernel_statement_execute contract. A non-nil zero explicitly
// selects unlimited execution through the timeout-aware C API.
ClientQueryTimeout *time.Duration
// KernelExperimental carries experimental, kernel-backend-only options that
// have no equivalent on the default (Thrift) path — currently the richer TLS
// surface (a trusted-CA bundle and an independent hostname-skip) the kernel
// exposes over its C ABI. It lives here on Config, NOT on UserConfig, so it
// stays off the stable exported/DSN surface (the same treatment TLSConfig and
// ArrowConfig get). nil means no experimental option was set. The Thrift
// backend rejects a non-nil value loudly; the kernel backend forwards it to
// the kernel C ABI. Mirrors Node's non-exported InternalConnectionOptions /
// Python's underscore-prefixed kwargs.
KernelExperimental *KernelExperimentalConfig
}
Driver Configurations. Only UserConfig are currently exposed to users
func (*Config) EffectiveSessionParams ¶ added in v1.15.0
EffectiveSessionParams returns the session confs to send to the server: the user-supplied SessionParams plus any conf derived from a higher-level option (currently metric-view metadata). Both backends call this, so the derivation is backend-neutral — neither special-cases the option nor hardcodes the conf literal. The returned map is always a fresh copy the caller may mutate freely.
func (*Config) ToEndpointURL ¶
ToEndpointURL generates the endpoint URL from Config that a Thrift client will connect to
type ConfigValue ¶ added in v1.10.0
type ConfigValue[T any] struct { // contains filtered or unexported fields }
ConfigValue represents a configuration value that can be set by client or resolved from server. This implements the config overlay pattern: client > server > default
T is the type of the configuration value (bool, string, int, etc.)
Example usage:
type MyConfig struct {
EnableFeature ConfigValue[bool]
BatchSize ConfigValue[int]
}
// Client explicitly sets value (overrides server)
config.EnableFeature = NewConfigValue(true)
// Client doesn't set value (use server)
config.EnableFeature = ConfigValue[bool]{} // nil/unset
// Resolve value with overlay priority
enabled := config.EnableFeature.Resolve(ctx, serverResolver, defaultValue)
func NewConfigValue ¶ added in v1.10.0
func NewConfigValue[T any](value T) ConfigValue[T]
NewConfigValue creates a ConfigValue with a client-set value. The value will override any server-side configuration.
func ParseBoolConfigValue ¶ added in v1.10.0
func ParseBoolConfigValue(params map[string]string, key string) ConfigValue[bool]
ParseBoolConfigValue parses a string value into a ConfigValue[bool]. Returns unset ConfigValue if the parameter is not present.
Example:
params := map[string]string{"enableFeature": "true"}
value := ParseBoolConfigValue(params, "enableFeature")
// value.IsSet() == true, value.Get() == (true, true)
func ParseIntConfigValue ¶ added in v1.10.0
func ParseIntConfigValue(params map[string]string, key string) ConfigValue[int]
ParseIntConfigValue parses a string value into a ConfigValue[int]. Returns unset ConfigValue if the parameter is not present or invalid.
func ParseStringConfigValue ¶ added in v1.10.0
func ParseStringConfigValue(params map[string]string, key string) ConfigValue[string]
ParseStringConfigValue parses a string value into a ConfigValue[string]. Returns unset ConfigValue if the parameter is not present.
func (ConfigValue[T]) Get ¶ added in v1.10.0
func (cv ConfigValue[T]) Get() (T, bool)
Get returns the client-set value and whether it was set. If not set, returns zero value and false.
func (ConfigValue[T]) IsSet ¶ added in v1.10.0
func (cv ConfigValue[T]) IsSet() bool
IsSet returns true if the client explicitly set this configuration value.
func (ConfigValue[T]) Resolve ¶ added in v1.10.0
func (cv ConfigValue[T]) Resolve( ctx context.Context, serverResolver ServerResolver[T], defaultValue T, ) T
Resolve applies config overlay priority to determine the final value:
Priority 1: Client Config - if explicitly set (overrides server) Priority 2: Server Config - resolved via serverResolver (when client doesn't set) Priority 3: Default Value - used when server unavailable/errors (fail-safe)
Parameters:
- ctx: Context for server requests
- serverResolver: How to fetch from server (can be nil if no server config)
- defaultValue: Fail-safe default when client unset and server unavailable
Returns: The resolved configuration value following overlay priority
func (ConfigValue[T]) ResolveWithContext ¶ added in v1.10.0
func (cv ConfigValue[T]) ResolveWithContext( ctx context.Context, host string, httpClient *http.Client, serverResolver ServerResolver[T], defaultValue T, ) T
ResolveWithContext is a more flexible version that takes host and httpClient. This is the recommended method for production use.
type KernelExperimentalConfig ¶ added in v1.15.0
type KernelExperimentalConfig struct {
// TLSTrustedCertsPEM is a PEM CA bundle added to the kernel's trust store on
// top of the system roots (maps to kernel_session_config_set_tls_trusted_certs).
// Needed because the kernel's rustls stack ignores SSL_CERT_FILE, so a custom
// CA must be handed to the kernel explicitly.
TLSTrustedCertsPEM []byte
// TLSClientCertPEM / TLSClientKeyPEM are the paired mTLS client identity
// forwarded through kernel_session_config_set_tls_client_certificate.
// TLSClientCertConfigured distinguishes an explicit call with empty values
// from the option never being set, so invalid input cannot fail open.
TLSClientCertPEM []byte
TLSClientKeyPEM []byte
TLSClientCertConfigured bool
// TLSSkipHostnameVerify skips only the certificate hostname check, independent
// of the blanket InsecureSkipVerify (which relaxes both chain and hostname).
// Maps to kernel_session_config_set_tls_skip_hostname_verification.
TLSSkipHostnameVerify bool
// ProxyURL / ProxyUsername / ProxyPassword / ProxyBypassHosts configure an
// explicit HTTP proxy on the kernel path (WithKernelProxy), overriding the
// HTTP(S)_PROXY / NO_PROXY environment the driver otherwise mirrors. The
// credentials and bypass list are the "advanced" fields the env-var path
// can't express (a structured no-proxy list, out-of-band basic auth). Empty
// ProxyURL leaves the environment-derived proxy in effect. Maps to
// kernel_session_config_set_proxy(url, username, password, bypass_hosts).
ProxyURL string
ProxyUsername string
ProxyPassword string
ProxyBypassHosts string
// RetryOverallTimeout is the cumulative retry budget across all attempts on
// the kernel path (WithKernelRetryOverallTimeout). Zero = keep the kernel
// default (900s). WithRetries only carries the per-attempt backoff bounds +
// max attempts (mirroring the Thrift RetryWaitMin/Max/RetryMax surface); the
// overall budget is a kernel-only knob the Thrift path has no equivalent for,
// so it lives here and maps to the 4th arg of
// kernel_session_config_set_retry_config (matching the pyo3/napi
// retry_overall_timeout knob).
RetryOverallTimeout time.Duration
// MaxConnections is the maximum number of idle HTTP connections retained per
// host by the kernel. Nil keeps the kernel default (100).
MaxConnections *int
// MaxChunksInMemory bounds how many decompressed CloudFetch chunks the kernel
// holds in memory at once (WithKernelMaxChunksInMemory) — the knob that trades
// throughput for peak RSS on large result sets. Zero = keep the kernel default
// (16). A positive value is forwarded as the client-only
// "cloudfetch_max_chunks_in_memory" session conf, which the kernel folds into
// its result config and strips before the SEA wire; it is kernel-only because
// the Thrift path has no such in-memory-chunk knob.
MaxChunksInMemory int
// DecimalAsFloat scans top-level DECIMAL columns to a lossy float64 instead of
// the exact string (WithKernelDecimalAsFloat). Kernel-only; off by default.
DecimalAsFloat bool
// TokenCacheEnabled controls the kernel's on-disk OAuth U2M token-cache persistence
// (WithTokenCache). When false (the default), tokens are held in memory only;
// when true, the refresh token is persisted encrypted to ~/.config/databricks-sql-kernel/oauth/.
// U2M-only: PAT and M2M ignore this setting. Maps to
// kernel_session_config_set_u2m_token_cache_config; a nil KernelExperimental or
// false TokenCacheEnabled calls the setter with enabled=false so the kernel does
// NOT persist to disk by default.
TokenCacheEnabled bool
// EnableGeoSpatialSupport selects the kernel's client-side GEOMETRY /
// GEOGRAPHY result representation. True requests the canonical
// struct<srid,wkb> Arrow value; false requests WKT / EWKT strings. A pointer
// distinguishes the omitted value (default false) from an explicit setting.
EnableGeoSpatialSupport *bool
}
KernelExperimentalConfig holds the experimental, kernel-only connection knobs set via the WithKernel* options. Every field is either forwarded to the kernel C ABI (kernel backend) or rejected (Thrift backend) — never silently dropped; the exhaustiveness guard TestKernelExperimentalFieldsClassified asserts this so a newly-added field can't slip through unclassified.
func (*KernelExperimentalConfig) DeepCopy ¶ added in v1.15.0
func (k *KernelExperimentalConfig) DeepCopy() *KernelExperimentalConfig
DeepCopy returns a deep copy of the experimental config, or nil for a nil receiver. The byte slice is copied so a mutation of the copy can't reach back into the original (the connector may DeepCopy the whole Config per conn).
type ServerResolver ¶ added in v1.10.0
type ServerResolver[T any] interface { // Resolve fetches the configuration value from the server. // Returns the value and any error encountered. // On error, the config overlay will fall back to the default value. Resolve(ctx context.Context, host string, httpClient *http.Client) (T, error) }
ServerResolver defines how to fetch a configuration value from the server. Implementations should handle caching, retries, and error handling.
type UserConfig ¶
type UserConfig struct {
Protocol string
Host string // from databricks UI
Port int // from databricks UI
HTTPPath string // from databricks UI
Catalog string
Schema string
Authenticator auth.Authenticator
AccessToken string // from databricks UI
MaxRows int // max rows per page
QueryTimeout time.Duration // Timeout passed to server for query processing
UserAgentEntry string
Location *time.Location
SessionParams map[string]string
RetryWaitMin time.Duration
RetryWaitMax time.Duration
RetryMax int
// Telemetry configuration
// Uses config overlay pattern: client > server > default.
// Unset = check server feature flag; explicitly true/false overrides the server.
EnableTelemetry ConfigValue[bool]
TelemetryBatchSize int // 0 = use default (200)
TelemetryFlushInterval time.Duration // 0 = use default (30s)
Transport http.RoundTripper
UseLz4Compression bool
EnableMetricViewMetadata bool
// UseArrowNativeDecimalDSN is a DSN-only carrier for the useArrowNativeDecimal
// parameter. The authoritative setting lives on ArrowConfig; ParseDSN records
// the DSN value here (because it can only return a UserConfig) and the
// connector copies it into Config.ArrowConfig.UseArrowNativeDecimal when it is
// assembled. The name intentionally differs from ArrowConfig's field so the
// promoted selector Config.UseArrowNativeDecimal stays unambiguous.
UseArrowNativeDecimalDSN bool
CloudFetchConfig
// UseKernel selects the SEA-via-kernel backend instead of Thrift. See the
// WithUseKernel connector option for the build requirements. DSN: useKernel=true.
UseKernel bool
// WarehouseID is the bare SQL warehouse id, used by the kernel backend (which
// addresses a warehouse by id) in preference to HTTPPath. The Thrift backend
// ignores it and routes by HTTPPath. DSN: warehouseId=<id>.
WarehouseID string
// TokenCacheEnabledDSN is a DSN-only carrier for the tokenCache parameter (U2M-only).
// The authoritative setting lives on KernelExperimentalConfig; ParseDSN records the
// DSN value here (because it can only return a UserConfig) and the connector copies
// it into Config.KernelExperimental.TokenCacheEnabled when it is assembled. The name
// intentionally differs from KernelExperimentalConfig's field so the setting stays
// unambiguous. False by default; when true, enables on-disk token-cache persistence
// for U2M OAuth. DSN: tokenCache=true.
TokenCacheEnabledDSN bool
// EnableGeoSpatialSupportDSN is a DSN-only carrier for
// enableGeoSpatialSupport=true|false. Nil means omitted (the default is
// false); the connector copies a present value into KernelExperimental.
EnableGeoSpatialSupportDSN *bool
}
UserConfig is the set of configurations exposed to users
func ParseDSN ¶
func ParseDSN(dsn string) (UserConfig, error)
ParseDSN constructs UserConfig and CloudFetchConfig by parsing DSN string supplied to `sql.Open()`
func (UserConfig) DeepCopy ¶
func (ucfg UserConfig) DeepCopy() UserConfig
DeepCopy returns a true deep copy of UserConfig
func (UserConfig) WithDefaults ¶
func (ucfg UserConfig) WithDefaults() UserConfig
WithDefaults provides default settings for optional fields in UserConfig