Documentation
¶
Index ¶
Constants ¶
const KernelMaxChunksInMemoryConfKey = "cloudfetch_max_chunks_in_memory"
KernelMaxChunksInMemoryConfKey is the CLIENT-only session conf the kernel reads to bound how many decompressed CloudFetch chunks it holds in memory (WithKernelMaxChunksInMemory). Unlike MetricViewMetadataConfKey this is NOT a server SET parameter and is NOT added by EffectiveSessionParams: the kernel backend injects it into its own SessionConf only, and the kernel strips it before the SEA wire (it is absent from the kernel's server allowlist). It must match the key the kernel's apply_client_result_overrides looks for.
const MetricViewMetadataConfKey = "spark.sql.thriftserver.metadata.metricview.enabled"
MetricViewMetadataConfKey is the server session conf that enables metric-view metadata (WithEnableMetricViewMetadata). Despite the "thriftserver" in its name — a historical server-side name — it is an ordinary Spark SQL session conf the server honors regardless of client transport, so both the Thrift and kernel backends send the identical key/value. Defined once here so no backend hardcodes the literal (see EffectiveSessionParams).
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type ArrowConfig ¶ added in v1.1.0
type ArrowConfig struct {
UseArrowBatches bool
UseArrowNativeDecimal bool
UseArrowNativeTimestamp bool
// the following are currently not supported
UseArrowNativeComplexTypes bool
UseArrowNativeIntervalTypes bool
}
func (ArrowConfig) DeepCopy ¶ added in v1.1.0
func (arrowConfig ArrowConfig) DeepCopy() ArrowConfig
DeepCopy returns a true deep copy of UserConfig
func (ArrowConfig) WithDefaults ¶ added in v1.1.0
func (ucfg ArrowConfig) WithDefaults() ArrowConfig
type CloudFetchConfig ¶ added in v1.4.0
type CloudFetchConfig struct {
UseCloudFetch bool
MaxDownloadThreads int
MaxFilesInMemory int
MinTimeToExpiry time.Duration
CloudFetchSpeedThresholdMbps float64 // Minimum download speed in MBps before WARN logging (default: 0.1)
HTTPClient *http.Client
}
func (CloudFetchConfig) DeepCopy ¶ added in v1.4.0
func (cfg CloudFetchConfig) DeepCopy() CloudFetchConfig
func (CloudFetchConfig) WithDefaults ¶ added in v1.4.0
func (cfg CloudFetchConfig) WithDefaults() CloudFetchConfig
type Config ¶
type Config struct {
UserConfig
TLSConfig *tls.Config // nil disables TLS
ArrowConfig
PollInterval time.Duration
ClientTimeout time.Duration // max time the http request can last
PingTimeout time.Duration // max time allowed for ping
CanUseMultipleCatalogs bool
DriverName string
DriverVersion string
ThriftProtocol string
ThriftTransport string
ThriftProtocolVersion cli_service.TProtocolVersion
ThriftDebugClientProtocol bool
// KernelExperimental carries experimental, kernel-backend-only options that
// have no equivalent on the default (Thrift) path — currently the richer TLS
// surface (a trusted-CA bundle and an independent hostname-skip) the kernel
// exposes over its C ABI. It lives here on Config, NOT on UserConfig, so it
// stays off the stable exported/DSN surface (the same treatment TLSConfig and
// ArrowConfig get). nil means no experimental option was set. The Thrift
// backend rejects a non-nil value loudly; the kernel backend forwards it to
// the kernel C ABI. Mirrors Node's non-exported InternalConnectionOptions /
// Python's underscore-prefixed kwargs.
KernelExperimental *KernelExperimentalConfig
}
Driver Configurations. Only UserConfig are currently exposed to users
func (*Config) EffectiveSessionParams ¶ added in v1.15.0
EffectiveSessionParams returns the session confs to send to the server: the user-supplied SessionParams plus any conf derived from a higher-level option (currently metric-view metadata). Both backends call this, so the derivation is backend-neutral — neither special-cases the option nor hardcodes the conf literal. The returned map is always a fresh copy the caller may mutate freely.
func (*Config) ToEndpointURL ¶
ToEndpointURL generates the endpoint URL from Config that a Thrift client will connect to
type ConfigValue ¶ added in v1.10.0
type ConfigValue[T any] struct { // contains filtered or unexported fields }
ConfigValue represents a configuration value that can be set by client or resolved from server. This implements the config overlay pattern: client > server > default
T is the type of the configuration value (bool, string, int, etc.)
Example usage:
type MyConfig struct {
EnableFeature ConfigValue[bool]
BatchSize ConfigValue[int]
}
// Client explicitly sets value (overrides server)
config.EnableFeature = NewConfigValue(true)
// Client doesn't set value (use server)
config.EnableFeature = ConfigValue[bool]{} // nil/unset
// Resolve value with overlay priority
enabled := config.EnableFeature.Resolve(ctx, serverResolver, defaultValue)
func NewConfigValue ¶ added in v1.10.0
func NewConfigValue[T any](value T) ConfigValue[T]
NewConfigValue creates a ConfigValue with a client-set value. The value will override any server-side configuration.
func ParseBoolConfigValue ¶ added in v1.10.0
func ParseBoolConfigValue(params map[string]string, key string) ConfigValue[bool]
ParseBoolConfigValue parses a string value into a ConfigValue[bool]. Returns unset ConfigValue if the parameter is not present.
Example:
params := map[string]string{"enableFeature": "true"}
value := ParseBoolConfigValue(params, "enableFeature")
// value.IsSet() == true, value.Get() == (true, true)
func ParseIntConfigValue ¶ added in v1.10.0
func ParseIntConfigValue(params map[string]string, key string) ConfigValue[int]
ParseIntConfigValue parses a string value into a ConfigValue[int]. Returns unset ConfigValue if the parameter is not present or invalid.
func ParseStringConfigValue ¶ added in v1.10.0
func ParseStringConfigValue(params map[string]string, key string) ConfigValue[string]
ParseStringConfigValue parses a string value into a ConfigValue[string]. Returns unset ConfigValue if the parameter is not present.
func (ConfigValue[T]) Get ¶ added in v1.10.0
func (cv ConfigValue[T]) Get() (T, bool)
Get returns the client-set value and whether it was set. If not set, returns zero value and false.
func (ConfigValue[T]) IsSet ¶ added in v1.10.0
func (cv ConfigValue[T]) IsSet() bool
IsSet returns true if the client explicitly set this configuration value.
func (ConfigValue[T]) Resolve ¶ added in v1.10.0
func (cv ConfigValue[T]) Resolve( ctx context.Context, serverResolver ServerResolver[T], defaultValue T, ) T
Resolve applies config overlay priority to determine the final value:
Priority 1: Client Config - if explicitly set (overrides server) Priority 2: Server Config - resolved via serverResolver (when client doesn't set) Priority 3: Default Value - used when server unavailable/errors (fail-safe)
Parameters:
- ctx: Context for server requests
- serverResolver: How to fetch from server (can be nil if no server config)
- defaultValue: Fail-safe default when client unset and server unavailable
Returns: The resolved configuration value following overlay priority
func (ConfigValue[T]) ResolveWithContext ¶ added in v1.10.0
func (cv ConfigValue[T]) ResolveWithContext( ctx context.Context, host string, httpClient *http.Client, serverResolver ServerResolver[T], defaultValue T, ) T
ResolveWithContext is a more flexible version that takes host and httpClient. This is the recommended method for production use.
type KernelExperimentalConfig ¶ added in v1.15.0
type KernelExperimentalConfig struct {
// TLSTrustedCertsPEM is a PEM CA bundle added to the kernel's trust store on
// top of the system roots (maps to kernel_session_config_set_tls_trusted_certs).
// Needed because the kernel's rustls stack ignores SSL_CERT_FILE, so a custom
// CA must be handed to the kernel explicitly.
TLSTrustedCertsPEM []byte
// TLSClientCertPEM / TLSClientKeyPEM are the paired mTLS client identity
// forwarded through kernel_session_config_set_tls_client_certificate.
// TLSClientCertConfigured distinguishes an explicit call with empty values
// from the option never being set, so invalid input cannot fail open.
TLSClientCertPEM []byte
TLSClientKeyPEM []byte
TLSClientCertConfigured bool
// TLSSkipHostnameVerify skips only the certificate hostname check, independent
// of the blanket InsecureSkipVerify (which relaxes both chain and hostname).
// Maps to kernel_session_config_set_tls_skip_hostname_verification.
TLSSkipHostnameVerify bool
// ProxyURL / ProxyUsername / ProxyPassword / ProxyBypassHosts configure an
// explicit HTTP proxy on the kernel path (WithKernelProxy), overriding the
// HTTP(S)_PROXY / NO_PROXY environment the driver otherwise mirrors. The
// credentials and bypass list are the "advanced" fields the env-var path
// can't express (a structured no-proxy list, out-of-band basic auth). Empty
// ProxyURL leaves the environment-derived proxy in effect. Maps to
// kernel_session_config_set_proxy(url, username, password, bypass_hosts).
ProxyURL string
ProxyUsername string
ProxyPassword string
ProxyBypassHosts string
// RetryOverallTimeout is the cumulative retry budget across all attempts on
// the kernel path (WithKernelRetryOverallTimeout). Zero = keep the kernel
// default (900s). WithRetries only carries the per-attempt backoff bounds +
// max attempts (mirroring the Thrift RetryWaitMin/Max/RetryMax surface); the
// overall budget is a kernel-only knob the Thrift path has no equivalent for,
// so it lives here and maps to the 4th arg of
// kernel_session_config_set_retry_config (matching the pyo3/napi
// retry_overall_timeout knob).
RetryOverallTimeout time.Duration
// MaxChunksInMemory bounds how many decompressed CloudFetch chunks the kernel
// holds in memory at once (WithKernelMaxChunksInMemory) — the knob that trades
// throughput for peak RSS on large result sets. Zero = keep the kernel default
// (16). A positive value is forwarded as the client-only
// "cloudfetch_max_chunks_in_memory" session conf, which the kernel folds into
// its result config and strips before the SEA wire; it is kernel-only because
// the Thrift path has no such in-memory-chunk knob.
MaxChunksInMemory int
// DecimalAsFloat scans top-level DECIMAL columns to a lossy float64 instead of
// the exact string (WithKernelDecimalAsFloat). Kernel-only; off by default.
DecimalAsFloat bool
// TokenCacheEnabled controls the kernel's on-disk OAuth U2M token-cache persistence
// (WithTokenCache). When false (the default), tokens are held in memory only;
// when true, the refresh token is persisted encrypted to ~/.config/databricks-sql-kernel/oauth/.
// U2M-only: PAT and M2M ignore this setting. Maps to
// kernel_session_config_set_u2m_token_cache_config; a nil KernelExperimental or
// false TokenCacheEnabled calls the setter with enabled=false so the kernel does
// NOT persist to disk by default.
TokenCacheEnabled bool
}
KernelExperimentalConfig holds the experimental, kernel-only connection knobs set via the WithKernel* options. Every field is either forwarded to the kernel C ABI (kernel backend) or rejected (Thrift backend) — never silently dropped; the exhaustiveness guard TestKernelExperimentalFieldsClassified asserts this so a newly-added field can't slip through unclassified.
func (*KernelExperimentalConfig) DeepCopy ¶ added in v1.15.0
func (k *KernelExperimentalConfig) DeepCopy() *KernelExperimentalConfig
DeepCopy returns a deep copy of the experimental config, or nil for a nil receiver. The byte slice is copied so a mutation of the copy can't reach back into the original (the connector may DeepCopy the whole Config per conn).
type ServerResolver ¶ added in v1.10.0
type ServerResolver[T any] interface { // Resolve fetches the configuration value from the server. // Returns the value and any error encountered. // On error, the config overlay will fall back to the default value. Resolve(ctx context.Context, host string, httpClient *http.Client) (T, error) }
ServerResolver defines how to fetch a configuration value from the server. Implementations should handle caching, retries, and error handling.
type UserConfig ¶
type UserConfig struct {
Protocol string
Host string // from databricks UI
Port int // from databricks UI
HTTPPath string // from databricks UI
Catalog string
Schema string
Authenticator auth.Authenticator
AccessToken string // from databricks UI
MaxRows int // max rows per page
QueryTimeout time.Duration // Timeout passed to server for query processing
UserAgentEntry string
Location *time.Location
SessionParams map[string]string
RetryWaitMin time.Duration
RetryWaitMax time.Duration
RetryMax int
// Telemetry configuration
// Uses config overlay pattern: client > server > default.
// Unset = check server feature flag; explicitly true/false overrides the server.
EnableTelemetry ConfigValue[bool]
TelemetryBatchSize int // 0 = use default (200)
TelemetryFlushInterval time.Duration // 0 = use default (30s)
Transport http.RoundTripper
UseLz4Compression bool
EnableMetricViewMetadata bool
// UseArrowNativeDecimalDSN is a DSN-only carrier for the useArrowNativeDecimal
// parameter. The authoritative setting lives on ArrowConfig; ParseDSN records
// the DSN value here (because it can only return a UserConfig) and the
// connector copies it into Config.ArrowConfig.UseArrowNativeDecimal when it is
// assembled. The name intentionally differs from ArrowConfig's field so the
// promoted selector Config.UseArrowNativeDecimal stays unambiguous.
UseArrowNativeDecimalDSN bool
CloudFetchConfig
// UseKernel selects the SEA-via-kernel backend instead of Thrift. See the
// WithUseKernel connector option for the build requirements. DSN: useKernel=true.
UseKernel bool
// WarehouseID is the bare SQL warehouse id, used by the kernel backend (which
// addresses a warehouse by id) in preference to HTTPPath. The Thrift backend
// ignores it and routes by HTTPPath. DSN: warehouseId=<id>.
WarehouseID string
// TokenCacheEnabledDSN is a DSN-only carrier for the tokenCache parameter (U2M-only).
// The authoritative setting lives on KernelExperimentalConfig; ParseDSN records the
// DSN value here (because it can only return a UserConfig) and the connector copies
// it into Config.KernelExperimental.TokenCacheEnabled when it is assembled. The name
// intentionally differs from KernelExperimentalConfig's field so the setting stays
// unambiguous. False by default; when true, enables on-disk token-cache persistence
// for U2M OAuth. DSN: tokenCache=true.
TokenCacheEnabledDSN bool
}
UserConfig is the set of configurations exposed to users
func ParseDSN ¶
func ParseDSN(dsn string) (UserConfig, error)
ParseDSN constructs UserConfig and CloudFetchConfig by parsing DSN string supplied to `sql.Open()`
func (UserConfig) DeepCopy ¶
func (ucfg UserConfig) DeepCopy() UserConfig
DeepCopy returns a true deep copy of UserConfig
func (UserConfig) WithDefaults ¶
func (ucfg UserConfig) WithDefaults() UserConfig
WithDefaults provides default settings for optional fields in UserConfig