config

package
v1.15.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 1, 2026 License: Apache-2.0 Imports: 19 Imported by: 0

Documentation

Index

Constants

View Source
const KernelMaxChunksInMemoryConfKey = "cloudfetch_max_chunks_in_memory"

KernelMaxChunksInMemoryConfKey is the CLIENT-only session conf the kernel reads to bound how many decompressed CloudFetch chunks it holds in memory (WithKernelMaxChunksInMemory). Unlike MetricViewMetadataConfKey this is NOT a server SET parameter and is NOT added by EffectiveSessionParams: the kernel backend injects it into its own SessionConf only, and the kernel strips it before the SEA wire (it is absent from the kernel's server allowlist). It must match the key the kernel's apply_client_result_overrides looks for.

View Source
const MetricViewMetadataConfKey = "spark.sql.thriftserver.metadata.metricview.enabled"

MetricViewMetadataConfKey is the server session conf that enables metric-view metadata (WithEnableMetricViewMetadata). Despite the "thriftserver" in its name — a historical server-side name — it is an ordinary Spark SQL session conf the server honors regardless of client transport, so both the Thrift and kernel backends send the identical key/value. Defined once here so no backend hardcodes the literal (see EffectiveSessionParams).

Variables

This section is empty.

Functions

This section is empty.

Types

type ArrowConfig added in v1.1.0

type ArrowConfig struct {
	UseArrowBatches         bool
	UseArrowNativeDecimal   bool
	UseArrowNativeTimestamp bool

	// the following are currently not supported
	UseArrowNativeComplexTypes  bool
	UseArrowNativeIntervalTypes bool
}

func (ArrowConfig) DeepCopy added in v1.1.0

func (arrowConfig ArrowConfig) DeepCopy() ArrowConfig

DeepCopy returns a true deep copy of UserConfig

func (ArrowConfig) WithDefaults added in v1.1.0

func (ucfg ArrowConfig) WithDefaults() ArrowConfig

type CloudFetchConfig added in v1.4.0

type CloudFetchConfig struct {
	UseCloudFetch                bool
	MaxDownloadThreads           int
	MaxFilesInMemory             int
	MinTimeToExpiry              time.Duration
	CloudFetchSpeedThresholdMbps float64 // Minimum download speed in MBps before WARN logging (default: 0.1)
	HTTPClient                   *http.Client
}

func (CloudFetchConfig) DeepCopy added in v1.4.0

func (cfg CloudFetchConfig) DeepCopy() CloudFetchConfig

func (CloudFetchConfig) WithDefaults added in v1.4.0

func (cfg CloudFetchConfig) WithDefaults() CloudFetchConfig

type Config

type Config struct {
	UserConfig
	TLSConfig *tls.Config // nil disables TLS
	ArrowConfig
	PollInterval              time.Duration
	ClientTimeout             time.Duration // max time the http request can last
	PingTimeout               time.Duration // max time allowed for ping
	CanUseMultipleCatalogs    bool
	DriverName                string
	DriverVersion             string
	ThriftProtocol            string
	ThriftTransport           string
	ThriftProtocolVersion     cli_service.TProtocolVersion
	ThriftDebugClientProtocol bool

	// KernelExperimental carries experimental, kernel-backend-only options that
	// have no equivalent on the default (Thrift) path — currently the richer TLS
	// surface (a trusted-CA bundle and an independent hostname-skip) the kernel
	// exposes over its C ABI. It lives here on Config, NOT on UserConfig, so it
	// stays off the stable exported/DSN surface (the same treatment TLSConfig and
	// ArrowConfig get). nil means no experimental option was set. The Thrift
	// backend rejects a non-nil value loudly; the kernel backend forwards it to
	// the kernel C ABI. Mirrors Node's non-exported InternalConnectionOptions /
	// Python's underscore-prefixed kwargs.
	KernelExperimental *KernelExperimentalConfig
}

Driver Configurations. Only UserConfig are currently exposed to users

func WithDefaults

func WithDefaults() *Config

WithDefaults provides default settings for Config

func (*Config) DeepCopy

func (c *Config) DeepCopy() *Config

DeepCopy returns a true deep copy of Config

func (*Config) EffectiveSessionParams added in v1.15.0

func (c *Config) EffectiveSessionParams() map[string]string

EffectiveSessionParams returns the session confs to send to the server: the user-supplied SessionParams plus any conf derived from a higher-level option (currently metric-view metadata). Both backends call this, so the derivation is backend-neutral — neither special-cases the option nor hardcodes the conf literal. The returned map is always a fresh copy the caller may mutate freely.

func (*Config) ToEndpointURL

func (c *Config) ToEndpointURL() (string, error)

ToEndpointURL generates the endpoint URL from Config that a Thrift client will connect to

type ConfigValue added in v1.10.0

type ConfigValue[T any] struct {
	// contains filtered or unexported fields
}

ConfigValue represents a configuration value that can be set by client or resolved from server. This implements the config overlay pattern: client > server > default

T is the type of the configuration value (bool, string, int, etc.)

Example usage:

type MyConfig struct {
    EnableFeature ConfigValue[bool]
    BatchSize     ConfigValue[int]
}

// Client explicitly sets value (overrides server)
config.EnableFeature = NewConfigValue(true)

// Client doesn't set value (use server)
config.EnableFeature = ConfigValue[bool]{} // nil/unset

// Resolve value with overlay priority
enabled := config.EnableFeature.Resolve(ctx, serverResolver, defaultValue)

func NewConfigValue added in v1.10.0

func NewConfigValue[T any](value T) ConfigValue[T]

NewConfigValue creates a ConfigValue with a client-set value. The value will override any server-side configuration.

func ParseBoolConfigValue added in v1.10.0

func ParseBoolConfigValue(params map[string]string, key string) ConfigValue[bool]

ParseBoolConfigValue parses a string value into a ConfigValue[bool]. Returns unset ConfigValue if the parameter is not present.

Example:

params := map[string]string{"enableFeature": "true"}
value := ParseBoolConfigValue(params, "enableFeature")
// value.IsSet() == true, value.Get() == (true, true)

func ParseIntConfigValue added in v1.10.0

func ParseIntConfigValue(params map[string]string, key string) ConfigValue[int]

ParseIntConfigValue parses a string value into a ConfigValue[int]. Returns unset ConfigValue if the parameter is not present or invalid.

func ParseStringConfigValue added in v1.10.0

func ParseStringConfigValue(params map[string]string, key string) ConfigValue[string]

ParseStringConfigValue parses a string value into a ConfigValue[string]. Returns unset ConfigValue if the parameter is not present.

func (ConfigValue[T]) Get added in v1.10.0

func (cv ConfigValue[T]) Get() (T, bool)

Get returns the client-set value and whether it was set. If not set, returns zero value and false.

func (ConfigValue[T]) IsSet added in v1.10.0

func (cv ConfigValue[T]) IsSet() bool

IsSet returns true if the client explicitly set this configuration value.

func (ConfigValue[T]) Resolve added in v1.10.0

func (cv ConfigValue[T]) Resolve(
	ctx context.Context,
	serverResolver ServerResolver[T],
	defaultValue T,
) T

Resolve applies config overlay priority to determine the final value:

Priority 1: Client Config - if explicitly set (overrides server)
Priority 2: Server Config - resolved via serverResolver (when client doesn't set)
Priority 3: Default Value - used when server unavailable/errors (fail-safe)

Parameters:

  • ctx: Context for server requests
  • serverResolver: How to fetch from server (can be nil if no server config)
  • defaultValue: Fail-safe default when client unset and server unavailable

Returns: The resolved configuration value following overlay priority

func (ConfigValue[T]) ResolveWithContext added in v1.10.0

func (cv ConfigValue[T]) ResolveWithContext(
	ctx context.Context,
	host string,
	httpClient *http.Client,
	serverResolver ServerResolver[T],
	defaultValue T,
) T

ResolveWithContext is a more flexible version that takes host and httpClient. This is the recommended method for production use.

type KernelExperimentalConfig added in v1.15.0

type KernelExperimentalConfig struct {
	// TLSTrustedCertsPEM is a PEM CA bundle added to the kernel's trust store on
	// top of the system roots (maps to kernel_session_config_set_tls_trusted_certs).
	// Needed because the kernel's rustls stack ignores SSL_CERT_FILE, so a custom
	// CA must be handed to the kernel explicitly.
	TLSTrustedCertsPEM []byte
	// TLSClientCertPEM / TLSClientKeyPEM are the paired mTLS client identity
	// forwarded through kernel_session_config_set_tls_client_certificate.
	// TLSClientCertConfigured distinguishes an explicit call with empty values
	// from the option never being set, so invalid input cannot fail open.
	TLSClientCertPEM        []byte
	TLSClientKeyPEM         []byte
	TLSClientCertConfigured bool
	// TLSSkipHostnameVerify skips only the certificate hostname check, independent
	// of the blanket InsecureSkipVerify (which relaxes both chain and hostname).
	// Maps to kernel_session_config_set_tls_skip_hostname_verification.
	TLSSkipHostnameVerify bool

	// ProxyURL / ProxyUsername / ProxyPassword / ProxyBypassHosts configure an
	// explicit HTTP proxy on the kernel path (WithKernelProxy), overriding the
	// HTTP(S)_PROXY / NO_PROXY environment the driver otherwise mirrors. The
	// credentials and bypass list are the "advanced" fields the env-var path
	// can't express (a structured no-proxy list, out-of-band basic auth). Empty
	// ProxyURL leaves the environment-derived proxy in effect. Maps to
	// kernel_session_config_set_proxy(url, username, password, bypass_hosts).
	ProxyURL         string
	ProxyUsername    string
	ProxyPassword    string
	ProxyBypassHosts string

	// RetryOverallTimeout is the cumulative retry budget across all attempts on
	// the kernel path (WithKernelRetryOverallTimeout). Zero = keep the kernel
	// default (900s). WithRetries only carries the per-attempt backoff bounds +
	// max attempts (mirroring the Thrift RetryWaitMin/Max/RetryMax surface); the
	// overall budget is a kernel-only knob the Thrift path has no equivalent for,
	// so it lives here and maps to the 4th arg of
	// kernel_session_config_set_retry_config (matching the pyo3/napi
	// retry_overall_timeout knob).
	RetryOverallTimeout time.Duration

	// MaxChunksInMemory bounds how many decompressed CloudFetch chunks the kernel
	// holds in memory at once (WithKernelMaxChunksInMemory) — the knob that trades
	// throughput for peak RSS on large result sets. Zero = keep the kernel default
	// (16). A positive value is forwarded as the client-only
	// "cloudfetch_max_chunks_in_memory" session conf, which the kernel folds into
	// its result config and strips before the SEA wire; it is kernel-only because
	// the Thrift path has no such in-memory-chunk knob.
	MaxChunksInMemory int

	// DecimalAsFloat scans top-level DECIMAL columns to a lossy float64 instead of
	// the exact string (WithKernelDecimalAsFloat). Kernel-only; off by default.
	DecimalAsFloat bool

	// TokenCacheEnabled controls the kernel's on-disk OAuth U2M token-cache persistence
	// (WithTokenCache). When false (the default), tokens are held in memory only;
	// when true, the refresh token is persisted encrypted to ~/.config/databricks-sql-kernel/oauth/.
	// U2M-only: PAT and M2M ignore this setting. Maps to
	// kernel_session_config_set_u2m_token_cache_config; a nil KernelExperimental or
	// false TokenCacheEnabled calls the setter with enabled=false so the kernel does
	// NOT persist to disk by default.
	TokenCacheEnabled bool
}

KernelExperimentalConfig holds the experimental, kernel-only connection knobs set via the WithKernel* options. Every field is either forwarded to the kernel C ABI (kernel backend) or rejected (Thrift backend) — never silently dropped; the exhaustiveness guard TestKernelExperimentalFieldsClassified asserts this so a newly-added field can't slip through unclassified.

func (*KernelExperimentalConfig) DeepCopy added in v1.15.0

DeepCopy returns a deep copy of the experimental config, or nil for a nil receiver. The byte slice is copied so a mutation of the copy can't reach back into the original (the connector may DeepCopy the whole Config per conn).

type ServerResolver added in v1.10.0

type ServerResolver[T any] interface {
	// Resolve fetches the configuration value from the server.
	// Returns the value and any error encountered.
	// On error, the config overlay will fall back to the default value.
	Resolve(ctx context.Context, host string, httpClient *http.Client) (T, error)
}

ServerResolver defines how to fetch a configuration value from the server. Implementations should handle caching, retries, and error handling.

type UserConfig

type UserConfig struct {
	Protocol       string
	Host           string // from databricks UI
	Port           int    // from databricks UI
	HTTPPath       string // from databricks UI
	Catalog        string
	Schema         string
	Authenticator  auth.Authenticator
	AccessToken    string        // from databricks UI
	MaxRows        int           // max rows per page
	QueryTimeout   time.Duration // Timeout passed to server for query processing
	UserAgentEntry string
	Location       *time.Location
	SessionParams  map[string]string
	RetryWaitMin   time.Duration
	RetryWaitMax   time.Duration
	RetryMax       int
	// Telemetry configuration
	// Uses config overlay pattern: client > server > default.
	// Unset = check server feature flag; explicitly true/false overrides the server.
	EnableTelemetry          ConfigValue[bool]
	TelemetryBatchSize       int           // 0 = use default (200)
	TelemetryFlushInterval   time.Duration // 0 = use default (30s)
	Transport                http.RoundTripper
	UseLz4Compression        bool
	EnableMetricViewMetadata bool
	// UseArrowNativeDecimalDSN is a DSN-only carrier for the useArrowNativeDecimal
	// parameter. The authoritative setting lives on ArrowConfig; ParseDSN records
	// the DSN value here (because it can only return a UserConfig) and the
	// connector copies it into Config.ArrowConfig.UseArrowNativeDecimal when it is
	// assembled. The name intentionally differs from ArrowConfig's field so the
	// promoted selector Config.UseArrowNativeDecimal stays unambiguous.
	UseArrowNativeDecimalDSN bool
	CloudFetchConfig
	// UseKernel selects the SEA-via-kernel backend instead of Thrift. See the
	// WithUseKernel connector option for the build requirements. DSN: useKernel=true.
	UseKernel bool
	// WarehouseID is the bare SQL warehouse id, used by the kernel backend (which
	// addresses a warehouse by id) in preference to HTTPPath. The Thrift backend
	// ignores it and routes by HTTPPath. DSN: warehouseId=<id>.
	WarehouseID string
	// TokenCacheEnabledDSN is a DSN-only carrier for the tokenCache parameter (U2M-only).
	// The authoritative setting lives on KernelExperimentalConfig; ParseDSN records the
	// DSN value here (because it can only return a UserConfig) and the connector copies
	// it into Config.KernelExperimental.TokenCacheEnabled when it is assembled. The name
	// intentionally differs from KernelExperimentalConfig's field so the setting stays
	// unambiguous. False by default; when true, enables on-disk token-cache persistence
	// for U2M OAuth. DSN: tokenCache=true.
	TokenCacheEnabledDSN bool
}

UserConfig is the set of configurations exposed to users

func ParseDSN

func ParseDSN(dsn string) (UserConfig, error)

ParseDSN constructs UserConfig and CloudFetchConfig by parsing DSN string supplied to `sql.Open()`

func (UserConfig) DeepCopy

func (ucfg UserConfig) DeepCopy() UserConfig

DeepCopy returns a true deep copy of UserConfig

func (UserConfig) WithDefaults

func (ucfg UserConfig) WithDefaults() UserConfig

WithDefaults provides default settings for optional fields in UserConfig

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL