server

package
v0.0.0-...-f03633c Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 13, 2026 License: MIT Imports: 61 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	AcquisitionOutcomeOK       = "ok"
	AcquisitionOutcomeCanceled = "canceled"
	AcquisitionOutcomeCapacity = "capacity"
	AcquisitionOutcomeDraining = "draining"
	AcquisitionOutcomeDisabled = "disabled"
	AcquisitionOutcomeError    = "error"
)

Bounded outcome labels shared by the tier's two acquisition metrics — duckgres_exploratory_escalations_total here and duckgres_session_activation_total in the control plane. Every label is derived from the CLASSIFIED SQLSTATE, never from the error text, so the label set stays closed no matter what a worker or the K8s API says.

View Source 
const (
	// QueryEventStart is emitted when a statement begins executing. Its
	// terminal counterpart shares the same query_id. A QueryStart with no
	// terminal is how a query that never came back — worker OOM-killed, pod
	// evicted — becomes visible; without it such a query leaves no trace at
	// all.
	QueryEventStart = "QueryStart"
	// QueryEventFinish is a statement that completed and returned to the client.
	QueryEventFinish = "QueryFinish"
	// QueryEventExceptionBeforeStart is a statement that failed BEFORE
	// EXECUTION BEGAN: auth or policy denial, a transpile error, a failure to
	// obtain a worker — and, in practice most often, an extended-protocol
	// Describe whose prepare the engine rejected (a binder error). That last
	// case is why the boundary is "execution began", not "an engine saw it":
	// Describe hands the statement to the worker to learn its result schema, so
	// the engine does see it, and it still never runs. ClickHouse draws the
	// line the same way — analysis-time failures are ExceptionBeforeStart.
	// There is no QueryStart for these, by definition.
	QueryEventExceptionBeforeStart = "ExceptionBeforeStart"
	// QueryEventExceptionWhileProcessing is a statement that failed after
	// execution began.
	QueryEventExceptionWhileProcessing = "ExceptionWhileProcessing"
)

Query-log event types, matching ClickHouse's system.query_log `type` column so a duckgres query log can be read — or exported — with the same vocabulary. ClickHouse encodes these as Enum8; we store the names, because existing rows already carry them and the DuckLake view is nicer to read. The codes are kept here so an export can map straight across.

View Source 
const (
	OidBool        int32 = 16
	OidBytea       int32 = 17
	OidChar        int32 = 18 // "char" - single-byte internal type
	OidName        int32 = 19 // name - 64-byte internal type for identifiers
	OidInt8        int32 = 20 // bigint
	OidInt2        int32 = 21 // smallint
	OidInt4        int32 = 23 // integer
	OidText        int32 = 25
	OidOid         int32 = 26
	OidFloat4      int32 = 700  // real
	OidFloat8      int32 = 701  // double precision
	OidBpchar      int32 = 1042 // blank-padded char
	OidVarchar     int32 = 1043
	OidDate        int32 = 1082
	OidTime        int32 = 1083
	OidTimestamp   int32 = 1114
	OidTimestamptz int32 = 1184
	OidInterval    int32 = 1186
	OidNumeric     int32 = 1700
	OidUUID        int32 = 2950
	OidTimetz      int32 = 1266
	OidJSON        int32 = 114
	OidJSONB       int32 = 3802

	// Array OIDs
	OidBoolArray        int32 = 1000
	OidInt2Array        int32 = 1005
	OidInt4Array        int32 = 1007
	OidTextArray        int32 = 1009
	OidVarcharArray     int32 = 1015
	OidInt8Array        int32 = 1016
	OidFloat4Array      int32 = 1021
	OidFloat8Array      int32 = 1022
	OidTimestampArray   int32 = 1115
	OidDateArray        int32 = 1182
	OidTimeArray        int32 = 1183
	OidTimestamptzArray int32 = 1185
	OidIntervalArray    int32 = 1187
	OidNumericArray     int32 = 1231
	OidTimetzArray      int32 = 1270
	OidUUIDArray        int32 = 2951
)

PostgreSQL type OIDs

View Source 
const (
	ExitSuccess     = 0  // Clean disconnect
	ExitError       = 1  // Error (crash, protocol error)
	ExitAuthFailure = 10 // Authentication failure (triggers rate limit update)
)

Exit codes for child processes

View Source 
const ClientIdleTimeoutGUCName = "duckgres.idle_timeout"

ClientIdleTimeoutGUCName is the connect-time option a client may use to request a longer idle timeout, for example: 

PGOPTIONS='-c duckgres.idle_timeout=15m' psql ...

It is deliberately a Duckgres-specific option: PostgreSQL's idle_session_timeout has different semantics and is currently ignored for compatibility. This option is accepted only when the operator sets a positive Config.ClientIdleTimeoutMax.

View Source 
const DefaultControlPlaneIdleTimeout = 15 * time.Minute

DefaultControlPlaneIdleTimeout is the connection idle timeout the control plane applies when none is configured. In remote/process control-plane mode an idle client connection pins a worker (a scarce k8s pod or local process), so an idle connection is closed after this long — its message loop hits the read deadline, returns, and the worker is released back to the hot-idle pool. Operators override it with --idle-timeout (a negative value disables it).

This was 60s, which reclaimed a pinned worker quickly but broke any client that pauses between statements. A batch client running its DAG across several parallel connections idles a connection whenever a thread waits on work elsewhere. It then finds the connection closed and reports a lost connection rather than an idle one — and a client cannot safely replay a write that may already have committed, so the failure reaches the user.

5m fixed the worst of that but still sat inside the gaps clients actually leave. Measured over a production deployment, the connections it reaped were not abandoned: the client came back a median of 8s after the reap and at worst 58s, so every observed idle gap fell between 5m00s and 5m58s. Those clients were missing the threshold by seconds and paying a cold worker respawn to get back, which is churn with no reclaim benefit — and only ~4% of reaps had no client return at all, so a longer window is not exposing many genuinely abandoned connections.

15m clears that cluster with margin for a slower run, and costs on the order of one continuously-held worker across the fleet. It is deliberately at the ceiling the accompanying test enforces: past this, an abandoned connection holds a pinned worker too long to justify by default. Deployments that want tighter worker density set --idle-timeout / DUCKGRES_IDLE_TIMEOUT; a client that needs longer asks for it per connection with duckgres.idle_timeout, bounded by DUCKGRES_CLIENT_IDLE_TIMEOUT_MAX.

This is a frequency reduction, not a correctness fix: a client whose gap can exceed any finite timeout still needs to treat the reap as retryable.

View Source 
const DefaultDuckLakeSpecVersion = ducklake.DefaultSpecVersion

DefaultDuckLakeSpecVersion is re-exported for callers that referenced the constant under the server package before the migration code moved.

View Source 
const DefaultRecentErrorCap = 500

DefaultRecentErrorCap is how many recent errors each server retains in memory for the admin Errors page. It's a bounded live-triage buffer, not durable history — long-term error history lives in the external query-log pipeline.

View Source 
const DefaultSessionInitTimeout = 10 * time.Second

DefaultSessionInitTimeout bounds startup metadata initialization and catalog probes.

View Source 
const ProfilingOutputPath = "/tmp/duckgres-profiling.json"

ProfilingOutputPath is where DuckDB writes the per-query profile JSON. The duckdbservice worker reads this file after each query and forwards the contents to the control plane via gRPC trailer (see duckdbservice.sendProfilingMetadata) where EnrichSpanWithProfiling turns it into OTEL child spans.

View Source 
const S3CacheGUCName = s3CacheGUCName

S3CacheGUCName is the startup-option / GUC name, exported for the control plane's startup-option parsing.

View Source 
const WorkerTTLGUCName = "duckgres.worker_ttl"

WorkerTTLGUCName is the duckgres-namespaced session GUC controlling how long the session's worker stays hot-idle (warm, reusable) after its last session ends, on the remote/k8s backend: 

SET duckgres.worker_ttl = '20m'

It is the mid-session form of the `-c duckgres.worker_ttl=...` startup option (controlplane/worker_profile.go), for clients that cannot set startup options. Used as the SHOW result column label.

Variables

View Source 
var (
	NewRateLimiter              = auth.NewRateLimiter
	DefaultRateLimitConfig      = auth.DefaultRateLimitConfig
	BeginRateLimitedAuthAttempt = auth.BeginRateLimitedAuthAttempt
	RecordFailedAuthAttempt     = auth.RecordFailedAuthAttempt
	RecordSuccessfulAuthAttempt = auth.RecordSuccessfulAuthAttempt
	ValidateUserPassword        = auth.ValidateUserPassword
)
View Source 
var (
	HasAttachedCatalog          = sessionmeta.HasAttachedCatalog
	InitSessionDatabaseMetadata = sessionmeta.InitSessionDatabaseMetadata
)

HasAttachedCatalog and InitSessionDatabaseMetadata moved to server/sessionmeta. Re-exports kept here for the dozen call sites in the control plane and elsewhere; new code should import server/sessionmeta directly.

View Source 
var (
	CheckDuckLakeMigrationVersion   = ducklake.CheckMigrationVersion
	CheckAndBackupDuckLakeMigration = ducklake.CheckAndBackupMigration
	BackupDuckLakeMetadata          = ducklake.BackupMetadata
	DefaultDeltaCatalogPath         = ducklake.DefaultDeltaCatalogPath
)

Re-exports of the migration / backup / delta-path entry points so callers that referenced them under the server package continue to compile after the implementation moved to server/ducklake. New code should import server/ducklake directly.

View Source 
var (
	IncrementOpenConnections = observe.IncrementOpenConnections
	DecrementOpenConnections = observe.DecrementOpenConnections
)

connectionsGauge, IncrementOpenConnections, DecrementOpenConnections moved to server/observe. The aliases below preserve the existing server.X spellings for the call sites in this package and the control plane.

View Source 
var (
	IsEmptyQuery          = sqlcore.IsEmptyQuery
	OTELGRPCClientHandler = sqlcore.OTELGRPCClientHandler
)
View Source 
var (
	SystemMemoryBytes   = sysinfo.SystemMemoryBytes
	ValidateMemoryLimit = sysinfo.ValidateMemoryLimit
	ParseMemoryBytes    = sysinfo.ParseMemoryBytes
)
View Source 
var (
	NewACMEManager     = tlscert.NewACMEManager
	NewACMEDNSManager  = tlscert.NewACMEDNSManager
	EnsureCertificates = tlscert.EnsureCertificates
)
View Source 
var GenerateSecretKey = wire.GenerateSecretKey

GenerateSecretKey re-exports wire.GenerateSecretKey so existing callers that imported it from server keep compiling. New code should use server/wire directly.

View Source 
var RedactSecrets = wire.RedactSecrets

RedactSecrets is a re-export var for callers that imported it from this package. See server/wire/redact.go for the implementation.

View Source 
var Tracer = observe.Tracer

Tracer re-exports observe.Tracer for callers that previously imported it from this package. The tracing helpers, profiling-output enrichment, and connection counters all live in server/observe; new code should import that package directly.

Functions

func AcquisitionFailureOutcome 

func AcquisitionFailureOutcome(code string) string

AcquisitionFailureOutcome maps a classified SQLSTATE to the bounded failure class label above. FAILURES only — each metric supplies its own success label (this package's counter uses AcquisitionOutcomeOK, the control plane's activation counter keeps its established "success"), because the two have been published under different names and dashboards read both.

Draining and disabled are broken out from the generic error bucket deliberately: a control plane rolling out, and an operator disabling an account, each look identical to a broken cluster otherwise.

func ActivateDBConnection 

func ActivateDBConnection(db *sql.DB, cfg Config, duckLakeSem chan struct{}, username string) error

ActivateDBConnection applies tenant-specific DuckLake runtime to an already initialized generic DuckDB connection used by a shared warm worker.

func ApplyConnectionS3CacheOption 

func ApplyConnectionS3CacheOption(cc *clientConn, raw string) error

ApplyConnectionS3CacheOption applies a pre-validated `-c duckgres.s3_cache=…` startup option on a control-plane connection whose session executor is already bound. A returned error means the worker could not swap the S3 transport — the caller should refuse the connection (FATAL) rather than start a session whose cache state doesn't match the requested option (a benchmark connecting with s3_cache=off must never silently run cached).

func ApplyProfilingSettings 

func ApplyProfilingSettings(ctx context.Context, conn *sql.Conn)

ApplyProfilingSettings runs ProfilingSetupSQL against the given connection, writing profiles to ProfilingOutputPath. Use this for connections that bypass ConfigureMainDB — primarily fresh per-session connections in the cluster-mode worker, where eviction between sessions discards whatever settings ConfigureMainDB applied.

func AttachDeltaCatalog 

func AttachDeltaCatalog(db *sql.DB, dlCfg DuckLakeConfig, sem chan struct{}) error

AttachDeltaCatalog attaches the configured Delta Lake catalog/table alongside DuckLake. It reuses the DuckLake S3 secret settings so Delta scans can access the same object store credentials.

Delta is enabled by default. When no path is derivable (e.g. a plain standalone DuckDB instance with no DuckLake object_store/data_path), this is a benign no-op: there's no Delta sibling to attach.

func AttachDuckLake 

func AttachDuckLake(db *sql.DB, dlCfg DuckLakeConfig, sem chan struct{}, dataDir string) error

AttachDuckLake attaches a DuckLake catalog if configured (but does NOT set it as default). Call setDuckLakeDefault after creating per-connection views in memory.main. This is a standalone function so it can be reused by control plane workers. dataDir is used for writing migration backup files if a schema upgrade is needed.

func BootstrapBundledExtensions 

func BootstrapBundledExtensions(dataDir string) error

BootstrapBundledExtensions eagerly seeds bundled extension binaries into the configured extension_directory cache once per data directory.

func BoundQueryLogText 

func BoundQueryLogText(text string) string

BoundQueryLogText is the shared 4096-byte UTF-8-safe cap used by the durable query log and the PostHog OTLP query-text handler.

func BuildDuckDBCopyFromSQL 

func BuildDuckDBCopyFromSQL(tableName, columnList, filePath string, opts *CopyFromOptions) string

BuildDuckDBCopyFromSQL generates a DuckDB COPY FROM statement

func CancelClientConn 

func CancelClientConn(cc *clientConn)

CancelClientConn cancels the context of a clientConn.

func CloseConnectionMetrics 

func CloseConnectionMetrics(cc *clientConn) time.Duration

CloseConnectionMetrics records the completed connection's lifetime in the duckgres_connection_duration_seconds histogram (per org) and returns the elapsed duration so the caller can log it. Call exactly once per connection at teardown. backendStart is always set in NewClientConn, so the duration is always meaningful for control-plane connections.

func ConfigureDBConnection 

func ConfigureDBConnection(db *sql.DB, cfg Config, duckLakeSem chan struct{}, username string, serverStartTime time.Time, serverVersion string) error

ConfigureDBConnection initializes an existing DuckDB connection with pg_catalog, information_schema, and DuckLake catalog attachment.

func ConfigureMainDB 

func ConfigureMainDB(db *sql.DB, cfg Config, username string) error

ConfigureMainDB applies the per-instance DuckDB settings (threads, memory, temp dir, extensions, profiling) that the client-query DB needs. Shared between openBaseDB (single-DB path) and OpenDuckDBPair (shared-connector path) so the main DB is configured identically either way.

func ConnectionBilling 

func ConnectionBilling(cc *clientConn) (orgID, username, querySource string, millicores, mib int64, dur time.Duration)

ConnectionBilling returns the data needed to meter one connection's compute-usage at teardown: the org, the authenticated username (used to resolve the informational team id stamped onto the bucket — the user's own team when it has one, else the org's oldest team), the session's query source (the `duckgres.query_source` GUC — "standard" unless the client set it), the provisioned worker size in milli-units, and the connection's elapsed lifetime. millicores == 0 means metering should be skipped (unknown worker size). Call at the same teardown point as CloseConnectionMetrics. A mid-connection GUC change is not split: the whole connection is metered under the final value (documented in docs/design/billing-pull-api.md).

The query source is clamped to the closed {standard, endpoints} set as defense in depth: every set path already validates (22023 at SET / startup time), so a non-canonical value here means a validation bypass — degrade it to the default rather than writing unbounded-cardinality client input into the billing bucket key (and onward into billing exports).

func CreateDBConnection 

func CreateDBConnection(cfg Config, duckLakeSem chan struct{}, username string, serverStartTime time.Time, serverVersion string) (*sql.DB, error)

CreateDBConnection creates a DuckDB connection for a client session. Uses in-memory database as an anchor for DuckLake attachment (actual data lives in RDS/S3). This is a standalone function so it can be reused by both the server and control plane workers. serverStartTime is the time the top-level server process started (may differ from processStartTime in process isolation mode where each child has its own processStartTime). serverVersion is the version of the top-level server/control-plane process.

func CreatePassthroughDBConnection 

func CreatePassthroughDBConnection(cfg Config, duckLakeSem chan struct{}, username string, serverStartTime time.Time, serverVersion string) (*sql.DB, error)

CreatePassthroughDBConnection creates a DuckDB connection without pg_catalog or information_schema initialization. DuckLake is still attached if configured so passthrough users can access the same data. This is used for passthrough users who send DuckDB-native SQL and don't need the PostgreSQL compatibility layer.

func DefaultDuckDBThreads 

func DefaultDuckDBThreads(cpuMillicores int64) int

DefaultDuckDBThreads returns the default DuckDB thread count for a CPU allocation expressed in millicores. DuckDB gets 2.5 threads per CPU, rounded up so fractional CPU allocations never lose their share of parallelism.

func DuckDBDSN 

func DuckDBDSN(cfg Config, username string) (string, error)

DuckDBDSN returns the DSN openBaseDB / the duckdbservice pair builder use for cfg/username. Exported so duckdbservice (which holds the duckdb-go-v2 import) can build a *duckdb.Connector against the same DSN that openBaseDB would have passed to sql.Open.

func InitMinimalServer 

func InitMinimalServer(s *Server, cfg Config, queryCancelCh <-chan struct{})

InitMinimalServer initializes a Server struct with minimal fields for use in control plane worker sessions.

func LegacySecretDirectory 

func LegacySecretDirectory(cfg Config) string

LegacySecretDirectory returns DuckDB's pre-pinning default persistent-secret location for a worker whose HOME is its DataDir (<DataDir>/.duckdb/stored_secrets). This is where secrets accumulated before SecretDirectory pinning existed, and the only reason it's a named helper is so the recycle wipe and this derivation stay colocated and can't drift apart. Returns "" when DataDir is unset.

func LoadExtensions 

func LoadExtensions(db *sql.DB, extensions []string) error

LoadExtensions installs and loads DuckDB extensions. This is a standalone function so it can be reused by control plane workers. Extension strings can include a source, e.g. "cache_httpfs FROM community". INSTALL uses the full string; LOAD uses just the extension name.

NOTE: Extension names come from trusted server config, not user input.

func MarkConnectionPinned 

func MarkConnectionPinned(cc *clientConn)

MarkConnectionPinned takes a connection off the exploratory tier WITHOUT a worker switch — used by the control-plane activator when the very first statement is already a pinning one and it therefore acquired the standard profile directly. Without this the connection would still believe it is on the small worker and escalate (destroy + re-acquire) one statement later.

func NewClientConn 

func NewClientConn(s *Server, conn net.Conn, reader *bufio.Reader, writer *bufio.Writer,
	username, orgID, database, applicationName string, executor QueryExecutor, pid, secretKey int32, workerID int, workerPod string) *clientConn

NewClientConn creates a clientConn with pre-initialized fields for use by the control plane worker. The returned value is opaque (*clientConn) but can be used with SendInitialParams and RunMessageLoop.

func NormalizeIdleTimeout 

func NormalizeIdleTimeout(configured, zeroDefault time.Duration) time.Duration

NormalizeIdleTimeout resolves a configured connection idle timeout: zero means "unset" → use zeroDefault; a negative value means "explicitly disabled" → 0 (no timeout); a positive value is used as-is. Shared by standalone (24h default) and the control plane (DefaultControlPlaneIdleTimeout).

func ParseStartupOptions 

func ParseStartupOptions(options string) map[string]string

ParseStartupOptions parses the Postgres startup-message `options` parameter, which carries `-c name=value` GUC settings (also accepted: `-cname=value` and `--name=value`), e.g. `-c search_path=ducklake.main`. libpq's `options` connection keyword, the PGOPTIONS env var, and pgjdbc's `currentSchema` all arrive here. Values may contain backslash-escaped spaces. Returns a map of setting name -> value (later settings win on duplicate names).

func ProcessVersion 

func ProcessVersion() string

ProcessVersion returns the version string for this process.

func ProfilingSetupSQL 

func ProfilingSetupSQL(outputPath string) []string

ProfilingSetupSQL returns the SQL statements that configure DuckDB profiling so the output is written to outputPath. These are session-scoped — DuckDB rejects `SET GLOBAL` for each one — so any caller that hands out fresh connections (e.g. after evictConnFromPool) must re-run them per connection.

func ReadMessage 

func ReadMessage(r io.Reader) (byte, []byte, error)

func ReadStartupMessage 

func ReadStartupMessage(r io.Reader) (wire.StartupMessage, error)

func RecoverAbortedTransaction 

func RecoverAbortedTransaction[T any](
	err error,
	canRollback bool,
	rollback func() error,
	retry func() (T, error),
) (T, error, bool)

RecoverAbortedTransaction rolls back and retries once when a DuckLake-backed connection is stuck in "Current transaction is aborted" state. This is only safe when the caller owns the transaction lifecycle (autocommit / no active user transaction). Callers should pass canRollback=false for explicit user transactions so the original error is surfaced unchanged.

func RefreshS3Secret 

func RefreshS3Secret(db *sql.DB, dlCfg DuckLakeConfig, duckLakeSem chan struct{}) error

RefreshS3Secret replaces the DuckDB S3 secret with updated credentials. Used when a hot-idle worker is reclaimed and STS credentials have rotated. Respects the configured S3 provider (config, aws_sdk, credential_chain).

func RegisterDuckDBAppender 

func RegisterDuckDBAppender(f DuckDBAppendFunc)

RegisterDuckDBAppender wires a real DuckDB Appender implementation into the COPY codepath. duckdbservice's init() calls this; binaries that don't link duckdbservice get the unavailable fallback below.

func RegisterValueNormalizer 

func RegisterValueNormalizer(n ValueNormalizer)

RegisterValueNormalizer adds a hook consulted by normalizeDriverValue before the binary encoders dispatch on the value's type. Intended for use from init() in importers that own driver-specific value types — duckdbservice registers a normalizer that converts duckdb.Interval and duckdb.Decimal to their arrowmap equivalents.

func RunChildMode 

func RunChildMode()

RunChildMode is the entry point for child worker processes. It reconstructs the TCP connection from FD 3, completes TLS handshake, authenticates the user, creates a DuckDB connection, and runs the message loop.

Configuration is read from stdin as JSON (more secure than env vars for passwords).

Exit codes:

  • 0: Success (clean disconnect)
  • 1: Error (crash, protocol error)
  • 10: Authentication failure

func RunMessageLoop 

func RunMessageLoop(cc *clientConn) error

RunMessageLoop runs the main message loop for a client connection. It cancels the connection context when the loop exits, ensuring in-flight query contexts (and any gRPC calls derived from them) are cancelled promptly.

func RunShell 

func RunShell(cfg Config)

RunShell starts an interactive SQL shell with a fully initialized DuckDB connection. It uses the same CreateDBConnection path as the PostgreSQL server, so extensions, DuckLake, and pg_catalog views are all available.

func S3ProviderForConfig 

func S3ProviderForConfig(dlCfg DuckLakeConfig) string

S3ProviderForConfig returns the effective S3 provider for the given DuckLake config.

func SanitizeSearchPath 

func SanitizeSearchPath(s string) (string, bool)

SanitizeSearchPath validates a client-supplied search_path so it can be safely embedded in `SET search_path = '<value>'`. Returns (trimmed, true) when the value is a plausible, injection-safe search_path; ("", false) otherwise (callers should then fall back to the default search_path).

func SecretDirectory 

func SecretDirectory(cfg Config) string

SecretDirectory returns the directory DuckDB should use for persistent secrets for this instance, or "" to leave DuckDB's default in place.

DuckDB defaults persistent secrets to $HOME/.duckdb/stored_secrets, independent of the database file. On a worker that means a CREATE PERSISTENT SECRET lands in whatever $HOME the process happens to have and survives across restarts on any non-ephemeral disk — long after the in-memory secret of the same name (recreated each activation) gets re-added, which is exactly what produces DuckDB's "secret occurs in multiple storage backends" ambiguity errors. Pinning it under DataDir makes the location deterministic and, crucially, wipeable on worker recycle (see duckdbservice.Warmup).

func SendInitialParams 

func SendInitialParams(cc *clientConn)

SendInitialParams sends the initial parameter status messages and backend key data.

func SetCatalogUseRewrite 

func SetCatalogUseRewrite(cc *clientConn, enabled bool)

SetCatalogUseRewrite records whether this session should expand a bare `USE ducklake` into its reliable two-part target. This is not masking — the catalog name is real; it only works around DuckDB's bare-catalog `USE` resolution.

func SetConnectionDatabase 

func SetConnectionDatabase(cc *clientConn, database string)

SetConnectionDatabase updates the PostgreSQL-visible database name for a control-plane connection after the fact. The eager connect path knows the resolved catalog before it builds the connection; the lazily-activated path learns it only when the session is created, so the activator restamps it.

func SetConnectionExploratory 

func SetConnectionExploratory(cc *clientConn, switcher WorkerSwitcher)

SetConnectionExploratory marks a control-plane connection as starting on the exploratory small worker and installs the switcher used to escalate it. Call before RunMessageLoop; the switcher runs on the message-loop goroutine.

func SetConnectionIdleTimeout 

func SetConnectionIdleTimeout(cc *clientConn, timeout time.Duration)

SetConnectionIdleTimeout applies a previously validated connect-time client idle-timeout request. The control plane calls this before RunMessageLoop.

func SetConnectionPhysicalCatalog 

func SetConnectionPhysicalCatalog(cc *clientConn, catalog string)

SetConnectionPhysicalCatalog records the resolved DuckDB catalog for control-plane proxy connections. The PostgreSQL-visible database remains on clientConn.database; this value is used only for execution/transpiler policy.

func SetConnectionTeamID 

func SetConnectionTeamID(cc *clientConn, teamID int64)

SetConnectionTeamID records the PostHog Team.id this connection is attributed to for product analytics (query_initiated / query_completed / query_failed), giving those per-org events a PostHog-native key. Resolved from the config snapshot at setup (the connecting user's team, else the org's oldest team); 0 when unknown or non-multitenant. Constant for the connection's life.

func SetConnectionWorkerSize 

func SetConnectionWorkerSize(cc *clientConn, millicores, mib int64)

SetConnectionWorkerSize records the provisioned worker pod size (in milli-units) on a control-plane connection for compute-usage billing. millicores == 0 means the size is unknown (non-remote / standalone) and metering is skipped. Constant for the connection's life, except that tier escalation may raise it once (largest size wins) from the message-loop goroutine — the same goroutine that reads it at teardown.

func SetConnectionWorkerTTLControl 

func SetConnectionWorkerTTLControl(cc *clientConn, ctl *WorkerTTLControl)

SetConnectionWorkerTTLControl installs the control-plane capability behind the `duckgres.worker_ttl` session GUC on a remote/k8s connection. Call before RunMessageLoop; the hooks run on the message-loop goroutine (the same one that handles SET/SHOW and tier escalation), so they are single-threaded with statement handling. Connections without it (standalone, process backend) get session-state-only SET/SHOW.

func SetPassthrough 

func SetPassthrough(cc *clientConn, enabled bool)

SetPassthrough flips this session into passthrough mode (bypasses the SQL transpiler + pg_catalog). The control plane resolves the per-org flag from the config store after auth and calls this before the message loop starts. Single-tenant mode keeps using server.Config.PassthroughUsers and never calls this.

func SetPendingS3CacheOption 

func SetPendingS3CacheOption(cc *clientConn, raw string)

SetPendingS3CacheOption parks a connect-time `-c duckgres.s3_cache=...` startup option (already validated with ValidateS3CacheOption) on a lazily activated connection. The option cannot be applied at connect — there is no worker to swap the S3 transport on — so ensureSessionActive applies it right after the activator installs the executor. A failure there fails the activation, which is connection-fatal, matching the eager path's refusal.

Do NOT reach for ApplyConnectionS3CacheOption from inside an activator: at that point the executor is still nil, the worker swap silently no-ops, and the session flag flips anyway.

func SetProcessVersion 

func SetProcessVersion(v string)

SetProcessVersion sets the version string for this process. Called from main().

func SetProgressFn 

func SetProgressFn(s *Server, fn func(pid int32) (pct float64, rows, totalRows uint64, stalled bool))

SetProgressFn sets the progress lookup function on a Server. Used by the control plane to provide cached query progress from worker health checks.

func SetQueryAccessPolicy 

func SetQueryAccessPolicy(cc *clientConn, policy *QueryAccessPolicy)

SetQueryAccessPolicy binds a fail-closed project policy before the message loop starts and prevents scoped sessions from bypassing the SQL layer. nil keeps root and internal users unrestricted.

func SetQueryLogSink 

func SetQueryLogSink(s *Server, sink QueryLogSink)

SetQueryLogSink sets the active query-log sink on a Server.

func SetQueryLogger 

func SetQueryLogger(s *Server, ql *QueryLogger)

SetQueryLogger sets the query logger on a Server. Used by the control plane to attach a query logger to the minimal server after creation.

func SetSessionActivator 

func SetSessionActivator(cc *clientConn, a SessionActivator)

SetSessionActivator installs the lazy first-acquisition hook on a control-plane connection created WITHOUT a worker (exploratory tier only). Call before RunMessageLoop; the activator runs on the message-loop goroutine. The connection must have been built with a nil executor — SetSessionActivator on a connection that already has one is inert by construction, since ensureSessionActive never replaces a live executor. See SessionActivator.

func SetUserSecretManager 

func SetUserSecretManager(s *Server, mgr UserSecretManager)

SetUserSecretManager installs the per-user persistent secret manager on a Server. Used by the multitenant control plane after the config store is up. Must be called before the server starts accepting connections.

func StartCredentialRefresh 

func StartCredentialRefresh(execer sqlExecer, dlCfg DuckLakeConfig, isTxActive ...func() bool) func()

StartCredentialRefresh starts a background goroutine that periodically refreshes S3 credentials for long-lived DuckDB connections using the credential_chain provider. This prevents credential expiration when running on EC2 with IAM instance roles, STS assume-role, or other temporary credential sources.

The execer parameter accepts either *sql.DB (standalone mode) or *sql.Conn (worker mode where the pool's only connection is pinned by the session).

The optional isTxActive callback reports whether the caller currently has an active user transaction on this connection. When provided and returning false, aborted transaction errors are auto-recovered by issuing ROLLBACK and retrying once. When omitted (or returning true), automatic rollback is skipped to avoid rolling back caller-owned transactions.

Note: ExecContext serializes behind any running query (pool contention for *sql.DB, internal mutex for *sql.Conn). This means credentials are refreshed between queries, not during them. A query that runs longer than the credential TTL (~6h for instance roles) could still fail if DuckDB makes S3 requests with stale cached credentials.

Returns a stop function that cancels the refresh goroutine. The caller must call the stop function when the connection is closed to prevent goroutine leaks. If credential refresh is not needed (static credentials, no S3, etc.), returns a no-op.

func ValidateClientIdleTimeoutOption 

func ValidateClientIdleTimeoutOption(raw string, max time.Duration) (time.Duration, error)

ValidateClientIdleTimeoutOption parses a client-requested idle timeout and enforces the operator-configured maximum. A non-positive maximum disables the feature. Client values must be positive: clients cannot disable idle reaping and retain a worker indefinitely.

func ValidateS3CacheOption 

func ValidateS3CacheOption(raw string) error

ValidateS3CacheOption validates a `-c duckgres.s3_cache=…` startup-option value without applying it. The control plane calls this BEFORE acquiring a worker so a bad option is rejected (FATAL 22023) without spending a spawn.

func WriteAuthCleartextPassword 

func WriteAuthCleartextPassword(w io.Writer) error

func WriteAuthOK 

func WriteAuthOK(w io.Writer) error

func WriteBackendKeyData 

func WriteBackendKeyData(w io.Writer, pid, secretKey int32) error

func WriteErrorResponse 

func WriteErrorResponse(w io.Writer, severity, code, message string) error

func WriteParameterStatus 

func WriteParameterStatus(w io.Writer, name, value string) error

func WriteReadyForQuery 

func WriteReadyForQuery(w io.Writer, txStatus byte) error

Types

type ACMEDNSManager 

type ACMEDNSManager = tlscert.ACMEDNSManager

type ACMEManager 

type ACMEManager = tlscert.ACMEManager

type BackendKey 

type BackendKey = wire.BackendKey

BackendKey moved to server/wire. Alias kept for back-compat with the dozens of references to server.BackendKey across this package and the control plane.

type ChildConfig 

type ChildConfig struct {
	// Connection info (RemoteAddr is known at spawn time; Username/Database are read after TLS)
	RemoteAddr string `json:"remote_addr"`

	// Server config
	DataDir              string   `json:"data_dir"`
	Extensions           []string `json:"extensions"`
	IdleTimeout          int64    `json:"idle_timeout"`            // nanoseconds
	ClientIdleTimeoutMax int64    `json:"client_idle_timeout_max"` // nanoseconds

	// TLS config
	TLSCertFile string `json:"tls_cert_file"`
	TLSKeyFile  string `json:"tls_key_file"`

	// DuckLake config
	DuckLake DuckLakeConfig `json:"ducklake"`

	// Authentication - map of username -> password
	// Child will look up after reading username from startup message
	Users map[string]string `json:"users"`

	// Backend key (pre-generated by parent for cancel request routing)
	// BackendPid is set to child's actual PID after fork
	BackendSecretKey int32 `json:"backend_secret_key"`

	// ServerStartTime is the parent server's start time (Unix nanoseconds).
	// Used to distinguish server uptime from child process uptime.
	ServerStartTime int64 `json:"server_start_time"`

	// ServerVersion is the parent server's version string.
	// Used to distinguish control_plane_version() from worker_version().
	ServerVersion string `json:"server_version,omitempty"`
}

ChildConfig contains all configuration needed by a child worker process. It is passed from parent to child via the DUCKGRES_CHILD_CONFIG env var as JSON.

type ChildProcess 

type ChildProcess struct {
	PID        int
	Cmd        *exec.Cmd
	Username   string
	RemoteAddr string
	BackendKey BackendKey
	StartTime  time.Time
	// contains filtered or unexported fields
}

ChildProcess represents a spawned child worker process

type ChildTracker 

type ChildTracker struct {
	// contains filtered or unexported fields
}

ChildTracker manages spawned child worker processes

func NewChildTracker 

func NewChildTracker() *ChildTracker

NewChildTracker creates a new child tracker

func (*ChildTracker) Add 

func (ct *ChildTracker) Add(child *ChildProcess)

Add registers a new child process

func (*ChildTracker) Count 

func (ct *ChildTracker) Count() int

Count returns the number of active child processes

func (*ChildTracker) FindByBackendKey 

func (ct *ChildTracker) FindByBackendKey(key BackendKey) *ChildProcess

FindByBackendKey finds a child process by its backend key (for cancel requests)

func (*ChildTracker) Get 

func (ct *ChildTracker) Get(pid int) *ChildProcess

Get returns a child process by PID

func (*ChildTracker) Remove 

func (ct *ChildTracker) Remove(pid int) *ChildProcess

Remove unregisters a child process by PID

func (*ChildTracker) SignalAll 

func (ct *ChildTracker) SignalAll(sig syscall.Signal)

SignalAll sends a signal to all child processes

func (*ChildTracker) WaitAll 

func (ct *ChildTracker) WaitAll() <-chan struct{}

WaitAll returns a channel that is closed when all children have exited. Caller should call this after SignalAll to wait for graceful shutdown.

NOTE: This method creates a new goroutine each time it's called. It captures a snapshot of current children at call time - children added after the call won't be waited on. For typical shutdown scenarios, call this once after SignalAll.

type ColumnTyper 

type ColumnTyper = sqlcore.ColumnTyper

The SQL/result interfaces moved to server/sqlcore so the Flight client and other duckdb-free callers can implement them without importing server. The aliases below preserve the old server.X spellings for the dozens of references inside this package and elsewhere.

type Config 

type Config struct {
	Host    string
	Port    int
	DataDir string
	Users   map[string]string // username -> password

	// TLS configuration (required unless ACME is configured)
	TLSCertFile string // Path to TLS certificate file
	TLSKeyFile  string // Path to TLS private key file

	// ACME/Let's Encrypt configuration (alternative to static TLS cert/key)
	ACMEDomain   string // Domain for ACME certificate (e.g., "decisive-mongoose-wine.us.duckgres.com")
	ACMEEmail    string // Contact email for Let's Encrypt notifications
	ACMECacheDir string // Directory for cached certificates (default: "./certs/acme")

	// ACME DNS-01 challenge configuration (for private/internal interfaces)
	// When ACMEDNSProvider is set, DNS-01 challenges are used instead of HTTP-01.
	// This allows certificate issuance for hosts without public port 80 access.
	ACMEDNSProvider string // DNS provider for ACME DNS-01 challenges (currently only "route53")
	ACMEDNSZoneID   string // Route53 hosted zone ID for DNS-01 challenges

	// Rate limiting configuration
	RateLimit RateLimitConfig

	// Extensions to load on database initialization
	Extensions []string

	// DuckLake configuration
	DuckLake DuckLakeConfig

	// AlwaysDuckLake forces the SQL transpiler into DuckLake mode for every
	// session even when the global DuckLake.MetadataStore is empty. The
	// multitenant control plane sets this because metadata stores are
	// per-org (loaded from configstore), so the global field stays empty
	// even though every worker is DuckLake-backed.
	AlwaysDuckLake bool

	// Graceful shutdown timeout (default: 30s)
	ShutdownTimeout time.Duration

	// IdleTimeout is the maximum time a connection can be idle before being closed.
	// This prevents accumulation of zombie connections from clients that disconnect
	// uncleanly. Default: 24 hours. Set to a negative value (e.g., -1) to disable.
	IdleTimeout time.Duration

	// ClientIdleTimeoutMax is the largest timeout a client may request through
	// the connect-time duckgres.idle_timeout option. A non-positive value disables
	// client overrides. This must remain bounded because idle control-plane
	// sessions retain workers and admission capacity.
	ClientIdleTimeoutMax time.Duration

	// SessionInitTimeout bounds startup metadata initialization and catalog probes.
	// Default: 10 seconds.
	SessionInitTimeout time.Duration

	// FilePersistence stores DuckDB data in <DataDir>/<username>.duckdb instead of :memory:.
	// DuckDB memory-maps the file and serves queries from RAM, so performance is similar
	// to in-memory mode while data persists across connections and restarts.
	FilePersistence bool

	// ProcessIsolation enables spawning each client connection in a separate OS process.
	// This prevents DuckDB C++ crashes from taking down the entire server.
	// When enabled, rate limiting and cancel requests are handled by the parent process,
	// while TLS, authentication, and query execution happen in child processes.
	ProcessIsolation bool

	// PinSecretDirectory pins DuckDB's persistent-secret directory under DataDir
	// (<DataDir>/secrets) instead of DuckDB's $HOME default. Set for worker
	// processes (see duckdbservice.OpenDuckDBPair): it makes the persisted-secret
	// location deterministic and wipeable on recycle, and — by redirecting away
	// from the $HOME default — stops stale secrets in the old location from being
	// loaded and colliding with the in-memory secrets re-created at activation.
	// Left false for standalone, so upgrading doesn't relocate an existing
	// standalone user's persistent secrets.
	//
	// Note: we deliberately do NOT also set allow_persistent_secrets=false.
	// Disabling persistent secrets unregisters DuckDB's local_file secret
	// storage backend, which the DuckLake ATTACH path depends on ("Unknown
	// secret storage found: 'local_file'") — so the directory pinning plus the
	// recycle wipe is how workers stay clean.
	PinSecretDirectory bool

	// UserSecrets persists per-user CREATE PERSISTENT SECRET statements
	// across sessions and worker pods. Set by the multitenant control plane
	// (remote backend, config-store-backed); nil everywhere else, in which
	// case secret DDL passes through to DuckDB untouched.
	UserSecrets UserSecretManager

	// MemoryLimit is the DuckDB memory_limit per session (e.g., "4GB").
	// If empty, auto-detected from system memory.
	MemoryLimit string

	// Threads is the DuckDB threads per session.
	// If zero, defaults to 2.5x runtime.NumCPU(), rounded up.
	Threads int

	// MemoryBudget is the total memory available for all DuckDB sessions (e.g., "24GB").
	// Used in control-plane mode for dynamic per-session memory allocation.
	// If empty, defaults to 75% of system RAM.
	MemoryBudget string

	// MemoryRebalance enables dynamic per-connection memory reallocation in control-plane mode.
	// When enabled, the memory budget is redistributed across all active sessions on every
	// connect/disconnect. When disabled (default), each session gets a static allocation
	// of budget/max_workers at creation time.
	MemoryRebalance bool

	// PassthroughUsers are users that bypass the SQL transpiler and pg_catalog initialization.
	// Queries from these users go directly to DuckDB without any PostgreSQL compatibility layer.
	PassthroughUsers map[string]bool

	// QueryLog configures query-log collection and flushing.
	QueryLog QueryLogConfig

	// DisableParquetPrefetching turns off DuckDB's parquet prefetch, which
	// coalesces reads across non-projected columns on remote files. Set for
	// deployments whose cache proxy runs in block-aligned mode, where
	// prefetch's coalesced reads cause byte amplification the proxy can't
	// avoid. See applyParquetPrefetchPolicy / parquetPrefetchPolicyStatements.
	DisableParquetPrefetching bool
}

type ConnDetail 

type ConnDetail struct {
	PID             int32
	OrgID           string
	Username        string
	Database        string
	ApplicationName string
	ClientAddr      string
	ClientPort      int32
	WorkerID        int
	WorkerPod       string
	State           string // active | idle | idle in transaction | idle in transaction (aborted)
	Query           string // redacted current/last query ("" when idle)
	BackendStart    time.Time
	QueryStart      time.Time // zero when no query is in flight
}

ConnDetail is a redacted snapshot of one live client connection, for the admin live-query detail view. Query is the ALREADY-redacted current/last query (usersecrets.RedactForLog, same as pg_stat_activity) — callers must never expose raw SQL here.

type ConnLiveSummary 

type ConnLiveSummary struct {
	State      string    // active | idle | idle in transaction | idle in transaction (aborted)
	QueryStart time.Time // zero when no query is in flight
}

ConnLiveSummary is the per-connection live state the admin Live list needs: the pg_stat_activity-style State and the current-query start. State is "active" when a query is in flight; anything else means no in-flight query.

type CopyFromOptions 

type CopyFromOptions struct {
	TableName  string
	ColumnList string // Empty string or "(col1, col2, ...)"
	Delimiter  string
	HasHeader  bool
	NullString string
	Quote      string // Quote character (default " for CSV)
	Escape     string // Escape character (default same as Quote)
	IsBinary   bool   // True if FORMAT binary
}

CopyFromOptions contains parsed options from a COPY FROM STDIN command

func ParseCopyFromOptions 

func ParseCopyFromOptions(query string) (*CopyFromOptions, error)

ParseCopyFromOptions extracts options from a COPY FROM STDIN command

type CopyToOptions 

type CopyToOptions struct {
	Source    string // Table name or (SELECT query)
	Delimiter string
	HasHeader bool
	IsQuery   bool // True if Source is a query in parentheses
}

CopyToOptions contains parsed options from a COPY TO STDOUT command

func ParseCopyToOptions 

func ParseCopyToOptions(query string) (*CopyToOptions, error)

ParseCopyToOptions extracts options from a COPY TO STDOUT command

type DuckDBAppendFunc 

type DuckDBAppendFunc func(rawConn sqlcore.RawConn, parts []string, rows [][]any) (int, error)

DuckDBAppendFunc bulk-inserts rows into a DuckDB table using the duckdb-go Appender API. The implementation calls rawConn.Raw to extract the underlying driver.Conn and then drives duckdb.NewAppender* against it.

The signature lives in the server package (rather than in duckdbservice) because the COPY codepath in clientConn dispatches through it; the duckdbservice package registers the actual implementation at init time via RegisterDuckDBAppender, which keeps server/conn_copy.go itself free of any duckdb-go imports.

parts is the result of splitQualifiedName(tableName) — a 1, 2, or 3 element slice of catalog/schema/table parts (preserving the precise args the original switch passed to NewAppender* / NewAppenderFromConn).

type DuckLakeCheckpointer 

type DuckLakeCheckpointer struct {
	// contains filtered or unexported fields
}

DuckLakeCheckpointer runs DuckLake CHECKPOINT on a schedule. CHECKPOINT performs full catalog maintenance: expires snapshots, merges adjacent files, rewrites data files, and cleans up orphaned files.

func NewDuckLakeCheckpointer 

func NewDuckLakeCheckpointer(cfg Config) (*DuckLakeCheckpointer, error)

NewDuckLakeCheckpointer opens a dedicated DuckDB connection, attaches DuckLake, and starts a background goroutine that runs CHECKPOINT on the configured interval.

func (*DuckLakeCheckpointer) Stop 

func (c *DuckLakeCheckpointer) Stop()

Stop shuts down the checkpoint scheduler, waits for any in-progress checkpoint to finish, and closes the database connection.

type DuckLakeConfig 

type DuckLakeConfig = ducklake.Config

DuckLakeConfig is an alias for ducklake.Config retained so the dozens of references to server.DuckLakeConfig across this package and others continue to compile after the migration code moved to server/ducklake. New code should import server/ducklake and use ducklake.Config directly.

type ExecResult 

type ExecResult = sqlcore.ExecResult

The SQL/result interfaces moved to server/sqlcore so the Flight client and other duckdb-free callers can implement them without importing server. The aliases below preserve the old server.X spellings for the dozens of references inside this package and elsewhere.

type LocalExecutor 

type LocalExecutor struct {
	// contains filtered or unexported fields
}

LocalExecutor wraps *sql.DB to implement QueryExecutor for local DuckDB access.

func NewLocalExecutor 

func NewLocalExecutor(db *sql.DB) *LocalExecutor

NewLocalExecutor creates a new LocalExecutor wrapping the given *sql.DB.

func (*LocalExecutor) Close 

func (e *LocalExecutor) Close() error

func (*LocalExecutor) ConnContext 

func (e *LocalExecutor) ConnContext(ctx context.Context) (RawConn, error)

func (*LocalExecutor) DB 

func (e *LocalExecutor) DB() *sql.DB

DB returns the underlying *sql.DB (for credential refresh and other direct access).

func (*LocalExecutor) Exec 

func (e *LocalExecutor) Exec(query string, args ...any) (ExecResult, error)

func (*LocalExecutor) ExecContext 

func (e *LocalExecutor) ExecContext(ctx context.Context, query string, args ...any) (ExecResult, error)

func (*LocalExecutor) LastProfilingOutput 

func (e *LocalExecutor) LastProfilingOutput() string

func (*LocalExecutor) PingContext 

func (e *LocalExecutor) PingContext(ctx context.Context) error

func (*LocalExecutor) Query 

func (e *LocalExecutor) Query(query string, args ...any) (RowSet, error)

func (*LocalExecutor) QueryContext 

func (e *LocalExecutor) QueryContext(ctx context.Context, query string, args ...any) (RowSet, error)

type LocalRowSet 

type LocalRowSet struct {
	// contains filtered or unexported fields
}

LocalRowSet wraps *sql.Rows to implement RowSet.

func (*LocalRowSet) Close 

func (r *LocalRowSet) Close() error

func (*LocalRowSet) ColumnTypes 

func (r *LocalRowSet) ColumnTypes() ([]ColumnTyper, error)

func (*LocalRowSet) Columns 

func (r *LocalRowSet) Columns() ([]string, error)

func (*LocalRowSet) Err 

func (r *LocalRowSet) Err() error

func (*LocalRowSet) Next 

func (r *LocalRowSet) Next() bool

func (*LocalRowSet) Scan 

func (r *LocalRowSet) Scan(dest ...any) error

type PinnedExecutor 

type PinnedExecutor struct {
	// contains filtered or unexported fields
}

PinnedExecutor wraps a pinned *sql.Conn from a shared *sql.DB pool to implement QueryExecutor for file-persistence mode.

func NewPinnedExecutor 

func NewPinnedExecutor(conn *sql.Conn, db *sql.DB) *PinnedExecutor

func (*PinnedExecutor) Close 

func (e *PinnedExecutor) Close() error

Close returns the pinned connection to the pool; it does not close the underlying DB.

func (*PinnedExecutor) ConnContext 

func (e *PinnedExecutor) ConnContext(ctx context.Context) (RawConn, error)

func (*PinnedExecutor) DB 

func (e *PinnedExecutor) DB() *sql.DB

DB returns the underlying *sql.DB (for credential refresh and other direct access).

func (*PinnedExecutor) Exec 

func (e *PinnedExecutor) Exec(query string, args ...any) (ExecResult, error)

func (*PinnedExecutor) ExecContext 

func (e *PinnedExecutor) ExecContext(ctx context.Context, query string, args ...any) (ExecResult, error)

func (*PinnedExecutor) LastProfilingOutput 

func (e *PinnedExecutor) LastProfilingOutput() string

func (*PinnedExecutor) PingContext 

func (e *PinnedExecutor) PingContext(ctx context.Context) error

func (*PinnedExecutor) Query 

func (e *PinnedExecutor) Query(query string, args ...any) (RowSet, error)

func (*PinnedExecutor) QueryContext 

func (e *PinnedExecutor) QueryContext(ctx context.Context, query string, args ...any) (RowSet, error)

type QueryAccessError 

type QueryAccessError struct {
	Reason string
}

QueryAccessError is returned when a project-scoped principal attempts an operation or relation outside its policy.

func (*QueryAccessError) Error 

func (e *QueryAccessError) Error() string

type QueryAccessPolicy 

type QueryAccessPolicy struct {
	ReadOnly         bool
	AllowedSchemas   []string
	AllowedRelations []string
}

QueryAccessPolicy is a fail-closed SQL policy for a project-scoped user. A nil policy means the internal/root principal and remains unrestricted.

ReadOnly separates the two scoped modes: a project reader (ReadOnly) may only SELECT, while a project user may additionally run DML and DDL — but only where every target resolves into AllowedSchemas/AllowedRelations. Neither can reach a relation outside the project, and neither gets the native-DuckDB escape hatches (arbitrary file/URL readers, secrets, settings).

func NormalizeQueryAccessPolicy 

func NormalizeQueryAccessPolicy(policy QueryAccessPolicy) QueryAccessPolicy

NormalizeQueryAccessPolicy makes policy snapshots deterministic for tests, logging, and cross-protocol conversion.

func (*QueryAccessPolicy) Authorize 

func (p *QueryAccessPolicy) Authorize(query string) error

Authorize verifies that every persistent relation a query touches is owned by the project, and that the statement itself is within the principal's mode: a project reader may only read, a project user may also write. Native DuckDB fallback is deliberately unavailable to scoped users because an unparsed statement cannot be authorized safely.

type QueryExecutor 

type QueryExecutor = sqlcore.QueryExecutor

The SQL/result interfaces moved to server/sqlcore so the Flight client and other duckdb-free callers can implement them without importing server. The aliases below preserve the old server.X spellings for the dozens of references inside this package and elsewhere.

type QueryLogConfig 

type QueryLogConfig struct {
	Enabled       bool
	FlushInterval time.Duration
	BatchSize     int
	// StartEvents selects which statements emit a QueryStart event. Empty
	// behaves as the default ("data") so a config that predates this field
	// keeps working.
	StartEvents QueryStartEvents
	// Metadata enables per-statement extraction of the relations, columns,
	// functions, and access classes a statement touches (server/querymeta).
	// It costs one parse per distinct statement text, memoized.
	Metadata bool
}

QueryLogConfig configures the query log feature.

type QueryLogEntry 

type QueryLogEntry = wire.QueryLogEntry

QueryLogEntry represents a single entry in the query log.

The concrete shape lives in server/wire so the DuckDB-free Flight client can forward entries to worker pods without importing the full server package.

type QueryLogSink 

type QueryLogSink interface {
	Log(QueryLogEntry)
	StopContext(context.Context) error
}

QueryLogSink accepts query log entries and drains them during shutdown.

func NewQueryLogSink 

func NewQueryLogSink(cfg Config) (QueryLogSink, error)

NewQueryLogSink creates the native Postgres-backed query-log sink.

type QueryLogger 

type QueryLogger struct {
	// contains filtered or unexported fields
}

QueryLogger batches query log entries and writes them to durable storage.

func NewPostgresQueryLoggerContext 

func NewPostgresQueryLoggerContext(ctx context.Context, dlCfg DuckLakeConfig, cfg QueryLogConfig) (*QueryLogger, error)

NewPostgresQueryLoggerContext creates a worker-local query-log sink backed by the tenant metadata Postgres database. The caller owns query-log routing; this sink only owns a small native pgx pool and batched append-only inserts.

func (*QueryLogger) Log 

func (ql *QueryLogger) Log(entry QueryLogEntry)

Log sends an entry to the query log. Non-blocking; drops if channel is full.

func (*QueryLogger) Stop 

func (ql *QueryLogger) Stop()

Stop drains remaining entries and shuts down the flush goroutine.

func (*QueryLogger) StopContext 

func (ql *QueryLogger) StopContext(ctx context.Context) error

StopContext drains remaining entries until ctx expires. If the deadline is reached, it cancels in-flight storage work; loggers that own a DB handle also close it to unblock shutdown.

type QueryStartEvents 

type QueryStartEvents string

QueryStartEvents selects which statements get a QueryStart event.

QueryStart roughly doubles query-log row count, and the statements that benefit least from it are the ones clients emit most: BEGIN/COMMIT, SET, and driver catalog introspection never hang and never need in-flight visibility. Terminal events stay universal regardless of this setting, so nothing disappears from the log — cheap statements simply have no paired start row. 

const (
	// QueryStartEventsData logs a start event for statements that touch data or
	// change schema, and skips transaction control, session settings, and
	// catalog introspection. The default.
	QueryStartEventsData QueryStartEvents = "data"
	// QueryStartEventsAll logs a start event for every statement.
	QueryStartEventsAll QueryStartEvents = "all"
	// QueryStartEventsOff disables start events.
	QueryStartEventsOff QueryStartEvents = "off"
)

func NormalizeQueryStartEvents 

func NormalizeQueryStartEvents(value string) QueryStartEvents

NormalizeQueryStartEvents maps a configured value onto the closed set, falling back to the default for anything unrecognized. Query-log volume policy must never fail a boot over a typo.

type RateLimitConfig 

type RateLimitConfig = auth.RateLimitConfig

type RateLimiter 

type RateLimiter = auth.RateLimiter

type RawConn 

type RawConn = sqlcore.RawConn

The SQL/result interfaces moved to server/sqlcore so the Flight client and other duckdb-free callers can implement them without importing server. The aliases below preserve the old server.X spellings for the dozens of references inside this package and elsewhere.

type RecentError 

type RecentError struct {
	Time       time.Time `json:"time"`
	OrgID      string    `json:"org"`
	Username   string    `json:"user"`
	PID        int32     `json:"pid"`
	WorkerID   int       `json:"worker_id"`
	WorkerPod  string    `json:"worker_pod"`
	SQLState   string    `json:"sqlstate"` // classifyErrorCode
	Category   string    `json:"category"` // user | system | conflict | metadata_connection_lost
	Message    string    `json:"message"`  // REDACTED
	Query      string    `json:"query"`    // REDACTED
	ClientAddr string    `json:"client_addr"`
	TraceID    string    `json:"trace_id"`
}

RecentError is a redacted snapshot of one failed query, for the admin Errors page. Message and Query are ALREADY redacted (usersecrets.RedactErrorForLog / RedactForLog) at the capture site — this struct must never carry raw SQL or a CREATE SECRET credential, since the Errors page surfaces both fields.

type RowSet 

type RowSet = sqlcore.RowSet

The SQL/result interfaces moved to server/sqlcore so the Flight client and other duckdb-free callers can implement them without importing server. The aliases below preserve the old server.X spellings for the dozens of references inside this package and elsewhere.

type S3CacheControl 

type S3CacheControl interface {
	SetS3CacheEnabled(ctx context.Context, enabled bool) error
}

S3CacheControl is the optional session-executor capability behind the `duckgres.s3_cache` GUC: swapping the worker-side S3 secret between the cache-proxy transport (enabled, the default) and the org's native HTTPS transport (bypassed). flightclient.FlightExecutor implements it for remote/k8s workers. Executors that don't implement it (standalone, in-process) get session-state-only SET/SHOW, which is correct because those deployments never route S3 traffic through a cache proxy — there is nothing to bypass.

type S3CacheModeControl 

type S3CacheModeControl interface {
	SetS3CacheMode(ctx context.Context, mode string) error
}

S3CacheModeControl extends S3CacheControl for workers that can distinguish the cache-on, cache-off, and proxy-passthrough transports. Passthrough keeps cache-proxy on the request path while ensuring it neither reads nor fills its cache.

type Server 

type Server struct {
	// contains filtered or unexported fields
}

func New 

func New(cfg Config) (*Server, error)

func (*Server) ActiveConnections 

func (s *Server) ActiveConnections() int64

ActiveConnections returns the number of active connections

func (*Server) CancelQuery 

func (s *Server) CancelQuery(key BackendKey) bool

CancelQuery cancels a running query by its backend key. Returns true if a query was found and cancelled, false otherwise.

func (*Server) CancelQueryBySignal 

func (s *Server) CancelQueryBySignal(key BackendKey) bool

CancelQueryBySignal sends SIGUSR1 to a child process to cancel its current query. Returns true if the signal was sent successfully.

func (*Server) Close 

func (s *Server) Close() error

func (*Server) ConnDetailByPID 

func (s *Server) ConnDetailByPID(pid int32) (ConnDetail, bool)

ConnDetailByPID returns a redacted snapshot of the live connection for pid, or ok=false if no such connection is registered on this server. Used by the control-plane admin API to render the live-query detail view.

func (*Server) ConnDetailByWorkerID 

func (s *Server) ConnDetailByWorkerID(workerID int) (ConnDetail, bool)

ConnDetailByWorkerID returns a redacted snapshot of the live connection bound to the given control-plane worker id, or ok=false if none is registered here.

Worker ids are CLUSTER-UNIQUE (config-store issued) and there is exactly one session per worker, so this is the collision-free address for the admin live-query detail. PID is NOT safe: the CP allocates backend pids per-org (every org's SessionManager starts at 1000), so two orgs can share a pid and a lookup keyed by pid can return the wrong org's connection. A negative workerID (standalone / transient describe conns) never matches.

func (*Server) ConnSummariesByWorkerID 

func (s *Server) ConnSummariesByWorkerID() map[int]ConnLiveSummary

ConnSummariesByWorkerID returns a one-pass snapshot of every live connection keyed by cluster-unique worker id: its state and current-query start. Used to attach running-query duration AND the active/idle state to the live list in a single lock acquisition (keyed on worker id, not the collision-prone per-org pid). Connections with no worker (standalone / transient) are omitted.

func (*Server) DrainOrgConnections 

func (s *Server) DrainOrgConnections(orgID string) int

DrainOrgConnections requests a clean close of every PostgreSQL connection for orgID at its next idle protocol boundary. Connections already blocked waiting for client input are woken immediately; executing queries are not cancelled and close after their next ReadyForQuery.

func (*Server) DrainUserConnections 

func (s *Server) DrainUserConnections(orgID, username string) int

DrainUserConnections requests a clean close of every PostgreSQL connection for one org user. It is used when a project reader's credentials or access policy changes so an established connection cannot retain stale access.

func (*Server) ListenAndServe 

func (s *Server) ListenAndServe() error

func (*Server) QueryLogger 

func (s *Server) QueryLogger() *QueryLogger

QueryLogger returns the server's query logger (may be nil).

func (*Server) RecentErrors 

func (s *Server) RecentErrors(limit int) []RecentError

RecentErrors returns up to limit of this server's most recent redacted query errors, newest first. Used by the control-plane admin Errors page (fanned out across replicas). limit <= 0 returns all retained.

func (*Server) RegisterQuery 

func (s *Server) RegisterQuery(key BackendKey, cancel context.CancelFunc)

RegisterQuery registers a cancel function for a backend key. This allows the query to be cancelled via a cancel request from another connection.

func (*Server) Shutdown 

func (s *Server) Shutdown(ctx context.Context) error

Shutdown performs a graceful shutdown with the given context

func (*Server) StopQueryLogging 

func (s *Server) StopQueryLogging(ctx context.Context) error

StopQueryLogging drains and stops the active query-log sink, if any.

func (*Server) UnregisterQuery 

func (s *Server) UnregisterQuery(key BackendKey)

UnregisterQuery removes the cancel function for a backend key. This should be called when a query completes (successfully or with error).

func (*Server) UserSecretManager 

func (s *Server) UserSecretManager() UserSecretManager

UserSecretManager returns the installed per-user persistent secret manager (nil when the feature is not configured).

type SessionAcquireError 

type SessionAcquireError struct {
	Code    string
	Message string
	Err     error
}

SessionAcquireError carries an ALREADY-CLASSIFIED session-acquisition failure from the control plane to the client. The control plane owns every sentinel involved (*WorkerCapacityExhaustedError, the vCPU admission rejection, draining, cancellation, catalog init) but reaches this package through a plain `error`, so it classifies the failure with the same logic the eager connect path uses (sessionCreationErrorResponse) and hands the result across in this type. Code is the SQLSTATE and Message is exactly what the client should see — NOT the wrapped internal error chain.

func (*SessionAcquireError) Error 

func (e *SessionAcquireError) Error() string

func (*SessionAcquireError) Unwrap 

func (e *SessionAcquireError) Unwrap() error

type SessionActivator 

type SessionActivator func(ctx context.Context, pinned bool) (exec QueryExecutor, workerID int, workerPod string, err error)

SessionActivator lazily acquires the connection's first worker/session. Installed by the control plane when the exploratory tier defers acquisition past connection startup, so a connection that never issues an engine-touching statement never spends a worker pod. Invoked on the message-loop goroutine by the first statement that needs an engine — the same goroutine that reads c.executor, so an activation can never race executor use or another activation.

pinned=true means the first statement is ALREADY a pinning one: the control plane then acquires the escalation-target (standard) profile directly and marks the connection off-tier (MarkConnectionPinned), instead of acquiring the small worker only to escalate off it one statement later.

type TypeInfo 

type TypeInfo struct {
	OID    int32
	Size   int16 // -1 for variable length
	Typmod int32 // -1 = no modifier; for NUMERIC: ((precision << 16) | scale) + 4
}

TypeInfo contains PostgreSQL type information

type UserSecretManager 

type UserSecretManager interface {
	// Ready reports whether secrets can be persisted (e.g. the encryption
	// key is configured). A non-nil error is shown to the client.
	Ready() error
	// PutSecret stores one statement. With ifNotExists set, an already-stored
	// name is left untouched (mirroring DuckDB's IF NOT EXISTS no-op on the
	// live session); otherwise any prior statement with the same name is
	// replaced. Called only after the statement executed successfully on the
	// live session.
	PutSecret(ctx context.Context, orgID, username, secretName, statement string, ifNotExists bool) error
	// DeleteSecret removes one stored secret, reporting whether it existed.
	DeleteSecret(ctx context.Context, orgID, username, secretName string) (existed bool, err error)
}

UserSecretManager persists per-user CREATE PERSISTENT SECRET statements so they survive across sessions and worker pods. Implemented by the multitenant control plane (backed by the config store); nil in standalone and process-worker modes, where secret DDL passes through untouched.

type ValueNormalizer 

type ValueNormalizer func(any) any

ValueNormalizer is a hook that converts driver-specific value types (e.g., duckdb.Interval, duckdb.Decimal) into the duckdb-free equivalents in arrowmap (IntervalValue, DecimalValue) so the binary-format encoders in types.go can handle them without importing the duckdb-go driver.

A normalizer must return the input unchanged when it doesn't recognize the type; the encode helpers fall back to AppendNull/return nil when the final value still isn't a recognized type.

type WorkerActivationPayload 

type WorkerActivationPayload = wire.WorkerActivationPayload

WorkerActivationPayload moved to server/wire so the control plane can use it without importing the rest of server. The alias preserves the existing server.WorkerActivationPayload spelling for current call sites.

type WorkerControlMetadata 

type WorkerControlMetadata = wire.WorkerControlMetadata

type WorkerCreateSessionPayload 

type WorkerCreateSessionPayload = wire.WorkerCreateSessionPayload

type WorkerDestroySessionPayload 

type WorkerDestroySessionPayload = wire.WorkerDestroySessionPayload

type WorkerHealthCheckPayload 

type WorkerHealthCheckPayload = wire.WorkerHealthCheckPayload

type WorkerQueryLogPayload 

type WorkerQueryLogPayload = wire.WorkerQueryLogPayload

type WorkerReleaseQueryHandlePayload 

type WorkerReleaseQueryHandlePayload = wire.WorkerReleaseQueryHandlePayload

type WorkerSetS3CachePayload 

type WorkerSetS3CachePayload = wire.WorkerSetS3CachePayload

type WorkerSwitcher 

type WorkerSwitcher func(ctx context.Context, reason string) (exec QueryExecutor, workerID int, workerPod string, err error)

WorkerSwitcher swaps a connection's backing worker/session: the control plane destroys the current (stateless, exploratory) session and creates one on a normal-size worker, returning the new executor + worker identity.

type WorkerTTLControl 

type WorkerTTLControl struct {
	// Baseline is the TTL resolved at connect time (startup GUC > org default
	// > deployment default > built-in 1m). RESET restores it on the worker,
	// and SHOW falls back to it when no worker is bound yet.
	Baseline time.Duration

	// Apply overrides the bound worker's hot-idle TTL, returning the value
	// actually applied (the control plane may clamp to WorkerMaxTTL). A
	// returned *transform.CodedError preserves its SQLSTATE to the client;
	// any other error surfaces as XX000.
	Apply func(ctx context.Context, ttl time.Duration) (applied time.Duration, err error)

	// Current reports the TTL the bound worker would park with NOW (ok=false
	// when no worker is bound — a lazily activated connection before its
	// first engine statement). It beats Baseline for SHOW because a reused
	// hot-idle worker can carry a previous request's TTL.
	Current func() (ttl time.Duration, ok bool)
}

WorkerTTLControl is the optional control-plane capability behind the `duckgres.worker_ttl` session GUC, installed on remote/k8s connections. It is how the connection layer reaches the bound worker's pool-side profile — the hot-idle TTL lives in the control plane's worker pool, not in the worker process, so unlike duckgres.s3_cache this is NOT an executor capability. Connections without it (standalone, process backend) get session-state-only SET/SHOW, which is correct because those deployments have no hot-idle worker TTL to override.

type WorkerWaitSessionIdlePayload 

type WorkerWaitSessionIdlePayload = wire.WorkerWaitSessionIdlePayload

Source Files

View all Source files

Directories

Path Synopsis
Package auth holds duckgres' connection rate-limiting and password validation policy.
 Package auth holds duckgres' connection rate-limiting and password validation policy.
Package ducklake holds DuckLake configuration, the metadata-store version migration check, and the SQL fragments needed to ATTACH a DuckLake catalog.
 Package ducklake holds DuckLake configuration, the metadata-store version migration check, and the SQL fragments needed to ATTACH a DuckLake catalog.
Package observe holds duckgres' OpenTelemetry tracing helpers, the connection-count gauge, and the per-query Prometheus metrics emitted from the trace path.
 Package observe holds duckgres' OpenTelemetry tracing helpers, the connection-count gauge, and the per-query Prometheus metrics emitted from the trace path.
Package pgbinary validates and normalizes PostgreSQL binary COPY streams before they are handed to DuckDB's postgres_scanner extension.
 Package pgbinary validates and normalizes PostgreSQL binary COPY streams before they are handed to DuckDB's postgres_scanner extension.
Package querymeta extracts what an inbound statement touches — catalogs, schemas, relations, columns, functions — and what class of access it is.
 Package querymeta extracts what an inbound statement touches — catalogs, schemas, relations, columns, functions — and what class of access it is.
Package sessionmeta installs session-local catalog/metadata overrides on a duckgres connection (current_database, pg_database, information_schema views) so they reflect the catalog the session defaults to on the PG wire.
 Package sessionmeta installs session-local catalog/metadata overrides on a duckgres connection (current_database, pg_database, information_schema views) so they reflect the catalog the session defaults to on the PG wire.
Package sqlcore holds the duckgres-internal SQL/result interfaces that span the wire-protocol/server layer and the Arrow Flight client.
 Package sqlcore holds the duckgres-internal SQL/result interfaces that span the wire-protocol/server layer and the Arrow Flight client.
Package sysinfo holds duckgres' system-memory detection helpers and the memory-limit string parser shared between the server, the control plane, and config resolution.
 Package sysinfo holds duckgres' system-memory detection helpers and the memory-limit string parser shared between the server, the control plane, and config resolution.
Package usersecrets implements the building blocks of the per-user persistent secret manager: classification of DuckDB secret DDL statements and authenticated encryption for storing them in the config store.
 Package usersecrets implements the building blocks of the per-user persistent secret manager: classification of DuckDB secret DDL statements and authenticated encryption for storing them in the config store.
Package wire holds duckgres wire-level types and helpers shared between the PG protocol layer and the control plane / worker RPC paths.
 Package wire holds duckgres wire-level types and helpers shared between the PG protocol layer and the control plane / worker RPC paths.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.