Documentation
¶
Index ¶
- Constants
- Variables
- type Config
- type DuckDB
- func (d *DuckDB) ClearHTTPCache()
- func (d *DuckDB) Close() error
- func (d *DuckDB) ConfigureS3(s3cfg *S3Config) error
- func (d *DuckDB) DB() *sql.DB
- func (d *DuckDB) Exec(query string, args ...interface{}) (sql.Result, error)
- func (d *DuckDB) Query(query string, args ...interface{}) (*sql.Rows, error)
- func (d *DuckDB) QueryContext(ctx context.Context, query string, args ...interface{}) (*sql.Rows, error)
- func (d *DuckDB) QueryWithProfile(query string) (*sql.Rows, *sql.Conn, *QueryProfile, error)
- func (d *DuckDB) QueryWithProfileContext(ctx context.Context, query string) (*sql.Rows, *sql.Conn, *QueryProfile, error)
- func (d *DuckDB) Stats() sql.DBStats
- type QueryCache
- type QueryProfile
- type S3Config
- type SQLTransformCache
Constants ¶
const DefaultQueryCacheMaxSize = DefaultSQLTransformCacheMaxSize
const DefaultSQLTransformCacheMaxSize = 10000
DefaultSQLTransformCacheMaxSize is the default maximum number of cached entries
const QueryCacheTTL = SQLTransformCacheTTL
Aliases for backwards compatibility
const SQLTransformCacheTTL = 60 * time.Second
SQLTransformCacheTTL is the default TTL for SQL transform cache entries (60 seconds)
Variables ¶
var ArrowEnabled bool
ArrowEnabled is set to true by duckdb_arrow.go init() when compiled with the duckdb_arrow tag.
Functions ¶
This section is empty.
Types ¶
type Config ¶
type Config struct {
MaxConnections int
MemoryLimit string
ThreadCount int
EnableWAL bool
// S3 configuration for httpfs extension
S3Region string
S3AccessKey string
S3SecretKey string
S3Endpoint string // Custom endpoint for MinIO or S3-compatible services
S3UseSSL bool
S3PathStyle bool // Use path-style addressing (required for MinIO)
// Azure Blob Storage configuration for azure extension
AzureAccountName string
AzureAccountKey string
AzureEndpoint string // Custom endpoint (optional)
// Query optimization configuration
EnableS3Cache bool // Enable S3 file caching via cache_httpfs extension
S3CacheSize int64 // Cache size in bytes
S3CacheTTLSeconds int // Cache entry TTL in seconds (default: 3600)
}
Config holds DuckDB configuration
type DuckDB ¶
type DuckDB struct {
// contains filtered or unexported fields
}
DuckDB manages DuckDB connections and query execution Note: No mutex is needed here because: 1. *sql.DB maintains its own connection pool with internal synchronization 2. DuckDB handles concurrent queries internally 3. Adding a mutex would only add overhead without safety benefits
func (*DuckDB) ClearHTTPCache ¶
func (d *DuckDB) ClearHTTPCache()
ClearHTTPCache clears DuckDB's cache_httpfs and parquet_metadata_cache. This should be called after compaction deletes files to prevent stale cache hits (glob results, file metadata, and data blocks pointing to deleted parquet files). Safe to call even if cache_httpfs is not loaded — the error is silently ignored.
func (*DuckDB) ConfigureS3 ¶
ConfigureS3 reconfigures DuckDB's S3 settings at runtime. This is useful when tiered storage uses different S3 credentials than the main storage. The httpfs extension must already be loaded.
func (*DuckDB) DB ¶
DB returns the underlying *sql.DB connection pool This is used for passing to components that need direct DB access (e.g., compaction)
func (*DuckDB) QueryContext ¶
func (d *DuckDB) QueryContext(ctx context.Context, query string, args ...interface{}) (*sql.Rows, error)
QueryContext executes a query with context support for timeout/cancellation
func (*DuckDB) QueryWithProfile ¶
QueryWithProfile executes a query and returns timing breakdown using DuckDB profiling This is used to measure parsing/planning overhead for optimization decisions
The caller MUST close both resources when done:
- rows.Close() — releases the result set
- conn.Close() — returns the pinned connection to the pool
func (*DuckDB) QueryWithProfileContext ¶
func (d *DuckDB) QueryWithProfileContext(ctx context.Context, query string) (*sql.Rows, *sql.Conn, *QueryProfile, error)
QueryWithProfileContext executes a query with context support for timeout/cancellation and returns timing breakdown using DuckDB profiling. All profiling PRAGMAs and the query are pinned to a single connection to avoid race conditions across the connection pool.
The caller MUST close both resources when done:
- rows.Close() — releases the result set
- conn.Close() — returns the pinned connection to the pool
type QueryCache ¶
type QueryCache = SQLTransformCache
QueryCache is an alias for SQLTransformCache (backwards compatibility)
type QueryProfile ¶
type QueryProfile struct {
TotalMs float64 `json:"total_ms"`
PlannerMs float64 `json:"planner_ms"`
ExecutionMs float64 `json:"execution_ms"`
RowsScanned uint64 `json:"rows_scanned"`
Latency float64 `json:"latency_ms"` // DuckDB reported latency
}
QueryProfile contains timing breakdown for a query execution
type S3Config ¶
type S3Config struct {
Region string
Endpoint string
AccessKey string
SecretKey string
UseSSL bool
PathStyle bool
}
S3Config holds S3 configuration for DuckDB httpfs extension
type SQLTransformCache ¶
type SQLTransformCache struct {
// contains filtered or unexported fields
}
SQLTransformCache provides a sharded TTL cache for SQL-to-storage-path transformations. This caches the result of converting table references (e.g., FROM mydb.cpu) to DuckDB read_parquet() calls (e.g., FROM read_parquet('./data/mydb/cpu/**/*.parquet')). This is NOT a query results cache or query plan cache - it only caches the regex-based string transformation that happens before DuckDB sees the query.
The cache is sharded into 16 buckets to reduce lock contention under concurrent load. Each shard has its own RWMutex, so concurrent queries hitting different shards don't block each other.
func NewQueryCache ¶
func NewQueryCache(ttl time.Duration, maxSize int) *SQLTransformCache
NewQueryCache creates a new query cache (alias for backwards compatibility)
func NewSQLTransformCache ¶
func NewSQLTransformCache(ttl time.Duration, maxSize int) *SQLTransformCache
NewSQLTransformCache creates a new SQL transform cache with the specified TTL and max size
func (*SQLTransformCache) Cleanup ¶
func (c *SQLTransformCache) Cleanup() int
Cleanup removes expired entries from the cache
func (*SQLTransformCache) Get ¶
func (c *SQLTransformCache) Get(sql string) (string, bool)
Get retrieves a cached transformed SQL if it exists and hasn't expired
func (*SQLTransformCache) Invalidate ¶
func (c *SQLTransformCache) Invalidate()
Invalidate removes all entries from the cache
func (*SQLTransformCache) Set ¶
func (c *SQLTransformCache) Set(sql, transformed string)
Set stores a transformed SQL in the cache
func (*SQLTransformCache) Size ¶
func (c *SQLTransformCache) Size() int
Size returns the current number of entries in the cache
func (*SQLTransformCache) Stats ¶
func (c *SQLTransformCache) Stats() map[string]interface{}
Stats returns cache statistics as a map
Source Files