Help us shape the future of Go! Take the 2026 Go Developer Survey

sqlconnect

package
v1.35.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 20, 2026 License: MIT Imports: 18 Imported by: 3

Documentation

Index

Constants

This section is empty.

Variables

View Source
var (
	ErrNotSupported         = errors.New("sqconnect: feature not supported")
	ErrDropOldTablePostCopy = errors.New("sqlconnect move table: dropping old table after copying its contents to the new table")
)

Functions

func QueryAsync

func QueryAsync[T any](ctx context.Context, db QueryDB, mapper RowMapper[T], query string, params ...any) (ch <-chan ValueOrError[T], leave func())

QueryAsync executes a query and returns a channel that will receive the results or an error, along with a function that the caller can use to leave the channel early. The channel will be closed when the query is done or when the context is canceled.

func QueryJSONAsync

func QueryJSONAsync(ctx context.Context, db JsonQueryDB, query string, params ...any) (ch <-chan ValueOrError[json.RawMessage], leave func())

QueryJSONAsync executes a query and returns a channel that will receive the results as json or an error, along with a function that the caller can use to leave the channel early. The channel will be closed when the query is done or when the context is canceled.

Non-finite floats (NaN, +Inf, -Inf) are emitted as JSON null, at any depth, because JSON cannot represent them and json.Marshal would otherwise fail the entire result set rather than the one offending cell. The substitution is silent and lossy in one specific way: in the emitted JSON a non-finite value is indistinguishable from a real SQL NULL, so a caller deriving null-rate style metrics from this output will count such rows as unpopulated. Use QueryJSONMapAsync, which does not perform the substitution, when the distinction matters.

func QueryJSONMapAsync

func QueryJSONMapAsync(ctx context.Context, db JsonQueryDB, query string, params ...any) (ch <-chan ValueOrError[map[string]any], leave func())

QueryJSONMapAsync executes a query and returns a channel that will receive the results as a map or an error, along with a function that the caller can use to leave the channel early. The channel will be closed when the query is done or when the context is canceled.

func RegisterDBFactory

func RegisterDBFactory(name string, factory DBFactory)

RegisterDBFactory registers a database factory. It replaces any option-aware factory previously registered under the same name.

func RegisterDBFactoryWithOptions added in v1.31.0

func RegisterDBFactoryWithOptions(name string, factory DBFactoryWithOptions)

RegisterDBFactoryWithOptions registers a database factory that accepts factory options.

func RegisterDialectFactory added in v1.23.0

func RegisterDialectFactory(name string, factory DialectFactory)

func SetEgressPolicy added in v1.34.0

func SetEgressPolicy(blockedCIDRs []string) error

SetEgressPolicy configures a process-wide list of CIDR ranges that warehouse connections must never reach (for example in-cluster service ranges), checked at dial time against the actual resolved address in addition to the always- blocked classes (loopback, link-local, metadata, ...). It is set once at startup. An empty or nil list clears it. Ranges are parsed with net.ParseCIDR; an invalid entry returns an error and the policy is left unchanged.

It is enforced only on connections opened through a guarded driver path: mysql, postgres, redshift over the Postgres/JDBC path, snowflake, databricks and trino. Other outbound paths are not covered and rely on the network layer instead — among them fabric (mssql, which has no guarded dialer), clickhouse (which has its own dial guard), the Redshift Data API (AWS SDK), bigquery's token_uri exchange, the SSH bastion dial, and OAuth/OIDC discovery and token requests. Treat it as defence in depth, not the sole egress control.

func WithValidationOptions added in v1.32.0

func WithValidationOptions(ctx context.Context, o ValidationOptions) context.Context

WithValidationOptions returns a context that carries the options.

Types

type CatalogAdmin added in v1.1.0

type CatalogAdmin interface {
	// CurrentCatalog returns the current catalog.
	// If this operation is not supported by the warehouse [ErrNotSupported] will be returned.
	CurrentCatalog(ctx context.Context) (CatalogRef, error)
	// ListCatalogs returns all available catalogs linked to the credentials. System catalogs are filtered out.
	// If this operation is not supported by the warehouse [ErrNotSupported] will be returned.
	ListCatalogs(ctx context.Context) ([]CatalogRef, error)
}

type CatalogRef added in v1.24.0

type CatalogRef struct {
	Name string `json:"name"` // the catalog/database name
}

CatalogRef provides a reference to a database catalog

func (CatalogRef) String added in v1.24.0

func (c CatalogRef) String() string

type ColumnRef

type ColumnRef struct {
	Name    string `json:"name"`
	Type    string `json:"type"`
	RawType string `json:"rawType"`
}

ColumnRef provides a reference to a table column

type DB

type DB interface {

	// SqlDB returns the underlying *sql.DB
	SqlDB() *sql.DB
	CatalogAdmin
	SchemaAdmin
	TableAdmin
	JsonRowMapper
	Dialect
	// contains filtered or unexported methods
}

func NewDB

func NewDB(name string, credentialsJSON json.RawMessage, opts ...DBOption) (DB, error)

NewDB creates a new database client for the provided name. It returns an error if options are supplied for a factory that does not support them.

type DBFactory

type DBFactory func(credentialsJSON json.RawMessage) (DB, error)

DBFactory creates a database client from its JSON configuration.

type DBFactoryOptions added in v1.31.0

type DBFactoryOptions struct {
	// contains filtered or unexported fields
}

DBFactoryOptions contains options for database factories that support them.

func (DBFactoryOptions) BigQueryTokenSource added in v1.31.0

func (o DBFactoryOptions) BigQueryTokenSource() oauth2.TokenSource

BigQueryTokenSource returns the caller-supplied token source for BigQuery, if any.

type DBFactoryWithOptions added in v1.31.0

type DBFactoryWithOptions func(credentialsJSON json.RawMessage, opts DBFactoryOptions) (DB, error)

DBFactoryWithOptions creates a database client with factory options.

type DBOption added in v1.31.0

type DBOption interface {
	// contains filtered or unexported methods
}

DBOption configures database client construction.

func WithBigQueryTokenSource added in v1.31.0

func WithBigQueryTokenSource(tokenSource oauth2.TokenSource) DBOption

WithBigQueryTokenSource configures BigQuery clients to authenticate with the provided OAuth2 token source instead of service account JSON credentials. The database config must then carry no credentials: NewDB returns an error rather than silently discarding a configured key. The token source must be safe for concurrent use, cache tokens (for example with oauth2.ReuseTokenSource), and provide tokens with BigQuery scopes. Nil and typed-nil token sources are rejected.

type Dialect

type Dialect interface {
	// QuoteTable quotes a table name
	QuoteTable(table RelationRef) string

	// QuoteIdentifier quotes an identifier, e.g. a column name
	QuoteIdentifier(name string) string

	// FormatTableName formats a table name, typically by lower or upper casing it, depending on the database.
	// Fabric preserves the original case because its default collation is case-sensitive.
	//
	// Deprecated: to be removed in future versions, since its behaviour is not consistent across databases, e.g. using lowercase for BigQuery while it shouldn't.
	// If you want to have a consistent behaviour across databases, use [NormaliseIdentifier] and [ParseRelationRef] instead.
	FormatTableName(name string) string

	// NormaliseIdentifier normalises the identifier's parts that are unquoted, typically by lower or upper casing them, depending on the database
	NormaliseIdentifier(identifier string) string

	// ParseRelationRef parses a string into a RelationRef after normalising the identifier and stripping out surrounding quotes.
	// The result is a RelationRef with case-sensitive fields, i.e. it can be safely quoted (see [QuoteTable] and, for instance, used for matching against the database's information schema.
	ParseRelationRef(identifier string) (RelationRef, error)

	// QueryCondition returns a dialect-specific query expression for the provided identifier, operator and value(s).
	//
	// E.g. QueryCondition("age", "gt", 18) returns "age > 18"
	//
	// Each operator has a different number of arguments, e.g. [eq] requires one argument, [in] requires at least one argument, etc.
	// See [op] package for the list of supported operators
	QueryCondition(identifier, operator string, args ...any) (Expression, error)

	// ParseGoquExpression converts a goqu Expression to an Expression
	ParseGoquExpression(goquExpression GoquExpression) (Expression, error)

	// Expressions returns the dialect-specific expressions
	Expressions() Expressions
}

func NewDialect added in v1.23.0

func NewDialect(name string, optionsJSON json.RawMessage) (Dialect, error)

NewDialect creates a new database dialect for the provided name.

type DialectFactory added in v1.23.0

type DialectFactory func(optionsJSON json.RawMessage) (Dialect, error)

type ErrorClassifier added in v1.32.0

type ErrorClassifier interface {
	ClassifyError(error) ErrorInfo
}

ErrorClassifier is an optional interface that a DB implements when it can classify its own errors. It sits beside DB and does not change it.

type ErrorInfo added in v1.32.0

type ErrorInfo struct {
	Category   string
	Code       string
	ServerCode int32
}

ErrorInfo is the classification of a driver error.

type Expression added in v1.10.0

type Expression interface {
	GoquExpression() GoquExpression
	fmt.Stringer
}

Expression represents a dialect-specific expression. One can get the expression's SQL string by calling [String()] on it.

type Expressions added in v1.10.0

type Expressions interface {
	// TimestampAdd returns an expression that adds the interval to the timestamp value.
	// The value can either be a string literal (column, timestamp, function etc.) or a [time.Time] value.
	TimestampAdd(timeValue any, interval int, unit string) (Expression, error)

	// DateAdd returns an expression that adds the interval to the date value.
	// The value can either be a string literal (column, timestamp, function etc.) or a [time.Time] value.
	// Values are cast to [DATE].
	DateAdd(dateValue any, interval int, unit string) (Expression, error)

	// Literal creates a literal sql expression
	Literal(sql string, args ...any) (Expression, error)
}

Expressions provides dialect-specific expressions

type FilterOptions added in v1.25.0

type FilterOptions struct {
	Catalog string
}

func NewFilterOptions added in v1.25.0

func NewFilterOptions(opts ...Option) (FilterOptions, error)

type GoquExpression added in v1.10.0

type GoquExpression = goqu.Expression

GoquExpression represents a goqu expression

type JsonQueryDB

type JsonQueryDB interface {
	JsonRowMapper
	QueryContext(ctx context.Context, query string, args ...any) (*sql.Rows, error)
}

type JsonRowMapper

type JsonRowMapper interface {
	// JSONRowMapper returns a row mapper that maps rows to map[string]any
	JSONRowMapper() RowMapper[map[string]any]
}

type MaterializationAdmin added in v1.32.0

type MaterializationAdmin interface {
	// CreateTableForQueryWithOptions runs DESCRIBE, CREATE and AwaitTable.
	// It runs no INSERT.
	CreateTableForQueryWithOptions(
		context.Context, QueryExecutor, RelationRef, string, MaterializationOptions,
	) (uuid string, err error)
	// InsertFromQueryWithOptions runs the INSERT ... SELECT into a table
	// that CreateTableForQueryWithOptions created with the same options.
	InsertFromQueryWithOptions(
		context.Context, QueryExecutor, RelationRef, string, MaterializationOptions,
	) error
	// CreateTableFromQueryWithOptions runs both calls above in sequence.
	CreateTableFromQueryWithOptions(
		context.Context, QueryExecutor, RelationRef, string, MaterializationOptions,
	) (uuid string, err error)
	MoveTableWithOptions(
		context.Context, QueryExecutor, RelationRef, RelationRef, MaterializationOptions,
	) (uuid string, err error)
}

MaterializationAdmin is an optional interface for drivers whose create from query needs an explicit engine, sorting key or visibility check.

type MaterializationOptions added in v1.32.0

type MaterializationOptions struct {
	SortingKey        []string
	RequireSortingKey bool
	AllowNullableKey  bool
	Columns           []ColumnRef
	Visibility        VisibilityPolicy
}

type NilAny

type NilAny struct {
	Value any
}

func (*NilAny) Scan

func (v *NilAny) Scan(src any) error

type Option

type Option func(options *Options)

func WithCatalog

func WithCatalog(catalog string) Option

func WithPrefix added in v1.25.0

func WithPrefix(prefix string) Option

func WithRelationType

func WithRelationType(relationType RelationType) Option

func WithSchema

func WithSchema(schema string) Option

type Options added in v1.25.0

type Options struct {
	Schema  string
	Catalog string
	Type    RelationType
	Prefix  string
}

func NewOptions added in v1.25.0

func NewOptions(opts ...Option) Options

type QueryCondition

type QueryCondition struct {
	Column   string `json:"column,omitempty"`
	Operator string `json:"operator,omitempty"`
	Value    string `json:"value,omitempty"`
}

QueryCondition defines a query condition.

type QueryController added in v1.35.0

type QueryController interface {
	// StopQuery stops the connector's own query with queryID on the replica
	// that answers. It never stops a query of another user. Query ids must be
	// unique per execution: a reused id lets a stop reach a later query. The
	// outcome is [StopError] exactly when err is not nil. Callers must verify
	// what a stopped write left behind from their own tables.
	StopQuery(ctx context.Context, queryID string) (StopOutcome, error)
}

QueryController stops a query of the connector's own user by query id. It reads no system table, so it needs no grant beyond the connector's own login. Cancelling a read's context closes the request. A read sent with readonly > 0 and cancel_http_readonly_queries_on_client_close = 1 then stops at its next input read after the client goes away, or at its max_execution_time. A CPU-bound read, such as a large join, can run on for many seconds. A server that ends TLS itself sees the close only when it next writes output. StopQuery stops a read by its id too.

type QueryDB

type QueryDB interface {
	QueryContext(ctx context.Context, query string, args ...any) (*sql.Rows, error)
}

type QueryDef

type QueryDef struct {
	Table      RelationRef       `json:"table"`                // Reference to table that should be queried
	Columns    []string          `json:"columns,omitempty"`    // Columns that should be included. Defaults to "*" if nil or empty.
	Conditions []*QueryCondition `json:"conditions,omitempty"` // Conditions is a list of query conditions.
	OrderBy    *QueryOrder       `json:"order_by,omitempty"`   // OrderBy defines the query's order by clause.
}

QueryDef describes a query that consists of a table and columns that should be queried.

func (*QueryDef) ToSQL

func (query *QueryDef) ToSQL(d Dialect) string

type QueryExecutor added in v1.32.0

type QueryExecutor interface {
	ExecContext(context.Context, string, ...any) (sql.Result, error)
	QueryContext(context.Context, string, ...any) (*sql.Rows, error)
	QueryRowContext(context.Context, string, ...any) *sql.Row
}

QueryExecutor is the statement surface shared by *sql.DB and *sql.Conn.

type QueryOrder

type QueryOrder struct {
	Column string // the order by column
	Order  string // supported values are ('ASC', 'DESC')
}

QueryOrder defines the query's order by clause.This only supports one order by column.

type RelationRef

type RelationRef struct {
	Name    string       `json:"name"`              // the relation's name
	Schema  string       `json:"schema,omitempty"`  // the relation's schema
	Catalog string       `json:"catalog,omitempty"` // the relation's catalog
	Type    RelationType `json:"type,omitempty"`    // the relation's type
}

RelationRef provides a reference to a database table

func NewRelationRef

func NewRelationRef(name string, options ...Option) RelationRef

func NewSchemaTableRef

func NewSchemaTableRef(schema, table string) RelationRef

NewSchemaTableRef creates a new RelationRef with a schema and a table

func (RelationRef) String

func (t RelationRef) String() string

func (*RelationRef) UnmarshalJSON

func (r *RelationRef) UnmarshalJSON(data []byte) error

type RelationType

type RelationType string
const (
	TableRelation RelationType = "table"
	ViewRelation  RelationType = "view"
)

type RowMapper

type RowMapper[T any] func(cols []*sql.ColumnType, row RowScan) (T, error)

RowMapper is a function that maps database rows to a value

func JSONRowMapper

func JSONRowMapper(valueMapper func(databaseTypeName string, value any) any) RowMapper[map[string]any]

JSONRowMapper returns a row mapper that scans rows and maps them to [map[string]any]

type RowScan

type RowScan interface {
	Scan(dest ...any) error
}

RowScan is an interface that represents a row scanner

type SchemaAdmin

type SchemaAdmin interface {
	// CreateSchema creates a schema
	CreateSchema(ctx context.Context, schema SchemaRef) error
	// ListSchemas returns a list of schemas.
	//
	// Supported options:
	//   - [WithCatalog]: scope the listing to a specific catalog.
	//
	//	schemas, err := db.ListSchemas(ctx, WithCatalog("my_catalog"))
	ListSchemas(ctx context.Context, opts ...Option) ([]SchemaRef, error)
	// SchemaExists returns true if the schema exists.
	//
	// Supported options:
	//   - [WithCatalog]: scope the check to a specific catalog.
	//
	//	exists, err := db.SchemaExists(ctx, schemaRef, WithCatalog("my_catalog"))
	SchemaExists(ctx context.Context, schemaRef SchemaRef, opts ...Option) (bool, error)
	// DropSchema drops a schema.
	DropSchema(ctx context.Context, schema SchemaRef) error
}

type SchemaRef

type SchemaRef struct {
	Name string `json:"name"` // the schema
}

SchemaRef provides a reference to a database schema

func (SchemaRef) String

func (s SchemaRef) String() string

type StopOutcome added in v1.35.0

type StopOutcome string

StopOutcome is the result of QueryController.StopQuery. It has three values only, so a caller can use it as a metric label.

const (
	// StopLanded means the first check saw the query, and the stop and its
	// confirmation succeeded. The query may have ended on its own in between.
	// On a multi-replica service the stop can reach a replica that does not
	// run the query, so only the query's own error proves the stop: the
	// server ends a stopped query with code 394 (CH_CANCELLED).
	StopLanded StopOutcome = "landed"
	// StopMissed means no query ran under the id on the answering replica. The
	// query has ended, has not reached the server yet, or runs on another
	// replica of a multi-replica service. A caller that still waits for the answer of its statement
	// knows that the stop missed it.
	StopMissed StopOutcome = "missed"
	// StopError means the stop outcome is unknown. The server may still stop
	// the query. The error says why.
	StopError StopOutcome = "error"
)

type TableAdmin

type TableAdmin interface {
	// CreateTestTable creates a test table
	CreateTestTable(ctx context.Context, relation RelationRef) error
	// ListTables returns a list of tables in the given schema.
	//
	// Supported options:
	//   - [WithCatalog]: scope the listing to a specific catalog.
	//   - [WithPrefix]: filter tables by name prefix.
	//
	//	tables, err := db.ListTables(ctx, schema, WithCatalog("my_catalog"), WithPrefix("test"))
	ListTables(ctx context.Context, schema SchemaRef, opts ...Option) ([]RelationRef, error)
	// TableExists returns true if the table exists. If RelationRef.Catalog is set, the check is scoped to that catalog.
	TableExists(ctx context.Context, relation RelationRef) (bool, error)
	// ListColumns returns a list of columns for the given table
	ListColumns(ctx context.Context, relation RelationRef) ([]ColumnRef, error)
	// ListColumnsForSqlQuery returns a list of columns for the given sql query
	ListColumnsForSqlQuery(ctx context.Context, sql string) ([]ColumnRef, error)
	// CountTableRows returns the number of rows in the given table
	CountTableRows(ctx context.Context, table RelationRef) (count int, err error)
	// DropTable drops a table
	DropTable(ctx context.Context, ref RelationRef) error
	// TruncateTable truncates a table
	TruncateTable(ctx context.Context, ref RelationRef) error
	// RenameTable renames a table. It might fall back to using MoveTable if the underlying database does not support renaming tables.
	RenameTable(ctx context.Context, oldRef, newRef RelationRef) error
	// MoveTable creates a new table by copying the old table's contents to it and then drops the old table. Returns [ErrDropOldTablePostCopy] if the old table could not be dropped after copy.
	MoveTable(ctx context.Context, oldRef, newRef RelationRef) error
	// CreateTableFromQuery creates a table from the results of a query
	CreateTableFromQuery(ctx context.Context, table RelationRef, query string) error
	// GetRowCountForQuery returns the number of rows returned by the query
	GetRowCountForQuery(ctx context.Context, query string, params ...any) (int, error)
}

type TableListOptions added in v1.25.0

type TableListOptions struct {
	Catalog string
	Prefix  string
}

func NewTableListOptions added in v1.25.0

func NewTableListOptions(opts ...Option) (TableListOptions, error)

type ValidationOptions added in v1.32.0

type ValidationOptions struct {
	SyncLogPruning bool
	// WorkingDatabase names the validation working database. ClickHouse has no
	// default and refuses an empty name with CH_CONFIG_INVALID.
	WorkingDatabase string
}

ValidationOptions carries run inputs that are not account fields.

func ValidationOptionsFrom added in v1.32.0

func ValidationOptionsFrom(ctx context.Context) (ValidationOptions, bool)

ValidationOptionsFrom returns the options that WithValidationOptions put on the context. It returns false when the context carries none, as in account-only validation.

type ValidationReporter added in v1.32.0

type ValidationReporter interface {
	ValidateContext(context.Context) (ValidationResult, error)
}

ValidationReporter is an optional interface that a DB implements when it can report a structured validation result.

type ValidationResult added in v1.32.0

type ValidationResult struct {
	ServerVersion string
	GrantsChecked bool
	Warnings      []ValidationWarning
}

type ValidationStageError added in v1.32.0

type ValidationStageError struct {
	Stage int
	Tag   string
	Err   error
}

ValidationStageError reports the validation stage that failed and wraps its cause. A consumer recovers the stage with errors.As.

func (ValidationStageError) Error added in v1.32.0

func (e ValidationStageError) Error() string

func (ValidationStageError) Unwrap added in v1.32.0

func (e ValidationStageError) Unwrap() error

type ValidationWarning added in v1.32.0

type ValidationWarning struct {
	Info      ErrorInfo
	Operation string
}

type ValueOrError

type ValueOrError[T any] struct {
	Value T
	Err   error
}

ValueOrError represents a value or an error

type VisibilityChecker added in v1.32.0

type VisibilityChecker interface {
	// AwaitTable returns the table's UUID once it is visible.
	// A non-empty wantUUID also requires that UUID.
	AwaitTable(ctx context.Context, exec QueryExecutor, ref RelationRef,
		wantUUID string, p VisibilityPolicy) (uuid string, err error)
	// AwaitTableAbsent returns once the name resolves to no table.
	AwaitTableAbsent(ctx context.Context, exec QueryExecutor, ref RelationRef,
		p VisibilityPolicy) error
}

VisibilityChecker is an optional interface. It confirms that a DDL step is visible before the next statement relies on it.

type VisibilityPolicy added in v1.32.0

type VisibilityPolicy struct {
	InitialBackoff time.Duration
	MaxBackoff     time.Duration
	Deadline       time.Duration
}

Directories

Path Synopsis
Package clickhousequery is the public surface that callers of the ClickHouse driver use for per-statement settings, query ids, the dial policy, the query guard and adapter error details.
Package clickhousequery is the public surface that callers of the ClickHouse driver use for per-statement settings, query ids, the dial policy, the query guard and adapter error details.
cmd
cleanup command
internal
chctx
Package chctx marks a context that carries a caller statement settings map, so the driver's connection wrapper leaves that map in place.
Package chctx marks a context that carries a caller statement settings map, so the driver's connection wrapper leaves that map in place.
cherr
Package cherr holds the ClickHouse adapter error type and the one error-code registry that the driver, the client, Lookout and the grants reference share.
Package cherr holds the ClickHouse adapter error type and the one error-code registry that the driver, the client, Lookout and the grants reference share.
chpolicy
Package chpolicy stores the process-wide ClickHouse dial policy that clickhousequery.SetDialPolicy installs and the driver's dialer reads.
Package chpolicy stores the process-wide ClickHouse dial policy that clickhousequery.SetDialPolicy installs and the driver's dialer reads.
chsql
Package chsql is the ClickHouse SQL lexer that the query guard and the SQL wrappers share.
Package chsql is the ClickHouse SQL lexer that the query guard and the SQL wrappers share.
clickhouse
Package clickhouse is the ClickHouse driver of sqlconnect.
Package clickhouse is the ClickHouse driver of sqlconnect.
clickhouse/chtest
Package chtest starts digest-pinned ClickHouse containers through the rudder-go-kit ClickHouse resource and puts a fault-injecting proxy in front of them.
Package chtest starts digest-pinned ClickHouse containers through the rudder-go-kit ClickHouse resource and puts a fault-injecting proxy in front of them.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL