Help us shape the future of Go! Take the 2026 Go Developer Survey

dbconn

package
v0.19.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 20, 2026 License: Apache-2.0 Imports: 23 Imported by: 6

README

Database Connections

The dbconn package provides MySQL database connection management and locking utilities for Spirit. It wraps database/sql with Spirit-specific concerns: retry logic, TLS auto-configuration, advisory locking, table locking, and the ability to kill blocking transactions.

Connection Setup

When creating a new connection, Spirit appends standardized DSN parameters to ensure consistent behavior across all connections. These include setting sql_mode="NO_AUTO_VALUE_ON_ZERO" (to be able to copy legacy data like 0000-00-00), time_zone=+00:00, transaction_isolation=read-committed, charset=utf8mb4, collation=utf8mb4_bin, and rejectReadOnly=true (for Aurora failover resilience). NO_AUTO_VALUE_ON_ZERO preserves explicit zero values in auto-increment columns. Spirit also sets session lock timeouts and checks write warnings, so using a non-strict mode does not silently permit lossy copies.

Pool sizing

Use SetPoolSize(db, n) rather than db.SetMaxOpenConns(n) directly. It sets the open and idle limits together, and they must stay together: database/sql closes a connection returned to a pool whose free list already holds MaxIdleConns entries, so an idle limit below the open limit turns every release past that point into a close and every subsequent acquire into a fresh dial, TLS handshake and MySQL auth. The copy phase — hundreds of read and write workers cycling connections continuously — is exactly that workload, and the churn is invisible on the status block, which does not report pool internals at all.

Holding the connections idle instead costs nothing that was not already reserved: the open limit is the budget, and matching the idle limit to it only stops the pool from discarding what it is entitled to keep. Several call sites ratchet a pool's size after it was created (the migration runner once thread counts are final, the checksum, cutover), which is why this lives in a helper rather than at construction only.

Connections are still recycled on maxConnLifetime (3 minutes), which pool sizing does not affect. With a large pool in steady use, that lifetime — not the idle limit — is the dominant source of reconnects.

TLS

Spirit supports five TLS modes: DISABLED, PREFERRED, REQUIRED, VERIFY_CA, and VERIFY_IDENTITY. The default is PREFERRED, which first attempts a TLS connection and falls back to plaintext if it fails. RDS hosts are auto-detected via hostname pattern matching (*.rds.amazonaws.com), and an embedded RDS CA bundle is used automatically.

Retryable Transactions

RetryableTransaction is the primary mechanism for executing statements that may encounter transient errors. It classifies MySQL errors into retryable (deadlocks, lock wait timeouts, connection loss, read-only mode, killed queries) and fatal (everything else). On transient errors, the entire transaction is retried up to MaxRetries times.

An important subtlety is that RetryableTransaction inspects SHOW WARNINGS after every statement. This catches issues that MySQL does not surface as errors, such as range_optimizer_max_mem_size exceeded warnings. This particular warning is treated as fatal because it indicates a table scan will occur instead of an index range scan.

Force Kill

Both ForceExec and NewTableLock kill blocking transactions after a delay. They wait for DBConfig.ForceKillAfter (zero defaults to 90% of LockWaitTimeout), then query performance_schema to identify and kill transactions that are blocking metadata lock acquisition. ForceExec always runs its kill worker; for NewTableLock the kill timer is gated on DBConfig.ForceKill (default true), which programmatic callers such as datasync's read-only source disable for connections that must never kill. LOCK TABLES returns as soon as it holds its locks, so NewTableLock's timer only ever fires while the statement is waiting. A ForceExec statement can keep running after it holds its locks, as a table rebuild does, and the sessions holding locks on the table beside it are then concurrent traffic, not blockers. So while its statement runs, ForceExec checks performance_schema.metadata_locks every 100ms, and at the moment the delay would be reached, and kills only after the statement has been waiting for a table metadata lock for the delay. That also covers a statement that starts waiting part-way through, such as a rebuild upgrading its lock to finish: its blockers get the full delay from when the wait started. The checks run over connections from the same pool as the statement, so the pool must be able to supply a second connection. Each check is bounded to one second. A check that fails, including one that cannot get a connection, kills nothing. A single failed check leaves a wait already seen in progress, so one slow check cannot push the kill past the lock wait timeout. A second failure in a row restarts the wait, because over a longer stretch the statement could have got its lock and started a new wait.

Each blocker is killed with KILL. When KILL is denied (error 1095: the user lacks CONNECTION_ADMIN and SUPER), the kill calls mysql.rds_kill instead, which RDS and Aurora MySQL provide and which needs only EXECUTE on the procedure. On Aurora MySQL 3, rds_superuser_role often lacks CONNECTION_ADMIN, and a blue/green switchover can remove it. The choice is made on every kill, so a privilege change during a run takes effect at the next one. If the procedure is missing or not executable, the error names both failures, and each failed kill is logged with its session id. CheckForceKillPrivileges accepts either way to kill: CONNECTION_ADMIN or SUPER, or EXECUTE on an existing mysql.rds_kill, as SHOW GRANTS lists them.

A kill that cannot list the blocking sessions kills nothing, so both ForceExec and NewTableLock look again every 100ms for as long as the statement still waits. The lookup reads information_schema.innodb_trx for each transaction's weight, and MySQL 9.7 fails every read of that table with error 3854 while any transaction is running a statement whose text holds a 4-byte character, such as an emoji. Such a statement usually ends well within the rest of the lock wait, and the next lookup then finds and kills the blocker.

ForceExec retries a statement that hits a lock wait timeout after its kill worker saw it waiting for the delay and killed, up to DBConfig.MaxRetries attempts in total (the same budget cutover uses). Every attempt runs a fresh kill worker, so a blocker that rolls back slowly or a new blocker that arrives between attempts is killed as well, instead of the retry timing out and the migration falling into a table copy. Between attempts it waits (bounded by 30 seconds) for the killed sessions to leave performance_schema.threads, because KILL is asynchronous. Errors from the kill and cleanup steps are logged but never joined into the statement's error: callers inspect that error to detect ambiguous DDL. Any error other than a lock wait timeout, or a timeout on which no kill ran, is returned immediately. When no kill ran because a check failed, ForceExec logs that at the point of return, so the skipped retries are not mistaken for an exhausted budget. A timeout whose kill left a blocker alive that no kill ends is also returned immediately: an explicit table lock, since the kill never ends a LOCK TABLES session; a transaction too heavy to roll back safely, which the kill skips; or a session the user may not kill, because it lacks CONNECTION_ADMIN or SUPER and cannot execute mysql.rds_kill. Another attempt succeeds only if that blocker happens to finish in time, and until then it holds up the table for another lock wait timeout. A kill that found nothing to end still retries, since its blocker may have finished on its own or been replaced by one the next attempt's kill can end.

ForceExec reserves a sql.Conn for the connection ID lookup, DDL, kill-worker join, and retries. Cancellation cannot return an idle session to the pool while its kill worker still runs. DDL is not wrapped in a transaction; MySQL implicitly commits ALTER TABLE. Use sql.DB.BeginTx when a real transaction is needed.

There are two important safety constraints:

  1. Transaction weight threshold: Transactions with a weight above 1,000,000 (as reported by information_schema.innodb_trx.trx_weight) are never killed, because their rollback would be expensive and disruptive.
  2. Explicit table locks: Connections holding LOCK TABLES are never killed. Instead, an ErrTableLockFound error is returned. This is because killing non-transactional locks is unsafe.

Advisory Lock

AdvisoryLock provides an advisory locking mechanism using MySQL's GET_LOCK() function. It runs on a dedicated single-connection database pool with a background goroutine that periodically refreshes the lock. If the connection drops, it automatically reconnects and re-acquires locks.

Lock names are deterministic hashes of schema.table, truncated with a SHA1 suffix to fit MySQL's 64-character limit for lock names. This is used to prevent concurrent Spirit migrations on the same table.

Table Lock

TableLock wraps MySQL's LOCK TABLES ... WRITE statement. It integrates with the force-kill mechanism to automatically kill blocking transactions if the lock cannot be acquired within the timeout. This is used during legacy snapshot checksum setup and cutover. The default lockless checksum takes no table lock.

Table locks are session-scoped, so the helper reserves a dedicated sql.Conn until Close. A transaction cannot provide that ownership: cancellation can automatically roll it back and return its connection to the pool while table locks remain held. Callers must defer Close after successful acquisition. It ignores the caller's cancellation and starts its own 30-second cleanup timeout when invoked. A successful unlock returns the connection to the pool; a failed unlock or failed acquisition discards the connection. This does not depend on driver session-reset support.

Discarding prevents reuse of the session, but does not guarantee that server-side locks have been released when Close returns. If a statement was interrupted, MySQL may retain the session and its locks until that statement detects the disconnected client or finishes.

ExecUnderLockWithWarnings is the DML path for a final change-feed flush. It checks SHOW WARNINGS on the locking connection after each statement, with the same policy as ordinary flush writes. Unsafe warnings stop the flush and prevent cutover. The DML is already committed when its warnings are read; the method does not start a transaction, which would release LOCK TABLES. DDL continues to use ExecUnderLock so a warning cannot obscure a completed rename.

Transaction Pool

TrxPool pre-creates a pool of REPEATABLE READ transactions with START TRANSACTION WITH CONSISTENT SNAPSHOT. This ensures all worker threads see the same point-in-time data, which is essential for parallel checksum verification.

See Also

Documentation

Overview

Package dbconn contains a series of database-related utility functions.

Index

Constants

View Source 
const (
	// TableLockQuery is used to find tables that are locked by a LOCK TABLES command.
	// It's not really possible to find out how long the lock has been held, so we don't consider
	// the length of the lock here.
	TableLockQuery = `` /* 440-byte string literal not displayed */

	LongRunningEventQuery = `` /* 751-byte string literal not displayed */

)
View Source 
const DefaultMaxConnections = 128

DefaultMaxConnections is the main pool default for migrations, moves and continuous syncs. Monitor and advisory-lock connections use separate dedicated pools.

View Source 
const DriverName = "block-mysql"

DriverName is the database/sql driver spirit opens every connection with.

It is exported because it is part of this package's contract, not an implementation detail: EnhanceDSNWithTLS returns a DSN whose tls= name refers to an entry this package registered, and a driver's TLS registry is a package-level global. A consumer that opens such a DSN with a different driver gets "invalid value / unknown config name: rds" at connect time — an error that says nothing about drivers. Open it with sql.Open( dbconn.DriverName, ...) and that stays correct through any future move.

View Source 
const MinMigrationPoolSize = 5

MinMigrationPoolSize is the smallest --max-connections a migration can complete on. It covers the connections that cutover cannot serialize.

The cutover sets the number: it needs the LOCK TABLES connection, the RENAME TABLE connection and the flush threads, and unlike every other phase it cannot trade connections for time — below the minimum it does not run slower, it cannot run. See migration's CutOver.Run, which holds the same number and will raise a pool that arrives under it.

Nothing else gets a say. The copy, the checksum and the drain all queue on a small pool and finish eventually, so a low --max-connections is the operator asking for a slow migration, which is theirs to ask for.

View Source 
const MonitorPoolSize = 2

MonitorPoolSize is the pool size MonitorOpener gives the Aurora monitor pool: both Aurora throttlers poll concurrently without serializing on one connection.

Variables

View Source 
var (

	// TransactionWeightThreshold is the maximum information_schema.innodb_trx.trx_weight
	// over which we consider a transaction too big to be safely killed. Rolling back a
	// heavy transaction can cause a huge impact on the database.
	TransactionWeightThreshold int64 = 1_000_000

	ErrTableLockFound = errors.New("explicit table lock found! spirit cannot proceed")
)
View Source 
var ErrForceKillPrivilegeMissing = errors.New("missing a privilege force-kill needs")

ErrForceKillPrivilegeMissing matches an error from CheckForceKillPrivileges that names a privilege the user lacks.

View Source 
var ErrLockHeld = errors.New("lock is held by another connection")

ErrLockHeld reports that another connection holds an advisory lock this one needs: another migration is already running against the table, or the schema for an atomic multi-table migration. It is not a failure of the migration that was refused, which can run once the holder releases the lock, so callers can tell it apart from one with errors.Is.

View Source 
var ErrStatementOutcomeUnknown = errors.New("statement outcome unknown")

ErrStatementOutcomeUnknown marks a statement that ExecUnderLock sent to the server but stopped waiting for. The server may have committed it: the caller must check the server state before it acts on the failure.

Functions

func AnalyzeTable added in v0.18.0 

func AnalyzeTable(ctx context.Context, db *sql.DB, logger *slog.Logger, schemaName, tableName string) error

AnalyzeTable runs ANALYZE TABLE for schemaName.tableName on db. An empty schemaName leaves the table unqualified, so it resolves against db's default database (required through a Vitess vtgate, where qualifying is wrong). It reads the result set rather than using Exec because ANALYZE reports a failure such as a missing table as a Msg_type="Error" row (not a statement error), which would otherwise be a silent no-op.

ANALYZE also reports a lock wait timeout as an Error row, with no error code to tell it apart from a permanent failure. A run calls this after its whole copy, so a transient lock should not end it: an Error row is retried up to analyzeAttempts times, and only the last one is returned. A statement error is returned at once.

func CheckForceKillPrivileges added in v0.16.0 

func CheckForceKillPrivileges(ctx context.Context, db *sql.DB) error

CheckForceKillPrivileges verifies that the connection's user holds every privilege force-kill needs: SELECT on the performance_schema tables its queries read (see GetTableLocks and getLockingTransactions), PROCESS to read information_schema.innodb_trx, and CONNECTION_ADMIN or SUPER, or EXECUTE on mysql.rds_kill, to kill another user's session (see checkKillPrivilege). It returns an error naming each one that is missing.

A privilege the user lacks matches ErrForceKillPrivilegeMissing. A read that fails for another reason, such as a lost connection, does not, so a caller can tell a missing grant from a check it could not run.

It is intended for preflight privilege checks. It reads SHOW GRANTS, one row of information_schema.innodb_metrics, no rows of the performance_schema lock tables, and, when the grants allow executing mysql.rds_kill, one row of information_schema.ROUTINES. It logs nothing, so unlike GetTableLocks / getLockingTransactions it neither scans server-wide locks nor emits "found locking transaction" log lines.

func EnhanceDSNWithTLS 

func EnhanceDSNWithTLS(inputDSN string, config *DBConfig) (string, error)

EnhanceDSNWithTLS enhances a DSN with TLS settings from the provided config if the DSN doesn't already contain TLS parameters. This allows replica connections to inherit TLS settings from the main connection while still respecting explicit TLS configuration in the DSN.

The returned DSN names a TLS configuration registered in DriverName's registry. Open it with that driver: a caller that hands the result to a different one fails at connect time with "invalid value / unknown config name", because TLS registries are per-driver package globals rather than anything the DSN carries.

This settles TLS and nothing else. The settings a spirit connection also needs — interpolation, cleartext-password gating, integer tinyint(1) — are applied by newDSN, so they reach a connection only via New. Spirit's own replica path passes this result to NewWithConnectionType for exactly that reason; a consumer that opens the result directly owns the rest itself.

func Exec 

func Exec(ctx context.Context, db *sql.DB, stmt string, args ...any) error

Exec is like db.Exec but only returns an error. This makes it a little bit easier to use in error handling. It accepts args which are escaped client side using the sqlescape library. i.e. %n is an identifier, %? is automatic type conversion on a variable, and %r splices a sqlescape.RawSQL argument in verbatim (for raw user SQL such as an ALTER clause, which must never be concatenated into the format string).

func ExecWithoutForeignKeyChecks added in v0.19.0 

func ExecWithoutForeignKeyChecks(ctx context.Context, db *sql.DB, stmt string, args ...any) error

ExecWithoutForeignKeyChecks is like Exec, but runs stmt in a session with foreign_key_checks turned off. It is for DDL, which a SET_VAR hint cannot apply to: with the checks off, MySQL adds a foreign key in place, as a metadata change, rather than copying the table to check its rows.

The session is turned back to the setting it had before it is returned to the pool. When that fails, the session is discarded instead, so no other statement can run with the checks off. The setting is read rather than reset with DEFAULT: SET foreign_key_checks = DEFAULT turns the checks off.

func ForceExec 

func ForceExec(ctx context.Context, db *sql.DB, tables []*table.TableInfo, dbConfig *DBConfig, logger *slog.Logger, stmt string, args ...any) error

ForceExec is like Exec but it has some added logic to force kill any connections that are holding up metadata locks preventing this from succeeding. It kills only while the statement is waiting for a table metadata lock, never while it holds its locks and runs. The statement holds one connection from db while the checks and the kill run over others, so db must be able to supply a second connection: a check that cannot get one in time fails, and a failed check kills nothing. Like Exec, stmt is a sqlescape format string: embed raw user SQL (such as an ALTER clause) with the %r verb and a sqlescape.RawSQL argument, never by concatenating it into stmt.

func GetTLSConfigForBinlog 

func GetTLSConfigForBinlog(config *DBConfig, host string) (*tls.Config, error)

GetTLSConfigForBinlog creates a TLS config for binary log connections using the same logic as main database connections

func IsConnectionLossError added in v0.15.0 

func IsConnectionLossError(err error) bool

IsConnectionLossError reports whether err indicates that the connection to MySQL failed or was lost, meaning the client cannot know whether the last statement it sent was executed by the server. Connection-level failures never surface as a *mysql.MySQLError: go-sql-driver returns driver.ErrBadConn when the failure was detected before anything was written, and mysql.ErrInvalidConn when the connection died mid-statement — possibly *after* the server executed the statement but before the client read the result. Raw io.EOF is included for paths that surface the TCP-level error directly, and the client-library codes CR_CONN_HOST_ERROR (2003) / CR_SERVER_LOST (2013) are included because proxies (e.g. ProxySQL, RDS Proxy) can relay them inside real server error packets.

In contrast to deterministic SQL errors (lock wait timeout, deadlock, ...), where the server has positively reported that the statement did NOT take effect, these errors are ambiguous. Callers retrying a non-idempotent statement (e.g. the cutover RENAME TABLE) must verify server-side state before deciding whether the statement was applied. The exception is ER_CLIENT_INTERACTION_TIMEOUT (4031): the server killed the session for inactivity *before* the observing statement arrived, so that statement positively did not execute — verification is still safe, just guaranteed to conclude "not applied".

func IsLockContentionError added in v0.17.0 

func IsLockContentionError(err error) bool

IsLockContentionError reports whether err is InnoDB lock contention: a lock wait timeout (1205) or a deadlock (1213). Both are already covered by canRetryError, but callers that can *adapt* — by backing off harder or by lowering their own write concurrency — need to tell contention apart from the other retryable classes, which no amount of self-throttling would fix.

This distinction matters because contention can be self-inflicted. Spirit runs on READ COMMITTED (see conn.go), so concurrent REPLACE batches with disjoint primary keys never gap-conflict on the clustered index. They do still take next-key locks during duplicate-key handling on every *secondary* index, where "disjoint by PK" buys nothing — so a wide enough flush fan-out deadlocks against itself with no external workload at all.

func IsOutcomeUnknown added in v0.18.0 

func IsOutcomeUnknown(err error) bool

IsOutcomeUnknown reports whether err leaves the outcome of the statement unknown: the connection was lost (see IsConnectionLossError), or TableLock.ExecUnderLock stopped waiting for the reply (ErrStatementOutcomeUnknown). The caller must check the server state before it treats the statement as failed.

func IsRDSHost 

func IsRDSHost(host string) bool

IsRDSHost reports whether host is an Amazon RDS or Aurora endpoint.

DriverName applies verified TLS to such a host by itself, so the database/sql paths in this package no longer need to ask. It stays exported and in use for two reasons: GetTLSConfigForBinlog serves the go-mysql binlog client, which is not a database/sql connection and so is not covered by the driver; and block/schemabot calls it to decide a TLS mode.

Two things changed by delegating. The match is now case-insensitive, which is a fix — DNS is case-insensitive, nothing normalizes the host, and a hostname that arrives uppercased is the same endpoint. And GovCloud and China endpoints now report false, because the bundle behind NewTLSConfig contains no roots for either partition, so verifying against it could only ever fail. Reach those with --tls-certificate-path and that partition's own bundle.

func KillLockingTransactions 

func KillLockingTransactions(ctx context.Context, db *sql.DB, tables []*table.TableInfo, logger *slog.Logger, ignorePIDs []int) error

func KillSessionAndWait added in v0.18.0 

func KillSessionAndWait(ctx context.Context, db *sql.DB, pid int) error

KillSessionAndWait kills the session pid and waits until it has exited, so any statement it was running has either finished or been rolled back. A session that is already gone counts as success. ctx bounds both steps.

It needs no extra privileges when pid belongs to the same user as db: a user can KILL its own sessions and see them in information_schema.PROCESSLIST without CONNECTION_ADMIN or PROCESS. Another user's session needs CONNECTION_ADMIN or SUPER, or, on RDS and Aurora, EXECUTE on mysql.rds_kill (see killConnection).

func LoadCertificateFromFile 

func LoadCertificateFromFile(filePath string) ([]byte, error)

LoadCertificateFromFile loads certificate data from a file

func MonitorOpener added in v0.19.0 

func MonitorOpener(dsn string, cfg *DBConfig, label string) func() (*sql.DB, error)

MonitorOpener returns the throttler.AuroraSetup.OpenMonitor closure: it opens a MonitorPoolSize pool on dsn from a copy of cfg (cfg is never mutated), labelled label in connect errors. The throttler package does not import dbconn, so the runners pass it this.

func New 

func New(inputDSN string, config *DBConfig) (db *sql.DB, err error)

New is similar to sql.Open except we take the inputDSN and append additional options to it to standardize the connection. It will also ping the connection to ensure it is valid.

func NewContext added in v0.18.0 

func NewContext(ctx context.Context, inputDSN string, config *DBConfig) (db *sql.DB, err error)

NewContext is like New, but the connect-time ping also ends when ctx is done, so a caller that is shutting down is not held for up to connectTimeout by a server that never completes the handshake.

func NewCustomTLSConfig 

func NewCustomTLSConfig(certData []byte, sslMode string) *tls.Config

NewCustomTLSConfig creates a TLS config based on SSL mode and certificate data.

An empty certData means "use the RDS roots", which is the fallback for a host that is not recognizably RDS and for which no --tls-certificate-path was given. That fallback is inherited behaviour and is rarely what anyone wants: a non-RDS server will not present an RDS-issued certificate, so VERIFY_CA and VERIFY_IDENTITY against it fail by construction. It is preserved here rather than changed, because tightening it is a behaviour change that belongs in its own commit.

func NewTLSConfig 

func NewTLSConfig() *tls.Config

NewTLSConfig returns a TLS config that verifies an Amazon RDS or Aurora server against the RDS root bundle.

The bundle and the pool behind this used to be spirit's own — an embedded copy of global-bundle.pem plus an x509.CertPool built from it. Both now come from DriverName, which carries the same bundle for its own auto-TLS, so there is one copy to refresh instead of two that can drift apart.

Each call returns a config with its own RootCAs, so callers may modify the result (GetTLSConfigForBinlog sets ServerName on it). It also pins MinVersion to TLS 1.2, which spirit's version did not.

func NewWithConnectionType 

func NewWithConnectionType(inputDSN string, config *DBConfig, connectionType string) (db *sql.DB, err error)

NewWithConnectionType is like New but includes context about the connection type for better error messages

func ReadBoundsForPool added in v0.17.0 

func ReadBoundsForPool(start, ceiling, maxConnections, reserve, holds int) (int, int)

ReadBoundsForPool fits BOTH bounds of a reader pool without growing the connection pool. Fitting just the ceiling fails because consumers floor it back to the start (see checksum.NewChecker and the copier's resolveReadCeiling). The legacy SingleChecker also pre-opens a snapshot transaction per ceiling slot under one table lock (initConnPool); the lockless checker borrows a connection per read. Migration, move and sync all fit with it. The caller supplies its lifecycle-specific reserve. holds is how many connections one reader takes from the busiest pool at once: one for a migration, more for a move whose source and target share a handle. Values below one mean one. Unresolved connection limits pass through; a small budget retains one reader so it can progress.

func RedactDSN added in v0.16.0 

func RedactDSN(dsn string) string

RedactDSN returns dsn with the password masked, safe for logging. It keeps the username, host and parameters so logs stay useful, masking only the password and only when one was actually present. If the driver can't parse the DSN it still never echoes a password: it redacts the credentials section before '@', or — lacking '@' — masks from the first ':' (a malformed "user:password" still has its password hidden).

func RequireDifferentDatabase added in v0.17.0 

func RequireDifferentDatabase(ctx context.Context, source, target *sql.DB) error

RequireDifferentDatabase refuses a copy whose target aliases its source. Connection strings are insufficient: different users or hostnames can reach the same database. Check the selected schemas and, when they overlap, the server UUIDs before any target writes or destructive recovery.

func RequireDifferentDatabases added in v0.17.0 

func RequireDifferentDatabases(ctx context.Context, sources, targets []*sql.DB) error

RequireDifferentDatabases refuses any source/target database alias. Each connection's identity is read at most once, which keeps sharded moves from issuing the same identity queries for every source/target pair.

func RetryableTransaction 

func RetryableTransaction(ctx context.Context, db *sql.DB, dupKeyHandling DupKeyHandling, config *DBConfig, stmts ...string) (int64, error)

RetryableTransaction retries all statements in a transaction, retrying if a statement errors, or there is a deadlock. It will retry up to maxRetries times.

func SetPoolSize added in v0.16.0 

func SetPoolSize(db *sql.DB, n int)

SetPoolSize sets a pool's connection limit, keeping the idle limit equal to it. Both must move together: database/sql closes a connection returned to a pool whose free list already holds MaxIdleConns entries, so an idle limit below the open limit turns every release past that point into a close and every subsequent acquire into a fresh dial, TLS handshake and MySQL auth. The copy phase is exactly that workload — up to a few hundred write and read workers cycling connections continuously — and the churn is invisible in the status block, which does not report pool internals at all.

Holding the connections idle instead costs nothing the caller has not already reserved: SetMaxOpenConns is the budget, and this only stops the pool from throwing away what it is entitled to keep. Note that connections are still recycled on maxConnLifetime, which this does not change.

n <= 0 means unlimited to SetMaxOpenConns; pass it through unchanged (and leave the idle limit alone, since "unlimited idle" is not expressible) rather than silently reinterpreting it.

func SplitDSNs added in v0.16.0 

func SplitDSNs(dsnList string) []string

SplitDSNs splits a comma-separated list of DSNs into a slice, trimming surrounding whitespace and dropping empty entries. An empty input returns nil. (Used for the replica DSN list, but takes no position on what the individual DSNs mean.)

func ValidateConnectionLimit added in v0.17.0 

func ValidateConnectionLimit(limit int) error

ValidateConnectionLimit rejects negative limits, which database/sql would otherwise interpret as unlimited. Zero means the runner should apply its default.

func ValidateMaxConnections added in v0.17.0 

func ValidateMaxConnections(maxConnections, readers, reserve int) error

ValidateMaxConnections validates an explicit pool budget against pinned checksum readers and runner-specific headroom. Zero is unresolved and is accepted so callers can apply their defaults after validation.

func WithMultiTableSchemaLock added in v0.16.0 

func WithMultiTableSchemaLock(schemaName string) func(*AdvisoryLock)

WithMultiTableSchemaLock adds a schema-scoped lock to the AdvisoryLock so that only one atomic multi-table migration runs per schema at a time. Multi-table migrations all coordinate through a single shared _spirit_checkpoint (and _spirit_sentinel), so they must not overlap; a second one fails to acquire this lock and aborts. Single-table migrations do not use it — they are serialized per-table by the table locks and may run concurrently.

Applied as an option so the lock name is prepended before the per-table names; it is held and released on the same dedicated session as the rest.

Types

type AdvisoryLock added in v0.16.0 

type AdvisoryLock struct {
	// contains filtered or unexported fields
}

AdvisoryLock ensures that only one spirit migration operates on a table at a time. It is a user-level advisory lock (GET_LOCK) held on a dedicated connection, so it is invisible to application traffic and never blocks queries or DDL. It may be confused for a table lock, which it is not.

func NewAdvisoryLock added in v0.16.0 

func NewAdvisoryLock(ctx context.Context, dsn string, tables []*table.TableInfo, config *DBConfig, logger *slog.Logger, optionFns ...func(*AdvisoryLock)) (*AdvisoryLock, error)

func (*AdvisoryLock) Close added in v0.16.0 

func (m *AdvisoryLock) Close() error

type DBConfig 

type DBConfig struct {
	ForceKillAfter           time.Duration // Zero preserves the default: 90% of LockWaitTimeout.
	LockWaitTimeout          int
	InnodbLockWaitTimeout    int
	MaxRetries               int // Total attempts, not retries after the first: 1 means a single attempt.
	MaxOpenConnections       int
	RangeOptimizerMaxMemSize int64
	InterpolateParams        bool
	ForceKill                bool // If true, kill locking transactions to acquire metadata locks (default: true)
	// There is deliberately no RejectReadOnly here. It used to map to the
	// driver's rejectReadOnly option and defaulted to true, guarding against
	// landing on a demoted, now-read-only Aurora primary after a blue/green
	// deploy or failover. [DriverName] now does that unconditionally and has
	// removed the option, so there is nothing left to configure.
	//
	// The field also carried an opt-out, used by the sync runner for a
	// read-only source. That turned out to be guarding against an error the
	// workload cannot raise: 1290/1792/1836 are raised by *writes*, and a
	// read-only source only reads. Verified against a super_read_only MySQL
	// 8.0 — SELECT, SHOW TABLES, SHOW CREATE TABLE, SHOW MASTER STATUS, every
	// session SET newDSN adds, and the binlog client's FLUSH BINARY LOGS all
	// succeed; an INSERT is what returns 1290.
	// TLS Configuration
	TLSMode            string // TLS connection mode (DISABLED, PREFERRED, REQUIRED, VERIFY_CA, VERIFY_IDENTITY)
	TLSCertificatePath string // Path to custom TLS certificate file
}

func NewDBConfig 

func NewDBConfig() *DBConfig

func (*DBConfig) StatementCompletionTimeout added in v0.18.0 

func (c *DBConfig) StatementCompletionTimeout() time.Duration

StatementCompletionTimeout is how long to wait for a statement whose outcome the caller must know, such as a cutover RENAME TABLE: the session's lock_wait_timeout plus lockedStatementCompletionMargin. The server reports a metadata lock wait as ER_LOCK_WAIT_TIMEOUT after lock_wait_timeout, and that error is conclusive, so the client-side bound must be longer. A statement still running when this bound expires has an unknown outcome.

func (*DBConfig) ValidateForceKillAfter added in v0.18.0 

func (c *DBConfig) ValidateForceKillAfter() error

ValidateForceKillAfter rejects delays that cannot leave time for lock acquisition. LockWaitTimeout is in whole seconds, matching MySQL's session variable.

type DupKeyHandling added in v0.16.0 

type DupKeyHandling int

DupKeyHandling selects how warning checks treat duplicate-key (1062) warnings. Copy / INSERT IGNORE paths legitimately expect dup-key warnings (e.g. resume re-inserts); checksum-fix DELETE/REPLACE/UPSERT paths do not and want them surfaced. Using a named int enum (rather than a bool) keeps call sites self-documenting and stops a bare positional bool (true/false) from compiling. 

const (
	// ErrorOnDupKey surfaces duplicate-key warnings as errors.
	ErrorOnDupKey DupKeyHandling = iota
	// IgnoreDupKeyWarnings tolerates duplicate-key warnings.
	IgnoreDupKeyWarnings
)

type LockDetail 

type LockDetail struct {
	PID          int
	User         sql.NullString
	Host         sql.NullString
	Info         sql.NullString
	ObjectType   sql.NullString
	ObjectSchema sql.NullString
	ObjectName   sql.NullString
	LockType     sql.NullString // e.g. "INTENTION_EXCLUSIVE", "SHARED_READ",
	LockDuration sql.NullString // e.g. "STATEMENT", "TRANSACTION"
	LockStatus   sql.NullString
	RunningTime  sql.NullString // Human-readable format of the timer_wait
	TimerWait    sql.NullInt64  // in picoseconds
	TrxWeight    sql.NullInt64  // Rows modified by the transaction
}

func GetTableLocks 

func GetTableLocks(ctx context.Context, db *sql.DB, tables []*table.TableInfo, logger *slog.Logger, ignorePIDs []int) ([]*LockDetail, error)

type RowQuerier added in v0.18.0 

type RowQuerier interface {
	QueryRowContext(ctx context.Context, query string, args ...any) *sql.Row
}

RowQuerier is the part of *sql.DB (or *sql.Conn) that reads one row.

type TableLock 

type TableLock struct {
	// contains filtered or unexported fields
}

func NewTableLock 

func NewTableLock(ctx context.Context, db *sql.DB, tables []*table.TableInfo, config *DBConfig, logger *slog.Logger) (*TableLock, error)

NewTableLock creates a new server wide lock on multiple tables. i.e. LOCK TABLES .. WRITE. It uses a short timeout and *does not retry*. The caller is expected to retry, which gives it a chance to first do things like catch up on replication apply before it does the next attempt.

config.ForceKill=true is the default, and will more or less ensure that the lock acquisition is successful by killing long-running queries that are blocking our lock acquisition after ForceKillAfter (by default, 90% of LockWaitTimeout). Programmatic callers that never take locks (e.g. datasync's read-only source) can disable it via DBConfig.ForceKill.

func NewTableLockReferencing added in v0.19.0 

func NewTableLockReferencing(ctx context.Context, db *sql.DB, tables, referenced []*table.TableInfo, config *DBConfig, logger *slog.Logger) (*TableLock, error)

NewTableLockReferencing is NewTableLock for tables whose foreign keys reference the tables in referenced. LOCK TABLES also locks the tables a locked table's foreign keys reference, for reading, so it waits for the sessions that write to them. DDL under the lock that adds, drops or renames a foreign key (RENAME TABLE included) waits for every open transaction that has read or written them. With config.ForceKill, the sessions blocking the lock or a statement run under it on a referenced table are killed, as they are on the locked tables.

A foreign key added under the lock can only reference a table that is locked this way: MySQL refuses any other with error 1100.

func (*TableLock) Close 

func (s *TableLock) Close(ctx context.Context) error

Close releases the table lock even if the caller's context has expired. The cleanup budget starts here, after all work under the lock has finished. A session whose unlock fails is discarded, never returned to the pool.

func (*TableLock) DB added in v0.15.0 

func (s *TableLock) DB() *sql.DB

DB returns the database connection pool this lock was acquired on. Because LOCK TABLES ... WRITE blocks writes from every other connection, any write to a locked table must go through this lock's own connection. Callers holding locks on multiple servers (e.g. one per shard) use this to match each lock to the target it belongs to.

func (*TableLock) ExecUnderLock 

func (s *TableLock) ExecUnderLock(ctx context.Context, stmts ...string) error

ExecUnderLock executes statements on the locking session, in order.

It does not start a statement once ctx is done. A statement it has started runs to completion even if ctx is then cancelled: with a cancellable context, a cancel during the statement would close the connection and return context.Canceled, but the server can still commit the statement, so the caller could not tell whether it took effect (issue #1338). Each statement instead runs on a context detached from ctx's cancellation and bounded by DBConfig.StatementCompletionTimeout. If that bound expires, the error wraps ErrStatementOutcomeUnknown. The client closes the connection then, but the server may still be running the statement: a read of the server state that shows it did not take effect is not conclusive.

A caller that must run its statements even after a cancel, such as the rename that retires a source after a traffic switch, passes context.WithoutCancel(ctx).

For a lock taken with NewTableLockReferencing, the sessions that block a statement on a referenced table are killed once it has waited the force-kill delay.

func (*TableLock) ExecUnderLockWithWarnings added in v0.19.0 

func (s *TableLock) ExecUnderLockWithWarnings(ctx context.Context, stmts ...string) error

ExecUnderLockWithWarnings executes DML on the locking session and checks SHOW WARNINGS after every statement, using the same policy as an ordinary change-feed flush (ErrorOnDupKey). A warning or a failed warning read stops execution before the next statement, so the caller must not proceed to cutover.

Like ExecUnderLock, a started statement finishes despite caller cancellation. The warning read shares that detached completion budget. No transaction is opened: BEGIN would release LOCK TABLES. A warning therefore reports a write that has already committed; it does not roll it back.

DDL callers use ExecUnderLock: a warning after a successful RENAME must not turn a known ownership transfer into an ordinary failed-attempt result.

func (*TableLock) ExecUnderLockWithoutForeignKeyChecks added in v0.19.0 

func (s *TableLock) ExecUnderLockWithoutForeignKeyChecks(ctx context.Context, stmts ...string) error

ExecUnderLockWithoutForeignKeyChecks is ExecUnderLock with the locking session's foreign_key_checks turned off for the statements, for DDL that must add a foreign key in place without checking the rows (see ExecWithoutForeignKeyChecks). The session gets its previous setting back afterwards; if it cannot, Close discards the session instead of returning it to the pool.

type TrxPool 

type TrxPool struct {
	sync.Mutex
	// contains filtered or unexported fields
}

func NewTrxPool 

func NewTrxPool(ctx context.Context, db *sql.DB, count int, logger *slog.Logger) (*TrxPool, error)

NewTrxPool creates a pool of transactions which have already had their read-view created in REPEATABLE READ isolation.

The pool is sized for the maximum concurrency the caller may ever scale up to, so some transactions can sit unused for hours. An idle transaction still counts against the server's wait_timeout, and managed configurations often set it low (e.g. 600s on Aurora): without intervention the server silently kills the connection and a later Get() hands out a dead transaction that fails with "driver: bad connection". To prevent that, the pool runs a background keepalive that periodically pings whatever transactions are idle in the pool; it stops when ctx is canceled or the pool is closed. A nil logger discards keepalive warnings.

func (*TrxPool) Close 

func (p *TrxPool) Close() error

Close closes all transactions in the pool.

func (*TrxPool) Get 

func (p *TrxPool) Get() (*sql.Tx, error)

Get gets a transaction from the pool.

func (*TrxPool) Put 

func (p *TrxPool) Put(trx *sql.Tx)

Put puts a transaction back in the pool.

type UnsafeWarningError added in v0.17.0 

type UnsafeWarningError struct {
	Warning *mysql.MySQLError
}

UnsafeWarningError reports a warning that MySQL raised on a statement Spirit executed without error, and that Spirit treats as fatal. Statements such as INSERT IGNORE succeed while discarding rows, so the warning is the only signal that the copy would silently lose data.

It unwraps to the underlying *mysql.MySQLError, so callers can classify the warning by its code with errors.As or errors.AsType rather than by matching on the message. The code matters because the same fatal branch covers unrelated conditions — a NOT NULL column with no default (1364), a duplicate on a unique key (1062), a value too long for its column (1406) — which a caller may want to report or act on differently.

func (*UnsafeWarningError) Error added in v0.17.0 

func (e *UnsafeWarningError) Error() string

Error reports the warning and its code. The type is exported, so a caller can hold one without a warning; Error stays callable on that value because the places an error's text is read — logs, %v, a failing test — are the last places a panic is affordable.

func (*UnsafeWarningError) Unwrap added in v0.17.0 

func (e *UnsafeWarningError) Unwrap() error

Unwrap returns the underlying warning, or nil when the error carries none. A nil return ends the chain, which is what errors.Is and errors.As expect.

Source Files

View all Source files

Directories

Path Synopsis
Package sqlescape provides SQL escaping functionality.
 Package sqlescape provides SQL escaping functionality.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.