Documentation
¶
Index ¶
- Constants
- func ApplyMigrationPoolTuning(db *gorm.DB) error
- func ApplyPoolTuning(db *gorm.DB, config *Config, logger ...schemas.Logger) error
- func BuildDSN(config *Config) string
- func Close(db *gorm.DB, logger schemas.Logger)
- func ConnectUnpooled(ctx context.Context, sqlDB *sql.DB) (conn *pgx.Conn, ok bool, err error)
- func DescribeTableLockHolders(ctx context.Context, db *gorm.DB, tables ...string) string
- func DisableStatementTimeout(ctx context.Context, conn sqlExecQueryer) (func(context.Context) error, error)
- func ExecWithBoundedLock(tx *gorm.DB, label string, stmts ...string) error
- func IsLockNotAvailableError(err error) bool
- func MigrationDSN(dsn string) string
- func Open(dsn string, config *Config, logger gormlogger.Interface) (*gorm.DB, error)
- func RestoreStatementTimeoutOrDiscard(ctx context.Context, conn rawConn, restore func(context.Context) error, ...) error
- func RunPasswordCommand(ctx context.Context, config *PasswordCommandConfig) (string, error)
- func RunWithLockRetry(ctx context.Context, logger schemas.Logger, label string, ...) error
- func SetApplicationName(base string)
- func TableIsSmall(tx *gorm.DB, table string, maxBytes int64, maxRows float64) (bool, error)
- func ValidTempFileLimit(s string) bool
- func Validate(config *Config, requireStaticPassword bool) error
- func WaitForAdvisoryLock(ctx context.Context, conn *sql.Conn, logger schemas.Logger, ...) error
- func WithApplicationName(dsn, role string) string
- type AdvisoryLockHolder
- type AdvisoryWaitOptions
- type Config
- type LockRetryOptions
- type MaintenanceSessionOptions
- type MaintenanceSessionResult
- type PasswordCommandConfig
Constants ¶
const ( // DefaultSmallTableMaxBytes and DefaultSmallTableMaxRows are the thresholds // below which an index is built without CONCURRENTLY: below both, a plain // build finishes in well under a second, so blocking the table's writes for // that long is cheaper than waiting on every long transaction in the database. DefaultSmallTableMaxBytes int64 = 256 << 20 DefaultSmallTableMaxRows float64 = 100_000 )
const ( // DefaultLockRetryBudget bounds how long a migration step keeps retrying after // lock timeouts, so a process that cannot get its table lock still exits // with a diagnosable error rather than hanging indefinitely. DefaultLockRetryBudget = 10 * time.Minute )
Variables ¶
This section is empty.
Functions ¶
func ApplyMigrationPoolTuning ¶ added in v1.5.5
ApplyMigrationPoolTuning pins a throwaway migration pool to a minimal size. Migration pools are opened, used for serial DDL, and closed; without this they inherit database/sql's defaults, where MaxOpenConns is unlimited.
func ApplyPoolTuning ¶
ApplyPoolTuning applies MaxIdleConns, MaxOpenConns, ConnMaxLifetime, and ConnMaxIdleTime. When a logger is supplied the effective pool shape is logged once, so operators can size against the server's max_connections without reading source — note that logstore and configstore open separate pools, so a pod's ceiling is the sum across them.
func ConnectUnpooled ¶ added in v1.9.0
ConnectUnpooled opens a connection that sqlDB does not manage, the same way sqlDB opens its own: the password command runs again and the session settings apply. The caller closes it. ok is false when sqlDB was not opened that way; its connections then need nothing beyond the settings an existing one was opened with.
func DescribeTableLockHolders ¶ added in v1.10.0
DescribeTableLockHolders lists the other sessions holding a granted lock on any of tables, oldest transaction first, with a pg_cancel_backend hint. It is meant for RunWithLockRetry's Describe: on a rolling upgrade the blockers are usually an old replica's long reads, which an operator can cancel safely. It returns "" when the lookup fails or nothing is held. The lookup is a diagnostic on a retry path, so it is bounded by describeLockHoldersTimeout rather than allowed to hold up the retry itself.
The statement text is never included: pg_stat_activity.query carries the session's current or most recent statement verbatim, literals and all, and this output lands in logs and in the boot error. Each holder is classified into a fixed set of statement kinds (see statementKind); together with the session's application_name that says enough to tell a cancellable read from a write and to identify the replica.
func DisableStatementTimeout ¶ added in v1.9.0
func DisableStatementTimeout(ctx context.Context, conn sqlExecQueryer) (func(context.Context) error, error)
DisableStatementTimeout lifts statement_timeout on one dedicated connection for runtime work that legitimately runs long (index builds, materialized view creation and refresh). It returns a function that restores the connection's previous value; call it before the connection goes back to the pool, or close the connection instead. RESET is not a substitute: the pool's timeout is set after connect, so RESET would drop the connection to the server default.
func ExecWithBoundedLock ¶ added in v1.10.0
ExecWithBoundedLock runs stmts in one short transaction with a bounded lock wait and statement time, retrying lock timeouts with backoff. It is for DDL on small tables that would otherwise use CONCURRENTLY, and runs fine from a migration with UseTransaction=false because it opens its own transaction.
func IsLockNotAvailableError ¶ added in v1.10.0
IsLockNotAvailableError reports whether err is PostgreSQL lock_not_available (SQLSTATE 55P03), which is what lock_timeout raises. Many migrations wrap the driver error with %s rather than %w, which drops the typed error, so the rendered "(SQLSTATE 55P03)" suffix pgx appends is matched as a fallback.
func MigrationDSN ¶ added in v1.10.0
MigrationDSN derives the DSN for a throwaway migration pool from a runtime DSN. simple_protocol keeps cached prepared plans from surviving the DDL and is also how Open recognises a migration pool. application_name identifies this process so lock waiters in other processes can see what the lock holder is doing.
func Open ¶
Open opens a *gorm.DB against the configured Postgres instance.
A runtime pool gets the configured statement_timeout and idle_in_transaction_session_timeout on every physical connection. A migration pool does not: configstore and logstore build their migration DSN by forcing default_query_exec_mode=simple_protocol, and that is how Open tells the two apart (see runtimeSessionSettingsSQL).
func RestoreStatementTimeoutOrDiscard ¶ added in v1.9.0
func RestoreStatementTimeoutOrDiscard(ctx context.Context, conn rawConn, restore func(context.Context) error, timeout time.Duration) error
RestoreStatementTimeoutOrDiscard runs restore, the function returned by DisableStatementTimeout, and enforces that function's contract: when the restore fails, conn is dropped (a Raw callback returning driver.ErrBadConn) so database/sql closes the physical connection instead of pooling it with statement_timeout lifted. The restore runs under timeout and ignores the cancellation of ctx, which is often already done when cleanup runs.
func RunPasswordCommand ¶
func RunPasswordCommand(ctx context.Context, config *PasswordCommandConfig) (string, error)
RunPasswordCommand executes a configured password command and returns stdout.
func RunWithLockRetry ¶ added in v1.10.0
func RunWithLockRetry(ctx context.Context, logger schemas.Logger, label string, opts LockRetryOptions, fn func() error) error
RunWithLockRetry runs fn and, while it fails with a lock timeout, runs it again with exponential backoff until the budget is spent. Only lock timeouts are retried: they mean another session held the lock for a while and retrying later can succeed, whereas every other error is returned at once. fn must be idempotent; migration steps are, because the migrator records completed IDs and the DDL helpers use IF [NOT] EXISTS.
func SetApplicationName ¶ added in v1.10.0
func SetApplicationName(base string)
SetApplicationName sets the application_name prefix every store pool opened after this call reports to Postgres, e.g. "bifrost-ip-10-0-1-13". Each store appends its role ("-config", "-logs") so pg_stat_activity can attribute a session to a node and a store. Empty disables it, which is the default.
func TableIsSmall ¶ added in v1.10.0
TableIsSmall reports whether table is below both maxBytes and maxRows, i.e. small enough to index without CONCURRENTLY.
CREATE INDEX CONCURRENTLY waits for every transaction in the database older than it, whatever table that transaction touches, so on a database shared with a large log store a tiny table's index waits out the longest open transaction (an old replica's materialized-view refresh or unbounded dashboard query). A plain build only takes a SHARE lock on its own table, which on a small table is held for milliseconds.
Size comes from pg_total_relation_size; the row estimate from pg_class.reltuples, which is -1 for a never-analyzed table, in which case the byte size alone decides. A missing table reports small: there is nothing to scan, and the build fails on its own with a clear error.
func ValidTempFileLimit ¶ added in v1.11.0
ValidTempFileLimit reports whether s is a temp_file_limit value Postgres accepts, so a bad config value is caught at startup instead of on every pass. The server stores the limit as an int in kB, so anything above math.MaxInt32 kB (just under 2TB) is rejected there and here.
func WaitForAdvisoryLock ¶ added in v1.10.0
func WaitForAdvisoryLock(ctx context.Context, conn *sql.Conn, logger schemas.Logger, opts AdvisoryWaitOptions) error
WaitForAdvisoryLock acquires a session advisory lock on conn with pg_try_advisory_lock, retrying until it succeeds or the wait budget runs out. It never blocks inside Postgres, so a crashed holder cannot hang the caller.
The budget depends on what the holder is doing. A holder whose process is running statements (a live instance doing slow but legitimate work, such as a CREATE INDEX CONCURRENTLY waiting on a long transaction) earns BusyTimeout; an idle holder only gets Timeout. The caller owns conn and closes it on error.
func WithApplicationName ¶ added in v1.10.0
WithApplicationName appends application_name=<base>-<role> to a libpq DSN when SetApplicationName has set a base. The base is trimmed so the role suffix survives the 63-byte limit. Postgres hex-escapes anything outside printable ASCII, which would lengthen the name past that limit and cut the role off server-side, so such characters become '_' before the byte cut; that also means the cut can never split a multibyte rune.
Types ¶
type AdvisoryLockHolder ¶ added in v1.10.0
type AdvisoryLockHolder struct {
PID int
ApplicationName string
ClientAddr string
State string
// Query is the holder session's own current (or, when idle, last) statement.
Query string
BackendStart time.Time
// BusySessions counts the holder's sibling sessions (same migration
// application_name, else same client address) that are running a statement,
// sitting inside an open transaction, or finished a statement in the last 15
// seconds, including the holder itself. A live instance between short polls (for
// example while it waits to pause matview refreshes) still counts as busy; a
// crashed instance's lingering sessions go quiet and do not.
BusySessions int
// BusyQuery and BusyFor describe the longest-running of those statements.
BusyQuery string
BusyFor time.Duration
// BusyWait is the wait event of that statement, e.g. "Lock/virtualxid".
BusyWait string
}
AdvisoryLockHolder describes the session holding a session-level advisory lock and what the process behind it is doing right now.
func FindAdvisoryLockHolder ¶ added in v1.10.0
func FindAdvisoryLockHolder(ctx context.Context, q rowQueryer, key int64) (*AdvisoryLockHolder, error)
FindAdvisoryLockHolder returns the session holding advisory key in the current database, or nil when nobody holds it.
func (*AdvisoryLockHolder) Busy ¶ added in v1.10.0
func (h *AdvisoryLockHolder) Busy() bool
Busy reports whether the holder's process is visibly doing work, as opposed to a crashed instance's lingering session that will never release the lock.
func (*AdvisoryLockHolder) String ¶ added in v1.10.0
func (h *AdvisoryLockHolder) String() string
String renders the holder for logs and errors.
type AdvisoryWaitOptions ¶ added in v1.10.0
type AdvisoryWaitOptions struct {
// Key is the advisory lock key.
Key int64
// Label names the lock in logs and errors, e.g. "configstore migration".
Label string
// LogPrefix prefixes log lines, e.g. "[configstore]".
LogPrefix string
// RetryInterval is the wait between pg_try_advisory_lock attempts.
RetryInterval time.Duration
// Timeout is how long to wait while the holder shows no sign of work, the
// signature of a crashed instance's lingering session.
Timeout time.Duration
// BusyTimeout is how long to keep waiting while the holder is visibly busy.
// Values at or below Timeout disable the extension.
BusyTimeout time.Duration
// HolderLogInterval throttles the log line describing the holder.
HolderLogInterval time.Duration
// RecentActivityWindow is how recently the holder's process must have run a
// statement to count as busy; defaultRecentActivityWindow when zero.
RecentActivityWindow time.Duration
}
AdvisoryWaitOptions configures WaitForAdvisoryLock.
type Config ¶
type Config struct {
Host *schemas.SecretVar `json:"host"`
Port *schemas.SecretVar `json:"port"`
User *schemas.SecretVar `json:"user"`
Password *schemas.SecretVar `json:"password"`
PasswordCommand *PasswordCommandConfig `json:"password_command,omitempty"`
DBName *schemas.SecretVar `json:"db_name"`
SSLMode *schemas.SecretVar `json:"ssl_mode"`
MaxIdleConns int `json:"max_idle_conns"`
MaxOpenConns int `json:"max_open_conns"`
ConnMaxLifetime string `json:"conn_max_lifetime,omitempty"`
// ConnMaxIdleTime bounds how long an idle physical connection is kept before
// being closed. Without it, the idle cap is the only thing controlling pool
// size, so every burst above MaxIdleConns closes connections on return and
// reopens them on the next query — each reopen forks a Postgres backend.
ConnMaxIdleTime string `json:"conn_max_idle_time,omitempty"`
// StatementTimeout is the server-side statement_timeout set on every runtime
// pool connection (Go duration string, default 120s; "0" or a negative value
// leaves the server default). It bounds a runaway query so it cannot hold a
// pooled connection and its locks indefinitely. Migration pools never get it:
// they run long index builds and backfills.
StatementTimeout string `json:"statement_timeout,omitempty"`
// IdleInTransactionSessionTimeout is the server-side
// idle_in_transaction_session_timeout set on every runtime pool connection
// (Go duration string, default 120s; "0" or a negative value leaves the server
// default). It ends a session left idle inside an open transaction, which
// otherwise holds row locks and blocks vacuum for as long as it stays open.
IdleInTransactionSessionTimeout string `json:"idle_in_transaction_session_timeout,omitempty"`
// MigrationLockRetryBudget is how long a schema migration step keeps
// retrying after lock timeouts before the process gives up and exits (Go
// duration string, default 10m). On a rolling upgrade a step that needs a
// table lock retries while older replicas' long queries hold the table; raise
// this on large deployments so the budget outlasts those queries, and keep
// the startup probe at least as long.
MigrationLockRetryBudget string `json:"migration_lock_retry_budget,omitempty"`
}
Config is the shared Postgres connection configuration used by framework stores.
type LockRetryOptions ¶ added in v1.10.0
type LockRetryOptions struct {
// Budget is the total time spent retrying; DefaultLockRetryBudget when zero.
Budget time.Duration
// InitialBackoff is the first wait between attempts; it doubles up to MaxBackoff.
InitialBackoff time.Duration
// MaxBackoff caps the wait between attempts.
MaxBackoff time.Duration
// Describe, when set, is called on the first lock timeout (logged with the
// retry warning, so operators see what is blocking while there is still time
// to act) and once the budget is exhausted to add context (typically the
// sessions holding the contended lock) to the final error.
Describe func(ctx context.Context) string
}
LockRetryOptions tunes RunWithLockRetry. Zero values take the package defaults.
func MigrationLockRetryOptions ¶ added in v1.10.0
func MigrationLockRetryOptions(config *Config, logger schemas.Logger) LockRetryOptions
MigrationLockRetryOptions returns the step-level retry options a store's config asks for: its migration_lock_retry_budget when set and valid, and otherwise the package default. An invalid or non-positive value is logged and ignored rather than failing boot. A nil config yields the default.
func (LockRetryOptions) WithBudgetFallback ¶ added in v1.10.0
func (opts LockRetryOptions) WithBudgetFallback(fallback LockRetryOptions) LockRetryOptions
WithBudgetFallback returns opts with fallback's budget filled in where opts has none, so a test or caller override keeps precedence over configuration.
type MaintenanceSessionOptions ¶ added in v1.11.0
type MaintenanceSessionOptions struct {
// StatementTimeout is the server-side statement_timeout for the session.
// Zero disables it, as DisableStatementTimeout does.
StatementTimeout time.Duration
// ClientCheckInterval sets client_connection_check_interval, so a backend
// running a long statement notices that its client went away and aborts it.
// Without it the statement runs to completion first. Zero leaves the
// session's value unchanged.
ClientCheckInterval time.Duration
// IdleSessionTimeout sets idle_session_timeout, so a session whose client
// vanished without closing its socket still ends once it sits idle. That is
// what frees session-scoped state such as an advisory lock after the
// statement timeout cancelled the statement. Zero leaves the session's value
// unchanged.
IdleSessionTimeout time.Duration
// TempFileLimit sets temp_file_limit (a Postgres size such as "20GB"), so one
// statement cannot fill the server's disk with temp files. Empty leaves the
// session's value unchanged. The parameter needs superuser or
// GRANT SET ON PARAMETER temp_file_limit; without that it is skipped.
TempFileLimit string
}
MaintenanceSessionOptions are the server-side bounds BoundMaintenanceSession applies to one dedicated connection. They exist because a client-side context deadline dies with the process: when the pod running a long statement is killed, only limits the server enforces itself still stop that statement.
type MaintenanceSessionResult ¶ added in v1.11.0
type MaintenanceSessionResult struct {
// TempFileLimitDenied is true when TempFileLimit was set but the role may not
// change temp_file_limit. The other bounds were still applied.
TempFileLimitDenied bool
// ClientCheckUnsupported is true when ClientCheckInterval was set but the
// server's platform cannot check client sockets (SQLSTATE 22023). The other
// bounds were still applied.
ClientCheckUnsupported bool
}
MaintenanceSessionResult reports what BoundMaintenanceSession could not apply.
func BoundMaintenanceSession ¶ added in v1.11.0
func BoundMaintenanceSession(ctx context.Context, conn sqlExecQueryer, opts MaintenanceSessionOptions) (func(context.Context) error, MaintenanceSessionResult, error)
BoundMaintenanceSession applies opts to conn for long maintenance work (for example materialized-view refreshes) and returns a function that restores the previous values. Call the restore before the connection goes back to the pool, through RestoreStatementTimeoutOrDiscard, exactly as with DisableStatementTimeout. On an error part way through, the returned restore is non-nil and undoes what was already applied; callers must still run it.
type PasswordCommandConfig ¶
type PasswordCommandConfig struct {
Command string `json:"command"`
Args []string `json:"args,omitempty"`
Timeout string `json:"timeout,omitempty"`
// CacheTTL is how long a successfully resolved password is reused across new
// physical connections (Go duration string, default 60s). Set it below the
// credential's validity window; a rotation shorter than the TTL will fail
// connections until the cached value expires.
CacheTTL string `json:"cache_ttl,omitempty"`
}
PasswordCommandConfig describes a command that prints a Postgres password to stdout.
Source Files