Documentation
¶
Overview ¶
Package dbbackup holds the database backup job specification, agent-side credential profiles and the run manifest. It is shared by the server (validation, redaction), the agent (execution) and the configuration loader, so it must stay free of process-execution and platform dependencies.
Index ¶
- Constants
- Variables
- func Bool(v *bool, def bool) bool
- func DefaultBackupID(s *Spec) string
- func DefaultPort(engine string) int
- func NormalizeEngine(name string) (string, bool)
- type BinlogPosition
- type BinlogPrevious
- type Config
- type CredentialProfile
- type Credentials
- type DatabaseEntry
- type FileEntry
- type JobState
- type JobStateRecord
- type K8sTarget
- type Manifest
- type ManifestGenerator
- type ManifestJob
- type ManifestServer
- type ManifestSnapshot
- type MongoManifest
- type MongoOptions
- type MySQLBinlog
- type MySQLManifest
- type MySQLOptions
- type MySQLState
- type PostgresBasebackup
- type PostgresGlobals
- type PostgresManifest
- type PostgresOptions
- type PostgresState
- type Restore
- type Spec
- type TLSSpec
- type Tool
- type ToolConfig
Constants ¶
const ( DefaultMinFreeBytes = int64(1 << 30) DefaultSizeSafetyFactor = 1.2 DefaultKillGrace = 20 * time.Second )
Defaults
const ( ManifestFileName = "fallbck-db.json" ManifestBlobName = "fallbck-db.json" // ArchiveName is the pxar archive holding the spool directory ArchiveName = "db" )
ManifestFileName is the manifest inside the archive; the blob is ManifestBlobName + ".blob"
const ( EnginePostgres = "postgres" EngineMySQL = "mysql" EngineMongoDB = "mongodb" )
Supported engines
const ( TLSDisable = "disable" TLSPrefer = "prefer" TLSRequire = "require" TLSVerifyCA = "verify-ca" TLSVerifyFull = "verify-full" )
TLS modes (libpq spelling; mapped per engine by the agent)
const ( CompressionNone = "none" CompressionGzip = "gzip" )
Compression settings for the spooled dumps. PBS deduplicates and compresses chunks itself, so the default is uncompressed dumps.
const ( PostgresModeDump = "dump" // pg_dumpall --globals-only + pg_dump per database PostgresModeBasebackup = "basebackup" // pg_basebackup with streamed WAL PostgresModeBoth = "both" )
PostgreSQL backup modes
const ManifestSchemaVersion = 1
ManifestSchemaVersion is bumped on incompatible manifest changes
Variables ¶
var Engines = []string{EnginePostgres, EngineMySQL, EngineMongoDB}
Engines lists the supported engines in display order
var ErrInvalidSpec = errors.New("invalid database job configuration")
ErrInvalidSpec wraps every validation failure
Functions ¶
func DefaultBackupID ¶
DefaultBackupID is the PBS backup group id for a database job: db-<engine>-<host>, keeping database snapshots apart from the host's filesystem backups. Kubernetes targets use db-<engine>-<namespace>-<service>.
func DefaultPort ¶
DefaultPort returns the engine's default TCP port
func NormalizeEngine ¶
NormalizeEngine maps aliases (postgresql, mariadb, mongo) to the canonical engine name
Types ¶
type BinlogPosition ¶
type BinlogPrevious ¶
type BinlogPrevious struct {
BinlogPosition
RunID string `json:"run_id,omitempty"`
Snapshot string `json:"snapshot,omitempty"`
}
type Config ¶
type Config struct {
// SpoolDir receives the dumps before upload (default <data_dir>/spool)
SpoolDir string `yaml:"spool_dir,omitempty" json:"spool_dir,omitempty"`
// StateDir keeps per-job state such as the last MySQL binlog position (default <data_dir>/state/db)
StateDir string `yaml:"state_dir,omitempty" json:"state_dir,omitempty"`
// MinFreeBytes must be free on the spool filesystem before and during a run (default 1 GiB)
MinFreeBytes int64 `yaml:"min_free_bytes,omitempty" json:"min_free_bytes,omitempty"`
// SizeSafetyFactor scales the server's size estimate for the disk check (default 1.2)
SizeSafetyFactor float64 `yaml:"size_safety_factor,omitempty" json:"size_safety_factor,omitempty"`
// KillGrace is the SIGTERM → SIGKILL grace period for client tools (default 20s)
KillGrace time.Duration `yaml:"kill_grace,omitempty" json:"kill_grace,omitempty"`
// KeepFailedSpool leaves the spool directory in place when a run fails (debugging)
KeepFailedSpool bool `yaml:"keep_failed_spool,omitempty" json:"keep_failed_spool,omitempty"`
// Checksums records a sha256 per spooled file in the manifest (default true)
Checksums *bool `yaml:"checksums,omitempty" json:"checksums,omitempty"`
Tools ToolConfig `yaml:"tools,omitempty" json:"tools"`
Profiles map[string]CredentialProfile `yaml:"profiles,omitempty" json:"profiles,omitempty"`
}
Config is the agent-side database backup configuration (db_backup: in agent.yaml)
func (*Config) ApplyDefaults ¶
ApplyDefaults fills unset values; dataDir anchors the spool and state directories
func (*Config) ChecksumsEnabled ¶
ChecksumsEnabled reports whether per-file checksums are recorded
type CredentialProfile ¶
type CredentialProfile struct {
Engine string `yaml:"engine" json:"engine"`
User string `yaml:"user,omitempty" json:"user,omitempty"`
Password string `yaml:"password,omitempty" json:"-"`
// PasswordFile is read at run time (trailing newline trimmed)
PasswordFile string `yaml:"password_file,omitempty" json:"password_file,omitempty"`
Host string `yaml:"host,omitempty" json:"host,omitempty"`
Port int `yaml:"port,omitempty" json:"port,omitempty"`
TLS TLSSpec `yaml:"tls,omitempty" json:"tls"`
}
CredentialProfile is a named set of database credentials kept on the agent host (db_backup.profiles in the agent configuration). A job that leaves its password blank is resolved against the profile it names (default: the engine name).
type Credentials ¶
type Credentials struct {
User string
Password string
Host string
Port int
TLS TLSSpec
// Source records where the password came from ("job", "profile:<name>", "none")
Source string
}
Credentials are the resolved connection settings for one run
func ResolveCredentials ¶
func ResolveCredentials(spec *Spec, password string, profiles map[string]CredentialProfile) (Credentials, error)
ResolveCredentials merges the job spec with the agent profile. password is the server-stored password for this run ("" when the job has none). The profile is consulted only when the job carries no password; it may also fill in user, host, port and TLS settings the job left blank.
type DatabaseEntry ¶
type JobState ¶
type JobState struct {
JobID string `json:"job_id"`
Engine string `json:"engine"`
LastSuccess *JobStateRecord `json:"last_success,omitempty"`
}
JobState is persisted per job on the agent after every successful run
type JobStateRecord ¶
type JobStateRecord struct {
RunID string `json:"run_id"`
Snapshot string `json:"snapshot"`
CompletedAt time.Time `json:"completed_at"`
MySQL *MySQLState `json:"mysql,omitempty"`
Postgres *PostgresState `json:"postgres,omitempty"`
}
type K8sTarget ¶ added in v0.9.5
type K8sTarget struct {
Namespace string `json:"namespace"`
// Service is the Service the dump pod connects to (<service>.<namespace>.svc);
// Spec.Host, when set, is used verbatim instead
Service string `json:"service,omitempty"`
// SecretName names a Secret in Namespace holding the credentials: PasswordKey
// (default "password") and optionally UserKey. A password stored on the job wins.
SecretName string `json:"secret_name,omitempty"`
PasswordKey string `json:"password_key,omitempty"`
UserKey string `json:"user_key,omitempty"`
// ToolsImage overrides the client-tools image of the dump pod (default: the agent's
// per-engine image template with the tag taken from the detected server version)
ToolsImage string `json:"tools_image,omitempty"`
}
K8sTarget places the database inside a Kubernetes cluster. Such a job is assigned to a kubernetes agent, which dumps the database from a short-lived pod in Namespace that runs the engine's own client image.
type Manifest ¶
type Manifest struct {
SchemaVersion int `json:"schema_version"`
Generator ManifestGenerator `json:"generator"`
Job ManifestJob `json:"job"`
Snapshot ManifestSnapshot `json:"snapshot"`
Engine string `json:"engine"`
StartedAt time.Time `json:"started_at"`
FinishedAt time.Time `json:"finished_at"`
// ConsistentAsOf is the point in time the backup represents
ConsistentAsOf time.Time `json:"consistent_as_of"`
Server ManifestServer `json:"server"`
Tools map[string]Tool `json:"tools"`
Spec *Spec `json:"spec"`
Databases []DatabaseEntry `json:"databases"`
Files []FileEntry `json:"files"`
Warnings []string `json:"warnings,omitempty"`
Notes []string `json:"notes,omitempty"`
Postgres *PostgresManifest `json:"postgres,omitempty"`
MySQL *MySQLManifest `json:"mysql,omitempty"`
MongoDB *MongoManifest `json:"mongodb,omitempty"`
Restore Restore `json:"restore"`
}
Manifest describes one database backup run; it is written into the archive root and uploaded as a separate blob so restores can start from it.
func ParseManifest ¶
ParseManifest decodes a manifest
type ManifestGenerator ¶
type ManifestJob ¶
type ManifestServer ¶
type ManifestSnapshot ¶
type MongoManifest ¶
type MongoOptions ¶
type MongoOptions struct {
// Oplog captures the oplog during the dump (point-in-time, requires a replica set and
// dumping all databases); default true when no database list is given
Oplog *bool `json:"oplog,omitempty"`
ReadPreference string `json:"read_preference,omitempty"`
AuthSource string `json:"auth_source,omitempty"` // default admin
ReplicaSet string `json:"replica_set,omitempty"`
NumParallelCollections int `json:"num_parallel_collections,omitempty"`
ExtraArgs []string `json:"extra_args,omitempty"`
}
MongoOptions are the MongoDB-specific job settings
type MySQLBinlog ¶
type MySQLBinlog struct {
Enabled bool `json:"enabled"`
PositionSource string `json:"position_source,omitempty"`
File string `json:"file,omitempty"`
Position int64 `json:"position,omitempty"`
GTID string `json:"gtid,omitempty"`
GTIDMode string `json:"gtid_mode,omitempty"`
ServerID int64 `json:"server_id,omitempty"`
ServerUUID string `json:"server_uuid,omitempty"`
Previous *BinlogPrevious `json:"previous,omitempty"`
Files []FileEntry `json:"files,omitempty"`
CoversFrom *BinlogPosition `json:"covers_from,omitempty"`
CoversTo *BinlogPosition `json:"covers_to,omitempty"`
ChainReset bool `json:"chain_reset"`
ChainGap bool `json:"chain_gap"`
}
type MySQLManifest ¶
type MySQLManifest struct {
Flavor string `json:"flavor"`
DumpFiles []string `json:"dump_files"`
AllDatabases bool `json:"all_databases"`
PerDatabase bool `json:"per_database"`
SingleTransaction bool `json:"single_transaction"`
FlushLogs bool `json:"flush_logs"`
NonTransactionalTables int `json:"non_transactional_tables"`
Binlog *MySQLBinlog `json:"binlog,omitempty"`
}
type MySQLOptions ¶
type MySQLOptions struct {
SingleTransaction *bool `json:"single_transaction,omitempty"` // default true
// IncludeBinlogs fetches the binary logs written since the previous successful run
IncludeBinlogs bool `json:"include_binlogs,omitempty"`
FlushLogs *bool `json:"flush_logs,omitempty"` // default = IncludeBinlogs
// IncludeSystemSchemas also dumps the mysql schema when an explicit list is given
IncludeSystemSchemas bool `json:"include_system_schemas,omitempty"`
// PerDatabase dumps every database to its own file instead of one --all-databases
// file (the mysql schema included), so a single database can be restored on its
// own; the dumps are taken one after another and are not mutually consistent.
// An explicit database or exclude list always dumps per database.
PerDatabase bool `json:"per_database,omitempty"`
Routines *bool `json:"routines,omitempty"` // default true
Triggers *bool `json:"triggers,omitempty"` // default true
Events *bool `json:"events,omitempty"` // default true
ExtraDumpArgs []string `json:"extra_dump_args,omitempty"`
}
MySQLOptions are the MySQL/MariaDB-specific job settings
type MySQLState ¶
type PostgresBasebackup ¶
type PostgresBasebackup struct {
Dir string `json:"dir"`
Files []string `json:"files"`
Label string `json:"label"`
Timeline int64 `json:"timeline,omitempty"`
WALStartLSN string `json:"wal_start_lsn,omitempty"`
WALEndLSN string `json:"wal_end_lsn,omitempty"`
WALStartFile string `json:"wal_start_file,omitempty"`
WALEndFile string `json:"wal_end_file,omitempty"`
WALSource string `json:"wal_source,omitempty"`
Compression string `json:"compression"`
}
type PostgresGlobals ¶
type PostgresManifest ¶
type PostgresManifest struct {
Mode string `json:"mode"`
Globals *PostgresGlobals `json:"globals,omitempty"`
Basebackup *PostgresBasebackup `json:"basebackup,omitempty"`
// WAL position of the server when the logical dumps were taken
WALLSN string `json:"wal_lsn,omitempty"`
}
type PostgresOptions ¶
type PostgresOptions struct {
Mode string `json:"mode,omitempty"`
// Format is "custom" (default, pg_restore) or "plain" (psql)
Format string `json:"format,omitempty"`
// Globals dumps roles and tablespaces with pg_dumpall --globals-only (default true)
Globals *bool `json:"globals,omitempty"`
// RolePasswords: "auto" (include when superuser), "include", "omit"
RolePasswords string `json:"role_passwords,omitempty"`
ExtraDumpArgs []string `json:"extra_dump_args,omitempty"`
}
PostgresOptions are the PostgreSQL-specific job settings
type PostgresState ¶
type Spec ¶
type Spec struct {
Engine string `json:"engine"`
// Host is a hostname/IP, or a path (leading "/") for a local unix socket; optional
// for kubernetes targets that name a Service
Host string `json:"host"`
Port int `json:"port,omitempty"`
User string `json:"user,omitempty"`
// Password is stored server-side; blank means the agent resolves it from Profile
Password string `json:"password,omitempty"`
// HasPassword is set on API responses instead of the password itself
HasPassword bool `json:"has_password,omitempty"`
// ClearPassword (request only) removes a stored password on update
ClearPassword bool `json:"clear_password,omitempty"`
// Profile names the agent-side credential profile (default: the engine name)
Profile string `json:"profile,omitempty"`
TLS TLSSpec `json:"tls"`
Databases []string `json:"databases,omitempty"`
ExcludeDatabases []string `json:"exclude_databases,omitempty"`
Compression string `json:"compression,omitempty"`
// BackupID overrides the PBS backup group id (default db-<engine>-<host>)
BackupID string `json:"backup_id,omitempty"`
Postgres *PostgresOptions `json:"postgres,omitempty"`
MySQL *MySQLOptions `json:"mysql,omitempty"`
MongoDB *MongoOptions `json:"mongodb,omitempty"`
// Kubernetes is set when the job runs on a kubernetes agent
Kubernetes *K8sTarget `json:"kubernetes,omitempty"`
}
Spec is the database job configuration stored server-side (backup_jobs.db_config) and handed to the agent (with the password redacted) for every run.
func (*Spec) EffectiveHost ¶ added in v0.9.5
EffectiveHost is the address the dump connects to: Host, or for a kubernetes target without an explicit host the Service's cluster DNS name
func (*Spec) EffectivePort ¶
EffectivePort returns the configured port or the engine default
func (*Spec) Redacted ¶
Redacted returns a copy without the password (HasPassword tells whether one is stored)
type TLSSpec ¶
type TLSSpec struct {
Mode string `json:"mode,omitempty" yaml:"mode,omitempty"`
CAFile string `json:"ca_file,omitempty" yaml:"ca_file,omitempty"`
CertFile string `json:"cert_file,omitempty" yaml:"cert_file,omitempty"`
KeyFile string `json:"key_file,omitempty" yaml:"key_file,omitempty"`
}
TLSSpec describes how the agent connects to the database server. File paths refer to the agent host.
type ToolConfig ¶
type ToolConfig struct {
PostgresBinDir string `yaml:"postgres_bindir,omitempty" json:"postgres_bindir,omitempty"`
MySQLBinDir string `yaml:"mysql_bindir,omitempty" json:"mysql_bindir,omitempty"`
MongoBinDir string `yaml:"mongo_bindir,omitempty" json:"mongo_bindir,omitempty"`
}
ToolConfig pins the client tool directories (default: PATH and well-known locations)