dump

package
v0.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 19, 2026 License: MIT Imports: 23 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	CompressFormatGzip = "gzip"
	CompressFormatZstd = "zstd"
)

Compression formats accepted by --compress-format.

View Source 
const BufferTypeFile = "file"

Variables

This section is empty.

Functions

func CompressExtension 

func CompressExtension(format string) string

CompressExtension returns the file extension for a compression format.

func GetDropTableIfExistSQL 

func GetDropTableIfExistSQL(table string) string

func GetLockAllTablesSQL 

func GetLockAllTablesSQL() string

func GetLockTablesSQL 

func GetLockTablesSQL(tasksPool []*Task, mode string) string

func GetMySQLConnection 

func GetMySQLConnection(host *MySQLHost, credentials *MySQLCredentials) (*sql.DB, error)

GetMySQLConnection opens and verifies a MySQL connection. Uses mysql.Config.FormatDSN so passwords with special characters are handled correctly.

func GetShowCreateTableSQL 

func GetShowCreateTableSQL(table string) string

func GetUseDatabaseSQL 

func GetUseDatabaseSQL(schema string) string

func NormalizeTableName 

func NormalizeTableName(tableName string) string

NormalizeTableName converts "schema.table" to "`schema`.`table`".

func ParseIniFile 

func ParseIniFile(iniFile string, do *DumpOptions, flagSet map[string]bool)

ParseIniFile loads opts from an INI file, skipping keys already set via flags.

func ParseString 

func ParseString(s interface{}) []byte

ParseString escapes a byte slice for safe use inside single-quoted SQL literals. Uses ” for single-quote (compatible with NO_BACKSLASH_ESCAPES).

func ParseWhereCondition 

func ParseWhereCondition(whereValue string, do *DumpOptions)

ParseWhereCondition parses a --where value and populates opts. Format: "condition" (global) or "db.tbl:condition,db.tbl2:condition2" (per-table).

func Run 

func Run(ctx context.Context, opts *DumpOptions)

Run executes the dump. Callers are responsible for validating opts before calling.

func RunChecksums 

func RunChecksums(tasks []*Task, db *sql.DB, dest string, meta *DumpMetadata) error

RunChecksums runs CHECKSUM TABLE for every task and writes <dest>/checksums.txt. It also updates the metadata file with each table's checksum value. This runs after UNLOCK TABLES so it does not extend the lock window.

func StripDefiner 

func StripDefiner(stmt string) string

StripDefiner removes the DEFINER=... clause from a CREATE statement so the object loads on servers where the definer account does not exist. The object is then created with the invoker as definer.

func TablesFromAllDatabases 

func TablesFromAllDatabases(db *sql.DB, includeSystem bool) map[string]bool

TablesFromAllDatabases returns all base tables across non-system databases. By default it excludes mysql, sys, information_schema, and performance_schema — safe for both on-prem and Cloud SQL (which manages mysql.* via IAM). Pass includeSystem=true to include mysql (minus slow_log/general_log) for on-prem full-cluster migrations that need to transfer accounts.

func TablesFromDatabase 

func TablesFromDatabase(databases string, db *sql.DB) map[string]bool

TablesFromDatabase returns all base tables in the given comma-separated list of databases. Fixed: was using defer inside a loop which held all connections open simultaneously.

func TablesFromString 

func TablesFromString(tablesParam string) map[string]bool

TablesFromString parses a comma-separated "schema.table" list.

func VerifyChecksums 

func VerifyChecksums(dest string, db *sql.DB) error

VerifyChecksums reads <dest>/checksums.txt and compares each entry against CHECKSUM TABLE on the provided database connection. Returns any mismatches as errors.

Types

type Buffer 

type Buffer struct {
	Type           string
	Buffer         *bufio.Writer
	Compressor     compressor
	FileDescriptor *os.File
}

Buffer is the default struct to write the data.

func NewBuffer 

func NewBuffer(options *BufferOptions) (*Buffer, error)

func NewChunkBuffer 

func NewChunkBuffer(c *DataChunk, workerId int) (*Buffer, error)

func NewFileBuffer 

func NewFileBuffer(fileName string, compress bool, compressFormat string, compressLevel int) *Buffer

func NewMasterDataBuffer 

func NewMasterDataBuffer(t *TaskManager) (*Buffer, error)

func NewSlaveDataBuffer 

func NewSlaveDataBuffer(t *TaskManager) (*Buffer, error)

func NewTableDefinitionBuffer 

func NewTableDefinitionBuffer(t *Task) (*Buffer, error)

func (*Buffer) Close 

func (b *Buffer) Close() error

func (*Buffer) Flush 

func (b *Buffer) Flush() error

func (*Buffer) Write 

func (b *Buffer) Write(p []byte) (int, error)

type BufferOptions 

type BufferOptions struct {
	Compress       bool
	CompressFormat string
	CompressLevel  int
	Type           string
	Path           string
}

type ChecksumResult 

type ChecksumResult struct {
	Schema    string
	Name      string
	Checksum  int64
	Timestamp time.Time
}

ChecksumResult holds the result of a single CHECKSUM TABLE call.

type ColumnsMap 

type ColumnsMap map[string]int

type DataChunk 

type DataChunk struct {
	Min           int64
	Max           int64
	Sequence      uint64
	Task          *Task
	IsSingleChunk bool
	IsLastChunk   bool
}

DataChunk holds the information for one read chunk of a table.

func NewDataChunk 

func NewDataChunk(task *Task) DataChunk

func NewDataLastChunk 

func NewDataLastChunk(task *Task) DataChunk

func NewSingleDataChunk 

func NewSingleDataChunk(task *Task) DataChunk

func (*DataChunk) GetOrderBYSQL 

func (dc *DataChunk) GetOrderBYSQL() string

func (*DataChunk) GetPrepareSQL 

func (dc *DataChunk) GetPrepareSQL() string

func (*DataChunk) GetSampleSQL 

func (dc *DataChunk) GetSampleSQL() string

func (*DataChunk) GetWhereSQL 

func (dc *DataChunk) GetWhereSQL() string

GetWhereSQL returns the WHERE clause for this chunk, merging chunking bounds with any per-table or global WHERE condition from DumpOptions.

func (*DataChunk) Parse 

func (dc *DataChunk) Parse(stmt *sql.Stmt, w io.Writer) error

Parse executes the chunk query and writes the resulting INSERT statements to w. Callers stage the output in memory so a failed chunk can be retried without duplicating rows already written to the dump file.

Rows are grouped into multi-row INSERTs, split on whichever limit is hit first: OutputChunkSize rows, or StatementSize bytes. The byte cap exists because wide rows (large TEXT/BLOB columns) can push a row-count-only group past the target server's max_allowed_packet, producing a dump that cannot be restored anywhere. The cap is checked at row boundaries, so a statement can exceed it by at most one row.

type DumpMetadata 

type DumpMetadata struct {
	GoDumpVersion  string           `json:"go_dump_version"`
	StartTime      time.Time        `json:"start_time"`
	EndTime        *time.Time       `json:"end_time,omitempty"`
	Status         string           `json:"status"` // "in_progress" | "complete" | "failed"
	MySQLHost      string           `json:"mysql_host"`
	MySQLPort      int              `json:"mysql_port,omitempty"`
	MySQLVersion   string           `json:"mysql_version"`
	BinlogFile     string           `json:"binlog_file,omitempty"`
	BinlogPosition int              `json:"binlog_position,omitempty"`
	GTIDSet        string           `json:"gtid_set,omitempty"`
	CharacterSet   string           `json:"character_set"`
	Objects        map[string]int   `json:"objects,omitempty"` // "triggers"/"routines"/"events" -> count dumped
	Tables         []*TableMetadata `json:"tables"`
	// contains filtered or unexported fields
}

DumpMetadata is written to <destination>/metadata.json throughout the dump run. It is the source of truth for resume logic and restore verification.

func LoadDumpMetadata 

func LoadDumpMetadata(destDir string) (*DumpMetadata, error)

LoadDumpMetadata reads an existing metadata.json for resume logic.

func NewDumpMetadata 

func NewDumpMetadata(destDir, appVersion, mysqlHost string, mysqlPort int, mysqlVersion string) *DumpMetadata

NewDumpMetadata initialises a metadata object and writes the initial in_progress file.

func (*DumpMetadata) AddTable 

func (m *DumpMetadata) AddTable(schema, name string, rowEstimate uint64)

AddTable registers a table in the metadata with pending status.

func (*DumpMetadata) Complete 

func (m *DumpMetadata) Complete()

Complete marks the dump finished and writes the final metadata file.

func (*DumpMetadata) Fail 

func (m *DumpMetadata) Fail()

Fail marks the dump as failed and writes the metadata file.

func (*DumpMetadata) MarkTableDone 

func (m *DumpMetadata) MarkTableDone(schema, name string, chunks uint64)

MarkTableDone updates a table's status and chunk count, then flushes the file.

func (*DumpMetadata) MergeDoneTables 

func (m *DumpMetadata) MergeDoneTables(prior *DumpMetadata)

MergeDoneTables copies tables recorded as done in a prior run's metadata into m, so a resumed run's metadata (and checksums) still describe the entire dump set on disk — not just the tables re-dumped in this run.

func (*DumpMetadata) ResumeState 

func (m *DumpMetadata) ResumeState() (done map[string]bool, inProgress map[string]bool)

ResumeState returns two sets derived from a prior run's metadata:

  • done: "schema.table" keys that completed successfully (safe to skip)
  • inProgress: "schema.table" keys that were pending/in_progress (partial files exist)

func (*DumpMetadata) SetBinlog 

func (m *DumpMetadata) SetBinlog(file string, position int, gtidSet string)

SetBinlog records the binlog position captured at the consistent snapshot point.

func (*DumpMetadata) SetChecksum 

func (m *DumpMetadata) SetChecksum(schema, name string, checksum int64)

SetChecksum records the CHECKSUM TABLE result for a table.

func (*DumpMetadata) SetObjects 

func (m *DumpMetadata) SetObjects(counts map[string]int)

SetObjects records how many triggers/routines/events were dumped.

func (*DumpMetadata) TablesSnapshot 

func (m *DumpMetadata) TablesSnapshot() []TableMetadata

TablesSnapshot returns a copy of the current table entries for safe iteration without holding the metadata lock.

func (*DumpMetadata) Write 

func (m *DumpMetadata) Write() error

Write serialises the metadata to a temp file and renames it atomically.

type DumpOptions 

type DumpOptions struct {
	AppVersion            string
	MySQLHost             *MySQLHost
	MySQLCredentials      *MySQLCredentials
	Threads               int
	ChunkSize             uint64
	OutputChunkSize       uint64
	StatementSize         uint64 // max bytes per INSERT statement; a row group is flushed once it exceeds this
	ChannelBufferSize     int
	LockTables            bool
	LockWaitTimeout       int // seconds to wait for FTWRL / LOCK TABLES before aborting
	TablesWithoutUKOption string
	DestinationDir        string
	AddDropTable          bool
	GetMasterStatus       bool
	GetSlaveStatus        bool
	SkipUseDatabase       bool
	Compress              bool
	CompressFormat        string // "gzip" or "zstd"
	CompressLevel         int
	IsolationLevel        sql.IsolationLevel
	Consistent            bool
	Checksum              bool
	Resume                bool
	DumpTriggers          bool
	DumpRoutines          bool // stored procedures and functions
	DumpEvents            bool
	SkipDefiner           bool              // strip DEFINER=... from trigger/routine/event definitions
	WhereConditions       map[string]string // table -> where condition
	GlobalWhereCondition  string            // fallback for all tables
	TemporalOptions       TemporalOptions
}

func GetDumpOptions 

func GetDumpOptions() *DumpOptions

GetDumpOptions returns a new DumpOptions with production-safe defaults.

type MySQLCredentials 

type MySQLCredentials struct {
	User     string
	Password string
}

type MySQLHost 

type MySQLHost struct {
	HostName   string
	SocketFile string
	Port       int
}

type Table 

type Table struct {
	CreateTableSQL string
	IsLocked       bool
	Engine         string
	Collation      string
	// contains filtered or unexported fields
}

Table contains the name and metadata of a MySQL table.

func NewTable 

func NewTable(schema string, name string, db *sql.DB) *Table

NewTable creates a Table and loads its metadata from MySQL.

func (*Table) GetFullName 

func (t *Table) GetFullName() string

func (*Table) GetName 

func (t *Table) GetName() string

func (*Table) GetPrimaryOrUniqueKey 

func (t *Table) GetPrimaryOrUniqueKey() string

GetPrimaryOrUniqueKey returns the single-column key used for chunking. Empty string means no usable key exists.

func (*Table) GetSchema 

func (t *Table) GetSchema() string

func (*Table) GetUnescapedFullName 

func (t *Table) GetUnescapedFullName() string

func (*Table) GetUnescapedName 

func (t *Table) GetUnescapedName() string

func (*Table) GetUnescapedSchema 

func (t *Table) GetUnescapedSchema() string

type TableMetadata 

type TableMetadata struct {
	Schema      string      `json:"schema"`
	Name        string      `json:"name"`
	RowEstimate uint64      `json:"row_estimate"`
	Chunks      uint64      `json:"chunks"`
	Status      TableStatus `json:"status"`
	Checksum    int64       `json:"checksum,omitempty"`
}

TableMetadata tracks per-table dump state. Used for resume and checksum verification.

type TableStatus 

type TableStatus string
const (
	TableStatusPending TableStatus = "pending"
	TableStatusDone    TableStatus = "done"
	TableStatusFailed  TableStatus = "failed"
)

type Task 

type Task struct {
	Table           *Table
	ChunkSize       uint64
	OutputChunkSize uint64
	TaskManager     *TaskManager
	Id              int64
	TotalChunks     uint64

	// Per-table completion tracking so a table can be marked done in
	// metadata.json as soon as its last chunk is written — not only at the end
	// of the whole dump. This is what gives --resume per-table granularity.
	ChunksCompleted int64 // incremented atomically by workers
	// contains filtered or unexported fields
}

func NewTask 

func NewTask(schema string, table string, chunkSize uint64, outputChunkSize uint64, tm *TaskManager) Task

func (*Task) AddChunk 

func (t *Task) AddChunk(chunk DataChunk)

func (*Task) CreateChunks 

func (t *Task) CreateChunks(db *sql.DB)

func (*Task) GetChunkSqlQuery 

func (t *Task) GetChunkSqlQuery() string

func (*Task) GetLastChunkSqlQuery 

func (t *Task) GetLastChunkSqlQuery() string

func (*Task) GetMinKeyQuery 

func (t *Task) GetMinKeyQuery() string

GetMinKeyQuery returns the query for the smallest key value, used to seed the chunk bounds.

func (*Task) GetSingleChunkTestQuery 

func (t *Task) GetSingleChunkTestQuery() string

func (*Task) NoteChunkCompleted 

func (t *Task) NoteChunkCompleted()

NoteChunkCompleted records one finished chunk and marks the table done in the dump metadata once chunk creation has finished and every chunk is written.

func (*Task) PrintInfo 

func (t *Task) PrintInfo()

type TaskManager 

type TaskManager struct {
	CreateChunksWaitGroup  *sync.WaitGroup
	ProcessChunksWaitGroup *sync.WaitGroup
	ChunksChannel          chan DataChunk
	DB                     *sql.DB
	ThreadsCount           int

	TotalChunks           int64
	Queue                 int64
	CompletedChunks       int64 // incremented atomically by workers as each chunk finishes
	StartTime             time.Time
	DestinationDir        string
	TablesWithoutPKOption string
	SkipUseDatabase       bool
	GetMasterStatus       bool
	GetSlaveStatus        bool
	Compress              bool
	CompressFormat        string
	CompressLevel         int
	IsolationLevel        sql.IsolationLevel

	DumpOptions *DumpOptions

	// Binlog info captured by getMasterData, written to metadata.
	BinlogFile     string
	BinlogPosition int
	GTIDSet        string
	// contains filtered or unexported fields
}

func NewTaskManager 

func NewTaskManager(
	wgC *sync.WaitGroup,
	wgP *sync.WaitGroup,
	cDC chan DataChunk,
	db *sql.DB,
	dumpOptions *DumpOptions) TaskManager

func (*TaskManager) AddChunk 

func (tm *TaskManager) AddChunk(chunk DataChunk)

func (*TaskManager) AddTask 

func (tm *TaskManager) AddTask(t *Task)

func (*TaskManager) AddWorkerDB 

func (tm *TaskManager) AddWorkerDB(db *sql.DB)

func (*TaskManager) AddWorkersDB 

func (tm *TaskManager) AddWorkersDB()

func (*TaskManager) CleanChunkChannel 

func (tm *TaskManager) CleanChunkChannel()

func (*TaskManager) CreateChunks 

func (tm *TaskManager) CreateChunks(db *sql.DB)

func (*TaskManager) DisplaySummary 

func (tm *TaskManager) DisplaySummary() error

func (*TaskManager) GetBufferOptions 

func (tm *TaskManager) GetBufferOptions() *BufferOptions

func (*TaskManager) GetTasksPool 

func (tm *TaskManager) GetTasksPool() []*Task

func (*TaskManager) GetTransactions 

func (tm *TaskManager) GetTransactions(lockTables bool, allDatabases bool)

func (*TaskManager) MySQLVersion 

func (tm *TaskManager) MySQLVersion() string

MySQLVersion returns the detected server version as a "X.Y.Z" string.

func (*TaskManager) PrintStatus 

func (tm *TaskManager) PrintStatus()

PrintStatus logs progress every 5 seconds until the queue drains or ctx is cancelled. Output: "Progress: 450/1500 (30.0%) | Rate: 12.3 chunks/s | ETA: ~1m42s"

func (*TaskManager) SetContext 

func (tm *TaskManager) SetContext(ctx context.Context)

SetContext sets the context used for transactions and DB operations.

func (*TaskManager) SetMetadata 

func (tm *TaskManager) SetMetadata(meta *DumpMetadata)

SetMetadata attaches the dump metadata so workers can mark tables done as they finish. Must be called before StartWorkers/CreateChunks to avoid races.

func (*TaskManager) StartWorker 

func (tm *TaskManager) StartWorker(workerId int)

func (*TaskManager) StartWorkers 

func (tm *TaskManager) StartWorkers() error

func (*TaskManager) WriteObjectsSQL 

func (tm *TaskManager) WriteObjectsSQL() map[string]int

WriteObjectsSQL dumps triggers, routines, and events per the Dump* options. It returns per-kind counts of objects written, or nil when no object dumping was requested. Discovery or SHOW CREATE failures (typically missing privileges) skip the object with a warning; file-write failures abort the dump, exactly like the schema files.

func (*TaskManager) WriteSchemaCreateSQL 

func (tm *TaskManager) WriteSchemaCreateSQL()

GetTransactions establishes the worker snapshots that define the backup's point in time.

Consistent path (lockTables=true): one dedicated connection acquires FLUSH TABLES WITH READ LOCK (--all-databases, or when non-InnoDB tables are present) or LOCK TABLES ... READ (InnoDB-only table list). While the lock is held, every worker issues START TRANSACTION WITH CONSISTENT SNAPSHOT and the binlog coordinates are captured — so all snapshots and the recorded position refer to the same instant. The lock is released immediately afterwards; the window is typically milliseconds.

Limitation: non-InnoDB tables are write-protected only during the lock window. Their data is read later without MVCC, so writes occurring after the unlock can appear in the dump. Holding the lock for the whole dump (mydumper's --no-locks=false behaviour for non-transactional tables) is not implemented. WriteSchemaCreateSQL writes one <schema>-schema-create.sql file per dumped schema so the dump restores onto a server where the database does not exist yet. SHOW CREATE DATABASE preserves charset/collation; IF NOT EXISTS is added mysqldump-style so restoring into an existing database is a no-op.

func (*TaskManager) WriteTablesSQL 

func (tm *TaskManager) WriteTablesSQL(addDropTable bool)

type TemporalOptions 

type TemporalOptions struct {
	Tables, Databases, IsolationLevel           string
	AllDatabases, Debug, DryRun, Execute, Quiet bool
	IncludeSystemDatabases                      bool
}

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.