README
¶
move
The move package implements multi-table database migration for MoveTables and resharding operations. While the migration package performs online schema changes (ALTER TABLE on a single table within the same server), move copies one or more tables between different MySQL servers, optionally resharding data across multiple targets.
Lifecycle
A move operation follows this sequence:
- Pre-run checks — validate configuration before opening database connections.
- Setup — discover source tables, create targets, run preflight/post-setup checks.
- Copy rows — bulk-copy all source tables to their targets using the copier.
- Initial checksum (post-copy phase) — flush the binlog backlog, restore any secondary indexes that were deferred during table creation (see
DeferSecondaryIndexesbelow), run ANALYZE TABLE, and verify data consistency between source and targets. This is the correctness gate; cutover does not proceed unless this checksum succeeds. - Sentinel wait — optionally pause before cutover, allowing external orchestration to confirm readiness. While the sentinel blocks cutover, a continuous checksum loop runs in the background to keep re-verifying the data; the loop is interrupted as soon as the sentinel is dropped.
- Cutover — acquire a table lock, flush remaining replication changes, execute the caller-provided cutover function, and rename original tables out of the way.
- Reverse window (optional) — when
ReverseWindowis set, the move does not exit after cutover; it holds a change-only reverse feed (targets → the source's_oldtables) for the configured duration so the move can be rolled back. See Reverse Window below.
Design Decisions
Multi-table Awareness
Unlike migration which operates on a single table, move discovers and copies all tables in a source database (or a specified subset). A single replication client tracks changes across all tables, and the cutover renames all source tables atomically.
Sharding Support
When a ShardingProvider is configured, each source table is annotated with a sharding column and hash function during discovery. The applier uses this metadata to route rows to the correct target based on key ranges. Without a sharding provider, the operation is a simple 1:1 move to a single target.
Deferred Secondary Indexes
The DeferSecondaryIndexes option creates target tables without secondary indexes, adding them back before cutover. This can significantly speed up the bulk copy phase since index maintenance is avoided until the data is in place.
However, this optimization is not always safe:
- You can run out of temporary disk space
- The History List Length will grow during re-adding indexes due to MySQL bug #113476.
Checkpoint and Resume
Move operations write periodic checkpoints to a _spirit_move_checkpoint table on the first target. If a move is interrupted, the runner detects the existing checkpoint during setup and resumes from the last recorded binlog position rather than starting over. DDL changes on source tables during a move invalidate the checkpoint to prevent resuming into an inconsistent state. The name is intentionally distinct from migration's shared _spirit_checkpoint and datasync's _spirit_sync_checkpoint — see below for why that distinctness matters.
The checkpoint records progress into the targets (the copier watermark, and the rows deleteAboveWatermark prunes on resume), so it lives alongside the data it describes. With N sources and M targets, both slices are sorted deterministically and the checkpoint always lands on the first target (targets[0]), so a resume looks in the same place regardless of the order the caller supplied sources or targets. (Earlier 1:N reshard versions stored the checkpoint on the single source; that convention is no longer used.)
A run interrupted before its first checkpoint dump leaves the checkpoint table present but empty. Because _spirit_move_checkpoint is uniquely named, only a move can have created it — transiently, after the target tables passed the empty-target validation and before any row is copied — so everything on the target is that attempt's partial copy, and a re-run recovers automatically: it wipes the target tables and starts a fresh copy, no --force needed. The unique name is what makes this safe: a leftover _spirit_checkpoint from an unrelated migration is not proof of a dead move, so move never keys ownership on it. --force remains for the states spirit cannot prove it owns (a non-empty target with no move checkpoint table, or a checkpoint written by an incompatible spirit version).
Sentinel Table
When CreateSentinel is enabled, the runner creates a _spirit_sentinel table on the first target (targets[0], alongside the checkpoint) during setup (before the copy starts) and then blocks before cutover until it is dropped by an external actor. The wait sits between the initial checksum and the cutover. This provides a coordination point for orchestration systems that need to perform additional steps between copy completion and cutover.
While the sentinel blocks the cutover, the runner re-runs the checksum in a loop (the "continuous checksum") so that the data is re-verified close to the moment of cutover, even if the sentinel sits for hours. The first iteration starts one hour after the initial checksum, and subsequent iterations are capped at one per hour so that small tables do not churn the table lock back-to-back; the wait is interrupted when the sentinel is dropped. One exception: if a pass had already detected a mismatch and is mid-recopy, the in-flight repair runs to completion (bounded by an internal per-chunk timeout) before cutover continues, because the DELETE-from-targets + re-apply-from-sources pair must stay atomic. See docs/move.md for the user-facing description.
Cutover Function
The cutover is split into two parts: a caller-provided function and a table rename. The caller's function runs under a table lock with all replication changes flushed, giving it a consistent view. If it succeeds, the runner renames the original tables to _old suffixes.
This separation allows orchestration systems to perform routing changes, such as updating a DNS server or Vitess topology server as part of the atomic cutover.
Reverse Window
ReverseWindow (the --reverse-window flag) makes a cutover reversible for a bounded period. Instead of exiting after the cutover rename, the runner stands up a change-only reverse feed — the former targets become change sources and the source's now-retired _old tables become the write target (reversefeed.go, from the reverse-feed foundations) — and holds it for the window (reversewindow.go). The feed's start position is captured in the cutover's postSwitch hook, under the source lock, so no target write is missed.
The window ends in one of three ways: it elapses (finalize forward — the source stays retired, checkpoint dropped), the feed dies (finalize forward — rollback is no longer safe), or a rollback is requested.
A rollback is requested by creating a _spirit_move_revert marker table on the first target (revertmarker.go), mirroring the sentinel but with inverted polarity (the operator creates it to act, rather than dropping it to proceed). The reverse cutover mirrors the forward one with roles swapped, plus one asymmetry: the source's _old tables are renamed back to their real names before traffic returns. The former targets are then retired to a _revert suffix — distinct from _old so a subsequent move can recognize and drop them (dropStaleRevertTables clears leftovers, making revert→retry idempotent). A separate reverseCutoverFunc (registered via SetReverseCutover, the mirror of the cutover function above) performs the routing switch back to the source.
Two constraints and one resume note:
- Unsharded source only — reverse-window is a 1→M forward move reversed as M→1; a sharded source would need an M:N reverse and is rejected at startup.
- Stale-marker guard — a
_spirit_move_revertpresent at pre-flight or pre-cutover aborts the run, so a leftover from an interrupted rollback is never read as a fresh request. - Resume — the checkpoint gains
move_phase(reverse_window/reverting) andcutover_atcolumns, so a move killed during the window resumes back into it rather than re-copying. Therevertingphase (mid-rollback) is not auto-resumed and must be completed manually.
Documentation
¶
Index ¶
- type CutOver
- type CutOverSource
- type Move
- type ReverseFeed
- type ReverseFeedConfig
- type ReverseSource
- type Runner
- func (r *Runner) Cancel()
- func (r *Runner) Close() error
- func (r *Runner) DumpCheckpoint(ctx context.Context) error
- func (r *Runner) Progress() status.Progress
- func (r *Runner) Run(ctx context.Context) error
- func (r *Runner) SetCutover(cutover func(ctx context.Context) error)
- func (r *Runner) SetLogger(logger *slog.Logger)
- func (r *Runner) SetReverseCutover(fn func(ctx context.Context) error)
- func (r *Runner) Status() string
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type CutOver ¶
type CutOver struct {
// contains filtered or unexported fields
}
type CutOverSource ¶ added in v0.13.0
CutOverSource holds per-source state needed for the cutover.
type Move ¶
type Move struct {
SourceDSN string `name:"source-dsn" help:"Where to copy the tables from." default:"spirit:spirit@tcp(127.0.0.1:3306)/src"`
TargetDSN string `name:"target-dsn" help:"Where to copy the tables to." default:"spirit:spirit@tcp(127.0.0.1:3306)/dest"`
TargetChunkTime time.Duration `` /* 156-byte string literal not displayed */
// TargetChunkSize is the in-memory byte budget the buffered copier sizes each
// copy chunk against (see table.DefaultTargetChunkBytes). Move always uses the
// buffered copier. A zero value means "use the default" (NewRunner fills it
// in). The Kong default below must stay equal to table.DefaultTargetChunkBytes.
TargetChunkSize uint64 `name:"target-chunk-size" help:"In-memory byte budget per copy chunk (in bytes)." default:"16777216"`
Threads int `name:"threads" help:"How many chunks to copy in parallel" default:"2"`
WriteThreads int `name:"write-threads" help:"How many concurrent write threads to use per target" default:"4"`
CreateSentinel bool `name:"create-sentinel" help:"Create a sentinel table on the first target database to block after table copy" default:"false"`
DeferSecondaryIndexes bool `name:"defer-secondary-indexes" help:"Create target tables without secondary indexes, add them before cutover" default:"false"`
CheckpointMaxAge time.Duration `name:"checkpoint-max-age" help:"Maximum age of a checkpoint before refusing to resume from it" optional:"" default:"168h"`
// Force makes the runner wipe the target tables and start the copy fresh when
// it cannot resume from a checkpoint (e.g. the checkpoint is from an
// incompatible spirit version, or the target is in a state resume can't
// validate). Without it, an unresumable non-empty target is a hard error.
Force bool `` /* 152-byte string literal not displayed */
// ReverseWindow, when > 0, keeps the move alive after cutover in change-only
// reverse mode (targets→source) for this long, so the move can be rolled back
// before the source is retired. 0 (the default) is a normal cutover. During
// the window the source's now-retired _old tables are kept current from the
// targets; an operator rolls back by creating the _spirit_move_revert table on
// the first target (see revertmarker.go), otherwise the window elapses and the
// move finalizes forward. A sharded (multi-DSN) source additionally requires
// ReverseShardingProvider and SourceKeyRanges so the reverse feed can route
// rows back to the correct source shard — see the guard in Runner.Run. The
// data plane is ReverseFeed (reversefeed.go); the post-cutover driver is
// reverseWindow (reversewindow.go).
ReverseWindow time.Duration `` /* 166-byte string literal not displayed */
// EnableExperimentalGTID switches the change source from binlog file+position to MySQL GTIDs.
// EXPERIMENTAL — see pkg/change/gtid.go. Requires gtid_mode=ON and
// enforce_gtid_consistency=ON on every source.
EnableExperimentalGTID bool `` /* 129-byte string literal not displayed */
// SourceTables optionally specifies a list of tables to move.
// If empty, all tables in the source database will be moved.
// This is useful for Vitess MoveTables operations where only specific tables should be moved.
SourceTables []string
// ShardingProvider optionally provides vindex metadata for resharding operations.
// If nil, tables will not have vindex configuration (suitable for simple MoveTables 1:1 operations).
// For resharding operations (1:many), this should be set to provide sharding key information.
// The provider is called during table discovery to configure ShardingColumn and HashFunc
// on each TableInfo.
// SourceDSNs optionally specifies multiple source DSNs for N:M resharding operations.
// When set, each DSN represents a separate source shard. All sources must have identical
// table schemas. If empty, SourceDSN is used as the single source.
SourceDSNs []string `kong:"-"`
// SourceKeyRanges optionally specifies each source shard's Vitess-style key
// range ("-80", "80-", ...), parallel to SourceDSNs (SourceKeyRanges[i] is
// SourceDSNs[i]'s range). Required, together with ReverseShardingProvider,
// when ReverseWindow > 0 and the source is sharded (len(SourceDSNs) > 1):
// the reverse feed routes rows flowing back from the targets to the source
// shard whose range contains the row's hash. Unused otherwise.
SourceKeyRanges []string `kong:"-"`
ShardingProvider table.ShardingMetadataProvider `kong:"-"`
// ReverseShardingProvider provides the SOURCE keyspace's sharding metadata
// (vindex column + hash) for the reverse feed of a reverse-window move with
// a sharded source. It is consulted for each moved table when the window
// opens; a table without metadata is a hard error there, because reverse
// writes could not be routed to a source shard. Note the asymmetry with
// ShardingProvider, which describes the TARGET keyspace for the forward copy.
ReverseShardingProvider table.ShardingMetadataProvider `kong:"-"`
Targets []applier.Target `kong:"-"`
}
func (*Move) Validate ¶ added in v0.15.0
Validate is called by Kong after parsing to check for invalid flag values. Zero values mean "use the default" (NewRunner fills them in), so they are not rejected here; only explicitly-negative or otherwise invalid values are caught. Mirrors migration.Migration.Validate.
type ReverseFeed ¶ added in v0.16.0
type ReverseFeed struct {
// contains filtered or unexported fields
}
ReverseFeed is a running change-only reverse feed: one change.Source per ReverseSource, all sharing a single applier that writes to U (mirrors the forward move, where N sources share one applier).
func NewReverseFeed ¶ added in v0.16.0
func NewReverseFeed(cfg ReverseFeedConfig) (_ *ReverseFeed, err error)
NewReverseFeed wires the feeds and their shared applier. It does not open any binlog stream; call Start or Run for that.
func (*ReverseFeed) Close ¶ added in v0.16.0
func (rf *ReverseFeed) Close()
Close stops periodic flush and closes all feeds. Safe to call more than once. The applier is never Started (the subscription apply path is synchronous), so there is nothing to Stop on it.
func (*ReverseFeed) Err ¶ added in v0.16.0
func (rf *ReverseFeed) Err() error
Err returns the first fatal feed error, if any.
func (*ReverseFeed) Flush ¶ added in v0.16.0
func (rf *ReverseFeed) Flush(ctx context.Context) error
Flush drains all feeds synchronously (e.g. before a health check or a reverse cutover, so U reflects everything written to the sources so far).
func (*ReverseFeed) Positions ¶ added in v0.16.0
func (rf *ReverseFeed) Positions() []string
Positions returns each source's current safe-to-resume position, in the same order as the configured sources. Intended for the caller's checkpoint so the window can resume in reverse mode after a restart.
func (*ReverseFeed) Run ¶ added in v0.16.0
Run opens the feeds and holds the rollback window for the given duration, keeping U current. It returns:
- nil when the window elapses normally (after a final flush);
- ctx.Err() if the context is cancelled;
- a fatal error if any feed dies (rollback is then unsafe and the caller must complete-forward).
Run does not Close the feeds; the caller does that after deciding the terminal action (complete-forward or roll back), since a reverse cutover needs the feeds flushed one last time first.
type ReverseFeedConfig ¶ added in v0.16.0
type ReverseFeedConfig struct {
Sources []ReverseSource
// Target is the U side when the former move source was a single database.
// Target.DB's default database MUST be U's schema (see ReverseSource.DB).
// Mutually exclusive with Targets.
Target applier.Target
// Targets is the U side when the former move source was SHARDED: one entry
// per former source shard, each with its Vitess-style key range set. Rows
// are routed by the WATCHED table's sharding metadata, so every
// ReverseSource table must have ShardingColumn and HashFunc set (the source
// keyspace's vindex) — NewReverseFeed fails otherwise. Each Targets[i].DB's
// default database MUST be that shard's schema (see ReverseSource.DB).
// Mutually exclusive with Target.
Targets []applier.Target
// TargetTables maps each watched (reverse-source) table NAME to the U-side
// TableInfo it is written to, built on Target.DB (with Targets, on any one
// shard: the schemas are identical and the name is unqualified, so each
// shard's own connection determines where the write lands). It is a map,
// not a slice, because the names can differ: after a forward cutover the
// source tables are renamed to their _old form, so a watched "t1" is
// written to "t1_old".
TargetTables map[string]*table.TableInfo
Logger *slog.Logger
DBConfig *dbconn.DBConfig
Threads int // applier write threads; 0 => default (4)
FlushInterval time.Duration // 0 => change.DefaultFlushInterval
// GTID selects the GTID-based change source (matching the forward move's
// --enable-experimental-gtid) instead of binlog file+offset, so the reverse
// feed uses the same coordinate scheme the operator chose for the move.
GTID bool
}
ReverseFeedConfig configures a ReverseFeed.
type ReverseSource ¶ added in v0.16.0
type ReverseSource struct {
DB *sql.DB // connection to S (default DB == its schema)
Addr string // host:port for the binlog syncer
User string // binlog syncer user
Password string // binlog syncer password
Tables []*table.TableInfo // S-side tables to watch, built on DB
// Position is the opaque change.Source position to resume from (captured at
// cutover). Empty means start from the source's current head.
Position string
}
ReverseSource is one reverse source: a former move target (shard S) whose binlog is tailed to keep the former move source (U) current.
DB's default database MUST be this source's schema. table.TableInfo.SetInfo resolves columns via information_schema WHERE table_schema=DATABASE() — the connection's default database, not the SchemaName argument — so a connection defaulting to the wrong schema silently loads the wrong table's definition, surfacing only at apply time as a fatal column-count mismatch.
type Runner ¶
type Runner struct {
// contains filtered or unexported fields
}
func (*Runner) DumpCheckpoint ¶
DumpCheckpoint is called approximately every minute. It writes the current state of the migration to the checkpoint table, which can be used in recovery. Previously resuming from checkpoint would always restart at the copier, but it can now also resume at the checksum phase.
func (*Runner) SetReverseCutover ¶ added in v0.16.0
SetReverseCutover registers the rollback traffic switch used if a revert is requested during the reverse window (route back to the source). It mirrors SetCutover and is only consulted when move.ReverseWindow > 0.
Source Files
Directories