status

Status

The status package provides state management, progress reporting, and periodic task monitoring for Spirit migrations. It defines the lifecycle states that a migration passes through and the infrastructure for background checkpointing and status logging.

State Machine

State is an int32 enum representing the current phase of a migration. It uses atomic.LoadInt32/atomic.StoreInt32 for lock-free concurrent access, since the migration runner and watcher goroutines read and write the state simultaneously.

The states are defined in lifecycle order:

Initial → CopyRows → ApplyChangeset → RestoreSecondaryIndexes → AnalyzeTable → Checksum → PostChecksum → WaitingOnSentinelTable → CutOver → ReverseWindow → Close → ErrCleanup

This ordering is deliberate — the code uses ordinal comparisons (e.g., state >= CutOver) to determine when to stop checkpointing and status reporting. WaitingOnSentinelTable sorts after PostChecksum because that is when it runs. The checkpoint gates in migration and move persist checksum resume evidence only in the window [Checksum, WaitingOnSentinelTable): the sentinel wait discards that evidence, so a restart during the wait re-verifies the whole range.

ReverseWindow is entered only by a move run with --reverse-window set. It sorts immediately after CutOver: the forward cutover is done and traffic is on the target, but Spirit keeps the source current in change-only mode so the move can still be rolled back. Because it is >= CutOver, the background status/checkpoint loops have already stopped (the reverse-window driver manages its own checkpoint writes) while orchestration can still observe that a revert is possible.

Tracker

Tracker wraps a State with per-state wall-clock timing, and is what the runners hold in place of a bare State field. Phases with a clear extent run under Do(state, fn), which transitions to state, runs fn, and attributes fn's elapsed time (panic inclusive) to that state:

err := r.status.Do(status.CopyRows, func() error {
    return r.copier.Run(ctx)
})

Set remains the transition primitive for states with no bracketable extent from the setter's perspective (Close, ErrCleanup); it closes the previous state's still-open interval (a gap after a completed Do stays unattributed), preserving the historical "one state ends when the next starts" semantics. In both cases the state stays current after the phase's code completes — Get() and the ordinal comparisons above behave exactly as before.

Because the tracker owns the timing, the runners no longer carry ad-hoc fields like copyDuration or sentinelWaitStartTime: the status block header renders Elapsed() (time in the current state) and final summaries render Duration(state) (total time attributed to a state, accumulating across repeat visits). The two can disagree after a bracket completes: Duration(state) freezes when the bracket closes, while Elapsed() keeps growing until the next transition.

The tracker assumes spirit's linear execution model — one goroutine advances through the phases in order, and the only concurrent transition is a fatal Set(ErrCleanup) racing an open bracket (time accrues to the bracketed state up to the fatal transition; the bracket's own exit becomes a no-op). It is not designed for concurrent or overlapping phases. Begin() marks the start of a run and resets all timing; runners call it once at the top of Run.

Metrics and reusable-run evidence

Every runner passes its existing metrics.Sink to Tracker. Generic sinks receive phase-entry and completed-duration values. A sink that also implements status.WorkflowMetricsSink receives typed, synchronous callbacks for each bracketed Do attempt (started, then finished with succeeded, failed, or cancelled) and one exact uint64 copy aggregate. These are attempt outcomes, not workflow/SLO outcomes: a failed checksum may be retried successfully, and a cancelled pod may later resume. Set-only transitions such as ErrCleanup do not enter the typed attempt stream.

The typed capability deliberately extends metrics.Sink rather than creating another observer mechanism. A runner with the default metrics.NoopSink disables transition delivery entirely and adds no transition allocations. Sink calls happen outside the tracker's timing mutex, their latency is excluded from phase duration, and a panic in a typed callback is recovered so telemetry cannot change migration behavior.

Copy totals count work settled during the current Run invocation and are emitted even when the copy attempt fails or is cancelled. The chunker restores its RowsCopied count from the checkpoint so that progress continues across a resume, while its chunk count starts afresh, and the runner subtracts the restored rows from the aggregate, so a resumed invocation reports only the rows and chunks it settled itself.

Durable mutation and physical ownership are correctness facts, not metrics. migration.Runner.Result and move.Runner.Result return status.WorkflowResult after Run; failures also preserve machine-checkable status.ErrDurableMutation and status.ErrOwnershipAmbiguous markers through errors.Is. Result-bearing forward and reverse cutover callbacks carry the same two independent facts, so a caller can report a confirmed partial write without inventing ownership ambiguity.

Task Interface

The Task interface defines the contract that a migration runner must implement: reporting progress, returning a status string, dumping checkpoints, and cancelling. Both the migration.Runner and move.Runner implement this interface.

Background Monitoring

WatchTask launches two background goroutines:

1. Status logger: Logs task.Status() every 30 seconds until the migration reaches cutover. This provides a regular heartbeat in the logs.
2. Checkpoint dumper: Calls task.DumpCheckpoint() every 50 seconds until cutover. If a checkpoint write fails (with anything other than ErrWatermarkNotReady or context.Canceled), the task is cancelled immediately. A task that implements Aborter is stopped with Abort(cause), where the cause is the write error marked by FatalAbort, so it can return that error instead of context.Canceled; any other task is stopped with Cancel(). The rationale is that it is better to fail early than to discover after a multi-day migration that progress was never being saved.

The checkpoint dumper also handles a race condition where the state transitions past cutover mid-checkpoint — the checkpoint table may have already been dropped, so this case is handled gracefully rather than treated as an error.

One report, not three lines

The status report is deliberately the only recurring INFO output a run emits. It used to compete with a per-checkpoint line and a per-flush line from the change feed, each on its own interval, which made the log hard to read (#329). Those events now report themselves here instead, and still log their detail at DEBUG.

Task.Status() still returns a string; each runner now builds a Block and returns its rendered form — a header line plus one indented row per subsystem. All three runners build it the same way.

migration status: state=copyRows total-time=2m6s copier-time=2m0s
  copier   30.84%  5048712/16370180  chunk-size=92220  eta=4m39s  throttled=false
  applier queue=128/128  workers=4  wait-p50=1.323s  write-p50=32ms  write-p90=127ms
  binlog  deltas=0  rotations=962 (0 forced)  flushed 0s ago (took 3µs, 0 rows)
  ckpt    20s ago  binlog.000123:41909012

The flush figures read as a phrase — flushed 30s ago (took 9µs, 0 rows) — because two of them are durations of different kinds. Side by side as key=0s pairs, "flushed just now" and "the flush was instant" are indistinguishable.

Two things the block gives up, deliberately: the whole report is one log record with newlines in it, which the default slog handler (what the CLI uses) prints as written but a quoting handler (TextHandler, JSON) will escape; and the applier-/binlog- field prefixes are gone, since the row label carries them.

conns-in-use was dropped: it reported sql.DB pool occupancy, which tracks the configured thread count and says nothing an operator acts on.

Progress Reporting

Progress is a struct (not just a string) containing the current state and a summary. It is designed as a struct specifically to allow future expansion for GUI wrappers and external tooling.

Alongside the summary it carries structured fields for the things a wrapper would otherwise have to parse out of prose or scrape from the logs: ETA, the runner-wide Copy counts, per-table Tables progress, Checksum progress, and — from #844 — Resume and Throttle.

Copy

Copy is the runner-wide row copy as a CopyProgress{RowsCopied, RowsTotal}. It is the sum of Tables, so the two always reconcile: both count settled rows against the tables' cardinality estimates, and neither is the optimistic chunker's keyspace position, which is what the copier paces on and what the ETA is derived from. It is populated as soon as the copy chunker exists and keeps its final reading through the later phases, so "how much did this run copy" stays answerable after the copy ends.

Two caveats carry over from the per-table counts. RowsCopied counts rows the copy settled, so a row the binlog applier wrote before the copy reached it is not counted (the copy inserts with INSERT IGNORE, which reports it as unaffected), and on a busy table the copy finishes short of RowsTotal; a resume restores the count from the checkpoint and continues it, though a move deletes and re-copies the rows at or above the resume position, so the rows among them settled before the checkpoint are counted again. And on an auto_increment key the ETA's DUE is paced on the keyspace, so over a sparse key range Copy can read close to complete while the ETA is still counting down, or the reverse; Summary carries both halves.

Resume

Resume is true when the run resumed from a checkpoint left by an earlier run. A resumed run walks the whole state machine again (CopyRows, Checksum, ...) even when those phases are near-instant, so a wrapper watching only CurrentState sees what looks like a migration starting over — confusing when the previous pod died while waiting on the sentinel table.

CurrentState is deliberately not overloaded with a synthetic "recovering" value: callers parse it for phase display, so a new state would be a breaking change. Pair Resume with the progress fields instead — a resumed run whose copy and checksum progress are both near-complete is one to render as "recovering" rather than "starting".

Throttle

Throttle reports whether the current phase is paused by a throttler, and why:

Two traps for consumers:

• Reason is for display, not for branching. It is empty when the configured throttler cannot explain itself (see ReasonedThrottler), and it is sampled independently of Throttled rather than atomically with it, so on a fast-changing signal the two can briefly disagree.
• Utilization is also 0 when no continuous load signal exists — notably when throttling is replica-lag-only, which is a budget rather than a load gauge. So a copy paused on replica lag reports Throttled with Utilization 0: treat 0 as "unknown" and hide the gauge, rather than drawing an idle server.

Which signals count depends on the phase, and the runner reports only the ones that phase actually honours — so Throttled means the same thing everywhere:

That last row matters for a wrapper polling after a run ends: a loaded server — or a replica-lag throttler that fails closed once its poll loop has stopped — must not make a finished migration, a cutover, or a sentinel wait look paused.

A move reports the combined load signal of its Aurora targets, whether or not autoscaling is enabled: the whole signal in CopyRows, load signals only in Checksum and — while its continuous checksum runs — in WaitingOnSentinelTable, and the zero value elsewhere. A sync reports its Aurora target's load signal in CopyRows and ApplyChangeset. A target that is not Aurora, or whose Aurora probe fails, contributes no signal.

See Also

• pkg/migration - Migration runner that implements the Task interface
• pkg/move - Move runner that implements the Task interface

Structured runner progress

Migration, move and datasync use TablesFromChunker to return table progress in a stable order. Multi-source identifiers retain the source qualifier so equally named tables remain distinct. Copy ETA is populated during CopyRows and cleared afterwards; the Copy counts persist past it. Migration and move also expose the finite initial checksum counts; datasync’s continuous verifier has no corresponding finite phase.

All three use multiline status blocks. Datasync includes copier-time while copying and state-time while restoring indexes, alongside its existing binlog and checkpoint rows. Sentinel progress polling returns a summary without emitting logs; periodic logging remains the responsibility of WatchTask.

Per-table RowsCopied is the actual settled row count from chunk feedback; RowsTotal is the estimated table cardinality, which may change and is not an upper bound. These are not the optimistic chunker’s keyspace-distance counters. The chunkers carry the count forward in their watermark, so a resumed copy includes the work done before the checkpoint; a watermark written before a chunker persisted its count resumes at zero. Use IsComplete for completion rather than requiring equality of the two counts.

Summary remains runner-specific: move and migration leave it empty in some phases (such as index restoration), while datasync falls back to the state name. Structured fields and multiline status formatting are aligned; summary text is not a shared API format.