migration

package
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 12, 2026 License: Apache-2.0 Imports: 21 Imported by: 0

Documentation

Overview

Package migration executes the three migration domains — filesystem, object storage and DBMS — on top of data-mig.

Its job is resolution and delegation. A plan carries ConnectionRefs, which name a source (honeybee, beetle*, or an inline ssh/minio/db block) rather than the access details themselves. The Resolve* and *Loc functions turn a ref plus a path into the concrete data-mig location or config the transfer needs, calling cm-honeybee or cm-beetle when the ref is a reference rather than an inline block. Migrate* then runs the transfer and reports progress as ProgressEvents.

Those resolvers are exported because pkg/core/validation reuses them: a migration and its validation must resolve a connection the same way, or a transfer that succeeded would fail verification.

Index

Constants

This section is empty.

Variables

View Source
var DefaultExecutor = &Executor{
	cancels: make(map[string]context.CancelFunc),
}

DefaultExecutor is the package-level singleton used by API handlers.

Functions

func CheckConcurrency

func CheckConcurrency(n int) error

CheckConcurrency checks a concurrency a request or the configuration names: 0 for the default, or 1 to datamig.MaxConcurrency. The error names the field itself ("concurrency 33 is out of range ...").

func CheckStagingDir

func CheckStagingDir(dir string) error

CheckStagingDir creates dir if it is missing and proves a file can be written in it. Both callers want the failure up front: at start-up for the configured default, and on the request for one a migration names, rather than as every item of an asynchronous migration failing the same way.

func DBConnConfigToDBMSLocation

func DBConnConfigToDBMSLocation(cfg commonmodel.DBConnConfig, database string) (datamig.DBMSLocation, error)

DBConnConfigToDBMSLocation converts the unified DBConnConfig (the type all DB sources resolve into) to a datamig.DBMSLocation.

DBConnConfig carries server-level access only, so the database name is a separate argument: which databases to migrate is a decision of the migration, not a property of the connection.

func DBMSLocationDirect

func DBMSLocationDirect(dbType, database string, direct *datamig.DirectConfig) datamig.DBMSLocation

DBMSLocationDirect builds a datamig.DBMSLocation for a direct TCP connection.

func DBMSLocationSSHTunnel

func DBMSLocationSSHTunnel(dbType, database string, tunnel *datamig.SSHTunnelConfig) datamig.DBMSLocation

DBMSLocationSSHTunnel builds a datamig.DBMSLocation for an SSH-tunnelled connection.

func DBMSLocationWithFilter

func DBMSLocationWithFilter(loc datamig.DBMSLocation, f *datamig.DBMSFilterOption) datamig.DBMSLocation

DBMSLocationWithFilter attaches a DBMSFilterOption (table/column/row/object exclusions) to an existing DBMSLocation. Returns a copy with Filter set.

func DropTargetDatabase

func DropTargetDatabase(dst commonmodel.ConnectionRef, dstLoc datamig.DBMSLocation) error

DropTargetDatabase removes a database centipede created, after its migration failed or was cancelled. It is never called for a database that already existed: that case is data-mig's rollback, which keeps the database.

func EnsureTargetDatabase

func EnsureTargetDatabase(
	dst commonmodel.ConnectionRef,
	dstLoc datamig.DBMSLocation,
	srcInfo *datamig.DBMSInfo,
) (bool, error)

EnsureTargetDatabase creates the destination database when it does not exist, and reports whether centipede created it.

data-mig requires the target to already exist and be empty, and never creates one itself, so this step belongs here. The returned flag decides who cleans up after a failure: a database centipede created is dropped whole (DropTargetDatabase), while one that already existed is left to data-mig's rollback, which empties it and keeps it.

srcInfo supplies the source database's character set so the new database matches it; without that the difference resurfaces later as a validation warning or a charset incompatibility error.

func FilesystemLoc

func FilesystemLoc(sshCfg *datamig.SSHConfig, path string, filter *datamig.PathFilterOption) datamig.StorageLocation

FilesystemLoc builds a datamig.StorageLocation for a remote filesystem path. Pass nil filter to scan all entries with no depth limit.

func HoneybeeToDBMSLocation

func HoneybeeToDBMSLocation(cfg honeybee.HoneybeeConnConfig, database string) datamig.DBMSLocation

HoneybeeToDBMSLocation maps a honeybee DBMS config to a datamig.DBMSLocation. When the honeybee connection has SSH tunnel info (db_access_type = "ssh-tunnel"), an SSH-tunnelled DBMSLocation is returned; otherwise a direct one is returned.

func InspectDB

func InspectDB(loc datamig.DBMSLocation) (*datamig.DBMSInfo, error)

InspectDB returns server and schema metadata for the database described by loc.

func MigrateDBMS

func MigrateDBMS(ctx context.Context, m targetmodel.MigrationDBModel, onFailure, stagingDir string, concurrency int, progressCh chan<- ProgressEvent) error

MigrateDBMS transfers every database listed in m from src to dst. One MigrationDBModel holds a single connection pair and as many database items as it migrates, so each item is resolved, prepared and transferred in turn and reports its own ProgressEvent with ItemPath "<dbType>/<srcName>".

Each item's target database is created when missing. The transfer scope is always full; a non-empty rule list drives the exclude-only pipeline that removes the listed tables/columns/rows/schema objects.

Cleanup after a failure follows ownership: RollbackOnFailure is enabled only for a database that already existed, in which case data-mig empties it and keeps it. A database centipede created is dropped whole instead, so the two paths never both run.

Filter feasibility (e.g. column/row/object exclusion is impossible over an ssh-tunnel) is checked up-front by datamig.ValidateDBMS — MigrateDBMSAsync does not validate on its own — and any violation is reported as a failed item.

stagingDir is the local staging root a dump is written under (see ResolveStagingDir); empty leaves it to data-mig's default. concurrency is how many connections a restore writes over at once (see ResolveConcurrency); zero leaves it to data-mig's default.

Returns ctx.Err() on cancellation; per-item errors are reported through progressCh and do not stop the remaining items.

func MigrateFileSystem

func MigrateFileSystem(ctx context.Context, m targetmodel.MigrationFileSystemModel, stagingDir string, concurrency int, progressCh chan<- ProgressEvent) error

MigrateFileSystem transfers each FSMigrationInfo directory from a filesystem source to the destination. The destination is another filesystem (SSH→SSH) or, when m.DstType is objectstorage, an object-storage bucket (filesystem→S3 cross- storage; data-mig handles the ssh→local→S3 relay). Processes folders in ascending Order and emits a ProgressEvent per folder. Returns ctx.Err() on cancellation; per-folder transfer errors are reported via ProgressEvent only (continues). stagingDir is the local staging root a relay or cross-storage transfer uses (see ResolveStagingDir); empty leaves it to data-mig's default. concurrency is how many rsync processes a folder runs at once (see ResolveConcurrency); zero leaves it to data-mig's default.

func MigrateObjectStorage

func MigrateObjectStorage(ctx context.Context, m targetmodel.MigrationObjectStorageModel, stagingDir string, concurrency int, progressCh chan<- ProgressEvent) error

MigrateObjectStorage transfers each ObjectMigrationInfo bucket from an object- storage source to the destination. The destination is another bucket (S3→S3) or, when m.DstType is filesystem, an SSH filesystem path (objectstorage→filesystem cross-storage; data-mig handles the S3→local→ssh relay). Processes buckets in ascending Order, defaultObjectConcurrency (one) at a time. Emits a ProgressEvent per bucket. Returns ctx.Err() on cancellation; per-bucket transfer errors are reported via ProgressEvent only. stagingDir is the local staging root a relay or cross-storage transfer uses (see ResolveStagingDir); empty leaves it to data-mig's default. concurrency is how many objects, and parts of one, a bucket moves at once (see ResolveConcurrency); zero leaves it to data-mig's default.

func MinioObjectStorageLoc

func MinioObjectStorageLoc(s3cfg *datamig.S3MinioConfig, bucketPath string, filter *datamig.PathFilterOption) datamig.StorageLocation

MinioObjectStorageLoc builds a datamig.StorageLocation for a MinIO (honeybee) bucket path. bucketPath format: "bucket-name" or "bucket-name/prefix/". Pass nil filter for no filtering.

func ResolveConcurrency

func ResolveConcurrency(requested int) int

ResolveConcurrency returns the concurrency a migration runs with: the one its request named, else the server's configured default. Zero means neither was set, and data-mig then runs at its own default (datamig.DefaultConcurrency).

It is resolved at execution rather than stored at creation, like the staging root (ResolveStagingDir), so a retry of a migration that named none follows the configuration in effect when the retry runs.

func ResolveDBMSLocation

func ResolveDBMSLocation(ref commonmodel.ConnectionRef, database, dbType string) (datamig.DBMSLocation, error)

ResolveDBMSLocation maps a ConnectionRef to a datamig.DBMSLocation. database is the database name to operate on (from the migration item). dbType is the engine hint; empty falls back to the value resolved from the source.

honeybee resolves via its own connection info; beetleDb is a managed RDB whose host/port/user/engine would be fetched from cm-beetle; db is inline credentials. Inline credentials and ref passwords are decrypted first.

func ResolveObjectStorageLoc

func ResolveObjectStorageLoc(ref commonmodel.ConnectionRef, bucketPath string, filter *datamig.PathFilterOption) (datamig.StorageLocation, error)

ResolveObjectStorageLoc maps a ConnectionRef to a data-mig StorageLocation for object storage. Supports honeybee (MinIO direct), beetleObjectStorage (managed bucket, reached through cb-tumblebug's API inside data-mig) and minio (inline S3 credentials). Inline credentials are decrypted first.

func ResolveS3Endpoint

func ResolveS3Endpoint(cfg commonmodel.MinioConnConfig) (endpoint, region string, useSSL bool, bucketLookup string, err error)

ResolveS3Endpoint derives the endpoint, signing region, TLS setting and bucket addressing style from a connection's provider. It mirrors cm-honeybee's resolveS3Endpoint so that the two systems address the same bucket the same way; keep the two in sync.

Defaults are HTTPS, no signing region, and automatic bucket lookup. Each branch below changes only what that provider does differently.

The region in the HOSTNAME and the region a request is SIGNED with are the same string for most providers and are not for all of them: NCP serves "kr.object.ncloudstorage.com" and signs "kr-standard". Where they differ the signing region is returned EMPTY rather than guessed, which makes minio-go ask the bucket for its location and sign with the answer — the same way cb-spider reaches these providers. Returning "us-east-1" instead would suppress that lookup and sign NCP wrongly, and spelling cfg.Region into both would build "kr-standard.object.ncloudstorage.com", a host that does not exist.

func ResolveSSHConfig

func ResolveSSHConfig(ref commonmodel.ConnectionRef) (*datamig.SSHConfig, error)

ResolveSSHConfig resolves a ConnectionRef to a data-mig SSHConfig. Supports honeybee and beetleSsh (references) and ssh (inline). Inline credentials are decrypted first. data-mig SSH uses key-based auth only, so the private key is required for filesystem transfers.

func ResolveStagingDir

func ResolveStagingDir(requested string) string

ResolveStagingDir returns the staging root a migration runs with: the one its request named, else the server's configured default. Empty means neither was set, and data-mig then picks its own default ($TMPDIR/datamig-staging, or /tmp/datamig-staging when TMPDIR is unset).

It is resolved at execution rather than stored at creation, so a retry of a migration that named no directory follows the configuration in effect when the retry runs.

func TargetBucketExists

func TargetBucketExists(dst commonmodel.ConnectionRef, bucketPath string) (bool, error)

TargetBucketExists reports whether the bucket an object storage destination writes into is already there. bucketPath is the DstPath: its first segment is the bucket.

Neither centipede nor data-mig creates a bucket, so the bucket has to be made on the provider first. A false result with a nil error means the store said it is not there; any other answer it could not give — a denied permission, an unreachable endpoint, a provider with no way to ask — is an error.

func TargetDatabaseExists

func TargetDatabaseExists(dst commonmodel.ConnectionRef, dstLoc datamig.DBMSLocation) (bool, error)

TargetDatabaseExists reports whether the destination database is already there.

The answer comes from whoever made the database: cm-beetle for a managed instance, the database connection itself for everything else. Asking the other one invites a split verdict, and that verdict is what decides whether centipede owns the database and has to drop it after a failure.

func ToDBMSFilter

func ToDBMSFilter(rules []commonmodel.DBMSFilterRule) (*datamig.DBMSFilterOption, error)

ToDBMSFilter is toDBMSFilter for a caller outside the migration — validation, which inspects the source through the very filter the migration dumped it with.

func ToDataMigFilter

func ToDataMigFilter(rules []commonmodel.PathFilterRule) (*datamig.PathFilterOption, error)

ToDataMigFilter converts mirror path-filter rules into a data-mig FilterOption. Returns (nil, nil) when there are no rules to apply.

Exported because validation has to reach the same verdict this conversion feeds the transfer: a file the migration was told to skip must not be read back as missing. Sharing the converter is what keeps the two from drifting.

func ToDataMigSSHConfig

func ToDataMigSSHConfig(cfg beetle.SSHConfig) *datamig.SSHConfig

ToDataMigSSHConfig maps a beetle SSHConfig to a data-mig SSHConfig. data-mig supports key-based and SSH-agent auth only (no password field).

func ToInlineMinioConfig

func ToInlineMinioConfig(cfg commonmodel.MinioConnConfig) (*datamig.S3MinioConfig, error)

ToInlineMinioConfig builds a datamig.S3MinioConfig from an inline MinioConnConfig, resolving the provider into a concrete endpoint first — datamig.S3MinioConfig has no provider field and expects resolved values.

func ToMinioS3Config

func ToMinioS3Config(cfg honeybee.HoneybeeConnConfig) (*datamig.S3MinioConfig, error)

ToMinioS3Config maps a honeybee ObjectStorage config to a data-mig S3MinioConfig.

It goes through ResolveS3Endpoint rather than copying the connection's fields across, because a honeybee connection carries only what its caller typed. cm-honeybee works out the endpoint, the signing region, the TLS setting and the bucket addressing style on every inspect of its own, and returns none of them: os_endpoint is optional for every provider whose host it builds itself, there is no field for the bucket addressing style at all, and os_use_ssl is the caller's wish rather than what the provider imposed. Reading the fields alone therefore produced an empty endpoint for every provider that needs none typed in — and, for Tencent, an endpoint that resolved and a bucket that could not be addressed.

The provider comes from the connection's source group; see honeybee.GetConnectionInfo. A minio group must name one, so the empty case is a group registered before that was required: there the caller's own endpoint is used as before.

A connection with os_access_type = "ssh-tunnel" names an endpoint only its SSH host can reach, so the host is passed on as the config's SSHTunnel and every connection to the store is opened from there.

func ToTumblebugOSConfig

func ToTumblebugOSConfig(ref commonmodel.BeetleOSRef) *datamig.TumblebugConfig

ToTumblebugOSConfig maps a BeetleOSRef to a data-mig TumblebugConfig.

data-mig reaches a managed bucket through cb-tumblebug's ObjectStorage API, so a beetleObjectStorage connection resolves to that access type at transfer time. cb-tumblebug is an internal detail here: it is not a selectable connection source, and the nsId/osId pair comes from the beetle ref.

The endpoint and Basic Auth credentials are sourced from cm-centipede.yaml.

func TumblebugObjectStorageLoc

func TumblebugObjectStorageLoc(tbCfg *datamig.TumblebugConfig, bucketPath string, filter *datamig.PathFilterOption) datamig.StorageLocation

TumblebugObjectStorageLoc builds a datamig.StorageLocation for a cb-tumblebug Object Storage bucket.

Types

type EventKind

type EventKind int

EventKind separates an item's outcome from an observation made while it runs.

EventDone is the zero value, so a ProgressEvent built without naming a kind is what it always was: one item, finished.

const (
	// EventDone reports that an item finished. It advances ProcessedItems and
	// writes one MigrationLog row.
	EventDone EventKind = iota
	// EventTick reports how far the current item has got. It writes no log row
	// and does not advance ProcessedItems; SizeBytes is the item's running total,
	// not a delta.
	EventTick
)

type Executor

type Executor struct {
	// contains filtered or unexported fields
}

Executor manages the lifecycle of asynchronous migration runs.

func (*Executor) Cancel

func (e *Executor) Cancel(migrationID string) error

Cancel signals the running migration to stop. The Execute goroutine sets Migration.Status = "cancelled" after the current item completes. Returns an error if the migration is not currently running.

func (*Executor) Execute

func (e *Executor) Execute(migrationID string)

Execute loads the migration from DB, runs all plan items in sequence, and updates Migration status and progress after each item completes. Must be launched as a goroutine: go DefaultExecutor.Execute(id)

func (*Executor) Retry

func (e *Executor) Retry(migrationID, retryMode string) (*model.Migration, error)

Retry creates a new Migration that re-runs items from the original migration according to retryMode, then launches retryRun as a goroutine.

retryMode "" or "partial": re-run only failed items; DBMS targets are DROP-ped first. retryMode "full": re-run all plan items; DBMS targets are DROP-ped first.

Returns an error if the original migration is not in "failed" or "cancelled" status, or if retryMode is not one of the accepted values.

type ProgressEvent

type ProgressEvent struct {
	Kind     EventKind
	ItemPath string
	// Current is the file, object or table the item is on right now — one level
	// below ItemPath. Ticks only.
	Current string
	// Step is the pipeline step SizeBytes counts within, and Steps how many the
	// item's transfer has: a relay's 1/2 stages, its 2/2 sends. Ticks only; zero
	// where the transfer does not report one.
	Step  int
	Steps int
	// StepName names Step when the transfer has its own word for it — a
	// database's "dumping" and "restoring". Empty for a file transfer, whose
	// steps the reader names by position.
	StepName string
	// StepDetail is what the step is doing beyond its name, where the
	// transfer says: a database restore's "indexes deferred: 2 tables", then
	// "rebuilding indexes". Ticks only.
	StepDetail string
	// Rows, Units and UnitsTotal are a database item's: rows written so far,
	// and tables (or collections) finished out of how many. BytesTotal is what
	// SizeBytes counts towards when that is known — the dump a restore reads.
	// Zero where the step cannot tell. Ticks only.
	Rows       int64
	Units      int
	UnitsTotal int
	BytesTotal int64
	Status     string
	SizeBytes  int64
	DurationMs int64
	// TargetCreated reports that this item's target database did not exist before
	// the migration and does because of it, which is what decides whether a
	// failure dropped the database or emptied it. DBMS items only. See
	// model.MigrationLog.TargetCreated for why MongoDB sets it without anyone
	// running a create statement.
	TargetCreated bool
	// Concurrency is how many transfers the item ran at once and
	// ConcurrencyNote why that is not what the migration asked for, as
	// data-mig reported them. Zero when it did not. Outcomes only.
	Concurrency     int
	ConcurrencyNote string
	Err             error
}

ProgressEvent is emitted on the progressCh channel: once per migration item (directory, bucket, or database) as it finishes, and optionally as it runs.

Status values: "success" | "failed" | "cancelled"

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL