Documentation
¶
Overview ¶
Package migration executes the three migration domains — filesystem, object storage and DBMS — on top of data-mig.
Its job is resolution and delegation. A plan carries ConnectionRefs, which name a source (honeybee, beetle*, or an inline ssh/minio/db block) rather than the access details themselves. The Resolve* and *Loc functions turn a ref plus a path into the concrete data-mig location or config the transfer needs, calling cm-honeybee or cm-beetle when the ref is a reference rather than an inline block. Migrate* then runs the transfer and reports progress as ProgressEvents.
Those resolvers are exported because pkg/core/validation reuses them: a migration and its validation must resolve a connection the same way, or a transfer that succeeded would fail verification.
Index ¶
- Variables
- func CheckConcurrency(n int) error
- func CheckStagingDir(dir string) error
- func DBConnConfigToDBMSLocation(cfg commonmodel.DBConnConfig, database string) (datamig.DBMSLocation, error)
- func DBMSLocationDirect(dbType, database string, direct *datamig.DirectConfig) datamig.DBMSLocation
- func DBMSLocationSSHTunnel(dbType, database string, tunnel *datamig.SSHTunnelConfig) datamig.DBMSLocation
- func DBMSLocationWithFilter(loc datamig.DBMSLocation, f *datamig.DBMSFilterOption) datamig.DBMSLocation
- func DropTargetDatabase(dst commonmodel.ConnectionRef, dstLoc datamig.DBMSLocation) error
- func EnsureTargetDatabase(dst commonmodel.ConnectionRef, dstLoc datamig.DBMSLocation, ...) (bool, error)
- func FilesystemLoc(sshCfg *datamig.SSHConfig, path string, filter *datamig.PathFilterOption) datamig.StorageLocation
- func HoneybeeToDBMSLocation(cfg honeybee.HoneybeeConnConfig, database string) datamig.DBMSLocation
- func InspectDB(loc datamig.DBMSLocation) (*datamig.DBMSInfo, error)
- func MigrateDBMS(ctx context.Context, m targetmodel.MigrationDBModel, ...) error
- func MigrateFileSystem(ctx context.Context, m targetmodel.MigrationFileSystemModel, stagingDir string, ...) error
- func MigrateObjectStorage(ctx context.Context, m targetmodel.MigrationObjectStorageModel, ...) error
- func MinioObjectStorageLoc(s3cfg *datamig.S3MinioConfig, bucketPath string, ...) datamig.StorageLocation
- func ResolveConcurrency(requested int) int
- func ResolveDBMSLocation(ref commonmodel.ConnectionRef, database, dbType string) (datamig.DBMSLocation, error)
- func ResolveObjectStorageLoc(ref commonmodel.ConnectionRef, bucketPath string, ...) (datamig.StorageLocation, error)
- func ResolveS3Endpoint(cfg commonmodel.MinioConnConfig) (endpoint, region string, useSSL bool, bucketLookup string, err error)
- func ResolveSSHConfig(ref commonmodel.ConnectionRef) (*datamig.SSHConfig, error)
- func ResolveStagingDir(requested string) string
- func TargetBucketExists(dst commonmodel.ConnectionRef, bucketPath string) (bool, error)
- func TargetDatabaseExists(dst commonmodel.ConnectionRef, dstLoc datamig.DBMSLocation) (bool, error)
- func ToDBMSFilter(rules []commonmodel.DBMSFilterRule) (*datamig.DBMSFilterOption, error)
- func ToDataMigFilter(rules []commonmodel.PathFilterRule) (*datamig.PathFilterOption, error)
- func ToDataMigSSHConfig(cfg beetle.SSHConfig) *datamig.SSHConfig
- func ToInlineMinioConfig(cfg commonmodel.MinioConnConfig) (*datamig.S3MinioConfig, error)
- func ToMinioS3Config(cfg honeybee.HoneybeeConnConfig) (*datamig.S3MinioConfig, error)
- func ToTumblebugOSConfig(ref commonmodel.BeetleOSRef) *datamig.TumblebugConfig
- func TumblebugObjectStorageLoc(tbCfg *datamig.TumblebugConfig, bucketPath string, ...) datamig.StorageLocation
- type EventKind
- type Executor
- type ProgressEvent
Constants ¶
This section is empty.
Variables ¶
var DefaultExecutor = &Executor{ cancels: make(map[string]context.CancelFunc), }
DefaultExecutor is the package-level singleton used by API handlers.
Functions ¶
func CheckConcurrency ¶
CheckConcurrency checks a concurrency a request or the configuration names: 0 for the default, or 1 to datamig.MaxConcurrency. The error names the field itself ("concurrency 33 is out of range ...").
func CheckStagingDir ¶
CheckStagingDir creates dir if it is missing and proves a file can be written in it. Both callers want the failure up front: at start-up for the configured default, and on the request for one a migration names, rather than as every item of an asynchronous migration failing the same way.
func DBConnConfigToDBMSLocation ¶
func DBConnConfigToDBMSLocation(cfg commonmodel.DBConnConfig, database string) (datamig.DBMSLocation, error)
DBConnConfigToDBMSLocation converts the unified DBConnConfig (the type all DB sources resolve into) to a datamig.DBMSLocation.
DBConnConfig carries server-level access only, so the database name is a separate argument: which databases to migrate is a decision of the migration, not a property of the connection.
func DBMSLocationDirect ¶
func DBMSLocationDirect(dbType, database string, direct *datamig.DirectConfig) datamig.DBMSLocation
DBMSLocationDirect builds a datamig.DBMSLocation for a direct TCP connection.
func DBMSLocationSSHTunnel ¶
func DBMSLocationSSHTunnel(dbType, database string, tunnel *datamig.SSHTunnelConfig) datamig.DBMSLocation
DBMSLocationSSHTunnel builds a datamig.DBMSLocation for an SSH-tunnelled connection.
func DBMSLocationWithFilter ¶
func DBMSLocationWithFilter(loc datamig.DBMSLocation, f *datamig.DBMSFilterOption) datamig.DBMSLocation
DBMSLocationWithFilter attaches a DBMSFilterOption (table/column/row/object exclusions) to an existing DBMSLocation. Returns a copy with Filter set.
func DropTargetDatabase ¶
func DropTargetDatabase(dst commonmodel.ConnectionRef, dstLoc datamig.DBMSLocation) error
DropTargetDatabase removes a database centipede created, after its migration failed or was cancelled. It is never called for a database that already existed: that case is data-mig's rollback, which keeps the database.
func EnsureTargetDatabase ¶
func EnsureTargetDatabase( dst commonmodel.ConnectionRef, dstLoc datamig.DBMSLocation, srcInfo *datamig.DBMSInfo, ) (bool, error)
EnsureTargetDatabase creates the destination database when it does not exist, and reports whether centipede created it.
data-mig requires the target to already exist and be empty, and never creates one itself, so this step belongs here. The returned flag decides who cleans up after a failure: a database centipede created is dropped whole (DropTargetDatabase), while one that already existed is left to data-mig's rollback, which empties it and keeps it.
srcInfo supplies the source database's character set so the new database matches it; without that the difference resurfaces later as a validation warning or a charset incompatibility error.
func FilesystemLoc ¶
func FilesystemLoc(sshCfg *datamig.SSHConfig, path string, filter *datamig.PathFilterOption) datamig.StorageLocation
FilesystemLoc builds a datamig.StorageLocation for a remote filesystem path. Pass nil filter to scan all entries with no depth limit.
func HoneybeeToDBMSLocation ¶
func HoneybeeToDBMSLocation(cfg honeybee.HoneybeeConnConfig, database string) datamig.DBMSLocation
HoneybeeToDBMSLocation maps a honeybee DBMS config to a datamig.DBMSLocation. When the honeybee connection has SSH tunnel info (db_access_type = "ssh-tunnel"), an SSH-tunnelled DBMSLocation is returned; otherwise a direct one is returned.
func InspectDB ¶
func InspectDB(loc datamig.DBMSLocation) (*datamig.DBMSInfo, error)
InspectDB returns server and schema metadata for the database described by loc.
func MigrateDBMS ¶
func MigrateDBMS(ctx context.Context, m targetmodel.MigrationDBModel, onFailure, stagingDir string, concurrency int, progressCh chan<- ProgressEvent) error
MigrateDBMS transfers every database listed in m from src to dst. One MigrationDBModel holds a single connection pair and as many database items as it migrates, so each item is resolved, prepared and transferred in turn and reports its own ProgressEvent with ItemPath "<dbType>/<srcName>".
Each item's target database is created when missing. The transfer scope is always full; a non-empty rule list drives the exclude-only pipeline that removes the listed tables/columns/rows/schema objects.
Cleanup after a failure follows ownership: RollbackOnFailure is enabled only for a database that already existed, in which case data-mig empties it and keeps it. A database centipede created is dropped whole instead, so the two paths never both run.
Filter feasibility (e.g. column/row/object exclusion is impossible over an ssh-tunnel) is checked up-front by datamig.ValidateDBMS — MigrateDBMSAsync does not validate on its own — and any violation is reported as a failed item.
stagingDir is the local staging root a dump is written under (see ResolveStagingDir); empty leaves it to data-mig's default. concurrency is how many connections a restore writes over at once (see ResolveConcurrency); zero leaves it to data-mig's default.
Returns ctx.Err() on cancellation; per-item errors are reported through progressCh and do not stop the remaining items.
func MigrateFileSystem ¶
func MigrateFileSystem(ctx context.Context, m targetmodel.MigrationFileSystemModel, stagingDir string, concurrency int, progressCh chan<- ProgressEvent) error
MigrateFileSystem transfers each FSMigrationInfo directory from a filesystem source to the destination. The destination is another filesystem (SSH→SSH) or, when m.DstType is objectstorage, an object-storage bucket (filesystem→S3 cross- storage; data-mig handles the ssh→local→S3 relay). Processes folders in ascending Order and emits a ProgressEvent per folder. Returns ctx.Err() on cancellation; per-folder transfer errors are reported via ProgressEvent only (continues). stagingDir is the local staging root a relay or cross-storage transfer uses (see ResolveStagingDir); empty leaves it to data-mig's default. concurrency is how many rsync processes a folder runs at once (see ResolveConcurrency); zero leaves it to data-mig's default.
func MigrateObjectStorage ¶
func MigrateObjectStorage(ctx context.Context, m targetmodel.MigrationObjectStorageModel, stagingDir string, concurrency int, progressCh chan<- ProgressEvent) error
MigrateObjectStorage transfers each ObjectMigrationInfo bucket from an object- storage source to the destination. The destination is another bucket (S3→S3) or, when m.DstType is filesystem, an SSH filesystem path (objectstorage→filesystem cross-storage; data-mig handles the S3→local→ssh relay). Processes buckets in ascending Order, defaultObjectConcurrency (one) at a time. Emits a ProgressEvent per bucket. Returns ctx.Err() on cancellation; per-bucket transfer errors are reported via ProgressEvent only. stagingDir is the local staging root a relay or cross-storage transfer uses (see ResolveStagingDir); empty leaves it to data-mig's default. concurrency is how many objects, and parts of one, a bucket moves at once (see ResolveConcurrency); zero leaves it to data-mig's default.
func MinioObjectStorageLoc ¶
func MinioObjectStorageLoc(s3cfg *datamig.S3MinioConfig, bucketPath string, filter *datamig.PathFilterOption) datamig.StorageLocation
MinioObjectStorageLoc builds a datamig.StorageLocation for a MinIO (honeybee) bucket path. bucketPath format: "bucket-name" or "bucket-name/prefix/". Pass nil filter for no filtering.
func ResolveConcurrency ¶
ResolveConcurrency returns the concurrency a migration runs with: the one its request named, else the server's configured default. Zero means neither was set, and data-mig then runs at its own default (datamig.DefaultConcurrency).
It is resolved at execution rather than stored at creation, like the staging root (ResolveStagingDir), so a retry of a migration that named none follows the configuration in effect when the retry runs.
func ResolveDBMSLocation ¶
func ResolveDBMSLocation(ref commonmodel.ConnectionRef, database, dbType string) (datamig.DBMSLocation, error)
ResolveDBMSLocation maps a ConnectionRef to a datamig.DBMSLocation. database is the database name to operate on (from the migration item). dbType is the engine hint; empty falls back to the value resolved from the source.
honeybee resolves via its own connection info; beetleDb is a managed RDB whose host/port/user/engine would be fetched from cm-beetle; db is inline credentials. Inline credentials and ref passwords are decrypted first.
func ResolveObjectStorageLoc ¶
func ResolveObjectStorageLoc(ref commonmodel.ConnectionRef, bucketPath string, filter *datamig.PathFilterOption) (datamig.StorageLocation, error)
ResolveObjectStorageLoc maps a ConnectionRef to a data-mig StorageLocation for object storage. Supports honeybee (MinIO direct), beetleObjectStorage (managed bucket, reached through cb-tumblebug's API inside data-mig) and minio (inline S3 credentials). Inline credentials are decrypted first.
func ResolveS3Endpoint ¶
func ResolveS3Endpoint(cfg commonmodel.MinioConnConfig) (endpoint, region string, useSSL bool, bucketLookup string, err error)
ResolveS3Endpoint derives the endpoint, signing region, TLS setting and bucket addressing style from a connection's provider. It mirrors cm-honeybee's resolveS3Endpoint so that the two systems address the same bucket the same way; keep the two in sync.
Defaults are HTTPS, no signing region, and automatic bucket lookup. Each branch below changes only what that provider does differently.
The region in the HOSTNAME and the region a request is SIGNED with are the same string for most providers and are not for all of them: NCP serves "kr.object.ncloudstorage.com" and signs "kr-standard". Where they differ the signing region is returned EMPTY rather than guessed, which makes minio-go ask the bucket for its location and sign with the answer — the same way cb-spider reaches these providers. Returning "us-east-1" instead would suppress that lookup and sign NCP wrongly, and spelling cfg.Region into both would build "kr-standard.object.ncloudstorage.com", a host that does not exist.
func ResolveSSHConfig ¶
func ResolveSSHConfig(ref commonmodel.ConnectionRef) (*datamig.SSHConfig, error)
ResolveSSHConfig resolves a ConnectionRef to a data-mig SSHConfig. Supports honeybee and beetleSsh (references) and ssh (inline). Inline credentials are decrypted first. data-mig SSH uses key-based auth only, so the private key is required for filesystem transfers.
func ResolveStagingDir ¶
ResolveStagingDir returns the staging root a migration runs with: the one its request named, else the server's configured default. Empty means neither was set, and data-mig then picks its own default ($TMPDIR/datamig-staging, or /tmp/datamig-staging when TMPDIR is unset).
It is resolved at execution rather than stored at creation, so a retry of a migration that named no directory follows the configuration in effect when the retry runs.
func TargetBucketExists ¶
func TargetBucketExists(dst commonmodel.ConnectionRef, bucketPath string) (bool, error)
TargetBucketExists reports whether the bucket an object storage destination writes into is already there. bucketPath is the DstPath: its first segment is the bucket.
Neither centipede nor data-mig creates a bucket, so the bucket has to be made on the provider first. A false result with a nil error means the store said it is not there; any other answer it could not give — a denied permission, an unreachable endpoint, a provider with no way to ask — is an error.
func TargetDatabaseExists ¶
func TargetDatabaseExists(dst commonmodel.ConnectionRef, dstLoc datamig.DBMSLocation) (bool, error)
TargetDatabaseExists reports whether the destination database is already there.
The answer comes from whoever made the database: cm-beetle for a managed instance, the database connection itself for everything else. Asking the other one invites a split verdict, and that verdict is what decides whether centipede owns the database and has to drop it after a failure.
func ToDBMSFilter ¶
func ToDBMSFilter(rules []commonmodel.DBMSFilterRule) (*datamig.DBMSFilterOption, error)
ToDBMSFilter is toDBMSFilter for a caller outside the migration — validation, which inspects the source through the very filter the migration dumped it with.
func ToDataMigFilter ¶
func ToDataMigFilter(rules []commonmodel.PathFilterRule) (*datamig.PathFilterOption, error)
ToDataMigFilter converts mirror path-filter rules into a data-mig FilterOption. Returns (nil, nil) when there are no rules to apply.
Exported because validation has to reach the same verdict this conversion feeds the transfer: a file the migration was told to skip must not be read back as missing. Sharing the converter is what keeps the two from drifting.
func ToDataMigSSHConfig ¶
ToDataMigSSHConfig maps a beetle SSHConfig to a data-mig SSHConfig. data-mig supports key-based and SSH-agent auth only (no password field).
func ToInlineMinioConfig ¶
func ToInlineMinioConfig(cfg commonmodel.MinioConnConfig) (*datamig.S3MinioConfig, error)
ToInlineMinioConfig builds a datamig.S3MinioConfig from an inline MinioConnConfig, resolving the provider into a concrete endpoint first — datamig.S3MinioConfig has no provider field and expects resolved values.
func ToMinioS3Config ¶
func ToMinioS3Config(cfg honeybee.HoneybeeConnConfig) (*datamig.S3MinioConfig, error)
ToMinioS3Config maps a honeybee ObjectStorage config to a data-mig S3MinioConfig.
It goes through ResolveS3Endpoint rather than copying the connection's fields across, because a honeybee connection carries only what its caller typed. cm-honeybee works out the endpoint, the signing region, the TLS setting and the bucket addressing style on every inspect of its own, and returns none of them: os_endpoint is optional for every provider whose host it builds itself, there is no field for the bucket addressing style at all, and os_use_ssl is the caller's wish rather than what the provider imposed. Reading the fields alone therefore produced an empty endpoint for every provider that needs none typed in — and, for Tencent, an endpoint that resolved and a bucket that could not be addressed.
The provider comes from the connection's source group; see honeybee.GetConnectionInfo. A minio group must name one, so the empty case is a group registered before that was required: there the caller's own endpoint is used as before.
A connection with os_access_type = "ssh-tunnel" names an endpoint only its SSH host can reach, so the host is passed on as the config's SSHTunnel and every connection to the store is opened from there.
func ToTumblebugOSConfig ¶
func ToTumblebugOSConfig(ref commonmodel.BeetleOSRef) *datamig.TumblebugConfig
ToTumblebugOSConfig maps a BeetleOSRef to a data-mig TumblebugConfig.
data-mig reaches a managed bucket through cb-tumblebug's ObjectStorage API, so a beetleObjectStorage connection resolves to that access type at transfer time. cb-tumblebug is an internal detail here: it is not a selectable connection source, and the nsId/osId pair comes from the beetle ref.
The endpoint and Basic Auth credentials are sourced from cm-centipede.yaml.
func TumblebugObjectStorageLoc ¶
func TumblebugObjectStorageLoc(tbCfg *datamig.TumblebugConfig, bucketPath string, filter *datamig.PathFilterOption) datamig.StorageLocation
TumblebugObjectStorageLoc builds a datamig.StorageLocation for a cb-tumblebug Object Storage bucket.
Types ¶
type EventKind ¶
type EventKind int
EventKind separates an item's outcome from an observation made while it runs.
EventDone is the zero value, so a ProgressEvent built without naming a kind is what it always was: one item, finished.
const ( // EventDone reports that an item finished. It advances ProcessedItems and // writes one MigrationLog row. EventDone EventKind = iota // EventTick reports how far the current item has got. It writes no log row // and does not advance ProcessedItems; SizeBytes is the item's running total, // not a delta. EventTick )
type Executor ¶
type Executor struct {
// contains filtered or unexported fields
}
Executor manages the lifecycle of asynchronous migration runs.
func (*Executor) Cancel ¶
Cancel signals the running migration to stop. The Execute goroutine sets Migration.Status = "cancelled" after the current item completes. Returns an error if the migration is not currently running.
func (*Executor) Execute ¶
Execute loads the migration from DB, runs all plan items in sequence, and updates Migration status and progress after each item completes. Must be launched as a goroutine: go DefaultExecutor.Execute(id)
func (*Executor) Retry ¶
Retry creates a new Migration that re-runs items from the original migration according to retryMode, then launches retryRun as a goroutine.
retryMode "" or "partial": re-run only failed items; DBMS targets are DROP-ped first. retryMode "full": re-run all plan items; DBMS targets are DROP-ped first.
Returns an error if the original migration is not in "failed" or "cancelled" status, or if retryMode is not one of the accepted values.
type ProgressEvent ¶
type ProgressEvent struct {
Kind EventKind
ItemPath string
// Current is the file, object or table the item is on right now — one level
// below ItemPath. Ticks only.
Current string
// Step is the pipeline step SizeBytes counts within, and Steps how many the
// item's transfer has: a relay's 1/2 stages, its 2/2 sends. Ticks only; zero
// where the transfer does not report one.
Step int
Steps int
// StepName names Step when the transfer has its own word for it — a
// database's "dumping" and "restoring". Empty for a file transfer, whose
// steps the reader names by position.
StepName string
// StepDetail is what the step is doing beyond its name, where the
// transfer says: a database restore's "indexes deferred: 2 tables", then
// "rebuilding indexes". Ticks only.
StepDetail string
// Rows, Units and UnitsTotal are a database item's: rows written so far,
// and tables (or collections) finished out of how many. BytesTotal is what
// SizeBytes counts towards when that is known — the dump a restore reads.
// Zero where the step cannot tell. Ticks only.
Rows int64
Units int
UnitsTotal int
BytesTotal int64
Status string
SizeBytes int64
DurationMs int64
// TargetCreated reports that this item's target database did not exist before
// the migration and does because of it, which is what decides whether a
// failure dropped the database or emptied it. DBMS items only. See
// model.MigrationLog.TargetCreated for why MongoDB sets it without anyone
// running a create statement.
TargetCreated bool
// Concurrency is how many transfers the item ran at once and
// ConcurrencyNote why that is not what the migration asked for, as
// data-mig reported them. Zero when it did not. Outcomes only.
Concurrency int
ConcurrencyNote string
Err error
}
ProgressEvent is emitted on the progressCh channel: once per migration item (directory, bucket, or database) as it finishes, and optionally as it runs.
Status values: "success" | "failed" | "cancelled"