Documentation
¶
Index ¶
- Constants
- func ParsePageParams(c echo.Context) (page, pageSize int)
- type ApiResponse
- type CreateMigrationReq
- type ListResponse
- type Migration
- type MigrationListFilter
- type MigrationLog
- type MigrationSummary
- type PlanEntry
- type PlanSummary
- type RetryReq
- type RetryResponse
- type SimpleMessageResponse
- type StartValidationRequest
- type TargetPlanReq
- type ValidationDetailItem
- type ValidationResultResponse
Constants ¶
const ( // DBMSOnFailureCleanup undoes a failed database item: one centipede created // is dropped whole, one that already existed is emptied by data-mig. DBMSOnFailureCleanup = "cleanup" // DBMSOnFailureKeep leaves a failed database exactly as the failure left it, // so the partial result can be inspected. Nothing is dropped and nothing is // emptied. DBMSOnFailureKeep = "keep" )
DBMSOnFailure values for CreateMigrationReq.DBMSOnFailure and Migration.DBMSOnFailure. Empty is read as DBMSOnFailureCleanup.
const ( // DateParamLayout is the wire format of dateFrom and dateTo. DateParamLayout = "2006-01-02" // MaxLastDays bounds lastDays. Ten years is well past any window a caller // would ask for and keeps a typo from turning into an unbounded scan. MaxLastDays = 3650 )
Variables ¶
This section is empty.
Functions ¶
func ParsePageParams ¶
ParsePageParams reads `page` and `pageSize` query parameters. Defaults: page=1, pageSize=20. pageSize is capped at 100.
Types ¶
type ApiResponse ¶
type ApiResponse[T any] struct { Success bool `json:"success"` Data T `json:"data"` Message string `json:"message,omitempty"` Error string `json:"error,omitempty"` }
ApiResponse is the generic envelope returned by every endpoint.
Data has no omitempty so that an empty list still serializes as "data": [].
func SimpleErrorResponse ¶
func SimpleErrorResponse(msg string) ApiResponse[any]
SimpleErrorResponse returns an error ApiResponse with no data.
func SuccessResponse ¶
func SuccessResponse[T any](data T) ApiResponse[T]
SuccessResponse wraps data in a successful ApiResponse.
func SuccessResponseWithMessage ¶
func SuccessResponseWithMessage[T any](data T, msg string) ApiResponse[T]
SuccessResponseWithMessage wraps data and attaches a message.
type CreateMigrationReq ¶
type CreateMigrationReq struct {
Name string `json:"name" validate:"required"`
Description string `json:"description"`
Plan targetmodel.TargetDataMigrationModel `json:"plan" validate:"required"`
// DBMSOnFailure decides what happens to a DBMS target when one of its
// database items fails. It is named for the DBMS domain because only that
// domain has an undo step: a filesystem or object storage transfer leaves
// whatever it had already written, with or without this field.
//
// "cleanup" (the default) undoes the failed item. "keep" leaves it in place
// for inspection — useful when the failure is in the data rather than in the
// connection, since cleanup destroys the evidence.
//
// Cancellation is not a failure and always cleans up, whatever this says.
DBMSOnFailure string `json:"dbmsOnFailure,omitempty" validate:"omitempty,oneof=cleanup keep"`
// StagingDir is the local directory, on the cm-centipede host, that
// transfers stage data under when they cannot go straight from source to
// destination: a filesystem or object storage relay, a cross-storage
// transfer, a DBMS dump. Must be an absolute path; it is created if missing.
// Empty uses centipede.migration.stagingDir from the configuration, and when
// that is empty too, $TMPDIR/datamig-staging or /tmp/datamig-staging.
StagingDir string `json:"stagingDir,omitempty" example:"/data/centipede-staging"`
// Concurrency is how many transfers each item runs at once: rsync processes
// for a filesystem, objects and the parts of one object for object storage,
// restore connections for a database. 1 to 32; empty or 0 uses
// centipede.migration.concurrency from the configuration, and when that is
// unset too, 4. Each item runs at most what its environment allows and logs
// what it ran at; a MySQL or PostgreSQL restore over an SSH tunnel runs one
// stream whatever this says.
Concurrency int `json:"concurrency,omitempty" example:"8"`
}
CreateMigrationReq is the request body for POST /centipede/migration.
type ListResponse ¶
type ListResponse[T any] struct { Total int64 `json:"total"` Page int `json:"page"` PageSize int `json:"pageSize"` Items []T `json:"items"` }
ListResponse wraps paginated list results.
type Migration ¶
type Migration struct {
ID string `gorm:"primaryKey" json:"id"`
Name string `gorm:"index:,column:name,unique;type:text collate nocase" json:"name" validate:"required"`
Description string `gorm:"column:description" json:"description"`
Plan targetmodel.TargetDataMigrationModel `gorm:"column:plan;serializer:json" json:"plan" validate:"required"`
// DBMSOnFailure is carried on the record rather than only on the request
// because execution is asynchronous and a retry re-reads the row: the policy
// has to outlive the call that set it. Empty means DBMSOnFailureCleanup.
DBMSOnFailure string `gorm:"column:dbms_on_failure" json:"dbmsOnFailure,omitempty"`
// StagingDir is the staging root the request named, kept as given for the
// same reason as DBMSOnFailure. Empty means the server's configured default,
// resolved when the migration runs.
StagingDir string `gorm:"column:staging_dir" json:"stagingDir,omitempty"`
// Concurrency is the concurrency the request named, kept as given for the
// same reason as DBMSOnFailure. Zero means the server's configured default,
// resolved when the migration runs. What each item actually ran at is on its
// log row (MigrationLog.AppliedConcurrency).
Concurrency int `gorm:"column:concurrency" json:"concurrency,omitempty"`
Status string `gorm:"column:status" json:"status"`
StatusMessage string `gorm:"column:status_message" json:"statusMessage,omitempty"`
TotalItems int64 `gorm:"column:total_items" json:"totalItems"`
ProcessedItems int64 `gorm:"column:processed_items" json:"processedItems"`
FailedItems int64 `gorm:"column:failed_items" json:"failedItems"`
TransferredBytes int64 `gorm:"column:transferred_bytes" json:"transferredBytes"`
// CurrentObject is the file, object or table being moved right now — one level
// below ItemPath, which names the folder, bucket or database. Empty when
// nothing is in flight.
CurrentObject string `gorm:"column:current_object" json:"currentObject,omitempty"`
// CurrentStep is the step of the item's transfer that CurrentObject and the
// in-flight part of TransferredBytes belong to, and CurrentSteps how many it
// has: a relay's 1/2 stages through local storage, its 2/2 sends to the
// destination, and each counts its bytes from zero. Zero when nothing is in
// flight or the transfer has no steps to report (DBMS).
CurrentStep int `gorm:"column:current_step" json:"currentStep,omitempty"`
CurrentSteps int `gorm:"column:current_steps" json:"currentSteps,omitempty"`
// CurrentStepName names CurrentStep when the transfer has a word for it: a
// database's "dumping" and "restoring". Empty for a file transfer.
CurrentStepName string `gorm:"column:current_step_name" json:"currentStepName,omitempty"`
// CurrentStepDetail is what CurrentStep is doing beyond its name, where
// the transfer says: a restore's "indexes deferred: 2 tables" while it
// writes rows with those tables' indexes held back, then "rebuilding
// indexes". Empty otherwise.
CurrentStepDetail string `gorm:"column:current_step_detail" json:"currentStepDetail,omitempty"`
// CurrentUnits counts the tables (or collections) of the item in flight
// that its current step has finished, out of CurrentUnitsTotal;
// CurrentRows the rows it has written; CurrentBytesTotal what the in-flight
// part of TransferredBytes is counting towards — the dump a restore reads.
// Databases only, and zero wherever the step cannot tell: a restore through
// a CLI over SSH counts bytes alone.
CurrentUnits int `gorm:"column:current_units" json:"currentUnits,omitempty"`
CurrentUnitsTotal int `gorm:"column:current_units_total" json:"currentUnitsTotal,omitempty"`
CurrentRows int64 `gorm:"column:current_rows" json:"currentRows,omitempty"`
CurrentBytesTotal int64 `gorm:"column:current_bytes_total" json:"currentBytesTotal,omitempty"`
ValidationStatus string `gorm:"column:validation_status" json:"validationStatus,omitempty"`
ValidationMessage string `gorm:"column:validation_message" json:"validationMessage,omitempty"`
ValidationDetails []ValidationDetailItem `gorm:"column:validation_details;serializer:json" json:"validationDetails,omitempty"`
// ValidationMode is how filesystem items were compared — "quick" (size and
// modification time) or "checksum" (size and SHA256). Object storage,
// databases and cross-storage items are compared one way whatever it says.
ValidationMode string `gorm:"column:validation_mode" json:"validationMode,omitempty"`
// ValidationStartedAt is when the latest validation started, so a running
// one can show how long it has been going.
ValidationStartedAt *time.Time `gorm:"column:validation_started_at" json:"validationStartedAt,omitempty"`
StartedAt *time.Time `gorm:"column:started_at" json:"startedAt,omitempty"`
CompletedAt *time.Time `gorm:"column:completed_at" json:"completedAt,omitempty"`
// CreatedAt is indexed because it is both the list ordering key and the only
// column the date filters compare against: without the index every list call
// is a full scan plus a sort.
CreatedAt time.Time `gorm:"column:created_at;index" json:"createdAt"`
UpdatedAt time.Time `gorm:"column:updated_at" json:"updatedAt"`
}
Migration represents a migration execution unit persisted to SQLite.
Status values: "pending" | "running" | "completed" | "failed" | "cancelled" ValidationStatus values: "passed" | "failed" | "not_run" | "running"
type MigrationListFilter ¶
type MigrationListFilter struct {
// Status filters on the status column. Empty means no filter; an unknown
// value is not rejected and simply matches nothing.
Status string
// DateFrom is the inclusive lower bound on created_at. Empty means none.
DateFrom string
// DateToExcl is the *exclusive* upper bound on created_at: the requested
// dateTo plus one day, since dateTo includes its own day. Empty means none.
DateToExcl string
// DateParamGiven records that the request carried at least one of dateFrom,
// dateTo or lastDays — lastDays=0 included, which asks for no window at
// all. ApplyDefaultWindow leaves such a filter untouched.
DateParamGiven bool
}
MigrationListFilter is the parsed form of the query parameters shared by GET /centipede/migration and GET /centipede/migration/all.
The date bounds are kept as yyyy-mm-dd strings rather than time.Time on purpose — see applyMigrationFilter in the dao package for why.
func ParseMigrationListFilter ¶
func ParseMigrationListFilter(c echo.Context) (MigrationListFilter, error)
ParseMigrationListFilter reads the list filter query parameters, following the same shape as ParsePageParams.
It returns an error instead of ignoring a malformed parameter: a date filter that silently dropped out would return *more* rows than asked for, and extra rows look like data rather than like a failure.
The parser is pure — it injects no defaults. GET /centipede/migration/all applies its own default window on top of the result.
func (*MigrationListFilter) ApplyDefaultWindow ¶
func (f *MigrationListFilter) ApplyDefaultWindow(lastDays int) bool
ApplyDefaultWindow fills in a lastDays window when the request named no date filter at all, and reports whether it did. A filter that already carries one — including the empty window lastDays=0 asks for — is left alone.
It exists for GET /centipede/migration/all, which is otherwise unbounded; the paginated endpoint is already bounded by pageSize and does not call it.
type MigrationLog ¶
type MigrationLog struct {
ID uint `gorm:"primaryKey;autoIncrement" json:"id"`
MigrationID string `gorm:"column:migration_id;index" json:"migrationId"`
// PlanEntryID names the plan entry this item came from
// (targetmodel.Migration*Model.PlanEntryID). ItemPath alone does not identify
// it: two entries of one plan legitimately carry the same path — one source
// fanned out to two destinations, or two sources that scanned the same
// directory — and their log lines were indistinguishable.
//
// It is the only thing about the entry kept here. Which host the item was
// read from, where it was written and under what rules are all in the plan,
// which GET /centipede/migration/{id} returns beside these logs; copying any
// of it into every row would not save the reader that lookup.
//
// Not indexed: nothing queries by it. One migration's logs are read whole, by
// the list endpoint and by the retry that groups them in memory, so an index
// here would only be paid for on every insert.
//
// Empty on rows written before this field existed.
PlanEntryID string `gorm:"column:plan_entry_id" json:"planEntryId,omitempty"`
// ItemPath: absolute file path (SSH) | object key (ObjectStorage) | table/collection name (DBMS)
ItemPath string `gorm:"column:item_path" json:"itemPath"`
Status string `gorm:"column:status" json:"status"`
ErrorMsg string `gorm:"column:error_msg" json:"errorMsg,omitempty"`
SizeBytes int64 `gorm:"column:size_bytes" json:"sizeBytes,omitempty"`
DurationMs int64 `gorm:"column:duration_ms" json:"durationMs"`
// FailedObjectKind and FailedObjectName name the object data-mig was
// working on when the item failed — a table, view, routine, file or object —
// so the failure can be located without reading ErrorMsg. FailedObjectName
// carries the enclosing database, schema or bucket when data-mig knew it
// ("centipede_test.v_customer_stats").
//
// Both stay empty when the failure was not tied to a single object: a
// connection refused, a dump command that failed as a whole.
FailedObjectKind string `gorm:"column:failed_object_kind" json:"failedObjectKind,omitempty"`
FailedObjectName string `gorm:"column:failed_object_name" json:"failedObjectName,omitempty"`
// TargetCreated records that this item's target database did not exist before
// the migration and does because of it, which is what decides whether a failure
// dropped the database or emptied it. DBMS items only.
//
// It is not a record of who ran a DDL statement. On MongoDB nobody does: a
// database begins to exist when the restore writes its first collection, and
// the flag is true there all the same, because dropping is still what undoing
// the item means.
TargetCreated bool `gorm:"column:target_created" json:"targetCreated,omitempty"`
// RollbackStatus reports the cleanup that followed a failure. What it refers
// to depends on TargetCreated: emptying a database that already existed
// (false), or dropping one centipede created (true).
RollbackStatus string `gorm:"column:rollback_status" json:"rollbackStatus,omitempty"`
RollbackErrorMsg string `gorm:"column:rollback_error_msg" json:"rollbackErrorMsg,omitempty"`
// AppliedConcurrency is how many transfers this item ran at once, as
// data-mig settled on them, and ConcurrencyNote why that is not what the
// migration asked for, when it is not: "capped from 16: ssh MaxSessions", or
// a restore through a CLI over SSH, which runs one stream. Zero when the
// item did not report it — rows written before this field existed among them.
AppliedConcurrency int `gorm:"column:applied_concurrency" json:"appliedConcurrency,omitempty"`
ConcurrencyNote string `gorm:"column:concurrency_note" json:"concurrencyNote,omitempty"`
CreatedAt time.Time `gorm:"column:created_at" json:"createdAt"`
}
MigrationLog records the result of a single item migration within a migration.
Status values: "success" | "failed" | "skipped" RollbackStatus values: "success" | "failed" | "skipped" | "" (non-DBMS, or no rollback performed)
type MigrationSummary ¶
type MigrationSummary struct {
ID string `json:"id"`
Name string `json:"name"`
Description string `json:"description"`
PlanSummary PlanSummary `json:"planSummary"`
DBMSOnFailure string `json:"dbmsOnFailure,omitempty"`
StagingDir string `json:"stagingDir,omitempty"`
Concurrency int `json:"concurrency,omitempty"`
Status string `json:"status"`
StatusMessage string `json:"statusMessage,omitempty"`
TotalItems int64 `json:"totalItems"`
ProcessedItems int64 `json:"processedItems"`
FailedItems int64 `json:"failedItems"`
TransferredBytes int64 `json:"transferredBytes"`
CurrentObject string `json:"currentObject,omitempty"`
CurrentStep int `json:"currentStep,omitempty"`
CurrentSteps int `json:"currentSteps,omitempty"`
CurrentStepName string `json:"currentStepName,omitempty"`
CurrentStepDetail string `json:"currentStepDetail,omitempty"`
CurrentUnits int `json:"currentUnits,omitempty"`
CurrentUnitsTotal int `json:"currentUnitsTotal,omitempty"`
CurrentRows int64 `json:"currentRows,omitempty"`
CurrentBytesTotal int64 `json:"currentBytesTotal,omitempty"`
ValidationStatus string `json:"validationStatus,omitempty"`
ValidationMessage string `json:"validationMessage,omitempty"`
ValidationDetails []ValidationDetailItem `json:"validationDetails,omitempty"`
ValidationMode string `json:"validationMode,omitempty"`
ValidationStartedAt *time.Time `json:"validationStartedAt,omitempty"`
StartedAt *time.Time `json:"startedAt,omitempty"`
CompletedAt *time.Time `json:"completedAt,omitempty"`
CreatedAt time.Time `json:"createdAt"`
UpdatedAt time.Time `json:"updatedAt"`
}
MigrationSummary is the list projection of Migration: every field of the record except Plan, which is replaced by PlanSummary.
The list endpoints return this rather than Migration because a list has no use for the connection details: a caller scanning migrations wants to know what each one moves, and the single-record read is where the plan itself belongs. Leaving the plan out also keeps a list call from touching credentials at all — the single-record read masks them (see connsec.MaskPlan), and what is never loaded needs no masking.
func NewMigrationSummary ¶
func NewMigrationSummary(m *Migration) MigrationSummary
NewMigrationSummary projects m into a MigrationSummary.
The unit counts are taken from the stored plan without decrypting it: encryption substitutes the connection inside each unit and never adds or removes a unit, so len() over an encrypted plan gives the same numbers.
type PlanEntry ¶
type PlanEntry struct {
SrcConnection commonmodel.ConnectionRef `json:"srcConnection" validate:"required"`
DstConnection commonmodel.ConnectionRef `json:"dstConnection" validate:"required"`
// FileSystemStrategy is the transfer strategy applied to filesystem (SSH→SSH)
// tasks. Empty is normalised to "direct" (agent-forward: the source host
// sends straight to the destination, with nothing staged here). Object
// storage tasks take ObjectStorageStrategy instead; DBMS tasks have none.
FileSystemStrategy string `json:"fileSystemStrategy,omitempty" validate:"omitempty,oneof=auto direct relay"`
FileSystemFilter *commonmodel.FileSystemFilter `json:"fileSystemFilter,omitempty"`
ObjectStorageFilter *commonmodel.ObjectStorageFilter `json:"objectStorageFilter,omitempty"`
DBMSFilter *commonmodel.DBMSFilter `json:"dbmsFilter,omitempty"`
// ObjectStorageStrategy is the transfer strategy applied to object storage
// (S3→S3) tasks. Empty is normalised to "direct": each object streams from
// source to destination, with nothing staged here. "relay" downloads to
// local staging first, then uploads. It has no effect on objectstorage→
// filesystem cross-storage, which always relays.
ObjectStorageStrategy string `json:"objectStorageStrategy,omitempty" validate:"omitempty,oneof=auto direct relay"`
}
PlanEntry selects one entry of TargetPlanReq.Source and pairs it with a destination.
SrcConnection names that entry. It is a ConnectionRef for symmetry with DstConnection and with the target model this plan produces, but only a honeybee ref is accepted: the source model is produced by cm-honeybee, so that is the only ref a discovery ever yields. Matching is by honeybee.connectionId alone (see plan.SrcConnKey) — never by comparing the structs, whose other sub-fields are rejected on a source ref and would only add ways for an equal connection to compare unequal.
A honeybee connection has exactly one ConnType, so the id also settles which of the three source domains the entry belongs to, and therefore which filter field this entry may set. The other three are rejected rather than ignored: an ignored filter leaves the caller believing a plan that migrated everything had honoured it.
The same SrcConnection may appear in several entries. Two entries that name the same source and the same destination but different paths are the way one connection migrates more than one folder or bucket, since the file system and object storage filters each select a single one.
type PlanSummary ¶
type PlanSummary struct {
FileSystems int `json:"fileSystems"`
ObjectStorages int `json:"objectStorages"`
Databases int `json:"databases"`
// CrossStorage counts the entries above that land on the other storage
// type: a file system into object storage, or object storage onto a file
// system. Validation compares those by path and size only, so a screen
// offering a validation mode needs to know without reading the plan.
CrossStorage int `json:"crossStorage"`
}
PlanSummary counts the migration units of a plan by category. It is what the list endpoints return in place of the plan itself, so that a caller can still tell a filesystem migration from a database one without the response carrying connection credentials.
type RetryReq ¶
type RetryReq struct {
RetryMode string `json:"retryMode,omitempty"`
}
RetryReq is the optional request body for POST /centipede/migration/{id}/retry.
RetryMode: "partial" (default) — re-run only failed items after rollback
"full" — empty every DBMS target (the database itself is kept), then re-run all plan items
type RetryResponse ¶
type RetryResponse struct {
OriginalMigrationID string `json:"originalMigrationId"`
NewMigration Migration `json:"newMigration"`
}
RetryResponse is returned when a retry migration is created successfully.
type SimpleMessageResponse ¶
type SimpleMessageResponse struct {
Message string `json:"message"`
}
SimpleMessageResponse carries a human-readable message in the data field.
type StartValidationRequest ¶
type StartValidationRequest struct {
// Mode is how filesystem items are compared: "quick" (size and
// modification time, seconds; the default) or "checksum" (size and SHA256
// of every migrated file). Object storage, databases and cross-storage
// items (compared by path and size) are compared one way whatever it says.
Mode string `json:"mode,omitempty" enums:"quick,checksum" example:"quick"`
}
StartValidationRequest is the optional body of POST /centipede/migration/{id}/validation.
type TargetPlanReq ¶
type TargetPlanReq struct {
Source sourcemodel.SourceDataMigrationModel `json:"source" validate:"required"`
Plans []PlanEntry `json:"plans" validate:"required,min=1"`
}
TargetPlanReq is the request body for POST /centipede/plans/target.
Source is the discovery result passed through verbatim — a cm-honeybee refined response, group-level included, which is why it can hold several entries. Plans selects which of those entries to migrate and pairs each with a destination: one entry per (source entry, destination), so two sources never share a destination by accident and the same source may appear twice to fan out. A source entry no plan entry names is not migrated.
Before this shape there was one dstConnection and one filter set for the whole request, which a group-level source model had no way to use: every entry was copied onto the same destination and the same mapping.
type ValidationDetailItem ¶
type ValidationDetailItem struct {
// PlanEntryID names the plan entry this detail is about, the same id
// MigrationLog carries. Without it a plan of several entries reports every
// one of them under the same ItemPath — "filesystem" for a whole entry that
// could not be resolved — and nothing says which.
PlanEntryID string `json:"planEntryId,omitempty"`
// ItemPath: file path | object key | DBMS target
// ("{database}/{table}", "{database}/{table}.{column}", "{database}/{kind}:{name}").
ItemPath string `json:"itemPath"`
// Status: "passed" | "failed" | "warning". A warning is reported but does not
// fail the validation — a character-set difference can be a managed service
// imposing its own default.
Status string `json:"status"`
Message string `json:"message,omitempty"`
}
ValidationDetailItem describes the verification result for a single item.
type ValidationResultResponse ¶
type ValidationResultResponse struct {
ValidationStatus string `json:"validationStatus"` // "passed" | "failed" | "not_run" | "running"
// ValidationMessage is the result's summary once the validation has
// finished, and while it runs, where it has got to — refreshed every few
// seconds.
ValidationMessage string `json:"validationMessage,omitempty"`
Details []ValidationDetailItem `json:"details,omitempty"` // list of failed items
Mode string `json:"mode,omitempty"` // "quick" | "checksum": how filesystem items were compared
StartedAt *time.Time `json:"startedAt,omitempty"`
}
ValidationResultResponse is returned by GET /centipede/migration/{id}/validation.