Documentation
¶
Overview ¶
Package dmsengine moves rows between databases and in-process stores for the DMS emulator: table-mapping evaluation, full load, and change data capture from MySQL binlogs and PostgreSQL logical decoding.
Index ¶
- Constants
- Variables
- func MatchPattern(pattern, s string) bool
- func ParseCDCTime(s string) (time.Time, bool)
- func QuoteIdent(f Family, name string) string
- func SupportedSource(engine string) bool
- func SupportedTarget(engine string) bool
- func TargetType(src, dst Family, native string) string
- func TestEndpoint(ctx context.Context, o Opener, ep EndpointConfig, source bool) error
- type Change
- type Column
- type DynamoKey
- type DynamoStore
- type DynamoValue
- type EndpointConfig
- type Family
- type Kind
- type Mappings
- type ObjectStore
- type Op
- type Opener
- type PrepMode
- type Progress
- type Result
- type Row
- type Rule
- type Source
- type Stores
- type StreamStart
- type StreamStore
- type Table
- type TableRef
- type TableStat
- type Target
- type TargetColumn
- type TargetTable
- type Task
- type TaskConfig
- type TaskSettings
Constants ¶
const ( MigrationFullLoad = "full-load" MigrationCDC = "cdc" MigrationFullLoadAndCDC = "full-load-and-cdc" )
Migration types.
const ( StopNormal = "Stop Reason NORMAL" StopFatalError = "Stop Reason FATAL_ERROR" StopFullLoadOnly = "Stop Reason FULL_LOAD_ONLY_FINISHED" StopAtServerTime = "Stop Reason STOPPED_AT_SERVER_TIME" StopAtCommitTime = "Stop Reason STOPPED_AT_COMMIT_TIME" )
Stop reasons as DescribeReplicationTasks reports them.
const DefaultConnectTimeout = 10 * time.Second
DefaultConnectTimeout bounds each new database connection.
Variables ¶
var ( ErrUnsupportedEngine = errors.New("unsupported endpoint engine") ErrNotConfigured = errors.New("in-process target not available") )
Errors reported while opening endpoints.
var ErrCertificate = errors.New("endpoint certificate is not a valid PEM CA bundle")
ErrCertificate reports an endpoint CertificateArn whose PEM holds no usable certificate.
var ErrInvalidSettings = errors.New("invalid ReplicationTaskSettings")
ErrInvalidSettings reports an unusable ReplicationTaskSettings document.
var ErrMapping = errors.New("invalid table mappings")
ErrMapping reports an invalid table-mappings document.
var ErrTableNotInTask = errors.New("table is not part of the replication task")
ErrTableNotInTask is returned when a reload names a table the task does not replicate.
Functions ¶
func MatchPattern ¶
MatchPattern matches s against a DMS locator pattern where % matches any run of characters.
func ParseCDCTime ¶
ParseCDCTime reads the timestamp forms CdcStartPosition and CdcStopPosition accept.
func QuoteIdent ¶
QuoteIdent quotes an identifier for the family's dialect.
func SupportedSource ¶
SupportedSource reports whether the engine can read from the endpoint engine.
func SupportedTarget ¶
SupportedTarget reports whether the engine can write to the endpoint engine.
func TargetType ¶
TargetType renders the column type to use on a target of family dst when the source is src.
func TestEndpoint ¶
TestEndpoint opens the endpoint and verifies it is reachable, returning the driver's own error on failure.
Types ¶
type Change ¶
type Change struct {
Time time.Time
Schema string
Table string
Op Op
Position string
Before Row
After Row
Commit bool
}
Change is one captured row change; Before is set for updates and deletes when the source logs it.
type Column ¶
Column describes one source column. Native is the engine's own type text (e.g. "varchar(20)").
type DynamoStore ¶
type DynamoStore interface {
CreateTable(ctx context.Context, name string, keys []DynamoKey) error
DropTable(ctx context.Context, name string) error
PutItem(ctx context.Context, table string, item map[string]DynamoValue) error
DeleteItem(ctx context.Context, table string, key map[string]DynamoValue) error
}
DynamoStore writes items into the emulator's DynamoDB.
type DynamoValue ¶
DynamoValue is a typed attribute value: N, S or B.
type EndpointConfig ¶
type EndpointConfig struct {
Engine string
Server string
Database string
User string
Password string
SSLMode string
// CertificatePEM is the endpoint's CertificateArn contents, trusted when SslMode asks for TLS.
CertificatePEM string
// SlotName names the PostgreSQL logical replication slot for source endpoints.
SlotName string
// S3Settings, KinesisSettings and DynamoDBSettings carry the raw settings JSON objects.
S3Settings string
KinesisSettings string
DynamoDBSettings string
Port int
// PollInterval spaces PostgreSQL change polls; zero picks a default.
PollInterval time.Duration
}
EndpointConfig is the engine's view of a DMS endpoint.
type Family ¶
type Family int
Family groups engine names that share a SQL dialect.
type Kind ¶
type Kind int
Kind is the value class a column's text form belongs to.
Column value classes.
type Mappings ¶
type Mappings struct {
Rules []Rule `json:"rules"`
}
Mappings is a parsed table-mappings document.
func ParseMappings ¶
ParseMappings parses and validates a TableMappings JSON document.
type ObjectStore ¶
type ObjectStore interface {
PutObject(ctx context.Context, bucket, key string, body []byte) error
CheckBucket(ctx context.Context, bucket string) error
}
ObjectStore writes objects into the emulator's S3.
type Opener ¶
type Opener interface {
OpenSource(ctx context.Context, ep EndpointConfig) (Source, error)
OpenTarget(ctx context.Context, ep EndpointConfig, src Family) (Target, error)
// ReleaseSlot drops the replication slot a source endpoint created, if any.
ReleaseSlot(ctx context.Context, ep EndpointConfig) error
}
Opener builds sources and targets for endpoints.
type Progress ¶
type Progress struct {
FullLoadStart time.Time
FullLoadFinish time.Time
Checkpoint string
Tables []TableStat
Running bool
}
Progress is a point-in-time view of a running task.
type Result ¶
Result is how a run ended: Reason is a StopReason string and Err is set when the task failed.
type Row ¶
type Row []any
Row holds a column's text form: nil (NULL), string, or []byte for binary columns. Values line up with a table's Columns.
type Rule ¶
type Rule struct {
Locator struct {
Schema string `json:"schema-name"`
Table string `json:"table-name"`
Column string `json:"column-name"`
} `json:"object-locator"`
ID flexString `json:"rule-id"`
Name flexString `json:"rule-name"`
Type string `json:"rule-type"`
Action string `json:"rule-action"`
Target string `json:"rule-target"`
Value string `json:"value"`
Old string `json:"old-value"`
}
Rule is one entry of a table-mappings document.
type Source ¶
type Source interface {
Ping(ctx context.Context) error
Tables(ctx context.Context) ([]Table, error)
Scan(ctx context.Context, t Table, fn func(Row) error) error
// Position captures the current change position; for PostgreSQL it also creates the replication slot.
Position(ctx context.Context) (string, error)
// Stream delivers changes from start until ctx is done or fn fails. Rows are in the table's column order.
Stream(
ctx context.Context,
start StreamStart,
tables func(schema, table string) (Table, bool),
fn func(Change) error,
) error
// Release drops source-side resources such as replication slots.
Release(ctx context.Context) error
Close() error
}
Source reads tables and change streams.
type Stores ¶
type Stores struct {
Objects ObjectStore
Streams StreamStore
Dynamo DynamoStore
}
Stores bundles the in-process services DMS targets can write to.
type StreamStart ¶
StreamStart says where change capture begins.
type StreamStore ¶
type StreamStore interface {
PutRecord(ctx context.Context, stream, partitionKey string, data []byte) error
CheckStream(ctx context.Context, stream string) error
}
StreamStore writes records into the emulator's Kinesis.
type TableStat ¶
type TableStat struct {
FullLoadStart time.Time
FullLoadEnd time.Time
LastUpdate time.Time
Schema string
Name string
State string
Error string
FullLoadRows int64
FullLoadErrorRows int64
Inserts int64
Updates int64
Deletes int64
FullLoadReloaded bool
}
TableStat is one row of DescribeTableStatistics; names are the source's.
type Target ¶
type Target interface {
Ping(ctx context.Context) error
Prepare(ctx context.Context, t TargetTable, mode PrepMode) error
Load(ctx context.Context, t TargetTable, rows []Row) error
EndLoad(ctx context.Context, t TargetTable) error
Apply(ctx context.Context, t TargetTable, c Change) error
Commit(ctx context.Context) error
Close() error
}
Target receives rows and changes.
type TargetColumn ¶
TargetColumn is a column on the target; Source indexes the originating source column.
type TargetTable ¶
type TargetTable struct {
Schema string
Name string
SourceSchema string
SourceName string
Columns []TargetColumn
PK []string
SourceFamily Family
}
TargetTable is a source table after mapping rules.
func (TargetTable) Project ¶
func (t TargetTable) Project(r Row) Row
Project selects and orders a source row's values for the target.
type Task ¶
type Task struct {
// contains filtered or unexported fields
}
Task runs one replication from start to finish.
type TaskConfig ¶
type TaskConfig struct {
Source Source
Target Target
Now func() time.Time
OnProgress func(Progress)
MigrationType string
CDCStart string
CDCStop string
Resume string
Settings TaskSettings
Mappings Mappings
PingBackoff time.Duration
SourceFamily Family
}
TaskConfig wires a task to its endpoints.
type TaskSettings ¶
TaskSettings is the subset of ReplicationTaskSettings the engine honours.
func ParseTaskSettings ¶
func ParseTaskSettings(doc string) (TaskSettings, error)
ParseTaskSettings reads ReplicationTaskSettings JSON; empty input yields the defaults.