Help us shape the future of Go! Take the 2026 Go Developer Survey

dmsengine

package
v1.9.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 1, 2026 License: MIT Imports: 26 Imported by: 0

Documentation

Overview

Package dmsengine moves rows between databases and in-process stores for the DMS emulator: table-mapping evaluation, full load, and change data capture from MySQL binlogs and PostgreSQL logical decoding.

Index

Constants

View Source
const (
	MigrationFullLoad       = "full-load"
	MigrationCDC            = "cdc"
	MigrationFullLoadAndCDC = "full-load-and-cdc"
)

Migration types.

View Source
const (
	StopNormal       = "Stop Reason NORMAL"
	StopFatalError   = "Stop Reason FATAL_ERROR"
	StopFullLoadOnly = "Stop Reason FULL_LOAD_ONLY_FINISHED"
	StopAtServerTime = "Stop Reason STOPPED_AT_SERVER_TIME"
	StopAtCommitTime = "Stop Reason STOPPED_AT_COMMIT_TIME"
)

Stop reasons as DescribeReplicationTasks reports them.

View Source
const DefaultConnectTimeout = 10 * time.Second

DefaultConnectTimeout bounds each new database connection.

Variables

View Source
var (
	ErrUnsupportedEngine = errors.New("unsupported endpoint engine")
	ErrNotConfigured     = errors.New("in-process target not available")
)

Errors reported while opening endpoints.

View Source
var ErrCertificate = errors.New("endpoint certificate is not a valid PEM CA bundle")

ErrCertificate reports an endpoint CertificateArn whose PEM holds no usable certificate.

View Source
var ErrInvalidSettings = errors.New("invalid ReplicationTaskSettings")

ErrInvalidSettings reports an unusable ReplicationTaskSettings document.

View Source
var ErrMapping = errors.New("invalid table mappings")

ErrMapping reports an invalid table-mappings document.

View Source
var ErrTableNotInTask = errors.New("table is not part of the replication task")

ErrTableNotInTask is returned when a reload names a table the task does not replicate.

Functions

func MatchPattern

func MatchPattern(pattern, s string) bool

MatchPattern matches s against a DMS locator pattern where % matches any run of characters.

func ParseCDCTime

func ParseCDCTime(s string) (time.Time, bool)

ParseCDCTime reads the timestamp forms CdcStartPosition and CdcStopPosition accept.

func QuoteIdent

func QuoteIdent(f Family, name string) string

QuoteIdent quotes an identifier for the family's dialect.

func SupportedSource

func SupportedSource(engine string) bool

SupportedSource reports whether the engine can read from the endpoint engine.

func SupportedTarget

func SupportedTarget(engine string) bool

SupportedTarget reports whether the engine can write to the endpoint engine.

func TargetType

func TargetType(src, dst Family, native string) string

TargetType renders the column type to use on a target of family dst when the source is src.

func TestEndpoint

func TestEndpoint(ctx context.Context, o Opener, ep EndpointConfig, source bool) error

TestEndpoint opens the endpoint and verifies it is reachable, returning the driver's own error on failure.

Types

type Change

type Change struct {
	Time     time.Time
	Schema   string
	Table    string
	Op       Op
	Position string
	Before   Row
	After    Row
	Commit   bool
}

Change is one captured row change; Before is set for updates and deletes when the source logs it.

type Column

type Column struct {
	Name     string
	Native   string
	Kind     Kind
	Nullable bool
}

Column describes one source column. Native is the engine's own type text (e.g. "varchar(20)").

type DynamoKey

type DynamoKey struct {
	Name   string
	Number bool
}

DynamoKey is one key attribute of a target table; Number selects type N over S.

type DynamoStore

type DynamoStore interface {
	CreateTable(ctx context.Context, name string, keys []DynamoKey) error
	DropTable(ctx context.Context, name string) error
	PutItem(ctx context.Context, table string, item map[string]DynamoValue) error
	DeleteItem(ctx context.Context, table string, key map[string]DynamoValue) error
}

DynamoStore writes items into the emulator's DynamoDB.

type DynamoValue

type DynamoValue struct {
	Type string
	S    string
	B    []byte
}

DynamoValue is a typed attribute value: N, S or B.

type EndpointConfig

type EndpointConfig struct {
	Engine   string
	Server   string
	Database string
	User     string
	Password string
	SSLMode  string
	// CertificatePEM is the endpoint's CertificateArn contents, trusted when SslMode asks for TLS.
	CertificatePEM string
	// SlotName names the PostgreSQL logical replication slot for source endpoints.
	SlotName string
	// S3Settings, KinesisSettings and DynamoDBSettings carry the raw settings JSON objects.
	S3Settings       string
	KinesisSettings  string
	DynamoDBSettings string
	Port             int
	// PollInterval spaces PostgreSQL change polls; zero picks a default.
	PollInterval time.Duration
}

EndpointConfig is the engine's view of a DMS endpoint.

type Family

type Family int

Family groups engine names that share a SQL dialect.

const (
	FamilyOther Family = iota
	FamilyMySQL
	FamilyPostgres
	FamilyS3
	FamilyDynamoDB
	FamilyKinesis
)

Engine families.

func FamilyOf

func FamilyOf(engine string) Family

FamilyOf maps a DMS engine name to its family.

type Kind

type Kind int

Kind is the value class a column's text form belongs to.

const (
	KindString Kind = iota
	KindInt
	KindDecimal
	KindFloat
	KindBool
	KindTime
	KindBytes
	KindJSON
)

Column value classes.

func KindOf

func KindOf(native string) Kind

KindOf classifies an engine-native type.

type Mappings

type Mappings struct {
	Rules []Rule `json:"rules"`
}

Mappings is a parsed table-mappings document.

func ParseMappings

func ParseMappings(doc string) (Mappings, error)

ParseMappings parses and validates a TableMappings JSON document.

func (Mappings) Plan

func (m Mappings) Plan(t Table, src Family, overrideSchema string) TargetTable

Plan maps a selected source table to its target shape. overrideSchema (TargetMetadata.TargetSchema) wins over schema renames when set.

func (Mappings) Select

func (m Mappings) Select(tables []Table) []Table

Select returns the source tables chosen by the selection rules, in source order.

type ObjectStore

type ObjectStore interface {
	PutObject(ctx context.Context, bucket, key string, body []byte) error
	CheckBucket(ctx context.Context, bucket string) error
}

ObjectStore writes objects into the emulator's S3.

type Op

type Op string

Op is a change operation.

const (
	OpInsert Op = "insert"
	OpUpdate Op = "update"
	OpDelete Op = "delete"
)

Change operations.

type Opener

type Opener interface {
	OpenSource(ctx context.Context, ep EndpointConfig) (Source, error)
	OpenTarget(ctx context.Context, ep EndpointConfig, src Family) (Target, error)
	// ReleaseSlot drops the replication slot a source endpoint created, if any.
	ReleaseSlot(ctx context.Context, ep EndpointConfig) error
}

Opener builds sources and targets for endpoints.

func NewOpener

func NewOpener(stores Stores) Opener

NewOpener returns the production Opener, writing in-process targets through stores.

type PrepMode

type PrepMode string

PrepMode is FullLoadSettings.TargetTablePrepMode.

const (
	PrepDropAndCreate  PrepMode = "DROP_AND_CREATE"
	PrepTruncateBefore PrepMode = "TRUNCATE_BEFORE_LOAD"
	PrepDoNothing      PrepMode = "DO_NOTHING"
)

Target table preparation modes.

type Progress

type Progress struct {
	FullLoadStart  time.Time
	FullLoadFinish time.Time
	Checkpoint     string
	Tables         []TableStat
	Running        bool
}

Progress is a point-in-time view of a running task.

type Result

type Result struct {
	Err    error
	Reason string
}

Result is how a run ended: Reason is a StopReason string and Err is set when the task failed.

type Row

type Row []any

Row holds a column's text form: nil (NULL), string, or []byte for binary columns. Values line up with a table's Columns.

type Rule

type Rule struct {
	Locator struct {
		Schema string `json:"schema-name"`
		Table  string `json:"table-name"`
		Column string `json:"column-name"`
	} `json:"object-locator"`
	ID     flexString `json:"rule-id"`
	Name   flexString `json:"rule-name"`
	Type   string     `json:"rule-type"`
	Action string     `json:"rule-action"`
	Target string     `json:"rule-target"`
	Value  string     `json:"value"`
	Old    string     `json:"old-value"`
}

Rule is one entry of a table-mappings document.

type Source

type Source interface {
	Ping(ctx context.Context) error
	Tables(ctx context.Context) ([]Table, error)
	Scan(ctx context.Context, t Table, fn func(Row) error) error
	// Position captures the current change position; for PostgreSQL it also creates the replication slot.
	Position(ctx context.Context) (string, error)
	// Stream delivers changes from start until ctx is done or fn fails. Rows are in the table's column order.
	Stream(
		ctx context.Context,
		start StreamStart,
		tables func(schema, table string) (Table, bool),
		fn func(Change) error,
	) error
	// Release drops source-side resources such as replication slots.
	Release(ctx context.Context) error
	Close() error
}

Source reads tables and change streams.

type Stores

type Stores struct {
	Objects ObjectStore
	Streams StreamStore
	Dynamo  DynamoStore
}

Stores bundles the in-process services DMS targets can write to.

type StreamStart

type StreamStart struct {
	Time     time.Time
	Position string
}

StreamStart says where change capture begins.

type StreamStore

type StreamStore interface {
	PutRecord(ctx context.Context, stream, partitionKey string, data []byte) error
	CheckStream(ctx context.Context, stream string) error
}

StreamStore writes records into the emulator's Kinesis.

type Table

type Table struct {
	Schema  string
	Name    string
	Columns []Column
	PK      []string
}

Table is a source table; PK lists primary-key column names in key order.

func (Table) Key

func (t Table) Key() string

Key is "schema.table".

type TableRef

type TableRef struct{ Schema, Name string }

TableRef names a source table.

type TableStat

type TableStat struct {
	FullLoadStart     time.Time
	FullLoadEnd       time.Time
	LastUpdate        time.Time
	Schema            string
	Name              string
	State             string
	Error             string
	FullLoadRows      int64
	FullLoadErrorRows int64
	Inserts           int64
	Updates           int64
	Deletes           int64
	FullLoadReloaded  bool
}

TableStat is one row of DescribeTableStatistics; names are the source's.

type Target

type Target interface {
	Ping(ctx context.Context) error
	Prepare(ctx context.Context, t TargetTable, mode PrepMode) error
	Load(ctx context.Context, t TargetTable, rows []Row) error
	EndLoad(ctx context.Context, t TargetTable) error
	Apply(ctx context.Context, t TargetTable, c Change) error
	Commit(ctx context.Context) error
	Close() error
}

Target receives rows and changes.

type TargetColumn

type TargetColumn struct {
	Column
	Source int
}

TargetColumn is a column on the target; Source indexes the originating source column.

type TargetTable

type TargetTable struct {
	Schema       string
	Name         string
	SourceSchema string
	SourceName   string
	Columns      []TargetColumn
	PK           []string
	SourceFamily Family
}

TargetTable is a source table after mapping rules.

func (TargetTable) Key

func (t TargetTable) Key() string

Key is "schema.table" on the target.

func (TargetTable) Project

func (t TargetTable) Project(r Row) Row

Project selects and orders a source row's values for the target.

type Task

type Task struct {
	// contains filtered or unexported fields
}

Task runs one replication from start to finish.

func NewTask

func NewTask(cfg TaskConfig) *Task

NewTask builds a task; call Run once.

func (*Task) Has

func (t *Task) Has(r TableRef) bool

Has reports whether the task replicates the table.

func (*Task) Reload

func (t *Task) Reload(ctx context.Context, refs []TableRef) error

Reload re-copies tables while change capture keeps running.

func (*Task) Run

func (t *Task) Run(ctx context.Context) Result

Run executes the task until it finishes, fails or ctx is cancelled.

type TaskConfig

type TaskConfig struct {
	Source        Source
	Target        Target
	Now           func() time.Time
	OnProgress    func(Progress)
	MigrationType string
	CDCStart      string
	CDCStop       string
	Resume        string
	Settings      TaskSettings
	Mappings      Mappings
	PingBackoff   time.Duration
	SourceFamily  Family
}

TaskConfig wires a task to its endpoints.

type TaskSettings

type TaskSettings struct {
	PrepMode     PrepMode
	TargetSchema string
	CommitRate   int
}

TaskSettings is the subset of ReplicationTaskSettings the engine honours.

func ParseTaskSettings

func ParseTaskSettings(doc string) (TaskSettings, error)

ParseTaskSettings reads ReplicationTaskSettings JSON; empty input yields the defaults.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL