datamigrate

package
v0.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 12, 2026 License: Apache-2.0 Imports: 22 Imported by: 0

Documentation

Overview

Package datamigrate copies live data from a source database into a managed database here, and checks a cutover is safe. It never stores source credentials.

Index

Constants

View Source
const (
	CheckPass = "pass"
	CheckWarn = "warn"
	CheckFail = "fail"
)

Check outcomes.

View Source
const (
	VerdictGo     = "go"
	VerdictWait   = "wait"
	VerdictNoGo   = "no-go"
	VerdictSwitch = "switched"
)

Cutover verdicts.

View Source
const (
	FailureTimeout    = "source_timeout"
	FailureRefused    = "source_refused"
	FailureDNS        = "source_dns"
	FailureAuth       = "auth_failed"
	FailureNoDatabase = "database_missing"
	FailureSSL        = "ssl_required"
	FailurePermission = "permission_denied"
	FailureDisk       = "out_of_disk"
	FailureNotRunning = "target_not_running"
	FailureMismatch   = "verify_mismatch"
	FailureUnknown    = "unknown"
)

Failure codes returned by Humanize.

View Source
const (
	SeverityOK    = "ok"
	SeverityWarn  = "warn"
	SeverityBlock = "block"
)

Severity of one preflight check.

View Source
const (
	StatusPending     = "pending"
	StatusCopying     = "copying"
	StatusVerified    = "verified"
	StatusFailed      = "failed"
	StatusUnsupported = "unsupported"
)

Copy statuses reported per database.

View Source
const (
	EnginePostgres = "postgres"
	EngineMySQL    = "mysql"
	EngineMariaDB  = "mariadb"
	EngineMongoDB  = "mongodb"
	EngineRedis    = "redis"
)

Engine ids this package can copy.

View Source
const DefaultHelperVersion = "17"

DefaultHelperVersion is the Postgres image tag the read-only helper runs. pg_dump needs a client at least as new as the source server.

View Source
const DefaultMaxTTL = 300

DefaultMaxTTL is the TTL in seconds above which a cutover should wait.

View Source
const PgvectorMinMajor = 13

PgvectorMinMajor is the oldest Postgres major the pgvector images exist for.

View Source
const PgvectorVariantAvailable = true

PgvectorVariantAvailable gates the "<major>-pgvector" managed Postgres version, which the database reconciler maps to the pgvector image.

View Source
const PgvectorVersionSuffix = "-pgvector"

PgvectorVersionSuffix is appended to a Postgres major to select the variant.

Variables

This section is empty.

Functions

func DefaultPort

func DefaultPort(engine string) int

DefaultPort is the engine's well known port.

func DeriveTargetNames

func DeriveTargetNames(sources []string, taken map[string]bool) map[string]string

DeriveTargetNames maps source database names to unique, valid managed names. Existing names (already taken) are avoided with a numeric suffix.

func DiskFreeCommand

func DiskFreeCommand() []string

DiskFreeCommand prints "free|bytes" for the filesystem the helper runs on, the host's Docker storage that the managed database will also write to.

func DumpCommand

func DumpCommand(engine string) ([]string, error)

DumpCommand streams a logical dump of the source on stdout. It runs in the helper container, where the SRC_* variables are set.

func EngineFromImage

func EngineFromImage(image string) string

EngineFromImage maps an image reference to an engine id, "" when it is not a database this package can read.

func ExplainReachFailure

func ExplainReachFailure(raw string, local bool) string

ExplainReachFailure turns a connection failure into a plain explanation, naming the Docker network isolation case when the source is not a local container. The raw error is always kept in full after it.

func HumanBytes

func HumanBytes(n int64) string

HumanBytes formats a byte count for messages.

func HumanDuration

func HumanDuration(sec int) string

HumanDuration formats seconds for messages.

func InventoryDatabaseCommand

func InventoryDatabaseCommand(dbName string) []string

InventoryDatabaseCommand prints "tables|N" and "ext|name" lines for the Postgres database passed as the script's first argument. The name travels as a positional argument, so it never reaches a shell or SQL string.

func InventoryListCommand

func InventoryListCommand(engine string) ([]string, error)

InventoryListCommand lists the server version and its databases. Postgres prints "db|name|bytes"; the others also print a table count.

func InventorySupported

func InventorySupported(engine string) bool

InventorySupported reports whether a server's databases can be listed.

func MajorOf

func MajorOf(version string) int

MajorOf reads the leading major number of a server version such as "16.4".

func Overall

func Overall(plans []DomainPlan) string

Overall is the verdict for a whole migration: the worst of its domains.

func ParseCounts

func ParseCounts(out string) map[string]int64

ParseCounts reads "name|count" lines, ignoring anything that is not one (server notices, blank lines). The last field is the count, so a table name containing "|" still parses.

func Redact

func Redact(raw string) string

Redact removes passwords and credentials embedded in connection strings from raw error text.

func SourceCountCommand

func SourceCountCommand(engine string) ([]string, error)

SourceCountCommand prints "name|count" lines for the source, run in the helper.

func Supported

func Supported(engine string) bool

Supported reports whether Copy can move engine.

func TargetCountCommand

func TargetCountCommand(engine string) ([]string, error)

TargetCountCommand is SourceCountCommand for the managed database, run in its own container where the engine's own credential variables are already set.

func ValidTargetName

func ValidTargetName(name string) bool

ValidTargetName reports whether name can be a managed database name.

func ValidateSSHTarget

func ValidateSSHTarget(s string) error

ValidateSSHTarget accepts only user@host.

Types

type Check

type Check struct {
	ID     string `json:"id"`
	Status string `json:"status"`
	Detail string `json:"detail"`
	Fix    string `json:"fix,omitempty"`
}

Check is one named result with what to do about it.

type Copier

type Copier struct {
	Runtime  docker.Runtime
	Restorer Restorer
	Logger   *slog.Logger
	// VerifyAttempts and VerifyInterval retry the target count while the
	// engine is still loading the restored data. Zero values use defaults.
	// HelperNetwork is a Docker network the helper joins instead of the
	// default bridge, so a source container on it is reachable.
	HelperNetwork  string
	VerifyAttempts int
	VerifyInterval time.Duration
}

Copier moves a source database into a managed database through a helper container on the target's node. It only uses the Docker Engine API.

func (*Copier) Copy

func (c *Copier) Copy(ctx context.Context, t Target, src Source) (Verification, error)

Copy runs the full copy and verification. Every error is scrubbed of the source password. The helper container is always removed.

type DNSResolver

type DNSResolver struct {
	Timeout time.Duration
}

DNSResolver asks the domain's authoritative servers for the real TTL and falls back to the system resolver, which can only show a decayed one.

func (DNSResolver) Lookup

func (r DNSResolver) Lookup(ctx context.Context, domain string) DomainState

Lookup implements Resolver.

type DatabaseInfo

type DatabaseInfo struct {
	Name       string   `json:"name"`
	SizeBytes  int64    `json:"size_bytes"`
	Tables     int      `json:"tables"`
	Extensions []string `json:"extensions,omitempty"`
}

DatabaseInfo is one database found on a source server.

type DomainPlan

type DomainPlan struct {
	App     string        `json:"app"`
	Domain  string        `json:"domain"`
	Verdict string        `json:"verdict"`
	Checks  []Check       `json:"checks"`
	DNS     DomainState   `json:"dns"`
	Change  *RecordChange `json:"change,omitempty"`
}

DomainPlan is the go or no-go for one domain.

func Evaluate

func Evaluate(in EvalInput) DomainPlan

Evaluate decides whether switching domain's DNS to this platform is safe. The source keeps serving until the operator makes the change.

type DomainState

type DomainState struct {
	Addrs []string `json:"addrs,omitempty"`
	CNAME string   `json:"cname,omitempty"`
	// TTL is the TTL in seconds of the record the operator edits, 0 when unknown.
	TTL uint32 `json:"ttl"`
	// Authoritative is false when TTL came from a caching resolver, so the
	// real value may be higher than what it shows.
	Authoritative bool   `json:"authoritative"`
	Error         string `json:"error,omitempty"`
}

DomainState is what public DNS says about a domain right now.

type EvalInput

type EvalInput struct {
	App         string
	Domain      string
	Ready       bool
	ReadyDetail string
	DNS         DomainState
	TargetIPs   []string
	MaxTTL      uint32
}

EvalInput is everything a cutover decision for one domain depends on.

type FailureInfo

type FailureInfo struct {
	Code      string `json:"code"`
	Summary   string `json:"summary"`
	Hint      string `json:"hint"`
	Technical string `json:"technical,omitempty"`
}

FailureInfo is a copy failure in words an operator can act on. Technical holds the redacted raw error, to be shown only on request.

func Humanize

func Humanize(raw, fallbackHost string, fallbackPort int) FailureInfo

Humanize maps a raw copy error to a short sentence with a fix hint. The fallback host and port come from the recorded copy and are used when the error text does not name them.

type Inspector

type Inspector struct {
	Runtime       docker.Runtime
	Logger        *slog.Logger
	HelperVersion string
	// HelperNetwork is the Docker network the helper joins, see Copier.
	HelperNetwork string
}

Inspector reads a source server through a short-lived helper container.

func (*Inspector) Inventory

func (i *Inspector) Inventory(ctx context.Context, engine string, src Source) (Inventory, error)

Inventory lists the databases on the source server. It only runs read-only statements, and every error is scrubbed of the source password.

type Inventory

type Inventory struct {
	Version   string         `json:"version"`
	Major     int            `json:"major"`
	FreeBytes int64          `json:"free_bytes"`
	Databases []DatabaseInfo `json:"databases"`
}

Inventory is what a read-only look at a source server found.

func ParseInventory

func ParseInventory(engine, out string) Inventory

ParseInventory reads the output of InventoryListCommand.

type LocalSource

type LocalSource struct {
	Container string `json:"container"`
	Image     string `json:"image"`
	Engine    string `json:"engine"`
	Running   bool   `json:"running"`
	Network   string `json:"network,omitempty"`
	Host      string `json:"host,omitempty"`
	Port      int    `json:"port,omitempty"`
	// Problem is why this container cannot be used, empty when it can.
	Problem string `json:"problem,omitempty"`
}

LocalSource is a database container on this daemon, resolved to how the helper must reach it: join Network and connect to Host:Port.

func SelectLocalSource

func SelectLocalSource(c docker.LocalContainer) LocalSource

SelectLocalSource decides which network the helper joins and what address it uses. A user-defined network wins and the container name is used (embedded DNS). Only the default bridge falls back to the internal IP. It never uses a published host address: Docker's bridge isolation drops that traffic.

type PreflightCheck

type PreflightCheck struct {
	ID         string `json:"id"`
	Severity   string `json:"severity"`
	Message    string `json:"message"`
	NextAction string `json:"next_action,omitempty"`
}

PreflightCheck is one preflight finding with what to do about it.

type PreflightInput

type PreflightInput struct {
	Engine        string
	DB            DatabaseInfo
	SourceMajor   int
	TargetName    string
	TargetVersion string
	// TargetExists is true when a managed database already has TargetName,
	// OwnTarget when this migration created it earlier (so a retry is fine).
	TargetExists bool
	OwnTarget    bool
	// FreeBytes is the target node's free disk, negative when unknown.
	FreeBytes int64
	// DiskMargin multiplies the database size to get the space required.
	DiskMargin float64
	// MBPerSecond is the assumed copy throughput for the time estimate.
	MBPerSecond float64
}

PreflightInput is everything a database's preflight needs, gathered by the caller.

type PreflightResult

type PreflightResult struct {
	Checks          []PreflightCheck `json:"checks"`
	TargetVersion   string           `json:"target_version"`
	RequiredBytes   int64            `json:"required_bytes"`
	EstimateSeconds int              `json:"estimate_seconds"`
	Blocked         bool             `json:"blocked"`
}

PreflightResult is the checks plus the target version the plan resolved to.

func Preflight

func Preflight(in PreflightInput) PreflightResult

Preflight evaluates one database before anything is written. It never drops an extension: a missing one blocks the copy with a next action.

type Prober

type Prober struct {
	Resolver Resolver
	// Dial opens a TCP connection, used to reach this node directly.
	Dial func(ctx context.Context, network, addr string) (net.Conn, error)
	// RootCAs overrides the system roots, for a private CA.
	RootCAs *x509.CertPool
	Timeout time.Duration
}

Prober is how PostSwitch reaches the network, replaceable in tests.

func (Prober) PostSwitch

func (p Prober) PostSwitch(ctx context.Context, domain, healthPath string, targets []string) []Check

PostSwitch checks domain is served from this node: DNS resolves here, a trusted certificate is presented, and healthPath answers 200. TLS and HTTP go to this node's addresses with the domain as SNI, so they pass or fail independently of how far DNS has propagated.

type RecordChange

type RecordChange struct {
	Type     string `json:"type"`
	Name     string `json:"name"`
	Value    string `json:"value"`
	TTL      uint32 `json:"ttl"`
	Replaces string `json:"replaces,omitempty"`
}

RecordChange is the exact DNS edit to make.

type Resolver

type Resolver interface {
	Lookup(ctx context.Context, domain string) DomainState
}

Resolver reads what public DNS says about a domain.

type Restorer

type Restorer interface {
	Restore(ctx context.Context, engine, containerName string, dump io.Reader) error
}

Restorer replays a dump into a running database container. internal/backup's ContainerRestorer satisfies it and replaces the target's contents, so a repeated copy is idempotent.

type Source

type Source struct {
	Host     string `json:"host"`
	Port     int    `json:"port,omitempty"`
	User     string `json:"user,omitempty"`
	Password string `json:"password,omitempty"`
	Database string `json:"database,omitempty"`
	// AuthDatabase is MongoDB's authentication database, "admin" when empty.
	AuthDatabase string `json:"auth_database,omitempty"`
	TLS          bool   `json:"tls,omitempty"`
}

Source locates the database to copy from. Password is held in memory only.

func (Source) Scrub

func (s Source) Scrub(text string) string

Scrub replaces the source password, raw or URL-escaped, with a placeholder.

func (*Source) Validate

func (s *Source) Validate(engine string) error

Validate checks the source is well formed for engine and fills the default port.

type TableCount

type TableCount struct {
	Name   string `json:"name"`
	Source int64  `json:"source"`
	Target int64  `json:"target"`
	OK     bool   `json:"ok"`
}

TableCount is one table (or collection) compared across both sides.

type Target

type Target struct {
	Name    string
	Engine  string
	Version string
}

Target names the managed database receiving the data.

type Verification

type Verification struct {
	Checked    int          `json:"checked"`
	Mismatched int          `json:"mismatched"`
	Tables     []TableCount `json:"tables,omitempty"`
}

Verification is the outcome of comparing per-table row counts.

func Compare

func Compare(source, target map[string]int64) Verification

Compare checks every source table exists on the target with the same count. Tables only the target has are ignored: restore tooling may add some.

func (Verification) Failure

func (v Verification) Failure(limit int) string

Failure summarises a verification that did not pass, listing at most max tables.

type VolumeGuide

type VolumeGuide struct {
	App           string `json:"app"`
	Kind          string `json:"kind"`
	Source        string `json:"source"`
	Target        string `json:"target"`
	ContainerPath string `json:"container_path"`
	Command       string `json:"command"`
}

VolumeGuide is a command the operator runs on this node to copy one volume. Volume contents are not moved by the control plane: it would need root on the source host, which is the operator's call to make.

func GuideVolumes

func GuideVolumes(app, appPrefix, sourceSSH string, vols []VolumeRef) []VolumeGuide

GuideVolumes builds one rsync command per volume, to run as root on the node that hosts app. appPrefix is the prefix imports put on volume names ("app-<app>-"), so the source volume name is recovered from it.

type VolumeRef

type VolumeRef struct {
	// Name is the Docker volume name here, empty for a bind mount.
	Name          string
	ContainerPath string
	HostPath      string
	// SourceName is the volume's real name on the source host, when it is
	// not derivable from Name by trimming the app prefix.
	SourceName string
}

VolumeRef is one persistent mount of an imported app.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL