Documentation
¶
Overview ¶
Package datamigrate copies live data from a source database into a managed database here, and checks a cutover is safe. It never stores source credentials.
Index ¶
- Constants
- func DefaultPort(engine string) int
- func DeriveTargetNames(sources []string, taken map[string]bool) map[string]string
- func DiskFreeCommand() []string
- func DumpCommand(engine string) ([]string, error)
- func EngineFromImage(image string) string
- func ExplainReachFailure(raw string, local bool) string
- func HumanBytes(n int64) string
- func HumanDuration(sec int) string
- func InventoryDatabaseCommand(dbName string) []string
- func InventoryListCommand(engine string) ([]string, error)
- func InventorySupported(engine string) bool
- func MajorOf(version string) int
- func Overall(plans []DomainPlan) string
- func ParseCounts(out string) map[string]int64
- func Redact(raw string) string
- func SourceCountCommand(engine string) ([]string, error)
- func Supported(engine string) bool
- func TargetCountCommand(engine string) ([]string, error)
- func ValidTargetName(name string) bool
- func ValidateSSHTarget(s string) error
- type Check
- type Copier
- type DNSResolver
- type DatabaseInfo
- type DomainPlan
- type DomainState
- type EvalInput
- type FailureInfo
- type Inspector
- type Inventory
- type LocalSource
- type PreflightCheck
- type PreflightInput
- type PreflightResult
- type Prober
- type RecordChange
- type Resolver
- type Restorer
- type Source
- type TableCount
- type Target
- type Verification
- type VolumeGuide
- type VolumeRef
Constants ¶
const ( CheckPass = "pass" CheckWarn = "warn" CheckFail = "fail" )
Check outcomes.
const ( VerdictGo = "go" VerdictWait = "wait" VerdictNoGo = "no-go" VerdictSwitch = "switched" )
Cutover verdicts.
const ( FailureTimeout = "source_timeout" FailureRefused = "source_refused" FailureDNS = "source_dns" FailureAuth = "auth_failed" FailureNoDatabase = "database_missing" FailureSSL = "ssl_required" FailurePermission = "permission_denied" FailureDisk = "out_of_disk" FailureNotRunning = "target_not_running" FailureMismatch = "verify_mismatch" FailureUnknown = "unknown" )
Failure codes returned by Humanize.
const ( SeverityOK = "ok" SeverityWarn = "warn" SeverityBlock = "block" )
Severity of one preflight check.
const ( StatusPending = "pending" StatusCopying = "copying" StatusVerified = "verified" StatusFailed = "failed" StatusUnsupported = "unsupported" )
Copy statuses reported per database.
const ( EnginePostgres = "postgres" EngineMySQL = "mysql" EngineMariaDB = "mariadb" EngineMongoDB = "mongodb" EngineRedis = "redis" )
Engine ids this package can copy.
const DefaultHelperVersion = "17"
DefaultHelperVersion is the Postgres image tag the read-only helper runs. pg_dump needs a client at least as new as the source server.
const DefaultMaxTTL = 300
DefaultMaxTTL is the TTL in seconds above which a cutover should wait.
const PgvectorMinMajor = 13
PgvectorMinMajor is the oldest Postgres major the pgvector images exist for.
const PgvectorVariantAvailable = true
PgvectorVariantAvailable gates the "<major>-pgvector" managed Postgres version, which the database reconciler maps to the pgvector image.
const PgvectorVersionSuffix = "-pgvector"
PgvectorVersionSuffix is appended to a Postgres major to select the variant.
Variables ¶
This section is empty.
Functions ¶
func DeriveTargetNames ¶
DeriveTargetNames maps source database names to unique, valid managed names. Existing names (already taken) are avoided with a numeric suffix.
func DiskFreeCommand ¶
func DiskFreeCommand() []string
DiskFreeCommand prints "free|bytes" for the filesystem the helper runs on, the host's Docker storage that the managed database will also write to.
func DumpCommand ¶
DumpCommand streams a logical dump of the source on stdout. It runs in the helper container, where the SRC_* variables are set.
func EngineFromImage ¶
EngineFromImage maps an image reference to an engine id, "" when it is not a database this package can read.
func ExplainReachFailure ¶
ExplainReachFailure turns a connection failure into a plain explanation, naming the Docker network isolation case when the source is not a local container. The raw error is always kept in full after it.
func InventoryDatabaseCommand ¶
InventoryDatabaseCommand prints "tables|N" and "ext|name" lines for the Postgres database passed as the script's first argument. The name travels as a positional argument, so it never reaches a shell or SQL string.
func InventoryListCommand ¶
InventoryListCommand lists the server version and its databases. Postgres prints "db|name|bytes"; the others also print a table count.
func InventorySupported ¶
InventorySupported reports whether a server's databases can be listed.
func Overall ¶
func Overall(plans []DomainPlan) string
Overall is the verdict for a whole migration: the worst of its domains.
func ParseCounts ¶
ParseCounts reads "name|count" lines, ignoring anything that is not one (server notices, blank lines). The last field is the count, so a table name containing "|" still parses.
func Redact ¶
Redact removes passwords and credentials embedded in connection strings from raw error text.
func SourceCountCommand ¶
SourceCountCommand prints "name|count" lines for the source, run in the helper.
func TargetCountCommand ¶
TargetCountCommand is SourceCountCommand for the managed database, run in its own container where the engine's own credential variables are already set.
func ValidTargetName ¶
ValidTargetName reports whether name can be a managed database name.
func ValidateSSHTarget ¶
ValidateSSHTarget accepts only user@host.
Types ¶
type Check ¶
type Check struct {
ID string `json:"id"`
Status string `json:"status"`
Detail string `json:"detail"`
Fix string `json:"fix,omitempty"`
}
Check is one named result with what to do about it.
type Copier ¶
type Copier struct {
Runtime docker.Runtime
Restorer Restorer
Logger *slog.Logger
// VerifyAttempts and VerifyInterval retry the target count while the
// engine is still loading the restored data. Zero values use defaults.
// HelperNetwork is a Docker network the helper joins instead of the
// default bridge, so a source container on it is reachable.
HelperNetwork string
VerifyAttempts int
VerifyInterval time.Duration
}
Copier moves a source database into a managed database through a helper container on the target's node. It only uses the Docker Engine API.
type DNSResolver ¶
DNSResolver asks the domain's authoritative servers for the real TTL and falls back to the system resolver, which can only show a decayed one.
func (DNSResolver) Lookup ¶
func (r DNSResolver) Lookup(ctx context.Context, domain string) DomainState
Lookup implements Resolver.
type DatabaseInfo ¶
type DatabaseInfo struct {
Name string `json:"name"`
SizeBytes int64 `json:"size_bytes"`
Tables int `json:"tables"`
Extensions []string `json:"extensions,omitempty"`
}
DatabaseInfo is one database found on a source server.
type DomainPlan ¶
type DomainPlan struct {
App string `json:"app"`
Domain string `json:"domain"`
Verdict string `json:"verdict"`
Checks []Check `json:"checks"`
DNS DomainState `json:"dns"`
Change *RecordChange `json:"change,omitempty"`
}
DomainPlan is the go or no-go for one domain.
func Evaluate ¶
func Evaluate(in EvalInput) DomainPlan
Evaluate decides whether switching domain's DNS to this platform is safe. The source keeps serving until the operator makes the change.
type DomainState ¶
type DomainState struct {
Addrs []string `json:"addrs,omitempty"`
CNAME string `json:"cname,omitempty"`
// TTL is the TTL in seconds of the record the operator edits, 0 when unknown.
TTL uint32 `json:"ttl"`
// Authoritative is false when TTL came from a caching resolver, so the
// real value may be higher than what it shows.
Authoritative bool `json:"authoritative"`
Error string `json:"error,omitempty"`
}
DomainState is what public DNS says about a domain right now.
type EvalInput ¶
type EvalInput struct {
App string
Domain string
Ready bool
ReadyDetail string
DNS DomainState
TargetIPs []string
MaxTTL uint32
}
EvalInput is everything a cutover decision for one domain depends on.
type FailureInfo ¶
type FailureInfo struct {
Code string `json:"code"`
Summary string `json:"summary"`
Hint string `json:"hint"`
Technical string `json:"technical,omitempty"`
}
FailureInfo is a copy failure in words an operator can act on. Technical holds the redacted raw error, to be shown only on request.
func Humanize ¶
func Humanize(raw, fallbackHost string, fallbackPort int) FailureInfo
Humanize maps a raw copy error to a short sentence with a fix hint. The fallback host and port come from the recorded copy and are used when the error text does not name them.
type Inspector ¶
type Inspector struct {
Runtime docker.Runtime
Logger *slog.Logger
HelperVersion string
// HelperNetwork is the Docker network the helper joins, see Copier.
HelperNetwork string
}
Inspector reads a source server through a short-lived helper container.
type Inventory ¶
type Inventory struct {
Version string `json:"version"`
Major int `json:"major"`
FreeBytes int64 `json:"free_bytes"`
Databases []DatabaseInfo `json:"databases"`
}
Inventory is what a read-only look at a source server found.
func ParseInventory ¶
ParseInventory reads the output of InventoryListCommand.
type LocalSource ¶
type LocalSource struct {
Container string `json:"container"`
Image string `json:"image"`
Engine string `json:"engine"`
Running bool `json:"running"`
Network string `json:"network,omitempty"`
Host string `json:"host,omitempty"`
Port int `json:"port,omitempty"`
// Problem is why this container cannot be used, empty when it can.
Problem string `json:"problem,omitempty"`
}
LocalSource is a database container on this daemon, resolved to how the helper must reach it: join Network and connect to Host:Port.
func SelectLocalSource ¶
func SelectLocalSource(c docker.LocalContainer) LocalSource
SelectLocalSource decides which network the helper joins and what address it uses. A user-defined network wins and the container name is used (embedded DNS). Only the default bridge falls back to the internal IP. It never uses a published host address: Docker's bridge isolation drops that traffic.
type PreflightCheck ¶
type PreflightCheck struct {
ID string `json:"id"`
Severity string `json:"severity"`
Message string `json:"message"`
NextAction string `json:"next_action,omitempty"`
}
PreflightCheck is one preflight finding with what to do about it.
type PreflightInput ¶
type PreflightInput struct {
Engine string
DB DatabaseInfo
SourceMajor int
TargetName string
TargetVersion string
// TargetExists is true when a managed database already has TargetName,
// OwnTarget when this migration created it earlier (so a retry is fine).
TargetExists bool
OwnTarget bool
// FreeBytes is the target node's free disk, negative when unknown.
FreeBytes int64
// DiskMargin multiplies the database size to get the space required.
DiskMargin float64
// MBPerSecond is the assumed copy throughput for the time estimate.
MBPerSecond float64
}
PreflightInput is everything a database's preflight needs, gathered by the caller.
type PreflightResult ¶
type PreflightResult struct {
Checks []PreflightCheck `json:"checks"`
TargetVersion string `json:"target_version"`
RequiredBytes int64 `json:"required_bytes"`
EstimateSeconds int `json:"estimate_seconds"`
Blocked bool `json:"blocked"`
}
PreflightResult is the checks plus the target version the plan resolved to.
func Preflight ¶
func Preflight(in PreflightInput) PreflightResult
Preflight evaluates one database before anything is written. It never drops an extension: a missing one blocks the copy with a next action.
type Prober ¶
type Prober struct {
Resolver Resolver
// Dial opens a TCP connection, used to reach this node directly.
Dial func(ctx context.Context, network, addr string) (net.Conn, error)
// RootCAs overrides the system roots, for a private CA.
RootCAs *x509.CertPool
Timeout time.Duration
}
Prober is how PostSwitch reaches the network, replaceable in tests.
func (Prober) PostSwitch ¶
func (p Prober) PostSwitch(ctx context.Context, domain, healthPath string, targets []string) []Check
PostSwitch checks domain is served from this node: DNS resolves here, a trusted certificate is presented, and healthPath answers 200. TLS and HTTP go to this node's addresses with the domain as SNI, so they pass or fail independently of how far DNS has propagated.
type RecordChange ¶
type RecordChange struct {
Type string `json:"type"`
Name string `json:"name"`
Value string `json:"value"`
TTL uint32 `json:"ttl"`
Replaces string `json:"replaces,omitempty"`
}
RecordChange is the exact DNS edit to make.
type Resolver ¶
type Resolver interface {
Lookup(ctx context.Context, domain string) DomainState
}
Resolver reads what public DNS says about a domain.
type Restorer ¶
type Restorer interface {
Restore(ctx context.Context, engine, containerName string, dump io.Reader) error
}
Restorer replays a dump into a running database container. internal/backup's ContainerRestorer satisfies it and replaces the target's contents, so a repeated copy is idempotent.
type Source ¶
type Source struct {
Host string `json:"host"`
Port int `json:"port,omitempty"`
User string `json:"user,omitempty"`
Password string `json:"password,omitempty"`
Database string `json:"database,omitempty"`
// AuthDatabase is MongoDB's authentication database, "admin" when empty.
AuthDatabase string `json:"auth_database,omitempty"`
TLS bool `json:"tls,omitempty"`
}
Source locates the database to copy from. Password is held in memory only.
type TableCount ¶
type TableCount struct {
Name string `json:"name"`
Source int64 `json:"source"`
Target int64 `json:"target"`
OK bool `json:"ok"`
}
TableCount is one table (or collection) compared across both sides.
type Verification ¶
type Verification struct {
Checked int `json:"checked"`
Mismatched int `json:"mismatched"`
Tables []TableCount `json:"tables,omitempty"`
}
Verification is the outcome of comparing per-table row counts.
func Compare ¶
func Compare(source, target map[string]int64) Verification
Compare checks every source table exists on the target with the same count. Tables only the target has are ignored: restore tooling may add some.
func (Verification) Failure ¶
func (v Verification) Failure(limit int) string
Failure summarises a verification that did not pass, listing at most max tables.
type VolumeGuide ¶
type VolumeGuide struct {
App string `json:"app"`
Kind string `json:"kind"`
Source string `json:"source"`
Target string `json:"target"`
ContainerPath string `json:"container_path"`
Command string `json:"command"`
}
VolumeGuide is a command the operator runs on this node to copy one volume. Volume contents are not moved by the control plane: it would need root on the source host, which is the operator's call to make.
func GuideVolumes ¶
func GuideVolumes(app, appPrefix, sourceSSH string, vols []VolumeRef) []VolumeGuide
GuideVolumes builds one rsync command per volume, to run as root on the node that hosts app. appPrefix is the prefix imports put on volume names ("app-<app>-"), so the source volume name is recovered from it.
type VolumeRef ¶
type VolumeRef struct {
// Name is the Docker volume name here, empty for a bind mount.
Name string
ContainerPath string
HostPath string
// SourceName is the volume's real name on the source host, when it is
// not derivable from Name by trimming the app prefix.
SourceName string
}
VolumeRef is one persistent mount of an imported app.