Documentation
¶
Overview ¶
Package datamigrate copies live data from a source database into a managed database here, and checks a cutover is safe. It never stores source credentials.
Index ¶
- Constants
- Variables
- func CreateRolesCommand(roles []string) ([]string, error)
- func DefaultPort(engine string) int
- func DeriveTargetNames(sources []string, taken map[string]bool) map[string]string
- func DiskFreeCommand() []string
- func DumpCommand(engine string) ([]string, error)
- func EngineFromImage(image string) string
- func ExplainReachFailure(raw string, local bool) string
- func HostedProbeCommand() []string
- func HumanBytes(n int64) string
- func HumanDuration(sec int) string
- func InventoryDatabaseCommand(dbName string) []string
- func InventoryListCommand(engine string) ([]string, error)
- func InventorySupported(engine string) bool
- func MajorOf(version string) int
- func MissingRoles(policyRoles, targetRoles []string) []string
- func ObjectCountsCommand() []string
- func Overall(plans []DomainPlan) string
- func ParseCounts(out string) map[string]int64
- func Redact(raw string) string
- func RedactURL(raw string) string
- func SourceCountCommand(engine string) ([]string, error)
- func Supported(engine string) bool
- func TargetCountCommand(engine string) ([]string, error)
- func TargetStateCommand() []string
- func ValidTargetName(name string) bool
- func ValidateSSHTarget(s string) error
- type Check
- type Copier
- func (c *Copier) Copy(ctx context.Context, t Target, src Source) (Verification, error)
- func (c *Copier) MoveHosted(ctx context.Context, plan HostedPlan, cfg HostedConfig, ...) (HostedResult, error)
- func (c *Copier) PreflightHosted(ctx context.Context, plan HostedPlan, cfg HostedConfig, ...) (HostedSource, HostedResult, error)
- type DNSResolver
- type DatabaseInfo
- type DomainPlan
- type DomainState
- type EvalInput
- type FailureInfo
- type HostedConfig
- type HostedPlan
- type HostedPreflightInput
- type HostedProbe
- type HostedResult
- type HostedSource
- type Inspector
- type Inventory
- type LocalSource
- type ObjectCounts
- type PreflightCheck
- type PreflightInput
- type PreflightResult
- type Prober
- type RecordChange
- type Resolver
- type Restorer
- type SecretURL
- type Source
- type TableCount
- type Target
- type TargetState
- type Verification
- type VolumeGuide
- type VolumeRef
Constants ¶
const ( CheckPass = "pass" CheckWarn = "warn" CheckFail = "fail" )
Check outcomes.
const ( VerdictGo = "go" VerdictWait = "wait" VerdictNoGo = "no-go" VerdictSwitch = "switched" )
Cutover verdicts.
const ( // EnvHostedAllowInsecure lets a public host be reached without TLS. EnvHostedAllowInsecure = "APP_MIGRATE_HOSTED_ALLOW_INSECURE" // EnvHostedMaxBytes is the size above which a dump and restore is not // recommended and needs an explicit override. EnvHostedMaxBytes = "APP_MIGRATE_HOSTED_MAX_BYTES" )
Env names for the hosted Postgres move, in the style of the migration hub.
const ( FailureTimeout = "source_timeout" FailureRefused = "source_refused" FailureDNS = "source_dns" FailureAuth = "auth_failed" FailureNoDatabase = "database_missing" FailureSSL = "ssl_required" FailurePermission = "permission_denied" FailureDisk = "out_of_disk" FailureNotRunning = "target_not_running" FailureMismatch = "verify_mismatch" FailureUnknown = "unknown" )
Failure codes returned by Humanize.
const ( SeverityOK = "ok" SeverityWarn = "warn" SeverityBlock = "block" )
Severity of one preflight check.
const ( StatusPending = "pending" StatusCopying = "copying" StatusVerified = "verified" StatusFailed = "failed" StatusUnsupported = "unsupported" )
Copy statuses reported per database.
const ( EnginePostgres = "postgres" EngineMySQL = "mysql" EngineMariaDB = "mariadb" EngineMongoDB = "mongodb" EngineRedis = "redis" )
Engine ids this package can copy.
const DefaultHelperVersion = "17"
DefaultHelperVersion is the Postgres image tag the read-only helper runs. pg_dump needs a client at least as new as the source server.
const DefaultHostedMaxBytes int64 = 100 << 30
DefaultHostedMaxBytes is Heroku's own dump and restore ceiling (about 100 GB), used only when EnvHostedMaxBytes is unset.
const DefaultMaxTTL = 300
DefaultMaxTTL is the TTL in seconds above which a cutover should wait.
const PgvectorMinMajor = 13
PgvectorMinMajor is the oldest Postgres major the pgvector images exist for.
const PgvectorVariantAvailable = true
PgvectorVariantAvailable gates the "<major>-pgvector" managed Postgres version, which the database reconciler maps to the pgvector image.
const PgvectorVersionSuffix = "-pgvector"
PgvectorVersionSuffix is appended to a Postgres major to select the variant.
Variables ¶
var ErrPreflightBlocked = errors.New("hosted database preflight is blocked")
ErrPreflightBlocked wraps the blocking findings of a hosted preflight.
var ErrWritesNotFrozen = errors.New("writes to the source are not frozen: put the old app in maintenance mode and set writes_frozen before moving")
ErrWritesNotFrozen is returned when a move is attempted before the caller acknowledged that the source stopped taking writes.
Functions ¶
func CreateRolesCommand ¶
CreateRolesCommand creates each role NOLOGIN NOINHERIT when absent, so it is safe to repeat. Names outside a conservative character set are refused rather than quoted, because they come from the source database.
func DeriveTargetNames ¶
DeriveTargetNames maps source database names to unique, valid managed names. Existing names (already taken) are avoided with a numeric suffix.
func DiskFreeCommand ¶
func DiskFreeCommand() []string
DiskFreeCommand prints "free|bytes" for the filesystem the helper runs on, the host's Docker storage that the managed database will also write to.
func DumpCommand ¶
DumpCommand streams a logical dump of the source on stdout. It runs in the helper container, where the SRC_* variables are set.
func EngineFromImage ¶
EngineFromImage maps an image reference to an engine id, "" when it is not a database this package can read.
func ExplainReachFailure ¶
ExplainReachFailure turns a connection failure into a plain explanation, naming the Docker network isolation case when the source is not a local container. The raw error is always kept in full after it.
func HostedProbeCommand ¶
func HostedProbeCommand() []string
HostedProbeCommand prints "key|value" lines for the connected database. It is read-only (the helper sets default_transaction_read_only) and the same facts as `SHOW server_version_num` and pg_database_size.
func InventoryDatabaseCommand ¶
InventoryDatabaseCommand prints "tables|N" and "ext|name" lines for the Postgres database passed as the script's first argument. The name travels as a positional argument, so it never reaches a shell or SQL string.
func InventoryListCommand ¶
InventoryListCommand lists the server version and its databases. Postgres prints "db|name|bytes"; the others also print a table count.
func InventorySupported ¶
InventorySupported reports whether a server's databases can be listed.
func MissingRoles ¶
MissingRoles lists the policy roles the target does not have. Role names are case sensitive in Postgres, so the comparison is exact.
func ObjectCountsCommand ¶
func ObjectCountsCommand() []string
ObjectCountsCommand counts the non-table objects on the target after the restore.
func Overall ¶
func Overall(plans []DomainPlan) string
Overall is the verdict for a whole migration: the worst of its domains.
func ParseCounts ¶
ParseCounts reads "name|count" lines, ignoring anything that is not one (server notices, blank lines). The last field is the count, so a table name containing "|" still parses.
func Redact ¶
Redact removes passwords and credentials embedded in connection strings from raw error text.
func RedactURL ¶
RedactURL masks the password of a connection URL without parsing it, so a malformed URL cannot make an error echo the secret.
func SourceCountCommand ¶
SourceCountCommand prints "name|count" lines for the source, run in the helper.
func TargetCountCommand ¶
TargetCountCommand is SourceCountCommand for the managed database, run in its own container where the engine's own credential variables are already set.
func TargetStateCommand ¶
func TargetStateCommand() []string
TargetStateCommand prints the target cluster's roles and its user table count. It runs inside the managed container, where the superuser variables are set.
func ValidTargetName ¶
ValidTargetName reports whether name can be a managed database name.
func ValidateSSHTarget ¶
ValidateSSHTarget accepts only user@host.
Types ¶
type Check ¶
type Check struct {
ID string `json:"id"`
Status string `json:"status"`
Detail string `json:"detail"`
Fix string `json:"fix,omitempty"`
}
Check is one named result with what to do about it.
type Copier ¶
type Copier struct {
Runtime docker.Runtime
Restorer Restorer
Logger *slog.Logger
// VerifyAttempts and VerifyInterval retry the target count while the
// engine is still loading the restored data. Zero values use defaults.
// HelperNetwork is a Docker network the helper joins instead of the
// default bridge, so a source container on it is reachable.
HelperNetwork string
VerifyAttempts int
VerifyInterval time.Duration
}
Copier moves a source database into a managed database through a helper container on the target's node. It only uses the Docker Engine API.
func (*Copier) Copy ¶
Copy runs the full copy and verification. Every error is scrubbed of the source password. The helper container is always removed.
func (*Copier) MoveHosted ¶
func (c *Copier) MoveHosted(ctx context.Context, plan HostedPlan, cfg HostedConfig, diskMargin, mbps float64) (HostedResult, error)
MoveHosted copies a hosted Postgres database into a managed one: refuse without writes_frozen, preflight, dump with --no-owner --no-privileges, restore through the filtering restorer and compare per-table row counts. The helper image tag is the target major, which preflight keeps at or above the source's. Errors are scrubbed of the URL and password.
func (*Copier) PreflightHosted ¶
func (c *Copier) PreflightHosted(ctx context.Context, plan HostedPlan, cfg HostedConfig, diskMargin, mbps float64) (HostedSource, HostedResult, error)
PreflightHosted validates the URL, probes the database with read-only queries and returns the findings. Nothing is dumped or written.
type DNSResolver ¶
DNSResolver asks the domain's authoritative servers for the real TTL and falls back to the system resolver, which can only show a decayed one.
func (DNSResolver) Lookup ¶
func (r DNSResolver) Lookup(ctx context.Context, domain string) DomainState
Lookup implements Resolver.
type DatabaseInfo ¶
type DatabaseInfo struct {
Name string `json:"name"`
SizeBytes int64 `json:"size_bytes"`
Tables int `json:"tables"`
Extensions []string `json:"extensions,omitempty"`
}
DatabaseInfo is one database found on a source server.
type DomainPlan ¶
type DomainPlan struct {
App string `json:"app"`
Domain string `json:"domain"`
Verdict string `json:"verdict"`
Checks []Check `json:"checks"`
DNS DomainState `json:"dns"`
Change *RecordChange `json:"change,omitempty"`
}
DomainPlan is the go or no-go for one domain.
func Evaluate ¶
func Evaluate(in EvalInput) DomainPlan
Evaluate decides whether switching domain's DNS to this platform is safe. The source keeps serving until the operator makes the change.
type DomainState ¶
type DomainState struct {
Addrs []string `json:"addrs,omitempty"`
CNAME string `json:"cname,omitempty"`
// TTL is the TTL in seconds of the record the operator edits, 0 when unknown.
TTL uint32 `json:"ttl"`
// Authoritative is false when TTL came from a caching resolver, so the
// real value may be higher than what it shows.
Authoritative bool `json:"authoritative"`
Error string `json:"error,omitempty"`
}
DomainState is what public DNS says about a domain right now.
type EvalInput ¶
type EvalInput struct {
App string
Domain string
Ready bool
ReadyDetail string
DNS DomainState
TargetIPs []string
MaxTTL uint32
}
EvalInput is everything a cutover decision for one domain depends on.
type FailureInfo ¶
type FailureInfo struct {
Code string `json:"code"`
Summary string `json:"summary"`
Hint string `json:"hint"`
Technical string `json:"technical,omitempty"`
}
FailureInfo is a copy failure in words an operator can act on. Technical holds the redacted raw error, to be shown only on request.
func Humanize ¶
func Humanize(raw, fallbackHost string, fallbackPort int) FailureInfo
Humanize maps a raw copy error to a short sentence with a fix hint. The fallback host and port come from the recorded copy and are used when the error text does not name them.
type HostedConfig ¶
HostedConfig is the operator policy for hosted Postgres sources.
func HostedConfigFromEnv ¶
func HostedConfigFromEnv(getenv func(string) string) (HostedConfig, error)
HostedConfigFromEnv reads the policy through getenv (os.Getenv in production).
func HostedConfigFromProcessEnv ¶
func HostedConfigFromProcessEnv() (HostedConfig, error)
HostedConfigFromProcessEnv is HostedConfigFromEnv over the process environment.
type HostedPlan ¶
type HostedPlan struct {
// URL is the source connection URL. It prints and marshals redacted.
URL SecretURL `json:"url"`
// TargetName is the new managed database name; TargetVersion defaults to
// the source's major (with the pgvector variant when needed).
TargetName string `json:"target_name"`
TargetVersion string `json:"target_version,omitempty"`
TargetExists bool `json:"target_exists,omitempty"`
OwnTarget bool `json:"own_target,omitempty"`
// WritesFrozen is the caller's acknowledgement that the old app is in
// maintenance mode (Heroku: `heroku maintenance:on -a <app>`) and nothing
// writes to the source. A dump taken while it changes is not a copy.
WritesFrozen bool `json:"writes_frozen"`
// AllowLarge overrides the size ceiling after reading the replication advice.
AllowLarge bool `json:"allow_large,omitempty"`
// CreateMissingRoles creates the roles named by row level security
// policies on the target (NOLOGIN NOINHERIT) before the restore.
CreateMissingRoles bool `json:"create_missing_roles,omitempty"`
}
HostedPlan is the caller's request to move one hosted database.
type HostedPreflightInput ¶
type HostedPreflightInput struct {
Plan HostedPlan
Source HostedSource
Probe HostedProbe
// Target is what the managed database already holds.
Target TargetState
Config HostedConfig
DiskMargin float64
MBPerSecond float64
}
HostedPreflightInput is the gathered facts for HostedPreflight.
type HostedProbe ¶
type HostedProbe struct {
Version string `json:"version"`
VersionNum int `json:"version_num"`
Major int `json:"major"`
SizeBytes int64 `json:"size_bytes"`
Tables int `json:"tables"`
Extensions []string `json:"extensions,omitempty"`
// PolicyRoles are roles named by row level security policies. Hosted
// providers own them, so they will not exist on the target.
PolicyRoles []string `json:"policy_roles,omitempty"`
FreeBytes int64 `json:"free_bytes"`
}
HostedProbe is what read-only queries found on a hosted database.
func ParseHostedProbe ¶
func ParseHostedProbe(out string) HostedProbe
ParseHostedProbe reads the output of HostedProbeCommand, ignoring notices.
type HostedResult ¶
type HostedResult struct {
Preflight PreflightResult `json:"preflight"`
Probe HostedProbe `json:"probe"`
// Restore holds what the restore changed to tolerate provider objects.
Restore []PreflightCheck `json:"restore,omitempty"`
// MissingRoles are the policy roles absent from the target at preflight time.
MissingRoles []string `json:"missing_roles,omitempty"`
Verification Verification `json:"verification"`
Redacted string `json:"source"`
}
HostedResult is the outcome of PreflightHosted or MoveHosted.
type HostedSource ¶
type HostedSource struct {
Source Source
// Redacted is the URL with the password masked, safe to log and store.
Redacted string
// Private is true when the host is a loopback, private or internal name.
Private bool
// contains filtered or unexported fields
}
HostedSource is a validated postgres:// URL, split into a Source. Obtain the URL from the provider, for Heroku `heroku config:get DATABASE_URL -a <app>`.
func ParseHostedURL ¶
func ParseHostedURL(raw string, cfg HostedConfig) (HostedSource, error)
ParseHostedURL validates raw and splits it. Errors never echo the URL. A public host must use TLS: a missing sslmode becomes require, and an explicit disable, allow or prefer is refused unless cfg.AllowInsecure.
func (HostedSource) GoString ¶
func (h HostedSource) GoString() string
GoString is String, so %#v is redacted too.
func (HostedSource) Scrub ¶
func (h HostedSource) Scrub(text string) string
Scrub removes the URL and its password from any text, for errors and plans.
func (HostedSource) String ¶
func (h HostedSource) String() string
String and GoString keep the credentials out of fmt output.
type Inspector ¶
type Inspector struct {
Runtime docker.Runtime
Logger *slog.Logger
HelperVersion string
// HelperNetwork is the Docker network the helper joins, see Copier.
HelperNetwork string
}
Inspector reads a source server through a short-lived helper container.
func (*Inspector) Inventory ¶
Inventory lists the databases on the source server. It only runs read-only statements, and every error is scrubbed of the source password.
func (*Inspector) ProbeHosted ¶
func (i *Inspector) ProbeHosted(ctx context.Context, hs HostedSource) (HostedProbe, error)
ProbeHosted measures a hosted database through a short-lived helper. Every error is scrubbed of the URL and password.
type Inventory ¶
type Inventory struct {
Version string `json:"version"`
Major int `json:"major"`
FreeBytes int64 `json:"free_bytes"`
Databases []DatabaseInfo `json:"databases"`
}
Inventory is what a read-only look at a source server found.
func ParseInventory ¶
ParseInventory reads the output of InventoryListCommand.
type LocalSource ¶
type LocalSource struct {
Container string `json:"container"`
Image string `json:"image"`
Engine string `json:"engine"`
Running bool `json:"running"`
Network string `json:"network,omitempty"`
Host string `json:"host,omitempty"`
Port int `json:"port,omitempty"`
// Problem is why this container cannot be used, empty when it can.
Problem string `json:"problem,omitempty"`
}
LocalSource is a database container on this daemon, resolved to how the helper must reach it: join Network and connect to Host:Port.
func SelectLocalSource ¶
func SelectLocalSource(c docker.LocalContainer) LocalSource
SelectLocalSource decides which network the helper joins and what address it uses. A user-defined network wins and the container name is used (embedded DNS). Only the default bridge falls back to the internal IP. It never uses a published host address: Docker's bridge isolation drops that traffic.
type ObjectCounts ¶
type ObjectCounts struct {
Sequences int `json:"sequences"`
Views int `json:"views"`
Functions int `json:"functions"`
Extensions int `json:"extensions"`
}
ObjectCounts are the non-table objects present on the target after a restore.
func ParseObjectCounts ¶
func ParseObjectCounts(out string) *ObjectCounts
ParseObjectCounts reads the output of ObjectCountsCommand.
type PreflightCheck ¶
type PreflightCheck struct {
ID string `json:"id"`
Severity string `json:"severity"`
Message string `json:"message"`
NextAction string `json:"next_action,omitempty"`
}
PreflightCheck is one preflight finding with what to do about it.
type PreflightInput ¶
type PreflightInput struct {
Engine string
DB DatabaseInfo
SourceMajor int
TargetName string
TargetVersion string
// TargetExists is true when a managed database already has TargetName,
// OwnTarget when this migration created it earlier (so a retry is fine).
TargetExists bool
OwnTarget bool
// FreeBytes is the target node's free disk, negative when unknown.
FreeBytes int64
// DiskMargin multiplies the database size to get the space required.
DiskMargin float64
// MBPerSecond is the assumed copy throughput for the time estimate.
MBPerSecond float64
}
PreflightInput is everything a database's preflight needs, gathered by the caller.
type PreflightResult ¶
type PreflightResult struct {
Checks []PreflightCheck `json:"checks"`
TargetVersion string `json:"target_version"`
RequiredBytes int64 `json:"required_bytes"`
EstimateSeconds int `json:"estimate_seconds"`
Blocked bool `json:"blocked"`
}
PreflightResult is the checks plus the target version the plan resolved to.
func HostedPreflight ¶
func HostedPreflight(in HostedPreflightInput) PreflightResult
HostedPreflight runs the standard Postgres preflight on the probe, then adds the hosted checks: quiesced writes, size ceiling, TLS, roles. Nothing has been dumped or written when it runs.
func Preflight ¶
func Preflight(in PreflightInput) PreflightResult
Preflight evaluates one database before anything is written. It never drops an extension: a missing one blocks the copy with a next action.
type Prober ¶
type Prober struct {
Resolver Resolver
// Dial opens a TCP connection, used to reach this node directly.
Dial func(ctx context.Context, network, addr string) (net.Conn, error)
// RootCAs overrides the system roots, for a private CA.
RootCAs *x509.CertPool
Timeout time.Duration
}
Prober is how PostSwitch reaches the network, replaceable in tests.
func (Prober) PostSwitch ¶
func (p Prober) PostSwitch(ctx context.Context, domain, healthPath string, targets []string) []Check
PostSwitch checks domain is served from this node: DNS resolves here, a trusted certificate is presented, and healthPath answers 200. TLS and HTTP go to this node's addresses with the domain as SNI, so they pass or fail independently of how far DNS has propagated.
type RecordChange ¶
type RecordChange struct {
Type string `json:"type"`
Name string `json:"name"`
Value string `json:"value"`
TTL uint32 `json:"ttl"`
Replaces string `json:"replaces,omitempty"`
}
RecordChange is the exact DNS edit to make.
type Resolver ¶
type Resolver interface {
Lookup(ctx context.Context, domain string) DomainState
}
Resolver reads what public DNS says about a domain.
type Restorer ¶
type Restorer interface {
Restore(ctx context.Context, engine, containerName string, dump io.Reader) error
}
Restorer replays a dump into a running database container. internal/backup's ContainerRestorer satisfies it and replaces the target's contents, so a repeated copy is idempotent.
type SecretURL ¶
type SecretURL string
SecretURL is a connection URL that prints, logs and marshals redacted, so a plan holding it cannot leak the password by accident.
func (SecretURL) MarshalText ¶
MarshalText stores only the redacted form.
type Source ¶
type Source struct {
Host string `json:"host"`
Port int `json:"port,omitempty"`
User string `json:"user,omitempty"`
Password string `json:"password,omitempty"`
Database string `json:"database,omitempty"`
// AuthDatabase is MongoDB's authentication database, "admin" when empty.
AuthDatabase string `json:"auth_database,omitempty"`
TLS bool `json:"tls,omitempty"`
// SSLMode is the libpq sslmode for Postgres when TLS is set, "require" when empty.
SSLMode string `json:"ssl_mode,omitempty"`
}
Source locates the database to copy from. Password is held in memory only.
type TableCount ¶
type TableCount struct {
Name string `json:"name"`
Source int64 `json:"source"`
Target int64 `json:"target"`
OK bool `json:"ok"`
}
TableCount is one table (or collection) compared across both sides.
type TargetState ¶
type TargetState struct {
// Known is false when the target could not be queried.
Known bool
Roles []string
// UserTables counts tables outside the system schemas: non-zero means an
// earlier run (or other data) lives there.
UserTables int
}
TargetState is what the managed target already holds, read before the move.
func ParseTargetState ¶
func ParseTargetState(out string) TargetState
ParseTargetState reads the output of TargetStateCommand.
type Verification ¶
type Verification struct {
Checked int `json:"checked"`
Mismatched int `json:"mismatched"`
Tables []TableCount `json:"tables,omitempty"`
// Objects counts what else arrived (hosted Postgres moves only).
Objects *ObjectCounts `json:"objects,omitempty"`
// Findings are what the move changed or created on the way (hosted moves only).
Findings []PreflightCheck `json:"findings,omitempty"`
}
Verification is the outcome of comparing per-table row counts.
func Compare ¶
func Compare(source, target map[string]int64) Verification
Compare checks every source table exists on the target with the same count. Tables only the target has are ignored: restore tooling may add some.
func (Verification) Failure ¶
func (v Verification) Failure(limit int) string
Failure summarises a verification that did not pass, listing at most max tables.
type VolumeGuide ¶
type VolumeGuide struct {
App string `json:"app"`
Kind string `json:"kind"`
Source string `json:"source"`
Target string `json:"target"`
ContainerPath string `json:"container_path"`
Command string `json:"command"`
}
VolumeGuide is a command the operator runs on this node to copy one volume. Volume contents are not moved by the control plane: it would need root on the source host, which is the operator's call to make.
func GuideVolumes ¶
func GuideVolumes(app, appPrefix, sourceSSH string, vols []VolumeRef) []VolumeGuide
GuideVolumes builds one rsync command per volume, to run as root on the node that hosts app. appPrefix is the prefix imports put on volume names ("app-<app>-"), so the source volume name is recovered from it.
type VolumeRef ¶
type VolumeRef struct {
// Name is the Docker volume name here, empty for a bind mount.
Name string
ContainerPath string
HostPath string
// SourceName is the volume's real name on the source host, when it is
// not derivable from Name by trimming the app prefix.
SourceName string
}
VolumeRef is one persistent mount of an imported app.