migrate

package module
v1.0.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 26, 2026 License: MIT Imports: 18 Imported by: 0

README

gFly DB Migrate

Copyright © 2026, gFly
https://www.gfly.dev
All rights reserved.

Runs versioned *.sql migration files — named YYYYMMDD_HHMMSS_description.up.sql / .down.sql, a UTC timestamp to the second — against PostgreSQL or MySQL, tracking each file's state — applied/rolled back, batch, run count, checksum — in a migrations table it manages itself.

The timestamp (not a small sequential counter, and not configurable to anything else) is deliberate: when two people on separate branches each pick "the next number," they either collide or, worse, silently don't collide but sort in an order neither of them intended once merged. A per-second timestamp makes that collision astronomically unlikely and keeps the merged order matching the order each file was actually written in. --new (below) generates one.

Install

go get -u github.com/gflydev/db/migrate@latest

# PostgreSQL
go get -u github.com/gflydev/db/migrate/postgres@latest
# MySQL
go get -u github.com/gflydev/db/migrate/mysql@latest

Usage

Wire one case into your app's CLI entrypoint:

import (
	"os"

	dbmigrate "github.com/gflydev/db/migrate"
	dbmigratePg "github.com/gflydev/db/migrate/postgres"
)

func main() {
	args := os.Args[1:]
	switch {
	case len(args) > 0 && args[0] == "db:migrate":
		os.Exit(dbmigrate.RunCLI(args[1:], dbmigratePg.Dialect{}, "database/migrations"))
	}
}

Connection settings are read from DB_HOST, DB_PORT, DB_NAME, DB_USERNAME, DB_PASSWORD, DB_SSL_MODE (the same variables github.com/gflydev/db/psql and github.com/gflydev/db/mysql already use).

./build/artisan db:migrate                              # apply the single next pending migration
./build/artisan db:migrate --all                        # apply every pending migration (one batch)
./build/artisan db:migrate --down                       # roll back the single most recent migration
./build/artisan db:migrate --down --all                 # roll back every migration in the latest batch
./build/artisan db:migrate --status                     # show every migration's state
./build/artisan db:migrate --dry-run                    # combine with the above: print, don't execute
./build/artisan db:migrate --new=create_widgets_table   # create a new migration pair, timestamped now
./build/artisan db:migrate --baseline=20260101_000000   # mark files up to that timestamp as already applied
./build/artisan db:migrate help                         # same as -h/--help: print the flag list and examples above

Design notes

  • Each migration file runs in its own transaction on PostgreSQL; MySQL's DDL auto-commits, so a mid-file failure there cannot be rolled back and is reported as such.
  • One advisory lock (pg_advisory_lock / GET_LOCK) is held for a whole invocation, so two concurrent db:migrate runs serialize rather than race.
  • Before any command runs, every migration currently recorded as up has its on-disk checksum re-verified against what was recorded; a mismatch aborts (--force to proceed anyway).
  • --baseline is a one-time bootstrap for a database that already has this schema outside the tool's tracking — it never executes SQL, and refuses to run on a non-empty tracking table without --force.
  • Rolling back a migration keeps its tracking row (status flips to down, run_count and batch stay as history) rather than deleting it, so run_count survives repeated rollback/reapply cycles.
  • --status also reports any tracking-table row whose .sql files are no longer in the migrations directory (deleted or renamed after it ran) as FILE MISSING — without this check such a row would simply vanish from every command's view, staying in the table forever with no way to roll it back until the files are restored.
  • Load accepts exactly one filename shape — YYYYMMDD_HHMMSS_description.(up|down).sql with a real, parseable UTC timestamp — and refuses the whole directory (not just the offending file) if anything else is in it, including golang-migrate's older NNNNNN_description.up.sql sequential-number style. A migrations directory adopting this module for the first time must rename its existing files to the timestamp style before db:migrate will run at all.

Testing

go test ./...                     # this package: no database required
go test ./postgres/...            # SQL-string unit tests: no database required
go test -tags=integration ./postgres/...  # full up/down/status cycle against a real Postgres
go test ./mysql/...                # SQL-string unit tests: no database required
go test -tags=integration ./mysql/...      # full up/down/status cycle against a real MySQL

The integration build tag needs DB_* environment variables pointed at a disposable database — it creates and drops its own migrations/widgets tables and skips (not fails) if it can't reach a database.

Documentation

Overview

Package migrate runs versioned *.sql migration files against a database, tracking each file's state (applied/rolled back, batch, run count, checksum) in a tracking table it manages itself. It is dialect-agnostic: see the postgres and mysql subpackages for the two supported databases, and RunCLI for wiring it into an application's own command-line entrypoint.

File layout

Migrations live as pairs of files named "YYYYMMDD_HHMMSS_description.up.sql" and "YYYYMMDD_HHMMSS_description.down.sql" — a UTC timestamp, to the second, then a snake_case description — in a single directory; Load discovers and validates them. The timestamp (not a small sequential counter) is deliberate: two people working on separate branches each pick "the next number" independently and collide or, worse, don't collide but sort in an order neither of them intended once merged. A timestamp taken at the moment the file is created makes that collision astronomically unlikely and keeps the merged order matching the order each file was actually written in. New creates a fresh pair with the current timestamp.

Concurrency

A Migrator is not safe for concurrent use by multiple goroutines within one process — each exported method holds the underlying *sql.DB open for its own duration and returns before the next call should start. Concurrent processes are handled separately: every method acquires the dialect's advisory lock for its full duration, so two OS processes (e.g. two deploys racing) serialize instead of corrupting the tracking table.

Index

Constants

View Source
const (
	// DefaultTable is the tracking table name every adopter of this package uses. It is not
	// configurable (see NewMigrator) — one gFly app has one migrations directory and one
	// tracking table, and a configurable name would only invite drift between the table a
	// health check reads and the one db:migrate writes.
	DefaultTable = "migrations"

	// StatusUp and StatusDown are the two values Record.Status takes. They are exported so a
	// caller inspecting Status's results doesn't need to guess the tracking table's raw string
	// values, and so this package's own SQL (store.go) and each Dialect's generated SQL
	// (postgres, mysql) share one definition instead of three independently hand-typed copies.
	StatusUp   = "up"
	StatusDown = "down"
)

Variables

View Source
var (
	// ErrChecksumMismatch means a migration recorded as applied ("up") no longer matches the
	// sha256 checksum stored when it ran — its .up.sql file changed on disk since then. Up,
	// Down and Baseline all refuse to proceed past this unless force is true.
	ErrChecksumMismatch = errors.New("checksum mismatch: an applied migration changed on disk since it ran")

	// ErrNonEmptyTable means Baseline was called with force == false against a tracking table
	// that already has at least one row. Baseline is a one-time bootstrap, not a merge tool.
	ErrNonEmptyTable = errors.New("baseline refuses to run on a non-empty tracking table")

	// ErrUnknownVersion means the version passed to Baseline does not match any migration
	// discovered by Load.
	ErrUnknownVersion = errors.New("baseline version does not match any known migration")

	// ErrFilesMissing means the tracking table has a row for a migration whose .sql files are
	// no longer present in the migrations directory, so Down has nothing to execute for it.
	ErrFilesMissing = errors.New("migration is recorded as applied but its files are missing from disk")

	// ErrConflictingFlags means RunCLI was given --baseline together with --down or --all,
	// which the CLI does not support (baseline never applies or rolls back a range of files).
	ErrConflictingFlags = errors.New("--baseline cannot be combined with --down or --all")
)

Sentinel errors returned by Migrator. Wrap these with errors.New("%w: ...", ErrX) (this package's convention — see Migrator's own error sites — is github.com/gflydev/core/errors, not the standard library's, matching the rest of the gFly framework) rather than building a new, unmatched error string for the same condition. Callers, including this package's own CLI and tests, must be able to distinguish them with errors.Is instead of parsing error text.

These stay distinct from github.com/gflydev/core/errors' own sentinels (ItemNotFound, InvalidParameter, ...): those are shaped for mapping an HTTP request to a status code, and reusing one here would make an unrelated part of a gFly app's API layer match on a condition that has nothing to do with it.

Functions

func RunCLI

func RunCLI(args []string, dialect Dialect, dir string) int

RunCLI parses args (everything after "db:migrate" on the command line — see the package README for the full flag list), runs the requested operation against dialect's database, and returns a process exit code: 0 on success (including "nothing to do"), 1 on any error (printed to os.Stderr). A typical caller wires it in as:

case args[0] == "db:migrate":
    os.Exit(migrate.RunCLI(args[1:], postgres.Dialect{}, "database/migrations"))

Types

type Dialect

type Dialect interface {
	// Name identifies the dialect in log/error output, e.g. "postgres", "mysql".
	Name() string

	// Open reads DB_HOST/DB_PORT/DB_NAME/DB_USERNAME/DB_PASSWORD/DB_SSL_MODE and returns a
	// ready connection pool.
	Open() (*sql.DB, error)

	// CreateMigrationsTableSQL returns the CREATE TABLE IF NOT EXISTS statement for the
	// tracking table named table.
	CreateMigrationsTableSQL(table string) string

	// UpsertMigrationSQL returns a statement that inserts a new row for table, or updates the
	// existing row for the same migration name, setting status='up'. Parameter order is
	// (migration, batch, run_count, checksum).
	UpsertMigrationSQL(table string) string

	// Placeholder returns the positional-parameter placeholder for argument position argPos
	// (1-based): "$1", "$2", ... for Postgres, "?" for every position in MySQL.
	Placeholder(argPos int) string

	// Lock acquires a database-wide advisory lock scoped to conn, blocking (up to the
	// dialect's own timeout) until it is available or returning an error if it cannot be
	// acquired.
	Lock(ctx context.Context, conn *sql.Conn, key string) error

	// Unlock releases a lock acquired by Lock, using the same conn.
	Unlock(ctx context.Context, conn *sql.Conn, key string) error

	// SupportsTransactionalDDL reports whether a failing statement partway through a
	// migration file can be rolled back (true for Postgres, false for MySQL).
	SupportsTransactionalDDL() bool
}

Dialect is the seam between this package's orchestration and a specific database.

type Migration

type Migration struct {
	// Version is the UTC timestamp prefix in timestampLayout's shape, e.g. "20260512_230412".
	Version string
	// Name is the filename's identity: the timestamp and description, without the
	// .up.sql/.down.sql suffix, e.g. "20260512_230412_create_widgets_table".
	Name string
	// UpPath and DownPath are absolute (or working-directory-relative) paths to the two files.
	UpPath, DownPath string
}

Migration is one file pair discovered on disk by Load.

func Load

func Load(dir string) ([]Migration, error)

Load reads every *.sql file directly inside dir (not recursively), validates that each one matches "YYYYMMDD_HHMMSS_description.(up|down).sql" with a real, parseable timestamp, and that every "up" file has a matching "down" file and vice versa, and returns the migrations sorted ascending by version (equivalently, by timestamp, equivalently, by creation order).

Returns an error listing every offending filename if any file fails the naming pattern, has an invalid timestamp (e.g. month 13), or is missing its up/down counterpart — Load fails all-or-nothing rather than returning a partial, silently-incomplete set.

Load lists dir with the standard library's os.ReadDir rather than gflydev/storage: the migrations directory is part of the deployed source tree (like the framework's own view templates), not a caller-configurable storage disk, and IStorage has no directory-listing method to begin with.

func New

func New(dir, description string) (Migration, error)

New creates a fresh migration pair in dir, named "{current UTC timestamp}_{description}", and returns it as a Migration (with UpPath/DownPath already pointing at the new files). Both files are created with a one-line placeholder comment and nothing else — valid, no-op SQL until filled in.

description must already be snake_case ([a-z0-9_]+); RunCLI's --new flag validates this before calling New, and New itself re-validates so a direct library caller gets the same guarantee. Returns an error without creating anything if description is invalid, or if either target file already exists (only possible if New is called twice for the same description within the same second).

type Migrator

type Migrator struct {
	// contains filtered or unexported fields
}

Migrator runs the *.sql files in a directory against a database, tracking each one's state in a table it manages itself. Construct one with NewMigrator; a Migrator is cheap to create and holds no long-lived connection until a method is called.

A Migrator is not safe for concurrent use by multiple goroutines in the same process — call its methods sequentially. Concurrent *processes* (e.g. two deploys racing) are handled by the Dialect's advisory lock, which every method holds for its full duration.

func NewMigrator

func NewMigrator(db *sql.DB, dialect Dialect, dir string, table string) *Migrator

NewMigrator constructs a Migrator that runs the migration files in dir against db, using dialect for every dialect-specific SQL fragment and table as the tracking table's name.

table is normally DefaultTable ("migrations"); RunCLI always passes DefaultTable, since the spec fixes the table name rather than making it configurable (a health check and db:migrate must agree on one name — see the migrate package's README).

func (*Migrator) Baseline

func (m *Migrator) Baseline(ctx context.Context, version string, force bool) error

Baseline marks every migration up to and including version as already applied — batch 0, run count 1, real checksum — without running any SQL. Use it once, against a database that already has this schema from before it was managed by this tool.

Returns ErrNonEmptyTable if the tracking table already has any rows and force is false. Returns ErrUnknownVersion if version does not match any migration Load discovers. With force == true, Baseline proceeds even over an existing baseline, overwriting the affected rows.

func (*Migrator) Down

func (m *Migrator) Down(ctx context.Context, all bool, dryRun bool) (*Result, error)

Down rolls back applied migrations, descending by version. With all == false (the default), it rolls back exactly one — the single most-recently-applied migration, regardless of which batch it belongs to. With all == true, it rolls back every migration in the current latest batch (every StatusUp row sharing the highest Record.Batch value). With dryRun == true, it reports which migration(s) it would roll back without executing any SQL.

Returns ErrChecksumMismatch under the same condition as Up. Returns ErrFilesMissing if a migration recorded as applied has no matching files in the migrations directory. As with Up, a failure partway through a multi-file --all rollback stops the run immediately.

func (*Migrator) Status

func (m *Migrator) Status(ctx context.Context) ([]StatusRow, error)

Status reports every migration file's state: joined with its tracking-table row if it has one, or reported as pending if it doesn't. Unlike Up and Down, Status never fails on ErrChecksumMismatch — it reports the mismatch per row (StatusRow.ChecksumMatches) instead of refusing to run, since inspecting state should always be safe.

It also reports every tracking-table row that has no matching file on disk anymore (StatusRow.FileMissing) — Load itself simply never sees these names, so without this check they would silently sit in the table forever, invisible to every other command.

func (*Migrator) Up

func (m *Migrator) Up(ctx context.Context, all bool, dryRun bool) (*Result, error)

Up applies pending migrations, ascending by version. With all == false (the default), it applies exactly one — the earliest pending migration. With all == true, it applies every pending migration, all sharing one freshly incremented batch number. With dryRun == true, it reports which migration(s) it would apply without executing any SQL or writing to the tracking table.

Returns ErrChecksumMismatch if a migration currently marked applied no longer matches its on-disk checksum (see prepare). If a file fails partway through a multi-file --all run, Up stops immediately: earlier files in the run stay applied and recorded, the failing file is not recorded as applied, and no later file is attempted.

type Record

type Record struct {
	// Migration is the tracked migration's identity — the same value as the matching
	// Migration.Name (e.g. "000024_create_widgets_table").
	Migration string
	// Batch is the batch number assigned the last time this migration was applied. It keeps
	// its value after a rollback (Status becomes StatusDown) so it remains a historical record;
	// it is only reassigned by a fresh "up".
	Batch int
	// Status is either StatusUp or StatusDown.
	Status string
	// RunCount is the number of times this migration has been successfully applied, including
	// re-applications after a rollback. It is never decremented.
	RunCount int
	// Checksum is the sha256 (hex-encoded) of the .up.sql file's contents as of the last time
	// this migration was applied. It is compared against the current file on disk before every
	// operation; see ErrChecksumMismatch.
	Checksum string
	// MigratedAt is the time of the last successful "up"; zero-valued (Valid == false) if this
	// migration has never been applied.
	MigratedAt sql.NullTime
	// RolledBackAt is the time of the last "down"; zero-valued (Valid == false) if this
	// migration has never been rolled back.
	RolledBackAt sql.NullTime
}

Record is one row of the tracking table, as last read by store.list.

type Result

type Result struct {
	Applied []string
}

Result reports what a single Up or Down call did, in the order it happened. An empty Applied means there was nothing pending (Up) or nothing applied to roll back (Down) — not an error.

type StatusRow

type StatusRow struct {
	Migration Migration
	// Record is nil if Migration has never been applied.
	Record *Record
	// ChecksumMatches is meaningless (true) if Record is nil; see Record.Checksum.
	ChecksumMatches bool
	// FileMissing is true when Record is non-nil but its .sql files are no longer present in
	// the migrations directory — e.g. deleted or renamed after the migration ran. Migration's
	// UpPath/DownPath are empty in that case; only Version (parsed from the tracking table's
	// migration name) and Name are populated. Such a row can never be rolled back by Down until
	// its files are restored (see ErrFilesMissing).
	FileMissing bool
}

StatusRow is one line of `db:migrate --status` output: a Migration joined with its Record, if any, plus whether its on-disk checksum still matches what was recorded.

Directories

Path Synopsis
postgres module

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL