migrate

package module
v0.0.0-...-276d082 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 11, 2025 License: MIT Imports: 13 Imported by: 0

README

GitHub Workflow Status (branch) GoDoc Coverage Status packagecloud.io Docker Pulls Supported Go Versions GitHub Release Go Report Card

HistoMigrate

Database migrations written in Go, now with out-of-order support! Use as CLI or import as library.

  • HistoMigrate reads migrations from sources and applies them in correct order to a database, intelligently handling non-linear application scenarios.
  • Drivers are "dumb"; HistoMigrate glues everything together and makes sure the logic is bulletproof (keeping drivers lightweight).
  • Database drivers don't assume things or try to correct user input. When in doubt, HistoMigrate fails.

Forked from golang-migrate/migrate

Out-of-Order Migration Support

HistoMigrate introduces robust support for out-of-order migrations, solving common pain points in modern deployment workflows, such as:

  • Cherry-picking Hotfixes: Apply critical hotfixes that include migrations with higher version numbers without issues, even if lower-version migrations from the main branch are yet to be applied.
  • Non-Linear Development: Seamlessly manage migration application when features or bug fixes are merged or deployed in a non-sequential order.

This is achieved by transitioning from a single "current version" tracking model to a history-based approach where the migration table records every individual migration that has been applied, rather than just the highest timestamp. This ensures your database's migration state is always an accurate reflection of exactly which scripts have run.

Databases

Database drivers run migrations. Add a new database?

Database URLs

Database connection strings are specified via URLs. The URL format is driver dependent but generally has the form: dbdriver://username:password@host:port/dbname?param1=true&param2=false

Any reserved URL characters need to be escaped. Note, the % character also needs to be escaped

Explicitly, the following characters need to be escaped: !, #, $, %, &, ', (, ), *, +, ,, /, :, ;, =, ?, @, [, ]

It's easiest to always run the URL parts of your DB connection URL (e.g. username, password, etc) through an URL encoder. See the example Python snippets below:

$ python3 -c 'import urllib.parse; print(urllib.parse.quote(input("String to encode: "), ""))'
String to encode: FAKEpassword!#$%&'()*+,/:;=?@[]
FAKEpassword%21%23%24%25%26%27%28%29%2A%2B%2C%2F%3A%3B%3D%3F%40%5B%5D
$ python2 -c 'import urllib; print urllib.quote(raw_input("String to encode: "), "")'
String to encode: FAKEpassword!#$%&'()*+,/:;=?@[]
FAKEpassword%21%23%24%25%26%27%28%29%2A%2B%2C%2F%3A%3B%3D%3F%40%5B%5D
$

Migration Sources

Source drivers read migrations from local or remote sources. Add a new source?

CLI usage

This section details how to use the migrate CLI tool for managing your database migrations.

  • Simple wrapper around this library.
  • Handles ctrl+c (SIGINT) gracefully.
  • No config search paths, no config files, no magic ENV var injections.

CLI Documentation (includes CLI install instructions)

Basic usage
$ migrate -source file://path/to/migrations -database postgres://localhost:5432/database up 2
Docker usage
$ docker run -v {{ migration dir }}:/migrations --network host migrate/migrate \
    -path=/migrations/ -database postgres://localhost:5432/database up 2
Database Migration Guide using migrate CLI

This guide explains how to use the migrate CLI tool for managing database schema migrations with HistoMigrate's out-of-order capabilities.

Prerequisites
  • Install migrate
  • Ensure you have a valid DATABASE_URL (e.g., postgres://user:pass@host:port/dbname?sslmode=disable)
  • Create a db/migrations directory or any folder where you store migration files
1. Create a New Migration File
migrate create -ext sql -dir db/migrations -tz Local $MIGRATION_TITLE
Example
migrate create -ext sql -dir db/migrations -tz Local create_users_table
Description

Creates two .sql files in the db/migrations directory with specified names and a current timestamp prefix:

  • xxxxxx_create_users_table.up.sql
  • xxxxxx_create_users_table.down.sql

Where xxxxxx is a UTC timestamp (or local time if -tz Local is used).

2. Apply All Available Migrations
migrate -path db/migrations -database postgres://user:pass@host:port/dbname?sslmode=disable up
Description

Applies all .up.sql migrations in order, skipping any already applied.

3. Apply a Limited Number of Migrations
migrate -path db/migrations -database postgres://user:pass@host:port/dbname?sslmode=disable up 1
Description

Applies only 1 new migration. Replace 1 with any desired number.

4. Apply a Specific Migration (do)
migrate -path db/migrations -database postgres://user:pass@host:port/dbname?sslmode=disable do $VERSION
Example
migrate -path db/migrations -database $DATABASE_URL do 20250525112233
Description

Applies the specified migration's .up.sql script to the database. This command explicitly applies a migration regardless of its version order relative to the current head, allowing for out-of-order application scenarios (e.g., hotfixes).

5. Revert a Specific Migration (undo)
migrate -path db/migrations -database postgres://user:pass@host:port/dbname?sslmode=disable undo $VERSION
Description

Rolls back a specific migration using its .down.sql script. Use the migration timestamp to undo. This command specifically targets and reverts a previously applied migration, even if it was applied out-of-order.

Example
migrate -path db/migrations -database $DATABASE_URL undo 20250525112233
6. Revert All Migrations (down)
migrate -path db/migrations -database postgres://user:pass@host:port/dbname?sslmode=disable down
Description

Rolls back all applied migrations in reverse order of application.

⚠️ Use with caution in production. This command will revert the entire schema history, including migrations applied with the do command.

7. Check Current Migration Version
migrate -path db/migrations -database $DATABASE_URL version
Description

Shows the last migration timestamp that was applied to the database.

Example PostgreSQL URL
postgres://username:password@localhost:5432/dbname?sslmode=disable
Common Commands Summary
Command Description
migrate create -ext sql -dir $PATH -tz Local name Create a new migration file
migrate -path $PATH -database $DATABASE_URL up Apply all new migrations
migrate -path $PATH -database $DATABASE_URL up $LIMIT Apply a limited number of new migrations
migrate -path $PATH -database $DATABASE_URL do $VERSION ✅ Apply a specific migration (out-of-order possible)
migrate -path $PATH -database $DATABASE_URL undo $VERSION ⬅️ Roll back specified migration
migrate -path $PATH -database $DATABASE_URL down 🔁 Roll back all migrations
migrate -path $PATH -database $DATABASE_URL down $LIMIT ⬅️ Roll back limited number of migrations
migrate -path $PATH -database $DATABASE_URL version Show last applied migration timestamp

Use in your Go project

  • API is stable and frozen for this release.
  • Uses Go modules to manage dependencies.
  • To help prevent database corruptions, it supports graceful stops via GracefulStop chan bool.
  • Bring your own logger.
  • Uses io.Reader streams internally for low memory overhead.
  • Thread-safe and no goroutine leaks.

Go Documentation

import (
    "[github.com/abramad-labs/histomigrate](https://github.com/abramad-labs/histomigrate)"
    _ "[github.com/abramad-labs/histomigrate/database/postgres](https://github.com/abramad-labs/histomigrate/database/postgres)"
    _ "[github.com/abramad-labs/histomigrate/source/github](https://github.com/abramad-labs/histomigrate/source/github)"
)

func main() {
    m, err := migrate.New(
        "github://mattes:personal-access-token@mattes/migrate_test",
        "postgres://localhost:5432/database?sslmode=enable")
    m.Steps(2)
}

Want to use an existing database client?

import (
    "database/sql"
    _ "[github.com/lib/pq](https://github.com/lib/pq)"
    "[github.com/abramad-labs/histomigrate](https://github.com/abramad-labs/histomigrate)"
    "[github.com/abramad-labs/histomigrate/database/postgres](https://github.com/abramad-labs/histomigrate/database/postgres)"
    _ "[github.com/abramad-labs/histomigrate/source/file](https://github.com/abramad-labs/histomigrate/source/file)"
)

func main() {
    db, err := sql.Open("postgres", "postgres://localhost:5432/database?sslmode=enable")
    driver, err := postgres.WithInstance(db, &postgres.Config{})
    m, err := migrate.NewWithDatabaseInstance(
        "file:///migrations",
        "postgres", driver)
    m.Up() // or m.Steps(2) if you want to explicitly set the number of migrations to run
}

Getting started

Go to getting started

Tutorials

(more tutorials to come)

Migration files

Each migration has an up and down migration. Why?

1481574547_create_users_table.up.sql
1481574547_create_users_table.down.sql

Best practices: How to write migrations.

Coming from another db migration tool?

Check out migradaptor. Note: migradaptor is not affiliated or supported by this project

Versions

Version Supported? Import Notes
master :white_check_mark: import "github.com/abramad-labs/histomigrate" New features and bug fixes arrive here first
v4 ❌ import "github.com/golang-migrate/migrate" (with package manager) DO NOT USE - This refers to the upstream golang-migrate v4.
v3 ❌ import "gopkg.in/golang-migrate/migrate.v3" DO NOT USE - This refers to the upstream golang-migrate v3.

Development and Contributing

Yes, please! Makefile is your friend, read the development guide.

Also have a look at the FAQ.

Documentation

Overview

Package migrate reads migrations from sources and runs them against databases. Sources are defined by the `source.Driver` and databases by the `database.Driver` interface. The driver interfaces are kept "dumb", all migration logic is kept in this package.

Index

Examples

Constants

This section is empty.

Variables

View Source 
var (
	ErrNoChange       = errors.New("no change")
	ErrNilVersion     = errors.New("no migration")
	ErrInvalidVersion = errors.New("version must be >= -1")
	ErrLocked         = errors.New("database locked")
	ErrLockTimeout    = errors.New("timeout: can't acquire database lock")
)
View Source 
var DefaultBufferSize = uint(100000)

DefaultBufferSize sets the in memory buffer size (in Bytes) for every pre-read migration (see DefaultPrefetchMigrations).

View Source 
var DefaultLockTimeout = 15 * time.Second

DefaultLockTimeout sets the max time a database driver has to acquire a lock.

View Source 
var DefaultPrefetchMigrations = uint(10)

DefaultPrefetchMigrations sets the number of migrations to pre-read from the source. This is helpful if the source is remote, but has little effect for a local source (i.e. file system). Please note that this setting has a major impact on the memory usage, since each pre-read migration is buffered in memory. See DefaultBufferSize.

Functions

func FilterCustomQuery 

func FilterCustomQuery(u *nurl.URL) *nurl.URL

FilterCustomQuery filters all query values starting with `x-`

Types

type ErrDirty 

type ErrDirty struct {
	Version int
}

func (ErrDirty) Error 

func (e ErrDirty) Error() string

type ErrShortLimit 

type ErrShortLimit struct {
	Short uint
}

ErrShortLimit is an error returned when not enough migrations can be returned by a source for a given limit.

func (ErrShortLimit) Error 

func (e ErrShortLimit) Error() string

Error implements the error interface.

type Logger 

type Logger interface {

	// Printf is like fmt.Printf
	Printf(format string, v ...interface{})

	// Verbose should return true when verbose logging output is wanted
	Verbose() bool
}

Logger is an interface so you can pass in your own logging implementation.

type Migrate 

type Migrate struct {

	// Log accepts a Logger interface
	Log Logger

	// GracefulStop accepts `true` and will stop executing migrations
	// as soon as possible at a safe break point, so that the database
	// is not corrupted.
	GracefulStop chan bool

	// PrefetchMigrations defaults to DefaultPrefetchMigrations,
	// but can be set per Migrate instance.
	PrefetchMigrations uint

	// LockTimeout defaults to DefaultLockTimeout,
	// but can be set per Migrate instance.
	LockTimeout time.Duration
	// contains filtered or unexported fields
}

func New 

func New(sourceURL, databaseURL string) (*Migrate, error)

New returns a new Migrate instance from a source URL and a database URL. The URL scheme is defined by each driver.

Example 
// Read migrations from /home/mattes/migrations and connect to a local postgres database.
m, err := New("file:///home/mattes/migrations", "postgres://mattes:secret@localhost:5432/database?sslmode=disable")
if err != nil {
	log.Fatal(err)
}

// Migrate all the way up ...
if err := m.Up(); err != nil && err != ErrNoChange {
	log.Fatal(err)
}

func NewWithDatabaseInstance 

func NewWithDatabaseInstance(sourceURL string, databaseName string, databaseInstance database.Driver) (*Migrate, error)

NewWithDatabaseInstance returns a new Migrate instance from a source URL and an existing database instance. The source URL scheme is defined by each driver. Use any string that can serve as an identifier during logging as databaseName. You are responsible for closing the underlying database client if necessary.

Example 
// Create and use an existing database instance.
db, err := sql.Open("postgres", "postgres://mattes:secret@localhost:5432/database?sslmode=disable")
if err != nil {
	log.Fatal(err)
}
defer func() {
	if err := db.Close(); err != nil {
		log.Fatal(err)
	}
}()

// Create driver instance from db.
// Check each driver if it supports the WithInstance function.
// `import "github.com/abramad-labs/histomigrate/database/postgres"`
instance, err := dStub.WithInstance(db, &dStub.Config{})
if err != nil {
	log.Fatal(err)
}

// Read migrations from /home/mattes/migrations and connect to a local postgres database.
m, err := NewWithDatabaseInstance("file:///home/mattes/migrations", "postgres", instance)
if err != nil {
	log.Fatal(err)
}

// Migrate all the way up ...
if err := m.Up(); err != nil {
	log.Fatal(err)
}

func NewWithInstance 

func NewWithInstance(sourceName string, sourceInstance source.Driver, databaseName string, databaseInstance database.Driver) (*Migrate, error)

NewWithInstance returns a new Migrate instance from an existing source and database instance. Use any string that can serve as an identifier during logging as sourceName and databaseName. You are responsible for closing down the underlying source and database client if necessary.

Example 
// See NewWithDatabaseInstance and NewWithSourceInstance for an example.

func NewWithSourceInstance 

func NewWithSourceInstance(sourceName string, sourceInstance source.Driver, databaseURL string) (*Migrate, error)

NewWithSourceInstance returns a new Migrate instance from an existing source instance and a database URL. The database URL scheme is defined by each driver. Use any string that can serve as an identifier during logging as sourceName. You are responsible for closing the underlying source client if necessary.

Example 
di := &DummyInstance{"think any client required for a source here"}

// Create driver instance from DummyInstance di.
// Check each driver if it support the WithInstance function.
// `import "github.com/abramad-labs/histomigrate/source/stub"`
instance, err := sStub.WithInstance(di, &sStub.Config{})
if err != nil {
	log.Fatal(err)
}

// Read migrations from Stub and connect to a local postgres database.
m, err := NewWithSourceInstance(srcDrvNameStub, instance, "postgres://mattes:secret@localhost:5432/database?sslmode=disable")
if err != nil {
	log.Fatal(err)
}

// Migrate all the way up ...
if err := m.Up(); err != nil {
	log.Fatal(err)
}

func (*Migrate) Close 

func (m *Migrate) Close() (source error, database error)

Close closes the source and the database.

func (*Migrate) DoMigration 

func (m *Migrate) DoMigration(version uint) error

DoMigration executes a single database migration. It acquires a lock, checks if the migration is already applied, queues it for processing (if not applied), runs it, and then releases the lock. It requires an ExtendedDriver.

func (*Migrate) Down 

func (m *Migrate) Down() error

Down looks at the currently active migration version and will migrate all the way down (applying all down migrations).

func (*Migrate) Drop 

func (m *Migrate) Drop() error

Drop deletes everything in the database.

func (*Migrate) Force 

func (m *Migrate) Force(version int) error

Force sets a migration version. It does not check any currently active version in database. It resets the dirty state to false.

func (*Migrate) Migrate 

func (m *Migrate) Migrate(version uint) error

Migrate looks at the currently active migration version, then migrates either up or down to the specified version.

func (*Migrate) Run 

func (m *Migrate) Run(migration ...*Migration) error

Run runs any migration provided by you against the database. It does not check any currently active version in database. Usually you don't need this function at all. Use Migrate, Steps, Up or Down instead.

func (*Migrate) Steps 

func (m *Migrate) Steps(n int) error

Steps looks at the currently active migration version. It will migrate up if n > 0, and down if n < 0.

func (*Migrate) UndoMigration 

func (m *Migrate) UndoMigration(version uint) error

UndoMigration rolls back a specific database migration. It acquires a lock, confirms the migration is currently applied (returning ErrNoChange if not), then queues and runs the "down" migration. It requires an ExtendedDriver.

func (*Migrate) Up 

func (m *Migrate) Up() error

Up looks at the currently active migration version and will migrate all the way up (applying all up migrations).

func (*Migrate) Version 

func (m *Migrate) Version() (version uint, dirty bool, err error)

Version returns the currently active migration version. If no migration has been applied, yet, it will return ErrNilVersion.

type Migration 

type Migration struct {
	// Identifier can be any string to help identifying
	// the migration in the source.
	Identifier string

	// Version is the version of this migration.
	Version uint

	// TargetVersion is the migration version after this migration
	// has been applied to the database.
	// Can be -1, implying that this is a NilVersion.
	TargetVersion int

	// Body holds an io.ReadCloser to the source.
	Body io.ReadCloser

	// BufferedBody holds an buffered io.Reader to the underlying Body.
	BufferedBody io.Reader

	// BufferSize defaults to DefaultBufferSize
	BufferSize uint

	// Scheduled is the time when the migration was scheduled/ queued.
	Scheduled time.Time

	// StartedBuffering is the time when buffering of the migration source started.
	StartedBuffering time.Time

	// FinishedBuffering is the time when buffering of the migration source finished.
	FinishedBuffering time.Time

	// FinishedReading is the time when the migration source is fully read.
	FinishedReading time.Time

	// BytesRead holds the number of Bytes read from the migration source.
	BytesRead int64

	// UpKindMigration indicates the direction of the migration.
	// true for "up" (applying/forward), false for "down" (rolling back/reverse).
	UpKindMigration bool
	// contains filtered or unexported fields
}

Migration holds information about a migration. It is initially created from data coming from the source and then used when run against the database.

func NewMigration 

func NewMigration(body io.ReadCloser, identifier string,
	version uint, targetVersion int) (*Migration, error)

NewMigration returns a new Migration and sets the body, identifier, version and targetVersion. Body can be nil, which turns this migration into a "NilMigration". If no identifier is provided, it will default to "<empty>". targetVersion can be -1, implying it is a NilVersion.

What is a NilMigration? Usually each migration version coming from source is expected to have an Up and Down migration. This is not a hard requirement though, leading to a situation where only the Up or Down migration is present. So let's say the user wants to migrate up to a version that doesn't have the actual Up migration, in that case we still want to apply the version, but with an empty body. We are calling that a NilMigration, a migration with an empty body.

What is a NilVersion? NilVersion is a const(-1). When running down migrations and we are at the last down migration, there is no next down migration, the targetVersion should be nil. Nil in this case is represented by -1 (because type int).

Example 
// Create a dummy migration body, this is coming from the source usually.
body := io.NopCloser(strings.NewReader("dumy migration that creates users table"))

// Create a new Migration that represents version 1486686016.
// Once this migration has been applied to the database, the new
// migration version will be 1486689359.
migr, err := NewMigration(body, "create_users_table", 1486686016, 1486689359)
if err != nil {
	log.Fatal(err)
}

fmt.Print(migr.LogString())

Output:
1486686016/u create_users_table
Example (NilMigration) 
// Create a new Migration that represents a NilMigration.
// Once this migration has been applied to the database, the new
// migration version will be 1486689359.
migr, err := NewMigration(nil, "", 1486686016, 1486689359)
if err != nil {
	log.Fatal(err)
}

fmt.Print(migr.LogString())

Output:
1486686016/u <empty>
Example (NilVersion) 
// Create a dummy migration body, this is coming from the source usually.
body := io.NopCloser(strings.NewReader("dumy migration that deletes users table"))

// Create a new Migration that represents version 1486686016.
// This is the last available down migration, so the migration version
// will be -1, meaning NilVersion once this migration ran.
migr, err := NewMigration(body, "drop_users_table", 1486686016, -1)
if err != nil {
	log.Fatal(err)
}

fmt.Print(migr.LogString())

Output:
1486686016/d drop_users_table

func (*Migration) Buffer 

func (m *Migration) Buffer() error

Buffer buffers Body up to BufferSize. Calling this function blocks. Call with goroutine.

func (*Migration) LogString 

func (m *Migration) LogString() string

LogString returns a string describing this migration to humans.

func (*Migration) String 

func (m *Migration) String() string

String implements string.Stringer and is used in tests.

type MultiError deprecated 

type MultiError struct {
	Errs []error
}

MultiError holds multiple errors.

Deprecated: Use github.com/hashicorp/go-multierror instead

func NewMultiError deprecated 

func NewMultiError(errs ...error) MultiError

NewMultiError returns an error type holding multiple errors.

Deprecated: Use github.com/hashicorp/go-multierror instead

func (MultiError) Error 

func (m MultiError) Error() string

Error implements error. Multiple errors are concatenated with 'and's.

Source Files

View all Source files

Directories

Path Synopsis
cmd
migrate command
Package database provides the Driver interface.
 Package database provides the Driver interface.
cassandra
clickhouse
cockroachdb
firebird
mongodb
multistmt
Package multistmt provides methods for parsing multi-statement database migrations
 Package multistmt provides methods for parsing multi-statement database migrations
mysql
neo4j
pgx
pgx/v5
postgres
ql
redshift
rqlite
snowflake
spanner
sqlcipher
sqlite
sqlite3
sqlserver
stub
testing
Package testing has the database tests.
 Package testing has the database tests.
yugabytedb
internal
cli
url
Package source provides the Source interface.
 Package source provides the Source interface.
aws_s3
bitbucket
file
github
github_ee
gitlab
go_bindata
go_bindata/examples/migrations
godoc_vfs
Package godoc_vfs contains a driver that reads migrations from a virtual file system.
 Package godoc_vfs contains a driver that reads migrations from a virtual file system.
google_cloud_storage
httpfs
iofs
Package iofs provides the Go 1.16+ io/fs#FS driver.
 Package iofs provides the Go 1.16+ io/fs#FS driver.
pkger
stub
testing
Package testing has the source tests.
 Package testing has the source tests.
Package testing is used in driver tests and should only be used by migrate tests.
 Package testing is used in driver tests and should only be used by migrate tests.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.