migrate

package module
v0.0.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Mar 28, 2025 License: BSD-3-Clause Imports: 11 Imported by: 0

README

Migrate module

Migrate is a simple database migration management package. It is designed to support non-sql databases as well as sql databases. Support sqlite is available with the migrate/sqlite package. Adding support for other sql databases is trivial.

See the example program in examples/simple for a usage example. The intended usage is to define migration steps in an init function and use a migrator to use them on a database.

This module is in alpha stage and may change without notice. Contributions are welcome.

Installation

To use this package, execute the instruction go get -u github.com/chmike/migrate@latest.

Migration steps

A migration step has a name, a version and an up and down function. A version is the step sequence number and a 16 byte checksum serving as a signature.

For sql databases, the up and down functions may be a transaction wrapped sequence of sql commands of go functions. Non transaction wrapped sql commands and go functions are also supported but the dry run flag is ignored. The intended use is to set some database specific flags that can't be set from inside a transaction.

The following code is an example of migration step definition.

import github.com/chmike/migrate/sqlite


var migrationSteps *sqlite.Steps

func init() {
    s := sqlite.NewSteps("my book database")

    // migration step with sql commands
    s.Append(
        // name
        "create some table",

        // up operation
        sqlite.Tx(sqlite.Cmd(`CREATE TABLE "example" ("id" INTEGER PRIMARY KEY AUTOINCREMENT, "text" TEXT NOT NULL)`)),

        // down operation
        sqlite.Tx(sqlite.Cmd(`DROP TABLE "example"`)),
    )

    // migration step with a go function as up operation and an sql command for down
    s.Append(
        // name
        "insert some rows",

        // up operation
        sqlite.TxF(func(tx sqlite.SQLTx, info sqlite.StepInfo, dryRun bool, log sqlite.Logger) error {
            for i := range 10 {
                text := fmt.Sprintf("%d test: %s: %v -> %v", i, info.Name(), info.From(), info.To())
                _, err := tx.Tx().Exec(`INSERT INTO "example" ("text") VALUES (?);`, text)
                if err != nil {
                    log.Error("failed inserting row", F("text", text))
                    return err // an error result in transaction rollback
                }
                log.Info("inserting row", F("text", text))
            }
            return nil // nil error result in commit unless dryRun is true
        }),

        // down operation
        sqlite.Tx(sqlite.Cmd(`DELETE FROM "example";`)),
    )

    // . . .

    migrationSteps = s
}

If the migration steps are defined in a dedicated file, it is also possible to import the sqlite package with a dot so that the code is a little lighter.

import . github.com/chmike/migrate/sqlite


var migrationSteps *Steps

func init() {
    s := NewSteps("my book database")

    // migration step with sql commands
    s.Append(
        // name
        "create some table",

        // up operation
        Tx(Cmd(`CREATE TABLE "example" ("id" INTEGER PRIMARY KEY AUTOINCREMENT, "text" TEXT NOT NULL)`)),

        // down operation
        Tx(Cmd(`DROP TABLE "example"`)),
    )

    // migration step with a go function as up operation and an sql command for down
    s.Append(
        // name
        "insert some rows",

        // up operation
        TxF(func(tx sSQLTx, info StepInfo, dryRun bool, log Logger) error {
            for i := range 10 {
                text := fmt.Sprintf("%d test: %s: %v -> %v", i, info.Name(), info.From(), info.To())
                _, err := tx.Tx().Exec(`INSERT INTO "example" ("text") VALUES (?);`, text)
                if err != nil {
                    log.Error("failed inserting row", F("text", text))
                    return err // an error result in transaction rollback
                }
                log.Info("inserting row", F("text", text))
            }
            return nil // nil error result in commit unless dryRun is true
        }),

        // down operation
        Tx(Cmd(`DELETE FROM "example";`)),
    )

    // . . .
}

Migrator

The interaction with a database is performed by use of a migrator. For an sql database, the migrator binds the steps with a small group of database specific queries used by this migration module.

Once the migrator is instantiated, it is required to call the Version or Init methods. It is safe to call the Init method as it will return an error if the database is already initialized, otherwise it will initialize the database. The version method will simply try ot read the database version.

Once the Version or Init methods have been called, one may call any of the methods that will perform a migration step. OneUp, OneDown, AllUp, AllDown. They all have a version with a context argument.

There are also a OneUpDryRun and OneDownDryRun methods whose effect should be obvious. Regardless if they return an error or not, the transaction will be rolled back.

Logger

The migrate logger is a wrapper for the different kind of loggers. A logger wrapper for the std log, slog, zap and zerolog are provided.

Se the example above how to log messages. This module supports Error, Warn, Info and Debug logging messages. It uses its own log level filtering.

Documentation

Index

Constants

This section is empty.

Variables

View Source
var BadVersion = badVersion

Functions

func NewSQLDB

func NewSQLDB(db *sql.DB, q *Queries) *sqlDB

NewSQLDB returns an SQLDB

Types

type Database

type Database interface {
	// InitVersion initialize the version information. Returns ErrAlreadyInitialized
	// if the database is already initialized. The ID of the given version MUST be 0.
	// Leaves the database unchanged if dryRun is true.
	InitVersion(ctx context.Context, v Version, dryRun bool) error

	// Version returns the current database version. Returns ErrNotInitialized if
	// the database is not initialized.
	Version(context.Context) (Version, error)

	// The DefaultStepFunc is called when the step function is nil. It changes the version
	// from info.From() to info.To() unless an error occurs or dryRun is true.
	DefaultStepFunc(ctx context.Context, info StepInfo, dryRun bool, log Logger) error
}

Database is the interface to a database.

type Error

type Error string
const (
	// ErrBadParameters is returned when a function received invalid parameters.
	ErrBadParameters Error = "bad parameters"

	// ErrAlreadyInitialized is returned by Init if the database is already initialized.
	ErrAlreadyInitialized Error = "already initialized"

	// ErrBadVersionID is returned when Init found an out of range version ID in the database.
	ErrBadVersionID Error = "bad version identifier"

	// ErrBadVersionChecksum is returned when Init found a version with invalid checksum which is a
	// hint that the migration steps don't match the one used for the database. Don't change
	// any migration steps as it will result in changing checksums.
	ErrBadVersionChecksum Error = "bad version checksum"

	// ErrNotInitialized is returned when Init has not been called.
	ErrNotInitialized Error = "not initialized"

	// ErrBadVersion is returned when the database is not in the expected version.
	ErrBadVersion Error = "bad version"

	// ErrEndOfSteps is returned when OneUp or OneDown has no more more migration steps to perform.
	ErrEndOfSteps Error = "end of steps"

	// ErrNotSQLDB is returned a database is not an SQL database.
	ErrNotSQLDB Error = "not an SQL database"

	// ErrBeginTx is returned when starting a transaction fails.
	ErrBeginTx Error = "begin transaction"

	// ErrCommitTx is returned when a committing a transaction fails.
	ErrCommitTx Error = "commit transaction"

	// ErrRollbackTx is return when the transaction rollback failed.
	ErrRollbackTx Error = "rollback transaction"

	// ErrCancel is returned by a user function to force a dry run.
	ErrCancel Error = "cancel transaction"

	// ErrAbort is returned by a user function to aborts a transaction.
	ErrAbort Error = "abort transaction"
)

func (Error) Error

func (e Error) Error() string

type Field

type Field struct {
	Key   string
	Value any
}

Field is a key value pairs.

func F

func F(key string, value any) Field

F is a helper function to create a key value pair field to log.

type LogAdapter

type LogAdapter struct {
	// contains filtered or unexported fields
}

LogAdapter adapts the std log to the logger.

func (*LogAdapter) Debug

func (a *LogAdapter) Debug(msg string, fields ...Field)

Debug logs an debug level message.

func (*LogAdapter) Error

func (a *LogAdapter) Error(msg string, fields ...Field)

Error logs an error level message.

func (*LogAdapter) Info

func (a *LogAdapter) Info(msg string, fields ...Field)

Info logs an info level message.

func (*LogAdapter) Level

func (a *LogAdapter) Level() LogLevel

Level returns the current logging level.

func (*LogAdapter) SetLevel

func (a *LogAdapter) SetLevel(lvl LogLevel)

SetLevel set the logging level.

func (*LogAdapter) Warn

func (a *LogAdapter) Warn(msg string, fields ...Field)

Warn logs a warning level message.

type LogLevel

type LogLevel int

Different log levels supported

const (
	LevelDebug LogLevel = iota
	LevelInfo
	LevelWarn
	LevelError
	LevelNoLog
)

type Logger

type Logger interface {
	// Error logs an error level message.
	Error(msg string, fields ...Field)

	// Warn logs a warning level message.
	Warn(msg string, fields ...Field)

	// Info logs an info level message.
	Info(msg string, fields ...Field)

	// Debug logs an debug level message.
	Debug(msg string, fields ...Field)

	// SetLevel sets the log level.
	SetLevel(level LogLevel)

	// Level returns the log level.
	Level() LogLevel
}

Logger is a common logging interface.

func NewLogLogger

func NewLogLogger(lvl LogLevel) Logger

NewLogLogger creates a new std logger using the default logger.

func NewLogLoggerWith

func NewLogLoggerWith(logger *log.Logger, lvl LogLevel) Logger

NewLogLoggerWith creates a new std logger using the given logger.

func NewNilLogger

func NewNilLogger() Logger

func NewSlogLogger

func NewSlogLogger(lvl LogLevel) Logger

NewSlogLogger returns a Logger using the default slog logger.

func NewSlogLoggerWith

func NewSlogLoggerWith(logger *slog.Logger, lvl LogLevel) Logger

NewSlogLoggerWith returns a Logger using the given slog logger.

type Migrater

type Migrater interface {
	// Init initializes the database version to v0 after verifying that it is not initialized.
	Init() error

	// InitDryRun simulates the database initialization to version v0 after verifying that
	// it is not initialized.
	InitDryRun() error

	// Version returns the current version of the database.
	Version() (Version, error)

	// OneUp executes one up step.
	OneUp(ctx context.Context) error

	// OneDown executes one down step.
	OneDown(ctx context.Context) error

	// OneUpDryRun simulates one up step.
	OneUpDryRun(ctx context.Context) error

	// OneDownDryRun simulates one down step.
	OneDownDryRun(ctx context.Context) error

	// AllUp executes all up steps.
	AllUp(ctx context.Context) error

	// AllDown executes all down steps.
	AllDown(ctx context.Context) error
}

Migrater manages database migration steps.

type Migrator

type Migrator struct {
	// contains filtered or unexported fields
}

Migrator is a Migrater for the given database, stepper and logger.

func New

func New(db Database, steps Stepper, l Logger) (*Migrator, error)

New creates a new migrator. Returns ErrBadParameters if the parameters are invalid, or ErrBadVersion if the version in the database isn't found in the stepper.

func (*Migrator) AllDown

func (m *Migrator) AllDown() error

AllDown attempts to executes all migration steps down.

func (*Migrator) AllDownCtx

func (m *Migrator) AllDownCtx(ctx context.Context) error

AllDownCtx attempts to executes all migration steps down.

func (*Migrator) AllUp

func (m *Migrator) AllUp() error

AllUp attempts to executes all migration steps up.

func (*Migrator) AllUpCtx

func (m *Migrator) AllUpCtx(ctx context.Context) error

AllUpCtx attempts to executes all migration steps up.

func (*Migrator) Init

func (m *Migrator) Init() error

Init initializes the database version to v0 after verifying that it is not initialized.

func (*Migrator) InitCtx

func (m *Migrator) InitCtx(ctx context.Context) error

InitCtx initializes the database version to v0 after verifying that it is not initialized.

func (*Migrator) InitDryRun

func (m *Migrator) InitDryRun() error

InitDryRun simulates the database initialization to version v0 after verifying that it is not initialized.

func (*Migrator) InitDryRunCtx

func (m *Migrator) InitDryRunCtx(ctx context.Context) error

InitDryRunCtx simulates the database initialization to version v0 after verifying that it is not initialized.

func (*Migrator) OneDown

func (m *Migrator) OneDown() error

OneDown attempts to execute one migration step down.

func (*Migrator) OneDownCtx

func (m *Migrator) OneDownCtx(ctx context.Context) error

OneDownCtx attempts to execute one migration step down.

func (*Migrator) OneDownDryRun

func (m *Migrator) OneDownDryRun() error

OneDownDryRun attempts to execute one migration step down.

func (*Migrator) OneDownDryRunCtx

func (m *Migrator) OneDownDryRunCtx(ctx context.Context) error

OneDownDryRunCtx attempts to execute one migration step down.

func (*Migrator) OneUp

func (m *Migrator) OneUp() error

OneUp attempts to execute one migration step up.

func (*Migrator) OneUpCtx

func (m *Migrator) OneUpCtx(ctx context.Context) error

OneUpCtx attempts to execute one migration step up.

func (*Migrator) OneUpDryRun

func (m *Migrator) OneUpDryRun() error

OneUpDryRun attempts to execute one migration step up.

func (*Migrator) OneUpDryRunCtx

func (m *Migrator) OneUpDryRunCtx(ctx context.Context) error

OneUpDryRunCtx attempts to execute one migration step up.

func (*Migrator) Version

func (m *Migrator) Version() (v Version, err error)

Version returns the current version of the database.

func (*Migrator) VersionCtx

func (m *Migrator) VersionCtx(ctx context.Context) (Version, error)

VersionCtx returns the current version of the database after checking its validity against the migrations steps.

type NilAdapter

type NilAdapter struct {
	// contains filtered or unexported fields
}

NilAdapter is a nil logger that doesn't produce any log.

func (*NilAdapter) Debug

func (a *NilAdapter) Debug(msg string, fields ...Field)

Debug logs an debug level message.

func (*NilAdapter) Error

func (a *NilAdapter) Error(msg string, fields ...Field)

Error logs an error level message.

func (*NilAdapter) Info

func (a *NilAdapter) Info(msg string, fields ...Field)

Info logs an info level message.

func (*NilAdapter) Level

func (a *NilAdapter) Level() LogLevel

Level returns the current logging level.

func (*NilAdapter) SetLevel

func (a *NilAdapter) SetLevel(lvl LogLevel)

SetLevel set the logging level.

func (*NilAdapter) Warn

func (a *NilAdapter) Warn(msg string, fields ...Field)

Warn logs a warning level message.

type NoTxFunc

type NoTxFunc func(ctx context.Context, db SQLDB, info StepInfo, log Logger) error

Func is a user provided function executed outside a transaction. It doesn't support dry run and the changes to the database won't be cancelled when the function returns an error.

type Queries

type Queries struct {
	// CreateTableQuery is the query to create a table holding its version.
	// The version is an integer ID and a 32 character string. The table
	// contains at most one row.
	CreateTableQuery string

	// InitTableQuery is the query to insert the database version.
	// The first parameter is the version ID which is an integer and the second
	// parameter is the checksum which is a 32 character string.
	InitTableQuery string

	// VersionQuery is the row query to get the database version. The
	// first value is the integer ID and the second is the 32 character
	// checksum value.
	VersionQuery string

	// SetVersionQuery is the update query to set the database version.
	// The first parameter is the version ID which is an integer and the second
	// parameter is the checksum which is a 32 character string.
	SetVersionQuery string // DB specific set version query.
}

queries are the sqlite queries.

func (*Queries) Replace

func (q *Queries) Replace(defaultTableName, newTableName string)

Replace replaces all occurrences of defaultTableName with newTableName in the queries.

type SQLCommand

type SQLCommand struct {
	Cmd  string
	Args []any
}

SQLCommand is an SQL query instruction with arguments.

func Cmd

func Cmd(cmd string, args ...any) SQLCommand

Cmd is a function simplifying the creation of a Command.

func (SQLCommand) String

func (c SQLCommand) String() string

type SQLDB

type SQLDB interface {
	Database

	// DB return the sql database handle.
	DB() *sql.DB

	// StartTransaction starts a transaction.
	StartTransaction(ctx context.Context, opts *sql.TxOptions) (SQLTx, error)

	// VersionTx returns the database version and is called in a transaction.
	VersionTx(tx SQLTx) (Version, error)

	// SetVersion changes the database version from info.From to info.To.
	SetVersion(ctx context.Context, info StepInfo, dryRun bool, log Logger) error

	// SetVersionTx changes the database version from info.From to info.To.
	// dryRun is provided for information purpose only.
	SetVersionTx(tx SQLTx, info StepInfo, dryRun bool, log Logger) error

	// Queries returns the database specific queries.
	Queries() *Queries
}

SQLDB is an sql database.

type SQLTx

type SQLTx interface {
	// Tx returns the sql transaction.
	Tx() *sql.Tx

	// FinalizeTransaction finalizes a transaction and should be called as a deferred
	// after starting the transaction. It commits the transaction when err is nil and dryRun
	// is false, otherwise it rolls back the transaction.
	FinalizeTransaction(err *error, dryRun bool)
}

SQLTx is an sql database transaction handle.

type SlogAdapter

type SlogAdapter struct {
	// contains filtered or unexported fields
}

SlogAdapter adapts slog.Logger to Logger

func (*SlogAdapter) Debug

func (a *SlogAdapter) Debug(msg string, fields ...Field)

Debug logs an debug level message.

func (*SlogAdapter) Error

func (a *SlogAdapter) Error(msg string, fields ...Field)

Error logs an error level message.

func (*SlogAdapter) Info

func (a *SlogAdapter) Info(msg string, fields ...Field)

Info logs an info level message.

func (*SlogAdapter) Level

func (a *SlogAdapter) Level() LogLevel

Level returns the current logging level.

func (*SlogAdapter) SetLevel

func (a *SlogAdapter) SetLevel(lvl LogLevel)

SetLevel set the logging level.

func (*SlogAdapter) Warn

func (a *SlogAdapter) Warn(msg string, fields ...Field)

Warn logs a warning level message.

type StepFunc

type StepFunc func(ctx context.Context, db Database, info StepInfo, dryRun bool, log Logger) error

StepFunc is a migration step operation. It is required to check the version prior to any operation by calling db.Version(), do its operations and then call db.SetVersion() if no error occurred. These operations should be wrapped in a transaction to ensure database integrity that should be rolled back in case of error of if dryRun is true.

func NoTx

func NoTx(cmds ...SQLCommand) StepFunc

NoTx returns a migration step function that executes the SQL commands in sequence without a wrapping transaction. It terminates as soon as a command returns an error. It doesn't execute any cmds when dryRun is true.

func NoTxF

func NoTxF(fs ...NoTxFunc) StepFunc

NoTxF returns a migration step function that executes the user provided functions in sequence without a wrapping transaction. It terminates as soon as a function returns an error. It doesn't execute any function when dryRun is true. The pseudo error ErrCancel is treated as ErrAbort as operations can't be cancelled and database version remains info.From().

Use with care as any error in the function may leave the database is an undefined state.

func Tx

func Tx(cmds ...SQLCommand) StepFunc

Tx returns a migration step function that executes all the SQL commands in sequence wrapped in a transaction. The execution stops and rolls back as soon as an error is returned by one of the commands. It is also rolled back when dryRun is true.

func TxF

func TxF(fs ...TxFunc) StepFunc

TxF returns a migration step function that executes all the user provided functions in sequence wrapped in a transaction. The execution stops and rolls back as soon as an error is returned by one of the function and the step function returns the error.

A user function may return the ErrAbort pseudo error to force a termination of the function execution and the AllUp or AllDown execution which will return the ErrAbort error. To force a roll back of the transaction without terminating the execution of subsequent functions and migration steps, it must return the ErrCancel pseudo error. The migration step function will return nil as error.

type StepInfo

type StepInfo interface {
	fmt.Stringer

	// Name returns the step name.
	Name() string

	// From is the version of the database before the migrate step.
	From() Version

	// To is the version of the database after the migration step if there
	// is no error or DryRun is false.
	To() Version
}

StepInfo is a step information.

type Stepper

type Stepper interface {
	// Len returns the number of steps.
	Len() int

	// Version returns the version of step ID. ID 0 is the initial state
	// of the database after it is initialized for migration. It is also
	// the state to which the database is returned with the migration
	// AllDown. It is the state where the database should be empty.
	Version(ID int) (Version, error)

	// Name returns the migration step name. ID 0 is the initial state
	// of the database after it is initialized for migration. It is also
	// to state to which the database is returned with the migration
	// AllDown. It is the state where the database should be empty.
	Name(ID int) (string, error)

	// Check returns an error if the given version is invalid. It may be
	// one of ErrBadVersionID when the ID is out of range, or ErrBadChecksum
	// if the checksum doesn't match the one in the stepper.
	Check(Version) error

	// Up returns the StepInfo and function for one step up migration
	// from the given version to the next version.
	// It returns a ErrBadVersion if the version is invalid or
	// ErrEndOfSteps if there are no more steps.
	Up(Version) (StepInfo, StepFunc, error)

	// Down returns the StepInfo and function for one step down migration
	// from the given version to the next version.
	// It returns a ErrBadVersion if the version is invalid or
	// ErrEndOfSteps if there are no more steps.
	Down(Version) (StepInfo, StepFunc, error)
}

A Stepper manages migration steps.

type Steps

type Steps struct {
	// contains filtered or unexported fields
}

Steps is a read only sequence of migration steps.

func NewSteps

func NewSteps(name string) *Steps

NewSteps instantiates a new migration step sequence. The name should not be empty and ideally unique to the database as it is used to compute the root checksum identifying the database.

func (*Steps) Append

func (s *Steps) Append(name string, up StepFunc, down StepFunc) error

Append appends a new migration step to the list. Name must not be empty as it is used to compute a checksum. The functions up or down may be nil.

func (*Steps) Check

func (s *Steps) Check(v Version) error

Check returns an error if the given version is invalid. It may be one of ErrBadVersionID when the ID is out of range, or ErrBadVersionChecksum if the checksum doesn't match the one in the stepper.

func (*Steps) Down

func (s *Steps) Down(v Version) (nfo StepInfo, f StepFunc, err error)

Down returns the StepInfo and function for one step down migration from the given version to the next version. It returns a ErrBadVersion if the version is invalid or ErrEndOfSteps if there are no more steps.

func (*Steps) Len

func (s *Steps) Len() int

Len returns the number of Steps.

func (*Steps) Name

func (s *Steps) Name(ID int) (name string, err error)

Name returns the migration step name. ID 0 is the initial state of the database after it is initialized for migration. It is also to state to which the database is returned with the migration AllDown. It is the state where the database should be empty.

func (*Steps) Up

func (s *Steps) Up(v Version) (nfo StepInfo, f StepFunc, err error)

Up returns the StepInfo and function for one step up migration from the given version to the next version. It returns a ErrBadVersion if the version is invalid or ErrEndOfSteps if there are no more steps.

func (*Steps) Version

func (s *Steps) Version(ID int) (v Version, err error)

Version returns the version of step ID. ID 0 is the initial state of the database after it is initialized for migration. It is also the state to which the database is returned with the migration AllDown. It is the state where the database should be empty.

type TxFunc

type TxFunc func(tx SQLTx, info StepInfo, dryRun bool, log Logger) error

TxFunc is a user provided function that is called wrapped in a transaction.

type Version

type Version struct {
	ID       int
	Checksum [32]byte
}

Version is a migration version.

func MakeVersion

func MakeVersion(id int, checksum string) (Version, error)

MakeVersion make a version from an id and checksum value.

func (Version) ChecksumString

func (v Version) ChecksumString() string

func (Version) String

func (v Version) String() string

Directories

Path Synopsis
examples
simple command

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL