Documentation
¶
Overview ¶
Package migrate applies a project's migrations from inside its own binary.
It is a module of its own, and that is the whole design. The rig CLI is a development tool: it wants Docker, cobra, a compiler. Production wants none of those, and a deployment that has to install a development tool to move its schema forward will eventually run a different version of it than the one the code was written against. The binary that has the code embeds the migrations that go with it, so the two cannot disagree.
It is not part of rig/runtime for the opposite reason: goose would then be in the go.mod of every generated application, including the ones that migrate some other way.
The files are the same goose files the CLI runs, read through the same library, so `rig db up` in development and this in production apply the schema identically.
More than one set ¶
A project may have several. rig owns a dozen tables of its own, and the modules that read them can carry their own DDL rather than have it vendored into the project's migrations directory — see github.com/simonjanss/rig/runtime/dbschema. Each set is then numbered from one in its own namespace and records itself in its own table, which is what lets those modules be released without agreeing a migration number.
Source is one such set and UpAll, ApplyAll and RequireAll are the functions that take a list of them. Order is the caller's, and rig's sets come before the project's own: rig's DDL never references a project's table, while a project's routinely references rig's.
A project that vendored the foundation has exactly one set and wants Apply or Require, unchanged.
Where to run them ¶
There are three answers and they are not equally good.
In development, `rig db up` starts a throwaway Postgres and migrates it. Nothing below applies.
As a step of the deployment is what this package is shaped for: Apply behind a subcommand, run as a job before the new version rolls out. One process migrates, the replicas only serve, and a migration that fails fails on its own — leaving the previous version running rather than taking the new one down with it.
At startup — Apply as the server's pre-listen hook — is one line, and for a single instance it is fine. Four things make it worse as soon as there is more than one:
- The application's database user needs rights to change the schema. A separate job can run as a role that may ALTER TABLE while the server runs as one that may not, and giving the process that answers public requests the ability to drop a table is not a trade that gets undone later.
- Every replica migrates at once. The advisory lock makes that safe — one applies and the rest wait — but waiting is the problem: a three-minute migration means every other replica sits on the lock for three minutes before it listens, and an orchestrator with a startup budget shorter than that kills them mid-wait. Safe, and still a crash loop.
- A bad migration takes down the new version instead of a job. With a rollout that keeps the old pods until the new ones are ready this is a stuck deploy rather than an outage, which is survivable — but it surfaces as replicas crash-looping rather than as a job that failed and stopped.
- Every restart runs it. A process cycling at three in the morning takes the migration lock on the recovery path. Almost always a no-op. Not always.
Checking rather than applying — Require as the same hook — is the middle, and the one to reach for by default. The server refuses to start when the database is behind it, so deploying before the migration job finished is a process that stops with a message naming what is missing, rather than one that serves against a schema it does not expect and fails a query at a time. The application keeps a database user that cannot change anything.
What not to do is apply migrations beside a server that is already answering. Every option above finishes before the first request; running them concurrently means serving against a half-applied schema, and the failures that produces are per-query, transient, and unreproducible by the time anybody looks.
Index ¶
- Constants
- func Apply(fsys fs.FS, opt Options) func(ctx context.Context, pool *pgxpool.Pool) error
- func ApplyAll(srcs []Source, opt Options) func(ctx context.Context, pool *pgxpool.Pool) error
- func FromPool(pool *pgxpool.Pool) *sql.DB
- func Pending(ctx context.Context, db *sql.DB, fsys fs.FS, opt Options) ([]string, error)
- func PendingAll(ctx context.Context, db *sql.DB, srcs []Source, opt Options) ([]string, error)
- func Require(fsys fs.FS, opt Options) func(ctx context.Context, pool *pgxpool.Pool) error
- func RequireAll(srcs []Source, opt Options) func(ctx context.Context, pool *pgxpool.Pool) error
- func Up(ctx context.Context, db *sql.DB, fsys fs.FS, opt Options) ([]string, error)
- func UpAll(ctx context.Context, db *sql.DB, srcs []Source, opt Options) ([]string, error)
- type Options
- type Source
Constants ¶
const DefaultTable = "rig_migrations"
DefaultTable is where the applied versions are recorded.
It is the same name `rig db up` uses, which is the point: development and production have to agree about which migrations have run, and two bookkeeping tables would mean the second reader thinks nothing has.
Variables ¶
This section is empty.
Functions ¶
func Apply ¶
Apply is Up as a one-argument function, for wiring straight into a server.
Tasks: map[string]serve.Task{"migrate": migrate.Apply(migrations, migrate.Options{})},
It borrows the pool, applies what is pending, and hands the borrowed handle back. That is three lines every project would otherwise write the same way, including the deferred close that is easy to leave out.
One set is a list of one, so this is ApplyAll with the list written out. The set has no name, which is what a failure names it by: a migration that will not apply is reported under the word "migrations" rather than under a name nobody supplied.
func ApplyAll ¶
ApplyAll is UpAll as a one-argument function, for wiring straight into a server — Apply's shape for a project whose modules carry their own schema.
func FromPool ¶
FromPool borrows a pool for migrations.
goose speaks database/sql and the application speaks pgx, so something has to bridge them. Doing it from the pool rather than from the connection string means the migration uses the credentials and the settings the application is actually running with.
The returned handle must be closed; closing it does not close the pool.
func Pending ¶
Pending is what Up would apply, without applying it.
A deployment that wants to fail before it starts rather than migrate on the way up can ask this first: an unexpected pending migration means the binary and the database disagree about which came first.
func PendingAll ¶
PendingAll is what UpAll would apply, without applying it.
func Require ¶
Require is Apply's careful sibling: it refuses to start when the database is behind, instead of bringing it forward.
Migrate: migrate.Require(migrations, migrate.Options{}),
This is the shape for a deployment that migrates as its own step. The server still notices — a binary started before its migration job finished stops with a message naming what is missing, rather than serving against a schema it does not expect and failing one query at a time. And the application's database user never needs the rights to change a schema, which is worth more than the convenience of applying it here.
One set is a list of one, so this is RequireAll with the list written out, and the message names the set the way that one does — for a set with no name, by the word "migrations".
func RequireAll ¶
RequireAll is Require over several sets: it refuses to start when any of them is behind, and says which.
Any rather than the project's own, because being behind on rig's schema is the same failure. A binary pinning a module one version newer than the database has applied is exactly the skew this catches, and it is the skew that gets worse once the modules carry their own migrations.
func Up ¶
Up applies every pending migration and returns what it applied.
It is safe to call from several processes at once: the first takes an advisory lock, the others wait for it and then find nothing to do. That is what makes it usable from a deployment that starts its replicas together.
func UpAll ¶
UpAll applies several sets in the order given and returns everything it applied.
The order is the caller's and it matters. rig's sets go first and the project's own last, because rig's DDL never references a project's table while a project's routinely references rig's — a join table pointing at rig_notification, a file column pointing at rig_file. So "all of rig's, then all of yours" is a total order that holds under upgrades too, and there is no need for anything cleverer.
Options.Dir and Options.Table are ignored: those are per-set facts and each Source carries its own. Logger and Unlocked apply to every set.
Applying the sets one after another rather than together is not a weaker arrangement. goose's advisory lock is a single fixed identifier, so several replicas starting at once still serialise across all of it — the per-set tables are bookkeeping, never a concurrency boundary.
Types ¶
type Options ¶
type Options struct {
// Dir is the subdirectory of the filesystem holding the .sql files. Empty
// means "migrations", which is where rig puts them and what an embed
// directive in the project root produces.
Dir string
// Table records which versions have been applied. Empty means
// DefaultTable. Naming it per project keeps two projects sharing one
// database from reading each other's history as their own.
//
// A project that changed migrations.table in rig.yaml has to pass the same
// name here. They are the same bookkeeping.
Table string
// Logger receives one line per migration applied, and a line for a run that
// found nothing to do. Both at INFO.
//
// Nil is not silence. It is [log/slog.Default], which is the reading every
// other Logger in rig gives it, and it is the whole point of this field
// being a logger rather than the writer it used to be: the generated boot
// passes an Options with nothing in it, so which migrations a deployment
// applied went nowhere at all. A caller that wants silence says so, and
// then it is a decision somebody made rather than one nobody noticed:
//
// migrate.Options{Logger: slog.New(slog.DiscardHandler)}
//
// INFO rather than the DEBUG most of what rig says about its own lifecycle
// goes at. Which migrations this database has had applied to it is a fact
// somebody goes looking for months later, and a level that has to have been
// turned on in advance cannot answer that.
Logger *slog.Logger
// Unlocked skips the advisory lock.
//
// The lock is what makes it safe for several instances to start at once:
// one applies, the rest wait and then find nothing to do. Skipping it is
// for a database whose user cannot take advisory locks, and it means
// arranging by hand that only one runner exists.
Unlocked bool
}
Options configure a run. The zero value is the sensible one.
type Source ¶
type Source struct {
// Name is who the set belongs to, for a line that has to say whose migration
// applied or whose is missing.
Name string
// FS holds the files, and Dir names the directory inside it. Empty Dir means
// "migrations", as in [Options].
FS fs.FS
Dir string
// Table records the applied versions for this set alone. Empty means
// [DefaultTable], which is the project's own; a module's set names its own.
Table string
}
A Source is one migration set: a filesystem, the directory inside it, and the table recording how far that set has been applied.
It exists because a project has more than one. rig's own tables — the identities, the file rows, the notifications — belong to the modules whose code reads them, and a project can leave those modules to carry their own schema instead of vendoring it. Then there are several sets to apply, each numbered from one in its own namespace, and each has to record its progress somewhere of its own: two sets sharing a table would share a numbering sequence, and two separately released modules would eventually collide on a version.
The fields restate github.com/simonjanss/rig/runtime/dbschema.Set without its manifest, because applying a set needs only these. The conversion is the caller's — usually three lines of generated code — and it is deliberate that the dependency does not exist: a module that carries schema should not thereby carry goose, which is the same reason this package is not part of rig/runtime.
Source Files