Documentation
¶
Overview ¶
Package pgtest provides a REAL ephemeral postgres for generate-time schema introspection and for hermetic tests that need a database.
Why real postgres ¶
forge is postgres-pinned. The schema a project's migrations declare — schema-qualified DDL (CREATE TABLE controlplane.foo), postgres-only types (TIMESTAMPTZ, JSONB, TEXT[], BIGSERIAL), '::type' casts, multi-ADD ALTERs, generated/identity columns — only round-trips faithfully on real postgres. The previous in-memory SQLite "shadow" approximated this and froze the ORM the moment a project used a construct SQLite couldn't parse (the controlplane.-schema bug). Real postgres needs no normalization: migrations apply verbatim.
The shared instance ¶
Booting postgres is expensive, so this package boots ONE server per process (sync.Once) and hands every caller its own freshly-created, uniquely-named database on that server. Databases are cheap; the server boot is the cost, paid once. This mirrors how the e2e corpus builds the forge binary once via sync.Once and runs fixtures in parallel against it.
By default the server is an embedded-postgres binary (github.com/fergusstrange/embedded-postgres) — a real postgres downloaded and cached under the user cache dir on first use, no Docker required. Set FORGE_TEST_POSTGRES_URL to a base postgres DSN (postgres://user:pass@host:port/postgres?sslmode=disable) to point at an already-running server instead (a dev docker-compose, CI service container, or a detected local postgres) and skip the embedded boot.
Leftovers ¶
Every per-caller database is dropped by its cleanup, and its name records the process that owns it. A process that dies before its cleanup runs leaves its databases behind; the next process to reach the server drops each one whose owner is provably gone (owner.go). Templates are reaped by age instead (template.go), since no single process owns one.
Index ¶
- Constants
- func AcquireShared() (baseURL string, release func(), err error)
- func CanReach(baseURL string) bool
- func EnsureDatabase(appDSN string) error
- func New() (*sql.DB, func(), error)
- func NewAtURL(baseURL string) (*sql.DB, func(), error)
- func NewClone(template string) (*sql.DB, func(), error)
- func NewCloneAtURL(baseURL, template string) (*sql.DB, func(), error)
- func NewFromTemplate(key string, build func(db *sql.DB, dsn string) error) (db *sql.DB, dsn string, cleanup func(), err error)
- func NewURL() (dsn string, cleanup func(), err error)
- func RecreateDatabase(appDSN string) error
- func ReserveLoopbackPort() (int, error)
- func ResolveVersion(declared string) (string, error)
- func ShadowScratchLike() string
- func Shutdown()
- func SupportedMajors() []string
- type Embedded
- type FetchError
Constants ¶
const DefaultTemplateTTL = 24 * time.Hour
DefaultTemplateTTL is how long a template database may go unused before the next NewFromTemplate on the same server drops it.
A template is keyed by what built it — usually a hash of the migration files — so every migration edit publishes a new one and the previous one is never asked for again. On the embedded server that is harmless: the server is torn down when its last user detaches, templates with it. On an external FORGE_TEST_POSTGRES_URL server, shared by every checkout and every agent on a machine and never torn down, each one used to persist forever. A day is long enough that a key in active use is never dropped (each process that uses a template restamps it) and short enough that a shared server does not accumulate a database per migration edit.
const DefaultVersion = string(embeddedpostgres.V18)
DefaultVersion is the postgres the embedded server runs when nothing declares one: embedded-postgres's own default.
const DriverName = "pgx"
DriverName is the database/sql driver every handle pgtest returns is opened with: jackc/pgx's stdlib adapter, the driver production opens its pool with (serverkit.OpenDB). A test database on a different driver than production tests a different program — arrays, time scanning, prepared statement caching and error types (*pgconn.PgError, not *pq.Error) all differ between pgx and lib/pq.
const EnvBaseURL = "FORGE_TEST_POSTGRES_URL"
EnvBaseURL names the env var that, when set, points pgtest at an already-running postgres instead of booting embedded-postgres. The value is a base DSN whose database is "postgres" (the maintenance DB pgtest connects to in order to CREATE DATABASE).
const EnvTemplateTTL = "FORGE_TEST_TEMPLATE_TTL"
EnvTemplateTTL names the env var that overrides DefaultTemplateTTL: a Go duration ("72h"). Zero or negative disables reaping.
const EnvVersion = "FORGE_TEST_POSTGRES_VERSION"
EnvVersion names the env var that selects the embedded server's postgres version for one run, over forge.yaml's database.postgres_version.
Variables ¶
This section is empty.
Functions ¶
func AcquireShared ¶
AcquireShared returns the base maintenance DSN of the postgres server this process should use for scratch databases, plus a release that must be called when the process is done with it.
Resolution: an explicit FORGE_TEST_POSTGRES_URL wins (external server, no teardown responsibility); otherwise the cross-process shared embedded instance is booted or reused. Multiple acquisitions in one process share a single underlying reference — the LAST release performs the cross-process detach (and teardown when this is the last live user across all processes).
It is additive to the pgtest API. New/NewURL use it internally, which is what makes a plain `go test ./...` fan-out share ONE instance across its per-package binaries — no orchestrator required. (`forge test` used to call it directly to pre-resolve a server and export the DSN to its subprocesses; with that command gone the in-process path is the only caller, and it gives the same sharing.)
func CanReach ¶
CanReach reports whether a postgres server is reachable at baseURL with working credentials — a fast open/ping/close with a short timeout. It creates and mutates NOTHING; a malformed URL or any connect/ping failure (server down, wrong password, missing role) returns false. Callers use it to pick among candidate servers before committing one to NewAtURL, so an unreachable or wrong-credential candidate is skipped rather than turned into a hard CREATE DATABASE error.
func EnsureDatabase ¶
EnsureDatabase makes the application database named in appDSN exist, idempotently — the RUNTIME counterpart to NewAtURL's throwaway scratch database. Where NewAtURL creates a uniquely-named database and DROPS it on cleanup (generate-time schema introspection), EnsureDatabase creates the ONE named database the app will actually run against and NEVER drops it: a fresh `forge env up` against a scaffolded dev DSN would otherwise die with `FATAL: database "<project>" does not exist (SQLSTATE 3D000)` because nothing had issued CREATE DATABASE.
It reuses the SAME maintenance-DB mechanism as NewAtURL: appDSN is reduced to its SERVER coordinates against the maintenance database "postgres" (the app's own database is never opened), a short maintenance connection is opened there (openBase), and CREATE DATABASE is issued only when the target is absent. A concurrent creator winning the race between the existence check and the CREATE (duplicate_database, SQLSTATE 42P04) is a no-op, not an error — the post-condition (the database exists) holds either way.
The maintenance connection FAILING (server down, wrong credentials) is a HARD error: the app cannot boot without it, so the caller should surface it loudly rather than let the app fail later with an opaque connect error. appDSN that will not parse, names no host, or names no database is likewise an error.
func New ¶
New creates a fresh, uniquely-named, empty database on the shared server and returns an open *sql.DB connected to it plus a cleanup function that closes the connection and drops the database. The first call boots the shared server (embedded download on the very first run of a new machine).
Callers own the returned cleanup; tests typically defer it or register it with t.Cleanup. The connection is opened with DriverName (pgx), the driver production uses. A process that dies before its cleanup runs leaves the database behind, and the next process to reach the server drops it: the name records its owner (see owner.go).
func NewAtURL ¶
NewAtURL creates a fresh, uniquely-named, empty database on the ALREADY-RUNNING postgres server addressed by baseURL and returns an open *sql.DB connected to it plus a cleanup that drops it. It is the "give me a scratch database on THIS server" primitive: unlike New it never boots embedded-postgres and never consults FORGE_TEST_POSTGRES_URL — the CALLER has already resolved which server to use (that policy lives above this package; pgtest stays a dumb ephemeral-postgres utility).
baseURL is a DSN whose database is the maintenance DB (conventionally "postgres"): NewAtURL connects there to CREATE the scratch database and never touches whatever application database also lives on the server. The scratch database is dropped-if-exists BEFORE creating it, so a leftover of the same name can never wedge a fresh run; cleanup drops it again. The leftovers a crashed run left under its own names are dropped by the sweep the first NewAtURL of a process makes on a server (owner.go).
Callers own the returned cleanup; the generate pipeline defers it.
func NewClone ¶ added in v0.1.46
NewClone is New with the fresh database created as a COPY of the existing database named template on the same shared server (CREATE DATABASE … TEMPLATE …): every table, index, constraint, function and row it holds, without re-running whatever built it.
Unlike NewFromTemplate, which builds and publishes a server-lifetime template under a key, NewClone copies a database the caller already built and owns, and whose lifetime the caller decides.
Postgres refuses the copy while any other session is connected to the template, so the caller closes its own connections to it first. The copy is independent from then on: nothing done to it reaches the template or a sibling copy, sequence positions included.
func NewCloneAtURL ¶ added in v0.1.46
NewCloneAtURL is NewAtURL with the scratch database created as a COPY of the database named template on the same server — NewClone's counterpart for a caller-resolved server, with the same contract: the caller holds no connection to template, and the copy is independent.
func NewFromTemplate ¶ added in v0.1.46
func NewFromTemplate(key string, build func(db *sql.DB, dsn string) error) (db *sql.DB, dsn string, cleanup func(), err error)
NewFromTemplate is New for a database that already has a schema: the schema is built ONCE per (server, key) by build and every caller gets its own clone (CREATE DATABASE ... TEMPLATE), which copies files instead of replaying the migrations. For a project with a few hundred migrations and a few hundred tests that is the difference between seconds and minutes of test time.
key names what build produces: change it whenever the schema build changes (a hash of the migration files is the usual choice) and a stale template is never served. The template lives on the shared server, so every process of a fan-out — one `go test` binary per package — builds it once between them: a postgres advisory lock serializes the builders, the finished template is published under its final name only when complete (a reader never sees a half-built one), and a builder that dies mid-way leaves a *_build database the next builder drops.
Templates are reaped by age. Each process that uses a template stamps it as used, and the first NewFromTemplate a process makes on a server drops every template there unused for longer than DefaultTemplateTTL (EnvTemplateTTL overrides it), plus any *_build database whose builder is gone. A key in use is therefore never dropped, and a superseded one — the migration hash before the last edit — is gone a day later instead of living as long as the server. A template reaped between two clones is rebuilt on demand.
build receives an open handle on the empty database to fill, and that database's DSN for a builder that connects itself (a migrator that takes a URL). Connections it leaves open — through either — are terminated before the template is published, and the DSN names a database that stops existing under that name when build returns, so build must not need either afterwards.
NewFromTemplate returns the clone twice, as an open handle and as its DSN (a postgres:// URL, the shape NewURL returns), because a test usually needs both: the handle for its own queries, the DSN for whatever connects by URL — a second pool, a migrator, a server it boots as a subprocess. cleanup closes the handle, terminates every other session on the clone, and drops it.
func NewURL ¶
NewURL creates a fresh, uniquely-named, empty database on the shared server like New, but returns its connection DSN (a postgres:// URL) instead of an open *sql.DB. Use this when the consumer is a separate process that connects itself via DATABASE_URL — e.g. an e2e test that boots a generated server. The returned cleanup drops the database; the caller must not hold connections past it.
func RecreateDatabase ¶
RecreateDatabase DROPs the application database named in appDSN and creates it again, empty — the destructive counterpart to EnsureDatabase, and the mechanism behind `forge db reset`.
The three functions here now span the lifecycle, and the difference between them is what they do to EXISTING data. NewAtURL makes a uniquely-named scratch database and drops it on cleanup (generate-time introspection). EnsureDatabase creates the ONE named database only when it is absent and never drops it (runtime boot). RecreateDatabase drops it WHETHER OR NOT it exists and creates it fresh: the post-condition is an empty database, which is exactly what a caller recovering from a wedged migration state needs — there is no dirty flag to clear and no offending row to repair when the database itself is new.
It reuses the same maintenance-DB mechanism as the other two: appDSN is reduced to its SERVER coordinates against the maintenance database "postgres" (the target is never opened, which is required — postgres will not drop a database you are connected to), and the name is quoted as an identifier so a project called "control-plane" is dropped verbatim rather than parsed as SQL.
Lingering backends on the target are terminated first. A connection held by a running dev server makes DROP DATABASE block indefinitely, and a reset that hangs with no explanation is worse than one that fails loudly.
THIS DESTROYS DATA AND HAS NO GUARD OF ITS OWN. Deciding that the target is a scratch dev database — that the environment is development, that the DSN is the one that environment declares, and that a human confirmed it — is the CALLER's job, and forge's CLI does all three before calling this. As with everything in pgtest, this is the dumb mechanic; the policy lives above it.
func ReserveLoopbackPort ¶
ReserveLoopbackPort binds an ephemeral loopback TCP port, reads the number the OS assigned, and releases it.
Exported because every harness that has to hand a concrete port to a process it is about to spawn needs exactly this: embedded-postgres here, and the service binary a generated e2e harness starts. Without it each harness re-implements the same bind-:0/read/close trick.
The port is FREE at the moment of return, not reserved — the caller must bind it promptly and tolerate the (small) race where another process claims it in between. There is no portable way to close that race; an OS-assigned ephemeral port is still far better than a hardcoded one.
func ResolveVersion ¶ added in v1.0.0
ResolveVersion turns a declaration — a major ("16") or an exact release ("16.9.0") — into the release the embedded server boots.
func ShadowScratchLike ¶ added in v1.0.0
func ShadowScratchLike() string
ShadowScratchLike is a LIKE pattern (backslash escapes) matching exactly the scratch databases this process has created with NewAtURL and NewCloneAtURL, for a test that asserts its code dropped every one it made:
SELECT count(*) FROM pg_database WHERE datname LIKE $1 -- ShadowScratchLike()
func Shutdown ¶
func Shutdown()
Shutdown releases this process's shared-server reference: it closes the maintenance connection and detaches from the cross-process pool. When this is the LAST live user of the shared embedded instance, the detach tears the server down cleanly (stop + remove data dir + release IPC). It is a cheap no-op when this process never opened a database, and is idempotent.
The forge CLI defers Shutdown so every `forge generate` process cleans up its share on exit; ordinary `go test` binaries that never call it are backstopped by reapStaleInstances. (`forge test` used to be the other deferring caller. It is gone — projects run their suite through `task test` — so the per-package `go test` binaries a suite spawns now reach the pool directly, which is the path reapStaleInstances covers.)
func SupportedMajors ¶ added in v1.0.0
func SupportedMajors() []string
SupportedMajors lists the postgres majors the embedded server can run, newest first.
Types ¶
type Embedded ¶ added in v1.0.0
type Embedded struct {
// contains filtered or unexported fields
}
Embedded is a running embedded postgres plus the temp log file that belongs to it.
func StartEmbedded ¶
func StartEmbedded(cfg embeddedpostgres.Config) (*Embedded, error)
StartEmbedded builds and starts an embedded postgres from cfg, fetching its binary (on a cold cache) through a retrying relay. A download failure comes back as a *FetchError naming the URL and the upstream status, never as the library's "no version found".
cfg must not set BinaryRepositoryURL: the relay takes that slot and forwards to Maven Central, the library's default.
The returned Embedded owns the temp log file the library created for this Start (see embeddedlog.go): Stop removes it after the library's final flush, and a caller whose server outlives the handle calls Detach. A failed Start removes its own file here. No other process's log file is ever touched.
func (*Embedded) Detach ¶ added in v1.0.0
func (e *Embedded) Detach()
Detach removes the log file this instance created without stopping the server, for a caller that will never Stop through this handle because the server outlives it. The running postmaster keeps its descriptor, so it is unaffected. Calling Stop after Detach is an error in the library whenever a Logger is configured (its final flush reopens the removed file by name).
func (*Embedded) StartedLogs ¶ added in v1.0.0
StartedLogs returns the log files this instance owns, for diagnostics.
type FetchError ¶
type FetchError struct {
// URL is the upstream artifact the library asked for.
URL string
// Status is the last HTTP status the repository returned, 0 when the
// request never got a response (DNS, connection refused, TLS, timeout).
Status int
// Attempts is how many times forge tried.
Attempts int
// Err is the last transport error, when Status is 0.
Err error
}
FetchError is a postgres binary download that failed. It replaces the library's "no version found matching X", which it returns for any non-200.
func (*FetchError) Error ¶
func (e *FetchError) Error() string
func (*FetchError) Unwrap ¶
func (e *FetchError) Unwrap() error
Source Files