postgres

package
v1.66.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 13, 2026 License: Apache-2.0 Imports: 33 Imported by: 0

Documentation

Overview

Package postgres backs up Postgres clusters: base backups, and the WAL a receiver streams through a replication slot, each into a set of its own.

Index

Constants

View Source
const (
	// DefaultCommitEvery is how often the receiver commits what is new.
	DefaultCommitEvery = time.Minute
	// DefaultCommitBytes is how much new WAL makes it commit early.
	DefaultCommitBytes = 256 << 20
	// MinCommitEvery is the shortest interval it takes, and the least time
	// between two commits.
	MinCommitEvery = 10 * time.Second
)

The receiver's commit cadence.

View Source
const (
	// MetaSystemID is the cluster's system identifier, in decimal.
	MetaSystemID = "pg.system_id"
	// MetaTimeline is the timeline a base backup started on, in decimal.
	MetaTimeline = "pg.timeline"
	// MetaStartLSN and MetaStopLSN bound a base backup, as Postgres prints
	// LSNs (0/2000028).
	MetaStartLSN = "pg.start_lsn"
	MetaStopLSN  = "pg.stop_lsn"
	// MetaStartWALFile names the WAL file a base backup started in.
	MetaStartWALFile = "pg.start_wal_file"
	// MetaFirstWALFile and MetaLastWALFile bound the segments a WAL commit
	// holds.
	MetaFirstWALFile = "pg.first_wal_file"
	MetaLastWALFile  = "pg.last_wal_file"
	// MetaEndLSN is where the WAL a WAL commit holds ends, its open
	// segment included; MetaTimeline then names the timeline it ends on.
	MetaEndLSN = "pg.end_lsn"
	// MetaCapture says how a WAL commit's WAL was collected: CaptureReplication,
	// or archive for WAL that archive_command delivered, which restore reads
	// alike.
	MetaCapture = "pg.capture"
	// MetaBaseSet and MetaWindow are, on a WAL commit, the set its base
	// backups go to and how far back point-in-time recovery reaches, as a
	// Go duration.
	MetaBaseSet = "pg.base_set"
	MetaWindow  = "pg.window"
)

The commit metadata the Postgres sets carry.

View Source
const CaptureReplication = "replication"

CaptureReplication is the MetaCapture of WAL a receiver streamed.

Variables

View Source
var ErrConflict = errors.New("the spool holds a different file of that name")

ErrConflict is returned when a file is received again under a name the spool holds with other contents.

View Source
var ErrForeignCluster = errors.New("base backup of another cluster")

ErrForeignCluster is returned for a base backup of another cluster than the set's.

View Source
var ErrForeignSegment = errors.New("WAL segment from another cluster")

ErrForeignSegment is returned for a WAL segment from another cluster than the set's.

View Source
var ErrGap = errors.New("WAL is missing a segment")

ErrGap is returned for WAL that skips a segment.

View Source
var ErrNoBase = errors.New("no base backup to restore from")

ErrNoBase is returned when no base backup can be restored to the moment asked for.

View Source
var ErrNoBaseStream = errors.New("base backups of Postgres before 15 are taken with pg_basebackup")

ErrNoBaseStream is returned by StartBase for a server before Postgres 15, whose base backup stream it doesn't read; pg_basebackup takes those.

View Source
var ErrServerShutDown = errors.New("the server shut down")

ErrServerShutDown is returned by Receiver.Run when the server shut down and the WAL it streamed up to its shutdown is committed. Run again once the server is back, streaming goes on where it ended.

View Source
var ErrSlotActive = errors.New("the replication slot is in use by another connection")

ErrSlotActive is returned when another connection streams through the receiver's slot.

View Source
var ErrSynchronousStandby = errors.New("the server counts the receiver as a synchronous standby; remove it from synchronous_standby_names")

ErrSynchronousStandby is returned when the server waits for the receiver to commit before its own commits return.

View Source
var ErrUncappedSlot = errors.New("max_slot_wal_keep_size is -1, so the slot may keep WAL until the database's disk is full; set it, or allow an uncapped slot")

ErrUncappedSlot is returned when max_slot_wal_keep_size leaves the receiver's slot free to keep every WAL file until the disk is full.

Functions

func Carry

func Carry(cutoff string) func(name string) bool

Carry keeps every timeline history and every WAL file at or after the cutoff, and whatever is not WAL at all.

func Check

func Check(ctx context.Context, connString, slot string, allowUncapped bool) []string

Check connects to the cluster at connString as the receiver and the base backups do, and lists what it lacks for replication capture: a supported version, the role's REPLICATION attribute, pg_hba.conf entries for both connections and, unless slot is empty because nothing streams, a capped slot unless allowUncapped and a free one. An empty list means nothing is missing.

func Cutoff

func Cutoff(bases []*proto.Commit, window time.Duration, now time.Time) (string, bool)

Cutoff names the WAL file the WAL set must reach back to: the one the oldest base backup received within window started in, or the newest base when none was. ok is false when no base backup records one, and then every WAL file is kept.

func DropSlot

func DropSlot(ctx context.Context, connString, slot string) error

DropSlot drops the receiver's replication slot, so the server no longer keeps WAL for it.

func FetchWAL

func FetchWAL(ctx context.Context, objects backup.ObjectStore, key *storekey.Key, ref *proto.Ref, name, dst string) error

FetchWAL writes the file name of the WAL commit at ref to dst, as restore_command does with %f and %p. A segment the commit holds only as the newest timeline's partial segment is written padded with zeros to its full size. It returns backup.ErrNotFound for a file the commit does not hold, which Postgres asks for as a matter of course, and for a segment off the history of the commit's newest timeline, which recovery follows: an older timeline's past where that history leaves it.

func FormatLSN

func FormatLSN(lsn uint64) string

FormatLSN prints an LSN as Postgres does.

func ParseLSN

func ParseLSN(s string) (uint64, error)

ParseLSN reads an LSN as Postgres prints it.

func SlotName

func SlotName(set string) string

SlotName is the replication slot a WAL set's receiver uses unless told otherwise: goback_ and the set's name, lowercased, with every other character as _, cut to 63 bytes.

Types

type BaseBackup

type BaseBackup struct {
	// Walker stores the commit; its Set is the base set. Run sets its
	// Stream and Metadata.
	Walker *backup.Walker
}

BaseBackup commits one base backup to the base set.

func (*BaseBackup) Run

func (b *BaseBackup) Run(ctx context.Context, tar io.Reader, wait func() error) (*backup.WalkResult, error)

Run stores tar, the output of pg_basebackup -D - -Ft -X fetch, as the set's next commit: the data directory as a tree, together with what the backup records about itself. wait, when set, is called once tar is read to its end and reports how its writer ended; an error from it leaves the set without a commit.

type BaseInfo

type BaseInfo struct {
	SystemID     uint64
	Timeline     uint32
	StartLSN     uint64
	StopLSN      uint64
	StartWALFile string
}

BaseInfo is what a base backup records about itself.

func (BaseInfo) Metadata

func (i BaseInfo) Metadata() map[string]string

Metadata is the commit metadata a base backup's set carries.

type BaseStream

type BaseStream struct {
	// Tar is the backup as pg_basebackup -D - -Ft -X fetch writes it: the
	// data directory with the WAL it needs, then backup_manifest.
	Tar io.Reader
	// Wait reports how the backup ended, once Tar is read to its end.
	Wait func() error
	// contains filtered or unexported fields
}

BaseStream is a base backup a server streams over a replication connection.

func StartBase

func StartBase(ctx context.Context, connString string) (*BaseStream, error)

StartBase connects to the cluster at connString, as the receiver does, and starts a base backup with a fast checkpoint and the WAL it needs. Only the data directory's tablespace is taken; a cluster with others is refused.

func (*BaseStream) Close

func (b *BaseStream) Close()

Close ends the connection, and with it a backup not read to its end.

type OpenSegment

type OpenSegment struct {
	// Name is the segment's name.
	Name string
	// Path is where it is on disk.
	Path string
	// Size is how much of it holds whole records.
	Size int64
}

OpenSegment is a WAL segment a receiver is still writing.

type Receiver

type Receiver struct {
	// ConnString is the libpq connection string of the replication
	// connection and of the monitoring one, which goes to the database it
	// names, postgres by default; empty leaves both to the PG* variables.
	ConnString string
	// Slot is the physical replication slot; SlotName of the WAL set when
	// empty. The receiver creates it when missing.
	Slot string
	// AllowUncappedSlot streams even when max_slot_wal_keep_size doesn't
	// bound what the slot keeps.
	AllowUncappedSlot bool
	// Spool is where WAL waits for its commit; one receiver per spool.
	Spool Spool
	// Walker returns the walker of one commit: its Set is the WAL set and
	// its Root the spool's directory.
	Walker func() *backup.Walker
	// BaseSet is the set the cluster's base backups go to.
	BaseSet string
	// Window is how far back point-in-time recovery reaches.
	Window time.Duration
	// CommitEvery is how often new WAL is committed; DefaultCommitEvery
	// when zero, and at least MinCommitEvery.
	CommitEvery time.Duration
	// CommitBytes of new WAL commit early; DefaultCommitBytes when zero.
	CommitBytes int64
	// Logger is where the receiver reports; nil means slog.Default.
	Logger *slog.Logger
	// Committed, when set, hears of each commit once it is stored.
	Committed func(*backup.WalkResult)
	// Report, when set, hears how the receiver stands every ten seconds.
	Report func(ReceiverStatus)
	// contains filtered or unexported fields
}

Receiver streams a cluster's WAL through a physical replication slot into a spool, and commits it to the WAL set on a timer: whole segments as they fill, and the segment being received up to its last whole record as <segment>.partial. It reports to the server as flushed only the end of what a stored commit holds, so the slot keeps everything else, and losing the receiver or its spool loses nothing the server has dropped.

func (*Receiver) Run

func (r *Receiver) Run(ctx context.Context, stop <-chan struct{}) error

Run receives and commits until stop is closed, then commits what it received and returns nil. When the server shuts down, it commits what the server sent up to its shutdown at once, so the shutdown doesn't wait for CommitEvery, and returns ErrServerShutDown. It returns early with an error that needs a person (ErrGap, ErrForeignCluster, ErrUncappedSlot, ErrSynchronousStandby, ErrSlotActive) or, on any other failure, one a restart may clear. When ctx ends, it stops without the last commit.

type ReceiverStatus

type ReceiverStatus struct {
	Timeline uint32 `json:"timeline"`
	// Received is where the WAL written to the spool ends.
	Received string `json:"received"`
	// Committed is where the WAL the newest stored commit holds ends.
	Committed string `json:"committed,omitempty"`
	// LagBytes is how much WAL the server has written that no commit
	// holds; LagSeconds how long the oldest of it has waited.
	LagBytes   int64   `json:"lag_bytes"`
	LagSeconds float64 `json:"lag_seconds"`
	SpoolBytes int64   `json:"spool_bytes"`
	// WALStatus and SafeWALSize are the slot's, as pg_replication_slots
	// shows them; SafeWALSize is -1 when nothing bounds it.
	WALStatus   string `json:"wal_status,omitempty"`
	SafeWALSize int64  `json:"safe_wal_size"`
	// RetainedBytes is how much WAL the slot keeps on the server.
	RetainedBytes int64 `json:"retained_bytes"`
	// Healthy is false while WAL has waited for a commit longer than
	// three commit intervals.
	Healthy bool `json:"-"`
}

ReceiverStatus is how a receiver stands.

type Restore

type Restore struct {
	Objects backup.ObjectStore
	Key     *storekey.Key
	// Latest finds a set's newest complete commit; it returns
	// backup.ErrNotFound for a set without one.
	Latest  func(ctx context.Context, set string) (*proto.Ref, error)
	BaseSet string
	// WALSet, when empty, restores the base alone: Postgres recovers to the
	// end of the base backup.
	WALSet string
	// At, when set, is the moment recovery stops at; zero recovers to the
	// end of the WAL set.
	At time.Time
	// FS is the tree the data directory is written onto; nil is the local
	// disk.
	FS backup.WriteFS
	// RestoreCommand, when set, is the restore_command that fetches the WAL
	// commit's files, with %f and %p for Postgres to fill in. When nil, the
	// WAL recovery reads is copied into the data directory's pg_wal.
	RestoreCommand func(walCommit *proto.Ref) string
}

Restore writes a base backup into a data directory and, given a WAL set, sets the cluster up to recover from it when Postgres starts on it. The owner of the data directory is left to the caller.

func (*Restore) Run

func (r *Restore) Run(ctx context.Context, dir string) (*RestoreResult, error)

Run restores into dir, which must be empty or missing.

type RestoreResult

type RestoreResult struct {
	Base    *proto.Commit
	BaseRef *proto.Ref
	// WAL is the WAL commit recovery reads from; nil when the WAL set has
	// none, and then the cluster recovers only to the end of its base.
	WAL *proto.Ref
	// Copied names the WAL files copied into pg_wal, as they are named
	// there; empty with a RestoreCommand.
	Copied []string
	// ModeUnset is true when FS could not give the data directory mode
	// 0700, which Postgres needs before it starts on it.
	ModeUnset bool
}

RestoreResult says what a restore chose and did.

type Spool

type Spool struct {
	Dir string
}

Spool is the directory a receiver keeps WAL in until a commit holds it: whole files under their names, and the segment being received in .receiving. A file appears under its name only once it is whole.

func (Spool) Files

func (s Spool) Files() ([]string, error)

Files lists the whole files in the spool, in name order, which is WAL order within a timeline.

func (Spool) Remove

func (s Spool) Remove(names ...string) error

Remove deletes files a backup has committed.

type WALBackup

type WALBackup struct {
	// Walker stores the commit; its Root is the spool's directory and its
	// Set the WAL set. Run sets its Include, Carry, Growing and Metadata.
	Walker *backup.Walker
	Spool  Spool
	// BaseSet is the set the cluster's base backups go to.
	BaseSet string
	Window  time.Duration
	Now     func() time.Time
	// Files are the spool's whole files to commit.
	Files []string
	// Open, when set, is the segment still being received; the commit
	// holds it up to its length as <name>.partial, in place of the previous
	// commit's partial segment of its timeline.
	Open *OpenSegment
}

WALBackup commits what a receiver's spool holds to the WAL set. Every commit holds each WAL file back to the oldest base backup inside Window, so any one commit replays on its own.

func (*WALBackup) Run

func (b *WALBackup) Run(ctx context.Context) (*backup.WalkResult, error)

Run commits Files and Open and then deletes Files from the spool. It commits nothing, and returns nil, when both are empty. It refuses WAL from another cluster than the set's and WAL that skips a segment, and then leaves the spool as it was.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL