Documentation
¶
Overview ¶
Package postgres backs up Postgres clusters: base backups, and the WAL a receiver streams through a replication slot, each into a set of its own.
Index ¶
- Constants
- Variables
- func Carry(cutoff string) func(name string) bool
- func Check(ctx context.Context, connString, slot string, allowUncapped bool) []string
- func Cutoff(bases []*proto.Commit, window time.Duration, now time.Time) (string, bool)
- func DropSlot(ctx context.Context, connString, slot string) error
- func FetchWAL(ctx context.Context, objects backup.ObjectStore, key *storekey.Key, ...) error
- func FormatLSN(lsn uint64) string
- func ParseLSN(s string) (uint64, error)
- func SlotName(set string) string
- type BaseBackup
- type BaseInfo
- type BaseStream
- type OpenSegment
- type Receiver
- type ReceiverStatus
- type Restore
- type RestoreResult
- type Spool
- type WALBackup
Constants ¶
const ( // DefaultCommitEvery is how often the receiver commits what is new. DefaultCommitEvery = time.Minute // DefaultCommitBytes is how much new WAL makes it commit early. DefaultCommitBytes = 256 << 20 // MinCommitEvery is the shortest interval it takes, and the least time // between two commits. MinCommitEvery = 10 * time.Second )
The receiver's commit cadence.
const ( // MetaSystemID is the cluster's system identifier, in decimal. MetaSystemID = "pg.system_id" // MetaTimeline is the timeline a base backup started on, in decimal. MetaTimeline = "pg.timeline" // MetaStartLSN and MetaStopLSN bound a base backup, as Postgres prints // LSNs (0/2000028). MetaStartLSN = "pg.start_lsn" MetaStopLSN = "pg.stop_lsn" // MetaStartWALFile names the WAL file a base backup started in. MetaStartWALFile = "pg.start_wal_file" // MetaFirstWALFile and MetaLastWALFile bound the segments a WAL commit // holds. MetaFirstWALFile = "pg.first_wal_file" MetaLastWALFile = "pg.last_wal_file" // MetaEndLSN is where the WAL a WAL commit holds ends, its open // segment included; MetaTimeline then names the timeline it ends on. MetaEndLSN = "pg.end_lsn" // MetaCapture says how a WAL commit's WAL was collected: CaptureReplication, // or archive for WAL that archive_command delivered, which restore reads // alike. MetaCapture = "pg.capture" // MetaBaseSet and MetaWindow are, on a WAL commit, the set its base // backups go to and how far back point-in-time recovery reaches, as a // Go duration. MetaBaseSet = "pg.base_set" MetaWindow = "pg.window" )
The commit metadata the Postgres sets carry.
const CaptureReplication = "replication"
CaptureReplication is the MetaCapture of WAL a receiver streamed.
Variables ¶
var ErrConflict = errors.New("the spool holds a different file of that name")
ErrConflict is returned when a file is received again under a name the spool holds with other contents.
var ErrForeignCluster = errors.New("base backup of another cluster")
ErrForeignCluster is returned for a base backup of another cluster than the set's.
var ErrForeignSegment = errors.New("WAL segment from another cluster")
ErrForeignSegment is returned for a WAL segment from another cluster than the set's.
var ErrGap = errors.New("WAL is missing a segment")
ErrGap is returned for WAL that skips a segment.
var ErrNoBase = errors.New("no base backup to restore from")
ErrNoBase is returned when no base backup can be restored to the moment asked for.
var ErrNoBaseStream = errors.New("base backups of Postgres before 15 are taken with pg_basebackup")
ErrNoBaseStream is returned by StartBase for a server before Postgres 15, whose base backup stream it doesn't read; pg_basebackup takes those.
var ErrServerShutDown = errors.New("the server shut down")
ErrServerShutDown is returned by Receiver.Run when the server shut down and the WAL it streamed up to its shutdown is committed. Run again once the server is back, streaming goes on where it ended.
var ErrSlotActive = errors.New("the replication slot is in use by another connection")
ErrSlotActive is returned when another connection streams through the receiver's slot.
var ErrSynchronousStandby = errors.New("the server counts the receiver as a synchronous standby; remove it from synchronous_standby_names")
ErrSynchronousStandby is returned when the server waits for the receiver to commit before its own commits return.
var ErrUncappedSlot = errors.New("max_slot_wal_keep_size is -1, so the slot may keep WAL until the database's disk is full; set it, or allow an uncapped slot")
ErrUncappedSlot is returned when max_slot_wal_keep_size leaves the receiver's slot free to keep every WAL file until the disk is full.
Functions ¶
func Carry ¶
Carry keeps every timeline history and every WAL file at or after the cutoff, and whatever is not WAL at all.
func Check ¶
Check connects to the cluster at connString as the receiver and the base backups do, and lists what it lacks for replication capture: a supported version, the role's REPLICATION attribute, pg_hba.conf entries for both connections and, unless slot is empty because nothing streams, a capped slot unless allowUncapped and a free one. An empty list means nothing is missing.
func Cutoff ¶
Cutoff names the WAL file the WAL set must reach back to: the one the oldest base backup received within window started in, or the newest base when none was. ok is false when no base backup records one, and then every WAL file is kept.
func DropSlot ¶
DropSlot drops the receiver's replication slot, so the server no longer keeps WAL for it.
func FetchWAL ¶
func FetchWAL(ctx context.Context, objects backup.ObjectStore, key *storekey.Key, ref *proto.Ref, name, dst string) error
FetchWAL writes the file name of the WAL commit at ref to dst, as restore_command does with %f and %p. A segment the commit holds only as the newest timeline's partial segment is written padded with zeros to its full size. It returns backup.ErrNotFound for a file the commit does not hold, which Postgres asks for as a matter of course, and for a segment off the history of the commit's newest timeline, which recovery follows: an older timeline's past where that history leaves it.
Types ¶
type BaseBackup ¶
type BaseBackup struct {
// Walker stores the commit; its Set is the base set. Run sets its
// Stream and Metadata.
Walker *backup.Walker
}
BaseBackup commits one base backup to the base set.
func (*BaseBackup) Run ¶
func (b *BaseBackup) Run(ctx context.Context, tar io.Reader, wait func() error) (*backup.WalkResult, error)
Run stores tar, the output of pg_basebackup -D - -Ft -X fetch, as the set's next commit: the data directory as a tree, together with what the backup records about itself. wait, when set, is called once tar is read to its end and reports how its writer ended; an error from it leaves the set without a commit.
type BaseInfo ¶
type BaseInfo struct {
SystemID uint64
Timeline uint32
StartLSN uint64
StopLSN uint64
StartWALFile string
}
BaseInfo is what a base backup records about itself.
type BaseStream ¶
type BaseStream struct {
// Tar is the backup as pg_basebackup -D - -Ft -X fetch writes it: the
// data directory with the WAL it needs, then backup_manifest.
Tar io.Reader
// Wait reports how the backup ended, once Tar is read to its end.
Wait func() error
// contains filtered or unexported fields
}
BaseStream is a base backup a server streams over a replication connection.
func StartBase ¶
func StartBase(ctx context.Context, connString string) (*BaseStream, error)
StartBase connects to the cluster at connString, as the receiver does, and starts a base backup with a fast checkpoint and the WAL it needs. Only the data directory's tablespace is taken; a cluster with others is refused.
func (*BaseStream) Close ¶
func (b *BaseStream) Close()
Close ends the connection, and with it a backup not read to its end.
type OpenSegment ¶
type OpenSegment struct {
// Name is the segment's name.
Name string
// Path is where it is on disk.
Path string
// Size is how much of it holds whole records.
Size int64
}
OpenSegment is a WAL segment a receiver is still writing.
type Receiver ¶
type Receiver struct {
// ConnString is the libpq connection string of the replication
// connection and of the monitoring one, which goes to the database it
// names, postgres by default; empty leaves both to the PG* variables.
ConnString string
// Slot is the physical replication slot; SlotName of the WAL set when
// empty. The receiver creates it when missing.
Slot string
// AllowUncappedSlot streams even when max_slot_wal_keep_size doesn't
// bound what the slot keeps.
AllowUncappedSlot bool
// Spool is where WAL waits for its commit; one receiver per spool.
Spool Spool
// Walker returns the walker of one commit: its Set is the WAL set and
// its Root the spool's directory.
Walker func() *backup.Walker
// BaseSet is the set the cluster's base backups go to.
BaseSet string
// Window is how far back point-in-time recovery reaches.
Window time.Duration
// CommitEvery is how often new WAL is committed; DefaultCommitEvery
// when zero, and at least MinCommitEvery.
CommitEvery time.Duration
// CommitBytes of new WAL commit early; DefaultCommitBytes when zero.
CommitBytes int64
// Logger is where the receiver reports; nil means slog.Default.
Logger *slog.Logger
// Committed, when set, hears of each commit once it is stored.
Committed func(*backup.WalkResult)
// Report, when set, hears how the receiver stands every ten seconds.
Report func(ReceiverStatus)
// contains filtered or unexported fields
}
Receiver streams a cluster's WAL through a physical replication slot into a spool, and commits it to the WAL set on a timer: whole segments as they fill, and the segment being received up to its last whole record as <segment>.partial. It reports to the server as flushed only the end of what a stored commit holds, so the slot keeps everything else, and losing the receiver or its spool loses nothing the server has dropped.
func (*Receiver) Run ¶
Run receives and commits until stop is closed, then commits what it received and returns nil. When the server shuts down, it commits what the server sent up to its shutdown at once, so the shutdown doesn't wait for CommitEvery, and returns ErrServerShutDown. It returns early with an error that needs a person (ErrGap, ErrForeignCluster, ErrUncappedSlot, ErrSynchronousStandby, ErrSlotActive) or, on any other failure, one a restart may clear. When ctx ends, it stops without the last commit.
type ReceiverStatus ¶
type ReceiverStatus struct {
Timeline uint32 `json:"timeline"`
// Received is where the WAL written to the spool ends.
Received string `json:"received"`
// Committed is where the WAL the newest stored commit holds ends.
Committed string `json:"committed,omitempty"`
// LagBytes is how much WAL the server has written that no commit
// holds; LagSeconds how long the oldest of it has waited.
LagBytes int64 `json:"lag_bytes"`
LagSeconds float64 `json:"lag_seconds"`
SpoolBytes int64 `json:"spool_bytes"`
// WALStatus and SafeWALSize are the slot's, as pg_replication_slots
// shows them; SafeWALSize is -1 when nothing bounds it.
WALStatus string `json:"wal_status,omitempty"`
SafeWALSize int64 `json:"safe_wal_size"`
// RetainedBytes is how much WAL the slot keeps on the server.
RetainedBytes int64 `json:"retained_bytes"`
// Healthy is false while WAL has waited for a commit longer than
// three commit intervals.
Healthy bool `json:"-"`
}
ReceiverStatus is how a receiver stands.
type Restore ¶
type Restore struct {
Objects backup.ObjectStore
Key *storekey.Key
// Latest finds a set's newest complete commit; it returns
// backup.ErrNotFound for a set without one.
Latest func(ctx context.Context, set string) (*proto.Ref, error)
BaseSet string
// WALSet, when empty, restores the base alone: Postgres recovers to the
// end of the base backup.
WALSet string
// At, when set, is the moment recovery stops at; zero recovers to the
// end of the WAL set.
At time.Time
// FS is the tree the data directory is written onto; nil is the local
// disk.
FS backup.WriteFS
// RestoreCommand, when set, is the restore_command that fetches the WAL
// commit's files, with %f and %p for Postgres to fill in. When nil, the
// WAL recovery reads is copied into the data directory's pg_wal.
RestoreCommand func(walCommit *proto.Ref) string
}
Restore writes a base backup into a data directory and, given a WAL set, sets the cluster up to recover from it when Postgres starts on it. The owner of the data directory is left to the caller.
type RestoreResult ¶
type RestoreResult struct {
Base *proto.Commit
BaseRef *proto.Ref
// WAL is the WAL commit recovery reads from; nil when the WAL set has
// none, and then the cluster recovers only to the end of its base.
WAL *proto.Ref
// Copied names the WAL files copied into pg_wal, as they are named
// there; empty with a RestoreCommand.
Copied []string
// ModeUnset is true when FS could not give the data directory mode
// 0700, which Postgres needs before it starts on it.
ModeUnset bool
}
RestoreResult says what a restore chose and did.
type Spool ¶
type Spool struct {
Dir string
}
Spool is the directory a receiver keeps WAL in until a commit holds it: whole files under their names, and the segment being received in .receiving. A file appears under its name only once it is whole.
type WALBackup ¶
type WALBackup struct {
// Walker stores the commit; its Root is the spool's directory and its
// Set the WAL set. Run sets its Include, Carry, Growing and Metadata.
Walker *backup.Walker
Spool Spool
// BaseSet is the set the cluster's base backups go to.
BaseSet string
Window time.Duration
Now func() time.Time
// Files are the spool's whole files to commit.
Files []string
// Open, when set, is the segment still being received; the commit
// holds it up to its length as <name>.partial, in place of the previous
// commit's partial segment of its timeline.
Open *OpenSegment
}
WALBackup commits what a receiver's spool holds to the WAL set. Every commit holds each WAL file back to the oldest base backup inside Window, so any one commit replays on its own.