Help us shape the future of Go! Take the 2026 Go Developer Survey

job

package
v1.79.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 20, 2026 License: Apache-2.0 Imports: 27 Imported by: 0

Documentation

Overview

Package job runs each kind of backup once against a store that is already open: a directory, a Postgres base backup or WAL, a MySQL dump or binlogs. A one-shot command and a long-running process call the same functions.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func AfterRun

func AfterRun(ctx context.Context, env *Env)

AfterRun is the upkeep between runs: it compacts the store when it can and trims the blob and tree caches, logging what fails.

func MySQLDump

func MySQLDump(ctx context.Context, env *Env, s MySQLDumpSettings) (*backup.WalkResult, error)

MySQLDump dumps the server with mysqldump and commits the dump to the set.

func MySQLReceive

func MySQLReceive(ctx context.Context, env *Env, s MySQLReceiveSettings, stop <-chan struct{},
	committed func(*backup.WalkResult), report func(mysql.ReceiverStatus),
) error

MySQLReceive receives the server's binlogs as a replica into the spool and commits them to the set, until stop is closed or, with UntilCaughtUp, until it is caught up; then it makes its last commit and returns nil. committed and report, when set, hear of each commit and of how the receiver stands.

func OpenObjectCache

func OpenObjectCache(dir string, logger *slog.Logger) (*badger.Store, error)

OpenObjectCache opens the object cache under dir, a store's backup cache directory. It returns nil, logging a warning, when another process has it open.

func OpenStatCache

func OpenStatCache(dir string, logger *slog.Logger) (*statcache.Cache, error)

OpenStatCache opens the stat cache under dir, a store's backup cache directory. It returns nil, logging a warning, when another process has it open.

func OpenTreeCache

func OpenTreeCache(dir string) (*blobcache.Cache, error)

OpenTreeCache opens the tree cache under dir, a store's backup cache directory.

func Path

func Path(ctx context.Context, env *Env, s PathSettings) (*backup.WalkResult, error)

Path commits the directory to the set as a diff against its latest commit, wrapped in the hooks. The post hook runs however the walk ends. A result that is Dirty committed with torn or unreadable files.

func PostgresBase

func PostgresBase(ctx context.Context, env *Env, s PostgresBaseSettings) (*backup.WalkResult, error)

PostgresBase takes a base backup and commits it to the set. When ctx ends first, the backup stops and nothing is committed.

func PostgresReceive

func PostgresReceive(ctx context.Context, env *Env, s PostgresReceiveSettings, stop <-chan struct{},
	committed func(*backup.WalkResult), report func(postgres.ReceiverStatus),
) error

PostgresReceive streams the cluster's WAL through a replication slot into the spool and commits it to the set, until stop is closed or, with UntilCaughtUp, until it is caught up; then it makes its last commit and returns nil. committed and report, when set, hear of each commit and of how the receiver stands.

Types

type Env

type Env struct {
	Store backup.ObjectStore
	Index backup.Index
	// Key seals what runs upload; nil backs up in the clear.
	Key     *storekey.Key
	AgentID string
	// Logger is where runs report; nil means slog.Default.
	Logger *slog.Logger
	// Stderr, when set, takes the stderr of the programs and hooks runs
	// start as it comes; nil logs it through Logger line by line.
	Stderr io.Writer

	// StatCache lets Path skip re-reading files whose stat data match.
	StatCache *statcache.Cache
	// ObjectCache keeps the commits and trees Path diffs against.
	ObjectCache *badger.Store
	// TreeCache keeps the commits and trees PostgresReceive and MySQLReceive
	// read every commit.
	TreeCache *blobcache.Cache
	// BlobCache receives every blob Path uploads.
	BlobCache *blobcache.Cache
}

Env is what runs share: the open store and index, the key, the agent they commit as, where they log, and the caches. The caller opens and closes all of it; a nil cache runs without it.

func (*Env) Close

func (e *Env) Close() error

Close closes the stat and object caches.

type Filter added in v1.71.0

type Filter func(rel string) bool

Filter reports whether the slash-separated path rel, relative to the backed-up directory, is backed up. A run asks once per path it lists, and a directory it keeps out is skipped whole.

func Globs added in v1.71.0

func Globs(logger *slog.Logger, include, exclude []string) Filter

Globs is a Filter of doublestar patterns over the path with a leading slash: a path an include matches is kept even when an exclude matches it too, and one no exclude matches is kept. A malformed pattern matches nothing and is reported to logger, nil meaning slog.Default.

type MySQLClient

type MySQLClient struct {
	// DefaultsExtraFile is the option file the programs read before their
	// own, for the connection and its password; empty reads ~/.my.cnf
	// alone.
	DefaultsExtraFile string
	Mysql             string
	Mysqldump         string
}

MySQLClient names the client programs that reach a MySQL server.

type MySQLDumpSettings

type MySQLDumpSettings struct {
	Set    string
	Client MySQLClient
	// BinlogSet is the set the server's binlogs go to, whose chain the dump
	// joins; empty when there is none.
	BinlogSet string
	// NoBinlogPosition records no binary log position even with log_bin
	// on, which spares the dump the global read lock.
	NoBinlogPosition bool
	// InitialRetention, when set, is the retention policy the set starts
	// with if this run creates it; a set that exists keeps its own.
	InitialRetention *retention.Policy
}

MySQLDumpSettings say how MySQLDump dumps a server.

type MySQLReceiveSettings

type MySQLReceiveSettings struct {
	Set string
	// DumpSet is the set the server's dumps go to.
	DumpSet string
	// Spool is where binlogs wait for their commit.
	Spool string
	// DefaultsExtraFile is the option file read before ~/.my.cnf for the
	// connection and its password.
	DefaultsExtraFile string
	// ServerID is the replica server id; mysql.ReplicaServerID of the set
	// when zero.
	ServerID    uint32
	CommitEvery time.Duration
	CommitBytes int64
	// Window is how far back point-in-time recovery reaches at least.
	Window time.Duration
	// InitialRetention, when set, is the retention policy the set starts
	// with if this run creates it; a set that exists keeps its own.
	InitialRetention *retention.Policy
	// UntilCaughtUp stops MySQLReceive once it has committed the binlog the
	// server had written when it started.
	UntilCaughtUp bool
}

MySQLReceiveSettings say how MySQLReceive receives a server's binlogs.

type PathSettings

type PathSettings struct {
	Set string
	// Dir is the directory to back up; on the local disk, relative to the
	// working directory when not absolute.
	Dir string
	// FS is the tree Dir is read from; nil means the local disk, and on
	// any other FS Dir is taken as that FS's path.
	FS backup.FS
	// Filters decide what is backed up: a path is when every filter keeps
	// it. Globs makes one from patterns.
	Filters []Filter
	// Unchanged, when set, reports directories known to be as in the set's
	// latest commit, everything under them included, such as from a change
	// journal; the walk takes them from that commit without reading them.
	// Filters must be the ones that commit was made with.
	Unchanged func(rel string) bool
	Workers   int
	// ForceHash is the percent of unchanged files re-hashed as a check.
	ForceHash int
	// Verify reads every file, changed or not.
	Verify             bool
	CheckpointInterval time.Duration
	ScanWorkers        int
	ReadRetries        int
	LostRetries        int
	// ProgressInterval is how often the walk reports; negative never.
	ProgressInterval time.Duration
	// Progress, when set, hears the walk's counters every ProgressInterval,
	// the default when it is not positive, and once more when it ends,
	// in place of the log line.
	Progress    func(backup.WalkResult)
	Meta        map[string]string
	PreHook     string
	PostHook    string
	HookTimeout time.Duration
	// HookOptional backs up anyway when the pre hook fails.
	HookOptional bool
	// VSS reads Dir from a shadow copy taken after the pre hook, on
	// Windows; when none can be taken, the live files are read.
	VSS bool
	// DownloadPlaceholders reads files a cloud sync client keeps only in
	// the cloud, which downloads them; otherwise they are recorded without
	// content.
	DownloadPlaceholders bool
	// InitialRetention, when set, is the retention policy the set starts
	// with if this run creates it; a set that exists keeps its own.
	InitialRetention *retention.Policy
}

PathSettings say how Path backs up a directory.

type PostgresBaseSettings

type PostgresBaseSettings struct {
	Set string
	// DBName is the connection string pg_basebackup connects with; empty
	// leaves it to the PG* variables.
	DBName string
	// PGBasebackup, when set, is the pg_basebackup that takes the backup;
	// its major version must match the server's. When empty, the backup
	// is taken over a replication connection of its own from Postgres 15
	// on, and by the pg_basebackup on the PATH before.
	PGBasebackup string
	// InitialRetention, when set, is the retention policy the set starts
	// with if this run creates it; a set that exists keeps its own.
	InitialRetention *retention.Policy
}

PostgresBaseSettings say how PostgresBase takes a base backup.

type PostgresReceiveSettings

type PostgresReceiveSettings struct {
	Set string
	// BaseSet is the set the cluster's base backups go to.
	BaseSet string
	// Spool is where WAL waits for its commit.
	Spool string
	// DBName is the connection string of the replication and monitoring
	// connections; empty leaves it to the PG* variables.
	DBName string
	// Slot is the replication slot; postgres.SlotName of the set when
	// empty.
	Slot string
	// AllowUncappedSlot streams when max_slot_wal_keep_size is -1.
	AllowUncappedSlot bool
	CommitEvery       time.Duration
	CommitBytes       int64
	// Window is how far back point-in-time recovery reaches at least.
	Window time.Duration
	// InitialRetention, when set, is the retention policy the set starts
	// with if this run creates it; a set that exists keeps its own.
	InitialRetention *retention.Policy
	// UntilCaughtUp stops PostgresReceive once it has committed the WAL the
	// server had flushed when it started.
	UntilCaughtUp bool
}

PostgresReceiveSettings say how PostgresReceive streams WAL.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL