Documentation
¶
Overview ¶
Package backup builds, verifies and restores manifest-based backup points across the three CrossCodex stores: PostgreSQL (WAL-G base backups plus continuous WAL archiving), tenant object storage (a content-addressed blob repository) and the JetStream audit streams (jsm.go snapshots).
A point is complete once points/<id>/manifest.json exists; the manifest is written last and is the commit marker. Manifests are untrusted on read and decoded strictly (see Decode).
Postgres is restored outside this package by deploy/db/crosscodex-db-restore (WAL-G needs an empty PGDATA with Postgres stopped). Restore here covers objects and streams. See docs/dev/backup.md.
Index ¶
- Constants
- Variables
- func Encode(w io.Writer, m *Manifest) error
- func ExecRunner(ctx context.Context, env []string, name string, args ...string) ([]byte, error)
- func NewPointID(now time.Time, r io.Reader) (string, error)
- func ValidatePointID(id string) error
- type CaptureStats
- type CommandRunner
- type Deps
- type ListResult
- type Locker
- type Manifest
- type MaxAge
- type NATSCapture
- type ObjectEntry
- type ObjectStoreOpener
- type ObjectsCapture
- type PointList
- type PointReport
- type PostgresCapture
- type Repository
- func (r *Repository) Begin(ctx context.Context, id string, started time.Time) error
- func (r *Repository) ClearInProgress(ctx context.Context, id string) error
- func (r *Repository) Close() error
- func (r *Repository) HasBlob(ctx context.Context, sum string) (bool, error)
- func (r *Repository) LoadManifest(ctx context.Context, id string) (*Manifest, error)
- func (r *Repository) OpenBlob(ctx context.Context, sum string) (io.ReadCloser, error)
- func (r *Repository) OpenStreamFile(ctx context.Context, id, stream, name string) (io.ReadCloser, error)
- func (r *Repository) Points(ctx context.Context) (PointList, error)
- func (r *Repository) PutBlob(ctx context.Context, sum string, data io.Reader) error
- func (r *Repository) PutStreamFile(ctx context.Context, id, stream, name string, data io.Reader) error
- func (r *Repository) WriteManifest(ctx context.Context, m *Manifest) error
- type SQLTenantLister
- type Service
- func (s *Service) List(ctx context.Context) (res ListResult, err error)
- func (s *Service) Restore(ctx context.Context, pointID string) (err error)
- func (s *Service) Run(ctx context.Context) (m *Manifest, err error)
- func (s *Service) Verify(ctx context.Context, opts VerifyOptions) (rep VerifyReport, err error)
- type StoreAge
- type StreamCapture
- type StreamFile
- type StreamState
- type StreamStore
- type TenantLister
- type VerifyOptions
- type VerifyReport
- type WALCheck
- type WALG
- type WALGBackup
- type Window
Constants ¶
const ( WALStatusOK = "OK" WALStatusWarning = "WARNING" // segments still uploading or delayed WALStatusFailure = "FAILURE" // segments lost: PITR across the gap is impossible )
Check statuses printed by `wal-g wal-verify --json`.
const MaxManifestBytes = 256 << 20
MaxManifestBytes bounds how much Decode reads. A manifest is untrusted input; the bound keeps a corrupt or hostile file from exhausting memory. About 150 bytes per object entry puts the cap near 1.7 million objects.
const Namespace = "backup"
Namespace is the tenant slot the repository occupies in its storage backend, so all backup data lives under <destination>/backup/. WAL-G writes its own data beside it under <destination>/postgres/.
const SchemaVersion = 1
SchemaVersion is the manifest format this package writes and accepts.
Variables ¶
var ( // ErrInvalidManifest wraps every manifest decoding or validation failure. ErrInvalidManifest = errors.New("invalid backup manifest") // ErrInvalidPointID wraps a malformed backup point ID. ErrInvalidPointID = errors.New("invalid backup point ID") )
Functions ¶
func ExecRunner ¶
ExecRunner is the production CommandRunner. It uses an argument vector, never a shell; WALG passes credentials in env, never in args. On context cancellation it sends SIGTERM (not Kill) and gives the process up to WaitDelay to exit cleanly, so backup-push can finish an in-flight upload step rather than leaving a half-written base backup at the destination.
func NewPointID ¶
NewPointID returns now as YYYYMMDDTHHMMSSZ (UTC) plus 8 random hex digits read from r, so IDs sort by time and concurrent runs don't collide.
func ValidatePointID ¶
ValidatePointID checks the point ID format. IDs become repository keys, so this is also the path-safety check.
Types ¶
type CaptureStats ¶
type CaptureStats struct {
Window Window `json:"window"`
Bytes int64 `json:"bytes"`
ThroughputBytesPerSec float64 `json:"throughput_bytes_per_sec"`
}
CaptureStats is common to every store's capture. Throughput is the capture rate and estimates the restore rate.
type CommandRunner ¶
type CommandRunner func(ctx context.Context, env []string, name string, args ...string) ([]byte, error)
CommandRunner runs name with args under exactly the environment env and returns stdout. A non-zero exit is an error carrying the command's stderr.
type Deps ¶
type Deps struct {
Repo *Repository
WALG *WALG
Tenants TenantLister
Objects ObjectStoreOpener
Streams StreamStore
StreamNames []string
Lock Locker
MaxAge MaxAge
TempDir string // spool and snapshot files; "" = os.TempDir()
Now func() time.Time // nil = time.Now
Rand io.Reader // point ID suffixes; nil = crypto/rand.Reader
Tracer trace.Tracer // nil = telemetry.Instrumentation("pkg/backup")
Meter metric.Meter // nil = telemetry.Instrumentation("pkg/backup")
}
Deps wires a Service. Each method uses only some of them:
- Run: Repo, WALG, Tenants, Objects, Streams, StreamNames, Lock
- List: Repo
- Verify: Repo, WALG, MaxAge
- Restore: Repo, WALG, Objects, Streams
type ListResult ¶
ListResult is the output of List.
type Locker ¶
type Locker interface {
TryAcquire(ctx context.Context) (release func(context.Context) error, err error)
}
Locker keeps a backup run and a non-dry-run retention scan apart. db.AdvisoryLocker implements it with db.BackupRetentionLockName.
type Manifest ¶
type Manifest struct {
SchemaVersion int `json:"schema_version"`
ID string `json:"id"`
StartedAt time.Time `json:"started_at"`
FinishedAt time.Time `json:"finished_at"`
Postgres PostgresCapture `json:"postgres"`
Objects ObjectsCapture `json:"objects"`
NATS NATSCapture `json:"nats"`
}
Manifest describes one backup point.
type NATSCapture ¶
type NATSCapture struct {
Streams []StreamCapture `json:"streams"`
CaptureStats
}
NATSCapture lists the audit stream snapshots.
type ObjectEntry ¶
type ObjectEntry struct {
Key string `json:"key"`
SHA256 string `json:"sha256"`
Size int64 `json:"size"`
}
ObjectEntry is one tenant object; its content is blob SHA256.
type ObjectStoreOpener ¶
ObjectStoreOpener opens one tenant's object store. Production binds storage.NewFromConfig to cfg.Storage.Objects, so capture and restore go through the same tenant-prefix and path-escape checks as the daemon.
type ObjectsCapture ¶
type ObjectsCapture struct {
Tenants map[string][]ObjectEntry `json:"tenants"`
CaptureStats
}
ObjectsCapture lists every tenant's objects.
type PointList ¶
type PointList struct {
Complete []string // manifest.json present
Incomplete []string // IN_PROGRESS only: the run failed or is still going
}
PointList holds point IDs in ascending (time) order.
type PointReport ¶
type PointReport struct {
ID string
Issues []string // integrity problems; empty means intact
EstimatedRestore time.Duration // captured bytes / capture throughput; 0 when unknown
}
PointReport is one point's verification result.
type PostgresCapture ¶
type PostgresCapture struct {
BackupName string `json:"backup_name"`
StartLSN uint64 `json:"start_lsn"`
FinishLSN uint64 `json:"finish_lsn"`
Timeline uint32 `json:"timeline"`
CaptureStats
}
PostgresCapture records the WAL-G base backup taken for this point.
type Repository ¶
type Repository struct {
// contains filtered or unexported fields
}
Repository is the backup destination. It inherits pkg/storage's atomic local writes and key/path-escape checks.
func NewRepository ¶
func NewRepository(store storage.Provider) *Repository
NewRepository wraps an already-open provider (tests pass a local one).
func OpenRepository ¶
func OpenRepository(dest config.BackupDestinationConfig) (*Repository, error)
OpenRepository opens the configured destination. An empty backend means backup is disabled, and this fails closed.
func (*Repository) ClearInProgress ¶
func (r *Repository) ClearInProgress(ctx context.Context, id string) error
ClearInProgress removes the IN_PROGRESS marker of a committed point.
func (*Repository) Close ¶
func (r *Repository) Close() error
Close releases the underlying provider.
func (*Repository) LoadManifest ¶
LoadManifest reads and strictly decodes point id's manifest.
func (*Repository) OpenBlob ¶
func (r *Repository) OpenBlob(ctx context.Context, sum string) (io.ReadCloser, error)
OpenBlob opens blob sum; storage.ErrNotFound when it's missing.
func (*Repository) OpenStreamFile ¶
func (r *Repository) OpenStreamFile(ctx context.Context, id, stream, name string) (io.ReadCloser, error)
OpenStreamFile opens one snapshot file; storage.ErrNotFound when it's missing.
func (*Repository) Points ¶
func (r *Repository) Points(ctx context.Context) (PointList, error)
Points lists every point. Files other than the two markers are ignored (e.g. a local backend's temp files); a malformed point directory is an error, because it means the repository was modified outside crosscodexd.
func (*Repository) PutBlob ¶
PutBlob stores data as blob sum. data must be seekable for S3 (pass a spooled *os.File).
func (*Repository) PutStreamFile ¶
func (r *Repository) PutStreamFile(ctx context.Context, id, stream, name string, data io.Reader) error
PutStreamFile stores one snapshot file of stream for point id.
func (*Repository) WriteManifest ¶
func (r *Repository) WriteManifest(ctx context.Context, m *Manifest) error
WriteManifest commits a point: once manifest.json exists the point is complete.
type SQLTenantLister ¶
type SQLTenantLister struct {
// contains filtered or unexported fields
}
SQLTenantLister reads tenant IDs as backup_user, which sees every row of tenants (tenant_id only) through its own RLS policy (backup_list_tenants).
func OpenSQLTenantLister ¶
func OpenSQLTenantLister(dsn string) (*SQLTenantLister, error)
OpenSQLTenantLister prepares a single-connection pool on dsn; it connects on first use.
type Service ¶
type Service struct {
// contains filtered or unexported fields
}
Service runs the backup commands.
func (*Service) List ¶
func (s *Service) List(ctx context.Context) (res ListResult, err error)
List returns every point. A complete point whose manifest fails to decode is an error; `verify` reports the details.
func (*Service) Restore ¶
Restore restores a point's objects and streams. Postgres must already be restored with crosscodex-db-restore (see deploy/README.md). The point is verified first and every target is checked empty before anything is written, so a refusal changes nothing.
func (*Service) Run ¶
Run captures Postgres, then objects, then JetStream into a new point and commits its manifest. Postgres goes first and is authoritative; the other stores may be slightly ahead of it, never behind (docs/dev/backup.md). Any failure leaves the point incomplete (no manifest). When the manifest is written but cleanup fails, Run returns the manifest and an error.
func (*Service) Verify ¶
func (s *Service) Verify(ctx context.Context, opts VerifyOptions) (rep VerifyReport, err error)
Verify checks blob and snapshot checksums, WAL-G base backup presence, WAL continuity (wal-g wal-verify) and staleness.
type StoreAge ¶
type StoreAge struct {
Store string
NewestPoint string // "" when no valid point exists
Age time.Duration
MaxAge time.Duration
Stale bool
}
StoreAge is one store's staleness, measured from its capture window in the newest valid point.
type StreamCapture ¶
type StreamCapture struct {
Stream string `json:"stream"`
Files []StreamFile `json:"files"`
State StreamState `json:"state"`
}
StreamCapture is one audit stream's snapshot.
type StreamFile ¶
type StreamFile struct {
Name string `json:"name"`
SHA256 string `json:"sha256"`
Size int64 `json:"size"`
}
StreamFile is one file of a stream snapshot.
type StreamState ¶
type StreamState struct {
FirstSeq uint64 `json:"first_seq"`
LastSeq uint64 `json:"last_seq"`
Msgs uint64 `json:"msgs"`
}
StreamState is a stream's sequence state at snapshot time.
type StreamStore ¶
type StreamStore interface {
// Exists reports whether stream exists on the server.
Exists(ctx context.Context, stream string) (bool, error)
// Snapshot writes a snapshot of stream into the empty directory dir and
// returns the state the snapshot holds.
Snapshot(ctx context.Context, stream, dir string) (StreamState, error)
// Restore recreates stream from the snapshot files in dir and returns the
// stream's state afterwards.
Restore(ctx context.Context, stream, dir string) (StreamState, error)
}
StreamStore is the JetStream surface backups need. NewJSMStreamStore implements it with jsm.go.
func NewJSMStreamStore ¶
func NewJSMStreamStore(nc *nats.Conn) (StreamStore, error)
NewJSMStreamStore wraps nc, a connection from natsbus.DialExternal.
type TenantLister ¶
TenantLister returns every tenant ID. SQLTenantLister reads them as backup_user through its dedicated RLS policy (backup_list_tenants).
type VerifyOptions ¶
type VerifyOptions struct {
PointID string
}
VerifyOptions selects the points Verify checks; PointID "" means all complete points. Staleness is always computed over the whole repository (the newest valid point), even when PointID is set, so `verify --point` can fail on staleness alone.
type VerifyReport ¶
type VerifyReport struct {
Points []PointReport
Incomplete []string
WAL []WALCheck
Staleness []StoreAge
}
VerifyReport is the output of Verify.
func (VerifyReport) OK ¶
func (r VerifyReport) OK() bool
OK is true when every checked point is intact, no WAL check reports FAILURE, and no store is stale.
type WALG ¶
type WALG struct {
// contains filtered or unexported fields
}
WALG runs WAL-G as backup_user against the configured destination.
func NewWALG ¶
func NewWALG(dsn string, dest config.BackupDestinationConfig, run CommandRunner) (*WALG, error)
NewWALG derives WAL-G's environment from backup.dsn and the destination.
func (*WALG) BackupPush ¶
BackupPush takes a full base backup over the replication protocol (remote mode: no PGDATA argument, so crosscodexd needs no access to the database's files).
type WALGBackup ¶
type WALGBackup struct {
Name string `json:"backup_name"`
WALFileName string `json:"wal_file_name"`
StartTime time.Time `json:"start_time"`
FinishTime time.Time `json:"finish_time"`
StartLSN uint64 `json:"start_lsn"`
FinishLSN uint64 `json:"finish_lsn"`
CompressedSize int64 `json:"compressed_size"`
}
WALGBackup is one entry of `wal-g backup-list --json --detail`.
func (WALGBackup) Timeline ¶
func (b WALGBackup) Timeline() (uint32, error)
Timeline is the PostgreSQL timeline: the first 8 hex digits of the WAL file name. WAL-G's listing has no timeline field.