backup

package
v1.76.9 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 14, 2026 License: MIT Imports: 23 Imported by: 0

Documentation

Overview

Package backup is the archive format of a sluis backup (format 2). It holds no exporter and no restore logic: those write and read records through Writer and Reader.

Objects

A backup is two objects in an S3-compatible bucket, written through a port.Blob:

backup/<instance>/<id>.tar.gz.age   the backup
backup/<instance>/<id>.json         the sidecar, written last

The id is the UTC start and a random suffix, 20261013T020000Z-ab12cd. The sidecar (Sidecar) is plain JSON: the object's key, size and SHA-256, the recipients it was encrypted to and, per module, the counts. It names no secret and holds no value. It is written only after the object, so it is the completion marker: an object without a sidecar is not a backup.

The object

The object is a POSIX tar (PAX), compressed with gzip and encrypted with age (https://age-encryption.org/v1) to the configured X25519 recipients, binary, not armored. It is read without sluis:

age -d -i key.txt <id>.tar.gz.age | tar xz

The tar's members, in this order:

manifest.json
<module>/state.jsonl
<module>/secrets.jsonl
<module>/blobs/<escaped object key>   (sorted by member name)

for each module in port.Modules order. A JSONL line is one export record (internal/backup/export); a blob is the object's bytes. Every member has mode 0600, the backup's start as its modification time and no owner names.

manifest.json (Manifest) is first and lists the SHA-256 of every other member and the counts of each module, so a reader verifies each member as it reads it and refuses a missing, extra, reordered or changed one.

Escaping a blob key

An object key becomes a relative path that cannot leave its directory and reads back to the same key (EscapeKey, UnescapeKey): the key is split at '/', and in each segment every byte that is not a letter, a digit, '-', '_' or '.' is written %XX (upper-case hex), as is a '.' that begins a segment (so no segment is '.', '..' or hidden); an empty segment is written '%'.

Encryption

The writer needs only the recipients' public keys: it holds no key that decrypts, and no KMS. The restore reads the age identity (AGE-SECRET-KEY-1...) from its own secret. age authenticates the whole stream, so a changed byte fails the decryption; the sidecar's SHA-256 is checked before that, and the manifest's hashes after it.

Versions

The sidecar and the manifest carry the format version; this release reads format 2 only (Version). Format 1 (chunks sealed under a KMS data key) is gone: no format 1 backup was ever kept.

Index

Constants

View Source
const (
	ObjectSuffix  = ".tar.gz.age"
	SidecarSuffix = ".json"
)

The suffixes of a backup's two objects.

View Source
const ManifestFile = "manifest.json"

ManifestFile is the name of the tar's first member.

View Source
const Version = 2

Version is the format version this release writes and reads.

Variables

View Source
var (
	// ErrFormat is an archive this release cannot read: no sidecar, not a
	// manifest, an unknown version, or a field out of range.
	ErrFormat = errors.New("backup: unreadable archive")
	// ErrIntegrity is an archive that was changed or truncated: the object's
	// size or SHA-256, its decryption, a member's hash or a count did not
	// match.
	ErrIntegrity = errors.New("backup: archive failed verification")
	// ErrKey is an archive the identity does not open: it is not one of the
	// recipients, or the identity could not be read. Nothing was read.
	ErrKey = errors.New("backup: the identity does not open the archive")
)

Errors a caller can match with errors.Is.

Functions

func BlobFile added in v1.76.1

func BlobFile(m port.Module, key string) string

BlobFile is the member name of one blob of a module.

func EscapeKey added in v1.76.1

func EscapeKey(key string) string

EscapeKey turns an object key into a relative path inside the tar: split at '/', each segment keeps letters, digits, '-', '_' and '.', and writes every other byte as %XX, as well as a '.' that begins the segment; an empty segment is "%". The result never holds "." or ".." as a segment, an absolute path or a byte a file system treats specially. UnescapeKey reverses it.

func ObjectName added in v1.76.1

func ObjectName(instance, id string) string

ObjectName is the key of a backup's encrypted object.

func ParseIdentities added in v1.76.1

func ParseIdentities(text string) ([]age.Identity, error)

ParseIdentities parses an age identity file: one or more AGE-SECRET-KEY-1... lines, with # comments and blank lines allowed (age-keygen's output).

func ParseRecipients added in v1.76.1

func ParseRecipients(in []string) ([]age.Recipient, error)

ParseRecipients parses age X25519 recipients (age1...). At least one is required.

func Publish added in v1.76.1

func Publish(ctx context.Context, store port.Blob, a *Archive) error

Publish uploads an archive: the encrypted object with one write, then the sidecar. A backup is complete only once Publish returned; an object without its sidecar is not a backup and a prune removes it.

func Root added in v1.76.1

func Root(instance string) string

Root is the prefix of every backup of an instance.

func SecretsFile added in v1.76.1

func SecretsFile(m port.Module) string

SecretsFile is the member name of a module's secrets.

func SidecarName added in v1.76.1

func SidecarName(instance, id string) string

SidecarName is the key of a backup's sidecar.

func StateFile added in v1.76.1

func StateFile(m port.Module) string

StateFile is the member name of a module's State records.

func UnescapeKey added in v1.76.1

func UnescapeKey(path string) (string, error)

UnescapeKey reverses EscapeKey. It accepts only the form EscapeKey writes, so one key has one path.

func ValidName

func ValidName(s string) bool

ValidName reports whether s is acceptable as an instance name or a backup id: one path segment of letters, digits, dot, dash and underscore.

Types

type Archive added in v1.76.1

type Archive struct {
	Manifest *Manifest
	// Sidecar describes the encrypted file; [Publish] writes it after it.
	Sidecar *Sidecar
	// Path is the encrypted file.
	Path string
}

Archive is a finished backup on local disk, not yet uploaded.

func (*Archive) Remove added in v1.76.1

func (a *Archive) Remove()

Remove deletes the local file.

type BlobsEntry added in v1.76.1

type BlobsEntry struct {
	Count int64 `json:"count"`
	Bytes int64 `json:"bytes"`
}

BlobsEntry counts a module's blob members.

type Identities added in v1.76.1

type Identities func(ctx context.Context) ([]age.Identity, error)

Identities returns the age identities a reader decrypts with; it is called once the object has been checked, so a backup that is not there never reads the secret.

type Layout

type Layout string

Layout names the storage layout of the installation that was backed up.

const (
	LayoutV4 Layout = "v4"
	LayoutV5 Layout = "v5"
)

The layouts a backup can describe.

func (Layout) Valid

func (l Layout) Valid() bool

Valid reports whether l is a known layout.

type Lines added in v1.76.1

type Lines struct {
	Records int64  `json:"records"`
	SHA256  string `json:"sha256"`
}

Lines is one JSONL member: its record count and SHA-256.

type Manifest

type Manifest struct {
	// Format is the archive format version (see [Version]).
	Format int `json:"format"`
	// Instance and ID name the backup; they are also in its object key.
	Instance string `json:"instance"`
	ID       string `json:"id"`
	// Layout is the storage layout of the source installation.
	Layout Layout `json:"layout"`
	// Created is the start of the backup, UTC, RFC 3339.
	Created string `json:"created"`
	// Creator names what wrote it, for example "sluis-backup 1.76.1".
	Creator string `json:"creator"`
	// Modules lists the modules in the backup in [port.Modules] order, each
	// once.
	Modules []ModuleEntry `json:"modules"`
	// Files is the SHA-256 (hex) of every other member, by member name.
	Files map[string]string `json:"files"`
}

Manifest is manifest.json, the first member of the tar.

func (*Manifest) Module added in v1.76.1

func (m *Manifest) Module(mod port.Module) *ModuleEntry

Module returns the entry of a module, or nil when the backup does not hold it.

func (*Manifest) Records

func (m *Manifest) Records(s Section) int64

Records is the number of records (lines, or blobs) of a section across all modules.

type ModuleEntry

type ModuleEntry struct {
	Module  port.Module `json:"module"`
	State   Lines       `json:"state"`
	Secrets Lines       `json:"secrets"`
	Blobs   BlobsEntry  `json:"blobs"`
}

ModuleEntry is one module's members.

type ObjectInfo added in v1.76.1

type ObjectInfo struct {
	Key    string `json:"key"`
	Size   int64  `json:"size"`
	SHA256 string `json:"sha256"`
}

ObjectInfo is the encrypted object as stored.

type Params

type Params struct {
	Instance string
	ID       string
	Layout   Layout
	// Creator names what writes it, for example "sluis-backup 1.76.1".
	Creator string
	// Recipients are the age X25519 public keys (age1...) the backup is
	// encrypted to; at least one.
	Recipients []string
	// Now is the clock; time.Now when nil.
	Now func() time.Time
	// TempDir holds the spool and the encrypted file while they are written;
	// os.TempDir() (/tmp on Lambda) when empty.
	TempDir string
}

Params describe the backup a Writer makes.

type Reader

type Reader struct {
	// contains filtered or unexported fields
}

Reader is one verified backup, decrypted into a local spool.

func Decrypt added in v1.76.1

func Decrypt(ctx context.Context, sc *Sidecar, r io.Reader, idents []age.Identity, tempDir string) (*Reader, error)

Decrypt reads the encrypted object of the sidecar from r: see Open.

func Open

func Open(ctx context.Context, store port.Blob, ids Identities, instance, id, tempDir string) (*Reader, error)

Open fetches a backup, checks the object against its sidecar, decrypts it with the identities, and verifies every member of the tar against the manifest (its hash, its place in the order, the counts) before it returns. A key that does not open it fails with ErrKey; anything changed, with ErrIntegrity. tempDir holds the spool ("" is os.TempDir()).

func (*Reader) Blobs added in v1.76.1

func (rd *Reader) Blobs(ctx context.Context, m port.Module, fn func(key string, body []byte) error) error

Blobs calls fn with each blob of a module, sorted by member name; the body is valid only during the call.

func (*Reader) Close

func (rd *Reader) Close()

Close removes the spool.

func (*Reader) Manifest

func (rd *Reader) Manifest() *Manifest

Manifest is the verified manifest.

func (*Reader) Records

func (rd *Reader) Records(ctx context.Context, s Section, m port.Module, fn func(record []byte) error) error

Records calls fn with each record of the state or the secrets file of a module, in order. The slice passed to fn is valid only during the call. A module the backup does not hold has no records.

func (*Reader) Sidecar added in v1.76.1

func (rd *Reader) Sidecar() *Sidecar

Sidecar is the backup's sidecar, checked against the manifest.

type Section

type Section string

Section is one of the kinds of content a module contributes.

const (
	State   Section = "state"
	Secrets Section = "secrets"
	Blobs   Section = "blobs"
)

The sections, in archive order.

func Sections

func Sections() []Section

Sections lists every section in archive order.

func (Section) Valid

func (s Section) Valid() bool

Valid reports whether s is one of Sections.

type Sidecar added in v1.76.1

type Sidecar struct {
	Format     int             `json:"format"`
	Instance   string          `json:"instance"`
	ID         string          `json:"id"`
	Created    string          `json:"created"`
	Creator    string          `json:"creator"`
	Layout     Layout          `json:"layout"`
	Object     ObjectInfo      `json:"object"`
	Recipients []string        `json:"recipients"`
	Modules    []SidecarModule `json:"modules"`
}

Sidecar is <id>.json, written after the object: what a listing shows and what a reader checks the object against. It names no secret.

func Fetch added in v1.76.1

func Fetch(ctx context.Context, store port.Blob, instance, id string) (*Sidecar, []byte, error)

Fetch reads a backup's sidecar and its encrypted object and checks the object's size and SHA-256 against the sidecar. It needs no identity.

func ReadSidecar added in v1.76.1

func ReadSidecar(ctx context.Context, store port.Blob, instance, id string) (*Sidecar, error)

ReadSidecar reads a backup's sidecar. Nothing authenticates it: it is for listing, and for checking the object against before it is decrypted.

func (*Sidecar) Records added in v1.76.1

func (s *Sidecar) Records() int64

Records is every record and blob the sidecar counts.

type SidecarModule added in v1.76.1

type SidecarModule struct {
	Module        port.Module `json:"module"`
	StateRecords  int64       `json:"stateRecords"`
	SecretRecords int64       `json:"secretRecords"`
	Blobs         int64       `json:"blobs"`
	BlobBytes     int64       `json:"blobBytes"`
}

SidecarModule is one module's counts.

type Writer

type Writer struct {
	// contains filtered or unexported fields
}

Writer writes one backup into a local encrypted file (Archive); Publish uploads it. Records are spooled to disk as they come, so memory is one record or one blob. It is not safe for concurrent use.

func NewWriter

func NewWriter(p Params) (*Writer, error)

NewWriter starts a backup. Nothing is written to a store.

func (*Writer) Abort

func (w *Writer) Abort()

Abort removes the spool without writing a backup.

func (*Writer) Add

func (w *Writer) Add(_ context.Context, s Section, m port.Module, record []byte) error

Add appends one record (one line of JSON) to the state or the secrets file of a module. The records of one file must come together: a file once left is closed.

func (*Writer) AddBlob added in v1.76.1

func (w *Writer) AddBlob(_ context.Context, m port.Module, key string, body []byte) error

AddBlob adds one blob of a module as a file of its own.

func (*Writer) Close

func (w *Writer) Close(_ context.Context) (*Archive, error)

Close finishes the backup: it writes manifest.json and every member into a tar, compresses it with gzip and encrypts it with age into a file, and returns that file with its manifest and sidecar. The spool is removed.

func (*Writer) Include added in v1.76.1

func (w *Writer) Include(m port.Module) error

Include puts a module in the backup even when it adds no record: its record files are written, empty.

Directories

Path Synopsis
Package app assembles the backup module as a process of its own (docs/decisions/0071): the storage it reads an installation from, the archive bucket and the age recipients it writes to (or, in the restore role, the age identity it reads with), the audit trail, and the job that runs under the backup lease (internal/backup/job).
Package app assembles the backup module as a process of its own (docs/decisions/0071): the storage it reads an installation from, the archive bucket and the age recipients it writes to (or, in the restore role, the age identity it reads with), the audit trail, and the job that runs under the backup lease (internal/backup/job).
Package export reads an installation through the ports and writes it into a backup archive (backup.Writer).
Package export reads an installation through the ports and writes it into a backup archive (backup.Writer).
Package job runs the backup module's work: one backup under the backup lease, its status and retention records, the audit records, the listing of the archive and the prune.
Package job runs the backup module's work: one backup under the backup lease, its status and retention records, the audit records, the listing of the archive and the prune.
Package restore writes a backup (backup.Reader, verified whole when it was opened) into an installation of layout v5, and shows beforehand what that would do.
Package restore writes a backup (backup.Reader, verified whole when it was opened) into an installation of layout v5, and shows beforehand what that would do.
Package restorejob is the restore function's work: one restore at a time, asynchronous, with every module under maintenance while it writes (docs/decisions/0071 and 0072, point 9).
Package restorejob is the restore function's work: one restore at a time, asynchronous, with every module under maintenance while it writes (docs/decisions/0071 and 0072, point 9).
Package rpc is the backup module as another module sees it (docs/decisions/0071): the status of the backups, the list of what the archive holds, and a backup now, as methods of the `backup` module over internal/modcall.
Package rpc is the backup module as another module sees it (docs/decisions/0071): the status of the backups, the list of what the archive holds, and a backup now, as methods of the `backup` module over internal/modcall.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL