backup

package
v1.77.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 19, 2026 License: MIT Imports: 23 Imported by: 0

Documentation

Overview

Package backup is the archive format of a sluis backup (format 2). It holds no exporter and no restore logic: those write and read records through Writer and Reader.

Objects

A backup is two objects in an S3-compatible bucket, written through a port.Blob:

backup/<instance>/<id>.tar.gz.age   the backup
backup/<instance>/<id>.json         the sidecar, written last

The id is the UTC start and a random suffix, 20261013T020000Z-ab12cd. The sidecar (Sidecar) is plain JSON: the object's key, size and SHA-256, the recipients it was encrypted to and, per module, the counts. It names no secret and holds no value. It is written only after the object, so it is the completion marker: an object without a sidecar is not a backup.

The object

The object is a POSIX tar (PAX), compressed with gzip and encrypted with age (https://age-encryption.org/v1) to the configured X25519 recipients, binary, not armored. It is read without sluis:

age -d -i key.txt <id>.tar.gz.age | tar xz

The tar's members, in this order:

manifest.json
<module>/state.jsonl
<module>/secrets.jsonl
<module>/blobs/<escaped object key>   (sorted by member name)

for each module in port.Modules order. A JSONL line is one export record (internal/backup/export); a blob is the object's bytes. Every member has mode 0600, the backup's start as its modification time and no owner names.

manifest.json (Manifest) is first and lists the SHA-256 of every other member and the counts of each module, so a reader verifies each member as it reads it and refuses a missing, extra, reordered or changed one.

Escaping a blob key

An object key becomes a relative path that cannot leave its directory and reads back to the same key (EscapeKey, UnescapeKey): the key is split at '/', and in each segment every byte that is not a letter, a digit, '-', '_' or '.' is written %XX (upper-case hex), as is a '.' that begins a segment (so no segment is '.', '..' or hidden); an empty segment is written '%'.

Encryption

The writer needs only the recipients' public keys: it holds no key that decrypts, and no KMS. The restore reads the age identity (AGE-SECRET-KEY-1...) from its own secret. age authenticates the whole stream, so a changed byte fails the decryption; the sidecar's SHA-256 is checked before that, and the manifest's hashes after it.

Versions

The sidecar and the manifest carry the format version; this release reads format 2 only (Version). Format 1 (chunks sealed under a KMS data key) is gone: no format 1 backup was ever kept.

Index

Constants

View Source
const (
	ObjectSuffix  = ".tar.gz.age"
	SidecarSuffix = ".json"
)

The suffixes of a backup's two objects.

View Source
const (
	ReasonUnattested  = "unattested"
	ReasonMismatch    = "mismatch"
	ReasonUnsealed    = "unsealed"
	ReasonUnavailable = "unavailable"
)

The verification words, in the audit record of an unverified restore.

View Source
const ManifestFile = "manifest.json"

ManifestFile is the name of the tar's first member.

View Source
const Version = 2

Version is the format version this release writes and reads.

Variables

View Source
var (
	// ErrFormat is an archive this release cannot read: no sidecar, not a
	// manifest, an unknown version, or a field out of range.
	ErrFormat = errors.New("backup: unreadable archive")
	// ErrIntegrity is an archive that was changed or truncated: the object's
	// size or SHA-256, its decryption, a member's hash or a count did not
	// match.
	ErrIntegrity = errors.New("backup: archive failed verification")
	// ErrKey is an archive the identity does not open: it is not one of the
	// recipients, or the identity could not be read. Nothing was read.
	ErrKey = errors.New("backup: the identity does not open the archive")
)

Errors a caller can match with errors.Is.

View Source
var (
	// ErrUnverified is a backup the sealed audit trail does not vouch for.
	ErrUnverified = errors.New("backup: not vouched for by the sealed audit trail")
	// ErrNotAttested is a backup of which the trail holds no completed-backup
	// record: planted in the bucket, or its record never arrived.
	ErrNotAttested = fmt.Errorf("%w: the trail holds no record that this backup was made", ErrUnverified)
	// ErrMismatch is a backup whose record names other bytes or other
	// recipients, or whose records disagree with each other: positive evidence
	// that the archive was tampered with. No flag overrides it.
	ErrMismatch = fmt.Errorf("%w: the record of this backup disagrees with what the archive holds", ErrUnverified)
	// ErrNotSealed is a backup whose record is in the trail but whose hour is
	// not sealed yet. Wait for the notary's next run and try again.
	ErrNotSealed = fmt.Errorf("%w: the hour holding this backup's record is not sealed yet", ErrUnverified)
	// ErrAttestUnavailable is a trail that could not be asked, or a restore
	// that is not configured to ask.
	ErrAttestUnavailable = fmt.Errorf("%w: the trail could not be asked", ErrUnverified)
	// ErrRecipients is a backup encrypted to other recipients than the restore
	// is configured with: a backup written before the recipients changed. It is
	// not an [ErrUnverified], and no flag overrides it. It is not a defence
	// against forgery: the recipients are public keys, and a forger lists them.
	ErrRecipients = errors.New("backup: the sidecar's recipients are not the configured ones")
	// ErrNoRecipients is an unverified restore with no recipients to compare
	// the sidecar with: it would have no check left.
	ErrNoRecipients = errors.New("backup: an unverified restore needs backup.recipients to compare the sidecar with")
)

Errors a caller can match with errors.Is. Each of the first four wraps ErrUnverified. `allowUnverified` overrides ErrNotAttested, ErrNotSealed and ErrAttestUnavailable: the trail says nothing yet. It does not override ErrMismatch, where the trail says something else.

Functions

func BlobFile added in v1.76.1

func BlobFile(m port.Module, key string) string

BlobFile is the member name of one blob of a module.

func EscapeKey added in v1.76.1

func EscapeKey(key string) string

EscapeKey turns an object key into a relative path inside the tar: split at '/', each segment keeps letters, digits, '-', '_' and '.', and writes every other byte as %XX, as well as a '.' that begins the segment; an empty segment is "%". The result never holds "." or ".." as a segment, an absolute path or a byte a file system treats specially. UnescapeKey reverses it.

func Fingerprint added in v1.77.0

func Fingerprint(recipient string) string

Fingerprint is the lower-case hex SHA-256 of an age recipient (the age1... string, trimmed). It is what the audit record of a backup names the recipients by.

func Fingerprints added in v1.77.0

func Fingerprints(recipients []string) []string

Fingerprints are the Fingerprint of each recipient, sorted and without repeats: two lists name the same recipients when their fingerprints are equal.

func ObjectName added in v1.76.1

func ObjectName(instance, id string) string

ObjectName is the key of a backup's encrypted object.

func OpenVerified added in v1.77.0

func OpenVerified(ctx context.Context, store port.Blob, ids Identities, v Verifier, instance, id, tempDir string) (*Reader, Verification, error)

OpenVerified is Open for a restore: the backup is fetched, and v decides whether it is genuine before the identity is read or a byte is decrypted (see Verifier). A backup v refuses is ErrUnverified (or ErrRecipients), and nothing is read of it but its sidecar and its hash. The verification it passed is returned with the reader.

func ParseIdentities added in v1.76.1

func ParseIdentities(text string) ([]age.Identity, error)

ParseIdentities parses an age identity file: one or more AGE-SECRET-KEY-1... lines, with # comments and blank lines allowed (age-keygen's output).

func ParseRecipients added in v1.76.1

func ParseRecipients(in []string) ([]age.Recipient, error)

ParseRecipients parses age X25519 recipients (age1...). At least one is required.

func Publish added in v1.76.1

func Publish(ctx context.Context, store port.Blob, a *Archive) error

Publish uploads an archive: the encrypted object with one write, then the sidecar. A backup is complete only once Publish returned; an object without its sidecar is not a backup and a prune removes it.

func ReasonOf added in v1.77.0

func ReasonOf(err error) string

ReasonOf is the short word for why err is an ErrUnverified; "" for any other error.

func Root added in v1.76.1

func Root(instance string) string

Root is the prefix of every backup of an instance.

func SecretsFile added in v1.76.1

func SecretsFile(m port.Module) string

SecretsFile is the member name of a module's secrets.

func SidecarName added in v1.76.1

func SidecarName(instance, id string) string

SidecarName is the key of a backup's sidecar.

func StateFile added in v1.76.1

func StateFile(m port.Module) string

StateFile is the member name of a module's State records.

func UnescapeKey added in v1.76.1

func UnescapeKey(path string) (string, error)

UnescapeKey reverses EscapeKey. It accepts only the form EscapeKey writes, so one key has one path.

func ValidName

func ValidName(s string) bool

ValidName reports whether s is acceptable as an instance name or a backup id: one path segment of letters, digits, dot, dash and underscore.

Types

type Archive added in v1.76.1

type Archive struct {
	Manifest *Manifest
	// Sidecar describes the encrypted file; [Publish] writes it after it.
	Sidecar *Sidecar
	// Path is the encrypted file.
	Path string
	// SidecarSHA256 is the SHA-256 (hex) of the sidecar exactly as [Publish]
	// wrote it; empty until it has.
	SidecarSHA256 string
}

Archive is a finished backup on local disk, not yet uploaded.

func (*Archive) Remove added in v1.76.1

func (a *Archive) Remove()

Remove deletes the local file.

type Attestor added in v1.77.0

type Attestor interface {
	// Attest returns every record of the backup, none when there is none. It
	// returns an [ErrAttestUnavailable] when it cannot tell.
	Attest(ctx context.Context, instance, id string) ([]Claim, error)
}

Attestor finds the completed-backup records of a backup in the audit trail.

type AttestorFunc added in v1.77.0

type AttestorFunc func(ctx context.Context, instance, id string) ([]Claim, error)

AttestorFunc is an Attestor that is a function.

func (AttestorFunc) Attest added in v1.77.0

func (f AttestorFunc) Attest(ctx context.Context, instance, id string) ([]Claim, error)

Attest calls f.

type BlobsEntry added in v1.76.1

type BlobsEntry struct {
	Count int64 `json:"count"`
	Bytes int64 `json:"bytes"`
}

BlobsEntry counts a module's blob members.

type Claim added in v1.77.0

type Claim struct {
	// RecordID is the record's id in the trail, for the log.
	RecordID string
	// Instance is the installation the record says the backup is of.
	Instance string
	// ObjectKey, ObjectSize and ObjectSHA256 describe the encrypted object;
	// SidecarSHA256 is the hash of the sidecar as written.
	ObjectKey     string
	ObjectSize    int64
	ObjectSHA256  string
	SidecarSHA256 string
	// Recipients are the [Fingerprints] of the recipients it was encrypted to.
	Recipients []string
	// Sealed is whether the hour holding the record is sealed, and SealKey the
	// seal that covers it.
	Sealed  bool
	SealKey string
}

Claim is what the audit trail says one backup is: the data of a completed-backup record.

type Fetched added in v1.77.0

type Fetched struct {
	Sidecar       *Sidecar
	SidecarSHA256 string
	Body          []byte
}

Fetched is a backup read from the archive: the sidecar, the SHA-256 of the sidecar's bytes, and the encrypted object, whose size and SHA-256 matched the sidecar's. None of it is authenticated: whoever may write the archive bucket wrote it. A Verifier is what says it is the backup that was made.

func FetchAll added in v1.77.0

func FetchAll(ctx context.Context, store port.Blob, instance, id string) (*Fetched, error)

FetchAll is Fetch that also reports the hash of the sidecar's bytes.

type Hashes added in v1.77.0

type Hashes struct{ Object, Sidecar string }

Hashes are the hashes of a backup's two objects.

type Identities added in v1.76.1

type Identities func(ctx context.Context) ([]age.Identity, error)

Identities returns the age identities a reader decrypts with; it is called once the object has been checked, so a backup that is not there never reads the secret.

type Layout

type Layout string

Layout names the storage layout of the installation that was backed up.

const (
	LayoutV4 Layout = "v4"
	LayoutV5 Layout = "v5"
)

The layouts a backup can describe.

func (Layout) Valid

func (l Layout) Valid() bool

Valid reports whether l is a known layout.

type Lines added in v1.76.1

type Lines struct {
	Records int64  `json:"records"`
	SHA256  string `json:"sha256"`
}

Lines is one JSONL member: its record count and SHA-256.

type Manifest

type Manifest struct {
	// Format is the archive format version (see [Version]).
	Format int `json:"format"`
	// Instance and ID name the backup; they are also in its object key.
	Instance string `json:"instance"`
	ID       string `json:"id"`
	// Layout is the storage layout of the source installation.
	Layout Layout `json:"layout"`
	// Created is the start of the backup, UTC, RFC 3339.
	Created string `json:"created"`
	// Creator names what wrote it, for example "sluis-backup 1.76.1".
	Creator string `json:"creator"`
	// Modules lists the modules in the backup in [port.Modules] order, each
	// once.
	Modules []ModuleEntry `json:"modules"`
	// Files is the SHA-256 (hex) of every other member, by member name.
	Files map[string]string `json:"files"`
}

Manifest is manifest.json, the first member of the tar.

func (*Manifest) Module added in v1.76.1

func (m *Manifest) Module(mod port.Module) *ModuleEntry

Module returns the entry of a module, or nil when the backup does not hold it.

func (*Manifest) Records

func (m *Manifest) Records(s Section) int64

Records is the number of records (lines, or blobs) of a section across all modules.

type ModuleEntry

type ModuleEntry struct {
	Module  port.Module `json:"module"`
	State   Lines       `json:"state"`
	Secrets Lines       `json:"secrets"`
	Blobs   BlobsEntry  `json:"blobs"`
}

ModuleEntry is one module's members.

type NoAttestor added in v1.77.0

type NoAttestor struct{}

NoAttestor is the attestor of a restore that is not configured to verify: it can tell nothing, so every backup is ErrAttestUnavailable until an operator passes allowUnverified.

func (NoAttestor) Attest added in v1.77.0

Attest refuses.

type ObjectInfo added in v1.76.1

type ObjectInfo struct {
	Key    string `json:"key"`
	Size   int64  `json:"size"`
	SHA256 string `json:"sha256"`
}

ObjectInfo is the encrypted object as stored.

type Params

type Params struct {
	Instance string
	ID       string
	Layout   Layout
	// Creator names what writes it, for example "sluis-backup 1.76.1".
	Creator string
	// Recipients are the age X25519 public keys (age1...) the backup is
	// encrypted to; at least one.
	Recipients []string
	// Now is the clock; time.Now when nil.
	Now func() time.Time
	// TempDir holds the spool and the encrypted file while they are written;
	// os.TempDir() (/tmp on Lambda) when empty.
	TempDir string
}

Params describe the backup a Writer makes.

type Reader

type Reader struct {
	// contains filtered or unexported fields
}

Reader is one verified backup, decrypted into a local spool.

func Decrypt added in v1.76.1

func Decrypt(ctx context.Context, sc *Sidecar, r io.Reader, idents []age.Identity, tempDir string) (*Reader, error)

Decrypt reads the encrypted object of the sidecar from r: see Open.

func Open

func Open(ctx context.Context, store port.Blob, ids Identities, instance, id, tempDir string) (*Reader, error)

Open fetches a backup, checks the object against its sidecar, decrypts it with the identities, and verifies every member of the tar against the manifest (its hash, its place in the order, the counts) before it returns. A key that does not open it fails with ErrKey; anything changed, with ErrIntegrity. tempDir holds the spool ("" is os.TempDir()).

Open trusts the archive bucket: it does not ask whether the backup is the one the backup function wrote. A restore uses OpenVerified.

func (*Reader) Blobs added in v1.76.1

func (rd *Reader) Blobs(ctx context.Context, m port.Module, fn func(key string, body []byte) error) error

Blobs calls fn with each blob of a module, sorted by member name; the body is valid only during the call.

func (*Reader) Close

func (rd *Reader) Close()

Close removes the spool.

func (*Reader) Manifest

func (rd *Reader) Manifest() *Manifest

Manifest is the verified manifest.

func (*Reader) Records

func (rd *Reader) Records(ctx context.Context, s Section, m port.Module, fn func(record []byte) error) error

Records calls fn with each record of the state or the secrets file of a module, in order. The slice passed to fn is valid only during the call. A module the backup does not hold has no records.

func (*Reader) Sidecar added in v1.76.1

func (rd *Reader) Sidecar() *Sidecar

Sidecar is the backup's sidecar, checked against the manifest.

type Section

type Section string

Section is one of the kinds of content a module contributes.

const (
	State   Section = "state"
	Secrets Section = "secrets"
	Blobs   Section = "blobs"
)

The sections, in archive order.

func Sections

func Sections() []Section

Sections lists every section in archive order.

func (Section) Valid

func (s Section) Valid() bool

Valid reports whether s is one of Sections.

type Sidecar added in v1.76.1

type Sidecar struct {
	Format     int             `json:"format"`
	Instance   string          `json:"instance"`
	ID         string          `json:"id"`
	Created    string          `json:"created"`
	Creator    string          `json:"creator"`
	Layout     Layout          `json:"layout"`
	Object     ObjectInfo      `json:"object"`
	Recipients []string        `json:"recipients"`
	Modules    []SidecarModule `json:"modules"`
}

Sidecar is <id>.json, written after the object: what a listing shows and what a reader checks the object against. It names no secret.

func Fetch added in v1.76.1

func Fetch(ctx context.Context, store port.Blob, instance, id string) (*Sidecar, []byte, error)

Fetch reads a backup's sidecar and its encrypted object and checks the object's size and SHA-256 against the sidecar. It needs no identity.

func ReadSidecar added in v1.76.1

func ReadSidecar(ctx context.Context, store port.Blob, instance, id string) (*Sidecar, error)

ReadSidecar reads a backup's sidecar. Nothing authenticates it: it is for listing, and for checking the object against before it is decrypted.

func (*Sidecar) Records added in v1.76.1

func (s *Sidecar) Records() int64

Records is every record and blob the sidecar counts.

type SidecarModule added in v1.76.1

type SidecarModule struct {
	Module        port.Module `json:"module"`
	StateRecords  int64       `json:"stateRecords"`
	SecretRecords int64       `json:"secretRecords"`
	Blobs         int64       `json:"blobs"`
	BlobBytes     int64       `json:"blobBytes"`
}

SidecarModule is one module's counts.

type Verification added in v1.77.0

type Verification struct {
	// Unverified is set when the backup went ahead without a sealed record of
	// it, and Reason says why (the Reason... words).
	Unverified bool
	Reason     string
	// Hashes are those of the backup that was checked.
	Hashes Hashes
	// RecordID and SealKey name the record and seal that vouched for it.
	RecordID, SealKey string
}

Verification is what was established about a fetched backup.

type Verifier added in v1.77.0

type Verifier struct {
	// Attestor asks the trail. Nil is [NoAttestor].
	Attestor Attestor
	// Recipients are the age recipients (age1...) the restore is configured
	// with. When set the sidecar must list exactly these, verified or not.
	Recipients []string
	// AllowUnverified lets a backup the trail says nothing of go ahead (no
	// record, an hour not sealed, a trail that cannot be asked), with
	// [Verification.Unverified] set. It needs Recipients, so that a backup of
	// an earlier set is still refused. It does not override [ErrMismatch] or
	// [ErrRecipients].
	AllowUnverified bool
	// Pinned, when set, is the hashes an earlier slice of a restore checked.
	// The trail is not asked again: the fetched backup must be those bytes,
	// and Was is the verification that slice reached.
	Pinned *Hashes
	Was    Verification
}

Verifier decides whether a fetched backup is the one the backup function made.

func (Verifier) Check added in v1.77.0

func (v Verifier) Check(ctx context.Context, f *Fetched) (Verification, error)

Check holds f to the rules, in this order:

  1. The sidecar's recipients are the configured ones, if any are configured. Always.
  2. Pinned, when set: f has those hashes, and nothing more is asked.
  3. The trail holds a record of this backup; every record of it agrees; its object and sidecar hashes, its object key and size and its recipients are those of f; and the hour holding it is sealed.

A failure of 3 is an ErrUnverified; with AllowUnverified the backup goes ahead and the verification says why it was not verified.

type Writer

type Writer struct {
	// contains filtered or unexported fields
}

Writer writes one backup into a local encrypted file (Archive); Publish uploads it. Records are spooled to disk as they come, so memory is one record or one blob. It is not safe for concurrent use.

func NewWriter

func NewWriter(p Params) (*Writer, error)

NewWriter starts a backup. Nothing is written to a store.

func (*Writer) Abort

func (w *Writer) Abort()

Abort removes the spool without writing a backup.

func (*Writer) Add

func (w *Writer) Add(_ context.Context, s Section, m port.Module, record []byte) error

Add appends one record (one line of JSON) to the state or the secrets file of a module. The records of one file must come together: a file once left is closed.

func (*Writer) AddBlob added in v1.76.1

func (w *Writer) AddBlob(_ context.Context, m port.Module, key string, body []byte) error

AddBlob adds one blob of a module as a file of its own.

func (*Writer) Close

func (w *Writer) Close(_ context.Context) (*Archive, error)

Close finishes the backup: it writes manifest.json and every member into a tar, compresses it with gzip and encrypts it with age into a file, and returns that file with its manifest and sidecar. The spool is removed.

func (*Writer) Include added in v1.76.1

func (w *Writer) Include(m port.Module) error

Include puts a module in the backup even when it adds no record: its record files are written, empty.

Directories

Path Synopsis
Package app assembles the backup module as a process of its own (docs/decisions/0071): the storage it reads an installation from, the archive bucket and the age recipients it writes to (or, in the restore role, the age identity it reads with), the audit trail, and the job that runs under the backup lease (internal/backup/job).
Package app assembles the backup module as a process of its own (docs/decisions/0071): the storage it reads an installation from, the archive bucket and the age recipients it writes to (or, in the restore role, the age identity it reads with), the audit trail, and the job that runs under the backup lease (internal/backup/job).
Package attest is how a restore asks the audit trail whether a backup is the one the backup function made (backup.Attestor), through the audit installation's query service.
Package attest is how a restore asks the audit trail whether a backup is the one the backup function made (backup.Attestor), through the audit installation's query service.
Package export reads an installation through the ports and writes it into a backup archive (backup.Writer).
Package export reads an installation through the ports and writes it into a backup archive (backup.Writer).
Package job runs the backup module's work: one backup under the backup lease, its status and retention records, the audit records, the listing of the archive and the prune.
Package job runs the backup module's work: one backup under the backup lease, its status and retention records, the audit records, the listing of the archive and the prune.
Package restore writes a backup (backup.Reader, verified whole when it was opened) into an installation of layout v5, and shows beforehand what that would do.
Package restore writes a backup (backup.Reader, verified whole when it was opened) into an installation of layout v5, and shows beforehand what that would do.
Package restorejob is the restore function's work: one restore at a time, asynchronous, with every module under maintenance while it writes (docs/decisions/0071 and 0072, point 9).
Package restorejob is the restore function's work: one restore at a time, asynchronous, with every module under maintenance while it writes (docs/decisions/0071 and 0072, point 9).
Package rpc is the backup module as another module sees it (docs/decisions/0071): the status of the backups, the list of what the archive holds, and a backup now, as methods of the `backup` module over internal/modcall.
Package rpc is the backup module as another module sees it (docs/decisions/0071): the status of the backups, the list of what the archive holds, and a backup now, as methods of the `backup` module over internal/modcall.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL