Documentation
¶
Overview ¶
Package backup is the archive format of a sluis backup (format 2). It holds no exporter and no restore logic: those write and read records through Writer and Reader.
Objects ¶
A backup is two objects in an S3-compatible bucket, written through a port.Blob:
backup/<instance>/<id>.tar.gz.age the backup backup/<instance>/<id>.json the sidecar, written last
The id is the UTC start and a random suffix, 20261013T020000Z-ab12cd. The sidecar (Sidecar) is plain JSON: the object's key, size and SHA-256, the recipients it was encrypted to and, per module, the counts. It names no secret and holds no value. It is written only after the object, so it is the completion marker: an object without a sidecar is not a backup.
The object ¶
The object is a POSIX tar (PAX), compressed with gzip and encrypted with age (https://age-encryption.org/v1) to the configured X25519 recipients, binary, not armored. It is read without sluis:
age -d -i key.txt <id>.tar.gz.age | tar xz
The tar's members, in this order:
manifest.json <module>/state.jsonl <module>/secrets.jsonl <module>/blobs/<escaped object key> (sorted by member name)
for each module in port.Modules order. A JSONL line is one export record (internal/backup/export); a blob is the object's bytes. Every member has mode 0600, the backup's start as its modification time and no owner names.
manifest.json (Manifest) is first and lists the SHA-256 of every other member and the counts of each module, so a reader verifies each member as it reads it and refuses a missing, extra, reordered or changed one.
Escaping a blob key ¶
An object key becomes a relative path that cannot leave its directory and reads back to the same key (EscapeKey, UnescapeKey): the key is split at '/', and in each segment every byte that is not a letter, a digit, '-', '_' or '.' is written %XX (upper-case hex), as is a '.' that begins a segment (so no segment is '.', '..' or hidden); an empty segment is written '%'.
Encryption ¶
The writer needs only the recipients' public keys: it holds no key that decrypts, and no KMS. The restore reads the age identity (AGE-SECRET-KEY-1...) from its own secret. age authenticates the whole stream, so a changed byte fails the decryption; the sidecar's SHA-256 is checked before that, and the manifest's hashes after it.
Versions ¶
The sidecar and the manifest carry the format version; this release reads format 2 only (Version). Format 1 (chunks sealed under a KMS data key) is gone: no format 1 backup was ever kept.
Index ¶
- Constants
- Variables
- func BlobFile(m port.Module, key string) string
- func EscapeKey(key string) string
- func ObjectName(instance, id string) string
- func ParseIdentities(text string) ([]age.Identity, error)
- func ParseRecipients(in []string) ([]age.Recipient, error)
- func Publish(ctx context.Context, store port.Blob, a *Archive) error
- func Root(instance string) string
- func SecretsFile(m port.Module) string
- func SidecarName(instance, id string) string
- func StateFile(m port.Module) string
- func UnescapeKey(path string) (string, error)
- func ValidName(s string) bool
- type Archive
- type BlobsEntry
- type Identities
- type Layout
- type Lines
- type Manifest
- type ModuleEntry
- type ObjectInfo
- type Params
- type Reader
- func (rd *Reader) Blobs(ctx context.Context, m port.Module, fn func(key string, body []byte) error) error
- func (rd *Reader) Close()
- func (rd *Reader) Manifest() *Manifest
- func (rd *Reader) Records(ctx context.Context, s Section, m port.Module, fn func(record []byte) error) error
- func (rd *Reader) Sidecar() *Sidecar
- type Section
- type Sidecar
- type SidecarModule
- type Writer
- func (w *Writer) Abort()
- func (w *Writer) Add(_ context.Context, s Section, m port.Module, record []byte) error
- func (w *Writer) AddBlob(_ context.Context, m port.Module, key string, body []byte) error
- func (w *Writer) Close(_ context.Context) (*Archive, error)
- func (w *Writer) Include(m port.Module) error
Constants ¶
const ( ObjectSuffix = ".tar.gz.age" SidecarSuffix = ".json" )
The suffixes of a backup's two objects.
const ManifestFile = "manifest.json"
ManifestFile is the name of the tar's first member.
const Version = 2
Version is the format version this release writes and reads.
Variables ¶
var ( // ErrFormat is an archive this release cannot read: no sidecar, not a // manifest, an unknown version, or a field out of range. ErrFormat = errors.New("backup: unreadable archive") // ErrIntegrity is an archive that was changed or truncated: the object's // size or SHA-256, its decryption, a member's hash or a count did not // match. ErrIntegrity = errors.New("backup: archive failed verification") // ErrKey is an archive the identity does not open: it is not one of the // recipients, or the identity could not be read. Nothing was read. ErrKey = errors.New("backup: the identity does not open the archive") )
Errors a caller can match with errors.Is.
Functions ¶
func EscapeKey ¶ added in v1.76.1
EscapeKey turns an object key into a relative path inside the tar: split at '/', each segment keeps letters, digits, '-', '_' and '.', and writes every other byte as %XX, as well as a '.' that begins the segment; an empty segment is "%". The result never holds "." or ".." as a segment, an absolute path or a byte a file system treats specially. UnescapeKey reverses it.
func ObjectName ¶ added in v1.76.1
ObjectName is the key of a backup's encrypted object.
func ParseIdentities ¶ added in v1.76.1
ParseIdentities parses an age identity file: one or more AGE-SECRET-KEY-1... lines, with # comments and blank lines allowed (age-keygen's output).
func ParseRecipients ¶ added in v1.76.1
ParseRecipients parses age X25519 recipients (age1...). At least one is required.
func Publish ¶ added in v1.76.1
Publish uploads an archive: the encrypted object with one write, then the sidecar. A backup is complete only once Publish returned; an object without its sidecar is not a backup and a prune removes it.
func SecretsFile ¶ added in v1.76.1
SecretsFile is the member name of a module's secrets.
func SidecarName ¶ added in v1.76.1
SidecarName is the key of a backup's sidecar.
func UnescapeKey ¶ added in v1.76.1
UnescapeKey reverses EscapeKey. It accepts only the form EscapeKey writes, so one key has one path.
Types ¶
type Archive ¶ added in v1.76.1
type Archive struct {
Manifest *Manifest
// Sidecar describes the encrypted file; [Publish] writes it after it.
Sidecar *Sidecar
// Path is the encrypted file.
Path string
}
Archive is a finished backup on local disk, not yet uploaded.
type BlobsEntry ¶ added in v1.76.1
BlobsEntry counts a module's blob members.
type Identities ¶ added in v1.76.1
Identities returns the age identities a reader decrypts with; it is called once the object has been checked, so a backup that is not there never reads the secret.
type Layout ¶
type Layout string
Layout names the storage layout of the installation that was backed up.
type Manifest ¶
type Manifest struct {
// Format is the archive format version (see [Version]).
Format int `json:"format"`
// Instance and ID name the backup; they are also in its object key.
Instance string `json:"instance"`
ID string `json:"id"`
// Layout is the storage layout of the source installation.
Layout Layout `json:"layout"`
// Created is the start of the backup, UTC, RFC 3339.
Created string `json:"created"`
// Creator names what wrote it, for example "sluis-backup 1.76.1".
Creator string `json:"creator"`
// Modules lists the modules in the backup in [port.Modules] order, each
// once.
Modules []ModuleEntry `json:"modules"`
// Files is the SHA-256 (hex) of every other member, by member name.
Files map[string]string `json:"files"`
}
Manifest is manifest.json, the first member of the tar.
type ModuleEntry ¶
type ModuleEntry struct {
Module port.Module `json:"module"`
State Lines `json:"state"`
Secrets Lines `json:"secrets"`
Blobs BlobsEntry `json:"blobs"`
}
ModuleEntry is one module's members.
type ObjectInfo ¶ added in v1.76.1
type ObjectInfo struct {
Key string `json:"key"`
Size int64 `json:"size"`
SHA256 string `json:"sha256"`
}
ObjectInfo is the encrypted object as stored.
type Params ¶
type Params struct {
Instance string
ID string
Layout Layout
// Creator names what writes it, for example "sluis-backup 1.76.1".
Creator string
// Recipients are the age X25519 public keys (age1...) the backup is
// encrypted to; at least one.
Recipients []string
// Now is the clock; time.Now when nil.
Now func() time.Time
// TempDir holds the spool and the encrypted file while they are written;
// os.TempDir() (/tmp on Lambda) when empty.
TempDir string
}
Params describe the backup a Writer makes.
type Reader ¶
type Reader struct {
// contains filtered or unexported fields
}
Reader is one verified backup, decrypted into a local spool.
func Decrypt ¶ added in v1.76.1
func Decrypt(ctx context.Context, sc *Sidecar, r io.Reader, idents []age.Identity, tempDir string) (*Reader, error)
Decrypt reads the encrypted object of the sidecar from r: see Open.
func Open ¶
func Open(ctx context.Context, store port.Blob, ids Identities, instance, id, tempDir string) (*Reader, error)
Open fetches a backup, checks the object against its sidecar, decrypts it with the identities, and verifies every member of the tar against the manifest (its hash, its place in the order, the counts) before it returns. A key that does not open it fails with ErrKey; anything changed, with ErrIntegrity. tempDir holds the spool ("" is os.TempDir()).
func (*Reader) Blobs ¶ added in v1.76.1
func (rd *Reader) Blobs(ctx context.Context, m port.Module, fn func(key string, body []byte) error) error
Blobs calls fn with each blob of a module, sorted by member name; the body is valid only during the call.
func (*Reader) Records ¶
func (rd *Reader) Records(ctx context.Context, s Section, m port.Module, fn func(record []byte) error) error
Records calls fn with each record of the state or the secrets file of a module, in order. The slice passed to fn is valid only during the call. A module the backup does not hold has no records.
type Section ¶
type Section string
Section is one of the kinds of content a module contributes.
The sections, in archive order.
type Sidecar ¶ added in v1.76.1
type Sidecar struct {
Format int `json:"format"`
Instance string `json:"instance"`
ID string `json:"id"`
Created string `json:"created"`
Creator string `json:"creator"`
Layout Layout `json:"layout"`
Object ObjectInfo `json:"object"`
Recipients []string `json:"recipients"`
Modules []SidecarModule `json:"modules"`
}
Sidecar is <id>.json, written after the object: what a listing shows and what a reader checks the object against. It names no secret.
func Fetch ¶ added in v1.76.1
Fetch reads a backup's sidecar and its encrypted object and checks the object's size and SHA-256 against the sidecar. It needs no identity.
func ReadSidecar ¶ added in v1.76.1
ReadSidecar reads a backup's sidecar. Nothing authenticates it: it is for listing, and for checking the object against before it is decrypted.
type SidecarModule ¶ added in v1.76.1
type SidecarModule struct {
Module port.Module `json:"module"`
StateRecords int64 `json:"stateRecords"`
SecretRecords int64 `json:"secretRecords"`
Blobs int64 `json:"blobs"`
BlobBytes int64 `json:"blobBytes"`
}
SidecarModule is one module's counts.
type Writer ¶
type Writer struct {
// contains filtered or unexported fields
}
Writer writes one backup into a local encrypted file (Archive); Publish uploads it. Records are spooled to disk as they come, so memory is one record or one blob. It is not safe for concurrent use.
func (*Writer) Add ¶
Add appends one record (one line of JSON) to the state or the secrets file of a module. The records of one file must come together: a file once left is closed.
Directories
¶
| Path | Synopsis |
|---|---|
|
Package app assembles the backup module as a process of its own (docs/decisions/0071): the storage it reads an installation from, the archive bucket and the age recipients it writes to (or, in the restore role, the age identity it reads with), the audit trail, and the job that runs under the backup lease (internal/backup/job).
|
Package app assembles the backup module as a process of its own (docs/decisions/0071): the storage it reads an installation from, the archive bucket and the age recipients it writes to (or, in the restore role, the age identity it reads with), the audit trail, and the job that runs under the backup lease (internal/backup/job). |
|
Package export reads an installation through the ports and writes it into a backup archive (backup.Writer).
|
Package export reads an installation through the ports and writes it into a backup archive (backup.Writer). |
|
Package job runs the backup module's work: one backup under the backup lease, its status and retention records, the audit records, the listing of the archive and the prune.
|
Package job runs the backup module's work: one backup under the backup lease, its status and retention records, the audit records, the listing of the archive and the prune. |
|
Package restore writes a backup (backup.Reader, verified whole when it was opened) into an installation of layout v5, and shows beforehand what that would do.
|
Package restore writes a backup (backup.Reader, verified whole when it was opened) into an installation of layout v5, and shows beforehand what that would do. |
|
Package restorejob is the restore function's work: one restore at a time, asynchronous, with every module under maintenance while it writes (docs/decisions/0071 and 0072, point 9).
|
Package restorejob is the restore function's work: one restore at a time, asynchronous, with every module under maintenance while it writes (docs/decisions/0071 and 0072, point 9). |
|
Package rpc is the backup module as another module sees it (docs/decisions/0071): the status of the backups, the list of what the archive holds, and a backup now, as methods of the `backup` module over internal/modcall.
|
Package rpc is the backup module as another module sees it (docs/decisions/0071): the status of the backups, the list of what the archive holds, and a backup now, as methods of the `backup` module over internal/modcall. |