Documentation
¶
Overview ¶
Package backup is the archive format of a sluis backup (format 2). It holds no exporter and no restore logic: those write and read records through Writer and Reader.
Objects ¶
A backup is two objects in an S3-compatible bucket, written through a port.Blob:
backup/<instance>/<id>.tar.gz.age the backup backup/<instance>/<id>.json the sidecar, written last
The id is the UTC start and a random suffix, 20261013T020000Z-ab12cd. The sidecar (Sidecar) is plain JSON: the object's key, size and SHA-256, the recipients it was encrypted to and, per module, the counts. It names no secret and holds no value. It is written only after the object, so it is the completion marker: an object without a sidecar is not a backup.
The object ¶
The object is a POSIX tar (PAX), compressed with gzip and encrypted with age (https://age-encryption.org/v1) to the configured X25519 recipients, binary, not armored. It is read without sluis:
age -d -i key.txt <id>.tar.gz.age | tar xz
The tar's members, in this order:
manifest.json <module>/state.jsonl <module>/secrets.jsonl <module>/blobs/<escaped object key> (sorted by member name)
for each module in port.Modules order. A JSONL line is one export record (internal/backup/export); a blob is the object's bytes. Every member has mode 0600, the backup's start as its modification time and no owner names.
manifest.json (Manifest) is first and lists the SHA-256 of every other member and the counts of each module, so a reader verifies each member as it reads it and refuses a missing, extra, reordered or changed one.
Escaping a blob key ¶
An object key becomes a relative path that cannot leave its directory and reads back to the same key (EscapeKey, UnescapeKey): the key is split at '/', and in each segment every byte that is not a letter, a digit, '-', '_' or '.' is written %XX (upper-case hex), as is a '.' that begins a segment (so no segment is '.', '..' or hidden); an empty segment is written '%'.
Encryption ¶
The writer needs only the recipients' public keys: it holds no key that decrypts, and no KMS. The restore reads the age identity (AGE-SECRET-KEY-1...) from its own secret. age authenticates the whole stream, so a changed byte fails the decryption; the sidecar's SHA-256 is checked before that, and the manifest's hashes after it.
Versions ¶
The sidecar and the manifest carry the format version; this release reads format 2 only (Version). Format 1 (chunks sealed under a KMS data key) is gone: no format 1 backup was ever kept.
Index ¶
- Constants
- Variables
- func BlobFile(m port.Module, key string) string
- func EscapeKey(key string) string
- func Fingerprint(recipient string) string
- func Fingerprints(recipients []string) []string
- func ObjectName(instance, id string) string
- func OpenVerified(ctx context.Context, store port.Blob, ids Identities, v Verifier, ...) (*Reader, Verification, error)
- func ParseIdentities(text string) ([]age.Identity, error)
- func ParseRecipients(in []string) ([]age.Recipient, error)
- func Publish(ctx context.Context, store port.Blob, a *Archive) error
- func ReasonOf(err error) string
- func Root(instance string) string
- func SecretsFile(m port.Module) string
- func SidecarName(instance, id string) string
- func StateFile(m port.Module) string
- func UnescapeKey(path string) (string, error)
- func ValidName(s string) bool
- type Archive
- type Attestor
- type AttestorFunc
- type BlobsEntry
- type Claim
- type Fetched
- type Hashes
- type Identities
- type Layout
- type Lines
- type Manifest
- type ModuleEntry
- type NoAttestor
- type ObjectInfo
- type Params
- type Reader
- func (rd *Reader) Blobs(ctx context.Context, m port.Module, fn func(key string, body []byte) error) error
- func (rd *Reader) Close()
- func (rd *Reader) Manifest() *Manifest
- func (rd *Reader) Records(ctx context.Context, s Section, m port.Module, fn func(record []byte) error) error
- func (rd *Reader) Sidecar() *Sidecar
- type Section
- type Sidecar
- type SidecarModule
- type Verification
- type Verifier
- type Writer
- func (w *Writer) Abort()
- func (w *Writer) Add(_ context.Context, s Section, m port.Module, record []byte) error
- func (w *Writer) AddBlob(_ context.Context, m port.Module, key string, body []byte) error
- func (w *Writer) Close(_ context.Context) (*Archive, error)
- func (w *Writer) Include(m port.Module) error
Constants ¶
const ( ObjectSuffix = ".tar.gz.age" SidecarSuffix = ".json" )
The suffixes of a backup's two objects.
const ( ReasonUnattested = "unattested" ReasonMismatch = "mismatch" ReasonUnsealed = "unsealed" )
The verification words, in the audit record of an unverified restore.
const ManifestFile = "manifest.json"
ManifestFile is the name of the tar's first member.
const Version = 2
Version is the format version this release writes and reads.
Variables ¶
var ( // ErrFormat is an archive this release cannot read: no sidecar, not a // manifest, an unknown version, or a field out of range. ErrFormat = errors.New("backup: unreadable archive") // ErrIntegrity is an archive that was changed or truncated: the object's // size or SHA-256, its decryption, a member's hash or a count did not // match. ErrIntegrity = errors.New("backup: archive failed verification") // ErrKey is an archive the identity does not open: it is not one of the // recipients, or the identity could not be read. Nothing was read. ErrKey = errors.New("backup: the identity does not open the archive") )
Errors a caller can match with errors.Is.
var ( // ErrUnverified is a backup the sealed audit trail does not vouch for. ErrUnverified = errors.New("backup: not vouched for by the sealed audit trail") // ErrNotAttested is a backup of which the trail holds no completed-backup // record: planted in the bucket, or its record never arrived. ErrNotAttested = fmt.Errorf("%w: the trail holds no record that this backup was made", ErrUnverified) // ErrMismatch is a backup whose record names other bytes or other // recipients, or whose records disagree with each other: positive evidence // that the archive was tampered with. No flag overrides it. ErrMismatch = fmt.Errorf("%w: the record of this backup disagrees with what the archive holds", ErrUnverified) // ErrNotSealed is a backup whose record is in the trail but whose hour is // not sealed yet. Wait for the notary's next run and try again. ErrNotSealed = fmt.Errorf("%w: the hour holding this backup's record is not sealed yet", ErrUnverified) // that is not configured to ask. ErrAttestUnavailable = fmt.Errorf("%w: the trail could not be asked", ErrUnverified) // ErrRecipients is a backup encrypted to other recipients than the restore // is configured with: a backup written before the recipients changed. It is // not an [ErrUnverified], and no flag overrides it. It is not a defence // against forgery: the recipients are public keys, and a forger lists them. ErrRecipients = errors.New("backup: the sidecar's recipients are not the configured ones") // ErrNoRecipients is an unverified restore with no recipients to compare // the sidecar with: it would have no check left. ErrNoRecipients = errors.New("backup: an unverified restore needs backup.recipients to compare the sidecar with") )
Errors a caller can match with errors.Is. Each of the first four wraps ErrUnverified. `allowUnverified` overrides ErrNotAttested, ErrNotSealed and ErrAttestUnavailable: the trail says nothing yet. It does not override ErrMismatch, where the trail says something else.
Functions ¶
func EscapeKey ¶ added in v1.76.1
EscapeKey turns an object key into a relative path inside the tar: split at '/', each segment keeps letters, digits, '-', '_' and '.', and writes every other byte as %XX, as well as a '.' that begins the segment; an empty segment is "%". The result never holds "." or ".." as a segment, an absolute path or a byte a file system treats specially. UnescapeKey reverses it.
func Fingerprint ¶ added in v1.77.0
Fingerprint is the lower-case hex SHA-256 of an age recipient (the age1... string, trimmed). It is what the audit record of a backup names the recipients by.
func Fingerprints ¶ added in v1.77.0
Fingerprints are the Fingerprint of each recipient, sorted and without repeats: two lists name the same recipients when their fingerprints are equal.
func ObjectName ¶ added in v1.76.1
ObjectName is the key of a backup's encrypted object.
func OpenVerified ¶ added in v1.77.0
func OpenVerified(ctx context.Context, store port.Blob, ids Identities, v Verifier, instance, id, tempDir string) (*Reader, Verification, error)
OpenVerified is Open for a restore: the backup is fetched, and v decides whether it is genuine before the identity is read or a byte is decrypted (see Verifier). A backup v refuses is ErrUnverified (or ErrRecipients), and nothing is read of it but its sidecar and its hash. The verification it passed is returned with the reader.
func ParseIdentities ¶ added in v1.76.1
ParseIdentities parses an age identity file: one or more AGE-SECRET-KEY-1... lines, with # comments and blank lines allowed (age-keygen's output).
func ParseRecipients ¶ added in v1.76.1
ParseRecipients parses age X25519 recipients (age1...). At least one is required.
func Publish ¶ added in v1.76.1
Publish uploads an archive: the encrypted object with one write, then the sidecar. A backup is complete only once Publish returned; an object without its sidecar is not a backup and a prune removes it.
func ReasonOf ¶ added in v1.77.0
ReasonOf is the short word for why err is an ErrUnverified; "" for any other error.
func SecretsFile ¶ added in v1.76.1
SecretsFile is the member name of a module's secrets.
func SidecarName ¶ added in v1.76.1
SidecarName is the key of a backup's sidecar.
func UnescapeKey ¶ added in v1.76.1
UnescapeKey reverses EscapeKey. It accepts only the form EscapeKey writes, so one key has one path.
Types ¶
type Archive ¶ added in v1.76.1
type Archive struct {
Manifest *Manifest
// Sidecar describes the encrypted file; [Publish] writes it after it.
Sidecar *Sidecar
// Path is the encrypted file.
Path string
// SidecarSHA256 is the SHA-256 (hex) of the sidecar exactly as [Publish]
// wrote it; empty until it has.
SidecarSHA256 string
}
Archive is a finished backup on local disk, not yet uploaded.
type Attestor ¶ added in v1.77.0
type Attestor interface {
// Attest returns every record of the backup, none when there is none. It
// returns an [ErrAttestUnavailable] when it cannot tell.
Attest(ctx context.Context, instance, id string) ([]Claim, error)
}
Attestor finds the completed-backup records of a backup in the audit trail.
type AttestorFunc ¶ added in v1.77.0
AttestorFunc is an Attestor that is a function.
type BlobsEntry ¶ added in v1.76.1
BlobsEntry counts a module's blob members.
type Claim ¶ added in v1.77.0
type Claim struct {
// RecordID is the record's id in the trail, for the log.
RecordID string
// Instance is the installation the record says the backup is of.
Instance string
// ObjectKey, ObjectSize and ObjectSHA256 describe the encrypted object;
// SidecarSHA256 is the hash of the sidecar as written.
ObjectKey string
ObjectSize int64
ObjectSHA256 string
SidecarSHA256 string
// Recipients are the [Fingerprints] of the recipients it was encrypted to.
Recipients []string
// Sealed is whether the hour holding the record is sealed, and SealKey the
// seal that covers it.
Sealed bool
SealKey string
}
Claim is what the audit trail says one backup is: the data of a completed-backup record.
type Fetched ¶ added in v1.77.0
Fetched is a backup read from the archive: the sidecar, the SHA-256 of the sidecar's bytes, and the encrypted object, whose size and SHA-256 matched the sidecar's. None of it is authenticated: whoever may write the archive bucket wrote it. A Verifier is what says it is the backup that was made.
type Hashes ¶ added in v1.77.0
type Hashes struct{ Object, Sidecar string }
Hashes are the hashes of a backup's two objects.
type Identities ¶ added in v1.76.1
Identities returns the age identities a reader decrypts with; it is called once the object has been checked, so a backup that is not there never reads the secret.
type Layout ¶
type Layout string
Layout names the storage layout of the installation that was backed up.
type Manifest ¶
type Manifest struct {
// Format is the archive format version (see [Version]).
Format int `json:"format"`
// Instance and ID name the backup; they are also in its object key.
Instance string `json:"instance"`
ID string `json:"id"`
// Layout is the storage layout of the source installation.
Layout Layout `json:"layout"`
// Created is the start of the backup, UTC, RFC 3339.
Created string `json:"created"`
// Creator names what wrote it, for example "sluis-backup 1.76.1".
Creator string `json:"creator"`
// Modules lists the modules in the backup in [port.Modules] order, each
// once.
Modules []ModuleEntry `json:"modules"`
// Files is the SHA-256 (hex) of every other member, by member name.
Files map[string]string `json:"files"`
}
Manifest is manifest.json, the first member of the tar.
type ModuleEntry ¶
type ModuleEntry struct {
Module port.Module `json:"module"`
State Lines `json:"state"`
Secrets Lines `json:"secrets"`
Blobs BlobsEntry `json:"blobs"`
}
ModuleEntry is one module's members.
type NoAttestor ¶ added in v1.77.0
type NoAttestor struct{}
NoAttestor is the attestor of a restore that is not configured to verify: it can tell nothing, so every backup is ErrAttestUnavailable until an operator passes allowUnverified.
type ObjectInfo ¶ added in v1.76.1
type ObjectInfo struct {
Key string `json:"key"`
Size int64 `json:"size"`
SHA256 string `json:"sha256"`
}
ObjectInfo is the encrypted object as stored.
type Params ¶
type Params struct {
Instance string
ID string
Layout Layout
// Creator names what writes it, for example "sluis-backup 1.76.1".
Creator string
// Recipients are the age X25519 public keys (age1...) the backup is
// encrypted to; at least one.
Recipients []string
// Now is the clock; time.Now when nil.
Now func() time.Time
// TempDir holds the spool and the encrypted file while they are written;
// os.TempDir() (/tmp on Lambda) when empty.
TempDir string
}
Params describe the backup a Writer makes.
type Reader ¶
type Reader struct {
// contains filtered or unexported fields
}
Reader is one verified backup, decrypted into a local spool.
func Decrypt ¶ added in v1.76.1
func Decrypt(ctx context.Context, sc *Sidecar, r io.Reader, idents []age.Identity, tempDir string) (*Reader, error)
Decrypt reads the encrypted object of the sidecar from r: see Open.
func Open ¶
func Open(ctx context.Context, store port.Blob, ids Identities, instance, id, tempDir string) (*Reader, error)
Open fetches a backup, checks the object against its sidecar, decrypts it with the identities, and verifies every member of the tar against the manifest (its hash, its place in the order, the counts) before it returns. A key that does not open it fails with ErrKey; anything changed, with ErrIntegrity. tempDir holds the spool ("" is os.TempDir()).
Open trusts the archive bucket: it does not ask whether the backup is the one the backup function wrote. A restore uses OpenVerified.
func (*Reader) Blobs ¶ added in v1.76.1
func (rd *Reader) Blobs(ctx context.Context, m port.Module, fn func(key string, body []byte) error) error
Blobs calls fn with each blob of a module, sorted by member name; the body is valid only during the call.
func (*Reader) Records ¶
func (rd *Reader) Records(ctx context.Context, s Section, m port.Module, fn func(record []byte) error) error
Records calls fn with each record of the state or the secrets file of a module, in order. The slice passed to fn is valid only during the call. A module the backup does not hold has no records.
type Section ¶
type Section string
Section is one of the kinds of content a module contributes.
The sections, in archive order.
type Sidecar ¶ added in v1.76.1
type Sidecar struct {
Format int `json:"format"`
Instance string `json:"instance"`
ID string `json:"id"`
Created string `json:"created"`
Creator string `json:"creator"`
Layout Layout `json:"layout"`
Object ObjectInfo `json:"object"`
Recipients []string `json:"recipients"`
Modules []SidecarModule `json:"modules"`
}
Sidecar is <id>.json, written after the object: what a listing shows and what a reader checks the object against. It names no secret.
func Fetch ¶ added in v1.76.1
Fetch reads a backup's sidecar and its encrypted object and checks the object's size and SHA-256 against the sidecar. It needs no identity.
func ReadSidecar ¶ added in v1.76.1
ReadSidecar reads a backup's sidecar. Nothing authenticates it: it is for listing, and for checking the object against before it is decrypted.
type SidecarModule ¶ added in v1.76.1
type SidecarModule struct {
Module port.Module `json:"module"`
StateRecords int64 `json:"stateRecords"`
SecretRecords int64 `json:"secretRecords"`
Blobs int64 `json:"blobs"`
BlobBytes int64 `json:"blobBytes"`
}
SidecarModule is one module's counts.
type Verification ¶ added in v1.77.0
type Verification struct {
// Unverified is set when the backup went ahead without a sealed record of
// it, and Reason says why (the Reason... words).
Unverified bool
Reason string
// Hashes are those of the backup that was checked.
Hashes Hashes
// RecordID and SealKey name the record and seal that vouched for it.
RecordID, SealKey string
}
Verification is what was established about a fetched backup.
type Verifier ¶ added in v1.77.0
type Verifier struct {
// Attestor asks the trail. Nil is [NoAttestor].
Attestor Attestor
// Recipients are the age recipients (age1...) the restore is configured
// with. When set the sidecar must list exactly these, verified or not.
Recipients []string
// AllowUnverified lets a backup the trail says nothing of go ahead (no
// record, an hour not sealed, a trail that cannot be asked), with
// [Verification.Unverified] set. It needs Recipients, so that a backup of
// an earlier set is still refused. It does not override [ErrMismatch] or
// [ErrRecipients].
AllowUnverified bool
// Pinned, when set, is the hashes an earlier slice of a restore checked.
// The trail is not asked again: the fetched backup must be those bytes,
// and Was is the verification that slice reached.
Pinned *Hashes
Was Verification
}
Verifier decides whether a fetched backup is the one the backup function made.
func (Verifier) Check ¶ added in v1.77.0
Check holds f to the rules, in this order:
- The sidecar's recipients are the configured ones, if any are configured. Always.
- Pinned, when set: f has those hashes, and nothing more is asked.
- The trail holds a record of this backup; every record of it agrees; its object and sidecar hashes, its object key and size and its recipients are those of f; and the hour holding it is sealed.
A failure of 3 is an ErrUnverified; with AllowUnverified the backup goes ahead and the verification says why it was not verified.
type Writer ¶
type Writer struct {
// contains filtered or unexported fields
}
Writer writes one backup into a local encrypted file (Archive); Publish uploads it. Records are spooled to disk as they come, so memory is one record or one blob. It is not safe for concurrent use.
func (*Writer) Add ¶
Add appends one record (one line of JSON) to the state or the secrets file of a module. The records of one file must come together: a file once left is closed.
Directories
¶
| Path | Synopsis |
|---|---|
|
Package app assembles the backup module as a process of its own (docs/decisions/0071): the storage it reads an installation from, the archive bucket and the age recipients it writes to (or, in the restore role, the age identity it reads with), the audit trail, and the job that runs under the backup lease (internal/backup/job).
|
Package app assembles the backup module as a process of its own (docs/decisions/0071): the storage it reads an installation from, the archive bucket and the age recipients it writes to (or, in the restore role, the age identity it reads with), the audit trail, and the job that runs under the backup lease (internal/backup/job). |
|
Package attest is how a restore asks the audit trail whether a backup is the one the backup function made (backup.Attestor), through the audit installation's query service.
|
Package attest is how a restore asks the audit trail whether a backup is the one the backup function made (backup.Attestor), through the audit installation's query service. |
|
Package export reads an installation through the ports and writes it into a backup archive (backup.Writer).
|
Package export reads an installation through the ports and writes it into a backup archive (backup.Writer). |
|
Package job runs the backup module's work: one backup under the backup lease, its status and retention records, the audit records, the listing of the archive and the prune.
|
Package job runs the backup module's work: one backup under the backup lease, its status and retention records, the audit records, the listing of the archive and the prune. |
|
Package restore writes a backup (backup.Reader, verified whole when it was opened) into an installation of layout v5, and shows beforehand what that would do.
|
Package restore writes a backup (backup.Reader, verified whole when it was opened) into an installation of layout v5, and shows beforehand what that would do. |
|
Package restorejob is the restore function's work: one restore at a time, asynchronous, with every module under maintenance while it writes (docs/decisions/0071 and 0072, point 9).
|
Package restorejob is the restore function's work: one restore at a time, asynchronous, with every module under maintenance while it writes (docs/decisions/0071 and 0072, point 9). |
|
Package rpc is the backup module as another module sees it (docs/decisions/0071): the status of the backups, the list of what the archive holds, and a backup now, as methods of the `backup` module over internal/modcall.
|
Package rpc is the backup module as another module sees it (docs/decisions/0071): the status of the backups, the list of what the archive holds, and a backup now, as methods of the `backup` module over internal/modcall. |