Documentation
¶
Overview ¶
Package dbbackup creates and restores hot backups of the cloop SQLite state database (Task 20115).
Why a dedicated package:
Operators need a safe way to capture the current project state without stopping the running cloop daemon. SQLite WAL mode (Task 20084) makes this possible — readers and writers do not block each other — but a naive `cp` of state.db is unsafe because uncommitted frames may still be in the WAL file. We use `VACUUM INTO 'path'` which the SQLite engine guarantees is a transactionally-consistent copy at the read transaction's snapshot point; it is the recommended hot-backup primitive when the C-level Online Backup API is not available, as is the case for the pure-Go modernc.org/sqlite driver we ship.
Backup flow:
- Open the source database (this triggers Migrate, but that is idempotent and does not write user data).
- PRAGMA wal_checkpoint(TRUNCATE) flushes pending WAL frames into the main database file so the snapshot is as current as possible.
- VACUUM INTO 'output.db' produces a self-contained, defragmented copy. The copy has no -wal/-shm sidecars; it is a single file.
- Compute SHA-256 of the copy and write a sidecar metadata JSON describing source path, size, schema version, and duration — and, read back from the copy, the sealing keys its sealed rows depend on and where its audit chain ends, the whole sealed under a key derived from CLOOP_SECRET_KEY (manifest.go).
Restore flow (Task 20410 put everything before the swap into Inspect):
- Preflight, read-only, before anything is touched: PRAGMA integrity_check, the schema this binary can open, the SHA-256 and the seal over the sidecar, and whether the CLOOP_SECRET_KEY the restored hub will run with derives every sealing key the backup's secrets are sealed under. Each refusal names the flag that overrides it, if any.
- Refuse to overwrite an existing destination unless --force is set.
- Atomically swap the file in via tmp+rename.
- Record the restore: an audit.restored row in the restored chain, and a restore record sealed to the audit checkpoint sinks, so `cloop hub audit verify --checkpoints` can tell this restore from a truncation.
Why VACUUM INTO instead of OS-level copy:
- Atomically consistent against in-flight writes.
- Skips freelist pages, so the backup is smaller and quicker to verify.
- Single self-contained file (no WAL/SHM sidecars to ship together).
Index ¶
- Constants
- Variables
- func KeyFingerprint(passphrase string) string
- type BackupOptions
- type BackupReport
- type KeyVerdict
- type ManifestChain
- type ManifestKey
- type Metadata
- type Preflight
- type PreflightOptions
- type Recording
- type Refusal
- type RefusedError
- type RestoreOptions
- type RestoreReport
- type SealState
- type SidecarState
- type UnrecordedError
Constants ¶
const ( KeyStateLegacy = "legacy" KeyStateUnknown = "unknown" )
Key states a manifest names beyond the registry's own (primary, active, retired): the pre-envelope construction, and a key id rows name that the registry does not hold.
const ( OverrideSkipChecksum = "--skip-checksum" OverrideAllowUnsealed = "--allow-unsealed" OverrideAcceptSecretLoss = "--accept-secret-loss" )
Flags that override a refusal, as the CLI spells them.
const EnvKey = auditcheckpoint.EnvKey
EnvKey is the variable the manifest seal is derived from: the secret broker's passphrase, the hub's one root secret.
const ManifestVersion = 1
ManifestVersion is the manifest format this build writes.
const MetadataSuffix = ".meta.json"
MetadataSuffix is appended to the backup file path to form the sidecar metadata path: "/path/foo.db" → "/path/foo.db.meta.json".
Variables ¶
var ( ErrIntegrity = errors.New("dbbackup: the backup fails PRAGMA integrity_check") ErrSchemaAhead = errors.New("dbbackup: the backup's schema is newer than this binary") ErrNoSidecar = errors.New("dbbackup: the backup has no sidecar") ErrChecksum = errors.New("dbbackup: checksum mismatch") ErrSealBroken = errors.New("dbbackup: the sidecar's seal does not verify") ErrUnsealed = errors.New("dbbackup: the sidecar's seal cannot be checked") ErrSecretLoss = errors.New("dbbackup: the key cannot open the backup's sealing keys") )
Sentinels a RefusedError wraps, one per kind of refusal.
var ErrHubServing = errors.New("dbbackup: a hub is serving the destination database")
ErrHubServing is returned when a restore is refused because a hub process still serves the destination database.
var ErrNoReason = errors.New("dbbackup: a restore needs a reason; it is recorded with the restore")
ErrNoReason is returned when a restore is asked for without a reason.
Functions ¶
func KeyFingerprint ¶ added in v0.0.5
KeyFingerprint names passphrase the way every audit checkpoint, escrow and sidecar sealed under it does; "" for no key.
Types ¶
type BackupOptions ¶ added in v0.0.5
type BackupOptions struct {
// Key is the CLOOP_SECRET_KEY the sidecar is sealed under; "" writes an
// unsealed sidecar, which a restore then refuses without --allow-unsealed.
Key string
}
BackupOptions configures BackupWithOptions.
type BackupReport ¶
type BackupReport struct {
Source string
Output string
MetadataPath string
SizeBytes int64
Duration time.Duration
SHA256 string
WALCheckpoint string // "PASSED" or human-readable diagnostic
SchemaVersion int
// Keys and Chain are what the manifest records.
Keys []ManifestKey
Chain *ManifestChain
// KeyFingerprint names the key the sidecar is sealed under; "" when it
// is unsealed, because there was no CLOOP_SECRET_KEY.
KeyFingerprint string
}
BackupReport summarises a successful backup run.
func Backup ¶
func Backup(srcPath, outPath string) (*BackupReport, error)
Backup creates a hot backup of the SQLite database at srcPath and writes it to outPath, sealing its sidecar under the hub's current root (pkg/rootkey). See BackupWithOptions.
func BackupWithOptions ¶ added in v0.0.5
func BackupWithOptions(srcPath, outPath string, opts BackupOptions) (*BackupReport, error)
BackupWithOptions creates a hot backup of the SQLite database at srcPath and writes it to outPath. The destination directory must exist. A sibling "<outPath>.meta.json" file is written alongside containing the SHA-256 checksum, source size, and other metadata used by Restore for validation, and the manifest: the copy's sealing keys and audit head, sealed under opts.Key.
If outPath already exists, it is overwritten — operators are expected to rotate backup filenames (timestamp, sequence, etc.) externally.
func (*BackupReport) Sealed ¶ added in v0.0.5
func (r *BackupReport) Sealed() bool
Sealed reports whether the sidecar was sealed.
type KeyVerdict ¶ added in v0.0.5
type KeyVerdict struct {
ManifestKey
// Opens: the key derives it (passes its check value; for the legacy
// key, opens a sample of what it sealed).
Opens bool `json:"opens"`
// Lost: rows sealed under it would be unreadable after the restore, and
// it is the key's fault — a restore would make them unreadable.
Lost bool `json:"lost,omitempty"`
// Why says why it does not open, when it does not.
Why string `json:"why,omitempty"`
// KMS names the KMS the key is held by (Task 20433); "" for a key
// derived from a passphrase. Read from the backup's own registry.
KMS string `json:"kms,omitempty"`
}
KeyVerdict is one sealing key the backup depends on, judged against the key a restore would run with.
type ManifestChain ¶ added in v0.0.5
type ManifestChain struct {
LastID int64 `json:"last_id"`
LastRowHash string `json:"last_row_hash,omitempty"`
LastTime string `json:"last_time,omitempty"`
FirstID int64 `json:"first_id,omitempty"`
Rows int64 `json:"rows"`
AnchorID int64 `json:"anchor_id,omitempty"`
AnchorThroughID int64 `json:"anchor_through_id,omitempty"`
AnchorHash string `json:"anchor_hash,omitempty"`
}
ManifestChain is where a backup's audit chain ends.
type ManifestKey ¶ added in v0.0.5
type ManifestKey struct {
ID string `json:"id"`
State string `json:"state"`
// Rows counts the rows sealed under it, by sealed set.
Rows map[string]int `json:"rows,omitempty"`
}
ManifestKey is one sealing key a backup depends on: in its registry, or named by its sealed rows.
func (ManifestKey) Total ¶ added in v0.0.5
func (k ManifestKey) Total() int
Total is how many rows are sealed under the key.
type Metadata ¶
type Metadata struct {
Source string `json:"source"`
Backup string `json:"backup"`
CreatedAt time.Time `json:"created_at"`
SizeBytes int64 `json:"size_bytes"`
SHA256 string `json:"sha256"`
DurationMS int64 `json:"duration_ms"`
SchemaVersion int `json:"schema_version"`
CloopVersion string `json:"cloop_version,omitempty"`
// Manifest is the manifest format, ManifestVersion.
Manifest int `json:"manifest,omitempty"`
// Keys are the sealing keys the backup's sealed rows depend on.
Keys []ManifestKey `json:"keys,omitempty"`
// Chain is where the backup's audit chain ends.
Chain *ManifestChain `json:"chain,omitempty"`
// KeyFingerprint names the CLOOP_SECRET_KEY the sidecar is sealed under,
// and Seal is the seal; both empty on an unsealed sidecar.
KeyFingerprint string `json:"key_fingerprint,omitempty"`
Seal string `json:"seal,omitempty"`
}
Metadata is persisted next to the backup file. Restore consults it to verify the backup before overwriting the target. Every field after CloopVersion is the manifest (manifest.go), absent from sidecars written before Task 20410; such a sidecar is unsealed, and a restore says so.
func LoadMetadata ¶
LoadMetadata reads the sidecar metadata file written next to a backup. Returns (nil, nil) if the file is missing; a restore then refuses unless SkipChecksum says the sidecar is not to be checked.
type Preflight ¶ added in v0.0.5
type Preflight struct {
Backup string
SizeBytes int64
// SHA256 is the backup file's digest, computed here.
SHA256 string
// Meta is the sidecar; nil when there is none.
Meta *Metadata
Sidecar SidecarState
// Seal is what the sidecar's seal says, when it was checked.
Seal SealState
// SchemaVersion is the highest migration the backup records.
SchemaVersion int
// Chain is where the backup's audit chain ends.
Chain statedb.AuditHead
// Keys are the sealing keys the backup depends on, by id.
Keys []KeyVerdict
// KeyFingerprint names the key checked against; "" when there was none.
// PreviousKeyFingerprint names the previous root also checked against,
// during a change of root.
KeyFingerprint string
PreviousKeyFingerprint string
// Refusals say why the restore is refused; none when it may go ahead.
Refusals []Refusal
// Notes are what a restore gives up, or cannot check, without being
// refused for it.
Notes []string
}
Preflight is what a backup is, checked against the key a restore would run with.
func Inspect ¶ added in v0.0.5
func Inspect(path string, opts PreflightOptions) (*Preflight, error)
Inspect checks the backup at path against opts without writing anything — not to the backup, not to the database it would replace. An error means it could not look at all; what it found is in the Preflight, and Err says whether a restore may proceed.
func (*Preflight) LostKeys ¶ added in v0.0.5
func (p *Preflight) LostKeys() []KeyVerdict
LostKeys are the keys whose material a restore gives up.
type PreflightOptions ¶ added in v0.0.5
type PreflightOptions struct {
// Key is the CLOOP_SECRET_KEY the restored hub will run with.
Key string
// Previous is the CLOOP_SECRET_KEY_PREVIOUS it will run with while a
// change of root is in progress (Task 20425): a backup taken before the
// change holds keys only the previous root derives, and a sidecar sealed
// under it. Both are as good as the current root's, because the restored
// hub opens them.
Previous string
// KMS are the key-management services the restored hub will run with
// (Task 20433): secrets.kms and, during a change of root,
// secrets.kms_previous. A backup's KMS-held keys open through one of
// them and nothing else — no key file, no escrow — so restoring a backup
// of a hub on a KMS needs access to that KMS key.
KMS []secretbroker.KMS
// SkipChecksum, AllowUnsealed and AcceptSecretLoss are RestoreOptions'.
SkipChecksum bool
AllowUnsealed bool
AcceptSecretLoss bool
}
PreflightOptions are what a backup is checked against.
type Recording ¶ added in v0.0.5
type Recording struct {
// RestoredHead is where the restored chain ended before the restore
// appended to it: the newest row the backup holds.
RestoredHead statedb.AuditHead
// Superseded is the newest checkpoint the restore superseded; nil when
// none saw history past the restored head, or none could be read.
Superseded *auditcheckpoint.Located
// Row is the audit.restored row; zero when it could not be appended.
Row statedb.AuditEvent
// Record is the restore record; Sealed whether it carries a seal, and
// Sinks where it was written ("file", "stderr").
Record auditcheckpoint.Record
Sealed bool
Sinks []string
// RetiredKeys are the sealing keys retired under --accept-secret-loss.
RetiredKeys []string
// Notes say what could not be done, and why.
Notes []string
}
Recording is how a restore was recorded.
type Refusal ¶ added in v0.0.5
type Refusal struct {
// Err is the sentinel; Message says it to an operator.
Err error
Message string
// Override is the flag that proceeds anyway; "" when none does.
Override string
}
Refusal is one reason a restore is refused.
type RefusedError ¶ added in v0.0.5
RefusedError is a restore refused before anything was touched.
func (*RefusedError) Error ¶ added in v0.0.5
func (e *RefusedError) Error() string
func (*RefusedError) Is ¶ added in v0.0.5
func (e *RefusedError) Is(target error) bool
Is matches every sentinel the refusals carry.
type RestoreOptions ¶
type RestoreOptions struct {
// Force allows overwriting an existing destination database. Without
// this flag, Restore refuses to overwrite if the destination file or
// any of its WAL/SHM sidecars exist (heuristic for "active database").
Force bool
// Key is the CLOOP_SECRET_KEY the restored hub will run with. The
// backup's sealing keys must derive from it, and its sidecar must carry
// its seal; the restore record is sealed under it. Restore never reads
// the environment itself: the caller says which key it means.
Key string
// Previous is the CLOOP_SECRET_KEY_PREVIOUS the restored hub will run
// with during a change of root (Task 20425): the backup's keys and its
// sidecar may derive from it instead, and checkpoints sealed under it are
// read as the hub's own. Nothing is sealed under it.
Previous string
// KMS are the key-management services the restored hub will run with
// (Task 20433); the backup's KMS-held keys must open through one of them.
KMS []secretbroker.KMS
// SkipChecksum restores without checking the sidecar at all — no digest,
// no seal. For a backup whose sidecar was lost; nothing then vouches that
// the backup was not edited, and the restore records that it was skipped.
SkipChecksum bool
// AllowUnsealed accepts a sidecar whose seal cannot be checked here:
// written unsealed (before Task 20410, or by a hub with no
// CLOOP_SECRET_KEY), sealed under another key, or with no Key to check
// it. The digest is still checked.
AllowUnsealed bool
// AcceptSecretLoss restores a backup whose sealing keys Key cannot
// derive, and retires those keys in the restored database so the hub can
// start and mint a fresh one; what they sealed is reported lost. The
// backup file keeps them. Recorded on the audit.restored row and the
// restore record.
AcceptSecretLoss bool
// Reason says why; it is recorded and required. Actor is who.
Reason string
Actor string
// CheckpointFile is the audit checkpoint file (audit.checkpoints.file):
// where the restore looks for the newest checkpoint it supersedes, and
// where it appends its restore record. "" reads and writes no file.
// CheckpointStderr receives the record's stderr copy; nil writes none.
CheckpointFile string
CheckpointStderr io.Writer
// Chain is the role the restore record names, auditcheckpoint.Chain*;
// "" takes the role the newest checkpoint for this database names, or
// project when there is none.
Chain string
// Now is the clock; nil means time.Now.
Now func() time.Time
}
RestoreOptions configures Restore.
type RestoreReport ¶
type RestoreReport struct {
Backup string
Destination string
SizeBytes int64
Duration time.Duration
BackedUpPath string // pre-restore copy of the destination, if any
// Preflight is what the backup was found to be before the swap.
Preflight *Preflight
// Recorded says how the restore was recorded; see Recording.
Recorded Recording
}
RestoreReport describes a successful restore.
func Restore ¶
func Restore(backupPath, dstPath string, opts RestoreOptions) (*RestoreReport, error)
Restore checks the backup at backupPath (Inspect), atomically swaps it into dstPath, and records the restore.
Nothing is touched until every check has passed: a refusal returns a *RefusedError naming each reason and the flag, if any, that overrides it. If dstPath already exists and Force is false, Restore returns an error. When Force is true, the existing destination is moved to a sibling .pre-restore backup before the swap, so a botched restore is recoverable.
Once the backup is in place the restore is recorded (record.go). A failure there returns the report with an *UnrecordedError: the database is restored, and the error says what is missing from the record of it.
type SealState ¶ added in v0.0.5
type SealState string
SealState is what a sidecar's seal says about it.
const ( // SealValid: sealed under the key checking it, and the seal verifies. SealValid SealState = "valid" // SealUnsigned: written without a key — by a hub with no // CLOOP_SECRET_KEY, or by a build that did not seal sidecars. SealUnsigned SealState = "unsigned" // SealUnchecked: sealed, and there is no key here to check it with. SealUnchecked SealState = "unchecked" // SealForeignKey: sealed under another CLOOP_SECRET_KEY than the one // checking it. SealForeignKey SealState = "foreign-key" // SealBad: claims the checking key and does not verify under it, or is // half a seal. The sidecar, or the backup with it, was edited. SealBad SealState = "bad" )
type SidecarState ¶ added in v0.0.5
type SidecarState string
SidecarState is how the sidecar was found.
const ( // SidecarSealed: present, the digest matches, and its seal verifies // under the key. SidecarSealed SidecarState = "sealed" // SidecarUnchecked: present and the digest matches, but its seal could // not be checked here (see the Refusal or note saying why). SidecarUnchecked SidecarState = "unchecked" // SidecarBroken: present, and either the digest or the seal says the // backup or the sidecar was changed. SidecarBroken SidecarState = "broken" // SidecarMissing: there is none. SidecarMissing SidecarState = "missing" // SidecarSkipped: not checked, under --skip-checksum. SidecarSkipped SidecarState = "skipped" )
type UnrecordedError ¶ added in v0.0.5
type UnrecordedError struct {
Problems []string
}
UnrecordedError is a restore that completed and could not be fully recorded: the database is the backup, and something is missing from the record of it. The message says what, and how to finish it by hand.
func (*UnrecordedError) Error ¶ added in v0.0.5
func (e *UnrecordedError) Error() string