Documentation
¶
Overview ¶
Package checkpoint holds the on-disk format primitives for engine recovery checkpoints (ADR-0133). A checkpoint is a consistent Pebble snapshot of the state store at a known applied log position, from which recovery replays only the WAL suffix instead of the whole log from genesis.
This package currently provides only the Manifest — the small, engine-owned, versioned descriptor that makes a checkpoint interpretable and verifiable — with a deterministic binary codec and validation. Creating and publishing checkpoints, restoring from them, and compacting WAL segments are later slices of ADR-0133; none of that lives here yet, and no WAL segment is deleted by anything in this package.
Index ¶
- Constants
- Variables
- func ChecksumDir(dir string) (uint64, error)
- func Dir(dataDir string) string
- func DirName(appliedPos uint64) string
- func List(root string) ([]uint64, error)
- func Prune(root string, keep int) error
- func Publish(root string, m *Manifest, snapshot func(dir string) error) (string, error)
- type DeploymentRef
- type Manifest
- type Staged
Constants ¶
const DirBase = "checkpoints"
DirBase is the checkpoint root's name inside a data directory. It is exported for the whole-instance snapshot, which names archive entries relative to the data dir and so needs the name rather than the path (ADR-0109/0131).
const FormatVersion uint32 = 1
FormatVersion is the manifest/checkpoint on-disk format version. Recovery ignores a checkpoint whose version it does not understand and falls back to an older checkpoint or genesis replay (ADR-0133): pre-1.0 there is no in-place migration.
const ManifestName = "manifest"
ManifestName is the manifest file's name inside a checkpoint directory.
Variables ¶
var ( ErrBadMagic = errors.New("checkpoint: bad manifest magic") ErrTruncated = errors.New("checkpoint: truncated manifest") ErrChecksum = errors.New("checkpoint: manifest checksum mismatch") ErrUnsupportedVersion = errors.New("checkpoint: unsupported manifest format version") ErrInconsistent = errors.New("checkpoint: inconsistent manifest fields") ErrTooLong = errors.New("checkpoint: manifest field too long to encode") )
Errors returned by Unmarshal and Validate. They are sentinels so callers (recovery, tests) can branch on the failure mode — a corrupt manifest is skipped, not fatal.
var ErrStateChecksum = errors.New("checkpoint: state checksum mismatch")
ErrStateChecksum reports that a published checkpoint's state files no longer match the checksum recorded in its manifest — a corrupt or truncated snapshot. Like the manifest decode errors it is a signal to skip this checkpoint and fall back, not to fail startup.
Functions ¶
func ChecksumDir ¶
ChecksumDir hashes a checkpoint directory's state files — every regular file except the manifest, in sorted relative-path order, mixing each path in with its content so a rename is caught as well as an edit. It is computed before the manifest is written and re-computed by Verify, so both see exactly the same set.
func Dir ¶
Dir is the checkpoint root inside an Atlas data directory, alongside the WAL and the state store. The server's checkpoint cadence and the recovery that reads what it publishes both resolve the path through this one function, so a checkpoint can never be written somewhere recovery does not look (ADR-0131).
func List ¶
List returns the applied positions of the checkpoints published under root, in ascending order — so the last element is the newest. Temporary directories from a crashed publish, stray files, and unrecognised names are ignored, which is what makes an interrupted publish invisible. A missing root is not an error: it just has no checkpoints.
func Prune ¶
Prune deletes all but the newest keep published checkpoints under root, bounding the disk a rotating checkpoint schedule uses. keep is clamped to at least one, so Prune never removes the only recovery source. Temporary directories left by a crashed publish are cleaned up too.
func Publish ¶
Publish stages a checkpoint and commits it in one call (ADR-0131).
Both halves run wherever the caller is, so this is for callers that have no writer to keep free — tests, and synchronous embedding. **The server does not use it**: it stages on the run loop and commits off it, because Staged.Commit reads the whole state store and would otherwise do so with the engine stopped.
Publishing at a position that is already published is a no-op, so a retry after a crash between the rename and the caller's bookkeeping is safe.
m.StateChecksum is filled in by Publish; the caller sets the rest.
Types ¶
type DeploymentRef ¶
DeploymentRef names a deployed definition (its key and version) that a checkpoint's replayed events assume is registered. Deployments are durable and reloaded independently before replay (ADR-0019); recording them lets recovery detect a checkpoint taken against a deployment set that no longer resolves.
type Manifest ¶
type Manifest struct {
// Partition is the partition this checkpoint belongs to; recovery ignores a
// checkpoint for a different partition.
Partition uint16
// AppliedPosition is the highest log position folded into the checkpoint's state.
// Recovery replays the WAL strictly after it. The load-bearing field.
AppliedPosition uint64
// HighestPosition is the highest log position seen when the checkpoint was taken
// (>= AppliedPosition); it restores the processor's position without scanning the
// pruned prefix.
HighestPosition uint64
// KeyCounter is the partition key-generator counter at the checkpoint, restored so
// live keys never collide with replayed ones without scanning the prefix.
KeyCounter uint64
// StateChecksum is a checksum over the checkpoint's state-store files, verified at
// restore so a torn or corrupt snapshot is rejected. It is metadata to this codec
// (computed by the creation path, a later slice); the codec only round-trips it.
StateChecksum uint64
// CreatedUnixNano is the wall-clock creation time (diagnostics only).
CreatedUnixNano int64
// AtlasVersion is the Atlas build that wrote the checkpoint (diagnostics only).
AtlasVersion string
// Deployments are the (key, version) references the replayed events assume exist.
Deployments []DeploymentRef
}
Manifest is the engine-owned descriptor of a recovery checkpoint (ADR-0133). It is self-describing and self-verifying: encoded with a magic prefix, the format version, the fields below, and a trailing checksum over the whole body.
func Load ¶
Load reads and validates the manifest of the checkpoint published at pos. A malformed, corrupt, or wrong-version manifest returns the codec's sentinel error so the caller can skip this checkpoint and fall back (ADR-0131).
func Unmarshal ¶
Unmarshal decodes a manifest and verifies its magic, format version, and trailing checksum. A malformed, wrong-version, or corrupt manifest returns a sentinel error (ErrBadMagic, ErrTruncated, ErrUnsupportedVersion, ErrChecksum) so recovery can skip the checkpoint and fall back rather than fail (ADR-0133).
func Verify ¶
Verify loads the manifest at pos and confirms the checkpoint's state files still hash to the checksum recorded when it was published, so a corrupt or truncated snapshot is rejected (ErrStateChecksum) rather than restored.
func (*Manifest) Marshal ¶
Marshal encodes the manifest into its deterministic on-disk form: magic, format version, the fields, and a trailing CRC-32 over everything preceding it. It errors only if a field cannot be represented (an over-long string or deployment list).
func (*Manifest) Validate ¶
Validate checks the semantic invariants a usable checkpoint manifest must hold, independent of encoding. Recovery calls it after Unmarshal; a failure means skip the checkpoint, not crash.
Encodability is deliberately not checked here — that is Marshal's job, and a decoded manifest cannot violate it anyway (the version string's length prefix is one byte).
type Staged ¶ added in v0.7.0
type Staged struct {
// contains filtered or unexported fields
}
Staged is a checkpoint whose snapshot has been taken but which is not yet published. It exists to split the publication into its two very different halves (ADR-0131).
Taking the snapshot needs the writer stopped: only between batches do the store's applied position and the state it holds agree exactly, which is what makes the recorded position describe the snapshotted state (invariant I3). Everything after it does not. Once Stage returns, the staged directory is a set of hard links to immutable SST files under a `tmp-` name nothing else looks at, so checksumming it, writing the manifest and renaming it into place can run with the engine free.
The split is not cosmetic. The checksum reads **every byte of the state store**, so on a store of any size it is by far the longest step — running it inside the single-writer turn stops command processing, and with it every request in the API, for as long as the read takes. That is the shape Prune was already kept out of the writer's way for; this keeps the much larger reader out of it too.
func Stage ¶ added in v0.7.0
Stage snapshots the state under a temporary directory and returns the staged checkpoint for Staged.Commit to publish.
**It must run on the partition's single-writer goroutine** — that is the whole reason it is a separate call. snapshot is given a fresh, non-existent directory path that it must create and populate (state.Store.Snapshot does exactly this).
Staging at a position that already has a published checkpoint takes no snapshot at all; the returned Staged commits to the existing path.
func (*Staged) Abandon ¶ added in v0.7.0
Abandon discards a staged checkpoint without publishing it, for a caller that staged and then could not go on (a shutdown, a failed pass). It is safe to call on a Staged that found its position already published, where it does nothing.
func (*Staged) AlreadyPublished ¶ added in v0.20.0
AlreadyPublished reports that a checkpoint was already published at this position when it was staged, so Commit writes and hashes nothing. A caller that treats a commit as a fresh verification of the files on disk must not do so then.
func (*Staged) AppliedPosition ¶ added in v0.7.0
AppliedPosition is the log position this checkpoint captures.
func (*Staged) Commit ¶ added in v0.7.0
Commit checksums the staged state, records it in the manifest, and **renames** the directory to its published name, fsyncing the directory and then root.
It runs with the writer free: everything it touches was fixed by Stage. The rename is the publication point — a crash before it leaves only a `tmp-` directory, which List ignores and the next attempt clears, so a checkpoint directory is never half-published. Any failure removes the temporary directory rather than leaving a partially-built checkpoint for the next publish (or a human) to mistake for work in progress.
m.StateChecksum is filled in here; the caller sets the rest.
Source Files