lifecycle

func ParseCloudDestination ¶

func ParseCloudDestination(
	r *DestinationRegistry,
	uri string,
) (CloudDestination, error)

ParseCloudDestination resolves uri to a CloudDestination using r's registered schemes. uri's scheme must have a factory registered on r (see RegisterS3/RegisterGCS/RegisterBuiltinDestinations) or this returns an error. r may be nil, which behaves as an empty registry.

func NewDestinationRegistry ¶

func NewDestinationRegistry() *DestinationRegistry

NewDestinationRegistry returns an empty registry with no cloud destination schemes registered.

func FetchCloudManifest ¶

func FetchCloudManifest(
	ctx context.Context,
	registry *DestinationRegistry,
	snapshotURI string,
	opts ...ManifestOption,
) (m Manifest, ok bool, err error)

FetchCloudManifest resolves the CloudDestination at the given exact snapshot URI (a specific snapshot's own location — see JoinCloudURI, not a base destination) and fetches its manifest, if that destination type implements ConfigurableCloudManifestFetcher. A manifest trust key is required. ok=false (nil error) means the destination type doesn't support authenticated fetches. ok=true with a non-nil err means an actual fetch was attempted and failed — check errors.Is(err, ErrCloudSnapshotNotFound) to tell "confirmed absent" apart from a real communication failure (auth, network, timeout); only the former should ever be treated as equivalent to "doesn't exist" by a caller.

func ParseManifest ¶

func ParseManifest(data []byte, opts ...ManifestOption) (Manifest, error)

ParseManifest validates and decodes a manifest already read into memory — the same validation ReadManifest performs against a local file, but against bytes fetched however the caller obtained them (e.g. a cloud destination's ListSnapshots fetching a single remote manifest.json object without downloading the whole snapshot).

func PeekManifest ¶

func PeekManifest(
	ctx context.Context,
	registry *DestinationRegistry,
	snapshotDir string,
	opts ...ManifestOption,
) (Manifest, error)

PeekManifest resolves snapshotDir (a local path or a cloud destination URI — see Restore's doc comment) and reads its manifest, without restoring anything. Intended for a caller that needs to validate a snapshot's recorded plugins/network/storage mode (Manifest. CheckPluginMatch, or comparing StorageMode/Network directly) against a target's actual configuration before Restore ever touches targetDataDir — Restore's own validateRestoredDatabase only checks the manifest against itself, since it opens the restored copy using the manifest's own recorded plugins, not necessarily whatever the caller actually intends to run it with afterward.

For a cloud snapshotDir, this tries FetchCloudManifest first — which fetches just the one manifest.json object via CloudManifestFetcher, without downloading the (possibly very large) blob/metadata backups alongside it — before falling back to resolveManifest below, which downloads manifest.json alone. FetchCloudManifest's ok=false covers two distinct cases resolveManifest already handles correctly on its own: snapshotDir isn't a recognized cloud URI at all (a plain local path), or it is one but that destination type doesn't implement CloudManifestFetcher — either way, falling through to resolveManifest is the right move, so its own error (if any) from the failed FetchCloudManifest attempt is deliberately discarded here rather than duplicating PeekManifest's cloud-vs-local branching a second time.

func ReadManifest ¶

func ReadManifest(dir string, opts ...ManifestOption) (Manifest, error)

ReadManifest reads and validates the manifest at dir/ManifestFileName, rejecting it if the checksum doesn't match its contents or if its format version is newer than this build understands.

func Restore ¶

func Restore(
	ctx context.Context,
	host *plugin.Host,
	registry *DestinationRegistry,
	snapshotDir string,
	targetDataDir string,
	storageConfig RestoreStorageConfig,
	opts ...ManifestOption,
) (Manifest, error)

Restore populates targetDataDir (which must not already exist, or must be empty) from the snapshot at snapshotDir, then opens the result with database.New to confirm it passes the same startup consistency checks (CheckNodeSettings, checkCommitTimestamp) any other dingo startup does, and that its tip matches what the manifest recorded. The opened database is closed again before returning — the caller is responsible for (re)opening it for real use.

The actual restore work happens in a sibling staging directory, only atomically renamed into targetDataDir once fully validated — see RestoreValidated's doc comment for why: an interruption (including one the caller's process cannot catch, e.g. a plain, non-signal-handled process kill) leaves targetDataDir completely untouched rather than half-restored.

snapshotDir may instead be a cloud destination URI (s3://bucket/prefix or gcs://bucket/prefix; see DestinationRegistry) — Restore downloads it into a local temp directory first, then proceeds exactly as it would for a local snapshotDir. This is also how a snapshot created on one node can be restored onto another, since the two never need to share a filesystem. registry may be nil if snapshotDir is always a local path.

host resolves the manifest-recorded blob/metadata plugins against targetDataDir; composition code (node.go, cmd/dingo, internal/dblifecycle, bark) builds and owns it — typically a fresh, single-use host built just for this call via internal/plugins.NewHost, mirroring how internal/plugins.OpenDatabase builds its own scratch host for a temporary open elsewhere. This package never constructs one itself: a domain package under the database import boundary registering providers or owning a plugin host of its own would split provider ownership away from the application's composition root.

storageConfig propagates the caller's actual configured provider settings into the plugins this resolves — see RestoreStorageConfig's doc comment. Pass the zero value if the caller has no such configuration to propagate.

This is an offline operation: targetDataDir must not be concurrently held open by another *database.Database (e.g. a running node), since it restores the metadata store before starting it (metadata.Restorer) and the blob store immediately after starting it empty (blob.Restorer) — two-phase orchestration that a live store's own Start/Stop lifecycle cannot safely interleave with.

func RestoreValidated ¶

func RestoreValidated(
	ctx context.Context,
	host *plugin.Host,
	registry *DestinationRegistry,
	snapshotDir string,
	targetDataDir string,
	validate func(Manifest) error,
	storageConfig RestoreStorageConfig,
	opts ...ManifestOption,
) (Manifest, error)

RestoreValidated is Restore, but — when validate is non-nil — calls validate(manifest) immediately after resolving the snapshot's manifest and before targetDataDir is touched in any way (not even the empty/absent check), returning validate's error without doing anything destructive if it fails.

This is the hook an offline caller (see internal/dblifecycle.Service. Restore) uses to run Manifest.CheckCompatibility against its own configured plugins/network/storage mode before committing to a restore, without paying for a second cloud download to re-resolve the manifest it already checked: calling PeekManifest and then Restore separately would download a cloud snapshotDir twice.

Interruption safety: every actual restore step (metadata restore, blob restore, validateRestoredDatabase) runs against a sibling staging directory with a unique .dingo-restore-staging- name, never targetDataDir itself. Only once all of them succeed is the staging directory atomically renamed into targetDataDir's place. A caller whose context is cancelled mid-restore, or whose process is killed outright (a plain SIGKILL, or any termination path that skips Go's deferred cleanup — notably, the offline `dingo database restore` CLI's default process termination does not install a signal-aware context, so an operator's Ctrl+C takes this path, not graceful cancellation), is left with targetDataDir exactly as it was before the call: absent, or the same empty directory opened and confirmed. The half-restored staging and payload directories are orphaned siblings. The next restore into the same parent removes them through retained directory handles before it starts: on Unix at once, since the killed process no longer holds them (see CleanStaleRestoreWorkDirs), and on Windows after the age cutoff.

Crash durability (power loss, not just an interrupted process that leaves the OS itself running): atomic rename alone only guarantees targetDataDir never shows a partially-visible directory -- it says nothing about whether the renamed files, or the rename itself, survive a power loss. Before activating the rename, every directory in the staged tree is fsynced (syncDirTree); after the rename, the parent directory is fsynced too, since the rename changed its own entries. If either sync fails, this returns an error without pretending the restore completed durably; a failed pre-rename sync leaves targetDataDir untouched exactly like any other failure above, and a failed post-rename sync is still surfaced even though targetDataDir was already activated, since durability could not be confirmed.

func Snapshot ¶

func Snapshot(
	ctx context.Context,
	db *database.Database,
	dir string,
	trigger string,
	dingoVersion string,
	blobPluginName string,
	metadataPluginName string,
	opts ...ManifestOption,
) (m Manifest, err error)

Snapshot captures a point-in-time backup of db's blob and metadata stores into dir, which must not already exist, writing a manifest alongside them. Both stores are backed up via their native MVCC/versioned mechanism (blob.Backuper, metadata.Backuper), each independently consistent as of whenever it runs — but the two backup calls run concurrently (not sequentially: see the comment further down on why neither exposes a way to separate "capture a consistent point" from "stream/copy it"), so without synchronization a commit landing during either one's own window would write its commit timestamp to one store's backup and not the other's, and the restored copy would fail its cross-store consistency check. Snapshot closes that window with Database.PauseCommitsContext, which blocks new read-write Txns from being constructed (not reads, and not a quiesce — nothing is torn down or disconnected) while both backups capture their stores. A successful snapshot holds the barrier through the slower backup. An aborted snapshot releases it as soon as cancellation or a backup failure makes the result unusable, then waits for both providers before removing partial files. This is safe to call against a database a live node is actively writing to.

The hold is observed in the dingo_snapshot_commit_pause_seconds histogram and can be bounded with WithMaxCommitPause.

dingoVersion is recorded in the manifest for cross-version restore detection; pass the running binary's version string. blobPluginName and metadataPluginName are recorded in the manifest for Restore's later plugin-match check (Manifest.CheckPluginMatch) — db itself no longer knows which provider names resolved its injected Stores, so the caller (which does know, from its own plugin selection) must supply them.

func SnapshotToCloud ¶

func SnapshotToCloud(
	ctx context.Context,
	registry *DestinationRegistry,
	db *database.Database,
	dir string,
	trigger string,
	dingoVersion string,
	blobPluginName string,
	metadataPluginName string,
	cloudDest string,
	name string,
	description string,
	opts ...ManifestOption,
) (Manifest, error)

SnapshotToCloud calls Snapshot to produce the local copy at dir exactly as before, then — if name or description is non-empty — labels it (see LabelSnapshot), then — if cloudDest is non-empty — additionally uploads dir's contents to that destination (a base URI like "s3://bucket/prefix" or "gcs://bucket/prefix"; see DestinationRegistry), nested one level under this snapshot's own ID (dir's base name), mirroring the local SnapshotDir/<snapshotID> layout: the actual upload target is cloudDest + "/" + filepath.Base(dir), not cloudDest itself. This is what makes ListCloudSnapshots able to enumerate multiple snapshots stored at the same configured cloudDest — a flat, unnested upload would silently overwrite every previous snapshot's files with the newest one's. The local copy is always kept; cloudDest is a mirror, not a replacement.

Labeling happens before mirroring, not after: MirrorToCloud uploads whatever is on disk at dir the moment it runs and then writes the cloud-mirrored marker recording that destination as fully done. A caller that labeled the local manifest only after this returned would leave the already-uploaded remote copy permanently without the name/description — and since the marker already says this destination is mirrored, nothing would ever retry the upload to pick up the label. Labeling first means the directory MirrorToCloud uploads already carries it.

cloudDest == "" skips the upload — existing local-only callers are unaffected, and registry may be nil in that case. A recognized cloud destination requires WithManifestKey; the request is rejected before the local snapshot is written when the key is absent.

If the upload fails, the local snapshot is still valid and left in place, but this still returns an error: the operator asked for both copies, so a cloud-only failure is a real (partial) failure, not a silent degrade to local-only.

func WithManifestID ¶ added in v0.87.0

func WithManifestID(id string) ManifestOption

WithManifestID sets the snapshot ID (local directory name or cloud object prefix segment) that a keyed verification requires the manifest to carry. ReadManifest defaults it to the base name of the directory it reads, so this is needed only where the manifest is read from a location whose name is not the snapshot ID, such as a temporary download directory or a cached catalog row. An empty id is the same as passing no option.

func WithManifestKey ¶ added in v0.81.0

func WithManifestKey(key []byte) ManifestOption

WithManifestKey sets the operator trust root: a shared secret that WriteManifest uses to authenticate a manifest, and that ParseManifest, ReadManifest and Manifest.Authenticate require a manifest to verify against. A manifest carries the digests of its payload files, so authenticating it authenticates them. An empty key is the same as passing no option.

func WithManifestMaxBytes ¶ added in v0.70.11

func WithManifestMaxBytes(maxBytes int64) ManifestOption

WithManifestMaxBytes sets the maximum encoded manifest size. Zero uses MaxManifestBytes (1 MiB); negative values are rejected before I/O.

func WithMaxCommitPause ¶ added in v0.79.1

func WithMaxCommitPause(limit time.Duration) ManifestOption

WithMaxCommitPause bounds how long Snapshot may hold the commit barrier once it has acquired it, including snapshot-state reads. When work remains at the limit, backups are cancelled and the barrier is released. Snapshot waits for both providers to stop before removing partial files and returning ErrCommitPauseExceeded, so its total call duration can exceed the barrier limit. Time spent waiting to acquire the barrier is not counted: that wait is bounded by the caller's context. Zero means no limit; negative values are rejected before any I/O.

func GetPendingTruncate ¶

func GetPendingTruncate(db *database.Database) (*PendingTruncate, error)

GetPendingTruncate reports a previously-started truncate that still needs completion. Its durable metadata marker prevents a partially committed blob deletion from going unnoticed on restart.

func ListCloudSnapshots ¶

func ListCloudSnapshots(
	ctx context.Context,
	registry *DestinationRegistry,
	cloudDest string,
	opts ...ManifestOption,
) (entries []SnapshotEntry, ok bool, err error)

ListCloudSnapshots lists the snapshots already stored at the base cloud destination URI cloudDest, if its scheme's implementation supports listing (SnapshotLister). ok reports whether listing was actually attempted: false (with a nil error) means cloudDest is empty or its destination type doesn't implement SnapshotLister, which callers like ListAvailableSnapshots should treat as "nothing to add," not a failure — cloud listing is an optional capability, not every CloudDestination implementation provides it. Invalid or unauthenticated entries are omitted; valid entries may be returned alongside an error describing omitted entries or an incomplete listing.

func ListSnapshots ¶

func ListSnapshots(baseDir string, opts ...ManifestOption) ([]SnapshotEntry, error)

ListSnapshots scans baseDir's immediate subdirectories for a valid manifest.json, returning one SnapshotEntry per readable snapshot, newest first (by Manifest.CreatedAt). This covers both manually and automatically (epoch-boundary) triggered snapshots, since both are written as ordinary subdirectories of the same configured snapshot directory — there is no separate catalog store.

A subdirectory that exists but has no manifest.json at all (a snapshot still being written) is silently skipped rather than treated as a hard error, since that is an expected transient state, not corruption of the catalog itself. baseDir not existing yet (no snapshot has ever been taken) returns an empty result, not an error.

Any other ReadManifest failure for a given subdirectory — a malformed manifest, a checksum mismatch (ErrManifestCorrupted), a permission error — is a real problem, not the expected in-progress case, and is not silently swallowed the same way: that subdirectory is still left out of the returned entries (one broken snapshot must not hide every other, otherwise-valid one from the catalog), but its error is accumulated and returned via errors.Join alongside the entries found, so a caller can log or surface it instead of the catalog silently looking one snapshot smaller than it should.

func ListSnapshotsContext ¶ added in v0.82.0

func ListSnapshotsContext(
	ctx context.Context,
	baseDir string,
	opts ...ManifestOption,
) ([]SnapshotEntry, error)

ListSnapshotsContext scans the local snapshot directory using ctx.