Documentation
¶
Overview ¶
Package blobbackup implements the shared backup/restore stream format used by cloud blob store plugins (s3, gcs) that have no native point-in-time snapshot primitive of their own -- a plain length-prefixed key/value stream produced by walking the store's existing Get/Set/NewIterator interface, distinct from badger's own native Backup/Load format. Both plugins are otherwise near-identical siblings (see their database.go files), so this one shared implementation replaces what would otherwise be two copies that could silently drift apart on a framing or version change.
Index ¶
- Constants
- Variables
- func Backup(ctx context.Context, store Store, w io.Writer, maxValueLen int64, ...) error
- func IsEmpty(ctx context.Context, store Store) (bool, error)
- func ReadRecord(r io.Reader, maxValueLen int64) (key, value []byte, err error)
- func Reset(ctx context.Context, store Store, ...) error
- func Restore(ctx context.Context, store Store, r io.Reader, maxValueLen int64, ...) error
- func Validate(ctx context.Context, r io.Reader, maxValueLen int64, errPrefix string) error
- func WriteRecord(w io.Writer, key, value []byte, maxValueLen int64) error
- type ErrTerminator
- type Store
Constants ¶
const ( DefaultRestoreBatchRecords = 1000 DefaultRestoreBatchBytes = 32 << 20 )
DefaultRestoreBatchRecords/DefaultRestoreBatchBytes bound how many records Restore accumulates per write transaction. A cloud store's transaction commit applies from an in-memory pending map, so a single transaction spanning an entire large store would hold the whole restored dataset in memory before ever issuing a write; batching bounds that regardless of how large the store being restored is.
const MaxKeyLen = 64 << 10
MaxKeyLen bounds a single record's declared key length, checked before allocating a buffer of that size -- blob keys are always small (a few dozen bytes), so this is generous headroom while still far below what a corrupted or adversarial stream's raw length prefix could otherwise claim (up to 2^32-1).
const Version = 2
Version 2 added the mandatory terminatorMarker (see its own doc comment) -- a version 1 stream (this whole cloud-backup mechanism's first cut, never released) had no way to distinguish a truncated file from a complete one, so there is no migration path from it and none is needed: nothing has shipped a version 1 backup for this to stay compatible with.
Variables ¶
var Magic = [4]byte{'D', 'B', 'L', 'B'}
Magic/Version identify this backup framing. Every batch is keyed only by content, so a version byte lets a future framing change be detected up front rather than misparsed.
Functions ¶
func Backup ¶
func Backup( ctx context.Context, store Store, w io.Writer, maxValueLen int64, errPrefix string, ) error
Backup streams every key/value currently in store to w, using the store's own read-only transaction and forward iterator rather than any cloud-native mechanism (neither S3 nor GCS has one). database/lifecycle.Snapshot already holds Database.PauseCommitsContext for the full duration of this call, so this needs no consistency mechanism of its own -- for a very large cloud-backed store this does mean the existing write-pause lasts as long as the full bucket walk, not just a brief lock, an inherent tradeoff of this backend family (see DATABASE.md). maxValueLen bounds a single value's length (mirroring the calling plugin's own read-side limit); errPrefix names the calling plugin (e.g. "s3 backup") in every returned error.
func ReadRecord ¶
ReadRecord reads one record written by WriteRecord. Returns an *ErrTerminator (which wraps io.EOF) only once it reads terminatorMarker in the key-length position, and has also read the fixed-size footer that follows it; a plain end-of-file at the key-length read itself (the stream simply having no more bytes) means the backup was truncated before ever reaching its terminator, and is a real error, not a clean end. Any other read failure, including a partial record, is also a real error. Declared lengths are validated against sane bounds before allocating a buffer of that size, so a corrupted or adversarial stream can only ever produce a normal error, not an attempted multi-gigabyte allocation -- mirrors badger's Restore validating its own length-prefixed framing the same way (see validateLoadRecordSizes in the badger plugin).
func Reset ¶ added in v0.70.1
func Reset( ctx context.Context, store Store, deleteBatch func(context.Context, [][]byte) error, errPrefix string, ) error
Reset removes every key from store in bounded batches. It is used only by live restore after a complete rollback backup has been written. A provider supplies deleteBatch when it can delete the collected keys without the ordinary transaction path's per-key compensation work; nil retains that path for stores without a provider-native reset operation. Restarting the iterator after every successful batch keeps iterator state independent of the deletes and makes a retry continue from the remaining keys.
func Restore ¶
func Restore( ctx context.Context, store Store, r io.Reader, maxValueLen int64, errPrefix string, ) error
Restore replaces store's contents by loading a backup stream produced by Backup. It must only be called against a freshly created, empty store, enforced below -- matching blob.Restorer's documented contract. A restore that fails partway through (a batch commit error, a malformed record) can leave some already-committed batches in the store: a cloud store's transaction commit applies and durably commits each batch independently, so those earlier batches are not retroactively undone by a later batch's failure. A failed Restore must not be retried against the same store -- IsEmpty's precondition check below will (correctly) refuse it as no longer empty; discard the store and start over instead. Every error return below that can only occur once at least one earlier batch already committed (see partialDataWarning) says so explicitly, so an operator reading the failure doesn't have to already know this internal batching detail to realize the store can't just be retried against.
func Validate ¶ added in v0.70.1
Validate reads and verifies a complete backup stream without writing any records to a store. Restore orchestration calls this before it captures or mutates the live target, so failures after the fixed header (including a truncated record, a bad terminator checksum/count, and trailing data) are rejected while the original database is still intact.
Types ¶
type ErrTerminator ¶
ErrTerminator is returned by ReadRecord (wrapping io.EOF, so an existing errors.Is(err, io.EOF) check still recognizes it as end-of-stream) instead of a plain io.EOF whenever it reads terminatorMarker in a key-length position. Its RecordCount/Checksum are the footer's own declared values -- see terminatorMarker's doc comment for why a caller (Restore) must use errors.As to recover them and compare against what it actually read before accepting this as a genuine, uncorrupted end, rather than treating the marker's mere presence as sufficient proof on its own.
func (*ErrTerminator) Error ¶
func (e *ErrTerminator) Error() string
func (*ErrTerminator) Unwrap ¶
func (e *ErrTerminator) Unwrap() error
type Store ¶
type Store interface {
NewTransaction(readWrite bool) types.Txn
NewIterator(
txn types.Txn,
opts types.BlobIteratorOptions,
) types.BlobIterator
Get(txn types.Txn, key []byte) ([]byte, error)
Set(txn types.Txn, key, value []byte) error
Delete(txn types.Txn, key []byte) error
}
Store is the subset of a cloud blob store's own interface Backup/Restore need: a plain key-iteration walk and batched writes over the store's existing transaction/iterator machinery, since neither S3 nor GCS has a native snapshot mechanism of its own.
Source Files