Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type BackupValidator ¶ added in v0.70.1
BackupValidator is implemented by blob store plugins that can validate a backup stream completely without writing to the target store. Live restore uses this before either store is mutated, so a truncated record, a checksum mismatch, or trailing data cannot be discovered only after metadata has already been replaced.
type Backuper ¶ added in v0.69.0
type Backuper interface {
// Backup streams a backup of the current contents of the store to w.
Backup(ctx context.Context, w io.Writer) error
}
Backuper is implemented by blob store plugins that can stream a consistent point-in-time copy of their contents. Implementations should use their native MVCC/versioned backup mechanism so this does not require blocking concurrent writers for the duration of the backup.
type BlobStore ¶
type BlobStore interface {
// Transaction management
Close() error
NewTransaction(bool) types.Txn
// KV operations (plugins map these to their internals with transaction context)
Get(txn types.Txn, key []byte) ([]byte, error)
Set(txn types.Txn, key, val []byte) error
Delete(txn types.Txn, key []byte) error
NewIterator(
txn types.Txn,
opts types.BlobIteratorOptions,
) types.BlobIterator
// Commit timestamp management
GetCommitTimestamp() (int64, error)
// SetCommitTimestamp stores the last commit timestamp; parameter order is
// (timestamp, txn) to keep the transaction as the final parameter.
SetCommitTimestamp(int64, types.Txn) error
// Block operations
SetBlock(
txn types.Txn,
slot uint64,
hash []byte,
cbor []byte,
id uint64,
blockType uint,
height uint64,
prevHash []byte,
) error
GetBlock(
txn types.Txn,
slot uint64,
hash []byte,
) ([]byte, types.BlockMetadata, error)
DeleteBlock(txn types.Txn, slot uint64, hash []byte, id uint64) error
// TombstoneBlock replaces the block's CBOR with an expired-history
// marker while leaving index pointers (bi, bh) and metadata in place.
// GetBlock and iterator ValueCopy must return types.ErrHistoryExpired
// for the block so archive wrappers can proxy it if configured.
// DeleteBlock continues to fully remove a block for orphan rollbacks
// where there is no retained history entry.
TombstoneBlock(txn types.Txn, slot uint64, hash []byte) error
GetBlockURL(
ctx context.Context,
txn types.Txn,
point ocommon.Point,
) (types.SignedURL, types.BlockMetadata, error)
// UTxO operations
SetUtxo(txn types.Txn, txId []byte, outputIdx uint32, cbor []byte) error
GetUtxo(txn types.Txn, txId []byte, outputIdx uint32) ([]byte, error)
DeleteUtxo(txn types.Txn, txId []byte, outputIdx uint32) error
// Transaction operations
SetTx(txn types.Txn, txHash []byte, offsetData []byte) error
GetTx(txn types.Txn, txHash []byte) ([]byte, error)
DeleteTx(txn types.Txn, txHash []byte) error
// DiskSize returns the on-disk size of the blob store in bytes.
// Returns 0 for cloud-backed stores where local size is not meaningful.
DiskSize() (int64, error)
// Sync flushes everything committed so far to durable storage, so it
// survives an unclean shutdown of the process or host.
//
// This exists because committing a blob transaction is not the same as
// making it durable. The combined blob+metadata commit in database.Txn
// deliberately commits blob first so the blob store can only ever be ahead
// of the metadata tip, never behind -- startup reconciliation knows how to
// trim a blob store that is ahead (cleanupOrphanedBlobs) but cannot
// reconstruct blocks the metadata tip already references. That ordering
// only holds on disk if the blob commit is durable before the metadata
// commit is, so Txn.Commit calls Sync between the two. Skipping it inverts
// the invariant on an unclean host shutdown, because the two stores flush on
// very different schedules: SQLite reaches disk at WAL checkpoints (every
// 1000 pages by default) while Badger can hold committed writes in a 128MiB
// memtable for hours at chain tip. The metadata tip then survives while the
// blocks it references are silently discarded.
//
// Implementations whose writes are already durable on commit (remote
// object stores) may return nil.
Sync() error
}
BlobStore defines the interface for a blob storage provider. All transactional methods (Get, Set, Delete, NewIterator, SetCommitTimestamp) require a non-nil types.Txn created by NewTransaction(). Passing nil will result in types.ErrNilTxn.
Important: iterators returned by `NewIterator` yield `Item()` values that must only be accessed while the transaction used to create the iterator is still active. Implementations may validate transaction state at access time (for example `ValueCopy` may fail if the transaction has been committed or rolled back). Typical usage iterates and accesses item values within the same transaction scope.
type LocalBlockReader ¶ added in v0.70.1
type LocalBlockReader interface {
GetBlockLocal(
txn types.Txn,
slot uint64,
hash []byte,
) ([]byte, types.BlockMetadata, error)
}
LocalBlockReader is an optional extension for wrappers that can bypass a remote archive fallback. Database code uses it for bounded local probes where a cache miss must remain a miss.
type ProviderDependencies ¶ added in v0.68.0
type ProviderDependencies struct {
DataDir string
RunMode string
StorageMode string
Logger *slog.Logger
PromRegistry prometheus.Registerer
}
ProviderDependencies are shared application settings injected into any storage.blob provider.
type Resettable ¶ added in v0.70.1
Resettable is implemented by remote blob stores whose configured target is independent of the local data directory passed to the provider. A live restore cannot obtain an empty target for these providers by restoring into a sibling directory, so it takes a rollback backup and then calls Reset before loading the replacement. Callers must not use Reset without first retaining a restorable copy of the current contents.
type Restorer ¶ added in v0.69.0
type Restorer interface {
// Restore replaces the store's contents with the backup read from r.
Restore(ctx context.Context, r io.Reader) error
}
Restorer is implemented by blob store plugins that can replace their contents from a backup produced by Backuper.Backup. Restore must only be called against a freshly created, empty store — it is not a merge.
type TxnBudget ¶ added in v0.71.1
type TxnBudget interface {
// RemainingTxnEntries reports how many further mutations, each costing
// entryBytes of key plus value, txn will still accept. Staging all of
// them fills the transaction, so a caller with anything left to write
// into it afterwards -- and every combined transaction has the commit
// timestamp -- keeps headroom of its own.
//
// The bool is false when the store cannot answer for this transaction
// -- a nil, finished, or foreign handle -- which callers read the same
// way as a store that does not implement the interface at all.
RemainingTxnEntries(txn types.Txn, entryBytes int) (int, bool)
}
TxnBudget is an optional extension for blob stores whose transactions hold only a bounded number of staged mutations.
badger is one: it rejects every further staged write once a transaction reaches its per-transaction entry count or byte budget, and the last write any combined transaction makes is the commit timestamp Txn.Commit puts into that same blob transaction. Staging an unbounded set therefore does not cost the caller only the tail of that set -- it costs the caller the whole commit, which on the startup rollback path leaves a node that fails identically on every start (blinklabs-io/dingo#4657). Callers that stage a set they did not size themselves ask how much room is left and stop short of it.
A store that does not implement this has no per-transaction bound for callers to respect: the cloud plugins stage mutations in memory and apply them in Commit. Callers treat such a store as unbounded.
Directories
¶
| Path | Synopsis |
|---|---|
|
internal
|
|
|
blobbackup
Package blobbackup implements the shared backup/restore stream format used by cloud blob store plugins (s3, gcs) that have no native point-in-time snapshot primitive of their own -- a plain length-prefixed key/value stream produced by walking the store's existing Get/Set/NewIterator interface, distinct from badger's own native Backup/Load format.
|
Package blobbackup implements the shared backup/restore stream format used by cloud blob store plugins (s3, gcs) that have no native point-in-time snapshot primitive of their own -- a plain length-prefixed key/value stream produced by walking the store's existing Get/Set/NewIterator interface, distinct from badger's own native Backup/Load format. |
|
committimestamp
Package committimestamp decodes a blob-stored commit timestamp shared by every blob backend (badger, S3, GCS), rejecting a value that does not actually fit the int64 the rest of the codebase carries it as, rather than each backend separately risking silent truncation or wraparound.
|
Package committimestamp decodes a blob-stored commit timestamp shared by every blob backend (badger, S3, GCS), rejecting a value that does not actually fit the int64 the rest of the codebase carries it as, rather than each backend separately risking silent truncation or wraparound. |
|
compensate
Package compensate provides a disk-spooled compensation log for cloud blob transactions.
|
Package compensate provides a disk-spooled compensation log for cloud blob transactions. |