Documentation
¶
Index ¶
- Variables
- func CopyFromBadgerToPebble(badgerDB *badger.DB, pebbleDB *pebble.DB, cfg MigrationConfig) error
- func CopyFromBadgerToPebbleSSTables(badgerDB *badger.DB, pebbleDB *pebble.DB, cfg MigrationConfig) error
- func ForceCompactPebbleDB(pebbleDir string) error
- func GenerateKeysShorterThanPrefix(n int) [][]byte
- func GeneratePrefixes(n int) [][]byte
- func RunMigration(badgerDir string, pebbleDir string, cfg MigrationConfig) error
- func RunMigrationAndCompaction(badgerDir string, pebbleDir string, cfg MigrationConfig) error
- type KVPair
- type KVPairs
- type MigrationConfig
- type ValidationMode
Constants ¶
This section is empty.
Variables ¶
var DefaultMigrationConfig = MigrationConfig{ BatchByteSize: 32_000_000, ReaderWorkerCount: 2, WriterWorkerCount: 2, ReaderShardPrefixBytes: 2, ValidationMode: PartialValidation, }
Functions ¶
func CopyFromBadgerToPebble ¶
func CopyFromBadgerToPebble(badgerDB *badger.DB, pebbleDB *pebble.DB, cfg MigrationConfig) error
CopyFromBadgerToPebble migrates all key-value pairs from a BadgerDB instance to a PebbleDB instance.
The migration is performed in parallel using a configurable number of reader and writer workers. Reader workers iterate over the BadgerDB by sharded key prefixes (based on ReaderShardPrefixBytes) and send key-value pairs to a shared channel. Writer workers consume from this channel and write batched entries into PebbleDB.
Configuration is provided via MigrationConfig:
- BatchByteSize: maximum size in bytes for a single Pebble write batch.
- ReaderWorkerCount: number of concurrent workers reading from Badger.
- WriterWorkerCount: number of concurrent workers writing to Pebble.
- ReaderShardPrefixBytes: number of bytes used to shard the keyspace for parallel iteration.
The function blocks until all keys are migrated and written successfully. It returns an error if any part of the process fails.
func CopyFromBadgerToPebbleSSTables ¶
func CopyFromBadgerToPebbleSSTables(badgerDB *badger.DB, pebbleDB *pebble.DB, cfg MigrationConfig) error
CopyFromBadgerToPebble copies all key-value pairs from a BadgerDB to a PebbleDB using SSTable ingestion. It reads BadgerDB in prefix-sharded ranges and writes those ranges into SSTable files, which are then ingested into Pebble.
func ForceCompactPebbleDB ¶
func GeneratePrefixes ¶
func RunMigration ¶
func RunMigration(badgerDir string, pebbleDir string, cfg MigrationConfig) error
RunMigration performs a complete migration of key-value data from a BadgerDB directory to a PebbleDB directory and verifies the integrity of the migrated data.
It executes the following steps:
- Validates that the Badger directory exists and is non-empty. Ensures that the Pebble directory does not already contain data.
- Opens both databases and runs the migration using CopyFromBadgerToPebble with the given config.
- Writes a "MIGRATION_STARTED" marker file with a timestamp in the Pebble directory.
- After migration, performs validation by: - For PartialValidation: Generates a list of prefix shards (based on 2-byte prefixes) and finds the min and max keys for each prefix group - For FullValidation: Validates all keys in the database
- Writes a "MIGRATION_COMPLETED" marker file with a timestamp to signal successful completion.
This function returns an error if any part of the process fails, including directory checks, database operations, or validation mismatches.
func RunMigrationAndCompaction ¶
func RunMigrationAndCompaction(badgerDir string, pebbleDir string, cfg MigrationConfig) error
Types ¶
type MigrationConfig ¶
type MigrationConfig struct {
PebbleDir string
BatchByteSize int // the size of each batch to write to pebble
ReaderWorkerCount int // the number of workers to read from badger
WriterWorkerCount int // the number of workers to write to the pebble
// number of prefix bytes used to assign iterator workload
// e.g, if the number is 1, it means the first byte of the key is used to divide into 256 key space,
// and each worker will be assigned to iterate all keys with the same first byte.
// Since keys are not evenly distributed, especially some table under a certain prefix byte may have
// a lot more data than others, we might choose to use 2 or 3 bytes to divide the key space, so that
// the redaer worker can concurrently iterate keys with the same prefix bytes (same table).
ReaderShardPrefixBytes int
// ValidationMode determines how thorough the validation should be
// - PartialValidation: only checks min/max keys for each prefix (faster)
// - FullValidation: checks all keys in the database (more thorough)
ValidationMode ValidationMode
ValidationOnly bool // if true, only validate the data in the badger db without copying it to pebble db
}
type ValidationMode ¶
type ValidationMode string
ValidationMode defines how thorough the validation should be
const ( // PartialValidation only checks min/max keys for each prefix PartialValidation ValidationMode = "partial" // FullValidation checks all keys in the database FullValidation ValidationMode = "full" )
func ParseValidationModeValid ¶
func ParseValidationModeValid(mode string) (ValidationMode, error)
Source Files