ethdb

package
v0.0.0-...-140c642 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Mar 24, 2021 License: GPL-3.0 Imports: 33 Imported by: 0

README

Ethdb package hold's bouquet of objects to access DB

Words "KV" and "DB" have special meaning here:

  • KV - key-value-style API to access data: let developer manage transactions, stateful cursors.
  • DB - object-oriented-style API to access data: Get/Put/Delete/WalkOverTable/MultiPut, managing transactions internally.

So, DB abstraction fits 95% times and leads to more maintainable code - because it's looks stateless.

About "key-value-style": Modern key-value databases don't provide Get/Put/Delete methods, because it's very hard-drive-unfriendly - it pushes developers do random-disk-access which is order of magnitude slower than sequential read. To enforce sequential-reads - introduced stateful cursors/iterators - they intentionally look as file-api: open_cursor/seek/write_data_from_current_position/move_to_end/step_back/step_forward/delete_key_on_current_position/append.

Class diagram:

// This is not call graph, just show classes from low-level to high-level. 
// And show which classes satisfy which interfaces.

+-----------------------------------+   +-----------------------------------+   +-----------------------------------+ 
|  github.com/ledgerwatch/lmdb-go   |   |  github.com/torquem-ch/mdbx-go    |   | google.golang.org/grpc.ClientConn |                    
|  (app-agnostic LMDB go bindings)  |   |  (app-agnostic MDBX go bindings)  |   | (app-agnostic RPC and streaming)  |
+-----------------------------------+   +-----------------------------------+   +-----------------------------------+
                 |                                        |                                      |
                 |                                        |                                      |
                 v                                        v                                      v
+-----------------------------------+   +-----------------------------------+   +-----------------------------------+
|      ethdb/kv_lmdb.go             |   |       ethdb/kv_mdbx.go            |   |       ethdb/kv_remote.go          |                
| (tg-specific LMDB implementaion)  |   |  (tg-specific MDBX implementaion) |   |   (tg-specific remote DB access)  |              
+-----------------------------------+   +-----------------------------------+   +-----------------------------------+
                 |                                        |                                      |
                 |                                        |                                      |
                 v                                        v                                      v
            +----------------------------------------------------------------------------------------------+
            |                                       ethdb/kv_abstract.go                                   |  
            |         (Common KV interface. DB-friendly, disk-friendly, cpu-cache-friendly.                |
            |           Same app code can work with local or remote database.                              |
            |           Allows experiment with another database implementations.                           |
            |          Supports context.Context for cancelation. Any operation can return error)           |
            +----------------------------------------------------------------------------------------------+
                 |                                        |                                      |
                 |                                        |                                      |
                 v                                        v                                      v
+-----------------------------------+   +-----------------------------------+   +-----------------------------------+
|       ethdb/object_db.go          |   |          ethdb/tx_db.go           |   |    ethdb/remote/remotedbserver    |                
|     (thread-safe, stateless,      |   | (non-thread-safe, more performant |   | (grpc server, using kv_abstract,  |  
|   opens/close short transactions  |   |   than object_db, method Begin    |   |   kv_remote call this server, 1   |
|      internally when need)        |   |  DOESN'T create new TxDb object)  |   | transaction maps on 1 grpc stream |
+-----------------------------------+   +-----------------------------------+   +-----------------------------------+
                |                                          |                                     
                |                                          |                                     
                v                                          v                                     
            +-----------------------------------------------------------------------------------------------+
            |                                    ethdb/interface.go                                         |  
            |     (Common DB interfaces. ethdb.Database and ethdb.DbWithPendingMutations are widely used)   |
            +-----------------------------------------------------------------------------------------------+
                |                      
                |                      
                v                      
+--------------------------------------------------+ 
|             ethdb/mutation.go                    |                 
| (also known as "batch", recording all writes and |  
|   them flush to DB in sorted way only when call  | 
|     .Commit(), use it to avoid random-writes.    | 
|   It use and satisfy ethdb.Database in same time |
+--------------------------------------------------+ 

ethdb.AbstractKV design:

  • InMemory, ReadOnly: NewLMDB().Flags(lmdb.ReadOnly).InMem().Open()

  • MultipleDatabases, Customization: NewLMDB().Path(path).WithBucketsConfig(config).Open()

  • 1 Transaction object can be used only withing 1 goroutine.

  • Only 1 write transaction can be active at a time (other will wait).

  • Unlimited read transactions can be active concurrently (not blocked by write transaction).

  • Methods db.Update, db.View - can be used to open and close short transaction.

  • Methods Begin/Commit/Rollback - for long transaction.

  • it's safe to call .Rollback() after .Commit(), multiple rollbacks are also safe. Common transaction patter:

tx, err := db.Begin(true, ethdb.RW)
if err != nil {
    return err
}
defer tx.Rollback() // important to avoid transactions leak at panic or early return

// ... code which uses database in transaction
 
err := tx.Commit()
if err != nil {
    return err
}
  • No internal copies/allocations. It means: 1. app must copy keys/values before put to database. 2. Data after read from db - valid only during current transaction - copy it if plan use data after transaction Commit/Rollback.

  • Methods .Bucket() and .Cursor(), can’t return nil, can't return error.

  • Bucket and Cursor - are interfaces - means different classes can satisfy it: for example LmdbCursor and LmdbDupSortCursor classes satisfy it. If your are not familiar with "DupSort" concept, please read indices.md first.

  • If Cursor returns err!=nil then key SHOULD be != nil (can be []byte{} for example). Then traversal code look as:

for k, v, err := c.First(); k != nil; k, v, err = c.Next() {
    if err != nil {
        return err
    }
    // logic
}
  • Move cursor: cursor.Seek(key)

ethdb.Database design:

  • Allows pass multiple implementations
  • Allows traversal tables by db.Walk

ethdb.TxDb design:

  • holds inside 1 long-running transaction and 1 cursor per table
  • method Begin DOESN'T create new TxDb object, it means this object can be passed into other objects by pointer, and high-level app code can start/commit transactions when it needs without re-creating all objects which holds TxDb pointer.
  • This is reason why txDb.CommitAndBegin() method works: inside it creating new transaction object, pinter to TxDb stays valid.

How to dump/load table

Install all database tools: make db-tools - tools with prefix mdb_ is for lmdb, lmdbgo_ is for lmdb written in go, mdbx_ is for mdbx.

./build/bin/mdbx_dump -a /path/to/chaindata | lz4 > dump.lz4
lz4 -d < dump.lz4 | ./build/bin/mdbx_load -an /path/to/chaindata

How to get table checksum

./build/bin/mdbx_dump -s table_name /path/to/chaindata | tail -n +4 | sha256sum # tail here is for excluding header 

Header example:
VERSION=3
geometry=l268435456,c268435456,u25769803776,s268435456,g268435456
mapsize=756375552
maxreaders=120
format=bytevalue
database=TBL0001
type=btree
db_pagesize=4096
duplicates=1
dupsort=1
HEADER=END

Documentation

Overview

Package ethdb defines the interfaces for an Ethereum data store.

Index

Constants

View Source
const (
	NonExistingDBI dbutils.DBI = 999_999_999
)

Variables

View Source
var (
	ErrAttemptToDeleteNonDeprecatedBucket = errors.New("only buckets from dbutils.DeprecatedBuckets can be deleted")
	ErrUnknownBucket                      = errors.New("unknown bucket. add it to dbutils.Buckets")
)
View Source
var (
	LMDBDefaultMapSize          = 2 * datasize.TB
	LMDBDefaultMaxFreelistReuse = uint(1000) // measured in pages
)
View Source
var DefaultStorageMode = StorageMode{History: true, Receipts: true, TxIndex: true, CallTraces: false}
View Source
var DeletedValue = []byte("it is deleted value")
View Source
var EndSuffix = []byte{0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff}
View Source
var ErrKeyNotFound = errors.New("db: key not found")

ErrKeyNotFound is returned when key isn't found in the database.

View Source
var ErrUnavailableSnapshot = errors.New("unavailable snapshot")

Functions

func Bytesmask

func Bytesmask(fixedbits int) (fixedbytes int, mask byte)

func CustomCmpFunc

func CustomCmpFunc(tx *lmdb.Txn, dbi lmdb.DBI) dbutils.CmpFunc

func CustomDupCmpFunc

func CustomDupCmpFunc(tx *lmdb.Txn, dbi lmdb.DBI) dbutils.CmpFunc

func DefaultBucketConfigs

func DefaultBucketConfigs(defaultBuckets dbutils.BucketsCfg) dbutils.BucketsCfg

func ForEach

func ForEach(c Cursor, walker func(k, v []byte) (bool, error)) error

func Get

func Get(tx Tx, bucket string, key []byte) ([]byte, error)

func InspectDatabase

func InspectDatabase(db Database) error

func KeyCmpBackward

func KeyCmpBackward(key1, key2 []byte) (int, bool)

func MultiPut

func MultiPut(tx Tx, tuples ...[]byte) error

func MultiWalk

func MultiWalk(c Cursor, startkeys [][]byte, fixedbits []int, walker func(int, []byte, []byte) error) error

MultiWalk is similar to multiple Walk calls folded into one.

func NewRemote

func NewRemote() remoteOpts

func NewSnapshot2KV

func NewSnapshot2KV() snapshotOpts2

func NewSplitCursor

func NewSplitCursor(c Cursor, startkey []byte, matchBits int, part1end, part2start, part3start int) *splitCursor

func SetStorageModeIfNotExist

func SetStorageModeIfNotExist(db Database, sm StorageMode) error

func Walk

func Walk(c Cursor, startkey []byte, fixedbits int, walker func(k, v []byte) (bool, error)) error

func WarmUp

func WarmUp(tx Tx, bucket string, logEvery *time.Ticker, quit <-chan struct{}) error

Types

type Backend

type Backend interface {
	AddLocal([]byte) ([]byte, error)
	Etherbase() (common.Address, error)
	NetVersion() (uint64, error)
	Subscribe(func(*remote.SubscribeReply)) error
}

type BucketConfigsFunc

type BucketConfigsFunc func(defaultBuckets dbutils.BucketsCfg) dbutils.BucketsCfg

type BucketMigrator

type BucketMigrator interface {
	DropBucket(string) error
	CreateBucket(string) error
	ExistsBucket(string) bool
	ClearBucket(string) error
	ExistingBuckets() ([]string, error)
}

Interface used for buckets migration, don't use it in usual app code

type BucketsMigrator

type BucketsMigrator interface {
	BucketExists(bucket string) (bool, error) // makes them empty
	ClearBuckets(buckets ...string) error     // makes them empty
	DropBuckets(buckets ...string) error      // drops them, use of them after drop will panic
}

type Closer

type Closer interface {
	Close()
}

type Cursor

type Cursor interface {
	First() ([]byte, []byte, error)               // First - position at first key/data item
	Seek(seek []byte) ([]byte, []byte, error)     // Seek - position at first key greater than or equal to specified key
	SeekExact(key []byte) ([]byte, []byte, error) // SeekExact - position at first key greater than or equal to specified key
	Next() ([]byte, []byte, error)                // Next - position at next key/value (can iterate over DupSort key/values automatically)
	Prev() ([]byte, []byte, error)                // Prev - position at previous key
	Last() ([]byte, []byte, error)                // Last - position at last key and last possible value
	Current() ([]byte, []byte, error)             // Current - return key/data at current cursor position

	Put(k, v []byte) error           // Put - based on order
	Append(k []byte, v []byte) error // Append - append the given key/data pair to the end of the database. This option allows fast bulk loading when keys are already known to be in the correct order.
	Delete(k, v []byte) error        // Delete - short version of SeekExact+DeleteCurrent or SeekBothExact+DeleteCurrent

	// DeleteCurrent This function deletes the key/data pair to which the cursor refers.
	// This does not invalidate the cursor, so operations such as MDB_NEXT
	// can still be used on it.
	// Both MDB_NEXT and MDB_GET_CURRENT will return the same record after
	// this operation.
	DeleteCurrent() error

	// PutNoOverwrite(key, value []byte) error
	Reserve(k []byte, n int) ([]byte, error)

	// PutCurrent - replace the item at the current cursor position.
	PutCurrent(key, value []byte) error

	Count() (uint64, error) // Count - fast way to calculate amount of keys in bucket. It counts all keys even if Prefix was set.

	Close()
}

Cursor - class for navigating through a database CursorDupSort are inherit this class

If methods (like First/Next/Seek) return error, then returned key SHOULD not be nil (can be []byte{} for example). Then looping code will look as: c := kv.Cursor(bucketName)

for k, v, err := c.First(); k != nil; k, v, err = c.Next() {
   if err != nil {
       return err
   }
   ... logic
}

type CursorDupSort

type CursorDupSort interface {
	Cursor

	// SeekBothExact -
	// second parameter can be nil only if searched key has no duplicates, or return error
	SeekBothExact(key, value []byte) ([]byte, []byte, error)
	SeekBothRange(key, value []byte) ([]byte, []byte, error)
	FirstDup() ([]byte, error)          // FirstDup - position at first data item of current key
	NextDup() ([]byte, []byte, error)   // NextDup - position at next data item of current key
	NextNoDup() ([]byte, []byte, error) // NextNoDup - position at first data item of next key
	LastDup(k []byte) ([]byte, error)   // LastDup - position at last data item of current key

	CountDuplicates() (uint64, error)  // CountDuplicates - number of duplicates for the current key
	DeleteCurrentDuplicates() error    // DeleteCurrentDuplicates - deletes all of the data items for the current key
	AppendDup(key, value []byte) error // AppendDup - same as Append, but for sorted dup data

}

type DBCounterStats

type DBCounterStats struct {
	Put           uint64
	Get           uint64
	GetS          uint64
	GetAsOf       uint64
	Has           uint64
	Walk          uint64
	WalkAsOf      uint64
	MultiWalkAsOf uint64
	Delete        uint64
	MultiPut      uint64
}

type Database

type Database interface {
	Getter
	Putter
	Deleter
	Closer

	// MultiPut inserts or updates multiple entries.
	// Entries are passed as an array:
	// bucket0, key0, val0, bucket1, key1, val1, ...
	MultiPut(tuples ...[]byte) (uint64, error)

	// NewBatch - starts in-mem batch
	//
	// Common pattern:
	//
	// batch := db.NewBatch()
	// defer batch.Rollback()
	// ... some calculations on `batch`
	// batch.Commit()
	//
	NewBatch() DbWithPendingMutations                                         //
	Begin(ctx context.Context, flags TxFlags) (DbWithPendingMutations, error) // starts db transaction
	Last(bucket string) ([]byte, []byte, error)

	// IdealBatchSize defines the size of the data batches should ideally add in one write.
	IdealBatchSize() int

	Keys() ([][]byte, error)

	Append(bucket string, key, value []byte) error
	AppendDup(bucket string, key, value []byte) error
	Sequence(bucket string, amount uint64) (uint64, error)
}

Database wraps all database operations. All methods are safe for concurrent use.

type DbCopier

type DbCopier interface {
	NewDbWithTheSameParameters() *ObjectDatabase
}

type DbProvider

type DbProvider uint8
const (
	Remote DbProvider = iota
	Lmdb
)

type DbWithPendingMutations

type DbWithPendingMutations interface {
	Database

	// Commit - commits transaction (or flush data into underlying db object in case of `mutation`)
	//
	// Common pattern:
	//
	// tx := db.Begin()
	// defer tx.Rollback()
	// ... some calculations on `tx`
	// tx.Commit()
	//
	Commit() (uint64, error)

	// CommitAndBegin - commits and starts new transaction inside same db object.
	// useful for periodical commits implementation.
	//
	// Common pattern:
	//
	// tx := db.Begin()
	// defer tx.Rollback()
	// for {
	// 		... some calculations on `tx`
	//       tx.CommitAndBegin()
	//       // defer here - is not useful, because 'tx' object is reused and first `defer` will work perfectly
	// }
	// tx.Commit()
	//
	CommitAndBegin(ctx context.Context) error
	Rollback()
	BatchSize() int

	Reserve(bucket string, key []byte, i int) ([]byte, error)
}

DbWithPendingMutations is an extended version of the Database, where all changes are first made in memory. Later they can either be committed to the database or rolled back.

type Deleter

type Deleter interface {
	// Delete removes a single entry.
	Delete(bucket string, k, v []byte) error
}

Deleter wraps the database delete operations.

type Getter

type Getter interface {
	// Get returns the value for a given key if it's present.
	Get(bucket string, key []byte) ([]byte, error)

	// Has indicates whether a key exists in the database.
	Has(bucket string, key []byte) (bool, error)

	// Walk iterates over entries with keys greater or equal to startkey.
	// Only the keys whose first fixedbits match those of startkey are iterated over.
	// walker is called for each eligible entry.
	// If walker returns false or an error, the walk stops.
	Walk(bucket string, startkey []byte, fixedbits int, walker func(k, v []byte) (bool, error)) error
}

Getter wraps the database read operations.

type GetterPutter

type GetterPutter interface {
	Getter
	Putter
}

type HasKV

type HasKV interface {
	KV() KV
	SetKV(kv KV)
}

type HasNetInterface

type HasNetInterface interface {
	DB() Database
}

type HasStats

type HasStats interface {
	DiskSize(context.Context) (uint64, error) // db size
}

type HasTx

type HasTx interface {
	Tx() Tx
}

type KV

type KV interface {
	View(ctx context.Context, f func(tx Tx) error) error
	Update(ctx context.Context, f func(tx Tx) error) error
	Close()

	// Begin - creates transaction
	// 	tx may be discarded by .Rollback() method
	//
	// A transaction and its cursors must only be used by a single
	// 	thread (not goroutine), and a thread may only have a single transaction at a time.
	//  It happen automatically by - because this method calls runtime.LockOSThread() inside (Rollback/Commit releases it)
	//  By this reason application code can't call runtime.UnlockOSThread() - it leads to undefined behavior.
	//
	// If this `parent` is non-NULL, the new transaction
	//	will be a nested transaction, with the transaction indicated by parent
	//	as its parent. Transactions may be nested to any level. A parent
	//	transaction and its cursors may not issue any other operations than
	//	Commit and Rollback while it has active child transactions.
	Begin(ctx context.Context, flags TxFlags) (Tx, error)
	AllBuckets() dbutils.BucketsCfg

	CollectMetrics()
}

KV low-level database interface - main target is - to provide common abstraction over top of LMDB and RemoteKV.

Common pattern for short-living transactions:

 if err := db.View(ctx, func(tx ethdb.Tx) error {
    ... code which uses database in transaction
 }); err != nil {
		return err
}

Common pattern for long-living transactions:

tx, err := db.Begin(ethdb.RW)
if err != nil {
	return err
}
defer tx.Rollback()

... code which uses database in transaction

err := tx.Commit()
if err != nil {
	return err
}

func GenStateData

func GenStateData(data []KvData) (KV, error)

type KvData

type KvData struct {
	K []byte
	V []byte
}

type LmdbCursor

type LmdbCursor struct {
	// contains filtered or unexported fields
}

func (*LmdbCursor) Append

func (c *LmdbCursor) Append(k []byte, v []byte) error

Append - speedy feature of lmdb which is not part of KV interface. Cast your cursor to *LmdbCursor to use this method. Return error - if provided data will not sorted (or bucket have old records which mess with new in sorting manner).

func (*LmdbCursor) Close

func (c *LmdbCursor) Close()

func (*LmdbCursor) Count

func (c *LmdbCursor) Count() (uint64, error)

func (*LmdbCursor) Current

func (c *LmdbCursor) Current() ([]byte, []byte, error)

Current - return key/data at current cursor position

func (*LmdbCursor) Delete

func (c *LmdbCursor) Delete(k, v []byte) error

func (*LmdbCursor) DeleteCurrent

func (c *LmdbCursor) DeleteCurrent() error

DeleteCurrent This function deletes the key/data pair to which the cursor refers. This does not invalidate the cursor, so operations such as MDB_NEXT can still be used on it. Both MDB_NEXT and MDB_GET_CURRENT will return the same record after this operation.

func (*LmdbCursor) First

func (c *LmdbCursor) First() ([]byte, []byte, error)

func (*LmdbCursor) Last

func (c *LmdbCursor) Last() ([]byte, []byte, error)

func (*LmdbCursor) Next

func (c *LmdbCursor) Next() (k, v []byte, err error)

func (*LmdbCursor) Prefetch

func (c *LmdbCursor) Prefetch(v uint) Cursor

func (*LmdbCursor) Prefix

func (c *LmdbCursor) Prefix(v []byte) Cursor

func (*LmdbCursor) Prev

func (c *LmdbCursor) Prev() (k, v []byte, err error)

func (*LmdbCursor) Put

func (c *LmdbCursor) Put(key []byte, value []byte) error

func (*LmdbCursor) PutCurrent

func (c *LmdbCursor) PutCurrent(key []byte, value []byte) error

func (*LmdbCursor) PutNoOverwrite

func (c *LmdbCursor) PutNoOverwrite(key []byte, value []byte) error

func (*LmdbCursor) Reserve

func (c *LmdbCursor) Reserve(k []byte, n int) ([]byte, error)

func (*LmdbCursor) Seek

func (c *LmdbCursor) Seek(seek []byte) (k, v []byte, err error)

func (*LmdbCursor) SeekExact

func (c *LmdbCursor) SeekExact(key []byte) ([]byte, []byte, error)

type LmdbDupSortCursor

type LmdbDupSortCursor struct {
	*LmdbCursor
}

func (*LmdbDupSortCursor) Append

func (c *LmdbDupSortCursor) Append(k []byte, v []byte) error

func (*LmdbDupSortCursor) AppendDup

func (c *LmdbDupSortCursor) AppendDup(k []byte, v []byte) error

func (*LmdbDupSortCursor) CountDuplicates

func (c *LmdbDupSortCursor) CountDuplicates() (uint64, error)

Count returns the number of duplicates for the current key. See mdb_cursor_count

func (*LmdbDupSortCursor) DeleteCurrentDuplicates

func (c *LmdbDupSortCursor) DeleteCurrentDuplicates() error

DeleteCurrentDuplicates - delete all of the data items for the current key.

func (*LmdbDupSortCursor) DeleteExact

func (c *LmdbDupSortCursor) DeleteExact(k1, k2 []byte) error

DeleteExact - does delete

func (*LmdbDupSortCursor) FirstDup

func (c *LmdbDupSortCursor) FirstDup() ([]byte, error)

func (*LmdbDupSortCursor) LastDup

func (c *LmdbDupSortCursor) LastDup(k []byte) ([]byte, error)

func (*LmdbDupSortCursor) NextDup

func (c *LmdbDupSortCursor) NextDup() ([]byte, []byte, error)

NextDup - iterate only over duplicates of current key

func (*LmdbDupSortCursor) NextNoDup

func (c *LmdbDupSortCursor) NextNoDup() ([]byte, []byte, error)

NextNoDup - iterate with skipping all duplicates

func (*LmdbDupSortCursor) PrevDup

func (c *LmdbDupSortCursor) PrevDup() ([]byte, []byte, error)

func (*LmdbDupSortCursor) PrevNoDup

func (c *LmdbDupSortCursor) PrevNoDup() ([]byte, []byte, error)

func (*LmdbDupSortCursor) PutCurrent

func (c *LmdbDupSortCursor) PutCurrent(k, v []byte) error

Warning! this method doesn't check order of keys, it means you can insert key in wrong place of bucket

The key parameter must still be provided, and must match it.
If using sorted duplicates (#MDB_DUPSORT) the data item must still
sort into the same place. This is intended to be used when the
new data is the same size as the old. Otherwise it will simply
perform a delete of the old record followed by an insert.

func (*LmdbDupSortCursor) PutNoDupData

func (c *LmdbDupSortCursor) PutNoDupData(key, value []byte) error

func (*LmdbDupSortCursor) SeekBothExact

func (c *LmdbDupSortCursor) SeekBothExact(key, value []byte) ([]byte, []byte, error)

func (*LmdbDupSortCursor) SeekBothRange

func (c *LmdbDupSortCursor) SeekBothRange(key, value []byte) ([]byte, []byte, error)

type LmdbKV

type LmdbKV struct {
	// contains filtered or unexported fields
}

func (*LmdbKV) AllBuckets

func (db *LmdbKV) AllBuckets() dbutils.BucketsCfg

func (*LmdbKV) AllDBI

func (db *LmdbKV) AllDBI() map[string]dbutils.DBI

func (*LmdbKV) Begin

func (db *LmdbKV) Begin(_ context.Context, flags TxFlags) (txn Tx, err error)

func (*LmdbKV) Close

func (db *LmdbKV) Close()

Close closes db All transactions must be closed before closing the database.

func (*LmdbKV) CollectMetrics

func (db *LmdbKV) CollectMetrics()

func (*LmdbKV) DiskSize

func (db *LmdbKV) DiskSize(_ context.Context) (uint64, error)

func (*LmdbKV) Env

func (db *LmdbKV) Env() *lmdb.Env

func (*LmdbKV) NewDbWithTheSameParameters

func (db *LmdbKV) NewDbWithTheSameParameters() *ObjectDatabase

func (*LmdbKV) Update

func (db *LmdbKV) Update(ctx context.Context, f func(tx Tx) error) (err error)

func (*LmdbKV) View

func (db *LmdbKV) View(ctx context.Context, f func(tx Tx) error) (err error)

type LmdbOpts

type LmdbOpts struct {
	// contains filtered or unexported fields
}

func NewLMDB

func NewLMDB() LmdbOpts

func NewMDBX

func NewMDBX() LmdbOpts

func (LmdbOpts) Exclusive

func (opts LmdbOpts) Exclusive() LmdbOpts

func (LmdbOpts) Flags

func (opts LmdbOpts) Flags(f func(uint) uint) LmdbOpts

func (LmdbOpts) InMem

func (opts LmdbOpts) InMem() LmdbOpts

func (LmdbOpts) MapSize

func (opts LmdbOpts) MapSize(sz datasize.ByteSize) LmdbOpts

func (LmdbOpts) MaxFreelistReuse

func (opts LmdbOpts) MaxFreelistReuse(pages uint) LmdbOpts

func (LmdbOpts) MustOpen

func (opts LmdbOpts) MustOpen() KV

func (LmdbOpts) Open

func (opts LmdbOpts) Open() (kv KV, err error)

func (LmdbOpts) Path

func (opts LmdbOpts) Path(path string) LmdbOpts

func (LmdbOpts) Set

func (opts LmdbOpts) Set(opt LmdbOpts) LmdbOpts

func (LmdbOpts) WithBucketsConfig

func (opts LmdbOpts) WithBucketsConfig(f BucketConfigsFunc) LmdbOpts

type MinDatabase

type MinDatabase interface {
	Get(bucket string, key []byte) ([]byte, error)
	Put(bucket string, key, value []byte) error
	Delete(bucket string, k, v []byte) error
}

MinDatabase is a minimalistic version of the Database interface.

type MultiPutTuples

type MultiPutTuples [][]byte

Type which expecting sequence of triplets: dbi, key, value, .... It sorts entries by dbi name, then inside dbi clusters sort by keys

func (MultiPutTuples) Len

func (t MultiPutTuples) Len() int

func (MultiPutTuples) Less

func (t MultiPutTuples) Less(i, j int) bool

func (MultiPutTuples) Swap

func (t MultiPutTuples) Swap(i, j int)

type MutationItem

type MutationItem struct {
	// contains filtered or unexported fields
}

func (*MutationItem) Less

func (mi *MutationItem) Less(than btree.Item) bool

type ObjectDatabase

type ObjectDatabase struct {
	// contains filtered or unexported fields
}

ObjectDatabase - is an object-style interface of DB accessing

func MustOpen

func MustOpen(path string) *ObjectDatabase

func NewDatabaseWithFreezer

func NewDatabaseWithFreezer(db *ObjectDatabase, dir, suffix string) (*ObjectDatabase, error)

func NewMemDatabase

func NewMemDatabase() *ObjectDatabase

func NewMemoryDatabase

func NewMemoryDatabase() *ObjectDatabase

NewMemoryDatabase is just an alias to simplify rebasing (the same name as `rawdb.NewMemoryDatabase` in vanilla geth)

func NewObjectDatabase

func NewObjectDatabase(kv KV) *ObjectDatabase

NewObjectDatabase returns a AbstractDB wrapper.

func Open

func Open(path string, readOnly bool) (*ObjectDatabase, error)

Open - main method to open database. Choosing driver based on path suffix. If env TEST_DB provided - choose driver based on it. Some test using this method to open non-in-memory db

func (*ObjectDatabase) Ancients

func (db *ObjectDatabase) Ancients() (uint64, error)

[TURBO-GETH] Freezer support (not implemented yet) Ancients returns an error as we don't have a backing chain freezer.

func (*ObjectDatabase) Append

func (db *ObjectDatabase) Append(bucket string, key []byte, value []byte) error

Append appends a single entry to the end of the bucket.

func (*ObjectDatabase) AppendDup

func (db *ObjectDatabase) AppendDup(bucket string, key []byte, value []byte) error

AppendDup appends a single entry to the end of the bucket.

func (*ObjectDatabase) Begin

func (*ObjectDatabase) BucketExists

func (db *ObjectDatabase) BucketExists(name string) (bool, error)

func (*ObjectDatabase) ClearBuckets

func (db *ObjectDatabase) ClearBuckets(buckets ...string) error

func (*ObjectDatabase) Close

func (db *ObjectDatabase) Close()

func (*ObjectDatabase) Delete

func (db *ObjectDatabase) Delete(bucket string, k, v []byte) error

Delete deletes the key from the queue and database

func (*ObjectDatabase) DiskSize

func (db *ObjectDatabase) DiskSize(ctx context.Context) (uint64, error)

func (*ObjectDatabase) DropBuckets

func (db *ObjectDatabase) DropBuckets(buckets ...string) error

func (*ObjectDatabase) Get

func (db *ObjectDatabase) Get(bucket string, key []byte) ([]byte, error)

Get returns the value for a given key if it's present.

func (*ObjectDatabase) Has

func (db *ObjectDatabase) Has(bucket string, key []byte) (bool, error)

func (*ObjectDatabase) IdealBatchSize

func (db *ObjectDatabase) IdealBatchSize() int

IdealBatchSize defines the size of the data batches should ideally add in one write.

func (*ObjectDatabase) KV

func (db *ObjectDatabase) KV() KV

func (*ObjectDatabase) Keys

func (db *ObjectDatabase) Keys() ([][]byte, error)

func (*ObjectDatabase) Last

func (db *ObjectDatabase) Last(bucket string) ([]byte, []byte, error)

func (*ObjectDatabase) MemCopy

func (db *ObjectDatabase) MemCopy() *ObjectDatabase

func (*ObjectDatabase) MultiPut

func (db *ObjectDatabase) MultiPut(tuples ...[]byte) (uint64, error)

MultiPut - requirements: input must be sorted and without duplicates

func (*ObjectDatabase) NewBatch

func (db *ObjectDatabase) NewBatch() DbWithPendingMutations

func (*ObjectDatabase) Put

func (db *ObjectDatabase) Put(bucket string, key []byte, value []byte) error

Put inserts or updates a single entry.

func (*ObjectDatabase) Reserve

func (db *ObjectDatabase) Reserve(bucket string, key []byte, i int) ([]byte, error)

func (*ObjectDatabase) Sequence

func (db *ObjectDatabase) Sequence(bucket string, amount uint64) (res uint64, err error)

func (*ObjectDatabase) SetKV

func (db *ObjectDatabase) SetKV(kv KV)

func (*ObjectDatabase) TruncateAncients

func (db *ObjectDatabase) TruncateAncients(items uint64) error

TruncateAncients returns an error as we don't have a backing chain freezer.

func (*ObjectDatabase) Walk

func (db *ObjectDatabase) Walk(bucket string, startkey []byte, fixedbits int, walker func(k, v []byte) (bool, error)) error

type Putter

type Putter interface {
	// Put inserts or updates a single entry.
	Put(bucket string, key, value []byte) error
}

Putter wraps the database write operations.

type RWCounterDecorator

type RWCounterDecorator struct {
	Database
	DBCounterStats
}

func NewRWDecorator

func NewRWDecorator(db Database) *RWCounterDecorator

func (*RWCounterDecorator) Delete

func (d *RWCounterDecorator) Delete(bucket string, k, v []byte) error

func (*RWCounterDecorator) Get

func (d *RWCounterDecorator) Get(bucket string, key []byte) ([]byte, error)

func (*RWCounterDecorator) Has

func (d *RWCounterDecorator) Has(bucket string, key []byte) (bool, error)

func (*RWCounterDecorator) MultiPut

func (d *RWCounterDecorator) MultiPut(tuples ...[]byte) (uint64, error)

func (*RWCounterDecorator) NewBatch

func (*RWCounterDecorator) Put

func (d *RWCounterDecorator) Put(bucket string, key, value []byte) error

func (*RWCounterDecorator) Walk

func (d *RWCounterDecorator) Walk(bucket string, startkey []byte, fixedbits int, walker func([]byte, []byte) (bool, error)) error

type RemoteBackend

type RemoteBackend struct {
	// contains filtered or unexported fields
}

func (*RemoteBackend) AddLocal

func (back *RemoteBackend) AddLocal(signedTx []byte) ([]byte, error)

func (*RemoteBackend) Etherbase

func (back *RemoteBackend) Etherbase() (common.Address, error)

func (*RemoteBackend) NetVersion

func (back *RemoteBackend) NetVersion() (uint64, error)

func (*RemoteBackend) Subscribe

func (back *RemoteBackend) Subscribe(onNewEvent func(*remote.SubscribeReply)) error

type RemoteKV

type RemoteKV struct {
	// contains filtered or unexported fields
}

func (*RemoteKV) AllBuckets

func (db *RemoteKV) AllBuckets() dbutils.BucketsCfg

func (*RemoteKV) Begin

func (db *RemoteKV) Begin(ctx context.Context, flags TxFlags) (Tx, error)

func (*RemoteKV) Close

func (db *RemoteKV) Close()

Close All transactions must be closed before closing the database.

func (*RemoteKV) CollectMetrics

func (db *RemoteKV) CollectMetrics()

func (*RemoteKV) DiskSize

func (db *RemoteKV) DiskSize(ctx context.Context) (uint64, error)

func (*RemoteKV) Update

func (db *RemoteKV) Update(ctx context.Context, f func(tx Tx) error) (err error)

func (*RemoteKV) View

func (db *RemoteKV) View(ctx context.Context, f func(tx Tx) error) (err error)

type SnapshotKV2

type SnapshotKV2 struct {
	// contains filtered or unexported fields
}

func (*SnapshotKV2) AllBuckets

func (s *SnapshotKV2) AllBuckets() dbutils.BucketsCfg

func (*SnapshotKV2) Begin

func (s *SnapshotKV2) Begin(ctx context.Context, flags TxFlags) (Tx, error)

func (*SnapshotKV2) Close

func (s *SnapshotKV2) Close()

func (*SnapshotKV2) CollectMetrics

func (s *SnapshotKV2) CollectMetrics()

func (*SnapshotKV2) Update

func (s *SnapshotKV2) Update(ctx context.Context, f func(tx Tx) error) error

func (*SnapshotKV2) View

func (s *SnapshotKV2) View(ctx context.Context, f func(tx Tx) error) error

type StorageMode

type StorageMode struct {
	History    bool
	Receipts   bool
	TxIndex    bool
	CallTraces bool
}

func GetStorageModeFromDB

func GetStorageModeFromDB(db Database) (StorageMode, error)

func StorageModeFromString

func StorageModeFromString(flags string) (StorageMode, error)

func (StorageMode) ToString

func (m StorageMode) ToString() string

type Tx

type Tx interface {
	// Cursor - creates cursor object on top of given bucket. Type of cursor - depends on bucket configuration.
	// If bucket was created with lmdb.DupSort flag, then cursor with interface CursorDupSort created
	// Otherwise - object of interface Cursor created
	//
	// Cursor, also provides a grain of magic - it can use a declarative configuration - and automatically break
	// long keys into DupSort key/values. See docs for `bucket.go:BucketConfigItem`
	Cursor(bucket string) Cursor
	CursorDupSort(bucket string) CursorDupSort // CursorDupSort - can be used if bucket has lmdb.DupSort flag
	GetOne(bucket string, key []byte) (val []byte, err error)
	HasOne(bucket string, key []byte) (bool, error)

	Commit(ctx context.Context) error // Commit all the operations of a transaction into the database.
	Rollback()                        // Rollback - abandon all the operations of the transaction instead of saving them.

	BucketSize(name string) (uint64, error)

	Comparator(bucket string) dbutils.CmpFunc
	Cmp(bucket string, a, b []byte) int
	DCmp(bucket string, a, b []byte) int

	// Allows to create a linear sequence of unique positive integers for each table.
	// Can be called for a read transaction to retrieve the current sequence value, and the increment must be zero.
	// Sequence changes become visible outside the current write transaction after it is committed, and discarded on abort.
	// Starts from 0.
	Sequence(bucket string, amount uint64) (uint64, error)

	CHandle() unsafe.Pointer // Pointer to the underlying C transaction handle (e.g. *C.MDB_txn)
}

type TxDb

type TxDb struct {
	// contains filtered or unexported fields
}

TxDb - provides Database interface around ethdb.Tx It's not thread-safe! TxDb not usable after .Commit()/.Rollback() call, but usable after .CommitAndBegin() call you can put unlimited amount of data into this class, call IdealBatchSize is unnecessary Walk and MultiWalk methods - work outside of Tx object yet, will implement it later

func NewTxDbWithoutTransaction

func NewTxDbWithoutTransaction(db Database, flags TxFlags) *TxDb

NewTxDbWithoutTransaction creates TxDb object without opening transaction, such TxDb not usable before .Begin() call on it It allows inject TxDb object into class hierarchy, but open write transaction later

func (*TxDb) Ancients

func (m *TxDb) Ancients() (uint64, error)

[TURBO-GETH] Freezer support (not implemented yet) Ancients returns an error as we don't have a backing chain freezer.

func (*TxDb) Append

func (m *TxDb) Append(bucket string, key []byte, value []byte) error

func (*TxDb) AppendDup

func (m *TxDb) AppendDup(bucket string, key []byte, value []byte) error

func (*TxDb) BatchSize

func (m *TxDb) BatchSize() int

func (*TxDb) Begin

func (m *TxDb) Begin(ctx context.Context, flags TxFlags) (DbWithPendingMutations, error)

func (*TxDb) BucketExists

func (m *TxDb) BucketExists(name string) (bool, error)

func (*TxDb) ClearBuckets

func (m *TxDb) ClearBuckets(buckets ...string) error

func (*TxDb) Close

func (m *TxDb) Close()

func (*TxDb) Commit

func (m *TxDb) Commit() (uint64, error)

func (*TxDb) CommitAndBegin

func (m *TxDb) CommitAndBegin(ctx context.Context) error

func (*TxDb) Delete

func (m *TxDb) Delete(bucket string, k, v []byte) error

func (*TxDb) DiskSize

func (m *TxDb) DiskSize(ctx context.Context) (common.StorageSize, error)

func (*TxDb) DropBuckets

func (m *TxDb) DropBuckets(buckets ...string) error

func (*TxDb) Get

func (m *TxDb) Get(bucket string, key []byte) ([]byte, error)

func (*TxDb) Has

func (m *TxDb) Has(bucket string, key []byte) (bool, error)

func (*TxDb) IdealBatchSize

func (m *TxDb) IdealBatchSize() int

IdealBatchSize defines the size of the data batches should ideally add in one write.

func (*TxDb) KV

func (m *TxDb) KV() KV

func (*TxDb) Keys

func (m *TxDb) Keys() ([][]byte, error)

func (*TxDb) Last

func (m *TxDb) Last(bucket string) ([]byte, []byte, error)

Last can only be called from the transaction thread

func (*TxDb) MultiPut

func (m *TxDb) MultiPut(tuples ...[]byte) (uint64, error)

func (*TxDb) NewBatch

func (m *TxDb) NewBatch() DbWithPendingMutations

func (*TxDb) Put

func (m *TxDb) Put(bucket string, key []byte, value []byte) error

func (*TxDb) Reserve

func (m *TxDb) Reserve(bucket string, key []byte, i int) ([]byte, error)

func (*TxDb) Rollback

func (m *TxDb) Rollback()

func (*TxDb) RollbackAndBegin

func (m *TxDb) RollbackAndBegin(ctx context.Context) error

func (*TxDb) Sequence

func (m *TxDb) Sequence(bucket string, amount uint64) (res uint64, err error)

func (*TxDb) TruncateAncients

func (m *TxDb) TruncateAncients(items uint64) error

TruncateAncients returns an error as we don't have a backing chain freezer.

func (*TxDb) Tx

func (m *TxDb) Tx() Tx

func (*TxDb) Walk

func (m *TxDb) Walk(bucket string, startkey []byte, fixedbits int, walker func([]byte, []byte) (bool, error)) error

type TxFlags

type TxFlags uint
const (
	RW     TxFlags = 0x00 // default
	RO     TxFlags = 0x02
	Try    TxFlags = 0x04
	NoSync TxFlags = 0x08
)

Directories

Path Synopsis
internal/lmdbarch
Package lmdbarch contains some architecture detection constants.
Package lmdbarch contains some architecture detection constants.
Code generated by go generate; DO NOT EDIT.
Code generated by go generate; DO NOT EDIT.
internal command

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL