database

package
v0.21.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jan 31, 2026 License: Apache-2.0 Imports: 27 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	BlockInitialIndex uint64 = 1
)
View Source 
const CborOffsetSize = 48

CborOffsetSize is the size in bytes of an encoded CborOffset. Layout: BlockSlot (8) + BlockHash (32) + ByteOffset (4) + ByteLength (4) = 48

Variables

View Source 
var DefaultConfig = &Config{
	BlobPlugin:     "badger",
	DataDir:        ".dingo",
	MetadataPlugin: "sqlite",
}
View Source 
var ErrDatumNotFound = errors.New("datum not found")
View Source 
var ErrNotImplemented = errors.New("not implemented")

ErrNotImplemented is returned when functionality is not yet implemented.

View Source 
var ErrUtxoNotFound = errors.New("utxo not found")

Functions

func BlockBeforeSlot added in v0.4.2 

func BlockBeforeSlot(db *Database, slotNumber uint64) (models.Block, error)

func BlockBeforeSlotTxn added in v0.4.2 

func BlockBeforeSlotTxn(txn *Txn, slotNumber uint64) (models.Block, error)

func BlockByHash added in v0.21.0 

func BlockByHash(db *Database, hash []byte) (models.Block, error)

func BlockByHashTxn added in v0.21.0 

func BlockByHashTxn(txn *Txn, hash []byte) (models.Block, error)

func BlockByPoint added in v0.4.2 

func BlockByPoint(db *Database, point ocommon.Point) (models.Block, error)

func BlockByPointTxn added in v0.4.2 

func BlockByPointTxn(txn *Txn, point ocommon.Point) (models.Block, error)

func BlockDeleteTxn added in v0.4.2 

func BlockDeleteTxn(txn *Txn, block models.Block) error

func BlocksAfterSlotTxn added in v0.4.2 

func BlocksAfterSlotTxn(txn *Txn, slotNumber uint64) ([]models.Block, error)

BlocksAfterSlotTxn returns all blocks after the specified slot; keep txn valid until results are consumed.

func BlocksRecent added in v0.4.2 

func BlocksRecent(db *Database, count int) ([]models.Block, error)

func BlocksRecentTxn added in v0.4.2 

func BlocksRecentTxn(txn *Txn, count int) ([]models.Block, error)

BlocksRecentTxn returns the N most recent blocks; keep txn valid until results are consumed.

Types

type BlockLRUCache added in v0.21.0 

type BlockLRUCache struct {
	// contains filtered or unexported fields
}

BlockLRUCache is a thread-safe LRU cache for recently accessed blocks. Blocks are keyed by (slot, hash) and evicted in LRU order when the cache exceeds maxEntries.

func NewBlockLRUCache added in v0.21.0 

func NewBlockLRUCache(maxEntries int) *BlockLRUCache

NewBlockLRUCache creates a new BlockLRUCache with the specified maximum number of entries. If maxEntries is negative, it is treated as zero (cache disabled).

func (*BlockLRUCache) Get added in v0.21.0 

func (c *BlockLRUCache) Get(slot uint64, hash [32]byte) (*CachedBlock, bool)

Get retrieves a cached block by slot and hash. Returns the block and true if found, or nil and false if not found. Accessing a block moves it to the front of the LRU list.

func (*BlockLRUCache) Put added in v0.21.0 

func (c *BlockLRUCache) Put(slot uint64, hash [32]byte, block *CachedBlock)

Put adds or updates a block in the cache. The block is moved to the front of the LRU list. If the cache exceeds maxEntries, the least recently used block is evicted.

type CacheMetrics added in v0.21.0 

type CacheMetrics struct {
	UtxoHotHits     atomic.Uint64
	UtxoHotMisses   atomic.Uint64
	TxHotHits       atomic.Uint64
	TxHotMisses     atomic.Uint64
	BlockLRUHits    atomic.Uint64
	BlockLRUMisses  atomic.Uint64
	ColdExtractions atomic.Uint64
}

CacheMetrics holds atomic counters for cache performance monitoring.

type CachedBlock added in v0.21.0 

type CachedBlock struct {
	// RawBytes contains the block's raw CBOR data.
	RawBytes []byte
	// TxIndex maps transaction hashes to their location in RawBytes.
	TxIndex map[[32]byte]Location
	// OutputIndex maps UTxO output keys to their location in RawBytes.
	OutputIndex map[OutputKey]Location
}

CachedBlock holds a block's raw CBOR data along with pre-computed indexes for fast extraction of transactions and UTxO outputs.

func (*CachedBlock) Extract added in v0.21.0 

func (cb *CachedBlock) Extract(offset, length uint32) []byte

Extract returns a slice of RawBytes from offset to offset+length. Returns nil if the range is out of bounds.

type CborCacheConfig added in v0.21.0 

type CborCacheConfig struct {
	HotUtxoEntries  int   // Number of UTxO CBOR entries in hot cache
	HotTxEntries    int   // Number of TX CBOR entries in hot cache
	HotTxMaxBytes   int64 // Memory limit for TX hot cache (0 = no limit)
	BlockLRUEntries int   // Number of blocks in LRU cache
}

CborCacheConfig holds configuration for the TieredCborCache.

type CborOffset added in v0.21.0 

type CborOffset struct {
	BlockSlot  uint64   // Slot number of the block containing the CBOR
	BlockHash  [32]byte // Hash of the block containing the CBOR
	ByteOffset uint32   // Byte offset within the block's CBOR data
	ByteLength uint32   // Length of the CBOR data in bytes
}

CborOffset represents a reference to CBOR data within a block. Instead of storing duplicate CBOR data, we store an offset reference that points to the CBOR within the block's raw data.

func DecodeCborOffset added in v0.21.0 

func DecodeCborOffset(data []byte) (*CborOffset, error)

DecodeCborOffset deserializes a 48-byte big-endian encoded slice into a CborOffset. Returns an error if the input data is not exactly 48 bytes.

func (*CborOffset) Encode added in v0.21.0 

func (c *CborOffset) Encode() []byte

Encode serializes the CborOffset to a 48-byte big-endian encoded slice. Layout:

  • bytes 0-7: BlockSlot (big-endian uint64)
  • bytes 8-39: BlockHash (32 bytes)
  • bytes 40-43: ByteOffset (big-endian uint32)
  • bytes 44-47: ByteLength (big-endian uint32)

type CommitTimestampError 

type CommitTimestampError struct {
	MetadataTimestamp int64
	BlobTimestamp     int64
}

CommitTimestampError contains the timestamps of the metadata and blob stores

func (CommitTimestampError) Error 

func (e CommitTimestampError) Error() string

Error returns the stringified error

type Config added in v0.12.1 

type Config struct {
	PromRegistry   prometheus.Registerer
	Logger         *slog.Logger
	BlobPlugin     string
	DataDir        string
	MetadataPlugin string
	MaxConnections int // Connection pool size for metadata plugin (should match DatabaseWorkers)
}

Config represents the configuration for a database instance

type Database 

type Database struct {
	// contains filtered or unexported fields
}

Database represents our data storage services

func New added in v0.4.3 

func New(
	config *Config,
) (*Database, error)

New creates a new database instance with optional persistence using the provided data directory. When config is nil, DefaultConfig is used (DataDir = ".dingo" for persistence). When config is provided but DataDir is empty, storage is in-memory only. When config.DataDir is non-empty, it specifies the persistent storage directory.

func (*Database) AddUtxos added in v0.8.0 

func (d *Database) AddUtxos(
	utxos []models.UtxoSlot,
	txn *Txn,
) error

func (*Database) ApplyPParamUpdates added in v0.4.4 

func (d *Database) ApplyPParamUpdates(
	slot, epoch uint64,
	era uint,
	currentPParams *lcommon.ProtocolParameters,
	decodeFunc func([]byte) (any, error),
	updateFunc func(lcommon.ProtocolParameters, any) (lcommon.ProtocolParameters, error),
	txn *Txn,
) error

func (*Database) Blob 

func (d *Database) Blob() blob.BlobStore

Blob returns the underling blob store instance

func (*Database) BlobTxn added in v0.4.3 

func (d *Database) BlobTxn(readWrite bool) *Txn

BlobTxn starts a new blob-only database transaction and returns a handle to it

func (*Database) BlockByIndex added in v0.4.5 

func (d *Database) BlockByIndex(
	blockIndex uint64,
	txn *Txn,
) (models.Block, error)

func (*Database) BlockCreate added in v0.4.5 

func (d *Database) BlockCreate(block models.Block, txn *Txn) error

func (*Database) Close 

func (d *Database) Close() error

Close cleans up the database connections

func (*Database) ComputeAndApplyPParamUpdates added in v0.21.0 

func (d *Database) ComputeAndApplyPParamUpdates(
	slot, epoch uint64,
	era uint,
	currentPParams lcommon.ProtocolParameters,
	decodeFunc func([]byte) (any, error),
	updateFunc func(
		lcommon.ProtocolParameters,
		any,
	) (lcommon.ProtocolParameters, error),
	txn *Txn,
) (lcommon.ProtocolParameters, error)

ComputeAndApplyPParamUpdates computes the new protocol parameters by applying pending updates for the given target epoch. The epoch parameter should be the epoch where updates take effect (currentEpoch + 1 during epoch rollover). This function takes currentPParams as a value and returns the updated parameters without mutating the input. This allows callers to capture the result in a transaction and apply it to in-memory state after the transaction commits.

func (*Database) Config added in v0.12.1 

func (d *Database) Config() *Config

Config returns the config object used for the database instance

func (*Database) DataDir added in v0.4.3 

func (d *Database) DataDir() string

DataDir returns the path to the data directory used for storage

func (*Database) DeleteBlockNoncesBeforeSlot added in v0.11.0 

func (d *Database) DeleteBlockNoncesBeforeSlot(
	slotNumber uint64,
	txn *Txn,
) error

DeleteBlockNoncesBeforeSlot removes all block_nonces older than the given slot number

func (*Database) DeleteBlockNoncesBeforeSlotWithoutCheckpoints added in v0.11.0 

func (d *Database) DeleteBlockNoncesBeforeSlotWithoutCheckpoints(
	slotNumber uint64,
	txn *Txn,
) error

DeleteBlockNoncesBeforeSlotWithoutCheckpoints removes non-checkpoint block_nonces older than the given slot number

func (*Database) GetAccount added in v0.6.0 

func (d *Database) GetAccount(
	stakeKey []byte,
	includeInactive bool,
	txn *Txn,
) (*models.Account, error)

GetAccount returns an account by staking key

func (*Database) GetActivePoolRelays added in v0.21.0 

func (d *Database) GetActivePoolRelays(
	txn *Txn,
) ([]models.PoolRegistrationRelay, error)

GetActivePoolRelays returns all relays from currently active pools. This is used for ledger peer discovery.

func (*Database) GetBlockNonce added in v0.11.0 

func (d *Database) GetBlockNonce(
	point ocommon.Point,
	txn *Txn,
) ([]byte, error)

GetBlockNonce fetches the block nonce for a given chain point

func (*Database) GetDatum added in v0.18.0 

func (d *Database) GetDatum(
	hash []byte,
	txn *Txn,
) (*models.Datum, error)

GetDatum retrieves a datum by its hash.

func (*Database) GetDrep added in v0.18.0 

func (d *Database) GetDrep(
	cred []byte,
	includeInactive bool,
	txn *Txn,
) (*models.Drep, error)

GetDrep returns a drep by credential

func (*Database) GetEpochs added in v0.6.0 

func (d *Database) GetEpochs(txn *Txn) ([]models.Epoch, error)

func (*Database) GetEpochsByEra added in v0.4.3 

func (d *Database) GetEpochsByEra(
	eraId uint,
	txn *Txn,
) ([]models.Epoch, error)

func (*Database) GetPParams added in v0.4.4 

func (d *Database) GetPParams(
	epoch uint64,
	decodeFunc func([]byte) (lcommon.ProtocolParameters, error),
	txn *Txn,
) (lcommon.ProtocolParameters, error)

func (*Database) GetPool added in v0.17.0 

func (d *Database) GetPool(
	pkh lcommon.PoolKeyHash,
	includeInactive bool,
	txn *Txn,
) (*models.Pool, error)

GetPool returns a pool by its key hash

func (*Database) GetPoolRegistrations added in v0.4.3 

func (d *Database) GetPoolRegistrations(
	poolKeyHash lcommon.PoolKeyHash,
	txn *Txn,
) ([]lcommon.PoolRegistrationCertificate, error)

GetPoolRegistrations returns a list of pool registration certificates

func (*Database) GetStakeRegistrations added in v0.4.3 

func (d *Database) GetStakeRegistrations(
	stakingKey []byte,
	txn *Txn,
) ([]lcommon.StakeRegistrationCertificate, error)

GetStakeRegistrations returns a list of stake registration certificates

func (*Database) GetTip added in v0.4.3 

func (d *Database) GetTip(txn *Txn) (ochainsync.Tip, error)

GetTip returns the current tip as represented by the protocol

func (*Database) GetTransactionByHash added in v0.21.0 

func (d *Database) GetTransactionByHash(
	hash []byte,
	txn *Txn,
) (*models.Transaction, error)

func (*Database) Logger added in v0.4.3 

func (d *Database) Logger() *slog.Logger

Logger returns the logger instance

func (*Database) Metadata 

func (d *Database) Metadata() metadata.MetadataStore

Metadata returns the underlying metadata store instance

func (*Database) MetadataTxn added in v0.4.3 

func (d *Database) MetadataTxn(readWrite bool) *Txn

MetadataTxn starts a new metadata-only database transaction and returns a handle to it

func (*Database) SetBlockNonce added in v0.11.0 

func (d *Database) SetBlockNonce(
	blockHash []byte,
	slotNumber uint64,
	nonce []byte,
	isCheckpoint bool,
	txn *Txn,
) error

func (*Database) SetDatum added in v0.9.0 

func (d *Database) SetDatum(
	rawDatum []byte,
	addedSlot uint64,
	txn *Txn,
) error

SetDatum saves the raw datum into the database by computing the hash before inserting.

func (*Database) SetEpoch added in v0.4.3 

func (d *Database) SetEpoch(
	slot, epoch uint64,
	nonce []byte,
	era, slotLength, lengthInSlots uint,
	txn *Txn,
) error

func (*Database) SetPParamUpdate added in v0.4.4 

func (d *Database) SetPParamUpdate(
	genesis, params []byte,
	slot, epoch uint64,
	txn *Txn,
) error

func (*Database) SetPParams added in v0.4.4 

func (d *Database) SetPParams(
	params []byte,
	slot, epoch uint64,
	era uint,
	txn *Txn,
) error

func (*Database) SetTip added in v0.4.3 

func (d *Database) SetTip(tip ochainsync.Tip, txn *Txn) error

SetTip saves the current tip

func (*Database) SetTransaction added in v0.18.0 

func (d *Database) SetTransaction(
	tx lcommon.Transaction,
	point ocommon.Point,
	idx uint32,
	updateEpoch uint64,
	pparamUpdates map[lcommon.Blake2b224]lcommon.ProtocolParameterUpdate,
	certDeposits map[int]uint64,
	txn *Txn,
) error

func (*Database) Transaction 

func (d *Database) Transaction(readWrite bool) *Txn

Transaction starts a new database transaction and returns a handle to it

func (*Database) UtxoByRef added in v0.4.3 

func (d *Database) UtxoByRef(
	txId []byte,
	outputIdx uint32,
	txn *Txn,
) (*models.Utxo, error)

func (*Database) UtxosByAddress added in v0.4.3 

func (d *Database) UtxosByAddress(
	addr ledger.Address,
	txn *Txn,
) ([]models.Utxo, error)

func (*Database) UtxosByAssets added in v0.21.0 

func (d *Database) UtxosByAssets(
	policyId []byte,
	assetName []byte,
	txn *Txn,
) ([]models.Utxo, error)

UtxosByAssets returns UTxOs that contain the specified assets policyId: the policy ID of the asset (required) assetName: the asset name (pass nil to match all assets under the policy, or empty []byte{} to match assets with empty names)

func (*Database) UtxosDeleteConsumed added in v0.4.4 

func (d *Database) UtxosDeleteConsumed(
	slot uint64,
	limit int,
	txn *Txn,
) (int, error)

func (*Database) UtxosDeleteRolledback added in v0.4.3 

func (d *Database) UtxosDeleteRolledback(
	slot uint64,
	txn *Txn,
) error

func (*Database) UtxosUnspend added in v0.4.3 

func (d *Database) UtxosUnspend(
	slot uint64,
	txn *Txn,
) error

type HotCache added in v0.21.0 

type HotCache struct {
	// contains filtered or unexported fields
}

HotCache provides a lock-free cache for frequently accessed CBOR data. It uses copy-on-write semantics for thread-safe concurrent access without locks. Eviction follows a Least-Frequently-Used (LFU) policy with probabilistic counting.

func NewHotCache added in v0.21.0 

func NewHotCache(maxSize int, maxBytes int64) *HotCache

NewHotCache creates a new HotCache with the given size and memory limits. Set maxSize to 0 for unlimited entries (limited only by maxBytes). Set maxBytes to 0 for unlimited memory (limited only by maxSize).

func (*HotCache) Get added in v0.21.0 

func (c *HotCache) Get(key []byte) ([]byte, bool)

Get retrieves a value from the cache by key. Returns the value and true if found, nil and false otherwise. This operation is lock-free and safe for concurrent use. Access counts are updated probabilistically (1 in accessSampleRate calls) to reduce overhead while maintaining approximate LFU behavior.

func (*HotCache) Put added in v0.21.0 

func (c *HotCache) Put(key []byte, cbor []byte)

Put adds or updates a value in the cache. If maxBytes > 0 and the entry size exceeds maxBytes/10, the entry is skipped. This operation uses copy-on-write semantics for thread safety.

type Location added in v0.21.0 

type Location struct {
	Offset uint32
	Length uint32
}

Location represents a byte range within the block's raw CBOR data.

type OutputKey added in v0.21.0 

type OutputKey struct {
	TxIndex     uint16
	OutputIndex uint16
}

OutputKey uniquely identifies a transaction output within a block.

type PartialCommitError added in v0.21.0 

type PartialCommitError struct {
	MetadataErr     error // The underlying metadata commit error
	CommitTimestamp int64 // Timestamp written to the blob store
}

PartialCommitError is returned when blob commits but metadata fails. This indicates the database is in an inconsistent state requiring recovery.

func (PartialCommitError) Error added in v0.21.0 

func (e PartialCommitError) Error() string

func (PartialCommitError) Is added in v0.21.0 

func (e PartialCommitError) Is(target error) bool

Is allows errors.Is(err, types.ErrPartialCommit) to match this error.

func (PartialCommitError) Unwrap added in v0.21.0 

func (e PartialCommitError) Unwrap() error

type PositionReader added in v0.21.0 

type PositionReader struct {
	// contains filtered or unexported fields
}

PositionReader wraps an io.Reader and tracks the current byte position. This is useful for tracking offsets during CBOR parsing to compute positions of transactions and UTxOs within block data.

func NewPositionReader added in v0.21.0 

func NewPositionReader(r io.Reader) *PositionReader

NewPositionReader creates a new PositionReader wrapping the given reader. The initial position is 0.

func (*PositionReader) Position added in v0.21.0 

func (pr *PositionReader) Position() int64

Position returns the current byte position in the reader.

func (*PositionReader) Read added in v0.21.0 

func (pr *PositionReader) Read(buf []byte) (int, error)

Read reads data from the underlying reader and updates the position. It implements the io.Reader interface.

type TieredCborCache added in v0.21.0 

type TieredCborCache struct {
	// contains filtered or unexported fields
}

TieredCborCache orchestrates the tiered cache system for CBOR data resolution. It checks hot caches first, then falls back to block extraction.

Cache tiers:

  • Tier 1: Hot caches (hotUtxo for UTxO CBOR, hotTx for transaction CBOR)
  • Tier 2: Block LRU cache (shared block cache with pre-computed indexes)
  • Tier 3: Cold extraction from blob store (not yet wired up)

func NewTieredCborCache added in v0.21.0 

func NewTieredCborCache(config CborCacheConfig) *TieredCborCache

NewTieredCborCache creates a new TieredCborCache with the given configuration.

func (*TieredCborCache) Metrics added in v0.21.0 

func (c *TieredCborCache) Metrics() *CacheMetrics

Metrics returns the cache metrics for monitoring and observability.

func (*TieredCborCache) ResolveTxCbor added in v0.21.0 

func (c *TieredCborCache) ResolveTxCbor(txHash []byte) ([]byte, error)

ResolveTxCbor resolves transaction CBOR data by transaction hash. It first checks the hot TX cache. On cache miss, it returns ErrNotImplemented since the cold path (fetching offsets and extracting from blocks) is not yet wired up.

func (*TieredCborCache) ResolveUtxoCbor added in v0.21.0 

func (c *TieredCborCache) ResolveUtxoCbor(
	txId []byte,
	outputIdx uint32,
) ([]byte, error)

ResolveUtxoCbor resolves UTxO CBOR data by transaction ID and output index. It first checks the hot UTxO cache. On cache miss, it returns ErrNotImplemented since the cold path (fetching offsets and extracting from blocks) is not yet wired up.

func (*TieredCborCache) ResolveUtxoCborBatch added in v0.21.0 

func (c *TieredCborCache) ResolveUtxoCborBatch(
	refs []UtxoRef,
) (map[UtxoRef][]byte, error)

ResolveUtxoCborBatch resolves multiple UTxO CBOR entries in a single batch. This is a stub that returns ErrNotImplemented. The full implementation will group requests by block to minimize blob store fetches.

type Txn 

type Txn struct {
	// contains filtered or unexported fields
}

Txn is a wrapper that coordinates both metadata and blob transactions. Metadata and blob are first-class siblings, not nested.

func NewBlobOnlyTxn added in v0.4.3 

func NewBlobOnlyTxn(db *Database, readWrite bool) *Txn

func NewMetadataOnlyTxn added in v0.4.3 

func NewMetadataOnlyTxn(db *Database, readWrite bool) *Txn

func NewTxn 

func NewTxn(db *Database, readWrite bool) *Txn

func (*Txn) Blob 

func (t *Txn) Blob() types.Txn

Blob returns the blob transaction handle

func (*Txn) Commit 

func (t *Txn) Commit() error

func (*Txn) DB added in v0.4.2 

func (t *Txn) DB() *Database

func (*Txn) Do 

func (t *Txn) Do(fn func(*Txn) error) error

Do executes the specified function in the context of the transaction. Any errors returned will result in the transaction being rolled back. If the function panics, the transaction is rolled back and the panic is re-raised after logging.

func (*Txn) Metadata 

func (t *Txn) Metadata() types.Txn

Metadata returns the underlying metadata transaction handle

func (*Txn) Release added in v0.21.0 

func (t *Txn) Release()

Release releases transaction resources. For read-only transactions, this releases locks and resources. For read-write transactions, this is equivalent to Rollback. Use this in defer statements for clean resource cleanup. Errors are logged but not returned, making this safe for deferred calls.

func (*Txn) Rollback 

func (t *Txn) Rollback() error

type UtxoRef added in v0.21.0 

type UtxoRef struct {
	TxId      [32]byte
	OutputIdx uint32
}

UtxoRef represents a reference to a UTxO by transaction ID and output index.

Source Files

View all Source files

Directories

Path Synopsis
blob
blob/aws
blob/badger
blob/gcs
metadata
metadata/mysql
metadata/postgres
metadata/sqlite

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.