Documentation
¶
Overview ¶
Package deferred holds the bulk-load deferred-index manifest.
The parent metadata package owns the DeferredIndexManager interface that callers type-assert against. This package owns the data: what to drop, what to rebuild, and the sync_state key that records crash-recovery state.
Index ¶
Constants ¶
const SyncStateKey = "metadata_indexes_pending"
SyncStateKey is the sync_state row that marks an in-flight (or interrupted) deferred-index drop/rebuild cycle. The value is the string "true" while the rebuild is outstanding and is removed once every manifest entry is present.
const SyncStateValue = "true"
SyncStateValue is the literal sync_state value written while a drop/rebuild cycle is outstanding.
Variables ¶
var Manifest = []Index{ { Name: "idx_utxo_payment_key", Table: "utxo", Columns: []string{"payment_key"}, Notes: "API address lookup", Critical: true, }, { Name: "idx_utxo_staking_key", Table: "utxo", Columns: []string{"staking_key"}, Notes: "API stake lookup", Critical: true, }, { Name: "idx_utxo_spent_at_tx_id", Table: "utxo", Columns: []string{"spent_at_tx_id"}, Notes: "Consumer transaction lookup and rollback repair", Critical: true, }, { Name: "idx_utxo_referenced_by_tx_id", Table: "utxo", Columns: []string{"referenced_by_tx_id"}, Notes: "Reference-input lookup and rollback repair", Critical: true, }, { Name: "idx_utxo_collateral_by_tx_id", Table: "utxo", Columns: []string{"collateral_by_tx_id"}, Notes: "Collateral lookup and rollback repair", Critical: true, }, { Name: "idx_utxo_added_slot", Table: "utxo", Columns: []string{"added_slot"}, Notes: "Rollback range scan", Critical: true, }, { Name: "idx_utxo_transaction_id", Table: "utxo", Columns: []string{"transaction_id"}, Notes: "Foreign-key reverse lookup", }, { Name: "idx_utxo_deleted_staking_amount", Table: "utxo", Columns: []string{ "deleted_slot", "credential_tag", "staking_key", "amount", }, Notes: "Primary UTxO RPC search path", Critical: true, }, { Name: "idx_utxo_deleted_payment_script", Table: "utxo", Columns: []string{"deleted_slot", "payment_script", "amount"}, Notes: "Script-locked supply", Critical: true, }, { Name: "idx_transaction_block_hash", Table: "transaction", Columns: []string{"block_hash"}, Notes: "Block transaction grouping", Critical: true, }, { Name: "idx_transaction_slot", Table: "transaction", Columns: []string{"slot"}, Notes: "Rollback and transaction history ordering", Critical: true, }, { Name: "idx_asset_name_hex", Table: "asset", Columns: []string{"name_hex"}, Notes: "Asset name lookup", }, { Name: "idx_asset_policy_id", Table: "asset", Columns: []string{"policy_id"}, Notes: "Policy lookup", Critical: true, }, { Name: "idx_asset_fingerprint", Table: "asset", Columns: []string{"fingerprint"}, Notes: "Fingerprint lookup", }, { Name: "idx_asset_amount", Table: "asset", Columns: []string{"amount"}, Notes: "Amount range scan", }, { Name: "idx_datum_added_slot", Table: "datum", Columns: []string{"added_slot"}, Notes: "Datum rollback scan", }, { Name: "idx_certs_block_hash", Table: "certs", Columns: []string{"block_hash"}, Notes: "Block certificate lookup", }, { Name: "idx_certs_certificate_id", Table: "certs", Columns: []string{"certificate_id"}, Notes: "Certificate reverse lookup", }, { Name: "idx_certs_slot", Table: "certs", Columns: []string{"slot"}, Notes: "Certificate rollback scan", Critical: true, }, { Name: "idx_certs_cert_type", Table: "certs", Columns: []string{"cert_type"}, Notes: "Certificate type filter", }, { Name: "idx_redeemer_index", Table: "redeemer", Columns: []string{"index"}, Notes: "Redeemer index lookup", }, { Name: "idx_redeemer_tag", Table: "redeemer", Columns: []string{"tag"}, Notes: "Redeemer tag filter", }, { Name: "idx_key_witness_type", Table: "key_witness", Columns: []string{"type"}, Notes: "Witness type filter", }, { Name: "idx_witness_scripts_script_hash", Table: "witness_scripts", Columns: []string{"script_hash"}, Notes: "Script hash lookup", }, { Name: "idx_witness_scripts_type", Table: "witness_scripts", Columns: []string{"type"}, Notes: "Script type filter", }, }
Manifest is the canonical list of metadata-store indexes that are dropped before bulk load and rebuilt before the database is marked ready.
The list is intentionally conservative: it targets the heaviest write paths (utxo, transaction, asset, datum, witness, certs/redeemer secondary indexes) where API backfill spends the bulk of its time.
Order matters at rebuild time only as a logging convenience; SQLite builds each index in a single statement and does not benefit from re-ordering.
var Retained = []Index{ { Name: "idx_utxo_staking_deleted_amount", Table: "utxo", Columns: []string{ "credential_tag", "staking_key", "deleted_slot", "amount", }, Notes: "Per-batch live-stake SUM during API-mode backfill", }, { Name: "idx_key_witness_transaction_id", Table: "key_witness", Columns: []string{"transaction_id"}, Notes: "SetTransaction witness idempotency delete", }, { Name: "idx_witness_scripts_transaction_id", Table: "witness_scripts", Columns: []string{"transaction_id"}, Notes: "SetTransaction witness idempotency delete", }, { Name: "idx_redeemer_transaction_id", Table: "redeemer", Columns: []string{"transaction_id"}, Notes: "SetTransaction witness idempotency delete", }, { Name: "idx_plutus_data_transaction_id", Table: "plutus_data", Columns: []string{"transaction_id"}, Notes: "SetTransaction witness idempotency delete", }, { Name: "idx_address_transaction_transaction_id", Table: "address_transaction", Columns: []string{"transaction_id"}, Notes: "SetTransaction address index idempotency delete", }, { Name: "idx_certs_transaction_id", Table: "certs", Columns: []string{"transaction_id"}, Notes: "SetTransaction certificate idempotency delete subquery", }, }
Retained names the indexes deliberately excluded from Manifest because an import path's own predicates need them resident during bulk load.
Removing an entry from Manifest does not restore it on databases already on disk. A shipped binary whose manifest still held the entry drops it at the start of a bulk-load cycle and recreates it only in the full rebuild, so a database interrupted inside that window -- a multi-hour Mithril backfill, for issue #3253 -- still has the index missing. The newer binary's BuildDeferredIndexes no longer carries the entry, and the versioned migration that created it is already recorded complete, so CREATE INDEX IF NOT EXISTS never runs again: the full-table scan the exclusion exists to prevent becomes permanent.
The drop and full-rebuild paths therefore create any absent entry here before touching the manifest, which repairs such a database at the start of the next cycle -- including the cycle the operator starts by upgrading to the binary that carries the exclusion. Entries already present cost one catalog lookup each.
Functions ¶
This section is empty.
Types ¶
type Index ¶
type Index struct {
// Name is the explicit index name in the versioned schema.
Name string
// Table is the SQL table name.
Table string
// Columns is the ordered SQL column list.
Columns []string
// Notes documents why this index is safe to defer. Surfaces
// in the manifest test failure message when the
// classification is questioned.
Notes string
// Critical marks indexes that must be present before the API
// can serve traffic. Critical indexes are rebuilt first so
// that the node can accept queries while the remaining lazy
// indexes finish in the background.
//
// Criteria for Critical=true:
// - Any WHERE predicate on the index column used by a live
// API query path (blockfrost, utxorpc, ledger queries).
// - Any WHERE predicate used by the rollback path
// (DeleteXAfterSlot), since rollbacks can occur as soon
// as live sync resumes.
//
// Everything else is lazy: FK reverse-lookups,
// witness/redeemer secondary indexes, and any column that is
// only SELECTed or SET but never filtered.
Critical bool
}
Index is one entry in the deferred-index manifest. Each entry names an index that is safe to drop while the database is in bulk-load mode (Mithril sync ledger-state import, immutable blob load, API-mode historical metadata backfill) and rebuild before the database is marked ready.
The manifest deliberately excludes:
- Primary keys (autoincrement IDs).
- Unique indexes that back ON CONFLICT clauses used during import (e.g. utxo.tx_id_output_idx, transaction.hash, asset.idx_asset_unique, datum.hash, script.hash, certs.uniq_tx_cert).
- Indexes on resume-checkpoint tables (import_checkpoint.import_key, backfill_checkpoint.phase).
- The utxo (tx_id, output_idx) lookup index, required to resolve transaction inputs during backfill UTxO spending.
- Indexes the import path's own predicates filter on. Those are listed in Retained, which the drop and rebuild paths keep present rather than dropping.
- Cross-row uniqueness constraints used by ledger-state import (pool_stake_snapshot, reward_snapshot, reward_pool_input, network_state, account.staking_key, drep.credential, etc.).
Adding a new index to the versioned metadata schema requires deciding its bulk-load behavior at the same time:
- Does any import path (ledger-state import, immutable blob load, backfill block replay) rely on the index for an ON CONFLICT target, FK enforcement, constraint lookup, or the WHERE clause of a per-row idempotency delete or aggregate refresh? If yes, leave it out of the manifest. A predicate an import path runs once per transaction cannot afford a scan of a table the same import path is growing.
- Does the index only serve API/query/rollback paths that do not run during Mithril sync? If yes, add it here.
- Composite indexes share state with their constituent columns. If a field has both a deferrable single-column query index and a protected composite unique index, give the single-column index an explicit name and list that name here instead of the field.
- Taking an index back out of the manifest means adding it to Retained. Deleting the entry alone leaves it missing on every database an older binary had already dropped it from.
See deferred_test.go for manifest invariants.
func CriticalManifest ¶ added in v0.51.0
func CriticalManifest() []Index
CriticalManifest returns the subset of Manifest entries that are marked Critical=true. These are the indexes that must be present before the API can serve traffic.
Source Files