parity

Overview ¶

Parity compare tool: checks Go ETL output against a reference database that holds the rows it is supposed to reproduce -- the pre-cutover rows written by the legacy Python indexer, or a Discovery Provider snapshot that a genesis migration was replayed from.

It runs three kinds of check, in increasing order of what they can see:

• row counts, which catch a table that did not get written at all;
• whole-table column aggregates, which catch a table that got the right number of rows with a column empty in all of them;
• a field-by-field comparison over a deterministic sample of rows, which catches values that are wrong rather than missing.

Usage:

go run ./pkg/etl/parity --db "$ETL_DB_URL" --prod-db "$PROD_DB_URL"

Check the queries against both schemas before committing to a long run:

go run ./pkg/etl/parity --db ... --prod-db ... --validate-only