Documentation
¶
Overview ¶
Command gen-dataset writes a synthetic dataset big enough to be worth measuring.
Both committed fixtures total seventy-three rows. Everything built for size — the engine's memory limit and query timeout, DuckDB's spill to temp_directory, the profiler's fan-out across columns, the anti-joins in checks/relate.go, the report — has therefore never run against data that would reach it. This writes a dataset that does, with defects planted at known counts and a manifest describing them, so `veritix eval` scores the same run that is being timed: a scale test that only measures seconds cannot tell a fast auditor from one that quietly stopped looking.
It is a generator rather than a committed fixture because a two-gigabyte fixture is not reviewable and does not belong in git history. The seed makes it reproducible instead: the same seed and scale write the same bytes, so a measurement can be repeated on another machine.
go run ./scripts/gen-dataset -out /var/tmp/big -scale 1 ./bin/veritix eval /var/tmp/big
Source Files