venn

venn

Row-level keyed diff of tabular datasets: files, directories, and lake tables. One binary, pure Go, no server and no SQL to write.

$ venn prod_export.parquet migrated.csv --key id
schema: identical
rows:   +50,000 added   -50,137 removed   ~100,008 changed   =9,849,855 unchanged   (left 10,000,000, right 9,999,863)
changed columns: price(41,203) qty(38,900) updated_at(31,077)
$ echo $?
1

Point it at two Iceberg snapshots and it tells you which rows changed, not just which files:

$ venn s3://lake/orders#8412 s3://lake/orders#8500 --key order_id
venn: s3://lake/orders: skipping 412 data files shared by both snapshots (96,401,220 rows per side)
schema: identical
rows:   +1,204,551 added   -0 removed   ~88,012 changed   =96,530,190 unchanged   (left 96,618,202, right 97,822,753)
changed columns: status(88,012)

Install

A single static binary, no runtime and no dependencies.

Download the archive for your platform from releases, extract, and put venn on your PATH. Linux, macOS and Windows, amd64 and arm64. Each release ships a checksums.txt.

# linux amd64, adjust the tag and platform
VER=0.1.1
curl -fsSL "https://github.com/KonMam/venn/releases/download/v${VER}/venn_${VER}_linux_amd64.tar.gz" \
  | tar -xz venn && sudo mv venn /usr/local/bin/

Homebrew (macOS):

brew install KonMam/tap/venn

Docker:

docker run --rm -v "$PWD:/data" ghcr.io/konmam/venn /data/left.parquet /data/right.parquet --key id

Python, if the rest of your stack is:

pip install venn-bin     # installs the same binary and puts venn on PATH

From source, needs Go 1.26 or newer:

go install github.com/KonMam/venn/cmd/venn@latest

Status

Pre-1.0 and in active development, but not experimental: every release is gated on a ground-truth oracle covering each supported format combination, a corruption torture suite, and a performance check that runs on every pull request. Expect flag names and output shapes to still move before 1.0. The parts to build CI on are the exit codes (0 equal, 1 differences, 2 error) and the --format json field names; those will not change without a major version. Issues and bug reports are welcome.

What it's for

• Migration validation. The old pipeline's output against the new one's, before you cut over. Any format diffs against any other, and values compare by logical type, so a parquet INT64 10 equals a CSV 10.0.
• Pipeline regression in CI. Yesterday's output against today's, with an allowed-change budget, an exit code, and a markdown report in the job summary. There is a GitHub Action.
• Lake table snapshot deltas. Which rows changed between two Iceberg snapshots or Delta versions. No engine, no cluster, no warehouse.

venn answers one question, which rows differ. Warehouse connectors and live-database diffing, dbt integration, lineage and data-quality rules, UIs and Excel are out of scope. If your data already lives in a warehouse, use the warehouse; venn is for data in files and lake tables, compared on the machine you are on.

What it reads

Diffing two snapshots of the same table skips data files that are live in both. A copy-on-write table holds each row in exactly one file, so shared files contribute identical rows to both sides and cancel exactly; a 1% churn on a huge table scans about 2% of it.

Usage

venn <left> <right> [--key <col>[,<col>...]] [flags]  row + schema diff
venn schema <left> <right>                            schema diff only
venn snapshot <file> --key <col> --output <b.snap>    save a hash baseline
venn <file> --against <b.snap>                        diff against a baseline

Exit codes: 0 identical or within budget, 1 differences, 2 error.

Full reference in docs/flags.md, worked examples in docs/usage.md, and the comparison rules in docs/semantics.md.

CI

venn old/ new/ --key id --summary          # fastest: counts only
venn old/ new/ --key id --max-diff 0.1%    # allow small drift
venn old/ new/ --key id --format json      # machine-readable
venn old/ new/ --key id --report out.md    # markdown for humans

GitHub Action

- uses: KonMam/venn@v0.1.1
  with:
    left: expected/orders.parquet
    right: build/orders.parquet
    key: order_id
    max-diff: "0.1%"

The step fails when differences exceed the budget, appends a markdown report to the job summary, and exposes added, removed, changed and unchanged as outputs.

Snapshot baselines

When the left side should not be re-read every run, freeze it once. A snapshot costs 16 bytes per row whatever the table's width:

venn snapshot expected.parquet --key id --output baseline.snap
venn build/output.parquet --against baseline.snap

Performance

An M1 Pro laptop with 16 GB, counts plus per-column attribution:

The batch engine keeps a bounded number of rows in flight whatever the input size, so the 1B-row diff peaks at roughly the footprint of the 10M-row one. --summary is one scan instead of two: the 1B diff drops to 61 s, csv.gz to 3.8 s.

The method, the pinned tool versions and the same workloads under a DuckDB SQL query are in docs/performance.md.

Embedding

pkg/venn wraps the engine for use as a library:

res, err := venn.Diff("a.parquet", "b.parquet", venn.Options{Keys: []string{"id"}})

Development

go test ./...              # manifest oracle, interop corpus, table fixtures, torture matrix
go test -race ./internal/...
golangci-lint run ./...
go run ./bench/gen --help  # fixture generator

The fixture generator plants a known set of diffs and writes the ground truth to manifest.json; the suite checks against it for every format combination in both join modes. Lake-table fixtures are written by pyiceberg and delta-rs, and the merge-on-read delete files are validated by reading them back with pyiceberg and DuckDB.

CONTRIBUTING.md covers what CI checks, including the per-PR performance gate, and which changes need a test. bench/README.md documents the performance and torture suites.

License

MIT. See LICENSE.