csvdiff

csvdiff

Diff two CSV files by their rows and columns instead of their bytes.

git diff on a CSV export is close to useless. Sort the file differently and every line turns red. Re-export it from a tool that quotes fields a little differently and the whole thing lights up. Meanwhile the one price that actually changed is buried somewhere in the noise.

csvdiff parses both files as tables, matches rows by a key column, and tells you which rows were added, which were removed, and which cells changed.

Status

v1.0.0. Everything documented here is built, tested and running.

Known limits, all of them covered in more detail below: files that are not UTF-8 are transcoded into memory up front rather than streamed, a side read from stdin is buffered for the same reason, and when a key value repeats, the pairing past the first identical match is a guess that csvdiff warns about.

Install

curl -fsSL https://raw.githubusercontent.com/Alyetama/csvdiff/main/install.sh | sh

The script picks up a prebuilt binary for macOS and Linux on x86-64 and arm64, checks it against the published SHA-256 sums, and refuses to install anything that does not match. If there is no binary for your platform it builds from source instead, which needs Go 1.22 or newer.

If you already have Go and would rather skip the script:

go install github.com/Alyetama/csvdiff@latest

Use it

csvdiff old.csv new.csv

That is the whole interface for the common case. csvdiff works out the delimiter, the encoding, and which column to match on, then prints what changed.

Real exports have a timestamp column that moves on every write. Drop it from the comparison:

csvdiff --ignore-columns updated_at old.csv new.csv

Match on a column that is not called id:

csvdiff --key sku old.csv new.csv

Keys can span several columns when no single one is unique:

csvdiff --key region,sku old.csv new.csv

How rows get matched

Without --key, csvdiff looks for a likely identifier column and checks that its values are unique in both files. The first candidate that passes wins. Candidates are id, key, uuid, guid, pk, rowid, row_id, anything ending in a separated id such as order_id or customerId, and then email, sku, code and slug. A bare id ending is not enough on its own, otherwise columns called paid or valid would look like identifiers.

If nothing qualifies, csvdiff falls back to matching whole rows, so a row is either present or not and nothing is ever reported as "modified".

When a key does appear more than once, rows that are byte-identical are paired up first and only the leftovers are matched in file order. Two rows swapping places inside a key group therefore read as unchanged rather than as two modifications. csvdiff warns when this happens, because past that point the pairing is a guess.

The header line of the output always says which choice it made:

  matched on "sku" (auto-detected) · 5 columns compared · ignoring updated_at

If that line says something you did not expect, pass --key and move on.

What it ignores unless you say otherwise

By default, three kinds of difference are treated as noise. Leading and trailing whitespace inside a field. Quoting style, because Ada and "Ada" parse to the same value. Column order, since columns are matched by name.

Row order is also ignored, but it is tracked. Pass --show-reordered and csvdiff reports which rows moved. It uses the longest run of rows that kept their relative order as the baseline, so a single row pulled to the top counts as one move rather than as everything else shifting down.

--strict turns all of that off. Values are compared byte for byte and a changed column order becomes a reported difference.

Output

The default is colored terminal output grouped by added, removed, and modified, with modified rows broken down cell by cell. Colors turn off automatically when stdout is not a terminal. NO_COLOR forces them off and FORCE_COLOR forces them on, but an explicit --color always or --color never wins over both.

--side-by-side prints each change as an old-vs-new table across all columns, with the changed cells highlighted. Useful for wide rows, where the cell-by-cell view loses context.

--format json gives you the whole result as one object, with a summary block for quick checks:

csvdiff --format json old.csv new.csv | jq .summary

{
  "added": 2,
  "removed": 1,
  "modified": 7,
  "unchanged": 0,
  "reordered": 0
}

That is the sample pair in examples/, where every row carries a bumped updated_at. Add --ignore-columns updated_at and the same summary reads "modified": 2, "unchanged": 5.

Every list in the JSON is present even when empty, so you never have to guard against null.

--format csv writes just the changed rows, tagged in a change column. Modified rows appear twice, once as modified_old and once as modified_new, so the export keeps both sides:

csvdiff --format csv old.csv new.csv > changes.csv

In CI

--exit-code makes csvdiff exit 1 when it finds differences. Exit code 2 always means csvdiff itself failed, so a broken file never reads as a clean diff:

csvdiff --exit-code --ignore-columns exported_at expected.csv actual.csv

- name: Check the fixture is still current
  run: csvdiff --exit-code --summary expected.csv build/actual.csv

Flags

Flags can go before, after, or between the file paths. Either path can be - to read that side from stdin.

Delimiters and encodings

The delimiter is detected by parsing the first few kilobytes with each candidate (comma, semicolon, tab, pipe, colon) and keeping whichever one produces a consistent number of fields per row. A comma inside a quoted sentence will not beat the semicolon that is actually splitting the columns. Override it with --delimiter when the guess is wrong.

UTF-8, UTF-8 with a BOM, and UTF-16 in either byte order are handled. Anything that is not valid UTF-8 is read as Latin-1, which is usually what a stray 0xE9 in an old export turns out to be.

Large files

csvdiff never holds both files in memory. It walks the old file once and keeps a small record per row: the key, a 128-bit fingerprint of the compared columns, and the byte offset where the row starts. Then it streams the new file against that index. A row's full text is only read back off disk when the row turns out to have changed.

Measured on an M-series Mac with /usr/bin/time -l, on generated files of 500,000 rows with 2,995 differences between them:

Wall-clock times on that machine ranged from 2.5 s to 13 s across repeated runs and were dominated by page cache and other load, so only the user CPU time is worth quoting. The RSS ranges are the spread across three runs, not error bars.

That gap between the two shapes is worth being straight about: the index itself is roughly 40 bytes per row, but the CSV parser allocates a string per record, so wide rows churn more garbage on the way through. Peak memory follows the row count first and the row width second. It does not follow the file size, which is the part that matters when the file is bigger than RAM.

Two caveats. Files that are not UTF-8 get transcoded into memory up front, because seeking into a re-encoded stream is not worth the complexity. And reading a side from stdin buffers it, since stdin cannot be seeked.

Build from source

git clone https://github.com/Alyetama/csvdiff
cd csvdiff
make build     # binary lands in bin/
make test
make demo      # runs against the sample files in examples/

No dependencies beyond the Go standard library.

License

MIT. See LICENSE.