Compare huge CSV files by key
Two exports of the same table rarely list their rows in the same order. A line diff then reports everything as changed.
Pick the key column by header name or by position:
rift --key csv:id a.csv b.csv
rift --key csv:1 a.csv b.csv
On a pair of generated CSV files, 32 MiB each, with a header row:
$ head -n 2 a.csv id,pickup_at,passengers,pickup_lat,pickup_lon,dropoff_lat,dropoff_lon,fare,payment 1,2024-12-25 11:56:34,4,40.549135,-73.402144,40.028372,-73.611914,12.97,3
$ rift --key csv:id a.csv b.csv > keys.txt $ head -n 8 keys.txt $ wc -l < keys.txt +191 - 140 +192 - 141 -144 145 - -145 146 - -146 147 - -147 148 - -148 149 - -149 150 - 15606
Each line is one key that differs, with tabs between the fields. + is a key only in the second file, - a key only in the first, ~ a key in both with different records. The two numbers are the line where the record starts in each file, from 1. The last line is the line count: the number of keys that differ.
Check the exit status
A key that occurs twice in one file stops the run with exit status 2. The run can stop after it printed some changes, so a script must check the exit status before it uses the output. See exit codes.
Memory
Key mode writes no temporary files. When the keys do not fit the budget, Rift splits them by hash and reads both files once per part. A generated 1 GiB CSV pair with shuffled rows took one pass at the default 256 MiB (seed 7, Apple M1). --stats reports the number of passes.
More options and rules are in the key mode guide.