Rift / Guide

Key mode

With --key, Rift matches records by key and ignores row order. Use it on exports where the same rows can arrive in a different order.

A line diff reports a reordered export as thousands of deletions and insertions. Key mode reports what changed per record.

Choose the key

Read the output

Rift prints one line per key that differs, with tabs between the fields and line numbers counted from 1:

A and B are the lines where the record starts, or - when it is absent.

Print the records

With --show-records, Rift prints the records under each change line. The record from the first file follows -, the one from the second file follows +:

~1	2	4
  - 1,x
  + 1,X
-2	3	-
  - 2,"two
  - lines"
+4	-	3
  + 4,"say ""hi"""

A record that spans lines carries the prefix on each line. Strip the prefixes and you get the record's bytes as the file has them, quotes and inner line breaks included, without the line terminator. Rift reads each record back from its file through a 64 KiB buffer, so a long record takes no more memory than a short one.

--show-records prints whole records. Output that names the fields that changed is planned for 0.4, see the roadmap.

This is a pair of generated CSV files, 32 MiB each, with the same keys in a different order and some rows added and removed:

$ head -n 2 a.csv
id,pickup_at,passengers,pickup_lat,pickup_lon,dropoff_lat,dropoff_lon,fare,payment
1,2024-12-25 11:56:34,4,40.549135,-73.402144,40.028372,-73.611914,12.97,3

$ rift --key csv:id a.csv b.csv > keys.txt
$ head -n 8 keys.txt
$ wc -l < keys.txt
+191	-	140
+192	-	141
-144	145	-
-145	146	-
-146	147	-
-147	148	-
-148	149	-
-149	150	-
   15606

The last line of that output is the line count of keys.txt, the number of keys that differ.

The same on NDJSON, by the id field. The key prints as the file has it, quotes included:

$ head -n 2 a.ndjson | cut -c1-110
{"id":"10000000001","type":"CreateEvent","actor":{"id":87359619,"login":"jorwailpan"},"repo":{"id":94868419,"n
{"id":"10000000002","type":"IssuesEvent","actor":{"id":31367866,"login":"ptxql"},"repo":{"id":81940066,"name":

$ rift --key json:id a.ndjson b.ndjson > keys.txt
$ head -n 8 keys.txt
$ wc -l < keys.txt
-"10000000206"	206	-
+"10000000641"	-	640
+"10000000642"	-	641
+"10000000643"	-	642
+"10000000644"	-	643
+"10000000645"	-	644
+"10000000646"	-	645
+"10000000647"	-	646
    4389

Rules to know

• Rift compares the bytes of a record before it calls the record unchanged.
• In CSV, a record can span lines inside quotes, and the header row counts as a record keyed by its own cell.
• In a pg_dump, only the rows of COPY blocks with the right column count take part, and the same key in two tables names two records.
• A key that occurs twice in one file stops the run with exit status 2 and lists the first few. A run can stop after it printed some changes. The output is then incomplete: check the exit status before you use it.
• A record too long for the budget stops the run with a budget error that names a --mem under which the record fits.
• --key cannot be combined with --summary, --interactive or --no-moves. --show-records needs --key, and --stats works with both.

Memory

Key mode writes no temporary files. When the keys do not fit --mem, Rift splits them by hash and reads both files once per part, so a smaller budget means more passes over the files. Each pass prints its changes sorted by line. --stats reports the number of passes.

Key mode prints the same set of lines at any --mem and on any number of threads. Their order is not fixed. It depends on --mem, since the passes split the keys by budget. Once key mode runs on several threads, from a refine half of 8 MiB, about --mem 16M, it can also differ between two runs with the same files and limit: the threads share the refine half, and the one that runs out first sets where the pass starts over. Sort the output before you compare runs.

--show-records takes 64 KiB of the budget for its read buffer, plus 24 bytes for each record that waits for its match or has changed, until the end of its pass. Under a tight --mem that can add passes.