Diff files larger than RAM

GNU diff holds both files in memory. On a big pair it gets killed, or the machine starts to swap.

Rift reads the files in fixed buffers and keeps a sample of their lines. Its heap stays at or under the limit you set with --mem.

What a small limit does

A 1 GiB pair of generated SQL dumps, with the heap limit at 64 MiB:

a 1 GiB pair at --mem 64M

$ rift --summary --mem 64M a.sql b.sql
a: 13327101 lines, 1073741876 bytes
b: 13337797 lines, 1074747414 bytes
stratum 6, 205021 anchors
equal    6512 segments, 12357315 lines in a, 12357315 in b
differs  6753 segments, 969786 lines in a, 980482 in b
first differing ranges (0-based, end exclusive):
  a 278..409  b 278..408
  a 904..1043  b 903..1042
  a 1231..1286  b 1230..1286
  a 8095..8135  b 8095..8126
  a 9608..9621  b 9599..9620
  a 11780..11829  b 11779..11836
  a 17445..17875  b 17452..17562
  a 21543..21562  b 21230..21246
  a 22519..22835  b 22203..22522
  a 24978..25053  b 24665..24730
  ... and 6743 more
moved    193 blocks, 33051 lines
  a 27627..27952  b 28206..28531
  a 65493..65499  b 65178..65184
  a 99590..99595  b 99830..99835
  a 146048..146445  b 146171..146568
  a 318177..318184  b 318943..318950
  a 387288..387295  b 388572..388579
  a 488830..488835  b 489579..489584
  a 630072..630534  b 630538..631000
  a 637483..637965  b 638942..639424
  a 767948..768364  b 771051..771467
  ... and 183 more
peak heap 21.0 MiB of 64.0 MiB (21970358 bytes)
time 0.59 s

The summary has the same shape as for a small pair. The sample is thinner, so fewer lines serve as anchors and more segments need refining.

What was measured

The largest pair measured so far is 10 GiB. On a Mac with 16 GB of RAM, GNU diff and git diff did not run on it, because they need more than 16 GB. Rift finished the full diff in 26.16 s with a heap of 94.1 MiB and a resident set of 136.5 MiB. The table and the commands are on the benchmarks page. No pair above 10 GiB has been measured.

Pair	Tool	Time	Peak memory
1 GiB	`rift` (full diff)	0.93 s	130.8 MiB
1 GiB	system `diff`	17.55 s	3.2 GiB
10 GiB	`rift` (full diff)	26.16 s	136.5 MiB
10 GiB	`diff`	did not run: needs more than 16 GB of RAM

Apple M1, 16 GB RAM, internal SSD, macOS, warm page cache. Source: the Benchmarks section of README.md at the v0.1.0 tag.

When the limit is too small

If the budget is below what the pair needs, Rift exits with status 2 and prints a limit to rerun with. The advice can name a higher limit again on the next run: see the memory guide.

What the limit does not cover

Only the heap is bounded. The resident set also holds the binary, thread stacks and the page cache the kernel counts against the process.