--mem and the budget error

The heap limit is the one promise Rift keeps for you: its heap stays at or under --mem.

rift --mem 1G a.csv b.csv

--mem takes bytes or a number with a K, M, G or T unit, in powers of 1024. The default is 256M and the smallest is 64K.

Rift keeps 48 KiB of the limit, plus 12 KiB per thread of its thread pool (one per core, or RAYON_NUM_THREADS), for heap that no budget counts: the command line, output buffers, messages and the thread pool. The rest is the budget. Rift reserves every allocation on the data path from it before it makes it, and reads files with pread into fixed buffers. It does not map files into memory. The tests and CI check the peak of the whole process with a counting allocator.

A larger file does not need a larger limit

On a 1 GiB pair of generated SQL dumps, with the limit at 64 MiB, Rift counted 13 million lines and finished:

a 1 GiB pair at --mem 64M

$ rift --summary --mem 64M a.sql b.sql
a: 13327101 lines, 1073741876 bytes
b: 13337797 lines, 1074747414 bytes
stratum 6, 205021 anchors
equal    6512 segments, 12357315 lines in a, 12357315 in b
differs  6753 segments, 969786 lines in a, 980482 in b
first differing ranges (0-based, end exclusive):
  a 278..409  b 278..408
  a 904..1043  b 903..1042
  a 1231..1286  b 1230..1286
  a 8095..8135  b 8095..8126
  a 9608..9621  b 9599..9620
  a 11780..11829  b 11779..11836
  a 17445..17875  b 17452..17562
  a 21543..21562  b 21230..21246
  a 22519..22835  b 22203..22522
  a 24978..25053  b 24665..24730
  ... and 6743 more
moved    193 blocks, 33051 lines
  a 27627..27952  b 28206..28531
  a 65493..65499  b 65178..65184
  a 99590..99595  b 99830..99835
  a 146048..146445  b 146171..146568
  a 318177..318184  b 318943..318950
  a 387288..387295  b 388572..388579
  a 488830..488835  b 489579..489584
  a 630072..630534  b 630538..631000
  a 637483..637965  b 638942..639424
  a 767948..768364  b 771051..771467
  ... and 183 more
peak heap 21.0 MiB of 64.0 MiB (21970358 bytes)
time 0.59 s

A smaller budget means a thinner sample of lines, so more segments are coarse and more work happens when a region is refined. A larger budget keeps more detail. In the benchmark runs on a 1 GiB pair, --mem 64M peaked at 21.0 MiB of heap and --mem 1G at 329.2 MiB, against 83.6 MiB at the default.

When the limit is too small

Below the smallest limit that works for a pair, Rift exits with status 2 and a budget error. The error ends with a limit to rerun with, the reserve included:

a limit that is too small

$ rift --summary --mem 512K a.sql b.sql
rift: skeleton budget exceeded: requested 253960 bytes, 122856 available; rerun with --mem 769K

The smallest limit that works depends on the files. The limit in the message is not a lower bound for the whole run. A run at that limit can stop at a later stage and name a higher one:

the suggested limit, run again

$ rift --summary --mem 769K a.sql b.sql
rift: skeleton budget exceeded: requested 266936 bytes, 254440 available; rerun with --mem 794K

So take the advice as a direction. Round up and try again, or start from a limit with room, such as 2M here:

a limit with room

$ rift --summary --mem 2M a.sql b.sql | tail -n 2
peak heap 816.0 KiB of 2.0 MiB (835622 bytes)
time 0.05 s

What the limit does not cover

Only the heap is bounded by --mem. The resident set also holds the binary, thread stacks and whatever page cache the kernel counts against the process. On the 1 GiB pair the resident set peaked at 113 MiB with the heap at 83.6 MiB, both at the default limit: see the benchmarks.

In key mode a larger limit means fewer passes over the files.