Reference

The manual page, rendered from man/rift.1 when the site is built. Run man rift after you install the .deb, or read it here.

Name

rift - Diff large text files under a hard memory limit

Synopsis

rift [--summary] [--mem] [-i|--interactive] [--no-moves] [--stats] [--key] [--show-records] [-h|--help] [-V|--version] <A> <B> [subcommands]

Description

rift compares two large text files under a heap limit set with --mem. It writes a unified diff that patch(1) applies to A to get B. Its other modes print a summary, open a terminal view, or compare records by key.

Rift hashes every line once and cuts the pair into segments at lines that occur once in each file. Segments with identical bytes are equal. The others are refined on demand.

Options

--summary: Print line counts, the segment map and resource use instead of a diff
--mem <MEM> [default: 256M]: Heap limit, in bytes or with a K, M, G or T unit (powers of 1024), at least 64K
-i, --interactive: Browse the diff in the terminal instead of printing it
--no-moves: Skip the search for moved blocks. The diff is the same
--stats: After a diff, print peak heap, time and approximate regions to standard error, and after an error the peak heap. With --summary it adds nothing: the summary prints the heap and the time, and refines nothing
--key <SPEC>: Compare records by key instead of lines, regardless of row order: csv:N (column, from 1), csv:NAME (header), json:PATH (dotted field) or copy:NAME (COPY column of a Postgres dump)
--show-records: With --key, print the records under each change: the one from A behind " - ", the one from B behind " + ", each of their lines behind the same prefix. Field-level output, which names the fields that changed, is planned for 0.4
-h, --help: Print help
-V, --version: Print version
<A>: First file
<B>: Second file

Modes

rift [options] A B

Writes a unified diff to standard output, as diff -u does. patch(1) applied to A gives B.

rift serve [options]

Serves Rift's protocol to an editor or another program; see SERVE. Added in 0.3.

rift git-diff [--mem SIZE] [--stats] path old-file old-hex old-mode new-file new-hex new-mode

The command git runs for an external diff; see GIT. Added in 0.3.

serve and git-diff are subcommands, so a file of either name is given with a path: rift ./serve other.

--summary

Writes no diff. Prints the line and byte count of each file, the number of segments that are equal or differing, the first 10 differing ranges (counted from 0, end excluded), the moved blocks, the peak heap and the time. It refines no segment, so it cannot say which ones would come out approximate; the full diff with --stats does.

-i, --interactive, rift tui A B

Opens a terminal view of the diff. The view is usable before the diff is complete. A hunk still being refined shows as one pending row and fills in by itself. The keys are:

n, p: Jump to the next and previous difference.
space, b: Page down and up.
g, :: Go to a line of A.
m: Jump from the first moved block on screen to its other end.
q: Quit.

rift tui is rift -i. With fewer than two arguments after tui, it is a file name. When standard input or output is not a terminal, it exits with status 2 before it reads the files.

--key SPEC

Compares records by key and ignores their order. Prints one line per key that differs, with tabs between the fields and line numbers counted from 1. ~ marks a key in both files with different records, - a key only in A, + a key only in B. A and B are the line where the record starts, or - when it is absent:

~KEY	A	B
-KEY	A	-
+KEY	-	B

SPEC is one of:

csv:N: CSV column N, from 1.
csv:NAME: CSV column with the header NAME.
json:PATH: Dotted field path in NDJSON records, such as user.id.
copy:NAME: Column NAME of the COPY blocks of a pg_dump file.

The key is printed as the file has it, quotes included. A tab, line break or backslash in a key prints as \t, \n or \\. A record counts as unchanged only after its bytes compare equal. A key that occurs twice in one file stops the run with exit status 2, possibly after changes were printed. The message on standard error then says the output is incomplete and must be discarded, so check the exit status before you use the output. --key cannot be combined with --summary or --interactive.

--show-records

With --key, prints the records under each change line, the one from A behind two spaces and -, the one from B behind two spaces and +, each followed by a space:

~KEY	A	B
  - record in A
  + record in B

A record that spans lines, such as a CSV record with a line break inside quotes, carries the prefix on each of its lines. Strip the prefixes and you get the record's bytes as the file has them, without its line terminator. Rift reads each record back from its file through a 64 KiB buffer, so a long record takes no more memory than a short one. Without --key, Rift refuses the option. Field-level output, which names the fields that changed, is planned for 0.4.

--stats

After a diff, prints to standard error the peak heap, the time, the number of approximate regions and the moved blocks. With --key it prints the peak heap, the time, the number of passes over the files and the number of records in each file. After an error it prints the peak heap. With --summary it adds nothing, since the summary already prints the peak heap and the time.

--no-moves

Skips the search for moved blocks. The unified diff is byte for byte the same; the search only adds notes to --summary, --stats and the terminal view, which then say that it was off. It cannot be combined with --key.

--mem SIZE

Sets the heap limit, at least 64K. The default is 256M. The smallest limit that works depends on the files. Below it, Rift exits with status 2 and a budget error that ends with a limit to rerun with, such as rerun with --mem 10333K. It is not a lower bound, and a run at that limit can still stop at a later stage and name a higher one. In key mode a larger limit means fewer passes over the files. Key mode prints the same set of lines at any limit and thread count. Their order depends on the limit, since each pass prints its own changes sorted by line, and once key mode runs on several threads (a refine half of 8 MiB or more, about --mem 16M and up) it can differ between runs.

Exit Status

Exit status follows diff(1).

0: The files are identical.
1: The files differ.
2: An error: an unreadable file, a budget too small, a duplicate key, an invalid option or -i in a build without the terminal view. The message goes to standard error.

When standard output closes before Rift finishes, as in rift a b | head, Rift stops without a message and exits 141, the status a shell reports for diff(1) killed by SIGPIPE.

Memory

The heap stays at or under --mem. Rift keeps 48 KiB of it, plus 12 KiB per thread of its thread pool (one per core, or RAYON_NUM_THREADS), for the heap no budget counts: the command line, output buffers, messages and the thread pool. The rest is the budget, half for the map of the files and half for refinement. Rift reserves each allocation on the data path from the budget before it makes it, and reads files with pread(2) into fixed buffers. It does not map files into memory. The test suite checks the peak with a counting allocator.

These are not guaranteed:

The resident set size. It also holds the binary, thread stacks and the page cache the kernel counts against the process.

A minimal diff. Like patience diff, the output can show a change as larger than the smallest possible one.

Detection of every moved block. Blocks are found from sampled lines, so some moves are missed.

Approximate Regions

When Rift cannot align a region within the budget, it replaces it whole: the output is still correct, but coarse. This happens in large regions with no line that is unique in both files, such as long runs of identical lines. A region whose two sides share no line at all is also replaced whole, but that is the smallest diff, so it is not approximate.

The unified diff marks such a hunk with approx: and the reason after the closing @@ of its header, where diff -p prints a function name. patch(1) ignores it. The reasons are:

too many changes among repeated lines to align them exactly

found no line here that occurs once in each file, so there is nothing to align on

too large to align within --mem; a larger --mem may help

The terminal view draws the hunk in yellow with a ~ and shows the reason in the status line when it fits. --stats counts the approximate segments and lines, and the segments for each reason.

Moved Blocks

A block of at least 3 lines that appears in both files at different positions is reported as moved. The unified diff still shows it as a deletion and an insertion. --summary and --stats list the blocks, and the m key in the terminal view jumps to the other end of one. When the budget leaves no room for the search, the output says so.

Git

Added in 0.3. rift git-diff is the command git runs for an external diff. Git passes it path old-file old-hex old-mode new-file new-hex new-mode, and for a rename or copy also the new path and git's rename lines. Rift prints a unified diff with git's headers (diff --git, mode lines, index, --- a/path, +++ b/path, and /dev/null for an added or deleted file) and exits 0 whether or not the files differ, because git treats any other status as a failure and stops with status 128. It also exits 0, without a message, when git's pager quits before the diff ends. The status of git diff is git's own: 0, or 1 with --exit-code when there are differences. Rift exits 2 on an error. Options such as --mem go before the arguments git adds. --stats prints the peak heap to standard error when the diff or the error is done.

To use it for large files only, name them in .gitattributes and define the driver. Git runs the command through the shell, so a quoted string can carry options:

*.sql diff=rift
*.csv diff=rift

git config diff.rift.command 'rift git-diff --mem 1G'

The headers match git diff, except that the hunk headers carry no function name, the hash of a work tree file is written as 0000000 because git does not pass one, and hashes are always 7 characters.

A file that is a Git LFS pointer on either side is replaced by the object in .git/lfs/objects of the common git directory, found with git rev-parse --git-common-dir, and read in place. If the object is missing Rift names its oid and suggests git lfs fetch. If its size differs from the pointer's, Rift stops. Rift does not check the hash, and ignores lfs.storage.

Git writes every blob that is not in the work tree to a temporary file before it runs the command. For a file that is not stored with LFS, that is a full copy: on a 10 GB file, 10 GB of disk and the time to write it. Comparing a work tree file with the index has one such copy; comparing two commits has two.

For the terminal view, use git difftool. It passes the two files as $LOCAL and $REMOTE and does not resolve LFS pointers:

git config difftool.rift.cmd 'rift -i "$LOCAL" "$REMOTE"'
git config difftool.prompt false
git difftool -t rift

Serve

Added in 0.3. rift serve [--max-session-mem SIZE] [--max-total-mem SIZE] [--stats] speaks Rift's protocol, JSON-RPC 2.0 framed as in LSP, on standard input and output, for editors and other programs. A client opens a pair of files as a session, then asks for counts, pages of differing segments and moved blocks, and the lines of one segment on both sides. Every answer has a size bound that does not depend on the files. The protocol is described in docs/protocol.md in the source repository.

--max-session-mem (default 1G) is the most memory one session may ask for; a session gets 256 MiB when it does not ask. --max-total-mem (default 2G) bounds the heap of the whole server: the sessions share it, less 8 MiB the protocol keeps for its buffers. At most four sessions are open, and none is closed for being idle. --stats prints the peak heap to standard error on exit.

The server exits with status 0 when standard input closes, and 2 when it cannot start.

Examples

Write a patch and apply it:

rift a.sql b.sql > changes.patch
patch a.sql changes.patch

Browse the differences:

rift -i a.sql b.sql

Count the differences without writing a diff, with a 1 GiB heap limit:

rift --summary --mem 1G a.csv b.csv

Write a patch and report the peak heap and time on standard error:

rift --stats a b > changes.patch

Compare two CSV files by the column id, whatever the row order:

rift --key csv:id a.csv b.csv

Compare NDJSON by a nested field, and a pg_dump by a COPY column:

rift --key json:user.id a.ndjson b.ndjson
rift --key copy:id a.sql b.sql