Compare NDJSON logs

NDJSON has one JSON object per line, so a line diff works. When the order changes, match by a field instead.

A line diff

On a pair of generated event logs, 64 MiB each:

a.ndjson, b.ndjson

$ rift --summary a.ndjson b.ndjson
a: 407536 lines, 67108957 bytes
b: 405238 lines, 66728329 bytes
stratum 0, 397791 anchors
equal    224 segments, 401211 lines in a, 401211 in b
differs  224 segments, 6325 lines in a, 4027 in b
first differing ranges (0-based, end exclusive):
  a 239..242  b 239..240
  a 4615..4616  b 4613..4623
  a 7950..7956  b 7957..7963
  a 11526..11530  b 11533..11534
  a 12436..12443  b 12440..12441
  a 13170..13171  b 13168..13172
  a 16145..16149  b 16146..16147
  a 16739..16750  b 16737..16738
  a 16950..16951  b 16938..16940
  a 17310..17311  b 17299..17303
  ... and 214 more
moved    32 blocks, 1258 lines
  a 11527..11530  b 13169..13172
  a 16146..16149  b 17300..17303
  a 19850..19860  b 20300..20310
  a 22290..22295  b 23026..23031
  a 27604..27614  b 29168..29178
  a 51237..51243  b 52244..52250
  a 59927..60393  b 60850..61316
  a 62237..62240  b 63434..63437
  a 66692..66696  b 68354..68358
  a 75122..75129  b 75975..75982
  ... and 22 more
peak heap 68.0 MiB of 256.0 MiB (71317724 bytes)
time 0.07 s

rift a.ndjson b.ndjson writes the unified diff, and patch applies it. The summary above peaked at a heap of 68.0 MiB.

Match by a field

When the same events can arrive in a different order, compare by key. json:PATH takes a dotted path, so json:user.id reaches a nested field. Here the records have an id field:

the first two records

$ head -n 2 a.ndjson | cut -c1-110
{"id":"10000000001","type":"CreateEvent","actor":{"id":87359619,"login":"jorwailpan"},"repo":{"id":94868419,"n
{"id":"10000000002","type":"IssuesEvent","actor":{"id":31367866,"login":"ptxql"},"repo":{"id":81940066,"name":

rift --key json:id

$ rift --key json:id a.ndjson b.ndjson > keys.txt
$ head -n 8 keys.txt
$ wc -l < keys.txt
-"10000000206"	206	-
+"10000000641"	-	640
+"10000000642"	-	641
+"10000000643"	-	642
+"10000000644"	-	643
+"10000000645"	-	644
+"10000000646"	-	645
+"10000000647"	-	646
    4389

The key prints as the file has it, quotes included. A tab, line break or backslash in a key prints as \t, \n or \\. See the key mode guide for the output format and the rules.