work / instruments / quench← abastrisMoMap →
● active · built; not released
quench
Opens multi-gigabyte JSONL files on a Mac to read, filter and profile them.
- job
- open large datasets
- domain
- developer tools · training data
- runs
- local
- class
- tool
- scale
- tested on a 40 GB file of 24.9 million lines
- since
- 2026
One call
- 01indexLine breaks found and indexed as the file streams in
- 02windowFirst rows on screen while the index grows
- 03schema5,000 lines sampled to infer the columns
- 04drawOnly the rows on screen are parsed
- 05profileFilters and a token histogram stream in the background
In · out
inputs
- JSONL file
- compressed .jsonl.gz file
- a local tokenizer file
outputs
- a table that scrolls millions of rows
- conversation detail pane
- token histogram
- filtered rows
What is measured
Against: Its own earlier version (58 s to open 40 GB) and the disk's raw read speed (6.13 s).
Opens a 40 GB file in 6 s, down from 58 s and at the disk's own read speed; first rows on screen in under 3 msmeasured
Reopens the same file in under a secondmeasured
Scrolls 20 million rows at 60 frames a secondmeasured
Filters about 475,000 rows a second and searches about 1.1 million; first results in about 35 msmeasured
Under 500 MB of memory for a 40 GB filemeasured
Stack · links
RustGPUIrayonmemchrsimd-jsontokenizers