Learning & reference
JSONL for LLM eval datasets, and the ways it breaks
Most eval tooling loads test cases from JSONL. The format is simple, and its failures are quiet: a bad line is often skipped rather than reported, so your pass rate describes fewer cases than you think. Below is the format, a valid sample, and ten broken variants you can point your own loader at.
Download a sample
Generated live, identical on every request, no sign-up.
Change records= for up to a million rows. Each line is one test case:
{"id":"case-000001","input":"What is 919 + 729? Reply with the number only.","expected":"1648","metadata":{"category":"arithmetic"}}
{"id":"case-000002","input":"Reverse this string exactly: qa...","expected":"...aq","metadata":{"category":"string"}}
{"id":"case-000003","input":"Repeat this word exactly, with no other text: résumé","expected":"résumé","metadata":{"category":"unicode"}}Why such easy questions: this file tests your loader and your grader, not a model. Every answer is mechanically checkable (a sum, a reversal, an echo), so if a correct model fails a case, the fault is in the harness.
The contract
Five rules. Most loading bugs are one of them being broken.
One value per line
Each line is a complete JSON value, normally an object. No pretty printing across lines, and no commas between records.
UTF-8, no BOM
Encode as UTF-8 without a byte order mark. Non-ASCII characters
can appear raw or as \u escapes; both are valid.
Newlines only between records
A newline inside a string must be escaped as
\n. A raw one splits the record across two lines and both halves fail.
A stable schema
The same fields on every line. A harness reading
expected treats a row that says expected_output as having no answer.
Unique ids
Many harnesses key results by id. A duplicate can overwrite an earlier result without any warning, so one case silently disappears from your score.
Ten ways it breaks, each one downloadable
The same 100 records, with exactly one fault injected. Point your loader at each and check whether it complains or quietly drops rows.
| Variant | What is wrong | Typical symptom |
|---|---|---|
| bom | UTF-8 byte order mark (EF BB BF) before line 1 | The first record fails, or its first key is read as \ufeffid. |
| crlf | Windows line endings | JSON parsers treat \r as whitespace, so parsing is fine. It bites tools that compare, hash or count lines as text, and every line shows as changed in a diff. |
| array | A JSON array saved as .jsonl | Line by line, only the last record parses. A lenient loader imports one row and reports success. |
| trailing-comma | A comma after every object | Every line is invalid JSON. Common after hand-editing an array export. |
| blank-lines | Empty lines between records | Strict loaders raise on the empty line; lenient ones skip it. Know which yours is. |
| dup-id | Every tenth id repeats the one before | All rows parse, and results keyed by id overwrite each other. |
| ragged | Every tenth row says expected_output and drops metadata | All rows parse; those rows are scored as having no answer. |
| raw-newline | An unescaped newline inside a string | One record becomes two invalid lines. |
| truncated | The last line is cut in half | What an interrupted download or a full disk leaves behind. |
| nfd | Expected answers stored decomposed (NFD) | Everything parses. Correct answers fail exact match because "é" is two code points in the key. |
The dangerous ones are the ones that parse. A file that throws on line 1 gets fixed the
same day. dup-id, ragged and nfd load cleanly and simply make your
score wrong, which can go unnoticed for months.
A loader that fails loudly
What to check, in the order the faults above appear.
Count what you expect
Compare the number of records loaded with the number of non-empty lines in the file. A mismatch means rows were dropped, and the run should stop.
Validate the schema per row
Reject a row missing a required field instead of treating the field as empty. Fail on unknown fields too, which is how a renamed field is caught.
Normalise before comparing
Apply NFC to both the expected answer and the model's output before an exact-match check. Why normalisation matters
Reject duplicate ids
Build a set of ids as you load and stop on a repeat.
Questions
What is JSONL?
JSON Lines: every line is one complete JSON value, separated by newlines. It streams, appends and diffs line by line, which suits eval datasets.
What is the difference between JSONL and a JSON array?
An array is one value parsed whole; JSONL is one value per line. An array saved as .jsonl can load as a single record and report success.
Why does my JSONL fail on the first line?
Usually a byte order mark. Save as UTF-8 without BOM, or strip it before parsing.
Why does my eval fail answers that look correct?
Often normalisation: "é" as one code point or two. Normalise both sides to NFC before comparing.
Where can I download sample JSONL for testing?
gen.hexaqa.com/gen?type=jsonl&records=100 for a valid file, plus bad= for any of the ten breakages above.
Keep going
Synthetic data for AI evals Character encodings → Generator →
More from Learning
Guides and references for test data, file handling and AI evals. All free, no sign-up. See the full hub.