Learning & reference
A RAG test corpus: documents that hide facts behind real extraction traps
Most wrong answers from a RAG pipeline are not retrieval failures. The text was read wrongly before retrieval ran, so the chunk that holds the answer never existed. This corpus is nine short documents in six formats, each hiding at least one fact behind a trap real documents contain, with an answer key that tells you which trap caught you.
Download
Free, no sign-up, generated from source. SHA-256 for every file is in the ZIP.
RAG test corpus (ZIP, 17 KB) Pack details →
Inside: rag/corpus/ (the documents), rag/baseline/all-facts.txt (the control) and
rag/answers.jsonl (the answer key). The company in them, Quillmere Instruments, is invented, and
every address is on a reserved domain.
How to run it
Two indexes, one set of questions. The comparison is the result.
1. Index the corpus
Load rag/corpus/ through your real pipeline:
your loaders, your chunker, your embeddings. Nothing else goes in this index.
2. Index the control, separately
Build a second index from
all-facts.txt alone. It holds every fact as one plain sentence, so nothing can go wrong
reading it.
3. Ask all 15 questions of each
Grade against expected and the
accept list. If the control scores well and the corpus does not, the model is fine and
extraction is failing.
Never index the answer key
answers.jsonl contains every answer. Put it
in the index and every question passes, which proves nothing.
Each line of the answer key is one eval case, in the JSONL shape most eval tools load:
{"id":"rag-07","input":"What is the gross price of the K2 field meter, including VAT?","expected":"EUR 571.20","metadata":{"source":"rag/corpus/price-list.xlsx","trap":"two-row-header","evidence":"K2 field meter | Net 480 | Gross incl. 19% VAT 571.2","accept":["571.20","571.2"],"trap_answer":"EUR 480"}}The useful field is trap_answer. A wrong answer tells you something failed.
An answer that matches trap_answer tells you which stage failed and why, which is the difference
between a bug report and a guess.
The traps
What each file does, and what we saw happen when a real tool read it.
| File | The trap | What goes wrong |
|---|---|---|
| product-sheet-two-column.pdf | Two columns drawn row by row | Extraction in content stream order (we tested pdftotext -raw) interleaves the warranty and returns text line by line. Layout-aware mode keeps them apart. |
| staff-handbook.pdf | A sentence split by a page break | Page 1 ends "an on-call allowance of", page 2 starts "EUR 42 per weekend", with a footer and a header in between. Per-page chunks answer nothing. |
| k2-specification.pdf | A footnote overrides the body | The spec says 50 degrees C; small print at the foot of the page says 40 for older serials. Pipelines that drop page furniture keep only the wrong one. |
| price-list.xlsx | A two-row, merged header | pandas read_excel by default names the gross column Unnamed: 2 and reads the sub-header as data, so "price" means net. |
| support-faq.html | Text hidden by CSS | An old notice with different support hours is hidden by a stylesheet rule. BeautifulSoup get_text() keeps it; only an extractor that applies CSS drops it. |
| support-thread.eml | The answer is in the quoted reply | The battery voltage appears only in the quoted earlier message. Strip quotes to de-duplicate a thread and the answer is gone. |
| firmware-3.1-notes-v1.md and v2 | A stale near-duplicate | Identical except the default sample interval. Only the front matter says v1 is superseded. Drop front matter and you cannot tell them apart. |
| service-manual.docx | Word splits text across runs | "12 months" is stored as "1" and "2 months" in separate runs, as Word does after edits. python-docx joins them correctly; join with spaces and you get "1 2 months". |
Two of the fifteen questions have no trap. They sit in trapped files, so if one of those fails too, the file was not read at all, which is a different problem from falling for the trap.
Fixing what it finds
The usual remedy for each trap. None of them needs a better model.
Columns and layout
Use a layout-aware extractor for PDFs, and spot-check what it produces for your ten most common document templates. Read the extracted text, not the rendered page.
Page breaks
Strip running headers and footers, then chunk across page boundaries rather than per page, so a sentence that crosses a page stays whole.
Footnotes
Keep a page's footnotes in the same chunk as the text that references them, or attach them as metadata. Never filter small type by font size alone.
Spreadsheet headers
Detect multi-row headers and flatten them into one name per
column ("Price (EUR) / Gross incl. 19% VAT"). In pandas, header=[0, 1] reads both rows.
Hidden HTML
Extract from rendered text where you can, or remove elements hidden
by display:none, the hidden attribute and aria-hidden before indexing.
Email threads
If you strip quoted text, keep it when the new message refers back to it ("see below", "as I said"), or index the whole thread once instead of each message.
Versions
Carry status and dates from front matter into chunk metadata, and filter superseded documents at query time.
Word runs
Concatenate runs within a paragraph with no separator. Separate paragraphs, not runs.
What it is not
It is small on purpose: nine documents and fifteen questions, so a failure is easy to trace to one file. It tests ingestion, not retrieval quality across millions of chunks, and it says nothing about whether your answers are right for your users. For that you need your own documents and your own answer key; the guide to eval data explains why nobody can publish that for you. Scanned pages with no text layer are not covered here; the document files include a PDF with no text layer for testing that OCR fallback exists.
Questions
What is a RAG test corpus?
Documents with known contents, used to check that a retrieval pipeline reads, chunks and retrieves them correctly. Knowing every answer in advance turns a wrong answer into a pointer at one stage of the pipeline.
Why does my RAG pipeline give wrong answers from correct documents?
Often the text was extracted wrongly first: interleaved columns, lost table headers, indexed hidden text, dropped footnotes. The model answers faithfully from text that was already wrong.
How do I test a RAG pipeline?
Test extraction before retrieval. Index awkward documents, ask questions you know the answers to, and compare with a plain-text control index.
Is there a free RAG evaluation dataset I can download?
Yes, this one: nine documents in six formats, a 15-question answer key and a control, in one ZIP. Free, no sign-up.
Why does pandas read my Excel header wrong?
read_excel takes one header row by default. For a two-row header, pass header=[0, 1] and flatten the result.
Keep going
Synthetic data for AI evals JSONL for evals → Mock API for agents →
More from Learning
Guides and references for test data, file handling and AI evals. All free, no sign-up. See the full hub.