RAG Test Corpus

Clean documents only test the easy path. These are the ones that make a correct model give a wrong answer, because the text it was handed was already wrong.

↓ Download rag-test-corpus.zip11 files · 16.9 KB zipped

SHA-256 b615b69e96935cd6476b00af1f229814d5a7f6daf1519bfb74c93d595f5b6de9

What is inside

Scroll the table sideways for more columns

File	Format	Size	Expected result
answers.jsonl	JSONL	4.7 KB	Load it as an eval dataset. Never index it: it contains every answer.
all-facts.txt	Text	1.5 KB	Index this alone and run the answer key against it. If this scores well and rag/corpus/ does not, the model is fine and the extraction is what is failing.
firmware-3.1-notes-v1.md	Markdown	491 bytes	Near-duplicate retrieval: both versions score almost identically for the question, and only the front matter says which is current. Pair with firmware-3.1-notes-v2.md.
firmware-3.1-notes-v2.md	Markdown	524 bytes	The correct source for the sample interval. A pipeline that drops front matter cannot tell it from v1.
k2-specification.pdf	PDF	1.5 KB	Retrieval returns the spec line (50 degrees C) without the footnote (40 degrees C for older serials), especially where small type or page furniture is filtered out.
price-list.xlsx	XLSX	1.8 KB	Loaders that take row 1 as the header (pandas read_excel default) name the gross column "Unnamed: 2" and read the sub-header row as data, so a question about the gross price often returns the net one.
product-sheet-two-column.pdf	PDF	1.6 KB	Extraction that follows content stream order (pdftotext -raw, for one) interleaves the warranty and returns sentences line by line. Layout-aware extraction (pdftotext in its default mode) keeps them apart.
service-manual.docx	DOCX	1.2 KB	Concatenating runs gives "every 12 months" and "QM-CAL-KIT". Any extractor that puts a space or newline between runs gives "every 1 2 months" and "QM- CAL- KIT".
staff-handbook.pdf	PDF	2.6 KB	Page-level chunking separates "an on-call allowance of" from "EUR 42 per weekend", with the page footer and next page header in between. Neither chunk answers the question on its own.
support-faq.html	HTML	1.1 KB	Tag-stripping extraction (BeautifulSoup get_text, for one) keeps the hidden notice. Only an extractor that applies CSS, such as a browser's innerText, drops it.
support-thread.eml	EML	1.0 KB	Quote stripping, used to de-duplicate email threads, removes the one line that answers the question and leaves "the voltage in my earlier reply is still correct".

Also in the archive: a README.md repeating this table with full digests, a manifest.json for scripting, and a SHA256SUMS you can check with sha256sum -c. Every file is generated from source, carries no third-party copyright, and is free to redistribute.

How to use it: 1. Index files/rag/corpus/ and nothing else. 2. Ask every question in files/rag/answers.jsonl and grade against "expected" and "accept". 3. Build a second index from files/rag/baseline/all-facts.txt alone and run the same questions. If the control scores well and the corpus does not, the model is fine and extraction is failing. Where an answer matches "trap_answer", the pipeline fell for exactly the trap that file contains. Never index answers.jsonl: it contains every answer.

Questions

How do I test a RAG pipeline?

Test extraction before retrieval. Index documents with known awkward layouts, ask questions whose answers you already know, and compare the score with a control built from the same facts as plain text. This pack contains both, plus the answer key.

Is this real company data?

No. Quillmere Instruments is invented, every address is on a reserved domain, and every file is generated from source code, so there is no copyright or personal data in it.

Other packs

Upload Torture Test Pack · Filename Compatibility Pack · Encoding and Unicode Pack · Archive and ZIP Security Pack · Document Parser Pack · Image Pipeline Pack · Structured Data Pack · Email Parser Pack