Learning & reference
Synthetic data for AI evals: what you can download, and what you have to build yourself
If you are looking for test data to evaluate an LLM app or an AI agent, most of what you need is generic and you can have it free today. One part is not, and no one can give it to you: the right answers for your own product. This page separates the two, then points at the parts this site provides.
Four different things get called "eval data"
They test different layers, and three of the four do not depend on your app at all.
| Layer | The question | Generic? | Get it |
|---|---|---|---|
| Ingestion | Can the pipeline read the file at all? | Yes | RAG test corpus, 208 test files |
| Tools and actions | Does the agent survive a failing API? | Yes | Mock API that fails on purpose |
| Robustness | Does hostile or odd input break it? | Mostly | Test strings, security fixtures |
| Answer quality | Is the response right for our users? | No | Only from your own traffic. See below. |
Most teams start at the bottom row and skip the top three. That is backwards: if the PDF was never extracted properly, no amount of prompt tuning will fix the answer, and you will spend a week blaming the model for a parser bug.
Why nobody can hand you the answer key
The one kind of eval data that cannot be published, and the reason is not competition.
The answer key is usually called a golden dataset: inputs paired with the correct response. Say you build a support assistant for a bank. One golden example might be:
| Input | Correct behaviour, for this bank |
|---|---|
| "I lost my card" | Offer to freeze it, give the 24 hour line, never ask for the full card number, never offer a loan. |
Every part of that "correct" is this bank's policy, tone and compliance rules. Another bank's correct answer differs, and a travel app's is unrecognisable. So a generic published golden set is a contradiction: the question only has an answer inside one specific product. Anyone offering you a downloadable one is offering someone else's policies.
Compare the files on this site. A PDF with a broken cross-reference table is broken for everyone who opens it. That is why test files can be a shared public resource and golden sets cannot.
What the eval frameworks actually ask of you
Read the docs closely and they all leave the same part to you.
Take DeepEval as a documented example. Its Synthesizer generates test inputs and expected outputs, from
documents you give it (including txt, docx, pdf and md) or from nothing, and it needs an LLM to do that. Its
documentation is explicit that it does not generate actual_output, because that has to
come from your own application. The pattern holds generally: a framework can help you produce questions, and
it runs your app to get answers, but it cannot know what your app should have said.
That is where the generic parts pay off. If the Synthesizer wants documents, it matters what documents you feed it. Clean, well-formed files produce an eval that only tests the easy path. The RAG test corpus gives it documents with the extraction traps real uploads contain.
Where graders quietly lie to you
An eval is only as trustworthy as the thing deciding pass or fail.
Exact match and invisible differences
"café" can be stored two ways, as one character or as "e" plus a combining accent. They look identical and never compare equal. An exact-match grader fails a correct answer. Normalisation, explained
Similarity rewards fluency
A score based on how close the wording is to the expected answer rewards a fluent paraphrase of the wrong fact over a terse correct one. Check a sample by hand before trusting the number.
A model grading itself
When the same model writes the answer and judges it, it can share the same blind spot and pass its own mistake. Use a different model, or a rule, where you can.
The eval file itself
A byte order mark, Windows line endings or one malformed line can make a harness skip rows without saying so. Your pass rate then describes fewer cases than you think. JSONL and how it breaks
What you can take from this site, free
Everything here is generated from source, carries a published SHA-256, and needs no sign-up.
| You need | Here | What it gives you |
|---|---|---|
| Documents to feed a RAG pipeline | RAG test corpus | The same facts in several formats, with extraction traps and an answer key |
| An API for an agent to call | Mock API | Deterministic records, pagination, and failures on demand |
| Fake people and accounts | Synthetic PII | Values from ranges reserved so they cannot reach anyone |
| Adversarial text | Test strings | Unicode, emoji, right-to-left and injection cases |
| Files that break parsers | Security fixtures, archives | Spoofed types, zip slip, zip bombs, truncated files |
| An eval file format that holds up | JSONL for evals | The contract, and broken variants to test your loader |
| Data at any volume | Generator | JSON, NDJSON, CSV or XML of any size, streamed |
Building the part only you can build
A short method for the answer key, since nobody can download it for you.
Start from real questions
Sample from what users actually ask, not what you imagine they ask. Synthetic questions drift towards the easy middle. Strip or replace personal data first, using reserved values.
Write down behaviour, not wording
"Offers to freeze the card, never asks for the full number" can be checked. "The ideal reply is the following paragraph" cannot, because a good answer phrased differently fails.
Keep the hard cases
When something breaks in production, add it. A small set of real failures catches more than a large set of easy passes.
Questions
Where can I download a dataset to evaluate my LLM app?
You can download the inputs but not the answer key. Documents, adversarial strings, fake personal data and mock APIs are generic and free here. The expected output for each input is specific to your product, so you build that from your own traffic.
What is a golden dataset?
Inputs paired with the correct response for your application: the answer key your eval scores against. Because "correct" depends on your product, a golden set is always specific to one app.
Can synthetic data replace real data in AI evals?
For ingestion, tool handling and robustness, yes. For answer quality, no, because that needs examples of what your users actually ask.
Does DeepEval generate the actual outputs?
No. Its documentation states the Synthesizer does not generate actual_output; that comes from your application. It generates inputs and expected outputs, and needs an LLM to do so.
What test data do I need for a RAG pipeline?
Documents in the formats your users upload, including awkward ones, plus a record of which document holds each fact. Test extraction before you tune retrieval.
Keep going
RAG test corpus Mock API for agents → Synthetic PII →
More from Learning
Guides and references for test data, file handling and AI evals. All free, no sign-up. See the full hub.