Encoding and Unicode Pack
Encoding bugs do not throw. They produce wrong text and carry on, and the damage usually surfaces in a different system days later. This pack makes each failure mode reproducible in one step.
SHA-256 9d62d955fc122113b2d27872b58f1353da1acc2073a023a63f64d16cbfd50646
What is inside
| File | Format | Size | Expected result |
|---|---|---|---|
| utf8-no-bom.txt | TXT | 99 bytes | The baseline. Everything else is compared against this. |
| utf8-with-bom.txt | TXT | 102 bytes | BOM stripped before parsing, not treated as part of the first value. |
| utf16le.txt | TXT | 162 bytes | Detected as UTF-16 LE from the FF FE mark and decoded correctly. |
| utf16be.txt | TXT | 162 bytes | Detected as UTF-16 BE from FE FF. Assuming little-endian reads garbage. |
| latin1.txt | TXT | 43 bytes | Decoded as Latin-1, not assumed to be UTF-8 and mangled. |
| zero-width-and-rtl.txt | TXT | 94 bytes | Invisible characters detected and stripped or rejected, not stored. |
| line-endings-crlf.txt | TXT | 1.0 KB | CRLF handled without leaving a stray carriage return on every value. |
| line-endings-lf.txt | TXT | 1.0 KB | The LF control for the CRLF case. |
| bom-and-semicolons.csv | CSV | 124 bytes | First column named id, not id. Delimiter sniffed, not assumed. |
| nfc-café.txt | TXT | 142 bytes | Normalised to NFC on the way in. |
| nfd-café.txt | TXT | 142 bytes | Recognised as the same name as the NFC copy, not stored twice. |
| файл-кириллица.txt | TXT | 142 bytes | Non-Latin filename round-trips unchanged. |
| 日本語ファイル名.txt | TXT | 142 bytes | Same, in a script that needs three bytes per character in UTF-8. |
| 🦊-emoji-name.txt | TXT | 142 bytes | Astral-plane characters survive. Length checks count the right unit. |
| unicode-entry-names.zip | ZIP | 563 bytes | Entry names decoded as UTF-8 via the flag bit, not the local code page. |
| unicode-headers.eml | EML | 427 bytes | RFC 2047 encoded-word subject decoded rather than shown raw. |
| quoted-printable.eml | EML | 535 bytes | Quoted-printable body decoded, soft line breaks removed. |
sha256sum -c. Every file is generated from source, carries no third-party copyright, and is free to redistribute.Questions
What is a BOM and why does it break my CSV?
A byte-order mark is an invisible marker at the start of a file declaring its encoding. A parser that does not strip it sees the first column named "id" instead of "id", so the import fails on exactly one field while the header looks perfect in every editor. The pack has the file that reproduces it.
How do I test for mojibake?
Feed the Latin-1 and UTF-8 files through the same path and compare the output. If the UTF-8 one comes back as café something decoded it as Latin-1. If the Latin-1 one comes back with replacement characters something assumed UTF-8. Both are in the pack.
What are NFC and NFD?
Two valid ways to write the same accented text. NFC composes é into one code point; NFD splits it into e plus a combining accent. They render identically and compare as unequal, which is how the same filename ends up stored twice. Both versions are in this pack.
Other packs
Upload Torture Test Pack · Filename Compatibility Pack · Archive and ZIP Security Pack · Document Parser Pack · Image Pipeline Pack · Structured Data Pack · Email Parser Pack