What is a byte-order mark (BOM) and why does it break my CSV?

A BOM is an invisible marker at the very start of a file that declares its encoding. In UTF-8 it is optional and has no byte-order to mark, so it is really just a signature. A parser that does not strip it sees the first column named "id" instead of "id", which is why a CSV import fails on the first header and nowhere else.

Why does my text show as cafÃ© instead of café?

The file is UTF-8 but something decoded it as Latin-1 or Windows-1252. In UTF-8 the character é is two bytes, C3 A9. Read one byte at a time as Latin-1, C3 is Ã and A9 is ©, so you get cafÃ©. It is silent because every byte from 00 to FF is a valid Latin-1 character, so the wrong decoder never errors.

What is the difference between NFC and NFD?

Two ways to write the same text. In NFC, é is the single code point U+00E9. In NFD it is a plain e followed by a combining acute accent, U+0301. They look identical and compare as unequal, so a filename saved on macOS (which uses NFD for filenames) can fail to match the same name typed on Windows.

Why is the length of an emoji not 1?

It depends which length you mean. The family emoji is 7 Unicode code points joined by zero-width joiners, 25 bytes in UTF-8, and 11 UTF-16 code units, which is what JavaScript .length and Java length() return. Users see one character, so a 20-character limit can reject what looks like two emoji.

Should I add a BOM to a UTF-8 file?

Usually not. It breaks shell scripts, JSON parsers, PHP output and anything concatenating files. The one common exception is a CSV meant to be opened in Excel on Windows, which needs the BOM to detect UTF-8 rather than guessing the system code page.

Is UTF-16 bigger than UTF-8?

For English, yes, about double, because every ASCII character takes two bytes. For text in scripts like Chinese, Japanese or Korean, UTF-16 is often smaller, since those characters are three bytes in UTF-8 but two in UTF-16. Neither is a fixed-width encoding.

Learning & reference

Character encodings: UTF-8, UTF-16, the BOM and normalization

Encoding bugs almost never look like encoding bugs. They arrive as a CSV whose first column will not match, a name that is in the database twice, a search that finds nothing, or a length check that rejects a perfectly ordinary comment. Here is what is actually happening, and a file you can download to reproduce each one.

Why encoding bugs are so hard to spot

Three properties conspire to make them silent. Nothing throws, nothing logs, and the damage usually appears several systems away from the cause.

The wrong decoder rarely fails

Every byte from 00 to FF is a valid Latin-1 character. Decoding UTF-8 as Latin-1 therefore cannot error. It just produces wrong text, and keeps going.

Identical text can differ in bytes

café has two legitimate spellings in Unicode that render identically. == says they are different, and so does your unique index.

Length has four different answers

Bytes, code points, UTF-16 units and what a person would count as characters are four different numbers. Your validator and your user are rarely using the same one.

The encodings you will actually meet

Everything else is a rounding error in practice, but these six account for nearly every real-world bug.

Encoding	Bytes per character	Signature	What to know
`UTF-8`	1 to 4	EF BB BF (optional)	The default for the web, JSON, and almost every modern API. ASCII is unchanged, so an ASCII-only file is byte-identical to its UTF-8 version. Self-synchronising: a decoder can recover after a bad byte.
`UTF-16 LE`	2 or 4	FF FE	The internal string format of Windows, Java, C# and JavaScript. Characters beyond U+FFFF take two units, called a surrogate pair. Doubles the size of English text.
`UTF-16 BE`	2 or 4	FE FF	Same encoding, opposite byte order. Rare in files, common in network protocols. Without a BOM you cannot tell the two apart except by guessing.
`ISO-8859-1` (Latin-1)	1	none	256 characters, no more. Cannot represent the euro sign, curly quotes or any non-Western script. Its real significance is that it never rejects input, which is what makes mojibake silent.
`Windows-1252`	1	none	Latin-1 with the unused `80`-`9F` range filled in with the euro sign, curly quotes and dashes. Mislabelled as Latin-1 constantly. This is why a smart quote sometimes appears as a control character.
`US-ASCII`	1	none	The first 128 characters, and a strict subset of UTF-8. Still the only safe assumption for protocol tokens, HTTP header names and email envelope addresses.

Practical rule: declare UTF-8 explicitly everywhere, including Content-Type: text/html; charset=utf-8, your database and connection collation, and the encoding= argument of every file open. A missing declaration is not neutral: something downstream will guess, and its guess depends on the machine's locale.

How mojibake happens, byte by byte

The word café in three encodings. Follow the bytes and the classic corruption becomes obvious.

Stored as	Bytes	Read back as UTF-8	Read back as Latin-1
`UTF-8`	63 61 66 C3 A9	café	cafÃ©
`Latin-1`	63 61 66 E9	caf� (invalid)	café
`UTF-16 LE`	63 00 61 00 66 00 E9 00	c a f é with nulls	same, with nulls

Read it in reverse to diagnose. If you see Ã© the source was UTF-8 read as Latin-1. If you see â€™ the source was a UTF-8 curly apostrophe read the same way. If you see � the source was not UTF-8 and the decoder correctly gave up. And if every second byte is a null, something wrote UTF-16 where UTF-8 was expected.

The byte-order mark

A BOM is the code point U+FEFF at the very start of a file. In UTF-16 it genuinely marks byte order. In UTF-8 there is no byte order to mark, so it is only a signature, and an unwelcome one in most places.

Encoding	BOM bytes	Helps	Hurts
`UTF-8`	EF BB BF	Excel on Windows needs it to open a CSV as UTF-8 rather than the system code page.	Shell scripts (the `#!` is no longer first), JSON parsers, PHP output before headers, concatenated files, and any CSV parser that does not strip it.
`UTF-16 LE`	FF FE	Effectively required. Without it a reader cannot know the byte order.	Tools that assume any text file is UTF-8 read it as two junk characters.
`UTF-16 BE`	FE FF	Same.	Same.

The failure to test for: a BOM in front of a CSV header makes the first column name id, not id. Every other column matches, so the import fails on exactly one field and the header looks perfect in every editor.

NFC and NFD: the same text, twice

Unicode lets an accented character be written either as one code point or as a base letter plus a combining mark. Both are correct, both render identically, and they are not equal.

Form	Code points	UTF-8 bytes	Where you meet it
`NFC` (composed)	4 — `c a f` U+00E9	63 61 66 C3 A9	What most systems produce. The form the W3C recommends for the web.
`NFD` (decomposed)	5 — `c a f e` U+0301	63 61 66 65 CC 81	macOS filenames. Drag a file from a Mac and its name arrives decomposed.

What this breaks: a user uploads café.pdf from a Mac and again from Windows and gets two files, because the names differ in bytes. A search for a composed name misses the decomposed rows. A unique constraint allows both. Fix: normalise to NFC at the boundary, once, on the way in. Do not normalise at comparison time only, or the duplicates are already stored.

"Length" is four different questions

Pick the wrong one and your 20-character limit rejects a two-emoji message, or your VARCHAR(255) truncates mid-character and corrupts the row.

Text	UTF-8 bytes	Code points	UTF-16 units	What a person counts
`café` (NFC)	5	4	4	4
`café` (NFD)	6	5	5	4
👍	4	1	2	1
👨‍👩‍👧‍👦	25	7	11	1

Which one does your language give you? JavaScript .length, Java String.length() and C# .Length all return UTF-16 code units. Python 3 len() returns code points. Go len() returns bytes. Rust .len() returns bytes and .chars().count() returns code points. None of them returns what the user sees, which needs grapheme-cluster segmentation (Intl.Segmenter in JavaScript).

Characters you cannot see

These are legal Unicode, survive a copy and paste, and are invisible in every editor that does not go looking for them.

Zero-width space & joiner

U+200B and U+200D have no width. They break exact-match search, defeat profanity filters, and make two visually identical usernames distinct. The joiner is also load-bearing inside emoji.

Right-to-left override

U+202E reverses display order. A file named invoice‮gnp.txt appears as invoicetxt.png. A classic way to disguise an executable as an image.

Non-breaking space

U+00A0 looks exactly like a space and is not one. trim() leaves it, split(' ') misses it, and a pasted value fails validation for no visible reason.

Homoglyphs

Cyrillic а (U+0430) renders identically to Latin a. The basis of lookalike domains and impersonated usernames. Only a codepoint-level check finds them.

Combining marks

Any number of accents can stack on one base character. Unbounded stacking is the "Zalgo text" effect, and can break layout or fixed-height rows.

Lone surrogates

Half of a surrogate pair, valid in JavaScript strings but not encodable as UTF-8. A common source of "invalid byte sequence" errors when JSON meets a database.

Test it with real files

Every case above has a file. Each one is generated from source, has a published SHA-256, and says on its own page exactly what it is built to catch.

What you want to prove	File
Your reader strips a UTF-8 BOM	utf8-with-bom.txt against utf8-no-bom.txt
A BOM does not poison your first CSV column	bom-and-semicolons.csv
You detect UTF-16 and both byte orders	utf16le.txt and utf16be.txt
Latin-1 input does not become mojibake	latin1.txt
NFC and NFD names are not stored twice	nfc-café.txt and nfd-café.txt
Invisible and RTL characters are handled	zero-width-and-rtl.txt
Emoji in a filename survive the round trip	🦊-emoji-name.txt
Non-Latin filenames survive	Cyrillic and Japanese filename fixtures
Encoded-word headers decode (RFC 2047)	unicode-headers.eml
Archive entry names are not garbled	unicode-entry-names.zip
CRLF and LF are both handled	line-endings-crlf.txt

Inspect a string yourself

Paste anything below to see the four lengths side by side, its normalization form, and any invisible characters it is carrying. Runs entirely in your browser; nothing is sent anywhere.

Keep going

Encoding test files QA test strings → Filename compatibility →