Text is not characters. Text is bytes, plus an agreement about what those bytes mean. Almost every encoding bug is a place where two pieces of software disagreed about that agreement.
Two separate things
Unicode is a catalogue. It assigns every character a number called a code point: A is U+0041, é is U+00E9, अ is U+0905, 🙂 is U+1F642. That is all Unicode is, a very large numbered list.
An encoding is a rule for turning those numbers into bytes. UTF-8, UTF-16 and UTF-32 are different rules for the same catalogue.
UTF-8 won, and deservedly. It uses one byte for the ASCII range, so every plain-English ASCII file is already valid UTF-8, and two to four bytes for everything else. It is self-synchronising and has no byte-order ambiguity.
Reading the symptoms
café → café: The text was encoded as UTF-8 and decoded as Latin-1 (or Windows-1252). In UTF-8, é is the two bytes C3 A9. Latin-1 has a character for every byte, so it renders them as two characters: à and ©. This is the single most common encoding bug in existence.
café → caf? or caf□: The opposite direction, the character could not be represented in the target encoding and was replaced. This is lossy. The information is gone, not merely displayed wrongly.
café → caf: Silent truncation at a byte that could not be decoded.
An invisible character at the very start of a file: A UTF-8 byte order mark (EF BB BF), which some editors show as \ufeff. Harmless in a text editor, and a real problem in a CSV header row or a JSON file, where the parser sees an unexpected character before the first field name.
The important insight: in the first case the underlying bytes are still correct. Re-decoding them properly recovers the text. In the second, the data is destroyed.
The two-byte rule of thumb
If one accented character becomes exactly two garbled characters, you are looking at a UTF-8/Latin-1 mismatch and the data is recoverable. If it becomes one replacement character, it is not.
Normalisation: two ways to write the same thing
Unicode sometimes offers more than one way to encode the same visible character. é can be a single code point U+00E9, or it can be e (U+0065) followed by a combining acute accent (U+0301).
They look identical. They are not equal as strings. "café" === "café" can be false.
This causes real problems: macOS traditionally stores filenames in decomposed form (NFD) while Linux and Windows use composed form (NFC), so the same filename can fail to match across systems. Database lookups miss. Deduplication misses.
The fix is normalisation, convert to a canonical form before comparing. NFC (composed) is the right default for storage and comparison on the web.
String length lies
"🙂".length is 2 in JavaScript, because JS strings are UTF-16 and characters outside the Basic Multilingual Plane are stored as surrogate pairs.
It gets worse. A family emoji like 👨👩👧 is several code points joined by zero-width joiners. A flag is two regional indicator symbols. "👨👩👧".length is 8.
If you are counting characters for a limit, a bio field, an SMS, a database column, decide which unit you actually mean:
- Bytes for storage limits.
- Code points for most technical purposes.
- Grapheme clusters for anything a human would call "characters".
Naively slicing a string at a byte or code-unit boundary will split a character in half and produce a replacement character.
Rules that prevent all of this
- UTF-8 everywhere. Files, databases, HTTP headers, HTML meta tags, connection strings. Every mismatch is a bug waiting to happen.
- Declare it explicitly.
<meta charset="utf-8">as the first thing in<head>;Content-Type: text/html; charset=utf-8on responses. - Never guess an encoding in your own code paths. Decide it and enforce it at the boundary.
- Normalise to NFC before comparing or storing user-supplied text.
- Do not add a BOM to UTF-8 files. It solves nothing and breaks parsers.
- In MySQL, use
utf8mb4, notutf8. The latter is a three-byte subset that cannot store emoji, a famous and still-common footgun.
Inspecting the bytes
When text looks wrong, look at the actual characters rather than guessing. The Unicode converter shows the code point of every character and the ASCII converter the values of plain ASCII text, usually enough to identify which of the failures above you are looking at.