When you type the letter A, your operating system records the number 65. When you type é, it records either 233 (in Latin-1) or the two bytes 0xC3 0xA9 (in UTF-8). When you type €, it records the three bytes 0xE2 0x82 0xAC in UTF-8 — there is no single-byte representation.
This is encoding: the agreement between humans (characters) and computers (numbers and bytes).
From Characters to Code Points
The Unicode Consortium maintains the definitive list of every character in every writing system. Each character has a code point — a unique number in the range U+0000 to U+10FFFF.
Examples:
A→ U+0041 (LATIN CAPITAL LETTER A)é→ U+00E9 (LATIN SMALL LETTER E WITH ACUTE)中→ U+4E2D (CJK UNIFIED IDEOGRAPH-4E2D)😀→ U+1F600 (GRINNING FACE)(zero-width space) → U+200B (ZERO WIDTH SPACE)
Unicode defines what characters exist, not how they are stored. Storage is the job of encodings.
UTF-8: The Dominant Encoding
UTF-8 stores each code point as 1, 2, 3, or 4 bytes:
| Code point range | Bytes | Example |
|---|---|---|
| U+0000 – U+007F | 1 byte | ASCII chars, A = 0x41 |
| U+0080 – U+07FF | 2 bytes | é = 0xC3 0xA9 |
| U+0800 – U+FFFF | 3 bytes | 中 = 0xE4 0xB8 0xAD |
| U+10000 – U+10FFFF | 4 bytes | 😀 = 0xF0 0x9F 0x98 0x80 |
The elegant design: all ASCII characters are stored identically in UTF-8 as in pure ASCII. A file with only ASCII content is valid UTF-8 by definition. Systems that assume ASCII will work correctly with UTF-8 ASCII-range content — and break on anything outside it.
Why Visual Similarity Lies
The Turkish dotless ı (U+0131) looks almost identical to the Latin i (U+0069) in many fonts. The Cyrillic а (U+0430) is visually indistinguishable from the Latin a (U+0061) in most fonts. These are different code points, different bytes, and different characters — but they render identically on screen.
This matters for:
- Security: domain
pаypal.comusing Cyrillicаis a different domain frompaypal.com - Search: searching for
adminwill not findаdminwith Cyrillicа - Equality checks:
"admin" == "аdmin"is false
What Happens When Encodings Mismatch
If data is written with one encoding and read with another, you get mojibake — garbled text. The string café stored as UTF-8 bytes and read as Latin-1 displays as café. The original bytes are intact; the interpretation is wrong.
The tester's move: submit non-ASCII text, navigate away, return, and compare. If it changed, there is an encoding mismatch somewhere in the pipeline.
Key Takeaways
- Every character has a Unicode code point (a number)
- UTF-8 stores code points as 1-4 bytes; ASCII chars are always 1 byte
- Visual similarity between characters from different scripts is not equality
- Encoding mismatches corrupt data silently — the bug appears only when data is retrieved