The question "how long is this string?" has multiple correct answers depending on who is counting.
Three Different Lengths
Take the string café:
| Measure | Count | Explanation |
|---|---|---|
| Characters (code points) | 4 | c, a, f, é |
| UTF-8 bytes | 5 | é is 2 bytes |
| UTF-16 code units | 4 | each code point ≤ U+FFFF = 1 unit |
| Latin-1 bytes | 4 | é is 1 byte in Latin-1 |
Now take café 😀:
| Measure | Count |
|---|---|
| Characters (code points) | 6 |
| UTF-8 bytes | 9 (café=5, space=1, 😀=4) |
| UTF-16 code units | 7 (😀 requires 2 units as a surrogate pair) |
Java String.length() |
7 (Java uses UTF-16 internally) |
Python len() |
6 (Python 3 counts code points) |
This is not a hypothetical. Different languages and frameworks return different values for the same string.
Where Length Limits Live
A common requirement: "username must be 3–20 characters." But which layer enforces it?
Application code: may use code points, code units, or bytes depending on the language and the library function called.
Database column: VARCHAR(20) in PostgreSQL counts characters (code points). VARCHAR(20) in MySQL with utf8mb4 charset counts characters but with a subtlety: storage takes up to 4 bytes per character, so the column may hit a row-size limit before 20 characters if they are all 4-byte emoji.
Network protocol: HTTP content-length is bytes. A 20-character username that is 40 bytes in UTF-8 will have a content-length of 40, not 20.
Frontend: JavaScript String.length returns UTF-16 code units. For strings containing only BMP characters (U+0000–U+FFFF), this equals the code point count. For emoji and other supplementary characters (U+10000+), it is larger.
The Truncation Bug
The dangerous scenario: a backend limits names to 20 bytes. The frontend limits to 20 characters. A user enters 20 CJK characters — visually 20, but up to 60 bytes in UTF-8. The backend truncates at byte 20, which falls mid-character. The stored value ends with a broken byte sequence. The database either rejects it or stores corrupted data.
The tester's approach: find the limit, find a character category that uses multiple bytes (CJK or emoji), and submit exactly at the limit. Then verify the stored/returned value matches the submitted one exactly.
The Off-By-One in UTF-16 Surrogate Pairs
JavaScript developers sometimes write: if (input.length > 20) return error. This uses String.length, which counts UTF-16 code units. For an emoji like 😀 (U+1F600, above U+FFFF), JavaScript reports length = 2. The user sees one character but the validator counts two.
A user who types 10 emoji gets rejected with "too long" when the display shows only 10 characters. This is a real UX bug with a confusing error message.
Testing Strategy
For any text field with a length limit, your test matrix must include:
- Exactly at the limit using ASCII characters (baseline)
- Exactly at the limit using 2-byte UTF-8 characters (Latin extended, Cyrillic)
- Exactly at the limit using 3-byte UTF-8 characters (CJK)
- Exactly at the limit using 4-byte UTF-8 characters (emoji)
- One over the limit using each character category
The first test passes on any reasonable implementation. Tests 2-4 reveal whether the limit is in bytes or characters. Test 5 reveals how truncation is handled and whether it creates invalid byte sequences.