Encoding Errors in the Wild

Encoding bugs in production all have the same root cause — two parts of the system disagree about how to interpret bytes — but they manifest in dozens of recognizable patterns.

Mojibake: The Classic Tell

Mojibake is the Japanese term for garbled characters from an encoding mismatch. Recognize it by these visual patterns:

The pattern: Ã followed by a lower-code character is a reliable signal of UTF-8 read as Latin-1. If you see é in a string, the original was é.

The Database Charset Bug

The most common source of mojibake in web applications: the database connection charset is misconfigured. The application sends UTF-8; the database stores it as Latin-1 bytes (discarding multi-byte sequences or storing them incorrectly); when retrieved, the bytes are re-interpreted as Latin-1, producing garbled output.

MySQL is particularly prone to this: the default charset was historically latin1, and configuring it correctly for UTF-8 requires setting the column charset, the table charset, the database charset, and the connection charset. Missing any one of these produces mojibake for non-ASCII input.

How to test for it: submit a string containing é, ñ, ü, or any accented Latin character. Retrieve it from the API or navigate to the profile page. If it comes back as é, ñ, or ü, the database is misconfigured.

Emoji Truncation

A subtler bug: the system accepts the submission without error, but the stored value is shorter than what was submitted. The user submits John 😊 (8 UTF-16 code units, 9 UTF-8 bytes). The backend truncates at byte 8, cutting the emoji mid-sequence. The result is John followed by a broken byte, which the database strips silently. The stored value is John — the emoji is gone.

This is a silent truncation — no error, no feedback, just missing data. The user has no idea their name was changed.

Signs to look for during testing:

• Input containing emoji is accepted without error
• The stored/displayed value is shorter than what was submitted
• The missing part is always the end of the string

Byte-Order Marks (BOM)

Some editors and tools add a Byte-Order Mark (BOM) at the start of UTF-8 files. The UTF-8 BOM is the byte sequence 0xEF 0xBB 0xBF. When a file with a BOM is uploaded or pasted, the BOM may appear as the invisible prefix character U+FEFF (ZERO WIDTH NO-BREAK SPACE).

A field that stores a username pasted from a BOM-marked text file will have U+FEFF prepended. String equality checks fail silently. The user cannot log in using the username typed manually because the stored version has an invisible prefix.

HTML Entity Confusion

Another class of encoding-adjacent bugs: HTML entities stored literally instead of as the character they represent.

A form field that stores < when the user typed < has double-encoded the input. When displayed, the page shows < instead of <. When returned from the API, the value is < instead of <. This is not technically an encoding error, but it is a character handling error with similar symptoms.

The reverse also happens: HTML in user input stored as literal <b>text</b> and rendered without escaping — an XSS vulnerability. The correct behavior: store the raw characters, escape on output.

Recognizing Encoding Bugs in Bug Reports

When users report these symptoms, think encoding:

• "My name has weird characters / question marks / boxes"
• "It worked before but now my profile name is different"
• "I can't log in even though I'm using the right password" (if the password field stores encoding-corrupted values)
• "The search doesn't find my name even when I type it exactly"
• "Some characters disappeared from my bio after I saved it"

The pattern: data that looks fine going in looks different coming out. The system accepted the input but did not preserve it faithfully.