Why does my text show as é or ’?

Bytes are not text

A file contains bytes. Nothing in the file says what those bytes mean, and text is what you get when a program decides how to interpret them. That decision is the encoding, and it is guessed far more often than it is stated.

For the first 128 characters this never matters. ASCII assigns them one byte each, and every encoding in common use agrees, so plain English text survives almost any mistake. The moment an accent, a curly quotation mark, a currency symbol or an emoji appears, the encodings disagree and the guess starts to show.

UTF-8 is now the answer nearly everywhere — over 98% of web pages. It stores those first 128 characters exactly as ASCII did, and everything else in two, three or four bytes. That backward compatibility is why it won, and it is also why encoding bugs are so easy to miss: a file can be tested with English text and fail the first time a real name is entered.

The older encodings still cause problems because files and databases outlive the decisions made when they were created. Windows-1252 and ISO 8859-1 give every byte a single character, which means they can never fail to decode — they will happily turn any byte sequence into something, and that something is what you end up looking at.

Reading the symptom

The specific garbage tells you what happened, which makes diagnosis much faster than guessing.

A single character becoming two or three, with the first being Ã,  or â, means UTF-8 bytes were read as Latin-1 or Windows-1252. This is the most common case by a wide margin. An é is two bytes in UTF-8, and reading each of them as a separate Latin-1 character gives é. Similarly, a curly apostrophe becomes ’ and an em dash becomes —.

A black diamond containing a question mark is the Unicode replacement character. It means a decoder read a byte sequence that is not valid in the encoding it was told to use and substituted a placeholder. Unlike the previous case this is lossy: the original bytes are gone and no amount of re-decoding brings them back.

Plain question marks in place of accented letters usually mean text was converted to an encoding that has no room for those characters — writing UTF-8 into a Latin-1 column, for instance. Boxes with hexadecimal digits inside are different again: the decoding worked and the font has no glyph for the character, which is a display problem rather than an encoding one.

Where it goes wrong

Databases are the most common source, and MySQL has a specific trap. Its encoding named utf8 is not UTF-8 — it stores a maximum of three bytes per character, which covers most of the world and excludes emoji and some CJK characters entirely. The genuine article is called utf8mb4, and any modern schema should use it. Text stored in the wrong one is truncated at the point of writing.

CSV files are the second. Excel on Windows writes CSV in the system codepage rather than UTF-8 unless explicitly told otherwise, and reads UTF-8 as the codepage unless the file carries a byte order mark. This is why exported spreadsheets so reliably arrive with mangled names, and why a byte order mark — normally a nuisance — is genuinely useful in this one context.

HTTP and HTML are a third. A page whose Content-Type header omits the charset leaves the browser guessing, and its guess depends on locale and version. A meta charset tag in the first kilobyte of the document is the fallback that makes this deterministic.

There is a fourth source worth naming because it is the hardest to see: double encoding. Text that has already been mangled once and is then stored, exported and re-imported through the same faulty path gets mangled again, so an é becomes é. Each pass roughly doubles the byte count of the affected characters, and each pass has to be reversed separately. Finding a record with four or five leading à characters is a reliable sign that an import job has been run more than once against data it had already damaged.

The general shape is always the same: some layer in the chain did not state its encoding, so another layer guessed. Fixing it means finding the layer that is silent rather than adding a conversion at the end.

Fixing it, and not causing it

If the pattern is the é kind, the data is recoverable. The bytes are intact and have simply been decoded twice — reversing it means encoding the mangled string back to Latin-1 bytes and decoding those as UTF-8. Do this once at the point of import rather than sprinkling conversions through the application, and verify against a known-bad record.

If the pattern is replacement characters, it is not recoverable from that copy. The bytes were discarded during decoding, and the only fix is to go back to the source and re-import with the correct encoding stated. This is why it is worth diagnosing before running a bulk conversion — a repair pass over already-damaged text produces confident-looking wrong answers.

Preventing it is a matter of being explicit at every boundary. Declare UTF-8 in the HTTP Content-Type header and in a meta tag. Use utf8mb4 for MySQL columns, connections and tables. Open and write files with the encoding named rather than relying on a platform default, which differs between Windows and everything else.

One last thing worth knowing: normalisation is a separate problem that looks similar. An accented letter can be stored as one code point or as a base letter plus a combining accent, and the two render identically while comparing as different strings. If two visually identical names refuse to match, that is normalisation rather than encoding, and the fix is to normalise both to the same form before comparing.

Questions