Why text turns into garbage

Characters are numbers

Computers store characters as numbers rather than pictures. That requires a table saying which number means which character, and the agreement defining that table is a character encoding. Early on there was only a small table built for English, and as each country added its own script in its own table, the tables multiplied. One of the oldest tables, ASCII, defined just 128 characters in 7 bits: upper and lower case English letters, digits, basic symbols and control characters. Korean environments long used EUC-KR and its extension CP949, and Japanese environments used Shift_JIS.

How garbling happens

When the table used to save a file differs from the one used to open it, the same numbers are read as different characters. That is why text appears as unknown symbols or question marks. The content is not damaged but misinterpreted, so reopening with the correct encoding usually restores it. If you save a garbled file again, though, the misread characters become permanent and hard to undo, so copy the original before trying to fix it.

What Unicode solved

Unicode is a single table assigning a unique number to every character in use. With one table, the problem of differing national tables disappears. How many bytes that number occupies in an actual file is a separate question, and the most widely used answer is UTF-8. Unicode contains all 11,172 precomposed Hangul syllables, so it can write characters that EUC-KR's basic set of 2,350 could not.

• Unicode: the table assigning a number to each character
• UTF-8: how those numbers are written as bytes
• Latin letters and digits take one byte; many scripts take three
• Emoji can take four

Characters versus bytes

In UTF-8 characters occupy differing numbers of bytes, so character count and byte count do not match. Ten Korean characters come to around thirty bytes. Where a length-limited field fills up faster in one script than another, the limit is most likely counted in bytes. When the basis is unclear, writing short is safer. The same Hangul character takes 3 bytes in UTF-8 but 2 bytes in EUC-KR or UTF-16, so the file size of the same text varies with the encoding.

The percent signs in a URL

Web addresses may originally contain only Latin letters and certain symbols. Characters outside that, including spaces, are written as a percent sign followed by two hexadecimal digits. That is what the unreadable string pasted from a copied address is: not corruption but conversion following the rule. For example, the Hangul syllable ga is EA B0 80 in UTF-8, three bytes, written as %EA%B0%80 in an address. A space becomes %20.

Diagnosing garbled text step by step

When text is garbled, checking in the order below finds the cause faster than changing settings at random. Most cases are solved simply by matching the encoding on the saving side and the opening side.

• Find out which program saved the original, and in which encoding
• Reopen it in an editor or browser with the encoding switched to UTF-8, CP949 and so on
• If a spreadsheet shows garbled text, open it through the import feature and specify the encoding
• Once solved, standardise on UTF-8 for future saves

When Hangul falls apart into letters

Sometimes a Korean file name appears with its syllables broken into separate jamo letters. This is not an encoding problem but a difference in normalisation. Unicode can write a syllable as one precomposed character or as a sequence of its component letters, and operating systems store file names differently. It often appears when files made on a Mac are moved to another system, and although the names look the same, search and comparison may treat them as different text.

An invisible marker: the byte order mark

Some UTF-8 files begin with the three bytes EF BB BF, known as the byte order mark. Some programs need this marker to recognise UTF-8, while others read it as a character and show a stray symbol before the first line. When a spreadsheet garbles Korean in a UTF-8 file, saving with the marker sometimes fixes it; for programming configuration files, leaving it off is usually safer.