UTF-8 vs Shift-JIS: Why Text Encoding Bugs Still Happen

A character is an abstract unit of text. Bytes are what files, network requests, and databases store or transmit. An encoding defines how characters become bytes and how bytes become characters again. UTF-8 is a Unicode encoding designed to represent all Unicode characters. Shift-JIS is a legacy Japanese encoding with a different mapping. A screen can show the right character only after the bytes are decoded with the intended encoding.

Mojibake is the garbled text that appears when bytes are decoded with the wrong encoding. A Japanese string saved as Shift-JIS and read as UTF-8 may produce replacement characters or unreadable symbols. The reverse can also happen. The important clue is that the visible text is not the original data; it is the result of an interpretation. To debug the problem, inspect the bytes and the declared or assumed encoding at every boundary.

Many modern systems default to UTF-8, which reduces encoding friction. But files copied from older software, regional spreadsheets, payment systems, label printers, and legacy CSV exports may still use Shift-JIS or another encoding. A file opening correctly in one editor does not prove it is UTF-8. Editors can guess, remember settings, or silently replace invalid bytes. Integrations should rely on explicit encoding rules rather than convenience guesses.

A round-trip test encodes text into bytes and decodes it back. If the decoded result differs from the original, the path lost information or used an incompatible encoding. This is especially important for punctuation, half-width and full-width characters, kana, rare kanji, emoji, and symbols. A sample containing only ASCII letters is a weak test because ASCII survives many encodings. Include the character that actually failed in the real workflow.

CSV files combine text encoding with delimiter, quoting, and spreadsheet behavior. A Japanese CSV may have correct Shift-JIS bytes but still break if the importer assumes UTF-8. Another file may decode correctly but split rows incorrectly because of commas or newlines inside quoted fields. When debugging, separate the questions: are the bytes decoded correctly, are rows parsed correctly, and are columns mapped correctly?

Encoding samples often come from customer names, addresses, order references, or payment descriptions. Replace sensitive values with synthetic text that preserves the failing characters and structure. For example, keep the same kind of kana, kanji, punctuation, and field length without using a real person name. The point is to reproduce the byte behavior, not to expose the original record.

Start with a tiny sample that contains one known failing character. View the bytes as hex under the suspected source encoding. Decode those bytes as UTF-8 and Shift-JIS and compare the result. Then test the full import path with an explicit encoding setting. If the issue disappears with explicit encoding, document the expected input format near the importer. Encoding bugs often return when the rule lives only in someone's memory.