Invisible Unicode Characters: The Zero-Width Space and Its Relatives

The characters, and where they come from

How they get into your data

Very few of these are typed on purpose. They travel. A user copies an address from a PDF, whose text layer inserts zero-width spaces at line-break opportunities, and pastes it into a form. A spreadsheet is saved as "CSV UTF-8" by Excel, which prepends a BOM that the importer reads as part of the first column name. A product description is written in Word, where the typographer's habit of a non-breaking space before a unit becomes U+00A0 in the export. A support agent pastes an id from a chat client that decorated it with a word joiner to stop it wrapping.

None of it is visible at any of those steps, which is why the same character ends up in a database, an API response and a log file before anyone notices. The symptoms are indirect: a lookup that fails for one record, a CSV with a column called email, a search that cannot find text that is plainly on the page, a filename that exists on the Mac but "does not exist" on the Linux server.

The ones that lie about order

The bidirectional control characters exist so that Arabic and Hebrew can be mixed with Latin text correctly. U+202E RIGHT-TO-LEFT OVERRIDE forces everything after it to display right-to-left until a pop character. Placed inside a filename, invoice‮fdp.exe is stored as invoice[RLO]fdp.exe but displays as invoiceexe.pdf — an executable wearing a PDF's name.

The same trick inside source code is the "Trojan Source" class of vulnerability (CVE-2021-42574): a comment or string containing bidi overrides can make code that a reviewer reads as harmless compile as something else, because the editor renders the reordered text while the compiler consumes the stored order. Modern GitHub, VS Code and most compilers now warn on these characters, and a code review tool that shows code points would have caught it from the start. Treat any bidi control outside genuinely right-to-left content as hostile.

Lookalikes: visible but wrong

A related family is not invisible at all — it looks exactly like something else. Cyrillic а (U+0430), Greek ο (U+03BF) and Latin a and o are indistinguishable in most fonts. A domain like раypal.com with two Cyrillic letters is the classic homograph attack, and the same substitution in a username or package name gets past a human reviewer every time. Browsers defend domains by refusing to display a name that mixes scripts within a label; you can apply the same rule to any identifier: a word whose letters come from more than one script is suspect. Punctuation has lookalikes too — curly quotes for straight ones, en dashes and minus signs for hyphens — which are harmless in prose and fatal in a config file or a command line.

The other way two strings differ

Even with no invisible characters, café can be two different strings: U+00E9 (one code point, "precomposed") or U+0065 U+0301 (e plus a combining acute accent). They render identically, they are canonically equivalent under Unicode, and === says they are different. Normalization fixes this by picking one form. NFC composes where possible and is what you should store; NFD decomposes; the K forms additionally fold compatibility characters, turning fi into fi and full-width Ａ into A, which is right for loose identifier matching and wrong for content.

The place this bites in practice is macOS filenames. HFS+ stored names in a decomposed form, so a file created on a Mac and synced to Linux can have a name that does not byte-match the composed name in your database. Normalize to NFC on the way in — str.normalize('NFC') in JavaScript, unicodedata.normalize('NFC', s) in Python, Normalizer.normalize(s, Form.NFC) in Java — and the problem does not arise.

Finding and removing them

To see what is there, print code points rather than characters: [...s].map(c => c.codePointAt(0).toString(16)) in JavaScript, [hex(ord(c)) for c in s] in Python, xxd or cat -A on a file. To remove the invisible ones in bulk, use Unicode property classes rather than a hand-written list:

// JavaScript: strip format characters, normalise spaces, compose accents
s.replace(/\p{Cf}/gu, '')          // ZWSP, ZWJ, BOM, bidi controls, soft hyphen …
 .replace(/[  -   　]/g, ' ')
 .normalize('NFC');

# Python
import re, unicodedata
s = ''.join(c for c in s if unicodedata.category(c) != 'Cf')
s = unicodedata.normalize('NFC', s)

Do this at the boundary — form submission, file import, API ingestion — and once. Be selective about ZWJ and ZWNJ if you handle Persian, Arabic, Indic scripts or emoji, where they are meaningful; stripping them corrupts the text. And keep the raw input somewhere for a while, because a cleaning step that is too aggressive is itself a data-loss bug.

Make the invisible visible first

Every problem in this guide is trivial once you can see the character and baffling until you can. A string is a sequence of code points, not of glyphs, and the tools that show you the code points — a hex dump, a code-point list, a diff that marks invisible characters — turn hours of "but they look the same" into a two-minute fix. Add one to the toolbox before the next bug report arrives.

Frequently asked questions