terminal

Overview ¶

Package terminal makes text that came from somewhere else safe to print.

Everything HEY serves — a sender's name, a subject, a filename, a label — was written by somebody else, and a terminal acts on the escape sequences and control characters it is handed. Stripping the sequence outright is what makes the string inert. Defacing its ESC byte only hides the trigger: the payload stays behind as visible debris of somebody else's choosing, and whatever lays the output out then measures that debris as text.

The bidirectional controls go the same way. They do not move the cursor, but they move what the reader sees: a right-to-left override (U+202E) between "invoice" and "fdp.exe" shows a PDF on screen and an executable on disk, and an isolate can swap the order of a sender's name and address. Nothing HEY shows in a single line needs them — an RTL name still reads right-to-left without an explicit override.

Confusables ¶

Past the controls lies text that reads as one thing and is another. The policy, class by class:

Format characters that draw nothing are stripped: the soft hyphen, the zero width space, the word joiner, the byte order mark, the combining grapheme joiner, the Mongolian vowel separator, the invisible mathematical operators and the deprecated format controls next to them. A zero width space in "paypal" or a soft hyphen in "invoicepdf.exe" renders as the honest spelling while being a different string, which is the same class as a bidi override. They are stripped rather than replaced, since a replacement mark is visible debris in a name column. The list is enumerated; it is not the Cf category wholesale, which also holds the joiners below, the Arabic number and verse signs that are visible, and the tag characters a subdivision flag is made of.

The zero width joiner and non-joiner are kept where they can join. They carry text: an emoji family is emoji joined by U+200D, and Persian, Urdu and the Indic scripts write with both. A joiner survives only between two non-ASCII, non-space base characters — the marks on the left base, a virama or an emoji presentation selector, are part of it, but a mark is not a base on its own — and a run of them collapses to one; at the start or end of a string, or next to an ASCII letter — a joiner inside "paypal" — it joins nothing and is dropped.

Combining marks are kept up to eight on a base and the rest of the run dropped. The deepest stacks a writing system produces reach five — a fully pointed Hebrew letter with its shin dot, dagesh, vowel, meteg and cantillation; a Tibetan stack with two subjoined letters and a vowel sign is four — where decomposed Vietnamese, Hindi and the keycap emoji use two, so eight is room for any of them and Zalgo needs dozens to climb out of its cell. Variation selectors are marks and sit under the same cap, so the U+FE0F that makes a heart red stays. A format character that is kept — a tag character, an Arabic number sign — is not a base: it draws nothing of its own, so the marks after it still belong to the letter before it and count against the same cap.

A byte that is not UTF-8 is dropped. Left as it is, a stray 0x85 is a C1 control to a terminal that reads bytes; and it goes the way everything else here goes, stripped rather than replaced, so the output is never longer than the input.

Spaces that are not U+0020 are left alone. A no-break space is ordinary in text that came out of HTML, as are the fixed-width spaces; they render as a space and read as one, and normalizing them would rewrite names for no safety gain.

Homoglyphs — a Cyrillic "а" in a Latin name — are not detected. Without confusable tables and a script heuristic the check cannot tell a forged name from a multilingual one, and the false positives land on real people. What a link's label says against where it goes is the Markdown serializer's business: a label that reads as a URL is written beside its destination rather than collapsed into it (see htmlutil.ToMarkdown).