Documentation
¶
Overview ¶
Package textnorm folds text into one canonical form for pattern matching and maps every folded byte back to the original text, so a detector can match the folded form and still redact the exact original bytes.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func IsHidden ¶
IsHidden reports whether r is an invisible rune that Fold removes: zero-width and joiner characters (U+200B to U+200F, U+2060 to U+2064, U+FEFF, U+180E, U+034F), the soft hyphen U+00AD, bidi controls (U+202A to U+202E, U+2066 to U+2069), tag characters (U+E0000 to U+E007F) and the supplementary variation selectors (U+E0100 to U+E01EF).
Types ¶
type Text ¶
type Text struct {
// Original is the text as given.
Original string
// Folded is the canonical form that patterns run on.
Folded string
// contains filtered or unexported fields
}
Text is a string and its folded form. Folded is NFKC normalised and lower case, has invisible runes removed (see IsHidden), has Cyrillic and Greek lookalikes inside Latin words replaced by their Latin letters, and has every whitespace run collapsed to one space. Build it with Fold.
func (Text) Hidden ¶
Hidden returns the number of removed runes that hid something: every invisible rune Fold removed except a leading byte order mark, joiners in emoji sequences and non-Latin words, subdivision flag tags, and the first non-joiner, LRM and RLM of a run (and an immediate repeat of one) that do not sit between two Latin letters.
func (Text) HiddenAt ¶
HiddenAt returns the original byte offset of every rune Hidden counts, in order.
func (Text) Lines ¶
Lines widens the original byte range [start, end) to the whole lines it touches: from just after the preceding '\n' to just before the next one. A range that ends with a '\n' does not pull in the following line. It binary-searches a newline index, so many calls on one long line stay cheap.
type Visible ¶
type Visible struct {
// Original is the text as given.
Original string
// Text is Original without its invisible runes.
Text string
// contains filtered or unexported fields
}
Visible is a string with its invisible runes (see IsHidden) removed and a map back to the original. Unlike Text it keeps case, width and whitespace, for detectors whose formats depend on them, such as secret tokens. Build it with Strip.
Source Files