Documentation
¶
Overview ¶
Package unifold folds Unicode strings into a normalised ASCII-lowercase form suitable for diacritic-insensitive equality and substring matching.
The pipeline is: hand-mapped ligature substitution → NFKD normalisation (golang.org/x/text/unicode/norm) → drop combining marks (unicode.Mn) → ToLower. The hand map covers non-decomposable diacritics that NFKD alone leaves intact (ß, æ, œ, ø, ł, þ, đ, ð, dotless ı, and their upper-case variants).
Scope: reproduces upstream PeeringDB's `unidecode.unidecode(v)` behaviour (rest.py:576) closely enough for filter-value matching in the pdbcompat layer. It is NOT a full Unicode-to-ASCII transliteration library — CJK, Arabic, Hebrew, and other non-Latin scripts pass through untouched so that foreign-language substring matches still work against the folded DB column. Correct matching depends on the SAME fold being applied on both sides of the comparison; this package is the single source of truth.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func Fold ¶
Fold normalises s for case-insensitive, diacritic-insensitive matching.
Fold is total: any UTF-8 input — including invalid UTF-8 bytes, null bytes, control characters, combining marks, ZWJ sequences, RTL text, or strings exceeding 64 KB — returns a string without panicking. The empty string maps to the empty string.
Fast-path contract: an input whose bytes are all within the 7-bit ASCII range (< 0x80) and contain no upper-case letters ('A'-'Z') is returned unchanged without allocating. The set admits digits, punctuation, and control characters in addition to 'a'-'z' — any such input is idempotent under the full fold pipeline (NFKD is a no-op on ASCII, the Mn guard never fires, ToLower is identity on non-letters and lower-case letters). This makes Fold cheap for the common case of folding an already-folded value (e.g. when the sync worker reads back a value it has just persisted into a `_fold` column).
Types ¶
This section is empty.
Source Files