Documentation
¶
Overview ¶
Package textnorm folds the differences between two copies of the same words that nobody means as differences: a run of whitespace where a line happened to wrap, and the typographic characters a book, a plain-text extraction and a Markdown file each spell their own way.
It exists because several guards need the same folding and must not disagree about it. `quotecheck` compares a quotation against the source it claims to come from, `a2check` compares a merged skill's language signals against its sources', and canonizer's `verify.Provenance` asks whether a rule's anchor appears in its source; a quotation that matched in one and not another would be a defect nobody could explain from the output.
That was not hypothetical, and is why this lives here rather than in exegesis. canonizer folded whitespace only, so an anchor copied from a source containing a curly apostrophe blocked there while the identical passage passed `quotecheck`. 231 of the 233 corpus skills contain typographic characters, so the disagreement was the normal case.
Case is deliberately preserved: a quotation differing only in case is a different quotation, and folding it would widen every guard that uses this.
Everything here is pure.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func Fold ¶
Fold reduces text to the form two copies of it are compared in: whitespace runs collapsed to one space, typographic characters folded to ASCII, ends trimmed.
It deliberately does not change case. Case can carry meaning in a quotation, and the caller that wants case-insensitivity can say so; a folder that quietly lower-cased would leave no way back.
Ensures: Fold(Fold(s)) == Fold(s); it is pure.
Types ¶
This section is empty.
Source Files