Documentation
¶
Overview ¶
Package textnorm cleans recovered text once, at the graph boundary.
Doing it here rather than per format is the same reasoning that makes every output render from a single graph: a rule applied in the Markdown writer and forgotten in the JSON writer is not a rule, it is a coincidence. Every emitted format inherits whatever happens in this package, and nothing downstream is permitted to re-clean text.
What this defends against ¶
Text extracted from a third-party page goes straight into a model's context. Unicode offers several ways to make a string read one way to a human and another way to a machine, and none of them are visible in a diff:
- Bidirectional overrides can reverse the displayed order of a run, so that text reading "delete nothing" on screen is "gnihton eteled" in the buffer -- or the reverse, which is the direction that matters here.
- Zero-width characters split a word so it survives a human read and a substring search but reaches the model as separate tokens, which is how a filtered phrase gets past a filter.
- Tag characters (U+E0000 block) are invisible everywhere and were used for exactly this in the wild.
None of these appear in legitimate rendered prose in a way that survives removal, so removal is safe and the absence of them is worth guaranteeing.
Why the version number exists ¶
Changing what this package does changes every artifact's content hash, which invalidates every cache everywhere. That has to be a decision, not an accident, so the version is part of the hash input: a consumer can see that the normalizer changed rather than inferring it from a hash that moved for no visible reason.
Index ¶
Constants ¶
const Version = 1
Version identifies the normalization rules. It is an input to every content hash, so bumping it deliberately invalidates caches.
Variables ¶
This section is empty.
Functions ¶
func CleanString ¶
CleanString is the convenience form for callers that do not need the report.
Types ¶
type Result ¶
type Result struct {
Text string
// Removed counts control characters stripped.
Removed int
// HadBidi is true when directional override characters were present. This
// is worth surfacing: unlike a stray zero-width space, a bidi override in
// body copy is almost never accidental.
HadBidi bool
// HadInvisible is true when zero-width or tag characters were present.
HadInvisible bool
}
Result reports what normalization had to remove. A caller that wants to know whether a page was playing games can look at this rather than diffing.
Source Files