Documentation
¶
Overview ¶
Package tokens is the mangler's token engine: the zero-copy token model plus the opinionated segmentation that produces it.
It is the mechanism the root mangler builds its rules on — it holds no naming policy (no initialisms, no fold tables, no target recipes). Stages that carry policy live in the root package and drive this model through its index-based API (Len/Text/Kind/Span/Rewrite) and the Overlay compaction primitive.
It is internal for now; a public custom-pipeline surface may re-export a curated part of it later.
Index ¶
- func IsCombiningMark(r rune) bool
- type Kind
- type Tokenizer
- type Tokens
- func (t *Tokens) Bounds(i int) (start, end int)
- func (t *Tokens) Kind(i int) Kind
- func (t *Tokens) Len() int
- func (t *Tokens) Overlay(match func(from int) (span int, kind Kind, override string))
- func (t *Tokens) Redeem()
- func (t *Tokens) Rewrite(i int, s string)
- func (t *Tokens) RuneLen() int
- func (t *Tokens) Runes() []rune
- func (t *Tokens) Span(i int) ([]rune, string)
- func (t *Tokens) Text(i int) string
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func IsCombiningMark ¶
IsCombiningMark reports whether r is a Unicode combining mark (categories Mn, Mc, Me).
Combining marks never start a token boundary (they attach to the current run) and are never valid identifier characters (assembly and the fold stage strip them).
Types ¶
type Kind ¶
type Kind uint8
Kind classifies a token produced by segmentation.
The tokenizer emits KindWord, KindNumber and KindSymbol. KindInitialism is set later by a rule overlay (the initialism pass), never by the tokenizer.
type Tokenizer ¶
type Tokenizer struct {
// Separator reports whether a rune is a token separator (elided, marks a boundary). The root mangler injects its
// full default (which also keeps verbalized symbols out of the separator set); a nil Separator falls back to a
// minimal whitespace/non-graphic rule so a bare Tokenizer is still usable.
Separator func(rune) bool
}
Tokenizer splits an UTF-8 string into tokens along opinionated segmentation rules.
Token boundaries ¶
A run of runes becomes a token; a boundary falls at any of:
- one or more consecutive separators (identified by Tokenizer.Separator); separators are elided, not emitted, e.g. "a, b" => [a, b];
- a symbol rune (non-letter, non-digit, non-separator) which becomes its own single-rune token, e.g. "a@b" => [a, @, b];
- a letter↔digit transition, e.g. "oauth2" => [oauth, 2], "v4" => [v, 4];
- a case alternance:
- lower→Upper, e.g. "fooBar" => [foo, Bar];
- an Upper-run→lower, with one-rune lookback, e.g. "HTTPServer" => [HTTP, Server].
Combining marks (Mn/Mc/Me) never start a boundary: they attach to the current token (and are stripped later, in the fold stage).
A script change between letters (e.g. Latin↔Cyrillic) is intentionally not a boundary yet: with ASCII folding on, non-Latin runes are romanized or elided before this point, so it rarely matters.
type Tokens ¶
type Tokens struct {
// contains filtered or unexported fields
}
Tokens is the mutable, pooled token model: a slice of [token] spans over one shared []rune (the only full copy of the input).
Pipeline stages mutate it in place; strings are materialized only at assembly. It is borrowed from a pool for the duration of one mangling and released with Tokens.Redeem; it must not be retained afterwards.
The surface is index-based (the [token] struct stays private): a stage reads with Len/Text/Kind/Span and edits with Rewrite, or merges runs with Overlay.
func Borrow ¶
Borrow borrows a Tokens and loads the input as one shared []rune.
It returns the wrapper by value so it stays on the caller's stack (no heap alloc): the pooled slices and their redeem closures are cached by pools, so nothing here allocates.
func (*Tokens) Bounds ¶
Bounds returns token i's half-open rune-span bounds [start,end) into Tokens.Runes.
It lets an overlay inspect contiguity between adjacent tokens (a separator elided between them leaves a gap) without exposing the [token] struct.
func (*Tokens) Overlay ¶
Overlay runs a single forward-compaction pass over the tokens.
At each position from, match may claim a run of span tokens [from, from+span): the run is merged into one token spanning their combined rune range, retagged to kind and carrying override; span == 0 leaves the token as-is. The merge shrinks the slice in place (matched runs never grow it).
This is the mechanism behind rule overlays such as the initialism pass (which merges break-crossing runs like [ipv,4] into one canonical token); the *matching* policy lives with the rule, the compaction stays here.
func (*Tokens) Redeem ¶
func (t *Tokens) Redeem()
Redeem returns the pooled backings.
The tokens must not be used afterwards.
func (*Tokens) Rewrite ¶
Rewrite replaces the rendered content of token i (transliteration, inflection, verbalization).
func (*Tokens) RuneLen ¶
RuneLen is the number of runes in the shared input (a size hint for assembly).
func (*Tokens) Runes ¶
Runes returns the shared input runes backing the tokens (read-only view — do not mutate).
Together with Tokens.Bounds it lets a rule read the raw runes of any token (e.g. case-insensitive initialism matching) without copying.
Source Files