search

type Analyzer ¶

type Analyzer interface {
	Analyze(text string) []Token
}

Analyzer converts raw text into the sequence of index terms used by both indexing and querying. Running the same Analyzer on documents and on queries is what keeps the two sides symmetric. Implementations in this package are stateless and safe for concurrent use by multiple goroutines.

func NewAnalyzer(normalizers []Normalizer, tokenizer Tokenizer, filters []TokenFilter) Analyzer

func NewNGramAnalyzer(n int) Analyzer

func NewScriptAwareAnalyzer() Analyzer

type AnalyzerFunc ¶

type AnalyzerFunc func(text string) []Token

AnalyzerFunc adapts an ordinary function to the Analyzer interface.

func (f AnalyzerFunc) Analyze(text string) []Token

type BM25 ¶

type BM25 struct {
	// K1 tunes term-frequency saturation (typical range 1.2–2.0).
	K1 float64
	// B tunes document-length normalization, from 0 (off) to 1 (full).
	B float64
}

BM25 scores a term/document pairing with the Okapi BM25 ranking function, the modern default for keyword relevance. It improves on plain TF-IDF in two ways: term frequency saturates (the 10th occurrence of a word adds far less than the 2nd, tuned by K1), and the score is normalized by document length so a long document does not rank highly just for being wordy (tuned by B). The inverse-document-frequency factor still rewards rare, discriminating terms over common ones.

The zero value is usable but disables length normalization (B == 0); prefer the parameters installed by NewInvertedIndex when scorer is nil, namely K1 = 1.2 and B = 0.75. A non-positive K1 falls back to 1.2, and B is clamped to [0, 1]. BM25 holds no mutable state and is safe for concurrent use.

func (s BM25) Score(stats TermStats) float64

type Bool ¶

type Bool bool

Bool adapts a boolean to Document as "true" or "false".

func (b Bool) String() string

type Bytes ¶

type Bytes []byte

Bytes adapts a raw byte slice to Document by interpreting it as text, so byte-backed UTF-8 content is indexable just like Text. Non-text bytes yield whatever terms the Analyzer extracts from that byte interpretation.

func (b Bytes) String() string

type ClassWeighted ¶ added in v0.16.0

type ClassWeighted struct {
	// Base scores each pairing before weighting; nil means standard BM25.
	Base Scorer
	// GramWeight multiplies ClassGram matches, clamped to [0, 1].
	GramWeight float64
}

ClassWeighted layers per-class weighting over a base Scorer (#888): a match on the auxiliary gram channel (ClassGram) is multiplied by GramWeight while primary evidence (ClassWord) passes through unchanged. With the dual-class ScriptAwareTokenizer this is what makes a whole-word match dominate the redundant intra-word fragments that only exist to keep infix recall: the fragments still surface a document, but at a fraction of the weight, so "arch" ranks the document containing the word above the ones merely containing the substring.

GramWeight is clamped to [0, 1]; 0 silences the auxiliary channel entirely and 1 restores unweighted scoring. A nil Base falls back to BM25 with the standard parameters, so the zero value scores like the default scorer with the gram channel muted. ClassWeighted holds no mutable state and is safe for concurrent use.

func (s ClassWeighted) Score(stats TermStats) float64

type DiacriticNormalizer ¶

type DiacriticNormalizer struct{}

DiacriticNormalizer folds accents and other nonspacing combining marks so matching is diacritic-insensitive: it decomposes text to Unicode NFD, drops every rune in category Mn (nonspacing mark), then recomposes to NFC. This makes "café" match "cafe", "naïve" match "naive", Greek "Ελλάδα" match "Ελλαδα", and Russian "ёж" match "еж" — across any script that encodes its accents as combining marks (Latin, Greek, Cyrillic, Vietnamese, …). It does not transliterate between scripts and never alters a base letter, so letters that are atomic rather than base-plus-mark — "ß", "ø", "ł" — are left intact.

Because it removes every nonspacing mark, it also strips the optional harakat of Arabic, the niqqud of Hebrew, and the (non-optional) vowel marks of Thai, trading some precision for recall in those scripts. It preserves spacing combining marks (category Mc), so Brahmic base+matra clusters such as Devanagari "भारत" keep their vowels, and the NFC recomposition keeps Hangul syllables and other composed forms stable. It folds independently of case, so its order relative to LowercaseNormalizer does not matter, and it is a no-op for scripts written without combining marks (CJK, Hangul).

func (DiacriticNormalizer) Normalize(text string) string

type Document ¶

type Document interface {
	String() string
}

Document is the indexable payload handed to an Indexer: anything that can serialize itself to the text to be analyzed. It is structurally identical to fmt.Stringer, so any existing Stringer already satisfies it. Implementing String lets a richer record decide which of its fields become searchable (for example concatenating a title and a body) instead of forcing callers to pre-flatten every document into a bare string.

type Duration ¶

type Duration time.Duration

Duration adapts a duration to Document in its canonical Go form (e.g. "1h30m0s").

func (d Duration) String() string

type EmptyTokenFilter ¶

type EmptyTokenFilter struct{}

EmptyTokenFilter drops tokens whose term is empty or all whitespace—cheap insurance after custom normalizers or filters.

func (EmptyTokenFilter) Filter(tokens []Token) []Token

type Float ¶

type Float float64

Float adapts a floating-point number to Document, using the shortest decimal that round-trips back to the same float64 (e.g. 3.14, 1e+20).

func (f Float) String() string

type IndexOption ¶ added in v0.16.0

type IndexOption func(*indexConfig)

IndexOption configures an InvertedIndex at construction time.

func WithPositions() IndexOption

func WithProximityWeight(w float64) IndexOption

type Indexer ¶

type Indexer[S comparable, D Document] interface {
	Index(id S, doc D)
	Delete(id S)
}

Indexer is the write side of a keyword-search engine. Index associates the document identified by id with the searchable terms extracted from doc, and Delete removes a previously indexed document. Indexing the same id again replaces its postings, so Index doubles as update. How text is analyzed and how postings are stored is left to the implementation. S is the caller's document-identifier type and D is the indexable document type.

type Int ¶

type Int int64

Int adapts a signed integer to Document, rendered in base 10.

func (i Int) String() string

type InvertedIndex ¶

type InvertedIndex[S comparable, D Document] struct {
	// contains filtered or unexported fields
}

InvertedIndex is a generic, in-memory inverted index that maps each analyzed term to the documents whose text contains it. It is the default Indexer + Searcher implementation: indexed documents (any Document, via doc.String()) and raw queries pass through the same Analyzer, so index-time and query-time terms stay symmetric.

It splits the two responsibilities of relevance search. The index owns matching and the statistics behind it — per-document term frequencies and document lengths — while a pluggable Scorer (BM25 by default) turns those statistics into the ranking. Search returns hits ordered by descending score.

Terms live in per-class tables (#888): each TokenClass has its own term dictionary, postings, and corpus statistics, so a dual-channel analyzer (ScriptAwareTokenizer's whole words + auxiliary intra-word grams) never mixes the two channels' document frequencies or lengths, and a class-aware Scorer (ClassWeighted) can weight them apart. A single-channel pipeline simply leaves the other class empty at zero cost.

InvertedIndex is safe for concurrent use by multiple goroutines: reads (Search) take a shared lock and writes (Index, Delete) take an exclusive one, and text analysis runs outside the lock to keep the critical section short.

func NewInvertedIndex[S comparable, D Document](analyzer Analyzer, scorer Scorer, opts ...IndexOption) *InvertedIndex[S, D]

func (idx *InvertedIndex[S, D]) Delete(id S)

func (idx *InvertedIndex[S, D]) DeleteMany(ids []S)

func (idx *InvertedIndex[S, D]) Index(id S, doc D)

func (idx *InvertedIndex[S, D]) IndexManyPrepared(items []PreparedItem[S])

func (idx *InvertedIndex[S, D]) IndexPrepared(id S, prepared PreparedDocument)

func (idx *InvertedIndex[S, D]) Prepare(doc D) PreparedDocument

func (idx *InvertedIndex[S, D]) Search(query string) []Result[S]

func (idx *InvertedIndex[S, D]) SearchMatch(query string, opts MatchOptions) []Result[S]

func (idx *InvertedIndex[S, D]) SearchMatchTopK(query string, k int, accept func(id S) bool, opts MatchOptions) []Result[S]

func (idx *InvertedIndex[S, D]) SearchPhrase(query string) []Result[S]

func (idx *InvertedIndex[S, D]) SearchPhraseTopK(query string, k int, accept func(id S) bool) []Result[S]

func (idx *InvertedIndex[S, D]) SearchTopK(query string, k int, accept func(id S) bool) []Result[S]

func (idx *InvertedIndex[S, D]) Stats() (terms, docs int)

type JSONStringValueNormalizer ¶ added in v0.12.0

type JSONStringValueNormalizer struct{}

JSONStringValueNormalizer rewrites text that is a JSON object or array into just the text of its string values, dropping every object field name (key) and every non-string scalar (number, boolean, null). Its purpose is to keep a full-text index free of JSON structure: when a vertex stores a serialized document such as {"role":"admin","score":9,"active":true}, only the human content ("admin") becomes searchable — not the field names ("role", "score", "active") and not the numeric or boolean scalars. String values are emitted in object-key sorted order (arrays keep their order) so the output is deterministic and re-indexing is idempotent.

Text that is not a JSON object or array passes through unchanged: a fast path returns the input verbatim when the first non-space byte is neither '{' nor '[', and any string that starts that way but fails to parse as JSON is also returned verbatim (so a value that merely looks like JSON is still indexed as raw text). A JSON object/array carrying no string values normalizes to the empty string.

Unlike the other normalizers in this package, this one is meant to run on a value in isolation — the document projection of a single stored value — not on a composed document such as "key value", because a key concatenated with a JSON blob is not itself valid JSON and the fast path would return it unchanged. Place it at the projection step, not in a shared analyzer chain.

func (JSONStringValueNormalizer) Normalize(text string) string

type LengthFilter ¶

type LengthFilter struct {
	Min int
	Max int
}

LengthFilter drops tokens whose length in runes is below Min or above Max. A Max of 0 means no upper bound. It removes noise such as stray single letters.

func (f LengthFilter) Filter(tokens []Token) []Token

type LowercaseFilter ¶

type LowercaseFilter struct{}

LowercaseFilter folds each token to lower case. Use it when tokenization runs before case folding; if you already lowercase with a LowercaseNormalizer it is redundant.

func (LowercaseFilter) Filter(tokens []Token) []Token

type LowercaseNormalizer ¶

type LowercaseNormalizer struct{}

LowercaseNormalizer folds text to lower case with Unicode case mapping, so matching is case-insensitive independent of language.

func (LowercaseNormalizer) Normalize(text string) string

type MatchMode ¶ added in v0.16.0

type MatchMode uint8

MatchMode selects how a multi-term query's word-channel terms combine to decide which documents match. It governs membership only — the Scorer still ranks whatever matches — and counts distinct query terms on the primary (word) channel, so the auxiliary intra-word grams a script-aware analyzer emits are evidence for ranking but never change how many terms a document must satisfy. For a single-channel (word-only) analyzer every match is a word match, so the modes reduce to the textbook boolean semantics.

const (
	// MatchAny keeps every document sharing at least one query term (the
	// OR-union). It is the zero value and the package default, because the
	// graph-seeding path Search feeds wants maximal recall.
	MatchAny MatchMode = iota
	// MatchAll keeps only documents containing every distinct query word term
	// (boolean AND), the precision end of the scale.
	MatchAll
	// MatchMinShould keeps documents containing at least MinShouldMatch distinct
	// query word terms — the tunable middle ground between Any and All.
	MatchMinShould
)

type MatchOptions ¶ added in v0.16.0

type MatchOptions struct {
	// Mode selects the boolean combination of the query's word terms.
	Mode MatchMode
	// MinShouldMatch is the minimum number of distinct query word terms a
	// document must contain when Mode is MatchMinShould. It is clamped to
	// [1, number of distinct query word terms]; other modes ignore it.
	MinShouldMatch int
	// Fuzziness is the maximum edit distance (0, 1, or 2) at which a word-channel
	// query term also matches dictionary terms, so a typo like "serach" still
	// finds "search". 0 (the default) disables fuzzy matching. CJK grams are
	// exempt. Expansion per query term is capped at MaxTermExpansions.
	Fuzziness int
	// PrefixTerms, when set, also matches dictionary terms that extend a
	// word-channel query term, so "lan" finds "lantern" (a prefix query). CJK
	// grams are exempt; expansion is capped at MaxTermExpansions.
	PrefixTerms bool
}

MatchOptions configures document membership for a query. The zero value is MatchAny with no term expansion, so an empty MatchOptions reproduces Search's OR-union exactly.

type NGramTokenizer ¶

type NGramTokenizer struct {
	N int
}

NGramTokenizer slides a fixed-width window of N runes across the whole input and emits one token per position, so "search" with N=3 yields "sea", "ear", "arc", "rch". Because it works on runes (not bytes) and never relies on word boundaries, it is the language-independent way to make substring queries and scripts with no whitespace between words (e.g. CJK) matchable, where the word-splitting tokenizers would otherwise emit one giant token.

The window moves over every rune—whitespace and punctuation included—so a caller who wants grams that do not straddle word boundaries should strip or collapse those runes with a Normalizer first. Every emitted term is exactly N runes long, and an input shorter than N yields no tokens. N < 1 is treated as 1 (unigrams), so the zero value tokenizes into single characters rather than misbehaving.

func (t NGramTokenizer) Tokenize(text string) []Token

type Normalizer ¶

type Normalizer interface {
	Normalize(text string) string
}

Normalizer rewrites raw text before tokenization—for example folding case or collapsing whitespace. Implementations in this package are stateless and therefore safe for concurrent use by multiple goroutines.

type NormalizerFunc ¶

type NormalizerFunc func(text string) string

NormalizerFunc adapts an ordinary function to the Normalizer interface.

func (f NormalizerFunc) Normalize(text string) string

type PreparedDocument ¶ added in v0.11.0

type PreparedDocument struct {
	// contains filtered or unexported fields
}

PreparedDocument carries the analyzed tokens of a document so the expensive analysis (doc.String() + tokenization) can run outside the index write lock. Produce one with Prepare and apply it with IndexPrepared or IndexManyPrepared. The zero value is a valid empty document: applying it removes the id, matching Index of a document with no analyzable terms.

type PreparedItem ¶ added in v0.11.0

type PreparedItem[S comparable] struct {
	ID       S
	Prepared PreparedDocument
}

PreparedItem pairs an id with its PreparedDocument for IndexManyPrepared.

type PunctuationFilter ¶

type PunctuationFilter struct{}

PunctuationFilter strips leading and trailing punctuation and symbol runes from each token and drops tokens left empty, while preserving inner marks (so "hello," becomes "hello" but "node-1" and "u.s.a" are untouched). It is the companion to WhitespaceTokenizer; UnicodeTokenizer already drops punctuation.

func (PunctuationFilter) Filter(tokens []Token) []Token

type PunctuationNormalizer ¶

type PunctuationNormalizer struct{}

PunctuationNormalizer replaces every Unicode punctuation or symbol rune with an ASCII space, turning marks such as '.', ',', '。', and '、' into word boundaries independent of language. It does not merge the spaces it introduces; chain SpaceNormalizer after it to collapse the resulting runs and trim the ends. Unlike PunctuationFilter, which only trims marks from the edges of an already-formed token (so "node-1" stays intact), this splits on every mark, so "node-1" becomes "node 1".

func (PunctuationNormalizer) Normalize(text string) string

type Result ¶

type Result[S comparable] struct {
	ID    S
	Score float64
}

Result is a single ranked search hit: the ID of a matching document and the relevance Score the Searcher's Scorer assigned it. Higher scores are more relevant.

type Scorer ¶

type Scorer interface {
	Score(stats TermStats) float64
}

Scorer assigns a relevance weight to a single (query term, document) pairing from the corpus statistics in stats. The Searcher sums a document's per-term scores to rank it, so a Scorer only has to express how one term contributes. Separating this from the index keeps "which documents match" (the index's job) independent of "how relevant each match is" (the Scorer's job), and lets the ranking formula be swapped without touching the postings. A higher score means more relevant; implementations should stay non-negative so that summing across terms is well behaved.

type ScorerFunc ¶

type ScorerFunc func(stats TermStats) float64

ScorerFunc adapts an ordinary function to the Scorer interface.

func (f ScorerFunc) Score(stats TermStats) float64

type ScriptAwareTokenizer ¶ added in v0.16.0

type ScriptAwareTokenizer struct {
	// N is the gram width, for unbounded-script runs and the auxiliary
	// intra-word grams alike. N < 2 is treated as 2.
	N int
}

ScriptAwareTokenizer is the production tokenizer for mixed-script content search (#888). It splits text into script runs and picks the right unit per run, instead of forcing one strategy onto every language the way a pure word or pure n-gram tokenizer does:

• A run of a space-delimited script (Latin, Cyrillic, Greek, digits, …) emits the whole word as a primary token (ClassWord) — the unit Lucene's StandardAnalyzer indexes, and what lets a whole-word match outrank an infix fragment. Words of at least N runes additionally emit every N-rune window as auxiliary tokens (ClassGram), which is what keeps the bigram pipeline's infix recall ("arch" finding "search") and typo tolerance; pair the index with ClassWeighted so this redundant channel stays evidence, not the ranking.
• A run of an unbounded script — one written without spaces between words: Han, Hiragana, Katakana, Hangul, Thai, Lao, Khmer, Myanmar — emits its N-rune windows as primary tokens, exactly the strategy of Lucene's CJKAnalyzer, because there the gram is the word-level unit. A run shorter than N (a lone ideograph) is emitted whole, so single- character words stay searchable.
• Every other rune (whitespace, punctuation, symbols) delimits runs and is dropped, so no gram ever straddles a word boundary and the WhitespaceFilter step of the plain n-gram pipeline is unnecessary.

Run it after the folding normalizers (width, diacritic, lowercase, punctuation, space) — NewScriptAwareAnalyzer wires that pipeline. In particular WidthNormalizer must run first so half-width katakana joins the katakana run it belongs to. N < 2 is treated as 2 (bigrams), so the zero value is the production configuration. It holds no state and is safe for concurrent use.

func (t ScriptAwareTokenizer) Tokenize(text string) []Token

type Searcher ¶

type Searcher[S comparable] interface {
	Search(query string) []Result[S]
}

Searcher is the read side of a keyword-search engine. Search turns a raw query string into the matching documents, each paired with a relevance score, ordered from most to least relevant — the ranked seed candidates a caller picks before a wider graph traversal, where the score can double as a seed's initial weight. Documents that share no analyzed term with the query are omitted, and a query with no analyzable terms yields no results. Ties in score have an unspecified relative order.

type SpaceNormalizer ¶

type SpaceNormalizer struct{}

SpaceNormalizer collapses every run of Unicode whitespace to a single ASCII space and trims the ends. It is useful in front of a WhitespaceTokenizer when the input may contain tabs, newlines, or repeated spaces.

func (SpaceNormalizer) Normalize(text string) string

type StopWordFilter ¶

type StopWordFilter struct {
	// contains filtered or unexported fields
}

StopWordFilter removes tokens that appear in a stop-word set. Matching is case-insensitive (terms and stop words are compared lower-cased), so the filter works whether or not case folding ran earlier.

func NewStopWordFilter(words ...string) *StopWordFilter

func (f *StopWordFilter) Filter(tokens []Token) []Token

type TermStats ¶

type TermStats struct {
	// TF is how many times the term occurs in this document (term frequency).
	TF int
	// DF is how many documents in the corpus contain the term (document
	// frequency); it drives the inverse-document-frequency weight.
	DF int
	// N is the number of documents carrying at least one token of the term's
	// class — the corpus size of the term's channel.
	N int
	// DocLen is the length of this document in tokens of the term's class.
	DocLen int
	// AvgLen is the mean DocLen across the class's N documents, used to
	// normalize DocLen so long and short documents compete fairly.
	AvgLen float64
	// Class is the matching channel the term belongs to, so a class-aware
	// Scorer (ClassWeighted) can weight primary and auxiliary evidence apart.
	Class TokenClass
}

TermStats carries the corpus statistics a Scorer needs to weight one (query term, matching document) pairing. The Searcher fills it in from the index while computing a result; a Scorer treats it as read-only input and holds no state of its own, so a single Scorer is safe for concurrent use.

Every count is scoped to the term's token class (#888): the index keeps separate postings and statistics per class, so DF, N, DocLen, and AvgLen describe only the channel the term matched on. For a single-channel pipeline that scoping is invisible — the class's corpus is the corpus.

type Text ¶

type Text string

Text adapts a plain string to Document so raw text can be indexed without a bespoke wrapper type: idx.Index(id, Text("some words")).

func (t Text) String() string

type Time ¶

type Time time.Time

Time adapts a timestamp to Document, formatted as RFC3339 with nanosecond precision — the same rendering Vertex timestamps use elsewhere.

func (t Time) String() string

type Token ¶

type Token struct {
	// Term is the analyzed text of this token.
	Term string
	// Class labels the matching channel the token belongs to; the zero value
	// (ClassWord) is the primary channel, so tokenizers that predate classes
	// keep their exact behavior.
	Class TokenClass
}

Token is the unit of analyzed text that flows through the package's analysis pipeline: a Tokenizer emits Tokens, TokenFilters rewrite or drop them, and an Analyzer returns the final slice that the index records and queries against.

type TokenClass ¶ added in v0.16.0

type TokenClass uint8

TokenClass separates a token's matching channel (#888). The inverted index keys terms, corpus statistics (document frequency, document length, corpus size), and postings per class, so the two channels never share BM25 statistics, and a class-aware Scorer (ClassWeighted) can weight them apart. Only the constants below are valid: indexing a token with an undefined class panics, and a query token with an undefined class simply matches nothing.

const (
	// ClassWord is the primary channel: whole words of space-delimited
	// scripts, the n-grams of unbounded (CJK-like) script runs — where the
	// gram is the word-level unit — and every token of a single-channel
	// pipeline such as NewNGramAnalyzer. It is deliberately the zero value.
	ClassWord TokenClass = iota
	// ClassGram is the auxiliary channel: intra-word n-gram fragments emitted
	// alongside the word they came from (ScriptAwareTokenizer) so infix and
	// typo-tolerant matching keeps working. Pair with ClassWeighted so this
	// redundant evidence never outranks a whole-word match.
	ClassGram
)

type TokenFilter ¶

type TokenFilter interface {
	Filter(tokens []Token) []Token
}

TokenFilter post-processes a token stream produced by a Tokenizer—dropping, rewriting, or compacting tokens. An Analyzer applies filters in order. Implementations in this package are stateless and safe for concurrent use, but they may reuse the input slice's backing array, so a caller must not keep using the slice it passes in after the call returns.

type TokenFilterFunc ¶

type TokenFilterFunc func(tokens []Token) []Token

TokenFilterFunc adapts an ordinary function to the TokenFilter interface.

func (f TokenFilterFunc) Filter(tokens []Token) []Token

type Tokenizer ¶

type Tokenizer interface {
	Tokenize(text string) []Token
}

Tokenizer splits text into tokens. Implementations in this package are stateless and therefore safe for concurrent use by multiple goroutines.

type TokenizerFunc ¶

type TokenizerFunc func(text string) []Token

TokenizerFunc adapts an ordinary function to the Tokenizer interface.

func (f TokenizerFunc) Tokenize(text string) []Token

type Uint ¶

type Uint uint64

Uint adapts an unsigned integer to Document, rendered in base 10.

func (u Uint) String() string

type UnicodeTokenizer ¶

type UnicodeTokenizer struct{}

UnicodeTokenizer splits text into maximal runs of letters and digits; every other rune (whitespace, punctuation, symbols) is treated as a delimiter and dropped. Tokenizing and stripping punctuation in a single pass makes it the simplest language-independent choice, so it is the default tokenizer.

func (UnicodeTokenizer) Tokenize(text string) []Token

type WhitespaceFilter ¶

type WhitespaceFilter struct{}

WhitespaceFilter drops every token whose term contains a Unicode whitespace rune. Its purpose is to raise the precision of n-gram search: run an NGramTokenizer over text whose words are already separated by single spaces (normalize first with PunctuationNormalizer + SpaceNormalizer) and the only grams that can contain a space are the ones straddling a word or punctuation boundary, e.g. "search" and "engine" yield the cross-boundary gram "h e" for N=3. Dropping them leaves just the intra-word grams, so a document is indexed under its real word content instead of boundary noise: BM25 document lengths reflect the words present, and a query no longer matches on a gram that merely spans two adjacent words. It is language-independent (any whitespace, any script) and a no-op for scripts written without spaces between words (e.g. CJK, Thai), so it is safe to leave in a multilingual pipeline.

func (WhitespaceFilter) Filter(tokens []Token) []Token

type WhitespaceTokenizer ¶

type WhitespaceTokenizer struct{}

WhitespaceTokenizer splits text on runs of Unicode whitespace, leaving every other rune—including punctuation—attached to its token (so "word." stays "word."). Pair it with a PunctuationFilter when that is not what you want.

func (WhitespaceTokenizer) Tokenize(text string) []Token

type WidthNormalizer ¶

type WidthNormalizer struct{}

WidthNormalizer folds full-width and half-width characters to their normal width with Unicode width folding, so the same word written at either width indexes and queries identically — a frequent need for CJK input. Full-width Latin letters, digits, and punctuation collapse to ASCII ("ＴＯＫＹＯ" → "TOKYO", "２０２４" → "2024", "！" → "!"), and half-width katakana expands to full-width ("ｶﾀｶﾅ" → "カタカナ"). It does not fold case, strip accents, or expand compatibility forms such as ligatures; chain LowercaseNormalizer and DiacriticNormalizer for those. Run it early — before PunctuationNormalizer — so width-folded punctuation is then treated as a word boundary. It is a no-op for text that already uses only normal-width characters, ordinary CJK ideographs ("東京") included.

func (WidthNormalizer) Normalize(text string) string