Documentation
¶
Overview ¶
Package searchtext is the tokenizer of tool search, the one `tokenize` of packages/coding-agent/src/extensions/tool-search/tool.ts and packages/durable/src/search.ts, which Pi keeps as two copies: Unicode word segmentation with identifier boundaries and English stop words and singular forms. The coding agent's ranker (coding/extension/builtin/toolsearch) and durable's (durable/internal/search) both call it.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func Tokenize ¶
Tokenize is Unicode word segmentation with identifier boundaries and English stop words and singular forms: the text is NFC-normalized, split at camelCase boundaries, lowercased, reduced to letters, marks and numbers, and split into the word-like segments of the ICU word break rules (internal/wordsegmenter, the Intl.Segmenter of Pi); plain ASCII splits at spaces instead, which gives the same words. ASCII words lose their stop words and plural endings.
pig additive (D92): a Piglet Binary built with pig_strip_word_dictionaries has no ICU dictionary data, so each run of Han, Hiragana, Katakana, Thai, Lao, Khmer or Myanmar text is one word, as ICU segments it without the dictionaries.
Types ¶
This section is empty.