searchtext

package
v0.7.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 5, 2026 License: MIT Imports: 7 Imported by: 0

Documentation

Overview

Package searchtext is the tokenizer of tool search, the one `tokenize` of packages/coding-agent/src/extensions/tool-search/tool.ts and packages/durable/src/search.ts, which Pi keeps as two copies: Unicode word segmentation with identifier boundaries and English stop words and singular forms. The coding agent's ranker (coding/extension/builtin/toolsearch) and durable's (durable/internal/search) both call it.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func Tokenize

func Tokenize(text string) []string

Tokenize is Unicode word segmentation with identifier boundaries and English stop words and singular forms: the text is NFC-normalized, split at camelCase boundaries, lowercased, reduced to letters, marks and numbers, and split into the word-like segments of the ICU word break rules (internal/wordsegmenter, the Intl.Segmenter of Pi); plain ASCII splits at spaces instead, which gives the same words. ASCII words lose their stop words and plural endings.

pig additive (D92): a Piglet Binary built with pig_strip_word_dictionaries has no ICU dictionary data, so each run of Han, Hiragana, Katakana, Thai, Lao, Khmer or Myanmar text is one word, as ICU segments it without the dictionaries.

Types

This section is empty.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL