lextoken

func FilterStopwords ¶

func FilterStopwords(terms []string) []string

FilterStopwords returns a copy of terms with stopwords removed. Returns nil when input is nil (preserves nil-vs-empty distinction).

func IsStopword ¶

func IsStopword(word string) bool

IsStopword reports whether word is in the canonical stopword set.

func KeywordTokenize ¶

func KeywordTokenize(query string) []string

KeywordTokenize splits a natural-language query into meaningful search terms by:

1. Lowercasing and splitting on non-alphanumeric characters.
2. Keeping terms ≥ 3 characters.
3. Filtering canonical stopwords (IsStopword).
4. Deduplicating.

It does NOT perform identifier splitting (use Tokenize for that). Mirrors the byte-identical logic from analyze/rank.go:286 (extractKeywordsForBoost) and embeddings/store_keyword.go:76 (ExtractQueryKeywords).

func SplitCamelCase ¶

func SplitCamelCase(s string) []string

SplitCamelCase splits a camelCase or PascalCase identifier into lowercase subwords. Subwords shorter than 2 characters are dropped. Mirrors analyze/context.go:138 (splitCamelCase).

func SplitIdentifier ¶

func SplitIdentifier(s string) []string

SplitIdentifier splits an identifier on underscores, then splits each part by camelCase. Empty parts (from leading/trailing/double underscores) are skipped. Mirrors analyze/context.go:186 (splitIdentifier).

func Tokenize ¶

func Tokenize(query string) []string

Tokenize splits a natural-language or identifier query into lowercase alphanumeric terms for symbol matching. It:

1. Lowercases and strips non-alphanumeric characters from each word.
2. Keeps terms ≥ 3 characters.
3. Also runs identifier splitting (SplitIdentifier) on each word, adding the resulting subwords as additional terms.
4. Deduplicates across both passes.

NOTE: Tokenize does NOT filter stopwords — it is used in analyze/context.go where stopwords are intentional query signals (e.g. "what", "are"). Use KeywordTokenize for stopword-filtered keyword extraction.

Mirrors analyze/context.go:202 (extractQueryTerms).