tokenizer

func TokenizeFile ¶

func TokenizeFile(path string, wordFilter func(string) bool) ([]string, error)

TokenizeFile reads a file and returns its tokens. File contents are read line by line to handle large files efficiently. See TokenizeString for tokenization details.

func TokenizeString ¶

func TokenizeString(s string, wordFilter func(string) bool) []string

TokenizeString splits an input string into word tokens. It can handle natural language text as well as source code. Tokens containing non-ASCII characters are filtered out.

An optional filter function can be provided to include/exclude specific words before tokenization. It should return true to include the word.