Versions in this module Expand all Collapse all v1 v1.0.0 Jul 24, 2024 Changes in this version + type Token = string + type TokenFilter interface + Filter func(list []Token) []Token + func NewEnglishStemmerFilter() TokenFilter + func NewNormalizerFilter(chars alphabet.Alphabet, pad string) TokenFilter + func NewRussianStemmerFilter() TokenFilter + type Tokenizer interface + Tokenize func(text string) []Token + func NewFilterTokenizer(tokenizer Tokenizer, filter TokenFilter) Tokenizer + func NewNGramTokenizer(nGramSize int) Tokenizer + func NewWordTokenizer(alphabet alphabet.Alphabet) Tokenizer + func NewWrapTokenizer(tokenizer Tokenizer, start, end string) Tokenizer
Go to main page