Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
View Source
var Stopwords = tokenizer.BuildStopWords(
"a", "an", "the",
"about", "above", "across", "after", "against",
"along", "among", "around", "at",
"before", "behind", "below", "between", "beyond",
"by", "down",
"dur",
"except", "for", "from",
"in", "into", "near", "of", "off", "on", "onto", "out",
"over", "through", "to", "toward",
"under", "until", "up", "upon",
"with", "within", "without",
"and", "as", "because", "but",
"either", "if", "neither", "nor", "or",
"since", "so", "than", "that", "though",
"unless", "when", "while", "yet",
"he", "her", "him",
"hi",
"i",
"it",
"me", "my", "our", "she",
"their", "them", "they",
"us", "we", "who", "you", "your",
"am", "are",
"be",
"been", "can", "could", "did",
"do",
"done", "get", "got", "had",
"ha",
"have",
"hav",
"is", "let", "may", "might", "must",
"need", "shall", "should",
"wa",
"were", "will", "would",
"all", "also", "any", "both", "each",
"even", "every", "few", "many", "more",
"most", "much", "no", "not", "other",
"own", "same", "some", "such",
"again", "already", "also", "back", "far",
"further", "here", "how", "just", "never",
"now", "off", "often", "only", "quite",
"rather", "really", "still", "then",
"ther",
"thi",
"tho",
"thu",
"too", "very", "well",
"what", "when",
"wher",
"which", "who", "why",
)
Functions ¶
func Tokenizer ¶
Tokenizer lowers tokens and strips a small set of inflectional suffixes. Deliberately light and lossy, tuned for recall. Use a Snowball port if you need linguistic accuracy.
Types ¶
This section is empty.
Click to show internal directories.
Click to hide internal directories.