en

package
v1.8.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 4, 2026 License: AGPL-3.0 Imports: 4 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

View Source
var Stopwords = tokenizer.BuildStopWords(

	"a", "an", "the",

	"about", "above", "across", "after", "against",
	"along", "among", "around", "at",
	"before", "behind", "below", "between", "beyond",
	"by", "down",
	"dur",
	"except", "for", "from",
	"in", "into", "near", "of", "off", "on", "onto", "out",
	"over", "through", "to", "toward",
	"under", "until", "up", "upon",
	"with", "within", "without",

	"and", "as", "because", "but",
	"either", "if", "neither", "nor", "or",
	"since", "so", "than", "that", "though",
	"unless", "when", "while", "yet",

	"he", "her", "him",
	"hi",
	"i",
	"it",
	"me", "my", "our", "she",
	"their", "them", "they",
	"us", "we", "who", "you", "your",

	"am", "are",
	"be",
	"been", "can", "could", "did",
	"do",
	"done", "get", "got", "had",
	"ha",
	"have",
	"hav",
	"is", "let", "may", "might", "must",
	"need", "shall", "should",
	"wa",
	"were", "will", "would",

	"all", "also", "any", "both", "each",
	"even", "every", "few", "many", "more",
	"most", "much", "no", "not", "other",
	"own", "same", "some", "such",

	"again", "already", "also", "back", "far",
	"further", "here", "how", "just", "never",
	"now", "off", "often", "only", "quite",
	"rather", "really", "still", "then",
	"ther",
	"thi",
	"tho",
	"thu",
	"too", "very", "well",
	"what", "when",
	"wher",
	"which", "who", "why",
)

Functions

func Fold

func Fold(b []byte) []byte

func HasVowel

func HasVowel(b []byte) bool

func Stemmer

func Stemmer(raw []byte) ([]byte, bool)

func Tokenizer

func Tokenizer(in []byte) iter.Seq[*tokenizer.Token]

Tokenizer lowers tokens and strips a small set of inflectional suffixes. Deliberately light and lossy, tuned for recall. Use a Snowball port if you need linguistic accuracy.

func TokenizerWithoutStopwords

func TokenizerWithoutStopwords(in []byte) iter.Seq[*tokenizer.Token]

func Undouble

func Undouble(raw []byte, keep int) int

Undouble drops one of a trailing doubled consonant after a suffix strip, e.g. runn -> run. Still a prefix, so the result can stay borrowed.

Types

This section is empty.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL