tokenizer

package

v1.33.13 Latest Latest Go to latest Published: Jan 22, 2026 License: BSD-3-Clause Imports: 14 Imported by: 0

Details

Valid go.mod file
Redistributable license
Tagged version
Stable version
Learn more about best practices

Repository

github.com/weaviate/weaviate

Links

Documentation ¶

Index ¶

Variables
func Tokenize(tokenization string, in string) []string
func TokenizeAndCountDuplicates(tokenization string, in string) ([]string, []int)
func TokenizeWithWildcards(tokenization string, in string) []string
type KagomeTokenizers

Constants ¶

This section is empty.

Variables ¶

View Source

var (
	UseGse          = false // Load Japanese dictionary and prepare tokenizer
	UseGseCh        = false // Load Chinese dictionary and prepare tokenizer
	KagomeKrEnabled = false // Load Korean dictionary and prepare tokenizer
	KagomeJaEnabled = false // Load Japanese dictionary and prepare tokenizer
	// The Tokenizer Libraries can consume a lot of memory, so we limit the number of parallel tokenizers
	ApacTokenizerThrottle = chan struct{}(nil) // Throttle for tokenizers

)

View Source

var Tokenizations []string = []string{
	models.PropertyTokenizationWord,
	models.PropertyTokenizationLowercase,
	models.PropertyTokenizationWhitespace,
	models.PropertyTokenizationField,
	models.PropertyTokenizationTrigram,
}

Optional tokenizers can be enabled with an environment variable like: 'ENABLE_TOKENIZER_XXX', e.g. 'ENABLE_TOKENIZER_GSE', 'ENABLE_TOKENIZER_KAGOME_KR', 'ENABLE_TOKENIZER_KAGOME_JA'

Functions ¶

func Tokenize ¶

func Tokenize(tokenization string, in string) []string

func TokenizeAndCountDuplicates ¶

func TokenizeAndCountDuplicates(tokenization string, in string) ([]string, []int)

func TokenizeWithWildcards ¶

func TokenizeWithWildcards(tokenization string, in string) []string

Types ¶

type KagomeTokenizers ¶

type KagomeTokenizers struct {
	Korean   *kagomeTokenizer.Tokenizer
	Japanese *kagomeTokenizer.Tokenizer
}

Source Files ¶

View all Source files

tokenizer.go

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL