wordlevel

package

v0.1.8 Latest Latest Go to latest Published: Oct 17, 2020 License: Apache-2.0 Imports: 7 Imported by: 5

Details

This section is empty.

This section is empty.

This section is empty.

type WordLevel struct {
	// contains filtered or unexported fields
}

WordLevel is a model for building WordLevel tokenizer

func NewWordLevel() *WordLevel

NewWordLevel initiates a new WordLevel

func NewWorldLevelFromFile(vocabFile string, unkToken string) (*WordLevel, error)

NewWordLevelFromFile initializes a WordLevel from file

func (wl WordLevel) IdToToken(id int) (string, bool)

IdToToken gets token of given id if existing

func (wl WordLevel) Save(dir string, nameOpt ...string) (err error)

Save saves vocab to a file

func (wl WordLevel) TokenToId(token string) (int, bool)

TokenToId returns id of a given token if existing

func (wl WordLevel) Tokenize(token string) []tokenizer.Token

Tokenize transforms given input to token

func (wl WordLevel) Vocab() map[string]int

Vocab returns a map of model vocab

type WordLevelBuilder struct {
	// contains filtered or unexported fields
}

WordLevelBuilder is a builder for WordLevel model

func NewWordLevelBuilder() *WordLevelBuilder

NewWordLevelBuilder creates a WordLevelBuilder with default values

func (wlb *WordLevelBuilder) Build() *WordLevel

Build builds a WordLevel using configuration

func (wlb *WordLevelBuilder) UnkToken(unkToken string)

UnkToken set `UNK` token for the vocab

func (wlb *WordLevelBuilder) Vocab(vocab map[string]int)

Vocab set the vocab (token -> id) mapping