Documentation
¶
Index ¶
- type WordLevel
- func (wl *WordLevel) GetVocab() (retVal map[string]int)
- func (wl *WordLevel) GetVocabSize() (retVal int)
- func (wl *WordLevel) IdToToken(id int) (string, bool)
- func (wl *WordLevel) Save(dir string, nameOpt ...string) (err error)
- func (wl *WordLevel) TokenToId(token string) (int, bool)
- func (wl *WordLevel) Tokenize(token string) ([]tokenizer.Token, error)
- type WordLevelBuilder
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type WordLevel ¶
type WordLevel struct {
// contains filtered or unexported fields
}
WordLevel is a model for building WordLevel tokenizer
func NewWorldLevelFromFile ¶
NewWordLevelFromFile initializes a WordLevel from file
func (*WordLevel) GetVocabSize ¶
GetVocabSize returns size of vocab.
type WordLevelBuilder ¶
type WordLevelBuilder struct {
// contains filtered or unexported fields
}
WordLevelBuilder is a builder for WordLevel model
func NewWordLevelBuilder ¶
func NewWordLevelBuilder() *WordLevelBuilder
NewWordLevelBuilder creates a WordLevelBuilder with default values
func (*WordLevelBuilder) Build ¶
func (wlb *WordLevelBuilder) Build() *WordLevel
Build builds a WordLevel using configuration
func (*WordLevelBuilder) UnkToken ¶
func (wlb *WordLevelBuilder) UnkToken(unkToken string)
UnkToken set `UNK` token for the vocab
func (*WordLevelBuilder) Vocab ¶
func (wlb *WordLevelBuilder) Vocab(vocab map[string]int)
Vocab set the vocab (token -> id) mapping
Source Files
Click to show internal directories.
Click to hide internal directories.