Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type BpeDecoder ¶
type BpeDecoder struct {
// contains filtered or unexported fields
}
Allows decoding Original BPE by joining all the tokens and then replacing the suffix used to identify end-of-words by whitespaces
func DefaultBpeDecoder ¶
func DefaultBpeDecoder() *BpeDecoder
DefaultBpeDecoder create a new BpeDecoder with default suffix (`</w>`)
func NewBpeDecoder ¶
func NewBpeDecoder(suffix string) *BpeDecoder
NewBpeDecoder creates a new BpeDecoder
func (*BpeDecoder) Decode ¶
func (bd *BpeDecoder) Decode(tokens []string) string
type WordPieceDecoder ¶
type WordPieceDecoder struct {
// contains filtered or unexported fields
}
WordPieceDecoder takes care of decoding a list of wordpiece tokens back into a readable string.
func DefaultWordpieceDecoder ¶
func DefaultWordpieceDecoder() *WordPieceDecoder
DefaultBpeDecoder create a new BpeDecoder with default suffix (`</w>`)
func NewWordPieceDecoder ¶
func NewWordPieceDecoder(prefix string, cleanup bool) *WordPieceDecoder
NewBpeDecoder creates a new BpeDecoder
func (*WordPieceDecoder) Decode ¶
func (wd *WordPieceDecoder) Decode(tokens []string) string
Source Files
Click to show internal directories.
Click to hide internal directories.