processor

package

v0.1.13 Latest Latest Go to latest Published: Nov 9, 2020 License: Apache-2.0 Imports: 2 Imported by: 20

Details

Valid go.mod file
Redistributable license
Tagged version
Stable version
Learn more about best practices

Repository

github.com/sugarme/tokenizer

Links

Open Source Insights

Documentation ¶

Index ¶

type BertProcessing
- func NewBertProcessing(sep, cls PostToken) (retVal *BertProcessing)
- func (bp *BertProcessing) AddedTokens(isPair bool) (retVal int)
- func (bp *BertProcessing) Process(encoding, pairEncoding *tokenizer.Encoding, addSpecialTokens bool) (retVal *tokenizer.Encoding)
type ByteLevelProcessing
- func NewByteLevelProcessing(pretok *pretokenizer.ByteLevel) (retVal *ByteLevelProcessing)
- func (blp *ByteLevelProcessing) AddedTokens(isPair bool) (retVal int)
- func (blp *ByteLevelProcessing) Process(encoding, pairEncoding *tokenizer.Encoding, addSpecialTokens bool) (retVal *tokenizer.Encoding)
type PostToken
type RobertaProcessing
- func DefaultRobertaProcessing() *RobertaProcessing
- func NewRobertaProcessing(sep, cls PostToken) *RobertaProcessing

Constants ¶

This section is empty.

Variables ¶

This section is empty.

Functions ¶

This section is empty.

Types ¶

type BertProcessing ¶

type BertProcessing struct {
	// contains filtered or unexported fields
}

func NewBertProcessing ¶

func NewBertProcessing(sep, cls PostToken) (retVal *BertProcessing)

func (*BertProcessing) AddedTokens ¶

func (bp *BertProcessing) AddedTokens(isPair bool) (retVal int)

func (*BertProcessing) Process ¶

func (bp *BertProcessing) Process(encoding, pairEncoding *tokenizer.Encoding, addSpecialTokens bool) (retVal *tokenizer.Encoding)

Process post-processes input encoding(s) by adding special tokens if specifying.

type ByteLevelProcessing ¶

type ByteLevelProcessing struct {
	// contains filtered or unexported fields
}

func NewByteLevelProcessing ¶

func NewByteLevelProcessing(pretok *pretokenizer.ByteLevel) (retVal *ByteLevelProcessing)

func (*ByteLevelProcessing) AddedTokens ¶

func (blp *ByteLevelProcessing) AddedTokens(isPair bool) (retVal int)

func (*ByteLevelProcessing) Process ¶

func (blp *ByteLevelProcessing) Process(encoding, pairEncoding *tokenizer.Encoding, addSpecialTokens bool) (retVal *tokenizer.Encoding)

type PostToken ¶

type PostToken struct {
	Value string
	Id    int
}

type RobertaProcessing ¶

type RobertaProcessing struct {
	// contains filtered or unexported fields
}

RobertaProcessing is a post post processor for Roberta model

func DefaultRobertaProcessing ¶

func DefaultRobertaProcessing() *RobertaProcessing

DefaultRobertaProcessing creates a RobertaProcessing with default values

func NewRobertaProcessing ¶

func NewRobertaProcessing(sep, cls PostToken) *RobertaProcessing

func (*RobertaProcessing) AddPrefixSpace ¶

func (rp *RobertaProcessing) AddPrefixSpace(addPrefixSpace bool)

AddPrefixSpace set whether the processor will add a prefix space

func (*RobertaProcessing) AddedTokens ¶

func (rp *RobertaProcessing) AddedTokens(isPair bool) int

func (*RobertaProcessing) Process ¶

func (rp *RobertaProcessing) Process(encoding, pairEncoding *tokenizer.Encoding, addSpecialTokens bool) *tokenizer.Encoding

Process post-processes input encoding(s) by adding special tokens if instructed to do so.

Specifically, if addSpecialToken=true, it will add special tokens patterns - Single encoding: <s> Sequence </s> - Pair encoding: <s> SequenceA </s> </s> SequenceB </s>

func (*RobertaProcessing) TrimOffsets ¶

func (rp *RobertaProcessing) TrimOffsets(trimOffsets bool)

TrimOffsets set whether the processor will trim offsets

Source Files ¶

View all Source files

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL