Documentation
¶
Overview ¶
Package tokenizers creates tokenizers from HuggingFace models.
Given a HuggingFace repository (see hub.New to create one), tokenizers will use its "tokenizer_config.json" and "tokenizer.json" to instantiate a Tokenizer.
Index ¶
Constants ¶
const ( TokBeginningOfSentence = api.TokBeginningOfSentence TokEndOfSentence = api.TokEndOfSentence TokUnknown = api.TokUnknown TokPad = api.TokPad TokMask = api.TokMask TokClassification = api.TokClassification TokSpecialTokensCount = api.TokSpecialTokensCount )
Variables ¶
This section is empty.
Functions ¶
func RegisterTokenizerClass ¶
func RegisterTokenizerClass(name string, constructor TokenizerConstructor)
RegisterTokenizerClass used by Tokenizer implementations.
Types ¶
type AnnotatedEncoding ¶ added in v0.3.5
type AnnotatedEncoding = api.AnnotatedEncoding
AnnotatedEncoding contains tokens with their spans.
type Config ¶
Config struct to hold HuggingFace's tokenizer_config.json contents. There is no formal schema for this file, but these are some common fields that may be of use. Specific tokenizer classes are free to implement additional features as they see fit.
The extra field ConfigFile holds the path to the file with the full config.
type SpecialToken ¶
type SpecialToken = api.SpecialToken
SpecialToken is an enum of commonly used special tokens.
type TokenSpan ¶ added in v0.3.2
TokenSpan represents the byte span of a token in the original text.
type Tokenizer ¶
Tokenizer interface allows one convert test to "tokens" (integer ids) and back.
It also allows mapping of special tokens: tokens with a comman semantic (like padding) but that may map to different ids (int) for different tokenizers.
func New ¶
New creates a new tokenizer from the given HuggingFace repo (see hub.New).
Currently, it only supports "SentencePiece" encoders, and it attempts to download details from the repo files "tokenizer_config.json" and "tokenizer.json".
If it fails to load those files, or create a tokenizer, it returns an error.
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
Package api defines the Tokenizer API.
|
Package api defines the Tokenizer API. |
|
Package bucket implements a streaming tokenizer of sentences into buckets (or batches) of discrete sizes, to minimize padding.
|
Package bucket implements a streaming tokenizer of sentences into buckets (or batches) of discrete sizes, to minimize padding. |
|
Package hftokenizer implements a tokenizer for HuggingFace's tokenizer.json format.
|
Package hftokenizer implements a tokenizer for HuggingFace's tokenizer.json format. |
|
Package sentencepiece implements a tokenizers.Tokenizer based on SentencePiece tokenizer.
|
Package sentencepiece implements a tokenizers.Tokenizer based on SentencePiece tokenizer. |
|
private/protos
Package protos have the Proto Buffer code for the sentencepiece_model.proto file, downloaded from https://github.com/google/sentencepiece/blob/master/src/sentencepiece_model.proto.
|
Package protos have the Proto Buffer code for the sentencepiece_model.proto file, downloaded from https://github.com/google/sentencepiece/blob/master/src/sentencepiece_model.proto. |