hftokenizer

package
v0.4.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 21, 2026 License: Apache-2.0 Imports: 12 Imported by: 7

Documentation

Overview

Package hftokenizer implements a tokenizer for HuggingFace's tokenizer.json format. This format is used by the HuggingFace Tokenizers library (the "fast" tokenizers) and supports WordPiece (BERT), BPE (GPT-2, RoBERTa), and Unigram models.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func New 

func New(config *api.Config, repo *hub.Repo) (api.Tokenizer, error)

New creates a HuggingFace tokenizer from the tokenizer.json file. It implements a tokenizer.TokenizerConstructor function signature.

Types

type AddedToken 

type AddedToken struct {
	ID         int    `json:"id"`
	Content    string `json:"content"`
	SingleWord bool   `json:"single_word"`
	Lstrip     bool   `json:"lstrip"`
	Rstrip     bool   `json:"rstrip"`
	Normalized bool   `json:"normalized"`
	Special    bool   `json:"special"`
}

AddedToken represents a special token added to the vocabulary.

type Decoder 

type Decoder struct {
	Type     string     `json:"type"`
	Prefix   string     `json:"prefix"`
	Suffix   string     `json:"suffix"`
	Decoders []*Decoder `json:"decoders"`
	Pattern  *Pattern   `json:"pattern"`

	Content       string `json:"content"`
	Replacement   string `json:"replacement"`
	PrependScheme string `json:"prepend_scheme"`
	Split         bool   `json:"split"`
	// contains filtered or unexported fields
}

Decoder represents the decoder configuration.

type Model 

type Model struct {
	Type                    string         `json:"type"`
	Vocab                   map[string]int `json:"-"` // Custom unmarshaling handles both map and array formats
	Merges                  []string       `json:"-"` // Custom unmarshaling handles both string and array formats
	UnkToken                string         `json:"unk_token"`
	ContinuingSubwordPrefix string         `json:"continuing_subword_prefix"`
	MaxInputCharsPerWord    int            `json:"max_input_chars_per_word"`
	FuseUnk                 bool           `json:"fuse_unk"`
	ByteFallback            bool           `json:"byte_fallback"`
	Dropout                 *float64       `json:"dropout"`
	EndOfWordSuffix         string         `json:"end_of_word_suffix"`
}

Model represents the tokenizer model (WordPiece, BPE, or Unigram).

func (*Model) UnmarshalJSON 

func (m *Model) UnmarshalJSON(data []byte) error

UnmarshalJSON implements custom unmarshaling to handle both vocab and merges formats: Vocab formats:

  1. Object format: {"token": id, ...} (WordPiece, BPE)
  2. Array format: [["token", score], ...] (Unigram) - ID is the array index

Merges formats:

  1. Array of strings: ["token1 token2", ...] (standard BPE)
  2. Array of arrays: [["token1", "token2"], ...] (some models like embeddinggemma)

type Normalizer 

type Normalizer struct {
	Type               string       `json:"type"`
	Lowercase          bool         `json:"lowercase"`
	CleanText          bool         `json:"clean_text"`
	HandleChineseChars bool         `json:"handle_chinese_chars"`
	StripAccents       *bool        `json:"strip_accents"`
	Normalizer         *Normalizer  `json:"normalizer"`
	Pattern            *Pattern     `json:"pattern"`
	Normalizers        []Normalizer `json:"normalizers"`
	Content            string       `json:"content"`
}

Normalizer represents the normalizer configuration.

type Pattern 

type Pattern struct {
	Regex  string `json:"Regex,omitempty"`
	String string `json:"String,omitempty"`
}

Pattern for regex-based operations.

type PostProcItem 

type PostProcItem struct {
	SpecialToken *struct {
		ID     string `json:"id"`
		TypeID int    `json:"type_id"`
	} `json:"SpecialToken,omitempty"`
	Sequence *struct {
		ID     string `json:"id"`
		TypeID int    `json:"type_id"`
	} `json:"Sequence,omitempty"`
}

PostProcItem is a tagged union item in TemplateProcessing templates. Exactly one of SpecialToken or Sequence is non-nil.

type PostProcSpecialToken 

type PostProcSpecialToken struct {
	ID     string   `json:"id"`
	IDs    []int    `json:"ids"`
	Tokens []string `json:"tokens"`
}

PostProcSpecialToken defines a special token for TemplateProcessing.

type PostProcessor 

type PostProcessor struct {
	Type          string                          `json:"type"`
	Single        []PostProcItem                  `json:"single"`
	Pair          []PostProcItem                  `json:"pair"`
	SpecialTokens map[string]PostProcSpecialToken `json:"special_tokens"`
	// Sep and Cls are used by BertProcessing and RobertaProcessing.
	// Format in JSON: ["[SEP]", 102] — a [token_string, token_id] tuple.
	Sep json.RawMessage `json:"sep"`
	Cls json.RawMessage `json:"cls"`
}

PostProcessor represents the post-processor configuration. Supports TemplateProcessing, BertProcessing, and RobertaProcessing types.

type PreTokenizer 

type PreTokenizer struct {
	Type           string         `json:"type"`
	AddPrefixSpace bool           `json:"add_prefix_space"`
	PreTokenizers  []PreTokenizer `json:"pretokenizers"`
	Pattern        *Pattern       `json:"pattern"`
	Behavior       string         `json:"behavior"`
	Invert         bool           `json:"invert"`
	Replacement    string         `json:"replacement"`
	PrependScheme  string         `json:"prepend_scheme"`
	Split          *bool          `json:"split"`
}

PreTokenizer represents the pre-tokenizer configuration.

type Tokenizer 

type Tokenizer struct {
	// contains filtered or unexported fields
}

Tokenizer implements the api.Tokenizer interface for HuggingFace tokenizer.json files.

func NewFromContent 

func NewFromContent(config *api.Config, content []byte) (*Tokenizer, error)

NewFromContent creates a HuggingFace tokenizer from tokenizer.json content.

func NewFromFile 

func NewFromFile(config *api.Config, filePath string) (*Tokenizer, error)

NewFromFile creates a HuggingFace tokenizer from a local tokenizer.json file path.

func (*Tokenizer) AddedTokensList 

func (t *Tokenizer) AddedTokensList() []AddedToken

AddedTokensList returns the list of added tokens sorted by ID.

func (*Tokenizer) Config added in v0.3.5 

func (t *Tokenizer) Config() *api.Config

Config returns the HuggingFace tokenizer configuration.

func (*Tokenizer) Decode 

func (t *Tokenizer) Decode(ids []int) string

Decode converts a sequence of token IDs back to text.

func (*Tokenizer) Encode 

func (t *Tokenizer) Encode(text string) []int

func (*Tokenizer) EncodeWithAnnotations added in v0.3.5 

func (t *Tokenizer) EncodeWithAnnotations(text string) api.AnnotatedEncoding

EncodeWithAnnotations returns the encoded text along with requested annotations.

func (*Tokenizer) GetTokenizerType 

func (t *Tokenizer) GetTokenizerType() string

GetTokenizerType returns the model type (WordPiece, BPE, Unigram).

func (*Tokenizer) GetVocab 

func (t *Tokenizer) GetVocab() map[string]int

GetVocab returns the full vocabulary mapping.

func (*Tokenizer) IDToToken 

func (t *Tokenizer) IDToToken(id int) (string, bool)

IDToToken converts a token ID to its string.

func (*Tokenizer) Normalize added in v0.3.5 

func (t *Tokenizer) Normalize(text string) string

Normalize returns the normalization used by the tokenizer (e.g.: BERT lower cases the string).

func (*Tokenizer) SpecialTokenID 

func (t *Tokenizer) SpecialTokenID(token api.SpecialToken) (int, error)

SpecialTokenID returns the ID for a given special token.

func (*Tokenizer) TokenToID 

func (t *Tokenizer) TokenToID(token string) (int, bool)

TokenToID converts a token string to its ID.

func (*Tokenizer) VocabSize 

func (t *Tokenizer) VocabSize() int

VocabSize returns the size of the vocabulary.

func (*Tokenizer) With added in v0.3.5 

func (t *Tokenizer) With(options api.EncodeOptions) error

With applies options to a tokenizer.

type TokenizerJSON 

type TokenizerJSON struct {
	Version       string          `json:"version"`
	Truncation    json.RawMessage `json:"truncation"`
	Padding       json.RawMessage `json:"padding"`
	AddedTokens   []AddedToken    `json:"added_tokens"`
	Normalizer    *Normalizer     `json:"normalizer"`
	PreTokenizer  *PreTokenizer   `json:"pre_tokenizer"`
	PostProcessor *PostProcessor  `json:"post_processor"`
	Decoder       *Decoder        `json:"decoder"`
	Model         Model           `json:"model"`
}

TokenizerJSON represents the structure of HuggingFace's tokenizer.json file.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.