Documentation
¶
Overview ¶
Package wordpiece implements BERT-style WordPiece tokenization in pure Go, with no cgo and no external tokenizer runtime. It turns text into the token IDs a BERT-vocabulary model expects: basic tokenization (Unicode cleanup, optional lowercasing and accent stripping, punctuation and CJK splitting) followed by greedy longest-match-first subword segmentation against a fixed vocabulary, wrapped with [CLS] and [SEP].
It is intended to feed a future in-process model stage so the binary can stay CGO_ENABLED=0; no such stage is wired yet. The common alternative, the rust tokenizers library, needs cgo and a statically linked archive.
Index ¶
Constants ¶
const ( DefaultUnknown = "[UNK]" DefaultClassify = "[CLS]" DefaultSeparator = "[SEP]" )
Default special tokens, matching the conventional BERT vocabulary.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Encoding ¶
Encoding is the model-ready result of tokenizing one input. The slices are parallel and always the same length: IDs feeds the model's input_ids, Mask its attention_mask (1 for every real token here, since we do not pad), and TypeIDs its token_type_ids (0 throughout for a single sequence). Tokens holds the matching string pieces, for debugging and tests.
type Option ¶
type Option func(*Tokenizer)
Option configures a Tokenizer at construction.
func Lowercase ¶
Lowercase controls whether input is lowercased and accent-stripped before matching (BERT "uncased" behaviour). Defaults to true.
func MaxCharsPerWord ¶
MaxCharsPerWord sets the length above which a single whitespace-delimited word is emitted as the unknown token rather than segmented. Defaults to 100.
func MaxLen ¶
MaxLen caps the total encoded length including [CLS] and [SEP]; longer input is truncated. Zero (the default) means no limit. Must be at least 2.
func SpecialTokens ¶
SpecialTokens overrides the unknown, classify, and separator token strings looked up in the vocabulary. Defaults to [UNK], [CLS], [SEP].
type Tokenizer ¶
type Tokenizer struct {
// contains filtered or unexported fields
}
Tokenizer encodes text against a fixed vocabulary. Build one with New, Parse, or Load; it is read-only and safe for concurrent use after construction.
func New ¶
New builds a Tokenizer from a vocabulary mapping each token to its ID. The special tokens must be present in the vocabulary.
func Parse ¶
Parse builds a Tokenizer from a vocab.txt stream: one token per line, the zero-based line number being its ID.
Source Files