tokenizer/

directory
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 17, 2026 License: MIT

Directories

Path Synopsis
Package wordpiece implements BERT-style WordPiece tokenization in pure Go, with no cgo and no external tokenizer runtime.
 Package wordpiece implements BERT-style WordPiece tokenization in pure Go, with no cgo and no external tokenizer runtime.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.