Versions in this module Expand all Collapse all v0 v0.3.0 Aug 25, 2025 Changes in this version type Tokenizer + func (t *Tokenizer) GetToken(id int) string v0.2.0 Aug 22, 2025 v0.1.0 Aug 4, 2025 Changes in this version + type Tokenizer struct + func NewTokenizer() *Tokenizer + func (t *Tokenizer) AddToken(token string) int + func (t *Tokenizer) Decode(tokenIDs []int) string + func (t *Tokenizer) Encode(text string) []int
Go to main page