Documentation
¶
Overview ¶
Package tokenizer provides a fast, pure-Go token counter.
It embeds the OpenAI cl100k_base vocabulary and implements byte-level BPE scoring, so token counts are deterministic and require no network calls.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type BPE ¶
type BPE struct {
// contains filtered or unexported fields
}
BPE is a byte-level byte-pair encoding tokenizer.
var Encoder *BPE
Encoder is a ready-to-use byte-level BPE tokenizer for the cl100k_base vocabulary (GPT-4 family, claude token counting compatible at the UTF-8 bytes level).
Source Files
Click to show internal directories.
Click to hide internal directories.