Documentation
¶
Overview ¶
Package tokenizer provides a fast, pure-Go token counter.
It embeds the OpenAI cl100k_base and o200k_base vocabularies and implements byte-level BPE scoring, so token counts are deterministic and require no network calls.
Index ¶
Constants ¶
const ( CL100KBase = "cl100k_base" O200KBase = "o200k_base" )
Encoding names, matching tiktoken's own naming.
Variables ¶
This section is empty.
Functions ¶
func CountForModel ¶ added in v1.1.2
CountForModel counts tokens in s using the tokenizer appropriate for model. exact is false when model has no known BPE vocabulary and the count is an approximation from cl100k_base.
Types ¶
type BPE ¶
type BPE struct {
// contains filtered or unexported fields
}
BPE is a byte-level byte-pair encoding tokenizer.
var CL100K *BPE
CL100K and O200K are the two embedded vocabularies. CL100K covers the GPT-3.5/GPT-4 era; O200K covers GPT-4o, GPT-4.1, the o-series, and GPT-5.
var Encoder *BPE
Encoder is a ready-to-use byte-level BPE tokenizer for the cl100k_base vocabulary (GPT-3.5/GPT-4 era). Kept for backward compatibility; prefer ForModel for model-accurate counting.
var O200K *BPE
func ForModel ¶ added in v1.1.2
ForModel returns the BPE encoder and encoding name for model, and whether that encoding is a confirmed match. OpenAI models resolve exactly; every other model (Claude, Gemini, local models, etc.) has no published public BPE vocabulary, so it falls back to cl100k_base as an approximation and exact is false.
Source Files