tokenize

Overview ¶

Package tokenize counts tokens with a real BPE tokenizer: the cl100k_base vocabulary (the GPT-4 / GPT-3.5 tokenizer), a reproducible public standard. Mesh budgets retrieval to this count and the eval harness measures tokens-to-completion with it, so Gate 1 (beat grep+read on token cost) rests on a genuine token count instead of a chars-per-token estimate.

Sovereignty: the vocabulary is compiled into the binary (the codec ships the merge ranks as Go source), so there is no runtime download. The BPE map is built once, lazily, on the first Count, so a mesh invocation that never budgets tokens pays nothing.

Why cl100k_base: the default coding agent is Claude, whose exact tokenizer is not public, so no count is exact. cl100k is a real, widely-used BPE within a few percent of both Claude and o200k for English and code, counts slightly conservatively (the honest direction for a "we save tokens" claim), and is half the compiled size of o200k. The point is a real BPE, not a chars/4 guess; both retrieval arms in the eval count with this same function, so the Gate-1 ratio is near tokenizer-invariant regardless.