tokens

Overview ¶

Package tokens estimates how many LLM tokens a given string costs.

We deliberately avoid a real tokenizer dependency (tiktoken-go, BPE vocab download, ~5MB binary bloat, init cost) because the call log only needs *trends* — comparing one invocation to another, finding heavy outputs, plotting cost over time. A small heuristic is enough for that and stays sub-microsecond.

The heuristic:

• ASCII bytes (< 0x80) cost ≈ 1/4 token each (matches the well-known "≈4 characters per token" rule for English/code).
• Non-ASCII runes (Cyrillic, CJK, emoji, ...) cost ≈ 1 token each (cl100k and similar BPEs split most Cyrillic chars into 1-2 subwords; using 1.0 gives an honest mid-range estimate that errs slightly conservative for English-only text and slightly low for CJK).

If you ever need exact counts for billing, swap this for a real tokenizer. The call sites take an int64, so the signature is stable.