Documentation
¶
Overview ¶
Package tokens provides tokenizer-aware token estimation for model budget reservations. Estimates only guard the reservation envelope: the provider-reported usage at Finish remains the authoritative settlement, so every estimator errs on the high side.
Envelope contract (P1-06). Fenced reserves against a proven conservative envelope, not an exact in-process tokenizer. The contract every estimator upholds is a one-sided guarantee: for any input the reservation is at or above the true token count a real tokenizer would report, never below it (envelope_corpus_test.go pins this across Latin, CJK, JSON, tool-call, reasoning, multilingual, and over-long inputs). An exact tokenizer (tiktoken, Qwen, SentencePiece, HF) is a per-provider plug-in installed via Register and selected by Executor.Tokenizer(); registering one may only tighten the envelope from above, so the budget invariant holds before and after. Until one is registered the heuristic and conservative estimators are the envelope: provider configuration loading rejects an unrecognized tokenizer name up front, and any runtime resolution failure falls back to conservative.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func Conservative ¶
Conservative estimates tokens with the doubled safety margin used for models whose tokenizer is unknown or non-standard (reasoning models, bespoke tokenizers). Failing safe here means reserving more, never less.
func Heuristic ¶
Heuristic estimates tokens from the text's script mix. CJK and other dense scripts tokenize at roughly one token per character — far above the bytes/4 average that badly under-reserved multilingual prompts — while Latin text keeps the conventional four-bytes-per-token average. The result includes a 10% safety margin and is never below bytes/4.
func Register ¶
Register installs a tokenizer implementation under a provider-configurable name — the seam for exact per-provider tokenizers (tiktoken, Qwen, SentencePiece, HF): a deployment imports its plugin package for side effects before configurations load, and provider configs select it with {"tokenizer": "<name>"}. A registered estimator must uphold the envelope contract from the package documentation: it may only tighten the reservation from above, never estimate below the true token count. Built-in names are reserved and duplicate registrations are rejected so one binary's link order can never silently swap an estimator.
Types ¶
type Estimator ¶
Estimator returns a conservative token estimate for one text chunk.
func ForName ¶
ForName resolves a configured tokenizer name to its estimator. The empty name (an unconfigured descriptor) uses the script-aware heuristic and the explicit "conservative" mode doubles the safety margin; a registered plug-in tokenizer (Register) resolves by its declared name. An unresolved name returns an error: provider configuration loading rejects it up front, and a caller that reaches resolution anyway must fail safe to the conservative estimator (as the kernel invoker does) rather than silently under-reserving.
Source Files