Documentation
¶
Overview ¶
Package tokens estimates how many tokens a piece of text will cost.
The estimate is not a tokenizer. Shipping a real BPE vocabulary would add megabytes of data and tie the artifact to one model family, and the numbers that matter -- the ones in the benchmark -- come from the API's own usage accounting, which is exact. This exists so that a manifest can tell an agent roughly what a section will cost before it asks for it, which is a decision that only needs to be right to within a few percent.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func Estimate ¶
Estimate approximates the token count of a string.
The model is a blend of two signals that fail in opposite directions. Characters-per-token is stable for prose but badly wrong for text full of punctuation, markup or IDs. Word count is stable for structured text but wrong for long words and for scripts without spaces. Taking the larger of the two keeps the estimate from collapsing on either kind of input, and erring high is the right direction: an agent that budgets too much reads the section, one that budgets too little truncates it.
func EstimateChars ¶
EstimateChars approximates tokens from a character count alone, for callers that have already discarded the text.
func EstimateHTML ¶
EstimateHTML approximates the cost of feeding raw markup to a model, which is the number the distilled artifact is measured against.
Markup tokenizes far worse than prose: angle brackets, attribute quoting, class-name soup and minified script all fragment heavily. Treating a megabyte of HTML as if it were a megabyte of English would understate the cost the artifact is avoiding by a wide margin, so raw markup is estimated at a denser ratio.
Types ¶
This section is empty.
Source Files