Documentation
¶
Overview ¶
Package tokens provides LLM token counting.
It prefers an exact BPE tokenizer (tiktoken-go with bundled offline vocabularies, so NO runtime network is required) for the common encodings (cl100k_base for GPT-3.5/4 legacy, o200k_base for GPT-4o/o1). When the real tokenizer cannot initialize (unsupported model, missing vocab, sandbox), EstimateTokens provides a deterministic char-count heuristic fallback so a tokenizer failure never breaks context-budget decisions.
Package tokens estimates LLM token counts cheaply and deterministically.
It lives in its own leaf package so both the context manager (compaction thresholds) and the provider adapters (cost tracking) can share one estimator without an import cycle.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func CountTokens ¶
CountTokens returns the exact BPE token count for text using the encoding appropriate to model. Falls back to the heuristic EstimateTokens when the real tokenizer is unavailable — token counting must never fail.
func CountTokensDefault ¶
CountTokensDefault is the model-agnostic convenience: uses the broadest default encoding and falls back to the heuristic on any error.
func EstimateTokens ¶
EstimateTokens approximates LLM token counts when the exact BPE encoder is not available. Calibrated: code ≈3.2-3.8 chars/tok, English prose ≈4 chars/tok, CJK ≈1.2 chars/tok.
Types ¶
This section is empty.
Source Files