Documentation
¶
Overview ¶
Package tokens provides LLM token counting.
It prefers an exact BPE tokenizer (tiktoken-go with bundled offline vocabularies, so NO runtime network is required) for the common encodings (cl100k_base for GPT-3.5/4 legacy, o200k_base for GPT-4o/o1/GPT-5, p50k_base for Claude — Anthropic ships no offline tokenizer, so p50k_base is the documented community approximation). When the real tokenizer cannot initialize (unsupported model, missing vocab, sandbox), EstimateTokens provides a deterministic char-count heuristic fallback so a tokenizer failure never breaks context-budget decisions.
Package tokens estimates LLM token counts cheaply and deterministically.
It lives in its own leaf package so both the context manager (compaction thresholds) and the provider adapters (cost tracking) can share one estimator without an import cycle.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func CountMethod ¶ added in v0.1.1
CountMethod describes how CountTokens counts for a model, so the UI can label estimates honestly instead of presenting every number as exact (PHILOSOPHY Principle 3 — forecast vs settlement must be distinguishable).
func CountTokens ¶
CountTokens returns the exact BPE token count for text using the encoding appropriate to model. Falls back to the model-aware heuristic EstimateTokensForModel when the real tokenizer is unavailable — token counting must never fail.
func CountTokensDefault ¶
CountTokensDefault is the model-agnostic convenience: uses the broadest default encoding and falls back to the heuristic on any error.
func EstimateTokens ¶
EstimateTokens approximates LLM token counts when the exact BPE encoder is not available. Calibrated: code ≈3.2-3.8 chars/tok, English prose ≈4 chars/tok, CJK ≈1.2 chars/tok.
func EstimateTokensForModel ¶ added in v0.1.1
EstimateTokensForModel is EstimateTokens with model-family calibration. Model families with an officially documented character→token ratio use it instead of the generic heuristic; unknown models fall back to EstimateTokens.
DeepSeek (official docs, api-docs.deepseek.com/quick_start/token_usage): 1 English char ≈ 0.3 token (≈3.33 chars/token), 1 Chinese char ≈ 0.6 token. The generic heuristic (4 chars/tok English, 0.85 tok/char CJK) overestimates DeepSeek CJK by ~40%, so the official ratios are applied for the family.
Types ¶
This section is empty.
Source Files