Documentation
¶
Overview ¶
Package tokens provides LLM token counting.
It prefers an exact BPE tokenizer (tiktoken-go with bundled offline vocabularies, so NO runtime network is required) for the common encodings (cl100k_base for GPT-3.5/4 legacy, o200k_base for GPT-4o/o1/GPT-5, p50k_base for Claude — Anthropic ships no offline tokenizer, so p50k_base is the documented community approximation). When the real tokenizer cannot initialize (unsupported model, missing vocab, sandbox), EstimateTokens provides a deterministic char-count heuristic fallback so a tokenizer failure never breaks context-budget decisions.
Package tokens estimates LLM token counts cheaply and deterministically.
It lives in its own leaf package so both the context manager (compaction thresholds) and the provider adapters (cost tracking) can share one estimator without an import cycle.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func CountMethod ¶ added in v0.1.1
CountMethod describes how CountTokens counts for a model, so the UI can label estimates honestly instead of presenting every number as exact (PHILOSOPHY Principle 3 — forecast vs settlement must be distinguishable).
func CountTokens ¶
CountTokens returns the exact BPE token count for text using the encoding appropriate to model. Falls back to the model-aware heuristic EstimateTokensForModel when the real tokenizer is unavailable — token counting must never fail.
func CountTokensDefault ¶
CountTokensDefault is the model-agnostic convenience: uses the broadest default encoding and falls back to the heuristic on any error.
func EstimateTokens ¶
EstimateTokens approximates LLM token counts when the exact BPE encoder is not available. Calibrated: code ≈3.2-3.8 chars/tok, English prose ≈4 chars/tok, CJK ≈1.2 chars/tok.
func EstimateTokensForModel ¶ added in v0.1.1
EstimateTokensForModel is EstimateTokens with model-family calibration. Model families with an officially documented character→token ratio use it instead of the generic heuristic; unknown models fall back to EstimateTokens.
DeepSeek (official docs, api-docs.deepseek.com/quick_start/token_usage): 1 English char ≈ 0.3 token (≈3.33 chars/token), 1 Chinese char ≈ 0.6 token. The generic heuristic (4 chars/tok English, 0.85 tok/char CJK) overestimates DeepSeek CJK by ~40%, so the official ratios are applied for the family.
Types ¶
type TurnTokenStats ¶ added in v0.1.2
type TurnTokenStats struct {
// TotalTokens is every completion token consumed by the turn.
TotalTokens int
// ProductiveTokens is the subset that produced the deliverable: a final
// answer, or a round that executed a file-mutating tool.
ProductiveTokens int
// WastedTokens is TotalTokens - ProductiveTokens (overhead/exploration).
WastedTokens int
}
TurnTokenStats summarizes a single agent turn's token economy: how many tokens were spent producing the deliverable (answer + file mutations) versus how many were overhead (exploration that changed nothing). The ratio is BroCode's north-star efficiency metric — a high ratio means the agent went straight to the result instead of thrashing through the codebase.
func NewTurnTokenStats ¶ added in v0.1.2
func NewTurnTokenStats(total, productive int) TurnTokenStats
NewTurnTokenStats builds stats from raw counters, deriving WastedTokens and clamping against negative or out-of-range inputs.
func (TurnTokenStats) Percent ¶ added in v0.1.2
func (s TurnTokenStats) Percent() int
Percent returns Ratio as a whole-number percentage (0–100).
func (TurnTokenStats) Ratio ¶ added in v0.1.2
func (s TurnTokenStats) Ratio() float64
Ratio returns ProductiveTokens / TotalTokens in [0,1]. It returns 1.0 when there were no tokens (nothing wasted, nothing to measure).
Source Files