Affected by 
Documentation
¶
Overview ¶
Package token provides token estimation for LLM context window management.
The primary source of truth for token counts is the model API's Usage field (types.TokenUsage), returned with every LLM response. This package serves as a supplementary estimator used in two scenarios:
Delta estimation — after an LLM call, new messages (assistant reply + tool results) are appended before the next call. The Estimator computes the token cost of these new messages so the engine can decide whether context compression is needed without making an extra API round-trip.
First-round fallback — on the very first round of a session, no prior API Usage is available, so the Estimator provides a full estimate.
The encoding used (cl100k_base) is an approximation. Different model families use different tokenizers, so the numbers will not be exact for non-OpenAI models. This is acceptable because the estimate only needs to be close enough to trigger compression at roughly the right time; over- or under-estimating by a small margin is corrected on the next API call.
Index ¶
- Constants
- type Estimator
- func (e *Estimator) EstimateMessage(msg *chat.Message) int
- func (e *Estimator) EstimateMessages(messages []chat.Message) int
- func (e *Estimator) EstimateReasoningArtifacts(msg *chat.Message) int
- func (e *Estimator) EstimateString(s string) int
- func (e *Estimator) EstimateTools(tools []chat.Tool) int
- func (e *Estimator) Scale() float64
- func (e *Estimator) SetScale(scale float64)
- func (e *Estimator) Unscaled() *Estimator
Constants ¶
const ( MinScale = 0.5 MaxScale = 1.5 )
Scale bounds. cl100k counts Chinese at about one token a character, where Qwen and DeepSeek spend about 0.6, so a scale near the floor is expected for Chinese sessions on those models. Anything outside the bounds says more about a provider's usage report than about its tokenizer.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Estimator ¶
type Estimator struct {
// contains filtered or unexported fields
}
Estimator counts tokens for messages and strings using BPE tokenization. It is intended for incremental (delta) estimation between LLM calls; the authoritative token count comes from the API's Usage response.
cl100k is not the provider's tokenizer, so text counts are multiplied by a scale learned from the provider's own counts (see SetScale). Every budget an estimate is compared with is in provider tokens, so one scaled estimator keeps them all consistent: history loading, the compaction trigger, and compaction's own cut point and summarizer budget.
func NewEstimator ¶
NewEstimator creates a token estimator using the cl100k_base encoding, at scale 1.
func (*Estimator) EstimateMessage ¶
EstimateMessage returns the token count for a single message.
Every field that goes out on the wire has to be counted here, including the ones that are easy to forget. Reasoning content is the expensive one: thinking models (DeepSeek V3.2/V4, MiMo) require the previous turns' reasoning_content to be sent back verbatim, and it is typically several times longer than the visible reply. Omitting it made a 130k context measure as 26k — compaction then cut in the wrong place, freed almost nothing, and ran again the next round. Reasoning content is counted alongside text and tool calls for that reason.
The reasoning artifacts that travel with it are counted too. They are opaque to us but not to the provider's billing: an Anthropic thinking signature is a few hundred base64 characters, and an OpenAI Responses encrypted reasoning item (or an OpenRouter reasoning_details blob) is several kilobytes replayed on every assistant turn. Leaving them out reproduces exactly the under-measurement described above, one replayed turn at a time.
func (*Estimator) EstimateMessages ¶
EstimateMessages returns the estimated token count for a slice of messages. Prefer using API Usage for the full context and this method only for deltas.
func (*Estimator) EstimateReasoningArtifacts ¶
EstimateReasoningArtifacts counts the opaque provider state replayed with an assistant turn. The signature carries a protocol tag that is stripped before it goes out, so only the payload after the tag is billed; the metadata is sent as-is, one JSON value per namespace key.
func (*Estimator) EstimateString ¶
EstimateString returns the token count for a string using BPE tokenization.
func (*Estimator) EstimateTools ¶
EstimateTools returns the token cost of the tool schemas sent with every request. They are part of the prompt the provider bills. They are not part of the compaction trigger, which estimates messages only; use this for diagnostics and request-budget clamping, not for ShouldCompact.
Source Files