Documentation
¶
Overview ¶
Package tokens estimates Claude token counts using cl100k_base — the same tokenizer Claude approximates for billing.
Mirrors src/services/tokenEstimation.ts and src/utils/tokens.ts — both use a tiktoken cl100k encoder for pre-flight estimates.
The encoder is lazy-initialized on first call. If tiktoken init fails (e.g. offline first run with no cached encoding), Estimate falls back to the chars/4 heuristic so callers always get a usable number.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func Estimate ¶
Estimate returns the cl100k_base token count for s. Falls back to a chars/4 estimate if the encoder isn't available.
func EstimateMany ¶
EstimateMany sums Estimate over a slice. Convenience for callers tallying over a list of message contents.
Types ¶
This section is empty.
Source Files
Click to show internal directories.
Click to hide internal directories.