tokenizer

package

v2.1.0 Latest Latest Go to latest Published: Oct 6, 2026 License: MIT Imports: 2 Imported by: 0

Details

Valid go.mod file
Redistributable license
Tagged version
Stable version
Learn more about best practices

Repository

github.com/xibodev/compa

Links

Open Source Insights

Documentation ¶

Index ¶

func EstimateMessageTokens(msg providers.Message) int
func EstimateTextTokens(s string) int
func EstimateToolDefsTokens(defs []providers.ToolDefinition) int
func IsCJK(r rune) bool

Constants ¶

This section is empty.

Variables ¶

This section is empty.

Functions ¶

func EstimateMessageTokens ¶

func EstimateMessageTokens(msg providers.Message) int

EstimateMessageTokens estimates the token count for a single message, including Content, ReasoningContent, ToolCalls arguments, ToolCallID metadata, and Media items. CJK runes count as one token each; other text as 2.5 characters per token.

func EstimateTextTokens ¶

func EstimateTextTokens(s string) int

EstimateTextTokens estimates the token count of plain text.

func EstimateToolDefsTokens ¶

func EstimateToolDefsTokens(defs []providers.ToolDefinition) int

EstimateToolDefsTokens estimates the total token cost of tool definitions as they appear in the LLM request.

func IsCJK ¶

func IsCJK(r rune) bool

IsCJK reports whether r is a CJK ideograph, a kana or a Hangul syllable (U+2E80–U+9FFF, U+F900–U+FAFF, U+AC00–U+D7AF). Tokenizers spend about one token on each of these runes, against about one per four for Latin text.

Types ¶

This section is empty.

Source Files ¶

View all Source files

estimator.go

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL