Documentation
¶
Overview ¶
Package token provides token counting for MCP tool call content.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
Types ¶
type Counter ¶
type Counter interface {
// Count returns the estimated number of tokens in the given text.
Count(text string) int
}
Counter estimates token counts for text content. Implementations may vary in accuracy — the interface allows swapping a heuristic counter for a tiktoken-based one without changing consumers.
type HeuristicCounter ¶
type HeuristicCounter struct {
// contains filtered or unexported fields
}
HeuristicCounter estimates tokens using a simple bytes-per-token ratio. This is fast and zero-dependency but approximate. Suitable for visibility purposes where exact counts are not required.
func NewHeuristicCounter ¶
func NewHeuristicCounter(bytesPerToken int) *HeuristicCounter
NewHeuristicCounter creates a counter that estimates tokens at the given bytes-per-token ratio. A ratio of 4 is a reasonable default for English text.
func (*HeuristicCounter) Count ¶
func (c *HeuristicCounter) Count(text string) int
Count returns the estimated token count for the given text.
Source Files
Click to show internal directories.
Click to hide internal directories.