Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type TokenCounter ¶
type TokenCounter struct {
// contains filtered or unexported fields
}
func NewTokenCounter ¶
func NewTokenCounter(encoding string) (*TokenCounter, error)
NewTokenCounter creates a new TokenCounter with the specified encoding. If encoding is empty, defaults to "o200k_base" (GPT-5 encoding). Supported encodings: o200k_base, cl100k_base, p50k_base, r50k_base, etc.
func (*TokenCounter) CountTokens ¶
func (tc *TokenCounter) CountTokens(text string) (int, error)
func (*TokenCounter) CountTokensWithPath ¶
func (tc *TokenCounter) CountTokensWithPath(text string, filePath string) (int, error)
Source Files
Click to show internal directories.
Click to hide internal directories.