Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type TokenCounter ¶
type TokenCounter struct {
// contains filtered or unexported fields
}
func NewTokenCounter ¶
func NewTokenCounter() *TokenCounter
NewTokenCounter creates a token counter with proper fallback
func (*TokenCounter) DebugTokenCount ¶ added in v0.4.1
func (tc *TokenCounter) DebugTokenCount(text string, label string) int
DebugTokenCount provides detailed breakdown of token estimation
func (*TokenCounter) EstimateTokens ¶
func (tc *TokenCounter) EstimateTokens(text string) int
EstimateTokens counts tokens using tiktoken or falls back to approximation
func (*TokenCounter) GetEncodingName ¶ added in v0.4.1
func (tc *TokenCounter) GetEncodingName() string
GetEncodingName returns the name of the encoding being used
func (*TokenCounter) IsFallbackMode ¶ added in v0.4.1
func (tc *TokenCounter) IsFallbackMode() bool
IsFallbackMode returns true if using approximation instead of tiktoken
Source Files
Click to show internal directories.
Click to hide internal directories.