tokens

func EstimateTokens ¶

func EstimateTokens(text string) int

EstimateTokens estimates the number of tokens in the given text string. It uses a rough approximation of 4 characters per token, which is a common heuristic for most language models. This function provides a quick estimation without requiring model-specific tokenizers.

The estimation may not be accurate for all models or text types, particularly for texts with many special characters, non-English languages, or code snippets. For more accurate token counting, use model-specific tokenizers when available.

Parameters:

• text: The input text string to estimate tokens for

Returns:

• int: The estimated number of tokens in the text

Example:

count := EstimateTokens("Hello, world!")  // Returns approximately 3

func InitializeTokenCounters ¶

func InitializeTokenCounters()

InitializeTokenCounters registers all available token counters for various language model providers. This function should be called during application startup to ensure that token counting functionality is available for all supported models.

Currently, this function is a placeholder for future provider-specific token counter implementations. As new providers are added (OpenAI, Anthropic, Google, etc.), their respective token counters will be registered here.

This function does not require any API keys and will only initialize counters that can work without authentication.

Example:

func main() {
    tokens.InitializeTokenCounters()
    // Token counting is now available
}

func InitializeTokenCountersWithKeys ¶

func InitializeTokenCountersWithKeys()

InitializeTokenCountersWithKeys registers token counters for various language model providers using the provided API keys. This function enables more accurate token counting by allowing access to provider-specific tokenization endpoints or libraries that require authentication.

This function should be called during application startup after API keys have been loaded from configuration or environment variables. It will initialize token counters for providers where API keys are available, enabling precise token counting that matches the provider's actual tokenization logic.

The function will silently skip providers for which no API keys are configured, allowing the application to continue with partial token counting capabilities.

Future implementations will accept provider-specific API keys through parameters or read them from a configuration context.

Example:

func main() {
    // Load API keys from environment or config
    tokens.InitializeTokenCountersWithKeys()
    // Provider-specific token counting is now available
}