tokenizer

package
v0.10.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 22, 2026 License: MIT Imports: 1 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func EstimateTokens 

func EstimateTokens(text string) int

EstimateTokens returns an approximate token count for the given text. Heuristic: calibrated to cl100k_base tokenization used by Claude and GPT-4. Accuracy: ±15% for typical English prose; may undercount for code or non-ASCII.

func FormatMemoriesWithBudget 

func FormatMemoriesWithBudget(memories []string, budget int) (string, int)

FormatMemoriesWithBudget formats multiple memory strings within a token budget. Returns the formatted string and the number of memories that fit.

func TruncateToTokenBudget 

func TruncateToTokenBudget(text string, budget int) string

TruncateToTokenBudget truncates text to approximately fit within a token budget.

Types

This section is empty.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.