tokenizer

package
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Feb 26, 2026 License: MIT Imports: 1 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func EstimateTokens

func EstimateTokens(text string) int

EstimateTokens provides a rough token count estimate. Uses the heuristic of ~4 characters per token for English text.

func FormatMemoriesWithBudget

func FormatMemoriesWithBudget(memories []string, budget int) (string, int)

FormatMemoriesWithBudget formats multiple memory strings within a token budget. Returns the formatted string and the number of memories that fit.

func TruncateToTokenBudget

func TruncateToTokenBudget(text string, budget int) string

TruncateToTokenBudget truncates text to approximately fit within a token budget.

Types

This section is empty.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL