tokens

package
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 6, 2025 License: MIT Imports: 4 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func Count 

func Count(text, modelName string) int

Count returns the number of tokens in the given text for the specified model

func CountWithFallback 

func CountWithFallback(text string) int

CountWithFallback counts tokens but falls back to a simple heuristic if tiktoken fails

func Truncate 

func Truncate(text string, tokenLimit int, modelName string) string

Truncate truncates text to fit within the specified token limit for the given model

Types

This section is empty.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.