tokenizer

package
v0.11.0-beta.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 23, 2026 License: Apache-2.0, MIT Imports: 3 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func CharsPerToken 

func CharsPerToken(modelFamily string) float64

func ModelFamily 

func ModelFamily(modelID string) string

Types

type Estimator 

type Estimator struct {
	// contains filtered or unexported fields
}

func NewEstimator 

func NewEstimator() *Estimator

func (*Estimator) ContextUsage 

func (e *Estimator) ContextUsage(text string, maxTokens int) float64

func (*Estimator) Count 

func (e *Estimator) Count(text string, model string) (int, error)

func (*Estimator) Estimate 

func (e *Estimator) Estimate(text string) int

func (*Estimator) MessageTokenCount 

func (e *Estimator) MessageTokenCount(messages []providers.Message) int

type Tokenizer 

type Tokenizer interface {
	Count(text string, model string) (int, error)
	Estimate(text string) int
	ContextUsage(text string, maxTokens int) float64
}

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.