tokenizer

package
v0.11.5 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 27, 2026 License: EUPL-1.2 Imports: 2 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func NewApproximationTokenizer

func NewApproximationTokenizer() ports.Tokenizer

NewApproximationTokenizer creates a new ApproximationTokenizer with default settings. Uses 4.0 characters per token as the default ratio.

func NewApproximationTokenizerWithRatio

func NewApproximationTokenizerWithRatio(charsPerToken float64) ports.Tokenizer

NewApproximationTokenizerWithRatio creates a new ApproximationTokenizer with a custom characters-per-token ratio. Useful for tuning accuracy for specific languages or domains.

Types

type ApproximationTokenizer

type ApproximationTokenizer struct {
	// contains filtered or unexported fields
}

ApproximationTokenizer implements ports.Tokenizer using character-based estimation. Provides fast, approximate token counting as a fallback when exact tokenization is not available or not needed. Uses a simple heuristic: ~4 characters per token.

func (*ApproximationTokenizer) CountTokens

func (a *ApproximationTokenizer) CountTokens(text string) (int, error)

CountTokens returns an approximate token count based on character length. Formula: token_count ≈ len(text) / charsPerToken

func (*ApproximationTokenizer) CountTurnsTokens

func (a *ApproximationTokenizer) CountTurnsTokens(turns []string) (int, error)

CountTurnsTokens returns the total approximate token count across multiple conversation turns. Sums the estimated token counts for each turn.

func (*ApproximationTokenizer) IsEstimate

func (a *ApproximationTokenizer) IsEstimate() bool

IsEstimate returns true because this tokenizer produces approximate counts.

func (*ApproximationTokenizer) ModelName

func (a *ApproximationTokenizer) ModelName() string

ModelName returns the identifier for this approximation method.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL