tokenizer

package

v0.6.1 Latest Latest Go to latest Published: Sep 9, 2026 License: Apache-2.0 Imports: 0 Imported by: 0

Details

Package tokenizer provides the stable Phase 1 token estimator.

View Source

const Version = "estimate/v1"

This section is empty.

func Estimate(content []byte) int64

type Estimator struct{}

func (e Estimator) Count(content []byte) (int, error)

func (Estimator) Estimate(content []byte) int64

func (Estimator) Name() string

type Tokenizer interface {
	Name() string
	Count([]byte) (int, error)
}