tokenize

package

v1.6.0 Latest Latest Go to latest Published: Oct 9, 2026 License: MIT Imports: 5 Imported by: 0

Details

Valid go.mod file
Redistributable license
Tagged version
Stable version
Learn more about best practices

Repository

github.com/bethropolis/sift

Links

Open Source Insights

Documentation ¶

Overview ¶

Package tokenize counts tokens offline and fits documents to a budget.

Index ¶

func FitToBudget(files []format.FileEntry, budget int) ([]format.FileEntry, int)
type Tokenizer
- func New(encoding string) (*Tokenizer, error)
- func (t *Tokenizer) Count(text []byte) (int, error)

Constants ¶

This section is empty.

Variables ¶

This section is empty.

Functions ¶

func FitToBudget ¶

func FitToBudget(files []format.FileEntry, budget int) ([]format.FileEntry, int)

FitToBudget greedily keeps files until the next one would exceed budget. Files are expected to be pre-sorted by priority; it returns the kept files and the total tokens used.

Types ¶

type Tokenizer ¶

type Tokenizer struct {
	// contains filtered or unexported fields
}

Tokenizer counts tokens using an offline BPE encoder.

func New ¶

func New(encoding string) (*Tokenizer, error)

New returns a Tokenizer for the named encoding (e.g. "cl100k_base", "o200k_base"). An empty name selects cl100k_base.

func (*Tokenizer) Count ¶

func (t *Tokenizer) Count(text []byte) (int, error)

Count returns the number of tokens in text.

Source Files ¶

View all Source files

tokenize.go

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL