tokenizer

package

v1.39.1 Latest Latest Go to latest Published: Oct 21, 2026 License: Apache-2.0 Imports: 9 Imported by: 0

Details

Valid go.mod file
Redistributable license
Tagged version
Stable version
Learn more about best practices

Repository

github.com/openai/openai-cli

Links

Open Source Insights

Documentation ¶

Overview ¶

Package tokenizer adapts embedded text encodings for local CLI token inspection.

Index ¶

Constants
Variables
func IsSupportedEncoding(name string) bool
func SupportedEncodings() []string
type Result
- func Encode(text, encoding string, inspect bool) (Result, error)

Constants ¶

View Source

const DefaultEncoding = "o200k_base"

DefaultEncoding is the encoding used when the caller makes no selection.

View Source

const MaxInputBytes = 1 << 20

MaxInputBytes bounds this local utility's input and result allocations. It does not limit API requests or guarantee short execution for unbroken text.

Variables ¶

View Source

var Licenses string

Licenses retains the tokenizer dependencies' distribution notices in the binary.

Functions ¶

func IsSupportedEncoding ¶

func IsSupportedEncoding(name string) bool

IsSupportedEncoding checks an exact encoding name without loading vocabulary data.

func SupportedEncodings ¶

func SupportedEncodings() []string

SupportedEncodings returns exact encoding names in presentation order.

Types ¶

type Result ¶

type Result struct {
	Encoding   string
	InputBytes int
	TokenCount int
	IDs        []uint
	Fragments  []string
}

Result preserves the encoding's exact token IDs and byte fragments when inspected. A fragment can contain only part of a UTF-8 character.

func Encode ¶

func Encode(text, encoding string, inspect bool) (Result, error)

Encode treats special-token markers as ordinary text and preserves input bytes. It does not normalize Unicode, strip a BOM, or modify whitespace. Calls are synchronous. The dependency does not expose context cancellation.

Source Files ¶

View all Source files

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL