tokenizer

package

v1.40.4 Latest Latest Go to latest Published: Oct 23, 2026 License: Apache-2.0 Imports: 0 Imported by: 0

Details

Valid go.mod file
Redistributable license
Tagged version
Stable version
Learn more about best practices

Repository

github.com/openai/openai-cli

Links

Open Source Insights

Documentation ¶

Overview ¶

Code generated by scripts/generate-tokenizer-unicode.go; DO NOT EDIT. UnicodeData 16.0.0 SHA-256: ff58e5823bd095166564a006e47d111130813dcf8bf234ef79fa51a870edb48f. Unicode-3.0 license: see ../../internal/tokenizer/LICENSES.txt.

Index ¶

Constants

Constants ¶

View Source

const Unicode16Letters = "" /* 9928-byte string literal not displayed */

Letters: 677 ranges, 141028 code points.

View Source

const Unicode16Lower = "" /* 14567-byte string literal not displayed */

Lower: 1274 ranges, 141640 code points.

View Source

const Unicode16Numbers = "" /* 2436-byte string literal not displayed */

Numbers: 144 ranges, 1911 code points.

View Source

const Unicode16Upper = "" /* 14714-byte string literal not displayed */

Upper: 1290 ranges, 141271 code points.

View Source

const Unicode16Whitespace = "\\x09-\\x0D\\x20\u0085\u00a0\u1680\u2000-\u200a\u2028-\u2029\u202f\u205f\u3000"

Whitespace: 10 ranges, 25 code points.

Variables ¶

This section is empty.

Functions ¶

This section is empty.

Types ¶

This section is empty.

Source Files ¶

View all Source files

unicode16.go

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL