tokenizer

package

v0.2.0 Latest Latest Go to latest Published: Oct 7, 2026 License: MIT, MIT Imports: 10 Imported by: 0

Details

Valid go.mod file
Redistributable license
Tagged version
Stable version
Learn more about best practices

Repository

github.com/yusing/mekugi

Links

Open Source Insights

Documentation ¶

Overview ¶

Source: github.com/tiktoken-go/tokenizer/codec/codec.go@v0.8.1 The splitting and byte-pair merge algorithm is retained verbatim; long pieces use an equivalent heap merge, and vocabulary storage is owned by tokenizer.go. See LICENSE for the upstream MIT license.

Package tokenizer provides Mekugi's pinned GPT-5 (o200k_base) tokenizer.

The vocabulary is embedded data, not a generated Go map. Token IDs and the splitting/BPE algorithm match github.com/tiktoken-go/tokenizer v0.8.1. Special-token spellings in source are ordinary text, not control tokens.

Constants ¶

This section is empty.

Variables ¶

This section is empty.

Functions ¶

This section is empty.

Types ¶

type Codec ¶

type Codec interface {
	GetName() string
	Count(string) (int, error)
	SelectRows(string, int, bool) (string, error)
	Encode(string) ([]uint, []string, error)
	Decode([]uint) (string, error)
}

Codec is the tokenization surface used by sanitized capture estimates.

func New ¶

func New() (Codec, error)

New returns a codec with immutable shared vocabulary and its own regexp.

Source Files ¶

View all Source files

Directories ¶

Path	Synopsis
regex Package codec registers the pinned upstream generated regex engines.	Package codec registers the pinned upstream generated regex engines.

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL