Documentation
¶
Overview ¶
Source: github.com/tiktoken-go/tokenizer/codec/codec.go@v0.8.1 The splitting and byte-pair merge algorithm is retained verbatim; long pieces use an equivalent heap merge, and vocabulary storage is owned by tokenizer.go. See LICENSE for the upstream MIT license.
Package tokenizer provides Mekugi's pinned GPT-5 (o200k_base) tokenizer.
The vocabulary is embedded data, not a generated Go map. Token IDs and the splitting/BPE algorithm match github.com/tiktoken-go/tokenizer v0.8.1. Special-token spellings in source are ordinary text, not control tokens.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Codec ¶
type Codec struct {
// contains filtered or unexported fields
}
Codec tokenizes text with the pinned o200k_base vocabulary. Use New to initialize a codec with its own regexp.
func (*Codec) SelectRows ¶
SelectRows admits LF-terminated rows in order, stopping at the first row that exceeds limit. Tail admission walks backwards but returns source order. Completed regexp pieces never change at a later row boundary. Only the piece crossing that boundary needs to be recounted, including newline/whitespace runs whose BPE counts need not grow monotonically.
Source Files
Directories