Documentation
¶
Overview ¶
Package tiktoken implements the core/tokenizer capabilities with OpenAI's tiktoken vocabularies, adapting github.com/pkoukk/tiktoken-go to the small interfaces Core defines.
The vocabulary is chosen explicitly, because no single encoding is correct across models:
tokenizer, err := tiktoken.New(tiktoken.O200KBase)
if err != nil {
return err
}
count, err := tokenizer.CountText(ctx, "hello")
An unknown encoding returns ErrInvalidEncoding at construction rather than silently falling back to another vocabulary, because a wrong token count is not visible until a request is rejected for length.
This module owns no provider request, no model-to-encoding routing, no text splitting, and no cache. Splitting belongs to etl; routing belongs to whatever knows the model.
Index ¶
Constants ¶
const ( O200KBase = Encoding(tiktokenlib.MODEL_O200K_BASE) CL100KBase = Encoding(tiktokenlib.MODEL_CL100K_BASE) P50KBase = Encoding(tiktokenlib.MODEL_P50K_BASE) P50KEdit = Encoding(tiktokenlib.MODEL_P50K_EDIT) R50KBase = Encoding(tiktokenlib.MODEL_R50K_BASE) )
Supported encodings.
Variables ¶
var ( ErrInvalidEncoding = errors.New("tiktoken: invalid encoding") ErrUninitialized = errors.New("tiktoken: tokenizer is not initialized") )
Functions ¶
This section is empty.
Types ¶
type Encoding ¶
type Encoding string
Encoding identifies a tiktoken vocabulary. Callers choose explicitly because token counts are vocabulary-specific; there is no model-independent default.
type Tokenizer ¶
type Tokenizer struct {
// contains filtered or unexported fields
}
Tokenizer encodes, decodes, and counts text with one tiktoken vocabulary. It is safe for concurrent use.
func New ¶
New resolves the vocabulary once at construction rather than per call, so a misspelled encoding fails where it is configured instead of on the first count. The vocabulary must be named explicitly because no single encoding is correct across models, and guessing one silently miscounts every budget derived from it.
Source Files