tiktoken

Overview ¶

Package tiktoken wraps the embedded tiktoken-go BPE codec so text can be split into model tokens with no runtime network fetch — the o200k_base vocabulary is compiled into the binary. The demo stream uses it to emit text at real token boundaries: a canned response then streams exactly like a live model turn (one SSE delta per token) instead of arbitrary fixed-width chunks.