Documentation
¶
Overview ¶
Command warmtiktoken populates the tiktoken vocabulary cache before a parallel test run.
tiktoken-go downloads the BPE on first use and finishes with an atomic rename into a shared cache directory. `go test ./...` runs packages in parallel, so several test binaries race to download the same file; on Windows the loser hits "Access is denied" on the rename and its whole package fails. It surfaced as bench/arms flaking with no code change anywhere near it.
Setting TIKTOKEN_CACHE_DIR alone does not fix that — it names a cache, and an empty one still races. Running this once first does, because every later construction is a cache hit that renames nothing.
It doubles as the SC-004 offline prerequisite: an outsider reproducing the deterministic figures needs the vocabulary present, and "warm the cache" is easier to follow as a command than as a paragraph.