warmtiktoken

command
v0.68.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 21, 2026 License: MIT Imports: 3 Imported by: 0

Documentation

Overview

Command warmtiktoken populates the tiktoken vocabulary cache before a parallel test run.

tiktoken-go downloads the BPE on first use and finishes with an atomic rename into a shared cache directory. `go test ./...` runs packages in parallel, so several test binaries race to download the same file; on Windows the loser hits "Access is denied" on the rename and its whole package fails. It surfaced as bench/arms flaking with no code change anywhere near it.

Setting TIKTOKEN_CACHE_DIR alone does not fix that — it names a cache, and an empty one still races. Running this once first does, because every later construction is a cache hit that renames nothing.

It doubles as the SC-004 offline prerequisite: an outsider reproducing the deterministic figures needs the vocabulary present, and "warm the cache" is easier to follow as a command than as a paragraph.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL