Documentation
¶
Overview ¶
Package tokenizer provides a tiktoken-backed agent.TokenCounter. It wraps github.com/tiktoken-go/tokenizer, whose byte-pair-encoding rank tables are COMPILED INTO the package as Go source (no go:embed of external files, no runtime download), so token counting is fully offline and deterministic — a hard requirement for a harness that must run air-gapped and reproducibly.
This adapter is wired only in the composition root (cmd/mecated); the agent package depends solely on the agent.TokenCounter interface, never on this concrete tokenizer. The default counter remains the dependency-free heuristic in engine/agent; this is the opt-in "real tokenizer" tier.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Counter ¶
type Counter struct {
// contains filtered or unexported fields
}
Counter is a tiktoken-backed agent.TokenCounter. It is safe for concurrent use: the underlying codec is read-only after construction.
func New ¶
New constructs a Counter for the given encoding. The rank tables are loaded from the compiled-in vocabulary (offline); an unknown encoding returns an error.
func NewForModel ¶
NewForModel constructs a Counter for the tiktoken encoding associated with the given model name, falling back to O200kBase when the model is unrecognised (every current OpenAI frontier model uses o200k_base).
type Encoding ¶
type Encoding string
Encoding identifies a tiktoken byte-pair-encoding table.
const ( // O200kBase is the encoding used by the GPT-4o / GPT-4.1 / o-series and // later frontier models. It is the sensible default for current OpenAI models. O200kBase Encoding = "o200k_base" // Cl100kBase is the encoding used by GPT-3.5-turbo and GPT-4 (pre-4o). Cl100kBase Encoding = "cl100k_base" )
Source Files