Documentation
¶
Overview ¶
Package tiktoken is the TokenCounter adapter: it counts tokens with a pure-Go, offline tiktoken BPE codec (o200k_base — the encoding text-embedding-3 and gpt-4o use). lore is provider-agnostic, so exact counts vary by model; this is treated as approximate-but-stable, which is enough to size chunks and bound retrieval budgets predictably. The dependency lives only here — the domain chunkers receive Count as an injected func, keeping stdlib-only domain clean.
Index ¶
Constants ¶
const EncodingName = "o200k_base"
EncodingName identifies the tiktoken encoding this adapter counts with. It is recorded in a Collection's ChunkerSpec so a change of tokenizer (which alters token counts and therefore chunk boundaries) is a chunker mismatch.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Counter ¶
type Counter struct {
// contains filtered or unexported fields
}
Counter counts tokens with a tiktoken codec. Construct with New; the zero value is not usable.
Source Files