Documentation
¶
Overview ¶
Package tokenwindow preserves source text when vocabulary tokens split UTF-8.
Index ¶
Constants ¶
This section is empty.
Variables ¶
var ErrSearchBudgetExceeded = errors.New("tokenwindow: search budget exceeded")
ErrSearchBudgetExceeded means the search stopped without proving absence.
Functions ¶
func Prefix ¶
func Prefix(ctx context.Context, codec tokenizer.Tokenizer, source string, limit, maxWork int, render func(string) string) (string, error)
Prefix returns a lossless source prefix whose rendered text fits the budget. Probe and final rendering are encoded independently because vocabulary boundaries can change after trimming or adding structural context. Chunk boundaries are selected from the probe, not a whole-document tokenization. Token-prefix decoding is only a candidate selector: if it finds no admissible prefix, every source rune boundary is checked by independent encoding. Token counts need not be monotonic. An empty result proves that no nonempty rendered source prefix fits. maxWork bounds input bytes passed to render and Encode, plus tokens passed to Decode; each operation costs at least one unit.
Types ¶
This section is empty.
Source Files