Documentation
¶
Overview ¶
Package tokenwindow preserves source text when vocabulary tokens split UTF-8.
Index ¶
Constants ¶
This section is empty.
Variables ¶
var ErrSearchBudgetExceeded = errors.New("tokenwindow: search budget exceeded")
ErrSearchBudgetExceeded means the search stopped without proving absence.
Functions ¶
func Prefix ¶
func Prefix(ctx context.Context, codec tokenizer.Tokenizer, source string, limit, maxWork int, render func(string) string) (string, error)
Prefix returns a lossless source prefix whose rendered text fits limit tokens. Probe and final rendering are encoded independently because vocabulary boundaries can change after trimming or adding structural context, and token counts need not be monotonic. Token-prefix decoding only selects a candidate; when it finds none, every source rune boundary is checked by independent encoding. An empty result proves that no nonempty rendered prefix fits. maxWork bounds input bytes passed to render and Encode plus tokens passed to Decode; each operation costs at least one unit.
Types ¶
This section is empty.
Source Files