Versions in this module Expand all Collapse all v0 v0.1.2 Jul 4, 2026 v0.1.1 Jun 25, 2026 Changes in this version + type CodeAwareTokenizer struct + CollapseWhitespace bool + ExtractSignatures bool + MaxLineLength int + PreserveStructure bool + RemoveComments bool + func NewCodeAwareTokenizer() *CodeAwareTokenizer + func (t *CodeAwareTokenizer) CompressCode(code string, maxTokens int, language string) string + func (t *CodeAwareTokenizer) EstimateTokens(code string) int + func (t *CodeAwareTokenizer) TokenizeCode(code string, language string) []CodeToken + type CodeToken struct + IsNoise bool + Line int + Type string + Value string
Go to main page