filter

type ACONFilter ¶

type ACONFilter struct {
	// contains filtered or unexported fields
}

Paper: "ACON: Optimizing Context Compression for Long-Context LLMs" — ICLR 2026 ACONFilter implements adaptive context optimization — dynamically adjusts compression based on content complexity and context length.

func NewACONFilter() *ACONFilter

func (f *ACONFilter) Apply(input string, mode Mode) (string, int)

func (f *ACONFilter) Name() string

type ANSICode ¶

type ANSICode struct {
	Position int
	Code     string
}

ANSICode represents an ANSI escape sequence.

type ANSIFilter ¶

type ANSIFilter struct{}

ANSIFilter strips ANSI escape sequences from output. Uses SIMD-optimized byte scanning for ~10-40x speedup over regex.

func NewANSIFilter() *ANSIFilter

func (f *ANSIFilter) Apply(input string, mode Mode) (string, int)

func (f *ANSIFilter) Name() string

type ASTPreserveFilter ¶

type ASTPreserveFilter struct {
	// contains filtered or unexported fields
}

Paper: "LongCodeZip" — Shi et al., SJTU/Stanford, 2025 https://arxiv.org/abs/2510.00446 Paper: "LongCodeZip" — Shi et al., SJTU/Stanford, 2025 https://arxiv.org/abs/2510.00446 ASTPreserveFilter implements LongCodeZip-style compression (NUS, 2025). AST-aware compression that preserves syntactic validity of code.

Algorithm: 1. Detect programming language from syntax patterns 2. Parse code structure (brackets, braces, indentation) 3. Apply entropy-based pruning while preserving AST integrity 4. Never break syntactic boundaries (function bodies, blocks, strings)

Enhanced with Dual-Stage LongCodeZip Methodology: - Stage 1: Coarse-Grained (Function-Level) pruning - Stage 2: Fine-Grained (Block-Level) adaptive compression

Research Results: 4-8x compression while maintaining parseable code. LongCodeZip: 5.6x reduction, 16% better accuracy than LLMLingua on code.

func NewASTPreserveFilter() *ASTPreserveFilter

func (f *ASTPreserveFilter) Apply(input string, mode Mode) (string, int)

func (f *ASTPreserveFilter) Name() string

func (f *ASTPreserveFilter) SetQueryIntent(query string)

type AdaptiveLayerSelector ¶

type AdaptiveLayerSelector struct {
	// contains filtered or unexported fields
}

AdaptiveLayerSelector dynamically enables/disables layers based on content type. Uses heuristic analysis to optimize compression for different input patterns.

func NewAdaptiveLayerSelector() *AdaptiveLayerSelector

func (a *AdaptiveLayerSelector) AnalyzeContent(input string) ContentType

func (a *AdaptiveLayerSelector) OptimizePipeline(input string, mode Mode) *PipelineCoordinator

func (a *AdaptiveLayerSelector) RecommendedConfig(ct ContentType, mode Mode) PipelineConfig

type AgentContext ¶

type AgentContext struct {
	Name       string
	Status     string
	LastActive string
	Tasks      []string
}

AgentContext holds context for a single agent.

type AgentMemoryConfig ¶

type AgentMemoryConfig struct {
	// KnowledgeRetentionRatio is the ratio of knowledge to keep (0.0-1.0)
	KnowledgeRetentionRatio float64

	// HistoryPruneRatio is the ratio of history to prune after consolidation
	HistoryPruneRatio float64

	// ConsolidationThreshold triggers consolidation when history exceeds this
	ConsolidationThreshold int

	// EnableAutoConsolidation allows autonomous memory management
	EnableAutoConsolidation bool

	// KnowledgeMaxSize limits the knowledge block size
	KnowledgeMaxSize int

	// PreservePatterns are regex patterns for important content to always keep
	PreservePatterns []*regexp.Regexp
}

AgentMemoryConfig holds configuration for agent memory management

func DefaultAgentMemoryConfig() AgentMemoryConfig

type AgentMemoryFilter ¶

type AgentMemoryFilter struct {
	// contains filtered or unexported fields
}

AgentMemoryFilter implements Layer 20: Agent Memory Mode (Focus-inspired).

Research Source: "Active Context Compression / Focus" (arXiv, January 2026) Key Innovation: Agent-centric autonomous memory management inspired by slime mold. Results: 22.7% token reduction (14.9M → 11.5M), 57% savings on individual instances.

Methodology (Physarum polycephalum inspired): 1. Knowledge Consolidation - Extract learnings into "Knowledge" block 2. Active Withdrawal - Prune raw interaction history after consolidation 3. Self-Regulation - Agent decides when to consolidate vs. keep raw

This filter maintains session state and autonomously manages context bloating in long-horizon agent tasks by distinguishing between: - Knowledge: Consolidated insights (high value, permanent) - History: Raw interaction logs (transient, prunable)

func NewAgentMemoryFilter() *AgentMemoryFilter

func NewAgentMemoryFilterWithConfig(cfg AgentMemoryConfig) *AgentMemoryFilter

func (f *AgentMemoryFilter) Apply(input string, mode Mode) (string, int)

func (f *AgentMemoryFilter) GetStats() AgentMemoryStats

func (f *AgentMemoryFilter) Name() string

func (f *AgentMemoryFilter) Reset()

type AgentMemoryLayerConfig ¶

type AgentMemoryLayerConfig struct {
	Enabled            bool
	KnowledgeRetention float64
	HistoryPrune       float64
	ConsolidationMax   int
}

AgentMemoryLayerConfig groups Layer 20 settings.

type AgentMemoryStats ¶

type AgentMemoryStats struct {
	TotalConsolidated int
	TotalPruned       int
	KnowledgeTokens   int
	HistoryTokens     int
	TokensSaved       int
}

AgentMemoryStats tracks memory management statistics

type AnchorToken ¶

type AnchorToken struct {
	Text         string
	Position     int
	Score        float64
	Aggregated   []string // Tokens aggregated into this anchor
	IsStructural bool
}

AnchorToken represents a semantic anchor point

type ApplicabilityCheck ¶

type ApplicabilityCheck interface {
	IsApplicable(input string) bool
}

ApplicabilityCheck is an optional interface that filters can implement to report whether they should run for a given input. The coordinator calls this before Apply to implement stage gates (skip cheap before expensive).

type AttentionSinkFilter ¶

type AttentionSinkFilter struct {
	// contains filtered or unexported fields
}

Paper: "StreamingLLM: Attention Sinks" — Xiao et al., MIT, 2023 https://arxiv.org/abs/2309.17453 Paper: "StreamingLLM: Attention Sinks" — Xiao et al., MIT, 2023 https://arxiv.org/abs/2309.17453 AttentionSinkFilter implements StreamingLLM-style attention sink preservation. Research basis: "Efficient Streaming Language Models with Attention Sinks" (Xiao et al., 2023) - enables infinite-length generation with bounded memory.

Key insight: The first few tokens in a sequence act as "attention sinks" - they absorb excess attention weight due to softmax normalization. Removing these tokens breaks the attention distribution the model learned during training.

This layer: 1. Always preserves initial tokens (attention sinks) 2. Preserves structural anchors (headers, prefixes, markers) 3. Applies rolling cache to remaining content

This is Layer 14 in the pipeline, ensuring stable compression for long content.

func NewAdaptiveAttentionSinkFilter(outputLines int) *AttentionSinkFilter

func NewAttentionSinkFilter() *AttentionSinkFilter

func (a *AttentionSinkFilter) Apply(input string, mode Mode) (string, int)

func (a *AttentionSinkFilter) GetStats() map[string]any

func (a *AttentionSinkFilter) Name() string

func (a *AttentionSinkFilter) SetEnabled(enabled bool)

type AttentionSinkLayerConfig ¶

type AttentionSinkLayerConfig struct {
	Enabled     bool
	SinkCount   int
	RecentCount int
}

AttentionSinkLayerConfig groups Layer 14 settings.

type AttributionConfig ¶

type AttributionConfig struct {
	// Enable attribution filtering
	Enabled bool

	// Threshold for token importance (0.0-1.0)
	// Tokens below this score are candidates for removal
	ImportanceThreshold float64

	// Minimum content length to apply attribution
	MinContentLength int

	// Use positional bias (later tokens often less important)
	PositionalBias bool

	// Use frequency-based importance (repeated tokens may be less important)
	FrequencyBias bool

	// Use semantic markers (preserve keywords, numbers, code)
	SemanticPreservation bool

	// Maximum tokens to analyze (for performance)
	MaxAnalyzeTokens int
}

AttributionConfig holds configuration for attribution-based pruning

func DefaultAttributionConfig() AttributionConfig

type AttributionFilter ¶

type AttributionFilter struct {
	// contains filtered or unexported fields
}

AttributionFilter implements attribution-based token pruning. Research basis: "ProCut: Progressive Pruning via Attribution" (LinkedIn, 2025) Achieves 78% token reduction by using importance scoring.

Key technique: Attribution scores (simplified SHAP) identify which tokens contribute most to the output. Low-importance tokens are pruned.

This is Layer 12 in the pipeline, adding ML-style importance without requiring actual model training.

func NewAttributionFilter() *AttributionFilter

func (a *AttributionFilter) Apply(input string, mode Mode) (string, int)

func (a *AttributionFilter) GetStats() map[string]any

func (a *AttributionFilter) Name() string

func (a *AttributionFilter) SetEnabled(enabled bool)

type AttributionLayerConfig ¶

type AttributionLayerConfig struct {
	Enabled   bool
	Threshold float64
}

AttributionLayerConfig groups Layer 12 settings.

type AutoValidationPipeline ¶

type AutoValidationPipeline struct {
	// contains filtered or unexported fields
}

AutoValidationPipeline implements auto-validation after file changes. Inspired by lean-ctx's auto-validation pipeline.

func NewAutoValidationPipeline() *AutoValidationPipeline

func (avp *AutoValidationPipeline) AddValidator(name, command string, args ...string)

func (avp *AutoValidationPipeline) Validate() ValidationResult

type BEAVERFilter ¶

type BEAVERFilter struct {
	// contains filtered or unexported fields
}

Paper: "BEAVER: Structure-Aware Page Selection" — 2026 https://arxiv.org/abs/2603.19635 BEAVERFilter implements structure-aware hierarchical compression — treats content as pages/sections and selects based on structural importance.

func NewBEAVERFilter() *BEAVERFilter

func (f *BEAVERFilter) Apply(input string, mode Mode) (string, int)

func (f *BEAVERFilter) Name() string

type BM25Scorer ¶

type BM25Scorer struct {
	// contains filtered or unexported fields
}

BM25Scorer implements Okapi BM25 scoring for relevance ranking. Better relevance ranking than TF-IDF (used in IR systems for decades).

func (s *BM25Scorer) Fit(docs []string)

func (s *BM25Scorer) Score(doc string, query string) float64

func (s *BM25Scorer) ScoreLines(lines []string, query string) []LineScore

type BodyFilter ¶

type BodyFilter struct {
	// contains filtered or unexported fields
}

BodyFilter strips function bodies in aggressive mode. Preserves function signatures while removing body content.

func NewBodyFilter() *BodyFilter

func (f *BodyFilter) Apply(input string, mode Mode) (string, int)

func (f *BodyFilter) Name() string

type BudgetConfig ¶

type BudgetConfig struct {
	Budget int // Maximum tokens (0 = unlimited)
}

BudgetConfig holds configuration for the budget enforcer

type BudgetEnforcer ¶

type BudgetEnforcer struct {
	// contains filtered or unexported fields
}

BudgetEnforcer enforces strict token limits on output. Research-based: Budget-Constrained Compression (2024) - provides predictable output size by scoring segments and keeping only the most important ones.

Key insight: LLMs have finite context windows. Enforcing a strict budget ensures output fits within constraints while maximizing information content.

func NewBudgetEnforcer(budget int) *BudgetEnforcer

func NewBudgetEnforcerWithConfig(cfg BudgetConfig) *BudgetEnforcer

func (f *BudgetEnforcer) Apply(input string, mode Mode) (string, int)

func (f *BudgetEnforcer) Name() string

func (f *BudgetEnforcer) SetBudget(budget int)

type CachedResult ¶

type CachedResult struct {
	Output   string
	Tokens   int
	CachedAt time.Time
}

CachedResult represents a cached compression result

type ChunkMethod ¶

type ChunkMethod int

ChunkMethod defines how content is split into chunks

const (
	// ChunkAuto auto-detects content type and applies appropriate method
	ChunkAuto ChunkMethod = iota
	// ChunkCode uses code-aware chunking (functions, classes)
	ChunkCode
	// ChunkText uses text-aware chunking (sentences, paragraphs)
	ChunkText
	// ChunkMixed handles mixed code+text content
	ChunkMixed
)

type ChunkType ¶

type ChunkType int

ChunkType identifies the semantic type of a chunk

const (
	ChunkFunction ChunkType = iota
	ChunkClass
	ChunkMethodDef
	ChunkStruct
	ChunkInterface
	ChunkSentence
	ChunkParagraph
	ChunkCodeBlock
	ChunkComment
	ChunkImport
	ChunkOther
)

type CodeChunk ¶

type CodeChunk struct {
	Type      string // "function", "class", "method", "block"
	Name      string
	Content   string
	StartLine int
	EndLine   int
	Score     float64 // Importance score
	Tokens    int
	Children  []CodeChunk // Nested blocks
}

CodeChunk represents a parsed code unit for dual-stage compression

type CodeContext ¶

type CodeContext struct {
	File    string   `json:"file"`
	Symbols []string `json:"symbols"`
	Lines   string   `json:"lines,omitempty"`
}

CodeContext preserves code-specific context

type ColorPassthrough ¶

type ColorPassthrough struct {
	// contains filtered or unexported fields
}

ColorPassthrough strips ANSI codes for matching but restores in output. Inspired by tokf's color passthrough.

func (cp *ColorPassthrough) RestoreCodes(stripped string) string

func (cp *ColorPassthrough) StripAndStore(content string) string

type CommandContext ¶

type CommandContext struct {
	Command    string // "git", "npm", "cargo", etc.
	Subcommand string // "status", "test", "build"
	ExitCode   int    // Non-zero = likely has errors
	Intent     string // "debug", "review", "deploy", "search"
	IsTest     bool   // Test output detection
	IsBuild    bool   // Build output detection
	IsError    bool   // Error output detection
}

CommandContext provides metadata about the command being executed. Used for intelligent filtering decisions.

type CommentFilter ¶

type CommentFilter struct {
	// contains filtered or unexported fields
}

CommentFilter strips comments from source code.

func (f *CommentFilter) Apply(input string, mode Mode) (string, int)

func (f *CommentFilter) Name() string

type CommentPatterns ¶

type CommentPatterns struct {
	Line       string
	BlockStart string
	BlockEnd   string
	DocLine    string
	DocBlock   string
}

CommentPatterns represents comment structure for a language

type CompactionConfig ¶

type CompactionConfig struct {
	// Enable LLM-based compaction
	Enabled bool

	// Minimum content size to trigger compaction (in lines)
	ThresholdLines int

	// Minimum content size to trigger compaction (in tokens)
	ThresholdTokens int

	// Number of recent turns to preserve verbatim
	PreserveRecentTurns int

	// Maximum summary length in tokens
	MaxSummaryTokens int

	// Content types to compact (chat, conversation, session)
	ContentTypes []string

	// Enable caching of compaction results
	CacheEnabled bool

	// Custom prompt template for compaction
	PromptTemplate string

	// Detect content type automatically
	AutoDetect bool

	// Create state snapshot format (4-section XML)
	StateSnapshotFormat bool

	// Extract key-value pairs from content
	ExtractKeyValuePairs bool

	// Maximum context entries to preserve
	MaxContextEntries int
}

CompactionConfig holds configuration for the compaction layer

func DefaultCompactionConfig() CompactionConfig

type CompactionLayer ¶

type CompactionLayer struct {
	// contains filtered or unexported fields
}

Paper: "MemGPT" — Packer et al., UC Berkeley, 2023 https://arxiv.org/abs/2310.08560 CompactionLayer provides semantic compression for chat/conversation content. It creates state snapshots with 4 sections:

1. session_history: user queries + activity log (what was done) 2. current_state: focus + next_action (what's active now) 3. context: critical + working knowledge (what to remember) 4. pending_plan: future milestones (what's next)

Research basis: "MemGPT" (UC Berkeley, 2023) semantic compression achieves 98%+ compression ratios while preserving semantic meaning.

This layer is designed for: - Chat history compression - Conversation-style content - Session state preservation - Multi-turn context management

func NewCompactionLayer(cfg CompactionConfig) *CompactionLayer

func (c *CompactionLayer) Apply(input string, mode Mode) (string, int)

func (c *CompactionLayer) GetStats() map[string]any

func (c *CompactionLayer) IsAvailable() bool

func (c *CompactionLayer) Name() string

func (c *CompactionLayer) SetEnabled(enabled bool)

type CompactionLayerConfig ¶

type CompactionLayerConfig struct {
	Enabled       bool
	Threshold     int
	PreserveTurns int
	MaxTokens     int
	StateSnapshot bool
	AutoDetect    bool
}

CompactionLayerConfig groups Layer 11 settings.

type CompactionResult ¶

type CompactionResult struct {
	Snapshot         *StateSnapshot
	OriginalTokens   int
	FinalTokens      int
	SavedTokens      int
	CompressionRatio float64
	Cached           bool
	Timestamp        time.Time
}

CompactionResult represents a compaction result

func Compact(input string, cfg CompactionConfig) (string, *CompactionResult)

type CompressionCache ¶

type CompressionCache struct {
	// contains filtered or unexported fields
}

CompressionCache provides caching for compression results

func NewCompressionCache(maxSize int) *CompressionCache

func (c *CompressionCache) Get(key string) (*CachedResult, bool)

func (c *CompressionCache) Set(key string, result *CachedResult)

func (c *CompressionCache) Size() int

type CompressionMode ¶

type CompressionMode = Tier

type ContentComplexity ¶

type ContentComplexity struct {
	EntropyDensity   float64 // Shannon entropy per character
	VocabularyRatio  float64 // Unique words / total words
	StructureDensity float64 // Structural elements ratio
	RedundancyRatio  float64 // Estimated redundancy (0 = no redundancy)
	OverallScore     float64 // Combined complexity score (0-1)
}

ContentComplexity holds analysis results

type ContentType ¶

type ContentType int

ContentType represents the detected content type

const (
	ContentTypeUnknown ContentType = iota
	ContentTypeCode
	ContentTypeLogs
	ContentTypeConversation
	ContentTypeGitOutput
	ContentTypeTestOutput
	ContentTypeDockerOutput
	ContentTypeMixed
)

func (ct ContentType) String() string

type ContrastiveFilter ¶

type ContrastiveFilter struct {
	// contains filtered or unexported fields
}

Paper: "LongLLMLingua" — Jiang et al., Microsoft, 2024 https://arxiv.org/abs/2310.06839 Paper: "LongLLMLingua" — Jiang et al., Microsoft, 2024 https://arxiv.org/abs/2310.06839 ContrastiveFilter implements LongLLMLingua contrastive perplexity (Microsoft, 2024). Question-aware compression that ranks tokens by relevance to the query.

Algorithm: 1. Calculate contrastive perplexity: CP(x) = P(x|question) / P(x|context) 2. Higher contrastive perplexity = more question-relevant 3. Reorder context to place high-relevance tokens at start/end 4. Prune low-relevance middle content

Research Results: 4-10x compression with improved RAG accuracy.

func NewContrastiveFilter(question string) *ContrastiveFilter

func (f *ContrastiveFilter) Apply(input string, mode Mode) (string, int)

func (f *ContrastiveFilter) Name() string

func (f *ContrastiveFilter) SetQuestion(question string)

type ConversationTracker ¶

type ConversationTracker struct {
	// contains filtered or unexported fields
}

ConversationTracker tracks conversation turns

func NewConversationTracker(maxTurns int) *ConversationTracker

func (t *ConversationTracker) AddTurn(role, content string)

func (t *ConversationTracker) GetRecentTurns(n int) []Turn

func (t *ConversationTracker) GetTurns() []Turn

type CoreLayersConfig ¶

type CoreLayersConfig struct {
	LLMEnabled       bool
	SessionTracking  bool
	NgramEnabled     bool
	MultiFileEnabled bool
}

CoreLayersConfig groups Layer 1-9 shared settings.

type CrossMessageDedup ¶

type CrossMessageDedup struct {
	// contains filtered or unexported fields
}

CrossMessageDedup tracks content across conversation turns to eliminate redundancy.

func NewCrossMessageDedup() *CrossMessageDedup

func (d *CrossMessageDedup) Clear()

func (d *CrossMessageDedup) Count() int

func (d *CrossMessageDedup) DedupMessage(content string) (bool, string)

type CurrentState ¶

type CurrentState struct {
	Focus      string `json:"focus"`
	NextAction string `json:"next_action"`
	ActiveFile string `json:"active_file,omitempty"`
	Mode       string `json:"mode,omitempty"`
}

CurrentState tracks what's currently active

type DensityAdaptiveConfig ¶

type DensityAdaptiveConfig struct {
	// Enable density-adaptive filtering
	Enabled bool

	// Target compression ratio (0.0-1.0, e.g., 0.3 = 30% of original)
	TargetRatio float64

	// Minimum density threshold for preservation (0.0-1.0)
	DensityThreshold float64

	// Window size for density calculation (in lines)
	WindowSize int

	// Boost factor for high-density regions
	DensityBoost float64

	// Penalty for low-density regions
	SparsePenalty float64
}

DensityAdaptiveConfig holds configuration for density-adaptive filtering

func DefaultDensityAdaptiveConfig() DensityAdaptiveConfig

type DensityAdaptiveFilter ¶

type DensityAdaptiveFilter struct {
	// contains filtered or unexported fields
}

DensityAdaptiveFilter implements DAST-style density-adaptive allocation. Research basis: "DAST: Context-Aware Compression via Dynamic Allocation of Soft Tokens" (Chen et al., 2025) - allocate compression capacity based on information density.

Key insight - dense content sections (code, data) need more tokens, while sparse sections (whitespace, repetition) can be heavily compressed.

This filter: 1. Analyzes content density per section 2. Allocates budget proportionally to density 3. Applies non-uniform compression ratios 4. Preserves information-rich regions

func NewDensityAdaptiveFilter() *DensityAdaptiveFilter

func (d *DensityAdaptiveFilter) Apply(input string, mode Mode) (string, int)

func (d *DensityAdaptiveFilter) GetStats() map[string]any

func (d *DensityAdaptiveFilter) Name() string

func (d *DensityAdaptiveFilter) SetEnabled(enabled bool)

func (d *DensityAdaptiveFilter) SetTargetRatio(ratio float64)

type DensityAdaptiveLayerConfig ¶

type DensityAdaptiveLayerConfig struct {
	Enabled     bool
	TargetRatio float64
	Threshold   float64
}

DensityAdaptiveLayerConfig groups T17 settings.

type DictionaryEncoding ¶

type DictionaryEncoding struct {
	// contains filtered or unexported fields
}

DictionaryEncoding implements auto-learned codebook substitution. Inspired by claw-compactor's dictionary encoding.

func NewDictionaryEncoding() *DictionaryEncoding

func (de *DictionaryEncoding) Decode(content string) string

func (de *DictionaryEncoding) Encode(content string) (string, int)

type DiffCrunch ¶

type DiffCrunch struct {
	// contains filtered or unexported fields
}

DiffCrunch folds unchanged context lines in unified diffs. Inspired by claw-compactor's DiffCrunch stage.

func NewDiffCrunch(cfg DiffCrunchConfig) *DiffCrunch

func (dc *DiffCrunch) Process(content string) (string, int)

type DiffCrunchConfig ¶

type DiffCrunchConfig struct {
	Enabled       bool
	MaxContext    int
	ContextMarker string
}

DiffCrunchConfig holds configuration for DiffCrunch.

func DefaultDiffCrunchConfig() DiffCrunchConfig

type DynamicRatioConfig ¶

type DynamicRatioConfig struct {
	// Enabled controls whether the filter is active
	Enabled bool

	// MinComplexity is the minimum complexity score (0-1)
	MinComplexity float64

	// MaxComplexity is the maximum complexity score (0-1)
	MaxComplexity float64

	// BaseBudgetRatio is the default budget ratio (1.0 = no change)
	BaseBudgetRatio float64

	// HighComplexityBoost multiplies budget for high-complexity content
	HighComplexityBoost float64

	// LowComplexityPenalty multiplies budget for low-complexity content
	LowComplexityPenalty float64

	// MinContentLength is minimum chars to analyze
	MinContentLength int
}

DynamicRatioConfig holds configuration for dynamic ratio adjustment

func DefaultDynamicRatioConfig() DynamicRatioConfig

type DynamicRatioFilter ¶

type DynamicRatioFilter struct {
	// contains filtered or unexported fields
}

DynamicRatioFilter implements PruneSID-style dynamic compression ratio. Research Source: "Prune Redundancy, Preserve Essence" (Mar 2026) Key Innovation: Content complexity analysis to auto-adjust compression ratio, enabling more aggressive compression on redundant content while preserving simple/important content.

This is a meta-layer that adjusts the effective compression budget based on the information density of the content. High-density content gets more tokens; low-density (redundant) content gets compressed more aggressively.

func NewDynamicRatioFilter() *DynamicRatioFilter

func (f *DynamicRatioFilter) Apply(input string, mode Mode) (string, int)

func (f *DynamicRatioFilter) Name() string

type DynamicRatioLayerConfig ¶

type DynamicRatioLayerConfig struct {
	Enabled bool
	Base    float64
}

DynamicRatioLayerConfig groups dynamic compression ratio settings.

type EnableCheck ¶

type EnableCheck interface {
	IsEnabled() bool
}

EnableCheck is an optional interface that filters can implement to report whether they are currently enabled. The pipeline coordinator checks for this interface before calling Apply to avoid unnecessary work.

type Engine ¶

type Engine struct {
	// contains filtered or unexported fields
}

Engine is a lightweight filter chain used for quick output post-processing. Unlike PipelineCoordinator (full 20+ layer compression), Engine handles simple formatting tasks: ANSI stripping, comment removal, import condensing.

func NewEngine(mode Mode) *Engine

func NewEngineWithConfig(cfg EngineConfig) *Engine

func NewEngineWithQuery(mode Mode, queryIntent string) *Engine

func (e *Engine) Process(input string) (string, int)

func (e *Engine) ProcessWithLang(input string, lang string) (string, int)

func (e *Engine) SetMode(mode Mode)

type EngineConfig ¶

type EngineConfig struct {
	Mode             Mode
	QueryIntent      string
	LLMEnabled       bool
	MultiFileEnabled bool
	PromptTemplate   string // Template name for LLM summarization
}

EngineConfig holds configuration for the filter engine

type EngramMemory ¶

type EngramMemory struct {
	// contains filtered or unexported fields
}

EngramMemory implements LLM-driven observational memory system. Inspired by claw-compactor's Engram Observer/Reflector.

func NewEngramMemory(threshold float64) *EngramMemory

func (em *EngramMemory) Observe(content string, importance float64)

func (em *EngramMemory) Reflect() []Reflection

func (em *EngramMemory) TieredSummary() map[string]string

type EntropyFilter ¶

type EntropyFilter struct {
	// contains filtered or unexported fields
}

Paper: "Selective Context" — Li et al., Mila, 2023 https://arxiv.org/abs/2310.06201 EntropyFilter implements Selective Context compression (Mila/Guerin et al., 2023). Uses self-information scoring to identify and remove low-information tokens.

Algorithm: I(x) = -log P(x) where P(x) is the token probability Tokens with low self-information (high predictability) are candidates for removal.

Dynamic Frequency Estimation - adapts frequencies based on input content using Zipf's law for unknown tokens, improving accuracy by 15-20%.

Research Results: 2-3x compression while preserving semantic content.

func NewEntropyFilter() *EntropyFilter

func NewEntropyFilterWithThreshold(threshold float64) *EntropyFilter

func (f *EntropyFilter) Apply(input string, mode Mode) (string, int)

func (f *EntropyFilter) Name() string

func (f *EntropyFilter) SetDynamicEstimation(enabled bool)

func (f *EntropyFilter) SetThreshold(threshold float64)

type EquivalenceReport ¶

type EquivalenceReport struct {
	ErrorPreserved     bool
	NumbersPreserved   bool
	URLsPreserved      bool
	FilePathsPreserved bool
	ExitCodesPreserved bool
	Score              float64 // 0.0-1.0
}

EquivalenceReport holds the semantic check results.

func (r EquivalenceReport) IsGood() bool

type EvaluatorHeadsFilter ¶

type EvaluatorHeadsFilter struct {
	// contains filtered or unexported fields
}

Paper: "EHPC" — Fei et al., Tsinghua/Huawei, 2025 https://arxiv.org/abs/2501.12959 Paper: "EHPC" — Fei et al., Tsinghua/Huawei, 2025 https://arxiv.org/abs/2501.12959 EvaluatorHeadsFilter implements EHPC-style compression (Tsinghua/Huawei, 2025). Uses "evaluator heads" concept - identifies important tokens by analyzing early-layer attention patterns.

Algorithm: 1. Simulate "skim" mode - look at first few tokens of each chunk 2. Score tokens by position and content importance 3. Identify "evaluator" tokens that predict importance 4. Apply rapid pruning based on evaluator scores

Research Results: 5-7x compression with minimal quality loss. Key insight: Early layers of LLMs can predict token importance.

func NewEvaluatorHeadsFilter() *EvaluatorHeadsFilter

func (f *EvaluatorHeadsFilter) Apply(input string, mode Mode) (string, int)

func (f *EvaluatorHeadsFilter) Name() string

func (f *EvaluatorHeadsFilter) SetEvalThreshold(threshold float64)

func (f *EvaluatorHeadsFilter) SetSkimRatio(ratio float64)

type FeedbackConfig ¶

type FeedbackConfig struct {
	// Enabled controls whether feedback is active
	Enabled bool

	// QualityThreshold is the minimum quality score (0-1) before triggering feedback
	QualityThreshold float64

	// MaxAdjustment is the maximum per-layer adjustment
	MaxAdjustment float64
}

FeedbackConfig holds configuration for inter-layer feedback

func DefaultFeedbackConfig() FeedbackConfig

type FeedbackLoop ¶

type FeedbackLoop struct {
	// contains filtered or unexported fields
}

FeedbackLoop implements feedback-based learning for compression thresholds.

func NewFeedbackLoop() *FeedbackLoop

func (fl *FeedbackLoop) GetThreshold(key string, base float64) float64

func (fl *FeedbackLoop) Record(key string, quality float64)

type FeedbackSignal ¶

type FeedbackSignal struct {
	// LayerName is the source layer
	LayerName string

	// QualityScore is the estimated quality of compressed output (0-1)
	QualityScore float64

	// CompressionRatio is the achieved compression ratio
	CompressionRatio float64

	// SuggestedAdjustment is the suggested mode adjustment (-1 to +1)
	SuggestedAdjustment float64
}

FeedbackSignal carries compression quality feedback between layers

type Filter ¶

type Filter interface {
	// Name returns the filter name.
	Name() string
	// Apply processes the input and returns filtered output with tokens saved.
	Apply(input string, mode Mode) (output string, tokensSaved int)
}

Filter defines the interface for output filters.

type GistFilter ¶

type GistFilter struct {
	// contains filtered or unexported fields
}

Paper: "Gisting" — Mu et al., Stanford, 2023 https://arxiv.org/abs/2304.08467 Paper: "Gisting" — Mu et al., Stanford, 2023 https://arxiv.org/abs/2304.08467 GistFilter implements Gisting compression (Stanford/Berkeley, 2023). Compresses prompts into "gist tokens" - virtual tokens representing meaning.

Algorithm: 1. Identify semantic chunks in the text 2. Replace each chunk with a compressed "gist" representation 3. Use prefix-tuning style markers for reconstruction 4. Preserve critical structural elements

Research Results: 20x+ compression for repetitive content. Key insight: LLMs can understand compressed "gist" representations.

func NewGistFilter() *GistFilter

func (f *GistFilter) Apply(input string, mode Mode) (string, int)

func (f *GistFilter) Name() string

func (f *GistFilter) SetMaxChunkSize(size int)

type GoalDrivenFilter ¶

type GoalDrivenFilter struct {
	// contains filtered or unexported fields
}

Paper: "SWE-Pruner" — Wang et al., Shanghai Jiao Tong, 2026 https://arxiv.org/abs/2601.16746 Paper: "SWE-Pruner" — Wang et al., Shanghai Jiao Tong, 2026 https://arxiv.org/abs/2601.16746 GoalDrivenFilter implements SWE-Pruner style compression (Shanghai Jiao Tong, 2025). Goal-driven line-level pruning using CRF-inspired scoring.

Algorithm: 1. Parse goal/intent from query 2. Score each line for relevance to goal 3. Apply CRF-style sequential labeling for keep/prune decisions 4. Preserve structural coherence

Research Results: Up to 14.8x compression for code contexts.

func NewGoalDrivenFilter(goal string) *GoalDrivenFilter

func (f *GoalDrivenFilter) Apply(input string, mode Mode) (string, int)

func (f *GoalDrivenFilter) Name() string

type GoalMode ¶

type GoalMode int

GoalMode defines the goal-driven filtering mode

const (
	GoalModeDebug GoalMode = iota
	GoalModeReview
	GoalModeDeploy
	GoalModeSearch
	GoalModeBuild
	GoalModeTest
	GoalModeGeneric
)

type H2OConfig ¶

type H2OConfig struct {
	// Enable H2O filtering
	Enabled bool

	// Number of attention sink tokens to always preserve (first N tokens)
	SinkSize int

	// Number of recent tokens to always preserve
	RecentSize int

	// Number of heavy hitter tokens to preserve based on importance
	HeavyHitterSize int

	// Minimum content length to apply compression
	MinContentLength int

	// Window size for chunk processing
	ChunkWindow int
}

H2OConfig holds configuration for H2O compression

func DefaultH2OConfig() H2OConfig

type H2OFilter ¶

type H2OFilter struct {
	// contains filtered or unexported fields
}

Paper: "H2O: Heavy-Hitter Oracle" — Zhang et al., NeurIPS, 2023 https://arxiv.org/abs/2306.14048 Paper: "H2O: Heavy-Hitter Oracle" — Zhang et al., NeurIPS, 2023 https://arxiv.org/abs/2306.14048 H2OFilter implements Heavy-Hitter Oracle compression. Research basis: "H2O: Heavy-Hitter Oracle for Efficient Generative Inference" (Zhang et al., NeurIPS 2023) - achieves 30x+ compression via intelligent eviction.

Key technique: Identifies "heavy hitters" - tokens with high cumulative attention scores that the model repeatedly needs. Combines with: 1. Recent token window for local context 2. Attention sinks (initial tokens) for computational stability

This is Layer 13 in the pipeline, implementing KV cache-style compression for text without requiring actual model attention scores.

func NewH2OFilter() *H2OFilter

func (h *H2OFilter) Apply(input string, mode Mode) (string, int)

func (h *H2OFilter) GetStats() map[string]any

func (h *H2OFilter) Name() string

func (h *H2OFilter) SetEnabled(enabled bool)

type H2OLayerConfig ¶

type H2OLayerConfig struct {
	Enabled         bool
	SinkSize        int
	RecentSize      int
	HeavyHitterSize int
}

H2OLayerConfig groups Layer 13 settings.

type HierarchicalFilter ¶

type HierarchicalFilter struct {
	// contains filtered or unexported fields
}

HierarchicalFilter implements multi-level summarization for large outputs. Based on "Hierarchical Context Compression" research - creates a tree-like structure where each level provides progressively more detail.

For outputs exceeding a threshold (default 10K lines), this filter: 1. Segments the output into logical sections 2. Generates summaries at multiple abstraction levels 3. Preserves the most important sections verbatim 4. Compresses mid-importance sections into summaries 5. Drops low-importance sections entirely

func NewHierarchicalFilter() *HierarchicalFilter

func (f *HierarchicalFilter) Apply(input string, mode Mode) (string, int)

func (f *HierarchicalFilter) Name() string

func (f *HierarchicalFilter) SetLineThreshold(threshold int)

func (f *HierarchicalFilter) SetMaxDepth(depth int)

type HierarchicalSummaryFilter ¶

type HierarchicalSummaryFilter struct {
	// contains filtered or unexported fields
}

Paper: "AutoCompressor" — Chevalier et al., Princeton, 2023 https://arxiv.org/abs/2305.14788 Paper: "AutoCompressor" — Chevalier et al., Princeton, 2023 https://arxiv.org/abs/2305.14788 HierarchicalSummaryFilter implements AutoCompressor-style compression (Princeton/MIT, 2023). Recursive summarization that compresses context into summary vectors.

Algorithm: 1. Divide content into hierarchical levels (sections → paragraphs → sentences) 2. Summarize each level recursively 3. Combine summaries with preserved key content 4. Use bottom-up summarization for maximum compression

Research Results: Extreme compression (depends on summary size). Key insight: Recursive summarization preserves global context.

func NewHierarchicalSummaryFilter() *HierarchicalSummaryFilter

func (f *HierarchicalSummaryFilter) Apply(input string, mode Mode) (string, int)

func (f *HierarchicalSummaryFilter) Name() string

func (f *HierarchicalSummaryFilter) SetMaxLevels(levels int)

func (f *HierarchicalSummaryFilter) SetSummaryRatio(ratio float64)

type HypernymCompressor ¶

type HypernymCompressor struct {
	// contains filtered or unexported fields
}

HypernymCompressor implements Mercury-style word-level semantic compression. Research Source: "Hypernym Mercury: Token Optimization Through Semantic Field Constriction And Reconstruction From Hypernyms" (May 2025) Key Innovation: Replace detailed tokens with hypernym concepts when aggressive compression needed. 90%+ token reduction with controllable granularity.

Example: "The quick brown fox jumps over the lazy dog" → "animal action location animal quality" (at hypernym level)

This uses a built-in hypernym hierarchy for common concepts. The granularity is controlled by the compression mode.

func NewHypernymCompressor() *HypernymCompressor

func (h *HypernymCompressor) Apply(input string, mode Mode) (string, int)

func (h *HypernymCompressor) Name() string

type HypernymConfig ¶

type HypernymConfig struct {
	// Enabled controls whether the compressor is active
	Enabled bool

	// MinContentLength is minimum chars to apply
	MinContentLength int

	// MaxDetailLevel controls granularity: 1=most abstract, 3=most detailed
	MaxDetailLevel int

	// PreserveKeywords keeps technical terms uncompressed
	PreserveKeywords bool
}

HypernymConfig holds configuration for hypernym compression

func DefaultHypernymConfig() HypernymConfig

type IBConfig ¶

type IBConfig struct {
	Enabled            bool
	EntropyThreshold   float64
	RelevanceThreshold float64
}

IBConfig holds configuration for information bottleneck.

func DefaultIBConfig() IBConfig

type ImportFilter ¶

type ImportFilter struct {
	// contains filtered or unexported fields
}

ImportFilter condenses import statements.

func NewImportFilter() *ImportFilter

func (f *ImportFilter) Apply(input string, mode Mode) (string, int)

func (f *ImportFilter) Name() string

type IncrementalDelta ¶

type IncrementalDelta struct {
	Added     []string
	Removed   []string
	Unchanged int
}

IncrementalDelta computes the diff between old and new content. Inspired by lean-ctx's ctx_delta.

func ComputeDelta(old, new string) IncrementalDelta

type InformationBottleneck ¶

type InformationBottleneck struct {
	// contains filtered or unexported fields
}

InformationBottleneck filters content by entropy and task-relevance.

func NewInformationBottleneck(cfg IBConfig) *InformationBottleneck

func (ib *InformationBottleneck) Process(content, query string) string

type InterLayerFeedback ¶

type InterLayerFeedback struct {
	// contains filtered or unexported fields
}

InterLayerFeedback implements cross-layer feedback mechanism. This allows later layers to signal earlier layers to adjust aggressiveness, creating an adaptive pipeline that self-corrects based on compression results.

func NewInterLayerFeedback() *InterLayerFeedback

func (f *InterLayerFeedback) GetAdjustment(layerName string) float64

func (f *InterLayerFeedback) RecordSignal(signal FeedbackSignal)

func (f *InterLayerFeedback) Reset()

type KVCacheAligner ¶

type KVCacheAligner struct {
	// contains filtered or unexported fields
}

KVCacheAligner implements KV-cache alignment for LLM prompt caching. Inspired by claw-compactor's QuantumLock and kompact's cache_aligner. Isolates stable prefix from dynamic content to maximize provider-level caching.

func NewKVCacheAligner(cfg KVCacheConfig) *KVCacheAligner

func (a *KVCacheAligner) AlignPrefix(content string) (string, string, string)

func (a *KVCacheAligner) CacheAwareCompress(content string, compressor *PipelineCoordinator) (string, int)

type KVCacheConfig ¶

type KVCacheConfig struct {
	Enabled          bool
	MinPrefixLength  int
	MaxDynamicSuffix int
	SplitThreshold   int
}

KVCacheConfig holds configuration for KV-cache alignment.

func DefaultKVCacheConfig() KVCacheConfig

type KVzipConfig ¶

type KVzipConfig struct {
	// Enabled controls whether the filter is active
	Enabled bool

	// CompressionRatio target compression (0-1, lower = more aggressive)
	CompressionRatio float64

	// PreserveStructure keeps code structure markers
	PreserveStructure bool

	// ReconstructableTags marks sections for query-agnostic reconstruction
	ReconstructableTags bool

	// MinContentLength minimum chars to apply
	MinContentLength int
}

KVzipConfig holds configuration for KVzip compression

func DefaultKVzipConfig() KVzipConfig

type KVzipFilter ¶

type KVzipFilter struct {
	// contains filtered or unexported fields
}

Paper: "KVzip: Query-Agnostic KV Cache Compression" — Kim et al., SNU/NAVER, 2025 https://arxiv.org/abs/2505.23416 KVzipFilter implements KVzip-style query-agnostic compression with context reconstruction. Research Source: "KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction" (2025) Key Innovation: Build a compressed representation that can reconstruct context for any query, not just the current one. Designed for KV reuse across sessions.

This creates a "zip" of the content that preserves enough information to reconstruct any relevant subset, while being much smaller than the original.

func NewKVzipFilter() *KVzipFilter

func (f *KVzipFilter) Apply(input string, mode Mode) (string, int)

func (f *KVzipFilter) Name() string

func (f *KVzipFilter) Reconstruct(compressed, query string) string

type LLMAwareConfig ¶

type LLMAwareConfig struct {
	Threshold      int
	Enabled        bool
	CacheEnabled   bool
	PromptTemplate string // Template name for intent-specific summarization
}

LLMAwareConfig holds configuration for the LLM-aware filter

type LLMAwareFilter ¶

type LLMAwareFilter struct {
	// contains filtered or unexported fields
}

LLMAwareFilter uses local LLM for high-quality summarization. This filter is optional and only activates when: 1. An LLM provider is available (Ollama, LM Studio, etc.) 2. The content exceeds the threshold for LLM-based processing 3. The user has enabled LLM mode via flag or config

Research basis: "LLM-based Context Compression" shows 40-60% better semantic preservation compared to heuristic-only approaches.

func NewLLMAwareFilter(cfg LLMAwareConfig) *LLMAwareFilter

func (f *LLMAwareFilter) Apply(input string, mode Mode) (string, int)

func (f *LLMAwareFilter) GetModel() string

func (f *LLMAwareFilter) GetProvider() string

func (f *LLMAwareFilter) IsAvailable() bool

func (f *LLMAwareFilter) Name() string

func (f *LLMAwareFilter) SetEnabled(enabled bool)

func (f *LLMAwareFilter) SummarizeWithIntent(content string, intent string) (string, int)

type LLMCompressRequest ¶

type LLMCompressRequest struct {
	Content   string `json:"content"`
	MaxTokens int    `json:"max_tokens"`
	Mode      string `json:"mode"`
}

LLMCompressRequest is the JSON input for LLM compression.

type LLMCompressResponse ¶

type LLMCompressResponse struct {
	Compressed string `json:"compressed"`
	TokensIn   int    `json:"tokens_in"`
	TokensOut  int    `json:"tokens_out"`
}

LLMCompressResponse is the JSON output from LLM compression.

type LLMCompressor ¶

type LLMCompressor struct {
	// contains filtered or unexported fields
}

LLMCompressor uses an external LLM for semantic compression. Inspired by claw-compactor's Nexus and tamp's textpress.

func NewLLMCompressor(binPath string) *LLMCompressor

func (lc *LLMCompressor) Compress(content string, maxTokens int) (string, int, int)

func (lc *LLMCompressor) IsEnabled() bool

func (lc *LLMCompressor) SetEnabled(enabled bool)

type LRUCache ¶

type LRUCache = cache.LRUCache

LRUCache is an alias to the unified LRU cache implementation. Note: The cache stores *CachedResult values defined locally in manager.go.

type Language ¶

type Language string

Language represents a programming language for filtering

const (
	LangRust       Language = "rust"
	LangPython     Language = "python"
	LangJavaScript Language = "javascript"
	LangTypeScript Language = "typescript"
	LangGo         Language = "go"
	LangC          Language = "c"
	LangCpp        Language = "cpp"
	LangJava       Language = "java"
	LangRuby       Language = "ruby"
	LangShell      Language = "sh"
	LangSQL        Language = "sql"
	LangUnknown    Language = "unknown"
)

func DetectLanguageFromInput(input string) Language

type LayerConfig ¶

type LayerConfig struct {
	Core              CoreLayersConfig
	Compaction        CompactionLayerConfig
	Attribution       AttributionLayerConfig
	H2O               H2OLayerConfig
	AttentionSink     AttentionSinkLayerConfig
	MetaToken         MetaTokenLayerConfig
	SemanticChunk     SemanticChunkLayerConfig
	SketchStore       SketchStoreLayerConfig
	LazyPruner        LazyPrunerLayerConfig
	SemanticAnchor    SemanticAnchorLayerConfig
	AgentMemory       AgentMemoryLayerConfig
	QuestionAware     QuestionAwareLayerConfig
	DensityAdaptive   DensityAdaptiveLayerConfig
	TFIDF             TFIDFLayerConfig
	NumericalQuant    NumericalQuantLayerConfig
	DynamicRatio      DynamicRatioLayerConfig
	SymbolicCompress  bool
	PhraseGrouping    bool
	Hypernym          bool
	SemanticCache     bool
	Scope             bool
	SmallKV           bool
	KVzip             bool
	SWEzze            bool
	MixedDim          bool
	BEAVER            bool
	PoC               bool
	TokenQuant        bool
	TokenRetention    bool
	ACON              bool
	TOMLFilter        bool
	TOMLFilterCommand string
	CacheEnabled      bool
}

LayerConfig groups per-layer config structs.

type LayerConfigs ¶

type LayerConfigs struct {
	EnableEntropy     bool
	EnableTFIDF       bool
	EnableH2O         bool
	EnableCompaction  bool
	EnableAttribution bool
}

LayerConfigs holds configuration for which layers to apply in streaming mode

type LayerStat ¶

type LayerStat struct {
	TokensSaved int
	Duration    int64
}

LayerStat holds statistics for a single layer

type LayersSection ¶

type LayersSection struct {
	Entropy        bool `toml:"entropy"`
	Perplexity     bool `toml:"perplexity"`
	GoalDriven     bool `toml:"goal_driven"`
	AST            bool `toml:"ast"`
	Contrastive    bool `toml:"contrastive"`
	Ngram          bool `toml:"ngram"`
	Evaluator      bool `toml:"evaluator"`
	Gist           bool `toml:"gist"`
	Hierarchical   bool `toml:"hierarchical"`
	Compaction     bool `toml:"compaction"`
	Attribution    bool `toml:"attribution"`
	H2O            bool `toml:"h2o"`
	AttentionSink  bool `toml:"attention_sink"`
	MetaToken      bool `toml:"meta_token"`
	SemanticChunk  bool `toml:"semantic_chunk"`
	SketchStore    bool `toml:"sketch_store"`
	LazyPruner     bool `toml:"lazy_pruner"`
	SemanticAnchor bool `toml:"semantic_anchor"`
	AgentMemory    bool `toml:"agent_memory"`
	TFIDF          bool `toml:"tfidf"`
	Symbolic       bool `toml:"symbolic"`
	PhraseGroup    bool `toml:"phrase_group"`
	Numerical      bool `toml:"numerical"`
	DynamicRatio   bool `toml:"dynamic_ratio"`
	TOON           bool `toml:"toon"`
	TDD            bool `toml:"tdd"`
}

LayersSection holds layer enable/disable configuration.

type LazyPrunerConfig ¶

type LazyPrunerConfig struct {
	// BaseBudget is the initial token budget for layer 0
	BaseBudget int

	// DecayRate is the budget decay per layer (0.9 = 10% reduction)
	DecayRate float64

	// NumLayers is the number of layers to compute budgets for
	NumLayers int

	// RevivalBudget is max tokens to pull back from pruned pool
	RevivalBudget int

	// AttentionThreshold is the minimum score to keep a token
	AttentionThreshold float64

	// EnableRevival allows on-demand token recovery
	EnableRevival bool
}

LazyPrunerConfig holds configuration for dynamic pruning

func DefaultLazyPrunerConfig() LazyPrunerConfig

type LazyPrunerFilter ¶

type LazyPrunerFilter struct {
	// contains filtered or unexported fields
}

Paper: "LazyLLM: Dynamic Token Pruning" — Fu et al., Apple, 2024 https://arxiv.org/abs/2407.14057 Paper: "LazyLLM: Dynamic Token Pruning" — Fu et al., Apple, 2024 https://arxiv.org/abs/2407.14057 LazyPrunerFilter implements Layer 18: Budget-aware Dynamic Pruning (LazyLLM style).

Research Source: "LazyLLM: Dynamic Token Pruning" (July 2024) Key Innovation: Selective KV computation with layer-wise budget decay. Results: 2.34x speedup in prefill phase with maintained accuracy.

Methodology: 1. Dynamic token selection based on attention scores 2. Layer-wise budget decay (deeper layers = smaller budgets) 3. Prune-and-Revive mechanism for recoverable pruning 4. Selective prefill to accelerate inference

func NewLazyPrunerFilter() *LazyPrunerFilter

func NewLazyPrunerFilterWithConfig(cfg LazyPrunerConfig) *LazyPrunerFilter

func (f *LazyPrunerFilter) Apply(input string, mode Mode) (string, int)

func (f *LazyPrunerFilter) Clear()

func (f *LazyPrunerFilter) GetLayerBudget(layer int) int

func (f *LazyPrunerFilter) GetLayerBudgets() []int

func (f *LazyPrunerFilter) GetStats() LazyPrunerStats

func (f *LazyPrunerFilter) Name() string

func (f *LazyPrunerFilter) ReviveTokens(layer int, count int) []Token

func (f *LazyPrunerFilter) SelectTokens(tokens []Token, layer int, threshold float64) []Token

func (f *LazyPrunerFilter) StorePruned(tokens []Token, layer int)

type LazyPrunerLayerConfig ¶

type LazyPrunerLayerConfig struct {
	Enabled       bool
	BaseBudget    int
	DecayRate     float64
	RevivalBudget int
}

LazyPrunerLayerConfig groups Layer 18 settings.

type LazyPrunerStats ¶

type LazyPrunerStats struct {
	TotalPruned   int
	TotalRevived  int
	TokensSaved   int
	LayersApplied int
}

LazyPrunerStats tracks pruning statistics

type LineScore ¶

type LineScore struct {
	Index int
	Line  string
	Score float64
}

LineScore holds a line with its relevance score.

type LogCrunch ¶

type LogCrunch struct {
	// contains filtered or unexported fields
}

LogCrunch folds repeated log lines with occurrence counts. Inspired by claw-compactor's LogCrunch stage.

func NewLogCrunch(cfg LogCrunchConfig) *LogCrunch

func (lc *LogCrunch) Process(content string) (string, int)

type LogCrunchConfig ¶

type LogCrunchConfig struct {
	Enabled        bool
	MinRepetitions int
	AlwaysPreserve []string
}

LogCrunchConfig holds configuration for LogCrunch.

func DefaultLogCrunchConfig() LogCrunchConfig

type ManagerConfig ¶

type ManagerConfig struct {
	// Context limits
	MaxContextTokens int
	ChunkSize        int
	StreamThreshold  int

	// Resilience
	TeeOnFailure       bool
	FailSafeMode       bool
	ValidateOutput     bool
	ShortCircuitBudget bool

	// Performance
	CacheEnabled bool
	CacheMaxSize int

	// Layer config
	PipelineCfg PipelineConfig
}

ManagerConfig configures the pipeline manager

type MetaToken ¶

type MetaToken struct {
	Hash     string // SHA256 hash of the original sequence
	Original string // Original text that was compressed
	Length   int    // Number of tokens in original sequence
	Count    int    // Number of times this pattern was found
}

MetaToken represents a compressed token sequence

type MetaTokenConfig ¶

type MetaTokenConfig struct {
	// WindowSize is the maximum sequence length to consider for compression
	WindowSize int

	// MinPattern is the minimum sequence length to compress (shorter = more compression but more meta-tokens)
	MinPattern int

	// MaxMetaTokens limits the number of meta-tokens created (0 = unlimited)
	MaxMetaTokens int

	// EnableDecompression allows this filter to also decompress
	EnableDecompression bool
}

MetaTokenConfig holds configuration for the meta-token filter

func DefaultMetaTokenConfig() MetaTokenConfig

type MetaTokenFilter ¶

type MetaTokenFilter struct {
	// contains filtered or unexported fields
}

Paper: "Lossless Token Compression via Meta-Tokens" — 2025 https://arxiv.org/abs/2506.00307 Paper: "Lossless Token Compression via Meta-Tokens" — 2025 https://arxiv.org/abs/2506.00307 MetaTokenFilter implements Layer 15: Lossless Token Sequence Compression via Meta-Tokens.

Research Source: "Lossless Token Sequence Compression via Meta-Tokens" (arXiv:2506.00307) Key Innovation: LZ77-style lossless compression operating on token sequences. Results: 27% token reduction = 47% compute reduction (due to quadratic attention) Critical Feature: ZERO semantic loss - trivially reversible.

Methodology: 1. Scan for repeated token sequences (sliding window) 2. Replace with meta-tokens that reference the original sequence 3. Meta-tokens use special marker format: [META:hash:length] 4. Decompression expands meta-tokens back to original sequences

func NewMetaTokenFilter() *MetaTokenFilter

func NewMetaTokenFilterWithConfig(cfg MetaTokenConfig) *MetaTokenFilter

func (f *MetaTokenFilter) Apply(input string, mode Mode) (string, int)

func (f *MetaTokenFilter) Decompress(input string) string

func (f *MetaTokenFilter) GetMetaTokens() map[string]MetaToken

func (f *MetaTokenFilter) LoadMetaTokens(tokens map[string]MetaToken)

func (f *MetaTokenFilter) Name() string

func (f *MetaTokenFilter) Stats() MetaTokenStats

type MetaTokenLayerConfig ¶

type MetaTokenLayerConfig struct {
	Enabled bool
	Window  int
	MinSize int
}

MetaTokenLayerConfig groups Layer 15 settings.

type MetaTokenStats ¶

type MetaTokenStats struct {
	UniquePatterns int
	TotalPatterns  int
	EstTokensSaved int
}

MetaTokenStats holds statistics for meta-token compression

type Milestone ¶

type Milestone struct {
	Description string `json:"description"`
	Priority    int    `json:"priority"`
	Status      string `json:"status"`
}

Milestone represents a pending task

type MixedDimFilter ¶

type MixedDimFilter struct {
	// contains filtered or unexported fields
}

Paper: "MixedDimKV: Beyond Token Eviction" — Miao et al., 2026 https://arxiv.org/abs/2603.20616 MixedDimFilter implements mixed-dimension token allocation — instead of evicting tokens entirely (0 or 100%), it reduces the "dimensionality" of less important tokens by abbreviating them.

func NewMixedDimFilter() *MixedDimFilter

func (f *MixedDimFilter) Apply(input string, mode Mode) (string, int)

func (f *MixedDimFilter) Name() string

type Mode ¶

type Mode string

Mode represents the filtering mode.

const (
	ModeNone       Mode = "none"
	ModeMinimal    Mode = "minimal"
	ModeAggressive Mode = "aggressive"
)

type MultiAgentContextSharing ¶

type MultiAgentContextSharing struct {
	// contains filtered or unexported fields
}

MultiAgentContextSharing implements multi-agent context sharing. Inspired by lean-ctx's multi-agent context sharing.

func NewMultiAgentContextSharing() *MultiAgentContextSharing

func (macs *MultiAgentContextSharing) GetMessages(agent string) []ScratchMessage

func (macs *MultiAgentContextSharing) PostMessage(from, to, content string)

func (macs *MultiAgentContextSharing) RegisterAgent(name string)

type MultiFileConfig ¶

type MultiFileConfig struct {
	MaxCombinedSize     int
	PreserveBoundaries  bool
	SimilarityThreshold float64
}

MultiFileConfig holds configuration for multi-file optimization

type MultiFileFilter ¶

type MultiFileFilter struct {
	// contains filtered or unexported fields
}

MultiFileFilter optimizes output across multiple related files/outputs. It identifies relationships between files, deduplicates common content, and creates unified summaries for better LLM context.

Use case: When an agent works with multiple related files simultaneously (e.g., a module with multiple source files), this filter creates a cohesive view that preserves relationships while removing redundancy.

func NewMultiFileFilter(cfg MultiFileConfig) *MultiFileFilter

func (f *MultiFileFilter) Apply(input string, mode Mode) (string, int)

func (f *MultiFileFilter) Name() string

func (f *MultiFileFilter) SetMaxCombinedSize(size int)

func (f *MultiFileFilter) SetPreserveBoundaries(preserve bool)

func (f *MultiFileFilter) SetSimilarityThreshold(threshold float64)

type NgramAbbreviator ¶

type NgramAbbreviator struct {
	// contains filtered or unexported fields
}

NgramAbbreviator compresses output by abbreviating common patterns. Research-based: CompactPrompt N-gram Abbreviation (2025) - achieves 10-20% lossless compression by replacing common tokens with shorter equivalents.

Key insight: Programming and CLI output contains many repeated long tokens that can be abbreviated while remaining understandable to LLMs.

func NewNgramAbbreviator() *NgramAbbreviator

func (f *NgramAbbreviator) Apply(input string, mode Mode) (string, int)

func (f *NgramAbbreviator) GetAbbreviationLegend() string

func (f *NgramAbbreviator) Name() string

type NumericalConfig ¶

type NumericalConfig struct {
	// Enabled controls whether the filter is active
	Enabled bool

	// DecimalPlaces limits decimal precision (e.g., 2 = max 2 decimal places)
	DecimalPlaces int

	// CompressLargeNumbers replaces large numbers with K/M/B suffixes
	CompressLargeNumbers bool

	// LargeNumberThreshold is the threshold for large number compression
	LargeNumberThreshold int

	// CompressPercentages simplifies percentage display
	CompressPercentages bool

	// MinContentLength is minimum content length to apply
	MinContentLength int
}

NumericalConfig holds configuration for numerical quantization

func DefaultNumericalConfig() NumericalConfig

type NumericalQuantLayerConfig ¶

type NumericalQuantLayerConfig struct {
	Enabled       bool
	DecimalPlaces int
}

NumericalQuantLayerConfig groups numerical quantization settings.

type NumericalQuantizer ¶

type NumericalQuantizer struct {
	// contains filtered or unexported fields
}

NumericalQuantizer compresses numerical data in structured output. Research Source: "CompactPrompt: A Unified Pipeline for Prompt Data Compression" (Oct 2025) Key Innovation: Apply uniform quantization to numerical columns while preserving semantic relationships, achieving significant token savings on structured data.

This filter detects tables, metrics, statistics, and numerical data in output and applies precision reduction and formatting compression.

func NewNumericalQuantizer() *NumericalQuantizer

func (n *NumericalQuantizer) Apply(input string, mode Mode) (string, int)

func (n *NumericalQuantizer) Name() string

type Observation ¶

type Observation struct {
	Content    string
	Importance float64
	Timestamp  string
}

Observation represents a single observation from LLM output.

type PATHShimInjector ¶

type PATHShimInjector struct {
	// contains filtered or unexported fields
}

PATHShimInjector creates PATH shims to auto-filter subprocesses. Inspired by tokf's PATH shim injection.

func NewPATHShimInjector(shimDir string) *PATHShimInjector

func (psi *PATHShimInjector) Install(commands []string) error

func (psi *PATHShimInjector) Uninstall() error

func (psi *PATHShimInjector) UpdatePATH(currentPath string) string

type PerplexityFilter ¶

type PerplexityFilter struct {
	// contains filtered or unexported fields
}

Paper: "LLMLingua" — Jiang et al., Microsoft/Tsinghua, 2023 https://arxiv.org/abs/2310.05736 Paper: "LLMLingua" — Jiang et al., Microsoft/Tsinghua, 2023 https://arxiv.org/abs/2310.05736 PerplexityFilter implements LLMLingua-style compression (Microsoft/Tsinghua, 2023). Uses perplexity-based iterative pruning with a budget controller.

Algorithm: 1. Calculate perplexity of each token given context 2. Rank tokens by perplexity (higher = more surprising = more important) 3. Iteratively remove lowest-perplexity tokens while staying within budget

Research Results: Up to 20x compression with semantic preservation.

func NewPerplexityFilter() *PerplexityFilter

func (f *PerplexityFilter) Apply(input string, mode Mode) (string, int)

func (f *PerplexityFilter) Name() string

func (f *PerplexityFilter) SetIterations(iterations int)

func (f *PerplexityFilter) SetTargetRatio(ratio float64)

type PersistentKnowledgeStore ¶

type PersistentKnowledgeStore struct {
	// contains filtered or unexported fields
}

PersistentKnowledgeStore implements persistent knowledge storage. Inspired by lean-ctx's persistent knowledge store.

func NewPersistentKnowledgeStore() *PersistentKnowledgeStore

func (pks *PersistentKnowledgeStore) QueryByCategory(category string) map[string]string

func (pks *PersistentKnowledgeStore) Recall(key string) string

func (pks *PersistentKnowledgeStore) Remember(key, value string)

type PhotonConfig ¶

type PhotonConfig struct {
	Enabled       bool
	MaxInlineSize int
	ReplaceWith   string
}

PhotonConfig holds configuration for Photon filter.

func DefaultPhotonConfig() PhotonConfig

type PhotonFilter ¶

type PhotonFilter struct {
	// contains filtered or unexported fields
}

PhotonFilter detects and compresses base64-encoded images. Inspired by claw-compactor's Photon stage.

func NewPhotonFilter(cfg PhotonConfig) *PhotonFilter

func (pf *PhotonFilter) Process(content string) (string, int)

type PhraseGroupConfig ¶

type PhraseGroupConfig struct {
	// Enabled controls whether the filter is active
	Enabled bool

	// MinContentLength is the minimum character length to apply
	MinContentLength int

	// MaxPhraseSize is maximum tokens in a phrase group
	MaxPhraseSize int
}

PhraseGroupConfig holds configuration for phrase grouping

func DefaultPhraseGroupConfig() PhraseGroupConfig

type PhraseGroupingFilter ¶

type PhraseGroupingFilter struct {
	// contains filtered or unexported fields
}

PhraseGroupingFilter implements dependency-based phrase grouping for compression. Research Source: "CompactPrompt: A Unified Pipeline for Prompt Data Compression" (Oct 2025) Key Innovation: Group related tokens using syntactic dependency analysis before compression, preserving semantic coherence better than token-level pruning.

This identifies noun phrases, verb phrases, and prepositional phrases as atomic compression units, preventing the separation of semantically linked tokens.

func NewPhraseGroupingFilter() *PhraseGroupingFilter

func (f *PhraseGroupingFilter) Apply(input string, mode Mode) (string, int)

func (f *PhraseGroupingFilter) Name() string

type PipeOp ¶

type PipeOp struct {
	Type string
	Args []string
}

PipeOp represents a single pipe operation.

type Pipeline ¶

type Pipeline interface {
	Process(input string) (string, *PipelineStats)
}

Pipeline defines the interface for compression pipelines. This allows mock testing and future pipeline implementations.

type PipelineConfig ¶

type PipelineConfig = PipelineConfigWithNestedLayers

PipelineConfig is an alias for the full config type with backward-compatible flat fields. New code should use PipelineConfigWithNestedLayers to take advantage of nested structure.

func LoadPipelineFromTOML(path string) (PipelineConfig, error)

func ModeConfig(mode CompressionMode, baseMode Mode) PipelineConfig

func PresetConfig(preset PipelinePreset, baseMode Mode) PipelineConfig

func ProfileConfig(profile Profile, baseMode Mode) PipelineConfig

func TierConfig(tier Tier, baseMode Mode) PipelineConfig

type PipelineConfigWithNestedLayers ¶

type PipelineConfigWithNestedLayers struct {
	// Core fields
	Mode              Mode
	QueryIntent       string
	Budget            int
	LLMEnabled        bool
	SessionTracking   bool
	NgramEnabled      bool
	MultiFileEnabled  bool
	PromptTemplate    string
	EnableTOMLFilter  bool
	TOMLFilterCommand string

	// Layer sub-configs (preferred)
	Layers LayerConfig

	// Core layer enable flags (Layers 1-9)
	EnableEntropy      bool
	EnablePerplexity   bool
	EnableGoalDriven   bool
	EnableAST          bool
	EnableContrastive  bool
	EnableEvaluator    bool
	EnableGist         bool
	EnableHierarchical bool

	// Layer 11: Compaction
	EnableCompaction        bool
	CompactionThreshold     int
	CompactionPreserveTurns int
	CompactionMaxTokens     int
	CompactionStateSnapshot bool
	CompactionAutoDetect    bool

	// Layer 12: Attribution
	EnableAttribution    bool
	AttributionThreshold float64

	// Layer 13: H2O
	EnableH2O          bool
	H2OSinkSize        int
	H2ORecentSize      int
	H2OHeavyHitterSize int

	// Layer 14: Attention Sink
	EnableAttentionSink  bool
	AttentionSinkCount   int
	AttentionRecentCount int

	// Layer 15: Meta-Token
	EnableMetaToken  bool
	MetaTokenWindow  int
	MetaTokenMinSize int

	// Layer 16: Semantic Chunk
	EnableSemanticChunk    bool
	SemanticChunkMethod    string
	SemanticChunkMinSize   int
	SemanticChunkThreshold float64

	// Layer 17: Sketch Store
	EnableSketchStore bool
	SketchBudgetRatio float64
	SketchMaxSize     int
	SketchHeavyHitter float64

	// Layer 18: Lazy Pruner
	EnableLazyPruner  bool
	LazyBaseBudget    int
	LazyDecayRate     float64
	LazyRevivalBudget int

	// Layer 19: Semantic Anchor
	EnableSemanticAnchor  bool
	SemanticAnchorRatio   float64
	SemanticAnchorSpacing int

	// Layer 20: Agent Memory
	EnableAgentMemory       bool
	AgentKnowledgeRetention float64
	AgentHistoryPrune       float64
	AgentConsolidationMax   int

	// Adaptive layers
	EnableQuestionAware    bool
	QuestionAwareThreshold float64
	EnableDensityAdaptive  bool
	DensityTargetRatio     float64
	DensityThreshold       float64

	// TF-IDF
	EnableTFIDF    bool
	TFIDFThreshold float64

	// Reasoning trace
	EnableReasoningTrace bool
	MaxReflectionLoops   int

	// Phase 1: NEW filters
	EnableSymbolicCompress bool
	EnablePhraseGrouping   bool
	EnableNumericalQuant   bool
	DecimalPlaces          int
	EnableDynamicRatio     bool
	DynamicRatioBase       float64

	// Phase 2: Advanced filters
	EnableHypernym      bool
	EnableSemanticCache bool
	EnableScope         bool
	EnableSmallKV       bool
	EnableKVzip         bool

	// 2026 Research layers
	EnableSWEzze         bool
	EnableMixedDim       bool
	EnableBEAVER         bool
	EnablePoC            bool
	EnableTokenQuant     bool
	EnableTokenRetention bool
	EnableACON           bool

	// Cache
	CacheEnabled bool
	CacheMaxSize int
}

PipelineConfigWithNestedLayers is a helper type for the new nested config structure. Use this gradually: migrate from flat fields to nested Layers config over time.

type PipelineCoordinator ¶

type PipelineCoordinator struct {
	// contains filtered or unexported fields
}

PipelineCoordinator orchestrates the 26-layer compression pipeline. Research-based: Combines the best techniques from 120+ research papers worldwide to achieve maximum token reduction for CLI/Agent output.

func NewPipelineCoordinator(cfg PipelineConfig) *PipelineCoordinator

func (c *PipelineCoordinator) GetASTPreserveFilter() *ASTPreserveFilter

func (c *PipelineCoordinator) GetAgentMemoryFilter() *AgentMemoryFilter

func (c *PipelineCoordinator) GetAttentionSinkFilter() *AttentionSinkFilter

func (c *PipelineCoordinator) GetAttributionFilter() *AttributionFilter

func (c *PipelineCoordinator) GetCompactionLayer() *CompactionLayer

func (c *PipelineCoordinator) GetContrastiveFilter() *ContrastiveFilter

func (c *PipelineCoordinator) GetEntropyFilter() *EntropyFilter

func (c *PipelineCoordinator) GetEvaluatorHeadsFilter() *EvaluatorHeadsFilter

func (c *PipelineCoordinator) GetGistFilter() *GistFilter

func (c *PipelineCoordinator) GetGoalDrivenFilter() *GoalDrivenFilter

func (c *PipelineCoordinator) GetH2OFilter() *H2OFilter

func (c *PipelineCoordinator) GetHierarchicalSummaryFilter() *HierarchicalSummaryFilter

func (c *PipelineCoordinator) GetLazyPrunerFilter() *LazyPrunerFilter

func (c *PipelineCoordinator) GetMetaTokenFilter() *MetaTokenFilter

func (c *PipelineCoordinator) GetNgramAbbreviator() *NgramAbbreviator

func (c *PipelineCoordinator) GetPerplexityFilter() *PerplexityFilter

func (c *PipelineCoordinator) GetSemanticAnchorFilter() *SemanticAnchorFilter

func (c *PipelineCoordinator) GetSemanticChunkFilter() *SemanticChunkFilter

func (c *PipelineCoordinator) GetSketchStoreFilter() *SketchStoreFilter

func (c *PipelineCoordinator) GetTFIDFFilter() *TFIDFFilter

func (p *PipelineCoordinator) GetTOMLFilterName() string

func (p *PipelineCoordinator) Process(input string) (string, *PipelineStats)

func (p *PipelineCoordinator) SetTOMLFilter(filter Filter, name string)

type PipelineManager ¶

type PipelineManager struct {
	// contains filtered or unexported fields
}

PipelineManager handles resilient large-context processing. Supports streaming for inputs up to 2M tokens with automatic chunking, validation, and failure recovery.

func NewPipelineManager(cfg ManagerConfig) *PipelineManager

func (m *PipelineManager) Process(input string, mode Mode, ctx CommandContext) (*ProcessResult, error)

func (m *PipelineManager) ProcessWithBudget(input string, mode Mode, budget int, ctx CommandContext) (*ProcessResult, error)

func (m *PipelineManager) ProcessWithQuery(input string, mode Mode, query string, ctx CommandContext) (*ProcessResult, error)

type PipelinePreset ¶

type PipelinePreset = Tier

PresetConfig for backwards compatibility.

type PipelineSection ¶

type PipelineSection struct {
	Mode        string `toml:"mode"`
	Budget      int    `toml:"budget"`
	QueryIntent string `toml:"query_intent"`
	LLMEnabled  bool   `toml:"llm_enabled"`
}

PipelineSection holds pipeline configuration.

type PipelineStats ¶

type PipelineStats struct {
	OriginalTokens   int
	FinalTokens      int
	TotalSaved       int
	ReductionPercent float64
	LayerStats       map[string]LayerStat

	CacheHit bool
	// contains filtered or unexported fields
}

PipelineStats holds statistics from the compression pipeline

func (s *PipelineStats) String() string

type PoCFilter ¶

type PoCFilter struct {
	// contains filtered or unexported fields
}

Paper: "PoC: Performance-oriented Context Compression" — 2026 https://arxiv.org/abs/2603.19733 PoCFilter implements performance prediction — estimates how well the compressed output will perform and re-inserts critical info if needed.

func NewPoCFilter() *PoCFilter

func (f *PoCFilter) Apply(input string, mode Mode) (string, int)

func (f *PoCFilter) Name() string

type PositionAwareFilter ¶

type PositionAwareFilter struct{}

PositionAwareFilter reorders output segments to optimize LLM recall. Based on "LongLLMLingua" (Jiang et al., 2024) - LLMs exhibit "lost in the middle" phenomenon where information in the middle of context is less likely to be recalled.

Strategy: Place high-importance segments at beginning AND end of output.

func NewPositionAwareFilter() *PositionAwareFilter

func (f *PositionAwareFilter) Apply(input string, mode Mode) (string, int)