filter

type ANSIFilter ¶

type ANSIFilter struct{}

ANSIFilter strips ANSI escape sequences from output.

func NewANSIFilter() *ANSIFilter

func (f *ANSIFilter) Apply(input string, mode Mode) (string, int)

func (f *ANSIFilter) Name() string

type ASTPreserveFilter ¶

type ASTPreserveFilter struct {
	// contains filtered or unexported fields
}

ASTPreserveFilter implements LongCodeZip-style compression (NUS, 2025). AST-aware compression that preserves syntactic validity of code.

Algorithm: 1. Detect programming language from syntax patterns 2. Parse code structure (brackets, braces, indentation) 3. Apply entropy-based pruning while preserving AST integrity 4. Never break syntactic boundaries (function bodies, blocks, strings)

Research Results: 4-8x compression while maintaining parseable code.

func NewASTPreserveFilter() *ASTPreserveFilter

func (f *ASTPreserveFilter) Apply(input string, mode Mode) (string, int)

func (f *ASTPreserveFilter) Name() string

type AdaptiveAttentionFilter ¶ added in v1.2.0

type AdaptiveAttentionFilter struct {
	// contains filtered or unexported fields
}

AdaptiveAttentionFilter implements ADSC-style compression (Berkeley/Clemson, 2025). Attention-driven self-compression that mimics LLM attention patterns.

Algorithm: 1. Simulate attention scoring for each token/line 2. Apply progressive downsampling based on attention scores 3. Use uniform downsampling for low-attention regions 4. Preserve high-attention regions in full detail

Research Results: 66.7% - 88.9% compression with 98.2% quality preservation. Key insight: Attention patterns reveal token importance.

func NewAdaptiveAttentionFilter() *AdaptiveAttentionFilter

func (f *AdaptiveAttentionFilter) Apply(input string, mode Mode) (string, int)

func (f *AdaptiveAttentionFilter) Name() string

func (f *AdaptiveAttentionFilter) SetDownsampleRatio(ratio float64)

func (f *AdaptiveAttentionFilter) SetThresholds(high, low float64)

type AttentionSinkFilter ¶

type AttentionSinkFilter struct {
	// contains filtered or unexported fields
}

AttentionSinkFilter implements StreamingLLM-style attention sink preservation. Research basis: "Efficient Streaming Language Models with Attention Sinks" (Xiao et al., 2023) - enables infinite-length generation with bounded memory.

Key insight: The first few tokens in a sequence act as "attention sinks" - they absorb excess attention weight due to softmax normalization. Removing these tokens breaks the attention distribution the model learned during training.

This layer: 1. Always preserves initial tokens (attention sinks) 2. Preserves structural anchors (headers, prefixes, markers) 3. Applies rolling cache to remaining content

This is Layer 14 in the pipeline, ensuring stable compression for long content.

func NewAttentionSinkFilter() *AttentionSinkFilter

func (a *AttentionSinkFilter) Apply(input string, mode Mode) (string, int)

func (a *AttentionSinkFilter) GetStats() map[string]interface{}

func (a *AttentionSinkFilter) Name() string

func (a *AttentionSinkFilter) SetEnabled(enabled bool)

type AttributionConfig ¶

type AttributionConfig struct {
	// Enable attribution filtering
	Enabled bool

	// Threshold for token importance (0.0-1.0)
	// Tokens below this score are candidates for removal
	ImportanceThreshold float64

	// Minimum content length to apply attribution
	MinContentLength int

	// Enable caching of importance scores
	CacheEnabled bool

	// Use positional bias (later tokens often less important)
	PositionalBias bool

	// Use frequency-based importance (repeated tokens may be less important)
	FrequencyBias bool

	// Use semantic markers (preserve keywords, numbers, code)
	SemanticPreservation bool

	// Maximum tokens to analyze (for performance)
	MaxAnalyzeTokens int
}

AttributionConfig holds configuration for attribution-based pruning

func DefaultAttributionConfig() AttributionConfig

type AttributionFilter ¶

type AttributionFilter struct {
	// contains filtered or unexported fields
}

AttributionFilter implements ProCut-style attribution-based token pruning. Research basis: "ProCut: Progressive Pruning via Attribution" (LinkedIn, 2025) Achieves 78% token reduction by using importance scoring.

Key technique: Attribution scores (simplified SHAP) identify which tokens contribute most to the output. Low-importance tokens are pruned.

This is Layer 12 in the pipeline, adding ML-style importance without requiring actual model training.

func NewAttributionFilter() *AttributionFilter

func (a *AttributionFilter) Apply(input string, mode Mode) (string, int)

func (a *AttributionFilter) GetStats() map[string]interface{}

func (a *AttributionFilter) Name() string

func (a *AttributionFilter) SetEnabled(enabled bool)

type BodyFilter ¶

type BodyFilter struct {
	// contains filtered or unexported fields
}

BodyFilter strips function bodies in aggressive mode. Preserves function signatures while removing body content.

func NewBodyFilter() *BodyFilter

func (f *BodyFilter) Apply(input string, mode Mode) (string, int)

func (f *BodyFilter) Name() string

type BudgetConfig ¶

type BudgetConfig struct {
	Budget int // Maximum tokens (0 = unlimited)
}

BudgetConfig holds configuration for the budget enforcer

type BudgetEnforcer ¶

type BudgetEnforcer struct {
	// contains filtered or unexported fields
}

BudgetEnforcer enforces strict token limits on output. Research-based: Budget-Constrained Compression (2024) - provides predictable output size by scoring segments and keeping only the most important ones.

Key insight: LLMs have finite context windows. Enforcing a strict budget ensures output fits within constraints while maximizing information content.

func NewBudgetEnforcer(budget int) *BudgetEnforcer

func NewBudgetEnforcerWithConfig(cfg BudgetConfig) *BudgetEnforcer

func (f *BudgetEnforcer) Apply(input string, mode Mode) (string, int)

func (f *BudgetEnforcer) Name() string

func (f *BudgetEnforcer) SetBudget(budget int)

type CachedResult ¶

type CachedResult struct {
	Output   string
	Tokens   int
	CachedAt time.Time
}

CachedResult represents a cached compression result

type CodeContext ¶

type CodeContext struct {
	File    string   `json:"file"`
	Symbols []string `json:"symbols"`
	Lines   string   `json:"lines,omitempty"`
}

CodeContext preserves code-specific context

type CommandContext ¶

type CommandContext struct {
	Command    string // "git", "npm", "cargo", etc.
	Subcommand string // "status", "test", "build"
	ExitCode   int    // Non-zero = likely has errors
	Intent     string // "debug", "review", "deploy", "search"
	IsTest     bool   // Test output detection
	IsBuild    bool   // Build output detection
	IsError    bool   // Error output detection
}

CommandContext provides metadata about the command being executed. Used for intelligent filtering decisions.

type CommentFilter ¶

type CommentFilter struct {
	// contains filtered or unexported fields
}

CommentFilter strips comments from source code.

func NewCommentFilter() *CommentFilter

func (f *CommentFilter) Apply(input string, mode Mode) (string, int)

func (f *CommentFilter) Name() string

type CommentPatterns ¶

type CommentPatterns struct {
	Line       string
	BlockStart string
	BlockEnd   string
	DocLine    string
	DocBlock   string
}

CommentPatterns represents comment structure for a language

func CommentPatternsForLang(lang Language) CommentPatterns

type CompactionConfig ¶

type CompactionConfig struct {
	// Enable LLM-based compaction
	Enabled bool

	// Minimum content size to trigger compaction (in lines)
	ThresholdLines int

	// Minimum content size to trigger compaction (in tokens)
	ThresholdTokens int

	// Number of recent turns to preserve verbatim
	PreserveRecentTurns int

	// Maximum summary length in tokens
	MaxSummaryTokens int

	// Content types to compact (chat, conversation, session)
	ContentTypes []string

	// Enable caching of compaction results
	CacheEnabled bool

	// Custom prompt template for compaction
	PromptTemplate string

	// Detect content type automatically
	AutoDetect bool

	// Create state snapshot format (4-section XML)
	StateSnapshotFormat bool

	// Extract key-value pairs from content
	ExtractKeyValuePairs bool

	// Maximum context entries to preserve
	MaxContextEntries int
}

CompactionConfig holds configuration for the compaction layer

func DefaultCompactionConfig() CompactionConfig

type CompactionLayer ¶

type CompactionLayer struct {
	// contains filtered or unexported fields
}

CompactionLayer provides semantic compression for chat/conversation content. It creates state snapshots with 4 sections:

1. session_history: user queries + activity log (what was done) 2. current_state: focus + next_action (what's active now) 3. context: critical + working knowledge (what to remember) 4. pending_plan: future milestones (what's next)

Research basis: "MemGPT" (UC Berkeley, 2023) semantic compression achieves 98%+ compression ratios while preserving semantic meaning.

This layer is designed for: - Chat history compression - Conversation-style content - Session state preservation - Multi-turn context management

func NewCompactionLayer(cfg CompactionConfig) *CompactionLayer

func (c *CompactionLayer) Apply(input string, mode Mode) (string, int)

func (c *CompactionLayer) GetStats() map[string]interface{}

func (c *CompactionLayer) IsAvailable() bool

func (c *CompactionLayer) Name() string

func (c *CompactionLayer) SetEnabled(enabled bool)

type CompactionResult ¶

type CompactionResult struct {
	Snapshot         *StateSnapshot
	OriginalTokens   int
	FinalTokens      int
	SavedTokens      int
	CompressionRatio float64
	Cached           bool
	Timestamp        time.Time
}

CompactionResult represents a compaction result

func Compact(input string, cfg CompactionConfig) (string, *CompactionResult)

type CompressionCache ¶

type CompressionCache struct {
	// contains filtered or unexported fields
}

CompressionCache provides caching for compression results

func NewCompressionCache(maxSize int) *CompressionCache

func (c *CompressionCache) Get(key string) (*CachedResult, bool)

func (c *CompressionCache) Set(key string, result *CachedResult)

func (c *CompressionCache) Size() int

type ContrastiveFilter ¶

type ContrastiveFilter struct {
	// contains filtered or unexported fields
}

ContrastiveFilter implements LongLLMLingua contrastive perplexity (Microsoft, 2024). Question-aware compression that ranks tokens by relevance to the query.

Algorithm: 1. Calculate contrastive perplexity: CP(x) = P(x|question) / P(x|context) 2. Higher contrastive perplexity = more question-relevant 3. Reorder context to place high-relevance tokens at start/end 4. Prune low-relevance middle content

Research Results: 4-10x compression with improved RAG accuracy.

func NewContrastiveFilter(question string) *ContrastiveFilter

func (f *ContrastiveFilter) Apply(input string, mode Mode) (string, int)

func (f *ContrastiveFilter) Name() string

func (f *ContrastiveFilter) SetQuestion(question string)

type ConversationTracker ¶

type ConversationTracker struct {
	// contains filtered or unexported fields
}

ConversationTracker tracks conversation turns

func NewConversationTracker(maxTurns int) *ConversationTracker

func (t *ConversationTracker) AddTurn(role, content string)

func (t *ConversationTracker) GetRecentTurns(n int) []Turn

func (t *ConversationTracker) GetTurns() []Turn

type CurrentState ¶

type CurrentState struct {
	Focus      string `json:"focus"`
	NextAction string `json:"next_action"`
	ActiveFile string `json:"active_file,omitempty"`
	Mode       string `json:"mode,omitempty"`
}

CurrentState tracks what's currently active

type Engine ¶

type Engine struct {
	// contains filtered or unexported fields
}

Engine combines multiple filters.

func NewEngine(mode Mode) *Engine

func NewEngineWithConfig(cfg EngineConfig) *Engine

func NewEngineWithLLM(mode Mode, queryIntent string, llmEnabled bool) *Engine

func NewEngineWithLLMAndConfig(cfg EngineConfig) *Engine

func NewEngineWithQuery(mode Mode, queryIntent string) *Engine

func (e *Engine) Process(input string) (string, int)

func (e *Engine) ProcessWithLang(input string, lang string) (string, int)

func (e *Engine) SetMode(mode Mode)

type EngineConfig ¶

type EngineConfig struct {
	Mode             Mode
	QueryIntent      string
	LLMEnabled       bool
	MultiFileEnabled bool
	PromptTemplate   string // Template name for LLM summarization
}

EngineConfig holds configuration for the filter engine

type EntropyFilter ¶

type EntropyFilter struct {
	// contains filtered or unexported fields
}

EntropyFilter implements Selective Context compression (Mila/Guerin et al., 2023). Uses self-information scoring to identify and remove low-information tokens.

Algorithm: I(x) = -log P(x) where P(x) is the token probability Tokens with low self-information (high predictability) are candidates for removal.

Research Results: 2-3x compression while preserving semantic content.

func NewEntropyFilter() *EntropyFilter

func (f *EntropyFilter) Apply(input string, mode Mode) (string, int)

func (f *EntropyFilter) Name() string

func (f *EntropyFilter) SetThreshold(threshold float64)

type ErrorTraceFilter ¶ added in v1.2.0

type ErrorTraceFilter struct {
	// contains filtered or unexported fields
}

ErrorTraceFilter compresses error stack traces to essential information. Research-based: Error Message Compression (2024) - achieves 4x compression by showing only: error type, first user file, line number, and message.

Key insight: Stack traces are often 50+ lines, but only 2-3 lines are actionable. Framework internals, standard library paths, and repetitive frames can be collapsed.

func NewErrorTraceFilter() *ErrorTraceFilter

func (f *ErrorTraceFilter) Apply(input string, mode Mode) (string, int)

func (f *ErrorTraceFilter) Name() string

type EvaluatorHeadsFilter ¶

type EvaluatorHeadsFilter struct {
	// contains filtered or unexported fields
}

EvaluatorHeadsFilter implements EHPC-style compression (Tsinghua/Huawei, 2025). Uses "evaluator heads" concept - identifies important tokens by analyzing early-layer attention patterns.

Algorithm: 1. Simulate "skim" mode - look at first few tokens of each chunk 2. Score tokens by position and content importance 3. Identify "evaluator" tokens that predict importance 4. Apply rapid pruning based on evaluator scores

Research Results: 5-7x compression with minimal quality loss. Key insight: Early layers of LLMs can predict token importance.

func NewEvaluatorHeadsFilter() *EvaluatorHeadsFilter

func (f *EvaluatorHeadsFilter) Apply(input string, mode Mode) (string, int)

func (f *EvaluatorHeadsFilter) Name() string

func (f *EvaluatorHeadsFilter) SetEvalThreshold(threshold float64)

func (f *EvaluatorHeadsFilter) SetSkimRatio(ratio float64)

type Filter ¶

type Filter interface {
	// Name returns the filter name.
	Name() string
	// Apply processes the input and returns filtered output with tokens saved.
	Apply(input string, mode Mode) (output string, tokensSaved int)
}

Filter defines the interface for output filters.

type GistFilter ¶

type GistFilter struct {
	// contains filtered or unexported fields
}

GistFilter implements Gisting compression (Stanford/Berkeley, 2023). Compresses prompts into "gist tokens" - virtual tokens representing meaning.

Algorithm: 1. Identify semantic chunks in the text 2. Replace each chunk with a compressed "gist" representation 3. Use prefix-tuning style markers for reconstruction 4. Preserve critical structural elements

Research Results: 20x+ compression for repetitive content. Key insight: LLMs can understand compressed "gist" representations.

func NewGistFilter() *GistFilter

func (f *GistFilter) Apply(input string, mode Mode) (string, int)

func (f *GistFilter) Name() string

func (f *GistFilter) SetMaxChunkSize(size int)

type GoalDrivenFilter ¶

type GoalDrivenFilter struct {
	// contains filtered or unexported fields
}

GoalDrivenFilter implements SWE-Pruner style compression (Shanghai Jiao Tong, 2025). Goal-driven line-level pruning using CRF-inspired scoring.

Algorithm: 1. Parse goal/intent from query 2. Score each line for relevance to goal 3. Apply CRF-style sequential labeling for keep/prune decisions 4. Preserve structural coherence

Research Results: Up to 14.8x compression for code contexts.

func NewGoalDrivenFilter(goal string) *GoalDrivenFilter

func (f *GoalDrivenFilter) Apply(input string, mode Mode) (string, int)

func (f *GoalDrivenFilter) Name() string

type GoalMode ¶

type GoalMode int

GoalMode defines the goal-driven filtering mode

const (
	GoalModeDebug GoalMode = iota
	GoalModeReview
	GoalModeDeploy
	GoalModeSearch
	GoalModeBuild
	GoalModeTest
	GoalModeGeneric
)

type H2OConfig ¶

type H2OConfig struct {
	// Enable H2O filtering
	Enabled bool

	// Number of attention sink tokens to always preserve (first N tokens)
	SinkSize int

	// Number of recent tokens to always preserve
	RecentSize int

	// Number of heavy hitter tokens to preserve based on importance
	HeavyHitterSize int

	// Minimum content length to apply compression
	MinContentLength int

	// Window size for chunk processing
	ChunkWindow int
}

H2OConfig holds configuration for H2O compression

func DefaultH2OConfig() H2OConfig

type H2OFilter ¶

type H2OFilter struct {
	// contains filtered or unexported fields
}

H2OFilter implements Heavy-Hitter Oracle compression. Research basis: "H2O: Heavy-Hitter Oracle for Efficient Generative Inference" (Zhang et al., NeurIPS 2023) - achieves 30x+ compression via intelligent eviction.

Key technique: Identifies "heavy hitters" - tokens with high cumulative attention scores that the model repeatedly needs. Combines with: 1. Recent token window for local context 2. Attention sinks (initial tokens) for computational stability

This is Layer 13 in the pipeline, implementing KV cache-style compression for text without requiring actual model attention scores.

func NewH2OFilter() *H2OFilter

func (h *H2OFilter) Apply(input string, mode Mode) (string, int)

func (h *H2OFilter) GetStats() map[string]interface{}

func (h *H2OFilter) Name() string

func (h *H2OFilter) SetEnabled(enabled bool)

type HierarchicalFilter ¶

type HierarchicalFilter struct {
	// contains filtered or unexported fields
}

HierarchicalFilter implements multi-level summarization for large outputs. Based on "Hierarchical Context Compression" research - creates a tree-like structure where each level provides progressively more detail.

For outputs exceeding a threshold (default 10K lines), this filter: 1. Segments the output into logical sections 2. Generates summaries at multiple abstraction levels 3. Preserves the most important sections verbatim 4. Compresses mid-importance sections into summaries 5. Drops low-importance sections entirely

func NewHierarchicalFilter() *HierarchicalFilter

func (f *HierarchicalFilter) Apply(input string, mode Mode) (string, int)

func (f *HierarchicalFilter) Name() string

func (f *HierarchicalFilter) SetLineThreshold(threshold int)

func (f *HierarchicalFilter) SetMaxDepth(depth int)

type HierarchicalSummaryFilter ¶

type HierarchicalSummaryFilter struct {
	// contains filtered or unexported fields
}

HierarchicalSummaryFilter implements AutoCompressor-style compression (Princeton/MIT, 2023). Recursive summarization that compresses context into summary vectors.

Algorithm: 1. Divide content into hierarchical levels (sections → paragraphs → sentences) 2. Summarize each level recursively 3. Combine summaries with preserved key content 4. Use bottom-up summarization for maximum compression

Research Results: Extreme compression (depends on summary size). Key insight: Recursive summarization preserves global context.

func NewHierarchicalSummaryFilter() *HierarchicalSummaryFilter

func (f *HierarchicalSummaryFilter) Apply(input string, mode Mode) (string, int)

func (f *HierarchicalSummaryFilter) Name() string

func (f *HierarchicalSummaryFilter) SetMaxLevels(levels int)

func (f *HierarchicalSummaryFilter) SetSummaryRatio(ratio float64)

type ImportFilter ¶

type ImportFilter struct {
	// contains filtered or unexported fields
}

ImportFilter condenses import statements.

func NewImportFilter() *ImportFilter

func (f *ImportFilter) Apply(input string, mode Mode) (string, int)

func (f *ImportFilter) Name() string

type LLMAwareConfig ¶

type LLMAwareConfig struct {
	Threshold      int
	Enabled        bool
	CacheEnabled   bool
	PromptTemplate string // Template name for intent-specific summarization
}

LLMAwareConfig holds configuration for the LLM-aware filter

type LLMAwareFilter ¶

type LLMAwareFilter struct {
	// contains filtered or unexported fields
}

LLMAwareFilter uses local LLM for high-quality summarization. This filter is optional and only activates when: 1. An LLM provider is available (Ollama, LM Studio, etc.) 2. The content exceeds the threshold for LLM-based processing 3. The user has enabled LLM mode via flag or config

Research basis: "LLM-based Context Compression" shows 40-60% better semantic preservation compared to heuristic-only approaches.

func NewLLMAwareFilter(cfg LLMAwareConfig) *LLMAwareFilter

func (f *LLMAwareFilter) Apply(input string, mode Mode) (string, int)

func (f *LLMAwareFilter) GetModel() string

func (f *LLMAwareFilter) GetProvider() string

func (f *LLMAwareFilter) IsAvailable() bool

func (f *LLMAwareFilter) Name() string

func (f *LLMAwareFilter) SetEnabled(enabled bool)

func (f *LLMAwareFilter) SummarizeWithIntent(content string, intent string) (string, int)

type Language ¶

type Language string

Language represents a programming language for filtering

const (
	LangRust       Language = "rust"
	LangPython     Language = "python"
	LangJavaScript Language = "javascript"
	LangTypeScript Language = "typescript"
	LangGo         Language = "go"
	LangC          Language = "c"
	LangCpp        Language = "cpp"
	LangJava       Language = "java"
	LangRuby       Language = "ruby"
	LangShell      Language = "sh"
	LangSQL        Language = "sql"
	LangUnknown    Language = "unknown"
)

func DetectLanguageFromInput(input string) Language

type LayerStat ¶

type LayerStat struct {
	TokensSaved int
}

LayerStat holds statistics for a single layer

type LogAggregator ¶ added in v1.2.0

type LogAggregator struct {
	// contains filtered or unexported fields
}

LogAggregator deduplicates and compresses log output.

func NewLogAggregator() *LogAggregator

func (f *LogAggregator) Apply(input string, mode Mode) (string, int)

func (f *LogAggregator) Name() string

type ManagerConfig ¶

type ManagerConfig struct {
	// Context limits
	MaxContextTokens int
	ChunkSize        int
	StreamThreshold  int

	// Resilience
	TeeOnFailure       bool
	FailSafeMode       bool
	ValidateOutput     bool
	ShortCircuitBudget bool

	// Performance
	CacheEnabled bool
	CacheMaxSize int

	// Layer config
	PipelineCfg PipelineConfig
}

ManagerConfig configures the pipeline manager

type Milestone ¶

type Milestone struct {
	Description string `json:"description"`
	Priority    int    `json:"priority"`
	Status      string `json:"status"`
}

Milestone represents a pending task

type Mode ¶

type Mode string

Mode represents the filtering mode.

const (
	ModeNone       Mode = "none"
	ModeMinimal    Mode = "minimal"
	ModeAggressive Mode = "aggressive"
)

type MultiFileConfig ¶

type MultiFileConfig struct {
	MaxCombinedSize     int
	PreserveBoundaries  bool
	SimilarityThreshold float64
}

MultiFileConfig holds configuration for multi-file optimization

type MultiFileFilter ¶

type MultiFileFilter struct {
	// contains filtered or unexported fields
}

MultiFileFilter optimizes output across multiple related files/outputs. It identifies relationships between files, deduplicates common content, and creates unified summaries for better LLM context.

Use case: When an agent works with multiple related files simultaneously (e.g., a module with multiple source files), this filter creates a cohesive view that preserves relationships while removing redundancy.

func NewMultiFileFilter(cfg MultiFileConfig) *MultiFileFilter

func (f *MultiFileFilter) Apply(input string, mode Mode) (string, int)

func (f *MultiFileFilter) Name() string

func (f *MultiFileFilter) SetMaxCombinedSize(size int)

func (f *MultiFileFilter) SetPreserveBoundaries(preserve bool)

func (f *MultiFileFilter) SetSimilarityThreshold(threshold float64)

type NgramAbbreviator ¶

type NgramAbbreviator struct {
	// contains filtered or unexported fields
}

NgramAbbreviator compresses output by abbreviating common patterns. Research-based: CompactPrompt N-gram Abbreviation (2025) - achieves 10-20% lossless compression by replacing common tokens with shorter equivalents.

Key insight: Programming and CLI output contains many repeated long tokens that can be abbreviated while remaining understandable to LLMs.

func NewNgramAbbreviator() *NgramAbbreviator

func (f *NgramAbbreviator) Apply(input string, mode Mode) (string, int)

func (f *NgramAbbreviator) GetAbbreviationLegend() string

func (f *NgramAbbreviator) Name() string

type PerplexityFilter ¶

type PerplexityFilter struct {
	// contains filtered or unexported fields
}

PerplexityFilter implements LLMLingua-style compression (Microsoft/Tsinghua, 2023). Uses perplexity-based iterative pruning with a budget controller.

Algorithm: 1. Calculate perplexity of each token given context 2. Rank tokens by perplexity (higher = more surprising = more important) 3. Iteratively remove lowest-perplexity tokens while staying within budget

Research Results: Up to 20x compression with semantic preservation.

func NewPerplexityFilter() *PerplexityFilter

func (f *PerplexityFilter) Apply(input string, mode Mode) (string, int)

func (f *PerplexityFilter) Name() string

func (f *PerplexityFilter) SetIterations(iterations int)

func (f *PerplexityFilter) SetTargetRatio(ratio float64)

type PipelineConfig ¶

type PipelineConfig struct {
	// Mode: none, minimal, aggressive
	Mode Mode

	// Query intent for query-aware compression
	QueryIntent string

	// Token budget (0 = unlimited)
	Budget int

	// Enable LLM-based compression
	LLMEnabled bool

	// Enable session tracking
	SessionTracking bool

	// Enable N-gram abbreviation
	NgramEnabled bool

	// Enable multi-file optimization
	MultiFileEnabled bool

	// Prompt template for LLM
	PromptTemplate string

	// Enable specific layers (all enabled by default)
	EnableEntropy      bool
	EnablePerplexity   bool
	EnableGoalDriven   bool
	EnableAST          bool
	EnableContrastive  bool
	EnableEvaluator    bool
	EnableGist         bool
	EnableHierarchical bool

	// Layer 11: Compaction (Semantic compression)
	EnableCompaction        bool
	CompactionThreshold     int
	CompactionPreserveTurns int
	CompactionMaxTokens     int
	CompactionStateSnapshot bool
	CompactionAutoDetect    bool

	// Layer 12: Attribution Filter (ProCut-style pruning)
	EnableAttribution    bool
	AttributionThreshold float64

	// Layer 13: H2O Filter (Heavy-Hitter Oracle)
	EnableH2O          bool
	H2OSinkSize        int
	H2ORecentSize      int
	H2OHeavyHitterSize int

	// Layer 14: Attention Sink Filter (StreamingLLM-style)
	EnableAttentionSink  bool
	AttentionSinkCount   int
	AttentionRecentCount int
}

PipelineConfig holds configuration for the compression pipeline

type PipelineCoordinator ¶

type PipelineCoordinator struct {
	// contains filtered or unexported fields
}

PipelineCoordinator orchestrates the 14-layer compression pipeline. Research-based: Combines the best techniques from 50+ research papers worldwide to achieve maximum token reduction for CLI/Agent output.

Layer order is critical - each layer builds on the previous:

Layer 1: Entropy Filtering (Selective Context, Mila 2023) - 2-3x Layer 2: Perplexity Pruning (LLMLingua, Microsoft/Tsinghua 2023) - 20x Layer 3: Goal-Driven Selection (SWE-Pruner, Shanghai Jiao Tong 2025) - 14.8x Layer 4: AST Preservation (LongCodeZip, NUS 2025) - 4-8x Layer 5: Contrastive Ranking (LongLLMLingua, Microsoft 2024) - 4-10x Layer 6: N-gram Abbreviation (CompactPrompt 2025) - 2.5x Layer 7: Evaluator Heads (EHPC, Tsinghua/Huawei 2025) - 5-7x Layer 8: Gist Compression (Stanford/Berkeley 2023) - 20x+ Layer 9: Hierarchical Summary (AutoCompressor, Princeton/MIT 2023) - Extreme Layer 10: Budget Enforcement (Industry standard) - Guaranteed Layer 11: Compaction Layer (Semantic compression) - Auto Layer 12: Attribution Filter (ProCut, LinkedIn 2025) - 78% Layer 13: H2O Filter (Heavy-Hitter Oracle, NeurIPS 2023) - 30x+ Layer 14: Attention Sink Filter (StreamingLLM, 2023) - Infinite context stability

func NewPipelineCoordinator(cfg PipelineConfig) *PipelineCoordinator

func (p *PipelineCoordinator) Process(input string) (string, *PipelineStats)

type PipelineManager ¶

type PipelineManager struct {
	// contains filtered or unexported fields
}

PipelineManager handles resilient large-context processing. Supports streaming for inputs up to 2M tokens with automatic chunking, validation, and failure recovery.

func NewPipelineManager(cfg ManagerConfig) *PipelineManager

func (m *PipelineManager) GetStats() map[string]interface{}

func (m *PipelineManager) Process(input string, mode Mode, ctx CommandContext) (*ProcessResult, error)

func (m *PipelineManager) ProcessFile(path string, mode Mode, ctx CommandContext) (*ProcessResult, error)

func (m *PipelineManager) ProcessWithBudget(input string, mode Mode, budget int, ctx CommandContext) (*ProcessResult, error)

func (m *PipelineManager) ProcessWithQuery(input string, mode Mode, query string, ctx CommandContext) (*ProcessResult, error)

type PipelineStats ¶

type PipelineStats struct {
	OriginalTokens   int
	FinalTokens      int
	TotalSaved       int
	ReductionPercent float64
	LayerStats       map[string]LayerStat
}

PipelineStats holds statistics from the compression pipeline

func (s *PipelineStats) String() string

type PluginConfig ¶ added in v1.2.0

type PluginConfig struct {
	Name         string            `json:"name"`
	Description  string            `json:"description"`
	Enabled      bool              `json:"enabled"`
	Patterns     []string          `json:"patterns"`
	Replacements map[string]string `json:"replacements"`
	Mode         string            `json:"mode"` // "hide" or "replace"
}

PluginConfig defines the structure for filter plugins in config.

func ExamplePluginConfigs() []PluginConfig

type PluginFilter ¶ added in v1.2.0

type PluginFilter struct {
	// contains filtered or unexported fields
}

PluginFilter represents a user-defined filter loaded from config.

func NewPluginFilter(cfg PluginConfig) (*PluginFilter, error)

func (f *PluginFilter) Apply(input string, mode Mode) (string, int)

func (f *PluginFilter) Description() string

func (f *PluginFilter) Name() string

type PluginManager ¶ added in v1.2.0

type PluginManager struct {
	// contains filtered or unexported fields
}

PluginManager handles loading and managing filter plugins.

func NewPluginManager(pluginsDir string) *PluginManager

func (pm *PluginManager) AddFilter(filter *PluginFilter)

func (pm *PluginManager) GetFilters() []*PluginFilter

func (pm *PluginManager) LoadPlugins() error

type PositionAwareFilter ¶

type PositionAwareFilter struct{}

PositionAwareFilter reorders output segments to optimize LLM recall. Based on "LongLLMLingua" (Jiang et al., 2024) - LLMs exhibit "lost in the middle" phenomenon where information in the middle of context is less likely to be recalled.

Strategy: Place high-importance segments at beginning AND end of output.

func NewPositionAwareFilter() *PositionAwareFilter

func (f *PositionAwareFilter) Apply(input string, mode Mode) (string, int)

func (f *PositionAwareFilter) Name() string

type ProcessResult ¶

type ProcessResult struct {
	Output           string
	OriginalTokens   int
	FinalTokens      int
	SavedTokens      int
	ReductionPercent float64
	LayerStats       map[string]LayerStat
	CacheHit         bool
	Chunks           int
	Validated        bool
	TeeFile          string // If failure occurred
	Warning          string
}

ProcessResult contains the result of processing

type QueryAwareFilter ¶

type QueryAwareFilter struct {
	// contains filtered or unexported fields
}

QueryAwareFilter prioritizes output segments based on the agent's query intent. Based on "LongLLMLingua" (Jiang et al., 2024) and "ACON" (Zhang et al., 2024).

Key insight: Different agent tasks need different output segments. A "debug" query needs errors/stack traces, not success messages. A "deploy" query needs status/version, not full logs.

func NewQueryAwareFilter(query ...string) *QueryAwareFilter

func (f *QueryAwareFilter) Apply(input string, mode Mode) (string, int)

func (f *QueryAwareFilter) Name() string

func (f *QueryAwareFilter) SetQuery(query string)

type QueryIntent ¶

type QueryIntent int

QueryIntent represents the type of agent query

const (
	IntentUnknown QueryIntent = iota
	IntentDebug               // Finding errors, failures, crashes
	IntentReview              // Code review, diff analysis
	IntentDeploy              // Deployment status, version info
	IntentSearch              // Finding files, functions, definitions
	IntentTest                // Running/analyzing tests
	IntentBuild               // Build/compilation status
)

type SemanticFilter ¶

type SemanticFilter struct {
	// contains filtered or unexported fields
}

SemanticFilter prunes low-information segments using statistical analysis. Based on "Selective Context" (Li et al., 2024) - uses self-information and information density to identify low-value content.

func NewSemanticFilter() *SemanticFilter

func (f *SemanticFilter) Apply(input string, mode Mode) (string, int)

func (f *SemanticFilter) Name() string

type SessionConfig ¶

type SessionConfig struct {
	SessionFile string // Path to session file
	MaxEntries  int    // Maximum entries to track (0 = unlimited)
}

SessionConfig holds configuration for the session tracker

type SessionHistory ¶

type SessionHistory struct {
	UserQueries []string `json:"user_queries"`
	ActivityLog []string `json:"activity_log"`
	FilesRead   []string `json:"files_read,omitempty"`
	FilesEdited []string `json:"files_edited,omitempty"`
	CommandsRun []string `json:"commands_run,omitempty"`
	Decisions   []string `json:"decisions,omitempty"`
}

SessionHistory tracks what happened in the session

type SessionStats ¶

type SessionStats struct {
	UniqueEntries    int
	TotalOccurrences int
	MultiOccurrences int
}

SessionStats holds session statistics

type SessionTracker ¶

type SessionTracker struct {
	// contains filtered or unexported fields
}

SessionTracker tracks content across commands to avoid repetition. Research-based: Context-Aware Compression (2024) - avoids repeating information already shown to the agent, achieving 5-10% additional reduction.

Key insight: Agents often run similar commands repeatedly. Tracking what has been shown allows collapsing repeated content to "[seen before]" markers.

func NewSessionTracker() *SessionTracker

func NewSessionTrackerWithConfig(cfg SessionConfig) *SessionTracker

func (f *SessionTracker) Apply(input string, mode Mode) (string, int)

func (f *SessionTracker) Clear() error

func (f *SessionTracker) Name() string

func (f *SessionTracker) Save() error

func (f *SessionTracker) Stats() SessionStats

type SinkConfig ¶

type SinkConfig struct {
	// Enable attention sink filtering
	Enabled bool

	// Number of initial tokens to always preserve as sinks
	SinkTokenCount int

	// Number of recent tokens to preserve in rolling cache
	RecentTokenCount int

	// Preserve structural markers (headers, prefixes)
	PreserveStructural bool

	// Minimum content length to apply
	MinContentLength int

	// Anchor patterns to always preserve
	AnchorPatterns []string
}

SinkConfig holds configuration for attention sink preservation

func DefaultSinkConfig() SinkConfig

type SnapshotContext ¶

type SnapshotContext struct {
	Critical    []string          `json:"critical"`  // Must preserve (can't rediscover)
	Working     []string          `json:"working"`   // Summarized knowledge
	KeyValue    map[string]string `json:"key_value"` // Extracted facts
	CodeContext []CodeContext     `json:"code_context,omitempty"`
}

SnapshotContext preserves important knowledge

type StateSnapshot ¶

type StateSnapshot struct {
	SessionHistory SessionHistory  `json:"session_history"`
	CurrentState   CurrentState    `json:"current_state"`
	Context        SnapshotContext `json:"context"`
	PendingPlan    []Milestone     `json:"pending_plan"`
}

StateSnapshot represents semantic compaction output

type Turn ¶

type Turn struct {
	Role      string // "user" or "assistant"
	Content   string
	Timestamp time.Time
	Hash      string
	Tokens    int
}

Turn represents a single conversation turn