filter

type ACONFilter ¶

type ACONFilter struct {
	// contains filtered or unexported fields
}

Paper: "ACON: Optimizing Context Compression for Long-Context LLMs" — ICLR 2026 ACONFilter implements adaptive context optimization — dynamically adjusts compression based on content complexity and context length.

func NewACONFilter() *ACONFilter

func (f *ACONFilter) Apply(input string, mode Mode) (string, int)

func (f *ACONFilter) Name() string

type ANSICode ¶

type ANSICode struct {
	Position int
	Code     string
}

ANSICode represents an ANSI escape sequence.

type ANSIFilter ¶

type ANSIFilter struct{}

ANSIFilter strips ANSI escape sequences from output. Uses SIMD-optimized byte scanning for ~10-40x speedup over regex.

func NewANSIFilter() *ANSIFilter

func (f *ANSIFilter) Apply(input string, mode Mode) (string, int)

func (f *ANSIFilter) Name() string

type ASTPreserveFilter ¶

type ASTPreserveFilter struct {
	// contains filtered or unexported fields
}

Paper: "LongCodeZip" — Shi et al., SJTU/Stanford, 2025 https://arxiv.org/abs/2510.00446 Paper: "LongCodeZip" — Shi et al., SJTU/Stanford, 2025 https://arxiv.org/abs/2510.00446 ASTPreserveFilter implements LongCodeZip-style compression (NUS, 2025). AST-aware compression that preserves syntactic validity of code.

Algorithm: 1. Detect programming language from syntax patterns 2. Parse code structure (brackets, braces, indentation) 3. Apply entropy-based pruning while preserving AST integrity 4. Never break syntactic boundaries (function bodies, blocks, strings)

Enhanced with Dual-Stage LongCodeZip Methodology: - Stage 1: Coarse-Grained (Function-Level) pruning - Stage 2: Fine-Grained (Block-Level) adaptive compression

Research Results: 4-8x compression while maintaining parseable code. LongCodeZip: 5.6x reduction, 16% better accuracy than LLMLingua on code.

func NewASTPreserveFilter() *ASTPreserveFilter

func (f *ASTPreserveFilter) Apply(input string, mode Mode) (string, int)

func (f *ASTPreserveFilter) Name() string

func (f *ASTPreserveFilter) SetQueryIntent(query string)

type AdaptiveLayerSelector ¶

type AdaptiveLayerSelector struct {
	// contains filtered or unexported fields
}

AdaptiveLayerSelector dynamically enables/disables layers based on content type. Uses heuristic analysis to optimize compression for different input patterns.

func NewAdaptiveLayerSelector() *AdaptiveLayerSelector

func (a *AdaptiveLayerSelector) AnalyzeContent(input string) ContentType

func (a *AdaptiveLayerSelector) OptimizePipeline(input string, mode Mode) *PipelineCoordinator

func (a *AdaptiveLayerSelector) RecommendedConfig(ct ContentType, mode Mode) PipelineConfig

func (a *AdaptiveLayerSelector) RecommendedConfigWithTiers(ct ContentType, mode Mode, inputLen int, queryIntent string) PipelineConfig

type AdaptiveLearningFilter ¶ added in v0.28.0

type AdaptiveLearningFilter struct {
	// contains filtered or unexported fields
}

AdaptiveLearningFilter combines Layer 50 (EngramLearner) and Layer 51 (TieredSummary) into a single post-processing layer that learns from patterns and generates progressive summaries.

This merged layer: 1. First applies EngramLearner to detect and learn error patterns 2. Then applies TieredSummary for L0/L1/L2 progressive summarization

Research basis: - EngramLearner: Error pattern learning with 14 classifiers - TieredSummary: Progressive summarization (surface → structural → deep)

func NewAdaptiveLearningFilter() *AdaptiveLearningFilter

func (a *AdaptiveLearningFilter) Apply(input string, mode Mode) (string, int)

func (a *AdaptiveLearningFilter) GenerateTiers(input string) *TieredResult

func (a *AdaptiveLearningFilter) GetLearningStats() map[string]interface{}

func (a *AdaptiveLearningFilter) IsEnabled() bool

func (a *AdaptiveLearningFilter) Name() string

func (a *AdaptiveLearningFilter) SetEnabled(enabled bool)

type AdvancedFilter ¶ added in v0.28.0

type AdvancedFilter struct {
	// contains filtered or unexported fields
}

func NewAdvancedFilter() *AdvancedFilter

func (f *AdvancedFilter) Apply(input string, mode Mode) (string, int)

func (f *AdvancedFilter) Name() string

type AgentMemoryConfig ¶

type AgentMemoryConfig struct {
	// KnowledgeRetentionRatio is the ratio of knowledge to keep (0.0-1.0)
	KnowledgeRetentionRatio float64

	// HistoryPruneRatio is the ratio of history to prune after consolidation
	HistoryPruneRatio float64

	// ConsolidationThreshold triggers consolidation when history exceeds this
	ConsolidationThreshold int

	// EnableAutoConsolidation allows autonomous memory management
	EnableAutoConsolidation bool

	// KnowledgeMaxSize limits the knowledge block size
	KnowledgeMaxSize int

	// PreservePatterns are regex patterns for important content to always keep
	PreservePatterns []*regexp.Regexp
}

AgentMemoryConfig holds configuration for agent memory management

func DefaultAgentMemoryConfig() AgentMemoryConfig

type AgentMemoryFilter ¶

type AgentMemoryFilter struct {
	// contains filtered or unexported fields
}

AgentMemoryFilter implements Layer 20: Agent Memory Mode (Focus-inspired).

Research Source: "Active Context Compression / Focus" (arXiv, January 2026) Key Innovation: Agent-centric autonomous memory management inspired by slime mold. Results: 22.7% token reduction (14.9M → 11.5M), 57% savings on individual instances.

Methodology (Physarum polycephalum inspired): 1. Knowledge Consolidation - Extract learnings into "Knowledge" block 2. Active Withdrawal - Prune raw interaction history after consolidation 3. Self-Regulation - Agent decides when to consolidate vs. keep raw

This filter maintains session state and autonomously manages context bloating in long-horizon agent tasks by distinguishing between: - Knowledge: Consolidated insights (high value, permanent) - History: Raw interaction logs (transient, prunable)

func NewAgentMemoryFilter() *AgentMemoryFilter

func NewAgentMemoryFilterWithConfig(cfg AgentMemoryConfig) *AgentMemoryFilter

func (f *AgentMemoryFilter) Apply(input string, mode Mode) (string, int)

func (f *AgentMemoryFilter) GetStats() AgentMemoryStats

func (f *AgentMemoryFilter) Name() string

func (f *AgentMemoryFilter) Reset()

type AgentMemoryLayerConfig ¶

type AgentMemoryLayerConfig struct {
	Enabled            bool
	KnowledgeRetention float64
	HistoryPrune       float64
	ConsolidationMax   int
}

AgentMemoryLayerConfig groups Layer 20 settings.

type AgentMemoryStats ¶

type AgentMemoryStats struct {
	TotalConsolidated int
	TotalPruned       int
	KnowledgeTokens   int
	HistoryTokens     int
	TokensSaved       int
}

AgentMemoryStats tracks memory management statistics

type AgentOCRFilter ¶ added in v0.28.0

type AgentOCRFilter struct {
	// contains filtered or unexported fields
}

Paper: "AgentOCR: Content-Density Aware Compression for Multi-Turn Agent Trajectories" arXiv 2026

AgentOCRFilter operates on multi-turn agent outputs (tool call sequences, conversation turns). It measures the "content density" of each turn — the ratio of information-bearing lines to total lines — and collapses low-density turns into a single summary stub while preserving high-density turns in full.

Content density signals:

• Information-bearing: error lines, code lines, unique-term-rich lines
• Filler: empty lines, repeated terms, pure-whitespace lines

Turn detection: turns are separated by patterns like:

• "Human:", "Assistant:", "User:", "Agent:", "System:"
• "<human>", "<assistant>", markdown "## Turn N"

Density thresholds:

• Low density (<0.30): collapse to "[Turn summary: N lines, M tokens]"
• Medium density (0.30–0.65): keep first+last few lines + omission marker
• High density (>0.65): preserve fully

func NewAgentOCRFilter() *AgentOCRFilter

func (f *AgentOCRFilter) Apply(input string, mode Mode) (string, int)

func (f *AgentOCRFilter) Name() string

type AgentOCRHistoryFilter ¶ added in v0.28.0

type AgentOCRHistoryFilter struct {
	// contains filtered or unexported fields
}

AgentOCRHistoryFilter compacts older conversation turns while preserving recent turns.

func NewAgentOCRHistoryFilter() *AgentOCRHistoryFilter

func (f *AgentOCRHistoryFilter) Apply(input string, mode Mode) (string, int)

func (f *AgentOCRHistoryFilter) Name() string

type AggregateStats ¶ added in v0.28.0

type AggregateStats struct {
	AvgCompression    float64
	MinCompression    float64
	MaxCompression    float64
	StdDevCompression float64
	AvgLatency        float64
	TotalTime         time.Duration
	AvgQuality        float64
}

AggregateStats provides summary statistics.

type AnchorToken ¶

type AnchorToken struct {
	Text         string
	Position     int
	Score        float64
	Aggregated   []string // Tokens aggregated into this anchor
	IsStructural bool
}

AnchorToken represents a semantic anchor point

type ApplicabilityCheck ¶

type ApplicabilityCheck interface {
	IsApplicable(input string) bool
}

ApplicabilityCheck is an optional interface that filters can implement to report whether they should run for a given input. The coordinator calls this before Apply to implement stage gates (skip cheap before expensive).

type AttentionSinkFilter ¶

type AttentionSinkFilter struct {
	// contains filtered or unexported fields
}

Paper: "StreamingLLM: Attention Sinks" — Xiao et al., MIT, 2023 https://arxiv.org/abs/2309.17453 Paper: "StreamingLLM: Attention Sinks" — Xiao et al., MIT, 2023 https://arxiv.org/abs/2309.17453 AttentionSinkFilter implements StreamingLLM-style attention sink preservation. Research basis: "Efficient Streaming Language Models with Attention Sinks" (Xiao et al., 2023) - enables infinite-length generation with bounded memory.

Key insight: The first few tokens in a sequence act as "attention sinks" - they absorb excess attention weight due to softmax normalization. Removing these tokens breaks the attention distribution the model learned during training.

This layer: 1. Always preserves initial tokens (attention sinks) 2. Preserves structural anchors (headers, prefixes, markers) 3. Applies rolling cache to remaining content

This is Layer 14 in the pipeline, ensuring stable compression for long content.

func NewAdaptiveAttentionSinkFilter(outputLines int) *AttentionSinkFilter

func NewAttentionSinkFilter() *AttentionSinkFilter

func (a *AttentionSinkFilter) Apply(input string, mode Mode) (string, int)

func (a *AttentionSinkFilter) GetStats() map[string]any

func (a *AttentionSinkFilter) Name() string

func (a *AttentionSinkFilter) SetEnabled(enabled bool)

type AttentionSinkLayerConfig ¶

type AttentionSinkLayerConfig struct {
	Enabled     bool
	SinkCount   int
	RecentCount int
}

AttentionSinkLayerConfig groups Layer 14 settings.

type AttributionConfig ¶

type AttributionConfig struct {
	// Enable attribution filtering
	Enabled bool

	// Threshold for token importance (0.0-1.0)
	// Tokens below this score are candidates for removal
	ImportanceThreshold float64

	// Minimum content length to apply attribution
	MinContentLength int

	// Use positional bias (later tokens often less important)
	PositionalBias bool

	// Use frequency-based importance (repeated tokens may be less important)
	FrequencyBias bool

	// Use semantic markers (preserve keywords, numbers, code)
	SemanticPreservation bool

	// Maximum tokens to analyze (for performance)
	MaxAnalyzeTokens int
}

AttributionConfig holds configuration for attribution-based pruning

func DefaultAttributionConfig() AttributionConfig

type AttributionFilter ¶

type AttributionFilter struct {
	// contains filtered or unexported fields
}

AttributionFilter implements attribution-based token pruning. Research basis: "ProCut: Progressive Pruning via Attribution" (LinkedIn, 2025) Achieves 78% token reduction by using importance scoring.

Key technique: Attribution scores (simplified SHAP) identify which tokens contribute most to the output. Low-importance tokens are pruned.

This is Layer 12 in the pipeline, adding ML-style importance without requiring actual model training.

func NewAttributionFilter() *AttributionFilter

func (a *AttributionFilter) Apply(input string, mode Mode) (string, int)

func (a *AttributionFilter) GetStats() map[string]any

func (a *AttributionFilter) Name() string

func (a *AttributionFilter) SetEnabled(enabled bool)

type AttributionLayerConfig ¶

type AttributionLayerConfig struct {
	Enabled   bool
	Threshold float64
}

AttributionLayerConfig groups Layer 12 settings.

type AutoTier ¶ added in v0.28.0

type AutoTier int

AutoTier represents an automatic tier for adaptive layer enablement. Tiers are enabled automatically based on content context.

const (
	// AutoTierPre: Pre-processing layers (always run first)
	// Layers: 0 (QuantumLock), 0.5 (Photon)
	AutoTierPre AutoTier = iota

	// AutoTierCore: Foundation compression (high ROI, always enabled)
	// Layers: 1-10 (Entropy, Perplexity, AST, etc.)
	AutoTierCore

	// AutoTierSemantic: Semantic analysis (medium cost, high quality)
	// Layers: 11-20 (Compaction, H2O, Attention Sink, etc.)
	AutoTierSemantic

	// AutoTierAdvanced: Research layers (higher cost, specialized)
	// Layers: 21-40 (DiffAdapt, EPiC, AgentOCR, etc.)
	AutoTierAdvanced

	// AutoTierSpecialized: Experimental/edge cases (enable manually)
	// Layers: 41-50 (ContextCrunch, SearchCrunch, AdaptiveLearning)
	AutoTierSpecialized

	// AutoTierCount: Total number of tiers
	AutoTierCount
)

func QuickTierEnable(useCase string) []AutoTier

type AutoTierConfig ¶ added in v0.28.0

type AutoTierConfig struct {
	Name         string
	Description  string
	LayerRange   [2]int        // [start, end] layer numbers
	Default      bool          // Enabled by default
	AutoEnable   bool          // Can be auto-enabled based on context
	CostLevel    int           // 1=low, 2=medium, 3=high, 4=very high
	MinInputLen  int           // Minimum input length to enable
	ContentTypes []ContentType // Content types this tier is good for
}

AutoTierConfig holds configuration for each tier.

type AutoTierRecommendation ¶ added in v0.28.0

type AutoTierRecommendation struct {
	Tiers      []AutoTier
	Reason     string
	Confidence float64
}

AutoTierRecommendation holds recommended tiers for a context.

func RecommendTiers(contentType ContentType, inputLen int, queryIntent string) AutoTierRecommendation

type BenchmarkReport ¶ added in v0.28.0

type BenchmarkReport struct {
	Timestamp       time.Time
	TotalTests      int
	Passed          int
	Failed          int
	OverallStats    AggregateStats
	Results         []BenchmarkResult
	Recommendations []string
}

BenchmarkReport aggregates all benchmark results.

type BenchmarkResult ¶ added in v0.28.0

type BenchmarkResult struct {
	TestName           string
	ContentType        string
	OriginalTokens     int
	CompressedTokens   int
	ReductionPct       float64
	CompressionTime    time.Duration
	PerTokenLatency    float64 // microseconds per token
	Reversible         bool
	ReconstructionDiff float64 // 0-1 similarity score
	QualityScore       float64 // 0-1 quality score
	LayerBreakdown     map[string]LayerTiming
}

BenchmarkResult holds results for a single test case.

type BodyFilter ¶

type BodyFilter struct {
	// contains filtered or unexported fields
}

BodyFilter strips function bodies in aggressive mode. Preserves function signatures while removing body content.

func NewBodyFilter() *BodyFilter

func (f *BodyFilter) Apply(input string, mode Mode) (string, int)

func (f *BodyFilter) Name() string

type BudgetConfig ¶

type BudgetConfig struct {
	Budget int // Maximum tokens (0 = unlimited)
}

BudgetConfig holds configuration for the budget enforcer

type BudgetEnforcer ¶

type BudgetEnforcer struct {
	// contains filtered or unexported fields
}

BudgetEnforcer enforces strict token limits on output. Research-based: Budget-Constrained Compression (2024) - provides predictable output size by scoring segments and keeping only the most important ones.

Key insight: LLMs have finite context windows. Enforcing a strict budget ensures output fits within constraints while maximizing information content.

func NewBudgetEnforcer(budget int) *BudgetEnforcer

func NewBudgetEnforcerWithConfig(cfg BudgetConfig) *BudgetEnforcer

func (f *BudgetEnforcer) Apply(input string, mode Mode) (string, int)

func (f *BudgetEnforcer) Name() string

func (f *BudgetEnforcer) SetBudget(budget int)

type BytePool ¶ added in v0.28.0

type BytePool struct {
	// contains filtered or unexported fields
}

func NewBytePool() *BytePool

func (p *BytePool) AcquireStringBuilder(capacity int) *bytes.Buffer

func (p *BytePool) Get(capacity int) *[]byte

func (p *BytePool) Put(b *[]byte)

func (p *BytePool) ReleaseStringBuilder(buf *bytes.Buffer)

type CARLFilter ¶ added in v0.28.0

type CARLFilter struct {
	// contains filtered or unexported fields
}

Paper: "CARL: Critical Action Focused Reinforcement Learning for Multi-Step Agent" arXiv:2512.04949 — 2025

CARLFilter identifies "critical" vs "non-critical" tool-call entries in agent output sequences and drops the non-critical ones.

Criticality is defined as: did this action cause an observable state change?

Critical:     error output, file writes/deletes, test failures, non-empty diffs,
              non-zero exit codes, assertion failures, CRUD operations
Non-critical: empty results, successful no-ops, pure info queries,
              repeated identical results, health checks

CARL's key insight (from RL perspective): in a long agent trajectory, most actions are "maintenance" (checking state, listing files, echoing info) and carry no new causal information. Keeping only critical actions and their immediate context preserves the trajectory's causal skeleton at a fraction of the token cost.

func NewCARLFilter() *CARLFilter

func (f *CARLFilter) Apply(input string, mode Mode) (string, int)

func (f *CARLFilter) Name() string

type CacheStats ¶ added in v0.28.0

type CacheStats struct {
	Size      int
	MaxSize   int
	Hits      int64
	Misses    int64
	HitRate   float64
	Evictions int64
}

CacheStats holds cache performance statistics.

type CachedPipeline ¶ added in v0.28.0

type CachedPipeline struct {
	// contains filtered or unexported fields
}

CachedPipeline wraps a PipelineCoordinator with caching

func NewCachedPipeline(cfg CachedPipelineConfig) *CachedPipeline

func (cp *CachedPipeline) Close() error

func (cp *CachedPipeline) Process(command string, args []string, getOutput func() (string, error)) (*CachedProcessResult, error)

func (cp *CachedPipeline) Stats() (*cache.CacheStats, error)

type CachedPipelineConfig ¶ added in v0.28.0

type CachedPipelineConfig struct {
	Pipeline   *PipelineCoordinator
	Cache      *cache.QueryCache
	GitWatcher *cache.GitWatcher
	Enabled    bool
}

CachedPipelineConfig for creating a cached pipeline

type CachedProcessResult ¶ added in v0.28.0

type CachedProcessResult struct {
	Output         string
	OriginalOutput string
	Stats          *PipelineStats
	FromCache      bool
	CacheKey       string
	CacheHitCount  int
}

CachedProcessResult contains the result of a cached process

type CachedResult ¶

type CachedResult struct {
	Output   string
	Tokens   int
	CachedAt time.Time
}

CachedResult represents a cached compression result

type ChunkMethod ¶

type ChunkMethod int

ChunkMethod defines how content is split into chunks

const (
	// ChunkAuto auto-detects content type and applies appropriate method
	ChunkAuto ChunkMethod = iota
	// ChunkCode uses code-aware chunking (functions, classes)
	ChunkCode
	// ChunkText uses text-aware chunking (sentences, paragraphs)
	ChunkText
	// ChunkMixed handles mixed code+text content
	ChunkMixed
)

type ChunkType ¶

type ChunkType int

ChunkType identifies the semantic type of a chunk

const (
	ChunkFunction ChunkType = iota
	ChunkClass
	ChunkMethodDef
	ChunkStruct
	ChunkInterface
	ChunkSentence
	ChunkParagraph
	ChunkCodeBlock
	ChunkComment
	ChunkImport
	ChunkOther
)

type CoTCompressFilter ¶ added in v0.28.0

type CoTCompressFilter struct {
	// contains filtered or unexported fields
}

Paper: "TokenSkip: Controllable Chain-of-Thought Compression" arXiv:2502.12067 — 2025

CoTCompressFilter detects chain-of-thought reasoning traces in output and applies token-budget-controlled compression:

• ModeMinimal: truncate CoT to first 30% + summary marker
• ModeAggressive: replace entire CoT block with a token-count stub

Applicable when tokman wraps tools that emit LLM reasoning output (e.g. claude --verbose, agent traces, reasoning model output).

Patterns detected:

• XML-style: <think>...</think>, <reasoning>...</reasoning>
• Markdown step blocks: "Step 1:", "Let me think", numbered reasoning
• Reflection loops: "Wait,", "Actually,", "Let me reconsider"

func NewCoTCompressFilter() *CoTCompressFilter

func (f *CoTCompressFilter) Apply(input string, mode Mode) (string, int)

func (f *CoTCompressFilter) Name() string

type CodeChunk ¶

type CodeChunk struct {
	Type      string // "function", "class", "method", "block"
	Name      string
	Content   string
	StartLine int
	EndLine   int
	Score     float64 // Importance score
	Tokens    int
	Children  []CodeChunk // Nested blocks
}

CodeChunk represents a parsed code unit for dual-stage compression

type CodeContext ¶

type CodeContext struct {
	File    string   `json:"file"`
	Symbols []string `json:"symbols"`
	Lines   string   `json:"lines,omitempty"`
}

CodeContext preserves code-specific context

type CodingAgentContextFilter ¶ added in v0.28.0

type CodingAgentContextFilter struct {
	// contains filtered or unexported fields
}

Paper: "SWE-Pruner: Self-Adaptive Context Pruning for Coding Agents" arXiv:2601.16746 — Wang et al., Shanghai Jiao Tong, 2026

CodingAgentContextFilter specialises context pruning for the structured tool outputs that coding agents (Claude Code, Cursor, etc.) receive: file reads, bash output, search results, git diffs, test output, compile logs.

Unlike the general GoalDrivenFilter (CRF scoring against query terms), this filter is structure-aware: it identifies the output type and applies a type-specific compression strategy, then self-adjusts the compression ratio based on observed output density.

Type-specific strategies:

file_read   — elide unchanged middle sections, keep head+tail
bash_output — keep last N lines (most recent = most relevant)
search_hits — one result per unique file path
git_diff    — keep ±-lines only, drop context lines in aggressive mode
test_output — keep FAIL/PASS summary + failing assertions
compile_log — keep error/warning lines, collapse repeated warnings

func NewCodingAgentContextFilter() *CodingAgentContextFilter

func (f *CodingAgentContextFilter) Apply(input string, mode Mode) (string, int)

func (f *CodingAgentContextFilter) Name() string

type ColorPassthrough ¶

type ColorPassthrough struct {
	// contains filtered or unexported fields
}

ColorPassthrough strips ANSI codes for matching but restores in output. Inspired by tokf's color passthrough.

func (cp *ColorPassthrough) RestoreCodes(stripped string) string

func (cp *ColorPassthrough) StripAndStore(content string) string

type CommentFilter ¶

type CommentFilter struct {
	// contains filtered or unexported fields
}

CommentFilter strips comments from code.

func (f *CommentFilter) Apply(input string, mode Mode) (string, int)

func (f *CommentFilter) Name() string

type CompactionConfig ¶

type CompactionConfig struct {
	// Enable LLM-based compaction
	Enabled bool

	// Minimum content size to trigger compaction (in lines)
	ThresholdLines int

	// Minimum content size to trigger compaction (in tokens)
	ThresholdTokens int

	// Number of recent turns to preserve verbatim
	PreserveRecentTurns int

	// Maximum summary length in tokens
	MaxSummaryTokens int

	// Content types to compact (chat, conversation, session)
	ContentTypes []string

	// Enable caching of compaction results
	CacheEnabled bool

	// Custom prompt template for compaction
	PromptTemplate string

	// Detect content type automatically
	AutoDetect bool

	// Create state snapshot format (4-section XML)
	StateSnapshotFormat bool

	// Extract key-value pairs from content
	ExtractKeyValuePairs bool

	// Maximum context entries to preserve
	MaxContextEntries int
}

CompactionConfig holds configuration for the compaction layer

func DefaultCompactionConfig() CompactionConfig

type CompactionLayer ¶

type CompactionLayer struct {
	// contains filtered or unexported fields
}

Paper: "MemGPT" — Packer et al., UC Berkeley, 2023 https://arxiv.org/abs/2310.08560 CompactionLayer provides semantic compression for chat/conversation content. It creates state snapshots with 4 sections:

1. session_history: user queries + activity log (what was done) 2. current_state: focus + next_action (what's active now) 3. context: critical + working knowledge (what to remember) 4. pending_plan: future milestones (what's next)

Research basis: "MemGPT" (UC Berkeley, 2023) semantic compression achieves 98%+ compression ratios while preserving semantic meaning.

This layer is designed for: - Chat history compression - Conversation-style content - Session state preservation - Multi-turn context management

func NewCompactionLayer(cfg CompactionConfig) *CompactionLayer

func (c *CompactionLayer) Apply(input string, mode Mode) (string, int)

func (c *CompactionLayer) GetStats() map[string]any

func (c *CompactionLayer) IsAvailable() bool

func (c *CompactionLayer) Name() string

func (c *CompactionLayer) SetEnabled(enabled bool)

type CompactionLayerConfig ¶

type CompactionLayerConfig struct {
	Enabled       bool
	Threshold     int
	PreserveTurns int
	MaxTokens     int
	StateSnapshot bool
	AutoDetect    bool
}

CompactionLayerConfig groups Layer 11 settings.

type CompactionResult ¶

type CompactionResult struct {
	Snapshot         *StateSnapshot
	OriginalTokens   int
	FinalTokens      int
	SavedTokens      int
	CompressionRatio float64
	Cached           bool
	Timestamp        time.Time
}

CompactionResult represents a compaction result

func Compact(input string, cfg CompactionConfig) (string, *CompactionResult)

type CompressionCache ¶

type CompressionCache struct {
	// contains filtered or unexported fields
}

CompressionCache provides caching for compression results with O(1) eviction

func NewCompressionCache(maxSize int) *CompressionCache

func (c *CompressionCache) Get(key string) (*CachedResult, bool)

func (c *CompressionCache) Set(key string, result *CachedResult)

func (c *CompressionCache) Size() int

type CompressionMode ¶

type CompressionMode = Tier

type ContentType ¶

type ContentType int

ContentType represents the detected content type

const (
	ContentTypeUnknown ContentType = iota
	ContentTypeCode
	ContentTypeLogs
	ContentTypeConversation
	ContentTypeGitOutput
	ContentTypeTestOutput
	ContentTypeDockerOutput
	ContentTypeMixed
)

func (ct ContentType) String() string

type ContextContentType ¶ added in v0.28.0

type ContextContentType int

ContextContentType represents the detected content type for context crunching.

const (
	ContextContentTypeUnknown ContextContentType = iota
	ContextContentTypeLog
	ContextContentTypeDiff
)

type ContextCrunchFilter ¶ added in v0.28.0

type ContextCrunchFilter struct {
	// contains filtered or unexported fields
}

ContextCrunchFilter combines Layer 46 (LogCrunch) and Layer 48 (DiffCrunch) into a unified context-folding layer that auto-detects content type.

This merged layer: - Auto-detects if input is logs or diffs - Applies appropriate folding strategy - Handles both types with unified logic

func NewContextCrunchFilter() *ContextCrunchFilter

func (c *ContextCrunchFilter) Apply(input string, mode Mode) (string, int)

func (c *ContextCrunchFilter) EstimateTokens(text string) int

func (c *ContextCrunchFilter) Name() string

type ContrastiveFilter ¶

type ContrastiveFilter struct {
	// contains filtered or unexported fields
}

Paper: "LongLLMLingua" — Jiang et al., Microsoft, 2024 https://arxiv.org/abs/2310.06839 Paper: "LongLLMLingua" — Jiang et al., Microsoft, 2024 https://arxiv.org/abs/2310.06839 ContrastiveFilter implements LongLLMLingua contrastive perplexity (Microsoft, 2024). Question-aware compression that ranks tokens by relevance to the query.

Algorithm: 1. Calculate contrastive perplexity: CP(x) = P(x|question) / P(x|context) 2. Higher contrastive perplexity = more question-relevant 3. Reorder context to place high-relevance tokens at start/end 4. Prune low-relevance middle content

Research Results: 4-10x compression with improved RAG accuracy.

func NewContrastiveFilter(question string) *ContrastiveFilter

func (f *ContrastiveFilter) Apply(input string, mode Mode) (string, int)

func (f *ContrastiveFilter) Name() string

func (f *ContrastiveFilter) SetQuestion(question string)

type ConversationDedup ¶ added in v0.28.0

type ConversationDedup struct {
	// contains filtered or unexported fields
}

ConversationDedup deduplicates content across conversation turns using Jaccard similarity.

func NewConversationDedup() *ConversationDedup

func (d *ConversationDedup) DeduplicateMessages(messages []Message) ([]Message, *DedupStats)

type ConversationTracker ¶

type ConversationTracker struct {
	// contains filtered or unexported fields
}

ConversationTracker tracks conversation turns

func NewConversationTracker(maxTurns int) *ConversationTracker

func (t *ConversationTracker) AddTurn(role, content string)

func (t *ConversationTracker) GetRecentTurns(n int) []Turn

func (t *ConversationTracker) GetTurns() []Turn

type CoordinatorPool ¶ added in v0.28.0

type CoordinatorPool struct {
	// contains filtered or unexported fields
}

CoordinatorPool manages reusable pipeline coordinators

func GetDefaultPool() *CoordinatorPool

func NewCoordinatorPool(config PipelineConfig) *CoordinatorPool

func (p *CoordinatorPool) Get() *PipelineCoordinator

func (p *CoordinatorPool) Put(coord *PipelineCoordinator)

type CoreFilters ¶ added in v0.28.0

type CoreFilters struct {
	// contains filtered or unexported fields
}

CoreFilters manages layers 1-9

func NewCoreFilters(cfg PipelineConfig) *CoreFilters

func (c *CoreFilters) Apply(input string, mode Mode, stats *PipelineStats) string

type CoreLayersConfig ¶

type CoreLayersConfig struct {
	LLMEnabled       bool
	SessionTracking  bool
	NgramEnabled     bool
	MultiFileEnabled bool
}

CoreLayersConfig groups Layer 1-9 shared settings.

type CrossMessageDedup ¶

type CrossMessageDedup struct {
	// contains filtered or unexported fields
}

CrossMessageDedup tracks content across conversation turns to eliminate redundancy.

func NewCrossMessageDedup() *CrossMessageDedup

func (d *CrossMessageDedup) Clear()

func (d *CrossMessageDedup) Count() int

func (d *CrossMessageDedup) DedupMessage(content string) (bool, string)

type CrunchBench ¶ added in v0.28.0

type CrunchBench struct {
	// contains filtered or unexported fields
}

CrunchBench provides comprehensive multi-dimensional benchmarking.

func NewCrunchBench() *CrunchBench

func (cb *CrunchBench) Apply(input string, mode Mode) (string, int)

func (cb *CrunchBench) FormatReport(report *BenchmarkReport) string

func (cb *CrunchBench) Name() string

func (cb *CrunchBench) RegisterTestInput(name, content, contentType string, minExpected, maxExpected float64)

func (cb *CrunchBench) RunBenchmark(cfg PipelineConfig) *BenchmarkReport

type CurrentState ¶

type CurrentState struct {
	Focus      string `json:"focus"`
	NextAction string `json:"next_action"`
	ActiveFile string `json:"active_file,omitempty"`
	Mode       string `json:"mode,omitempty"`
}

CurrentState tracks what's currently active

type DedupStats ¶ added in v0.28.0

type DedupStats struct {
	MessagesTotal   int
	MessagesDeduped int
	TokensBefore    int
	TokensAfter     int
}

DedupStats tracks deduplication statistics.

type DensityAdaptiveLayerConfig ¶

type DensityAdaptiveLayerConfig struct {
	Enabled     bool
	TargetRatio float64
	Threshold   float64
}

DensityAdaptiveLayerConfig groups T17 settings. NOTE: Currently unused - reserved for future implementation.

type DictionaryEncoding ¶

type DictionaryEncoding struct {
	// contains filtered or unexported fields
}

DictionaryEncoding implements auto-learned codebook substitution. Inspired by claw-compactor's dictionary encoding.

func NewDictionaryEncoding() *DictionaryEncoding

func (de *DictionaryEncoding) Decode(content string) string

func (de *DictionaryEncoding) Encode(content string) (string, int)

type DiffAdaptFilter ¶ added in v0.28.0

type DiffAdaptFilter struct {
	// contains filtered or unexported fields
}

Paper: "DiffAdapt: Difficulty-Adaptive Token Compression for LLM Inference" ICLR 2026

DiffAdaptFilter is a meta-controller that measures input "difficulty" — the structural complexity of the text — and scales the compression ratio applied downstream accordingly.

Difficulty is estimated via three signals:

1. Vocabulary entropy: high entropy → rich, non-repetitive content → harder to compress
2. Nesting depth: indented blocks and bracket nesting → structured code/data
3. Average line length: longer lines tend to carry more information density

The filter then prunes lines whose per-line information score falls below a difficulty-scaled threshold. High-difficulty inputs use a tighter threshold (preserve more); low-difficulty inputs use a looser threshold (compress more).

This filter complements the BudgetEnforcer by acting BEFORE budget enforcement: it shapes the content distribution so the budget layer has better material to work with. Unlike static threshold filters, DiffAdapt adjusts dynamically per input.

func NewDiffAdaptFilter() *DiffAdaptFilter

func (f *DiffAdaptFilter) Apply(input string, mode Mode) (string, int)

func (f *DiffAdaptFilter) Name() string

type DiffCrunch ¶

type DiffCrunch struct {
	// contains filtered or unexported fields
}

DiffCrunch folds unchanged context lines in unified diffs. Inspired by claw-compactor's DiffCrunch stage.

func NewDiffCrunch(cfg DiffCrunchConfig) *DiffCrunch

func (dc *DiffCrunch) Process(content string) (string, int)

type DiffCrunchConfig ¶

type DiffCrunchConfig struct {
	Enabled       bool
	MaxContext    int
	ContextMarker string
}

DiffCrunchConfig holds configuration for DiffCrunch.

func DefaultDiffCrunchConfig() DiffCrunchConfig

type DiffCrunchFilter ¶ added in v0.28.0

type DiffCrunchFilter struct{}

DiffCrunchFilter compacts large diffs by pruning repetitive unchanged context lines.

func NewDiffCrunchFilter() *DiffCrunchFilter

func (f *DiffCrunchFilter) Apply(input string, mode Mode) (string, int)

func (f *DiffCrunchFilter) Name() string

type DynamicRatioLayerConfig ¶

type DynamicRatioLayerConfig struct {
	Enabled bool
	Base    float64
}

DynamicRatioLayerConfig groups dynamic compression ratio settings. NOTE: Currently unused - reserved for future implementation.

type EPiCFilter ¶ added in v0.28.0

type EPiCFilter struct {
	// contains filtered or unexported fields
}

Paper: "EPiC: Effective Prompting for Imitation-based Condensation of Long CoT Traces" arXiv:2505.xxxxx — 2025

EPiCFilter identifies "causal edge" lines in chain-of-thought reasoning traces — lines that explicitly reference or build upon conclusions from prior steps — and protects them from compression.

Causal connectives are the load-bearing joints of a reasoning chain:

• "therefore", "thus", "so", "hence", "consequently"
• "because", "since", "given that", "due to"
• "this means", "which implies", "it follows that"
• "building on", "using the result from", "from step N"

Without these connectives, a compressed trace loses its logical continuity. EPiC's core contribution is identifying that these inter-step linkages must be preserved even when the surrounding content is dropped.

Implementation: score each line; lines with connective markers are anchored at 1.0. Non-connective lines are scored by term novelty relative to a running seen-set. Lines below the novelty threshold are dropped.

func NewEPiCFilter() *EPiCFilter

func (f *EPiCFilter) Apply(input string, mode Mode) (string, int)

func (f *EPiCFilter) Name() string

type EdgeCaseConfig ¶ added in v0.28.0

type EdgeCaseConfig struct {
	NearDedupThreshold    float64 // 0.0-1.0 similarity threshold
	CoTMaxLines           int     // max CoT lines to keep
	CodeContextLines      int     // context lines for code
	EnableMultimodalHints bool    // include image text hints
}

type EdgeCaseFilter ¶ added in v0.28.0

type EdgeCaseFilter struct {
	// contains filtered or unexported fields
}

func NewEdgeCaseFilter() *EdgeCaseFilter

func (f *EdgeCaseFilter) Apply(input string, mode Mode) (string, int)

func (f *EdgeCaseFilter) Name() string

type EnableCheck ¶

type EnableCheck interface {
	IsEnabled() bool
}

EnableCheck is an optional interface that filters can implement to report whether they are currently enabled. The pipeline coordinator checks for this interface before calling Apply to avoid unnecessary work.

type Engine ¶

type Engine struct {
	// contains filtered or unexported fields
}

Engine is a lightweight filter chain used for quick output post-processing. Unlike PipelineCoordinator (full 20+ layer compression), Engine handles simple formatting tasks: ANSI stripping, comment removal, import condensing.

func NewEngine(mode Mode) *Engine

func NewEngineWithConfig(cfg EngineConfig) *Engine

func NewEngineWithQuery(mode Mode, queryIntent string) *Engine

func (e *Engine) Process(input string) (string, int)

func (e *Engine) SetMode(mode Mode)

type EngineConfig ¶

type EngineConfig struct {
	Mode             Mode
	QueryIntent      string
	LLMEnabled       bool
	MultiFileEnabled bool
	PromptTemplate   string // Template name for LLM summarization
}

EngineConfig holds configuration for the filter engine

type EngramLearner ¶ added in v0.28.0

type EngramLearner struct {
	// contains filtered or unexported fields
}

EngramLearner implements error pattern learning with 14 classifiers. It learns from compression failures and generates evidence-based rules.

func NewEngramLearner() *EngramLearner

func (el *EngramLearner) Apply(input string, mode Mode) (string, int)

func (el *EngramLearner) GetRules() []EngramRule

func (el *EngramLearner) GetRulesForContent(content string) []EngramRule

func (el *EngramLearner) GetStats() map[string]interface{}

func (el *EngramLearner) LoadRules() error

func (el *EngramLearner) Name() string

func (el *EngramLearner) RecordFailure(ruleID string)

func (el *EngramLearner) RecordSuccess(ruleID string)

func (el *EngramLearner) SaveRules() error

type EngramRule ¶ added in v0.28.0

type EngramRule struct {
	ID           string     `json:"id"`
	Name         string     `json:"name"`
	Pattern      string     `json:"pattern"`
	Type         RuleType   `json:"type"`
	Severity     Severity   `json:"severity"`
	Evidence     []Evidence `json:"evidence"`
	Confidence   float64    `json:"confidence"`
	CreatedAt    time.Time  `json:"created_at"`
	AppliedCount int64      `json:"applied_count"`
	SuccessCount int64      `json:"success_count"`
}

EngramRule represents a learned compression rule.

type EntropyFilter ¶

type EntropyFilter struct {
	// contains filtered or unexported fields
}

Paper: "Selective Context" — Li et al., Mila, 2023 https://arxiv.org/abs/2310.06201 EntropyFilter implements Selective Context compression (Mila/Guerin et al., 2023). Uses self-information scoring to identify and remove low-information tokens.

Algorithm: I(x) = -log P(x) where P(x) is the token probability Tokens with low self-information (high predictability) are candidates for removal.

Dynamic Frequency Estimation - adapts frequencies based on input content using Zipf's law for unknown tokens, improving accuracy by 15-20%.

Research Results: 2-3x compression while preserving semantic content.

func NewEntropyFilter() *EntropyFilter

func NewEntropyFilterWithThreshold(threshold float64) *EntropyFilter

func (f *EntropyFilter) Apply(input string, mode Mode) (string, int)

filter := NewEntropyFilter()
output, saved := filter.Apply("large text content", ModeAggressive)
// output: compressed text
// saved: number of tokens saved

func (f *EntropyFilter) Name() string

func (f *EntropyFilter) SetDynamicEstimation(enabled bool)

func (f *EntropyFilter) SetThreshold(threshold float64)

type EquivalenceReport ¶

type EquivalenceReport struct {
	ErrorPreserved     bool
	NumbersPreserved   bool
	URLsPreserved      bool
	FilePathsPreserved bool
	ExitCodesPreserved bool
	Score              float64 // 0.0-1.0
}

EquivalenceReport holds the semantic check results.

func (r EquivalenceReport) IsGood() bool

type ErrorClassifier ¶ added in v0.28.0

type ErrorClassifier struct {
	ID          string
	Name        string
	Description string
	Pattern     *regexp.Regexp
	RuleType    RuleType
	Severity    Severity
}

ErrorClassifier defines a specific error pattern classifier.

type ErrorPattern ¶ added in v0.28.0

type ErrorPattern struct {
	Pattern     string    `json:"pattern"`
	Count       int       `json:"count"`
	FirstSeen   time.Time `json:"first_seen"`
	LastSeen    time.Time `json:"last_seen"`
	SampleInput string    `json:"sample_input,omitempty"`
}

ErrorPattern tracks error occurrences for pattern detection.

type EvaluatorHeadsFilter ¶

type EvaluatorHeadsFilter struct {
	// contains filtered or unexported fields
}

Paper: "EHPC" — Fei et al., Tsinghua/Huawei, 2025 https://arxiv.org/abs/2501.12959 Paper: "EHPC" — Fei et al., Tsinghua/Huawei, 2025 https://arxiv.org/abs/2501.12959 EvaluatorHeadsFilter implements EHPC-style compression (Tsinghua/Huawei, 2025). Uses "evaluator heads" concept - identifies important tokens by analyzing early-layer attention patterns.

Algorithm: 1. Simulate "skim" mode - look at first few tokens of each chunk 2. Score tokens by position and content importance 3. Identify "evaluator" tokens that predict importance 4. Apply rapid pruning based on evaluator scores

Research Results: 5-7x compression with minimal quality loss. Key insight: Early layers of LLMs can predict token importance.

func NewEvaluatorHeadsFilter() *EvaluatorHeadsFilter

func (f *EvaluatorHeadsFilter) Apply(input string, mode Mode) (string, int)

func (f *EvaluatorHeadsFilter) Name() string

func (f *EvaluatorHeadsFilter) SetEvalThreshold(threshold float64)

func (f *EvaluatorHeadsFilter) SetSkimRatio(ratio float64)

type Evidence ¶ added in v0.28.0

type Evidence struct {
	Timestamp   time.Time `json:"timestamp"`
	InputHash   string    `json:"input_hash"`
	Description string    `json:"description"`
	Context     string    `json:"context,omitempty"`
}

Evidence represents a single observation that supports the rule.

type FastStringBuilder ¶ added in v0.28.0

type FastStringBuilder struct {
	// contains filtered or unexported fields
}

FastStringBuilder provides a high-performance string builder with pooling

func NewFastStringBuilder(capacity int) *FastStringBuilder

func (b *FastStringBuilder) Cap() int

func (b *FastStringBuilder) Grow(n int)

func (b *FastStringBuilder) Len() int

func (b *FastStringBuilder) Reset()

func (b *FastStringBuilder) String() string

func (b *FastStringBuilder) Write(p []byte)

func (b *FastStringBuilder) WriteByte(c byte) error

func (b *FastStringBuilder) WriteString(s string)

type FeedbackConfig ¶

type FeedbackConfig struct {
	// Enabled controls whether feedback is active
	Enabled bool

	// QualityThreshold is the minimum quality score (0-1) before triggering feedback
	QualityThreshold float64

	// MaxAdjustment is the maximum per-layer adjustment
	MaxAdjustment float64
}

FeedbackConfig holds configuration for inter-layer feedback

func DefaultFeedbackConfig() FeedbackConfig

type FeedbackLoop ¶

type FeedbackLoop struct {
	// contains filtered or unexported fields
}

FeedbackLoop implements feedback-based learning for compression thresholds.

func NewFeedbackLoop() *FeedbackLoop

func (fl *FeedbackLoop) GetThreshold(key string, base float64) float64

func (fl *FeedbackLoop) Record(key string, quality float64)

type FeedbackSignal ¶

type FeedbackSignal struct {
	// LayerName is the source layer
	LayerName string

	// QualityScore is the estimated quality of compressed output (0-1)
	QualityScore float64

	// CompressionRatio is the achieved compression ratio
	CompressionRatio float64

	// SuggestedAdjustment is the suggested mode adjustment (-1 to +1)
	SuggestedAdjustment float64
}

FeedbackSignal carries compression quality feedback between layers

type Filter ¶

type Filter interface {
	// Name returns the filter name.
	Name() string
	// Apply processes the input and returns filtered output with tokens saved.
	Apply(input string, mode Mode) (output string, tokensSaved int)
}

Filter defines the interface for output filters.

func SafeFilterFunc(filter Filter, name string) Filter

type FusionStageMap ¶ added in v0.28.0

type FusionStageMap struct {
	Stage    string
	LayerIDs []string
}

FusionStageMap describes Claw-style 14-stage coverage mapped onto TokMan layers.

func ClawFusionStageCoverage() []FusionStageMap

type GMSAFilter ¶ added in v0.28.0

type GMSAFilter struct {
	// contains filtered or unexported fields
}

Paper: "GMSA: Enhancing Context Compression via Group Merging and Layer Semantic Alignment" arXiv:2505.12215 — 2025

GMSAFilter operates at paragraph/chunk level (blank-line separated blocks), complementing NearDedupFilter (line-level) and PerceptionCompressFilter (window-level).

Two phases:

1. Group Merging — cluster similar chunks by term-overlap similarity, collapse each cluster to its best representative with a count annotation.
2. Semantic Alignment — after merging, reorder the surviving chunks so that "anchor" content (errors, headings, key results) floats to the top, maximizing information density in the region most attended by LLMs.

Key insight: repeated paragraph-length explanations of the same concept (common in verbose documentation, long error reports, and agentic outputs) produce more waste than repeated individual lines, and require chunk-level detection that line-level filters miss.

func NewGMSAFilter() *GMSAFilter

func (f *GMSAFilter) Apply(input string, mode Mode) (string, int)

func (f *GMSAFilter) Name() string

type GistFilter ¶

type GistFilter struct {
	// contains filtered or unexported fields
}

Paper: "Gisting" — Mu et al., Stanford, 2023 https://arxiv.org/abs/2304.08467 Paper: "Gisting" — Mu et al., Stanford, 2023 https://arxiv.org/abs/2304.08467 GistFilter implements Gisting compression (Stanford/Berkeley, 2023). Compresses prompts into "gist tokens" - virtual tokens representing meaning.

Algorithm: 1. Identify semantic chunks in the text 2. Replace each chunk with a compressed "gist" representation 3. Use prefix-tuning style markers for reconstruction 4. Preserve critical structural elements

Research Results: 20x+ compression for repetitive content. Key insight: LLMs can understand compressed "gist" representations.

func NewGistFilter() *GistFilter

func (f *GistFilter) Apply(input string, mode Mode) (string, int)

func (f *GistFilter) Name() string

func (f *GistFilter) SetMaxChunkSize(size int)

type GoalDrivenFilter ¶

type GoalDrivenFilter struct {
	// contains filtered or unexported fields
}

Paper: "SWE-Pruner" — Wang et al., Shanghai Jiao Tong, 2026 https://arxiv.org/abs/2601.16746 Paper: "SWE-Pruner" — Wang et al., Shanghai Jiao Tong, 2026 https://arxiv.org/abs/2601.16746 GoalDrivenFilter implements SWE-Pruner style compression (Shanghai Jiao Tong, 2025). Goal-driven line-level pruning using CRF-inspired scoring.

Algorithm: 1. Parse goal/intent from query 2. Score each line for relevance to goal 3. Apply CRF-style sequential labeling for keep/prune decisions 4. Preserve structural coherence

Research Results: Up to 14.8x compression for code contexts.

func NewGoalDrivenFilter(goal string) *GoalDrivenFilter

func (f *GoalDrivenFilter) Apply(input string, mode Mode) (string, int)

func (f *GoalDrivenFilter) Name() string

type GoalMode ¶

type GoalMode int

GoalMode defines the goal-driven filtering mode

const (
	GoalModeDebug GoalMode = iota
	GoalModeReview
	GoalModeDeploy
	GoalModeSearch
	GoalModeBuild
	GoalModeTest
	GoalModeGeneric
)

type GraphCoTFilter ¶ added in v0.28.0

type GraphCoTFilter struct {
	// contains filtered or unexported fields
}

GraphCoTFilter keeps high-centrality reasoning lines in long traces.

func NewGraphCoTFilter() *GraphCoTFilter

func (f *GraphCoTFilter) Apply(input string, mode Mode) (string, int)

func (f *GraphCoTFilter) Name() string

type GuardrailResult ¶ added in v0.28.0

type GuardrailResult struct {
	Passed bool
	Reason string
}

type H2OConfig ¶

type H2OConfig struct {
	// Enable H2O filtering
	Enabled bool

	// Number of attention sink tokens to always preserve (first N tokens)
	SinkSize int

	// Number of recent tokens to always preserve
	RecentSize int

	// Number of heavy hitter tokens to preserve based on importance
	HeavyHitterSize int

	// Minimum content length to apply compression
	MinContentLength int

	// Window size for chunk processing
	ChunkWindow int
}

H2OConfig holds configuration for H2O compression

func DefaultH2OConfig() H2OConfig

type H2OFilter ¶

type H2OFilter struct {
	// contains filtered or unexported fields
}

Paper: "H2O: Heavy-Hitter Oracle" — Zhang et al., NeurIPS, 2023 https://arxiv.org/abs/2306.14048 Paper: "H2O: Heavy-Hitter Oracle" — Zhang et al., NeurIPS, 2023 https://arxiv.org/abs/2306.14048 H2OFilter implements Heavy-Hitter Oracle compression. Research basis: "H2O: Heavy-Hitter Oracle for Efficient Generative Inference" (Zhang et al., NeurIPS 2023) - achieves 30x+ compression via intelligent eviction.

Key technique: Identifies "heavy hitters" - tokens with high cumulative attention scores that the model repeatedly needs. Combines with: 1. Recent token window for local context 2. Attention sinks (initial tokens) for computational stability

This is Layer 13 in the pipeline, implementing KV cache-style compression for text without requiring actual model attention scores.

func NewH2OFilter() *H2OFilter

func (h *H2OFilter) Apply(input string, mode Mode) (string, int)

func (h *H2OFilter) GetStats() map[string]any

func (h *H2OFilter) Name() string

func (h *H2OFilter) SetEnabled(enabled bool)

type H2OLayerConfig ¶

type H2OLayerConfig struct {
	Enabled         bool
	SinkSize        int
	RecentSize      int
	HeavyHitterSize int
}

H2OLayerConfig groups Layer 13 settings.

type HierarchicalFilter ¶

type HierarchicalFilter struct {
	// contains filtered or unexported fields
}

HierarchicalFilter implements multi-level summarization for large outputs. Based on "Hierarchical Context Compression" research - creates a tree-like structure where each level provides progressively more detail.

For outputs exceeding a threshold (default 10K lines), this filter: 1. Segments the output into logical sections 2. Generates summaries at multiple abstraction levels 3. Preserves the most important sections verbatim 4. Compresses mid-importance sections into summaries 5. Drops low-importance sections entirely

func NewHierarchicalFilter() *HierarchicalFilter

func (f *HierarchicalFilter) Apply(input string, mode Mode) (string, int)

func (f *HierarchicalFilter) Name() string

func (f *HierarchicalFilter) SetLineThreshold(threshold int)

func (f *HierarchicalFilter) SetMaxDepth(depth int)

type HierarchicalSummaryFilter ¶

type HierarchicalSummaryFilter struct {
	// contains filtered or unexported fields
}

Paper: "AutoCompressor" — Chevalier et al., Princeton, 2023 https://arxiv.org/abs/2305.14788 Paper: "AutoCompressor" — Chevalier et al., Princeton, 2023 https://arxiv.org/abs/2305.14788 HierarchicalSummaryFilter implements AutoCompressor-style compression (Princeton/MIT, 2023). Recursive summarization that compresses context into summary vectors.

Algorithm: 1. Divide content into hierarchical levels (sections → paragraphs → sentences) 2. Summarize each level recursively 3. Combine summaries with preserved key content 4. Use bottom-up summarization for maximum compression

Research Results: Extreme compression (depends on summary size). Key insight: Recursive summarization preserves global context.

func NewHierarchicalSummaryFilter() *HierarchicalSummaryFilter

func (f *HierarchicalSummaryFilter) Apply(input string, mode Mode) (string, int)

func (f *HierarchicalSummaryFilter) Name() string

func (f *HierarchicalSummaryFilter) SetMaxLevels(levels int)

func (f *HierarchicalSummaryFilter) SetSummaryRatio(ratio float64)

type IBConfig ¶

type IBConfig struct {
	Enabled            bool
	EntropyThreshold   float64
	RelevanceThreshold float64
}

IBConfig holds configuration for information bottleneck.

func DefaultIBConfig() IBConfig

type ImportFilter ¶

type ImportFilter struct {
	// contains filtered or unexported fields
}

ImportFilter condenses import statements.

func NewImportFilter() *ImportFilter

func (f *ImportFilter) Apply(input string, mode Mode) (string, int)

func (f *ImportFilter) Name() string

type IncrementalDelta ¶

type IncrementalDelta struct {
	Added     []string
	Removed   []string
	Unchanged int
}

IncrementalDelta computes the diff between old and new content. Inspired by lean-ctx's ctx_delta.

func ComputeDelta(old, new string) IncrementalDelta

type InformationBottleneck ¶

type InformationBottleneck struct {
	// contains filtered or unexported fields
}

InformationBottleneck filters content by entropy and task-relevance.

func NewInformationBottleneck(cfg IBConfig) *InformationBottleneck

func (ib *InformationBottleneck) Process(content, query string) string

type InterLayerFeedback ¶

type InterLayerFeedback struct {
	// contains filtered or unexported fields
}

InterLayerFeedback implements cross-layer feedback mechanism. This allows later layers to signal earlier layers to adjust aggressiveness, creating an adaptive pipeline that self-corrects based on compression results.

func NewInterLayerFeedback() *InterLayerFeedback

func (f *InterLayerFeedback) GetAdjustment(layerName string) float64

func (f *InterLayerFeedback) RecordSignal(signal FeedbackSignal)

func (f *InterLayerFeedback) Reset()

type JSONSamplerFilter ¶ added in v0.28.0

type JSONSamplerFilter struct{}

JSONSamplerFilter down-samples dense JSON line streams while preserving anchors.

func NewJSONSamplerFilter() *JSONSamplerFilter

func (f *JSONSamplerFilter) Apply(input string, mode Mode) (string, int)

func (f *JSONSamplerFilter) Name() string

type KVCacheAligner ¶

type KVCacheAligner struct {
	// contains filtered or unexported fields
}

KVCacheAligner implements KV-cache alignment for LLM prompt caching. Inspired by claw-compactor's QuantumLock and kompact's cache_aligner. Isolates stable prefix from dynamic content to maximize provider-level caching.

func NewKVCacheAligner(cfg KVCacheConfig) *KVCacheAligner

func (a *KVCacheAligner) AlignPrefix(content string) (string, string, string)

func (a *KVCacheAligner) CacheAwareCompress(content string, compressor *PipelineCoordinator) (string, int)

type KVCacheConfig ¶

type KVCacheConfig struct {
	Enabled          bool
	MinPrefixLength  int
	MaxDynamicSuffix int
	SplitThreshold   int
}

KVCacheConfig holds configuration for KV-cache alignment.

func DefaultKVCacheConfig() KVCacheConfig

type L0Summary ¶ added in v0.28.0

type L0Summary struct {
	Keywords   []string `json:"keywords"`
	Entities   []string `json:"entities"`
	Topics     []string `json:"topics"`
	TokenCount int      `json:"token_count"`
}

L0Summary is the surface-level summary.

type L1Summary ¶ added in v0.28.0

type L1Summary struct {
	Title      string    `json:"title"`
	Sections   []Section `json:"sections"`
	Outline    string    `json:"outline"`
	TokenCount int       `json:"token_count"`
}

L1Summary is the structural summary.

type L2Summary ¶ added in v0.28.0

type L2Summary struct {
	Summary      string   `json:"summary"`
	KeyPoints    []string `json:"key_points"`
	Implications []string `json:"implications,omitempty"`
	TokenCount   int      `json:"token_count"`
}

L2Summary is the deep semantic summary.

type LLMCompressRequest ¶

type LLMCompressRequest struct {
	Content   string `json:"content"`
	MaxTokens int    `json:"max_tokens"`
	Mode      string `json:"mode"`
}

LLMCompressRequest is the JSON input for LLM compression.

type LLMCompressResponse ¶

type LLMCompressResponse struct {
	Compressed string `json:"compressed"`
	TokensIn   int    `json:"tokens_in"`
	TokensOut  int    `json:"tokens_out"`
}

LLMCompressResponse is the JSON output from LLM compression.

type LLMCompressor ¶

type LLMCompressor struct {
	// contains filtered or unexported fields
}

LLMCompressor uses an external LLM for semantic compression. Inspired by claw-compactor's Nexus and tamp's textpress.

func NewLLMCompressor(binPath string) *LLMCompressor

func (lc *LLMCompressor) Compress(content string, maxTokens int) (string, int, int)

func (lc *LLMCompressor) IsEnabled() bool

func (lc *LLMCompressor) SetEnabled(enabled bool)

type LRUCache ¶

type LRUCache = cache.LRUCache

LRUCache is an alias to the unified LRU cache implementation. Note: The cache stores *CachedResult values defined locally in manager.go.

type Language ¶

type Language string

Language represents a programming language for filtering

const (
	LangRust       Language = "rust"
	LangPython     Language = "python"
	LangJavaScript Language = "javascript"
	LangTypeScript Language = "typescript"
	LangGo         Language = "go"
	LangC          Language = "c"
	LangCpp        Language = "cpp"
	LangJava       Language = "java"
	LangRuby       Language = "ruby"
	LangShell      Language = "sh"
	LangSQL        Language = "sql"
	LangUnknown    Language = "unknown"
)

func DetectLanguageFromInput(input string) Language

type LatentCollabFilter ¶ added in v0.28.0

type LatentCollabFilter struct {
	// contains filtered or unexported fields
}

LatentCollabFilter approximates latent-space collaboration by collapsing semantically equivalent multi-agent turns into compact markers.

func NewLatentCollabFilter() *LatentCollabFilter

func (f *LatentCollabFilter) Apply(input string, mode Mode) (string, int)

func (f *LatentCollabFilter) Name() string

type LayerCache ¶ added in v0.28.0

type LayerCache struct {
	// contains filtered or unexported fields
}

LayerCache provides content-addressable caching for filter results. Uses SHA-256 hashing for cache keys with LRU eviction.

func GetGlobalLayerCache() *LayerCache

func NewLayerCache(maxSize int, ttl time.Duration) *LayerCache

func (c *LayerCache) Clear()

func (c *LayerCache) Get(layerName, input string, mode Mode) (*LayerCacheEntry, bool)

func (c *LayerCache) Put(layerName, input string, mode Mode, output string, tokensSaved int)

func (c *LayerCache) Stats() CacheStats

type LayerCacheEntry ¶ added in v0.28.0

type LayerCacheEntry struct {
	InputHash   string
	Output      string
	TokensSaved int
	LayerName   string
	Mode        Mode
	Timestamp   time.Time
	HitCount    int
}

LayerCacheEntry stores cached filter results.

type LayerConfig ¶

type LayerConfig struct {
	Core              CoreLayersConfig
	Compaction        CompactionLayerConfig
	Attribution       AttributionLayerConfig
	H2O               H2OLayerConfig
	AttentionSink     AttentionSinkLayerConfig
	MetaToken         MetaTokenLayerConfig
	SemanticChunk     SemanticChunkLayerConfig
	SketchStore       SketchStoreLayerConfig
	LazyPruner        LazyPrunerLayerConfig
	SemanticAnchor    SemanticAnchorLayerConfig
	AgentMemory       AgentMemoryLayerConfig
	QuestionAware     QuestionAwareLayerConfig
	DensityAdaptive   DensityAdaptiveLayerConfig
	TFIDF             TFIDFLayerConfig
	NumericalQuant    NumericalQuantLayerConfig
	DynamicRatio      DynamicRatioLayerConfig
	SymbolicCompress  bool
	PhraseGrouping    bool
	Hypernym          bool
	SemanticCache     bool
	Scope             bool
	SmallKV           bool
	KVzip             bool
	SWEzze            bool
	MixedDim          bool
	BEAVER            bool
	PoC               bool
	TokenQuant        bool
	TokenRetention    bool
	ACON              bool
	TOMLFilter        bool
	TOMLFilterCommand string
	CacheEnabled      bool

	// New: Claw Compactor features
	EnableAdaptiveLearning bool // Enable adaptive learning (merged EngramLearner + TieredSummary)
	EnableCrunchBench      bool // Enable comprehensive benchmarking
}