Documentation
¶
Overview ¶
Package config provides configuration management for TokMan.
The config package handles loading, validating, and saving configuration from TOML files, environment variables, and command-line flags.
Configuration Loading ¶
Use Load() to read configuration from files and environment:
cfg, err := config.Load(cfgFile)
if err != nil { ... }
Defaults ¶
The Defaults() function returns a configuration with sensible defaults:
cfg := config.Defaults()
Validation ¶
All configurations are validated via the Validate() method to ensure threshold values, limits, and cross-field constraints are satisfied.
Index ¶
- Constants
- func ConfigDir() string
- func ConfigPath() string
- func DataPath() string
- func DatabasePath() string
- func FiltersDir() string
- func FiltersPath() string
- func HooksPath() string
- func LogPath() string
- func ProjectPath() string
- type AlertsConfig
- type CommandContext
- type Config
- type DashboardConfig
- type ExportConfig
- type FilterConfig
- type HooksConfig
- type PipelineConfig
- type TrackingConfig
Constants ¶
const ( DefaultMaxContextTokens = 2000000 // 2M tokens max DefaultChunkSize = 100000 // 100K tokens per chunk DefaultStreamThreshold = 500000 // Stream if > 500K tokens DefaultCacheMaxSize = 1000 // Max cache entries )
Default pipeline limits.
Variables ¶
This section is empty.
Functions ¶
func ConfigPath ¶
func ConfigPath() string
ConfigPath returns the path to the configuration file. Follows XDG Base Directory Specification on Unix. Uses %APPDATA% on Windows. Falls back to temp directory if no home directory is available.
func DataPath ¶
func DataPath() string
DataPath returns the path to the data directory. Follows XDG Base Directory Specification on Unix. Uses %LOCALAPPDATA% on Windows. Falls back to temp directory if no home directory is available.
func DatabasePath ¶
func DatabasePath() string
DatabasePath returns the path to the SQLite database.
func FiltersDir ¶
func FiltersDir() string
FiltersDir returns the path to the user filters directory.
func FiltersPath ¶ added in v0.28.0
func FiltersPath() string
FiltersPath returns the path to the filters directory (alias for FiltersDir).
func ProjectPath ¶
func ProjectPath() string
ProjectPath returns the canonical path for the current working directory. Resolves symlinks for accurate project matching.
Types ¶
type AlertsConfig ¶
type AlertsConfig struct {
Enabled bool `mapstructure:"enabled"`
DailyTokenLimit int64 `mapstructure:"daily_token_limit"`
WeeklyTokenLimit int64 `mapstructure:"weekly_token_limit"`
UsageSpikeThreshold float64 `mapstructure:"usage_spike_threshold"`
}
AlertsConfig controls alert thresholds.
type CommandContext ¶
type CommandContext struct {
Command string `mapstructure:"command"` // "git", "npm", "cargo", etc.
Subcommand string `mapstructure:"subcommand"` // "status", "test", "build"
ExitCode int `mapstructure:"exit_code"` // Non-zero = likely has errors
Intent string `mapstructure:"intent"` // "debug", "review", "deploy", "search"
IsTest bool `mapstructure:"is_test"` // Test output detection
IsBuild bool `mapstructure:"is_build"` // Build output detection
IsError bool `mapstructure:"is_error"` // Error output detection
}
CommandContext provides metadata about the command being executed. Used for intelligent filtering decisions.
type Config ¶
type Config struct {
Tracking TrackingConfig `mapstructure:"tracking"`
Filter FilterConfig `mapstructure:"filter"`
Pipeline PipelineConfig `mapstructure:"pipeline"`
Hooks HooksConfig `mapstructure:"hooks"`
Dashboard DashboardConfig `mapstructure:"dashboard"`
Alerts AlertsConfig `mapstructure:"alerts"`
Export ExportConfig `mapstructure:"export"`
}
Config represents the main configuration structure.
func LoadFromFile ¶
LoadFromFile reads a TOML configuration file directly.
func (*Config) GetDatabasePath ¶
GetDatabasePath returns the effective database path.
type DashboardConfig ¶
type DashboardConfig struct {
Port int `mapstructure:"port"`
Bind string `mapstructure:"bind"`
UpdateInterval int `mapstructure:"update_interval"`
Theme string `mapstructure:"theme"`
EnableExport bool `mapstructure:"enable_export"`
}
DashboardConfig controls dashboard behavior.
type ExportConfig ¶
type ExportConfig struct {
DefaultFormat string `mapstructure:"default_format"`
IncludeTimestamp bool `mapstructure:"include_timestamps"`
MaxRecords int `mapstructure:"max_records"`
}
ExportConfig controls export behavior.
type FilterConfig ¶
type FilterConfig struct {
NoiseDirs []string `mapstructure:"noise_dirs"`
IgnoreFiles []string `mapstructure:"ignore_files"` // File patterns to ignore (e.g., "*.lock", "*.min.js")
Mode string `mapstructure:"mode"` // "minimal" or "aggressive"
MaxWidth int `mapstructure:"max_width"` // Max display width (0 = auto)
}
FilterConfig controls output filtering behavior.
type HooksConfig ¶
type HooksConfig struct {
ExcludedCommands []string `mapstructure:"excluded_commands"`
AuditDir string `mapstructure:"audit_dir"` // Directory for hook audit logs
TeeDir string `mapstructure:"tee_dir"` // Directory for failure tee logs
}
HooksConfig controls shell hook behavior.
type PipelineConfig ¶
type PipelineConfig struct {
// Context limits
MaxContextTokens int `mapstructure:"max_context_tokens"` // Max input context (default: 2M)
ChunkSize int `mapstructure:"chunk_size"` // Processing chunk size for large inputs
// Layer enable/disable
EnableEntropy bool `mapstructure:"enable_entropy"`
EnablePerplexity bool `mapstructure:"enable_perplexity"`
EnableGoalDriven bool `mapstructure:"enable_goal_driven"`
EnableAST bool `mapstructure:"enable_ast"`
EnableContrastive bool `mapstructure:"enable_contrastive"`
EnableNgram bool `mapstructure:"enable_ngram"`
EnableEvaluator bool `mapstructure:"enable_evaluator"`
EnableGist bool `mapstructure:"enable_gist"`
EnableHierarchical bool `mapstructure:"enable_hierarchical"`
EnableBudget bool `mapstructure:"enable_budget"`
// Layer thresholds (tunable)
EntropyThreshold float64 `mapstructure:"entropy_threshold"` // Layer 1: 0.0-1.0 (default: 0.3)
PerplexityThreshold float64 `mapstructure:"perplexity_threshold"` // Layer 2: 0.0-1.0 (default: 0.5)
GoalDrivenThreshold float64 `mapstructure:"goal_driven_threshold"` // Layer 3: 0.0-1.0 (default: 0.4)
ASTPreserveThreshold float64 `mapstructure:"ast_preserve_threshold"` // Layer 4: 0.0-1.0 (default: 0.6)
ContrastiveThreshold float64 `mapstructure:"contrastive_threshold"` // Layer 5: 0.0-1.0 (default: 0.5)
NgramMinOccurrences int `mapstructure:"ngram_min_occurrences"` // Layer 6: min repeats (default: 3)
EvaluatorThreshold float64 `mapstructure:"evaluator_threshold"` // Layer 7: 0.0-1.0 (default: 0.4)
GistMinChunkSize int `mapstructure:"gist_min_chunk_size"` // Layer 8: chars (default: 100)
HierarchicalMaxLevels int `mapstructure:"hierarchical_max_levels"` // Layer 9: depth (default: 3)
HierarchicalRatio float64 `mapstructure:"hierarchical_ratio"` // Layer 9: 0.0-1.0 (default: 0.3)
// Budget enforcement
DefaultBudget int `mapstructure:"default_budget"` // Default token budget (0 = unlimited)
HardBudgetLimit bool `mapstructure:"hard_budget_limit"` // Strict enforcement
BudgetOverflowFile string `mapstructure:"budget_overflow_file"` // File to save overflow content
// Resilience
TeeOnFailure bool `mapstructure:"tee_on_failure"` // Save raw output on failure
TeeDir string `mapstructure:"tee_dir"` // Directory for tee files
FailSafeMode bool `mapstructure:"failsafe_mode"` // Return original on corruption
ValidateOutput bool `mapstructure:"validate_output"` // Check output validity
ShortCircuitBudget bool `mapstructure:"short_circuit_budget"` // Skip layers if budget met
// Performance
ParallelLayers bool `mapstructure:"parallel_layers"` // Run independent layers in parallel
CacheEnabled bool `mapstructure:"cache_enabled"` // Cache compression results
CacheMaxSize int `mapstructure:"cache_max_size"` // Max cache entries
StreamThreshold int `mapstructure:"stream_threshold"` // Stream if input > N tokens
// Adaptive routing and extractive prefilter
EnablePolicyRouter bool `mapstructure:"enable_policy_router"` // Infer intent from output
EnableExtractiveFilter bool `mapstructure:"enable_extractive_prefilter"` // Enable extractive prefilter
ExtractiveMaxLines int `mapstructure:"extractive_max_lines"` // Trigger threshold
ExtractiveHeadLines int `mapstructure:"extractive_head_lines"` // Head lines to preserve
ExtractiveTailLines int `mapstructure:"extractive_tail_lines"` // Tail lines to preserve
ExtractiveSignalLines int `mapstructure:"extractive_signal_lines"` // Signal lines to preserve
EnableQualityGuardrail bool `mapstructure:"enable_quality_guardrail"` // Auto-fallback on quality risk
EnablePlannedLayers bool `mapstructure:"enable_planned_layers"` // Enable experimental 30-49 layer pack
// LLM Compaction (Layer 11) - Semantic compression
EnableCompaction bool `mapstructure:"enable_compaction"` // Enable LLM-based compaction
CompactionThreshold int `mapstructure:"compaction_threshold"` // Minimum tokens to trigger
CompactionPreserveTurns int `mapstructure:"compaction_preserve_turns"` // Recent turns to keep verbatim
CompactionMaxTokens int `mapstructure:"compaction_max_tokens"` // Max summary tokens
CompactionStateSnapshot bool `mapstructure:"compaction_state_snapshot"` // Use state snapshot format
CompactionAutoDetect bool `mapstructure:"compaction_auto_detect"` // Auto-detect conversation content
LLMProvider string `mapstructure:"llm_provider"` // ollama, lmstudio, openai
LLMModel string `mapstructure:"llm_model"` // Model name
LLMBaseURL string `mapstructure:"llm_base_url"` // API endpoint
// Attribution Filter (Layer 12) - ProCut-style pruning
EnableAttribution bool `mapstructure:"enable_attribution"` // Enable attribution filtering
AttributionThreshold float64 `mapstructure:"attribution_threshold"` // Importance threshold (0.0-1.0)
AttributionPositional bool `mapstructure:"attribution_positional"` // Use positional bias
AttributionFrequency bool `mapstructure:"attribution_frequency"` // Use frequency bias
AttributionSemantic bool `mapstructure:"attribution_semantic"` // Use semantic preservation
// H2O Filter (Layer 13) - Heavy-Hitter Oracle
EnableH2O bool `mapstructure:"enable_h2o"` // Enable H2O compression
H2OSinkSize int `mapstructure:"h2o_sink_size"` // Attention sink tokens to preserve
H2ORecentSize int `mapstructure:"h2o_recent_size"` // Recent tokens to preserve
H2OHeavyHitterSize int `mapstructure:"h2o_heavy_hitter_size"` // Heavy hitter tokens to preserve
// Attention Sink Filter (Layer 14) - StreamingLLM-style
EnableAttentionSink bool `mapstructure:"enable_attention_sink"` // Enable attention sink filtering
AttentionSinkCount int `mapstructure:"attention_sink_count"` // Initial tokens to preserve as sinks
AttentionRecentCount int `mapstructure:"attention_recent_count"` // Recent lines to preserve
// Meta-Token Compression (Layer 15) - LZ77-style lossless compression
EnableMetaToken bool `mapstructure:"enable_meta_token"` // Enable meta-token compression
MetaTokenWindow int `mapstructure:"meta_token_window"` // Sliding window size for token matching
MetaTokenMinMatch int `mapstructure:"meta_token_min_match"` // Minimum match length
// Semantic Chunking (Layer 16) - Dynamic boundary detection
EnableSemanticChunk bool `mapstructure:"enable_semantic_chunk"` // Enable semantic chunking
SemanticThreshold float64 `mapstructure:"semantic_threshold"` // Semantic shift threshold (0.0-1.0)
ChunkMinSize int `mapstructure:"chunk_min_size"` // Minimum chunk size in tokens
ChunkMaxSize int `mapstructure:"chunk_max_size"` // Maximum chunk size in tokens
// Sketch Store (Layer 17) - Reversible compression with KVReviver
EnableSketchStore bool `mapstructure:"enable_sketch_store"` // Enable sketch-based storage
SketchMemoryRatio int `mapstructure:"sketch_memory_ratio"` // Memory reduction ratio (default: 90%)
SketchOnDemand bool `mapstructure:"sketch_on_demand"` // Reconstruct on-demand when needed
// Lazy Pruner (Layer 18) - Budget-aware dynamic pruning
EnableLazyPruner bool `mapstructure:"enable_lazy_pruner"` // Enable lazy pruning
LazyBudgetRatio float64 `mapstructure:"lazy_budget_ratio"` // Ratio of budget to use for lazy pruning
LazyLayerDecay float64 `mapstructure:"lazy_layer_decay"` // Decay factor for layer-wise pruning
// Semantic Anchor (Layer 19) - Attention gradient detection
EnableSemanticAnchor bool `mapstructure:"enable_semantic_anchor"` // Enable semantic anchor preservation
AnchorThreshold float64 `mapstructure:"anchor_threshold"` // Gradient threshold for anchors
AnchorMinPreserve int `mapstructure:"anchor_min_preserve"` // Minimum anchors to preserve
// Agent Memory (Layer 20) - Knowledge graph extraction
EnableAgentMemory bool `mapstructure:"enable_agent_memory"` // Enable agent memory extraction
AgentMemoryMaxNodes int `mapstructure:"agent_memory_max_nodes"` // Max nodes in knowledge graph
AgentMemoryMaxEdges int `mapstructure:"agent_memory_max_edges"` // Max edges in knowledge graph
AgentMemoryExtractFn string `mapstructure:"agent_memory_extract_fn"` // Extraction function type
// Research Layers (31-49)
EnableDiffAdapt bool `mapstructure:"enable_difft_adapt"` // Difficulty-adaptive pruning
EnableEPiC bool `mapstructure:"enable_epic"` // Causal-edge preservation
EnableSSDP bool `mapstructure:"enable_ssdp"` // ToT branch pruning
EnableAgentOCR bool `mapstructure:"enable_agent_ocr"` // Turn-density compression
EnableS2MAD bool `mapstructure:"enable_s2_mad"` // Agreement collapse
EnableACON bool `mapstructure:"enable_acon"` // Adaptive context optimization
EnableLatentCollab bool `mapstructure:"enable_latent_collab"` // Latent collaboration merge
EnableGraphCoT bool `mapstructure:"enable_graph_cot"` // Graph-CoT compression
EnableRoleBudget bool `mapstructure:"enable_role_budget"` // Role-aware budgeting
EnableSWEAdaptive bool `mapstructure:"enable_swe_adaptive_loop"` // SWE adaptive prune loop
EnableAgentOCRHist bool `mapstructure:"enable_agent_ocr_history"` // AgentOCR history compaction
EnablePlanBudget bool `mapstructure:"enable_plan_budget"` // Plan-and-budget controller
EnableLightMem bool `mapstructure:"enable_lightmem"` // Lightweight memory reuse
EnablePathShorten bool `mapstructure:"enable_path_shorten"` // Path/identifier shortening
EnableJSONSampler bool `mapstructure:"enable_json_sampler"` // JSON statistical sampler
EnableContextCrunch bool `mapstructure:"enable_context_crunch"` // Context crunch (merged log + diff folding)
EnableSearchCrunch bool `mapstructure:"enable_search_crunch"` // Search result dedup stage
EnableStructColl bool `mapstructure:"enable_structural_collapse"` // Structural boilerplate collapse
EnableResearchPack bool `mapstructure:"enable_research_pack"` // One-toggle research bundle
// Perplexity Filter (Layer 2) - Detailed settings
PerplexityTargetRatio float64 `mapstructure:"perplexity_target_ratio"` // Target compression ratio (default: 0.3)
PerplexityIterationSteps int `mapstructure:"perplexity_iteration_steps"` // Number of pruning iterations (default: 2)
PerplexityPruneRatio float64 `mapstructure:"perplexity_prune_ratio"` // Prune ratio per iteration (default: 0.7)
PerplexityConvergenceThreshold float64 `mapstructure:"perplexity_convergence_threshold"` // Early exit threshold (default: 0.05)
PerplexityContextWindow int `mapstructure:"perplexity_context_window"` // Context window size (default: 10)
// Entropy Filter (Layer 1) - Sampling settings
EntropySamplingLarge int `mapstructure:"entropy_sampling_large"` // Sample threshold for large inputs (default: 100000)
EntropySamplingHuge int `mapstructure:"entropy_sampling_huge"` // Sample threshold for huge inputs (default: 500000)
EntropySamplingRate int `mapstructure:"entropy_sampling_rate"` // Sample rate for huge inputs (default: 10)
EntropyMinWordLength int `mapstructure:"entropy_min_word_length"` // Minimum word length (default: 2)
}
PipelineConfig controls the 20-layer compression pipeline. Supports contexts up to 2M tokens with streaming processing. Based on 120+ research papers from top institutions worldwide.
type TrackingConfig ¶
type TrackingConfig struct {
Enabled bool `mapstructure:"enabled"`
DatabasePath string `mapstructure:"database_path"`
Telemetry bool `mapstructure:"telemetry"`
}
TrackingConfig controls token tracking behavior.