config

package
v0.28.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Apr 16, 2026 License: MIT Imports: 8 Imported by: 0

Documentation

Overview

Package config provides configuration management for TokMan.

The config package handles loading, validating, and saving configuration from TOML files, environment variables, and command-line flags.

Configuration Loading

Use Load() to read configuration from files and environment:

cfg, err := config.Load(cfgFile)
if err != nil { ... }

Defaults

The Defaults() function returns a configuration with sensible defaults:

cfg := config.Defaults()

Validation

All configurations are validated via the Validate() method to ensure threshold values, limits, and cross-field constraints are satisfied.

Index

Constants

View Source
const (
	DefaultMaxContextTokens = 2000000 // 2M tokens max
	DefaultChunkSize        = 100000  // 100K tokens per chunk
	DefaultStreamThreshold  = 500000  // Stream if > 500K tokens
	DefaultCacheMaxSize     = 1000    // Max cache entries
)

Default pipeline limits.

Variables

This section is empty.

Functions

func ConfigDir

func ConfigDir() string

ConfigDir returns the path to the tokman config directory.

func ConfigPath

func ConfigPath() string

ConfigPath returns the path to the configuration file. Follows XDG Base Directory Specification on Unix. Uses %APPDATA% on Windows. Falls back to temp directory if no home directory is available.

func DataPath

func DataPath() string

DataPath returns the path to the data directory. Follows XDG Base Directory Specification on Unix. Uses %LOCALAPPDATA% on Windows. Falls back to temp directory if no home directory is available.

func DatabasePath

func DatabasePath() string

DatabasePath returns the path to the SQLite database.

func FiltersDir

func FiltersDir() string

FiltersDir returns the path to the user filters directory.

func FiltersPath added in v0.28.0

func FiltersPath() string

FiltersPath returns the path to the filters directory (alias for FiltersDir).

func HooksPath

func HooksPath() string

HooksPath returns the path to the hooks directory.

func LogPath

func LogPath() string

LogPath returns the path to the log file.

func ProjectPath

func ProjectPath() string

ProjectPath returns the canonical path for the current working directory. Resolves symlinks for accurate project matching.

Types

type AlertsConfig

type AlertsConfig struct {
	Enabled             bool    `mapstructure:"enabled"`
	DailyTokenLimit     int64   `mapstructure:"daily_token_limit"`
	WeeklyTokenLimit    int64   `mapstructure:"weekly_token_limit"`
	UsageSpikeThreshold float64 `mapstructure:"usage_spike_threshold"`
}

AlertsConfig controls alert thresholds.

type CommandContext

type CommandContext struct {
	Command    string `mapstructure:"command"`    // "git", "npm", "cargo", etc.
	Subcommand string `mapstructure:"subcommand"` // "status", "test", "build"
	ExitCode   int    `mapstructure:"exit_code"`  // Non-zero = likely has errors
	Intent     string `mapstructure:"intent"`     // "debug", "review", "deploy", "search"
	IsTest     bool   `mapstructure:"is_test"`    // Test output detection
	IsBuild    bool   `mapstructure:"is_build"`   // Build output detection
	IsError    bool   `mapstructure:"is_error"`   // Error output detection
}

CommandContext provides metadata about the command being executed. Used for intelligent filtering decisions.

type Config

type Config struct {
	Tracking  TrackingConfig  `mapstructure:"tracking"`
	Filter    FilterConfig    `mapstructure:"filter"`
	Pipeline  PipelineConfig  `mapstructure:"pipeline"`
	Hooks     HooksConfig     `mapstructure:"hooks"`
	Dashboard DashboardConfig `mapstructure:"dashboard"`
	Alerts    AlertsConfig    `mapstructure:"alerts"`
	Export    ExportConfig    `mapstructure:"export"`
}

Config represents the main configuration structure.

func Defaults

func Defaults() *Config

Defaults returns the default configuration.

func Load

func Load(cfgFile string) (*Config, error)

Load reads configuration from file and environment.

func LoadFromFile

func LoadFromFile(path string) (*Config, error)

LoadFromFile reads a TOML configuration file directly.

func (*Config) GetDatabasePath

func (c *Config) GetDatabasePath() string

GetDatabasePath returns the effective database path.

func (*Config) Save

func (c *Config) Save(path string) (retErr error)

Save writes the configuration to a TOML file.

func (*Config) Validate

func (c *Config) Validate() error

Validate checks configuration values for correctness and applies corrections.

type DashboardConfig

type DashboardConfig struct {
	Port           int    `mapstructure:"port"`
	Bind           string `mapstructure:"bind"`
	UpdateInterval int    `mapstructure:"update_interval"`
	Theme          string `mapstructure:"theme"`
	EnableExport   bool   `mapstructure:"enable_export"`
}

DashboardConfig controls dashboard behavior.

type ExportConfig

type ExportConfig struct {
	DefaultFormat    string `mapstructure:"default_format"`
	IncludeTimestamp bool   `mapstructure:"include_timestamps"`
	MaxRecords       int    `mapstructure:"max_records"`
}

ExportConfig controls export behavior.

type FilterConfig

type FilterConfig struct {
	NoiseDirs   []string `mapstructure:"noise_dirs"`
	IgnoreFiles []string `mapstructure:"ignore_files"` // File patterns to ignore (e.g., "*.lock", "*.min.js")
	Mode        string   `mapstructure:"mode"`         // "minimal" or "aggressive"
	MaxWidth    int      `mapstructure:"max_width"`    // Max display width (0 = auto)
}

FilterConfig controls output filtering behavior.

type HooksConfig

type HooksConfig struct {
	ExcludedCommands []string `mapstructure:"excluded_commands"`
	AuditDir         string   `mapstructure:"audit_dir"` // Directory for hook audit logs
	TeeDir           string   `mapstructure:"tee_dir"`   // Directory for failure tee logs
}

HooksConfig controls shell hook behavior.

type PipelineConfig

type PipelineConfig struct {
	// Context limits
	MaxContextTokens int `mapstructure:"max_context_tokens"` // Max input context (default: 2M)
	ChunkSize        int `mapstructure:"chunk_size"`         // Processing chunk size for large inputs

	// Layer enable/disable
	EnableEntropy      bool `mapstructure:"enable_entropy"`
	EnablePerplexity   bool `mapstructure:"enable_perplexity"`
	EnableGoalDriven   bool `mapstructure:"enable_goal_driven"`
	EnableAST          bool `mapstructure:"enable_ast"`
	EnableContrastive  bool `mapstructure:"enable_contrastive"`
	EnableNgram        bool `mapstructure:"enable_ngram"`
	EnableEvaluator    bool `mapstructure:"enable_evaluator"`
	EnableGist         bool `mapstructure:"enable_gist"`
	EnableHierarchical bool `mapstructure:"enable_hierarchical"`
	EnableBudget       bool `mapstructure:"enable_budget"`

	// Layer thresholds (tunable)
	EntropyThreshold      float64 `mapstructure:"entropy_threshold"`       // Layer 1: 0.0-1.0 (default: 0.3)
	PerplexityThreshold   float64 `mapstructure:"perplexity_threshold"`    // Layer 2: 0.0-1.0 (default: 0.5)
	GoalDrivenThreshold   float64 `mapstructure:"goal_driven_threshold"`   // Layer 3: 0.0-1.0 (default: 0.4)
	ASTPreserveThreshold  float64 `mapstructure:"ast_preserve_threshold"`  // Layer 4: 0.0-1.0 (default: 0.6)
	ContrastiveThreshold  float64 `mapstructure:"contrastive_threshold"`   // Layer 5: 0.0-1.0 (default: 0.5)
	NgramMinOccurrences   int     `mapstructure:"ngram_min_occurrences"`   // Layer 6: min repeats (default: 3)
	EvaluatorThreshold    float64 `mapstructure:"evaluator_threshold"`     // Layer 7: 0.0-1.0 (default: 0.4)
	GistMinChunkSize      int     `mapstructure:"gist_min_chunk_size"`     // Layer 8: chars (default: 100)
	HierarchicalMaxLevels int     `mapstructure:"hierarchical_max_levels"` // Layer 9: depth (default: 3)
	HierarchicalRatio     float64 `mapstructure:"hierarchical_ratio"`      // Layer 9: 0.0-1.0 (default: 0.3)

	// Budget enforcement
	DefaultBudget      int    `mapstructure:"default_budget"`       // Default token budget (0 = unlimited)
	HardBudgetLimit    bool   `mapstructure:"hard_budget_limit"`    // Strict enforcement
	BudgetOverflowFile string `mapstructure:"budget_overflow_file"` // File to save overflow content

	// Resilience
	TeeOnFailure       bool   `mapstructure:"tee_on_failure"`       // Save raw output on failure
	TeeDir             string `mapstructure:"tee_dir"`              // Directory for tee files
	FailSafeMode       bool   `mapstructure:"failsafe_mode"`        // Return original on corruption
	ValidateOutput     bool   `mapstructure:"validate_output"`      // Check output validity
	ShortCircuitBudget bool   `mapstructure:"short_circuit_budget"` // Skip layers if budget met

	// Performance
	ParallelLayers  bool `mapstructure:"parallel_layers"`  // Run independent layers in parallel
	CacheEnabled    bool `mapstructure:"cache_enabled"`    // Cache compression results
	CacheMaxSize    int  `mapstructure:"cache_max_size"`   // Max cache entries
	StreamThreshold int  `mapstructure:"stream_threshold"` // Stream if input > N tokens

	// Adaptive routing and extractive prefilter
	EnablePolicyRouter     bool `mapstructure:"enable_policy_router"`        // Infer intent from output
	EnableExtractiveFilter bool `mapstructure:"enable_extractive_prefilter"` // Enable extractive prefilter
	ExtractiveMaxLines     int  `mapstructure:"extractive_max_lines"`        // Trigger threshold
	ExtractiveHeadLines    int  `mapstructure:"extractive_head_lines"`       // Head lines to preserve
	ExtractiveTailLines    int  `mapstructure:"extractive_tail_lines"`       // Tail lines to preserve
	ExtractiveSignalLines  int  `mapstructure:"extractive_signal_lines"`     // Signal lines to preserve
	EnableQualityGuardrail bool `mapstructure:"enable_quality_guardrail"`    // Auto-fallback on quality risk
	EnablePlannedLayers    bool `mapstructure:"enable_planned_layers"`       // Enable experimental 30-49 layer pack

	// LLM Compaction (Layer 11) - Semantic compression
	EnableCompaction        bool   `mapstructure:"enable_compaction"`         // Enable LLM-based compaction
	CompactionThreshold     int    `mapstructure:"compaction_threshold"`      // Minimum tokens to trigger
	CompactionPreserveTurns int    `mapstructure:"compaction_preserve_turns"` // Recent turns to keep verbatim
	CompactionMaxTokens     int    `mapstructure:"compaction_max_tokens"`     // Max summary tokens
	CompactionStateSnapshot bool   `mapstructure:"compaction_state_snapshot"` // Use state snapshot format
	CompactionAutoDetect    bool   `mapstructure:"compaction_auto_detect"`    // Auto-detect conversation content
	LLMProvider             string `mapstructure:"llm_provider"`              // ollama, lmstudio, openai
	LLMModel                string `mapstructure:"llm_model"`                 // Model name
	LLMBaseURL              string `mapstructure:"llm_base_url"`              // API endpoint

	// Attribution Filter (Layer 12) - ProCut-style pruning
	EnableAttribution     bool    `mapstructure:"enable_attribution"`     // Enable attribution filtering
	AttributionThreshold  float64 `mapstructure:"attribution_threshold"`  // Importance threshold (0.0-1.0)
	AttributionPositional bool    `mapstructure:"attribution_positional"` // Use positional bias
	AttributionFrequency  bool    `mapstructure:"attribution_frequency"`  // Use frequency bias
	AttributionSemantic   bool    `mapstructure:"attribution_semantic"`   // Use semantic preservation

	// H2O Filter (Layer 13) - Heavy-Hitter Oracle
	EnableH2O          bool `mapstructure:"enable_h2o"`            // Enable H2O compression
	H2OSinkSize        int  `mapstructure:"h2o_sink_size"`         // Attention sink tokens to preserve
	H2ORecentSize      int  `mapstructure:"h2o_recent_size"`       // Recent tokens to preserve
	H2OHeavyHitterSize int  `mapstructure:"h2o_heavy_hitter_size"` // Heavy hitter tokens to preserve

	// Attention Sink Filter (Layer 14) - StreamingLLM-style
	EnableAttentionSink  bool `mapstructure:"enable_attention_sink"`  // Enable attention sink filtering
	AttentionSinkCount   int  `mapstructure:"attention_sink_count"`   // Initial tokens to preserve as sinks
	AttentionRecentCount int  `mapstructure:"attention_recent_count"` // Recent lines to preserve

	// Meta-Token Compression (Layer 15) - LZ77-style lossless compression
	EnableMetaToken   bool `mapstructure:"enable_meta_token"`    // Enable meta-token compression
	MetaTokenWindow   int  `mapstructure:"meta_token_window"`    // Sliding window size for token matching
	MetaTokenMinMatch int  `mapstructure:"meta_token_min_match"` // Minimum match length

	// Semantic Chunking (Layer 16) - Dynamic boundary detection
	EnableSemanticChunk bool    `mapstructure:"enable_semantic_chunk"` // Enable semantic chunking
	SemanticThreshold   float64 `mapstructure:"semantic_threshold"`    // Semantic shift threshold (0.0-1.0)
	ChunkMinSize        int     `mapstructure:"chunk_min_size"`        // Minimum chunk size in tokens
	ChunkMaxSize        int     `mapstructure:"chunk_max_size"`        // Maximum chunk size in tokens

	// Sketch Store (Layer 17) - Reversible compression with KVReviver
	EnableSketchStore bool `mapstructure:"enable_sketch_store"` // Enable sketch-based storage
	SketchMemoryRatio int  `mapstructure:"sketch_memory_ratio"` // Memory reduction ratio (default: 90%)
	SketchOnDemand    bool `mapstructure:"sketch_on_demand"`    // Reconstruct on-demand when needed

	// Lazy Pruner (Layer 18) - Budget-aware dynamic pruning
	EnableLazyPruner bool    `mapstructure:"enable_lazy_pruner"` // Enable lazy pruning
	LazyBudgetRatio  float64 `mapstructure:"lazy_budget_ratio"`  // Ratio of budget to use for lazy pruning
	LazyLayerDecay   float64 `mapstructure:"lazy_layer_decay"`   // Decay factor for layer-wise pruning

	// Semantic Anchor (Layer 19) - Attention gradient detection
	EnableSemanticAnchor bool    `mapstructure:"enable_semantic_anchor"` // Enable semantic anchor preservation
	AnchorThreshold      float64 `mapstructure:"anchor_threshold"`       // Gradient threshold for anchors
	AnchorMinPreserve    int     `mapstructure:"anchor_min_preserve"`    // Minimum anchors to preserve

	// Agent Memory (Layer 20) - Knowledge graph extraction
	EnableAgentMemory    bool   `mapstructure:"enable_agent_memory"`     // Enable agent memory extraction
	AgentMemoryMaxNodes  int    `mapstructure:"agent_memory_max_nodes"`  // Max nodes in knowledge graph
	AgentMemoryMaxEdges  int    `mapstructure:"agent_memory_max_edges"`  // Max edges in knowledge graph
	AgentMemoryExtractFn string `mapstructure:"agent_memory_extract_fn"` // Extraction function type

	// Research Layers (31-49)
	EnableDiffAdapt     bool `mapstructure:"enable_difft_adapt"`         // Difficulty-adaptive pruning
	EnableEPiC          bool `mapstructure:"enable_epic"`                // Causal-edge preservation
	EnableSSDP          bool `mapstructure:"enable_ssdp"`                // ToT branch pruning
	EnableAgentOCR      bool `mapstructure:"enable_agent_ocr"`           // Turn-density compression
	EnableS2MAD         bool `mapstructure:"enable_s2_mad"`              // Agreement collapse
	EnableACON          bool `mapstructure:"enable_acon"`                // Adaptive context optimization
	EnableLatentCollab  bool `mapstructure:"enable_latent_collab"`       // Latent collaboration merge
	EnableGraphCoT      bool `mapstructure:"enable_graph_cot"`           // Graph-CoT compression
	EnableRoleBudget    bool `mapstructure:"enable_role_budget"`         // Role-aware budgeting
	EnableSWEAdaptive   bool `mapstructure:"enable_swe_adaptive_loop"`   // SWE adaptive prune loop
	EnableAgentOCRHist  bool `mapstructure:"enable_agent_ocr_history"`   // AgentOCR history compaction
	EnablePlanBudget    bool `mapstructure:"enable_plan_budget"`         // Plan-and-budget controller
	EnableLightMem      bool `mapstructure:"enable_lightmem"`            // Lightweight memory reuse
	EnablePathShorten   bool `mapstructure:"enable_path_shorten"`        // Path/identifier shortening
	EnableJSONSampler   bool `mapstructure:"enable_json_sampler"`        // JSON statistical sampler
	EnableContextCrunch bool `mapstructure:"enable_context_crunch"`      // Context crunch (merged log + diff folding)
	EnableSearchCrunch  bool `mapstructure:"enable_search_crunch"`       // Search result dedup stage
	EnableStructColl    bool `mapstructure:"enable_structural_collapse"` // Structural boilerplate collapse
	EnableResearchPack  bool `mapstructure:"enable_research_pack"`       // One-toggle research bundle

	// Perplexity Filter (Layer 2) - Detailed settings
	PerplexityTargetRatio          float64 `mapstructure:"perplexity_target_ratio"`          // Target compression ratio (default: 0.3)
	PerplexityIterationSteps       int     `mapstructure:"perplexity_iteration_steps"`       // Number of pruning iterations (default: 2)
	PerplexityPruneRatio           float64 `mapstructure:"perplexity_prune_ratio"`           // Prune ratio per iteration (default: 0.7)
	PerplexityConvergenceThreshold float64 `mapstructure:"perplexity_convergence_threshold"` // Early exit threshold (default: 0.05)
	PerplexityContextWindow        int     `mapstructure:"perplexity_context_window"`        // Context window size (default: 10)

	// Entropy Filter (Layer 1) - Sampling settings
	EntropySamplingLarge int `mapstructure:"entropy_sampling_large"`  // Sample threshold for large inputs (default: 100000)
	EntropySamplingHuge  int `mapstructure:"entropy_sampling_huge"`   // Sample threshold for huge inputs (default: 500000)
	EntropySamplingRate  int `mapstructure:"entropy_sampling_rate"`   // Sample rate for huge inputs (default: 10)
	EntropyMinWordLength int `mapstructure:"entropy_min_word_length"` // Minimum word length (default: 2)
}

PipelineConfig controls the 20-layer compression pipeline. Supports contexts up to 2M tokens with streaming processing. Based on 120+ research papers from top institutions worldwide.

type TrackingConfig

type TrackingConfig struct {
	Enabled      bool   `mapstructure:"enabled"`
	DatabasePath string `mapstructure:"database_path"`
	Telemetry    bool   `mapstructure:"telemetry"`
}

TrackingConfig controls token tracking behavior.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL