config

package
v1.5.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Apr 2, 2026 License: MIT Imports: 9 Imported by: 0

Documentation

Index

Constants

View Source
const (
	DefaultMaxContextTokens = 2000000 // 2M tokens max
	DefaultChunkSize        = 100000  // 100K tokens per chunk
	DefaultStreamThreshold  = 500000  // Stream if > 500K tokens
	DefaultCacheMaxSize     = 1000    // Max cache entries
)

Default pipeline limits.

Variables

This section is empty.

Functions

func ConfigDir

func ConfigDir() string

ConfigDir returns the path to the tokman config directory.

func ConfigPath

func ConfigPath() string

ConfigPath returns the path to the configuration file. Follows XDG Base Directory Specification on Unix. Uses %APPDATA% on Windows.

func DataPath

func DataPath() string

DataPath returns the path to the data directory. Follows XDG Base Directory Specification on Unix. Uses %LOCALAPPDATA% on Windows.

func DatabasePath

func DatabasePath() string

DatabasePath returns the path to the SQLite database.

func FiltersDir

func FiltersDir() string

FiltersDir returns the path to the user filters directory.

func HooksPath

func HooksPath() string

HooksPath returns the path to the hooks directory.

func LogPath

func LogPath() string

LogPath returns the path to the log file.

func ProjectPath

func ProjectPath() string

ProjectPath returns the canonical path for the current working directory. Resolves symlinks for accurate project matching.

Types

type AlertsConfig

type AlertsConfig struct {
	Enabled             bool    `mapstructure:"enabled"`
	DailyTokenLimit     int64   `mapstructure:"daily_token_limit"`
	WeeklyTokenLimit    int64   `mapstructure:"weekly_token_limit"`
	UsageSpikeThreshold float64 `mapstructure:"usage_spike_threshold"`
}

AlertsConfig controls alert thresholds.

type CommandContext

type CommandContext struct {
	Command    string `mapstructure:"command"`    // "git", "npm", "cargo", etc.
	Subcommand string `mapstructure:"subcommand"` // "status", "test", "build"
	ExitCode   int    `mapstructure:"exit_code"`  // Non-zero = likely has errors
	Intent     string `mapstructure:"intent"`     // "debug", "review", "deploy", "search"
	IsTest     bool   `mapstructure:"is_test"`    // Test output detection
	IsBuild    bool   `mapstructure:"is_build"`   // Build output detection
	IsError    bool   `mapstructure:"is_error"`   // Error output detection
}

CommandContext provides metadata about the command being executed. Used for intelligent filtering decisions.

type Config

type Config struct {
	Tracking  TrackingConfig  `mapstructure:"tracking"`
	Filter    FilterConfig    `mapstructure:"filter"`
	Pipeline  PipelineConfig  `mapstructure:"pipeline"`
	Hooks     HooksConfig     `mapstructure:"hooks"`
	Dashboard DashboardConfig `mapstructure:"dashboard"`
	Alerts    AlertsConfig    `mapstructure:"alerts"`
	Export    ExportConfig    `mapstructure:"export"`
}

Config represents the main configuration structure.

func Defaults

func Defaults() *Config

Defaults returns the default configuration.

func Load

func Load(cfgFile string) (*Config, error)

Load reads configuration from file and environment.

func LoadFromFile

func LoadFromFile(path string) (*Config, error)

LoadFromFile reads a TOML configuration file directly.

func (*Config) GetDatabasePath

func (c *Config) GetDatabasePath() string

GetDatabasePath returns the effective database path.

func (*Config) Save

func (c *Config) Save(path string) (retErr error)

Save writes the configuration to a TOML file.

func (*Config) Validate

func (c *Config) Validate() error

Validate checks configuration values for correctness and applies corrections.

type DashboardConfig

type DashboardConfig struct {
	Port           int    `mapstructure:"port"`
	Bind           string `mapstructure:"bind"`
	UpdateInterval int    `mapstructure:"update_interval"`
	Theme          string `mapstructure:"theme"`
	EnableExport   bool   `mapstructure:"enable_export"`
}

DashboardConfig controls dashboard behavior.

type ExportConfig

type ExportConfig struct {
	DefaultFormat    string `mapstructure:"default_format"`
	IncludeTimestamp bool   `mapstructure:"include_timestamps"`
	MaxRecords       int    `mapstructure:"max_records"`
}

ExportConfig controls export behavior.

type FilterConfig

type FilterConfig struct {
	NoiseDirs   []string `mapstructure:"noise_dirs"`
	IgnoreFiles []string `mapstructure:"ignore_files"` // File patterns to ignore (e.g., "*.lock", "*.min.js")
	Mode        string   `mapstructure:"mode"`         // "minimal" or "aggressive"
	MaxWidth    int      `mapstructure:"max_width"`    // Max display width (0 = auto)
}

FilterConfig controls output filtering behavior.

type HooksConfig

type HooksConfig struct {
	ExcludedCommands []string `mapstructure:"excluded_commands"`
	AuditDir         string   `mapstructure:"audit_dir"` // Directory for hook audit logs
	TeeDir           string   `mapstructure:"tee_dir"`   // Directory for failure tee logs
}

HooksConfig controls shell hook behavior.

type PipelineConfig

type PipelineConfig struct {
	// Context limits
	MaxContextTokens int `mapstructure:"max_context_tokens"` // Max input context (default: 2M)
	ChunkSize        int `mapstructure:"chunk_size"`         // Processing chunk size for large inputs

	// Layer enable/disable
	EnableEntropy      bool `mapstructure:"enable_entropy"`
	EnablePerplexity   bool `mapstructure:"enable_perplexity"`
	EnableGoalDriven   bool `mapstructure:"enable_goal_driven"`
	EnableAST          bool `mapstructure:"enable_ast"`
	EnableContrastive  bool `mapstructure:"enable_contrastive"`
	EnableNgram        bool `mapstructure:"enable_ngram"`
	EnableEvaluator    bool `mapstructure:"enable_evaluator"`
	EnableGist         bool `mapstructure:"enable_gist"`
	EnableHierarchical bool `mapstructure:"enable_hierarchical"`
	EnableBudget       bool `mapstructure:"enable_budget"`

	// Layer thresholds (tunable)
	EntropyThreshold      float64 `mapstructure:"entropy_threshold"`       // Layer 1: 0.0-1.0 (default: 0.3)
	PerplexityThreshold   float64 `mapstructure:"perplexity_threshold"`    // Layer 2: 0.0-1.0 (default: 0.5)
	GoalDrivenThreshold   float64 `mapstructure:"goal_driven_threshold"`   // Layer 3: 0.0-1.0 (default: 0.4)
	ASTPreserveThreshold  float64 `mapstructure:"ast_preserve_threshold"`  // Layer 4: 0.0-1.0 (default: 0.6)
	ContrastiveThreshold  float64 `mapstructure:"contrastive_threshold"`   // Layer 5: 0.0-1.0 (default: 0.5)
	NgramMinOccurrences   int     `mapstructure:"ngram_min_occurrences"`   // Layer 6: min repeats (default: 3)
	EvaluatorThreshold    float64 `mapstructure:"evaluator_threshold"`     // Layer 7: 0.0-1.0 (default: 0.4)
	GistMinChunkSize      int     `mapstructure:"gist_min_chunk_size"`     // Layer 8: chars (default: 100)
	HierarchicalMaxLevels int     `mapstructure:"hierarchical_max_levels"` // Layer 9: depth (default: 3)
	HierarchicalRatio     float64 `mapstructure:"hierarchical_ratio"`      // Layer 9: 0.0-1.0 (default: 0.3)

	// Budget enforcement
	DefaultBudget      int    `mapstructure:"default_budget"`       // Default token budget (0 = unlimited)
	HardBudgetLimit    bool   `mapstructure:"hard_budget_limit"`    // Strict enforcement
	BudgetOverflowFile string `mapstructure:"budget_overflow_file"` // File to save overflow content

	// Resilience
	TeeOnFailure       bool   `mapstructure:"tee_on_failure"`       // Save raw output on failure
	TeeDir             string `mapstructure:"tee_dir"`              // Directory for tee files
	FailSafeMode       bool   `mapstructure:"failsafe_mode"`        // Return original on corruption
	ValidateOutput     bool   `mapstructure:"validate_output"`      // Check output validity
	ShortCircuitBudget bool   `mapstructure:"short_circuit_budget"` // Skip layers if budget met

	// Performance
	ParallelLayers  bool `mapstructure:"parallel_layers"`  // Run independent layers in parallel
	CacheEnabled    bool `mapstructure:"cache_enabled"`    // Cache compression results
	CacheMaxSize    int  `mapstructure:"cache_max_size"`   // Max cache entries
	StreamThreshold int  `mapstructure:"stream_threshold"` // Stream if input > N tokens

	// LLM Compaction (Layer 11) - Semantic compression
	EnableCompaction        bool   `mapstructure:"enable_compaction"`         // Enable LLM-based compaction
	CompactionThreshold     int    `mapstructure:"compaction_threshold"`      // Minimum tokens to trigger
	CompactionPreserveTurns int    `mapstructure:"compaction_preserve_turns"` // Recent turns to keep verbatim
	CompactionMaxTokens     int    `mapstructure:"compaction_max_tokens"`     // Max summary tokens
	CompactionStateSnapshot bool   `mapstructure:"compaction_state_snapshot"` // Use state snapshot format
	CompactionAutoDetect    bool   `mapstructure:"compaction_auto_detect"`    // Auto-detect conversation content
	LLMProvider             string `mapstructure:"llm_provider"`              // ollama, lmstudio, openai
	LLMModel                string `mapstructure:"llm_model"`                 // Model name
	LLMBaseURL              string `mapstructure:"llm_base_url"`              // API endpoint

	// Attribution Filter (Layer 12) - ProCut-style pruning
	EnableAttribution     bool    `mapstructure:"enable_attribution"`     // Enable attribution filtering
	AttributionThreshold  float64 `mapstructure:"attribution_threshold"`  // Importance threshold (0.0-1.0)
	AttributionPositional bool    `mapstructure:"attribution_positional"` // Use positional bias
	AttributionFrequency  bool    `mapstructure:"attribution_frequency"`  // Use frequency bias
	AttributionSemantic   bool    `mapstructure:"attribution_semantic"`   // Use semantic preservation

	// H2O Filter (Layer 13) - Heavy-Hitter Oracle
	EnableH2O          bool `mapstructure:"enable_h2o"`            // Enable H2O compression
	H2OSinkSize        int  `mapstructure:"h2o_sink_size"`         // Attention sink tokens to preserve
	H2ORecentSize      int  `mapstructure:"h2o_recent_size"`       // Recent tokens to preserve
	H2OHeavyHitterSize int  `mapstructure:"h2o_heavy_hitter_size"` // Heavy hitter tokens to preserve

	// Attention Sink Filter (Layer 14) - StreamingLLM-style
	EnableAttentionSink  bool `mapstructure:"enable_attention_sink"`  // Enable attention sink filtering
	AttentionSinkCount   int  `mapstructure:"attention_sink_count"`   // Initial tokens to preserve as sinks
	AttentionRecentCount int  `mapstructure:"attention_recent_count"` // Recent lines to preserve

	// Meta-Token Compression (Layer 15) - LZ77-style lossless compression
	EnableMetaToken   bool `mapstructure:"enable_meta_token"`    // Enable meta-token compression
	MetaTokenWindow   int  `mapstructure:"meta_token_window"`    // Sliding window size for token matching
	MetaTokenMinMatch int  `mapstructure:"meta_token_min_match"` // Minimum match length

	// Semantic Chunking (Layer 16) - Dynamic boundary detection
	EnableSemanticChunk bool    `mapstructure:"enable_semantic_chunk"` // Enable semantic chunking
	SemanticThreshold   float64 `mapstructure:"semantic_threshold"`    // Semantic shift threshold (0.0-1.0)
	ChunkMinSize        int     `mapstructure:"chunk_min_size"`        // Minimum chunk size in tokens
	ChunkMaxSize        int     `mapstructure:"chunk_max_size"`        // Maximum chunk size in tokens

	// Sketch Store (Layer 17) - Reversible compression with KVReviver
	EnableSketchStore bool `mapstructure:"enable_sketch_store"` // Enable sketch-based storage
	SketchMemoryRatio int  `mapstructure:"sketch_memory_ratio"` // Memory reduction ratio (default: 90%)
	SketchOnDemand    bool `mapstructure:"sketch_on_demand"`    // Reconstruct on-demand when needed

	// Lazy Pruner (Layer 18) - Budget-aware dynamic pruning
	EnableLazyPruner bool    `mapstructure:"enable_lazy_pruner"` // Enable lazy pruning
	LazyBudgetRatio  float64 `mapstructure:"lazy_budget_ratio"`  // Ratio of budget to use for lazy pruning
	LazyLayerDecay   float64 `mapstructure:"lazy_layer_decay"`   // Decay factor for layer-wise pruning

	// Semantic Anchor (Layer 19) - Attention gradient detection
	EnableSemanticAnchor bool    `mapstructure:"enable_semantic_anchor"` // Enable semantic anchor preservation
	AnchorThreshold      float64 `mapstructure:"anchor_threshold"`       // Gradient threshold for anchors
	AnchorMinPreserve    int     `mapstructure:"anchor_min_preserve"`    // Minimum anchors to preserve

	// Agent Memory (Layer 20) - Knowledge graph extraction
	EnableAgentMemory    bool   `mapstructure:"enable_agent_memory"`     // Enable agent memory extraction
	AgentMemoryMaxNodes  int    `mapstructure:"agent_memory_max_nodes"`  // Max nodes in knowledge graph
	AgentMemoryMaxEdges  int    `mapstructure:"agent_memory_max_edges"`  // Max edges in knowledge graph
	AgentMemoryExtractFn string `mapstructure:"agent_memory_extract_fn"` // Extraction function type

	// Perplexity Filter (Layer 2) - Detailed settings
	PerplexityTargetRatio          float64 `mapstructure:"perplexity_target_ratio"`          // Target compression ratio (default: 0.3)
	PerplexityIterationSteps       int     `mapstructure:"perplexity_iteration_steps"`       // Number of pruning iterations (default: 2)
	PerplexityPruneRatio           float64 `mapstructure:"perplexity_prune_ratio"`           // Prune ratio per iteration (default: 0.7)
	PerplexityConvergenceThreshold float64 `mapstructure:"perplexity_convergence_threshold"` // Early exit threshold (default: 0.05)
	PerplexityContextWindow        int     `mapstructure:"perplexity_context_window"`        // Context window size (default: 10)

	// Entropy Filter (Layer 1) - Sampling settings
	EntropySamplingLarge int `mapstructure:"entropy_sampling_large"`  // Sample threshold for large inputs (default: 100000)
	EntropySamplingHuge  int `mapstructure:"entropy_sampling_huge"`   // Sample threshold for huge inputs (default: 500000)
	EntropySamplingRate  int `mapstructure:"entropy_sampling_rate"`   // Sample rate for huge inputs (default: 10)
	EntropyMinWordLength int `mapstructure:"entropy_min_word_length"` // Minimum word length (default: 2)
}

PipelineConfig controls the 20-layer compression pipeline. Supports contexts up to 2M tokens with streaming processing. Based on 120+ research papers from top institutions worldwide.

func (PipelineConfig) ToFilterPipelineConfig

func (c PipelineConfig) ToFilterPipelineConfig(opts PipelineRuntimeOptions) filter.PipelineConfig

ToFilterPipelineConfig converts user-facing config into the runtime pipeline config. Some fields are best-effort mappings because the public config and runtime pipeline have diverged over time; centralizing that mapping keeps behavior consistent.

type PipelineRuntimeOptions

type PipelineRuntimeOptions struct {
	Mode        filter.Mode
	QueryIntent string
	Budget      int
	LLMEnabled  bool
}

PipelineRuntimeOptions carries per-request overrides for the runtime pipeline.

type TrackingConfig

type TrackingConfig struct {
	Enabled      bool   `mapstructure:"enabled"`
	DatabasePath string `mapstructure:"database_path"`
	Telemetry    bool   `mapstructure:"telemetry"`
}

TrackingConfig controls token tracking behavior.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL