config

package
v0.1.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 4, 2026 License: MIT Imports: 8 Imported by: 0

Documentation

Overview

Package config provides configuration management for tok.

The config package handles loading, validating, and saving configuration from TOML files, environment variables, and command-line flags.

Configuration Loading

Use Load() to read configuration from files and environment:

cfg, err := config.Load(cfgFile)
if err != nil { ... }

Defaults

The Defaults() function returns a configuration with sensible defaults:

cfg := config.Defaults()

Validation

All configurations are validated via the Validate() method to ensure threshold values, limits, and cross-field constraints are satisfied.

Index

Constants

View Source
const (
	DefaultMaxContextTokens = 2000000 // 2M tokens max
	DefaultChunkSize        = 100000  // 100K tokens per chunk
	DefaultStreamThreshold  = 500000  // Stream if > 500K tokens
	DefaultCacheMaxSize     = 1000    // Max cache entries
)

Default pipeline limits.

Variables

This section is empty.

Functions

func ConfigDir

func ConfigDir() string

ConfigDir returns the path to the tok config directory.

func ConfigPath

func ConfigPath() string

ConfigPath returns the path to the configuration file. Follows XDG Base Directory Specification on Unix. Uses %APPDATA% on Windows. Falls back to temp directory if no home directory is available.

func DataPath

func DataPath() string

DataPath returns the path to the data directory. Follows XDG Base Directory Specification on Unix. Uses %LOCALAPPDATA% on Windows. Falls back to temp directory if no home directory is available.

func DatabasePath

func DatabasePath() string

DatabasePath returns the path to the SQLite database.

func FiltersDir

func FiltersDir() string

FiltersDir returns the path to the user filters directory.

func FiltersPath

func FiltersPath() string

FiltersPath returns the path to the filters directory (alias for FiltersDir).

func HooksPath

func HooksPath() string

HooksPath returns the path to the hooks directory.

func LogPath

func LogPath() string

LogPath returns the path to the log file.

func ProjectPath

func ProjectPath() string

ProjectPath returns the canonical path for the current working directory. Resolves symlinks for accurate project matching.

Types

type AlertsConfig

type AlertsConfig struct {
	Enabled             bool    `mapstructure:"enabled"`
	DailyTokenLimit     int64   `mapstructure:"daily_token_limit"`
	WeeklyTokenLimit    int64   `mapstructure:"weekly_token_limit"`
	UsageSpikeThreshold float64 `mapstructure:"usage_spike_threshold"`
}

AlertsConfig controls alert thresholds.

type BudgetConfig

type BudgetConfig struct {
	DailyTokens      int `mapstructure:"daily_tokens"`      // Daily token budget (0 = unlimited)
	WarningThreshold int `mapstructure:"warning_threshold"` // Percent for warning (default: 80)
}

BudgetConfig controls token budget tracking and alerts.

type CommandContext

type CommandContext struct {
	Command    string `mapstructure:"command"`    // "git", "npm", "cargo", etc.
	Subcommand string `mapstructure:"subcommand"` // "status", "test", "build"
	ExitCode   int    `mapstructure:"exit_code"`  // Non-zero = likely has errors
	Intent     string `mapstructure:"intent"`     // "debug", "review", "deploy", "search"
	IsTest     bool   `mapstructure:"is_test"`    // Test output detection
	IsBuild    bool   `mapstructure:"is_build"`   // Build output detection
	IsError    bool   `mapstructure:"is_error"`   // Error output detection
}

CommandContext provides metadata about the command being executed. Used for intelligent filtering decisions.

type CompactionConfig added in v0.1.1

type CompactionConfig struct {
	EnableCompaction        bool   `mapstructure:"enable_compaction"`         // Enable LLM-based compaction
	CompactionThreshold     int    `mapstructure:"compaction_threshold"`      // Minimum tokens to trigger
	CompactionPreserveTurns int    `mapstructure:"compaction_preserve_turns"` // Recent turns to keep verbatim
	CompactionMaxTokens     int    `mapstructure:"compaction_max_tokens"`     // Max summary tokens
	CompactionStateSnapshot bool   `mapstructure:"compaction_state_snapshot"` // Use state snapshot format
	CompactionAutoDetect    bool   `mapstructure:"compaction_auto_detect"`    // Auto-detect conversation content
	LLMProvider             string `mapstructure:"llm_provider"`              // ollama, lmstudio, openai
	LLMModel                string `mapstructure:"llm_model"`                 // Model name
	LLMBaseURL              string `mapstructure:"llm_base_url"`              // API endpoint
}

CompactionConfig groups LLM compaction (Layer 11) settings. Controls semantic compression of conversation history. Field names match the original flat PipelineConfig fields so that embedding promotes them transparently (backward compatible).

type Config

type Config struct {
	Tracking    TrackingConfig    `mapstructure:"tracking"`
	Filter      FilterConfig      `mapstructure:"filter"`
	Pipeline    PipelineConfig    `mapstructure:"pipeline"`
	Hooks       HooksConfig       `mapstructure:"hooks"`
	Dashboard   DashboardConfig   `mapstructure:"dashboard"`
	Alerts      AlertsConfig      `mapstructure:"alerts"`
	Budget      BudgetConfig      `mapstructure:"budget"`
	Export      ExportConfig      `mapstructure:"export"`
	Edit        EditConfig        `mapstructure:"edit"`        // Edit batching configuration
	Keybindings KeybindingsConfig `mapstructure:"keybindings"` // TUI keybindings overrides
}

Config represents the main configuration structure.

func Defaults

func Defaults() *Config

Defaults returns the default configuration.

func Load

func Load(cfgFile string) (*Config, error)

Load reads configuration from file and environment.

func LoadFromFile

func LoadFromFile(path string) (*Config, error)

LoadFromFile reads a TOML configuration file directly.

func (*Config) GetDatabasePath

func (c *Config) GetDatabasePath() string

GetDatabasePath returns the effective database path.

func (*Config) Save

func (c *Config) Save(path string) (retErr error)

Save writes the configuration to a TOML file.

func (*Config) Validate

func (c *Config) Validate() error

Validate checks configuration values for correctness and applies corrections.

type DashboardConfig

type DashboardConfig struct {
	Port           int    `mapstructure:"port"`
	Bind           string `mapstructure:"bind"`
	UpdateInterval int    `mapstructure:"update_interval"`
	Theme          string `mapstructure:"theme"`
	EnableExport   bool   `mapstructure:"enable_export"`
}

DashboardConfig controls dashboard behavior.

type EditConfig

type EditConfig struct {
	BatchEnabled    bool     `mapstructure:"batch_enabled"`         // Enable batching of edit operations
	BatchSize       int      `mapstructure:"batch_size"`            // Max edits per batch (default: 10)
	BatchTimeoutMs  int      `mapstructure:"batch_timeout_ms"`      // Timeout for batch collection (default: 500ms)
	DryRun          bool     `mapstructure:"edit_dry_run"`          // Show what would be changed without writing
	CreateBackups   bool     `mapstructure:"create_backups"`        // Create .bak files before editing
	AtomicWrites    bool     `mapstructure:"atomic_writes"`         // Use atomic file replacement
	MaxFileSize     int      `mapstructure:"max_edit_file_size"`    // Maximum file size for editing (bytes, 0=unlimited)
	AllowedPatterns []string `mapstructure:"allowed_edit_patterns"` // Glob patterns of files allowed to edit
	DeniedPatterns  []string `mapstructure:"denied_edit_patterns"`  // Glob patterns of files denied
}

EditConfig controls edit batching behavior.

type EntropyFilterConfig added in v0.1.1

type EntropyFilterConfig struct {
	EntropySamplingLarge int `mapstructure:"entropy_sampling_large"`  // Sample threshold for large inputs (default: 100000)
	EntropySamplingHuge  int `mapstructure:"entropy_sampling_huge"`   // Sample threshold for huge inputs (default: 500000)
	EntropySamplingRate  int `mapstructure:"entropy_sampling_rate"`   // Sample rate for huge inputs (default: 10)
	EntropyMinWordLength int `mapstructure:"entropy_min_word_length"` // Minimum word length (default: 2)
}

EntropyFilterConfig groups Layer 1 (Entropy Filtering) detailed settings. Field names match the original flat PipelineConfig fields for backward compatibility.

type ExportConfig

type ExportConfig struct {
	DefaultFormat    string `mapstructure:"default_format"`
	IncludeTimestamp bool   `mapstructure:"include_timestamps"`
	MaxRecords       int    `mapstructure:"max_records"`
}

ExportConfig controls export behavior.

type FilterConfig

type FilterConfig struct {
	NoiseDirs   []string `mapstructure:"noise_dirs"`
	IgnoreFiles []string `mapstructure:"ignore_files"` // File patterns to ignore (e.g., "*.lock", "*.min.js")
	Mode        string   `mapstructure:"mode"`         // "minimal" or "aggressive"
	MaxWidth    int      `mapstructure:"max_width"`    // Max display width (0 = auto)
}

FilterConfig controls output filtering behavior.

type HooksConfig

type HooksConfig struct {
	ExcludedCommands []string `mapstructure:"excluded_commands"`
	AuditDir         string   `mapstructure:"audit_dir"`   // Directory for hook audit logs
	TeeDir           string   `mapstructure:"tee_dir"`     // Directory for failure tee logs
	TeeEnabled       bool     `mapstructure:"tee_enabled"` // Enable tee file writing (default: true)
}

HooksConfig controls shell hook behavior.

type KeybindingsConfig

type KeybindingsConfig struct {
	Quit           string `mapstructure:"quit"`            // default: q, ctrl+c
	NextSection    string `mapstructure:"next_section"`    // default: tab, right, l
	PrevSection    string `mapstructure:"prev_section"`    // default: shift+tab, left, h
	HistoryBack    string `mapstructure:"history_back"`    // default: H
	HistoryForward string `mapstructure:"history_forward"` // default: L
	Refresh        string `mapstructure:"refresh"`         // default: r
	Up             string `mapstructure:"up"`              // default: up, k
	Down           string `mapstructure:"down"`            // default: down, j
	PageUp         string `mapstructure:"page_up"`         // default: pgup, ctrl+b
	PageDown       string `mapstructure:"page_down"`       // default: pgdn, ctrl+f
	Top            string `mapstructure:"top"`             // default: g, home
	Bottom         string `mapstructure:"bottom"`          // default: G, end
	Enter          string `mapstructure:"enter"`           // default: enter
	Back           string `mapstructure:"back"`            // default: backspace
	Yank           string `mapstructure:"yank"`            // default: y
	Export         string `mapstructure:"export"`          // default: e
	Palette        string `mapstructure:"palette"`         // default: :
	Search         string `mapstructure:"search"`          // default: /
	Help           string `mapstructure:"help"`            // default: ?
}

KeybindingsConfig allows users to remap TUI keys. Keys use bubbletea's key syntax (e.g., "ctrl+c", "tab", "H").

type PerplexityFilterConfig added in v0.1.1

type PerplexityFilterConfig struct {
	PerplexityTargetRatio          float64 `mapstructure:"perplexity_target_ratio"`          // Target compression ratio (default: 0.3)
	PerplexityIterationSteps       int     `mapstructure:"perplexity_iteration_steps"`       // Number of pruning iterations (default: 2)
	PerplexityPruneRatio           float64 `mapstructure:"perplexity_prune_ratio"`           // Prune ratio per iteration (default: 0.7)
	PerplexityConvergenceThreshold float64 `mapstructure:"perplexity_convergence_threshold"` // Early exit threshold (default: 0.05)
	PerplexityContextWindow        int     `mapstructure:"perplexity_context_window"`        // Context window size (default: 10)
}

PerplexityFilterConfig groups Layer 2 (Perplexity Pruning) detailed settings. Field names match the original flat PipelineConfig fields for backward compatibility.

type PipelineConfig

type PipelineConfig struct {
	// Embedded sub-configs for logical grouping.
	// Fields are promoted, preserving backward compatibility.
	CompactionConfig       `mapstructure:",squash"`
	PerplexityFilterConfig `mapstructure:",squash"`
	EntropyFilterConfig    `mapstructure:",squash"`
	ResearchLayersConfig   `mapstructure:",squash"`
	// Context limits
	MaxContextTokens int `mapstructure:"max_context_tokens"` // Max input context (default: 2M)
	ChunkSize        int `mapstructure:"chunk_size"`         // Processing chunk size for large inputs

	// Agent preset selection
	Preset string `mapstructure:"preset"` // fast, balanced, full, T90, ultra, research

	// Layer enable/disable
	EnableEntropy      bool `mapstructure:"enable_entropy"`
	EnablePerplexity   bool `mapstructure:"enable_perplexity"`
	EnableGoalDriven   bool `mapstructure:"enable_goal_driven"`
	EnableAST          bool `mapstructure:"enable_ast"`
	EnableContrastive  bool `mapstructure:"enable_contrastive"`
	EnableNgram        bool `mapstructure:"enable_ngram"`
	EnableEvaluator    bool `mapstructure:"enable_evaluator"`
	EnableGist         bool `mapstructure:"enable_gist"`
	EnableHierarchical bool `mapstructure:"enable_hierarchical"`
	EnableBudget       bool `mapstructure:"enable_budget"`

	// Layer thresholds (tunable)
	EntropyThreshold      float64 `mapstructure:"entropy_threshold"`       // Layer 1: 0.0-1.0 (default: 0.3)
	PerplexityThreshold   float64 `mapstructure:"perplexity_threshold"`    // Layer 2: 0.0-1.0 (default: 0.5)
	GoalDrivenThreshold   float64 `mapstructure:"goal_driven_threshold"`   // Layer 3: 0.0-1.0 (default: 0.4)
	ASTPreserveThreshold  float64 `mapstructure:"ast_preserve_threshold"`  // Layer 4: 0.0-1.0 (default: 0.6)
	ContrastiveThreshold  float64 `mapstructure:"contrastive_threshold"`   // Layer 5: 0.0-1.0 (default: 0.5)
	NgramMinOccurrences   int     `mapstructure:"ngram_min_occurrences"`   // Layer 6: min repeats (default: 3)
	EvaluatorThreshold    float64 `mapstructure:"evaluator_threshold"`     // Layer 7: 0.0-1.0 (default: 0.4)
	GistMinChunkSize      int     `mapstructure:"gist_min_chunk_size"`     // Layer 8: chars (default: 100)
	HierarchicalMaxLevels int     `mapstructure:"hierarchical_max_levels"` // Layer 9: depth (default: 3)
	HierarchicalRatio     float64 `mapstructure:"hierarchical_ratio"`      // Layer 9: 0.0-1.0 (default: 0.3)

	// Budget enforcement
	DefaultBudget      int    `mapstructure:"default_budget"`       // Default token budget (0 = unlimited)
	HardBudgetLimit    bool   `mapstructure:"hard_budget_limit"`    // Strict enforcement
	BudgetOverflowFile string `mapstructure:"budget_overflow_file"` // File to save overflow content

	// Resilience
	TeeOnFailure       bool   `mapstructure:"tee_on_failure"`       // Save raw output on failure
	TeeDir             string `mapstructure:"tee_dir"`              // Directory for tee files
	FailSafeMode       bool   `mapstructure:"failsafe_mode"`        // Return original on corruption
	ValidateOutput     bool   `mapstructure:"validate_output"`      // Check output validity
	ShortCircuitBudget bool   `mapstructure:"short_circuit_budget"` // Skip layers if budget met

	// Performance
	ParallelLayers  bool `mapstructure:"parallel_layers"`  // Run independent layers in parallel
	CacheEnabled    bool `mapstructure:"cache_enabled"`    // Cache compression results
	CacheMaxSize    int  `mapstructure:"cache_max_size"`   // Max cache entries
	StreamThreshold int  `mapstructure:"stream_threshold"` // Stream if input > N tokens

	// Adaptive routing and extractive prefilter
	EnablePolicyRouter     bool `mapstructure:"enable_policy_router"`        // Infer intent from output
	EnableExtractiveFilter bool `mapstructure:"enable_extractive_prefilter"` // Enable extractive prefilter
	ExtractiveMaxLines     int  `mapstructure:"extractive_max_lines"`        // Trigger threshold
	ExtractiveHeadLines    int  `mapstructure:"extractive_head_lines"`       // Head lines to preserve
	ExtractiveTailLines    int  `mapstructure:"extractive_tail_lines"`       // Tail lines to preserve
	ExtractiveSignalLines  int  `mapstructure:"extractive_signal_lines"`     // Signal lines to preserve
	EnableQualityGuardrail bool `mapstructure:"enable_quality_guardrail"`    // Auto-fallback on quality risk

	// Attribution Filter (Layer 12) - ProCut-style pruning
	EnableAttribution     bool    `mapstructure:"enable_attribution"`     // Enable attribution filtering
	AttributionThreshold  float64 `mapstructure:"attribution_threshold"`  // Importance threshold (0.0-1.0)
	AttributionPositional bool    `mapstructure:"attribution_positional"` // Use positional bias
	AttributionFrequency  bool    `mapstructure:"attribution_frequency"`  // Use frequency bias
	AttributionSemantic   bool    `mapstructure:"attribution_semantic"`   // Use semantic preservation

	// H2O Filter (Layer 13) - Heavy-Hitter Oracle
	EnableH2O          bool `mapstructure:"enable_h2o"`            // Enable H2O compression
	H2OSinkSize        int  `mapstructure:"h2o_sink_size"`         // Attention sink tokens to preserve
	H2ORecentSize      int  `mapstructure:"h2o_recent_size"`       // Recent tokens to preserve
	H2OHeavyHitterSize int  `mapstructure:"h2o_heavy_hitter_size"` // Heavy hitter tokens to preserve

	// Attention Sink Filter (Layer 14) - StreamingLLM-style
	EnableAttentionSink  bool `mapstructure:"enable_attention_sink"`  // Enable attention sink filtering
	AttentionSinkCount   int  `mapstructure:"attention_sink_count"`   // Initial tokens to preserve as sinks
	AttentionRecentCount int  `mapstructure:"attention_recent_count"` // Recent lines to preserve

	// Meta-Token Compression (Layer 15) - LZ77-style lossless compression
	EnableMetaToken   bool `mapstructure:"enable_meta_token"`    // Enable meta-token compression
	MetaTokenWindow   int  `mapstructure:"meta_token_window"`    // Sliding window size for token matching
	MetaTokenMinMatch int  `mapstructure:"meta_token_min_match"` // Minimum match length

	// Semantic Chunking (Layer 16) - Dynamic boundary detection
	EnableSemanticChunk bool    `mapstructure:"enable_semantic_chunk"` // Enable semantic chunking
	SemanticThreshold   float64 `mapstructure:"semantic_threshold"`    // Semantic shift threshold (0.0-1.0)
	ChunkMinSize        int     `mapstructure:"chunk_min_size"`        // Minimum chunk size in tokens
	ChunkMaxSize        int     `mapstructure:"chunk_max_size"`        // Maximum chunk size in tokens

	// Sketch Store (Layer 17) - Reversible compression with KVReviver
	EnableSketchStore bool `mapstructure:"enable_sketch_store"` // Enable sketch-based storage
	SketchMemoryRatio int  `mapstructure:"sketch_memory_ratio"` // Memory reduction ratio (default: 90%)
	SketchOnDemand    bool `mapstructure:"sketch_on_demand"`    // Reconstruct on-demand when needed

	// Lazy Pruner (Layer 18) - Budget-aware dynamic pruning
	EnableLazyPruner bool    `mapstructure:"enable_lazy_pruner"` // Enable lazy pruning
	LazyBudgetRatio  float64 `mapstructure:"lazy_budget_ratio"`  // Ratio of budget to use for lazy pruning
	LazyLayerDecay   float64 `mapstructure:"lazy_layer_decay"`   // Decay factor for layer-wise pruning

	// Semantic Anchor (Layer 19) - Attention gradient detection
	EnableSemanticAnchor bool    `mapstructure:"enable_semantic_anchor"` // Enable semantic anchor preservation
	AnchorThreshold      float64 `mapstructure:"anchor_threshold"`       // Gradient threshold for anchors
	AnchorMinPreserve    int     `mapstructure:"anchor_min_preserve"`    // Minimum anchors to preserve

	// Agent Memory (Layer 20) - Knowledge graph extraction
	EnableAgentMemory    bool   `mapstructure:"enable_agent_memory"`     // Enable agent memory extraction
	AgentMemoryMaxNodes  int    `mapstructure:"agent_memory_max_nodes"`  // Max nodes in knowledge graph
	AgentMemoryMaxEdges  int    `mapstructure:"agent_memory_max_edges"`  // Max edges in knowledge graph
	AgentMemoryExtractFn string `mapstructure:"agent_memory_extract_fn"` // Extraction function type

}

PipelineConfig controls the 20-layer compression pipeline. Supports contexts up to 2M tokens with streaming processing. Based on 120+ research papers from top institutions worldwide.

Logical sub-groups are defined as embedded structs (CompactionConfig, PerplexityFilterConfig, EntropyFilterConfig, ResearchLayersConfig). Their fields are promoted to PipelineConfig, so existing field access like cfg.CompactionThreshold still compiles.

type ResearchLayersConfig added in v0.1.1

type ResearchLayersConfig struct {
	EnableResearchPack bool `mapstructure:"enable_research_pack"` // One-toggle research bundle
}

ResearchLayersConfig groups experimental research layer flags (Layers 31-49). All disabled by default unless a preset/profile enables them. Field names match the original flat PipelineConfig fields for backward compatibility.

type TrackingConfig

type TrackingConfig struct {
	Enabled      bool   `mapstructure:"enabled"`
	DatabasePath string `mapstructure:"database_path"`
	Telemetry    bool   `mapstructure:"telemetry"`
}

TrackingConfig controls token tracking behavior.

type ValidatedFilterConfig added in v0.1.1

type ValidatedFilterConfig struct {
	SchemaVersion int                               `toml:"schema_version"`
	Filters       map[string]ValidatedFilterSection `toml:"-"`
}

ValidatedFilterConfig represents a validated TOML filter configuration.

type ValidatedFilterSection added in v0.1.1

type ValidatedFilterSection struct {
	MatchCommand    string   `toml:"match_command"`
	StripAnsi       bool     `toml:"strip_ansi"`
	StripLinesMatch []string `toml:"strip_lines_matching"`
	KeepLinesMatch  []string `toml:"keep_lines_matching"`
	MaxLines        int      `toml:"max_lines"`
	MaxTokens       int      `toml:"max_tokens"`
	Priority        int      `toml:"priority"`
	Enabled         *bool    `toml:"enabled"`
}

ValidatedFilterSection represents a single filter section within a TOML file.

type ValidationError added in v0.1.1

type ValidationError struct {
	File    string
	Section string
	Field   string
	Message string
}

ValidationError represents a single validation error in a filter config.

func ValidateAllFilters added in v0.1.1

func ValidateAllFilters(dir string) ([]ValidationError, error)

ValidateAllFilters validates all TOML filter files in a directory.

func ValidateFilterFile added in v0.1.1

func ValidateFilterFile(path string) []ValidationError

ValidateFilterFile validates a single TOML filter configuration file.

func (ValidationError) Error added in v0.1.1

func (e ValidationError) Error() string

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL