Documentation
¶
Overview ¶
Package config provides configuration management for tok.
The config package handles loading, validating, and saving configuration from TOML files, environment variables, and command-line flags.
Configuration Loading ¶
Use Load() to read configuration from files and environment:
cfg, err := config.Load(cfgFile)
if err != nil { ... }
Defaults ¶
The Defaults() function returns a configuration with sensible defaults:
cfg := config.Defaults()
Validation ¶
All configurations are validated via the Validate() method to ensure threshold values, limits, and cross-field constraints are satisfied.
Index ¶
- Constants
- func ConfigDir() string
- func ConfigPath() string
- func DataPath() string
- func DatabasePath() string
- func FiltersDir() string
- func FiltersPath() string
- func HooksPath() string
- func LogPath() string
- func ProjectPath() string
- type AlertsConfig
- type BudgetConfig
- type CommandContext
- type CompactionConfig
- type Config
- type DashboardConfig
- type EditConfig
- type EntropyFilterConfig
- type ExportConfig
- type FilterConfig
- type HooksConfig
- type KeybindingsConfig
- type PerplexityFilterConfig
- type PipelineConfig
- type ResearchLayersConfig
- type TrackingConfig
- type ValidatedFilterConfig
- type ValidatedFilterSection
- type ValidationError
Constants ¶
const ( DefaultMaxContextTokens = 2000000 // 2M tokens max DefaultChunkSize = 100000 // 100K tokens per chunk DefaultStreamThreshold = 500000 // Stream if > 500K tokens DefaultCacheMaxSize = 1000 // Max cache entries )
Default pipeline limits.
Variables ¶
This section is empty.
Functions ¶
func ConfigPath ¶
func ConfigPath() string
ConfigPath returns the path to the configuration file. Follows XDG Base Directory Specification on Unix. Uses %APPDATA% on Windows. Falls back to temp directory if no home directory is available.
func DataPath ¶
func DataPath() string
DataPath returns the path to the data directory. Follows XDG Base Directory Specification on Unix. Uses %LOCALAPPDATA% on Windows. Falls back to temp directory if no home directory is available.
func DatabasePath ¶
func DatabasePath() string
DatabasePath returns the path to the SQLite database.
func FiltersDir ¶
func FiltersDir() string
FiltersDir returns the path to the user filters directory.
func FiltersPath ¶
func FiltersPath() string
FiltersPath returns the path to the filters directory (alias for FiltersDir).
func ProjectPath ¶
func ProjectPath() string
ProjectPath returns the canonical path for the current working directory. Resolves symlinks for accurate project matching.
Types ¶
type AlertsConfig ¶
type AlertsConfig struct {
Enabled bool `mapstructure:"enabled"`
DailyTokenLimit int64 `mapstructure:"daily_token_limit"`
WeeklyTokenLimit int64 `mapstructure:"weekly_token_limit"`
UsageSpikeThreshold float64 `mapstructure:"usage_spike_threshold"`
}
AlertsConfig controls alert thresholds.
type BudgetConfig ¶
type BudgetConfig struct {
DailyTokens int `mapstructure:"daily_tokens"` // Daily token budget (0 = unlimited)
WarningThreshold int `mapstructure:"warning_threshold"` // Percent for warning (default: 80)
}
BudgetConfig controls token budget tracking and alerts.
type CommandContext ¶
type CommandContext struct {
Command string `mapstructure:"command"` // "git", "npm", "cargo", etc.
Subcommand string `mapstructure:"subcommand"` // "status", "test", "build"
ExitCode int `mapstructure:"exit_code"` // Non-zero = likely has errors
Intent string `mapstructure:"intent"` // "debug", "review", "deploy", "search"
IsTest bool `mapstructure:"is_test"` // Test output detection
IsBuild bool `mapstructure:"is_build"` // Build output detection
IsError bool `mapstructure:"is_error"` // Error output detection
}
CommandContext provides metadata about the command being executed. Used for intelligent filtering decisions.
type CompactionConfig ¶ added in v0.1.1
type CompactionConfig struct {
EnableCompaction bool `mapstructure:"enable_compaction"` // Enable LLM-based compaction
CompactionThreshold int `mapstructure:"compaction_threshold"` // Minimum tokens to trigger
CompactionPreserveTurns int `mapstructure:"compaction_preserve_turns"` // Recent turns to keep verbatim
CompactionMaxTokens int `mapstructure:"compaction_max_tokens"` // Max summary tokens
CompactionStateSnapshot bool `mapstructure:"compaction_state_snapshot"` // Use state snapshot format
CompactionAutoDetect bool `mapstructure:"compaction_auto_detect"` // Auto-detect conversation content
LLMProvider string `mapstructure:"llm_provider"` // ollama, lmstudio, openai
LLMModel string `mapstructure:"llm_model"` // Model name
LLMBaseURL string `mapstructure:"llm_base_url"` // API endpoint
}
CompactionConfig groups LLM compaction (Layer 11) settings. Controls semantic compression of conversation history. Field names match the original flat PipelineConfig fields so that embedding promotes them transparently (backward compatible).
type Config ¶
type Config struct {
Tracking TrackingConfig `mapstructure:"tracking"`
Filter FilterConfig `mapstructure:"filter"`
Pipeline PipelineConfig `mapstructure:"pipeline"`
Hooks HooksConfig `mapstructure:"hooks"`
Dashboard DashboardConfig `mapstructure:"dashboard"`
Alerts AlertsConfig `mapstructure:"alerts"`
Budget BudgetConfig `mapstructure:"budget"`
Export ExportConfig `mapstructure:"export"`
Edit EditConfig `mapstructure:"edit"` // Edit batching configuration
Keybindings KeybindingsConfig `mapstructure:"keybindings"` // TUI keybindings overrides
}
Config represents the main configuration structure.
func LoadFromFile ¶
LoadFromFile reads a TOML configuration file directly.
func (*Config) GetDatabasePath ¶
GetDatabasePath returns the effective database path.
type DashboardConfig ¶
type DashboardConfig struct {
Port int `mapstructure:"port"`
Bind string `mapstructure:"bind"`
UpdateInterval int `mapstructure:"update_interval"`
Theme string `mapstructure:"theme"`
EnableExport bool `mapstructure:"enable_export"`
}
DashboardConfig controls dashboard behavior.
type EditConfig ¶
type EditConfig struct {
BatchEnabled bool `mapstructure:"batch_enabled"` // Enable batching of edit operations
BatchSize int `mapstructure:"batch_size"` // Max edits per batch (default: 10)
BatchTimeoutMs int `mapstructure:"batch_timeout_ms"` // Timeout for batch collection (default: 500ms)
DryRun bool `mapstructure:"edit_dry_run"` // Show what would be changed without writing
CreateBackups bool `mapstructure:"create_backups"` // Create .bak files before editing
AtomicWrites bool `mapstructure:"atomic_writes"` // Use atomic file replacement
MaxFileSize int `mapstructure:"max_edit_file_size"` // Maximum file size for editing (bytes, 0=unlimited)
AllowedPatterns []string `mapstructure:"allowed_edit_patterns"` // Glob patterns of files allowed to edit
DeniedPatterns []string `mapstructure:"denied_edit_patterns"` // Glob patterns of files denied
}
EditConfig controls edit batching behavior.
type EntropyFilterConfig ¶ added in v0.1.1
type EntropyFilterConfig struct {
EntropySamplingLarge int `mapstructure:"entropy_sampling_large"` // Sample threshold for large inputs (default: 100000)
EntropySamplingHuge int `mapstructure:"entropy_sampling_huge"` // Sample threshold for huge inputs (default: 500000)
EntropySamplingRate int `mapstructure:"entropy_sampling_rate"` // Sample rate for huge inputs (default: 10)
EntropyMinWordLength int `mapstructure:"entropy_min_word_length"` // Minimum word length (default: 2)
}
EntropyFilterConfig groups Layer 1 (Entropy Filtering) detailed settings. Field names match the original flat PipelineConfig fields for backward compatibility.
type ExportConfig ¶
type ExportConfig struct {
DefaultFormat string `mapstructure:"default_format"`
IncludeTimestamp bool `mapstructure:"include_timestamps"`
MaxRecords int `mapstructure:"max_records"`
}
ExportConfig controls export behavior.
type FilterConfig ¶
type FilterConfig struct {
NoiseDirs []string `mapstructure:"noise_dirs"`
IgnoreFiles []string `mapstructure:"ignore_files"` // File patterns to ignore (e.g., "*.lock", "*.min.js")
Mode string `mapstructure:"mode"` // "minimal" or "aggressive"
MaxWidth int `mapstructure:"max_width"` // Max display width (0 = auto)
}
FilterConfig controls output filtering behavior.
type HooksConfig ¶
type HooksConfig struct {
ExcludedCommands []string `mapstructure:"excluded_commands"`
AuditDir string `mapstructure:"audit_dir"` // Directory for hook audit logs
TeeDir string `mapstructure:"tee_dir"` // Directory for failure tee logs
TeeEnabled bool `mapstructure:"tee_enabled"` // Enable tee file writing (default: true)
}
HooksConfig controls shell hook behavior.
type KeybindingsConfig ¶
type KeybindingsConfig struct {
Quit string `mapstructure:"quit"` // default: q, ctrl+c
NextSection string `mapstructure:"next_section"` // default: tab, right, l
PrevSection string `mapstructure:"prev_section"` // default: shift+tab, left, h
HistoryBack string `mapstructure:"history_back"` // default: H
HistoryForward string `mapstructure:"history_forward"` // default: L
Refresh string `mapstructure:"refresh"` // default: r
Up string `mapstructure:"up"` // default: up, k
Down string `mapstructure:"down"` // default: down, j
PageUp string `mapstructure:"page_up"` // default: pgup, ctrl+b
PageDown string `mapstructure:"page_down"` // default: pgdn, ctrl+f
Top string `mapstructure:"top"` // default: g, home
Bottom string `mapstructure:"bottom"` // default: G, end
Enter string `mapstructure:"enter"` // default: enter
Back string `mapstructure:"back"` // default: backspace
Yank string `mapstructure:"yank"` // default: y
Export string `mapstructure:"export"` // default: e
Palette string `mapstructure:"palette"` // default: :
Search string `mapstructure:"search"` // default: /
Help string `mapstructure:"help"` // default: ?
}
KeybindingsConfig allows users to remap TUI keys. Keys use bubbletea's key syntax (e.g., "ctrl+c", "tab", "H").
type PerplexityFilterConfig ¶ added in v0.1.1
type PerplexityFilterConfig struct {
PerplexityTargetRatio float64 `mapstructure:"perplexity_target_ratio"` // Target compression ratio (default: 0.3)
PerplexityIterationSteps int `mapstructure:"perplexity_iteration_steps"` // Number of pruning iterations (default: 2)
PerplexityPruneRatio float64 `mapstructure:"perplexity_prune_ratio"` // Prune ratio per iteration (default: 0.7)
PerplexityConvergenceThreshold float64 `mapstructure:"perplexity_convergence_threshold"` // Early exit threshold (default: 0.05)
PerplexityContextWindow int `mapstructure:"perplexity_context_window"` // Context window size (default: 10)
}
PerplexityFilterConfig groups Layer 2 (Perplexity Pruning) detailed settings. Field names match the original flat PipelineConfig fields for backward compatibility.
type PipelineConfig ¶
type PipelineConfig struct {
// Embedded sub-configs for logical grouping.
// Fields are promoted, preserving backward compatibility.
CompactionConfig `mapstructure:",squash"`
PerplexityFilterConfig `mapstructure:",squash"`
EntropyFilterConfig `mapstructure:",squash"`
ResearchLayersConfig `mapstructure:",squash"`
// Context limits
MaxContextTokens int `mapstructure:"max_context_tokens"` // Max input context (default: 2M)
ChunkSize int `mapstructure:"chunk_size"` // Processing chunk size for large inputs
// Agent preset selection
Preset string `mapstructure:"preset"` // fast, balanced, full, T90, ultra, research
// Layer enable/disable
EnableEntropy bool `mapstructure:"enable_entropy"`
EnablePerplexity bool `mapstructure:"enable_perplexity"`
EnableGoalDriven bool `mapstructure:"enable_goal_driven"`
EnableAST bool `mapstructure:"enable_ast"`
EnableContrastive bool `mapstructure:"enable_contrastive"`
EnableNgram bool `mapstructure:"enable_ngram"`
EnableEvaluator bool `mapstructure:"enable_evaluator"`
EnableGist bool `mapstructure:"enable_gist"`
EnableHierarchical bool `mapstructure:"enable_hierarchical"`
EnableBudget bool `mapstructure:"enable_budget"`
// Layer thresholds (tunable)
EntropyThreshold float64 `mapstructure:"entropy_threshold"` // Layer 1: 0.0-1.0 (default: 0.3)
PerplexityThreshold float64 `mapstructure:"perplexity_threshold"` // Layer 2: 0.0-1.0 (default: 0.5)
GoalDrivenThreshold float64 `mapstructure:"goal_driven_threshold"` // Layer 3: 0.0-1.0 (default: 0.4)
ASTPreserveThreshold float64 `mapstructure:"ast_preserve_threshold"` // Layer 4: 0.0-1.0 (default: 0.6)
ContrastiveThreshold float64 `mapstructure:"contrastive_threshold"` // Layer 5: 0.0-1.0 (default: 0.5)
NgramMinOccurrences int `mapstructure:"ngram_min_occurrences"` // Layer 6: min repeats (default: 3)
EvaluatorThreshold float64 `mapstructure:"evaluator_threshold"` // Layer 7: 0.0-1.0 (default: 0.4)
GistMinChunkSize int `mapstructure:"gist_min_chunk_size"` // Layer 8: chars (default: 100)
HierarchicalMaxLevels int `mapstructure:"hierarchical_max_levels"` // Layer 9: depth (default: 3)
HierarchicalRatio float64 `mapstructure:"hierarchical_ratio"` // Layer 9: 0.0-1.0 (default: 0.3)
// Budget enforcement
DefaultBudget int `mapstructure:"default_budget"` // Default token budget (0 = unlimited)
HardBudgetLimit bool `mapstructure:"hard_budget_limit"` // Strict enforcement
BudgetOverflowFile string `mapstructure:"budget_overflow_file"` // File to save overflow content
// Resilience
TeeOnFailure bool `mapstructure:"tee_on_failure"` // Save raw output on failure
TeeDir string `mapstructure:"tee_dir"` // Directory for tee files
FailSafeMode bool `mapstructure:"failsafe_mode"` // Return original on corruption
ValidateOutput bool `mapstructure:"validate_output"` // Check output validity
ShortCircuitBudget bool `mapstructure:"short_circuit_budget"` // Skip layers if budget met
// Performance
ParallelLayers bool `mapstructure:"parallel_layers"` // Run independent layers in parallel
CacheEnabled bool `mapstructure:"cache_enabled"` // Cache compression results
CacheMaxSize int `mapstructure:"cache_max_size"` // Max cache entries
StreamThreshold int `mapstructure:"stream_threshold"` // Stream if input > N tokens
// Adaptive routing and extractive prefilter
EnablePolicyRouter bool `mapstructure:"enable_policy_router"` // Infer intent from output
EnableExtractiveFilter bool `mapstructure:"enable_extractive_prefilter"` // Enable extractive prefilter
ExtractiveMaxLines int `mapstructure:"extractive_max_lines"` // Trigger threshold
ExtractiveHeadLines int `mapstructure:"extractive_head_lines"` // Head lines to preserve
ExtractiveTailLines int `mapstructure:"extractive_tail_lines"` // Tail lines to preserve
ExtractiveSignalLines int `mapstructure:"extractive_signal_lines"` // Signal lines to preserve
EnableQualityGuardrail bool `mapstructure:"enable_quality_guardrail"` // Auto-fallback on quality risk
// Attribution Filter (Layer 12) - ProCut-style pruning
EnableAttribution bool `mapstructure:"enable_attribution"` // Enable attribution filtering
AttributionThreshold float64 `mapstructure:"attribution_threshold"` // Importance threshold (0.0-1.0)
AttributionPositional bool `mapstructure:"attribution_positional"` // Use positional bias
AttributionFrequency bool `mapstructure:"attribution_frequency"` // Use frequency bias
AttributionSemantic bool `mapstructure:"attribution_semantic"` // Use semantic preservation
// H2O Filter (Layer 13) - Heavy-Hitter Oracle
EnableH2O bool `mapstructure:"enable_h2o"` // Enable H2O compression
H2OSinkSize int `mapstructure:"h2o_sink_size"` // Attention sink tokens to preserve
H2ORecentSize int `mapstructure:"h2o_recent_size"` // Recent tokens to preserve
H2OHeavyHitterSize int `mapstructure:"h2o_heavy_hitter_size"` // Heavy hitter tokens to preserve
// Attention Sink Filter (Layer 14) - StreamingLLM-style
EnableAttentionSink bool `mapstructure:"enable_attention_sink"` // Enable attention sink filtering
AttentionSinkCount int `mapstructure:"attention_sink_count"` // Initial tokens to preserve as sinks
AttentionRecentCount int `mapstructure:"attention_recent_count"` // Recent lines to preserve
// Meta-Token Compression (Layer 15) - LZ77-style lossless compression
EnableMetaToken bool `mapstructure:"enable_meta_token"` // Enable meta-token compression
MetaTokenWindow int `mapstructure:"meta_token_window"` // Sliding window size for token matching
MetaTokenMinMatch int `mapstructure:"meta_token_min_match"` // Minimum match length
// Semantic Chunking (Layer 16) - Dynamic boundary detection
EnableSemanticChunk bool `mapstructure:"enable_semantic_chunk"` // Enable semantic chunking
SemanticThreshold float64 `mapstructure:"semantic_threshold"` // Semantic shift threshold (0.0-1.0)
ChunkMinSize int `mapstructure:"chunk_min_size"` // Minimum chunk size in tokens
ChunkMaxSize int `mapstructure:"chunk_max_size"` // Maximum chunk size in tokens
// Sketch Store (Layer 17) - Reversible compression with KVReviver
EnableSketchStore bool `mapstructure:"enable_sketch_store"` // Enable sketch-based storage
SketchMemoryRatio int `mapstructure:"sketch_memory_ratio"` // Memory reduction ratio (default: 90%)
SketchOnDemand bool `mapstructure:"sketch_on_demand"` // Reconstruct on-demand when needed
// Lazy Pruner (Layer 18) - Budget-aware dynamic pruning
EnableLazyPruner bool `mapstructure:"enable_lazy_pruner"` // Enable lazy pruning
LazyBudgetRatio float64 `mapstructure:"lazy_budget_ratio"` // Ratio of budget to use for lazy pruning
LazyLayerDecay float64 `mapstructure:"lazy_layer_decay"` // Decay factor for layer-wise pruning
// Semantic Anchor (Layer 19) - Attention gradient detection
EnableSemanticAnchor bool `mapstructure:"enable_semantic_anchor"` // Enable semantic anchor preservation
AnchorThreshold float64 `mapstructure:"anchor_threshold"` // Gradient threshold for anchors
AnchorMinPreserve int `mapstructure:"anchor_min_preserve"` // Minimum anchors to preserve
// Agent Memory (Layer 20) - Knowledge graph extraction
EnableAgentMemory bool `mapstructure:"enable_agent_memory"` // Enable agent memory extraction
AgentMemoryMaxNodes int `mapstructure:"agent_memory_max_nodes"` // Max nodes in knowledge graph
AgentMemoryMaxEdges int `mapstructure:"agent_memory_max_edges"` // Max edges in knowledge graph
AgentMemoryExtractFn string `mapstructure:"agent_memory_extract_fn"` // Extraction function type
}
PipelineConfig controls the 20-layer compression pipeline. Supports contexts up to 2M tokens with streaming processing. Based on 120+ research papers from top institutions worldwide.
Logical sub-groups are defined as embedded structs (CompactionConfig, PerplexityFilterConfig, EntropyFilterConfig, ResearchLayersConfig). Their fields are promoted to PipelineConfig, so existing field access like cfg.CompactionThreshold still compiles.
type ResearchLayersConfig ¶ added in v0.1.1
type ResearchLayersConfig struct {
EnableResearchPack bool `mapstructure:"enable_research_pack"` // One-toggle research bundle
}
ResearchLayersConfig groups experimental research layer flags (Layers 31-49). All disabled by default unless a preset/profile enables them. Field names match the original flat PipelineConfig fields for backward compatibility.
type TrackingConfig ¶
type TrackingConfig struct {
Enabled bool `mapstructure:"enabled"`
DatabasePath string `mapstructure:"database_path"`
Telemetry bool `mapstructure:"telemetry"`
}
TrackingConfig controls token tracking behavior.
type ValidatedFilterConfig ¶ added in v0.1.1
type ValidatedFilterConfig struct {
SchemaVersion int `toml:"schema_version"`
Filters map[string]ValidatedFilterSection `toml:"-"`
}
ValidatedFilterConfig represents a validated TOML filter configuration.
type ValidatedFilterSection ¶ added in v0.1.1
type ValidatedFilterSection struct {
MatchCommand string `toml:"match_command"`
StripAnsi bool `toml:"strip_ansi"`
StripLinesMatch []string `toml:"strip_lines_matching"`
KeepLinesMatch []string `toml:"keep_lines_matching"`
MaxLines int `toml:"max_lines"`
MaxTokens int `toml:"max_tokens"`
Priority int `toml:"priority"`
Enabled *bool `toml:"enabled"`
}
ValidatedFilterSection represents a single filter section within a TOML file.
type ValidationError ¶ added in v0.1.1
ValidationError represents a single validation error in a filter config.
func ValidateAllFilters ¶ added in v0.1.1
func ValidateAllFilters(dir string) ([]ValidationError, error)
ValidateAllFilters validates all TOML filter files in a directory.
func ValidateFilterFile ¶ added in v0.1.1
func ValidateFilterFile(path string) []ValidationError
ValidateFilterFile validates a single TOML filter configuration file.
func (ValidationError) Error ¶ added in v0.1.1
func (e ValidationError) Error() string