tokman

module
v0.28.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 16, 2026 License: MIT

README

🚀 TokMan

Token-aware CLI proxy & AI gateway for coding assistants

Practical 20-layer pipeline focused on real-world compression quality

Go Version License CI Security codecov Go Report Card Discord

🌐 Website · 📖 Documentation · 💬 Discord · 🐛 Issues

💡 What is TokMan?

TokMan intercepts CLI commands and applies an intelligent 20-layer compression pipeline to drastically reduce token usage for AI coding assistants with practical, high-impact defaults.

┌──────────────────────────────────────────────────────────────┐
│  Input: 10,000 tokens  →  TokMan Pipeline  →  Output: 1,500  │
│                                                                │
│  💰 Cost Savings:    $0.085 → $0.013  (85% reduction)        │
│  ⚡ Speed Boost:     Faster AI responses                       │
│  🎯 Quality:         Preserves critical information            │
└──────────────────────────────────────────────────────────────┘

✨ Key Features

🔥 Performance
  • 60-90% token reduction on typical dev workflows
  • 2-3x speedup with SIMD optimization (Go 1.26+)
  • Sub-millisecond processing for most commands
  • Zero configuration needed
🎯 Intelligence
  • 20 practical compression layers from academic research
  • Content-aware filtering (code, logs, JSON, etc.)
  • Context preservation - keeps what matters
  • Semantic understanding of command output
🔌 Extensibility
  • WASM plugin system for custom filters
  • 97+ TOML filters for popular tools
  • Scriptable via CLI or HTTP proxy
  • API access for programmatic use
🛡️ Enterprise Ready
  • Production tested on large codebases
  • Privacy first - all processing local
  • Audit logs and analytics dashboard
  • Team cost tracking and budgets

📊 Real-World Impact

Token Savings (30-minute Claude Code session)
Command Uses Before After Savings
📁 ls / tree 10× 2,000 400 80% ↓
📄 cat / read 20× 40,000 12,000 70% ↓
🔍 grep / rg 8× 16,000 3,200 80% ↓
🎯 git status 10× 3,000 600 80% ↓
📝 git diff 5× 10,000 2,500 75% ↓
📜 git log 5× 2,500 500 80% ↓
✅ git commit 8× 1,600 120 92% ↓
🧪 npm test 5× 25,000 2,500 90% ↓
🔬 pytest 4× 8,000 800 90% ↓
📦 npm ls 3× 900 180 80% ↓
📊 Total ~118,000 ~23,500 🎉 80% ↓
💰 Cost Reduction
Usage Pattern Without TokMan With TokMan Monthly Savings
🧑‍💻 Individual (30 min/day) $15 $2.25 $12.75
👥 Small Team (5 devs) $75 $11.25 $63.75
🏢 Team (20 devs) $300 $45 $255
🏭 Enterprise (100 devs) $1,500 $225 $1,275

Based on Claude Sonnet 3.5 pricing ($3/MTok input, $15/MTok output)

🚀 Quick Start

Installation
brew tap GrayCodeAI/tokman
brew install tokman
🚀 Install Script (Linux/macOS/Windows)
curl -fsSL https://raw.githubusercontent.com/GrayCodeAI/tokman/main/install.sh | sh
📦 Pre-built Binaries

Download from GitHub Releases:

  • macOS: tokman-darwin-amd64.tar.gz, tokman-darwin-arm64.tar.gz
  • Linux: tokman-linux-amd64.tar.gz, tokman-linux-arm64.tar.gz
  • Windows: tokman-windows-amd64.zip
🐹 Go Install
go install github.com/GrayCodeAI/tokman/cmd/tokman@latest
🔨 Build from Source
git clone https://github.com/GrayCodeAI/tokman.git
cd tokman
make build

# Or build for all platforms
make build-all
Setup for Your AI Tool
# Claude Code / GitHub Copilot
tokman init -g

# Cursor
tokman init --agent cursor

# Windsurf
tokman init --agent windsurf

# Cline / Roo Code
tokman init --agent cline

# Gemini CLI
tokman init -g --gemini
Verify Installation
tokman --version      # Check version
tokman doctor         # Verify setup
tokman gain           # View savings stats
Usage

Once installed, TokMan automatically intercepts commands:

# These are automatically compressed:
git status
npm ls
npm test
cat large-file.json

# Or use standalone:
tokman compress < input.txt
tokman benchmark --suite git-status
tokman dashboard  # Launch analytics dashboard

🧠 How It Works

TokMan uses a 20-layer pipeline inspired by cutting-edge research:

Input → Content Detection → Pipeline Selection → Compression → Output
         ↓                    ↓                   ↓
      [JSON, Code,        [Surface, Trim,    [20 layers:
       Logs, etc.]         Extract, Core]     Entropy, H2O,
                                               AST, Gist, etc.]
Compression Tiers
Tier Layers Reduction Use Case
🟢 Surface 3 30-50% Quick cleanup, preserve everything
🟡 Trim 12 50-70% Balanced compression
🟠 Extract 24 70-90% Aggressive, preserve essence
🔴 Core 20 90%+ Maximum practical compression
Specialized Profiles
  • 💻 Code: Syntax-aware, preserves structure (50-70%)
  • 📝 Log: Deduplication, pattern grouping (60-80%)
  • 💬 Thread: Conversation-aware, context preservation (55-75%)

📦 Supported Tools

TokMan has built-in filters for 97+ development tools:

🔧 Version Control
  • Git, GitHub CLI, GitLab CLI
  • Mercurial, SVN
🐳 Containers & Orchestration
  • Docker, Docker Compose
  • Kubernetes (kubectl), Helm
  • Podman, containerd
📦 Package Managers
  • npm, yarn, pnpm, bun
  • pip, uv, poetry
  • cargo, go mod
  • maven, gradle
🧪 Testing & Linting
  • Jest, Vitest, Playwright
  • pytest, unittest
  • cargo test, go test
  • ESLint, Ruff, golangci-lint
☁️ Cloud & Infrastructure
  • AWS CLI, gcloud, az
  • Terraform, Ansible
  • PostgreSQL, MySQL

See full list →

🆕 New Features (RTK-Compatible)

Generic Test Runner

Auto-detect and run project tests with a single command:

tokman test-runner              # Auto-detect test runner
tokman test-runner cargo test   # Run Rust tests
tokman test-runner npm test     # Run Node.js tests
tokman test-runner pytest       # Run Python tests

Supported test runners: Cargo, Go, Vitest, Jest, npm, pnpm, Pytest, RSpec, Rake Test, Playwright

Quota Estimation

Estimate subscription tier usage based on your token consumption:

tokman gain --quota pro         # Estimate 'pro' tier usage
tokman gain --quota 5x          # Estimate '5x' tier usage
tokman gain --quota 20x         # Estimate '20x' tier usage

Shows projected monthly usage, tier limits, and upgrade recommendations.

Session Adoption Tracking

View TokMan adoption across your Claude Code sessions:

tokman adoption                 # Show last 10 sessions
tokman adoption --limit 20      # Show last 20 sessions
Errors-Only Mode

Run any command and show only errors/warnings:

tokman err npm run build
tokman err cargo build
tokman err go test ./...
Smart File Summaries

Generate 2-line summaries of any file:

tokman smart main.go            # Go file summary
tokman smart package.json       # NPM package summary
tokman smart README.md          # Documentation summary

🔬 Technical Details

Architecture
┌─────────────────────────────────────────────────────────────┐
│                         TokMan CLI                           │
├─────────────────────────────────────────────────────────────┤
│  🎯 Command Router  →  📊 Content Detector  →  ⚙️  Pipeline  │
├─────────────────────────────────────────────────────────────┤
│                    20 Compression Layers                     │
│  ├─ Entropy Filtering        ├─ H2O (Heavy-Hitter Oracle)  │
│  ├─ Perplexity Pruning       ├─ Attention Sink             │
│  ├─ AST Preservation         ├─ Meta-Token Compression     │
│  ├─ Goal-Driven Selection    ├─ Semantic Chunking          │
│  └─ ... 23 more layers                                      │
├─────────────────────────────────────────────────────────────┤
│  💾 Cache Layer  →  📈 Analytics  →  🔌 Plugin System       │
└─────────────────────────────────────────────────────────────┘
Performance Optimizations
  • SIMD acceleration: AVX2, AVX-512, ARM NEON support
  • Fingerprint caching: Skip redundant processing
  • Streaming mode: Handle large inputs (>500K tokens)
  • Parallel execution: Multi-threaded pipeline
  • Early exit: Stop when budget met
Research Foundation

Built on 30+ academic papers including:

  • Selective Context (Mila 2023) - Entropy filtering
  • LLMLingua (Microsoft 2023) - Perplexity pruning
  • H2O (NeurIPS 2023) - Heavy-hitter oracle
  • StreamingLLM (2023) - Attention sink
  • AutoCompressor (Princeton/MIT 2023) - Hierarchical compression

Full research list →

📖 Documentation

🤝 Contributing

We welcome contributions! See CONTRIBUTING.md for guidelines.

Quick Contribution Guide
# 1. Fork and clone
git clone https://github.com/YOUR_USERNAME/tokman.git
cd tokman

# 2. Create a branch
git checkout -b feature/my-new-feature

# 3. Make changes and test
make test
make lint

# 4. Commit and push
git commit -m "feat: add amazing feature"
git push origin feature/my-new-feature

# 5. Open a Pull Request
Development Tools
make build          # Build binary
make test           # Run tests
make test-cover     # Generate coverage report
make lint           # Run linters
make bench          # Run benchmarks
make check          # Run all checks

📊 Project Stats

  • Language: Go 1.26+
  • Packages: 150+ internal packages
  • Tests: 144 packages with tests
  • Lines of Code: ~50,000
  • Built-in Filters: 97 TOML filters
  • Compression Layers: 20
  • Platforms: Linux, macOS, Windows (amd64/arm64)

🗺️ Roadmap

  • Core compression pipeline (20 layers)
  • TOML filter system
  • Agent integration (Claude, Cursor, Copilot, etc.)
  • Analytics dashboard
  • SIMD optimization
  • WASM plugin system
  • Cloud sync for team settings
  • Browser extension
  • IDE plugins (VS Code, JetBrains)
  • Real-time collaboration features
  • Advanced ML-based compression

🏆 How TokMan Compares

Feature TokMan RTK OMNI Snip Token-MCP
Language Go Rust Rust Go TypeScript
Compression Layers 20 practical ~15 Semantic YAML Cache
Token Reduction 60-90% 60-90% ~90% 60-90% 60-90%
Quality Metrics ✅ 6 metrics ❌ ❌ ❌ ❌
Research-Backed ✅ 120+ papers ❌ ❌ ❌ ❌
Multi-File ✅ ❌ ❌ ❌ ❌
TOML Filters ✅ 97+ ❌ ✅ ❌ ❌
Dashboard ✅ ❌ ✅ ❌ ❌
Cost Analysis ✅ ❌ ❌ ❌ ❌
Standalone Binary ✅ ✅ ✅ ✅ ❌

TokMan focuses on a practical 20-layer stack tuned for real-world usefulness and quality retention.

❓ FAQ

How does TokMan reduce tokens?

TokMan applies a layered compression pipeline that removes noise, groups similar content, truncates redundancy, and preserves critical information. Core stages are production-oriented, with additional experimental layers available.

Does it lose important information?

TokMan uses quality metrics (6-metric grading, A+ to F) to ensure compression preserves signal. The goal is to remove noise while keeping everything the AI needs.

Which AI tools does it support?

Claude Code, Cursor, GitHub Copilot, Windsurf, Cline/Roo Code, Gemini CLI, Codex, and Aider. Basically any tool that runs shell commands.

Is it fast enough for real-time use?

Yes. Most commands complete in <20ms of overhead. SIMD optimizations planned for Go 1.26+ will reduce this further.

Can I add custom filters?

Yes! Create .toml filter files in ~/.config/tokman/filters/. See the filter writing guide for details.

Is my data safe?

TokMan processes everything locally. No data is sent externally. Telemetry is opt-in and never collects file contents. See SECURITY.md.

How does it compare to RTK?

RTK is a Rust-based competitor with ~15 filtering strategies. TokMan has a practical 20-layer stack, quality metrics, multi-file intelligence, and cost analysis that RTK lacks.

Can I use it in CI/CD?

Yes! See the deployment guide for GitHub Actions, GitLab CI, and Docker integration.

🔧 Troubleshooting

TokMan not found after installation 
# Check if in PATH
which tokman

# Add Go bin to PATH
export PATH="$HOME/go/bin:$PATH"
# Add to ~/.bashrc or ~/.zshrc for persistence
Hooks not intercepting commands 
# Reinstall hooks
tokman init --uninstall
tokman init -g

# Verify with doctor
tokman doctor
Database errors 
# Reset database
rm ~/.local/share/tokman/tokman.db
tokman status  # Recreates automatically
High memory usage on large files 
# Use streaming mode (auto-enabled for >500K tokens)
# Or set a budget to limit output
tokman --budget 2000 cat large_file.txt

For more help, see the full troubleshooting guide or open an issue.

📄 License

TokMan is released under the MIT License.

🙏 Acknowledgments

Built with research from:

  • Microsoft Research (LLMLingua, LongLLMLingua)
  • Stanford University (Gist Compression)
  • MIT CSAIL (AutoCompressor)
  • Princeton University (AutoCompressor)
  • UC Berkeley (MemGPT, H2O)
  • Tsinghua University (EHPC)
  • Mila (Selective Context)
  • NUS (LongCodeZip)
  • Shanghai Jiao Tong University (SWE-Pruner)
  • LinkedIn (ProCut)
  • And 20+ other institutions

See CITATION.cff for academic citation information.

Special thanks to the open-source community and all contributors. See AUTHORS.md.

💬 Community & Support

⭐ Star us on GitHub if TokMan helps you save tokens!

Made with ❤️ by the TokMan team

Directories

Path Synopsis
cmd
eval-adaptive command
tokman command
internal
audit
breaker
cache
Package cache provides persistent query caching for TokMan.
 Package cache provides persistent query caching for TokMan.
commands
commands/build
Package build provides build tool commands (golangci, next, prisma, tsc).
 Package build provides build tool commands (golangci, next, prisma, tsc).
commands/cloud
commands/compression
commands/configcmd
Package configcmd provides configuration management commands.
 Package configcmd provides configuration management commands.
commands/container
Package container provides container and orchestration commands (docker, kubectl, psql).
 Package container provides container and orchestration commands (docker, kubectl, psql).
commands/core
Package core provides core CLI commands (doctor, completion, alias, enable, status, plugin, mcp, marketplace, trust).
 Package core provides core CLI commands (doctor, completion, alias, enable, status, plugin, mcp, marketplace, trust).
commands/filtercmd
Package filtercmd provides filter and pipeline management commands.
 Package filtercmd provides filter and pipeline management commands.
commands/hooks
Package hooks provides hook management commands.
 Package hooks provides hook management commands.
commands/infra
commands/lang
Package lang provides language runtime commands (go, dotnet).
 Package lang provides language runtime commands (go, dotnet).
commands/lang/elixir
commands/lang/haskell
commands/lang/rust
commands/lang/swift
commands/linter
Package linter provides linter and formatter commands.
 Package linter provides linter and formatter commands.
commands/output
Package output provides output processing and formatting commands.
 Package output provides output processing and formatting commands.
commands/pattern
commands/pkgmgr
Package pkgmgr provides package manager commands (cargo, npm, npx, pip, pnpm).
 Package pkgmgr provides package manager commands (cargo, npm, npx, pip, pnpm).
commands/registry
Package registry provides command registration for sub-packages.
 Package registry provides command registration for sub-packages.
commands/session
commands/shared
Package shared provides shared state and configuration for TokMan CLI commands.
 Package shared provides shared state and configuration for TokMan CLI commands.
commands/system
Package system provides system utility commands (ls, grep, find, tree, wc, watch, read, search, log, deps, clean, ccusage, discover, tee, profile).
 Package system provides system utility commands (ls, grep, find, tree, wc, watch, read, search, log, deps, clean, ccusage, discover, tee, profile).
commands/test
Package test provides test runner commands (jest, pytest, vitest, playwright).
 Package test provides test runner commands (jest, pytest, vitest, playwright).
commands/vcs
Package vcs provides version control system commands (git, gh, gt).
 Package vcs provides version control system commands (git, gh, gt).
commands/web
Package web provides HTTP-related commands for TokMan.
 Package web provides HTTP-related commands for TokMan.
compression
config
Package config provides configuration management for TokMan.
 Package config provides configuration management for TokMan.
contextread
Package contextread provides context reading capabilities (stub implementation).
 Package contextread provides context reading capabilities (stub implementation).
core
Package core provides core interfaces and utilities for TokMan.
 Package core provides core interfaces and utilities for TokMan.
discover
Package discover provides command discovery and auto-rewrite functionality.
 Package discover provides command discovery and auto-rewrite functionality.
filter
Package filter provides LRU caching using the unified cache package.
 Package filter provides LRU caching using the unified cache package.
graph
Package graph provides graph analysis functionality (stub implementation).
 Package graph provides graph analysis functionality (stub implementation).
health
Package health provides health check functionality for TokMan.
 Package health provides health check functionality for TokMan.
integrity
Package integrity provides hook integrity verification via SHA-256.
 Package integrity provides hook integrity verification via SHA-256.
llm
Package llm provides LLM-based summarization capabilities (stub implementation).
 Package llm provides LLM-based summarization capabilities (stub implementation).
logging
mcp
Package mcp implements the Model Context Protocol (MCP) server for TokMan.
 Package mcp implements the Model Context Protocol (MCP) server for TokMan.
memory
Package memory provides memory management functionality (stub implementation).
 Package memory provides memory management functionality (stub implementation).
metrics
Package metrics provides comprehensive metrics for TokMan.
 Package metrics provides comprehensive metrics for TokMan.
pattern
ratelimit
retry
security
Package security provides content scanning and PII redaction for TokMan.
 Package security provides content scanning and PII redaction for TokMan.
session
simd
state
tee
Package tee provides raw output recovery for command failures.
 Package tee provides raw output recovery for command failures.
telemetry
toml
tracking
Package tracking provides cost estimation and reporting.
 Package tracking provides cost estimation and reporting.
ttlcache
utils
validation
test
fixtures
integration/helpers
Package helpers provides utilities for integration tests
 Package helpers provides utilities for integration tests

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.