tokman

module
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 6, 2026 License: MIT

README

🚀 TokMan

Token-aware CLI proxy & AI gateway for coding assistants

31-layer compression pipeline achieving 60–90% token savings

Go Version License Tests Coverage Discord Build Status

🌐 Website · 📖 Documentation · 💬 Discord · 🐛 Issues

💡 What is TokMan?

TokMan intercepts CLI commands and applies an intelligent 31-layer compression pipeline to drastically reduce token usage for AI coding assistants. Built on research from 30+ academic papers, it achieves 60–90% token reduction on common development operations.

┌──────────────────────────────────────────────────────────────┐
│  Input: 10,000 tokens  →  TokMan Pipeline  →  Output: 1,500  │
│                                                                │
│  💰 Cost Savings:    $0.085 → $0.013  (85% reduction)        │
│  ⚡ Speed Boost:     Faster AI responses                       │
│  🎯 Quality:         Preserves critical information            │
└──────────────────────────────────────────────────────────────┘

✨ Key Features

🔥 Performance
  • 60-90% token reduction on typical dev workflows
  • 2-3x speedup with SIMD optimization (Go 1.26+)
  • Sub-millisecond processing for most commands
  • Zero configuration needed
🎯 Intelligence
  • 31 compression layers from academic research
  • Content-aware filtering (code, logs, JSON, etc.)
  • Context preservation - keeps what matters
  • Semantic understanding of command output
🔌 Extensibility
  • WASM plugin system for custom filters
  • 97+ TOML filters for popular tools
  • Scriptable via CLI or HTTP proxy
  • API access for programmatic use
🛡️ Enterprise Ready
  • Production tested on large codebases
  • Privacy first - all processing local
  • Audit logs and analytics dashboard
  • Team cost tracking and budgets

📊 Real-World Impact

Token Savings (30-minute Claude Code session)
Command Uses Before After Savings
📁 ls / tree 10× 2,000 400 80% ↓
📄 cat / read 20× 40,000 12,000 70% ↓
🔍 grep / rg 8× 16,000 3,200 80% ↓
🎯 git status 10× 3,000 600 80% ↓
📝 git diff 5× 10,000 2,500 75% ↓
📜 git log 5× 2,500 500 80% ↓
✅ git commit 8× 1,600 120 92% ↓
🧪 npm test 5× 25,000 2,500 90% ↓
🔬 pytest 4× 8,000 800 90% ↓
🐳 docker ps 3× 900 180 80% ↓
📊 Total ~118,000 ~23,500 🎉 80% ↓
💰 Cost Reduction
Usage Pattern Without TokMan With TokMan Monthly Savings
🧑‍💻 Individual (30 min/day) $15 $2.25 $12.75
👥 Small Team (5 devs) $75 $11.25 $63.75
🏢 Team (20 devs) $300 $45 $255
🏭 Enterprise (100 devs) $1,500 $225 $1,275

Based on Claude Sonnet 3.5 pricing ($3/MTok input, $15/MTok output)

🚀 Quick Start

Installation
# Using Go (recommended)
go install github.com/GrayCodeAI/tokman/cmd/tokman@latest

# From source
git clone https://github.com/GrayCodeAI/tokman.git
cd tokman
make build

# SIMD-optimized (2-3x faster)
make build-simd
Setup for Your AI Tool
# Claude Code / GitHub Copilot
tokman init -g

# Cursor
tokman init --agent cursor

# Windsurf
tokman init --agent windsurf

# Cline / Roo Code
tokman init --agent cline

# Gemini CLI
tokman init -g --gemini
Verify Installation
tokman --version      # Check version
tokman doctor         # Verify setup
tokman gain           # View savings stats
Usage

Once installed, TokMan automatically intercepts commands:

# These are automatically compressed:
git status
docker ps
npm test
cat large-file.json

# Or use standalone:
tokman compress < input.txt
tokman benchmark --suite git-status
tokman tui  # Launch analytics dashboard

🧠 How It Works

TokMan uses a 31-layer compression pipeline inspired by cutting-edge research:

Input → Content Detection → Pipeline Selection → Compression → Output
         ↓                    ↓                   ↓
      [JSON, Code,        [Surface, Trim,    [31 layers:
       Logs, etc.]         Extract, Core]     Entropy, H2O,
                                               AST, Gist, etc.]
Compression Tiers
Tier Layers Reduction Use Case
🟢 Surface 3 30-50% Quick cleanup, preserve everything
🟡 Trim 12 50-70% Balanced compression
🟠 Extract 24 70-90% Aggressive, preserve essence
🔴 Core 31 90%+ Maximum compression
Specialized Profiles
  • 💻 Code: Syntax-aware, preserves structure (50-70%)
  • 📝 Log: Deduplication, pattern grouping (60-80%)
  • 💬 Thread: Conversation-aware, context preservation (55-75%)

📦 Supported Tools

TokMan has built-in filters for 97+ development tools:

🔧 Version Control
  • Git, GitHub CLI, GitLab CLI
  • Mercurial, SVN
🐳 Containers & Orchestration
  • Docker, Docker Compose
  • Kubernetes (kubectl), Helm
  • Podman, containerd
📦 Package Managers
  • npm, yarn, pnpm, bun
  • pip, uv, poetry
  • cargo, go mod
  • maven, gradle
🧪 Testing & Linting
  • Jest, Vitest, Playwright
  • pytest, unittest
  • cargo test, go test
  • ESLint, Ruff, golangci-lint
☁️ Cloud & Infrastructure
  • AWS CLI, gcloud, az
  • Terraform, Ansible
  • PostgreSQL, MySQL

See full list →

🔬 Technical Details

Architecture
┌─────────────────────────────────────────────────────────────┐
│                         TokMan CLI                           │
├─────────────────────────────────────────────────────────────┤
│  🎯 Command Router  →  📊 Content Detector  →  ⚙️  Pipeline  │
├─────────────────────────────────────────────────────────────┤
│                    31 Compression Layers                     │
│  ├─ Entropy Filtering        ├─ H2O (Heavy-Hitter Oracle)  │
│  ├─ Perplexity Pruning       ├─ Attention Sink             │
│  ├─ AST Preservation         ├─ Meta-Token Compression     │
│  ├─ Goal-Driven Selection    ├─ Semantic Chunking          │
│  └─ ... 23 more layers                                      │
├─────────────────────────────────────────────────────────────┤
│  💾 Cache Layer  →  📈 Analytics  →  🔌 Plugin System       │
└─────────────────────────────────────────────────────────────┘
Performance Optimizations
  • SIMD acceleration: AVX2, AVX-512, ARM NEON support
  • Fingerprint caching: Skip redundant processing
  • Streaming mode: Handle large inputs (>500K tokens)
  • Parallel execution: Multi-threaded pipeline
  • Early exit: Stop when budget met
Research Foundation

Built on 30+ academic papers including:

  • Selective Context (Mila 2023) - Entropy filtering
  • LLMLingua (Microsoft 2023) - Perplexity pruning
  • H2O (NeurIPS 2023) - Heavy-hitter oracle
  • StreamingLLM (2023) - Attention sink
  • AutoCompressor (Princeton/MIT 2023) - Hierarchical compression

Full research list →

📖 Documentation

🤝 Contributing

We welcome contributions! See CONTRIBUTING.md for guidelines.

Quick Contribution Guide
# 1. Fork and clone
git clone https://github.com/YOUR_USERNAME/tokman.git
cd tokman

# 2. Create a branch
git checkout -b feature/my-new-feature

# 3. Make changes and test
make test
make lint

# 4. Commit and push
git commit -m "feat: add amazing feature"
git push origin feature/my-new-feature

# 5. Open a Pull Request
Development Tools
make build          # Build binary
make test           # Run tests
make test-cover     # Generate coverage report
make lint           # Run linters
make bench          # Run benchmarks
make check          # Run all checks

📊 Project Stats

  • Language: Go 1.26+
  • Packages: 150+ internal packages
  • Tests: 144 packages with tests
  • Lines of Code: ~50,000
  • Built-in Filters: 97 TOML filters
  • Compression Layers: 31 layers
  • Platforms: Linux, macOS, Windows (amd64/arm64)

🗺️ Roadmap

  • Core compression pipeline (31 layers)
  • TOML filter system
  • Agent integration (Claude, Cursor, Copilot, etc.)
  • Analytics dashboard
  • SIMD optimization
  • WASM plugin system
  • Cloud sync for team settings
  • Browser extension
  • IDE plugins (VS Code, JetBrains)
  • Real-time collaboration features
  • Advanced ML-based compression

📄 License

TokMan is released under the MIT License.

🙏 Acknowledgments

Built with research from:

  • Microsoft Research
  • Stanford University
  • MIT CSAIL
  • Princeton University
  • UC Berkeley
  • And 25+ other institutions

Special thanks to the open-source community and all contributors.

💬 Community & Support

⭐ Star us on GitHub if TokMan helps you save tokens!

Made with ❤️ by the TokMan team

Directories

Path Synopsis
cmd
tokman command
tokman-server command
Package main provides the TokMan server entry point.
 Package main provides the TokMan server entry point.
tokman/cmd
Package cmd provides BFCL validation command.
 Package cmd provides BFCL validation command.
internal
abtest
Package abtest provides A/B testing capabilities for TokMan
 Package abtest provides A/B testing capabilities for TokMan
agent
agents
Package agents provides integration with various AI coding agents.
 Package agents provides integration with various AI coding agents.
anomaly
Package anomaly provides cost anomaly detection
 Package anomaly provides cost anomaly detection
api
Package api provides cost intelligence API endpoints
 Package api provides cost intelligence API endpoints
ast
Package ast provides language-specific AST parsing for context-aware compression.
 Package ast provides language-specific AST parsing for context-aware compression.
asteng
audit
Package audit provides audit logging capabilities
 Package audit provides audit logging capabilities
autotune
Package autotune provides auto-tuning capabilities
 Package autotune provides auto-tuning capabilities
background
benchmarking
Package benchmarking provides chart generation capabilities
 Package benchmarking provides chart generation capabilities
budgetalerts
Package budgetalerts provides budget alerting capabilities for TokMan
 Package budgetalerts provides budget alerting capabilities for TokMan
cache
canary
Package canary provides canary deployment capabilities for TokMan
 Package canary provides canary deployment capabilities for TokMan
ccp
ccusage
Package ccusage provides integration with the ccusage CLI for fetching Claude Code API usage metrics.
 Package ccusage provides integration with the ccusage CLI for fetching Claude Code API usage metrics.
chaos
Package chaos provides chaos engineering capabilities for TokMan
 Package chaos provides chaos engineering capabilities for TokMan
circuitbreaker
Package circuitbreaker implements the circuit breaker pattern to prevent cascading failures when external services (LLM, gRPC, SQLite) degrade.
 Package circuitbreaker implements the circuit breaker pattern to prevent cascading failures when external services (LLM, gRPC, SQLite) degrade.
cli
Package cli provides CLI enhancement utilities
 Package cli provides CLI enhancement utilities
commandhistory
Package commandhistory provides command history management
 Package commandhistory provides command history management
commandmatch
commands
commands/agents
Package agents provides AI agent management commands.
 Package agents provides AI agent management commands.
commands/analysis
Package analysis provides analysis, metrics, and reporting commands.
 Package analysis provides analysis, metrics, and reporting commands.
commands/build
Package build provides build tool commands (golangci, next, prisma, tsc).
 Package build provides build tool commands (golangci, next, prisma, tsc).
commands/cloud
Package cloud provides cloud infrastructure commands (aws).
 Package cloud provides cloud infrastructure commands (aws).
commands/configcmd
Package configcmd provides configuration management commands.
 Package configcmd provides configuration management commands.
commands/container
Package container provides container and orchestration commands (docker, kubectl, psql).
 Package container provides container and orchestration commands (docker, kubectl, psql).
commands/core
Package core provides core CLI commands (doctor, completion, alias, enable, status, plugin, mcp, marketplace, trust).
 Package core provides core CLI commands (doctor, completion, alias, enable, status, plugin, mcp, marketplace, trust).
commands/enterprise
commands/filtercmd
Package filtercmd provides filter and pipeline management commands.
 Package filtercmd provides filter and pipeline management commands.
commands/hooks
Package hooks provides hook management commands.
 Package hooks provides hook management commands.
commands/init
Package initpkg provides initialization commands for AI agents and editors.
 Package initpkg provides initialization commands for AI agents and editors.
commands/lang
Package lang provides language runtime commands (go, dotnet).
 Package lang provides language runtime commands (go, dotnet).
commands/linter
Package linter provides linter and formatter commands.
 Package linter provides linter and formatter commands.
commands/output
Package output provides output processing and formatting commands.
 Package output provides output processing and formatting commands.
commands/pkgmgr
Package pkgmgr provides package manager commands (cargo, npm, npx, pip, pnpm).
 Package pkgmgr provides package manager commands (cargo, npm, npx, pip, pnpm).
commands/registry
Package registry provides command registration for sub-packages.
 Package registry provides command registration for sub-packages.
commands/sessioncmd
Package sessioncmd provides session and snapshot management commands.
 Package sessioncmd provides session and snapshot management commands.
commands/shared
Package shared provides common types, variables, and utilities for all command packages.
 Package shared provides common types, variables, and utilities for all command packages.
commands/system
Package system provides system utility commands (ls, grep, find, tree, wc, watch, read, search, log, deps, clean, ccusage, discover, tee, profile).
 Package system provides system utility commands (ls, grep, find, tree, wc, watch, read, search, log, deps, clean, ccusage, discover, tee, profile).
commands/test
Package test provides test runner commands (jest, pytest, vitest, playwright).
 Package test provides test runner commands (jest, pytest, vitest, playwright).
commands/vcs
Package vcs provides version control system commands (git, gh, gt).
 Package vcs provides version control system commands (git, gh, gt).
commands/web
Package web provides web, proxy, and HTTP utility commands.
 Package web provides web, proxy, and HTTP utility commands.
compliance
config
contextread
contributiongraph
core
cortex
Package cortex provides content-aware gate system (Claw Compactor's Cortex detection).
 Package cortex provides content-aware gate system (Claw Compactor's Cortex detection).
csvexport
cursorsync
dashboard
Package dashboard provides cost intelligence dashboard components
 Package dashboard provides cost intelligence dashboard components
datavalidation
defense
delta
Package delta implements file version tracking and diff-based compression.
 Package delta implements file version tracking and diff-based compression.
deps
digest
Package digest provides weekly digest generation for TokMan
 Package digest provides weekly digest generation for TokMan
discover
doctor
economics
Package economics provides Claude Code spending vs savings analysis.
 Package economics provides Claude Code spending vs savings analysis.
encryption
Package encryption provides data encryption at rest
 Package encryption provides data encryption at rest
favorites
Package favorites provides favorite commands management
 Package favorites provides favorite commands management
fetchclean
filecache
filter
Package filter implements the 31-layer token compression pipeline.
 Package filter implements the 31-layer token compression pipeline.
filter/cache
Package cache provides caching support for the filter pipeline.
 Package cache provides caching support for the filter pipeline.
filter/engine
Package engine provides the lightweight filter engine for quick output post-processing, distinct from the full 26+ layer pipeline.
 Package engine provides the lightweight filter engine for quick output post-processing, distinct from the full 26+ layer pipeline.
filter/layers
Package layers contains all compression layer implementations.
 Package layers contains all compression layer implementations.
filterdsl
filterejection
filtervariants
filterverify
firewall
fusionpipeline
gateway
Package gateway provides multi-provider LLM gateway.
 Package gateway provides multi-provider LLM gateway.
gatewayctl
githubembed
graph
grpcserver
Package grpcserver provides analytics gRPC service.
 Package grpcserver provides analytics gRPC service.
guardrails
headless
heatmap
historybloat
html
Package html provides HTML content extraction for token savings.
 Package html provides HTML content extraction for token savings.
httpmw
httpproxy
i18n
Package i18n provides internationalization support.
 Package i18n provides internationalization support.
injection
installer
integrity
Package integrity provides hook integrity verification via SHA-256.
 Package integrity provides hook integrity verification via SHA-256.
iteragent
Package iteragent provides iterative agent framework integration for TokMan
 Package iteragent provides iterative agent framework integration for TokMan
jsonpath
kvcache
langcompress
litm
llm
llmproviders
Package llmproviders provides LLM provider integration for TokMan agents
 Package llmproviders provides LLM provider integration for TokMan agents
luahatch
luau
Package luau provides debug functionality for Lua scripts.
 Package luau provides debug functionality for Lua scripts.
malware
mcp
Package mcp provides hash-based caching for MCP context server.
 Package mcp provides hash-based caching for MCP context server.
mcp/persistence
Package persistence provides SQLite-based persistence for MCP cache.
 Package persistence provides SQLite-based persistence for MCP cache.
mcp/reversible
Package reversible provides compression implementations.
 Package reversible provides compression implementations.
mcp/server
Package server provides HTTP and stdio transports for MCP.
 Package server provides HTTP and stdio transports for MCP.
mcp/tools
Package tools provides the 27 MCP tool implementations.
 Package tools provides the 27 MCP tool implementations.
memory
metrics
Package metrics provides Prometheus metrics for TokMan services.
 Package metrics provides Prometheus metrics for TokMan services.
modelcalllimits
multiclient
multitenancy
observability
onboarding
otelm
parser
pathshim
permissions
pii
pipeline
Package pipeline provides pre-processing and post-processing stages.
 Package pipeline provides pre-processing and post-processing stages.
playground
plugin
Package plugin provides an extensible plugin system for TokMan.
 Package plugin provides an extensible plugin system for TokMan.
policy
policydiff
pricing
prometheus
proxy
prune
quality
Package quality provides compression quality scoring and validation.
 Package quality provides compression quality scoring and validation.
ragkb
rbac
Package rbac provides role-based access control
 Package rbac provides role-based access control
readaggressive
reasoning
recommendations
registrycli
remotegain
responsecompress
retention
Package retention provides data retention policy management
 Package retention provides data retention policy management
rewind
rightsizing
routing
rtkcommands
rtksmart
scanning
scheduling
Package scheduling provides command scheduling capabilities
 Package scheduling provides command scheduling capabilities
secrets
security
Package security provides prompt injection detection.
 Package security provides prompt injection detection.
selfupdate
sentiment
Package sentiment provides sentiment analysis for content
 Package sentiment provides sentiment analysis for content
server
shellhooks
siem
simd
Package simd provides SIMD-optimized operations for Tokman compression.
 Package simd provides SIMD-optimized operations for Tokman compression.
specializedagents
streaming
Package streaming provides memory-efficient processing for large inputs (>500K tokens).
 Package streaming provides memory-efficient processing for large inputs (>500K tokens).
stress
Package stress provides distributed stress testing capabilities
 Package stress provides distributed stress testing capabilities
tampstages
tdd
teamcosts
Package teamcosts provides team cost allocation capabilities for TokMan
 Package teamcosts provides team cost allocation capabilities for TokMan
tee
teecommand
telemetry
templatepipes
templatetrees
tfidf
tieredpricing
tokenizer
tokftemplating
tokftest
Package tokftest provides the `tokman verify` command.
 Package tokftest provides the `tokman verify` command.
tokscalenavigation
toml
toon
tracing
Package tracing provides lightweight distributed tracing for the CLI execution path.
 Package tracing provides lightweight distributed tracing for the CLI execution path.
tracking
Package tracking provides cost estimation and reporting.
 Package tracking provides cost estimation and reporting.
tui
Package tui provides a Terminal User Interface using BubbleTea.
 Package tui provides a Terminal User Interface using BubbleTea.
tuiviews
undo
Package undo provides undo/redo functionality for CLI operations
 Package undo provides undo/redo functionality for CLI operations
utils
visual
Package visual provides visual comparison tools for compression results.
 Package visual provides visual comparison tools for compression results.
vulnscan
waste
webclean
workload
Package workload provides workload prediction
 Package workload provides workload prediction
wrapped
wsfeed
pkg
api
Package api provides public API types for TokMan services.
 Package api provides public API types for TokMan services.
api/proto/analyticsv1
api/proto/compressionv1
client
Package client provides gRPC clients for TokMan services.
 Package client provides gRPC clients for TokMan services.
services
analytics
Package analytics provides the analytics service interface.
 Package analytics provides the analytics service interface.
analytics/grpc
Package grpc provides gRPC server implementation for the analytics service.
 Package grpc provides gRPC server implementation for the analytics service.
compression
Package compression provides the compression service interface.
 Package compression provides the compression service interface.
compression/grpc
Package grpc provides gRPC server implementation for the compression service.
 Package grpc provides gRPC server implementation for the compression service.
tests
fixtures

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.