tokman

TokMan

Token-aware CLI proxy & AI gateway for coding assistants. 31-layer compression pipeline built on 30+ research papers, achieving 60–90% token savings.

Website · Install · Quick Start · Discord · Architecture · Contributing

TokMan intercepts CLI commands and applies a 31-layer compression pipeline to reduce token usage for AI coding assistants. Built on research from 30+ papers, it achieves 60–90% token reduction on common development operations — far beyond simple output filtering.

Input:  10,000 tokens  ──►  TokMan  ──►  Output:  1,500 tokens
                                        Savings:   85%
                                        Cost:      $0.085 → $0.013

Token Savings (30-min Claude Code Session)

Estimates based on medium-sized projects. Actual savings vary by project size and command complexity.

Cost Reduction

Installation

Go Install (recommended)

go install github.com/GrayCodeAI/tokman/cmd/tokman@latest

Build from Source

git clone https://github.com/GrayCodeAI/tokman.git
cd tokman
make build

SIMD-Optimized Build

make build-simd

Verify Installation

tokman --version
tokman gain   # Show token savings stats

Quick Start

# 1. Install for your AI tool
tokman init -g                     # Claude Code / Copilot (default)
tokman init -g --gemini            # Gemini CLI
tokman init -g --codex             # Codex (OpenAI)
tokman init --agent cursor         # Cursor
tokman init --agent windsurf       # Windsurf
tokman init --agent cline          # Cline / Roo Code

# 2. Restart your AI tool, then test
git status  # Automatically rewritten to tokman git status

# 3. Or use standalone
tokman summary --preset full < input.txt
tokman http-proxy start --listen :8080
tokman tui                         # Analytics dashboard

How It Works

  Without TokMan:                              With TokMan:

  Claude  --git status-->  shell  -->  git    Claude  --git status-->  TokMan  -->  git
    ^                                   |        ^                      |           |
    |       ~2,000 tokens (raw)         |        |    ~200 tokens       | 31-layer  |
    +-----------------------------------+        +---- (filtered) ------+ pipeline  +

Unlike simple output filters (dedup + truncation), TokMan applies a 31-layer research-backed compression pipeline with semantic understanding, AST preservation, goal-driven selection, and inter-layer feedback.

Commands

Files

tokman ls .                        # Token-optimized directory tree
tokman read file.go                # Smart file reading
tokman read file.go -l aggressive  # Signatures only (strips bodies)
tokman read file.go --mode graph   # Target file + related project files
tokman ctx read file.go --mode auto --max-tokens 300
tokman ctx delta file.go           # Show change since last snapshot
tokman ctx read file.go --mode graph --related-files 4
tokman smart file.go               # 2-line heuristic code summary
tokman find "*.go" .               # Compact find results
tokman grep "pattern" .            # Grouped search results
tokman diff file1 file2            # Condensed diff

Context workflow:

# Budget a single file for an agent
tokman ctx read internal/server/server.go --mode auto --max-tokens 350

# Show only what changed since the last snapshot
tokman ctx delta internal/server/server.go

# Pull related files into the same context bundle
tokman ctx read internal/server/server.go --mode graph --related-files 4

TokMan now tracks and visualizes smart context delivery separately from normal command filtering, including:

• direct file reads
• delta reads
• MCP-served reads
• top files and projects by context savings
• graph-aware MCP bundles for agent context delivery
• structured smart-read analytics by kind, mode, target, and bundle quality

Git

tokman git status                  # Compact status
tokman git log -n 10               # One-line commits
tokman git diff                    # Condensed diff
tokman git add                     # → "ok"
tokman git commit -m "msg"         # → "ok abc1234"
tokman git push                    # → "ok main"
tokman git pull                    # → "ok 3 files +10 -2"

GitHub CLI

tokman gh pr list                  # Compact PR listing
tokman gh pr view 42               # PR details + checks
tokman gh issue list               # Compact issue listing
tokman gh run list                 # Workflow run status

Test Runners

tokman test cargo test             # Show failures only (-90%)
tokman err npm run build           # Errors/warnings only
tokman vitest run                  # Vitest compact (failures only)
tokman playwright test             # E2E results (failures only)
tokman pytest                      # Python tests (-90%)
tokman go test                     # Go tests (NDJSON, -90%)
tokman cargo test                  # Cargo tests (-90%)
tokman rake test                   # Ruby minitest (-90%)
tokman rspec                       # RSpec tests (JSON, -60%+)

Build & Lint

tokman lint                        # ESLint grouped by rule/file
tokman lint biome                  # Supports other linters
tokman tsc                         # TypeScript errors grouped by file
tokman next build                  # Next.js build compact
tokman prettier --check .          # Files needing formatting
tokman cargo build                 # Cargo build (-80%)
tokman cargo clippy                # Cargo clippy (-80%)
tokman ruff check                  # Python linting (JSON, -80%)
tokman golangci-lint run           # Go linting (JSON, -85%)
tokman rubocop                     # Ruby linting (JSON, -60%+)

Package Managers

tokman pnpm list                   # Compact dependency tree
tokman pip list                    # Python packages (auto-detect uv)
tokman pip outdated                # Outdated packages
tokman bundle install              # Ruby gems (strip Using lines)
tokman prisma generate             # Schema generation (no ASCII art)

Containers

tokman docker ps                   # Compact container list
tokman docker images               # Compact image list
tokman docker logs <container>     # Deduplicated logs
tokman docker compose ps           # Compose services
tokman kubectl pods                # Compact pod list
tokman kubectl logs <pod>          # Deduplicated logs
tokman kubectl services            # Compact service list

Data & Analytics

tokman json config.json            # Structure without values
tokman deps                        # Dependencies summary
tokman env -f AWS                  # Filtered env vars
tokman log app.log                 # Deduplicated logs
tokman curl <url>                  # Auto-detect JSON + schema
tokman wget <url>                  # Download, strip progress bars
tokman summary <long command>      # Heuristic summary
tokman proxy <command>             # Raw passthrough + tracking

Token Savings Analytics

tokman gain                        # Summary stats
tokman gain --graph                # ASCII graph (last 30 days)
tokman gain --history              # Recent command history
tokman gain --daily                # Day-by-day breakdown
tokman gain --all --format json    # JSON export for dashboards

tokman discover                    # Find missed savings opportunities
tokman discover --all --since 7    # All projects, last 7 days

tokman session                     # Show TokMan adoption across recent sessions

Global Flags

-u, --ultra-compact    # ASCII icons, inline format (extra token savings)
-v, --verbose          # Increase verbosity (-v, -vv, -vvv)

Examples

Directory listing:

# ls -la (45 lines, ~800 tokens)         # tokman ls (12 lines, ~150 tokens)
drwxr-xr-x  15 user staff 480 ...        my-project/
-rw-r--r--   1 user staff 1234 ...        +-- src/ (8 files)
...                                       |   +-- main.go
                                          +-- go.mod

Git operations:

# git push (15 lines, ~200 tokens)        # tokman git push (1 line, ~10 tokens)
Enumerating objects: 5, done.              ok main
Counting objects: 100% (5/5), done.
Delta compression using up to 8 threads
...

Test output:

# cargo test (200+ lines on failure)      # tokman test cargo test (~20 lines)
running 15 tests                           FAILED: 2/15 tests
test utils::test_parse ... ok                test_edge_case: assertion failed
test utils::test_format ... ok               test_overflow: panic at utils.rs:18
...

Compression Pipeline

TokMan runs a 31-layer compression pipeline with stage gates, early-exit, and adaptive selection.

Pipeline Features

• Stage gates — skip layers when not applicable (zero cost)
• Early exit — stop pipeline when budget already met
• Inter-layer feedback — quality estimation between layers
• Result caching — O(1) LRU with SHA-256 fingerprinting
• Parallel execution — SIMD-accelerized where possible

HTTP Proxy Mode

Transparent proxy that intercepts LLM API calls — works with any AI agent without hooks.

tokman http-proxy start --listen :8080
# Then point your AI agent to http://localhost:8080

AI Gateway

tokman gateway --action set-quota --model gpt-4 --quota 10000
tokman gateway --action alias --model gpt-4 --alias gpt-4o-mini
tokman gateway --action kill-switch --model gpt-4 --kill-switch

Analytics & Monitoring

tokman tui                          # Interactive dashboard
tokman analytics --action anomaly   # Anomaly detection
tokman analytics --action forecast  # Spend forecasting
tokman analytics --action heatmap   # Token breakdown

Enterprise Features

TokMan includes 40+ enterprise-grade packages for production-ready token management, cost intelligence, testing, and AI agent orchestration.

Performance Testing

Cost Intelligence

AI Agent Framework

Deployment & Reliability

CLI Enhancements

CI/CD Integration

TokMan includes GitHub Actions workflows for:

• Benchmark testing — automated performance regression detection
• Stress testing — load validation on every PR
• Multi-platform builds — Linux, macOS, Windows (amd64/arm64)
• Security scanning — dependency vulnerability checks
• Automated releases — tag-based release with changelog generation

Security

tokman security --action scan < input.txt    # Scan for vulnerabilities
tokman security --action redact < input.txt  # Redact PII

Supported AI Tools

TokMan supports 10+ AI coding tools. Each integration transparently rewrites shell commands to tokman equivalents for 60–90% token savings.

Auto-Rewrite Hook

The most effective way to use TokMan. The hook transparently intercepts Bash commands and rewrites them to TokMan equivalents before execution.

Result: 100% TokMan adoption across all conversations and subagents, zero token overhead.

Setup

tokman init -g                 # Install hook + instructions (recommended)
tokman init -g --auto-patch    # Non-interactive (CI/CD)
tokman init -g --hook-only     # Hook only, no instructions
tokman init --show             # Verify installation
tokman init -g --uninstall     # Remove

After install, restart your AI tool.

Supported Commands

Commands already using tokman, heredocs (<<), and unrecognized commands pass through unchanged.

Configuration

Config file at ~/.config/tokman/config.toml:

[tracking]
enabled = true
database_path = "~/.local/share/tokman/tokman.db"

[filter]
mode = "minimal"  # or "aggressive"

[pipeline]
max_context_tokens = 2000000
enable_entropy = true
enable_compaction = true

[hooks]
excluded_commands = ["curl", "playwright"]

[dashboard]
port = 8080

Environment Variables

Architecture

┌─────────────┐    ┌──────────────────┐    ┌──────────────┐
│ CLI Command │───►│ 31-Layer Pipeline│───►│ Compressed   │
│ (git, npm,  │    │ (filter/)        │    │ Output       │
│  docker...) │    │                  │    │              │
└─────────────┘    └──────────────────┘    └──────────────┘
                          │
                    ┌─────┴─────┐
                    │ HTTP Proxy│
                    │ (proxy/)  │
                    └───────────┘

Performance

vs Competitors

Documentation

• ARCHITECTURE.md — Technical architecture
• LAYERS.md — Detailed layer descriptions
• CONTRIBUTING.md — How to contribute
• SECURITY.md — Security policy

Contributing

Contributions welcome! Please open an issue or PR on GitHub.

# Run tests
make test

# Run linter
make lint

# Run benchmarks
make benchmark

License

MIT — see LICENSE