tokman

TokMan 🌸

World's Most Advanced Token Reduction System — 14-layer research-based compression pipeline achieving 95-99% token reduction.

TokMan intercepts CLI commands, applies 14 research-backed compression layers, and tracks token savings. Built on 50+ papers from top institutions (Mila, Microsoft, Stanford, Berkeley, NeurIPS 2023-2025).

Compression Performance

Features

• 🧠 14-Layer Compression Pipeline — Research-based token reduction (95-99%)
• 🔧 Git Command Wrappers — Filtered status, diff, log, add, commit, push, pull, and more
• 📁 LS Handler — Hide noise directories (.git, node_modules, target, etc.)
• 🐳 Infrastructure Wrappers — Docker, kubectl, AWS CLI with filtered output
• 📦 Package Managers — npm, pnpm, pip, cargo with compact output
• 🧪 Test Runners — Go, pytest, vitest, jest, npm test, playwright with aggregated results
• 🔨 Build Tools — Go, cargo, next.js with error-only output
• 📊 Token Tracking — SQLite-based metrics on tokens saved
• 🔄 Shell Integration — Automatic command rewriting via shell hooks
• 🔐 Integrity Verification — SHA-256 hook verification for security
• 💰 Economics Analysis — Compare spending vs savings with quota estimates
• 💾 Tee on Failure — Auto-saves full output when commands fail for debugging

14-Layer Compression Pipeline

TokMan implements the world's most advanced token reduction system based on 50+ research papers:

See docs/LAYERS.md for detailed documentation of each layer.

Installation

# Clone and build
git clone https://github.com/GrayCodeAI/tokman.git
cd tokman
go build -o tokman ./cmd/tokman

# Install to PATH (optional)
sudo mv tokman /usr/local/bin/

Docker

# Pull from GitHub Container Registry (once published)
docker pull ghcr.io/graycodeai/tokman:latest

# Or build locally
docker build -f docker/Dockerfile -t tokman:latest .

# Run with dashboard
docker run -d -p 8080:8080 -v tokman-data:/home/tokman/.local/share/tokman tokman:latest

# Run with docker-compose
cd docker && docker-compose up -d

Homebrew (macOS/Linux)

brew install GrayCodeAI/tap/tokman

Quick Start

# Initialize TokMan (install shell hook)
tokman init

# Check token savings
tokman status

# View comprehensive savings analysis
tokman gain

# Use wrapped commands
tokman git status
tokman ls
tokman test ./...

Commands

Core Commands

Git Wrappers

Infrastructure Wrappers

Test Runners

Package Managers

Utilities

Analysis Commands

Rewriting

Shell Integration

Token Counting (tiktoken)

TokMan includes direct token counting using OpenAI's tiktoken tokenizer:

# Count tokens in text
tokman count "Hello, world!"

# Count tokens from stdin
cat file.txt | tokman count

# Count tokens in a file
tokman count file.go

# Use specific model encoding
tokman count --model gpt-4o "Hello, world!"
tokman count --model claude-3-sonnet "Hello, world!"

# Compare heuristic vs actual count
tokman count --compare "Your text here"

# Count multiple files
tokman count --files *.go

Supported encodings:

• cl100k_base — GPT-4, GPT-3.5-turbo, Claude, text-embedding-ada-002
• o200k_base — GPT-4o, GPT-4o-mini
• p50k_base — GPT-3 (davinci, curie, babbage, ada)

Add to your .bashrc or .zshrc:

source /path/to/tokman/hooks/tokman-rewrite.sh

This enables:

• Automatic command rewriting for all supported commands
• ts — alias for tokman status
• tr — alias for tokman rewrite
• tokman_install_hook — install hook to shell config
• tokman_status — show integration status

Shell Completions

Enable autocompletions for your shell:

# Bash
source <(tokman completion bash)

# Zsh
source <(tokman completion zsh)

# Fish
tokman completion fish | source

Or save the completion files from completions/ directory.

Custom Filter Plugins

Create JSON-based filter plugins in ~/.config/tokman/plugins/:

{
  "name": "hide-npm-warnings",
  "description": "Hide npm deprecation warnings",
  "enabled": true,
  "patterns": ["npm WARN deprecated"],
  "mode": "hide"
}

# Plugin management
tokman plugin list           # List loaded plugins
tokman plugin create myfilter # Create new plugin template
tokman plugin enable myfilter # Enable a plugin
tokman plugin disable myfilter # Disable a plugin
tokman plugin examples       # Generate example plugins

Web Dashboard

Launch an interactive dashboard to visualize token savings:

# Start dashboard on default port (8080)
tokman dashboard

# Custom port
tokman dashboard --port 3000

# Open browser automatically
tokman dashboard --open

Features:

• Real-time token savings charts
• Daily/weekly/monthly breakdowns
• Command-level analytics
• Cost tracking with Claude API rates
• RESTful API endpoints (/api/stats, /api/daily, /api/commands)

CI/CD Integration

Include TokMan in your CI pipelines for automated reporting:

GitHub Actions

# .github/workflows/tokman.yml
name: Token Savings Report
on: [workflow_run]

jobs:
  report:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-go@v5
        with: { go-version: '1.21' }
      - run: go install github.com/GrayCodeAI/tokman/cmd/tokman@latest
      - run: tokman summary --format markdown >> $GITHUB_STEP_SUMMARY

See templates/ for complete GitHub Actions and GitLab CI examples.

Configuration

Config file: ~/.config/tokman/config.toml

[tracking]
enabled = true
telemetry = false

[filter]
mode = "minimal"  # "minimal" or "aggressive"
noise_dirs = [
    ".git", "node_modules", "target",
    "__pycache__", ".venv", "vendor",
    ".idea", ".vscode", "dist", "build"
]

[hooks]
excluded_commands = []

Gain Command Features

The tokman gain command provides comprehensive savings analysis:

# Basic summary
tokman gain

# Show ASCII graph of daily savings
tokman gain --graph

# Show recent command history
tokman gain --history

# Show quota analysis (subscription tier estimate)
tokman gain --quota --tier pro

# Time breakdowns
tokman gain --daily
tokman gain --weekly
tokman gain --monthly
tokman gain --all

# Export data
tokman gain --format json
tokman gain --format csv

# Filter to current project
tokman gain --project

Security: Integrity Verification

TokMan protects against hook tampering:

# Verify hook integrity
tokman verify

# Runtime checks are automatic for operational commands
# Hooks are verified via SHA-256 hash stored during init

Architecture

┌─────────────────────────────────────────────────────────────┐
│                       TokMan CLI                            │
├─────────────────────────────────────────────────────────────┤
│  Command Handlers (internal/commands/)                      │
│  ├── git.go       - Git wrappers                            │
│  ├── docker.go    - Docker filtering                        │
│  ├── go.go        - Go build/test                           │
│  ├── cargo.go     - Rust commands                           │
│  ├── npm.go       - npm/pnpm/npx                            │
│  ├── pytest.go    - Python tests                            │
│  ├── gain.go      - Savings analysis                        │
│  ├── economics.go - Cost analysis                           │
│  └── ...          - 40+ command handlers                    │
├─────────────────────────────────────────────────────────────┤
│  14-Layer Compression Pipeline (internal/filter/)           │
│  ├── Layer 1-9:   Research compression (95%+)               │
│  ├── Layer 10:    Budget enforcement                        │
│  ├── Layer 11:    Compaction (MemGPT-style)                 │
│  ├── Layer 12:    Attribution (ProCut-style)                │
│  ├── Layer 13:    H2O (Heavy-Hitter Oracle)                 │
│  └── Layer 14:    Attention Sink (StreamingLLM)             │
├─────────────────────────────────────────────────────────────┤
│  Core Engine (internal/)                                    │
│  ├── tracking/    - SQLite token tracking                   │
│  ├── config/      - TOML config loader                      │
│  ├── integrity/   - SHA-256 hook verification               │
│  └── economics/   - Cost analysis engine                    │
├─────────────────────────────────────────────────────────────┤
│  Shell Integration (hooks/)                                 │
│  └── tokman-rewrite.sh - Bash/Zsh hook                      │
└─────────────────────────────────────────────────────────────┘

Development

# Run tests
go test ./...

# Build
go build -o tokman ./cmd/tokman

# Run with verbose output
./tokman -v git status

Token Estimation

TokMan uses a simple heuristic for token estimation:

tokens = ceil(text.length / 4.0)

This provides a reasonable approximation for tracking savings without requiring an actual tokenizer.

Data Storage

Following XDG Base Directory Specification:

Override with environment variables:

• XDG_CONFIG_HOME
• TOKMAN_DATABASE_PATH

Roadmap

• Git command wrappers
• Token tracking database
• Shell integration hooks
• Integrity verification
• Economics analysis
• Python tool wrappers (pytest, ruff, mypy)
• Rust/Cargo test aggregation
• Windows support
• Custom filter plugins
• Web dashboard for analytics
• Shell completions (bash/zsh/fish)
• CI/CD integration templates
• LLM API integration (direct token counting)
• Homebrew formula
• Docker image

License

MIT License - see LICENSE for details.

🌸 TokMan — Save tokens, save money, save the context window.