README
¶
Sift
sift turns a project directory into a single LLM-ready context document. It
walks the tree, respects .gitignore, and renders the result as plain text,
Markdown, JSON, or XML, with token budgeting, secret redaction, signature-only
compression, and git-relevance ranking so the most important code fits inside
a model's context window.
What it does
Sift scans a repo, filters out the noise (.gitignore, hidden files,
binaries, lockfiles), and packs what's left into one document you hand to an
LLM. Along the way it:
- Counts tokens against a
--budgetand keeps the highest-priority files when the budget is tight, using git relevance to decide what matters: modified files first, then recent diffs, then recent commits. - Redacts secrets (AWS, GitHub, Slack, Google, Stripe keys) before anything gets rendered.
- Compresses files to signatures only (
--mode signatures) via tree-sitter, keeping declarations and comments while dropping function bodies. Go, Rust, JavaScript/TypeScript (incl. TSX), Python, PHP, Java, Kotlin, C#, C/C++, Ruby, and Swift are supported. - Copies straight to the clipboard with
--clipboard, or writes to a file, or streams to stdout.
dump, pick, select, diff, delta, and watch cover the different
ways you'd want to generate that document: a full one-shot scan, an
interactive selection, an automatic budget-aware selection, just what
changed relative to a git ref, just what changed since your last dump, or a
document that keeps re-rendering as you edit.
Interactive picker
sift pick (or bare sift) opens a dual-pane TUI: a foldable file tree on
the left with three-state checkboxes and live token counts, a preview pane
with syntax highlighting and secret warnings on the right, and a budget bar
in the footer.
Each file can be FULL, SIGS (signature-only), or SKIP, cycled with m.
Press s to smart-select by git relevance and budget, / to fuzzy-filter
the tree, p to attach a task prompt (with presets for review, refactor,
bug investigation, and so on), and y to copy the result to the clipboard.
d opens the delta modal for picking a commit range interactively. Nerd
Font glyphs are on by default; pass --no-nerd-fonts for plain ASCII.
See docs/picker.md for the full keybinding reference.
Incremental deltas
A full dump gets expensive to re-send on every turn. sift delta renders
only what changed since the last recorded dump, keyed by the project's
absolute path and stored under ~/.config/sift/state.json, outside the repo.
sift delta # full content of files changed since last dump
sift delta --patch # raw unified diff in a <context_update> block
sift delta --since main # compare against a specific ref or branch
sift delta --patch --clipboard
--patch is the cheaper option token-wise: it emits the raw git diff
between the recorded baseline and HEAD, wrapped in a <context_update>
block, instead of full file contents.
Installation
macOS and Linux:
curl -fsSL https://bethropolis.github.io/sift/install.sh | sh
This installs to ~/.local/bin; set SIFT_INSTALL_DIR to change that.
Homebrew:
brew install bethropolis/tap/sift
Windows (Scoop):
scoop bucket add bethropolis https://github.com/bethropolis/scoop-bucket
scoop install sift
Go:
go install github.com/bethropolis/sift/cmd/sift@latest
Prebuilt archives for Linux, Windows, macOS, FreeBSD, and Android (amd64 and arm64) are on the releases page. Or build from source:
git clone https://github.com/bethropolis/sift.git
cd sift
./scripts/install.sh
Usage
sift # bare invocation launches the interactive picker
sift dump [path] # scan a directory and render its contents
sift pick [path] # interactively choose files, then render
sift select [path] # automatically select useful files
sift diff [ref] # dump files changed relative to a git ref
sift delta [path] # dump only changes since the last recorded dump
sift watch [path] # re-render the document on file changes
sift dump writes codebase.md next to the scanned root by default; use
--output - to write to stdout instead.
sift dump --style markdown
sift dump --dir ./src --style xml --clipboard
sift dump --budget 50000
sift dump --mode signatures --style xml
sift dump --prompt "Review this codebase for race conditions."
sift select . --selection-only --print-selection
sift select . --selection-only --selection-format json
sift pick --no-nerd-fonts
Run sift --help or sift <command> --help for the full flag list,
including profiles, ignore rules, secret scanning, and window-title
configuration.
Profiles
A .sift.toml in the scanned directory, or ~/.config/sift/config.toml
globally, packages a set of flags under a name you reference with
--profile:
[profiles.claude]
style = "xml"
budget = 60000
[profiles.rust-strict]
extensions = ["rs"]
mode = "signatures"
prompt = "Review this Rust codebase for unsafe usage."
sift dump --profile claude
Command-line flags always win over profile values.
Documentation
- Installation
- Usage
- Interactive picker
- Configuration and profiles
- Output formats and prompts
- Incremental deltas
Development
Needs Go 1.26+ and a POSIX shell for the helper scripts.
go test ./...
go vet ./...
go test -race ./...
just test, just test-race, just build, just dump, and just pick
cover the common justfile recipes.
Contributing
Issues and pull requests are welcome.
License
Directories
¶
| Path | Synopsis |
|---|---|
|
cmd
|
|
|
sift
command
Command sift packs a codebase into an LLM-friendly context document.
|
Command sift packs a codebase into an LLM-friendly context document. |
|
internal
|
|
|
clipboard
Package clipboard copies text to the system clipboard using the platform's native tool, avoiding X11-specific dependencies.
|
Package clipboard copies text to the system clipboard using the platform's native tool, avoiding X11-specific dependencies. |
|
compress
Package compress extracts signature-only summaries of source files using tree-sitter grammars registered in internal/lang.
|
Package compress extracts signature-only summaries of source files using tree-sitter grammars registered in internal/lang. |
|
format
Package format renders a scanned document in configurable output styles.
|
Package format renders a scanned document in configurable output styles. |
|
highlight
Package highlight provides small, terminal-safe syntax highlighting for human-facing previews.
|
Package highlight provides small, terminal-safe syntax highlighting for human-facing previews. |
|
ignore
Code generated by generate_rules.py.
|
Code generated by generate_rules.py. |
|
lang
Package lang is the facade external consumers import.
|
Package lang is the facade external consumers import. |
|
lang/registry
Package registry owns the global language lookup tables and the shared path-convention rules.
|
Package registry owns the global language lookup tables and the shared path-convention rules. |
|
lang/signature
Package signature owns the tree-sitter signature engine: per-language SignatureSpec registration and a pooled parser engine that reduces source to declaration signatures.
|
Package signature owns the tree-sitter signature engine: per-language SignatureSpec registration and a pooled parser engine that reduces source to declaration signatures. |
|
lang/types
Package types defines the core language contracts shared by every driver and the registry.
|
Package types defines the core language contracts shared by every driver and the registry. |
|
picker
Package picker runs the interactive picker workflow: structure-first skeleton launch, a background streaming scan, git-relevance rank patching, and selection-to-output conversion.
|
Package picker runs the interactive picker workflow: structure-first skeleton launch, a background streaming scan, git-relevance rank patching, and selection-to-output conversion. |
|
rank
Package rank scores files by git relevance so an LLM gets the most important context first when a token budget is in play.
|
Package rank scores files by git relevance so an LLM gets the most important context first when a token budget is in play. |
|
scan
Package scan owns the per-file processing pipeline: smart filtering, token counting, and optional signature compression.
|
Package scan owns the per-file processing pipeline: smart filtering, token counting, and optional signature compression. |
|
secrets
Code generated by generate_rules.py.
|
Code generated by generate_rules.py. |
|
selection
Package selection chooses the most useful representation of collected files under a token budget.
|
Package selection chooses the most useful representation of collected files under a token budget. |
|
setup
Package setup provides initialization and configuration functions
|
Package setup provides initialization and configuration functions |
|
smart
Package smart filters files that are unlikely to be useful LLM context: generated artifacts, lockfiles, minified bundles, and oversized files that .gitignore often misses.
|
Package smart filters files that are unlikely to be useful LLM context: generated artifacts, lockfiles, minified bundles, and oversized files that .gitignore often misses. |
|
state
Package state persists per-project dump records in the user's application data directory so delta dumps can know the last commit a project was dumped from, without polluting the repository itself.
|
Package state persists per-project dump records in the user's application data directory so delta dumps can know the last commit a project was dumped from, without polluting the repository itself. |
|
summary
Package summary handles display of scan results and statistics
|
Package summary handles display of scan results and statistics |
|
tokenize
Package tokenize counts tokens offline and fits documents to a budget.
|
Package tokenize counts tokens offline and fits documents to a budget. |
|
tui
Package tui renders an interactive picker for the pick subcommand.
|
Package tui renders an interactive picker for the pick subcommand. |
|
utils
Package utils provides common utilities shared across packages
|
Package utils provides common utilities shared across packages |
|
walker
Package walker handles directory traversal and file processing
|
Package walker handles directory traversal and file processing |
|
watch
Package watch re-renders a directory's context document when files change.
|
Package watch re-renders a directory's context document when files change. |