sift

module

v1.5.0 Latest Latest Go to latest Published: Oct 8, 2026 License: MIT

Details

Valid go.mod file
Redistributable license
Tagged version
Stable version
Learn more about best practices

Repository

github.com/bethropolis/sift

Links

Open Source Insights

README ¶

Sift

Your repo is too big to paste into a chatbot. That's the whole problem. sift fixes it: point it at a directory, get back one clean document with exactly the code the model needs.

Sift's interactive picker showing the file tree, live token counts, and a preview pane

What it does

sift walks a project directory, skips what you wouldn't paste into a prompt (.gitignored files, dotfiles, binaries, lockfiles), and writes the rest to a single document for an LLM. Output can be plain text, Markdown, JSON, or XML.

Files are ranked by git relevance: modified files first, then files in recent diffs, then files in recent commits. Pass --budget 200k and sift fills the document in that order and stops when the budget is used up.

Secrets are redacted before anything is rendered. This covers AWS, GitHub, Slack, Google, and Stripe keys.

With --mode signatures, sift uses tree-sitter to keep declarations and comments and drop function bodies. You keep the structure of the code at a fraction of the tokens.

The result can go to a file, stdout, or the clipboard (--clipboard). sift copy puts the last codebase.md back on the clipboard.

Languages supported in signature mode

Go, Rust, JavaScript/TypeScript (including TSX), Python, PHP, Java, Kotlin, C#, C/C++, Ruby, Swift, Dart/Flutter, Zig.

Commands

Command	Description
`dump`	Render every eligible file
`pick`	Choose files yourself in the TUI, then render
`select`	Let git relevance and the budget choose the files
`clone`	Clone a repo, then run on it
`copy`	Re-copy the last output to the clipboard
`diff`	Render files changed relative to a git ref
`delta`	Render files changed since the last dump
`follow`	Render a file plus its import-graph neighbors
`watch`	Re-render whenever files change
`serve`	Run sift from a browser

Running sift with no command opens the picker.

Picker

sift pick (or just sift) opens a two-pane TUI: file tree on the left, preview on the right, and a budget bar at the bottom.

Each file is in one of three modes: FULL, SIGS, or SKIP. Press m to cycle a file through them, or press s to have git relevance assign modes for you. / starts a fuzzy search. p attaches a task prompt, with presets for review, refactor, and debugging. Y builds codebase.md and copies it to the clipboard. d opens the commit-range picker.

Nerd Font icons are on by default. Pass --no-nerd-fonts to turn them off.

The full key list is in docs/picker.md.

Rather click? Use the browser.

sift serve runs the same picker as a web UI. sift serve --app goes one further: chromeless window, auto-login, and closing the window stops the server.

Sift's web UI with the same file tree, live token counts, and preview pane, running in a browser

Details: docs/serve.md.

Incremental deltas

sift delta renders only what changed since the last recorded dump. --patch emits a raw unified diff instead, which is cheaper token-wise:

sift delta --patch --clipboard
sift delta --since main

Installation

macOS / Linux — installs to ~/.local/bin:

curl -fsSL https://bethropolis.github.io/sift/install.sh | sh

macOS (Homebrew):

brew install bethropolis/tap/sift

Arch Linux (AUR):

yay -S sift-context-bin

Windows (Scoop):

scoop bucket add bethropolis https://github.com/bethropolis/scoop-bucket
scoop install sift

Go toolchain (CLI only — no embedded serve web UI):

go install github.com/bethropolis/sift/cmd/sift@latest

From source or a prebuilt archive:

git clone https://github.com/bethropolis/sift.git
cd sift
./scripts/install.sh

Prebuilt archives for Linux, Windows, macOS, FreeBSD, and Android (amd64 and arm64) are on the releases page.

Uninstall

From a checkout: ./scripts/uninstall.sh (--purge also removes ~/.config/sift). Otherwise use your package manager (brew uninstall, scoop uninstall, rm ~/.local/bin/sift). Full details: docs/installation.md.

Usage

sift                # interactive picker
sift dump [path]    # scan a directory (writes codebase.md next to the root)
sift pick [path]    # interactively choose files, then render
sift select [path]  # automatically select useful files
sift diff [ref]     # files changed relative to a git ref
sift delta [path]   # only changes since the last recorded dump
sift follow <file>  # render a file plus its import-graph neighbors
sift watch [path]   # re-render on file changes
sift serve          # browser picker at http://127.0.0.1:7777

More examples

sift dump --style markdown
sift dump --dir ./src --style xml --clipboard
sift dump --budget 50000
sift dump --mode signatures --style xml
sift dump --prompt "Review this codebase for race conditions."
sift select . --selection-only --print-selection
sift pick --no-nerd-fonts

sift dump writes codebase.md by default; --output - writes to stdout instead. sift --help lists everything else (profiles, ignore rules, secret scanning).

Profiles

A .sift.toml in the scanned directory, or ~/.config/sift/config.toml globally, bundles flags under a --profile name. Flags always win over profile values:

[profiles.claude]
style = "xml"
budget = 60000
compress_mode = "signatures"
extensions = ["go", "md"]
ignore = ["vendor/**", "testdata/**"]
prompt_file = "prompts/review.md"
smart_filter = true
copy_on_generate = true

sift dump --profile claude

Documentation

Development

Needs Go 1.26+ and a POSIX shell (just test, just test-race, just build, just dump, just pick cover the common recipes):

go test ./...
go vet ./...
go test -race ./...

Contributing

Issues and pull requests are welcome.

License

MIT

Directories ¶

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL

Path	Synopsis
cmd
sift command Command sift packs a codebase into an LLM-friendly context document.	Command sift packs a codebase into an LLM-friendly context document.
internal
app
clipboard Package clipboard copies text to the system clipboard using the platform's native tool, avoiding X11-specific dependencies.	Package clipboard copies text to the system clipboard using the platform's native tool, avoiding X11-specific dependencies.
compress Package compress extracts signature-only summaries of source files using tree-sitter grammars registered in internal/lang.	Package compress extracts signature-only summaries of source files using tree-sitter grammars registered in internal/lang.
config
desktop Package desktop installs and removes OS launcher entries that open `sift serve --app`: an XDG .desktop file on Linux and a Sift.app bundle on macOS.	Package desktop installs and removes OS launcher entries that open `sift serve --app`: an XDG .desktop file on Linux and a Sift.app bundle on macOS.
format Package format renders a scanned document in configurable output styles.	Package format renders a scanned document in configurable output styles.
gitclone Package gitclone is the one place sift runs `git clone`.	Package gitclone is the one place sift runs `git clone`.
highlight Package highlight provides small, terminal-safe syntax highlighting for human-facing previews.	Package highlight provides small, terminal-safe syntax highlighting for human-facing previews.
ignore Code generated by generate_rules.py.	Code generated by generate_rules.py.
lang Package lang is the facade external consumers import.	Package lang is the facade external consumers import.
lang/go
lang/javascript
lang/other
lang/python
lang/registry Package registry owns the global language lookup tables and the shared path-convention rules.	Package registry owns the global language lookup tables and the shared path-convention rules.
lang/rust
lang/signature Package signature owns the tree-sitter signature engine: per-language SignatureSpec registration and a pooled parser engine that reduces source to declaration signatures.	Package signature owns the tree-sitter signature engine: per-language SignatureSpec registration and a pooled parser engine that reduces source to declaration signatures.
lang/types Package types defines the core language contracts shared by every driver and the registry.	Package types defines the core language contracts shared by every driver and the registry.
lang/typescript
logger
mcp
picker Package picker runs the interactive picker workflow: structure-first skeleton launch, a background streaming scan, git-relevance rank patching, and selection-to-output conversion.	Package picker runs the interactive picker workflow: structure-first skeleton launch, a background streaming scan, git-relevance rank patching, and selection-to-output conversion.
rank Package rank scores files by git relevance so an LLM gets the most important context first when a token budget is in play.	Package rank scores files by git relevance so an LLM gets the most important context first when a token budget is in play.
scan Package scan owns the per-file processing pipeline: smart filtering, token counting, and optional signature compression.	Package scan owns the per-file processing pipeline: smart filtering, token counting, and optional signature compression.
secrets Code generated by generate_rules.py.	Code generated by generate_rules.py.
selection Package selection chooses the most useful representation of collected files under a token budget.	Package selection chooses the most useful representation of collected files under a token budget.
serve appmode.go owns the app-window lifecycle: it counts the window's liveness stream and shuts the server down when the window is gone, so an app-mode server never outlives the UI it was opened for.	appmode.go owns the app-window lifecycle: it counts the window's liveness stream and shuts the server down when the window is gone, so an app-mode server never outlives the UI it was opened for.
serve/auth Package auth holds the serve authentication primitives: startup token or configured password verification, stateless signed session cookies, and login throttling.	Package auth holds the serve authentication primitives: startup token or configured password verification, stateless signed session cookies, and login throttling.
serve/jail Package jail resolves every client-supplied path against the configured roots.	Package jail resolves every client-supplied path against the configured roots.
setup Package setup provides initialization and configuration functions	Package setup provides initialization and configuration functions
smart Package smart filters files that are unlikely to be useful LLM context: generated artifacts, lockfiles, minified bundles, and oversized files that .gitignore often misses.	Package smart filters files that are unlikely to be useful LLM context: generated artifacts, lockfiles, minified bundles, and oversized files that .gitignore often misses.
state Package state persists per-project dump records in the user's application data directory so delta dumps can know the last commit a project was dumped from, without polluting the repository itself.	Package state persists per-project dump records in the user's application data directory so delta dumps can know the last commit a project was dumped from, without polluting the repository itself.
summary
tokenize Package tokenize counts tokens offline and fits documents to a budget.	Package tokenize counts tokens offline and fits documents to a budget.
tui Package tui renders an interactive picker for the pick subcommand.	Package tui renders an interactive picker for the pick subcommand.
utils Package utils provides common utilities shared across packages	Package utils provides common utilities shared across packages
walker Package walker handles directory traversal and file processing	Package walker handles directory traversal and file processing
watch Package watch re-renders a directory's context document when files change.	Package watch re-renders a directory's context document when files change.
web Package web embeds the built Svelte single-page app served by `sift serve`.	Package web embeds the built Svelte single-page app served by `sift serve`.