git-dedup

git-dedup

Faster checkouts, a fraction of the disk space.

git-dedup is a wrapper around git built for fleets of coding agents. It forwards ordinary Git commands unchanged and automatically keeps one shared copy of Git history, so every new clone, worktree, and submodule reuses what is already on disk instead of downloading it again.

• 6x faster checkouts. Large checkouts drop from over a minute to about 10 seconds.
• 70% less disk. Across 117 checkouts, Git data dropped from 35.5 GB to 11.1 GB.
• No workflow changes. Checkouts are normal Git repositories. Use git, your editor, and your agents as before.

Install and use

npm install --global git-dedup

# check out a new repo automatically using the dedup store
git-dedup clone https://github.com/you/project.git

# dedup an existing repo into the store
git-dedup store add ./my-existing-repo

That's it. git-dedup automatically consolidates the new or existing project's history into a shared store in ~/.git-dedup or if its history already existed there, it reuses it automatically.

git-dedup is a native binary for macOS, Linux, and Windows on x64 and arm64; it only needs Git. The npm package runs the binary for your platform through a small launcher (Node.js 18+). Without Node.js, download an archive, .deb, .rpm, or .apk from GitHub Releases.

Documentation · CLI reference · Agent setup

Why I built it

I run fleets of coding agents, each in its own checkout. Every task started by cloning repositories and submodules from scratch, so agents sat idle waiting on downloads. Then I started running out of disk space, because every checkout held another full copy of the same history. git-dedup fixed both. We have dogfooded it heavily across our own agent fleets and repositories to make it robust and efficient.

More commands

# Worktrees and submodules get their history from the store too.
git-dedup clone --recurse-submodules https://github.com/you/project.git
git-dedup worktree add -b my-task ../my-task
git-dedup submodule update --init --recursive

# Reclaim space from repositories you already have.
git-dedup store add .
git-dedup store add ~/Coding --all

# Inspect the store.
git-dedup store

Ordinary commands, such as git-dedup status, are forwarded to Git. Unsupported clone forms, such as shallow, single-branch, or partial clones, fall back to plain Git. See the CLI reference.

Configuration

The default store is ~/.git-dedup (%USERPROFILE%\.git-dedup on Windows). Override it with GIT_DEDUP_STORE:

export GIT_DEDUP_STORE="$HOME/my-git-dedup-store"

$env:GIT_DEDUP_STORE = "$HOME\my-git-dedup-store"

Changing GIT_DEDUP_STORE selects a different store; it does not move the existing one. git-dedup does not change global Git configuration.

Keep the store

Your checkouts read their history from the shared store. Deleting it breaks every repository consolidated into it. See the store guide for moving or detaching checkouts.

Repository layout

After installing, the agent setup guide provides instructions to add to AGENTS.md or CLAUDE.md so agents call git-dedup explicitly.

Development

pnpm install --frozen-lockfile
pnpm build
pnpm check
pnpm test:proof

Development needs Go (the version in go.mod), Node.js, and pnpm. pnpm build compiles the binary for this machine. pnpm check runs Oxlint, Oxfmt, go vet, the Vitest behavior suites against the binary, workflow tests, npm package checks, and the documentation build; go test ./... runs the Go unit tests. The proof script creates temporary loopback Git remotes and verifies that concurrent clones share one object pool and remain valid after pool maintenance.

See CONTRIBUTING.md for the development workflow and RELEASING.md for release setup.

Contributing

See CONTRIBUTING.md for the issue, branch, and PR workflow, and GitHub Releases for release notes.

License

MIT

Author

Ben Houston