dupehound

dupehound

Detect code duplication across multiple languages. Single binary, no runtime dependencies, fast.

Finds type-1 (identical), type-2 (renamed identifiers), and type-3 (near-miss) clones using token-based detection with function-level granularity — skips imports, declarations, and config blocks to focus on actual logic duplication.

dupehound run against axeforging/tacomex-8bit-shop — a real React + Fastify sample app. See EXAMPLES.md for the full cookbook.

Contents

• Supported languages
• Install
• Usage
• Quick examples — basic recipes
• More examples & use cases → EXAMPLES.md for the full cookbook
• Baseline ratchet
• Quality gate
• Pre-commit hook
• Configuration — including per-path rules
• Clone types
• How it works
• Output formats
• Exit codes

Supported languages

Go, Python, JavaScript, TypeScript, Java, Kotlin, Rust, C, C++, C#, Swift, Scala, PHP, Ruby, Shell, SQL, Lua, Elixir, Dart, R

Install

Download the binary for your platform from Releases or install the latest release with the checksum-verifying install script:

curl -fsSL https://raw.githubusercontent.com/AxeForging/dupehound/main/install.sh | sh

For Gauntlet or custom GitHub Actions pipelines, install without running:

- uses: AxeForging/dupehound/setup@v0.3.0
  with:
    version: v0.3.0

Pin both references for reproducible CI; use version: latest only when intentionally opting into floating releases.

To intentionally track the newest published release, set version: latest.

The script installs to /usr/local/bin when writable, otherwise to $HOME/.local/bin. It verifies the release archive against the published SHA-256 checksums.

Pin a version or install somewhere other than /usr/local/bin:

curl -fsSL https://raw.githubusercontent.com/AxeForging/dupehound/main/install.sh | \
  DUPEHOUND_VERSION=v0.3.0 DUPEHOUND_INSTALL_DIR="$HOME/.local/bin" sh

Upgrading from 0.1

0.2 changes three defaults, so a first scan reports different clones:

• Clones may span functions or sit outside any (--scope file).
• Clones under five lines are dropped (--min-clone-lines 5).
• Memory is capped at half of what is available (--max-memory auto); when that is tight, type-3 detection is skipped and the report says so.

If you use a baseline, re-record it once with --write-baseline. To keep the 0.1 behaviour: --scope function --min-clone-lines 0 --max-memory 0. --dead-code now fails the scan only for high confidence findings.

Or build from source:

make build-local
sudo mv dupehound /usr/local/bin/

Usage

dupehound scan [options]

Options:
  --path, -p           Path to scan (file or directory) [default: .]
  --min-tokens, -t     Minimum tokens to consider a duplicate block [default: 50, ~5 lines]
  --format, -f         Output format: text, json, sarif, md, github [default: text]
                       github emits ::warning workflow commands for inline PR
                       annotations with zero SARIF/upload setup
  --output, -o         Output file (default: stdout)
  --include, -i        Glob patterns to include (repeatable, supports **)
  --exclude, -e        Glob patterns to exclude (repeatable, supports **)
  --language, -L       Filter by language (e.g. go, python, javascript)
  --similarity         Minimum similarity for type-3 detection (0.50–1.00; 1.0 disables) [default: 0.7]
  --max-bucket         Max candidates per fuzzy hash bucket [default: 5000]
  --min-duplication    Fail if global duplication % exceeds this value (0 = disabled) [default: 0]
  --staged             Only report clones involving git-staged files (for pre-commit hooks)
  --since              Git ref for diff-aware scanning (e.g. main, HEAD~5, v1.0.0).
                       The detector itself skips work for clones outside the diff —
                       this is a true cost reduction, not just a post-filter.
  --top                Max clones to show in text/md output (0 = show all) [default: 10]
  --git-churn          Annotate clones with git commit churn and re-sort by churn
  --churn-days         Rolling window in days for git churn (default 90)
  --dead-code          Detect likely-dead functions (heuristic only)
  --show-suppressed    Include suppressed clones in output (tagged [suppressed])
  --max-files          Hard cap on collected source files (0 = no cap); fail-fast safety net
  --max-pairs          Runaway backstop on fuzzy candidate pairs; past the limit, type-3
                       detection stops and returns a partial result [default: 50000000, 0 = unlimited]
  --max-file-size      Skip files larger than this many bytes BEFORE reading them
                       [default: 5242880 (5MiB), 0 = unlimited]; oversized files are
                       almost always minified/generated and would dominate memory
  --min-clone-lines    Drop clones spanning fewer lines than this [default: 5, 0 = no floor].
                       --min-tokens alone lets two or three dense lines qualify
  --scope              Where type-1/2 clones may lie [default: file]. 'file' looks across the
                       whole file, so a copied class made of small methods or a run of
                       block-style tests is one finding; outside function bodies names must
                       match, and import statements are ignored. 'function' looks inside one
                       function body at a time (the behaviour before this option existed)
  --max-memory         Memory budget, e.g. 2G or 512M [default: auto = half of system or
                       container memory, 0 = unlimited]. A scan that would exceed it
                       switches to low-memory mode: files are held as a compact index and
                       re-read on demand. Type-1/2 results are identical; type-3 detection
                       is skipped (partial result). If even that does not fit, the scan
                       stops with an error instead of being killed by the OS
  --baseline           Compare against a baseline file: recorded clones are accepted
                       debt, only NEW clones (or grown instance counts) fail the scan
  --write-baseline     Write current clones to a baseline file (accepted debt) and
                       exit 0; commit it and use --baseline in hooks/CI
  --scan-generated     Include machine-generated files (*.pb.go, @generated / "DO NOT EDIT"
                       headers, etc.); skipped by default as they inflate duplication
  --exit-zero          Always exit 0 even when clones are found
  --config, -c         Path to config file (default: auto-discover .dupehound.yml)
  --no-rules           Ignore the rules: block of the config for this run
  --verbose, -v        Enable verbose logging
  --quiet, -q          Suppress info-level progress logs (the clone report on stdout
                       is unaffected, so failing hooks still show the failure reason)

Quick examples

Scan current directory:

dupehound scan

Scan with a lower threshold:

dupehound scan --path ./src --min-tokens 30

JSON output for CI:

dupehound scan --format json --output report.json

Markdown output for GitHub PR comments:

dupehound scan --format md --output report.md

SARIF output for GitHub Code Scanning:

dupehound scan --format sarif --output results.sarif

Generated files (*.pb.go, @generated / DO NOT EDIT headers, etc.) are skipped by default; pass --scan-generated to include them:

dupehound scan --scan-generated

Scan only Python files:

dupehound scan --language python

More examples & use cases

The recipes above cover the basics. For a full cookbook with copy-pasteable patterns for pre-commit hooks, PR comments, diff-aware CI, suppression rules, churn ranking, dead-code detection, and safe-mode profiles for large monorepos, see EXAMPLES.md.

Jump straight to a topic:

• Pre-commit hook (lefthook + plain git)
• CI on pull requests (GitHub Actions + sticky comments)
• Baseline ratchet (--write-baseline / --baseline)
• Inline PR annotations (--format github)
• Diff-aware scanning with --since
• Filtering files with --include / --exclude
• Suppressing known duplicates (.dupehound-ignore + inline markers)
• Test↔Prod leak detection
• Git churn ranking (--git-churn)
• Dead function detection (--dead-code)
• Safe-mode profiles for large repos (--max-files, --max-pairs, --similarity 1.0)
• Hook output and AI readability (--quiet)
• Output formats (text, md, json, sarif)
• Real-world trial on cli/cli

Baseline ratchet (adopt on an existing codebase)

Most repos already have duplication; failing every scan on day one just gets the tool removed. The baseline ratchet records today's clones as accepted debt and fails only on new duplication — including pasting yet another copy of an already-known clone:

# once: record current duplication and commit the file
dupehound scan --write-baseline .dupehound-baseline.json
git add .dupehound-baseline.json

# in hooks / CI: fails (exit 1) only when NEW clones appear
dupehound scan --baseline .dupehound-baseline.json

Baseline entries are content-based fingerprints of the normalized token structure, so they survive line shifts, file renames, and unrelated edits without churn. The file is deterministic and diff-friendly. Known clones are hidden from the report (the summary still counts them); refactor a known clone away, rewrite the baseline, and the debt shrinks — it can only ratchet down.

Quality gate

Fail if duplication exceeds a threshold:

dupehound scan --min-duplication 5.0
# exit 1 if more than 5% of lines are duplicated

Gauntlet integration

Gauntlet owns orchestration: ordinary Go formatting, vet, lint, tests, and builds, plus Structlint for repository structure and Dupehound for duplication. Both custom gates reuse their existing GitHub annotations.

custom_gates:
  structlint:
    command: ["structlint", "validate", "--format", "github"]
    parser: github-annotations
    line_scoped: false

  dupehound:
    command: ["dupehound", "scan", "--format", "github", "--quiet"]
    parser: github-annotations
    line_scoped: true

Install the released tools and run the same contract locally:

go install github.com/AxeForging/gauntlet/cmd/gauntlet@v0.1.0
go install github.com/AxeForging/structlint@v0.6.0
go install github.com/AxeForging/dupehound@v0.3.0
gauntlet check
gauntlet check --staged --format agent

Structlint is whole-run because structural violations often have no source line. Dupehound is line-scoped, so diff runs suppress existing clone debt while blocking duplication introduced on changed lines.

Pre-commit hook

Use --staged to only check files being committed — fast, focused, no existing-debt noise:

#!/bin/sh
dupehound scan --staged

The --staged flag scans all files for cross-comparison but only reports clones where at least one instance is in a staged file. This catches both within-staged and staged-vs-existing duplication.

Combine with a baseline for the strictest useful hook — block only new duplication touching the commit:

#!/bin/sh
dupehound scan --staged --baseline .dupehound-baseline.json --quiet

Configuration

Create a .dupehound.yml in your project root (or run dupehound init):

scan:
  path: ./src
  min-tokens: 50
  exclude:
    - "*_test.go"
    - "*.spec.ts"
  language: ""
  format: text
  exit-zero: false

Config is auto-discovered by walking up from cwd. CLI flags override config values.

Per-path rules

One threshold rarely fits a whole repository: copied test fakes are 20–40 tokens, while production at that size is full of classes that share a shape by design. rules: sets thresholds per path or naming convention, in one scan:

scan:
  min-tokens: 50                # production
  exclude: [".venv/**"]
rules:                          # first matching rule wins; other files use scan
  - name: tests
    paths: ["**/*_test.py", "**/tests/**", "**/*.test.ts", "**/*_test.go"]
    min-tokens: 25
  - name: migrations
    paths: ["**/migrations/versions/*.py"]
    min-tokens: 120
  - name: generated
    paths: ["**/generated/**"]
    skip: true                  # not scanned at all

• A rule may set min-tokens, similarity (type-3 floor) or skip: true; unset values come from scan. name is optional (unnamed rules show as rules[N]).
• Patterns are matched against the path relative to --path, in the --exclude dialect: no / matches the file name anywhere, ** crosses directories, and a leading **/ also matches the top level.
• A clone spanning files under different rules is judged by the highest min-tokens among them. A 30-token block shared by a test and a production file is not reported when production is at 50 — but the same block copied between test files is, with only its test locations.
• Every format names the rule that judged a clone: [rule: tests ≥25] in text and GitHub annotations, "rule" / "rule_min_tokens" in JSON, properties.rule in SARIF, a 📐 badge in Markdown.
• --min-tokens / --similarity on the command line replace the scan values; rules still apply on top. --no-rules turns them off for one run.
• Baseline fingerprints, .dupehound-ignore, inline markers, --staged and --since work as before. Mistakes in rules: (unknown key, empty paths, malformed glob, threshold ≤ 0) stop the scan with the rule and line.

Cost: one exact-detection pass per distinct min-tokens. On Go's net package (378 files) a tests rule at 25 took a scan from 6.2 s to 10.2 s, with peak memory up 3%.

Clone types

How it works

dupehound tokenizes source files into a normalized token stream (stripping comments, collapsing whitespace), then uses sliding-window hashing to find identical (type-1/2) and near-miss (type-3, via mini-window Jaccard similarity) code blocks.

Detection is restricted to function and method bodies — imports, top-level declarations, struct definitions, config blocks, and keyword maps are automatically excluded. This dramatically reduces false positives and focuses on actionable logic duplication.

Data tables inside functions are excluded too. The rows of a collection literal are structurally identical by design, so a clone whose every instance falls inside a single literal is reported as data, not duplication — the table-driven test pattern no longer flags itself.

This covers Go ([]T{…}, map[K]V{…}, []struct{…}{…}), Python (lists and dicts), JavaScript/TypeScript, Ruby, Rust, PHP, Elixir, Swift, Dart, and Lua. Java, C, C++, C# and Kotlin are deliberately left out: there [ is an index or an array type and { is also a block, so a literal cannot be told from code by tokens alone.

Logic still counts. A callback column (run func(t *testing.T), run: () => {…}) is checked normally, two identical tables in different places are still flagged, and a short inline []T{a, b} never breaks up the block around it.

Output formats

• text — human-readable with hotspots, top clones by impact, and full clone list
• json — machine-readable with per-file stats, duplication percentage, and all clone data
• sarif — SARIF 2.1.0 for GitHub Code Scanning and other SARIF-compatible tools
• md — GitHub-flavored Markdown with expandable <details> sections, designed for PR comments
• github — GitHub Actions workflow commands (::warning file=…,line=…::) for inline PR annotations; just run dupehound scan --format github in any Actions step, no SARIF upload needed

Every clone instance is attributed to its enclosing function (api/user.go:42-63 (in loadUser)) across all supported languages, and each clone carries:

• saves ~N lines — refactor value: lines removable by deduplicating ((instances−1) × block length); the summary shows the repo-wide total
• scope — same-file (trivial extraction), cross-file (same directory), or cross-dir (usually needs a shared package)
• a stable content-based fingerprint (hash) usable in .dupehound-ignore suppression rules and baselines

If a safety guard reduced type-3 coverage (--max-pairs, --max-bucket, --max-memory), every format carries an explicit partial result marker (!! PARTIAL RESULT banner, partial/partial_reason JSON fields, SARIF run properties) — type-1/2 results are always complete.

Exit codes

License

MIT