secrets-in-source

fasthog

A concurrent secrets scanner for source code repositories. Fasthog detects hardcoded credentials, API keys, and other sensitive information using configurable regex patterns.

Features

• Concurrent scanning: Leverages multiple CPU cores for parallel file processing
• Two-stage detection: Fast preliminary screening followed by thorough pattern matching
• False positive filtering: Configurable exclusion patterns to reduce noise
• Multiple output formats: Terminal UI with progress tracking, JSON output, and optional file output
• Extensible patterns: Customizable regex files for different secret types, including optional pattern file overrides via config
• Cross-platform: Builds for Linux, macOS, and Windows

Installation

Prerequisites

• Go 1.22 or later (download)

Install from source

git clone https://github.com/bordenet/secrets-in-source.git
cd secrets-in-source
go install

Ensure ~/go/bin is in your PATH to run the installed binary.

Build from source

git clone https://github.com/bordenet/secrets-in-source.git
cd secrets-in-source
make build

The binary will be created in the current directory.

Cross-compilation

Build for different platforms:

# Windows
GOOS=windows GOARCH=amd64 go build -o fasthog.exe

# Linux
GOOS=linux GOARCH=amd64 go build -o fasthog

# macOS (Apple Silicon)
GOOS=darwin GOARCH=arm64 go build -o fasthog

Usage

Basic scanning

# Scan a directory
fasthog /path/to/repository

# Scan specific file types
fasthog /path/to/repository --types=py,js,go

# Save results to file
fasthog /path/to/repository --output=results.txt

# Combine options
fasthog /path/to/repository --types=yml,yaml,env,tf --output=results.txt

Running from source

go run fasthog.go /path/to/repository

Examples

# Scan Python and C# files
fasthog ~/projects/myapp --types=py,cs --output=secrets_report.txt

# Scan infrastructure files
fasthog ~/terraform --types=tf,yml,yaml,env

# Scan entire codebase with default extensions
fasthog ~/repositories/myproject

JSON Output

Fasthog can produce machine-readable JSON output suitable for CI pipelines and tooling.

# Write JSON results to a file
fasthog /path/to/repository --format=json --output=results.json

# Equivalent shorthand
fasthog /path/to/repository --json --output=results.json

The JSON structure includes fields for the scanned directory, extensions, matches, a summary, and top files by match count. A truncated example:

{
  "directory": "/path/to/repository",
  "extensions": [".py", ".js"],
  "summary": {
    "total_matches": 3,
    "total_files_with_matches": 2,
    "total_files_scanned": 127
  }
}

Configuration File

Fasthog supports an optional configuration file in the current working directory named fasthog.yaml, or a custom path supplied via --config.

Example fasthog.yaml:

# Directory is optional; the CLI argument remains the primary source
# directory: ./

extensions:
  - .py
  - .js

exclude_dirs:
  - build
  - vendor

output:
  format: json
  path: results.json

patterns:
  fast: custom_fast_patterns.regex
  strict: custom_strict_patterns.regex
  exclude: custom_exclude_patterns.regex

Precedence rules:

• CLI flags override config file values.
• Config file values override built-in defaults.
• Additional exclude_dirs entries extend the built-in excluded directories (such as .git and node_modules).

Use --config to select a specific configuration file:

fasthog /path/to/repository --config=/path/to/fasthog.yaml

Example Output

Directory: /path/to/repository
Extensions: [.py .js .yml .yaml .env]
Output: results.txt
Current file: config/database.yml  Matches: 3
███████████████████████████████████ 100%

Results:
config/database.yml:0012 password: "mySecretPassword123"
src/api/keys.js:0045 const API_KEY = "sk_live_1234567890abcdef"
.env:0003 DATABASE_URL=postgres://user:pass@localhost/db

Completed in 1.2s: 3 matches across 3 of 127 files
Results written to results.txt

Architecture

Scanning Process

Fasthog uses a two-stage detection pipeline:

1. Fast screening: Initial pass with lightweight patterns (fast_patterns.regex)
2. Strict validation: Thorough analysis with comprehensive patterns (strict_patterns.regex)
3. False positive filtering: Exclusion of known benign patterns (exclude_patterns.regex)

Performance Optimizations

• Precompiled patterns: Regex patterns are compiled once at startup
• Concurrent processing: Parallel file scanning using worker pools (up to runtime.NumCPU() workers)
• Streaming I/O: Line-by-line buffered reading to optimize memory usage
• In-memory processing: No temporary files or external process invocations

Pattern Files

Fasthog uses multiple regex pattern files for flexible detection:

Customizing Patterns

Edit the .regex files to add or modify detection patterns. Each file contains one regex pattern per line. Empty lines and lines starting with # are ignored.

Testing

Running Tests

# Run all tests
go test ./...

# Run tests with coverage
make test-coverage

# Run benchmarks
make bench

Test Cases

The test suite validates pattern accuracy using:

• test/Positives.txt: Known secrets that must be detected
• test/False_Positives.txt: Benign patterns that should not trigger alerts

Adding Test Cases

For false positives:

1. Add the pattern to test/False_Positives.txt
2. Update exclude_patterns.regex to filter it
3. Run go test to verify

For missed secrets:

1. Add the secret to test/Positives.txt
2. Update fast_patterns.regex and/or strict_patterns.regex
3. Run go test to verify detection

Comparison with TruffleHog

A gap analysis tool is provided in the test directory to compare results with TruffleHog:

# Generate TruffleHog results
trufflehog filesystem ~/repositories --json --concurrency=36 > trufflehog.json

# Generate fasthog results
fasthog ~/repositories --output=fasthog.txt

# Compare results
cd test
go run secrets_gap_analysis.go -t ../trufflehog.json -p ../fasthog.txt

Note: Fasthog does not currently parse compressed archives (.zip, .tar.gz, etc.).

Development

Building

# Build binary
make build

# Install locally
make install

# Run linters
make lint

# Format code
make fmt

# Run all checks
make check

Project Structure

.
├── fasthog.go              # Main application
├── *_test.go               # Test files
├── *.regex                 # Pattern definition files
├── test/
│   ├── Positives.txt       # Known secrets for testing
│   ├── False_Positives.txt # Benign patterns for testing
│   └── secrets_gap_analysis.go  # TruffleHog comparison tool
├── .github/workflows/      # CI/CD configuration
└── Makefile                # Build automation

Contributing

Contributions are welcome. Please:

1. Add test cases for new patterns
2. Ensure all tests pass (go test ./...)
3. Run linters (make lint)
4. Update documentation as needed

Code Coverage

Secrets-in-Source maintains comprehensive test coverage with ongoing improvements. The coverage visualization below shows detailed coverage by module:

What this means:

• Green: Well-tested code (>80% coverage)
• Yellow: Moderate coverage (60-80%)
• Red: Needs more tests (<60%)
• Size: Larger boxes = more lines of code

Click the image to explore detailed coverage reports on Codecov, including line-by-line coverage, branch coverage, and historical trends.

License

This project is licensed under the MIT License. See the LICENSE file for details.

Acknowledgments

Originally developed as a bash script, migrated to Go in collaboration with danielgtaylor.