dns-toolkit

DNS Toolkit

A command-line utility for downloading, processing, resolving, and consolidating DNS blocklists and allowlists from multiple sources. Performs DNS-to-IP resolution, reverse lookups, and overlap detection. Generates ready-to-use lists for DNS sinkholes, Pi-hole, AdGuard Home, and other network security tools.

Pipeline Flow

---
config:
  theme: mc
---
flowchart LR
    B["Process<br>via Processors"] --> C["Consolidate<br>Group<br>Categorize<br>Top Entries<br>Overlaps"]
    C --> D["Blocklist/Allowlist Files"] & E["Main README"] & F["Overlap Analysis"] & G["Summary Report"] & H["Statistics Report"] & n2["Format Conversion<br>via Converters"] & n4["Graph Generation<br>via Graphers"]
    D --> I["Archive"]
    E --> I
    F --> I
    H --> I
    G --> I
    n1["Data files"] --> B
    A["Download<br>via Downloaders"] --> n1
    n2 --> n3["Converted Files"]
    n4 --> n5["Graph Images"]
    n3 --> I
    n5 --> I
    B@{ shape: procs}
    C@{ shape: h-cyl}
    D@{ shape: docs}
    n2@{ shape: h-cyl}
    n4@{ shape: paper-tape}
    n1@{ shape: docs}
    n3@{ shape: docs}
    style C stroke-width:4px,stroke-dasharray: 0
    style n2 stroke-width:2px,stroke-dasharray: 2

Published Outputs

Ready-to-use blocklist files are published daily to the output branch:

• Domain and IP blocklists/allowlists compatible with Pi-hole, pfBlockerNG, AdGuard Home
• Lists organized by size (mini, lite, normal, big) and category (ads, malware, privacy)
• Top entries based on source frequency for high-confidence blocking

⚡ Quick Usage: Add https://raw.githubusercontent.com/phani-kb/dns-toolkit/output/[filename] to your DNS filtering tool (Pi-hole, AdGuard Home, pfBlockerNG).

Most commonly used outputs; Click any link to open the file.

View Detailed Overlap Analysis Comprehensive analysis showing how entries are shared across different DNS sources.

Conflicts Report (allowlist vs. blocklist) A daily-generated report listing entries found in both allowlists and blocklists, including the source(s) where they were found.

Processing summaries and metadata are archived in the summaries branch with 1-year retention.

Quick add (copy-ready URLs)

Copy one of these raw URLs directly into your DNS filtering tool.

Pi-hole (domain lists)

https://raw.githubusercontent.com/phani-kb/dns-toolkit/output/domain_blocklist.txt

AdGuard Home (AdGuard format)

https://raw.githubusercontent.com/phani-kb/dns-toolkit/output/adguard_blocklist.txt

pfBlockerNG (pfSense)

Use the 'DNSBL' > 'DNSBL Groups' section and add a custom list using the raw URL. Example domain list (paste into pfBlockerNG custom list):

https://raw.githubusercontent.com/phani-kb/dns-toolkit/output/domain_blocklist.txt

Branch Sizes

Note: The repo size badge above only reflects the default branch (release/1.0.0).

• Output branch size: 368.37 MB
• Summaries branch size: 6.03 MB

Source Statistics

Automatically generated statistics from source configuration files

Credits

This project is made possible by the following blocklist and allowlist sources:

Legend: S = Status, C/U/X = Count / Unique / Conflicts

Source Configuration (Important!)

Sources are configured in data/config/sources*.json files. Each source specifies:

• Download URL and frequency
• Source type (domain, IPv4, IPv6, AdGuard, etc.)
• Categories (ads, malware, privacy, etc.)
• License and website information

Special Note on Top Domains (tranco-list.eu)

Top domains sourced from the tranco-list.eu list (domain_top type) are treated as an allowlist.

Allowlist Generation Flow

---
config:
  theme: mc
---
flowchart LR
 subgraph horizontal[" "]
    direction LR
        OutputIPv4["allowlist_ipv4.txt"]
  end
    ReadSources["Read Sources"] --> ExtractDomains["Extract Domains from Source URLs"]
    ExtractDomains --> LoadCustom["Load Custom Files"]
    LoadCustom --> Combine["Combine & Deduplicate"]
    Combine --> OutputDomains["allowlist_domains.txt"] & OutputAdGuard["allowlist_adg.txt<br>(@@||domain^ format)"] & ResolveDNS["Resolve to IPv4"]
    ResolveDNS --> OutputIPv4

    OutputIPv4@{ shape: doc}
    ReadSources@{ shape: procs}
    ExtractDomains@{ shape: procs}
    OutputDomains@{ shape: doc}
    OutputAdGuard@{ shape: doc}
    ResolveDNS@{ shape: proc}
    style Combine stroke-width:2px,stroke-dasharray: 2
    style horizontal fill:transparent,stroke:transparent

Installation

git clone https://github.com/phani-kb/dns-toolkit.git
cd dns-toolkit
go build -o bin/dns-toolkit main.go

Quick Start

# Download and process all sources
dns-toolkit download
dns-toolkit process
dns-toolkit consolidate
dns-toolkit generate output

# Search for a domain
dns-toolkit search example.com

# Analyze overlaps between sources
dns-toolkit overlap

# Find top entries across sources
dns-toolkit top

Key Commands

DNS Toolkit

Usage:
  dns-toolkit [command]

Available Commands:
  archive          Archive DNS toolkit data
  consolidate      Consolidate processed files
  download         Download enabled sources
  generate         Generate different types of outputs
  help             Help about any command
  overlap          Find overlap between source files
  process          Process downloaded files
  search           Search for a domain or IP in the processed files
  sts              Prints the source types summary
  top              Find top entry(s) in each generic source type
  validate-sources Validate the sources configuration
  version          Print the version number of DNS Toolkit

Flags:
  -h, --help   help for dns-toolkit

Use "dns-toolkit [command] --help" for more information about a command.

Output Structure

data/output/
├── *_blocklist.txt    # Blocklists for various source types (adguard, domain, ipv4, ipv6, cidr)
├── *_allowlist.txt    # Allowlists for various source types (adguard, domain, etc.)
├── categories/        # Lists by category (ads, malware, privacy, etc.)
├── groups/            # Lists by size (mini, lite, normal, big)
├── top/               # Top entries based on source frequency
└── summaries/         # Processing metadata and statistics

Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Submit a pull request

Issues

If you encounter a bug, have a feature request, or want to suggest an improvement, please open an issue on the GitHub Issues page.

License

This project is licensed under the terms specified in the LICENSE file.