gocrawl

gocrawl

A highly-customizable, free and open-source (FOSS) website crawler for SEO and SEA audits, written in Go.

gocrawl walks a website concurrently and runs a pipeline of pluggable analyzers over every page — checking technical SEO, redirects, broken links, robots.txt, sitemap.xml coverage, structured data, and more — then writes a JSON, CSV, or HTML report.

Status: early, working vertical slice. Raw-HTML crawling, the core SEO analyzers, the SEA analyzers (UTM auditing, tracking-pixel detection, landing-page relevance), the AI-search analyzers (Answer Engine and Generative Engine Optimization), and headless rendering with lab-mode Core Web Vitals (LCP, FCP, CLS, TBT, TTFB) are all implemented. The design's whole point is that checks slot in as new analyzers without touching the engine — see the Roadmap for what's next.

Why gocrawl

• Customizable by design. Every check is an independent analyzer you can enable, disable, or configure. Crawl scope (depth, page cap, include/exclude patterns, subdomains, rate limiting, robots compliance) is fully configurable.
• Fast & portable. Concurrent Go engine, single static binary, no runtime deps.
• SEO, SEA, and AI search. Ships with SEO/technical analyzers, SEA analyzers (UTM auditing, tracking-pixel detection, landing-page relevance), and AI-search analyzers (Answer Engine and Generative Engine Optimization) — each an independent check on the same interface.
• Reports you can pipe or share. JSON for tooling, CSV for spreadsheets, and a self-contained HTML page (inline CSS, no JS) for handing to stakeholders.

Install

Works on Windows, macOS, and Linux. With Go 1.26+:

go install github.com/Patience-dot-devl/gocrawl/cmd/gocrawl@latest

Or build from source:

git clone https://github.com/Patience-dot-devl/gocrawl
cd gocrawl
make build      # produces ./gocrawl  (on Windows: go build -o gocrawl.exe ./cmd/gocrawl)

📖 See docs/install.md for full per-platform instructions — PATH setup on each OS, building without make on Windows, verifying the install, and the optional Chromium browser needed for --render headless.

Quick start

# Crawl one level deep and write a JSON report
gocrawl crawl https://example.com --depth 1 --out report.json

# CSV instead, with a page cap and higher concurrency
gocrawl crawl https://example.com --max-pages 200 --concurrency 8 \
  --format csv --out report.csv

# Self-contained HTML report to open in a browser (with a visual site-map tab)
gocrawl crawl https://example.com --format html --out report.html

# Re-render a saved JSON report into HTML — no recrawl
gocrawl crawl  https://example.com --format json --out report.json
gocrawl render report.json --out report.html

# Only run specific analyzers
gocrawl crawl https://example.com --analyzers seo,links,redirects

# Save a crawl, then track progress over time
gocrawl crawl https://example.com --save           # store this crawl
gocrawl history                                     # list saved crawls
# ...fix some issues, recrawl with --save, then:
gocrawl compare example.com/<earlier> latest        # what's new / resolved
gocrawl compare before.json after.json --fail-on-new  # CI gate on regressions

# List every available analyzer
gocrawl analyzers list

# Write a fully-commented example config you can edit
gocrawl init
gocrawl crawl https://example.com --config gocrawl.yaml

# Verify a HubSpot redirect-rule export against the live site
gocrawl check-redirects --input redirects.csv --domain example.com --output results.csv

Documentation

Full reference docs live in docs/:

• Configuration — every option, flag, env var, and default.
• Analyzers — what each analyzer checks, with every issue code.
• Output / report — the JSON, CSV, and HTML report formats.
• MCP server — running as an MCP server and the tool schemas.
• Web UI — running gocrawl serve, the browser UI, and the REST API.
• Redirect-rule verification — checking a redirect-rule CSV export against a live site.
• Architecture — how the engine and analyzer pipeline fit together.
• Roadmap — what's shipped, stubbed, and planned.

Use as an MCP server (Conductor / Claude Code)

gocrawl can run as a Model Context Protocol server over stdio, so agentic tools can drive crawls directly:

gocrawl mcp

It exposes two tools:

• crawl — run a crawl + analysis and return a structured JSON report. Arguments: url (required), depth, max_pages, concurrency, render, analyzers, specialized, security_audit, respect_robots, subdomains, include, exclude.
• list_analyzers — list the available analyzers.

Register it with an MCP client. For example, Claude Code:

claude mcp add gocrawl -- gocrawl mcp

Or in a Conductor / Claude Code mcp config block:

{
  "mcpServers": {
    "gocrawl": { "command": "gocrawl", "args": ["mcp"] }
  }
}

The agent can then call crawl with {"url": "https://example.com", "depth": 2} and reason over the returned issues. See the full MCP server guide for tool schemas and examples.

Use as a web app

gocrawl serve

Starts an HTTP server on :8080 (override with --addr) exposing a REST API to start, poll, cancel, and export crawls, and serves the built-in browser UI at the same address — a form to start a crawl, a live-polling report view, and crawl history. See the full Web UI guide for the API reference and how to build the frontend from source.

Configuration

Configuration is layered, in increasing precedence: defaults → YAML config file → environment variables (GOCRAWL_*) → command-line flags. Generate a starting file with gocrawl init (see configs/example.yaml). The full option, env-var, and default reference is in docs/configuration.md.

Key crawl options:

Analyzers (v1)

The aeo direct-answer-lead and geo quotable-density checks, plus the wordpress security-endpoint probes, are opt-in specialized checks, off by default; enable them with --specialized. See docs/analyzers.md for every issue code, severity, and threshold.

Security audit (opt-in)

--security-audit extends the security analyzer with a transport- and cookie-level pass:

• TLS — negotiated protocol version and cipher suite (obsolete versions, insecure suites).
• Certificates — expiry (30-day warning, 14-day error), validity window, self-signed certificates, chains missing their intermediates, SHA-1/MD5 signatures, undersized keys.
• Cookies — Secure, HttpOnly on session cookies, SameSite, the __Host-/__Secure- name prefixes, and lifetimes past the 400-day browser cap.
• Response headers — HSTS quality (max-age, includeSubDomains), framing protection, Referrer-Policy, and software-version disclosure.

It is passive: it reads the crawl's own responses and opens no extra connections, sends no probes, and changes nothing about how the site is crawled. Because these findings describe server configuration rather than individual pages, they are reported once per host. TLS and certificate checks need --render raw (the default) — headless rendering does not expose the handshake.

gocrawl crawl https://example.com --security-audit

How it works

seed URL ──▶ crawler engine ──▶ Result (pages, redirects, robots) ──▶ analyzer pipeline ──▶ report (json/csv/html)
              (concurrent,                                              (seo, links,
               scope + robots,                                          sitemap, …)
               redirect capture)

The engine never knows about specific checks; analyzers never fetch the crawl. The analyze.Analyzer interface is the single seam between them — which is what makes new checks cheap to add. See CONTRIBUTING.md.

Roadmap

Recently shipped: headless rendering via chromedp, lab-mode Core Web Vitals (LCP, FCP, CLS, TBT, TTFB) in the perf analyzer, and the HTML report format. Next up: resumable crawls and export integrations. See the full feature roadmap for status on each.

License

MIT © Patience-dot-devl