acurl

module

v1.0.0 Latest Latest Go to latest Published: Oct 19, 2026 License: Apache-2.0

Details

Valid go.mod file
Redistributable license
Tagged version
Stable version
Learn more about best practices

Repository

github.com/tragent-io/acurl

Links

Open Source Insights

README ¶

acurl

See your website the way AI agents see it.
acurl.dev · Documentation · Command reference · tragent.io

acurl fetches a page as a real Chrome browser and as ChatGPT, Claude, Perplexity, Googlebot and 18 other agents. It shows which ones were blocked, challenged, cloaked or shown a different price, scores how ready the page is for AI agents, and tells you how to fix what fails.

It's a free, open-source tool by tragent.io, built for online stores and useful on any website.

$ acurl shop.example/products/wool-runner
Fetching https://shop.example/products/wool-runner with 22 profiles...

PROFILE            STATUS  TIME    RESULT             DETAILS
Chrome (Browser)   200     412ms   allowed            Wool Runner | Example Store
ChatGPT-User       403     51ms    challenged         challenge: Cloudflare
Claude-User        200     388ms   allowed            OK
Perplexity-User    200     402ms   allowed            product: price 89.00 vs browser 98.00
OAI-SearchBot      200     97ms    cloaked            31% similar to browser
GPTBot             200     365ms   robots-disallowed  robots.txt: disallowed
Google-Extended    -       -       robots-disallowed  robots.txt token (not fetched)

AI readiness 71/100 (C)
  ✗ AI assistants can fetch the page            16.7/25  ChatGPT-User (challenged)
  ✗ AI search crawlers can fetch the page       10.0/20  OAI-SearchBot (cloaked)
  ✓ Content is in the HTML without JavaScript   15.0/15
  ✓ Product structured data is present          10.0/10

How to improve
  1. AI assistants can fetch the page
     Allow ChatGPT-User, Claude-User and Perplexity-User in your WAF or bot-management rules...

Install


Homebrew	`brew install tragent-io/tap/acurl`
Shell	`curl -fsSL https://acurl.dev/install.sh \| sh`
Go	`go install github.com/tragent-io/acurl/cmd/acurl@latest`
npm	`npx @tragent/acurl <url>`
pip	`pipx install acurl`
Docker	`docker run --rm ghcr.io/tragent-io/acurl <url>`
Binaries	GitHub Releases, with checksums, SBOMs and cosign signatures

What it checks

22 agent profiles in five categories: AI assistants (ChatGPT-User, Claude-User, Perplexity-User, ...), AI search (OAI-SearchBot, Claude-SearchBot, PerplexityBot, ...), AI training (GPTBot, ClaudeBot, CCBot, Google-Extended, ...), search engines, and a curl baseline
A real browser baseline with Chrome's TLS and HTTP/2 fingerprint. Bot profiles send only their own headers, like real crawlers.
WAF challenges from Cloudflare, Akamai, DataDome, PerimeterX, Imperva, Kasada, AWS WAF, Sucuri and CAPTCHAs
Cloaking: agents shown substantially different visible text than the browser, with --diff to see what's missing
robots.txt (RFC 9309), Content Signals, X-Robots-Tag, meta robots and llms.txt
Product data parity: schema.org Product from JSON-LD, microdata or OpenGraph, and agents shown a different price or stock
JavaScript dependence: client-rendered shells, and --render to measure JavaScript-only text in headless Chrome
An AI readiness score (0-100) from documented checks, with a fix for each failure

Commands

Command
`acurl <url>`	Check a page with every agent and score it
`acurl robots <url>`	robots.txt verdict per agent, Content Signals, sitemaps and llms.txt
`acurl batch [file]`	URL lists, CSV, stdin or whole sitemaps (`--sitemap auto:shop.example`)
`acurl watch <url>...`	Re-check on a schedule and alert Slack or a webhook on changes
`acurl report <file>`	Render saved JSON as HTML, Markdown, SARIF, JUnit or CSV
`acurl serve`	Local web UI and JSON API
`acurl mcp`	MCP server so Claude, Cursor and other assistants can run checks
`acurl keygen`	Ed25519 key for Web Bot Auth request signing
`acurl profiles list`	The agent catalog
`acurl init`	Create a `.acurl.yaml` config file

Every flag is in the command reference.

In CI

- uses: tragent-io/acurl@v1
  with:
    urls: |
      https://staging.shop.example/
      https://staging.shop.example/products/best-seller
    fail-on: blocked,challenged,score<80

Output formats: table, json, markdown, html, sarif, junit and csv. Exit codes follow --fail-on. See the CI guide.

With your AI assistant

claude mcp add acurl -- acurl mcp

Then ask "Can ChatGPT read our new product page?" See MCP setup for Claude Desktop and Cursor.

As a Go library

c, _ := acurl.New(acurl.Options{BlockPrivate: true})
rep, _ := c.Check(ctx, "https://shop.example/products/1")
fmt.Println(rep.Score.Value)

Caveat

acurl sends agent user-agent strings from your own IP address. Real AI agents come from published IP ranges and may sign their requests, so results show how a site responds to agent user agents, not guaranteed agent access. It's the fastest first check, and it catches most real-world blocking.

Contributing

Bug reports, new agent profiles and detection signatures are welcome. See CONTRIBUTING.md. Security issues: SECURITY.md.

License

Apache 2.0. Copyright tragent. acurl is built and maintained by tragent.io, which helps online stores get found, understood and bought from by AI shopping agents.

Directories ¶

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL

Path	Synopsis
cmd
acurl command Command acurl shows how AI agents see a URL compared to a real browser.	Command acurl shows how AI agents see a URL compared to a real browser.
examples
library command Command library shows how to use acurl as a Go library: check a list of pages with AI assistant profiles only and print the ones that fail.	Command library shows how to use acurl as a Go library: check a list of pages with AI assistant profiles only and print the ones that fail.
internal
batch Package batch runs acurl checks over many URLs.	Package batch runs acurl checks over many URLs.
classifier Package classifier labels each fetch result with how the site treated it.	Package classifier labels each fetch result with how the site treated it.
cli Package cli implements the acurl command line interface.	Package cli implements the acurl command line interface.
config Package config applies settings from .acurl.yaml files and ACURL_* environment variables to command flags.	Package config applies settings from .acurl.yaml files and ACURL_* environment variables to command flags.
detect Package detect recognises bot-challenge and WAF block pages.	Package detect recognises bot-challenge and WAF block pages.
extract Package extract pulls schema.org product data out of HTML pages, so acurl can show whether an agent was given the price and availability a browser sees.	Package extract pulls schema.org product data out of HTML pages, so acurl can show whether an agent was given the price and availability a browser sees.
fetcher Package fetcher retrieves a URL once per agent profile.	Package fetcher retrieves a URL once per agent profile.
llmstxt Package llmstxt finds and validates /llms.txt files (https://llmstxt.org).	Package llmstxt finds and validates /llms.txt files (https://llmstxt.org).
mcp Package mcp implements a Model Context Protocol server over stdio that exposes acurl checks as tools for AI assistants.	Package mcp implements a Model Context Protocol server over stdio that exposes acurl checks as tools for AI assistants.
netguard Package netguard stops requests to private and internal networks.	Package netguard stops requests to private and internal networks.
output Package output renders acurl reports in human and machine formats.	Package output renders acurl reports in human and machine formats.
profiles Package profiles loads and filters the agent profiles acurl fetches with.	Package profiles loads and filters the agent profiles acurl fetches with.
render Package render estimates how much of a page depends on JavaScript, which most AI agents do not execute.	Package render estimates how much of a page depends on JavaScript, which most AI agents do not execute.
robots Package robots fetches robots.txt and answers RFC 9309 access questions.	Package robots fetches robots.txt and answers RFC 9309 access questions.
score Package score turns an acurl report into an AI-readiness score from 0 to 100, with the checks behind it.	Package score turns an acurl report into an AI-readiness score from 0 to 100, with the checks behind it.
server Package server implements acurl serve: a local web UI and JSON API.	Package server implements acurl serve: a local web UI and JSON API.
sign Package sign implements HTTP Message Signatures (RFC 9421) with Ed25519, as used by the Web Bot Auth proposal for cryptographically identifying automated clients.	Package sign implements HTTP Message Signatures (RFC 9421) with Ed25519, as used by the Web Bot Auth proposal for cryptographically identifying automated clients.
textdiff Package textdiff compares what different agents were shown on a page.	Package textdiff compares what different agents were shown on a page.
watch Package watch detects changes in how a site treats AI agents between runs.	Package watch detects changes in how a site treats AI agents between runs.
pkg
acurl Package acurl is the Go API behind the acurl command: it fetches a URL once per agent profile and reports how each agent was treated.	Package acurl is the Go API behind the acurl command: it fetches a URL once per agent profile and reports how each agent was treated.
types Package types holds the data structures shared across acurl packages.	Package types holds the data structures shared across acurl packages.