README
¶
acurl
See your website the way AI agents see it.
acurl fetches a URL several times, once per agent profile: a real Chrome browser, AI agents
such as ChatGPT-User and Claude-Web, search crawlers, and plain curl. It then shows how each one was
treated side by side.
It is built for store owners and developers who want to know whether AI shopping and search agents can actually read their pages.
Install
go install github.com/tragent-io/acurl/cmd/acurl@latest
Usage
acurl https://example.com
acurl --agents ChatGPT-User,Googlebot https://yourstore.com/products/1
acurl --category ai-assistant,ai-search https://yourstore.com
acurl --diff GPTBot https://yourstore.com # what text did GPTBot not see?
acurl robots https://yourstore.com/products/1 # robots.txt, content signals, llms.txt
acurl profiles list
Each result is classified as allowed, blocked, challenged (WAF or captcha page),
rate-limited, cloaked (visible text differs a lot from what Chrome got),
robots-disallowed or error.
More: profiles · robots.txt and llms.txt
Output formats: --format table (default), json or markdown. Use -o report.md to write to a file.
Exit codes
| Code | Meaning |
|---|---|
| 0 | Every profile was allowed |
| 1 | At least one profile was blocked or challenged |
| 2 | A request failed |
Status
Early development. Expect breaking changes until v1.0.
License
Apache 2.0. See LICENSE.
Directories
¶
| Path | Synopsis |
|---|---|
|
cmd
|
|
|
acurl
command
Command acurl shows how AI agents see a URL compared to a real browser.
|
Command acurl shows how AI agents see a URL compared to a real browser. |
|
internal
|
|
|
classifier
Package classifier labels each fetch result with how the site treated it.
|
Package classifier labels each fetch result with how the site treated it. |
|
cli
Package cli implements the acurl command line interface.
|
Package cli implements the acurl command line interface. |
|
detect
Package detect recognises bot-challenge and WAF block pages.
|
Package detect recognises bot-challenge and WAF block pages. |
|
fetcher
Package fetcher retrieves a URL once per agent profile.
|
Package fetcher retrieves a URL once per agent profile. |
|
llmstxt
Package llmstxt finds and validates /llms.txt files (https://llmstxt.org).
|
Package llmstxt finds and validates /llms.txt files (https://llmstxt.org). |
|
output
Package output renders acurl reports in human and machine formats.
|
Package output renders acurl reports in human and machine formats. |
|
profiles
Package profiles loads and filters the agent profiles acurl fetches with.
|
Package profiles loads and filters the agent profiles acurl fetches with. |
|
robots
Package robots fetches robots.txt and answers RFC 9309 access questions.
|
Package robots fetches robots.txt and answers RFC 9309 access questions. |
|
textdiff
Package textdiff compares what different agents were shown on a page.
|
Package textdiff compares what different agents were shown on a page. |
|
pkg
|
|
|
types
Package types holds the data structures shared across acurl packages.
|
Package types holds the data structures shared across acurl packages. |