Documentation
¶
Overview ¶
Package aicrawl aggregates $ai_crawl events: the AI crawlers' own visits to the customer's site, recorded server-side.
This is the half of AI visibility that a JavaScript pixel physically cannot see. GPTBot, ClaudeBot and PerplexityBot execute no JavaScript, so they never fire a $pageview — every client-side analytics tool on the market reports zero of them, and the GEO monitors that would care never touch the customer's server. The fix is a few lines of edge middleware that report the hit from the request itself; the events land on the same instance as everything else, which is what lets the four stages of the loop finally sit on one log:
crawled → readable → named in the answer → sent a human who converted
Every incumbent holds exactly one of those.
Event contract (properties on $ai_crawl):
crawler string — normalized name, e.g. "GPTBot", "ClaudeBot" operator string — who runs it: "OpenAI", "Anthropic", "Perplexity", ... purpose string — "training" | "search" | "user" (see Purpose*) path string — request path, query stripped by the middleware status number — HTTP status the crawler received site string — hostname, same key the dashboard's site selector uses
Index ¶
Constants ¶
const ( PurposeTraining = "training" PurposeSearch = "search" PurposeUser = "user" )
The three reasons an AI company fetches a page, kept apart because they mean completely different things to the person reading the report:
- training: your words may end up in a future model. No traffic, no attribution.
- search: you are being indexed for answers. This is the one that feeds citations.
- user: a person asked an assistant about you RIGHT NOW and it fetched this page to answer them. That is a live human intent signal, not a robot.
Lumping them together is the mistake most crawler dashboards make: it turns the most valuable signal on the list into a rounding error inside "bot traffic".
const CrawlEvent = "$ai_crawl"
CrawlEvent is the event name the edge middleware writes. Exported because several packages must recognise it: ingest exempts it from the bot filter (it is BY DEFINITION a bot hit, reported by the server), and the billing/pulse counters exclude it so a crawl wave never reads as a traffic spike or an invoice.
Variables ¶
This section is empty.
Functions ¶
Types ¶
type CrawlerRow ¶
type CrawlerRow struct {
Crawler string `json:"crawler"`
Operator string `json:"operator"`
Purpose string `json:"purpose"`
Hits int `json:"hits"`
Pages int `json:"pages"` // distinct paths it fetched
Errors int `json:"errors"` // hits it received a 4xx/5xx for
LastAt string `json:"last_at"` // RFC3339, "" if never
LastPath string `json:"last_path"` // the newest path it fetched
}
CrawlerRow is one crawler's activity over the window.
type DayPoint ¶
type DayPoint struct {
Day string `json:"day"` // YYYY-MM-DD
Hits int `json:"hits"`
Training int `json:"training"`
Search int `json:"search"`
User int `json:"user"`
}
DayPoint is one day of the crawl trend, split by purpose so a training sweep never hides the fact that live user fetches went to zero.
type GapRow ¶
type GapRow struct {
Path string `json:"path"`
Views int `json:"views"` // human pageviews over the window
Visitors int `json:"visitors"`
}
GapRow is a page real visitors read that no AI crawler has ever fetched. The most actionable row in the module: it is a specific URL, it has proven human demand, and no assistant can cite what it has never read.
type OperatorRow ¶
type OperatorRow struct {
Operator string `json:"operator"`
Hits int `json:"hits"`
Pages int `json:"pages"`
Training int `json:"training"`
Search int `json:"search"`
User int `json:"user"`
LastAt string `json:"last_at"`
}
OperatorRow rolls the crawlers up to the company behind them — the level a reader actually thinks in ("has OpenAI read my site", not "has OAI-SearchBot").
type PathRow ¶
type PathRow struct {
Path string `json:"path"`
Hits int `json:"hits"`
Crawlers int `json:"crawlers"`
Errors int `json:"errors"`
LastAt string `json:"last_at"`
}
PathRow is one page's crawl coverage. Crawlers is the count of DISTINCT crawlers that fetched it, which is the number that matters: a page every operator has read is a page that can be cited by every assistant.
type Result ¶
type Result struct {
Days int `json:"days"`
Reporting bool `json:"reporting"` // any $ai_crawl event ever seen
Hits int `json:"hits"`
Pages int `json:"pages"`
// Probes are requests dropped as vulnerability scans wearing a bot user-agent (/.env,
// /@fs/etc/passwd and friends). Reported rather than silently discarded: an operator whose
// total quietly shrank is a number nobody can reconcile, and "we ignored 11 fake requests" is
// a more trustworthy line than a total that happens to be right.
Probes int `json:"probes"`
Crawlers []CrawlerRow `json:"crawlers"`
Operators []OperatorRow `json:"operators"`
Paths []PathRow `json:"paths"`
Trend []DayPoint `json:"trend"`
Gaps []GapRow `json:"gaps"`
Training int `json:"training"`
Search int `json:"search"`
User int `json:"user"`
Errors int `json:"errors"`
FirstAt string `json:"first_at"`
LastAt string `json:"last_at"`
Note string `json:"note,omitempty"`
}
Result is the whole report. Empty is a legitimate answer and Reporting says so — "no crawler has ever fetched this site" and "nobody has installed the reporter" look identical in the data and mean opposite things, so they are never conflated.
Source Files