Documentation
¶
Overview ¶
Package robots fetches robots.txt and answers RFC 9309 access questions.
Index ¶
Constants ¶
const MaxSize = 500 * 1024
MaxSize is the largest robots.txt body that is parsed. RFC 9309 requires crawlers to parse at least 500 KiB.
Variables ¶
This section is empty.
Functions ¶
func Directives ¶ added in v0.2.0
Directives returns the indexing directives (noindex, nofollow, noai, ...) that apply to the robots.txt token agent, from X-Robots-Tag headers and <meta name="robots"> or <meta name="{agent}"> tags. Generic and agent-specific directives are merged without duplicates.
Types ¶
type Cache ¶
type Cache struct {
// contains filtered or unexported fields
}
Cache fetches robots.txt at most once per origin and is safe for concurrent use.
type Data ¶
type Data struct {
URL string `json:"url"`
Status int `json:"status"`
Content string `json:"-"`
// Err is set when robots.txt could not be fetched or returned 5xx.
Err error `json:"-"`
}
Data is the robots.txt state for one origin.
func (*Data) ContentSignals ¶ added in v0.2.0
ContentSignals returns the Content-Signal directives (https://contentsignals.org) that apply to agent, such as {"search": "yes", "ai-train": "no"}. It returns nil when there are none.
func (*Data) Unreachable ¶
Unreachable reports whether robots.txt failed with a network error or a server error. RFC 9309 §2.3.1.4 says crawlers must then assume full disallow.
Source Files