Documentation
¶
Overview ¶
Package robots parses robots.txt per RFC 9309 and answers the one question the frontier asks: may meguri fetch this path?
The parser reduces a whole file to a single Rules value already specialized to meguri's user agent. It picks the most specific matching group, keeps that group's Allow and Disallow rules and Crawl-delay, and gathers the file-global Sitemap lines. Everything else is dropped.
The matcher follows the Google lineage that RFC 9309 standardized: longest pattern match wins, an Allow breaks a tie against an equally long Disallow, and only the two metacharacters * and $ are special.
Index ¶
Constants ¶
const MaxBytes = 500 << 10
MaxBytes is the RFC 9309 parse cap: a crawler must parse at least the first 500 KiB of a robots.txt and may stop after that. We stop exactly there so a hostile or runaway file cannot cost us more.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Rules ¶
type Rules struct {
// contains filtered or unexported fields
}
Rules is meguri's compact parsed form of one robots.txt, already reduced to the single group that applies to meguri's user agent. A nil *Rules means allow-all: no file, an empty file, or no matching group.
func Parse ¶
Parse parses a robots.txt body and returns the rules for userAgent, matched case-insensitively as a prefix and falling back to the * group. It strips a leading UTF-8 BOM, tolerates malformed lines, and reads at most MaxBytes. A nil result means allow-all.
func (*Rules) Allowed ¶
Allowed reports whether path may be fetched. path is a percent-decoded URL path such as "/a/b?x=1". A nil receiver allows everything. The decision is the longest-match rule, with an Allow winning a tie against an equally long Disallow.
func (*Rules) CrawlDelay ¶
CrawlDelay returns the group's Crawl-delay, or 0 when none was published.
Source Files