Documentation
¶
Overview ¶
Package robots renders and serves /robots.txt (RFC 9309): groups of user agents with their allow and disallow rules, plus Sitemap lines. Nothing is served unless the app registers it:
err := robots.Register(r, robots.File{
Groups: []robots.Group{
{UserAgents: []string{"*"}, Disallow: []string{"/admin/", "/trap/"}},
},
Sitemaps: []string{"https://shop.example/sitemap.xml"},
NoIndex: app.Config.NoIndex, // staging: disallow everything
})
Values are single-line: a line break or control character in a user agent or rule is an error, so input cannot add rules.
Index ¶
Constants ¶
View Source
const DisallowAll = "User-agent: *\nDisallow: /\n"
DisallowAll is the body served when File.NoIndex is set.
Variables ¶
This section is empty.
Functions ¶
Types ¶
type File ¶
type File struct {
Groups []Group
// Sitemaps are absolute http(s) URLs written as Sitemap lines.
Sitemaps []string
// NoIndex replaces the groups with one that disallows everything for
// every crawler (a staging site); the sitemaps are left out too.
NoIndex bool
}
File describes robots.txt.
type Group ¶
type Group struct {
// UserAgents are the product tokens the group applies to ("*",
// "Googlebot"). At least one is required.
UserAgents []string
// Allow and Disallow are path rules: "/" followed by the path, with
// "*" and a final "$" as RFC 9309 wildcards; "#" (a comment) is
// refused. An empty Disallow entry
// writes "Disallow:", which allows everything.
Allow []string
Disallow []string
}
Group is one record of the file: rules for the crawlers it names.
Source Files
Click to show internal directories.
Click to hide internal directories.