Documentation
¶
Overview ¶
Package robots provides robots.txt validation functionality.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Checker ¶
type Checker struct {
// contains filtered or unexported fields
}
Checker handles robots.txt validation for web crawling
func NewChecker ¶
NewChecker creates a new robots.txt checker
Source Files
Click to show internal directories.
Click to hide internal directories.