Why Go
- Case Studies
  
  Common problems companies solve with Go
- Use Cases
  
  Stories about how and why companies use Go
- Security
  
  How Go can help keep you secure by default
Learn
Docs
- Effective Go
  
  Tips for writing clear, performant, and idiomatic Go code
- Go User Manual
  
  A complete introduction to building software with Go
- Standard library
  
  Reference documentation for Go's standard library
- Release Notes
  
  Learn what's new in each Go release
Packages
Community
- Recorded Talks
  
  Videos from prior events
- Meetups
  
  Meet other local Go developers
- Conferences
  
  Learn and network with Go developers from around the world
- Go blog
  
  The Go project's official blog.
- Go project
  
  Get help and stay informed from Go
- Get connected

robotstxt

package module

Go to main page

Versions in this module

v1

v1.1.2

Mar 31, 2021

v1.1.1

Jul 3, 2019

v1.1.0

Jul 3, 2019

Changes in this version

+ var WhitespaceChars = []rune

+ type Group struct

+ Agent string

+ CrawlDelay time.Duration

+ func (g *Group) Test(path string) bool

+ type ParseError struct

+ Errs []error

+ func (e ParseError) Error() string

+ type RobotsData struct

+ Host string

+ Sitemaps []string

+ func FromBytes(body []byte) (r *RobotsData, err error)

+ func FromResponse(res *http.Response) (*RobotsData, error)

+ func FromStatusAndBytes(statusCode int, body []byte) (*RobotsData, error)

+ func FromStatusAndString(statusCode int, body string) (*RobotsData, error)

+ func FromString(body string) (r *RobotsData, err error)

+ func (r *RobotsData) FindGroup(agent string) (ret *Group)

+ func (r *RobotsData) TestAgent(path, agent string) bool