Why Go
- Case Studies
  
  Common problems companies solve with Go
- Use Cases
  
  Stories about how and why companies use Go
- Security
  
  How Go can help keep you secure by default
Learn
Docs
- Effective Go
  
  Tips for writing clear, performant, and idiomatic Go code
- Go User Manual
  
  A complete introduction to building software with Go
- Standard library
  
  Reference documentation for Go's standard library
- Release Notes
  
  Learn what's new in each Go release
Packages
Community
- Recorded Talks
  
  Videos from prior events
- Meetups
  
  Meet other local Go developers
- Conferences
  
  Learn and network with Go developers from around the world
- Go blog
  
  The Go project's official blog.
- Go project
  
  Get help and stay informed from Go
- Get connected

utils

package

Go to main page

Versions in this module

v1

v1.1.0

Aug 18, 2025

v1.0.0

Aug 16, 2025

v0

v0.0.0

Aug 18, 2025

Changes in this version

+ func CheckAbility(visited map[string]struct{}, rules Rules, normURL string) bool

+ func CheckDomain(domain *url.URL, rawURL string) (bool, error)

+ func GetHTML(rawURL string) ([]byte, error)

+ func GetRobots(rawURL string) ([]byte, error)

+ func Normalize(rawURL string) (string, error)

+ type Queue []string

+ func (q *Queue) CheckEmpty() bool

+ func (q *Queue) Dequeue() (string, error)

+ func (q *Queue) Enqueue(url string)

+ func (q *Queue) Peek() (string, error)

+ func (q *Queue) Size() int

+ type QueueOps interface

+ Dequeue func() (string, error)

+ Empty func() bool

+ Enqueue func(string)

+ Peek func() (string, error)

+ Size func() int

+ type Response struct

+ Content []string

+ Links []string

+ Title string

+ func ParseHTML(domain *url.URL, page []byte) (Response, error)

+ type Rules struct

+ Allowed []string

+ Delay int

+ Disallowed []string

+ func ParseRobots(normURL string, textFile []byte) (Rules, error)