extensions

package

v1.2.0 Latest Latest Go to latest Published: Feb 13, 2019 License: Apache-2.0 Imports: 3 Imported by: 0

Details

Valid go.mod file
Redistributable license
Tagged version
Stable version
Learn more about best practices

Repository

github.com/darjun/colly

Links

Open Source Insights

Documentation ¶

Overview ¶

Package extensions implements various helper addons for Colly

Index ¶

func RandomUserAgent(c *colly.Collector)
func Referer(c *colly.Collector)
func URLLengthFilter(c *colly.Collector, URLLengthLimit int)

Constants ¶

This section is empty.

Variables ¶

This section is empty.

Functions ¶

func RandomUserAgent ¶

func RandomUserAgent(c *colly.Collector)

RandomUserAgent generates a random browser user agent on every request

func Referer ¶ added in v1.2.0

func Referer(c *colly.Collector)

Referer sets valid Referer HTTP header to requests. Warning: this extension works only if you use Request.Visit from callbacks instead of Collector.Visit.

func URLLengthFilter ¶

func URLLengthFilter(c *colly.Collector, URLLengthLimit int)

URLLengthFilter filters out requests with URLs longer than URLLengthLimit

Types ¶

This section is empty.

Source Files ¶

View all Source files

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL