Documentation
¶
Index ¶
- func Collector(ctx context.Context, url string, projectPath string, cookieJar *cookiejar.Jar, ...) error
- func Crawl(ctx context.Context, site string, projectPath string, cookieJar *cookiejar.Jar, ...) error
- func Extractor(link string, projectPath string)
- func HTMLExtractor(link string, projectPath string) error
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func Collector ¶
func Collector(ctx context.Context, url string, projectPath string, cookieJar *cookiejar.Jar, proxyString string, userAgent string) error
Collector searches for css, js, and images within a given link TODO improve for better performance
func Crawl ¶
func Crawl(ctx context.Context, site string, projectPath string, cookieJar *cookiejar.Jar, proxyString string, userAgent string) error
Crawl asks the necessary crawlers for collecting links for building the web page
func Extractor ¶
Extractor visits a link determines if its a page or sublink downloads the contents to a correct directory in project folder TODO add functionality for determining if page or sublink
func HTMLExtractor ¶
HTMLExtractor downloads the HTML content from a URL and saves it to index.html
Types ¶
This section is empty.
Source Files
Click to show internal directories.
Click to hide internal directories.