Documentation
¶
Overview ¶
Package scraper provides functionality to scrape web content and extract metadata.
This package offers a configurable web scraper that can extract content, metadata, and crawl websites while respecting rate limits and timeouts. It provides functionality to parse HTML, extract metadata like title and description, and collect all paths from a website.
Index ¶
- type Config
- type Metadata
- type Parser
- type Scraper
- func (s *Scraper) Crawl(baseURL, currentURL *url.URL, paths, visited map[string]bool, depth int) error
- func (s *Scraper) GetAllPaths() error
- func (s *Scraper) GetContent() error
- func (s *Scraper) GetMetadata() error
- func (s *Scraper) GetSubPathHTMLContent() map[string]string
- func (s *Scraper) GetSubPathMarkdownContent() map[string]string
- func (s *Scraper) ScrapeContent() (string, error)
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Config ¶
type Config struct {
// UserAgent is the User-Agent header value sent with HTTP requests
UserAgent string
// Timeout specifies the maximum duration to wait for an HTTP request to complete
Timeout time.Duration
// MaxDepth defines how deep the crawler will follow links from the starting URL
// A value of 0 means only the starting page, 1 means the starting page and all directly linked pages, etc.
MaxDepth int
// RequestDelay specifies the minimum time between requests to the same host
// This helps prevent overwhelming servers with too many rapid requests
RequestDelay time.Duration
// MaxConcurrent limits the total number of concurrent HTTP requests
// This applies across all hosts being scraped
MaxConcurrent int
// Verbose enables verbose output with ASCII graphics
Verbose bool
}
Config holds configuration options for the scraper.
This struct allows customization of the scraper's behavior including request parameters, timeouts, crawling depth, and rate limiting settings.
func DefaultConfig ¶
func DefaultConfig() *Config
DefaultConfig returns a default configuration with reasonable values.
The default configuration includes a standard user agent, reasonable timeout, moderate crawl depth, and conservative rate limiting to be respectful to websites.
Returns:
- A Config struct with default values
type Metadata ¶
type Metadata struct {
// Title is the content of the <title> tag
Title string
// Description is the content of the meta description tag
Description string
}
Metadata holds metadata information extracted from a webpage.
This struct stores the extracted title and description from a web page's HTML content.
type Scraper ¶
type Scraper struct {
// URL is the base URL to scrape
URL string
// Metadata contains extracted metadata from the scraped content
Metadata Metadata
// Content holds the parsed HTML content
Content *html.Node
// SubPaths contains all discovered paths during crawling
SubPaths []string
// Config contains all the configuration options for this scraper
Config *Config
// SubPathsHTMLContent stores the HTML content of each subpath
SubPathsHTMLContent map[string]string
// SubPathsMarkdownContent stores the Markdown content of each subpath
SubPathsMarkdownContent map[string]string
// Verbose enables verbose output
Verbose bool
// contains filtered or unexported fields
}
Scraper is responsible for scraping web content.
It handles fetching web pages, extracting content and metadata, and crawling websites while respecting rate limits and timeouts.
func New ¶
New creates a new scraper with the given URL and configuration.
This is the recommended way to create a new Scraper instance. It initializes all necessary internal structures and the HTTP client. If config is nil, default configuration will be used.
Parameters:
- url: The base URL to scrape
- config: The configuration to use for this scraper (or nil for defaults)
Returns:
- A new Scraper instance ready to use
func (*Scraper) Crawl ¶
func (s *Scraper) Crawl(baseURL, currentURL *url.URL, paths, visited map[string]bool, depth int) error
Crawl recursively crawls a website starting from the given URL.
This method implements a depth-first crawl of the website, following links within the same host up to the configured maximum depth. It respects rate limiting settings and tracks visited URLs to avoid cycles.
Parameters:
- baseURL: The original base URL of the website
- currentURL: The current URL being crawled
- paths: A map to collect all unique paths found
- visited: A map of already visited URLs to avoid duplicates
- depth: The current crawl depth (0 for the starting URL)
Returns:
- An error if the crawling fails catastrophically (individual page errors are logged but don't stop the crawl)
func (*Scraper) GetAllPaths ¶
GetAllPaths crawls the website and collects all paths.
This method performs a depth-first crawl of the website starting from the base URL. It respects the MaxDepth configuration and only follows links within the same host. The method stores all discovered paths in the SubPaths field.
Returns:
- An error if the crawling fails, nil otherwise
func (*Scraper) GetContent ¶
GetContent fetches the content of the URL and parses it.
This method retrieves the HTML content from the scraper's URL and parses it. The method applies rate limiting and respects the configured timeout.
Returns:
- An error if the content cannot be fetched or parsed, nil otherwise
func (*Scraper) GetMetadata ¶
GetMetadata extracts metadata (title, description) from the HTML content.
This method extracts the title and description from the HTML content. It requires that GetContent has been called first to populate the Content field.
Returns:
- An error if the content hasn't been fetched yet, nil otherwise
func (*Scraper) GetSubPathHTMLContent ¶
GetSubPathHTMLContent returns the HTML content of a subpath.
func (*Scraper) GetSubPathMarkdownContent ¶
GetSubPathMarkdownContent returns the Markdown content of a subpath page
func (*Scraper) ScrapeContent ¶
ScrapeContent fetches the URL and scrapes the main content.
This method fetches the content of the scraper's URL and extracts the main text content from the page. It utilizes the existing scraper infrastructure for rate limiting and error handling.
Returns:
- The extracted text content or an error if scraping fails
Source Files