Documentation
¶
Index ¶
- func CheckLightpanda(autoStart bool) error
- func CheckSolimen(autoStart bool) error
- func CloseSharedPlaywright()
- func HeadersFromParams(params map[string]any) map[string]string
- type AnonymizedScraper
- func (s *AnonymizedScraper) ClosePlaywright()
- func (s *AnonymizedScraper) HTTPClient() *http.Client
- func (s *AnonymizedScraper) ScrapeContent(url string, headers map[string]string) (dom *goquery.Selection, err error)
- func (s *AnonymizedScraper) ScrapeContentWithPlaywright(url string, customHeaders map[string]string) (*goquery.Selection, error)
- type ArticleExtractor
- type RSSFeedScraper
- type Scraper
- type TwitterScraper
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func CheckLightpanda ¶
CheckLightpanda checks if Lightpanda is reachable on port 9222. If not, it either prompts the user interactively or auto-starts the container. Set autoStart to true to skip the interactive prompt.
func CheckSolimen ¶
CheckSolimen checks if the Solimen service is reachable on port 5011. If not, it either prompts the user interactively or auto-starts the container.
func CloseSharedPlaywright ¶
func CloseSharedPlaywright()
CloseSharedPlaywright releases resources used by the shared Playwright instance
func HeadersFromParams ¶
HeadersFromParams extracts custom HTTP headers stored under params["headers"]. Values may arrive as map[string]string (in-process) or map[string]any (after a JSON/protobuf round-trip), so both shapes are handled. Returns nil when absent.
Types ¶
type AnonymizedScraper ¶
type AnonymizedScraper struct {
Collector *colly.Collector
// contains filtered or unexported fields
}
AnonymizedScraper wraps a Colly collector and implements anonymization on each request.
func GetSharedAnonymizedScraper ¶
func GetSharedAnonymizedScraper(domain string) *AnonymizedScraper
GetSharedAnonymizedScraper returns a shared AnonymizedScraper instance. If domain is provided, it updates the allowed domains.
func NewAnonymizedScraper ¶
func NewAnonymizedScraper(domain string) *AnonymizedScraper
NewAnonymizedScraper creates a new instance of AnonymizedScraper for the given domain.
func (*AnonymizedScraper) ClosePlaywright ¶
func (s *AnonymizedScraper) ClosePlaywright()
ClosePlaywright cleans up Playwright resources.
func (*AnonymizedScraper) HTTPClient ¶
func (s *AnonymizedScraper) HTTPClient() *http.Client
HTTPClient returns an *http.Client that carries this scraper's anon profile (rotating User-Agent from s.userAgents + spoofed headers + Alt-Used) on every request via anonRoundTripper. Unlike the colly path it follows redirects across hosts and imposes no domain allowlist, so it suits feed/RSS fetching.
func (*AnonymizedScraper) ScrapeContent ¶
func (s *AnonymizedScraper) ScrapeContent(url string, headers map[string]string) (dom *goquery.Selection, err error)
ScrapeContent visits the URL, processes the HTML content and returns the largest content block. Custom headers, when provided, are carried via colly.Context and overlaid after the anon profile so they take precedence (custom headers win).
func (*AnonymizedScraper) ScrapeContentWithPlaywright ¶
func (s *AnonymizedScraper) ScrapeContentWithPlaywright(url string, customHeaders map[string]string) (*goquery.Selection, error)
ScrapeContentWithPlaywright uses Playwright to fetch a page, waiting for JavaScript to fully load before retrieving the DOM. Custom headers, when provided, are overlaid after the anon profile (custom wins).
type ArticleExtractor ¶
type ArticleExtractor struct {
// contains filtered or unexported fields
}
ArticleExtractor implements the Scraper interface for article content extraction
func NewArticleExtractor ¶
func NewArticleExtractor(configSelectors *models.Selectors) *ArticleExtractor
NewArticleExtractor creates a new ArticleExtractor instance
func (*ArticleExtractor) ExtractFromDOM ¶
func (*ArticleExtractor) GetSelectors ¶
func (ae *ArticleExtractor) GetSelectors() *models.Selectors
GetSelectors returns the configured selectors for this extractor
func (*ArticleExtractor) SetSelectors ¶
func (ae *ArticleExtractor) SetSelectors(selectors *models.Selectors)
SetSelectors updates the config selectors
type RSSFeedScraper ¶
type RSSFeedScraper struct {
// contains filtered or unexported fields
}
RSSFeedScraper implements the Scraper interface for RSS feeds
func NewRSSFeedScraper ¶
func NewRSSFeedScraper(configSelectors *models.Selectors) *RSSFeedScraper
NewRSSFeedScraper creates a new RSSFeedScraper instance
func (*RSSFeedScraper) ScrapeContent ¶
type Scraper ¶
type Scraper interface {
Fetch(url string, params map[string]any) ([]models.FeedItem, error)
ScrapeContent(url string, params map[string]any) (string, error)
}
Scraper defines the interface for feed scrapers
type TwitterScraper ¶
type TwitterScraper struct {
// contains filtered or unexported fields
}
TwitterScraper is a placeholder for a Twitter scraper This is just a skeleton - you would implement the actual Twitter scraping logic
func NewTwitterScraper ¶
func NewTwitterScraper(apiKey, apiSecret, bearerToken string) *TwitterScraper
NewTwitterScraper creates a new TwitterScraper instance
Source Files