Documentation
¶
Index ¶
- func AnalyzeFeedBody(raw RawResponse, parseErr error, itemCount int) models.FeedDiagnosis
- func CheckLightpanda(autoStart bool) error
- func CheckSolimen(autoStart bool) error
- func DiagnoseFeedURL(feedURL string, headers map[string]string) models.FeedDiagnosis
- func HeadersFromParams(params map[string]any) map[string]string
- func Usable(text string) (bool, string)
- type AnonymizedScraper
- type ArticleExtractor
- type ExtractResult
- type FeedInspection
- type RSSFeedScraper
- type RawResponse
- type Scraper
- type SelectorCandidate
- type SelectorScore
- type TwitterScraper
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func AnalyzeFeedBody ¶ added in v0.2.0
func AnalyzeFeedBody(raw RawResponse, parseErr error, itemCount int) models.FeedDiagnosis
AnalyzeFeedBody turns a raw response and its parse outcome into a structured diagnosis. It is a pure function over the bytes (no IO) so it can be unit tested directly against crafted payloads.
func CheckLightpanda ¶
CheckLightpanda checks if Lightpanda is reachable on port 9222. If not, it either prompts the user interactively or auto-starts the container. Set autoStart to true to skip the interactive prompt.
func CheckSolimen ¶
CheckSolimen checks if the Solimen service is reachable on port 5011. If not, it either prompts the user interactively or auto-starts the container.
func DiagnoseFeedURL ¶ added in v0.2.0
func DiagnoseFeedURL(feedURL string, headers map[string]string) models.FeedDiagnosis
DiagnoseFeedURL fetches a feed URL and returns a structured diagnosis. It is a thin wrapper over InspectFeedURL for callers that only need the diagnosis.
func HeadersFromParams ¶
HeadersFromParams extracts custom HTTP headers stored under params["headers"]. Values may arrive as map[string]string (in-process) or map[string]any (after a JSON/protobuf round-trip), so both shapes are handled. Returns nil when absent.
Types ¶
type AnonymizedScraper ¶
type AnonymizedScraper struct {
// contains filtered or unexported fields
}
AnonymizedScraper owns the User-Agent pool and applies anonymization to each request. It holds no browser or collector state: ScrapeContent builds a fresh colly collector per call and ScrapeContentDynamic opens a fresh CDP connection to Lightpanda per call, so concurrent scrapes never share mutable state.
func GetSharedAnonymizedScraper ¶
func GetSharedAnonymizedScraper() *AnonymizedScraper
GetSharedAnonymizedScraper returns the process-wide AnonymizedScraper, which owns the User-Agent pool. It holds no browser or collector state — ScrapeContent builds a per-call colly collector and ScrapeContentDynamic opens a per-call CDP connection to Lightpanda — so this needs no configuration and nothing to close.
func NewAnonymizedScraper ¶
func NewAnonymizedScraper() *AnonymizedScraper
NewAnonymizedScraper creates a new AnonymizedScraper.
func (*AnonymizedScraper) HTTPClient ¶
func (s *AnonymizedScraper) HTTPClient() *http.Client
HTTPClient returns an *http.Client that carries this scraper's anon profile (rotating User-Agent from s.userAgents + spoofed headers + Alt-Used) on every request via anonRoundTripper. Unlike the colly path it follows redirects across hosts and imposes no domain allowlist, so it suits feed/RSS fetching.
func (*AnonymizedScraper) ScrapeContent ¶
func (s *AnonymizedScraper) ScrapeContent(url string, headers map[string]string) (dom *goquery.Selection, err error)
ScrapeContent visits the URL, processes the HTML content and returns the largest content block. Custom headers, when provided, are carried via colly.Context and overlaid after the anon profile so they take precedence (custom headers win).
func (*AnonymizedScraper) ScrapeContentDynamic ¶ added in v0.2.0
func (s *AnonymizedScraper) ScrapeContentDynamic(url string, customHeaders map[string]string) (*goquery.Selection, error)
ScrapeContentDynamic renders a page with JavaScript by driving Lightpanda over CDP and returns its body DOM. It uses chromedp — a pure-Go CDP client — so it needs no Playwright driver, no Node, and no bundled browser: it connects to the Lightpanda instance already running on the CDP port. Drop-in replacement for the old Playwright path (same signature and return shape). Custom headers are overlaid after the anon profile (custom wins).
type ArticleExtractor ¶
type ArticleExtractor struct {
// contains filtered or unexported fields
}
ArticleExtractor implements the Scraper interface for article content extraction
func NewArticleExtractor ¶
func NewArticleExtractor(configSelectors *models.Selectors) *ArticleExtractor
NewArticleExtractor creates a new ArticleExtractor instance
func (*ArticleExtractor) ExtractFromDOM ¶
func (*ArticleExtractor) GetSelectors ¶
func (ae *ArticleExtractor) GetSelectors() *models.Selectors
GetSelectors returns the configured selectors for this extractor
func (*ArticleExtractor) SetSelectors ¶
func (ae *ArticleExtractor) SetSelectors(selectors *models.Selectors)
SetSelectors updates the config selectors
type ExtractResult ¶ added in v0.2.0
type ExtractResult struct {
URL string
Chars int
Matched bool // the article selector matched an element
}
ExtractResult is one selector test against one article URL.
type FeedInspection ¶ added in v0.2.0
type FeedInspection struct {
Diagnosis models.FeedDiagnosis
SampleLinks []string
// SampleContentChars holds the plain-text length of each sampled entry's feed
// content (content:encoded, falling back to description), aligned 1:1 with
// SampleLinks. It lets a caller tell whether the feed already ships full bodies.
SampleContentChars []int
Title string
}
FeedInspection bundles a feed diagnosis with sample article links and the detected feed title, for scaffolding a feed configuration.
func InspectFeedURL ¶ added in v0.2.0
func InspectFeedURL(feedURL string, headers map[string]string, maxLinks int) FeedInspection
InspectFeedURL fetches a feed URL, attempts to parse it, and returns a full diagnosis plus up to maxLinks sample article links and the feed title (when the feed parses). It is read-only: nothing is stored. The raw body is always saved to disk (even when tracing is off) so the offending bytes stay inspectable.
type RSSFeedScraper ¶
type RSSFeedScraper struct {
// contains filtered or unexported fields
}
RSSFeedScraper implements the Scraper interface for RSS feeds
func NewRSSFeedScraper ¶
func NewRSSFeedScraper(configSelectors *models.Selectors) *RSSFeedScraper
NewRSSFeedScraper creates a new RSSFeedScraper instance
func (*RSSFeedScraper) ScrapeContent ¶
type RawResponse ¶ added in v0.2.0
type RawResponse struct {
URL string
FinalURL string // after redirects
Status int
ContentType string
Body []byte
Duration time.Duration
}
RawResponse holds the raw bytes and metadata of a single feed HTTP fetch, captured before any parsing so failures stay inspectable.
func FetchRaw ¶ added in v0.2.0
func FetchRaw(feedURL string, headers map[string]string) (RawResponse, error)
FetchRaw performs a GET for a feed URL through the shared anonymized HTTP client — the same client gofeed uses for ParseURL — and returns the raw body plus response metadata. Custom headers are overlaid after the anon profile, so per-feed headers behave exactly as they do on the normal fetch path.
type Scraper ¶
type Scraper interface {
Fetch(url string, params map[string]any) ([]models.FeedItem, error)
ScrapeContent(url string, params map[string]any) (string, error)
}
Scraper defines the interface for feed scrapers
type SelectorCandidate ¶ added in v0.2.0
type SelectorCandidate struct {
Selector string `json:"selector"`
Chars int `json:"chars"` // trimmed text length
LinkDensity float64 `json:"link_density"` // anchor text ÷ total text (0..1); high = nav/menu
Snippet string `json:"snippet"` // first printable chars of the element's text
}
SelectorCandidate is a ranked guess at the element that wraps an article body.
func SuggestSelectors ¶ added in v0.2.0
func SuggestSelectors(dom *goquery.Selection, max int) []SelectorCandidate
SuggestSelectors walks a page DOM and returns the most likely article-content selectors, ranked by text length penalized by link density (so nav/menus/footers, which are mostly links, sink below prose). It is a deterministic, token-cheap alternative to feeding raw HTML to an LLM: the agent picks from measured candidates and confirms with test_selector. Pure over the DOM, so it is unit-testable.
type SelectorScore ¶ added in v0.2.0
type SelectorScore struct {
Samples int
Usable int // count of results that passed Usable
MeanChars int // mean extracted length over usable results
Score float64 // 0..1: usable ratio penalized by length variance
Reliable bool // Score high enough to recommend without hesitation
}
SelectorScore summarizes how a candidate selector performed across several sample articles, so a selector that works on one page but flukes on others is not mistaken for a reliable one.
func ScoreSelector ¶ added in v0.2.0
func ScoreSelector(results []ExtractResult) SelectorScore
ScoreSelector aggregates per-article results into a stability score. The score is the fraction of usable samples, scaled down when usable lengths vary wildly (high relative variance means the selector grabs different things per page).
type TwitterScraper ¶
type TwitterScraper struct {
// contains filtered or unexported fields
}
TwitterScraper is a placeholder for a Twitter scraper This is just a skeleton - you would implement the actual Twitter scraping logic
func NewTwitterScraper ¶
func NewTwitterScraper(apiKey, apiSecret, bearerToken string) *TwitterScraper
NewTwitterScraper creates a new TwitterScraper instance
Source Files