Documentation
¶
Index ¶
- Constants
- func RefineByUrlPattern(allEntries, validEntries []*model.Recipe) (string, int)
- func ReplayDiscovered(data *model.DataInput, feed *model.Feed, d *model.DiscoveredFeed) error
- func ScrapeFeed(data *model.DataInput, feed *model.Feed, sampling SamplingOptions, ...) error
- func UrlPathPattern(urls []string) string
- type GroupValidator
- type SamplingOptions
- type ScoringOptions
Constants ¶
const ( SourceRSSLink model.DiscoverySource = "rss-link" SourceSitemap model.DiscoverySource = "sitemap" SourceDOMContainer model.DiscoverySource = "dom-container" )
Source constants for DiscoveredFeed.Source.
Variables ¶
This section is empty.
Functions ¶
func RefineByUrlPattern ¶
RefineByUrlPattern derives the common URL path prefix from validEntries and returns the pattern and count of allEntries whose URL matches that prefix.
func ReplayDiscovered ¶
ReplayDiscovered replays a previously discovered feed configuration.
func ScrapeFeed ¶
func ScrapeFeed(data *model.DataInput, feed *model.Feed, sampling SamplingOptions, scoring ScoringOptions) error
ScrapeFeed runs the discovery pipeline against the given DataInput. On success, feed.Entries will contain stub recipes (Url only) and feed.Discovered will describe how they were found. When sampling.Validator is set, it is called with a sample of URLs from each DOM candidate group before committing it. Groups where fewer than half the sampled URLs are recipes are skipped; the next-best group is tried instead.
func UrlPathPattern ¶
UrlPathPattern returns the common path prefix of URLs as a string. Example: ["/recipes/pasta", "/recipes/chicken"] → "/recipes/"
Types ¶
type GroupValidator ¶ added in v0.18.0
GroupValidator samples a slice of candidate URLs and returns the subset that are confirmed recipe pages (already scraped, ready to merge into feed entries).
type SamplingOptions ¶ added in v0.18.0
type SamplingOptions struct {
// Validator is called with a sample of candidate URLs; nil disables validation.
Validator GroupValidator
// SampleSize is the number of URLs to sample per candidate group; 0 disables sampling.
SampleSize int
}
SamplingOptions configures optional URL validation during DOM discovery. The zero value disables sampling entirely.
type ScoringOptions ¶ added in v0.18.1
type ScoringOptions struct {
// AcceptThreshold overrides the confidence score above which a group is accepted without sampling.
AcceptThreshold float64
// SampleThreshold overrides the confidence score above which sampling is required.
SampleThreshold float64
// MinGroupSize overrides the minimum number of links a finalized container group must have to be considered.
MinGroupSize int
// MinSiblingsToMerge overrides the minimum number of sibling child groups required before mergeSiblingGroups collapses them into one container.
MinSiblingsToMerge int
// MaxGroupsCheck overrides the maximum number of candidate groups to validate via sampling.
MaxGroupsCheck int
}
ScoringOptions tunes the DOM heuristic scoring thresholds. Zero values fall back to the package-level defaults.
Source Files