Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type SitemapProcessor ¶
type SitemapProcessor struct {
// contains filtered or unexported fields
}
SitemapProcessor handles fetching, parsing, and processing sitemaps
func NewSitemapProcessor ¶
func NewSitemapProcessor( sitemapQueue chan string, pq *queue.ThreadSafePriorityQueue, store storage.PageStore, fetcher fetch.HTTPFetcher, rateLimiter *fetch.RateLimiter, globalSemaphore *semaphore.Weighted, compiledDisallowedPatterns []*regexp.Regexp, siteCfg *config.SiteConfig, appCfg *config.AppConfig, log *logrus.Entry, wg *sync.WaitGroup, ) *SitemapProcessor
NewSitemapProcessor creates a new SitemapProcessor
func (*SitemapProcessor) MarkSitemapProcessed ¶
func (sp *SitemapProcessor) MarkSitemapProcessed(sitemapURL string) bool
MarkSitemapProcessed records that a sitemap URL has been queued for processing Returns true if it was newly marked, false if already marked
func (*SitemapProcessor) Start ¶
func (sp *SitemapProcessor) Start(ctx context.Context)
Start runs the sitemap processing loop in a goroutine
Source Files
Click to show internal directories.
Click to hide internal directories.