orchestrate

package

v1.3.3 Latest Latest Go to latest Published: Feb 21, 2026 License: Apache-2.0 Imports: 10 Imported by: 0

Details

This section is empty.

This section is empty.

func GetAllSiteKeys(appCfg *config.AppConfig) []string

GetAllSiteKeys returns all site keys from the config

func ValidateSiteKeys(appCfg *config.AppConfig, siteKeys []string) error

ValidateSiteKeys checks that all provided site keys exist in the config

type Orchestrator struct {
	// contains filtered or unexported fields
}

Orchestrator manages parallel crawling of multiple sites

func NewOrchestrator(appCfg *config.AppConfig, siteKeys []string, resume bool, log *logrus.Entry) *Orchestrator

NewOrchestrator creates a new orchestrator for parallel site crawling

func (o *Orchestrator) Cancel()

Cancel cancels all running crawls

func (o *Orchestrator) GetProgress() []crawler.CrawlerProgress

GetProgress returns the current progress of all sites

func (o *Orchestrator) Run() []SiteResult

Run starts crawling all sites in parallel and waits for completion

type SiteResult struct {
	SiteKey        string
	Success        bool
	Error          error
	PagesProcessed int64
	Duration       time.Duration
}

SiteResult contains the result of crawling a single site