Documentation
¶
Overview ¶
Package scraper provides utilities for scraper registration and management.
Index ¶
- func Create(name string, settings config.ScraperSettings, db *database.DB, ...) (models.Scraper, error)
- func GetRegisteredDefaults() map[string]DefaultSettings
- func GetScraperConstructors() map[string]ScraperConstructor
- func NewDefaultScraperRegistry(cfg *config.Config, db *database.DB) (*models.ScraperRegistry, error)
- func ResetAllRegistries()
- type DefaultSettings
- type ScraperConstructor
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func Create ¶
func Create( name string, settings config.ScraperSettings, db *database.DB, globalScrapersConfig *config.ScrapersConfig, ) (models.Scraper, error)
func GetRegisteredDefaults ¶
func GetRegisteredDefaults() map[string]DefaultSettings
func GetScraperConstructors ¶
func GetScraperConstructors() map[string]ScraperConstructor
func NewDefaultScraperRegistry ¶
func NewDefaultScraperRegistry(cfg *config.Config, db *database.DB) (*models.ScraperRegistry, error)
NewDefaultScraperRegistry creates a new scraper registry with all default scrapers. This is the single source of truth for scraper registration across all modes (API, TUI, CLI).
Parameters:
- cfg: The application configuration
- db: The database connection (for ContentIDMappingRepository)
Returns:
- *models.ScraperRegistry: The configured registry
- error: Any error encountered during scraper initialization
The registry uses GetScraperConstructors() to discover all registered scrapers via init().
func ResetAllRegistries ¶
func ResetAllRegistries()
Types ¶
type DefaultSettings ¶
type DefaultSettings struct {
Settings config.ScraperSettings
Priority int
}
type ScraperConstructor ¶
type ScraperConstructor func(config.ScraperSettings, *database.DB, *config.ScrapersConfig) (models.Scraper, error)
Source Files
Directories
Click to show internal directories.
Click to hide internal directories.