Documentation
¶
Overview ¶
Package scraper provides utilities for scraper registration and management.
Index ¶
- func Create(name string, settings config.ScraperSettings, db *database.DB, ...) (models.Scraper, error)
- func GetRegisteredDefaults() map[string]DefaultSettings
- func GetScraperConstructors() map[string]ScraperConstructor
- func NewDefaultScraperRegistry(cfg *config.Config, db *database.DB) (*models.ScraperRegistry, error)
- func RegisterScraper(name string, constructor ScraperConstructor)
- func RegisterScraperDefaults(name string, defaults DefaultSettings)
- func ResetConstructors()
- func ResetDefaults()
- type DefaultSettings
- type ScraperConstructor
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func Create ¶
func Create( name string, settings config.ScraperSettings, db *database.DB, globalScrapersConfig *config.ScrapersConfig, ) (models.Scraper, error)
Create instantiates a single scraper by name with the provided settings. This is the factory method that enables dynamic scraper instantiation with custom settings, rather than NewDefaultScraperRegistry which creates all scrapers at once.
Parameters:
- name: The scraper name (e.g., "r18dev", "dmm")
- settings: The scraper configuration settings
- db: The database connection (can be nil for scrapers that don't need it)
- globalScrapersConfig: The global scrapers configuration (can be nil)
Returns:
- A new Scraper instance configured with the provided settings
- error if the scraper name is not registered or instantiation fails
Example usage:
settings := config.ScraperSettings{Enabled: true, Language: "en"}
scraper, err := scraper.Create("r18dev", settings, db, &cfg.Scrapers)
if err != nil {
log.Fatalf("Failed to create r18dev scraper: %v", err)
}
func GetRegisteredDefaults ¶
func GetRegisteredDefaults() map[string]DefaultSettings
GetRegisteredDefaults returns a copy of all registered scraper defaults.
func GetScraperConstructors ¶
func GetScraperConstructors() map[string]ScraperConstructor
GetScraperConstructors returns a copy of all registered scraper constructors. Primarily used by NewDefaultScraperRegistry.
func NewDefaultScraperRegistry ¶
func NewDefaultScraperRegistry(cfg *config.Config, db *database.DB) (*models.ScraperRegistry, error)
NewDefaultScraperRegistry creates a new scraper registry with all default scrapers. This is the single source of truth for scraper registration across all modes (API, TUI, CLI).
Parameters:
- cfg: The application configuration
- db: The database connection (for ContentIDMappingRepository)
Returns:
- *models.ScraperRegistry: The configured registry
- error: Any error encountered during scraper initialization
The registry uses GetScraperConstructors() to discover all registered scrapers via init().
func RegisterScraper ¶
func RegisterScraper(name string, constructor ScraperConstructor)
RegisterScraper registers a scraper constructor for init()-based auto-registration. This is called from each scraper package's init() function. The constructor will be called by NewDefaultScraperRegistry with actual config and db.
func RegisterScraperDefaults ¶
func RegisterScraperDefaults(name string, defaults DefaultSettings)
RegisterScraperDefaults registers default settings and priority for a scraper. Called from each scraper package's init() function. Also registers with scraperutil for config.go to use via GetDefaultScraperSettings().
func ResetConstructors ¶
func ResetConstructors()
ResetConstructors clears the constructor registry. Primarily used for test isolation.
func ResetDefaults ¶
func ResetDefaults()
ResetDefaults clears the defaults registry. Primarily used for test isolation.
Types ¶
type DefaultSettings ¶
type DefaultSettings struct {
Settings config.ScraperSettings
Priority int
}
DefaultSettings holds a scraper's default configuration and priority. Used by RegisterScraperDefaults for self-reported scraper priorities.
type ScraperConstructor ¶
type ScraperConstructor func(config.ScraperSettings, *database.DB, *config.ScrapersConfig) (models.Scraper, error)
ScraperConstructor is a function that creates a scraper instance. Parameters: settings, db, globalScrapersConfig
Source Files
Directories