scraper

package
v2.18.0-beta.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 15, 2026 License: GPL-3.0 Imports: 10 Imported by: 0

Documentation

Overview

Package scraper defines the metadata scraper types and generic run loop.

Concrete scrapers (gamelist.xml, ScreenScraper, TheGamesDB, etc.) call RunScraper from their [platforms.Scraper].Scrape callback, passing their record-specific load/match/map functions directly.

The sentinel tag pattern (scraper.<id>:scraped on the Media record) ensures that a crashed mid-write run is safely retried on the next invocation.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func ApplyScopedTargets added in v2.18.0

func ApplyScopedTargets(
	ctx context.Context, db database.MediaDBI, opts ScrapeOptions, selection ScopedSelection,
	targets []database.ScrapeWriteTarget, ch chan<- ScrapeUpdate,
)

ApplyScopedTargets counts selected media, not source records. It also checks write identities independently of scraper matching and deduplicates sources.

func RunScraper

func RunScraper(fn func(chan<- ScrapeUpdate) error) <-chan ScrapeUpdate

RunScraper creates a ScrapeUpdate channel, calls fn (which must start its own goroutine and close the channel when done), and returns the read end. If fn returns an error synchronously, a terminal FatalErr update is emitted and the channel is closed.

func RunTagInfo added in v2.14.1

func RunTagInfo(scraperID, runID string) database.TagInfo

RunTagInfo returns the per-run marker written to media rows completed during a persisted scraper operation.

func SentinelTagInfo

func SentinelTagInfo(scraperID string) database.TagInfo

SentinelTagInfo returns the sentinel TagInfo for the given scraper ID. Scrape callbacks write this after a successful record write to mark it done.

func VirtualMediaKey added in v2.18.0

func VirtualMediaKey(path string) string

VirtualMediaKey ignores mutable display text while preserving scheme and ID.

Types

type ContainerResolver added in v2.18.0

type ContainerResolver interface {
	Resolve(string) *database.Media
	HasMedia(string) bool
}

ContainerResolver separates allowed write targets from directory context. A single selected row must not make an otherwise ambiguous folder collapse.

func ScopedContainers added in v2.18.0

func ScopedContainers(
	ctx context.Context, db database.MediaDBI, systemID string, media []database.MediaWithFullPath,
) (ContainerResolver, error)

type MapResult

type MapResult struct {
	MediaTags  []database.TagInfo
	TitleTags  []database.TagInfo
	TitleProps []database.MediaProperty
	MediaProps []database.MediaProperty
}

MapResult holds the tag and property writes produced by a MapToDB call.

type MatchResult

type MatchResult struct {
	MediaDBID      int64
	MediaTitleDBID int64
}

MatchResult is the output of a successful Match call. Both IDs must be positive database IDs; invalid IDs are treated as an implementation error and skipped by the generic run loop before any writes are attempted.

type ScopedSelection added in v2.18.0

type ScopedSelection struct {
	Completed map[int64]struct{}
	Media     []database.MediaWithFullPath
	Titles    []database.TitleWithSystem
}

func LoadScopedSelection added in v2.18.0

func LoadScopedSelection(
	ctx context.Context, db database.MediaDBI, opts ScrapeOptions, scraperID string,
) (ScopedSelection, error)

func (ScopedSelection) Pending added in v2.18.0

Pending keeps already-completed media out of matching and force cleanup.

type ScrapeOptions

type ScrapeOptions struct {
	// Scope narrows database selection; nil retains the legacy Systems behavior.
	Scope *database.ScrapeScope

	// Pauser pauses scrape work while another foreground activity needs the system.
	Pauser *syncutil.Pauser

	// RunID identifies a persisted scraper operation across restarts.
	RunID string

	// Systems limits scraping to these system IDs. Nil or empty means all systems.
	Systems []string

	// Force re-processes records that already have a sentinel tag.
	Force bool

	// FillMissing revisits records but never replaces existing metadata.
	// It is mutually exclusive with Force and requires scraper support.
	FillMissing bool
}

ScrapeOptions configures a scrape run.

func (ScrapeOptions) SystemIDs added in v2.18.0

func (o ScrapeOptions) SystemIDs() []string

SystemIDs makes a resolved scope authoritative even for non-API callers.

type ScrapeSystem

type ScrapeSystem struct {
	ID       string
	ROMPaths []string
	// Extensions is the union of file extensions the platform's launchers index
	// for this system, lower-cased and dot-prefixed. It is empty when the set
	// cannot be stated exactly, which happens when a launcher accepts files
	// through a Test function instead of an extension list. Consumers must treat
	// an empty slice as "unknown", never as "indexes nothing".
	Extensions []string
	DBID       int64
}

ScrapeSystem carries the DB identity and filesystem paths needed by the scrape loop and concrete scraper implementations.

type ScrapeUpdate

type ScrapeUpdate struct {
	Err         error
	FatalErr    error
	SystemID    string
	Processed   int
	Total       int
	Matched     int
	Skipped     int
	TotalSteps  int
	CurrentStep int
	Done        bool
}

ScrapeUpdate is one progress event emitted on the channel returned by Scrape.

type SourceIndex added in v2.18.0

type SourceIndex struct {
	// contains filtered or unexported fields
}

SourceIndex provides temporary lookup keys over source rows already selected from MediaDB. It never discovers launcher configuration or broadens scope.

func NewSourceIndex added in v2.18.0

func NewSourceIndex(sources []database.MediaSource) *SourceIndex

func (*SourceIndex) ForMedia added in v2.18.0

func (s *SourceIndex) ForMedia(path string) (database.MediaSource, bool)

func (*SourceIndex) ForPath added in v2.18.0

func (s *SourceIndex) ForPath(path string) (database.MediaSource, bool)

func (*SourceIndex) HasMedia added in v2.18.0

func (s *SourceIndex) HasMedia(path string) bool

Directories

Path Synopsis
Package gamelistxml implements a scraper that reads EmulationStation gamelist.xml files to enrich the Zaparoo MediaDB with developer, publisher, genre, rating, year, artwork paths, and descriptions.
Package gamelistxml implements a scraper that reads EmulationStation gamelist.xml files to enrich the Zaparoo MediaDB with developer, publisher, genre, rating, year, artwork paths, and descriptions.
Package localmedia imports EmulationStation-style media folder artwork.
Package localmedia imports EmulationStation-style media folder artwork.
Package misterdocs imports metadata from MiSTer Downloader content installed under docs/<system> directories.
Package misterdocs imports metadata from MiSTer Downloader content installed under docs/<system> directories.
Package pinuppopper imports table metadata and artwork from a PinUP Popper install into the media database.
Package pinuppopper imports table metadata and artwork from a PinUP Popper install into the media database.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL