scraper

package
v2.18.0-beta.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 22, 2026 License: GPL-3.0 Imports: 10 Imported by: 0

Documentation

Overview

Package scraper defines the metadata scraper types and generic run loop.

Concrete scrapers (gamelist.xml, ScreenScraper, TheGamesDB, etc.) call RunScraper from their [platforms.Scraper].Scrape callback, passing their record-specific load/match/map functions directly.

The sentinel tag pattern (scraper.<id>:scraped on the Media record) ensures that a crashed mid-write run is safely retried on the next invocation.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func ApplyScopedTargets added in v2.18.0

func ApplyScopedTargets(
	ctx context.Context, db database.MediaDBI, opts ScrapeOptions, selection ScopedSelection,
	targets []database.ScrapeWriteTarget, ch chan<- ScrapeUpdate,
)

ApplyScopedTargets counts selected media, not source records. It also checks write identities independently of scraper matching and deduplicates sources.

func RunScraper

func RunScraper(fn func(chan<- ScrapeUpdate) error) <-chan ScrapeUpdate

RunScraper creates a ScrapeUpdate channel, calls fn (which must start its own goroutine and close the channel when done), and returns the read end. If fn returns an error synchronously, a terminal FatalErr update is emitted and the channel is closed.

func RunTagInfo added in v2.14.1

func RunTagInfo(scraperID, runID string) database.TagInfo

RunTagInfo returns the per-run marker written to media rows completed during a persisted scraper operation.

func SentinelTagInfo

func SentinelTagInfo(scraperID string) database.TagInfo

SentinelTagInfo returns the sentinel TagInfo for the given scraper ID. Scrape callbacks write this after a successful record write to mark it done.

func VirtualMediaKey added in v2.18.0

func VirtualMediaKey(path string) string

VirtualMediaKey ignores mutable display text while preserving scheme and ID.

Types

type ContainerResolver added in v2.18.0

type ContainerResolver interface {
	Resolve(string) *database.Media
	HasMedia(string) bool
}

ContainerResolver separates allowed write targets from directory context. A single selected row must not make an otherwise ambiguous folder collapse.

func ScopedContainers added in v2.18.0

func ScopedContainers(
	ctx context.Context, db database.MediaDBI, systemID string, media []database.MediaWithFullPath,
) (ContainerResolver, error)

type MapResult

type MapResult struct {
	MediaTags  []database.TagInfo
	TitleTags  []database.TagInfo
	TitleProps []database.MediaProperty
	MediaProps []database.MediaProperty
}

MapResult holds the tag and property writes produced by a MapToDB call.

type MatchResult

type MatchResult struct {
	MediaDBID      int64
	MediaTitleDBID int64
}

MatchResult is the output of a successful Match call. Both IDs must be positive database IDs; invalid IDs are treated as an implementation error and skipped by the generic run loop before any writes are attempted.

type ScopedSelection added in v2.18.0

type ScopedSelection struct {
	Completed map[int64]struct{}
	Media     []database.MediaWithFullPath
	Titles    []database.TitleWithSystem
}

func LoadScopedSelection added in v2.18.0

func LoadScopedSelection(
	ctx context.Context, db database.MediaDBI, opts ScrapeOptions, scraperID string,
) (ScopedSelection, error)

func (ScopedSelection) Pending added in v2.18.0

Pending keeps already-completed media out of matching and force cleanup.

type ScrapeOptions

type ScrapeOptions struct {
	// Scope narrows database selection; nil retains the legacy Systems behavior.
	Scope *database.ScrapeScope

	// Pauser pauses scrape work while another foreground activity needs the system.
	Pauser *syncutil.Pauser

	// RunID identifies a persisted scraper operation across restarts.
	RunID string

	// Systems limits scraping to these system IDs. Nil or empty means all systems.
	Systems []string

	// Force re-processes records that already have a sentinel tag.
	Force bool

	// FillMissing revisits records but never replaces existing metadata.
	// It is mutually exclusive with Force and requires scraper support.
	FillMissing bool
}

ScrapeOptions configures a scrape run.

func (ScrapeOptions) SystemIDs added in v2.18.0

func (o ScrapeOptions) SystemIDs() []string

SystemIDs makes a resolved scope authoritative even for non-API callers.

type ScrapeSystem

type ScrapeSystem struct {
	ID       string
	ROMPaths []string
	// Extensions is the union of file extensions the platform's launchers index
	// for this system, lower-cased and dot-prefixed. It is empty when the set
	// cannot be stated exactly, which happens when a launcher accepts files
	// through a Test function instead of an extension list. Consumers must treat
	// an empty slice as "unknown", never as "indexes nothing".
	Extensions []string
	DBID       int64
}

ScrapeSystem carries the DB identity and filesystem paths needed by the scrape loop and concrete scraper implementations.

type ScrapeUpdate

type ScrapeUpdate struct {
	Err         error
	FatalErr    error
	SystemID    string
	Processed   int
	Total       int
	Matched     int
	Skipped     int
	TotalSteps  int
	CurrentStep int
	Done        bool
}

ScrapeUpdate is one progress event emitted on the channel returned by Scrape.

type SourceError added in v2.18.0

type SourceError struct {
	Err  error
	Path string
}

SourceError reports a metadata source file that exists but could not be read or parsed. It travels in ScrapeUpdate.Err: the run carries on with whatever other sources it has, and the caller decides how to tell the user.

func (*SourceError) Error added in v2.18.0

func (e *SourceError) Error() string

func (*SourceError) Unwrap added in v2.18.0

func (e *SourceError) Unwrap() error

type SourceIndex added in v2.18.0

type SourceIndex struct {
	// contains filtered or unexported fields
}

SourceIndex provides temporary lookup keys over source rows already selected from MediaDB. It never discovers launcher configuration or broadens scope.

func NewSourceIndex added in v2.18.0

func NewSourceIndex(sources []database.MediaSource) *SourceIndex

func (*SourceIndex) ForMedia added in v2.18.0

func (s *SourceIndex) ForMedia(path string) (database.MediaSource, bool)

func (*SourceIndex) ForPath added in v2.18.0

func (s *SourceIndex) ForPath(path string) (database.MediaSource, bool)

func (*SourceIndex) HasMedia added in v2.18.0

func (s *SourceIndex) HasMedia(path string) bool

func (*SourceIndex) UnderParent added in v2.18.0

func (s *SourceIndex) UnderParent(path string) []database.MediaSource

UnderParent returns the unambiguous directory sources held directly inside path, so metadata describing a game folder can reach the variants in it.

Directories

Path Synopsis
Package gamelistxml implements a scraper that reads EmulationStation gamelist.xml files to enrich the Zaparoo MediaDB with developer, publisher, genre, rating, year, artwork paths, and descriptions.
Package gamelistxml implements a scraper that reads EmulationStation gamelist.xml files to enrich the Zaparoo MediaDB with developer, publisher, genre, rating, year, artwork paths, and descriptions.
Package localmedia imports EmulationStation-style media folder artwork.
Package localmedia imports EmulationStation-style media folder artwork.
Package misterarcade imports the MiSTer arcade catalog's metadata onto indexed .mra rows.
Package misterarcade imports the MiSTer arcade catalog's metadata onto indexed .mra rows.
Package misterdocs imports metadata from MiSTer Downloader content installed under docs/<system> directories.
Package misterdocs imports metadata from MiSTer Downloader content installed under docs/<system> directories.
Package mra reads the MAME set name out of a MiSTer arcade descriptor.
Package mra reads the MAME set name out of a MiSTer arcade descriptor.
Package pinuppopper imports table metadata and artwork from a PinUP Popper install into the media database.
Package pinuppopper imports table metadata and artwork from a PinUP Popper install into the media database.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL