scraper

package
v2.20.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 10, 2026 License: GPL-3.0 Imports: 19 Imported by: 0

Documentation

Overview

Package scraper defines the metadata scraper types and generic run loop.

Concrete scrapers (gamelist.xml, ScreenScraper, TheGamesDB, etc.) call RunScraper from their [platforms.Scraper].Scrape callback, passing their record-specific load/match/map functions directly.

The sentinel tag pattern (scraper.<id>:scraped on the Media record) ensures that a crashed mid-write run is safely retried on the next invocation.

Zaparoo Core Copyright (c) 2026 The Zaparoo Project Contributors. SPDX-License-Identifier: GPL-3.0-or-later

This file is part of Zaparoo Core.

Zaparoo Core is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

Zaparoo Core is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with Zaparoo Core. If not, see <http://www.gnu.org/licenses/>.

Index

Constants

View Source 
const WriteBatchSize = 100

WriteBatchSize is how many targets one write transaction carries.

Variables

This section is empty.

Functions

func ApplyScopedTargets added in v2.18.0 

func ApplyScopedTargets(
	ctx context.Context, db database.MediaDBI, opts ScrapeOptions, selection ScopedSelection,
	targets []database.ScrapeWriteTarget, ch chan<- ScrapeUpdate,
)

ApplyScopedTargets counts selected media, not source records. It also checks write identities independently of scraper matching and deduplicates sources.

func ApplyTargets added in v2.20.0 

func ApplyTargets(
	ctx context.Context,
	db database.MediaDBI,
	opts ScrapeOptions,
	scraperID string,
	targets []database.ScrapeWriteTarget,
	onBatch func(from, to int),
) error

ApplyTargets writes targets in batches of WriteBatchSize, calling onBatch with the half-open index range of each batch once it has committed. The run yields to the pauser before every batch.

A write failure is fatal: work that committed only part of a batch must not be reported as complete, or the sentinel would keep the remaining rows from ever being filled.

func ListingState added in v2.20.0 

func ListingState(fs afero.Fs, dir string) string

ListingState describes a directory by how many entries it holds and a digest of their names, from one listing and no stat. It moves whenever an entry is added, removed or renamed, which is when a scraper that matches files by name can find something new. A directory that cannot be read is described as such, so one that appears later reads as a change.

func NameDigest added in v2.20.0 

func NameDigest(name string) uint64

NameDigest hashes one entry name for an order-independent listing digest.

func RememberSystem added in v2.20.0 

func RememberSystem(
	ctx context.Context, db database.MediaDBI, scraperID, systemID string, version int, sourceState string,
) error

RememberSystem records the state a scraper completed a system in. Pass the source state read before the scraper loaded its source, so a source that changed while the system was being scraped no longer matches and is read again next time.

func RunScraper 

func RunScraper(fn func(chan<- ScrapeUpdate) error) <-chan ScrapeUpdate

RunScraper creates a ScrapeUpdate channel, calls fn (which must start its own goroutine and close the channel when done), and returns the read end. If fn returns an error synchronously, a terminal FatalErr update is emitted and the channel is closed.

func RunTagInfo added in v2.14.1 

func RunTagInfo(scraperID, runID string) database.TagInfo

RunTagInfo returns the per-run marker written to media rows completed during a persisted scraper operation.

func SentinelTagInfo 

func SentinelTagInfo(scraperID string) database.TagInfo

SentinelTagInfo returns the sentinel TagInfo for the given scraper ID. Scrape callbacks write this after a successful record write to mark it done.

func SystemFingerprint added in v2.20.0 

func SystemFingerprint(version int, sourceState string, libraryRevision int64) string

SystemFingerprint joins the state of a scraper's source for one system with that system's library revision. A fill-missing run that finds the fingerprint it stored last time would resolve the same records to the same rows and add nothing, so it can leave the system alone.

sourceState is whatever the scraper's result depends on outside the database, described without doing the scrape's own work. version is the scraper's own, bumped when a change to its matching or its writes means an unchanged system has to be scraped again.

func SystemUnchanged added in v2.20.0 

func SystemUnchanged(
	ctx context.Context, db database.MediaDBI, scraperID, systemID string, version int, sourceState string,
) bool

SystemUnchanged reports whether a system's source and library are as the scraper's last completed run left them. It costs two keyed reads. A failure to tell is not an error for the run: the scraper just does its work.

func VirtualMediaKey added in v2.18.0 

func VirtualMediaKey(path string) string

VirtualMediaKey ignores mutable display text while preserving scheme and ID.

func Wait added in v2.20.0 

func Wait(ctx context.Context, opts ScrapeOptions, scraperID string) error

Wait returns once the run may continue: straight away when nothing holds it, after the pauser's pacing or pause otherwise. It returns the context's error when the run was cancelled.

Types

type ContainerResolver added in v2.18.0 

type ContainerResolver interface {
	Resolve(string) *database.Media
	HasMedia(string) bool
}

ContainerResolver separates allowed write targets from directory context. A single selected row must not make an otherwise ambiguous folder collapse.

func ScopedContainers added in v2.18.0 

func ScopedContainers(
	ctx context.Context, db database.MediaDBI, systemID string, media []database.MediaWithFullPath,
) (ContainerResolver, error)

type MapResult 

type MapResult struct {
	MediaTags  []database.TagInfo
	TitleTags  []database.TagInfo
	TitleProps []database.MediaProperty
	MediaProps []database.MediaProperty
}

MapResult holds the tag and property writes produced by a MapToDB call.

type MatchResult 

type MatchResult struct {
	MediaDBID      int64
	MediaTitleDBID int64
}

MatchResult is the output of a successful Match call. Both IDs must be positive database IDs; invalid IDs are treated as an implementation error and skipped by the generic run loop before any writes are attempted.

type ScopedSelection added in v2.18.0 

type ScopedSelection struct {
	Completed map[int64]struct{}
	Media     []database.MediaWithFullPath
	Titles    []database.TitleWithSystem
}

func LoadScopedSelection added in v2.18.0 

func LoadScopedSelection(
	ctx context.Context, db database.MediaDBI, opts ScrapeOptions, scraperID string,
) (ScopedSelection, error)

func (ScopedSelection) Pending added in v2.18.0 

func (s ScopedSelection) Pending() ([]database.MediaWithFullPath, []database.TitleWithSystem)

Pending keeps already-completed media out of matching and force cleanup.

type ScrapeOptions 

type ScrapeOptions struct {
	// Scope narrows database selection; nil retains the legacy Systems behavior.
	Scope *database.ScrapeScope

	// Pauser pauses scrape work while another foreground activity needs the system.
	Pauser *syncutil.Pauser

	// RunID identifies a persisted scraper operation across restarts.
	RunID string

	// Systems limits scraping to these system IDs. Nil or empty means all systems.
	Systems []string

	// Force re-processes records that already have a sentinel tag.
	Force bool

	// FillMissing revisits records but never replaces existing metadata.
	// It is mutually exclusive with Force and requires scraper support.
	FillMissing bool
}

ScrapeOptions configures a scrape run.

func (ScrapeOptions) SystemIDs added in v2.18.0 

func (o ScrapeOptions) SystemIDs() []string

SystemIDs makes a resolved scope authoritative even for non-API callers.

type ScrapeSystem 

type ScrapeSystem struct {
	ID       string
	ROMPaths []string
	// Extensions is the union of file extensions the platform's launchers index
	// for this system, lower-cased and dot-prefixed. It is empty when the set
	// cannot be stated exactly, which happens when a launcher accepts files
	// through a Test function instead of an extension list. Consumers must treat
	// an empty slice as "unknown", never as "indexes nothing".
	Extensions []string
	DBID       int64
}

ScrapeSystem carries the DB identity and filesystem paths needed by the scrape loop and concrete scraper implementations.

type ScrapeUpdate 

type ScrapeUpdate struct {
	Err         error
	FatalErr    error
	SystemID    string
	Processed   int
	Total       int
	Matched     int
	Skipped     int
	TotalSteps  int
	CurrentStep int
	Done        bool
}

ScrapeUpdate is one progress event emitted on the channel returned by Scrape.

type SourceError added in v2.18.0 

type SourceError struct {
	Err  error
	Path string
}

SourceError reports a metadata source file that exists but could not be read or parsed. It travels in ScrapeUpdate.Err: the run carries on with whatever other sources it has, and the caller decides how to tell the user.

func (*SourceError) Error added in v2.18.0 

func (e *SourceError) Error() string

func (*SourceError) Unwrap added in v2.18.0 

func (e *SourceError) Unwrap() error

type SourceIndex added in v2.18.0 

type SourceIndex struct {
	// contains filtered or unexported fields
}

SourceIndex provides temporary lookup keys over source rows already selected from MediaDB. It never discovers launcher configuration or broadens scope.

func NewSourceIndex added in v2.18.0 

func NewSourceIndex(sources []database.MediaSource) *SourceIndex

NewSourceIndex indexes sources for lookup. A directory shared by variants of one game, such as ScummVM's one target per language of a multilingual disc, is a group: metadata for the directory applies to every target on it. Any other shared source is ambiguous and matches nothing by path.

func (*SourceIndex) ForMedia added in v2.18.0 

func (s *SourceIndex) ForMedia(path string) (database.MediaSource, bool)

func (*SourceIndex) ForPath added in v2.18.0 

func (s *SourceIndex) ForPath(path string) (database.MediaSource, bool)

func (*SourceIndex) Group added in v2.18.0 

func (s *SourceIndex) Group(path string) []database.MediaSource

Group returns every target configured on a directory shared by variants of one game, or nil when path is not such a directory.

func (*SourceIndex) Grouped added in v2.18.0 

func (s *SourceIndex) Grouped(source *database.MediaSource) bool

Grouped reports whether source is one of the targets sharing a directory that Group returns.

func (*SourceIndex) HasMedia added in v2.18.0 

func (s *SourceIndex) HasMedia(path string) bool

func (*SourceIndex) UnderParent added in v2.18.0 

func (s *SourceIndex) UnderParent(path string) []database.MediaSource

UnderParent returns the directory sources held directly inside path, unique or grouped, so metadata describing a game folder can reach the variants in it.

type UnmappedValues added in v2.18.0 

type UnmappedValues struct {
	// contains filtered or unexported fields
}

UnmappedValues counts the source values a scraper run dropped because they had no mapping. Safe for concurrent use.

func (*UnmappedValues) Count added in v2.18.0 

func (u *UnmappedValues) Count(tagType tags.TagType) int

Count returns how many distinct values were dropped for a tag type.

func (*UnmappedValues) LogSummary added in v2.18.0 

func (u *UnmappedValues) LogSummary(scraperID string)

LogSummary logs one info line per tag type that had unmapped values, with counts and a few examples. Call it once at the end of a run.

func (*UnmappedValues) Note added in v2.18.0 

func (u *UnmappedValues) Note(scraperID string, tagType tags.TagType, raw string)

Note records one dropped value. The first sighting of each value is logged at debug, with the scraper's name, so a single value can be traced.

Source Files

View all Source files

Directories

Path Synopsis
Package gamelistxml implements a scraper that reads EmulationStation gamelist.xml files to enrich the Zaparoo MediaDB with developer, publisher, genre, rating, year, artwork paths, and descriptions.
 Package gamelistxml implements a scraper that reads EmulationStation gamelist.xml files to enrich the Zaparoo MediaDB with developer, publisher, genre, rating, year, artwork paths, and descriptions.
Package libretrothumbs downloads box art, screenshots and title screens from the libretro thumbnail server, the source RetroArch uses.
 Package libretrothumbs downloads box art, screenshots and title screens from the libretro thumbnail server, the source RetroArch uses.
Package localmedia imports EmulationStation-style media folder artwork.
 Package localmedia imports EmulationStation-style media folder artwork.
Package misterarcade imports the MiSTer arcade catalog's metadata onto indexed .mra rows.
 Package misterarcade imports the MiSTer arcade catalog's metadata onto indexed .mra rows.
Package misterdocs imports metadata from MiSTer Downloader content installed under docs/<system> directories.
 Package misterdocs imports metadata from MiSTer Downloader content installed under docs/<system> directories.
Package mra reads the MAME set name out of a MiSTer arcade descriptor.
 Package mra reads the MAME set name out of a MiSTer arcade descriptor.
Package pinuppopper imports table metadata and artwork from a PinUP Popper install into the media database.
 Package pinuppopper imports table metadata and artwork from a PinUP Popper install into the media database.
Package scrapertest holds assertions shared by the scraper packages' tests.
 Package scrapertest holds assertions shared by the scraper packages' tests.
Package ssgenre maps ScreenScraper genre names onto Core's genre vocabulary.
 Package ssgenre maps ScreenScraper genre names onto Core's genre vocabulary.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.