postprocessor

package

v2.0.26 Latest Latest Go to latest Published: Jun 2, 2026 License: AGPL-3.0 Imports: 21 Imported by: 0

Details

Valid go.mod file
Redistributable license
Tagged version
Stable version
Learn more about best practices

Repository

github.com/internetarchive/Zeno

Links

Open Source Insights

Documentation ¶

Index ¶

Variables
func ExtractAssetsOutlinks(item *models.Item) (assets, outlinks []*models.URL, err error)
func Extractors(item *models.Item) (assets, outlinks []*models.URL, err error)
func SanitizeAssetsOutlinks(item *models.Item, assets []*models.URL, outlinks []*models.URL, err error) ([]*models.URL, []*models.URL, error)
func Start(inputChan, outputChan chan *models.Item) error
func Stop()
type OutlinkExtractor

Constants ¶

This section is empty.

Variables ¶

View Source

var (
	// ErrPostprocessorAlreadyInitialized is the error returned when the postprocessor is already initialized
	ErrPostprocessorAlreadyInitialized = errors.New("postprocessor already initialized")
)

Functions ¶

func ExtractAssetsOutlinks ¶

func ExtractAssetsOutlinks(item *models.Item) (assets, outlinks []*models.URL, err error)

ExtractAssetsOutlinks extracts assets from the item's body and returns them. It also potentially returns outlinks if the body contains URLs that are not assets.

func Extractors ¶

func Extractors(item *models.Item) (assets, outlinks []*models.URL, err error)

Extract assets and outlinks from the body using the appropriate extractor Order is important, we want to check for more specific things first, as they may trigger more general extractors (e.g. HTML) TODO this should be refactored using interfaces

func SanitizeAssetsOutlinks ¶

func SanitizeAssetsOutlinks(item *models.Item, assets []*models.URL, outlinks []*models.URL, err error) ([]*models.URL, []*models.URL, error)

func Start ¶

func Start(inputChan, outputChan chan *models.Item) error

This functions starts the preprocessor responsible for preparing the seeds sent by the reactor for captures

func Stop ¶

func Stop()

Types ¶

type OutlinkExtractor ¶

type OutlinkExtractor interface {
	Support(extractor.Mode) bool // Support checks if the extractor supports the given mode
	Match(*models.URL) bool
	Extract(*models.URL) ([]*models.URL, error)
}

Source Files ¶

View all Source files

Directories ¶

Path	Synopsis
domainscrawl Package domainscrawl is a postprocessing component that parse domains from a given input and stores them for later matching.	Package domainscrawl is a postprocessing component that parse domains from a given input and stores them for later matching.
extractor
sitespecific
facebook
github
ina
reddit
truthsocial

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL