README
¶
Preprocessor Package Documentation
Overview
The preprocessor package provides functionality to prepare seeds for capture. It includes mechanisms for validating URLs and preprocessoring items before they are sent for capture. The package ensures that operations are atomic and synchronized, maintaining consistency and avoiding race conditions.
The preprocessor package is designed to be used in a concurrent environment, where multiple goroutines may interact with the preprocessor. It uses channels to manage the flow of items and their preprocessoring status. The package is thread-safe and provides error handling for common scenarios.
Installation
To use the preprocessor package, import it into your package:
import "github.com/internetarchive/Zeno/internal/pkg/preprocessor"
Usage
Initialization
Before using the preprocessor, you need to initialize it with input and output channels:
inputChan := make(chan *models.Item)
outputChan := make(chan *models.Item)
err := preprocessor.Start(inputChan, outputChan)
if err != nil {
log.Fatalf("Error starting preprocessor: %v", err)
}
defer preprocessor.Stop()
The initialization should happen once or it will error out with
ErrPreprocessorAlreadyInitialized || ErrPreprocessorNotInitialized
Preprocessoring Items
To preprocessor an item, send it to the input channel:
item := &models.Item{
UUID: uuid.New(),
URL: &gocrawlhq.URL{Value: "http://example.com"},
Status: models.ItemFresh,
}
inputChan <- item
The preprocessored item will be sent to the output channel after preprocessoring.
Internals
Preprocessor Struct
The preprocessor struct holds the state and channels for managing item preprocessoring:
type preprocessor struct {
wg sync.WaitGroup
ctx context.Context
cancel context.CancelFunc
input chan *models.Item
output chan *models.Item
}
Documentation
¶
Overview ¶
Package preprocessor is the stage of the pipeline that :
1. Checks that the received seed is consistent and has the correct status 2. Normalizes the seed's lowest level URLs 3. Checks if the URLs should be excluded 4. Removes any false-positive assets 5. Deduplicate the items 6. Seencheck the items 7. Builds the requests before handling them to the archiver
Index ¶
Constants ¶
This section is empty.
Variables ¶
var ( // ErrPreprocessorAlreadyInitialized is the error returned when the preprocessor is already initialized ErrPreprocessorAlreadyInitialized = errors.New("preprocessor already initialized") // ErrSchemeIsInvalid is the error returned when the scheme of a URL is not http or http ErrUnsupportedScheme = errors.New("URL scheme is unsupported") // ErrUnsupportedHost is the error returned when the host of a URL is unsupported ErrUnsupportedHost = errors.New("unsupported host") // ErrPathLoopDetected is the error returned when a URL path contains repeating segments indicating a crawler trap ErrPathLoopDetected = errors.New("URL path or query loop detected") // ErrURLTooLong is the error returned when a URL exceeds the maximum allowed length ErrURLTooLong = errors.New("URL exceeds maximum allowed length") )
var (
GlobalPreprocessor *preprocessor
)
Functions ¶
func NormalizeURL ¶
Normalize the URL by removing fragments, attempting to add URL scheme if missing, and converting relative URLs into absolute URLs. An error is returned if the URL cannot be normalized.
func SetSeenchecker ¶
func SetSeenchecker(seenchecker SeencheckerFunc)
SetSeenchecker sets the seenchecker function to be used by the preprocessor. It should be called only once, and it will panic if called more than once or if the preprocessor is not initialized.
Source Files
Directories