Documentation
¶
Overview ¶
Package source defines the interface for a data source that can produce items to crawl and accept items that have been crawled.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type FeedEmptyReporter ¶
type FeedEmptyReporter struct {
// contains filtered or unexported fields
}
FeedEmptyReporter logs when the feed is empty for a long time.
func NewFeedEmptyReporter ¶
func NewFeedEmptyReporter(logger *log.FieldedLogger) *FeedEmptyReporter
func (*FeedEmptyReporter) Report ¶
func (e *FeedEmptyReporter) Report(urls int)
type Source ¶
type Source interface {
// Start initializes the source with the given input and output channels.
Start(finishChan, produceChan chan *models.Item) error
// Stop stops the source.
Stop()
// Name returns the name of the source.
Name() string
}
Source is an interface for a data source that can produce items to crawl and accept items that have been crawled.
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
Package hq provides a way to interact with the HQv3 API and consumes, produces and mark items as finished asynchronusly.
|
Package hq provides a way to interact with the HQv3 API and consumes, produces and mark items as finished asynchronusly. |
Click to show internal directories.
Click to hide internal directories.