hq

package

v2.0.26 Latest Latest Go to latest Published: Jun 2, 2026 License: AGPL-3.0 Imports: 23 Imported by: 0

Details

Valid go.mod file
Redistributable license
Tagged version
Stable version
Learn more about best practices

Repository

github.com/internetarchive/Zeno

Links

Open Source Insights

Documentation ¶

Overview ¶

Package hq provides a way to interact with the HQv3 API and consumes, produces and mark items as finished asynchronusly.

Index ¶

Variables
type HQ
- func New(HQKey, HQSecret, HQProject, HQAddress string, timeout, seencheckCacheSize int, ...) *HQ

Constants ¶

This section is empty.

Variables ¶

View Source

var (
	// ErrHQAlreadyInitialized is the error returned when the postprocessor is already initialized
	ErrHQAlreadyInitialized = errors.New("hq client already initialized")
)

Functions ¶

This section is empty.

Types ¶

type HQ ¶

type HQ struct {
	HQKey        string
	HQSecret     string
	HQProject    string
	HQAddress    string
	Timeout      int
	GZIPRequests bool
	SeencheckURL string
	// contains filtered or unexported fields
}

func New ¶

func New(HQKey, HQSecret, HQProject, HQAddress string, timeout, seencheckCacheSize int, gzipRequests bool, seencheckURL string) *HQ

func (*HQ) Name ¶

func (s *HQ) Name() string

Name returns the name of the source, used for logging and identification.

func (*HQ) SeencheckItem ¶

func (s *HQ) SeencheckItem(item *models.Item) error

SeencheckItem gets the MaxDepth children of the given item and sends a seencheck request to the crawl HQ for the URLs found. The items that were seen before will be marked as seen. A local otter cache is used to avoid sending redundant seencheck requests to HQ for URLs that have already been checked.

func (*HQ) Start ¶

func (s *HQ) Start(finishChan, produceChan chan *models.Item) error

Start initializes HQ async routines with the given input and output channels.

func (*HQ) Stop ¶

func (s *HQ) Stop()

Stop stops the global HQ and waits for all goroutines to finish. Finisher must be stopped first and Reactor must be frozen before stopping HQ.

Source Files ¶

View all Source files

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL