Versions in this module Expand all Collapse all v0 v0.1.0 Mar 22, 2026 Changes in this version + type Crawler struct + func New(fetcher *Fetcher, store store.Store, src source.Source, workers int) *Crawler + func (c *Crawler) Run(ctx context.Context, opts RunOptions) error + type Fetcher struct + func NewFetcher(requestsPerSecond int, maxRetry int, userAgent string) *Fetcher + func (f *Fetcher) Close() + func (f *Fetcher) Fetch(ctx context.Context, url string) ([]byte, error) + type RunOptions struct + Resume bool + RetryFailed bool
Go to main page