Documentation
¶
Overview ¶
Package run wires the stages together: it streams a seed source through the fetch engine on a large worker pool, then funnels every result to a single pack consumer that writes the captures. Reading the network is the bottleneck, so one writer keeps the output files simple and still keeps up with thousands of concurrent fetches.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func FetchBatch ¶ added in v0.2.0
func FetchBatch(ctx context.Context, cfg config.Config, urls []string, out chan<- fetch.Result) error
FetchBatch re-fetches a list of URLs using the same engine as Run. Results are written to out as they complete. FetchBatch blocks until all URLs have been attempted or ctx is cancelled, then returns ctx.Err() (nil if the list drained normally).
When cfg.Reorder is on (the default) the URL list is spread across hosts before it reaches the workers, so a host-clustered input (a raw Common Crawl shard arrives SURT-sorted, with long runs of same-host URLs) does not pin the pool against the per-host concurrency cap while most workers sit idle. This makes FetchBatch honor Reorder the way the full Run path does, so a caller no longer has to pre-spread the list itself.
Types ¶
type Runner ¶
type Runner struct {
// contains filtered or unexported fields
}
Runner executes one crawl end to end.
Source Files