Documentation
¶
Overview ¶
Package run wires the stages together: it streams a seed source through the fetch engine on a large worker pool, then funnels every result to a single pack consumer that writes the captures. Reading the network is the bottleneck, so one writer keeps the output files simple and still keeps up with thousands of concurrent fetches.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func FetchBatch ¶ added in v0.2.0
func FetchBatch(ctx context.Context, cfg config.Config, urls []string, out chan<- fetch.Result) error
FetchBatch re-fetches a list of URLs using the same engine as Run. Results are written to out as they complete. FetchBatch blocks until all URLs have been attempted or ctx is cancelled, then returns ctx.Err() (nil if the list drained normally).
Types ¶
type Runner ¶
type Runner struct {
// contains filtered or unexported fields
}
Runner executes one crawl end to end.
Source Files
Click to show internal directories.
Click to hide internal directories.