Documentation
¶
Overview ¶
Package fetch is the network engine: it turns a stream of seeds into a stream of results as fast as one machine sustains. It owns DNS, a sharded pool of keep-alive transports, per-host and per-IP concurrency limits, an adaptive timeout, and dead-host/dead-domain tracking so a run does not waste workers on targets that will never answer.
Index ¶
Constants ¶
This section is empty.
Variables ¶
var ErrCongested = errors.New("gave up after congestion retries")
ErrCongested is recorded when a fetch still fails after exhausting its congestion retries. It is counted as a failure, not a skip, and it never trips the dead-domain breaker: the host may be fine, we simply could not get through within the retry budget.
Functions ¶
Types ¶
type Fetcher ¶
type Fetcher struct {
// contains filtered or unexported fields
}
Fetcher performs concurrent re-fetches with post-fetch digest comparison.
func (*Fetcher) Fetch ¶
Fetch performs one re-fetch and returns its Result, setting Unchanged when the fetched body's sha1 matches the seed's prior digest. It never returns an error itself; failures are reported in Result.Err.
type Result ¶
type Result struct {
URL string
FetchedAt time.Time
// Set on a completed HTTP exchange.
Status int
Header http.Header
ReqHeader http.Header
Body []byte
Digest string // sha1 hex of Body
// ETag and LastModified are the response validators, carried into the
// capture index so a later recrawl can issue a conditional request. On a
// 304 they fall back to the validators the request sent.
ETag string
LastModified string
// Unchanged is true when Digest matches the seed's prior digest, so the
// content did not change since it was last captured.
Unchanged bool
// Meta is carried verbatim from the seed.
Meta map[string]string
// Err is non-nil for transport/DNS/timeout failures (no HTTP response).
Err error
}
Result is the outcome of fetching one seed.
type SeedURL ¶
SeedURL is the subset of a seed the fetcher needs, decoupling fetch from the seed package's concrete type.
Digest, ETag, and ModTime are validators from a prior capture of this URL. Digest drives post-fetch content comparison; ETag and ModTime drive conditional requests (If-None-Match / If-Modified-Since) so an unchanged page comes back as a bodiless 304.
Source Files