Documentation
¶
Overview ¶
Package fetch retrieves web content under egress, politeness, and rate controls, and classifies every attempt.
The guards are not bolted on around a fetcher — they are the fetcher. §3.3 and §3.4 require them present from the first commit precisely because retrofitting them means auditing every call site that has since appeared.
Index ¶
- Constants
- func DefaultDenyDomains() []string
- func DomainOf(rawURL string) string
- func HasStructuredData(html string) bool
- func ProviderSupplied(outcomes []*store.FetchOutcome) map[string]bool
- type Config
- type DomainDeny
- type Fetcher
- type Guard
- type GuardError
- type HTTPFetcher
- type Options
- type Outcome
- type Result
- type RobotsCache
- type Rules
Constants ¶
const DefaultUserAgent = "Mole/0.1 (+https://github.com/lajosdeme/mole)"
DefaultUserAgent identifies the crawler and points at the project.
const MaxCrawlDelay = 30 * time.Second
MaxCrawlDelay bounds what a site's Crawl-delay can cost us.
const MinUsableText = 250
MinUsableText is the extracted-character floor below which a page is treated as having produced nothing. Short legitimate pages exist, but below this a summarizer has nothing to work with either way.
Exported because the extractor and the search providers apply the same floor; three independent copies of this number would drift.
Variables ¶
This section is empty.
Functions ¶
func DefaultDenyDomains ¶
func DefaultDenyDomains() []string
DefaultDenyDomains ships with the binary. These are the sites whose terms clearly disallow automated access — the ones rev 1 removed first-party scrapers for. Removing the scraper without denying the domain would have left the exposure exactly where it was, just reached by a different code path.
func DomainOf ¶
registrableish reduces a host to something stable enough to group by in the per-cause domain ranking (§10.4).
It is NOT a public-suffix implementation: "bbc.co.uk" reduces to "co.uk". That is acceptable for ranking which domains cause which failures, and a real PSL is a dependency this does not yet justify. Revisit if the M2 report turns out to be misleading because of it. DomainOf reduces a URL to the same grouping key Result.Domain carries, so callers recording an outcome for a URL they never fetched bucket it identically to one they did.
func HasStructuredData ¶
HasStructuredData reports whether the document carries content in a machine-readable block that a parser could recover without executing JS.
OpenGraph deliberately does not qualify. og:title and og:description are metadata a page emits alongside its content, capped at a sentence or two — no parser turns them into the MinUsableText characters that would make this fetch OK. Counting them here filed every SPA as structured_only and held js_required near zero, which is the one number §17.1's decision reads.
func ProviderSupplied ¶
func ProviderSupplied(outcomes []*store.FetchOutcome) map[string]bool
ProviderSupplied is the set of URLs whose text came from the search provider rather than a fetch (§10.4).
Shared because re-reading one is a mistake two callers independently have to avoid: the text was extracted by the provider, so a fresh HTML extraction of the same URL produces different bytes and any comparison against it manufactures a mismatch. Both eval's citation accuracy and §11.5's grounding check need exactly this set, and the verifier's copy was written citing eval's reasoning without sharing its code.
Types ¶
type Config ¶
type Config struct {
// UserAgent identifies Mole. Required: an anonymous crawler is impolite and
// gives site operators no way to contact anyone or block selectively.
UserAgent string
MaxBytes int64
Timeout time.Duration
MaxRedirects int
// IgnoreRobots disables robots.txt. Off by default; every use is logged at
// WARN so it cannot happen quietly.
IgnoreRobots bool
// RobotsTTL is how long a robots.txt is cached.
RobotsTTL time.Duration
// PerDomain is the default rate limit applied to each host.
PerDomain limiter.Limit
}
Config tunes an HTTPFetcher.
type DomainDeny ¶
type DomainDeny struct {
// contains filtered or unexported fields
}
DomainDeny matches hosts against a suffix list.
func NewDomainDeny ¶
func NewDomainDeny(domains []string) *DomainDeny
func (*DomainDeny) Denied ¶
func (d *DomainDeny) Denied(host string) bool
Denied reports whether host is on the list, matching the domain and any subdomain of it but not a domain that merely ends with the same letters ("notlinkedin.com" is not "linkedin.com").
type Fetcher ¶
Fetcher retrieves a URL.
It returns a non-nil *Result even on failure, because the outcome of a failed fetch is data the project needs (§10.4) — a fetcher that returned only an error would throw away the input to the headless-browser decision.
type Guard ¶
type Guard struct {
// AllowedSchemes defaults to http and https.
AllowedSchemes []string
// AllowedPorts defaults to 80 and 443.
AllowedPorts []int
// ExtraDeny blocks additional ranges — an operator's internal supernets.
ExtraDeny []netip.Prefix
// AllowPrivateNetworks permits loopback and RFC1918 destinations.
//
// DANGEROUS. It exists so tests can reach an httptest server on 127.0.0.1
// and so an operator can deliberately point Mole at an intranet. Never
// enable it for a session that fetches attacker-influenced URLs.
//
// It unblocks loopback and RFC1918 and nothing else. Link-local,
// multicast, the unspecified address, and every prefix in deniedPrefixes
// stay refused — see CheckAddr. Cloud metadata stays unreachable either
// way, including via its NAT64 and 6to4 encodings.
AllowPrivateNetworks bool
// Resolver defaults to net.DefaultResolver.
Resolver *net.Resolver
// DialTimeout defaults to 10s.
DialTimeout time.Duration
}
Guard is the egress control on outbound fetches.
The Fetcher is the daemon's network position, and it fetches URLs chosen by a search engine and by an LLM. Without this, "summarize https://…" is a request to read the cloud metadata endpoint, an internal admin panel, or a service bound to loopback.
Two independent layers, because either alone has a hole:
- CheckURL rejects on scheme, port, and literal-IP host. It runs per redirect hop, since a public URL that 302s to 169.254.169.254 is the standard bypass.
- DialContext resolves the hostname, checks every returned address, and dials the address it checked. Checking and then handing the *name* to the dialer leaves a DNS-rebinding window where the second lookup returns a different address.
func (*Guard) CheckURL ¶
CheckURL validates scheme, port, and a literal-IP host. It does not resolve; DialContext does that, so that the check and the connection cannot disagree.
func (*Guard) DialContext ¶
DialContext resolves, checks every address, and connects to one it checked.
If ANY resolved address is denied the whole dial is refused, rather than picking an allowed one. A host that round-robins between a public and a private address would otherwise be reachable on retry.
type GuardError ¶
GuardError explains a refusal. Callers map it to FetchGuardDenied, which is deliberately not counted as evidence that a headless browser is needed (§17.1) — a blocked request is the system working.
func (*GuardError) Error ¶
func (e *GuardError) Error() string
type HTTPFetcher ¶
type HTTPFetcher struct {
// contains filtered or unexported fields
}
HTTPFetcher is the only web-fetch mechanism. There is no per-site adapter, and whether a headless browser is ever added is gated on §17.1.
func NewHTTP ¶
func NewHTTP(cfg Config, opts Options) *HTTPFetcher
NewHTTP builds a fetcher. A nil Guard gets the default (deny private networks), which is the safe direction for a missing argument.
func (*HTTPFetcher) Limiter ¶
func (f *HTTPFetcher) Limiter() *limiter.Limiter
Limiter exposes the rate limiter so callers can tighten a specific host.
type Options ¶
type Options struct {
Guard *Guard
Deny *DomainDeny
Lim *limiter.Limiter
Log *slog.Logger
// Transport lets the record/replay cassette layer wrap the guarded
// transport, so tests are deterministic and cost nothing.
Transport func(base http.RoundTripper) http.RoundTripper
}
Options are the collaborators an HTTPFetcher needs.
type Outcome ¶
type Outcome string
Outcome classifies why a fetch did or did not yield usable text.
This taxonomy is cheap to record and is the only thing that turns "do we need a headless browser?" from an argument into a number. Recording it is an M1 deliverable precisely because adding it later means re-running the whole eval corpus to get the data.
const ( OutcomeOK Outcome = "ok" // OutcomeProviderContent is a source read without fetching it, because the // search provider already returned usable page text. // // Distinct from ok on purpose. Every per-cause rate in §10.4 is a fraction // of attempted fetches, and filing these as ok inflated that denominator // with requests never made — dragging js_required and bot_block down by // however much of the corpus the provider happened to cover. OutcomeProviderContent Outcome = "provider_content" // OutcomeJSRequired is the SPA signature: the extractor produced almost // nothing, but the document is script-heavy and has an app-root element. // This is the ONLY outcome that counts as evidence for a headless browser. OutcomeJSRequired Outcome = "js_required" // OutcomeStructuredOnly means readability failed but __NEXT_DATA__, // JSON-LD, or OpenGraph carried the content. A day of parser work converts // these into OK — which is why they must not be lumped in with js_required. OutcomeStructuredOnly Outcome = "structured_only" OutcomeConsentWall Outcome = "consent_wall" OutcomeBotBlock Outcome = "bot_block" OutcomePaywall Outcome = "paywall" OutcomeNotFound Outcome = "not_found" OutcomeTimeout Outcome = "timeout" // OutcomeRobotsDenied and OutcomeGuardDenied are the system working // correctly. They are separate constants so they can be excluded from any // "failure rate" that informs a capability decision; folding them in would // inflate the apparent case for a browser. OutcomeRobotsDenied Outcome = "robots_denied" OutcomeGuardDenied Outcome = "guard_denied" OutcomeExtractFailed Outcome = "extract_failed" OutcomeServerError Outcome = "server_error" OutcomeTooLarge Outcome = "too_large" OutcomeUnsupported Outcome = "unsupported_type" OutcomeNetworkError Outcome = "network_error" )
func ClassifyBody ¶
ClassifyBody decides the outcome for a 2xx response whose extraction has already been attempted.
extractedLen is the length of the text a readability pass produced. Order matters: bot-block and consent markers are checked before the SPA signature, because a challenge page is also script-heavy with an empty body and would otherwise be miscounted as js_required — inflating the one number §17.1's decision actually turns on.
func ClassifyStatus ¶
ClassifyStatus maps a non-2xx response to an outcome.
func (Outcome) Attempted ¶
Attempted reports whether a request was actually made. It is the denominator for every per-cause rate: provider_content never touched the network.
func (Outcome) CapabilityGap ¶
CapabilityGap reports whether this outcome is evidence that Mole lacks a capability, as opposed to the system correctly declining or the server failing. Only js_required qualifies — see §17.1's gate.
func (Outcome) SystemWorking ¶
SystemWorking reports whether the outcome is a deliberate refusal by Mole rather than a failure.
type Result ¶
type Result struct {
URL string
Domain string
Outcome Outcome
StatusCode int
Bytes int64
Duration time.Duration
Err string
// ContentType and Content are populated on a usable fetch.
ContentType string
Content []byte
// FinalURL differs from URL when redirects were followed.
FinalURL string
}
Result is one fetch attempt, recorded whether or not it succeeded.
type RobotsCache ¶
type RobotsCache struct {
// Fetch retrieves a robots.txt body. Injected so the cache can be tested
// without a network and so the real one reuses the guarded transport.
Fetch func(ctx context.Context, robotsURL string) (status int, body []byte, err error)
// contains filtered or unexported fields
}
RobotsCache fetches and caches robots.txt per origin.
func NewRobotsCache ¶
func (*RobotsCache) Get ¶
func (c *RobotsCache) Get(ctx context.Context, origin string) *Rules
Get returns the rules for an origin ("https://example.com").
Concurrent callers for the same origin share one fetch. The entry must be created once and then mutated in place — replacing it on each cache miss hands every caller its own sync.Once, which is a stampede wearing the costume of a singleflight.
type Rules ¶
type Rules struct {
// contains filtered or unexported fields
}
Rules is a parsed robots.txt.
func AllowAllRules ¶
func AllowAllRules() *Rules
AllowAllRules is used when a site has no robots.txt (404), which the standard treats as unrestricted.
func DenyAllRules ¶
func DenyAllRules() *Rules
DenyAllRules is used when robots.txt could not be read due to a server error. The standard treats a 5xx as a full disallow; erring the other way would mean a flaky server silently converts into permission we were never given.
func (*Rules) Allowed ¶
Allowed reports whether userAgent may fetch path.
Precedence follows the standard: the longest matching pattern wins, and on a tie Allow beats Disallow. Getting this backwards would silently over-block (harmless but useless) or under-block (the thing we are trying not to do).
Source Files