Documentation
¶
Overview ¶
Package fetch is tier 0: a plain HTTP GET, and the cheap checks that decide whether anything has changed since last time.
Why change detection needs its own ladder ¶
The artifact cache is keyed on content hash, which sounds like it makes re-distilling an unchanged site free. It does not, quite: computing that hash requires rendering the site, which is the expensive step. "Re-distilling an unchanged site is free" is really "storing an unchanged site is free" unless something cheaper can prove the page has not moved.
So there is a ladder, cheapest first, and most sites reveal the answer on the first or second rung:
- A conditional request. A 304 ends the question at almost no cost.
- Sitemap lastmod, where a sitemap exists.
- The raw HTML byte hash. Noisy -- it changes on every rebuild -- but a match is conclusive proof of no change.
- The static-text hash, taken after boilerplate removal. This survives asset-hash churn and build-id rotation, which is what makes rung 3 noisy.
- Render.
Only genuine ambiguity reaches rung 5.
Index ¶
Constants ¶
const UserAgent = "Mozilla/5.0 (compatible; sieve/0.1; +https://github.com/qcoderx/sieve)"
UserAgent identifies sieve honestly, with a contact URL.
Publishing an identity is permanent: once there is a documented user agent and a contact address, one badly behaved release makes the project recognisable and blockable forever. That is the argument for conservative rate limits that cannot be configured away, not for hiding.
Variables ¶
This section is empty.
Functions ¶
func HashString ¶
func LearnTTL ¶
LearnTTL adjusts how long a domain is trusted between checks.
This is a small table, not a model: a site that keeps coming back unchanged earns a longer interval, and one that changes earns a short one. The ceiling exists because an artifact that is a week stale without saying so is worse than one that costs a conditional request.
func StaticTextHash ¶
StaticTextHash is the rung-4 signal for a fetched document.
Types ¶
type ChangeVerdict ¶
type ChangeVerdict struct {
Changed bool
// Rung names which check answered, so the manifest can say how the
// freshness conclusion was reached rather than merely asserting it.
Rung string
// Confident reports whether the check is conclusive. A raw-hash match is
// conclusive; a raw-hash mismatch is not.
Confident bool
Response *Response
}
ChangeVerdict is the outcome of the ladder.
type Client ¶
type Client struct {
// contains filtered or unexported fields
}
Client performs tier-0 fetches.
func (*Client) CheckChanged ¶
func (c *Client) CheckChanged(ctx context.Context, rawURL string, prior *Freshness) (*ChangeVerdict, error)
CheckChanged walks the change-detection ladder.
type Freshness ¶
type Freshness struct {
ETag string `json:"etag,omitempty"`
LastModified string `json:"last_modified,omitempty"`
// RawHash is the byte hash of the served HTML. It changes on every rebuild
// even when nothing a reader would notice changed, so a mismatch proves
// nothing -- but a match is conclusive.
RawHash string `json:"raw_hash,omitempty"`
// TextHash is the hash of the extracted static text. It survives asset
// churn and build-id rotation, which is exactly what makes RawHash noisy.
TextHash string `json:"text_hash,omitempty"`
// ContentHash is the artifact's own semantic hash from the last run.
ContentHash string `json:"content_hash,omitempty"`
CheckedAt time.Time `json:"checked_at"`
// TTL is how long this domain has earned before being re-checked.
TTL time.Duration `json:"ttl,omitempty"`
}
Freshness is what a previous run stored so the next one can skip work.
type Options ¶
type Options struct {
// Timeout bounds a single request.
Timeout time.Duration
// MaxBytes bounds a response body. Remote input is untrusted, and an
// unbounded read from an arbitrary host is a memory-exhaustion primitive.
MaxBytes int64
// MaxRedirects bounds a redirect chain.
MaxRedirects int
// Guard vets every URL, including every redirect hop.
Guard *safety.Guard
// AcceptLanguage is pinned rather than inherited.
AcceptLanguage string
}
Options configures the fetcher.
func DefaultOptions ¶
func DefaultOptions() Options
DefaultOptions returns sane, conservative settings.
type Response ¶
type Response struct {
URL string
FinalURL string
Status int
Body []byte
ContentType string
// ETag and LastModified are the rung-1 signals, stored so the next run can
// make a conditional request.
ETag string
LastModified string
// NotModified reports that the server answered 304.
NotModified bool
// Blocked reports that the response looks like a refusal rather than a page.
Blocked bool
BlockedReason string
Elapsed time.Duration
// Redirects is the chain that was followed, recorded because a redirect to
// a login page is a common and otherwise invisible failure.
Redirects []string
}
Response is a fetched document plus the freshness signals it carried.