fetch

package
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 17, 2026 License: Apache-2.0 Imports: 13 Imported by: 0

Documentation

Overview

Package fetch is tier 0: a plain HTTP GET, and the cheap checks that decide whether anything has changed since last time.

Why change detection needs its own ladder

The artifact cache is keyed on content hash, which sounds like it makes re-distilling an unchanged site free. It does not, quite: computing that hash requires rendering the site, which is the expensive step. "Re-distilling an unchanged site is free" is really "storing an unchanged site is free" unless something cheaper can prove the page has not moved.

So there is a ladder, cheapest first, and most sites reveal the answer on the first or second rung:

  1. A conditional request. A 304 ends the question at almost no cost.
  2. Sitemap lastmod, where a sitemap exists.
  3. The raw HTML byte hash. Noisy -- it changes on every rebuild -- but a match is conclusive proof of no change.
  4. The static-text hash, taken after boilerplate removal. This survives asset-hash churn and build-id rotation, which is what makes rung 3 noisy.
  5. Render.

Only genuine ambiguity reaches rung 5.

Index

Constants

View Source
const UserAgent = "Mozilla/5.0 (compatible; sieve/0.1; +https://github.com/qcoderx/sieve)"

UserAgent identifies sieve honestly, with a contact URL.

Publishing an identity is permanent: once there is a documented user agent and a contact address, one badly behaved release makes the project recognisable and blockable forever. That is the argument for conservative rate limits that cannot be configured away, not for hiding.

Variables

This section is empty.

Functions

func HashBytes

func HashBytes(b []byte) string

HashBytes and HashString produce the stored freshness signals.

func HashString

func HashString(s string) string

func LearnTTL

func LearnTTL(prev time.Duration, changed bool) time.Duration

LearnTTL adjusts how long a domain is trusted between checks.

This is a small table, not a model: a site that keeps coming back unchanged earns a longer interval, and one that changes earns a short one. The ceiling exists because an artifact that is a week stale without saying so is worse than one that costs a conditional request.

func StaticTextHash

func StaticTextHash(r *static.Result) string

StaticTextHash is the rung-4 signal for a fetched document.

Types

type ChangeVerdict

type ChangeVerdict struct {
	Changed bool
	// Rung names which check answered, so the manifest can say how the
	// freshness conclusion was reached rather than merely asserting it.
	Rung string
	// Confident reports whether the check is conclusive. A raw-hash match is
	// conclusive; a raw-hash mismatch is not.
	Confident bool
	Response  *Response
}

ChangeVerdict is the outcome of the ladder.

type Client

type Client struct {
	// contains filtered or unexported fields
}

Client performs tier-0 fetches.

func New

func New(opts Options) *Client

New builds a client whose transport refuses to connect to addresses the guard rejects.

func (*Client) CheckChanged

func (c *Client) CheckChanged(ctx context.Context, rawURL string, prior *Freshness) (*ChangeVerdict, error)

CheckChanged walks the change-detection ladder.

func (*Client) Get

func (c *Client) Get(ctx context.Context, rawURL string, prior *Freshness) (*Response, error)

Get fetches a URL, optionally conditionally.

type Freshness

type Freshness struct {
	ETag         string `json:"etag,omitempty"`
	LastModified string `json:"last_modified,omitempty"`
	// RawHash is the byte hash of the served HTML. It changes on every rebuild
	// even when nothing a reader would notice changed, so a mismatch proves
	// nothing -- but a match is conclusive.
	RawHash string `json:"raw_hash,omitempty"`
	// TextHash is the hash of the extracted static text. It survives asset
	// churn and build-id rotation, which is exactly what makes RawHash noisy.
	TextHash string `json:"text_hash,omitempty"`
	// ContentHash is the artifact's own semantic hash from the last run.
	ContentHash string    `json:"content_hash,omitempty"`
	CheckedAt   time.Time `json:"checked_at"`
	// TTL is how long this domain has earned before being re-checked.
	TTL time.Duration `json:"ttl,omitempty"`
}

Freshness is what a previous run stored so the next one can skip work.

type Options

type Options struct {
	// Timeout bounds a single request.
	Timeout time.Duration
	// MaxBytes bounds a response body. Remote input is untrusted, and an
	// unbounded read from an arbitrary host is a memory-exhaustion primitive.
	MaxBytes int64
	// MaxRedirects bounds a redirect chain.
	MaxRedirects int
	// Guard vets every URL, including every redirect hop.
	Guard *safety.Guard
	// AcceptLanguage is pinned rather than inherited.
	AcceptLanguage string
}

Options configures the fetcher.

func DefaultOptions

func DefaultOptions() Options

DefaultOptions returns sane, conservative settings.

type Response

type Response struct {
	URL         string
	FinalURL    string
	Status      int
	Body        []byte
	ContentType string
	// ETag and LastModified are the rung-1 signals, stored so the next run can
	// make a conditional request.
	ETag         string
	LastModified string
	// NotModified reports that the server answered 304.
	NotModified bool
	// Blocked reports that the response looks like a refusal rather than a page.
	Blocked       bool
	BlockedReason string
	Elapsed       time.Duration
	// Redirects is the chain that was followed, recorded because a redirect to
	// a login page is a common and otherwise invisible failure.
	Redirects []string
}

Response is a fetched document plus the freshness signals it carried.

func (*Response) IsHTML

func (r *Response) IsHTML() bool

IsHTML reports whether a response is worth parsing as a document.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL