Documentation
¶
Overview ¶
Package fetch is quarry's optional polite HTTP client: cross-goroutine rate limiting, bounded retries with backoff, and Retry-After respect, in front of a plain GET. It is the only quarry package that touches the network, and it is deliberately outside the deterministic core — import it for the batteries-included case, or ignore it and feed quarry HTML from anywhere.
It is not a crawler: no link discovery, no frontier, no robots.txt. One page per call, politely.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Client ¶
type Client struct {
// contains filtered or unexported fields
}
Client is a polite HTTP client. The zero value is not usable; construct with New. A Client is safe for concurrent use; the rate limit applies across all goroutines sharing it.
func (*Client) Get ¶
Get fetches one URL and returns the response body. Each attempt (including retries) takes a rate-limit slot; retry waits honor a Retry-After header when it demands more patience than the default exponential backoff (0.5s, 1s, 2s, ... capped at 30s). A Retry-After beyond 5 minutes stops retrying and returns the error. Context cancellation interrupts rate-limit waits, backoff waits, and the request itself.
type Option ¶
type Option func(*Client)
Option configures a Client.
func WithHTTPClient ¶
WithHTTPClient substitutes the underlying *http.Client (proxies, cookies, custom transports, timeouts). The default has a 30s timeout.
func WithMaxBodySize ¶
WithMaxBodySize caps the response body; a larger body is an error, not a truncation — truncated HTML extracts garbage silently, and silence is the enemy. The default is no cap.
func WithMinInterval ¶
WithMinInterval enforces at least d between request starts, shared across goroutines. The default is no rate limit.
func WithRetries ¶
WithRetries sets how many times a failed request is retried (beyond the first attempt). Only transport errors, 429 and 5xx responses are retried. The default is 2.
func WithUserAgent ¶
WithUserAgent sets the User-Agent header. Set one that identifies your project; the default identifies quarry.
type StatusError ¶
StatusError reports a non-2xx response. Responses that exhaust retries (429, 5xx) and responses that were never retryable (other 4xx) both surface as *StatusError; use errors.As.
func (*StatusError) Error ¶
func (e *StatusError) Error() string
Source Files