Documentation
¶
Overview ¶
Package fetch is the chassis's safe public-HTTP fetcher: one GET of a public web resource on a stack's behalf, bounded in every dimension.
It is deliberately narrow. The URL must be http(s) on port 80 or 443 with no credentials; the host is resolved here and every candidate address goes through the chassis egress guard before it is dialed, so the address checked is the address connected (no second lookup for DNS rebinding to exploit), and every redirect hop is checked the same way. There is no proxy, no cookie jar, no authentication, no request body; redirects, header bytes, body bytes and the total time are capped.
It knows nothing about what the body is for: callers (txco://html/ extract) parse it. Errors are *Error values with a stable Code and a message that never names an address.
Index ¶
Constants ¶
const ( CodeInvalidURL = "invalid_url" CodeDestinationDenied = "destination_denied" CodeDNSFailed = "dns_failed" CodeConnectionFailed = "connection_failed" CodeTimeout = "timeout" CodeRedirectLimit = "redirect_limit" CodeResponseTooLarge = "response_too_large" CodeUnsupportedType = "unsupported_content_type" CodeUnsupportedEncoding = "unsupported_encoding" CodeHTTPError = "http_error" )
Error codes. Callers prefix them with their op family (txco_html_…).
const ( DefaultMaxBytes = 3 << 20 DefaultMaxRedirects = 3 DefaultTimeout = 5 * time.Second DefaultMaxHeaderBytes = 64 << 10 )
Defaults for the zero values in Options.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type DialFunc ¶
DialFunc connects to an address the guard has already approved: always an "ip:port", never a hostname.
type Error ¶
type Error struct {
Code string
Message string
Status int
ContentType string
// Bytes is how many body bytes were read before the failure (for
// metering a refused oversize body).
Bytes int64
}
Error is a failed fetch. Message is fixed text written for the rule author: it never names an address, a resolver or a connection detail. Status and ContentType are set when a response arrived.
type Fetcher ¶
type Fetcher struct {
// contains filtered or unexported fields
}
Fetcher is safe for concurrent use. Its connection pool is keyed by scheme and host, so a reused connection is one dialed (and checked) for that same host.
type Options ¶
type Options struct {
Guard egress.Guard
MaxBytes int64 // body bytes; one more fails the fetch
MaxRedirects int // hops followed; one more fails the fetch
Timeout time.Duration // the whole fetch: dial, TLS, headers, body
// MediaTypes are the Content-Type media types accepted (lowercase,
// no parameters). Empty accepts any.
MediaTypes []string
// Accept is sent as the Accept header.
Accept string
// Resolver, Dial and TLSConfig are seams for tests; nil means the
// system resolver, a guarded TCP dial and the system roots.
Resolver Resolver
Dial DialFunc
TLSConfig *tls.Config
}
Options configure a Fetcher. Guard is required.
type Resolver ¶
type Resolver interface {
LookupNetIP(ctx context.Context, network, host string) ([]netip.Addr, error)
}
Resolver looks a host up. net.DefaultResolver satisfies it; tests substitute their own answers.
type Result ¶
type Result struct {
URL string // as requested (normalized)
FinalURL string // after redirects
Status int
ContentType string // the header as sent
Body []byte
Redirects int
Bytes int64 // body bytes read off the wire (compressed, if it was)
}
Result is a successful fetch: a 2xx response of an accepted media type whose body fit.
Source Files