Documentation
¶
Overview ¶
Package fetcher provides an HTTP client with header rotation, proxy support, cookie handling, and retry logic. It produces unified Response objects that integrate with the parser for immediate HTML extraction.
Index ¶
- type Fetcher
- func (f *Fetcher) Close()
- func (f *Fetcher) Do(ctx context.Context, method, rawURL string, body io.Reader) (*Response, error)
- func (f *Fetcher) Get(ctx context.Context, rawURL string) (*Response, error)
- func (f *Fetcher) Post(ctx context.Context, rawURL string, body io.Reader) (*Response, error)
- func (f *Fetcher) SetCookies(rawURL string, cookies []*http.Cookie) error
- type Option
- type Response
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Fetcher ¶
type Fetcher struct {
// contains filtered or unexported fields
}
Fetcher is an HTTP client with browser-like header rotation and retry logic.
func (*Fetcher) Do ¶
Do performs an HTTP request with retry logic and header rotation. It retries on network errors and HTTP 5xx responses. For POST/PUT/PATCH requests with a body, the body is buffered so it can be replayed on retries.
type Option ¶
type Option func(*Fetcher)
Option configures a Fetcher.
func WithHeaders ¶
WithHeaders sets custom HTTP headers.
func WithMaxRetries ¶
WithMaxRetries sets the maximum number of retries.
func WithTimeout ¶
WithTimeout sets the HTTP request timeout.
type Response ¶
type Response struct {
URL string
StatusCode int
Headers http.Header
Body []byte
Encoding string
Cookies []*http.Cookie
}
Response wraps an HTTP response with parsed HTML body access.
func (*Response) ContentLength ¶
ContentLength returns the Content-Length header as int, or -1 if not set or if the value overflows.
func (*Response) ContentType ¶
ContentType returns the Content-Type header value.
Source Files