Documentation
¶
Overview ¶
Package httpx gives source plugins HTTP clients that follow the network profiles their host pushes per source (SetNetworkProfile): the proxy (http, https or socks5, with credentials), a challenge solver (FlareSolverr/Byparr), a rate limit (per source or per host), a concurrency cap, the user agent, extra headers, DNS over HTTPS, a cookie jar (its own or shared with other sources) and timeouts.
A plugin keeps one Sources and asks it for each source's Client; its SetNetworkProfile implements the RPC. A Client's *http.Client stays the same; requests made after a new profile use the new settings.
Index ¶
- func IsChallenge(err error) bool
- func StatusError(resp *http.Response) error
- func Wrap(err error) error
- type ChallengeError
- type Client
- type Options
- type Sources
- func (s *Sources) Client(sourceID string) *Client
- func (s *Sources) HTTP(sourceID string) *http.Client
- func (s *Sources) SetDefault(p *sourcev1.NetworkProfile) error
- func (s *Sources) SetNetworkProfile(_ context.Context, req *sourcev1.SetNetworkProfileRequest) (*sourcev1.SetNetworkProfileResponse, error)
- func (s *Sources) SetProfile(sourceID string, p *sourcev1.NetworkProfile) error
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func IsChallenge ¶
IsChallenge reports whether err is (or wraps) a ChallengeError.
func StatusError ¶
StatusError turns an unsuccessful response into the error the source API expects: 429 → RATE_LIMITED (with Retry-After), 404 → NOT_FOUND, 401 and 403 → LOGIN_REQUIRED, others → SOURCE_DOWN. A 2xx or 3xx response gives nil. The body is left to the caller.
Types ¶
type ChallengeError ¶
ChallengeError means a site answered with an anti-bot challenge that couldn't be passed.
func (*ChallengeError) Error ¶
func (e *ChallengeError) Error() string
type Client ¶
type Client struct {
// contains filtered or unexported fields
}
Client is an HTTP client that follows a network profile.
func (*Client) HTTP ¶
HTTP returns the client to make requests with. It is the same value for the life of the Client.
func (*Client) Profile ¶
func (c *Client) Profile() *sourcev1.NetworkProfile
Profile returns the profile in effect (nil when direct). Don't change it.
func (*Client) SetProfile ¶
func (c *Client) SetProfile(p *sourcev1.NetworkProfile) error
SetProfile switches to a profile; nil means a direct connection with the plugin's defaults. Connections made under the old profile are closed.
type Options ¶
type Options struct {
// UserAgent is the plugin's own user agent, used unless the profile
// says otherwise.
UserAgent string
// Defaults when the profile sets no timeout. Zero means none.
ConnectTimeout time.Duration
ReadTimeout time.Duration
CallTimeout time.Duration
Log *slog.Logger
// that set shared_cookie_jar. [Sources] fills it in; without it every
// client keeps its own jar.
SharedJar func(name string) http.CookieJar
}
Options configure a Client.
type Sources ¶
type Sources struct {
// contains filtered or unexported fields
}
Sources keeps one Client per source, each following the network profile the host pushed for that source, or the default profile when none was pushed. Sources with the same shared_cookie_jar name share cookies.
func NewSources ¶
NewSources returns an empty set; clients are made on first use.
func (*Sources) SetDefault ¶
func (s *Sources) SetDefault(p *sourcev1.NetworkProfile) error
SetDefault sets the profile of sources the host pushed none for, e.g. a plugin's own configuration. nil means a direct connection.
func (*Sources) SetNetworkProfile ¶
func (s *Sources) SetNetworkProfile(_ context.Context, req *sourcev1.SetNetworkProfileRequest) (*sourcev1.SetNetworkProfileResponse, error)
SetNetworkProfile implements the plugin RPC of the same name, so a plugin can pass it straight through:
func (p *Plugin) SetNetworkProfile(ctx context.Context, r *sourcev1.SetNetworkProfileRequest) (*sourcev1.SetNetworkProfileResponse, error) {
return p.net.SetNetworkProfile(ctx, r)
}
Check that the source exists before, if the plugin cares.
func (*Sources) SetProfile ¶
func (s *Sources) SetProfile(sourceID string, p *sourcev1.NetworkProfile) error
SetProfile applies a pushed profile to a source; nil removes it, so the default applies again.