httpx

package
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 9, 2026 License: Apache-2.0 Imports: 24 Imported by: 0

Documentation

Overview

Package httpx gives source plugins HTTP clients that follow the network profiles their host pushes per source (SetNetworkProfile): the proxy (http, https or socks5, with credentials), a challenge solver (FlareSolverr/Byparr), a rate limit (per source or per host), a concurrency cap, the user agent, extra headers, DNS over HTTPS, a cookie jar (its own or shared with other sources) and timeouts.

A plugin keeps one Sources and asks it for each source's Client; its SetNetworkProfile implements the RPC. A Client's *http.Client stays the same; requests made after a new profile use the new settings.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func IsChallenge

func IsChallenge(err error) bool

IsChallenge reports whether err is (or wraps) a ChallengeError.

func StatusError

func StatusError(resp *http.Response) error

StatusError turns an unsuccessful response into the error the source API expects: 429 → RATE_LIMITED (with Retry-After), 404 → NOT_FOUND, 401 and 403 → LOGIN_REQUIRED, others → SOURCE_DOWN. A 2xx or 3xx response gives nil. The body is left to the caller.

func Wrap

func Wrap(err error) error

Wrap turns an error from a request made with a Client into the error the source API expects: a challenge that couldn't be passed → CLOUDFLARE_CHALLENGE, network failures and timeouts → SOURCE_DOWN. Plugin errors and context cancellation pass through.

Types

type ChallengeError

type ChallengeError struct {
	Host   string
	Reason string
}

ChallengeError means a site answered with an anti-bot challenge that couldn't be passed.

func (*ChallengeError) Error

func (e *ChallengeError) Error() string

type Client

type Client struct {
	// contains filtered or unexported fields
}

Client is an HTTP client that follows a network profile.

func New

func New(opts Options) *Client

New returns a client with a direct connection until SetProfile is called.

func (*Client) HTTP

func (c *Client) HTTP() *http.Client

HTTP returns the client to make requests with. It is the same value for the life of the Client.

func (*Client) Profile

func (c *Client) Profile() *sourcev1.NetworkProfile

Profile returns the profile in effect (nil when direct). Don't change it.

func (*Client) SetProfile

func (c *Client) SetProfile(p *sourcev1.NetworkProfile) error

SetProfile switches to a profile; nil means a direct connection with the plugin's defaults. Connections made under the old profile are closed.

type Options

type Options struct {
	// UserAgent is the plugin's own user agent, used unless the profile
	// says otherwise.
	UserAgent string
	// Defaults when the profile sets no timeout. Zero means none.
	ConnectTimeout time.Duration
	ReadTimeout    time.Duration
	CallTimeout    time.Duration
	Log            *slog.Logger
	// SharedJar returns the cookie jar shared under a name, for profiles
	// that set shared_cookie_jar. [Sources] fills it in; without it every
	// client keeps its own jar.
	SharedJar func(name string) http.CookieJar
}

Options configure a Client.

type Sources

type Sources struct {
	// contains filtered or unexported fields
}

Sources keeps one Client per source, each following the network profile the host pushed for that source, or the default profile when none was pushed. Sources with the same shared_cookie_jar name share cookies.

func NewSources

func NewSources(opts Options) *Sources

NewSources returns an empty set; clients are made on first use.

func (*Sources) Client

func (s *Sources) Client(sourceID string) *Client

Client returns the client for a source.

func (*Sources) HTTP

func (s *Sources) HTTP(sourceID string) *http.Client

HTTP returns the *http.Client for a source.

func (*Sources) SetDefault

func (s *Sources) SetDefault(p *sourcev1.NetworkProfile) error

SetDefault sets the profile of sources the host pushed none for, e.g. a plugin's own configuration. nil means a direct connection.

func (*Sources) SetNetworkProfile

SetNetworkProfile implements the plugin RPC of the same name, so a plugin can pass it straight through:

func (p *Plugin) SetNetworkProfile(ctx context.Context, r *sourcev1.SetNetworkProfileRequest) (*sourcev1.SetNetworkProfileResponse, error) {
	return p.net.SetNetworkProfile(ctx, r)
}

Check that the source exists before, if the plugin cares.

func (*Sources) SetProfile

func (s *Sources) SetProfile(sourceID string, p *sourcev1.NetworkProfile) error

SetProfile applies a pushed profile to a source; nil removes it, so the default applies again.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL