Documentation
¶
Overview ¶
Package proxy manages HTTP/SOCKS proxy pools, health checking, and rotation strategies for the Foxhound scraping framework.
Index ¶
- Variables
- type HealthChecker
- type Pool
- func (p *Pool) Ban(px *Proxy, domain string)
- func (p *Pool) Close() error
- func (p *Pool) Get(ctx context.Context) (*Proxy, error)
- func (p *Pool) GetForDomain(ctx context.Context, domain string) (*Proxy, error)
- func (p *Pool) GetForGeo(ctx context.Context, country, city string) (*Proxy, error)
- func (p *Pool) GetForSession(ctx context.Context, sessionID string) (*Proxy, error)
- func (p *Pool) Health(px *Proxy) ProxyHealth
- func (p *Pool) Len() int
- func (p *Pool) Release(px *Proxy, success bool)
- func (p *Pool) SetCooldown(d time.Duration)
- func (p *Pool) SetMaxRequests(n int)
- func (p *Pool) SetRotation(strategy RotationStrategy)
- type Provider
- type Proxy
- type ProxyHealth
- type RotationStrategy
- type StaticProvider
Constants ¶
This section is empty.
Variables ¶
var ErrNoProxy = errors.New("proxy: no proxy available")
ErrNoProxy is returned when the pool has no usable proxy available.
Functions ¶
This section is empty.
Types ¶
type HealthChecker ¶
type HealthChecker struct {
// contains filtered or unexported fields
}
HealthChecker probes proxies against a test endpoint to determine liveness and measure latency.
func NewHealthChecker ¶
func NewHealthChecker(checkURL string, timeout time.Duration) *HealthChecker
NewHealthChecker creates a HealthChecker that probes proxies by requesting checkURL. timeout is applied to each individual probe.
func (*HealthChecker) Check ¶
func (hc *HealthChecker) Check(ctx context.Context, p *Proxy) ProxyHealth
Check probes a single proxy and returns its updated health. The ProxyHealth.Score is set to 1.0 for alive proxies and 0.0 for dead ones.
The request is routed through the proxy under test so that the latency and reachability measurement reflects actual proxy connectivity.
type Pool ¶
type Pool struct {
// contains filtered or unexported fields
}
Pool manages a set of proxies sourced from one or more Provider instances. It tracks health, enforces cooldowns, and selects the best proxy on each Get.
func NewPool ¶
NewPool creates a Pool pre-loaded with proxies from the given providers. Providers are queried once during construction using a background context. If a provider fails it is logged and skipped.
func (*Pool) Get ¶
Get returns a proxy according to the configured rotation strategy. It blocks until a proxy is available or the context is cancelled.
When all proxies are on cooldown it sleeps until the earliest cooldown expiry rather than busy-waiting. If the pool is empty it returns ErrNoProxy immediately.
func (*Pool) GetForDomain ¶
GetForDomain returns a sticky proxy for the given domain. The same proxy is returned on every subsequent call with the same domain as long as the proxy remains available. If the sticky proxy is no longer available a new one is assigned.
func (*Pool) GetForGeo ¶
GetForGeo returns a proxy matching the requested country (and optionally city). Falls back to any available proxy if no geo match is found.
func (*Pool) GetForSession ¶
GetForSession returns a sticky proxy for the given sessionID. The same proxy is returned on every subsequent call with the same sessionID as long as the proxy remains available. If the sticky proxy is no longer available a new one is assigned.
func (*Pool) Health ¶
func (p *Pool) Health(px *Proxy) ProxyHealth
Health returns a snapshot of the ProxyHealth for the given proxy.
func (*Pool) Release ¶
Release returns a proxy to the pool after use, updating its health score. success=true increments the success rate; success=false decreases the score. When maxRequests > 0 and the proxy has now reached that limit it is placed on cooldown automatically.
func (*Pool) SetCooldown ¶
SetCooldown changes the duration proxies remain on cooldown after being banned.
func (*Pool) SetMaxRequests ¶
SetMaxRequests sets the maximum number of requests a proxy handles before automatic cooldown. 0 means unlimited.
func (*Pool) SetRotation ¶
func (p *Pool) SetRotation(strategy RotationStrategy)
SetRotation changes the rotation strategy. Safe to call at any time.
type Provider ¶
type Provider interface {
// Proxies returns the current list of available proxies.
Proxies(ctx context.Context) ([]*Proxy, error)
}
Provider is the source of proxy addresses for a pool.
type Proxy ¶
type Proxy struct {
// URL is the full proxy URL, e.g. "http://user:pass@host:port".
URL string
// Protocol is one of "http", "https", or "socks5".
Protocol string
// Host is the proxy hostname or IP address.
Host string
// Port is the proxy port number as a string.
Port string
// Username is the proxy auth username (may be empty).
Username string
// Password is the proxy auth password (may be empty).
Password string
// Country is the ISO 3166-1 alpha-2 country code of the proxy (optional).
Country string
// City is the city name of the proxy (optional).
City string
}
Proxy holds the parsed coordinates of a single proxy server.
func Parse ¶
Parse converts a raw proxy string into a Proxy value.
Accepted formats (in order of detection):
- protocol://user:pass@host:port — standard URL (already had scheme)
- user:pass@host:port — URL without scheme (http assumed)
- host:port — bare address (http, no auth)
- host:port:user:pass — colon-separated, port is numeric
- user:pass:host:port — colon-separated, port is last field
- protocol:host:port:user:pass — five colon-separated fields
type ProxyHealth ¶
type ProxyHealth struct {
// Alive indicates whether the proxy is reachable.
Alive bool
// Latency is the round-trip time to the proxy check endpoint.
Latency time.Duration
// SuccessRate is the fraction of requests that succeeded (0.0–1.0).
SuccessRate float64
// BlockRate is the fraction of requests that were blocked (0.0–1.0).
BlockRate float64
// BanCount is the total number of domain bans recorded for this proxy.
BanCount int
// CooldownUntil is the time after which the proxy may be used again.
CooldownUntil time.Time
// Score is an aggregate quality score from 0.0 (dead) to 1.0 (perfect).
Score float64
}
ProxyHealth records the current health state of a proxy.
type RotationStrategy ¶
type RotationStrategy int
RotationStrategy controls when the pool selects a new proxy.
const ( // PerRequest rotates the proxy on every outbound request. PerRequest RotationStrategy = iota // PerSession keeps the same proxy for the lifetime of a walker session. PerSession // PerDomain keeps the same proxy per target domain. PerDomain // OnBlock only rotates when a proxy is detected as blocked. OnBlock )
type StaticProvider ¶
type StaticProvider struct {
// contains filtered or unexported fields
}
StaticProvider serves a fixed list of proxy URLs.
Source Files
Directories