proxyinabox

package module
v0.9.33 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 2, 2026 License: MIT Imports: 6 Imported by: 3

README

Proxy-in-a-Box

Go Go Report Card

Automatic proxy pool for web scraping. Crawls proxies from YAML-defined sources, validates them, and provides HTTP/HTTPS proxy servers with automatic rotation and TLS fingerprint spoofing.

中文说明

Features

  • YAML-driven sources — All proxy sources defined as YAML configs with Lua scripting for complex logic
  • Headless browser scraping — Integrated Obscura with anti-detection enabled by default for complex JS-rendered pages
  • Auto-validation — Concurrent proxy verification with configurable worker pool
  • Smart rotation — Automatic proxy assignment based on domain and IP limits
  • TLS fingerprint spoofing — Uses uTLS to mimic Chrome browser fingerprints
  • MITM support — Built-in man-in-the-middle proxy for HTTPS traffic
  • SQLite storage — Lightweight embedded database, no external dependencies

Quick Start

# docker-compose.yml
services:
  proxy-in-a-box:
    image: ghcr.io/naiba/proxy-in-a-box
    restart: unless-stopped
    volumes:
      - ./data:/app/data
    ports:
      - "8080:8080"   # HTTP proxy
      - "8081:8081"   # HTTPS proxy
      - "8083:8083"   # Dashboard + API
From Source
go install github.com/naiba/proxyinabox/cmd/proxy-in-a-box@latest
mkdir -p data/sources
# Create data/pb.yaml and data/sources/*.yaml (see below)
proxy-in-a-box

Usage

Usage:
  proxy-in-a-box [flags]
  proxy-in-a-box [command]

Available Commands:
  test-source    Test a single proxy source YAML file (fetch + verify availability)

Flags:
  -c, --conf string   config file (default "./data/pb.yaml")
  -p, --ha string     http proxy server addr (default "0.0.0.0:8080")
  -s, --sa string     https proxy server addr (default "0.0.0.0:8081")
  -m, --ma string     management/dashboard addr (default "0.0.0.0:8083")
  -h, --help          help for proxy-in-a-box
Test a Source
proxy-in-a-box test-source data/sources/my-source.yaml [-w 20]

Fetches proxies from the specified source YAML file and verifies their availability. Use -w to set concurrent verification workers (default: 20).

Configure your application to use the proxy:

HTTP Proxy:  http://127.0.0.1:8080
HTTPS Proxy: https://127.0.0.1:8081

Management Dashboard & API:

GET /             — Web dashboard (pool, verification backlog, quarantined retries, source status)
GET /stat         — Pool statistics (plain text)
GET /get          — Get one available proxy
GET /api/stats    — Pool statistics (JSON: available/quarantined totals, request/maintenance stats, process runtime and crawler load)
GET /api/proxies  — Full proxy list (JSON)
GET /api/sources  — Source fetch statuses (JSON)

maintenance reports healthy proxies due for checking, quarantined proxies ready for retry or still waiting (including IP locks), source errors and pending first fetches, and periodic/source-candidate check outcomes since process start. Check totals count checks, not individual HTTP retries, and reset on restart.

Dashboard colors use a separate rolling 60-minute check count and 10-minute backlog trend sampled every 5 minutes. High routine failure is shown as expected churn while the queue is draining; it becomes critical only when no checks succeed or it coincides with a large sustained backlog. Source-candidate checks are shown separately. Quarantined endpoints waiting for retry are expected and stay neutral unless the waiting count grows substantially. Runtime cards expose CPU, RSS/heap, goroutines, file descriptors, worker activity, real in-memory queues, crawler attempts, and response bytes. Recent counts and trends reset on restart. Checks due turn yellow after ten minutes without a decline, and red when still growing above max(20, 20% of available proxies). Waiting retries turn yellow only after consecutive growth, at least a doubling, and max(10, 10% of available proxies); larger threefold growth can turn red. Source errors turn red when at least half of the sources are failing.

Configuration

data/pb.yaml:

debug: true

sys:
  name: MyProxy
  proxy_verify_worker: 20    # shared network-concurrency limit for candidate and routine checks (default 20, max 256)

# HTTPS MITM decryption (default: false)
# When enabled, the proxy decrypts HTTPS traffic using a self-signed CA — clients must disable TLS verification or trust the CA.
# When disabled (default), HTTPS CONNECT requests are tunneled as-is — clients use standard TLS verification.
enable_mitm: false

# Upstream proxy resilience (all optional; defaults shown)
upstream:
  max_attempts: 3              # distinct exits tried for GET/HEAD/OPTIONS/TRACE and HTTPS CONNECT
  connect_timeout: 5s          # TCP connection timeout to an upstream proxy
  handshake_timeout: 7s        # CONNECT/SOCKS/TLS handshake timeout
  response_header_timeout: 12s # response-header timeout
  request_timeout: 20s         # total timeout for one upstream HTTP attempt
  target_failure_ttl: 10m      # circuit-breaker TTL for one proxy/target pair

# Proxy health checks (all optional; defaults shown)
verification:
  interval: 2h                 # routine checks for available proxies
  deep_check_interval: 24h     # additional TLS interception probe
  retries: 2                   # attempts per health-check round
  response_body_limit: 16384   # maximum health-check response bytes
  max_attempts_per_second: 50  # process-wide validation rate limit (maximum 1000)
  attempt_burst: 20            # token-bucket burst, capped at the rate limit
  routine_reserved_workers: 4  # concurrency unavailable to source candidates

# Source downloads (all optional; direct by default so bad proxies cannot delay them)
source_fetch:
  proxy_fallback: false         # try one random proxy after direct access fails
  retries: 2                    # direct attempts (maximum 5)
  timeout: 20s                  # timeout per attempt
  response_body_limit: 4194304  # source response-body limit (default 4 MiB)
  max_candidates_per_fetch: 200 # initial per-source validation admission limit

# Headless browser for JS-rendered pages (optional)
# Requires Obscura v0.2.1+ with the stealth build feature — included in the Docker image
# Proxy-in-a-Box starts Obscura with --stealth by default.
obscura:
  bin: obscura                # binary path (leave empty to use PATH default)

Failed endpoints back off for 30 minutes, 2 hours, 6 hours, and then 24 hours. Repeated candidates returned by a source respect the same delay instead of consuming bandwidth on every source refresh.

Candidate endpoints are deduplicated before entering the in-memory queue. After a source has 100 completed candidate checks, its per-fetch admission limit automatically drops to 100, 50, or 25 when its yield over the latest 256 results is below 1%, 0.5%, or 0.25%. The rolling window keeps sampling low-yield sources and automatically restores their limit if quality recovers. Runtime statistics expose the last-second, 60-second average and 60-second peak attempt rates; the token bucket prevents immediately failing endpoints from turning a concurrency limit into an unbounded request rate.

Proxy Sources

Sources are YAML files in data/sources/. Three types supported:

text — Plain text IP:Port lists
name: example-text
type: text
url: "https://proxy-source.example/http.txt"
protocol: http
interval: 5m
json — JSON API with field paths
name: example-json
type: json
url: "https://proxy-source.example/api/proxies"
ip_field: "proxies.*.ip"
port_field: "proxies.*.port"
protocol_field: "proxies.*.protocol"
interval: 5m
script — Lua scripts for complex logic

Lua globals: fetch(url, headers?), sleep(ms), json_decode(str), json_encode(table), browser_fetch(url), browser_eval(expression)

name: example-script
type: script
interval: 10m
script: |
  local proxies = {}
  for page = 1, 5 do
    sleep(3000)
    local body = fetch("https://proxy-source.example/pages/" .. page)
    if body then
      local match = string.match(body, "fpsList = (.-);%s*\n")
      if match then
        local list = json_decode(match)
        if list then
          for _, item in ipairs(list) do
            proxies[#proxies+1] = {ip = item.ip, port = item.port, protocol = "http"}
          end
        end
      end
    end
  end
  return proxies
Browser-powered scraping (for JS-rendered pages)

Uses obscura config. browser_fetch(url) navigates the headless browser and returns rendered HTML. browser_eval(expression) executes JavaScript on the loaded page. Proxy-in-a-Box starts the stealth-enabled Obscura build with --stealth and aligns Obscura's navigation, script, fetch, and CDP deadlines with the 60-second client timeout.

name: example-browser
type: script
interval: 30m
script: |
  local proxies = {}
  local html = browser_fetch("https://proxy-source.example/rendered-list")
  if not html then return proxies end
  local raw = browser_eval([[(function(){
    var rows = document.querySelectorAll('div.grid.min-w-\\[600px\\]');
    var r = [];
    for (var i = 0; i < rows.length; i++) {
      var ch = rows[i].children;
      if (ch.length >= 3) {
        var ip = ch[0].textContent.trim();
        if (/^\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}$/.test(ip))
          r.push({ip: ip, port: ch[1].textContent.trim(), protocol: ch[2].textContent.trim().toLowerCase()});
      }
    }
    return JSON.stringify(r);
  })()]])
  if raw then
    local data = json_decode(raw)
    if data then
      for _, item in ipairs(data) do
        proxies[#proxies+1] = {ip = item.ip, port = item.port, protocol = item.protocol}
      end
    end
  end
  return proxies

Architecture

                    ┌─────────────────────────────────────────┐
                    │           Proxy-in-a-Box                │
                    ├─────────────────────────────────────────┤
 Your App ────────► │  HTTP Proxy :8080 / HTTPS Proxy :8081  │
                    ├─────────────────────────────────────────┤
                    │              Proxy Pool                 │
                    │   ┌─────┐ ┌─────┐ ┌─────┐ ┌─────┐      │
                    │   │ IP1 │ │ IP2 │ │ IP3 │ │ ... │      │
                    │   └─────┘ └─────┘ └─────┘ └─────┘      │
                    ├─────────────────────────────────────────┤
                    │  YAML Sources   │ Validators            │
                    │  text/json/lua  │ (concurrent workers)  │
                    ├─────────────────────────────────────────┤
                    │       Obscura (headless browser)         │
                    └─────────────────────────────────────────┘
                                     │
                                     ▼
                              ┌─────────────┐
                              │   SQLite    │
                              └─────────────┘

Benchmark

ab -v4 -n100 -c10 -X 127.0.0.1:8080 https://target.example/resource

Tech Stack

  • Language: Go 1.25
  • Database: SQLite (via glebarez/sqlite + GORM)
  • Scripting: gopher-lua (Lua 5.1 VM)
  • Browser: Obscura
  • TLS: uTLS for fingerprint spoofing
  • HTTP: Standard library + custom MITM proxy

License

MIT

Documentation

Index

Constants

This section is empty.

Variables

View Source
var DB *gorm.DB

DB instance

View Source
var DataDir string

Functions

func Init

func Init(configFilePath string)

Init init system

Types

type BlockedIP added in v0.8.0

type BlockedIP struct {
	IP                  string    `gorm:"type:varchar(15);primaryKey"`
	ConsecutiveFailures int       `gorm:"default:0"`
	LockedUntil         time.Time `gorm:"index"`
}

type Cache

type Cache interface {
	// --- 代理池读取 ---
	RandomProxy() (string, bool)
	GetProxy() (string, bool)
	ProxyLength() int
	PickProxy(req *http.Request) (string, error)
	HasProxy(p string) bool
	// IsProxyValidationDue prevents source refreshes from bypassing the
	// persisted retry delay of a quarantined endpoint.
	IsProxyValidationDue(proxyURI string) bool
	GetAllProxies() []Proxy

	// UpsertProxy 新代理首次验证成功时调用。
	// 原子完成:清除 blocked_ips 旧记录 → 写入/更新 DB → 替换内存 entry。
	UpsertProxy(p Proxy) error

	// MarkVerifySuccess 已有代理定期验证成功时调用。
	// 原子完成:清除 blocked_ips → 更新 DB delay/last_verify → 同步内存。
	MarkVerifySuccess(p Proxy, delay int64, verifyTime time.Time, deepVerified bool)

	// MarkVerifyFailed 已有代理定期验证失败但未达锁定阈值时调用。
	// 原子完成:在 DB 中持久化不可用状态 → 从内存移除。
	// 返回该端点更新后的连续失败次数。
	MarkVerifyFailed(p Proxy) int

	// MarkProxyUnavailable 处理代理自身明确返回 403/407 的情况。
	// 按 URI 隔离单个端点,等待定期健康检查恢复,不累计为整 IP 封禁。
	MarkProxyUnavailable(proxyURI string)

	// MarkProxyTargetFailure 对某个代理与目标的组合做短期熔断。它不会把
	// 仍可服务其他目标的代理从全局池中删除。
	MarkProxyTargetFailure(proxyURI string, target string)

	// RecordFailure 代理请求或验证失败时调用,累计失败次数。
	// 达到阈值时锁定 IP,并将该 IP 的代理持久化为不可用;记录不会删除,
	// 锁定到期后仍可由健康检查或数据源重新验证恢复。
	// 返回 true 表示触发了锁定。
	RecordFailure(ip string) bool

	// IsIPLocked 检查 IP 是否在锁定期内。
	IsIPLocked(ip string) bool

	// LoadLockedIPs 启动时从 DB 加载锁定状态到内存。
	LoadLockedIPs()

	// CleanupStaleProxies 删除 last_verify 超过阈值的陈旧代理,同步清理内存。
	CleanupStaleProxies(threshold time.Duration)
}
var CI Cache

CI cache instance

type Conf

type Conf struct {
	Debug bool
	Redis struct {
		Host string
		Port string
		Pass string
		Db   int
	}
	Sys struct {
		Name              string
		ProxyVerifyWorker int `mapstructure:"proxy_verify_worker"`
	}
	Obscura struct {
		// obscura 二进制路径,留空则使用 PATH 中的默认命令
		Bin string `mapstructure:"bin"`
	} `mapstructure:"obscura"`
	// Upstream controls how requests are attempted through proxies selected from
	// the pool. Zero values use the safe defaults defined by the MITM package.
	Upstream struct {
		MaxAttempts           int           `mapstructure:"max_attempts"`
		ConnectTimeout        time.Duration `mapstructure:"connect_timeout"`
		HandshakeTimeout      time.Duration `mapstructure:"handshake_timeout"`
		ResponseHeaderTimeout time.Duration `mapstructure:"response_header_timeout"`
		RequestTimeout        time.Duration `mapstructure:"request_timeout"`
		TargetFailureTTL      time.Duration `mapstructure:"target_failure_ttl"`
	} `mapstructure:"upstream"`
	// Verification controls recurring health checks. Zero values use the
	// conservative defaults defined by the crawler and service packages.
	Verification struct {
		Interval               time.Duration `mapstructure:"interval"`
		DeepCheckInterval      time.Duration `mapstructure:"deep_check_interval"`
		Retries                int           `mapstructure:"retries"`
		ResponseBodyLimit      int64         `mapstructure:"response_body_limit"`
		MaxAttemptsPerSecond   int           `mapstructure:"max_attempts_per_second"`
		AttemptBurst           int           `mapstructure:"attempt_burst"`
		RoutineReservedWorkers int           `mapstructure:"routine_reserved_workers"`
	} `mapstructure:"verification"`
	// SourceFetch controls downloads of proxy lists. Source documents are
	// fetched directly by default: routing them through an untrusted free proxy
	// makes a healthy source wait behind a failing proxy and can corrupt the
	// returned list. Proxy fallback remains opt-in for installations that need
	// it for reachability.
	SourceFetch struct {
		ProxyFallback         bool          `mapstructure:"proxy_fallback"`
		Retries               int           `mapstructure:"retries"`
		Timeout               time.Duration `mapstructure:"timeout"`
		ResponseBodyLimit     int64         `mapstructure:"response_body_limit"`
		MaxCandidatesPerFetch int           `mapstructure:"max_candidates_per_fetch"`
	} `mapstructure:"source_fetch"`
	// EnableMITM 是否启用 HTTPS 中间人解密,默认 false(关闭时走 TCP 隧道透传,客户端无需关闭 TLS 验证)
	EnableMITM bool `mapstructure:"enable_mitm"`
}

Conf config struct

var Config Conf

Config system config

type Proxy

type Proxy struct {
	gorm.Model
	// BUG-FIX: GORM v2 不识别 unique_index(v1 语法),导致 IP 唯一约束从未生效,
	// 同 IP 不同端口/协议的代理在 DB 中可以重复插入。改用 uniqueIndex 复合索引,
	// 以 (IP, Port, Protocol) 为唯一粒度,允许同 IP 不同端口合法共存。
	IP       string `gorm:"type:varchar(15);uniqueIndex:idx_proxy_endpoint"`
	Port     string `gorm:"type:varchar(5);uniqueIndex:idx_proxy_endpoint"`
	Country  string `gorm:"type:varchar(15)"`
	Provence string `gorm:"type:varchar(15)"`
	Source   string
	Protocol string `gorm:"uniqueIndex:idx_proxy_endpoint"`
	Delay    int64
	// Available is persisted so a proxy removed after a failed health check is
	// not silently restored to the live pool by a process restart.
	Available           bool      `gorm:"not null;default:true;index:idx_proxy_reverify,priority:1"`
	LastVerify          time.Time `gorm:"index:idx_proxy_reverify,priority:2"`
	LastDeepVerify      time.Time
	ConsecutiveFailures int       `gorm:"not null;default:0"`
	NextVerifyAt        time.Time `gorm:"index"`
}

Proxy proxy model

func (*Proxy) String

func (p *Proxy) String() string

func (*Proxy) URI

func (p *Proxy) URI() string

type ProxyService

type ProxyService interface {
	GetUnVerified() ([]Proxy, error)
	IsUnVerified(Proxy) (bool, error)
}

ProxyService proxy service

Directories

Path Synopsis
cmd
proxy-in-a-box command

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL